A command-line interface is provided to convert TensorFlow checkpoints in PyTorch models. This PyTorch implementation of Transformer-XL is an adaptation of the original PyTorch implementation which has been slightly modified to match the performances of the TensorFlow implementation and allow to re-use the pretrained weights. Google/CMU's Transformer-XL was released together with the paper Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. This PyTorch implementation of OpenAI GPT is an adaptation of the PyTorch implementation by HuggingFace and is provided with OpenAI's pre-trained model and a command-line interface that was used to convert the pre-trained NumPy checkpoint in PyTorch. OpenAI GPT was released together with the paper Improving Language Understanding by Generative Pre-Training by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. This PyTorch implementation of BERT is provided with Google's pre-trained models, examples, notebooks and a command-line interface to load any pre-trained TensorFlow checkpoint for BERT is also provided. Here are some information on these models:īERT was released together with the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. You can find more details in the Examples section below. ~91 F1 on SQuAD for BERT, ~88 F1 on RocStories for OpenAI GPT and ~18.3 perplexity on WikiText 103 for the Transformer-XL). These implementations have been tested on several datasets (see the examples) and should match the performances of the associated TensorFlow implementations (e.g. This repository contains op-for-op PyTorch reimplementations, pre-trained models and fine-tuning examples for: PyTorch Pretrained BERT: The Big & Extending Repository of pretrained Transformers
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |