one year on
Google posts BERT paper on arXiv, claims state-of-the-art across 11 NLP tasks
Bidirectional encoder representations from Transformers, pretrained on unlabeled text and fine-tuned with a single output layer, achieve state-of-the-art results on 11 NLP tasks.
Google today posted a paper on arXiv introducing BERT — Bidirectional Encoder Representations from Transformers — a language representation model that achieves state-of-the-art results on 11 natural language processing tasks. On the GLUE benchmark, BERT scores 80.5%, a 7.7-point absolute improvement. On SQuAD 1.1 question answering, it reaches 93.2 F1, surpassing both the previous best (91.6%) and human performance (91.2%). The model is pretrained on unlabeled text from Wikipedia using a masked language model objective, then fine-tuned with a single additional output layer for specific tasks.
BERT is deeply bidirectional, conditioning on both left and right context in all layers, unlike OpenAI’s GPT (unidirectional) or ELMo (shallowly bidirectional). The key innovation is a masking strategy that prevents the target word from ‘seeing itself’ in the deeper layers. Researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova also introduce a next-sentence prediction task to model relationships between sentences. The source code and pretrained models were open-sourced on TensorFlow, with fine-tuning possible in about 30 minutes on a Cloud TPU.
The record
One year later — open only if you can handle spoilers
Within two years, BERT and its derivatives would become the default foundation for virtually every major NLP system, from search engines to chatbots. The paper has accumulated tens of thousands of citations and sparked a wave of pretrained Transformer models, fundamentally shifting how the field approaches language understanding.
The Weekly Replay · free by email
This week, one year ago — every Sunday.
One email each Sunday: the week's replayed AI news, with the one-year-later annotations included. Written like it's breaking — dated like it isn't.
Free · double opt-in · unsubscribe anytime · privacy