Research Notes

word2vec and GloVe Embeddings

By Satwik · January 10, 2026

word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) taught the field that words could be represented as dense vectors carrying real semantic structure. word2vec learned embeddings by predicting context, either skip-gram or CBOW, while GloVe fit vectors to global co-occurrence counts. Both produced the now-famous property that vector arithmetic tracked analogies, with king minus man plus woman landing near queen.

Why it mattered

These embeddings were the first widely used transfer learning in NLP. You could train vectors once on a large corpus and drop them into many downstream models, giving them a head start on meaning. For years this was the default first layer of serious NLP systems.

Their key limitation, visible in hindsight, is that each word gets exactly one vector regardless of context, so "bank" collapses river and finance into one point. That flaw is precisely what ELMo and BERT would later dissolve.

Reading these papers today, the security lesson is sharp and early: the vectors absorbed the biases of their training corpus, encoding gender and racial stereotypes in measurable, exploitable ways. Bolukbasi et al. (2016) demonstrated this concretely with occupation analogies. Embeddings made plain that a model's representations are a compressed image of its data, prejudices included. Every debate we now have about training-data provenance and inherited bias was already legible in these small, elegant vector spaces.