DeepSeek-Coder trained open code models from 1.3B to 33B parameters on a two-trillion-token project-level corpus, combining code-heavy pretraining, long context and infilling during the rapid open-model race of 2023–2024.
StarCoder paired a strong open code model with an unusually visible discussion of training data, licensing, opt-out mechanisms and model governance through the BigCode collaboration.
Meta's 2023 Code Llama release showed how a general foundation model could be further trained into code-specialized, Python-specialized and instruction-following variants with infilling support.