PLaMo-13B

PLaMo-13B (Preferred Language Model) was released by Preferred Networks on September 28, 2023. The 13-billion-parameter causal model was pretrained from scratch on 1.5 trillion tokens of English and Japanese open data (1.32T English, 0.18T Japanese) using 480 NVIDIA A100 GPUs on Japan's ABCI infrastructure over nearly a month. It demonstrated top-tier Japanese-plus-English benchmark performance among similar-size open models at release.

Unusually for the era, it shipped under Apache 2.0 permitting commercial use, trained only on publicly available datasets. PFN also released PLaMo-13B-Instruct. The parallel-layer architecture differs slightly from LLaMA v2, which required custom llama.cpp support.