Orca 2

Orca 2 is a pair of small language models from Microsoft Research, released November 21, 2023 in 7B and 13B sizes built on Llama 2. Where the original Orca imitated GPT-4's reasoning traces verbatim, Orca 2 introduced 'explanation tuning': teaching the small model to select the right reasoning strategy per task — step-by-step for math, direct answers for simple facts, recall-then-generate for knowledge questions — guided by cautious system prompts.

The results were remarkable for the era: Orca-2-13B matched or beat Llama-2-Chat-70B on zero-shot reasoning benchmarks including BigBench-Hard, AGIEval, and GSM8K, showing that better training signals could let small models punch far above their weight.

An honest caveat: both models were released under the Microsoft Research License, restricted to non-commercial research use only. The weights are downloadable and runnable locally, but commercial deployment is not permitted.