AI’s next coding test: keep the old system working

OpenAI published new coding evaluations on October 19. MultiLangBench asks models to build a SQLite-compatible database engine across 20 languages. A second evaluation checks whether migrated COBOL programs preserve their observable outputs.

OpenAI reports an average 87.4% test pass rate for GPT-6 Astra on MultiLangBench, compared with 26.8% for GPT-5.6 Sol. It also reports 86.6% accuracy for GPT-6.1 Sol at low reasoning effort on its COBOL migration test. These are OpenAI’s results on defined evaluations, not Roseram tests of customer systems.

A rewrite can compile and still change what a business does. Our view: a useful migration starts by recording the old program’s behaviour. That record becomes a test the new version must pass, including awkward inputs that a clean demonstration might miss.

Choose a small, isolated component before attempting a whole system. Compare outputs, check dependencies, and have someone who knows the business rules review the change. A benchmark can help choose a tool to investigate. It cannot tell you whether your own migration is ready to release.