How OpenAI tests new models on complex tasks

Head of Codex Andrew Ambrosino spoke about a unique method of testing new neural networks within the company. When a fresh model is released, it is assigned a large-scale practical task - to completely rewrite the desktop application Codex in the Rust language and the GPUI framework.

The goal of the experiment is not to create a demonstration prototype, but to develop a full-fledged native product that can stably function on various platforms. The main difficulty lies in the high speed of change of the product itself by the team, due to which the AI has to work in conditions of a constantly changing code base.

In the future, OpenAI plans to achieve that one sufficiently powerful model could autonomously maintain several versions at once: for Mac on AppKit, for Windows on WinUI 3, as well as a separate version on Rust.

Although the developers admit that the technology has not yet reached this level, the current results show a significant approach to the set goal.