As AI-powered software engineering tools continue to flood the market, knowing which model to trust with your codebase has become one of the most consequential decisions a development team can make. Graham Neubig, Co-Founder of OpenHands, is trying to bring some clarity to that question. OpenHands built the OpenHands Index, a benchmark designed to measure LLM performance across the full breadth of real-world software engineering work — from bug fixes and frontend iteration to test generation and long-horizon app creation — rather than the narrow slices that traditional benchmarks like SWE-Bench have historically focused on.
In this exclusive VMblog Q&A, Neubig cuts through the noise on some of the industry’s most hotly debated topics: open source versus closed flagship models, how teams should evaluate and select LLMs for their specific workloads, and whether organizations are mature enough to run multiple models in parallel. His take is refreshingly pragmatic — the best model isn’t necessarily the one topping the leaderboards, but the one that delivers reliable performance on your actual workflows at an acceptable price and latency. For teams feeling overwhelmed by the pace of model releases, Neubig’s advice is straightforward: start simple, evaluate on your own tasks, and specialize only when the data tells you to.
++
VMblog: What is the OpenHands Index for those who aren’t familiar?
Graham Neubig: The OpenHands Index is a benchmark for AI models used in software engineering agents. It measures how well models perform across a range of real coding tasks – not just fixing bugs, but also building new apps, working on frontends, generating tests, and gathering technical information.
VMblog: Why did your company build this index and how is it different from other benchmarking solutions?
Neubig: We built it because most benchmarks (like the famous SWE-Bench benchmark) look at one narrow slice of coding, such as fixing GitHub issues on Python repositories. On the other hand, real software engineering is much broader. Teams use AI agents for issue resolution, long-horizon app creation, frontend iteration, test generation, and research – and we wanted a benchmark that reflects that reality.
VMblog: How do you feel about open source vs the closed flagship LLMs? Is one approach superior to the other?
Neubig: We don’t think there’s a single winner in principle – it depends on what a team values. Right now, the closed flagship models tend to lead on overall performance and speed in demanding software engineering tasks, but open models are improving quickly and offer advantages in control, self-hosting, flexibility, and cost.
So we see this less as an ideological debate and more as an engineering tradeoff. If you want the strongest out-of-the-box performance, closed models often lead today (although open models are also very good). If you want portability, transparency, or infrastructure control, open models can be very attractive. We see plenty of teams on both sides.
VMblog: With so many options on the market how do you recommend teams approach choosing which LLM is best for them?
Neubig: The best starting point is to choose based on workload, not hype. A model that is excellent at long-horizon implementation may not be the best choice for fast bug fixes, and a model that tops a single benchmark may not deliver the best mix of quality, speed, and cost for your actual use case.
We’d recommend teams first look at publicly available resources like the OpenHands Index to get a first idea, narrow down to a few candidates, and then evaluate models on their own representative tasks. This evaluation can be as simple as a “vibe check” or as complex as A/B testing In practice, the best model is usually the one that performs reliably on your workflow at an acceptable price and latency.
VMblog: How difficult is it to swap out an LLM if a team changes their mind or a better model launches?
Neubig: If your system is model-agnostic, switching models can be relatively easy at the infrastructure level. That’s one of the main benefits of using an agent platform that isn’t tightly coupled to a single model family such as OpenHands.
That said, it’s never completely frictionless. Different models behave differently in tool use, context management, latency, and cost, so swapping models should still be followed by evaluation and tuning. Done right, though, it should feel like optimization – not rebuilding your stack from scratch.
VMblog: Are most teams sophisticated enough to use different LLMs and coding agents for different tasks to optimize for performance and cost? How would you compare the simple approach of having a single LLM across everything versus picking and choosing based on the needs of the specific task?
Neubig: Most teams should start simple. A single strong default model is easier to manage, easier to budget, and easier to govern – and for many organizations, that gets most of the value with far less complexity.
But as usage grows, the case for specialization gets stronger. Some models are better for long-running implementation, others are faster or cheaper for routine tasks, and others may be especially good at testing or research. Over time, the most sophisticated teams will likely route work to different models, but the practical path is to start with one good default and layer in specialization when the data supports it.
##
