Understanding Synchronous vs Asynchronous AI
One of the easiest ways to understand the difference between these approaches is to think about parenting:
Synchronous agents require your active participation. You ask, it responds, you refine, it adjusts. Like helping a young child with chores - you must guide them through each step: "pick up your shoes," "put them in the closet," "now close the door." Progress only happens while you're standing there.
Asynchronous agents offer a different experience. You set the task, agree on the plan, then step away. The agent works independently, checking in only when needed. It's like when a child gets older - you can say, "Clean your room before I get back" and trust the work will be done without constant supervision.
This shift from step-by-step guidance to delegated autonomy is what makes asynchronous AI so exciting. It frees you from being the bottleneck.
The Surprising Results
The central question: could these agents truly deliver on the promise of asynchronous autonomy? The short answer is: no.
In testing, Genspark and Abacus.AI consistently outperformed Jules and Devin, not just by finishing more tasks but by delivering more polished, production-ready products.
While all four agents show potential, none are yet at a level where you could hand them a task and walk away. Instead, they require supervision similar to a synchronous chatbot, defeating the core promise of asynchronous work.
The autonomy that makes these tools exciting was almost more frustrating - asynchronous agents tended to ask more complex questions requiring more context to answer. If you're going to be asked questions anyway, they might as well be quick and easy to answer.
Benchmark Phase 2: Feature Build (Prediction App)
After the struggles of the messy repo challenge, all four agents were given a clean-slate challenge: build a full-stack "Prediction App" from scratch. The prompt included detailed specs for a SQLite database, FastAPI API with Brier scoring, a Next.js UI, tests, and documentation.
While all four agents produced a working app, the difference in quality was massive:
Genspark and AbacusAI delivered substantially more polished and user-friendly experiences. Their designs were modern and intuitive, with clear flows and better UX. AbacusAI's ability to build and deploy the entire application within its own ecosystem was particularly impressive.
Jules and Devin both produced working apps, but the final products were rough and sparse. Jules' UI looked like "a basic form designed in Infopath 2003," while Devin's was a minimalist wireframe.
Comparative Summary
The results from this benchmark are clear. While the field of autonomous AI agents is still in its infancy, there is already a significant performance gap between the leaders.
Genspark and AbacusAI are materially ahead of Jules and Devin in both their ability to handle complex repo-scale tasks and in their final product quality - a surprising result for the author.
They are capable of executing more robust, end-to-end changes with a higher degree of autonomy, though still not at the level of true "set it and forget it" development.