I recently sat in on a call with a major data analytics engine vendor. This is a company that builds a very popular product targeting is the enterprise. I’m going to refer to them as $VENDOR going forward.
As part of this, $VENDOR demoed some of their AI tooling with the aim of selling us on it as a means of converting our legacy code base.I don’t use much in the way of LLMs at work. This is because our parent company restricts access. I do use them extensively in a range of side projects. Their tool amounts to a chatbot built into a notebook, and the demo they showed us consisted of them taking said notebook and walking the chatbot through the conversion of a small piece of code. This is sufficient for generating small scripts, but insufficient for building against a large codebase. An LLM chatbot embedded in a notebook is a toy: It is can’t ingest the larger context of your project and your codebase, and it’s incapable of stringently checking the code it has written. At best, it doesn’t offer much in the way of productivity improvements, and charitably, in the context of what $VENDOR is offering, it is meant for analysts who can’t code running interactive queries rather than an organization maintaining and building on a large codebase, or building new products. If you’re going to use a forklift, the value is in giving it more information than a human can hold.
A forklift is a tool that a single human uses to move objects many times her own body weight, provided that the object meets certain criteria, and that the environment is suitable for operating a forklift.AI agents are information forklifts.
— Tilmon Edwards (Dirigo) (@tilmonedwards.com) December 30, 2025 at 4:54 PM
$VENDOR is a well-known company. You take a demo from them, and they promise a simple point and click solution to all of your problems. It’s disappointing, but it raises the question of “why are they selling snake oil?” I think I know the answer.
To illustrate this, I want to explain what holding LLMs right looks like—for me. This is a workflow that I use for development on my many side projects, and that has worked well for others. It’s based on the ed3d-plugins skills for Claude Code. This workflow should not be read as prescriptive—it’s a workflow that encodes a lot of what I think is important to get the tool to build the things you’re asking it to build, and build them right.I should emphasize that right now I don’t think anybody knows exactly what they’re doing. These things are very powerful technologies, and we are at the very beginning of understanding how they can be wielded effectively. Note: this is longish, but bear with me.
Agentic Development Workflow
0. What are you trying to build?
If you don’t know this, you have a larger problem than your manner of doing development, and I’d urge you to stop doing whatever you’re doing and figure that out first. Like engineers, the LLM can do many things—they can build large scale, elegant code and systems for you, but only if you know what you want to build and can articulate that. Unlike engineers, the LLM can’t save you from yourself.
1. Design
The first thing I do is give the robot a Product Requirements Document, a JIRA Epic, or at minimum, several paragraphs of textI recommend installing something like handy, spokenly, or hex, and dictating things to the robot. Local speech-to-text models work well now. This way you can go for a walk and talk to the robot. about what it is that I’m looking to build. The goal here is to give the robot enough information about what it is I want to build that it can generate testable acceptance criteria. If you have acceptance criteria to begin with, that’s great, supply those.
With this, the robot will gather context. It will investigate the existing code base, look for prior art, look for APIs and libraries that you’re already using, it will ask you questions to clarify the requirements. It might present some alternative approaches to doing things along the way.
2. Planning
Depending on what you’re doing, planning may look a bit different. For example, if you’re burning through an epic’s worth of issues in one pass, you’ll probably want the robot to set up multiple work trees. What’s important here is that the robot builds out a written plan of how to proceed through each phase of development, that it can then hand off to a fresh context for implementation. Plans are detailed and often include code examples, testing requirements, and for each phase, list the acceptance criteria. The good news is that the robot can generate all of them, and it can assign reviewing agents to review the plan and discover and then correct flaws.
You must then review the plan prior to handing it off for implementation. You must review the plan. This is your last opportunity before the robot writes code to make changes to the plan, and it’s easier to do now than to wait for a PR.
3. Execution
This is the part where the robot’s going to act autonomously, in a loop. It’s going to take your plan, and work through it phase by phase. It will build the code for a phase, and then run tests against that code. If those tests fail, it will go back and fix the cause until the tests pass. Then it will assign a subagent to do a code review.Why? While the model has infinite attention, the model is also, for lack of a better term, lazy, and will take shortcuts. We use subagents and fresh contexts to prevent the model from rationalizing these away. If that subagent identifies concerns, it will call another subagent with a fresh context to fix those issues. It will iterate until no new issues are identified. Once a given phase is complete, it will move on to the next phase and repeat this loop.
4. Final Review
Wrapping up development usually involves committing the code and opening a pull request from that branch to my main branch, and then calling a pair of agents to review the entire PR.At this point, you should not be handing a PR off to another human to review without having an agent review it, even if it is code you write artisanally. Otherwise, you’re wasting their time. I’m a big fan of this reviewing skill by hailey. In the case of this skill, one is a balanced reviewer that’s looking for correctness, it’s checking security, it’s checking tests. And the other is an adversarial reviewer designed to catch what the first reviewer might miss or rationalize away. It assumes things are broken, and then tries to break those things.
If these return issues, then another agent should be assigned to go fix the discovered issues, then call the reviewers again. The loop should repeat until all blocking or high impact issues are resolved and the PR updated. It is only at this point that I would consider the code ready for human review.
Product vs Process
If you’ve stuck with me this long, my condolences, but this serves to illustrate a point: What I’ve described is a process, not a product. There are demonstrably correct and incorrect ways to hold the tool. There are in fact multiple correct ways to hold the tool, but they all require substantial work to implement effectively. You need to understand your engineering processes, what it is you’re trying to build, and your customer before you can figure out your process. The fundamental pitch of $VENDOR is “what if someone can do agentic development without needing to do work?”
Process is individualized (to the organization at least), whereas large enterprise scale product offerings are generally sold as a generic solution that will magically solve all the issues with complex, highly individual and idiosyncratic organizational workflows. It won’t, but it will make your Salesforce consultant a lot of money.$VENDOR is not Salesforce. Enterprise software vendors know this and they know that the folks signing contracts are not generally positioned to understand organizational processes. I’m increasingly convinced that the corporate backlash to AI is not because it doesn’t work as a technology, but because executives were sold a bill of goods, rather than an actual solution.
What does this all mean?
I think this is a competitive fork in the road. The organizations doing this right are a slim minority, and no study that I’m aware of segments outcomes by adopter capability. However, organizations with strong testing and feedback loops will convert the gains. Those with weak foundations will convert them into delivery instability. Holding it correctly means that organizations will experience Jevons paradox and both their productivity and the quality of their product will improve.I’m aware of the METR study from 2025. I suspect its conclusions are out of date at this point and they’re having to update the methodology to handle the current state of things: We are Changing our Developer Productivity Experiment Design Holding it wrong is probably worse than not using the tool at all.
A friend raised the question “How do you convince the relevant stakeholders that a well-known legitimate company is trying to sell you snake oil?” I don’t have a great answer, but I have a few ideas.
Make the vendor prove their solution works: Ask them to pull in your entire codebase and have their solution attempt to solve a known problem. Get the vendor to prove that the agent workflow can ingest the relevant context, use provided tests, and deploy subagents to engage in adversarial review. It’s easy to demo a toy script in a notebook, hard to ingest a large repo and make changes without cascading effects. Ask them how the solution integrates with your version control and issue tracker.For what it is worth, if the agent harness can’t interact with the command line, it’s not going to be useful for a developer workflow. How does it integrate with your product team? Quality control workflows?
I do think it is imperative to spike adoption of subpar tools early. An organization that buys the swampland in Florida and discovers that cannot, in fact, build on it, is likely forgo future investment, to their long-term detriment.