Building AI Tools: 4 Wrong Assumptions We Had

9 months of building an AI developer tool. Four assumptions we had on day 1 that turned out to be completely wrong. And what we had to rethink because of it.

The setup

When we started WalnutAI, we had a vision of what developers needed from AI tools. We were confident. We had smart people, we’d read the research, we’d talked to a few teams. We thought we knew.

We were wrong about almost everything. Not wrong in a way that made us pivot the whole company, but wrong enough that every wrong assumption cost us 4–6 weeks of development, user confusion, or both. Looking back at 9 months of building, I can identify four concrete assumptions that shaped early decisions and then had to be completely rethought.

Here they are, in the order we learned them:

Wrong assumption #1: Developers want AI to do everything autonomously

What we thought: developers hire AI to save time. Time is saved by having the AI handle the whole problem end-to-end. Write requirements, generate code, generate tests, all automatically.

What happened: we built a mode where you could point WalnutAI at a user story and it would generate tests automatically, fully formed. No draft. No review. Just: “here are your 50 test cases, they’re done.”

The reaction from teams was ice cold. They didn’t use it. Not because it didn’t work technically. It worked fine. But they said: “I don’t trust what I didn’t write. I need to understand what’s happening and make choices.”

We pivoted. Now WalnutAI generates a draft, and the developer reviews it. They delete the tests they don’t want, edit the ones that are close, and add new ones. It’s a 10-minute editing job instead of a 2-hour writing job. Much higher adoption. The draft is the value, not the automation.

Lesson: Autonomy is not what developers want. Control is. Developers want to know why the AI did what it did. They want to disagree with it. The AI’s job is to draft. The developer’s job is to judge.

Wrong assumption #2: QA engineers are the primary buyer

What we thought: test generation is obviously a QA problem. QA engineers write tests. More tests need writing. Sell to QA.

What happened: our first few customers were QA leads who loved the product and used it every day. But they didn’t buy licenses. Their CTOs bought licenses. And the CTOs were buying it for a different reason than the QA leads were using it.

The QA lead cared about “I have fewer tests to write.” The CTO cared about “my teams ship faster and with fewer defects.” Different metrics. Different value. Different buyer.

We started talking to CTOs first. Different pitch entirely. Same product, different angle. Turns out we needed to sell to the person who owns the outcome (ship speed, release quality) not the person who does the work (write tests).

We didn’t fire QA leads. We just stopped positioning WalnutAI as “a QA tool” and started positioning it as “a tool that helps engineering leaders prove quality to the business.” Marketing changed. Sales changed. Customer conversations changed.

Lesson: the person who uses a tool is not always the person who buys it. Talk to the buyer about the outcome they own.

Wrong assumption #3: Test generation and test maintenance are the same use case

What we thought: testing is testing. If we solve “generating test cases,” we solve the test problem.

What happened: we got feedback from teams saying “your tool generates 100 tests great, but I have 2,000 legacy tests that break every sprint and need manual fixes. Can you help with that?”

We said “sure, same tool.” We were wrong. Generating tests from a new requirement is a totally different problem than maintaining tests against a constantly-changing codebase. The first is about understanding requirements. The second is about resilience and maintenance workflow.

If we’d tried to solve both in the same product, we’d have solved neither well. We made a hard call: we’re solving generation. Maintenance is a different product. Different team, different roadmap.

That hard no changed everything. It made our product cleaner, our roadmap clearer, our messaging simpler. And it made us better at what we actually do.

Lesson: some use cases look the same but are actually different products. Resist bundling them.

Wrong assumption #4: IDE-native beats web

What we thought: developers live in their editor. Sell them a VS Code extension. Web interfaces are for non-developers.

What happened: we built a VS Code extension first. It was technically elegant. The team was proud of it. Almost nobody used it.

Then we built the web app as a “secondary” thing. Teams started using the web app as their primary interface. The VS Code extension is useful for certain workflows now, but it’s not the primary.

We spent time investigating why. The friction was surprising. For VS Code, the engineer has to install the extension (requires admin), authenticate (adds a new login to remember), and learn a new UI (lives in the editor but works like a web app). For the web app, they click a link.

That’s not about being in the IDE. That’s about friction. Turns out adoption is a friction problem more than anything else. The fastest way to a tool is to reduce friction.

We didn’t abandon the VS Code extension. But we stopped treating it as the primary and the web as secondary. Friction is the real metric.

Lesson: IDE-native is elegant, but friction beats elegance every time. Optimize for the path of least resistance.

What changed as a result

Each of these lessons reshaped the product:

• We built a review/edit interface instead of full automation. It’s slower to use but faster to adopt.

• Our sales motion changed from “call QA leads” to “work with CTOs.” Messaging changed. Customer profiles changed.

• We narrowed scope aggressively. We’re the test generation and traceability company, not the everything company.

• We stopped prioritizing IDE-native and started prioritizing “lowest friction path to first success.” Sometimes that’s the web. Sometimes it’s a Slack bot. IDE integration is a nice-to-have for the 10% who care deeply about it.

What this means for building developer tools

If you’re building AI tools for developers, here’s what I’d tell you based on 9 months of being wrong:

• Developers don’t want autonomy. They want drafts and control. The AI’s job is to propose. The developer’s job is to decide.

• Talk to buyers, not users. The engineer who uses your tool is not the engineer who buys it.

• Scope aggressively. You’re not solving “the problem.” You’re solving one specific problem really well.

• Friction is real. A web app with lower friction beats an elegant IDE integration. Measure adoption, not architecture.

What we still don’t know

Nine months in and we’re still learning. The things we’re genuinely uncertain about now:

• Whether test generation is a temporary tool or a permanent layer. As AI gets better, maybe it becomes “AI writes the tests and you review” instead of “you use AI to draft.” Time will tell.

• Whether business users (PMs, product teams) will ever use these tools, or whether it’s always going to be filtered through engineers. Our current roadmap assumes “engineers are the interface.” But maybe that’s wrong too.

• How much of adoption is actually product and how much is just sales credibility. Are we winning because WalnutAI is the best tool, or because teams trust us? Probably both, but the ratio matters and we still don’t know.

Looking forward

We’re going to keep building with the assumption that we’re wrong about something. Probably several things. The teams that catch those wrong assumptions early and fix them fast are the ones that win.

We’re applying these lessons every day. See what we’ve built. Try WalnutAI.