When AI Should Write Reviews

From the beginning of building Windmill, one of our most important design decisions has been when AI should be used. And just as importantly, when it shouldn’t.

Take two extreme examples:

The first is what the current performance review process looks like, which is entirely manual. You’re met with an empty form with no context, and it’s entirely up to the manager to read through every Slack channel and note and GitHub PR and Figma comment just to try to figure out what’s going on. Or worse, they skip that step and just try to do it based on memory—which, as everyone knows, is littered with cognitive biases. Most obviously recency bias. I often told people, just focus on the few months before the performance review, that’s all that matters. Unfortunately, that’s not just advice, it’s also how most people have learned to game the system.

The other extreme is probably even worse. Managers go to ChatGPT and just say “write a performance review.” Maybe they give a few bullets. The output at first glance looks really good, like they did their job, but the substance is sloppy. It’s not based on truth, not based on actual performance. And it can lead to bad outcomes beyond the review process, for employee development and for company-wide culture.

Both extremes fail for the same reason, they optimize for convenience over quality. So how do we find the middle ground? This is at the core of how we think about AI: how can we use it to not just make the process faster, but actually make it better?

A Deliberate Multi-Stage Process

At Windmill, we ended up with a deliberate multi-stage process to strike the right balance.

Step 1: The Review Step

Before a manager writes a review, they need to understand what’s going on. Rather than relying on human memory to do so, we give the manager real data to work with.

We precompile a report with an unbiased analysis of the individual over the time period, say the last six months. What were their key achievements? What feedback did they receive from other employees—whether explicitly via Windmill or casually through Slack, meeting notes, Figma comments, GitHub PRs? Where were they strong? Where could they improve? What came up in one-on-ones?

We also let managers save private notes throughout the period, and we surface those to jog their memory so they don’t forget anything.

Step 2: Observations

This is where the real magic happens. We want managers interpreting and synthesizing the information, but we don’t want them starting from scratch.

So we pre-compile a list of potential observations. All based on the ground truth of what’s going on. Say you have an engineer who’s received repeated feedback that their pull requests are too big—all over a thousand lines of code, and they keep getting this feedback. Windmill lets the engineering manager quickly see this observation and decide: “Hey, I agree with this, this is important, I want to highlight it in the performance review.” Managers can always add their own observations as well.

The AI gives you observations, all built around the actual performance data. But the manager still decides what they agree with and what’s important. That’s where AI struggles today—deciphering what’s noise and what’s not. So we’re extremely careful about that.

It’s up to the manager to decide what gets included and highlighted. But they’re not working off memory, and they don’t have to dig through everything to find these observations.

Step 3: The Draft

From those observations, Windy writes a draft. This handles the grammatical and language work to write a performance review in the right style. Some of this can be configured by HR—what a good review looks like, how to give constructive feedback.

Windy writes the draft based on the actual data and what the manager decided was important. Customers tell us the initial draft is usually 70-90% complete.

Step 4: The Final Polish

This is often the most critical step. We still believe the manager should be in full control of every single word that gets submitted.

After the draft, the manager polishes it up. Small wording changes, maybe bigger changes to get it into their own style and tone. They make sure it highlights exactly what they want to convey and aligns with the ratings they’re giving.

AI and Human Collaboration

This process keeps both AI and humans focused on what they’re good at.

AI is good at remembering, finding information, and giving you all the available options. Humans are good at deciding what’s important, synthesizing information, and making judgment calls.

With this approach, performance reviews are written faster, more grounded in truth, and more actionable for employees.

The feedback we’ve received has been strong. 93% of people prefer AI-assisted reviews over completely manual ones. And it’s not just time savings—it’s accuracy, it’s synthesis, it’s how grounded the review is in what actually happened rather than what someone vaguely remembers.

The Real Time Flip

Here’s what’s interesting. In the old process, managers spent 70-80% of their time just trying to remember and find information. Very little time went into the actual judgment calls, the rating probably got a few seconds.

With Windmill, this flips. The overall process is much shorter. But the time spent on judgment is actually higher than before, because the rest of the stuff is handed to you on a silver platter.

As AI gets better, this workflow might evolve. But even as AI improves, we believe the ultimate judgment calls should stay with managers. Sifting through Slack channels for six months isn’t time well spent. Making the judgment calls, however, that’s where the manager should be spending their time.