GPT-4, Two Days In: An Engineer's Working Notes

OpenAI released GPT-4 on Tuesday, and like apparently everyone else in this industry, our engineering team has spent the last forty-eight hours feeding it our actual problems instead of doing some of our actual work. These are working notes, written fast and dated on purpose. Everything here is a two-day impression of a system we do not fully understand yet, published in the spirit of showing our work.

What is noticeably different from 3.5

The headline for us is not that it writes better, although it does. It is that it holds more of a problem in its head at once. GPT-3.5 could write a function; GPT-4 can follow a refactor across several files' worth of pasted context and keep the constraints straight. We handed it a gnarly TypeScript generics error from a real client codebase, the kind where the compiler message is forty lines of nested despair, and it identified the actual variance issue on the first pass. That was a junior-to-mid-level debugging session compressed into thirty seconds.

It is also markedly better at "boring translation" tasks that make up more of engineering than anyone admits:

Prompt pattern that is earning its keep this week:
"Here is a component using [old pattern]. Rewrite it using
[new pattern]. Preserve behavior exactly. List anything
you were unsure about at the end."

That last sentence matters. Asking the model to surface its own uncertainty does not make it honest, but it gives you a review checklist, and review is the whole game.

What has not changed

It still confabulates, confidently, and the improved fluency makes the failures more dangerous rather than less. Yesterday it invented a plausible-looking configuration option for a library we use daily. Correct naming convention, correct casing, does not exist. A junior engineer would have burned an hour on that. The model's confidence is not evidence, and nothing about this release changes our rule: generated code is a draft from an eager collaborator who read the whole internet and tests nothing.

Also unchanged: it does not know your system. It cannot know that the checkout service has an undocumented rate limit, that the CMS webhook fires twice on Tuesdays for reasons lost to history, that the client's legal team banned a dependency last spring. Institutional context remains our job, which is convenient, because institutional context is most of the job.

How we are actually using it this week

Our current working rules, subject to revision as the shine wears off:

• Prototyping and spikes: yes, aggressively. Getting a rough interactive prototype in front of a designer in an afternoon changes conversations. Prototype code was always disposable; now it is disposable and fast.
• Production code: through the normal door. Anything headed for a client repo goes through review like code from any other author, with the added rule that you cannot approve what you cannot explain.
• Nothing sensitive in the prompt. No client code containing credentials, no unreleased brand work, no personal data. This is policy, written down, this week, because the temptation to paste is real.

The honest takeaway

Two days in, GPT-4 feels less like a new tool and more like a raise in the floor. The tasks it absorbs are real, and the engineers here are not mourning them: nobody's favorite part of the week was writing the fourth CRUD form. What it has not touched is the part where someone decides what to build, notices the requirement that is missing, or takes responsibility for the thing in production at 2am.

We will report back when the honeymoon data comes in. For now: impressed, cautious, and slightly behind on this sprint, for reasons we can honestly explain.