So, I've been playing with ChatGPT and consorts in anger for the past 2 weeks.
Let me show you today's experience with ChatGPT (3.5 turbo).
I started out by creating a simple, self-contained HTML file with some JS, to parse out a subset of data from JSON. That worked fine, after giving ChatGPT a better idea of the data structure.
Next, I got a little bolder, trying to make ChatGPT load the file(s) and pass it to an existing API, which i tried to describe to it, which coiuld display the file content in a graphical way.
It was a constant back and forth, without me being able to find a prompt that would make it do what it was asked. It's not a terribly hard use case either. But I figured it's bad at compositing simple tasks like that. So I started from scratch.
My goal was to get to a stage, where I can load 3 files and put the results into a JS map filename -> base64 encoded file content.
We'll here's the result.
https://sharegpt.com/c/gmye2Qc
It took me about 25 minutes to get to that stage, mostly because ChatGPT response times are very bad, and because ChatGPT would just keep doing stupid things.
In that time, I could have written this myself, trice.
So, my take away is this:
If you think this will replace programmers anytime soon, I would like to get the phone number of your dealer.
If you give this to a junior dev without human supervision, you will forever have a junior dev churning out bad code that ignores any and all error states and corner cases.
Response times make this an inefficient tool to work on anything that's longer than a simple, self-contained function.
And even for the "single, self-contained function" use case, it usually shits the bed so hard the further you get away from "standard" problems, you'll spend more time massaging it via natural language input than writing things from scratch yourself.
Yes, GPT-4 is slightly better, given the bigger token window. But that's more than counter acted by the further increased response times. Also, it still shits the bed with anything that's "do a qsort in Python".
This is going to be a tool that will be used by many people, only to realize that diminishes their code comprehension and design skills. We'll have to unlearn all this again, once we see the real world negative impact it has on businesses and software quality.
ChatGPT/Copilot finally give programmers a chance to experience how non-technical people feel when trying to coerce programmers to create a thing.
You too can now throw imprecise natural language prompts at a magic coding box and get disillusioned by the results.