I did a little experiment tonight, the results of which have me kind of shook.
I had a spreadsheet column, each cell of which contained multiple merged values separated by semicolons. I wanted to separate them out into a separate column for each value, with a “1” in each row in each of those columns for the values present in the merged cell for that row.
I could have written a Python or Perl script to do this in about a half hour, but given that this work I was doing was part of a project to assess AI usage at my company and make recommendations for what to do about AI moving forward, I figured what the heck, let’s ask Copilot Chat to write the script. I should have some experience with the technology I’m making recommendations about, right?
I gave Copilot Chat a 2-sentence description of what I wanted:
“write me a python script which reads as input multi-value lines with the values separated by semicolons, and outputs a CSV with column headers corresponding to the various values on the input lines and with a ‘1’ as the value in each column for the input lines where that value was specified. The output rows should be in the same order as the input lines”
A few seconds later, it spit out a clean, working Python script which conformed exactly to my specifications.
The script was putting an extra blank value on every row because the input data were all semicolon-terminated. I told Copilot to fix that, and it did.
I asked it to show me a diff of the two versions of the script, and it did (it kept showing me diffs for the subsequent changes below without being asked).
Then I told it to make three additional changes: blank lines on the input should result in an empty row in the output rather than being skipped; a couple of the values which occurred only a few times in the input should be inserted into an “Other” column in the output rather than getting their own dedicated columns; and the names of several of the values should be shortened in the output column headers.
Copilot got all those changes right, the first time.
I told it to move a couple more values into the “Other” column, and it did it.
I am still skeptical about people producing thousands of tens of thousands of lines of code that they couldn’t possibly have read and simply assuming that they are all correct. But I have to admit that, at least for short scripts or single functions that you can easily read and verify after the LLM gives them to you, these tools are absolutely at the point where they are incredible time-savers. I could have done all this myself, but the AI did it much, much faster.
This does not excuse the massive intellectual property theft that went into building the models, nor the environmental impact, nor the continuous inflated claims, nor Sam Altman trying to lock in “intelligence as a service” and make everyone dumb, nor the literal death and violence that these tools are facilitating, nor the fact that if junior people don’t get to exercise their critical thinking skills on the job they will never turn into senior people who know how to guide the LLMs, make sure their outputs are correct, and navigate fixing them when they aren’t. All of these are still problems, and I am still extremely worried about them.
But I am doubtful that I will be able to convince my employer to care about any of them. I’m going to have to stop saying that these tools can’t possibly help our staff do their jobs better, because that is clearly not true. Instead, I am going to have to work with my colleagues to figure out what our best practices should be for using these tools safely and effectively, ensure that everyone in the company is effectively trained in those best practices on an ongoing basis, and put processes in place to ensure that they’re being followed.
