Case study: straightforward tech question, genAI is repeatedly, confidently wrong

I recently discovered that when I migrated one of my CentOS servers from using if-up/down scripts to configure its network interfaces, to using NetworkManager (a migration which was forced on me by CentOS), a critical aspect of my network configuration was lost: the server stopped using the correct source address for outbound IPv6 connections. My server is assigned a dedicated /64 block by the VM hosting company, and outbound connections are supposed to come from that block. Prior to the switch to NetworkManager, they were; after the switch, they were not.

I spent several hours web-searching for and reading online documentation and discussions unsuccessfully trying to figure out how to tell NetworkManager how to configure the default source address for outbound IPv6 connections. I finally decided to resort to Microsoft Copilot Chat to see if it could guide me in the right direction.

On the plus side, by the end of my chat with Copilot, I had found an adequate solution to the problem.

On the other hand, throughout the chat Copilot was repeatedly, confidently wrong, and if I had followed its instructions without question it would have screwed up my server’s configuration.

Below, for those who are curious, I detail all wrong and dangerous suggestions Copilot gave me. However, before doing that, I want to talk about why this all concerns me so much and should concern you too. None of this is new. Many other people more qualified to comment on this than I am have said what I am about to say. But I feel like we need to keep saying it, because we need people to listen.

Generative AI gets things wrong. A lot. If you have a lot of experience in whatever you are using it to help you with, you may be able to spot the mistakes it makes and redirect it in a more correct direction. However, if you don’t have a lot of experience, and you’re relying on the AI to be smart, you’re eventually going to get screwed when it leads you astray and that costs you (or others) something real.

Two things happen when people become habituated to using generative AI to produce work product in areas the people are not experienced in:

1. The stuff they produce has shitty quality which negatively impacts real people in the real world.
2. Those people never become experienced enough to be able to spot when the AI is wrong, because they’re not doing the work to become experienced, they’re having the AI do it for them.

As bad as things are now, with the negative impacts of AIs already showing up in the real world on a daily basis, if we continue down this path of outsourcing our thinking to AIs, things are going to get much, much worse.

Sam Altman, the CEO of OpenAI, said, “We see a future where intelligence is a utility, like electricity or water, and people buy it from us on a meter.” The problem with this statement is AIs are not intelligent, and they will never be intelligent without major breakthroughs in the technology. There is no sign of any such breakthroughs coming any time soon. That isn’t going to stop Altman and the other AI grifters from continuing to try to convince us that AIs can replace human intelligence. We need to push back.

---

So, let’s talk about all of Copilot’s screw-ups when I asked it for help with a seemingly simple NetworkManager problem.

I asked: “In NetworkManager using nmcli how can I configure the source address for outbound IPv6 connections.”

In retrospect, I see that my question was vague. I should have been more specific about the fact that my server has two different IPv6 addresses in two different address blocks assigned to it and I wanted to control which one was used as the outbound source address. If Copilot were an actual person with actual intelligence in the field being discussed, they would have recognized the ambiguity and asked for clarification. Instead, Copilot jumped into a confidently wrong answer:

“Option A” above is dangerously wrong for me, because the network interface in question already has a ipv6.addresses setting with two different addresses in it, so what Copilot told me to do would have erased one of the two addresses, thus preventing the server from continuing to accept inbound connections on that address.

“Option B” above is functionally wrong for the reasons described below.

After the three wrong answers Copilot gave me several more screens of useless text, as is its habit. Then I asked this:

When I do “nmcli c modify ipv6.routes ‘default table=100’ it says: “Error: failed to modify ipv6.routes: invalid route: Invalid IPv6 address ‘default’. The valid syntax is: ‘ip[/prefix] [next-hop] [metric] [attribute=val]… [,ip[/prefix] …]’.”

In response, Copilot said this:

Notice how it says nothing here about how the command that isn’t working is a command that it told me to use?

After this it once again spit out several screens full of useless text.

OK, so, I replaced “default” with “::/0”, which allowed me to run the command Copilot had suggested, but that didn’t have any impact on the source address for outbound connections. I told Copilot that, and it explained, at unendurably great length, why its earlier answer was wrong, actually. Then it tried to make some guesses about why: Rather than reading through all that nonsense I responded with this:

OK let’s reset and start over please.
The problem is that I have two different IPv6 addresses assigned to my host. I have to accept inbound connections on both of them.
One of them is a /128, i.e., an individual address, and one of them is a /64, a range.
I need outbound connections to come from an address inside the range.
I need to configure this with NetworkManager (nmcli).
I can’t remove the other address from the interface because I need to accept connections on it.

Here’s the answer it gave me in response:

Hoo boy, let’s talk about all the ways this is wrong.

First of all, remember, this is after I’ve told Copilot explicitly that I have two IPv6 addresses configured and I need both of them to keep working for inbound connections. And yet it again told me to change the ipv6.addresses setting on the interface to contain only a single address, which if I had listened would have immediately stopped the interface from accepting connections on the other address.

Second, disabling ipv6.ip6-privacy is irrelevant here. Both of the addresses in question are static.

Third, it says this solution will only work if “The /64 address has a better prefix match for most destinations”, but how could it possibly know whether that’s the case? It’s just assuming that it’s true and giving me an answer that fails if it isn’t?

Fourth, it’s not necessary to do nmcli connection down before nmcli connection up when changing the configuration of a NetworkManager connection; it’s sufficient to just do the up command. Note that it “knew” that earlier in the conversation and did not previously tell me to run down before up.

Fifth, it conveniently failed to mention that if I had followed its instructions and run nmcli connection down enp0s3 the server would have immedately dropped off the network, which would have been kind of problematic given that I was dong all this work over the network. 🤦

Needless to say, there were several more screens full of verbiage after the wrong answer above.

I responded asking about the first problem above:

Won’t this command erase the other addresses assigned to the interface? nmcli connection modify enp0s3 ipv6.addresses “2600:3c03:e000:29c::1234/64”

It responded by telling me that I was absolutely right and that I should use nmcli connection modify enp0s3 +ipv6.addresses instead, to preserve the existing addresses. Which, of course, is still wrong, because it ignores the fact that the address I’m trying to make the source address for outbound connections is already assigned to the interface.

I responded that I tried its new suggestion, and once again it did not work. It spit back a bunch of verbiage explaining various theories for why it wasn’t working and then asked me to send it the output of ip -6 addr show dev enp0s3 to help it figure out the problem. I sent it the output, after which it claimed that the problem was that “my /128 and my /64 are on different global prefixes” and that’s why Linux was picking the wrong one. I’m pretty sure this is just wrong, i.e., I’m pretty sure that what it was suggesting wouldn’t have worked even if that hadn’t been the case, but at best this is the assumption I mentioned above (“it says this solution will only work if…”) biting it in the ass.

Next it told me to run this command to force the address I don’t want to be use for outbound connections to be “deprecated”: “nmcli connection modify enp0s3 +ipv6.addresses "2600:3c03::f03c:91ff:fe67:dac1/128 preferred_lft 0“. This is wrong for two reasons:

1. It is, again, telling me to add an address to the interface which is already on the interface.
2. The syntax it suggested for doing that is simply invalid.

It offered two other options, moving the /128 address to the loopback interface (which I confess I didn’t try because it’s gross), and “Using policy routing to force the /64”, but right after suggesting that option it said, no joke, “But this still doesn’t override RFC 6724 unless the /128 is deprecated. So even with policy routing, you still need Option 1.” So why the fuck did it suggest an option that it already “knows” doesn’t work? Answer: because it knows nothing, there is no intelligence here.

I responded by sending it a message showing that the command it had told me to run is simply invalid:

# nmcli c modify enp0s3 +ipv6.addresses “2600:3c03::f03c:91ff:fe67:dac1/128 preferred_lft 0”

Error: failed to modify ipv6.addresses: invalid prefix ‘128 preferred_lft 0’; <0-128> allowed.

It replied by explaining to me in detail how that wasn’t actually a valid NetworkManager command, again not acknowledging that it was the exact command it had previously told me to run.

Then it offered me three different solutions for how to do this given that all of the solutions it had previously offered were wrong. The first one, also wrong:

This is wrong because for some inexplicable reason it mentions ipv6.addr-gen-mode in the title of the method and then never actually tells me to do anything with that setting, and because it doesn’t work, for reasons I explain below.

The second method it suggested was the one I mentioned before, moving the /128 address to the loopback interface. Maybe this would actually work 🤷, but it’s gross and I didn’t try it.

And the third method it offered was yet another example of suggesting something it already “knew” wouldn’t work:

So, what the heck, I spent like 20 minutes trying to get its first suggested method, the dispatcher.d script, to work. It wouldn’t work. I sent Copilot this:

The dispatcher.d script idea you suggested isn’t working because something is changing the preferred_lft value for the address back to forever after the dispatcher.d script runs. I confirmed this by adding an ip -6 addr show enp0s3 command to the end of the script and capturing its output. The preferred_lft value is correctly set to 0sec when the script exits, but then when I run the ip -6 addr show command from the command line afterward it’s back to forever.

It responded by explaining to me in detail why the solution it had just suggested to me would not, in fact work.

At this point it offered me three additional suggestions:

1. It again suggested moving the /128 address to the loopback interface, calling this the “best practical option.”
2. It suggested that I tell the individual programs that are making outbound connections what network address to use as the source when doing so, and gave examples of how to do that with curl, ssh, and ping6 commands.
3. Stop letting NetworkManager manage the interface and manage it with systemd-networkd or ifupdown instead. Ugh.

Now, I had known all along, even before starting this conversation with Copilot, that configuring sendmail to bind to a specific local address when making outbound IPv6 connections was an option. At this point I decided I’d wasted enough time trying to make it work at the network configuration level, so I replied to Copilot with this:

Do you know how to tell sendmail to use a specfic source address when establishing outbound connections?

It gave me this answer:

I suspect it will come as no surprise to you to hear that this, too, was wrong. This is the correct syntax:

CLIENT_OPTIONS(`Family=inet6,Address=2600:3c03:e000:29c::1')dnl

I added this to my sendmail.mc file, regenerated sendmail.cf, and restarted sendmail, and the problem was fixed.