He mentioned that in the talk at the question time part when someone brought up the DEFCON announcement of DARPA and a bunch of private companies running an AI bug hunting program. That’s what the entire talk is basically about. The deluge of AI slop they’ve received in their bug bounty program from people who are using AI to find bugs and not knowing anything about the bugs they find and when questioned for further information their continued use of AI causes it to go off on irrelevant tangents to questions asked and do nothing but waste their time. It was also shocking to see the scraper bandwidth abuse on the curl website. 65 terabytes a month, average 4000 requests per second, downloads are less that 0.01% of the requests made. Thankfully, the curl project doesn’t have to pay for that bandwidth.
I also like the response to Stenberg’s post on Mastodon
2 Likes
Indeed - I read a bug report where the user emphatically stated he had a proof of concept about a buffer overflow in curl where the PoC didn’t actually call curl at all - he’d had an AI agent generate code that had a buff overflow and then the same agent found its own overflow and confidently declared it to be in curl.
The user knew no better - he was just trying to make a few quid from a bug bounty program.
The mastodon discussion is the opposite side of things - someone using AI as a tool, competently, and correctly finding actual issues with the code which could be fixed but were convoluted enough that the previous multiple source code reviews of the curl source had completely failed to spot them.
I guess this is where the distinction lies - @bob490 had some issues he wanted to solve, asked an Anthropic agent about it and got answers. In this case they were apparently correct (or correct enough) to resolve the users immediate requirements. Commands were issued and nothing bad happened.
The shit hits the fan where someone is fed a command which does something cataclysmically bad (dd with the if and of swapped is the first example which springs to mind) and because the user has no clue what that dd command will actually do it’s a copy/paste catastrophe and much hilarity ensues.
“But AI said it was OK” is something I suspect we’ll hear a lot more of in future, right up until everyone realises that no-one is making any money from it and much of it vanishes, leaving a mess of half-baked, tech-debt ridden, poorly crafted code out there.
4 Likes