Nuked 
Hello followers! Today, let’s have some fun exploring how our beloved AI developer helpers might have a sneaky side.
Marketers tout AI tools like GitLab’s Duo chatbot as indispensable for developers, capable of quick tasks like creating to-do lists by simply asking. However, there’s a dark side they rarely mention—it turns out these tools can be tricked into harmful actions by malicious users.
Recently, security researchers from Legit demonstrated an attack that made Duo insert malicious code into a script. This could lead to leaking private code or sensitive issue data, like zero-day vulnerabilities, with just a simple instruction from a user to interact with external content.
The attack primarily involves prompt injections—hidden instructions embedded within code, comments, or other development content—that trick AI assistants into doing things they shouldn’t. For instance, researchers hid a command instructing Duo to add a malicious URL in the code, which the chatbot then included in its explanation, disguised with invisible characters to evade detection.
These embedded instructions can be crafted in markdown or HTML, and because Duo processes responses line by line, active tags like and can execute unintended actions, potentially exposing private repositories or leaking confidential data. The researchers showed how they achieved this by embedding invisible Unicode characters and malicious URLs.
GitLab responded by disabling unsafe HTML tags outside their domain, effectively stopping the demonstrated attacks. Still, this highlights an important point: AI assistants integrated into development workflows need robust safeguards, as they can become attack surfaces themselves.
In conclusion, while AI tools boost productivity, users must remain vigilant in inspecting outputs for malicious hints. As Mayraz succinctly put it, AI assistants are powerful—yet, without proper security, they can become risks rather than assets.
