Description

Models might generate code that causes harm or unintentionally affects other systems.

Why is harmful code generation a concern for foundation models?

The execution of harmful code might open vulnerabilities in IT systems.

Example

Generation of Less Secure Code

According to their paper, researchers at Stanford University investigated the impact of code-generation tools on code quality and found that programmers tend to include more bugs in their final code when they use AI assistants. These bugs might increase the code's security vulnerabilities, yet the programmers believed their code to be more secure.

Sources:

Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do Users Write More Insecure Code with AI Assistants?. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23), November 26-30, 2023, Copenhagen, Denmark. ACM, New York, NY, USA, 15 pages.

Related Risks

• LLM02: Insecure Output Handling (OWASP Top 10 for Large Language Model Applications v1.1)
• Information Security (NIST AI Risk Management Framework (AI RMF))

Parent topic: AI risk atlas

We provide examples covered by the press to help explain many of the foundation models' risks. Many of these events covered by the press are either still evolving or have been resolved, and referencing them can help the reader understand the potential risks and work towards mitigations. Highlighting these examples are for illustrative purposes only.