Vocabulary:
Repeat each word, definition, out loud.
| Vulnerability | A weakness in a system or software that can be exploited by attackers to gain unauthorized access or cause harm. |
| Remote Code Execution (RCE) | A type of attack where a hacker can run malicious code on someone else’s computer or server from a distance. |
| Insecure Direct Object Reference (IDOR) | A security flaw where unauthorized users can access restricted data by manipulating references to objects, such as files or user accounts. |
| Timing Attack | A method where hackers observe the time it takes for a system to respond to different inputs, using this information to guess sensitive data, such as passwords or API keys. |
| Access Control | The methods and rules used to determine who can access certain data, systems, or resources within a network. |
Article:
Read the article aloud on your own.
More than 30 security issues have been discovered in various open-source artificial intelligence (AI) and machine learning (ML) systems. Some of these flaws could allow hackers to run malicious code or steal sensitive data.
The vulnerabilities were found in platforms such as ChuanhuChatGPT, Lunary, and LocalAI, and were reported through Protect AI’s bug bounty program, Huntr.
The most serious problems affect Lunary, a toolkit used to manage large language models (LLMs):
- CVE-2024-7474 (CVSS score: 9.1) – An insecure direct object reference (IDOR) flaw that lets an authenticated user view or delete external accounts, potentially leading to data loss and unauthorized access.
- CVE-2024-7475 (CVSS score: 9.1) – A flaw in access control that allows attackers to change the SAML configuration, letting them log in as unauthorized users and access sensitive information.
- CVE-2024-7473 (CVSS score: 7.5) – Another IDOR vulnerability that could let hackers update other users’ prompts by tampering with user-controlled parameters.
In one example shared by Protect AI, an attacker could log in as one user, intercept the request to change a prompt, and modify the “id” parameter to update a prompt belonging to another user.
A critical vulnerability was also found in ChuanhuChatGPT’s upload feature (CVE-2024-5982), which could let attackers run code, create directories, and expose sensitive data.
In LocalAI, two security flaws were found: one that lets hackers execute code by uploading a malicious configuration file (CVE-2024-6983), and another that allows attackers to guess API keys based on server response time (CVE-2024-7010), a method known as a timing attack.
Additionally, a flaw in the Deep Java Library (CVE-2024-8396) was discovered, allowing remote code execution through a file overwrite bug in the untar function.
These disclosures came as NVIDIA released patches for a security issue in its NeMo AI framework that could lead to code execution and data tampering. Users are advised to update their software to protect their AI/ML systems from attacks.
Protect AI has also introduced Vulnhuntr, a tool that uses LLMs to find vulnerabilities in Python codebases. It breaks the code into smaller parts and scans for security issues, focusing on areas that handle user input first.
Finally, a new technique to bypass security safeguards in AI models, discovered by Mozilla’s 0Day Investigative Network (0Din), shows that certain prompts encoded in hex or emojis can trick models like OpenAI’s ChatGPT into performing harmful actions. This happens because the AI follows instructions step-by-step but doesn’t always understand the bigger picture, leading to security risks.
Questions:
Answer the following questions about the article.
- What types of vulnerabilities were discovered in AI and ML models?
- What is the impact of the CVE-2024-7474 vulnerability on Lunary?
- How can an attacker exploit the ChuanhuChatGPT vulnerability?
- Explain what a timing attack is and how it was used in LocalAI.
- Why is it important to update AI and ML tools after vulnerabilities are disclosed?
Discussion:
Use these questions for discussion.
Why do you think open-source software is more prone to vulnerabilities?
In your opinion, what could companies do to improve the security of AI/ML systems?
The article mentions that AI models like ChatGPT can be tricked by hex-encoded prompts. What are the potential dangers of this?
How do bug bounty programs like Protect AI’s Huntr help improve the security of AI and ML tools?
How do you think security vulnerabilities in AI systems might affect industries that heavily rely on AI, such as healthcare or finance?
Source: This exercise is based off an article from Hacker News
English