Promptfoo

Promptfoo (promptfoo.dev) is a company specializing in AI security solutions. It offers an AI security platform designed to help developers and enterprises build secure and reliable AI applications by catching vulnerabilities during development. The platform is widely adopted, trusted by major enterprises, and used by over 200,000 developers worldwide. Promptfoo's main AI product focuses on testing and securing large language models (LLMs) through features like red teaming, bias detection, and vulnerability identification. The target audience includes developers, security teams, and enterprises deploying generative AI products and LLMs at scale.

Promptfoo is a platform focused on AI security testing and evaluation, helping organizations identify and fix vulnerabilities in AI applications. Here are the main features and functions of Promptfoo: - Red Teaming: Automated red teaming for AI agents and Retrieval-Augmented Generation (RAG) systems, simulating real user attacks to uncover vulnerabilities such as prompt injections, jailbreaks, data leaks, business rule violations, insecure tool use, and toxic content generation. - Guardrails: Real-time protection against jailbreaks and adversarial attacks to secure AI applications during operation. - Model Security: Comprehensive security testing and monitoring for AI models, ensuring ongoing protection and compliance. - MCP Proxy: Secure proxy for Model Context Protocol communications, adding a layer of security to model interactions. - Code Scanning: Finds LLM (Large Language Model) vulnerabilities directly in your IDE and CI/CD pipelines, integrating security into the development workflow. - Evaluations: Tools to test and evaluate prompts, models, and RAG pipelines, supporting continuous improvement and reliability. - Integrations: Connects with CI/CD pipelines, GitHub, GitLab, Jenkins, and supports both on-premise and cloud deployments. - Remediation: Provides actionable remediation guidance directly in pull requests and developer workflows, with continuous monitoring and tracking of fixes. - Real-time Threat Intelligence: Leverages a large community for up-to-date threat intelligence and automatic deployment of new attack vectors. - Enterprise-Scale Automation: Scales security testing from single applications to large enterprise environments, with deep automation and minimal manual intervention. Promptfoo is trusted by major enterprises and integrates security into every stage of AI development and deployment.

Promptfoo.dev primarily supports the practice area of Privacy & data protection, specifically focusing on AI security, LLM (large language model) evaluation, and red teaming for vulnerabilities in AI systems. Features and functions of promptfoo.dev include: - Automated LLM prompt testing and evaluation - Red teaming for AI systems to detect vulnerabilities and misconfigurations - Synthetic dataset generation for robust testing - Guardrails to ensure secure and reliable AI outputs - Configuration of test cases, assertions, and metrics for LLM output validation - Support for multiple LLM providers and prompt templates - Performance and security reporting for AI models.

Promptfoo is covered under the following security and compliance frameworks: - SOC 2 Type II - ISO 27001 - GDPR compliant - HIPAA compliant There is no explicit mention of CCPA compliance in the available sources.

Promptfoo explicitly states the following about data handling: - No data retention: With an API key and telemetry disabled, no data is sent to Promptfoo servers. Local generation and grading are possible, and no prompt content, responses, or personally identifiable information are included in telemetry (which can be disabled). - Client data not used for training: Model weights or training data are never sent to Promptfoo, and files from your filesystem are not sent unless explicitly configured. - On-prem deployment: Promptfoo Enterprise On-Prem provides a dedicated runner within your network, supports full air-gapped operation, and ensures no data transmission to external servers. - Private cloud: Enterprise deployment options include self-hosted inference for all plugins, supporting private cloud or on-premises use. - Customer-managed encryption keys: This is not explicitly mentioned in the documentation page reviewed. For more details, see Promptfoo’s data handling and privacy documentation.