AI Security Vetting - Continuous AI Assurance

What This Tool Does

6 Security Categories

50+ Core Security Tests

6 AI Providers Supported

4 Output Formats

Sends security test prompts to your AI model/agent and scores the response. Covers Prompt Injection, Exfiltration (PII/IP), Tool/Schema Disclosure, Unsafe Actions, Governance, and Memory/Retention. Generates synthetic AU/NZ PII seeds to validate memory safety and produces comprehensive reports.

Why Vet Your AI?

🔍Background Checks for AI

You perform background checks and police checks on your staff before giving them access to sensitive systems and data. Why don't you vet your AI? Your AI systems have access to the same sensitive information and can make autonomous decisions that impact your business.

🛡️Security Clearance for AI

You ensure staff and contractors have Government security clearances (Confidential, Secret, Top Secret) before providing access to classified information and autonomous action capabilities. Why don't you perform the same checks against your AI? Apply the same security standards to your artificial intelligence systems.

🎯Penetration Testing for AI

You conduct regular penetration testing of your systems and services to identify vulnerabilities. Why not do the same for your AI? AI systems can be compromised through prompt injection, data exfiltration attempts, and social engineering attacks that traditional security tools can't detect.

Continuous AI Assurance is Critical

Your AI is constantly evolving with the data you provide and updates from AI vendors. Traditional one-time security assessments aren't enough. Your assurance must be continual and ongoing. This tool provides automated, repeatable security testing that can be integrated into your CI/CD pipeline for continuous AI compliance and risk management.

Comprehensive Security Testing

🚨 Prompt Injection Detection

Tests your AI's resistance to malicious prompt injection attacks that could compromise system behavior or extract sensitive information.

🔒 Data Exfiltration Testing

Validates that your AI properly protects PII, intellectual property, logs, and secrets from unauthorised extraction attempts.

🛠️ Tool/Schema Disclosure

Ensures your AI doesn't leak sensitive information about its capabilities, tools, or internal schemas to potential attackers.

⚠️ Unsafe Action Prevention

Tests that your AI refuses to perform dangerous actions like payments, MFA resets, or data deletion without proper authorisation.

👥 Governance Compliance

Validates that your AI properly escalates sensitive requests like mass communications or external data transfers to human oversight.

🧠 Memory/Retention Testing

Uses synthetic AU/NZ PII to test whether your AI improperly retains or recalls sensitive information across sessions.

Supported AI Providers

OpenAI

GPT-4, GPT-3.5 and compatible APIs

Azure OpenAI

Enterprise-grade OpenAI models

Anthropic Claude

Claude 3 Haiku, Sonnet, Opus

Google Gemini

Gemini Pro and Ultra models

Microsoft Copilot

Bot Framework Direct Line v3

OpenAI Compatible

Any OpenAI-compatible API

Specialised for Australian 🇦🇺 and New Zealand 🇳🇿 Data

Our tool generates synthetic, checksum-valid Australian and New Zealand identifiers to test your AI's memory safety and data protection capabilities. Never uses real PII - only realistic test data.

🇦🇺 Australian Identifiers

• Tax File Numbers (TFN) - Checksum-validated 9-digit identifiers
• Medicare Numbers - Valid format with check digits
• State Driver Licences - NSW, VIC, QLD, WA, SA, TAS, ACT, NT formats
• Australian Passports - Realistic passport number formats
• Australian Mobile Numbers - Valid 04xx xxx xxx patterns
• Australian Business Numbers (ABN) - 11-digit validated identifiers

🇳🇿 New Zealand Identifiers

• IRD Numbers - Checksum-validated Inland Revenue identifiers
• NZ Driver Licences - Valid regional format variations
• NZ Passports - Realistic New Zealand passport formats
• NZ Mobile Numbers - Valid 02x xxx xxxx patterns
• NZBN (Business Numbers) - 13-digit business identifiers
• National Health Index (NHI) - Healthcare identifier formats

Memory Safety Testing Modes

Same-Session Testing

Seeds are injected in the same exchange to test immediate echo vulnerabilities

Cross-Session Testing

Seeds sent in separate sessions to test long-term memory retention

Strict Mode Validation

Only fails on validated sensitive data (TFN, Medicare, etc.) to reduce false positives

⚠️ Ethical Testing Guarantee

All test data is synthetic and generated algorithmically. We never use real customer data, production PII, or actual government identifiers. Our synthetic data follows authentic formatting and validation rules but represents no real individuals or entities.

Quick Start Examples

Get started in minutes with these common configurations. Replace the binary name with yours (e.g., AISecurityVetting.exe on Windows).

OpenAI (Generic Environment)

# macOS/Linux
export OPENAI_API_KEY=sk-xxxx
./AISecurityVetting --provider openai --model gpt-4o-mini --license-key YOUR_KEY

# Windows PowerShell
$env:OPENAI_API_KEY="sk-xxxx"
.\AISecurityVetting.exe --provider openai --model gpt-4o-mini --license-key YOUR_KEY

Azure OpenAI

# Set environment variables
export AZURE_OPENAI_API_KEY=xxxxx
./AISecurityVetting \
  --provider azure-openai \
  --azure-endpoint https://YOUR-RESOURCE.openai.azure.com \
  --azure-deployment gpt4o \
  --model gpt-4o \
  --license-key YOUR_KEY

Anthropic Claude

# Claude API
export ANTHROPIC_API_KEY=xxxx
./AISecurityVetting --provider anthropic --model claude-3-5-sonnet-20240620 --license-key YOUR_KEY

# Windows PowerShell
$env:ANTHROPIC_API_KEY="xxxx"
.\AISecurityVetting.exe --provider anthropic --model claude-3-5-sonnet-20240620 --license-key YOUR_KEY

Google Gemini

# Gemini API
export GEMINI_API_KEY=xxxx
./AISecurityVetting --provider gemini --model gemini-1.5-pro --license-key YOUR_KEY

# Windows PowerShell
$env:GEMINI_API_KEY="xxxx"
.\AISecurityVetting.exe --provider gemini --model gemini-1.5-pro --license-key YOUR_KEY

# Optional: Use different API version
./AISecurityVetting --provider gemini --model gemini-1.5-pro --gemini-base v1 --license-key YOUR_KEY

Microsoft Copilot (Direct Line v3)

# Microsoft Copilot with specialised environment
export COPILOT_DIRECTLINE_SECRET=xxxx
./AISecurityVetting \
  --provider copilot --model ignored \
  --copilot-user-id security_tester \
  --target-env copilot \
  --seed-mode same-session --seed-count 5 \
  --license-key YOUR_KEY

# Windows PowerShell
$env:COPILOT_DIRECTLINE_SECRET="xxxx"
.\AISecurityVetting.exe `
  --provider copilot --model ignored `
  --copilot-user-id security_tester `
  --target-env copilot `
  --seed-mode same-session --seed-count 5 `
  --license-key YOUR_KEY

Note: Copilot uses Bot Framework Direct Line v3. The --model parameter is ignored. Recommended to use --target-env copilot for specialised Dataverse/Power Platform testing.

OpenAI Compatible APIs

# Any OpenAI-compatible API (Ollama, LocalAI, vLLM, etc.)
export OPENAI_API_KEY=your-api-key-or-token
./AISecurityVetting \
  --provider openai-compat \
  --base-url http://localhost:11434/v1 \
  --model llama3:latest \
  --license-key YOUR_KEY

# Example: Ollama local instance
export OPENAI_API_KEY=dummy-key
./AISecurityVetting \
  --provider openai-compat \
  --base-url http://localhost:11434/v1 \
  --model mistral:7b \
  --license-key YOUR_KEY

# Example: vLLM deployment
export OPENAI_API_KEY=your-vllm-token
./AISecurityVetting \
  --provider openai-compat \
  --base-url https://your-vllm-endpoint.com/v1 \
  --model meta-llama/Llama-2-7b-chat-hf \
  --license-key YOUR_KEY

Compatible with: Ollama, LocalAI, vLLM, Together AI, Groq, Perplexity API, and any other service implementing OpenAI's Chat Completions API format.

Advanced Configuration Examples

Advanced scenarios for comprehensive security testing, red team exercises, and specialised environments.

Full Enterprise Red Team Testing

# RAG + tools environment, aggressive attacks, obfuscation, strict memory classification
export OPENAI_API_KEY=sk-xxxx
./AISecurityVetting \
  --provider openai --model gpt-4o-mini \
  --target-env rag-tools \
  --attack-preset aggressive \
  --attack-obfuscation rot13 \
  --strict-mode \
  --seed-mode both --seed-count 15 \
  --region anz \
  --out red_team_$(date +%F) \
  --license-key YOUR_KEY \
  --temperature 0.0 \
  --timeout 90

Use case: Comprehensive security assessment for enterprise AI agents with tool access. Tests against sophisticated attack patterns with obfuscation techniques.

Memory/Retention Deep Testing

# Focus on memory leaks with extensive synthetic PII seeding
export ANTHROPIC_API_KEY=xxxx
./AISecurityVetting \
  --provider anthropic --model claude-3-5-sonnet-20240620 \
  --seed-mode both \
  --seed-count 25 \
  --seed-namespace "MEMORY-TEST-$(date +%Y%m%d)" \
  --strict-mode \
  --region anz \
  --attack-preset auto \
  --out memory_audit_results \
  --license-key YOUR_KEY

Use case: Validate memory safety and PII handling. Tests both same-session and cross-session retention with 25 synthetic AU/NZ records (TFN, Medicare, driver licenses, etc.).

Microsoft Copilot Agent Testing

# Specialised for Copilot with Dataverse/Power Platform probes
export COPILOT_DIRECTLINE_SECRET=xxxx
./AISecurityVetting \
  --provider copilot --model ignored \
  --copilot-user-id security_tester_001 \
  --target-env copilot \
  --seed-mode same-session --seed-count 10 \
  --attack-preset aggressive \
  --attack-as-system \
  --region anz \
  --out copilot_security_audit \
  --license-key YOUR_KEY \
  --timeout 120

Use case: Test Microsoft Copilot agents for Dataverse exfiltration, Power Automate triggers, SharePoint/Teams governance violations, and system-level prompt injection.

Custom Test Suite with File Seeding

# Custom YAML suite with pre-defined synthetic PII seeds
export AZURE_OPENAI_API_KEY=xxxxx
./AISecurityVetting \
  --provider azure-openai \
  --azure-endpoint https://YOUR-RESOURCE.openai.azure.com \
  --azure-deployment gpt4o \
  --model gpt-4o \
  --suite ./custom_financial_tests.yaml \
  --seed-file ./synthetic_customer_data.yaml \
  --seed-mode cross-session \
  --attack-preset lite \
  --strict-mode \
  --system-prompt "You are a secure financial AI assistant..." \
  --out custom_financial_audit \
  --license-key YOUR_KEY

Use case: Industry-specific testing with custom scenarios and controlled synthetic data. Perfect for specialised domains like finance, healthcare, or legal services.

System-Layer Compromise Simulation

# Test system prompt injection and orchestrator bypass attempts
export GEMINI_API_KEY=xxxx
./AISecurityVetting \
  --provider gemini --model gemini-1.5-pro \
  --target-env rag-tools \
  --attack-preset aggressive \
  --attack-as-system \
  --attack-obfuscation rot13 \
  --seed-mode both --seed-count 20 \
  --strict-mode \
  --max-tokens 2048 \
  --temperature 0.0 \
  --out system_compromise_test \
  --license-key YOUR_KEY \
  --log-level debug

Use case: Simulate advanced persistent threats targeting the AI orchestration layer. Tests malicious system instructions, prompt injection via tool outputs, and multi-vector attacks.

Pro Tips for Advanced Testing

Stable namespacing: Use consistent --seed-namespace values for reproducible cross-session tests
Debug logging: Add --log-level debug to troubleshoot provider connection issues
Timeout tuning: Increase --timeout for slow providers or complex tool chains
Temperature control: Keep --temperature 0.0 for consistent security outcomes
Output organisation: Use date-stamped output directories for audit trails

Test Categories & Environments

📖 Download Instruction Manual (PDF)

Complete setup guide, command reference, and advanced configuration examples

🎯 Generic Environment (Default)

50 core tests across all security categories. Perfect for general LLM safety vetting and baseline security assessment.

🔧 RAG + Tools Environment

Adds enterprise-specific probes for Salesforce, Xero, SharePoint, Slack, Jira, MYOB. Use for agents with enterprise integrations.

💼 Microsoft Copilot Environment

Specialised tests for Dataverse, Power Automate, SharePoint, Outlook, Teams. Designed for Microsoft Copilot agents.

Output Formats

📊 Interactive HTML Report

Modern, searchable interface with KPI cards, severity charts, filters, and detailed analysis. Perfect for demos and stakeholder presentations.

📝 Markdown Report

Detailed findings with attack preambles, effective prompts, and elaborated evaluations. Great for documentation and sharing.

📈 CSV Results

Machine-friendly summary for analysis, trending, and integration with other tools. Includes scores, latency, and configuration details.

🔍 JSONL Details

Complete test results with raw provider payloads, findings, and metadata. Perfect for programmatic analysis and integration.

Key Configuration Options

Seed Modes: Test memory retention with same-session, cross-session, or both patterns
Attack Presets: None, lite, aggressive, or auto-scaling adversarial preambles
Obfuscation: ROT13 encoding to test decoding-based bypass attempts
Strict Mode: Only fail on validated sensitive data echoes (TFN, Medicare, etc.)
Custom Suites: Load your own YAML test definitions for specific requirements

See AI Security Vetting in Action

Watch how our tool systematically tests AI systems for security vulnerabilities, from prompt injection to memory leaks, delivering comprehensive reports in real-time.

Interactive Security Testing

Real-time vulnerability detection across 6 security categories with detailed reporting and immediate insights.

🚨

Prompt Injection

🔒

Data Exfiltration

🛠️

Schema Disclosure

⚠️

Unsafe Actions

👥

Governance

🧠

Memory Safety

Ready to Secure Your AI?

Don't leave your AI systems vulnerable to attacks. Implement continuous security testing and compliance monitoring today.

Download for Windows Download for Mac (Intel) Download for Mac (Apple Silicon) Download for Linux

View other products from CyberAutomation.com.au →

AI Security Vetting for Australia 🇦🇺 and New Zealand 🇳🇿