[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader

[3nesdeniz]

Description

Closes #2171.

This PR adds a first-class PyRIT loader for the public Turkish Conversation Prompt-Injection Dataset (CC BY 4.0). The dataset contains 750 curated Turkish examples across 10 prompt-injection families, including 150 benign boundary cases paired with related attacks. It is intended to support both attack detection and over-refusal testing in Turkish LLM security workflows.

The new loader:

• defaults to attack examples from all published splits for red-team use;
• provides typed filters for attack, benign, or all labels;
• provides typed train, validation, test, and combined-split selection;
• supports filtering by one or more attack families;
• preserves row-level provenance, including source context, pair ID, source type, split, label, and attack family;
• maps attack families to harm_categories while leaving benign examples uncategorized;
• supports the existing Hugging Face token and cache behavior; and
• raises an explicit error when a valid filter produces an empty dataset.

The loader is registered with SeedDatasetProvider, documented in the dataset-loading guide, and linked to a bibliography entry. Attack examples contain adversarial instructions and should only be used in controlled, authorized security testing.

Tests and Documentation

• Added 17 focused unit tests covering defaults, label filters, multi-family filtering, all split mappings, provenance preservation, empty-result behavior, enum validation, token handling, and discovery metadata.
• uv run pytest tests/unit/datasets/test_turkish_conversation_prompt_injection_dataset.py tests/unit/datasets/test_seed_dataset_provider.py -q: 206 passed.
• Full unit suite: 10,306 passed, 119 skipped.
• Targeted pre-commit run for all changed files: passed, including Ruff, notebook Ruff, ty, notebook sanitation, documentation structure, and secret detection.
• jupytext --test-strict for the updated dataset-loading notebook: passed.
• The updated notebook was executed end to end against the live public dataset.
• Targeted strict Jupyter Book site build for the updated loading guide and bibliography: passed.
• Live dataset smoke validation confirmed 750 rows, the published 600/150 label distribution, 530/100/120 split distribution, all 10 attack families, and provider discovery.

AI assistance was used for implementation drafting and test planning. I reviewed the resulting code and documentation, validated them against the live dataset, and ran all checks listed above.

[3nesdeniz]

@microsoft-github-policy-service agree

[romanlutz]

Is there a paper associated with this? How were they collected or curated? Do they actually work on some model?

[3nesdeniz]

Thanks for the questions. There is no associated paper or preprint at this stage. The citation refers to the versioned Hugging Face dataset release itself.

The dataset was created through a paired curation process. I defined ten prompt-injection attack families and prepared 150 attack examples. For every attack, I created a matched legitimate request that remains close in topic, context, and vocabulary but does not attempt to override instructions or cross a trust boundary. These boundary pairs were combined with 450 additional legitimate requests, resulting in 600 benign examples and 150 attacks.

This structure is intentional. A detector should not classify a request as malicious merely because it contains terms such as "system prompt," "admin," or "API key." It must distinguish between legitimate user intent and an actual attempt to override authority or extract protected information. The matched benign examples serve as hard negatives and are intended to support false-positive analysis and reduction.

The attack examples were also executed in AltayDuel, AltaySec's continuously operating AI-vs-AI prompt-injection arena. AltayDuel runs Red and Blue agents across multiple model providers and records the complete conversation transcript, model and provider metadata, and judge verdicts. The attack examples included in this release produced successful prompt-injection outcomes under the arena's documented evaluation criteria.

These results should not be interpreted as universal bypasses against every model or configuration. Prompt-injection success remains dependent on the target model, system prompt, available tools, and surrounding application context.

The dataset repository also includes deterministic pair-preserving splits and validation for schema consistency, duplicate rows, pair integrity, split leakage, attack-family balance, and common sensitive-data patterns.