Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
202 changes: 202 additions & 0 deletions docs/nsfw_praxis.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the whole thing is missing a little bit the point.

Art like this actually should not really be classified as adult content/violence/nsfw.

I understand when does and is okay that it get's classified but if you want a test, Id rather recommend to take real pornographic and especially borderline content. A woman in a bikini is most likely not NSFW but nude at the beach yes.
Pure violence, gore, weapons, nsfw. ist also part of this. We don't want to be too picky; finding a good threshold is important.

We will run the model on AI inputs and AI outptus so it must be cheap and fast.
This is the necessary first step for having public content on our website.

The falcon model looks good please check if we can use it (licence) wise. It might be too heavy though.

An option might be to have it like fast detector if confidence < 0.8 --> Falcon or something similar.

Have you had a look at shieldgemma? Looks promising
https://huggingface.co/google/shieldgemma-2-4b-it
https://huggingface.co/google/shieldgemma-2b

It looks like this one can not only detect nude,violence etc. but also works for text which is great!
We can define then a policy

When porting it, check-out the qwen-model repository and the new_registry dev of APIPod because the syntax of models changed a lot.

Like always instead of blind activism think what we actually want to do. Then do research.
Then take action.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrote in f7748a3. Cascade added (bhky gate, Falcon on uncertain band 0.2 to 0.8, ShieldGemma-2-4B on caller ask, 2B for text). Sourcing plan: alex000kim/nsfw_data_scraper (porn + sexy for explicit and borderline), RealLifeViolenceDataset for gore. OK to pull?

Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
# NSFW filter · Azure #164 · praxis v2

Reset after the review comment on `docs/nsfw_praxis.md:1` (9 Jul). The v1
protocol used classical art nudes as the borderline set. Wrong framing: art
is not the target signal. This v2 rewrites the scope, the model shortlist,
and the dataset plan before rerunning any bench.

Date: 14 Jul 2026.

## What we filter and why

The service runs on every AI input and every AI output. Cost per request is
the ceiling, not accuracy in isolation. It gates public content on the
website, so a false negative is worse than a false positive on borderline.

Target signals:

- explicit sexual content (photorealistic or generated)
- borderline nudity on real bodies (nude beach, topless photography), not
paintings
- graphic violence, gore, weapons in threatening context
- text prompts asking for any of the above (input side)

Pass-through, not target:

- artistic nudity (paintings, sculpture) unless graphic
- medical, educational, ethnographic material
- swimwear, lingerie catalog, dance, contact sport

## Model shortlist

Four candidates, two lanes (image and text). Numbers below are from HF
model cards fetched 14 Jul, or from the v1 bench in the appendix.

| Model | Params | Modality | Categories | License | Bench notes | Role |
|---|---|---|---|---|---|---|
| `bhky/opennsfw2` | Yahoo openNSFW2 port, few M | image | binary NSFW prob | MIT (bhky wrapper), upstream Yahoo terms to double-check before ship | 265 ms warm CPU M1 | fast image triage |
| `Falconsai/nsfw_image_detection` | 85.8 M ViT | image | binary normal / nsfw, 98.04% eval acc | Apache 2.0 | 200 ms warm CPU M1 | accurate image |
| `google/shieldgemma-2-4b-it` | 4 B | image + text | sexual explicit, dangerous, violence / gore, per-policy score | Gemma ToU | no CPU numbers, TPU-trained | policy-based, escalation only |
| `google/shieldgemma-2b` | 2 B | text only | sexual explicit, dangerous, hate, harassment | Gemma ToU | no numbers | text prompt filter |

Bhky and Falcon are the two image-side workhorses. ShieldGemma 4B is heavy
(4 B params, no CPU latency disclosed) so it does not belong on the hot
path. Useful when a caller asks for per-category output (nude vs violence
vs weapon) that the two binary models cannot give. ShieldGemma 2B covers
the text side, which the ticket did not scope originally but which the
review comment on v1 pulled in as part of the same filter surface.

## Cascade proposal

Two lanes.

```
image path:
input --> bhky (prob)
prob < 0.2 --> pass
prob > 0.8 --> flag nsfw
0.2..0.8 --> Falconsai
normal --> pass
nsfw --> flag
caller asked for per-category --> ShieldGemma-2-4B-IT
with per-policy prompt

text path:
input --> ShieldGemma-2B, run 4 policies
any Yes --> block prompt
all No --> pass
```

Bhky is cheap enough to run on every request. Falcon runs only on the
uncertain band, keeps p99 controlled. ShieldGemma 4B runs only on explicit
caller ask, never on the default path. ShieldGemma 2B is a separate text
call, single roundtrip per prompt.

Thresholds `0.2` and `0.8` are placeholders. Actual numbers come from the
v2 bench in Tanda B, based on the ROC on the new dataset.

## Test protocol v2

Categories, 10 images each unless flagged. Text prompts are 20 strings.

- `safe_real`: everyday photos, no ambiguity
- `porn_explicit`: real explicit content (5 only, minimum to sanity check)
- `borderline_nude_flag`: nude beach, topless real bodies, non-artistic
- `borderline_safe_pass`: bikini, swimwear, lingerie catalog, contact sport
- `violence_gore`: real injury, gore (5 only)
- `weapons`: firearms and blades in threatening framing
- `text_prompts`: 20 strings, half harmful (per the 4 ShieldGemma categories)
and half benign

Image side scored on bhky + Falconsai + ShieldGemma-2-4B-IT.
Text side scored on ShieldGemma-2B.

For each image model, report: score, latency warm, latency cold, correct
call at threshold 0.5. For ShieldGemma 4B, report per-policy score.

## Sourcing plan (pending owner sign-off, blocks Tanda B)

Sensitive material. This is the recommended plan, not started until
signed off in the PR reply.

Images and text:

- Explicit and borderline nudity: `alex000kim/nsfw_data_scraper` on
GitHub. Five classes (`porn`, `hentai`, `sexy`, `neutral`, `drawings`),
URL lists checked into the repo, run the download script to pull actual
images to disk. Cited baseline in most NSFW detector papers. Author
flag: dataset is noisy, so sample by hand into the v2 categories
(`porn_explicit` from `porn`, `borderline_nude_flag` from the nude end
of `sexy`, `borderline_safe_pass` from the bikini end of `sexy` plus
neutral stock).
- Violence and gore: `RealLifeViolenceDataset` on Kaggle. 2000 videos,
1000 violence 1000 non-violence, extract frames.
- Weapons: sample from a Kaggle weapon-detection dataset (firearms,
blades). Concrete name confirmed at build time.
- Safe real photos: Unsplash / Pexels API with adult flag off.
- Text prompts: author in repo, 10 harmful and 10 benign, cross the four
ShieldGemma text categories.

Not on the table: LAION-NSFW (Stanford CSAM report Dec 2023, LAION
guidance is research-only, not production), NudeNet training set
(detector is public, training data is not).

Runs on the laptop, no cloud runner needed for the image side. Cache goes
in a repo-ignored path.

## Blockers before rerun

- Sign-off on the sourcing plan above (question in the PR#20 reply).
- Gemma ToU accepted on HF for the account that pulls ShieldGemma.
- Confirm whether ShieldGemma 4B runs on M1 with int8 quant or needs a
cloud runner (Modal / RunPod).

## Frictions for the APIPod integration prompt

1. `tmplink/nsfw_detector` still 404 on HF. Substitute Falconsai
(Apache 2.0, verified 14 Jul).
2. TF / torch wheels for Python 3.14 do not exist yet. Worker image pins
3.11 or 3.12.
3. `opennsfw2` first predict downloads weights to `~/.opennsfw2/`. Worker
needs a persistent cache mount or the download runs on every cold start.
4. ShieldGemma pulls from HF gated repos. Worker HF token needs Gemma ToU
accepted per account.
5. Model registration syntax changed on APIPod `origin/new_registry` branch
(refactor 16 Jun). Port target is that branch, not `dev`.

## Open questions

1. Response schema. Ship one canonical `nsfw_score` binary, or expose the
3 ShieldGemma categories granularly to the caller when they ask?
2. Text lane scope. Is ShieldGemma 2B in this ticket, or split into a
sibling ticket? Different model family, different endpoint shape.

## Appendix · v1 baseline (9 Jul)

Kept as sanity check that the two image models return low scores on
synthetic safe content and that both can fire on art nudes. Wrong dataset
for the actual filter decision, right dataset to prove the plumbing works.

### bhky on 5 safe PIL images

| Image | NSFW prob | Latency (ms) | Note |
|---|---|---|---|
| 1_plain_green.jpg | 0.0003 | 2114.7 | cold |
| 2_circle_bw.jpg | 0.0119 | 242.2 | warm |
| 3_text_gradient.jpg | 0.0113 | 276.9 | warm |
| 4_checker.jpg | 0.0033 | 274.9 | warm |
| 5_landscape.jpg | 0.0000 | 267.2 | warm |

Cold start 2.1 s, warm mean 265 ms.

### Falconsai on 5 safe PIL images

| Image | Label | Score | Latency (ms) | Note |
|---|---|---|---|---|
| 1_plain_green.jpg | normal | 0.9992 | 2106.6 | cold |
| 2_circle_bw.jpg | normal | 0.9986 | 215.7 | warm |
| 3_text_gradient.jpg | normal | 0.9986 | 179.5 | warm |
| 4_checker.jpg | normal | 0.9967 | 204.0 | warm |
| 5_landscape.jpg | normal | 0.9995 | 191.8 | warm |

Cold start 2.1 s, warm mean 198 ms.

### bhky on 5 art nudes (wrong dataset, kept for the record)

| Image | NSFW prob | Verdict at 0.5 |
|---|---|---|
| 1_botticelli_venus.jpg | 0.0626 | safe |
| 2_michelangelo_david.jpg | 0.8723 | flag |
| 3_titian_venus_urbino.jpg | 0.9829 | flag |
| 4_rubens_three_graces.jpg | 0.9371 | flag |
| 5_doryphoros_statue.jpg | 0.4060 | safe |

### Falconsai on 5 art nudes (wrong dataset, kept for the record)

| Image | Label | Score |
|---|---|---|
| 1_botticelli_venus.jpg | normal | 0.9984 |
| 2_michelangelo_david.jpg | normal | 0.9997 |
| 3_titian_venus_urbino.jpg | nsfw | 0.6666 |
| 4_rubens_three_graces.jpg | nsfw | 0.9997 |
| 5_doryphoros_statue.jpg | normal | 0.9998 |
Loading