-
-
Notifications
You must be signed in to change notification settings - Fork 2
add nsfw praxis notes for Azure #164 #20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
naattiiiiiiiii
wants to merge
3
commits into
main
Choose a base branch
from
docs/nsfw-praxis-164
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,202 @@ | ||
| # NSFW filter · Azure #164 · praxis v2 | ||
|
|
||
| Reset after the review comment on `docs/nsfw_praxis.md:1` (9 Jul). The v1 | ||
| protocol used classical art nudes as the borderline set. Wrong framing: art | ||
| is not the target signal. This v2 rewrites the scope, the model shortlist, | ||
| and the dataset plan before rerunning any bench. | ||
|
|
||
| Date: 14 Jul 2026. | ||
|
|
||
| ## What we filter and why | ||
|
|
||
| The service runs on every AI input and every AI output. Cost per request is | ||
| the ceiling, not accuracy in isolation. It gates public content on the | ||
| website, so a false negative is worse than a false positive on borderline. | ||
|
|
||
| Target signals: | ||
|
|
||
| - explicit sexual content (photorealistic or generated) | ||
| - borderline nudity on real bodies (nude beach, topless photography), not | ||
| paintings | ||
| - graphic violence, gore, weapons in threatening context | ||
| - text prompts asking for any of the above (input side) | ||
|
|
||
| Pass-through, not target: | ||
|
|
||
| - artistic nudity (paintings, sculpture) unless graphic | ||
| - medical, educational, ethnographic material | ||
| - swimwear, lingerie catalog, dance, contact sport | ||
|
|
||
| ## Model shortlist | ||
|
|
||
| Four candidates, two lanes (image and text). Numbers below are from HF | ||
| model cards fetched 14 Jul, or from the v1 bench in the appendix. | ||
|
|
||
| | Model | Params | Modality | Categories | License | Bench notes | Role | | ||
| |---|---|---|---|---|---|---| | ||
| | `bhky/opennsfw2` | Yahoo openNSFW2 port, few M | image | binary NSFW prob | MIT (bhky wrapper), upstream Yahoo terms to double-check before ship | 265 ms warm CPU M1 | fast image triage | | ||
| | `Falconsai/nsfw_image_detection` | 85.8 M ViT | image | binary normal / nsfw, 98.04% eval acc | Apache 2.0 | 200 ms warm CPU M1 | accurate image | | ||
| | `google/shieldgemma-2-4b-it` | 4 B | image + text | sexual explicit, dangerous, violence / gore, per-policy score | Gemma ToU | no CPU numbers, TPU-trained | policy-based, escalation only | | ||
| | `google/shieldgemma-2b` | 2 B | text only | sexual explicit, dangerous, hate, harassment | Gemma ToU | no numbers | text prompt filter | | ||
|
|
||
| Bhky and Falcon are the two image-side workhorses. ShieldGemma 4B is heavy | ||
| (4 B params, no CPU latency disclosed) so it does not belong on the hot | ||
| path. Useful when a caller asks for per-category output (nude vs violence | ||
| vs weapon) that the two binary models cannot give. ShieldGemma 2B covers | ||
| the text side, which the ticket did not scope originally but which the | ||
| review comment on v1 pulled in as part of the same filter surface. | ||
|
|
||
| ## Cascade proposal | ||
|
|
||
| Two lanes. | ||
|
|
||
| ``` | ||
| image path: | ||
| input --> bhky (prob) | ||
| prob < 0.2 --> pass | ||
| prob > 0.8 --> flag nsfw | ||
| 0.2..0.8 --> Falconsai | ||
| normal --> pass | ||
| nsfw --> flag | ||
| caller asked for per-category --> ShieldGemma-2-4B-IT | ||
| with per-policy prompt | ||
|
|
||
| text path: | ||
| input --> ShieldGemma-2B, run 4 policies | ||
| any Yes --> block prompt | ||
| all No --> pass | ||
| ``` | ||
|
|
||
| Bhky is cheap enough to run on every request. Falcon runs only on the | ||
| uncertain band, keeps p99 controlled. ShieldGemma 4B runs only on explicit | ||
| caller ask, never on the default path. ShieldGemma 2B is a separate text | ||
| call, single roundtrip per prompt. | ||
|
|
||
| Thresholds `0.2` and `0.8` are placeholders. Actual numbers come from the | ||
| v2 bench in Tanda B, based on the ROC on the new dataset. | ||
|
|
||
| ## Test protocol v2 | ||
|
|
||
| Categories, 10 images each unless flagged. Text prompts are 20 strings. | ||
|
|
||
| - `safe_real`: everyday photos, no ambiguity | ||
| - `porn_explicit`: real explicit content (5 only, minimum to sanity check) | ||
| - `borderline_nude_flag`: nude beach, topless real bodies, non-artistic | ||
| - `borderline_safe_pass`: bikini, swimwear, lingerie catalog, contact sport | ||
| - `violence_gore`: real injury, gore (5 only) | ||
| - `weapons`: firearms and blades in threatening framing | ||
| - `text_prompts`: 20 strings, half harmful (per the 4 ShieldGemma categories) | ||
| and half benign | ||
|
|
||
| Image side scored on bhky + Falconsai + ShieldGemma-2-4B-IT. | ||
| Text side scored on ShieldGemma-2B. | ||
|
|
||
| For each image model, report: score, latency warm, latency cold, correct | ||
| call at threshold 0.5. For ShieldGemma 4B, report per-policy score. | ||
|
|
||
| ## Sourcing plan (pending owner sign-off, blocks Tanda B) | ||
|
|
||
| Sensitive material. This is the recommended plan, not started until | ||
| signed off in the PR reply. | ||
|
|
||
| Images and text: | ||
|
|
||
| - Explicit and borderline nudity: `alex000kim/nsfw_data_scraper` on | ||
| GitHub. Five classes (`porn`, `hentai`, `sexy`, `neutral`, `drawings`), | ||
| URL lists checked into the repo, run the download script to pull actual | ||
| images to disk. Cited baseline in most NSFW detector papers. Author | ||
| flag: dataset is noisy, so sample by hand into the v2 categories | ||
| (`porn_explicit` from `porn`, `borderline_nude_flag` from the nude end | ||
| of `sexy`, `borderline_safe_pass` from the bikini end of `sexy` plus | ||
| neutral stock). | ||
| - Violence and gore: `RealLifeViolenceDataset` on Kaggle. 2000 videos, | ||
| 1000 violence 1000 non-violence, extract frames. | ||
| - Weapons: sample from a Kaggle weapon-detection dataset (firearms, | ||
| blades). Concrete name confirmed at build time. | ||
| - Safe real photos: Unsplash / Pexels API with adult flag off. | ||
| - Text prompts: author in repo, 10 harmful and 10 benign, cross the four | ||
| ShieldGemma text categories. | ||
|
|
||
| Not on the table: LAION-NSFW (Stanford CSAM report Dec 2023, LAION | ||
| guidance is research-only, not production), NudeNet training set | ||
| (detector is public, training data is not). | ||
|
|
||
| Runs on the laptop, no cloud runner needed for the image side. Cache goes | ||
| in a repo-ignored path. | ||
|
|
||
| ## Blockers before rerun | ||
|
|
||
| - Sign-off on the sourcing plan above (question in the PR#20 reply). | ||
| - Gemma ToU accepted on HF for the account that pulls ShieldGemma. | ||
| - Confirm whether ShieldGemma 4B runs on M1 with int8 quant or needs a | ||
| cloud runner (Modal / RunPod). | ||
|
|
||
| ## Frictions for the APIPod integration prompt | ||
|
|
||
| 1. `tmplink/nsfw_detector` still 404 on HF. Substitute Falconsai | ||
| (Apache 2.0, verified 14 Jul). | ||
| 2. TF / torch wheels for Python 3.14 do not exist yet. Worker image pins | ||
| 3.11 or 3.12. | ||
| 3. `opennsfw2` first predict downloads weights to `~/.opennsfw2/`. Worker | ||
| needs a persistent cache mount or the download runs on every cold start. | ||
| 4. ShieldGemma pulls from HF gated repos. Worker HF token needs Gemma ToU | ||
| accepted per account. | ||
| 5. Model registration syntax changed on APIPod `origin/new_registry` branch | ||
| (refactor 16 Jun). Port target is that branch, not `dev`. | ||
|
|
||
| ## Open questions | ||
|
|
||
| 1. Response schema. Ship one canonical `nsfw_score` binary, or expose the | ||
| 3 ShieldGemma categories granularly to the caller when they ask? | ||
| 2. Text lane scope. Is ShieldGemma 2B in this ticket, or split into a | ||
| sibling ticket? Different model family, different endpoint shape. | ||
|
|
||
| ## Appendix · v1 baseline (9 Jul) | ||
|
|
||
| Kept as sanity check that the two image models return low scores on | ||
| synthetic safe content and that both can fire on art nudes. Wrong dataset | ||
| for the actual filter decision, right dataset to prove the plumbing works. | ||
|
|
||
| ### bhky on 5 safe PIL images | ||
|
|
||
| | Image | NSFW prob | Latency (ms) | Note | | ||
| |---|---|---|---| | ||
| | 1_plain_green.jpg | 0.0003 | 2114.7 | cold | | ||
| | 2_circle_bw.jpg | 0.0119 | 242.2 | warm | | ||
| | 3_text_gradient.jpg | 0.0113 | 276.9 | warm | | ||
| | 4_checker.jpg | 0.0033 | 274.9 | warm | | ||
| | 5_landscape.jpg | 0.0000 | 267.2 | warm | | ||
|
|
||
| Cold start 2.1 s, warm mean 265 ms. | ||
|
|
||
| ### Falconsai on 5 safe PIL images | ||
|
|
||
| | Image | Label | Score | Latency (ms) | Note | | ||
| |---|---|---|---|---| | ||
| | 1_plain_green.jpg | normal | 0.9992 | 2106.6 | cold | | ||
| | 2_circle_bw.jpg | normal | 0.9986 | 215.7 | warm | | ||
| | 3_text_gradient.jpg | normal | 0.9986 | 179.5 | warm | | ||
| | 4_checker.jpg | normal | 0.9967 | 204.0 | warm | | ||
| | 5_landscape.jpg | normal | 0.9995 | 191.8 | warm | | ||
|
|
||
| Cold start 2.1 s, warm mean 198 ms. | ||
|
|
||
| ### bhky on 5 art nudes (wrong dataset, kept for the record) | ||
|
|
||
| | Image | NSFW prob | Verdict at 0.5 | | ||
| |---|---|---| | ||
| | 1_botticelli_venus.jpg | 0.0626 | safe | | ||
| | 2_michelangelo_david.jpg | 0.8723 | flag | | ||
| | 3_titian_venus_urbino.jpg | 0.9829 | flag | | ||
| | 4_rubens_three_graces.jpg | 0.9371 | flag | | ||
| | 5_doryphoros_statue.jpg | 0.4060 | safe | | ||
|
|
||
| ### Falconsai on 5 art nudes (wrong dataset, kept for the record) | ||
|
|
||
| | Image | Label | Score | | ||
| |---|---|---| | ||
| | 1_botticelli_venus.jpg | normal | 0.9984 | | ||
| | 2_michelangelo_david.jpg | normal | 0.9997 | | ||
| | 3_titian_venus_urbino.jpg | nsfw | 0.6666 | | ||
| | 4_rubens_three_graces.jpg | nsfw | 0.9997 | | ||
| | 5_doryphoros_statue.jpg | normal | 0.9998 | |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the whole thing is missing a little bit the point.
Art like this actually should not really be classified as adult content/violence/nsfw.
I understand when does and is okay that it get's classified but if you want a test, Id rather recommend to take real pornographic and especially borderline content. A woman in a bikini is most likely not NSFW but nude at the beach yes.
Pure violence, gore, weapons, nsfw. ist also part of this. We don't want to be too picky; finding a good threshold is important.
We will run the model on AI inputs and AI outptus so it must be cheap and fast.
This is the necessary first step for having public content on our website.
The falcon model looks good please check if we can use it (licence) wise. It might be too heavy though.
An option might be to have it like fast detector if confidence < 0.8 --> Falcon or something similar.
Have you had a look at shieldgemma? Looks promising
https://huggingface.co/google/shieldgemma-2-4b-it
https://huggingface.co/google/shieldgemma-2b
It looks like this one can not only detect nude,violence etc. but also works for text which is great!
We can define then a policy
When porting it, check-out the qwen-model repository and the new_registry dev of APIPod because the syntax of models changed a lot.
Like always instead of blind activism think what we actually want to do. Then do research.
Then take action.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rewrote in
f7748a3. Cascade added (bhky gate, Falcon on uncertain band 0.2 to 0.8, ShieldGemma-2-4B on caller ask, 2B for text). Sourcing plan:alex000kim/nsfw_data_scraper(porn + sexy for explicit and borderline),RealLifeViolenceDatasetfor gore. OK to pull?