add huggingface id to .LICENSE yaml frontmatter

[armijnhemel]

Short Description

Huggingface uses different descriptions than in SPDX or other places. Adding a huggingface id to the .LICENSE yaml frontmatter might make referencing things easier.

Possible Labels

• new feature

Select Category

• Enhancement
• Add License/Copyright
• Scan Feature
• Packaging
• Documentation
• Expand Support
• Other

Describe the Update

Inspired by #4900 : huggingface calls this "apple-sample-code-license" (as seen when browsing the HuggingFace website for the project linked in #4900 ). This isn't the same name as used in SPDX or the ScanCode licensedb.

Just as there already is spdx_license_key it might make sense to add something like huggingface_license_key.

How This Feature will help you/your organization

I think that the onslaught of licenses coming from huggingface in the near future will make this a useful feature.

Possible Solution/Implementation Details

A good first start would be to add data from this list:

https://huggingface.co/docs/hub/repositories-licenses

Example/Links if Any

Can you help with this Feature

[mjherzog]

The SPDX license id has special standing because of its widespread adoption as a standard. Adding specific keys for other ecosystems that use non-standard license naming is potentially a slippery slope to special license key proliferation. For other cases we have historically used keywords in the DejaCode license record so a better solution might be to continue using the keyword/alias approach and publish that data to ScanCode LicenseDB.
The longer term solution is to help huggingface to use standard license keys in general.

[armijnhemel]

Some other keyword would also help for sure.

[mjherzog]

@DennisClark @AyanSinhaMahapatra I am interested in your feedback on this idea.

[DennisClark]

we should encourage huggingface to use standard license keys

[armijnhemel]

we should encourage huggingface to use standard license keys

So what if this isn't going to happen? @pombredanne

[mjherzog]

We encourage them to use ScanCode LicenseDB - because we are much more responsive than SPDX to add new licenses.
We cannot start adding data fields for every org that has poor license naming practices.

[pombredanne]

@armijnhemel Until now for anything beyond SPDX and the OSI, we have erred towards having either rules for alternate ids or using one of key_aliases rather than tracking each new if in a new field. For instance with Fedora license ids, we do not have a new field for that.

See

scancode-toolkit/src/licensedcode/models.py

Line 283 in 6570c13

key_aliases = attr.ib(

But that feature is not used anywhere in the data and not used in code either, only an attribute and nothing more.

We could nicely use that and also add a rule for "Apple-Sample-Code-License"

We could also starting using key aliases for detection, but that would require we ensure these are globally unique in the database across all keys.

[armijnhemel]

@armijnhemel Until now for anything beyond SPDX and the OSI, we have erred towards having either rules for alternate ids or using one of key_aliases rather than tracking each new if in a new field. For instance with Fedora license ids, we do not have a new field for that.

See

scancode-toolkit/src/licensedcode/models.py

Line 283 in 6570c13
key_aliases = attr.ib(

But that feature is not used anywhere in the data and not used in code either, only an attribute and nothing more.

We could nicely use that and also add a rule for "Apple-Sample-Code-License"

We could also starting using key aliases for detection, but that would require we ensure these are globally unique in the database across all keys.

All I want is a field that basically indicates "hey, this is what HuggingFace calls it". Nothing more, nothing less. It should be quite easy to do (we are talking about 40 or so, see https://huggingface.co/docs/hub/repositories-licenses ). These values are used in queries on the HuggingFace website (and perhaps also in other places?) so it would, in my opinion, be useful to record it.

[mjherzog]

We should/can not add a field for each repo or org that uses an idiosyncratic license list!

[pombredanne]

We should/can not add a field for each repo or org that uses an idiosyncratic license list!

@mjherzog actually may be we could, this would be a small data structure change and we could generalize that for OSI and others, for instance, something like this:

key: aml
short_name: Apple MIT License
name: Apple Sample Code License 2006
category: Permissive
owner: Apple
homepage_url: https://fedoraproject.org/wiki/Licensing/Apple_MIT_License
spdx_license_key: AML
other_urls:
    - https://huggingface.co/apple/OpenELM/blob/main/LICENSE
other_keys:
  - source_url: https://huggingface.co/docs/hub/repositories-licenses
    key: foo
  - source_url: https://opensource.org/license/cvw
    key: cvw

Let me reach out to HF too

[armijnhemel]

I apologize if I wasn't clear, but I don't want any changes to the code, just to the YAML frontmatter, so it becomes easier for cross referencing with other data sources.

I could imagine something like (building on what @pombredanne said):

- origin: huggingface
  key: some-stupid-license-name

We are talking about 80 license names (just verified) which would be about 3% of the current license texts in scancode. Adding this data would make cross referencing with other sources much easier as I would be able to do it programmatically, instead of first having to do a manual lookup and recording the data anyway to be used by a script.

And yes, HuggingFace should be using standardized licenses, but they aren't and I don't see this happening soon as their notation is used in different parts of the HuggingFace infrastructure code and people are relying on it.

Also see #4914 where HuggingFace is using a different name than the license text.