make referenced_filenames required phrases

[armijnhemel]

Short Description

It seems that the referenced_filenames are ignored by the license scan, while these should be required phrases.

Possible Labels

• new feature

Select Category

• Enhancement
• Add License/Copyright
• Scan Feature
• Packaging
• Documentation
• Expand Support
• Other

Describe the Update

Currently referenced_filenames from .RULE files seem to be ignored by the license scanner, while these should be required phrases. An example of a rule where this is going wrong is #4949.

How This Feature will help you/your organization

It would mean fewer false positives

Possible Solution/Implementation Details

Treat referenced_filenames as required phrases.

Example/Links if Any

Can you help with this Feature

[armijnhemel]

My reasoning: if scancode includes the license results for referenced_filenames (which is, in my opinion, absolutely wrong) then you want the names of the referenced filenames to be in the license text. Imagine that there is a rule like this:

See file COPYING.

and there is a partial match to a rule that has LICENSE (but not COPYING) in the referenced_filenames, then including the results for LICENSE would be very wrong. Therefore, anything named in referenced_filenames should become part of the required phrases.

[armijnhemel]

So here is a good example. I just scanned this file: https://github.com/mirror/busybox/blob/master/util-linux/flock.c

The header is:

/*
 * Copyright (C) 2010 Timo Teras <timo.teras@iki.fi>
 *
 * This is free software, licensed under the GNU General Public License v2.
 */

As you can see, there is no reference to a local file.

The scancode result is:

      "license_detections": [
        {
          "license_expression": "gpl-2.0",
          "license_expression_spdx": "GPL-2.0-only",
          "matches": [
            {
              "license_expression": "gpl-2.0",
              "license_expression_spdx": "GPL-2.0-only",
              "from_file": "busybox-1.34.1/util-linux/flock.c",
              "start_line": 4,
              "end_line": 4,
              "matcher": "3-seq",
              "score": 70.59,
              "matched_length": 12,
              "match_coverage": 70.59,
              "rule_relevance": 100,
              "rule_identifier": "gpl-2.0_849.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/gpl-2.0_849.RULE"
            },
            {
              "license_expression": "gpl-2.0",
              "license_expression_spdx": "GPL-2.0-only",
              "from_file": "busybox-1.34.1/LICENSE",
              "start_line": 1,
              "end_line": 6,
              "matcher": "2-aho",
              "score": 100.0,
              "matched_length": 45,
              "match_coverage": 100.0,
              "rule_relevance": 100,
              "rule_identifier": "gpl-2.0_only2.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/gpl-2.0_only2.RULE"
            },
            {
              "license_expression": "gpl-2.0",
              "license_expression_spdx": "GPL-2.0-only",
              "from_file": "busybox-1.34.1/LICENSE",
              "start_line": 9,
              "end_line": 348,
              "matcher": "2-aho",
              "score": 100.0,
              "matched_length": 2931,
              "match_coverage": 100.0,
              "rule_relevance": 100,
              "rule_identifier": "gpl-2.0_1022.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/gpl-2.0_1022.RULE"
            }
          ],
          "identifier": "gpl_2_0-75e5fe36-cc99-32e8-5fa1-d70543ad2e0b",
          "detection_log": [
            "unknown-reference-to-local-file"
          ]
        }
     ],

The rule that partially matched is: https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/rules/gpl-2.0_849.RULE

---
license_expression: gpl-2.0
is_license_notice: yes
relevance: 100
referenced_filenames:
    - LICENSE
---

This is free software, licensed under the {{GNU General Public License v2}}.
See /LICENSE for more information.

In the header there is NO reference to LICENSE, but because of the partial match the results of the file LICENSE are included in the results when scanning this file. That's very wrong (and yes, in this case they are both GPL2, but that is besides the point).

If LICENSE would have been a required phrase, then this wouldn't have happened.

[armijnhemel]

As an experiment I made LICENSE a required word, rebuilt the index, scanned and that definitely helps as the wrong results are no longer included:

      "license_detections": [
        {
          "license_expression": "gpl-2.0",
          "license_expression_spdx": "GPL-2.0-only",
          "matches": [
            {
              "license_expression": "gpl-2.0",
              "license_expression_spdx": "GPL-2.0-only",
              "from_file": "busybox-1.34.1/util-linux/flock.c",
              "start_line": 4,
              "end_line": 4,
              "matcher": "2-aho",
              "score": 100.0,
              "matched_length": 8,
              "match_coverage": 100.0,
              "rule_relevance": 100,
              "rule_identifier": "gpl-2.0_1045.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/gpl-2.0_1045.RULE"
            }
          ],
          "identifier": "gpl_2_0-4756f608-a844-a9d7-a257-3b8286f3aa04"
        }
      ],