OR identified as AND

[hesa]

Description

Looking at Cairo 1.16 and the following file https://cgit.freedesktop.org/cairo/tree/src/cairo-analysis-surface-private.h?h=1.16.0

The file has the following license text at the top:

..... snip
 * This library is free software; you can redistribute it and/or                                                                                                                                                   
 * modify it either under the terms of the GNU Lesser General Public                                                                                                                                               
 * License version 2.1 as published by the Free Software Foundation                                                                                                                                                
 * (the "LGPL") or, at your option, under the terms of the Mozilla                                                                                                                                                 
 * Public License Version 1.1 (the "MPL").                                                                                                                                                                  
.... snip

The interesting part is "GNU Lesser General Public License version 2.1 as published by the Free Software Foundation (the "LGPL") or, at your option, under the terms of the Mozilla Public License Version 1.1 (the "MPL"). "

Reported license Scancode version 31.2.6:
"license_expressions": [ "lgpl-2.1 OR mpl-1.1" ]

Reported license Scancode version 32.0.6:

      "detected_license_expression": "lgpl-2.1 AND mpl-1.1",
      "detected_license_expression_spdx": "LGPL-2.1-only AND MPL-1.1",

So, Scancode 32.0.6 does not correctly identify "or, at your option" as "OR".

How To Reproduce

Download and unpack https://www.cairographics.org/releases/cairo-1.16.0.tar.xz

My settings/command line args for 32.0.6:

  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "31.2.6",
      "options": {
        "input": [
          "cairo-1.16.0"
        ],
        "--classify": true,
        "--copyright": true,
        "--email": true,
        "--info": true,
        "--json-pp": "cairo-1.16.0-scan.json",
        "--license": true,
        "--license-clarity-score": true,
        "--license-text": true,
        "--license-text-diagnostics": true,
        "--package": true,
        "--processes": "16",
        "--summary": true
      },
....snip

My settings/command line args for 32.0.6:

  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "32.0.6",
      "options": {
        "input": [
          "cairo-1.16.0"
        ],
        "--classify": true,
        "--copyright": true,
        "--email": true,
        "--info": true,
        "--json-pp": "cairo-1.16.0-scancode-32.0.06-scan.json",
        "--license": true,
        "--license-clarity-score": true,
        "--license-text": true,
        "--license-text-diagnostics": true,
        "--package": true,
        "--processes": "16",
        "--summary": true
      },
 .... snip

System configuration

Scancode:

$ pip list | grep scancode
scancode-toolkit            32.0.6

OS:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

[DennisClark]

@AyanSinhaMahapatra I believe the issue as reported is correct:
Scancode 32.0.6 does not correctly identify "or, at your option" as "OR"

[DennisClark]

It might be tricky, since it is very similar to the wording used for "or later" licenses.

[pombredanne]

@hesa there is a reference to SCTK 31.2.6 and then 32.0.6 above .... I get this with 32.0.6:

- matches:
    - score: '94.64'
      matcher: 3-seq
      end_line: 29
      rule_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lgpl-2.1_or_mpl-1.1_2.RULE
      start_line: 4
      matched_text: |
         * This library is free software; you can redistribute it and/or
         * modify it either under the terms of the GNU Lesser General Public
         * License version 2.1 as published by the Free Software Foundation
         * (the "LGPL") or, at your option, under the terms of the Mozilla
         * Public License Version 1.1 (the "MPL"). If you do not alter this
         * notice, a recipient may use your version of this file under either
         * the MPL or the LGPL.
         *
         * You should have received a copy of the LGPL along with this library
         * in the file COPYING-LGPL-2.1; if not, write to the Free Software
         * Foundation, Inc., 51 Franklin Street, Suite 500, Boston, MA 02110-1335, USA
         * You should have received a copy of the MPL along with this library
         * in the file COPYING-MPL-1.1
         *
         * The contents of this file are subject to the Mozilla Public License
         * Version 1.1 (the "License"); you may not use this file except in
         * compliance with the License. You may obtain a copy of the License at
         * http://www.mozilla.org/MPL/
         *
         * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY
         * OF ANY KIND, either express or implied. See the LGPL or the MPL for
         * the specific language governing rights and limitations.
         *
         * The Original Code is the cairo graphics library.
         *
         * The Initial Developer of the Original Code is Keith Packard
      match_coverage: '94.64'
      matched_length: 212
      rule_relevance: 100
      rule_identifier: lgpl-2.1_or_mpl-1.1_2.RULE
      license_expression: lgpl-2.1 OR mpl-1.1
  identifier: lgpl_2_1_or_mpl_1_1-8f4efaf4-1022-63ec-f250-bae5e2a90bfd
  license_expression: lgpl-2.1 OR mpl-1.1

This is correct?? (but not perfect in my book, we will need an improve rule to ensure we match this exactly and no approximately)

[hesa]

This is weird ..... I get different results when scanning the single file compared to the entire project.

Below is the output from running a small script I wrote (see even further down below) to show the weirdness (I am not ruling out I have made a mistake).

About Scancode
==============

Scancode: 32.0.6

Scan results for file
=====================

Entire project: Cairo 1.16.0
----------------------------
lgpl-2.1 AND mpl-1.1  from rule: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lgpl-2.1_or_mpl-1.1_2.RULE
lgpl-2.1 AND mpl-1.1  from rule: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lgpl-2.1_alternative.RULE
lgpl-2.1 AND mpl-1.1  from rule: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/mpl-1.1.LICENSE

Single file: cairo-analysis-surface-private.h
---------------------------------------------
lgpl-2.1 OR mpl-1.1  from rule: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lgpl-2.1_or_mpl-1.1_2.RULE

Diff between header files
=========================

sdiff
-----

md5
---
fdba194b1e7e431389d45a625fd13169  cairo-1.16.0/src/cairo-analysis-surface-private.h
fdba194b1e7e431389d45a625fd13169  single_file/cairo-analysis-surface-private.h

Script:

#!/bin/bash

FILE_URL="https://cgit.freedesktop.org/cairo/plain/src/cairo-analysis-surface-private.h?h=1.16.0"
CAIRO_URL="https://www.cairographics.org/releases/cairo-1.16.0.tar.xz"

TMP_DIR=Scancode-3523
mkdir -p ${TMP_DIR}
cd ${TMP_DIR}

do_scan()
{
    scancode --classify --copyright --email --info --json-pp ${1}-scan.json \
             --license --license-clarity-score --license-text \
             --license-text-diagnostics --package --processes 16 --summary $1

}

scan_project()
{
    curl -LJO ${CAIRO_URL}
    xz -d cairo-1.16.0.tar.xz
    tar xvf cairo-1.16.0.tar

    do_scan cairo-1.16.0
}

scan_file()
{
    mkdir single_file
    curl -LJO ${FILE_URL}
    mv cairo-analysis-surface-private.h single_file
    
    do_scan single_file
}

scan_project
scan_file

echo "About Scancode"
echo "=============="
echo
echo -n "Scancode: "
cat cairo-1.16.0-scan.json |  jq -r .headers[0].tool_version
echo 

echo "Scan results for file"
echo "====================="
echo
echo "Entire project: Cairo 1.16.0"
echo "----------------------------"
cat cairo-1.16.0-scan.json |  jq -r '.files[] | select(.path=="cairo-1.16.0/src/cairo-analysis-surface-private.h") | .license_detections[] | "\(.license_expression)  from rule: \(.matches[].rule_url)"'
echo

echo "Single file: cairo-analysis-surface-private.h"
echo "---------------------------------------------"
cat single_file-scan.json |  jq -r '.files[] | select(.path=="single_file/cairo-analysis-surface-private.h") | .license_detections[] | "\(.license_expression)  from rule: \(.matches[].rule_url)"'
echo

echo "Diff between header files"
echo "========================="
echo
echo "sdiff"
echo "-----"
sdiff -s "cairo-1.16.0/src/cairo-analysis-surface-private.h"  "single_file/cairo-analysis-surface-private.h"
echo
echo "md5"
echo "---"
md5sum "cairo-1.16.0/src/cairo-analysis-surface-private.h"  "single_file/cairo-analysis-surface-private.h"
echo

[pombredanne]

This is weird ..... I get different results when scanning the single file compared to the entire project.

@AyanSinhaMahapatra could this be because we are following (incorrectly) the license file references?

[pombredanne]

@AyanSinhaMahapatra the issue is clearly because we follow references.
I have detailed the issues in #3547
Here we have two referenced filenames and if present they are followed and replace the license choice incorrectly by a the referenced licenses.

• This is a simplified zip of https://www.cairographics.org/releases/cairo-1.16.0.tar.xz cairo-1.16.0.zip with just the files we need. It could be made smaller
• this is the scan of the root of the extracted archive and it has incorrect AND cairo-multi.yml.txt
• this is the scan of the src/ of the extracted archive with only a single C file and it has the ocrrect OR cairo-single.yml.txt

[DennisClark]

This issue continues to be a problem. I scanned cairo-1.18.4 (from https://www.cairographics.org/releases/cairo-1.18.4.tar.xz) using scancode-toolkit-32.3.3 and it reported the following:

  "summary": {
    "declared_license_expression": "lgpl-2.1 AND lgpl-2.1-plus AND mpl-1.0 AND mpl-1.1",
    "license_clarity_score": {
      "score": 70,
      "declared_license": true,
      "identification_precision": true,
      "has_license_text": true,
      "declared_copyrights": true,
      "conflicting_license_categories": true,
      "ambiguous_compound_licensing": true
    }

even though there are multiple places in the project files that clearly declare the choice of lgpl-2.1 OR mpl-1.1. Scan results attached.

cairo-1.18.4-results.json.zip

[hesa]

Get the cairo source code and scan it:

$ curl -LJO "https://www.cairographics.org/releases/cairo-1.18.4.tar.xz"
$ tar xvf cairo-1.18.4.tar.xz
$ scancode -clipe --license-text --license-text-diagnostics --classify --license-clarity-score --todo --license-diagnostics --summary -n 16 --json-pp cairo-1.18.4-scan.json cairo-1.18.4

Looking at the report, using scarfer, with scancode version 32.5.0, and listing only files with the AND in between the licenses we can see that the problem is reduced to the following three files:

$ scarfer cairo-1.18.4-scan.json -dde -il "lgpl-2.1 AND mpl-1.1"
cairo-1.18.4/NEWS
cairo-1.18.4/README.md
cairo-1.18.4/util/cairo-script/COPYING

... so the header file above is NOT identified as lgpl-2.1 AND mpl-1.1 anymore. Same applies to all the source code :)

Looking closer at the three files above:

1. cairo-1.18.4/NEWS:

The most significant news with this release is that the license of
cairo has changed. It is now dual-licensed under the LGPL and the
MPL. For details see the COPYING file as well as COPYING-LGPL-2.1 and
COPYING-MPL-1.1.

The words "LGPL and the MPL" triggers the identification, so I think this is a problem we need not fix.

2. cairo-1.18.4/README.md
   In this file there is actually or between the licenses.

3. cairo-1.18.4/util/cairo-script/COPYING

See the following files in this directory for the precise terms and
conditions of either license:

        COPYING-LGPL-2.1
        COPYING-MPL-1.1

Adding logic to have (the content of) these files checked against previous license identifications would add too much complexity (I think!) so I think we will have to live with this and solve this manually or automatically in some kind of curation.

CONCLUSION
.... my conclusion is that the problem is fixed for the source code files but remain for cairo-1.18.4/README.md.

• Text: GNU Lesser General Public\nLicense (LGPL) version 2.1 or the Mozilla Public License (MPL) version 1.1 (edited)
• Incorrectly identified as: lgpl-2.1 - which will be ANDed with the mpl-1.0 identification.
• Better identification: lgpl-2.1 OR mpl-1.1 - not ANDing the identifications (edited)

UPDATE: The README says:

Cairo is free software and is available to be redistributed and/or
modified under the terms of either the GNU Lesser General Public
License (LGPL) version 2.1 or the Mozilla Public License (MPL) version
1.1.

Note the 1.1 on a separate line. I missed that earlier

Note: Mozilla Public License (MPL) version refers to MPL-1.1 in this context, but I don't think we can/should add a rule for that.

[AyanSinhaMahapatra]

@hesa @pombredanne I have all the mentioned issues fixed at #5233

We improved earlier with #3681 the behaviour of following license references to other files, the key change was introducing some heuristics to decide whether or not we need to actually use/substitute the licenses mentioned in the referenced files. So in this case we are updating the heuristics there to preserve AND/OR license expression cases.

Note that we were already getting this correctly in the COPYING, where behind the scenes we check the referenced files COPYING-MPL-1.1 and COPYING-LGPL-2.1, identify that we don't detect any extra/fishy licenses in those files, and thus do nothing. If the licenses in the referenced files didn't match, we would have added the license keys from there.
In this case we encountered an edge case where we had an extra license key: (lgpl-2.1 OR mpl-1.1) AND other-permissive, we now handle this correctly.

1. cairo-1.16.0/src/cairo-analysis-surface-private.h was already fixed
2. cairo-1.18.4/NEWS just needed updated license rules
3. cairo-1.18.4/README.md just needed updated license rules
4. the summary: just needed updated license rules
5. cairo-1.18.4/util/cairo-script/COPYING fixed with updated code and heuristics (I've added this as a test in the fixing PR)