Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions src/cluecode/copyrights.py
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,9 @@ def get_tokens(numbered_lines, splitter=re.compile(r'[\t =;]+').split):
if TRACE_TOK:
logger_debug(' get_tokens: preped line: ' + repr(line))

# Author:Frankie.Chu -> Author: Frankie.Chu for AUTH + name tokenization
line = re.sub(r'(?i)\b(authors?|@authors?)(:)(?=\S)', r'\1\2 ', line)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you write a test for this? this is a really weird and complicated regex.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Covered by tests/cluecode/data/authors/author_frankie_chu_cpp-tm1637.cpp and its expected yml. That fixture is the glued Author:Frankie.Chu case from #4229; detect returns author Frankie.Chu.


for tok in splitter(line):
# strip trailing quotes+comma
if tok.endswith("',"):
Expand Down Expand Up @@ -2227,6 +2230,10 @@ def build_detection_from_node(
# proper noun with some separator and trailing comma
(r'^[A-Z]+\.[A-Z][a-z]+,?$', 'NNP'),

# Mixed-case dotted name (Frankie.Chu). Not NNP: AUTHOR uses AUTH+NAME-DOT only.
# Exclude common org suffixes and license-prose glue tails (Assignment.This).
(r'^[A-Z][a-z0-9]+(?:\.(?!Org,?$|Com,?$|Net,?$|Edu,?$|Gov,?$|Inc,?$|Ltd,?$|Co,?$|This,?$|In,?$|All,?$|The,?$|Of,?$|And,?$|For,?$|By,?$|With,?$|From,?$|To,?$|As,?$|Or,?$|An,?$|At,?$|On,?$|Is,?$|Are,?$|Was,?$|Be,?$|If,?$|Not,?$|No,?$|Any,?$|Such,?$|That,?$|These,?$|Those,?$|When,?$|Where,?$|Which,?$|Who,?$|Will,?$|Shall,?$|May,?$|Can,?$|Must,?$|Should,?$|Would,?$|Could,?$|Rights,?$|Reserved,?$|Agreement,?$|Event,?$|Breach,?$|License,?$|Software,?$|Source,?$|Code,?$|File,?$|Files,?$|Notice,?$|Notices,?$|Subject,?$|Terms,?$|Conditions,?$|Section,?$|Clause,?$|Party,?$|Parties,?$)[A-Z][a-z0-9]+)+,?$', 'NAME-DOT'),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this blob? Where does it come from?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NAME-DOT is a POS tag for one mixed-case dotted personal name token (e.g. Frankie.Chu). It is intentionally not generic NNP. AUTHOR stays AUTH + NAME-DOT so bare AUTH + any proper noun is not treated as an author.


# proper noun with apostrophe ': D'Orleans, D'Arcy, T'so, Ts'o
(r"^[A-Z][a-z]?'[A-Z]?[a-z]+[,\.]?$", 'NNP'),

Expand Down Expand Up @@ -2552,6 +2559,8 @@ def build_detection_from_node(

# NAME-YEAR starts or ends with a YEAR range
NAME-YEAR: {<YR-RANGE> <NNP> <NNP>+} #350
NAME-YEAR: {<YR-RANGE> <NAME-DOT> <COMP|COMPANY|NNP|NN>+} #350.05

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why tow lines?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two NAME-YEAR rules on purpose: (1) year range + dotted name alone, (2) year range + dotted name + company/name tokens. Same split as the existing NNP NAME-YEAR forms, with NAME-DOT dual-homed for holders. I can fold them if you want a smaller grammar.

NAME-YEAR: {<YR-RANGE> <NAME-DOT>} #350.06

COPYRIGHT: {<COPY> <YR-RANGE> <NNP> <NN> <NNP> <NNP> <NNP> <EMAIL>} #350.1

Expand Down Expand Up @@ -2699,6 +2708,8 @@ def build_detection_from_node(
# Companies
COMPANY: {<NAME|NAME-EMAIL|NAME-YEAR|NNP>+ <OF> <NN>? <COMPANY|COMP> <NNP>?} #770
COMPANY: {<NNP> <COMP|COMPANY> <COMP|COMPANY>} #780
# NAME-DOT + company words only (not bare NAME-DOT)
COMPANY: {<NAME-DOT> <COMP|COMPANY>+} #780.1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Holders that start with a dotted token then company words (e.g. Rocket.Chat followed by Technologies / Corp-style tails). Bare NAME-DOT as COMPANY was dropped; only NAME-DOT + COMP/COMPANY+ remains.

COMPANY: {<NN>? <COMPANY|NAME|NAME-EMAIL> <CC> <COMPANY|NAME|NAME-EMAIL>} #790
COMPANY: {<COMP|COMPANY|NNP> <NN> <COMPANY|COMPANY> <NNP>+} #800

Expand Down Expand Up @@ -3475,6 +3486,8 @@ def build_detection_from_node(
# author (Panagiotis Tsirigotis)
AUTHOR: {<AUTH> <NNP><NNP>+} #author Foo Bar

AUTHOR: {<AUTH> <NAME-DOT>} #author Frankie.Chu

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "author Frankie.Chu" seen in the wild?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — from #4229, this wild file still has it:
https://github.com/Wind-River/Lander/blob/98b5986d56f4ee6ae58febd496495acb2413ed33/source/satelliteware/libraries/DigitalTube/TM1637.cpp

//  Author:Frankie.Chu
//  Date:9 April,2012

That glued Author:Name line is what this PR detects.


# Author: Tim (xtimor@gmail.com)
AUTHOR: {<AUTH> <NNP>+ <EMAIL>+} #Author Foo joe@email.com

Expand Down
2 changes: 2 additions & 0 deletions tests/cluecode/data/authors/author_frankie_chu_cpp-tm1637.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
// Date:9 April,2012
// Author:Frankie.Chu
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
what:
- authors
- authors_summary
authors:
- Frankie.Chu
authors_summary:
- value: Frankie.Chu
count: 1
notes: Detect glued C++ Author:Name tags without space after colon (issue #4229)
Loading