Skip to content

Adding WebsiteMetadataProcessor to preprocessing_utils - #49

Merged
shanyas10 merged 33 commits into
bigscience-workshop:masterfrom
shanyas10:ds_preprocessing_website
Nov 23, 2021
Merged

Adding WebsiteMetadataProcessor to preprocessing_utils#49
shanyas10 merged 33 commits into
bigscience-workshop:masterfrom
shanyas10:ds_preprocessing_website

Commits

Commits on Oct 15, 2021

Commits on Oct 29, 2021

Commits on Oct 31, 2021

Commits on Nov 1, 2021

  • adding processor for website metadata

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    7aa1797

  • Create download_wiki_dump.sh

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    561867a

  • run style and quality checks

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    7a59eba

  • Update website_desc_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    fce9980

  • adding tokenization for sentence

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    ab7bd72

  • Update download_wiki_dump.sh

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    60e6620

  • add test

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    0299c88

  • Update preprocessing_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    5a3f0f1

  • Update preprocessing_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    2bf8ca2

  • Update test_preprocessing_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    1ef9493

  • Update test_preprocessing_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    19ff034

  • adding tests

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    a6761db

  • fixing a bug in mocking

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    9afa782

  • Update test.yml

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    46773a9

  • updating name in workflow

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    8cd384d

  • adding nltk to requirements

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    e6e0342

  • Merge branch 'ds_preprocessing_website' into test_ds_preprocess

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    a3785e3

  • Update test_preprocessing_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    a82cd78

  • Update test_preprocessing_utils.py

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    8160bda

  • fixing tests

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    f8c05e2

  • reverting changes from test

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    7ea07b4

  • fixing quality

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    7ef1d9a

Commits on Nov 2, 2021

Commits on Nov 23, 2021

  • Merge branch 'master' into ds_preprocessing_website

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    21c15c1

  • addressing PR comments

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    81af40a

  • make quality

    Shanya Sharma - s0s0cr3Shanya Sharma - s0s0cr3
    Shanya Sharma - s0s0cr3
    authored and
    Shanya Sharma - s0s0cr3
    committed
    1611733