Skip to content

Catch2 integration does not handle Generators inside sections #205

@Manu343726

Description

@Manu343726

Per Catch2 documentation:

GENERATE can be seen as an implicit SECTION, that goes from the place GENERATE is used, to the end of the scope.
[...]
The fact that GENERATE introduces a virtual SECTION can also be used to make a generator replay only some SECTIONs, without having to explicitly add a SECTION

I looked into the Catch2 integration and it seems like the Catch2 API does not include generator information as part of the sector:

// Approvals/integrations/catch/Cathc2Approvals.h

struct Catch2TestCommitRevert : Catch::TestEventListenerBase
{
    ...
    virtual void sectionStarting(Catch::SectionInfo const& sectionInfo) override
    {
        currentTest.sections.push_back(sectionInfo.name); // Name here does not include the generator value
    }
    ...
};

In fact, if you look at the Catch::SectionInfo type it only includes name and source location. I'm no expert on the Catch2 internals but at first glance, I would say Catch2 generators are handled more like part of the body of a section rather than variants of the parent section (Like generated tests cases in other frameworks).

The result of this behavior is that if you run an approval inside a section with a generator, the approval is uniquely registered for the section rather than each generated variant of the section, resulting in approval failures following the execution of the different variants of the section:

enum class Keyword
{
    Unknown, Foo, Bar, Quux
};

Keyword parseKeyword(const std::string_view word);

SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
{
     const std::string_view word = GENERATE("foo", "bar", "quux");
     Approvals::verify(parseKeyword(word)); // If we approve "foo", it fails in the next iteration of the generator
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions