Exit without any error log when test fails during the collection

[guybowden]

Not sure if this is by design or a bug.. but it would be helpful to output the errors somehow if the test errors out during the test collection phase.

We recently were experiencing a test exiting when a test script was importing some code that relied on a database being there in order to execute. The import was at the top of the file and should have been inside a test method.

The output from the tap stream was:

1..0

Running the test without the --tap-stream argument showed what the error was:

-------- ERROR collecting tests.py -------- 
tests.py:16: in <module>
    from my_module import MyClass
...
E   ProgrammingError: relation “x” does not exist
E   LINE 1: ...ion”, “x”.“y” FROM “x...
E                                                                ^

!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

[mblayman]

@guybowden thanks for the interesting report. This plugin works by hooking into pytest hooks. I'm not sure if pytest offers a hook that can show output in the erroring case that you've described.

I think minimally this sounds like a good place to dump out pytests error data as TAP diagnostics. I'll have to look and see if/how that would be possible.

Any chance you could put together a basic example that would show the kind of failure mode you've described? Thanks!

[guybowden]

A simple exception will do it:

pip install pytest-tap
echo "raise Exception" > test.py
py.test test.py --tap-stream
# 1..0

[mblayman]

Excellent. I'm not a regular pytest user so do you have any opinions about what might appear instead? Would something like what suggested about outputting as TAP diagnostics work?

# E   ProgrammingError: relation “x” does not exist
# E   LINE 1: ...ion”, “x”.“y” FROM “x...
# E                                                                ^
#
# !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

[guybowden]

To be honest, I'm not sure what is and isn't possible — When you get an actual test fail or error, you can see the output and fix it, this one just had us stumped for a while until we turned off the tap-stream option and could see the error. If it could output the error in the same way as it'd output a test error that'd be good?

[mblayman]

I finally dug into this problem to see what's going on since I was working on a different issue this evening and wanted to see if I could fix this too.

I don't think I'll be able to manage fixing it tonight, but here's what I've learned.

The error output that users don't see when using --tap-stream stems from unregistering the terminalreporter plugin. This plugin is what outputs all of pytest's standard test output that people are accustomed to. By unregistering the plugin, the --tap-stream option helps to ensure that pytest is not polluting the TAP stream.

The problem with this strategy is that the terminal reporter does quite a bit more than printing output. It's job also includes collecting stats about the run from all the tests. When the session flushes at the end, it reports any errors. Those error reports are exactly the kind of info that --tap-stream suppresses by unregistering the plugin.

To me, the code question becomes: how can the plugin collect all the stats without the side effect of output from the terminal reporter going to stdout? I see a few possible options to address this:

1. Continue to unregister TerminalReporter and then register some alternative implementation that collects stats and reports on errors.
2. Switch the plugin to leave TerminalReporter registered, then customize it somehow to change its behavior. Since the output functions are meant for printing to stdout, I could probably do this in a crude by using mock.patch.object, but that definitely feels like a hack.
3. Leave TerminalReporter registered and supply config options that suppress all the output that I need to while leaving other output. I don't think this route will work because I think I need to wrap all the output and treat it as diagnostics (i.e., prefix with #).

Unregistering the reporter bit me on another issue, so I would like to have some kind of registered reporter. Maybe I could strategically subclass the report and register my subclass instead. That feels a bit nicer than mocking. That would also give me the chance to wrap stuff as diagnostics.

[mblayman]

Since I'm not working on this feature now, here's a test that I might use in the future if I get around to this issue:

def test_stream_collection_exception(testdir):
    """Streaming displays error info when a problem happens during collection."""
    testdir.makepyfile("raise Exception('boom')")

    result = testdir.runpytest_subprocess("--tap-stream")

    expected_option_flags = ["*boom*"]
    result.stdout.fnmatch_lines(expected_option_flags)