[fix] Eliminate silent query limits

[bruntib]

Many report_server API functions have a limit parameter for limiting the number of results. When the parameter is 0 or None, then the MAX_QUERY_SIZE is used which is currently 500.

Silently limiting the results may cause practical problems, too. For example, when we filter on runs with a joker character (e.g. myrun*) then the results from the first 500 runs return only.

[barnabasdomozi]

In general, it's considered a good practice to have some hard limits on the amount of data and API endpoint can return from the database.
As an example, it's not uncommon to have production databases with more than 1 million rows in the RunHistory table. Without any enforced limit, the client could request all information at once which is an excessive amount of data.

Silently limiting the results may cause practical problems, too. For example, when we filter on runs with a joker character (e.g. myrun*) then the results from the first 500 runs return only.

For which API endpoints is this a problem?

[bruntib]

In general, it's considered a good practice to have some hard limits on the amount of data and API endpoint can return from the database. As an example, it's not uncommon to have production databases with more than 1 million rows in the RunHistory table. Without any enforced limit, the client could request all information at once which is an excessive amount of data.

I think, that's why these functions have a limit parameter. Capping this number arbitrarily on the server side is not an intuitive behavior and leads to bugs like in this specific case.

Silently limiting the results may cause practical problems, too. For example, when we filter on runs with a joker character (e.g. myrun*) then the results from the first 500 runs return only.

For which API endpoints is this a problem?

In this specific case the client queried the run IDs matching a run name pattern run_name*. The server returned only 500 IDs even if there was more. These were passed to getRunResultCount() (and to several other functions) through the run_ids parameter.

[barnabasdomozi]

I think, that's why these functions have a limit parameter. Capping this number arbitrarily on the server side is not an intuitive behavior and leads to bugs like in this specific case.

Yes, but if we remove the guard, essentially the client controls this limit parameter. If the limit set by the client is 0 or None or a large number, the server will try to return all rows with that limit. That means, in case of getRunHistory function, to load millions of rows of data into the server's memory and then it tries to return that excessive amount of data via Thrift / HTTP.

These were passed to getRunResultCount() (and to several other functions) through the run_ids parameter.

Maybe you meant a different function, but as of today, getRunResultCount has no enforced limit.

I think this server-side limit enforcement should be investigated on a case by case basis, and not removed from everywhere.

[barnabasdomozi]

Consider the following if expression which is more Pythonic and easier to read:

Suggested change

	if not limit or limit not in range(constants.MAX_QUERY_SIZE + 1):
	if limit not in range(1, constants.MAX_QUERY_SIZE + 1):

[barnabasdomozi]

Why was .getClient() dropped from here? ccService has no exported function named getRunData.