Add pabutools.recommendation: a recommendation system for participatory budgeting

[roeiyanku]

Implements "A Recommendation System for Participatory Budgeting" by Gil
Leibiker and Nimrod Talmon (AAMAS 2023):
https://optlearnmas23.github.io/files/p17.pdf

The module addresses the paper's information overload problem — the cognitive
burden of judging every project on a long ballot. Instead of asking every voter
about every project, it queries each voter for a partial ballot, estimates the
votes it did not ask for, and runs the voting rule on the completion.

Contents

pabutools.recommendation runs the system:

• The five sampling setups of §3.1: random, offline by popularity, consensus
  and controversiality, and the online adaptive setup. next_adaptive_question
  exposes a single step of the adaptive loop so it can drive a live process,
  not only a simulation.
• The three prediction modules of §2.1: binary classification with XGBoost,
  matrix factorization, and factorization machines.
• The LV/TV partiality arithmetic of §3.0.1 (plan_sampling, split_lv_tv),
  greedy approval, and elect for running a real process where no ground
  truth exists.

pabutools.analysis.recommendationanalytics measures how well it did:

• The accuracy metrics of §5: classification_metrics (precision, recall and
  F1 over the projects a voter was never asked about) and
  fractional_allocation_score (Definition 5.1).
• run_all_experiments, the treatment matrix of §6 — every setup x predictor
  x sample degree x LV degree, averaged over repeated random splits.

Tests: 40 in tests/test_recommendation.py and 3 in tests/test_analysis.py,
plus 49 doctests collected by tests/test_doctests.py.

Dependencies

The machine-learning libraries are an optional recommendation extra and are
imported lazily inside the functions that need them, so the module imports and
every non-ML algorithm works without them. Tests that fit a model skip rather
than fail when the library is absent, which keeps CI green on the standard
.[dev] install.

Notes on the paper

Two points are under-specified or inconsistent; both are documented in the
docstrings alongside the reading that was implemented:

• §2.1.1 states that binary classification is used and that XGBoost suits it,
  but never defines what a training row is or which features it carries.
• §5.2.1 defines symmetric distance as |rb △ pb|, yet its second example gives
  SD({1,2,3}, {1,3,4}) = 1 where the definition yields 2. The first and third
  examples agree with the definition, which is what is implemented.

[Simon-Rey]

Hey :) Thank you for your contribution. I don't fully understand what the code offers, could you explain to me how a user would use these functions and what they would provide?

[roeiyanku]

Thank you for the quick response! I'll explain below and also add an rst file.

In short: it lets a PB election run without asking every voter about every project. You ask each person about a handful, estimate the votes you never asked for, and the usual voting rule takes it from there. (For context, this came out of #58, which Erel opened for our Research Algorithms course.)

There are two flows.

Running an actual PB process where you cannot ask everyone about everything. You hold the ballots of the people who did fill one in completely, you choose which projects to put to everyone else, collect their partial answers, and get an outcome:

asked = offline_controversiality(instance, lv_profile, k=2)

{Library, Shade} <- the two projects the complete ballots most disagree about

answers = {
"v5": partial_ballot(approved={garden}, disapproved={crossings}),
"v6": partial_ballot(approved={crossings}, disapproved={garden}),
}

elect(instance, lv_profile, answers, predictor="classification")

BudgetAllocation([Garden, Shade])

So a user provides the instance, the ballots collected in full, and one partial ballot per remaining voter. They get back a BudgetAllocation, the same as from any other rule. The prediction only fills in projects a voter was never asked about; anything she actually answered is kept as given.

Evaluating the idea instead.
Here a user provides a complete profile and two numbers — how much of the vote to collect, and how much of that should come from complete ballots — and gets back the allocation that partial information would have produced, to compare against the real one:

real = set(greedy_approval(instance, profile))
predicted = set(run_experiment(instance, profile, 0.5, 0.5,
setup="offline_controversiality", seed=17))

fractional_allocation_score(real, predicted, instance.budget_limit) # 0.75

run_experiment is a simulation:
it splits the profile you already hold into the voters who answer in full and the ones who do not, reads each of the latter's answers to her k asked projects off her real ballot, and withholds the rest for the predictor to estimate. That is why it needs a complete profile, and why elect is the one to use when you actually collected the answers yourself. run_all_experiments repeats it across a whole grid of those two numbers, for every sampling setup and prediction module at once.

Everything else is the individual steps of those two flows, for anyone who wants to assemble them differently.

I pushed a usage page, docs-source/source/usage/recommendation.rst. It covers the above with runnable examples. Building the docs to check it also turned up two markup errors in my own docstrings (|LV| is substitution syntax in reStructuredText, and one :py:func: target was wrong); both are fixed in the same commit.

Happy to restructure any of it to fit your conventions.

[roeiyanku]

Following review feedback, I moved the evaluation code to a more appropriate place in the library.

pabutools.recommendation was mixing two different things: running the system, and measuring how well it ran. The Section 5 accuracy metrics and the Section 6 experiment sweep are analysis of an outcome, which is what pabutools.analysis exists for — fractional_allocation_score in particular just compares two bundles and isn't recommendation-specific at all. They now live in pabutools/analysis/recommendationanalytics.py:

• classification_metrics, fractional_allocation_score
• run_all_experiments, SAMPLE_DEGREES, LV_DEGREES

run_pipeline and run_experiment stayed put — they're simulations, but they return bundles rather than scores, and moving them would have dragged complete_ballots along too.

Imports run analysis -> recommendation, the same direction mesanalytics already uses to reach pabutools.rules, and nothing in recommendation imports analysis, so there is no cycle. The ML libraries are still imported lazily inside the functions that need them, so pulling recommendation into pabutools.analysis's __init__ does not make them hard dependencies.

One related cleanup: the two if __name__ == "__main__": doctest.testmod() blocks are gone, replaced by tests/test_doctests.py. Since the project has no pytest configuration, those blocks were the only thing running the modules' doctests, so removing them without a replacement would have silently dropped 49 tests.

Full suite: 280 passed, 9 skipped.

[roeiyanku]

Hey!Just wanted to follow up on the PR I submitted a couple of weeks ago. Whenever you get a chance, I’d really appreciate a review. Let me know if there’s anything you’d like me to change or add. Thanks!

[DominikPeters]

Hi! It doesn't feel to me like this material belongs to a library like pabutools, since it doesn't seem to be generally useful. Edit: I see that Simon in the issue discussion seems to have indicated this would be welcome, it's his call.

Generally, I think it isn't good form to let Claude write any text like PR descriptions or comments that you expect other people to read. It's better to write yourself to explain what you're doing and why you think it is a good addition.

[roeiyanku]

Thank you for the constructive criticism. If @Simon-Rey thinks there's much to cut down that's not a problem, I'll edit as necessary. In general the pr is meant to give the user the tools, based on machine learning algorithms, to predict the results of the full ballots, as most voters in real life will leave many of the projects unanswered.

[Simon-Rey]

I think my initial concern (and from what I understand also that of Dominik) is that I don't really think that users of the package will use that piece of code. The functionalities offered by a package need to be general, solve problem that several people will want to look into. Right now it all feels too specific for me to really believe that others will get advantage from it.

There are thousands of lines of code that all seem to me to have been AI generated (no problem with that, but it does not give me great confidence) and so I want to be a bit careful before introducing that to the package as they will need to be maintained at some point.

Overall, I don't know what to do with all of this and since Dominik also raised concerns I am tempted to close without merging.