Dependence aware tests for ppc*ecdf#428
Dependence aware tests for ppc*ecdf#428florence-bockting wants to merge 166 commits intostan-dev:masterfrom
ppc*ecdf#428Conversation
…improve input validation checks
ppc_*_ecdfppc*ecdf
|
As mentioned in the PR introduction text, the idea would be migration in multiple stages. Informing the user in the first stage is currently implemented as follows: |
jgabry
left a comment
There was a problem hiding this comment.
Here are a few comments and questions. So far I've only looked at some of the code, so I may have more, but I thought I would give you these now so we could start discussing.
| linewidth <- linewidth %||% 0.3 | ||
| color <- color %||% c(ecdf = "grey60", highlight = "red") | ||
| help_text <- help_text %||% TRUE | ||
| pareto_pit <- pareto_pit %||% is.null(pit) && test %in% c("POT", "PIET") |
There was a problem hiding this comment.
I think R will parse this as
(pareto_pit %||% is.null(pit)) && (test %in% c("POT", "PIET"))and not as
pareto_pit %||% (is.null(pit) && test %in% c("POT", "PIET"))So if we have
pareto_pit = TRUE
pit = NULL
test = "PRIT"then I think this will parse as TRUE && FALSE -> FALSE.
Do we not want this instead?
pareto_pit <- pareto_pit %||% (is.null(pit) && test %in% c("POT", "PIET"))This is a bit confusing for me, so I could definitely be wrong!
There was a problem hiding this comment.
Thank you for catching this behavior! I fixed the notation and created corresponding unittests.
Furthermore, I noticed that I need to catch the situation where a user provides both pareto_pit = TRUE and pit = pit (thus, pit is not NULL). I added an error with the user message:
`pareto_pit = TRUE` cannot be used together with a non-`NULL` `pit` value. Set either `pareto_pit = FALSE` or `pit = NULL`.
… corresponding unittests
…e-bockting/bayesplot into dependence-aware-LOO-PIT
|
@florence-bockting The changes look good, but I think there are the same issues (with the |
|
@jgabry I have refactored the code such that it removes duplicate code and documentation by creating new helper functions or template files. |
|
@florence-bockting please also officially add yourself to the DESCRIPTION file as part of this PR! |
Description
The current approach in
ppc_loo_pit_ecdfandppc_pit_ecdfassumes independence of LOO-PIT values which is not valid (Marhunenda et al., 2005). The corresponding graphical test yields an envelope that is too wide, reducing the test's ability to reveal model miscalibration.Tesso & Vehtari (2026, see preprint) propose three testing procedures that can handle any dependent uniform values and provide an updated graphical representation that uses color coding to indicate influential regions or most influential points of the ECDF. This PR implements the new development, by adding the updated approach (method = "correlated") additionally to the previous approach (method = "independent").
TODOs
ppc_loo_pit_ecdf()function inppc-loo.Rppc_pit_ecdf()andppc_pit_ecdf_grouped()function inppc-distributions.Rmethodargumentmethodargument