Interpreting POWERUP results

POWERUP produces several complementary outputs. Cross-validation metrics describe how well a perturbation can be predicted in held-out reference samples, prediction distributions describe the expected response and its uncertainty in a new sample, SHAP values explain how molecular features contribute to a prediction, and posterior distributions incorporate new experimental evidence.

No single number captures all of these questions. In particular, model performance, confidence in an individual prediction, and confidence that a prediction crosses a biologically meaningful response cutoff are related but distinct quantities.

Model performance

summarize_models() returns one row per perturbation. POWERUP reports both continuous-response metrics and classification-style metrics because they evaluate different aspects of held-out predictive performance.

model_summary <- summarize_models(models)
model_summary

All validation metrics are calculated separately in each held-out cross-validation fold. summarize_models() reports the mean of each metric across the repeated folds.

Continuous response performance

The three primary continuous metrics answer different questions:

Column	Interpretation
r	Mean Pearson correlation between predicted and observed responses across validation folds
R2	Mean held-out coefficient of determination across validation folds
rmse	Mean held-out root mean squared error across validation folds

Pearson r measures whether predicted and observed responses vary together. A high correlation means that samples predicted to have relatively high responses also tend to have relatively high observed responses, and likewise for low responses. Correlation is therefore useful for assessing relative ordering, but it does not require predictions to be on the correct numerical scale. Predictions can be systematically shifted or compressed and still correlate well with the observations.

R2 evaluates the numerical accuracy of the predictions relative to simply predicting the mean observed response within each validation fold. An R2 of 1 represents perfect prediction, 0 indicates no improvement over the fold mean by this measure, and negative values indicate that the held-out predictions are worse than using that mean. A model can therefore have a useful r but a substantially lower R2 when it captures relative ordering better than response magnitude.

RMSE measures prediction error directly on the response scale. Lower values indicate smaller errors, with larger misses receiving more weight because errors are squared before averaging.

These metrics are most informative when read together. For example, a model with relatively high r but much lower R2 may rank samples reasonably well while predicting response magnitudes less accurately. RMSE then indicates the size of those numerical errors on the original response scale.

Binarized response performance

POWERUP also evaluates whether held-out predictions fall on the correct side of the model’s response_cutoff. These metrics convert both the predicted response and observed response into the same target-event definition:

• when decreasing = FALSE, values greater than or equal to response_cutoff define the target event;
• when decreasing = TRUE, values less than or equal to response_cutoff define the target event.

The classification-style metrics in summarize_models() are:

Column	Question it answers
d_sensitivity	Of samples that truly satisfy the target event, what fraction did POWERUP identify?
d_specificity	Of samples that truly do not satisfy the target event, what fraction did POWERUP correctly reject?
d_FPR	Of samples that truly do not satisfy the target event, what fraction were incorrectly called target events?
d_PPV	Of samples POWERUP called target events, what fraction truly satisfied the event?
d_NPV	Of samples POWERUP called non-events, what fraction truly did not satisfy the event?
d_accuracy	What fraction of all target-event and non-event calls were correct?

Sensitivity and PPV are complementary ways of thinking about positive target-event calls. Sensitivity asks how many true target events are recovered. PPV asks how often a positive POWERUP call is correct. A model can have high PPV but modest sensitivity if its positive calls are reliable but capture only a subset of the true events.

Specificity describes correct rejection of true non-events, while d_FPR describes the corresponding false-positive rate and is equal to 1 - specificity when both are defined. NPV describes the reliability of negative calls. Accuracy summarizes all correct calls together, but it can be misleading when target events are much more or much less common than non-events.

PPV and NPV also depend on the prevalence of the target event in the validation samples. They should therefore be interpreted in the context of the reference dataset and the chosen response_cutoff, rather than as prevalence-independent properties of the model.

A discrete metric can be undefined in an individual validation fold. For example, PPV is undefined if the model makes no positive calls in that fold, and sensitivity is undefined if the fold contains no observed target events. POWERUP records those fold values as NA, and summarize_models() averages the available finite values only.

Model retention and metadata

The remaining summary fields describe model status or fitted-model characteristics rather than held-out performance:

Column	Interpretation
skipped	Whether the fitted response model was not retained
n_terms	Number of fitted features with nonzero aggregate SHAP contribution across the training samples
mean_pred_sd	Mean training-sample prediction SD estimated by the uncertainty model

A model is retained only when its mean validation correlation and mean validation R2 are finite and mean validation R2 is at least min_score. The default min_score is 0.01. Models that fail this check are marked as skipped and are not carried into prediction or SHAP analysis.

mean_pred_sd is the average prediction standard deviation estimated by the uncertainty model across the training samples. Smaller values indicate that the model assigns narrower predictive distributions on average.

For details on repeated cross-validation, model retention, and how the uncertainty model is trained and calibrated, see Training and evaluating POWERUP models.

Predictions and uncertainty

The default summarize_predictions() output is a sample-by-perturbation matrix of predicted response means. Use format = "long" when uncertainty and target-event probabilities are needed:

predictions <- summarize_predictions(models, format = "long")

The important fields are:

Column	Interpretation
pred_mean	Expected model response for the sample
pred_sd	Estimated prediction SD
pred_pi_lower_95, pred_pi_upper_95	Approximate 95% Gaussian prediction interval
response_cutoff	Response value defining the target event
decreasing	Which side of the cutoff defines the response of interest
prob_below_cutoff	Probability that the response is at or below the cutoff
prob_above_cutoff	Probability that the response is at or above the cutoff
prob_target_event	Probability that the response falls on the response-of-interest side of the cutoff
target_event_definition	Text showing which side of the cutoff defines the response of interest

pred_mean is the point estimate of the expected response. pred_sd describes the uncertainty estimated by POWERUP around that prediction, and the prediction interval gives the corresponding range under the fitted Gaussian predictive distribution.

The target-event probabilities integrate that full predictive distribution relative to response_cutoff. When decreasing = TRUE, prob_target_event is the probability that the response is at or below the cutoff. Otherwise it is the probability that the response is at or above the cutoff.

A predicted mean and a target-event probability therefore answer different questions. The predicted mean asks, “What response value does POWERUP expect?” The event probability asks, “Given both the predicted response and its uncertainty, how much probability lies on the target-event side of the cutoff?”

The validation d_* metrics and prediction probabilities should not be conflated. The d_* metrics evaluate hard event calls made from held-out point predictions during cross-validation; prob_target_event is an uncertainty-aware probability for an individual prediction.

Note that prediction SD represents uncertainty estimated by the fitted POWERUP procedure. It does not include every source of biological, measurement, or dataset-shift uncertainty.

SHAP explanations

POWERUP calculates SHAP values for both training and prediction samples. SHAP contributions describe how the fitted model moves from its baseline prediction to the prediction for an individual sample. Positive contributions move the predicted response upward and negative contributions move it downward.

SHAP values explain the fitted model, not biological causality. A feature with a large contribution strongly influenced the prediction under the fitted model, but this does not establish that experimentally changing the feature would change the biological response.

For detailed interpretation of individual waterfalls, SHAP scatter plots, cohort-level feature summaries, and recurrent explanation paths, see Explaining POWERUP predictions.

Experimental observations and posterior results

calculate_powerup_posteriors() combines a predictive prior with a prepared experimental observation for matching sample-perturbation pairs.

Column	Interpretation
prior_mean, prior_sd	POWERUP prediction before incorporating the observation
prior_prob_target_event	Target-event probability before the observation
observation_mean, observation_sd	Experimental estimate and uncertainty
posterior_mean, posterior_sd	Updated response estimate and uncertainty
posterior_prob_target_event	Target-event probability after incorporating the observation
posterior_status	Whether the requested posterior update was available and, if not, why
response_transform	Scale used for Gaussian evidence integration

The posterior is an inverse-variance weighted Gaussian update on the selected integration scale. response_transform = "identity" preserves the original same-scale Gaussian update. "logit" supports responses constrained to (0, 1), while "log" and "log10" support strictly positive responses. Original-scale posterior means, SDs, intervals, and plots are back-transformed for interpretation.

More precise information receives more weight on the integration scale. Barcode-count observations can provide logit-scale uncertainty directly from the hierarchical bootstrap, while generic prediction or observation mean/SD pairs are deterministically mapped to the requested transformed-normal family.

The possible posterior_status values are:

Status	Interpretation
ok	Prior and observation were valid and the posterior was calculated
not_available_bad_prior	The prior mean or SD could not define a valid predictive distribution
not_available_bad_observation	The observation mean or SD could not define a valid experimental distribution
not_available_bad_prior_transform	The prior moments could not define a valid distribution on the requested transformed scale
not_available_bad_observation_transform	The observation could not define a valid distribution on the requested transformed scale

plot_posterior() makes the relative weighting visible by plotting the prior, observation, and posterior distributions together.

For preparation of experimental measurements and details of the update, see Experimental observations and posterior updating.

Choosing the right output

Different outputs answer different questions:

Question	Primary output
Does the model rank high- and low-response samples well?	r
Does the model predict held-out response magnitudes well?	R2 and rmse
When POWERUP calls a target event, how often is that call correct?	d_PPV
How many true target events does POWERUP recover?	d_sensitivity
How well does POWERUP reject true non-events?	d_specificity and d_FPR
How reliable are negative event calls?	d_NPV
What response does POWERUP predict for this sample?	pred_mean
How uncertain is that individual prediction?	pred_sd and the prediction interval
How likely is the response of interest for this sample?	prob_target_event
Which molecular features contributed to this prediction?	SHAP contributions
How did experimental evidence change the prediction?	Prior vs. posterior mean, SD, and target-event probability

Session information

sessionInfo()

## R version 4.4.2 (2024-10-31)
## Platform: aarch64-apple-darwin20
## Running under: macOS Sequoia 15.7.3
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## time zone: America/New_York
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] powerup_1.0.99
## 
## loaded via a namespace (and not attached):
##  [1] digest_0.6.37     desc_1.4.3        R6_2.6.1          fastmap_1.2.0    
##  [5] xfun_0.51         cachem_1.1.0      knitr_1.50        htmltools_0.5.8.1
##  [9] rmarkdown_2.29    lifecycle_1.0.4   cli_3.6.5         sass_0.4.9       
## [13] pkgdown_2.2.0     textshaping_1.0.0 jquerylib_0.1.4   systemfonts_1.2.2
## [17] compiler_4.4.2    tools_4.4.2       ragg_1.5.1        bslib_0.9.0      
## [21] evaluate_1.0.3    yaml_2.3.10       jsonlite_2.0.0    rlang_1.1.6      
## [25] fs_1.6.5          htmlwidgets_1.6.4