Package 'NBvarsel'

ADNEX Exhaustive Variable Selection Results

Description

Pre-computed results from an exhaustive Net Benefit-based variable selection applied to ovarian tumour data from the International Ovarian Tumour Analysis (IOTA) consortium (phases 1–3). The dataset contains model-level summary statistics for the top 20 models per predictor count, drawn from a full exhaustive search over all 65,535 (2^16 - 1) predictor combinations.

Usage

adnex_results

Format

A data frame with 293 rows and 23 columns:

Model

Comma-separated list of predictor names included in the model.

n_Preds

Number of predictors in the model.

AUC

Cross-validated area under the ROC curve.

Brier

Cross-validated Brier score.

Total_Cost

Sum of predictor group costs for the model.

Avg_Adj_Net_Benefit

Average cost-adjusted Net Benefit across thresholds and CV folds.

Avg_Net_Benefit

Average Net Benefit (without cost adjustment) across thresholds and CV folds.

VIF_age, VIF_ca125, VIF_family_history, VIF_locules_gt_10, VIF_oncology_center, VIF_max_diam_lesion, VIF_papillary_count, VIF_acoustic_shadows, VIF_ascites, VIF_ireg_walls, VIF_bilateral, VIF_color_score, VIF_pain, VIF_max_diam_solid, VIF_papillary_presence, VIF_prop_solid

Permutation importance (delta Net Benefit) for each predictor. NA when the predictor is not included in the model.

Details

The original patient-level data are not publicly available. This dataset contains only aggregated model performance metrics and can be used to demonstrate the all_subset_plot() and VIF_plot() visualisation functions.

The analysis used 16 candidate predictors for a binary outcome (malignant vs. benign ovarian tumour) with grouped predictor costs reflecting clinical history (free), ultrasound examination (moderate cost), and blood biomarker (higher cost). Five-fold cross-validation was used with restricted cubic splines (3 knots) for continuous predictors.

The 16 candidate predictors are:

• Clinical history (no cost): patient age, family history of ovarian cancer, oncology centre, pain.

• Ultrasound (moderate cost): maximum lesion diameter, proportion solid, number of locules > 10, papillary count, papillary presence, acoustic shadows, ascites, irregular walls, bilateral, colour score, maximum solid diameter.

• Blood biomarker (higher cost): CA-125.

This is a filtered subset (top 20 models per number of predictors, ranked by Avg_Adj_Net_Benefit) of the full 65,535-model exhaustive search. The attribute "best_model_stats" contains the overall best model, and "n_total_models" records the total number of evaluated models.

Source

IOTA consortium data (phases 1–3). See https://iotaplus.org/en for information about the IOTA studies.

Examples

data(adnex_results)
head(adnex_results)

# Best model from the full search
attr(adnex_results, "best_model_stats")

# Visualise
all_subset_plot(adnex_results, filter = 7, size_dot = 1)
VIF_plot(adnex_results)$plot

---

All-Subset Model Comparison Plot

Description

Creates a two-panel figure comparing model performance across all evaluated subsets. The upper panel shows the selected metric for each model, and the lower panel shows which predictors are included in each model.

Usage

all_subset_plot(
  all_models,
  metric = "Avg_Net_Benefit",
  y_axis = "Average Net Benefit",
  filter = 5,
  size_dot = 3,
  highlight_color = "red",
  tile_color = "#2A6EBB",
  p1_theme = NULL,
  p2_theme = NULL,
  data_dict = NULL
)

Arguments

all_models	Data frame returned in the all_models element of nb_varsel(). This should contain the model summary output used for plotting.
metric	Character string. Name of the column in all_models to plot on the y-axis. Defaults to "Avg_Net_Benefit".
y_axis	Character string. Label for the y-axis in the top panel. Defaults to "Average Net Benefit".
filter	Integer. Maximum number of models to display per number of predictors. Defaults to 5.
size_dot	Numeric. Point size used in the metric plot. Defaults to 3.
highlight_color	Character string. Color for the dashed reference lines marking the best model. Defaults to "red".
tile_color	Character string. Fill color used for included predictors in the heatmap. Defaults to "#2A6EBB".
p1_theme	A ggplot2::theme() object applied to the top panel, or NULL.
p2_theme	A ggplot2::theme() object applied to the bottom panel, or NULL.
data_dict	A named character vector used to relabel predictors. Format: c("raw_name" = "Display Label"). Defaults to NULL.

Value

A patchwork object containing two stacked ggplot panels.

---

Variable Selection via Cross-Validated Net Benefit

Description

Performs exhaustive or groupwise variable selection for binary-outcome prediction models. Models are evaluated using cross-validated net benefit, with optional adjustment for predictor costs.

Usage

nb_varsel(
  data,
  outcome_var,
  costs = NULL,
  thresholds = seq(0.05, 0.3, by = 0.01),
  include_interactions = FALSE,
  mode = c("exhaustive", "groupwise"),
  group_size = 2,
  cv_folds = 5,
  seed = 123,
  verbose = TRUE,
  allow_parallel = TRUE,
  permutation = FALSE,
  splines = TRUE,
  n_knots = 3
)

Arguments

data	A data frame containing the predictor variables and the binary outcome.
outcome_var	Character string naming the binary outcome column (coded as 0/1 or as a two-level factor).
costs	Predictor costs. Can be: NULL (default): no costs. A named numeric vector: per-predictor costs, e.g. c(X1 = 0.1, X2 = 0.05). An unnamed scalar: the same cost applied to every predictor. A list of groups, each with elements vars (character vector) and cost (numeric). The group cost is added once if any variable in the group is selected.
thresholds	Numeric vector of decision thresholds at which net benefit is evaluated. Defaults to seq(0.05, 0.3, by = 0.01).
include_interactions	Logical. If TRUE, interaction terms ((X1 + X2 + ...)^2) are also evaluated in exhaustive mode. Defaults to FALSE.
mode	Character string specifying the selection strategy: "exhaustive" evaluates all predictor subsets, while "groupwise" performs backward elimination in groups of size group_size. Defaults to "exhaustive".
group_size	Integer. Number of variables removed at each step in groupwise mode. Defaults to 2.
cv_folds	Integer. Number of cross-validation folds. Defaults to 5.
seed	Integer. Random seed for reproducibility. Defaults to 123.
verbose	Logical. If TRUE, progress messages are printed. Defaults to TRUE.
allow_parallel	Logical. If TRUE, use parallel computation through foreach and doParallel. Defaults to TRUE.
permutation	Logical. If TRUE, compute permutation importance scores for each predictor. Defaults to FALSE.
splines	Logical. If TRUE, apply restricted cubic splines (via rms::rcs()) to continuous predictors. Defaults to TRUE.
n_knots	Integer. Number of knots used for restricted cubic splines. Defaults to 3.

Value

A list with the following elements:

best_model_stats

A one-row data frame describing the top-performing model.

all_models

A data frame with one row per evaluated model, containing columns Model, n_Preds, AUC, Brier, Total_Cost, Avg_Adj_Net_Benefit, and Avg_Net_Benefit. If permutation = TRUE, additional VIF_ columns containing permutation importance scores are included.

Examples

set.seed(42)
n <- 500
df <- data.frame(
  X1 = rnorm(n), X2 = rbinom(n, 1, 0.7),
  X3 = rnorm(n), X4 = rbinom(n, 1, 0.5)
)
df$Y <- rbinom(n, 1, plogis(2 * df$X1 + 1.5 * df$X2))

harms <- c(X1 = 0.1, X2 = 0.05, X3 = 0.1, X4 = 0.0001)

result <- nb_varsel(
  data = df, outcome_var = "Y", costs = harms,
  mode = "exhaustive", splines = FALSE,
  allow_parallel = FALSE
)
result$best_model_stats

---

Permutation Importance Bar Plot

Description

Creates a horizontal bar chart showing the average permutation importance of each predictor, measured as the change in net benefit relative to the full set of evaluated models.

Usage

VIF_plot(all_models, filter = NULL, color = "#2A6EBB", data_dict = NULL)

Arguments

all_models	Data frame returned in the all_models element of nb_varsel(). Must include VIF_ columns generated when permutation = TRUE.
filter	Integer or NULL. If specified, only the top filter models by Avg_Net_Benefit are used to compute average importance. Defaults to NULL, which uses all models.
color	Character string. Fill color for the bars. Defaults to "#2A6EBB".
data_dict	A named character vector used to relabel predictors. Format: c("raw_name" = "Display Label"). Defaults to NULL.

Value

A list with two elements:

plot

A ggplot2::ggplot object showing the bar chart.

data

A data frame containing the Variable, Average_Delta_NB, and mapped Label values.

Examples

data(adnex_results)
vif_results <- VIF_plot(adnex_results)
vif_results$plot
head(vif_results$data)

---