R/pool_compare_models.R
pool_compare_models.Rdpool_compare_model Compares the fit and performance of prediction models
in multiply imputed data sets by using clinical important performance measures
pool_compare_models(
pobj,
compare.predictors = NULL,
compare.group = NULL,
cutoff = 0.5,
boot_auc = FALSE,
nboot = 1000
)An object of class pmods (pooled models), produced by a previous
call to psfmi_lr.
Character vector with the names of the predictors that are compared. See details.
Character vector with the names of the group of predictors that are compared. See details.
A numerical scalar. Cutoff used for the categorical NRI value. More than one cutoff value can be used.
If TRUE the standard error of the AUC is calculated with stratified bootstrapping. If FALSE (is default), the standard error is calculated with De Long's method.
A numerical scalar. The number of bootstrap samples for the AUC standard error, used when boot_auc is TRUE. Default is 1000.
An object from which the following objects can be extracted:
DR_stats p-value of the D3 statistic, the D3 statistic, LRT fixed is the
likelihood Ratio test value of the constrained models.
stats_compare Mean of LogLik0, LogLik1, AIC0, AIC1, AIC_diff values of the
restricted (containing a 0) and full models (containing a 1).
NRI pooled values for the categorical and continuous Net Reclassification
improvement values and the Integrated Discrimination improvement.
AUC_stats Pooled Area Under the Curve of restricted and full models.
AUC_diff Pooled difference in AUC.
formula_test regression formula of full model.
cutoff Cutoff value used for reclassification values.
formula_null regression formula of null model
compare_predictors Predictors used in full model.
compare_group group of predictors used in full model.
The fit of the models are compared by using the D3 method for pooling Likelihood ratio
statistics (method of Meng and Rubin). The pooled AIC difference is calculated according to
the formula AIC = D - 2*p, where D is the pooled likelihood ratio tests of
constrained models (numerator in D3 statistic) and p is the difference in number of parameters
between the full and restricted models that are compared. The pooled AUC difference
is calculated, after the standard error is obtained in each imputed data set by method
DeLong or bootstrapping. The NRI categorical and continuous and IDI are calculated in each
imputed data set and pooled.
Eekhout I, van de Wiel MA, Heymans MW. Methods for significance testing of categorical covariates in logistic regression models after multiple imputation: power and applicability analysis. BMC Med Res Methodol. 2017;17(1):129.
Consentino F, Claeskens G. Order Selection tests with multiply imputed data Computational Statistics and Data Analysis.2010;54:2284-2295.
pool_lr <- psfmi_lr(data=lbpmilr, p.crit = 1, direction="FW", nimp=10, impvar="Impnr",
Outcome="Chronic", predictors=c("Radiation"), cat.predictors = ("Satisfaction"),
int.predictors = NULL, spline.predictors="Tampascale", nknots=3, method="D1")
res_compare <- pool_compare_models(pool_lr, compare.predictors = c("Pain", "Duration",
"Function"), cutoff = 0.4)
res_compare
#> $D3_stats
#> p-values D3 statistic LRT fixed
#> Pain 0.00925 7.85968 13.20684
#> Duration 0.90667 0.01375 0.01434
#> Function 0.05494 3.76136 4.55739
#>
#> $stats_compare
#> LogLik0 Loglik1 AIC0 AIC1 AIC_diff
#> Pain 146.1053 132.3418 158.1053 146.3418 11.206842
#> Duration 146.1053 146.0558 158.1053 160.0558 -1.985660
#> Function 146.1053 141.3747 158.1053 155.3747 2.557387
#>
#> $NRI
#> $NRI$Pain
#> NRI SE 95% CI LO 95% CI UP
#> NRI_cat 0.05536 0.06686 -0.07846 0.18919
#> NRI_ev 0.04203 0.04661 -0.05024 0.13429
#> NRI_ne 0.01333 0.04399 -0.07496 0.10163
#> NRI_cont 0.61237 0.17957 0.25664 0.96810
#> IDI 0.07144 0.02911 0.01250 0.13038
#>
#> $NRI$Duration
#> NRI SE 95% CI LO 95% CI UP
#> NRI_cat 0.00111 0.01854 -0.03731 0.03953
#> NRI_ev 0.00000 0.00963 -0.01969 0.01969
#> NRI_ne 0.00111 0.01921 -0.03982 0.04204
#> NRI_cont 0.07961 0.15268 -0.21988 0.37910
#> IDI 0.00029 0.00135 -0.00235 0.00293
#>
#> $NRI$Function
#> NRI SE 95% CI LO 95% CI UP
#> NRI_cat 0.03957 0.05027 -0.05995 0.13908
#> NRI_ev 0.03623 0.03757 -0.03791 0.11038
#> NRI_ne 0.00333 0.03595 -0.06886 0.07553
#> NRI_cont 0.39720 0.17485 0.05265 0.74174
#> IDI 0.02122 0.01315 -0.00462 0.04706
#>
#>
#> $cutoff
#> [1] 0.4
#>
#> $AUC
#> $AUC$Pain
#> 95% Low AUC 95% Up
#> AUC M0 0.7863 0.8567 0.9067
#> AUC M1 0.8196 0.8865 0.9307
#>
#> $AUC$Duration
#> 95% Low AUC 95% Up
#> AUC M0 0.7863 0.8567 0.9067
#> AUC M1 0.7872 0.8577 0.9077
#>
#> $AUC$Function
#> 95% Low AUC 95% Up
#> AUC M0 0.7863 0.8567 0.9067
#> AUC M1 0.7978 0.8648 0.9120
#>
#>
#> $AUC_diff
#> $AUC_diff$Pain
#> AUC SE 95% CI LO 95% CI HI
#> Diff DeLong 0.02974 0.0185 -0.00668 0.06616
#>
#> $AUC_diff$Duration
#> AUC SE 95% CI LO 95% CI HI
#> Diff DeLong 0.00103 0.00257 -0.00408 0.00614
#>
#> $AUC_diff$Function
#> AUC SE 95% CI LO 95% CI HI
#> Diff DeLong 0.00819 0.01105 -0.01351 0.02989
#>
#>
#> $formula_test
#> $formula_test$`Test - Pain`
#> [1] "Chronic ~ Radiation + factor(Satisfaction) + rcs(Tampascale, 3) + Pain"
#>
#> $formula_test$`Test - Duration`
#> [1] "Chronic ~ Radiation + factor(Satisfaction) + rcs(Tampascale, 3) + Duration"
#>
#> $formula_test$`Test - Function`
#> [1] "Chronic ~ Radiation + factor(Satisfaction) + rcs(Tampascale, 3) + Function"
#>
#>
#> $null_model
#> Chronic ~ Radiation + factor(Satisfaction) + rcs(Tampascale,
#> 3)
#> <environment: 0x0000020fbc166140>
#>
#> $compare_predictors
#> [1] "Pain" "Duration" "Function"
#>
#> $compare_group
#> NULL
#>