@bobbrown@patrick I am just following Patrick’s advice and posting the few things I know about VPC here. (I am not an expert in vpc by any means, pls feel free to correct me)
Probably you have already known, vpc is a simulation-based graphical diagnostic tool to evaluate a given model. What we do is first we simulate a dataset for multiple times using the final estimated parameters of the model. Then we compare different percentiles (median, 25th percentiles, 75th percentiles) of the simulated data to the corresponding percentiles of the observed data. The usual diagnostic tools, i.e. individual fits, goodness of fit, residual plots, are more about diagnosing a model at the individual level. VPC could help us to diagnose a model as a whole in one graph. We would be able to answer questions like '‘whether the average/high/low level of my data is being captured adequately or not’. It has another benefit: there are cases where individual fits look fine, but the model is indeed misspecified. I am attaching a document from Drs. Holford and Karlsson. They illustrated how misspecification of a model was hidden in the individual fits but was revealed in the VPC plot when shrinkage is high (page 21-25). Therefore, it could be more useful. https://www.page-meeting.org/pdf_assets/8694-Karlsson_Holford_VPC_Tutorial_hires.pdf.
@shli6161 Thank you for sharing your experience and the reference. Few thoughts of my own:
There was a reference to individual fits vs vpc results. Aren’t the simulations independent of the observed data for vpc? If so, the individual fits represent more so the structural model. However, the distribution of simulated data is being compared to that of the observed for vpc - which seems to be as a population level comparison.
The authors - they are famous scientists in this field. I will tread carefully. An effect compartmental model and an indirect response model (another famous scientist developed this, Dr Gerhard Levy also a pharmacist by training) were fitted to the warfarin data. Then vpc was applied. My understanding is that the choice of the PD model here ought to be driven by the biology and pharmacology, and not based on statistical reasoning. Does this make sense? Meaning, take the contrary outcome. If vpc showed both models performed similarly then would I accept the effect compartment model? I dont think that seems like a prudent thing to do. Thoughts?
Bob
No problem. These are really good points and I am happy to share my thoughts too:
I wouldn’t say the simulation results are definitely independent of the observed data- it depends on how you do the vpc. Most of the time, what we do is the non-parametric vpc where the final estimated parameters from the model were used for simulation. These parameters were essentially estimated from the data itself; so, the simulation results are still dependent on the data. To me, both individual fits and non-parametric VPC are dependent on the data itself. Individual fits are to check the model performance on an individual level, data point by data point. VPC is just a way to tell on average how well the model performs in capturing both the median and the variability (25th, 75th percentiles) of the data, and what is the uncertainty around the results.
Definitely! The pharmacological mechanism should override the statistics reasoning if we have a good idea what the mechanism is. Like the warfarin case: we know that warfarin’s mechanism is to inhibit prothrombin formation; therefore, only the turnover delay model makes sense (even if the statistics says otherwise). The example in the doc is mainly for illustrating the merits of vpc: they simulated a dataset using a turnover model and then estimated the simulated data using both the effect compartment delay model (the false model) and the turnover delay model (the true model). Under the case of high shrinkage, individual fits can not discern the false model from the true model, but vpc can. It is just a theoretical example where we know what the true model is. But in reality, we don’t know what the true model is, and we don’t always know what causes the effect delay (unlike warfarin). So let’s say we are trying to model the data of a new moiety with a delay in the PD effect. But we don’t know the reason for the delay. In this case, VPC might provide more insights if the individual fits couldn’t.
Does anyone know the statistical reasons behind why individual fits lose their magic under high shrinkage situations, but vpc doesn’t?
I wouldn’t say the simulation results are definitely independent of the observed data- it depends on how you do the vpc. Most of the time, what we do is the non-parametric vpc where the final estimated parameters from the model were used for simulation. These parameters were essentially estimated from the data itself; so, the simulation results are still dependent on the data. To me, both individual fits and non-parametric VPC are dependent on the data itself.
I believe most VPCs are actually done with parametric simulations, so the simulations are dependent on the model, not the observed data (at least in terms of how we usu think of this). Non-parametric simulations are what’s usu considered as being dependent on the data, since you will do something like a bootstrap.
Does anyone know the statistical reasons behind why individual fits lose their magic under high shrinkage situations, but vpc doesn’t?
I believe this is more so with epsilon shrinkage than eta shrinkage. It’s fairly intuitive when there’s epsilon shrinkage. But, the PPT you linked mentioned that this also can happen with high eta shrinkage and random correlation.
The main concerns with high shrinkage are noted here:
Identified consequences of η-shrinkage on EBE-based model diagnostics include non-normal and/or asymmetric distribution of EBEs with their mean values (“ETABAR”) significantly different from zero, even for a correctly specified model; EBE–EBE correlations and covariate relationships may be masked, falsely induced, or the shape of the true relationship distorted. Consequences of ε-shrinkage included low power of IPRED and IWRES to diagnose structural and residual error model misspecification, respectively.