Approximate Bayesian computation
Public peer review comments can be seen here
Contents
- 1 History
- 2 Method
- 3 Example
- 4 Model comparison with ABC
- 5 Pitfalls and remedies
- 6 Software for application of ABC
- 7 Conclusion
- 8 Acknowledgements
- 9 References
Public peer review comments will be posted here
(*contributed equally; ^corresponding authors)
Approximate Bayesian computation (ABC) constitutes a class of computational methods rooted in Bayesian statistics. In all model-based statistical inference, the likelihood function is of central importance, since it expresses the probability of the observed data under a particular statistical model, and thus quantifies the support data lend to particular values of parameters and to choices among different models. For simple models, an analytical formula for the likelihood function can typically be derived. However, for more complex models, an analytical formula might be elusive or the likelihood function might be computationally very costly to evaluate.
ABC methods bypass the evaluation of the likelihood function. In this way, ABC methods widen the realm of models for which statistical inference can be considered. ABC methods are mathematically well-founded, but they inevitably make assumptions and approximations whose impact needs to be carefully assessed. Furthermore, the wider application domain of ABC exacerbates the challenges of parameter estimation and model selection.
ABC has rapidly gained popularity over the last years and in particular for the analysis of complex problems arising in biological sciences, e.g. in population genetics, ecology, epidemiology, and systems biology.
History
The first ABC-related ideas date back to 1980’s. Donald Rubin, when discussing the interpretation of Bayesian statements in 1984^{[1]}, described a hypothetical sampling mechanism that yields a sample from the posterior distribution. This scheme was more of a conceptual thought experiment to demonstrate what type of manipulations are done when inferring the posterior distributions of parameters. The description of the sampling mechanism coincides exactly with that of the ABC-rejection scheme, and this paper can be considered to be the first to describe approximate Bayesian computation. Another prescient point was made when Rubin argued that in Bayesian inference, applied statisticians should not settle for analytically tractable models only, but instead consider computational methods that allow them to estimate the posterior distribution of interest. This way, a wider range of models can be considered. These arguments are particularly relevant in the context of ABC.
In 1984, Peter Diggle and Richard Gratton^{[2]} suggested using a systematic simulation scheme to approximate the likelihood function in situations where its analytic form is intractable. Their method was based on defining a grid in the parameter space and using it to approximate the likelihood by running several simulations for each grid point. The approximation was then improved by applying smoothing techniques to the outcomes of the simulations. While the idea of using simulation for hypothesis testing was not new,^{[3]}^{[4]} Diggle and Gratton seemingly introduced the first procedure using simulation to do statistical inference under a circumstance where the likelihood is intractable.
Although Diggle and Gratton’s approach had opened a new frontier, their method was not yet exactly identical to what is now known as ABC, as it aimed at approximating the likelihood rather than the posterior distribution. A paper of Simon Tavaré et al.^{[5]} was first to propose an ABC algorithm for posterior inference. In their seminal work, inference about the genealogy of DNA sequence data was considered, and in particular the problem of deciding the posterior distribution of the time to the most recent common ancestor of the sampled individuals. Such inference is analytically intractable for many demographical models, but the authors presented ways of simulating coalescent trees under the putative models. A sample from the posterior of model parameters was obtained by accepting/rejecting proposals based on comparing the number of segregating sites in the synthetic and real data. This work was followed by an applied study on modeling the variation in human Y chromosome by Jonathan K. Pritchard et al.^{[6]} using the ABC method. Finally, the term Approximate Bayesian Computation was established by Mark Beaumont et al.,^{[7]} extending further the ABC methodology and discussing the suitability of the ABC-approach more specifically for problems in population genetics. Since then, ABC has spread to applications outside population genetics, such as systems biology, epidemiology, or phylogeography.
Method
Motivation
A common incarnation of the Bayes’ theorem relates the conditional probability (or density) of a particular parameter value Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} given data Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} to the probability of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} given Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} by the rule
- Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|D) = \frac{p(D|\theta)p(\theta)}{p(D)}}
where Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|D)} denotes the posterior, Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(D|\theta)} the likelihood, Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta)} the prior, and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(D)} the evidence (also referred to as the marginal likelihood or the prior predictive probability of the data).
The prior represents beliefs about Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} before D is available, and it is often specified by choosing a particular distribution among a set of well-known and tractable families of distributions, such that both the evaluation of prior probabilities and random generation of values of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} are relatively straightforward. For certain kinds of models, it is more pragmatic to specify the prior Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta)} using a factorization of the joint distribution of all the elements of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} in terms of a sequence of their conditional distributions. If one is only interested in the relative posterior plausibilities of different values of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} , the evidence Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(D)} can be ignored, as it constitutes a normalising constant, which cancels for any ratio of posterior probabilities. It remains, however, necessary to evaluate the likelihood Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(D|\theta)} and the prior Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta)} . For numerous applications, it is computationally expensive, or even completely infeasible, to evaluate the likelihood^{[8]}, which motivates the use of ABC to circumvent this issue.
The ABC rejection algorithm
All ABC based methods approximate the likelihood function by simulations, the outcomes of which are compared with the observed data.^{[9]}^{[10]}^{[11]} More specifically, with the ABC rejection algorithm—the most basic form of ABC—a set of parameter points is first sampled from the prior distribution. Given a sampled parameter point Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} , a data set Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \hat{D}} is then simulated under the statistical model Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M} specified by Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} . If the generated Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \hat{D}} is too different from the observed data Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} , the sampled parameter value is discarded. In precise terms, Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \hat{D}} is accepted with tolerance Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon \ge 0} if
- Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \rho (\hat{D},D)\le\epsilon}
where the distance measure Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \rho(\hat{D},D)} determines the level of discrepancy between Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \hat{D}} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} based on a given metric (e.g., the Euclidean distance). A strictly positive tolerance is usually necessary, since the probability that the simulation outcome coincides exactly with the data (event Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \hat{D}=D} ) is negligible for all but trivial applications of ABC, which would in practice lead to rejection of nearly all sampled parameter points. The outcome of the ABC rejection algorithm is a sample of parameter values approximately distributed according to the desired posterior distribution, and, crucially, obtained without the need of explicitly evaluating the likelihood function (Figure 1).
Summary statistics
The probability of generating a data set Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \hat{D}} with a small distance to Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} typically decreases as the dimensionality of the data increases. This leads to a substantial decrease in the computational efficiency of the above basic ABC rejection algorithm. A common approach to lessen this problem is to replace Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} with a set of lower dimensional summary statistics Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle S(D)} (frequency of transition between the two states), which are selected to capture the relevant information in Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} . The acceptance criterion in ABC rejection algorithm becomes
- Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \rho(S(\hat{D}),S(D))\le\epsilon}
If the summary statistics are sufficient with respect to the model parameters Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} , the efficiency increase obtained in this way does not introduce any error.^{[12]} Indeed, by definition, sufficiency implies that all information in Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} about Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} is captured by Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle S(D)} .
As elaborated below, it is typically impossible, outside the exponential family of distributions, to identify a finite-dimensional set of sufficient statistics. Nevertheless, informative, but possibly non-sufficient, summary statistics are often used in applications where inference is performed with ABC methods.
Example
An illustrative example is a bistable system that can be characterized by a hidden Markov model (HMM) subject to measurement noise (Figure 2). Such models are employed for many biological systems: they have for example been used in Development, signaling, activation/deactivation, logical processing and non-equilibrium thermodynamics. For instance, the behavior of the Sonic Hedgehog (Shh) transcription factor in Drosophila melanogaster can be modeled with a HMM.^{[13]} The (biological) dynamical model consists of two states; A and B. If the probability of a transition from one state to the other is defined as Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} in both directions, the probability to remain in the same state at each time step is 1-Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} . The probability to measure the state correctly is Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \gamma} (conversely, the probability of an incorrect measurement is 1-Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \gamma} ).
Due to the conditional dependencies between states at different time points, calculation of the likelihood of time series data is somewhat tedious, which illustrates the motivation to use ABC. A computational issue for the basic ABC is the large dimensionality of the data in an application like this. This can be reduced using the summary statistic S, which is the frequency of switches between the two states. As a distance measure Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \rho(\cdot,\cdot)} , the absolute difference is used, combined with a tolerance Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon=2} . The posterior inference about the parameter Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} can be done following the five steps presented in Figure 1:
Step 1: Assume that the observed data is the state sequence AAAABAABBAAAAAABAAAA, which was generated using Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta=0.25} . The associated summary statistic, the number of switches between the states in the experimental data, is Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \omega_E=6} .
Step 2: Assuming nothing is known about Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} , a uniform prior in the interval Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle [0,1]} is employed. A number n of parameter points are drawn from the prior, and the model is simulated for each of the parameter points Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta_i, i = 1,\ldots, n} , which results in Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle n} sequences of simulated data. In this example, n=5, with each drawn parameter and simulated dataset recorded in Table 1, column 2-3. In practice, n would need to be much larger to obtain an appropriate approximation.
i | Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta_i} | Simulated datasets (step 2) | Summary statistic Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \omega_{S,i}} (step 3) |
Distance Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \rho(\omega_{S,i}, \omega_E)}
(step 4) |
Outcome (step 4) |
---|---|---|---|---|---|
1 | 0.08 | AABAAAABAABAAABAAAAA | 8 | 2 | accepted |
2 | 0.68 | AABBABABAAABBABABBAB | 13 | 7 | rejected |
3 | 0.87 | BBBABBABBBBABABBBBBA | 9 | 3 | rejected |
4 | 0.43 | AABAAAAABBABBBBBBBBA | 6 | 0 | accepted |
5 | 0.53 | ABBBBBAABBABBABAABBB | 9 | 3 | rejected |
Step 3: The summary statistic is being computed for each sequence of simulated data, Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \omega_{S,i}, i = 1,\ldots,n} (Table 1, column 4).
Step 4: The distance between the observed and simulated transition frequencies Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \rho(\omega_{S,i}, \omega_E) = |\omega_{S,i} - \omega_{E}|} is computed for all parameter points (Table 1, column 5). Parameter points for which the distance is smaller than or equal to Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} are accepted as approximate samples from the posterior (Table 1, column 6).
Step 5: The posterior distribution is approximated with the accepted parameter points. The posterior distribution should have a non-negligible probability for parameter values in a region around the true value of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} in the system, if the data are sufficiently informative. In this example, the posterior probability mass is evenly split between the values 0.08 and 0.43.
Figure 3 shows the posterior probabilities obtained by ABC and a large n using either the summary statistic combined with (Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon = 0 } and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon = 2 } ) or the full data sequence. These are compared with the true posterior, which can be computed exactly and efficiently using the Viterbi algorithm. The used summary statistic is not sufficient, and it is seen that even with Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon = 0 } , the deviation from the theoretical posterior is considerable. Of note, a much longer observed data sequence would be required to obtain a posterior that is concentrated around the true value of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta} (Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \theta = 0.25} ).
This example application of ABC used simplifications for illustrative purposes. A number of review articles provide pointers to more realistic applications of ABC.^{[9]}^{[10]}^{[11]}^{[14]}
Model comparison with ABC
Besides parameter estimation, the ABC-framework can be used to compute the posterior probabilities of different candidate models.^{[15]}^{[16]}^{[17]} In such applications, one possibility is to use the rejection-sampling in a hierarchical manner. First, a model is sampled from the prior distribution for the models, then given the model sampled, the model parameters are sampled from the prior distribution assigned to that model. Finally, a simulation is performed as in the single-model ABC. The relative acceptance frequencies for the different models now approximate the posterior distribution for these models. Again, computational improvements for ABC in the space of models have been proposed, such as constructing a particle filter in the joint space of models and parameters.^{[17]}.
Once the posterior probabilities of models have been estimated, one can make full use of the techniques of Bayesian model comparison. For instance, to compare the relative plausibilities of two models Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_1} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_2} , one can compute their posterior ratio, which is related to the Bayes factor Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}} :
- Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \frac{p(M_1 |D)}{p(M_2|D)}=\frac{p(D|M_1)}{p(D|M_2)}\frac{p(M_1)}{p(M_2)} = B_{1,2}\frac{p(M_1)}{p(M_2)}}
If the model priors are equal (Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(M_1)=p(M_2)} ), the Bayes factor equals the posterior ratio.
In practice, as discussed below, these measures can be highly sensitive to the choice of parameter prior distributions and summary statistics, and thus conclusions of model comparison should be drawn with caution.
Pitfalls and remedies
Error source | Potential issue | Solution | Subsection |
---|---|---|---|
Non-zero tolerance ε | The inexactness introduces a bias in the computed posterior distribution. | Theoretical/practical studies of the sensitivity of the posterior distribution to the tolerance. / Noisy ABC. | #Approximation of the posterior |
Non-sufficient summary statistics | The information loss causes inflated credible intervals. | Automatic selection/semi-automatic identification of sufficient statistics. / Model validation checks (e.g. Templeton 2009^{[18]}). | #Choice and sufficiency of summary statistics |
Small nr of models/Mis-specified models | The investigated models are not representative/lack predictive power. | Careful selection of models./ Evaluation of the predictive power. | #Small number of models |
Priors and parameter ranges | Conclusions may be sensitive to the choice of priors. / Model choice may be meaningless. | Check sensitivity of Bayes factors to the choice of priors. / Some theoretical results regarding choice of priors are available. / Use alternative methods for model validation. | #Prior distribution and parameter ranges |
Curse-of-dimensionality | Low parameter acceptance rates. / Model errors cannot be distinguished from an insufficient exploration of the parameter space. / Risk of overfitting. | Methods for model reduction if applicable. / Methods to speed up the parameter exploration. / Quality controls to detect overfitting. | #Curse-of-dimensionality |
Model ranking with summary statistics | The computation of Bayes factors on summary statistics may not be related to the Bayes factors on the original data, which may therefore render the results meaningless. | Only use summary statistics that fulfill the necessary and sufficient conditions to produce a consistent Bayesian model choice. / Use alternative methods for model validation. | #Bayes factor with ABC and summary statistics |
Implementation | Low protection to common assumptions in the simulation and the inference process. | Sanity checks of results. / Standardization of software. | #Indispensable quality controls |
As for all statistical methods, a number of assumptions and approximations are inherently required for the application of ABC-based methods to real modeling problems. For example, setting the tolerance parameter Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon } to zero ensures an exact result, but typically makes computations prohibitively expensive. Thus, values of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} larger than zero are used in practice, which introduces a bias. Likewise, sufficient statistics are typically not available and instead, other summary statistics are used, which introduces an additional bias due to the loss of information. Additional sources of bias —e.g., in the context of model selection—may be more subtle.^{[12]}^{[19]}
At the same time, some of the criticisms that have been directed at the ABC methods, in particular within the field of phylogeography,^{[20]}^{[21]}^{[18]} are not specific to ABC and apply to all Bayesian methods or even all statistical methods (e.g., the choice of prior distribution and parameter ranges).^{[9]}^{[22]} However, because of the ability of ABC-methods to handle much more complex models, some of these general pitfalls are of particular relevance in the context of ABC analyses.
This section discusses these potential risks and reviews possible ways to address them (Table 2).
Approximation of the posterior
A non-negligible Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} comes with the price that one samples from Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|\rho(\hat{D},D)\le\epsilon)} instead of the true posterior Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|D)} . With a sufficiently small tolerance, and a sensible distance measure, the resulting distribution Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|\rho(\hat{D},D)\le\epsilon)} should often approximate the actual target distribution Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|D)} reasonably well. On the other hand, a tolerance that is large enough that every point in the parameter space becomes accepted will yield a replica of the prior distribution. There are empirical studies of the difference between Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|\rho(\hat{D},D)\le\epsilon)} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(\theta|D)} as a function of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} ,^{[23]} and theoretical results for an upper Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} -dependent bound for the error in parameter estimates.^{[24]} The accuracy of the posterior (defined as the expected quadratic loss) delivered by ABC as a function of Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} has also been investigated.^{[25]} However, the convergence of the distributions when Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} approaches zero, and how it depends on the distance measure used, is an important topic that has yet to be investigated in greater detail. In particular, it remains difficult to disentangle errors introduced by this approximation from errors due to model mis-specification.^{[9]}
As an attempt to correct some of the error due to a non-zero Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} , the usage of local linear weighted regression with ABC to reduce the variance of the posterior estimates has been suggested.^{[7]} The method assigns weights to the parameters according to how well simulated summaries adhere to the observed ones, and performs linear regression between the summaries and the weighted parameters in the vicinity of observed summaries. The obtained regression coefficients are used to correct sampled parameters in the direction of observed summaries. An improvement was suggested in the form of non-linear regression using a feed-forward neural network model.^{[26]} However, it has been shown that the posterior distributions obtained with these approaches are not always consistent with the prior distribution, which did lead to a reformulation of the regression adjustment that respects the prior distribution.^{[27]}
Finally, statistical inference using ABC with a non-zero tolerance Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} is not inherently flawed: under the assumption of measurement errors, the optimal Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \epsilon} can in fact be shown to be not zero.^{[25]}^{[28]} Indeed, the bias caused by a non-zero tolerance can be characterized and compensated by introducing a specific form of noise to the summary statistics. Asymptotic consistency for such "noisy ABC", has been established, together with formulas for the asymptotic variance of the parameter estimates for a fixed tolerance.^{[25]}
Choice and sufficiency of summary statistics
Summary statistics may be used to increase the acceptance rate of ABC for high-dimensional data. Low-dimensional sufficient statistics are optimal for this purpose, as they capture all relevant information present in the data in the simplest possible form.^{[11]} However, low-dimensional sufficient statistics are typically unattainable for statistical models where ABC-based inference is most relevant, and consequently, some heuristic is usually necessary to identify useful low-dimensional summary statistics. The use of a set of poorly chosen summary statistics will often lead to inflated credible intervals due to the implied loss of information,^{[11]} which can also bias the discrimination between models. A review of methods for choosing summary statistics is available,^{[29]} which may provide valuable guidance in practice.
One approach to capture most of the information present in data would be to use many statistics, but the accuracy and stability of ABC appears to decrease rapidly with an increasing numbers of summary statistics.^{[9]}^{[11]} Instead, a better strategy is to focus on the relevant statistics only—relevancy depending on the whole inference problem, on the model used, and on the data at hand.^{[30]}
An algorithm has been proposed for identifying a representative subset of summary statistics, by iteratively assessing whether an additional statistic introduces a meaningful modification of the posterior.^{[31]} One of the challenges here is that a large ABC approximation error may heavily influence the conclusions about the usefulness of a statistic at any stage of the procedure. Another method^{[30]} decomposes into two main steps. First, a reference approximation of the posterior is constructed by minimizing the entropy. Sets of candidate summaries are then evaluated by comparing the ABC-approximated posteriors with the reference posterior.
With both of these strategies, a subset of statistics is selected from a large set of candidate statistics. Instead, the partial least squares regression approach uses information from all the candidate statistics, each being weighted appropriately.^{[32]} Recently, a method for constructing summaries in a semi-automatic manner has attained a considerable interest.^{[25]} This method is based on the observation that the optimal choice of summary statistics, when minimizing the quadratic loss of the parameter point estimates, can be obtained through the posterior mean of the parameters, which is approximated by performing a linear regression based on the simulated data.
Methods for the identification of summary statistics that could also simultaneously assess the influence on the approximation of the posterior would be of substantial value. ^{[33]} This is because the choice of summary statistics and the choice of tolerance constitute two sources of error in the resulting posterior distribution. These errors may corrupt the ranking of models, and may also lead to incorrect model predictions. Indeed, none of the methods above assesses the choice of summaries for the purpose of model selection.
Bayes factor with ABC and summary statistics
It has been shown that the combination of insufficient summary statistics and ABC for model selection can be problematic.^{[12]}^{[19]} Indeed, if one lets the Bayes factor based on the summary statistic Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle S(D)} be denoted by Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}^s} , the relation between Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}^s} takes the form^{[12]}
- Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}=\frac{p(D|M_1)}{p(D|M_2)}=\frac{p(D|S(D),M_1)}{p(D|S(D),M_2)} \frac{p(S(D)|M_1)}{p(S(D)|M_2)}=\frac{p(D|S(D),M_1)}{p(D|S(D),M_2)} B_{1,2}^s}
Thus, a summary statistic Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle S(D)} is sufficient for comparing two models Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_1} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_2} if and only if,
- Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle p(D|S(D),M_1)=p(D|S(D),M_2)}
which results in that Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}=B_{1,2}^s} . It is also clear from the equation above that there might be a huge difference between Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle B_{1,2}^s} if the condition is not satisfied, as can be demonstrated by toy examples.^{[12]}^{[16]}^{[19]} Crucially, it was shown that sufficiency for Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_1} or Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_2} alone, or for both models, does not guarantee sufficiency for ranking the models.^{[12]} However, it was also shown that any sufficient summary statistic for a model Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M} in which both Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_1} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle M_2} are nested is valid for ranking the nested models.^{[12]}
The computation of Bayes factors on Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle S(D)} may therefore be misleading for model selection purposes, unless the ratio between the Bayes factors on Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle D} and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle S(D)} would be available, or at least could be approximated reasonably well. Alternatively, necessary and sufficient conditions on summary statistics for a consistent Bayesian model choice have recently been derived,^{[34]} which can provide useful guidance.
However, this issue is only relevant for model selection when the dimension of the data has been reduced. ABC-based inference, in which the actual data sets are directly compared—as is the case for some systems biology applications (e.g., see ^{[35]})—circumvents this problem.
Indispensable quality controls
As the above discussion makes clear, any ABC analysis requires choices and trade-offs that can have a considerable impact on its outcomes. Specifically, the choice of competing models/hypotheses, the number of simulations, the choice of summary statistics, or the acceptance threshold cannot currently be based on general rules, but the effect of these choices should be evaluated and tested in each study.^{[10]}
A number of heuristic approaches to the quality control of ABC have been proposed, such as the quantification of the fraction of parameter variance explained by the summary statistics.^{[10]} A common class of methods aims at assessing whether or not the inference yields valid results, regardless of the actually observed data. For instance, given a set of parameter values, which are typically drawn from the prior or the posterior distributions for a model, one can generate a large number of artificial data sets. In this way, the quality and robustness of ABC inference can be assessed in a controlled setting, by gauging how well the chosen ABC inference method recovers the true parameter values, and also models if multiple structurally different models are considered simultaneously.
Another class of methods assesses whether the inference was successful in light of the given observed data, for example, by comparing the posterior predictive distribution of summary statistics to the summary statistics observed.^{[10]} Beyond that, cross-validation techniques^{[36]} and predictive checks^{[37]}^{[38]} represent promising future strategies to evaluate the stability and out-of-sample predictive validity of ABC inferences. This is particularly important when modeling large data sets, because then the posterior support of a particular model can appear overwhelmingly conclusive, even if all proposed models in fact are poor representations of the stochastic system underlying the observation data. Out-of-sample predictive checks can reveal potential systematic biases within a model and provide clues on to how to improve its structure or parametrization.
Interestingly, fundamentally novel approaches for model choice that incorporate quality control as an integral step in the process have recently been proposed. ABC allows, by construction, estimation of the discrepancies between the observed data and the model predictions, with respect to a comprehensive set of statistics. These statistics are not necessarily the same as those used in the acceptance criterion. The resulting discrepancy distributions have been used for selecting models that are in agreement with many aspects of the data simultaneously,^{[39]} and model inconsistency is detected from conflicting and co-dependent summaries. Another quality-control based method for model selection employs ABC to approximate the effective number of model parameters and the deviance of the posterior predictive distributions of summaries and parameters.^{[40]} The deviance information criterion is then used as measure of model fit. It has also been shown that the models preferred based on this criterion can conflict with those supported by Bayes factors. For this reason, it is useful to combine different methods for model selection to obtain correct conclusions.
Quality controls are achievable and indeed performed in many ABC-based works, but for certain problems, the assessment of the impact of the method-related parameters can be challenging. However, the rapidly increasing use of ABC can be expected to provide a more thorough understanding of the limitations and applicability of the method.
General risks in statistical inference exacerbated in ABC
This section reviews risks that are strictly speaking not specific to ABC, but also relevant for other statistical methods as well. However, the flexibility offered by ABC to analyze very complex models makes them highly relevant to discuss here.
Prior distribution and parameter ranges
The specification of the range and the prior distribution of parameters strongly benefits from previous knowledge about the properties of the system. One criticism has been that in some studies the “parameter ranges and distributions are only guessed based upon the subjective opinion of the investigators” ^{[41]}, which is connected to classical objections of Bayesian approaches.^{[42]}
With any computational method, it is typically necessary to constrain the investigated parameter ranges. The parameter ranges should if possible be defined based on known properties of the studied system, but may for practical applications necessitate an educated guess. However, theoretical results regarding objective priors are available, which may for example be based on the principle of indifference or the principle of maximum entropy.^{[43]}^{[44]} On the other hand, automated or semi-automated methods for choosing a prior distribution often yield improper densities. As most ABC procedures require generating samples from the prior, improper priors are not directly applicable to ABC.
One should also keep the purpose of the analysis in mind when choosing the prior distribution. In principle, uninformative and flat priors, that exaggerate our subjective ignorance about the parameters, may still yield reasonable parameter estimates. However, Bayes factors are highly sensitive to the prior distribution of parameters. Conclusions on model choice based on Bayes factor can be misleading unless the sensitivity of conclusions to the choice of priors is carefully considered.
Small number of models
Model-based methods have been criticized for not exhaustively covering the hypothesis space.^{[21]}. Indeed, model-based studies often revolve around a small number of models and due to the high computational cost to evaluate a single model in some instances, it may then be difficult to cover a large part of the hypothesis space.
An upper limit to the number of considered candidate models is typically set by the substantial effort required to define the models and to choose between many alternative options.^{[10]} There is no commonly accepted ABC-specific procedure for model construction, so experience and prior knowledge are used instead.^{[11]} Although more robust procedures for a priori model choice and formulation would be beneficial, there is no one-size-fits-all strategy for model development in statistics: sensible characterization of complex systems will always necessitate a great deal of detective work and use of expert knowledge from the problem domain.
Some opponents of ABC contend that since only few models—subjectively chosen and probably all wrong—can be realistically considered, ABC analyses provide only limited insight.^{[21]} However, there is an important distinction between identifying a plausible null hypothesis and assessing the relative fit of alternative hypotheses.^{[9]} Since useful null hypotheses, that potentially hold true, can extremely seldom be put forward in the context of complex models, predictive ability of statistical models as explanations of complex phenomena is far more important than the test of a statistical null hypothesis in this context. It is also common to average over the investigated models, weighted based on their relative plausibility, to infer model features (e.g., parameter values) and to make predictions.
Large data sets
Large data sets may constitute a computational bottleneck for model-based methods. It was for example pointed out that in some ABC-based analyses, part of the data have to be omitted.^{[21]} A number of authors have argued that large data sets are not a practical limitation,^{[10]}^{[42]} although the severity of this issue depends strongly on the characteristics of the models. Several aspects of a modeling problem can contribute to the computational complexity, such as the sample size, number of observed variables or features, time or spatial resolution, etc. However, with increasing computing power, this issue will potentially be less important.
Instead of sampling parameters for each simulation from the prior, it has been proposed alternatively to combine the Metropolis-Hastings algorithm with ABC, which was reported to result in a higher acceptance rate than for plain ABC.^{[33]} Naturally, such an approach inherits the general burdens of MCMC methods, such as the difficulty to assess convergence, correlation among the samples from the posterior,^{[23]} and relatively poor parallelizability.^{[10]}
Likewise, the ideas of sequential Monte Carlo (SMC) and population Monte Carlo (PMC) methods have been adapted to the ABC setting.^{[23]}^{[45]} The general idea is to iteratively approach the posterior from the prior through a sequence of target distributions. An advantage of such methods, compared to ABC-MCMC, is that the samples from the resulting posterior are independent. In addition, with sequential methods the tolerance levels must not be specified prior to the analysis, but are adjusted adaptively.^{[46]}
It is relatively straightforward to parallelize a number of steps in ABC algorithms based on rejection sampling and sequential Monte Carlo methods. It has also been demonstrated that parallel algorithms may yield significant speedups for MCMC based inference in phylogenetics,^{[47]}, which may be a tractable approach also for ABC based methods. Yet an adequate model for a complex system is very likely to require intensive computation, irrespectively of the chosen method of inference, and it is up to the user to select a method that is suitable for the particular application in question.
Curse-of-dimensionality
High-dimensional data sets and high-dimensional parameter spaces can require an extremely large number of parameter points to be simulated in ABC based studies to obtain a reasonable level of accuracy for the posterior inferences. In such situations, the computational cost is severely increased and may in the worst case render the computational analysis intractable. These are examples of well-known phenomena, which are usually referred to with the umbrella term curse-of-dimensionality.^{[48]}
To assess how severely the dimensionality of a data set affects the analysis within the context of ABC, analytical formulae have been derived for the error of the ABC estimators as functions of the dimension of the summary statistics.^{[49]}^{[50]} In addition, Blum and François have investigated how the dimension of the summary statistics is related to the mean squared error for different correction adjustments to the error of ABC estimators. It was also argued that dimension reduction techniques are useful to avoid the curse-of-dimensionality, due to a potentially lower dimensional underlying structure of summary statistics.^{[49]} Motivated by minimizing the quadratic loss of ABC estimators, Fearnhead and Prangle have proposed a scheme to project (possibly high-dimensional) data into estimates of the parameter posterior means; these means, now having the same dimension as the parameters, are then used as summary statistics for ABC.^{[50]}
ABC can be used to infer problems in high-dimensional parameter spaces, although one should account for the possibility of overfitting (e.g., see the model selection methods in ^{[39]} and ^{[40]}). However, the probability of accepting the simulated values for the parameters under a given tolerance with the ABC rejection algorithm typically decreases exponentially with increasing dimensionality of the parameter space (due to the global acceptance criterion).^{[11]} Although no computational method (based on ABC or not) seems to be able to break the curse-of-dimensionality, methods have recently been developed to handle high-dimensional parameter spaces under certain assumptions (e.g., based on polynomial approximation on sparse grids,^{[51]} which could potentially heavily reduce the simulation times for ABC). However, the applicability of such methods is problem dependent, and the difficulty of exploring parameter spaces should in general not be underestimated. For example, the introduction of deterministic global parameter estimation led to reports that the global optima obtained in several previous studies of low-dimensional problems were incorrect.^{[52]} For certain problems, it might therefore be difficult to know whether the model is incorrect or, as discussed above, whether the explored region of the parameter space is inappropriate.^{[21]} A more pragmatic approach is to cut the scope of the problem through model reduction ^{[11]}.
Software for application of ABC
A number of software packages are currently available for application of ABC to particular classes of statistical models. An assortment of ABC based software is presented in Table 3.
Software | Keywords and features | Reference |
---|---|---|
DIY-ABC | Software for fit of genetic data to complex situations. Comparison of competing models. Parameter estimation. Computation of bias and precision measures for a given model and known parameters values. | ^{[53]} |
ABC R package | Several ABC algorithms for performing parameter estimation and model selection. Nonlinear heteroscedastic regression methods for ABC. Cross-validation tool. | ^{[54]} |
ABC-SysBio | Python package. Parameter inference and model selection for dynamical systems. Combines ABC rejection sampler, ABC SMC for parameter inference, and ABC SMC for model selection. Compatible with models written in Systems Biology Markup Language (SBML). Deterministic and stochastic models. | ^{[55]} |
ABCtoolbox | Open source programs for various ABC algorithms including rejection sampling, MCMC without likelihood, a particle-based sampler, and ABC-GLM. Compatibility with most simulation and summary statistics computation programs. | ^{[56]} |
msBayes | Open source software package consisting of several C and R programs that are run with a Perl "front-end". Hierarchical coalescent models. Population genetic data from multiple co-distributed species. | ^{[57]} |
PopABC | Software package for inference of the pattern of demographic divergence. Coalescent simulation. Bayesian model choice. | ^{[58]} |
ONeSAMP | Web-based program to estimate the effective population size from a sample of microsatellite genotypes. Estimates of effective population size, together with 95% credible limits. | ^{[59]} |
ABC4F | Software for estimation of F-statistics for dominant data. | ^{[60]} |
2BAD | 2-event Bayesian ADmixture. Software allowing up to two independent admixture events with up to three parental populations. Estimation of several parameters (admixture, effective sizes, etc.). Comparison of pairs of admixture models. | ^{[61]} |
The suitability of individual software packages are dependent on the specific application at hand, the computer system environment, and the algorithms required.
Conclusion
In conclusion, ABC represents a class of well-founded and powerful methods for Bayesian statistical inference. However, reliable application of ABC requires additional caution to be considered, due to the approximations and biases introduced at the different stages of the approach. In its current incarnation, the ABC toolkit as a whole is best suited for inference about parameters or predictive inferences about observables in the presence of a single or few candidate model(s). How to make ABC practically feasible for problems involving large sets of models and/or high-dimensional target parameter spaces is currently largely an open issue. Since the computation of the likelihood function is bypassed, it can be tempting to attack high-dimensional problems using ABC, but inevitably this comes bundled with new challenges that investigators need to be aware of at each step of their analyses.
Acknowledgements
We would like to thank Matthew Astley, Joachim M. Buhmann, Sotiris Dimopoulos, Nick Goldman, Darren Logan, Joerg Stelling, and Elias Zamora-Sillero for useful discussions and/or feedback on the manuscript. We also thank Christian P. Robert and Dennis Prangle for their constructive referee reports, and Daniel Mietchen for editorial work and help to bring the article up to Wikipedia standards. MS was supported by SystemsX.ch (RTD project YeastX). AGB was supported by SystemsX.ch (RTD projects YeastX and LiverX). JC was supported by the European Research Council grant no. 239784. EN was supported by the FICS graduate school. MF was supported by a Swiss NSF grant No 3100-126074 to Laurent Excoffier. CD was supported by SNSF advanced researcher fellowship no. 136461. This article started as assignment for the graduate course ‘Reviews in Computational Biology’ (263-5151-00L) at ETH Zurich.
References
<references> ^{[9]} ^{[10]} ^{[11]} ^{[1]} ^{[33]} ^{[23]} ^{[32]} ^{[20]} ^{[21]} ^{[18]} ^{[22]} ^{[12]} ^{[19]} ^{[8]} ^{[36]} ^{[37]} ^{[38]} ^{[39]} ^{[40]} ^{[45]} ^{[46]} ^{[7]} ^{[26]} ^{[27]} ^{[42]} ^{[41]} ^{[56]} ^{[43]} ^{[47]} ^{[48]} ^{[51]} ^{[52]} ^{[24]} ^{[25]} ^{[28]} ^{[30]} ^{[31]} ^{[16]} ^{[34]} ^{[35]} ^{[5]} ^{[17]} ^{[6]} ^{[2]} ^{[4]} ^{[13]} ^{[3]} ^{[29]} ^{[50]} ^{[49]} ^{[53]} ^{[54]} ^{[55]} ^{[57]} ^{[58]} ^{[59]} ^{[60]} ^{[61]} ^{[14]} ^{[15]} ^{[44]}
- ^ ^{a} ^{b} Rubin DB (1984) Bayesianly Justifiable and Relevant Frequency Calculations for the Applies Statistician. The Annals of Statistics 12: pp. 1151-1172.
- ^ ^{a} ^{b} Diggle PJ, J. GR. (1984) Monte Carlo Methods of Inference for Implicit Statistical Models, Journal of the Royal Statistical Society, Series B 46:193-227.
- ^ ^{a} ^{b} Bartlett MS (1963) The spectral analysis of point processes. Journal of the Royal Statistical Society, Series B 25:264-296
- ^ ^{a} ^{b} Hoel DG, Mitchell TJ (1971) The simulation, fitting and testing of a stochastic cellular proliferation model. Biometrics, 27, 191-199.
- ^ ^{a} ^{b} Tavare S, Balding DJ, Griffiths RC et al. (1997) Inferring Coalescence Times From DNA Sequence Data, Genetics 145:505-518.
- ^ ^{a} ^{b} Pritchard JK, Seielstad MT, Perez-Lezaun A et al. (1999) Population Growth of Human Y Chromosomes: A Study of Y Chromosome Microsatellites, Molecular Biology and Evolution 16:1791-1798.
- ^ ^{a} ^{b} ^{c} Beaumont MA, Zhang W, Balding DJ (2002) Approximate Bayesian Computation in Population Genetics. Genetics 162: 2025-2035.
- ^ ^{a} ^{b} Busetto A.G., Buhmann J. Stable Bayesian Parameter Estimation for Biological Dynamical Systems.; 2009. IEEE Computer Society Press pp. 148-157.
- ^ ^{a} ^{b} ^{c} ^{d} ^{e} ^{f} ^{g} Beaumont MA (2010) Approximate Bayesian Computation in Evolution and Ecology. Annual Review of Ecology, Evolution, and Systematics 41: 379-406.
- ^ ^{a} ^{b} ^{c} ^{d} ^{e} ^{f} ^{g} ^{h} ^{i} Bertorelle G, Benazzo A, Mona S (2010) ABC as a flexible framework to estimate demography over space and time: some cons, many pros. Molecular Ecology 19: 2609-2625.
- ^ ^{a} ^{b} ^{c} ^{d} ^{e} ^{f} ^{g} ^{h} ^{i} Csilléry K, Blum MGB, Gaggiotti OE, François O (2010) Approximate Bayesian Computation (ABC) in practice. Trends in Ecology & Evolution 25: 410-418.
- ^ ^{a} ^{b} ^{c} ^{d} ^{e} ^{f} ^{g} ^{h} Didelot X, Everitt RG, Johansen AM, Lawson DJ (2011) Likelihood-free estimation of model evidence. Bayesian Analysis 6: 49-76.
- ^ ^{a} ^{b} Lai K, Robertson MJ, Schaffer DV (2004) The sonic hedgehog signaling system as a bistable genetic switch., Biophys J.: 86,2748-2757.
- ^ ^{a} ^{b} Marin JM, Pudlo P, Robert CP, Ryder RJ (2011) Approximate Bayesian computational methods, Statistics and Computing
- ^ ^{a} ^{b} Wilkinson, R. G. (2007). Bayesian Estimation of Primate Divergence Times, Ph.D. thesis, University of Cambridge.
- ^ ^{a} ^{b} ^{c} Grelaud A, Marin J-M, Robert C, Rodolphe F, Tally F (2009) Likelihood-free methods for model choice in Gibbs random fields. Bayesian Analysis 3: 427-442.
- ^ ^{a} ^{b} ^{c} Toni T, Stumpf MPH (2010). Simulation-based model selection for dynamical systems in systems and population biology, Bioinformatics 26 (1):104–10.
- ^ ^{a} ^{b} ^{c} Templeton AR (2009) Why does a method that fails continue to be used? The answer. Evolution 63: 807-812.
- ^ ^{a} ^{b} ^{c} ^{d} Robert CP, Cornuet J-M, Marin J-M, Pillai NS (2011) Lack of confidence in approximate Bayesian computation model choice. Proc Natl Acad Sci U S A 108: 15112-15117.
- ^ ^{a} ^{b} Templeton AR (2008) Nested clade analysis: an extensively validated method for strong phylogeographic inference. Molecular Ecology 17: 1877-1880.
- ^ ^{a} ^{b} ^{c} ^{d} ^{e} ^{f} Templeton AR (2009) Statistical hypothesis testing in intraspecific phylogeography: nested clade phylogeographical analysis vs. approximate Bayesian computation. Molecular Ecology 18: 319-331.
- ^ ^{a} ^{b} Berger JO, Fienberg SE, Raftery AE, Robert CP (2010) Incoherent phylogeographic inference. Proceedings of the National Academy of Sciences of the United States of America 107: E157-E157.
- ^ ^{a} ^{b} ^{c} ^{d} Sisson SA, Fan Y, Tanaka MM (2007) Sequential Monte Carlo without likelihoods. Proc Natl Acad Sci U S A 104: 1760-1765.
- ^ ^{a} ^{b} Dean TA, Singh SS, Jasra A, Peters GW (2011) Parameter estimation for hidden markov models with intractable likelihoods. arXiv:11035399v1 [mathST] 28 Mar 2011.
- ^ ^{a} ^{b} ^{c} ^{d} ^{e} Fearnhead P, Prangle D (2011) Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC. ArXiv:10041112v2 [statME] 13 Apr 2011.
- ^ ^{a} ^{b} Blum M, Francois O (2010) Non-linear regression models for approximate Bayesian computation. Stat Comp 20: 63-73.
- ^ ^{a} ^{b} Leuenberger C, Wegmann D (2009) Bayesian Computation and Model Selection Without Likelihoods. Genetics 184: 243-252.
- ^ ^{a} ^{b} Wilkinson RD (2009) Approximate Bayesian computation (ABC) gives exact results under the assumption of model error. arXiv:08113355.
- ^ ^{a} ^{b} Blum MGB, Nunes MA, Prangle D, Sisson SA (2012) A comparative review of dimension reduction methods in approximate Bayesian computation. arxiv.org/abs/1202.3819
- ^ ^{a} ^{b} ^{c} Nunes MA, Balding DJ (2010) On optimal selection of summary statistics for approximate Bayesian computation. Stat Appl Genet Mol Biol 9: Article 34.
- ^ ^{a} ^{b} Joyce P, Marjoram P (2008) Approximately sufficient statistics and bayesian computation. Stat Appl Genet Mol Biol 7: Article 26.
- ^ ^{a} ^{b} Wegmann D, Leuenberger C, Excoffier L (2009) Efficient approximate Bayesian computation coupled with Markov chain Monte Carlo without likelihood. Genetics 182: 1207-1218.
- ^ ^{a} ^{b} ^{c} Marjoram P, Molitor J, Plagnol V, Tavare S (2003) Markov chain Monte Carlo without likelihoods. Proc Natl Acad Sci U S A 100: 15324-15328.
- ^ ^{a} ^{b} Marin J-M, Pillai NS, Robert CP, Rosseau J (2011) Relevant statistics for Bayesian model choice. ArXiv:11104700v1 [mathST] 21 Oct 2011: 1-24.
- ^ ^{a} ^{b} Toni T, Welch D, Strelkowa N, Ipsen A, Stumpf M (2007) Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems. J R Soc Interface 6: 187-202.
- ^ ^{a} ^{b} Arlot S, Celisse A (2010) A survey of cross-validation procedures for model selection. Statistical surveys 4: 40-79.
- ^ ^{a} ^{b} Dawid A Present position and potential developments: Some personal views: Statistical theory: The prequential approach. Journal of the Royal Statistical Society Series A 1984: 278-292.
- ^ ^{a} ^{b} Vehtari A, Lampinen J (2002) Bayesian model assessment and comparison using cross-validation predictive densities. Neural Computation 14: 2439-2468.
- ^ ^{a} ^{b} ^{c} Ratmann O, Andrieu C, Wiuf C, Richardson S (2009) Model criticism based on likelihood-free inference, with an application to protein network evolution. Proceedings of the National Academy of Sciences of the United States of America 106: 10576-10581.
- ^ ^{a} ^{b} ^{c} Francois O, Laval G (2011) Deviance Information Criteria for Model Selection in Approximate Bayesian Computation. Stat Appl Genet Mol Biol 10: Article 33.
- ^ ^{a} ^{b} Templeton AR (2010) Coherent and incoherent inference in phylogeography and human evolution. Proceedings of the National Academy of Sciences of the United States of America 107: 6376-6381.
- ^ ^{a} ^{b} ^{c} Beaumont MA, Nielsen R, Robert C, Hey J, Gaggiotti O, et al. (2010) In defence of model-based inference in phylogeography. Molecular Ecology 19: 436-446.
- ^ ^{a} ^{b} Jaynes ET (1968) Prior Probabilities. IEEE Transactions on Systems Science and Cybernetics 4.
- ^ ^{a} ^{b} Berger, J. O. (2006). The case for objective Bayesian analysis. Bayesian Analysis 1 385–402 and 457–464.
- ^ ^{a} ^{b} Beaumont MA, Cornuet J-M, Marin J-M, Robert CP (2009) Adaptive approximate Bayesian computation. Biometrika 96: 983-990.
- ^ ^{a} ^{b} Del Moral P, Doucet A, Jasra A (2011 (in press)) An adaptive sequential Monte Carlo method for approximate Bayesian computation. Statistics and computing.
- ^ ^{a} ^{b} Feng X, Buell DA, Rose JR, Waddellb PJ (2003) Parallel Algorithms for Bayesian Phylogenetic Inference. Journal of Parallel and Distributed Computing 63: 707-718.
- ^ ^{a} ^{b} Bellman R (1961) Adaptive Control Processes: A Guided Tour: Princeton University Press.
- ^ ^{a} ^{b} ^{c} Blum MGB (2010) Approximate Bayesian Computation: a nonparametric perspective, Journal of the American Statistical Association (105): 1178-1187
- ^ ^{a} ^{b} ^{c} Fearnhead P, Prangle D (2012) Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate Bayesian computation, Journal of the Royal Statistical Society Series B 74(3): 419–474.
- ^ ^{a} ^{b} Gerstner T, Griebel M (2003) Dimension-Adaptive Tensor-Product Quadrature. Computing 71: 65-87.
- ^ ^{a} ^{b} Singer AB, Taylor JW, Barton PI, Green WH (2006) Global dynamic optimization for parameter estimation in chemical kinetics. J Phys Chem A 110: 971-976.
- ^ ^{a} ^{b} Cornuet J-M, Santos F, Beaumont M, et al. (2008) Inferring population history with DIY ABC: a user-friendly approach to approximate Bayesian computation, Bioinformatics, 24(23): 2713-2719
- ^ ^{a} ^{b} Csilléry K, François O, Blum MGB (2012), abc: an R package for approximate Bayesian computation (ABC) Methods in Ecology and Evolution, 3: 475–479
- ^ ^{a} ^{b} Liepe J, Barnes C, Cule E, Erguler K, Kirk P, Toni T, Stumpf MP (2010) ABC-SysBio—approximate Bayesian computation in Python with GPU support, Bioinformatics, 26: 1797–1799
- ^ ^{a} ^{b} Wegmann D, Leuenberger C, Neuenschwander S, Excoffier L (2010) ABCtoolbox: a versatile toolkit for approximate Bayesian computations. BMC Bioinformatics 11: 116.
- ^ ^{a} ^{b} Hickerson MJ, Stahl E, Takebayashi N (2007) msBayes: Pipeline for testing comparative phylogeographic histories using hierarchical approximate Bayesian computation, BMC Bioinformatics, 8, 268: 1471-2105
- ^ ^{a} ^{b} Lopes JS, Balding D, Beaumont MA (2009) PopABC: a program to infer historical demographic parameters, Bioinformatics, 25: 2747–2749
- ^ ^{a} ^{b} Tallmon DA, Koyuk A, Luikart G, Beaumont MA (2008) COMPUTER PROGRAMS: onesamp: a program to estimate effective population size using approximate Bayesian computation, Molecular Ecology Resources, 8: 299–301
- ^ ^{a} ^{b} Foll M, Baumont MA, Gaggiotti OE (2008) An Approximate Bayesian Computation approach to overcome biases that arise when using AFLP markers to study population structure, Genetics, 179: 927-939
- ^ ^{a} ^{b} Bray TC, Sousa VC, Parreira B, Bruford MW, Chikhi L (2010) 2BAD: an application to estimate the parental contributions during two independent admisture events, Molecular Ecology Resources, 10(3): 538-541