A comparison of approximate versus exact techniques for Bayesian parameter inference in nonlinear ordinary differential equation models

The behaviour of many processes in science and engineering can be accurately described by dynamical system models consisting of a set of ordinary differential equations (ODEs). Often these models have several unknown parameters that are difficult to estimate from experimental data, in which case Bayesian inference can be a useful tool. In principle, exact Bayesian inference using Markov chain Monte Carlo (MCMC) techniques is possible; however, in practice, such methods may suffer from slow convergence and poor mixing. To address this problem, several approaches based on approximate Bayesian computation (ABC) have been introduced, including Markov chain Monte Carlo ABC (MCMC ABC) and sequential Monte Carlo ABC (SMC ABC). While the system of ODEs describes the underlying process that generates the data, the observed measurements invariably include errors. In this paper, we argue that several popular ABC approaches fail to adequately model these errors because the acceptance probability depends on the choice of the discrepancy function and the tolerance without any consideration of the error term. We observe that the so-called posterior distributions derived from such methods do not accurately reflect the epistemic uncertainties in parameter values. Moreover, we demonstrate that these methods provide minimal computational advantages over exact Bayesian methods when applied to two ODE epidemiological models with simulated data and one with real data concerning malaria transmission in Afghanistan.

The reviewers and handling editors have recommended publication, but also suggest some minor revisions to your manuscript. Therefore, I invite you to respond to the comments and revise your manuscript.
• Ethics statement If your study uses humans or animals please include details of the ethical approval received, including the name of the committee that granted approval. For human studies please also detail whether informed consent was obtained. For field studies on animals please include details of all permissions, licences and/or approvals granted to carry out the fieldwork.
• Data accessibility It is a condition of publication that all supporting data are made available either as supplementary information or preferably in a suitable permanent repository. The data accessibility section should state where the article's supporting data can be accessed. This section should also include details, where possible of where to access other relevant research materials such as statistical tools, protocols, software etc can be accessed. If the data has been deposited in an external repository this section should list the database, accession number and link to the DOI for all data from the article that has been made publicly available. Data sets that have been deposited in an external repository and have a DOI should also be appropriately cited in the manuscript and included in the reference list.
If you wish to submit your supporting data or code to Dryad (http://datadryad.org/), or modify your current submission to dryad, please use the following link: http://datadryad.org/submit?journalID=RSOS&manu=RSOS-191315 • Competing interests Please declare any financial or non-financial competing interests, or state that you have no competing interests.
• Authors' contributions All submissions, other than those with a single author, must include an Authors' Contributions section which individually lists the specific contribution of each author. The list of Authors should meet all of the following criteria; 1) substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published.
All contributors who do not meet all of these criteria should be included in the acknowledgements.
We suggest the following format: AB carried out the molecular lab work, participated in data analysis, carried out sequence alignments, participated in the design of the study and drafted the manuscript; CD carried out the statistical analyses; EF collected field data; GH conceived of the study, designed the study, coordinated the study and helped draft the manuscript. All authors gave final approval for publication.
• Acknowledgements Please acknowledge anyone who contributed to the study but did not meet the authorship criteria.
• Funding statement Please list the source of funding for each author.
Please ensure you have prepared your revision in accordance with the guidance at https://royalsociety.org/journals/authors/author-guidelines/ --please note that we cannot publish your manuscript without the end statements. We have included a screenshot example of the end statements for reference. If you feel that a given heading is not relevant to your paper, please nevertheless include the heading and explicitly state that it is not relevant to your work.
Because the schedule for publication is very tight, it is a condition of publication that you submit the revised version of your manuscript before 25-Dec-2019. Please note that the revision deadline will expire at 00.00am on this date. If you do not think you will be able to meet this date please let me know immediately.
To revise your manuscript, log into https://mc.manuscriptcentral.com/rsos and enter your Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions". Under "Actions," click on "Create a Revision." You will be unable to make your revisions on the originally submitted version of the manuscript. Instead, revise your manuscript and upload a new version through your Author Centre.
When submitting your revised manuscript, you will be able to respond to the comments made by the referees and upload a file "Response to Referees" in "Section 6 -File Upload". You can use this to document any changes you make to the original manuscript. In order to expedite the processing of the revised manuscript, please be as specific as possible in your response to the referees. We strongly recommend uploading two versions of your revised manuscript: 1) Identifying all the changes that have been made (for instance, in coloured highlight, in bold text, or tracked changes); 2) A 'clean' version of the new manuscript that incorporates the changes made, but does not highlight them.
When uploading your revised files please make sure that you have: 1) A text file of the manuscript (tex, txt, rtf, docx or doc), references, tables (including captions) and figure captions. Do not upload a PDF as your "Main Document"; 2) A separate electronic file of each figure (EPS or print-quality PDF preferred (either format should be produced directly from original creation package), or original software format); 3) Included a 100 word media summary of your paper when requested at submission. Please ensure you have entered correct contact details (email, institution and telephone) in your user account; 4) Included the raw data to support the claims made in your paper. You can either include your data as electronic supplementary material or upload to a repository and include the relevant doi within your manuscript. Make sure it is clear in your data accessibility statement how the data can be accessed; 5) All supplementary materials accompanying an accepted article will be treated as in their final form. Note that the Royal Society will neither edit nor typeset supplementary material and it will be hosted as provided. Please ensure that the supplementary material includes the paper details where possible (authors, article title, journal name).
Supplementary files will be published alongside the paper on the journal website and posted on the online figshare repository (https://rs.figshare.com/). The heading and legend provided for each supplementary file during the submission process will be used to create the figshare page, so please ensure these are accurate and informative so that your files can be found in searches. Files on figshare will be made available approximately one week before the accompanying article so that the supplementary material can be attributed a unique DOI.
Please note that Royal Society Open Science charge article processing charges for all new submissions that are accepted for publication. Charges will also apply to papers transferred to Royal Society Open Science from other Royal Society Publishing journals, as well as papers submitted as part of our collaboration with the Royal Society of Chemistry (https://royalsocietypublishing.org/rsos/chemistry).
If your manuscript is newly submitted and subsequently accepted for publication, you will be asked to pay the article processing charge, unless you request a waiver and this is approved by Royal Society Publishing. You can find out more about the charges at https://royalsocietypublishing.org/rsos/charges. Should you have any queries, please contact openscience@royalsociety.org.
Once again, thank you for submitting your manuscript to Royal Society Open Science and I look forward to receiving your revision. If you have any questions at all, please do not hesitate to get in touch. The manuscript has been reviewed by three researchers, all of whom were positive about it. However, the first reviewer in particular raised a number of points that need addressing before the manuscript can be published, so I am recommending one more round of revisions. Please address the concerns of this reviewer, and the minor points raised by the second reviewer; I will be happy to look over a revised version and make a recommendation without any further need for peer review. The first reviewer also noted that they obtained an error message when they tried to run the code associated with the paper, so please also double check the code. It is a pleasure to accept your manuscript entitled "A comparison of approximate versus exact techniques for Bayesian parameter inference in non-linear ODE models" in its current form for publication in Royal Society Open Science. The comments of the reviewer(s) who reviewed your manuscript are included at the foot of this letter.
We note that 'davis.cochrane@monash.edu' is not currently accepting messages from the journal's emails -please can you ensure the editorial office is provided with a correct and active email address for Dr Cochrane as soon as possible?
Additionally, the Editors have recommended that you update the acknowledgement section of your manuscript to better reflect the support of a number of additional referees (and, indeed, Editors) in the review of your paper -can we suggest that you respond to this with an updated acknowledgement, please?
You can expect to receive a proof of your article in the near future. Please contact the editorial office (openscience_proofs@royalsociety.org) and the production office (openscience@royalsociety.org) to let us know if you are likely to be away from e-mail contact --if you are going to be away, please nominate a co-author (if available) to manage the proofing process, and ensure they are copied into your email to the journal.
Due to rapid publication and an extremely tight schedule, if comments are not received, your paper may experience a delay in publication.
Please see the Royal Society Publishing guidance on how you may share your accepted author manuscript at https://royalsociety.org/journals/ethics-policies/media-embargo/. Thank-you for responding to the reviews in such a complete way. I am recommending acceptance.
One minor point, however. By my count, the paper has now received 7 reviews (4 from Interface and 3 from Open Science), and yet you only thank two reviewers in your acknowledgements. "We thank Prof. Richard Wilkinson and the anonymous reviewer, whose comments helped us improve our work." Can you please communicate with the editorial team to amend this to a more appropriate statement of the input you received.  2. Pg 6 "We then discuss application of these Bayesian frameworks. . . " This doesn't seem to follow on logically from the previous sentence. Maybe say something like "In this section we discuss..." (2), I don't think the notation f (. . .) has been defined yet. 4. Pg 6 uses the notation α for the quantile used to update , while in Appendix A it is q instead. 5. "A second departure from standard ABC practice is. . . Toni directly computes a distance between simulated and observed data, originally using Euclidean distance." Why is this a departure from standard ABC practice? This seems quite common to me! 6. Pg 13 What are the units of t? 7. Pg 14 I'm not convinced that MAE is a sensible comparison to use. This would seem to favour over-concentrated posterior approximations. Table 1, why is the number of SMC ABC iterations "NA"? 9. Pg 15 What is a "challenge tolerance" and why was this particular value used?

Pg 15 In
10. Pg 21 "σ 2 is the noise associated with the data". What is the error model -normal noise?
11. Pg 22 Why is it necessary to transform the parameters to be supported over the real line?
12. Pg 23 "find all distances between these solutions and the true data". What distance function was used?
Why this particular sequence? What tolerance was used for MCMC ABC?
14. Pg 24 Table 5's caption mentions mean absolute error but this isn't included in the table.
15. Pg 24 "Estimation of the noise parameter is standard using exact Bayesian inference (MCMC), but not with the highly popular ABC based approaches we have investigated here". In my experience noise parameter estimation is standard in applications of ABC, and papers which avoid it are unusual.
16. Pg 25 "We can see in all examples presented in this paper that the computational time consumed by MCMC ABC is shorter compared to the other methods. . . This is a significant advantage. . . " I think MCMC ABC was only used once in the paper (Tables 4 and 5), which is not enough to make a conclusion like this.
17. Pg 26 "the problem with this algorithm is that there is no existing criteria to identify an appropriate iteration at which to terminate". I think this sentence is a bit misleading.
The authors have proposed an unusual application of this algorithm to the case where no noise is added to ABC simulations. I would argue that the lack an appropriate termination criterion in this case is a problem with this unusual application, not an general problem of the algorithm. We thank the reviewers for there helpful comments. Below we systematically address the comments and highlight changes made to the resubmitted manuscript. Page references made in our response are those from the resubmitted document. Changes to the manuscript are highlighted in yellow.

Referee # 1
General comments 1. I think the paper makes an interesting and novel point about using ABC for ODE models. Also in my opinion this paper is well suited to Royal Society Open Science.
We would like to thank the reviewer for this positive evaluation.
Main issues 1. The paper gives a good criticism of the "ABC with no noise on simulations" approach to inference. Page 4 motivates this with two references: Toni et al (2009) and Gupta et al (2018), which I think is enough for publication in this journal. The other references listed use the "ABC with known simulation noise" setting, which is not investigated by this paper as far as I can tell. I suggest removing these references, or explaining why this is relevant in more detail somewhere in the paper. We agree that also non-adaptive algorithms will not accurately reflect the uncertainties in parameter values, but we used di↵erent adaptive ABC methods such as SMC ABC and MCMC ABC, to confirm that even using the most recent ABC methods fail to accurately approximate the posterior distributions.
Regarding to the stopping rule we mention it among others methods that been used in the literature to terminate algorithm and we "found that none of these methods terminate the algorithm (in ODE case) in such a way as to produce the correct shape and spread of the posterior distribution" as we stated on page 17 and 18.
3. Surely some conditions are needed for the highlighted block at the bottom of page 11 to hold? For instance this conclusion would seem not to be the case if some parameters were non-identifiable.
In response to this comment we clarify in the paper that under an ideal conditions the highlighted block will be hold. In addition, it is standard in SMC ABC that extra reductions of the tolerance ✏ 0 will leads to low acceptance rates without adding a significant improvement to the ABC posterior. Figure 2 seems very unlikely under the normal noise model stated. Firstly, the vast majority of the observations are above the true infection curve. Secondly, no observations appear to be negative.

The data in
We randomly added normal noise to the true ODE solution and because the data is represents the proportion of infected individuals negative value (less than zero) is not appropriate. Therefore, we repeat the generation of the noise randomly until we have positive data.
5. MCMC only converges to its target asympytotically, so statements along the lines of "MCMC required 12401 steps to reach convergence" are incorrect. Fixed, page 15.
6. The tuning choices used for MCMC and MCMC ABC should be summarised (e.g. choice of proposal distributions).
7. The supplied code generally looks excellent, but it did eventually crash when I ran Run file final.R.

Fixed.
Minor issues 1. Pg 5 "MCMC and ABC. . . involves sampling the posterior density". ABC only samples from an approximation to the posterior density. Fixed, page 5.

Pg 6 "
We then discuss application of these Bayesian frameworks. . . " This does not seem to follow on logically from the previous sentence. Maybe say something like ? In this section we discuss...? Fixed, page 5. (2), I do not think the notation f(. . .) has been defined yet. Fixed, page 6.

In Equation
4. Pg 6 uses the notation ↵ for the quantile used to update, while in Appendix A it is q instead. Fixed.
5. "A second departure from standard ABC practice is. . . Toni directly computes a distance between simulated and observed data, originally using Euclidean distance."? Why is this a departure from standard ABC practice? This seems quite common to me!.
Our claim here is that the common practice with ABC approach is using a discrepancy function based on the distance between vectors of summary statistics s(z ⇤ ) and s(y) not directly computes a distance between simulated and observed data as in  approach. Because the summary statistics have much lower dimension than the simulated and observed data vectors z ⇤ and y. This explanation is appear on paragraph 3 page 11.
6. Pg 13 What are the units of t?
The units of the time is (weeks), we added this on page 13.
7. Pg 14 I'm not convinced that MAE is a sensible comparison to use. This would seem to favour over-concentrated posterior approximations.
We used range of performance measures such as CPU times, the number of iterations and the median of the posterior in addition to the mean absolute value, which give a good comparisons of the methods. In response to this comment, we agree with the reviewer and we clarify in the paper that MAE may favour over-concentrated posterior approximations. Table 1, why is the number of SMC ABC iterations "NA"? Fixed, we add the number of iterations.

Pg 15 In
9. Pg 15 What is a "challenge tolerance" and why was this particular value used?
In example 1 the challenge tolerance been chosen by finding the distance between the true ODE solution and the generate observations y. We clarify this at page 15.
10. Pg 21 is the noise associated with the data?. What is the error model ? normal noise? True, we assumed normal noise and we clarify this at page 22.
11. Pg 22 Why is it necessary to transform the parameters to be supported over the real line?
The parameters were transformed to improve the acceptance rate of the proposals, we clarify this at page 23.
12. Pg 23 "find all distances between these solutions and the true data". What distance function was used?
13. Pg 23 "we applied SMC ABC for 6 populations with a sequence of tolerance" Why this particular sequence? What tolerance was used for MCMC ABC?
The sequence of tolerance have been chosen adaptively, we clarify this on page 24. Regarding the way to choose MCMC ABC tolerance was demonstrated on the last paragraph on page 24.
14. Pg 24 Table 5 caption mentions mean absolute error but this is not included in the table. Fixed.
15. Pg 24 "Estimation of the noise parameter is standard using exact Bayesian inference (MCMC), but not with the highly popular ABC based approaches we have investigated here". In my experience noise parameter estimation is standard in applications of ABC, and papers which avoid it are unusual.
In response to this comments we have revised the sentence as follows on page 25: "Estimation of the noise parameter is standard using exact Bayesian inference (MCMC), but not with the current practice with ABC based approaches when applying to a system of ODEs that we investigated here". This clarify that the neglecting of the estimation of the noise when using ABC based approach have been done in some literatures when it applied on ODEs system.

Pg 25 "
We can see in all examples presented in this paper that the computational time consumed by MCMC ABC is shorter compared to the other methods. . . This is a significant advantage. . . " I think MCMC ABC was only used once in the paper (Tables 4 and 5), which is not enough to make a conclusion like this.
We agree with the reviewer and in response to this comment we remove this sentence.
17. Pg 26 "the problem with this algorithm is that there is no existing criteria to identify an appropriate iteration at which to terminate". I think this sentence is a bit misleading. The authors have proposed an unusual application of this algorithm to the case where no noise is added to ABC simulations. I would argue that the lack an appropriate termination criterion in this case is a problem with this unusual application, not an general problem of the algorithm.
To the best of our knowledge there is no current SMC ABC approach applied to the ODE models that terminates the iteration before it is shrink to point estimate as we explained in the paper. Estimating the number of population T required in such way the approximated ABC posterior reflects the noise on the data can be investigated further in a future work.

Pg 26-27
The extra experiments involving Figure 10 should be in a results section, not the conclusion.
Fixed as suggested.
19. Pg 26 "The comparison conducted in this paper demonstrates that using exact Bayesian inference (MCMC) for ODE parameter estimation is a practical alternative". This is very well known and not a novel finding. See for example Gelman et al. (1996).
We agree with the reviewer that the study conducted by Gelman et al. (1996) have observed that using MCMC with complex model has many features, but in our paper we concluded to this finding after the comparison between the exact and approximate Bayesian inference when we dealing with a system of ODEs. One aim is to shed light on which method is perform better. In response we added the provided reference on page 27.
20. Pg 28 "Both MCMC and SMC ABC method incur similar computational cost? MCMC took 6.6 minutes while SMC ABC took 11.2 minutes -roughly double the cost. Fixed, page 28.
21. Pg 27 "the needs to solve the ODE models too many times". Too many? compared to what? Fixed, page 28.
22. Pg 27 "in the second example we found that more e↵ort is needed to construct the likelihood functions when applying MCMC? I do not understand this comment ? is not the likelihood just based on using normal noise again? This does not seem like much e↵ort.
That's true for example 1, but for example 2 the likelihood was not straightforward because the ODEs system is coupled and complex. While computation of the Gaussian terms is straightforward, solving the system of equations to obtain the means is not.
Possible typos 1. Pg 3 "An SMC ABC approach developed by" should be "An SMC ABC approach was developed by". Fixed as suggested.
2. Pg 8 What is "zn". z n is the n th simulated data as we defined it on page 7.
4. Pg 13 "The parameter of interest is" should be "The parameters of interest are". Fixed, page 13.
5. Pg 15 "SMC ABC consumed the longest run times amongst the methods", ". . . has the smallest variance compared with other methods?. These sentences should be reworded to reflect that only two methods were compared. Fixed.

Referee # 2
General comments " Modifications and shortening of the paper have improved the newly submitted version".
We would like to thank the reviewer for this positive evaluation.
Minor remarks 1. The authors have implemented the method described by Vaart et al. (2018), which seems promising, even though time consuming. The authors should not refer to these results only in the discussion section but should instead show and describe them in the test problem section.
As mentioned in response to Referee # 1, we have moved the implementation of Vaart et al. (2018) method to the results section.