The role of deliberate practice in expert performance: revisiting Ericsson, Krampe & Tesch-Römer (1993)

We sought to replicate Ericsson, Krampe & Tesch-Römer's (Ericsson, Krampe & Tesch-Römer 1993 Psychol. Rev. 100, 363–406) seminal study on deliberate practice. Ericsson et al. found that differences in retrospective estimates of accumulated amounts of deliberate practice corresponded to each skill level of student violinists. They concluded, ‘individual differences in ultimate performance can largely be accounted for by differential amounts of past and current levels of practice’ (p. 392). We reproduced the methodology with notable exceptions, namely (i) employing a double-blind procedure, (ii) conducting analyses better suited to the study design, and (iii) testing previously unanswered questions about teacher-designed practice—that is, we examined the way Ericsson et al. operationalized deliberate practice (practice alone), and their theoretical but previously unmeasured definition of deliberate practice (teacher-designed practice), and compared them. We did not replicate the core finding that accumulated amounts of deliberate practice corresponded to each skill level. Overall, the size of the effect was substantial, but considerably smaller than the original study's effect size. Teacher-designed practice was perceived as less relevant to improving performance on the violin than practice alone. Further, amount of teacher-designed practice did not account for more variance in performance than amount of practice alone. Implications for the deliberate practice theory are discussed.


11-Mar-2019
Dear Dr Macnamara On behalf of the Editors, I am pleased to inform you that your Manuscript RSOS-190327 entitled "The Role of Deliberate Practice in Expert Performance: Revisiting Ericsson, Krampe, & Tesch-Römer (1993)" deemed suitable for in-principle acceptance in Royal Society Open Science subject to minor revision in accordance with the referee and editor suggestions. Please find their comments at the end of this email.
The reviewers and handling editors have recommended publication, but also suggest some minor revisions to your manuscript. Therefore, I invite you to respond to the comments and revise your manuscript.
Please you submit the revised version of your manuscript within 7 days (i.e. by the 19-Mar-2019). If you do not think you will be able to meet this date please let me know immediately.
To revise your manuscript, log into https://mc.manuscriptcentral.com/rsos and enter your Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions". Under "Actions," click on "Create a Revision." You will be unable to make your revisions on the originally submitted version of the manuscript. Instead, revise your manuscript and upload a new version through your Author Centre.
When submitting your revised manuscript, you will be able to respond to the comments made by the referees and upload a file "Response to Referees" in the "File Upload" step. You can use this to document any changes you make to the original manuscript. In order to expedite the processing of the revised manuscript, please be as specific as possible in your response to the referees.
Once again, thank you for submitting your manuscript to Royal Society Open Science and I look forward to receiving your revision. If you have any questions at all, please do not hesitate to get in touch.
Kind regards, Professor Chris Chambers Royal Society Open Science openscience@royalsociety.org on behalf of Chris Chambers (Registered Reports Editor, Royal Society Open Science) openscience@royalsociety.org Associate Editor Comments to Author (Professor Chris Chambers): Two expert reviewers have now appraised the manuscript. The reviews are positive, with both reviewers judging the Stage 1 Primary Criteria to be met, and both recommending Stage 1 in principle acceptance. Reviewer 2, however, raises two questions about the materials and statistical analyses. If it is not possible to alter these then the response to reviewers (and where appropriate, the manuscript itself) should fully justify these elements of the design. A minor revision is recommended to address these points.
Reviewers' comments to Author: Reviewer: 1 Comments to the Author(s) This looks like a well thought out and important study to conduct given the influence of the original 1993 study. I look forward to finding out the results.

Reviewer: 2
Comments to the Author(s) This study is superbly designed. Indeed, many of the shortcoming of the original study are corrected, while still enabling the authors to conduct a meaningful replication of the original piece of research. With the exception of one possibility, the statistical analyses were all meaningful to me. This is an ambitious design, and I recommend going forward with the study.
The Time 1 assessment protocol is impressive. Very systematic and thoroughly thought out; I especially liked the Swedish Music Discrimination Task and the Raven assessments. Yet, all of the proposed assessments seems highly appropriate to me. The only other thing I might think of is the Wonderlic (Gottfredson, 1997), given the familiarity of this measure to occupational psychology broadly defined. I know that Raven is translatable into IQ units (Jensen et al, 1988), and I very much like that you are using the Raven; but, if possible, the Wonderlic would be highly complementary.
The one statistical analysis that gave me pause was the discriminant function analysis noted on page 37. I have used this application in my own research on multiple occasions, and I have found it to be most informative (as it would be in the current context). But is sample size an issue here given the number of assessments relevant to this analysis? Like factor analysis and multiple regression, the rule of thumb is around 10 to 20 subjects per variable (15 is typically seen as fine). If sample size could be increased to handle this appropriately, I totally agree discriminant function analyses would be a nice feature of this study and would complement the excellent design that the authors have crafted.
Overall, the authors are to be congratulated for detailing and very well-designed study that is important.

21-Mar-2019
Dear Dr Macnamara On behalf of the Editor, I am pleased to inform you that your Stage 1 Replication RSOS-190327.R1 entitled "The Role of Deliberate Practice in Expert Performance: Revisiting Ericsson, Krampe, & Tesch-Römer (1993)" has been accepted in principle for publication in Royal Society Open Science.
You may now progress to Stage 2 and complete the study as approved.
Please note that you must now register your approved protocol on the Open Science Framework (https://osf.io/rr), using the "Submit your approved Registered Report" option and then the "Registered Report Protocol Preregistration" option. Please use the Registered Report option even though your article is being accepted as a Stage 1 Replication. Further into the registration process, in the Journal Title field enter "Royal Society Open Science (Replication article type, Results-Blind track)". Please note that a time-stamped, independent registration of the protocol is mandatory under journal policy, and manuscripts that do not conform to this requirement cannot be considered at Stage 2. The protocol should be registered unchanged from its current approved state. Please include a URL to the protocol in your Stage 2 manuscript, and because you submitted via the Results-Blind track please note in the manuscript that the preregistration was performed after data analysis (e.g. "This article received results-blind in-principle acceptance (IPA) at Royal Society Open Science. Following IPA, the accepted Stage 1 version of the manuscript, not including results and discussion, was preregistered on the OSF (URL). This preregistration was performed after data analysis.") Following completion of your study, we invite you to resubmit your paper for peer review as a Stage 2 Replication. Please note that your manuscript can still be rejected for publication at Stage 2 if the Editors consider any of the following conditions to be met: • The Introduction and methods deviated from the approved Stage 1 submission (required). • The authors' conclusions were not considered justified given the data.
We encourage you to read the complete guidelines for authors concerning Stage 2 submissions at: http://rsos.royalsocietypublishing.org/page/replication-studies#AuthorsGuidance. Please especially note the requirements for data sharing and that withdrawing your manuscript will result in publication of a Withdrawn Registration.
Once again, thank you for submitting your manuscript to Royal Society Open Science and I look forward to receiving your Stage 2 submission. If you have any questions at all, please do not hesitate to get in touch. We look forward to hearing from you shortly with the anticipated submission date for your stage two manuscript.

Recommendation? Accept with minor revision
Comments to the Author(s) Thanks for this paper, it is an important contribution to the literature. I only have a few more suggestions to strengthen the manuscript.
1. You motivate your article in the introduction by explaining how this particular paper has had a large impact both within the academic community and also outside it among the public. Perhaps this could be mentioned as an additional motivating factor for this attempted replication in the beginning of your discussion section given I think this is a key reason why you are attempting to see if the paper stands even with your additional improved method and analysis additions.
2. Though I think you are reasonably fair to Ericsson in regards to his definitional shifts or creeps, at the same time I'd encourage you to reread especially the abstract, intro, and discussion in particular to ensure it is as neutral as possible.
3. You note that the size of the core finding was much smaller. Is this smaller finding still of importance? Is there a way to communicate that clearly to the reader in the abstract and in the discussion/conclusion? 4. It seems that you are basically stating that you tried to replicate this really influential expertise paper and can't really replicate it and in fact the conclusions of the original study are not supported as stated by the original authors. It seems most plausible that the lack of replication is due to the your additional improvements in the methods. Though this is not a direct replication given you've made some improvements. Could perhaps discuss a bit about what this type of replication is? (is it a constructive replication, following Lykken's formulation?) Thanks for conducting this important research and I hope these suggestions are helpful.

08-Jul-2019
Dear Dr Macnamara On behalf of the Editor, I am pleased to inform you that your Stage 2 Replication submission RSOS-190327.R2 entitled "The Role of Deliberate Practice in Expert Performance: Revisiting Ericsson, Krampe, & Tesch-Römer (1993)" has been accepted for publication in Royal Society Open Science subject to minor revision in accordance with the referee suggestions. Please find the referees' comments at the end of this email.
The reviewers and Subject Editor have recommended publication, but also suggest some minor revisions to your manuscript. Therefore, I invite you to respond to the comments and revise your manuscript.
Please also ensure that all the below editorial sections are included where appropriate (a nonexhaustive example is included in an attachment): • Ethics statement If your study uses humans or animals please include details of the ethical approval received, including the name of the committee that granted approval. For human studies please also detail whether informed consent was obtained. For field studies on animals please include details of all permissions, licences and/or approvals granted to carry out the fieldwork.
• Data accessibility It is a condition of publication that all supporting data are made available either as supplementary information or preferably in a suitable permanent repository. The data accessibility section should state where the article's supporting data can be accessed. This section should also include details, where possible of where to access other relevant research materials such as statistical tools, protocols, software etc can be accessed. If the data has been deposited in an external repository this section should list the database, accession number and link to the DOI for all data from the article that has been made publicly available. Data sets that have been deposited in an external repository and have a DOI should also be appropriately cited in the manuscript and included in the reference list.
If you wish to submit your supporting data or code to Dryad (http://datadryad.org/), or modify your current submission to dryad, please use the following link: http://datadryad.org/submit?journalID=RSOS&manu=RSOS-190327.R2 • Competing interests Please declare any financial or non-financial competing interests, or state that you have no competing interests.
• Authors' contributions All submissions, other than those with a single author, must include an Authors' Contributions section which individually lists the specific contribution of each author. The list of Authors should meet all of the following criteria; 1) substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published.
All contributors who do not meet all of these criteria should be included in the acknowledgements.
We suggest the following format: AB carried out the molecular lab work, participated in data analysis, carried out sequence alignments, participated in the design of the study and drafted the manuscript; CD carried out the statistical analyses; EF collected field data; GH conceived of the study, designed the study, coordinated the study and helped draft the manuscript. All authors gave final approval for publication.
• Acknowledgements Please acknowledge anyone who contributed to the study but did not meet the authorship criteria.
• Funding statement Please list the source of funding for each author.
Because the schedule for publication is very tight, it is a condition of publication that you submit the revised version of your manuscript within 7 days (i.e. by the 16-Jul-2019). If you do not think you will be able to meet this date please let me know immediately.
To revise your manuscript, log into https://mc.manuscriptcentral.com/rsos and enter your Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions". Under "Actions," click on "Create a Revision." You will be unable to make your revisions on the originally submitted version of the manuscript. Instead, revise your manuscript and upload a new version through your Author Centre.
When submitting your revised manuscript, you will be able to respond to the comments made by the referees and upload a file "Response to Referees" in "Section 6 -File Upload". You can use this to document any changes you make to the original manuscript. In order to expedite the processing of the revised manuscript, please be as specific as possible in your response to the referees.
When uploading your revised files please make sure that you have: 1) A text file of the manuscript (tex, txt, rtf, docx or doc), references, tables (including captions) and figure captions. Do not upload a PDF as your "Main Document". 2) A separate electronic file of each figure (EPS or print-quality PDF preferred (either format should be produced directly from original creation package), or original software format) 3) Included a 100 word media summary of your paper when requested at submission. Please ensure you have entered correct contact details (email, institution and telephone) in your user account 4) Included the raw data to support the claims made in your paper. You can either include your data as electronic supplementary material or upload to a repository and include the relevant DOI within your manuscript 5) Included your supplementary files in a format you are happy with (no line numbers, Vancouver referencing, track changes removed etc) as these files will NOT be edited in production Once again, thank you for submitting your manuscript to Royal Society Open Science and I look forward to receiving your revision. If you have any questions at all, please do not hesitate to get in touch.

Kind regards, Professor Chris Chambers
Registered Reports Editor Royal Society Open Science openscience@royalsociety.org Editor Comments to Author (Professor Chris Chambers): The Stage 2 manuscript was returned to the two expert reviewers who assessed it at Stage 1. Both are positive about the submission while offering some suggestions for minor revision. In revising the manuscript, please be sure to make no unnecessary changes to the Introduction and Methods that were assessed and approved at Stage 1. Provided the authors respond thoroughly to reviewers' comments, full acceptance should be forthcoming without requiring further in-depth review.
Reviewers' comments to Author: Reviewer: 1 Comments to the Author(s) Thanks for this paper, it is an important contribution to the literature. I only have a few more suggestions to strengthen the manuscript.
1. You motivate your article in the introduction by explaining how this particular paper has had a large impact both within the academic community and also outside it among the public. Perhaps this could be mentioned as an additional motivating factor for this attempted replication in the beginning of your discussion section given I think this is a key reason why you are attempting to see if the paper stands even with your additional improved method and analysis additions.
2. Though I think you are reasonably fair to Ericsson in regards to his definitional shifts or creeps, at the same time I'd encourage you to reread especially the abstract, intro, and discussion in particular to ensure it is as neutral as possible.
3. You note that the size of the core finding was much smaller. Is this smaller finding still of importance? Is there a way to communicate that clearly to the reader in the abstract and in the discussion/conclusion? 4. It seems that you are basically stating that you tried to replicate this really influential expertise paper and can't really replicate it and in fact the conclusions of the original study are not supported as stated by the original authors. It seems most plausible that the lack of replication is due to the your additional improvements in the methods. Though this is not a direct replication given you've made some improvements. Could perhaps discuss a bit about what this type of replication is? (is it a constructive replication, following Lykken's formulation?) Thanks for conducting this important research and I hope these suggestions are helpful.
Reviewer: 2 Comments for the Authors: I have read this study twice now and believe that it is an excellent contribution to the literature. The authors have provided a compelling conceptual and statistical rational for their approach and, in many ways, the way in which they have operationalized each analytic move is exemplary. I was particularly impressed by the way in which they linked each analysis to embedded text (and quotes) that they pulled from the literature.
In addition to executing a superb replication, the authors have added important (additional) controls of their own to advance this area. This is very careful, impressive, and important work.
A few things to think about in crafting a revision:

Abstract
Rather than "operational definition" how about the way in which Ericsson et al. (1993) operationalize deliberate practice (practice alone), and their theoretical but previously unmeasured definition of deliberate practice (teacher-designed practice), …" This way, your description won't be tied to earlier ideas about operational definitions (viz., a concept IS what its measure IS).
Page 3 (first paragraph). I couldn't help thinking of Epstein's (2011) excellent book, The sports gene. Because in addition to height and body size, he has several other physical and physiological parameters of human individuality that clearly come into play.
Page 3 (second paragraph). Was it Gladwell's 10,000-hour rule? I thought this was suggested by someone else earlier.
Pages 13-14 constitute an excellent discussion of "sample size," very compelling. It is a pleasure to accept your Stage 2 Replication entitled "The Role of Deliberate Practice in Expert Performance: Revisiting Ericsson, Krampe, & Tesch-Römer (1993)" in its current form for publication in Royal Society Open Science.
Thank you for your fine contribution. On behalf of the Editors of Royal Society Open Science, we look forward to your continued contributions to the Journal. Two expert reviewers have now appraised the manuscript. The reviews are positive, with both reviewers judging the Stage 1 Primary Criteria to be met, and both recommending Stage 1 in principle acceptance. Reviewer 2, however, raises two questions about the materials and statistical analyses. If it is not possible to alter these then the response to reviewers (and where appropriate, the manuscript itself) should fully justify these elements of the design. A minor revision is recommended to address these points.
Reply: Thank you for the opportunity to submit minor revisions.
Reviewers' comments to Author: Reviewer: 1 Comments to the Author(s) This looks like a well thought out and important study to conduct given the influence of the original 1993 study. I look forward to finding out the results.
Reply: Thank you for your comments. We look forward to presenting the results.

Reviewer: 2
Comments to the Author(s) This study is superbly designed. Indeed, many of the shortcoming of the original study are corrected, while still enabling the authors to conduct a meaningful replication of the original piece of research. With the exception of one possibility, the statistical analyses were all meaningful to me. This is an ambitious design, and I recommend going forward with the study.
Reply: Thank you for these comments.
The Time 1 assessment protocol is impressive. Very systematic and thoroughly thought out; I especially liked the Swedish Music Discrimination Task and the Raven assessments. Yet, all of the proposed assessments seems highly appropriate to me. The only other thing I might think of is the Wonderlic (Gottfredson, 1997), given the familiarity of this measure to occupational psychology broadly defined. I know that Raven is translatable into IQ units (Jensen et al, 1988), and I very much like that you are using the Raven; but, if possible, the Wonderlic would be highly complementary.
Reply: Thank you for the suggestion. Ideally, with enough resources, other tests could be included. There are a number of reasons why we don't think the Wonderlic is ideal. First, while it is a popular measure, evidence of its validity is limited. Notably, the Wonderlic does not predict academic performance (Chamorro-Premuzic & Furnham, 2008;Furnham, Chamorro-Premuzic, & McDougall, 2002;McKelvie, 1994) and is inconsistent as a predictor of job performance (Barrick, Mount, & Strauss, 1993;Frei & McDaniel, 1998;Hogan & Hogan, 1995;Lyons, Hoffman, & Michel, 2009;Rode, Arthaud-Day, Mooney, Near, & Baldwin, 2008). Further, recent work by Hicks, Harrison and Engle (2015) found that, when controlling for working memory capacity, the Wonderlic has no direct association with fluid intelligence. Hicks et al. (2015) also found that the Wonderlic varies in whether it predicts working memory capacity or not depending on participants' fluid intelligence. We have more succinctly added this rationale to the manuscript. The Fluid Intelligence measure sub-section now reads: Fluid intelligence. We included two measures of fluid intelligence: Raven's Advanced Progressive Matrices [S5] and Letter Sets [S6] . The mean proportion correct on the two measures serves as the composite fluid intelligence score. A number of other intelligence measures could have been included, such as the Wonderlic test [S7] . We did not include the Wonderlic because evidence for its validity is limited both in terms of being a consistent predictor of job performance and in terms of correlations with fluid intelligence (see, e.g., [S8] ).
The one statistical analysis that gave me pause was the discriminant function analysis noted on page 37. I have used this application in my own research on multiple occasions, and I have found it to be most informative (as it would be in the current context). But is sample size an issue here given the number of assessments relevant to this analysis? Like factor analysis and multiple regression, the rule of thumb is around 10 to 20 subjects per variable (15 is typically seen as fine). If sample size could be increased to handle this appropriately, I totally agree discriminant function analyses would be a nice feature of this study and would complement the excellent design that the authors have crafted.