Testing the reproducibility and robustness of the cancer biology literature by robot

1.1. The reproducibility crisis

Ever since the seventeenth-century scientific revolution a fundamental pillar of science has been the requirement for reproducible results [1]. However, despite reproducibility being fundamental to science, the reproducibility of relatively few biomedical results is currently tested; and when reproducibility is tested, difficulty is often experienced in observing reproducibility [2–9]. This situation is termed the ‘reproducibility crisis’: ‘the ability to reproduce experiments is at the heart of science, yet failure to do so is a routine part of research’ [10]; ‘More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments' [11].

There are a number of reasons for difficulty in reproducing published results: the original result may have been very specific and only true under specific circumstances or the original results may not have been described in sufficient detail to enable reproducibility, stochasticity in the results, etc. Scientific fraud is another possible reason, but this is probably relatively rare [12].

The most direct solution to the reproducibility crisis would be for more scientists to attempt to reproduce other scientists' results. However, there are strong sociological and career disincentives against this: it is hard to get funding for such work, it is hard to publish such studies, authors can react badly to having their results doubted, etc. [13]. Attempts have also been made to identify what factors are important in reproducibility [14,15].

Recognition of the reproducibility crisis has led to multiple initiatives; for example, the Meta-Research Innovation Center at Stanford University (https://metrics.stanford.edu/), the National Research Council of the Netherlands (NOW) Replication Studies pilot programme [16] and the Reproducibility Project Cancer Biology (RPCB) [7–9]. However, these initiatives are limited in extent, and their significance is still to be determined.

Given the high cost and difficulty involved in confirming experimental results, and the current funding model, it is unlikely that human scientists will ever experimentally confirm more than a small fraction of published results. We therefore argue that the only feasible way to increase the proportion of reproduced results is to automate the process. To achieve such automation, it will be necessary to integrate text mining (to extract results from the literature) and artificial intelligence-based laboratory automation (to experimentally test the reproducibility of the results).

1.2. Forms of experimental confirmation

Here, we recognize distinctions between results that are ‘repeatable’, ‘reproducible’ and ‘robust’. The International Vocabulary of Metrology [17,18] defines ‘repeatability’ as ‘precision in measurements under conditions that include the same measurement procedure, same operators, same measuring system, same operating conditions and same location, and replicate measurements on the same or similar objects over a short period of time’ [19]. ‘Precision’ is defined as ‘closeness of agreement between measured quantities obtained by replicate measurements on the same or similar objects under conditions of repeatability or reproducibility’ [19]. We believe that most published biomedical results are repeatable: laboratories can generally replicate their own published results. Here, we operationally define a statement about cancer from the literature to be ‘repeatable’ if in one set of our semi-automated experiments (same protocol/cell line) we found statistically significant evidence for a result.

‘Reproducibility’ is defined as ‘precision in measurements under conditions that may involve different locations, operators, measuring systems and replicate measurements on the same or similar objects. The different measuring systems may use different measurement procedures’. It is a lack of reproducibility in published biomedical results that is causing the crisis. Here, we operationally define a statement to be ‘reproducible’ if, in our semi-automated experiments, we find the same result as automatically extracted from the literature using our standard experimental approach, and using the same cell line as was originally used.

The term ‘robust’ does not seem to be as formally defined, but results are generally described as ‘robust’ when they are more generally replicable than the above definition of ‘reproducible’ [20,21]. For robust results, the basic biological systems may be different, as well as the experimental apparatus and protocol; however, the conclusions are in agreement. We operationally define a statement about cancer to be ‘robust’ if in our semi-automated experiments we find the same result as automatically extracted from the literature using our standard experimental approach, but using a different cell line from the one that was originally used.

These definitions are consistent with existing formal definitions [17–19]. However, they differ from those proposed in the US National Academies of Sciences/Engineering/Medicine report [6]. This report proposes that ‘reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis. This definition is synonymous with "computational reproducibility"’. ‘Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data’. We do not follow these definitions as their emphasis is data analysis/computational. The US National Academies of Sciences/Engineering/Medicine definition of replicability is close to what we define as robust reproducibility.

Our operational definitions of ‘repeatable’, ‘reproducible’ and ‘robust’ enable the practical automation of testing for reproducibility, as different textual statements (propositions) can be tested using the same standard experimental protocol and conditions, making the laboratory automation easier to implement. It is currently impossible to automate experiments that closely replicate the whole spectrum of original published experiments because: many published experimental protocols are incomplete, i.e. they do not contain sufficient information to enable even human scientists to reliably repeat them [22,23]—the use of natural language to describe protocols, with all their inherent ambiguities exacerbates this problem [22,23]; even if a published protocol is completely specified for a human scientist, it is currently not possible with existing text-mining technology to extract sufficient information for a robot to implement the protocol in a laboratory; and it is not feasible with existing laboratory automation to more fully test the robust reproducibility of statements by executing a wide variety of orthogonal experimental procedures.

2.1. Text mining

We focused on textual statements (propositions) taken from the breast cancer literature that were predicted to describe experimental results regarding a change of gene expression in response to a drug (small molecule) treatment. We chose to test such statements because of their medical importance, and because it was expected to be possible to test them using laboratory automation.

A corpus of 12 260 full papers on breast cancer was constructed as part of our work for the Big Mechanism program [24]. To form the corpus, full papers were retrieved from the PubMed Central Open Access repository using ‘breast cancer’ and its synonyms as keywords, combined with names of breast cancer cell lines, e.g. ‘T-47 D’, ‘MCF-7’ (and their variants) [25,26]. These papers were then processed to automatically extract information in a form of ‘index cards’: about 35 925 statements predicted to describe experimental results regarding the change of expression of genes in response to drug treatment. The index cards are ‘json’ files. The content of each card holds information about a statement extracted from the literature. Five main pieces of information are provided: the meta-data, evidence, interaction, submitter and the identifier for the paper the index card is based on.

We used tools from the UK National Centre for Text Mining [25–27]. To extract events, we employed named entity recognition (NER) methods integrated into a unified processing pipeline, which enables the development and execution of reconfigurable, modular NER and event extraction workflows. For event extraction, we applied the machine-learning-based EventMine [27]. EventMine finds trigger words indicating events (e.g. inhibits), which are assigned event types (e.g. negative regulation). This process is described in [25,26].

2.2. Heuristic text filtering

Resource constraints meant that it was not possible to experimentally test all the extracted statements. We therefore chose to select statements of greater biomedical significance. Several heuristics were used to select these (see Material and methods). We first selected events with both qualities ‘entity : simple_chemical’ and ‘event : gene_expression’. This eliminated all events in which gene expression was not affected by a small molecule. The results were further filtered against a list of all genes in two systems biology models of breast cancer: one involving RAS signalling and the other ESR1 signalling. The motivation for this was to examine gene expression in genes known to be important in breast cancer. Finally, statements were filtered against lists of compounds known to be either commercially easily available or unsuitable. This resulted in a set of events of format ‘compound affects gene expression’, known to be in our models, which were possible to test using available compounds. Compounds were manually checked to identify those known to be used or under investigation as cancer therapeutics and those known to be common dietary supplements. The output of filtering was 74 events regarding genes of interest and involving cancer therapeutics or dietary supplements. The full list is presented in table 1.

2.3. Repeatable changes in gene expressions

Using the artificial intelligence (AI)-based laboratory automation system ‘Eve’ [28], we experimentally tested the 74 statements obtained from heuristic text filtering. We used two breast cancer cell lines, MCF7 and MDA-MB-231 (ATCC, USA). MCF7 is the most studied human breast cancer cell line, with over 25 000 scientific publications using it [29]. It originated as an invasive breast ductal carcinoma, is ‘luminal’ type and oestrogen receptors (ERs) are present. The breast cancer cell line MDA-MB-231 is ‘basal’ type and triple negative, i.e. missing three markers: ER, progesterone receptor (PR) and HER2/Neu oncogene [30].

Eve was originally designed as a ‘robot scientist’, an AI-directed laboratory automation system that automatically: originates hypotheses to explain observations, devises experiments to test these hypotheses, physically runs the experiments using laboratory robotics, interprets the results to change the probability of hypotheses and then repeats the cycle [28,31]. Eve has multiple functionalities and was originally designed for automatic early-stage drug development. In this work, we did not run Eve in a closed-loop automation manner. This was because the hypotheses to test were taken directly from the literature, and there was no need to modify them based on experimental results.

Using Eve we applied two different but closely related protocols. Each protocol was run by a different team (team 1 and team 2) using multiple replicates. The same laboratory was used by both teams, but the teams worked months apart. The protocols were based on real-time polymerase chain reaction (rtPCR) to measure the expression of a targeted gene. The protocols were designed for moderate throughput semi-automated experimentation. For full details, see Material and methods.

The results for each statement were first evaluated for repeatability, i.e. could we repeatedly obtain this same result using Eve? Specifically, we defined a statement to be repeatable if with Eve one team in one cell line found statistically significant evidence for replication of a result. To decide on significance, we applied a classical sign test: the number of replicates with increased expression versus the number of replicates with decreased expression. This test is straightforward and robust to assumptions about the underlying distribution. We found that 43 statements had significant evidence for repeatability (at least one of the teams on one of the cell lines; p < 0.05). These are shown in table 2.

2.4. Manual testing of the text mining

To evaluate the automated text mining that found the 43 repeatable statements, we manually read the original papers from which the statements were extracted. We assessed whether the compound was reported to cause inhibition or an increase in expression of the gene, and whether this statement was about MCF7 or MDA-MB-231 cells (table 2). We found that the text-mining software had generally done a good job in identifying statements that describe experimental results regarding the change of gene expression in response to drug (small molecule) treatment. Only four statements were false positives—where the human readers failed to identify the proposition recognized by the text mining (table 2). The text-mining software also performed well in identifying the correct direction of change in the propositions. In only one case, for the drug fulvestrant and the gene ERBB3, was the sign extracted wrongly.

The text-mining software did not attempt to determine cell culture type referred to in the statements beyond looking for the keywords ‘MCF7’ and ‘MDA-MB-231’ in papers. Automatically determining cell culture type is difficult as this experimental detail is often far from the textual statement about the effect of a drug. Manual reading found that 19 statements referred to experiments using MCF7. In four cases, the statements referred to experiments using MDA-MB-231.

2.5. Experimental testing of reproducibility and robustness

We operationally defined a statement about cancer to be ‘reproducible’ if, using Eve, we found statistically significant evidence for a change of gene expression in the same cell line, and in the same direction as in the original paper. As with replicability, we used a sign test to decide significance. We found statistically significant evidence for experimental reproducibility of six statements (table 3).

We investigated two forms of robust reproducibility: minor and major. In minor robustness, the proposition in the paper was found using MCF7, yet was confirmed by Eve in MDA-MB-231. There were four cases of minor robustness (table 4). In major robustness, the original proposition was about neither MCF7 nor MDA-MB-231, but we found conformational evidence in either MCF7 or MDA-MB-231 cells. There were 12 cases of major robustness (table 5).

2.6. Novel knowledge about changes in gene expressions

In two cases, we found statistically significant results where the original paper stated that the result was not statistically significant: drug 4OHT inhibits gene AKT1 expression in cell line MCF7 and drug curcumin inhibits gene PDK1 expression. The first case is one of reproducibility in MCF7, the second of robust reproducibility. Eve has therefore semi-automatically provided the first statistically significant evidence for these medically important statements.

It is often stated that because machines do not make mistakes they cannot therefore make serendipitous discoveries. This argument is incorrect as machines do make mistakes. In two cases, the text-mining software incorrectly identified statements that it believed to involve the drug silibinin inhibiting genes: drug silibinin inhibits gene PTEN expression and drug silibinin inhibits gene uPA expression. Although these statements were not found by human readers in the original papers, statistically significant experimental evidence was found for these statements in the cell line MCF7, i.e. they were both repeatable.

2.7. Limitations

Our approach is limited in a number of ways:

4.1. Materials

The cell lines used were MCF7 (Sigma, 86012803) and MDA-MB-231 (ATCC, HTB-26). Compounds were individually ordered (Selleck, Tocris and Sigma). CellsDirect Resuspension and Lysis Buffers (ThermoFisher, 11739010). CCK-8 cell counting kit (Sigma, 96992). NEB Luna Universal Probe One-Step RT-qPCR Kit (NEB, E3006 L). Thermo Fisher single-tube Taqman gene expression assays: AKT1, Hs00178289_m1; ATF4, Hs00909569_g1; BIRC5, Hs00977611_g1; BRCA1, Hs01556193_m1; BRCA2, Hs00609073_m1; CASP3, Hs00234387_m1; CCND1, Hs00765553_m1; CTNNB1, Hs00355049_m1; EGFR, Hs01076092_m1; ERBB2, Hs01001580_m1; ERBB3, Hs00176538_m1; ERBB3, Hs00951444_m1; ERBB3, Hs00951455_m1; ESR1, Hs01046812_m1; ESR1, Hs01046816_m1; HDAC1, Hs00606262_g1; HIF1A, Hs00153153_m1; HSP90, Hs00743767_sH; IL-8, Hs00174103_m1; MAPT, Hs00902194_m1; MELK, Hs01106440_m1; MMP-2, Hs01548727_m1; MMP-9, Hs00234579_m1; MTOR, Hs00234508_m1; NF-KB1, Hs00765730_m1; p21, Hs00355782_m1; p27, Hs01597588_m1; p300, Hs00914223_m1; p53, Hs01034249_m1; PDK1, Hs01561850_m1; PGR, Hs00172183_m1; PTEN, Hs02621230_s1; RASSF1, Hs00176538_m1; STAT3, Hs00374280_m1; STK11, Hs00975988_m1; TNF, Hs01113624_g1; TXNIP, Hs00197750_m1; uPA, Hs01547054_m1; VEGFA, Hs00900055_m1. All code used is available on request.

4.2. Team 1 assay methods

4.3. Team 2 assay methods

4.4. Event filtering

The output of the text-mining process was 35 925 ‘events’. Every event is of the form: a small chemical affecting a gene/protein. Several heuristics were used to filter the events for biological significance.

There are three broad stages to the pipeline. The first stage refined the text mining to more certain statements (table 6). The second stage focused on statements relevant to research interests of the group, on change of gene expression (or protein), and where the compounds where commercially available (table 7). The final manual stage integrated the heuristics, focused on breast cancer and chose cheaper compounds (table 6).