Response to Doctor's Data Inc (DDI) Funded Study

GI-MAP identified and quantified all spiked samples with a 100% accuracy, though the study misstated the results to imply GI-MAP only identified 80% of spiked samples.

• The authors incorrectly and arbitrarily assigned a False Negatives (FN) to GI-MAP findings. Two spiked organisms identified by GI-MAP (Vibrio, sample #10, and Yersinia, sample #12) were arbitrarily designated as false negatives because the amounts found were not noted as "High," even though they were correctly identified and quantified.

---

Table 1: GI-MAP Identification of Spiked Organisms

Top

The study erroneously concluded that GI-MAP had a low specificity level of 27% because it identified several organisms that BioFire and Meridian did not identify. Such a conclusion cannot be drawn in a comparative study that lacks a generally-accepted gold standard control method as the basis for comparing results. Without a legitimate gold-standard comparison control, the GI-MAP assigned "false positive" results may in fact be organisms not identified by BioFire or Meridian, giving them a high "false negative" rate. (see Table 2 below).

 

1. The comparator assays (BioFire and Meridian) lacked sensitivity: If a true comparator had been utilized, the inability of BioFire and Meridian Bioscience to identify the additional organisms would reduce their apparent sensitivity from 100% to 46%.
   1. Diagnostic Solutions Laboratory moved away from using qualitative assay panels like the BioFire for GI-MAP because they lacked the desired level of sensitivity. Qualitative assays, such as Biofire FilmArray and Meridian antigen testing, are only intended to correlate with disease not quantification of organisms. Unlike quantitative analysis, qualitative assays do not produce a "positive" result for organisms that exist below disease levels. BioFire's own FDA application states, "obtained from individuals with signs and or symptoms of gastrointestinal infection."
   2. GI-MAP relies exclusively on quantitative polymerase chain reaction (qPCR) technology which offers a much more accurate way to detect and quantify organisms. In comparison, a qualitative assay only reports Positive/Negative results. The BioFire and Meridian assays are designed to report 'Positive' only at high levels of organisms, and a negative result means an organism is below the cut-off, down to true negative. Quantification can detect organisms present at low levels.
   3. In real-world studies, researchers have found variation in the rates of pathogen detection and reproducibility from the BioFire FilmArray. Hitchcock, M., et al. (2018) noted that "The FilmArray GI panel has low yield in adult patients hospitalized for >72 h, similar to conventional stool microbiology tests, and it is reasonable to restrict its use in this population." ¹
   4. GI-MAP quantifies H. pylori and can detect significantly lower levels in clinical samples compared to commercially available non-molecular detection assays, such as the Meridian Bioscience antigen test. GI-MAP's validation studies were able to detect target genes in all strains of H. pylori based on in silico data and confirmation using three strains of H. pylori.
2. The study lacks accepted scientific practices: The authors seem unaware that the study is a poorly designed and conducted comparison of methods and not a properly controlled study. The study's conclusions are based on the unproven assumption that the process used to confirm spiked organisms, via BioFire and Meridian Bioscience, have adequate sensitivity and specificity to serve as valid controls for comparison to the GI-MAP. The study lacks true control sample(s) of the organisms in question and, as such, conclusions regarding potential false positive results are invalid. Given that a major assertion of the study claims that the GI-MAP generates a high rate of false positives resulting in an extremely low specificity, the author's failure to include valid controls is particularly egregious.
   1. The study itself states: "The initial stool sample/matrix was only tested by the BioFire FilmArray GI Panel and the H. pylori antigen detection test by Meridian Bioscience. Both screening assays have specific limits of detection, sensitivity, and specificity that may be different from the test assay, and therefore testing results may not match. … The lack of comparator quantitative PCR assays prevented a direct comparison with the quantitative results of the GI-MAP."
   2. In contrast, a study that followed accepted practices compared BioFire FilmArray to Luminex xTAG and noted that statements of false negative or false positive results could not be made due to the lack of a gold standard, even though the authors utilized a third assessment to compare results against. Interestingly, the study utilized qPCR as a comparator.²
   3. The DDI study lacks a description of sample data. The article states that the samples were sent to a "reference lab" that was not identified in the Methods section, raising concerns. True independent analysis (by culture on differential media, reporting of the BioFire test results, use of one of the POCT for a majority of these microbes, or even actual data based on pathologist examination for the detectable presence of parasitic/protozoan targets) are noticeably absent. Peer-review publications typically have a substantially higher level of confirmatory evidence, such as reporting all data.

---

Table 2: Identification of all Organisms by GI-MAP, BioFire and Meridian Bioscience

Top

The study analysis was funded by a DDI, a direct DSL competitor, with oversight by one of its employees, raising important questions of bias. Proportionally, government funding of research has dropped, leading to an increase in corporate research.³ While the majority of corporate research adheres to strict ethical standards, some do not. This study was funded by DDI with the stated purpose of evaluating its biggest competitor DSL's GI-MAP, using spiked and potentially contaminated stool samples, raising bias-related concerns.

 

1. There are significant unexplained discrepancies of samples submitted for GI-MAP analysis, such as samples arriving at several different dates (over several months) and samples that appear to be from several different subjects, contradictory to what is noted in the study.
   1. There is concern that pertinent data was withheld, as normal flora results from the samples submitted to GI-MAP suggest that there were ~7 distinct samples. H. pylori agreement between the consistent flora finding support the fact that this was not a true single pooled sample, thus leading to significant variation in H. pylori results. Both the normal flora and the protein analytes (conspicuously absent from this report) indicate that samples submitted for GI-MAP testing were from at least 7 different sources (either 7 different times of collection, 7 independent donors, or 7 unique pools).
   2. Data on 12 submitted samples was excluded from the study, further raising concerns of data being cherry-picked.
   3. Samples were received from two separate entities (clinicians) on multiple dates spanning at least four months – indicating that either samples were stored under unstated conditions prior to being sent in for testing or that samples were collected at different times, which directly contradicts stated procedures. Extended storage (greater than 7 days) in Cary-Blair vials at any temperature above -20C will alter both the microbial composition (including the increase in very low-level E. coli and other low-level microbial components commonly found in the gastrointestinal tract). This alteration in protein and microbial composition is one of the primary reasons DSL has established stringent transit times for valid test results. If study samples were collected and pooled into a single sample prior to subsequent aliquoting and shipment to DSL for testing, the correct collection dates were incorrectly stated on the required Test Requisition Forms, again demonstrating valid and accurate testing by DSL was not the intention of the study. If the samples had the correct collection dates that noted and identified that they were collected four months earlier, DSL would have immediately rejected the samples as falling outside of the allowable transit time. Therefore, either the collection process was incorrectly and/or inaccurately stated in the study, or incorrect data was entered.
2. Significant conflict of interests. While the authors state there is no conflict of interest, the study was paid for exclusively by DDI, and they fail to state that DDI is in direct competition with Diagnostic Solutions Laboratory (DSL), giving it a clear and distinct conflict of interest.
   1. IIT Research Institute, a contract research organization (CRO), was paid by DDI to run the study.
   2. Notably missing from the acknowledgment/author section is any participant from DSL, the laboratory conducting the BioFire study, or the clinician(s) on record submitting the samples.
3. The study samples were spiked with an unknown concentration of organisms from American Type Culture Collection (ATCC). ATCC dried organisms generally only guarantee an organism's presence confirmed by polymerase chain reaction (PCR) and does not guarantee it is pure and free of additional organisms. Purified strains are significantly more costly. The study did not clearly indicate which ATCC samples were included in the study. As per the article, ATCC organisms were grown in trypticase soy agar, which is a highly nutritive media that can grow a wide variety of organisms aerobically. Common GI pathogens are often anaerobic organisms that are classified as difficult to grow (with the exception of the facultative anaerobe E. coli). The stated growth conditions indicate that unintentional E. coli contamination is highly likely.
   1. In addition, the use of densitometry to "confirm" the concentration of the target organism is highly inaccurate and can only be used as an estimation. A colony-based approach for quantification of cultured organisms would have provided both an accurate calculation of viable target organisms as well as a determination of contamination, especially if said colony-based assay was performed on selective and or differential growth media. Lastly, a clear and accurate spiking scheme was not included. The authors state that aliquots were inoculated with 106 organisms, but they do not provide the volume inoculated or how much of the spiked aliquot was used and subsequently mixed with what volume of sample to produce the final sample.
4. Access Microbiology, who published the study, noted that it follows the Committee on Publication Ethics (COPE). Diagnostic Solutions Laboratory is contacting (COPE) concerning how such an obviously flawed and biased study passed its review process.

 

Top