Drop-Ship Kits Today – Our Current GI-MAP Turnaround Time: 7–10 Business Days

Response to Doctor's Data Inc (DDI) Funded Study

Wednesday, September 30, 2020

A qualitative assay (reporting only Positive/Negative results) cannot be used to determine the sensitivity of a quantitative assay (which reports numbers of detected organisms at levels as low as 10 cells/gram of stool). The BioFire and Meridian assays are designed to detect only high levels of organisms, intended to correlate to acute disease states. They are not intended to report low levels of organisms and cannot do so. A negative result does not rule out that the organism is present at levels below disease levels.

Diagnostic Solutions Laboratory's (DSL) Primary Concerns with the DDI Funded Study Include:

  1. GI-MAP® identified and quantified all spiked samples with a 100% accuracy, though the DDI study misrepresented the results to imply GI-MAP only identified 80% of spiked samples.
    See Details
  2. The DDI-funded study erroneously concluded that GI-MAP had a low specificity (27%) because it identified several organisms that BioFire and Meridian did not identify. Such a conclusion cannot be drawn in a comparative study that lacks a generally-accepted gold standard control method as the basis for comparing results.
    • The comparator assays (BioFire and Meridian) lacked sensitivity: If a true comparator had been utilized, the inability of BioFire and Meridian Bioscience to identify the additional organisms would reduce their apparent sensitivity from 100% to 46%.
    • The study lacks accepted scientific practices: The authors seem unaware that the study is a poorly designed and conducted comparison of methods – and not a properly controlled study.
      See Details
  3. The analysis was funded by DDI which is a direct DSL competitor. Study oversight was conducted by one of DDI's employees, raising questions of bias. There are significant unexplained study discrepancies, including the fact that a number of study samples were omitted from the report, and several different sample groups were submitted over several months – suggesting that multiple different negative controls were used, and a general lack of accepted scientific processes was followed.
    See Details

 

 


Expanded Statements Below

 

Section 1 – Expanded Statements

GI-MAP identified and quantified all spiked samples with a 100% accuracy, though the study misstated the results to imply GI-MAP only identified 80% of spiked samples.
  • The authors incorrectly and arbitrarily assigned a False Negatives (FN) to GI-MAP findings. Two spiked organisms identified by GI-MAP (Vibrio, sample #10, and Yersinia, sample #12) were arbitrarily designated as false negatives because the amounts found were not noted as "High," even though they were correctly identified and quantified.

Table 1: GI-MAP Identification of Spiked Organisms

# ATCC Spiked Organisms Found GI-MAP Identification and Quantification
1 Clostridium difficile Toxin A+, B+ ATCC 9689 C. difficile, Toxin A – High, 1.4×103 c.f.u. gm−1
C. difficile, Toxin B – High, 2.15×107 c.f.u. gm−1
2 Clostridium difficile Toxin A-, B- ATCC 700057 C. difficile, Toxin A – within normal limits
C. difficile, Toxin B – within normal limits
3 Clostridium difficile Toxin A-, B+ ATCC 43598 C. difficile, Toxin A – High, 1.9×105 c.f.u. gm−1
C. difficile, Toxin B – High, 6.32×107 c.f.u. gm−1
4 Escherichia coli O157:H7 ATCC 35150 Enterohemorrhagic E. coli – High, 5.19×105 c.f.u. gm−1
E. coli O157 – High, 3.4×105 c.f.u. gm−1
Enterotoxigenic E. coli LT/ST – High, 1.35×107 c.f.u. gm−1
5 Shigella boydii ATCC 9207 Enteroinvasive E. coli/Shigella – High, 1.14×107 c.f.u. gm−1
6 Escherichia coli O26:H11; ATCC BAA-2196 Enterohemorrhagic E. coli – High, 1.97×107 c.f.u. gm−1
Shiga-like Toxin E. coli stx1 – High, 8.84×106 c.f.u. gm−1
Shiga-like Toxin E. coli stx2 – High, 4.99×105 c.f.u. gm−1
7 Escherichia coli STX1+, STX2- O103:H11 CDC3008 Shiga-like Toxin E. coli – 1.56×101 c.f.u. gm−1
8 Shigella sonnei ATCC 29930 Enteroinvasive E. coli/Shigella – High, 4.22×106 c.f.u. gm−1
9 Salmonella bongori ATCC 43975 Salmonella – High, 3.48×108 c.f.u. gm−1
10 Vibrio cholerae ATCC 25870 Vibrio cholerae – 1.24×103 c.f.u. gm−1
11 Salmonella Enteritidis ATCC 13076 Salmonella – High, 2.89×107 c.f.u. gm−1
12 Yersinia enterocolitica O:8 ATCC 9610 Yersinia enterocolitica – 2.58×102 c.f.u. gm−1
13 Campylobacter jejuni ATCC 33560 Campylobacter – High, 2.96×106 c.f.u. gm−1
14 Campylobacter coli ATCC 51729 Campylobacter – High, 1.33×106 c.f.u. gm−1
15 Giardia intestinalis Giardia – High, 1.43×106 c.f.u. gm−1
16 Cryptosporidium parvum Cryptosporidium – High, 5.48×105 c.f.u. gm−1
    16/16 All spike samples were identified and quantified
100% No spiked sample was mis-identified or not quantified
Top

 

Section 2 – Expanded Statements

The study erroneously concluded that GI-MAP had a low specificity level of 27% because it identified several organisms that BioFire and Meridian did not identify. Such a conclusion cannot be drawn in a comparative study that lacks a generally-accepted gold standard control method as the basis for comparing results. Without a legitimate gold-standard comparison control, the GI-MAP assigned "false positive" results may in fact be organisms not identified by BioFire or Meridian, giving them a high "false negative" rate. (see Table 2 below).

 

  1. The comparator assays (BioFire and Meridian) lacked sensitivity: If a true comparator had been utilized, the inability of BioFire and Meridian Bioscience to identify the additional organisms would reduce their apparent sensitivity from 100% to 46%.
    1. Diagnostic Solutions Laboratory moved away from using qualitative assay panels like the BioFire for GI-MAP because they lacked the desired level of sensitivity. Qualitative assays, such as Biofire FilmArray and Meridian antigen testing, are only intended to correlate with disease not quantification of organisms. Unlike quantitative analysis, qualitative assays do not produce a "positive" result for organisms that exist below disease levels. BioFire's own FDA application states, "obtained from individuals with signs and or symptoms of gastrointestinal infection."
    2. GI-MAP relies exclusively on quantitative polymerase chain reaction (qPCR) technology which offers a much more accurate way to detect and quantify organisms. In comparison, a qualitative assay only reports Positive/Negative results. The BioFire and Meridian assays are designed to report 'Positive' only at high levels of organisms, and a negative result means an organism is below the cut-off, down to true negative. Quantification can detect organisms present at low levels.
    3. In real-world studies, researchers have found variation in the rates of pathogen detection and reproducibility from the BioFire FilmArray. Hitchcock, M., et al. (2018) noted that "The FilmArray GI panel has low yield in adult patients hospitalized for >72 h, similar to conventional stool microbiology tests, and it is reasonable to restrict its use in this population." 1
    4. GI-MAP quantifies H. pylori and can detect significantly lower levels in clinical samples compared to commercially available non-molecular detection assays, such as the Meridian Bioscience antigen test. GI-MAP's validation studies were able to detect target genes in all strains of H. pylori based on in silico data and confirmation using three strains of H. pylori.
  2. The study lacks accepted scientific practices: The authors seem unaware that the study is a poorly designed and conducted comparison of methods and not a properly controlled study. The study's conclusions are based on the unproven assumption that the process used to confirm spiked organisms, via BioFire and Meridian Bioscience, have adequate sensitivity and specificity to serve as valid controls for comparison to the GI-MAP. The study lacks true control sample(s) of the organisms in question and, as such, conclusions regarding potential false positive results are invalid. Given that a major assertion of the study claims that the GI-MAP generates a high rate of false positives resulting in an extremely low specificity, the author's failure to include valid controls is particularly egregious.
    1. The study itself states: "The initial stool sample/matrix was only tested by the BioFire FilmArray GI Panel and the H. pylori antigen detection test by Meridian Bioscience. Both screening assays have specific limits of detection, sensitivity, and specificity that may be different from the test assay, and therefore testing results may not match. … The lack of comparator quantitative PCR assays prevented a direct comparison with the quantitative results of the GI-MAP."
    2. In contrast, a study that followed accepted practices compared BioFire FilmArray to Luminex xTAG and noted that statements of false negative or false positive results could not be made due to the lack of a gold standard, even though the authors utilized a third assessment to compare results against. Interestingly, the study utilized qPCR as a comparator.2
    3. The DDI study lacks a description of sample data. The article states that the samples were sent to a "reference lab" that was not identified in the Methods section, raising concerns. True independent analysis (by culture on differential media, reporting of the BioFire test results, use of one of the POCT for a majority of these microbes, or even actual data based on pathologist examination for the detectable presence of parasitic/protozoan targets) are noticeably absent. Peer-review publications typically have a substantially higher level of confirmatory evidence, such as reporting all data.

Table 2: Identification of all Organisms by GI-MAP, BioFire and Meridian Bioscience

# All Organisms (*Spiked Organisms) Identified by GI-MAP Identified by BioFire & Meridian
1 Clostridium difficile Toxin A+, B+* Yes Yes
H. pylori Yes No
2 Clostridium difficile Toxin A-, B-* Yes Yes
3 Clostridium difficile Toxin A-, B+* Yes Yes
Enterohemorrhagic E. coli Yes No
H. pylori Yes No
Vibrio cholerae Yes No
4 Escherichia coli O157:H7* Yes Yes
H. pylori Yes No
5 Shigella boydii* Yes Yes
6 Escherichia coli O26:H11* Yes Yes
H. pylori Yes No
7 Escherichia coli STX1+, STX2-* Yes Yes
8 Shigella sonnei A* Yes Yes
9 Salmonella bongori* Yes Yes
10 Vibrio cholerae* Yes Yes
Enterohemorrhagic E. coli Yes No
H. pylori Yes No
11 Salmonella Enteritidis* Yes Yes
Adenovirus Yes No
H. pylori Yes No
12 Yersinia enterocolitica O:8* Yes Yes
Enterohemorrhagic E. coli Yes No
Giardia Yes No
H. pylori Yes No
13 Campylobacter jejuni* Yes Yes
14 Campylobacter coli* Yes Yes
15 Giardia intestinalis* Yes Yes
16 Cryptosporidium parvum* Yes Yes
17 Negative Control
18 Negative Control
19 Contaminated Control: Giardia Yes No
20 Contaminated Control: Enteroinvasive E. coli/Shigella Yes No
H. pylori Yes No
21 Contaminated Control: H. pylori Yes No
22 Contaminated Control: H. pylori Yes No
23 Contaminated Control: Enterotoxigenic E. coli LT/ST Yes No
    35/35 – 100% 16/35 – 46%
Top

 

Section 3 – Expanded Statements

The study analysis was funded by a DDI, a direct DSL competitor, with oversight by one of its employees, raising important questions of bias. Proportionally, government funding of research has dropped, leading to an increase in corporate research.3 While the majority of corporate research adheres to strict ethical standards, some do not. This study was funded by DDI with the stated purpose of evaluating its biggest competitor DSL's GI-MAP, using spiked and potentially contaminated stool samples, raising bias-related concerns.

 

  1. There are significant unexplained discrepancies of samples submitted for GI-MAP analysis, such as samples arriving at several different dates (over several months) and samples that appear to be from several different subjects, contradictory to what is noted in the study.
    1. There is concern that pertinent data was withheld, as normal flora results from the samples submitted to GI-MAP suggest that there were ~7 distinct samples. H. pylori agreement between the consistent flora finding support the fact that this was not a true single pooled sample, thus leading to significant variation in H. pylori results. Both the normal flora and the protein analytes (conspicuously absent from this report) indicate that samples submitted for GI-MAP testing were from at least 7 different sources (either 7 different times of collection, 7 independent donors, or 7 unique pools).
    2. Data on 12 submitted samples was excluded from the study, further raising concerns of data being cherry-picked.
    3. Samples were received from two separate entities (clinicians) on multiple dates spanning at least four months – indicating that either samples were stored under unstated conditions prior to being sent in for testing or that samples were collected at different times, which directly contradicts stated procedures. Extended storage (greater than 7 days) in Cary-Blair vials at any temperature above -20C will alter both the microbial composition (including the increase in very low-level E. coli and other low-level microbial components commonly found in the gastrointestinal tract). This alteration in protein and microbial composition is one of the primary reasons DSL has established stringent transit times for valid test results. If study samples were collected and pooled into a single sample prior to subsequent aliquoting and shipment to DSL for testing, the correct collection dates were incorrectly stated on the required Test Requisition Forms, again demonstrating valid and accurate testing by DSL was not the intention of the study. If the samples had the correct collection dates that noted and identified that they were collected four months earlier, DSL would have immediately rejected the samples as falling outside of the allowable transit time. Therefore, either the collection process was incorrectly and/or inaccurately stated in the study, or incorrect data was entered.
  2. Significant conflict of interests. While the authors state there is no conflict of interest, the study was paid for exclusively by DDI, and they fail to state that DDI is in direct competition with Diagnostic Solutions Laboratory (DSL), giving it a clear and distinct conflict of interest.
    1. IIT Research Institute, a contract research organization (CRO), was paid by DDI to run the study.
    2. Notably missing from the acknowledgment/author section is any participant from DSL, the laboratory conducting the BioFire study, or the clinician(s) on record submitting the samples.
  3. The study samples were spiked with an unknown concentration of organisms from American Type Culture Collection (ATCC). ATCC dried organisms generally only guarantee an organism's presence confirmed by polymerase chain reaction (PCR) and does not guarantee it is pure and free of additional organisms. Purified strains are significantly more costly. The study did not clearly indicate which ATCC samples were included in the study. As per the article, ATCC organisms were grown in trypticase soy agar, which is a highly nutritive media that can grow a wide variety of organisms aerobically. Common GI pathogens are often anaerobic organisms that are classified as difficult to grow (with the exception of the facultative anaerobe E. coli). The stated growth conditions indicate that unintentional E. coli contamination is highly likely.
    1. In addition, the use of densitometry to "confirm" the concentration of the target organism is highly inaccurate and can only be used as an estimation. A colony-based approach for quantification of cultured organisms would have provided both an accurate calculation of viable target organisms as well as a determination of contamination, especially if said colony-based assay was performed on selective and or differential growth media. Lastly, a clear and accurate spiking scheme was not included. The authors state that aliquots were inoculated with 106 organisms, but they do not provide the volume inoculated or how much of the spiked aliquot was used and subsequently mixed with what volume of sample to produce the final sample.
  4. Access Microbiology, who published the study, noted that it follows the Committee on Publication Ethics (COPE). Diagnostic Solutions Laboratory is contacting (COPE) concerning how such an obviously flawed and biased study passed its review process.

 

Top

REFERENCES


  1. Hitchcock MM, Gomez CA, Banaei N. Low Yield of FilmArray GI Panel in Hospitalized Patients with Diarrhea: an Opportunity for Diagnostic Stewardship Intervention. J Clin Microbiol. 2018 Feb 22;56(3):e01558-17. pubmed.ncbi.nlm.nih.gov/29237784/
  2. Zhan Z, Guo J, Xiao Y, He Z, Xia X, Huang Z, Guan H, Ling X, Li J, Diao B, Zhao H, Kan B, Zhang J. Comparison of BioFire FilmArray gastrointestinal panel versus Luminex xTAG Gastrointestinal Pathogen Panel (xTAG GPP) for diarrheal pathogen detection in China. Int J Infect Dis. 2020 Aug 13;99:414-420. pubmed.ncbi.nlm.nih.gov/32800862/
  3. Mervis, J. Data check: U.S. government share of basic research funding falls below 50%. Science. 2020 Mar 9; 370(6512). www.sciencemag.org/news/2017/03/data-check-us-government-share-basic-research-funding-falls-below-50