Blood Glucose Monitoring System Surveillance Program

Congressional diabetes Caucus & HHS Correspondence About DTS BGMS Surveillance Program

August 27, 2018 Response to Department of Health and Human Services from Congressional Diabetes Caucus

January 24, 2018 Response to Congressional Diabetes Caucus from Department of Health and Human Services

December 6, 2017 Letter to Department of Health and Human Services from Chairs of Congressional Diabetes Caucus

Blood Glucose Monitor Surveillance Article

Investigation of the Accuracy of 18 Marketed Blood Glucose Monitors. David C. Klonoff, Joan Lee Parkes, Boris P. Kovatchev, David Kerr, Wendy C. Bevier, Ronald L. Brazg, Mark Christiansen, Timothy S. Bailey, James H. Nichols and Michael A. Kohn. Diabetes Care 2018 Aug; 41(8): 1681-1688.

Summary of Results

David C. Klonoff, M.D., FACP, FRCP (Edin), Fellow AIMBE1, Joan L. Parkes, PhD2, Boris P. Kovatchev, PhD3
David Kerr, MBChB, DM, FRCPE4, Wendy C. Bevier, PhD4, Ronald L. Brazg, MD5, Mark P. Christiansen, MD6
Timothy S. Bailey, MD, FACE, CPI7, Jim H. Nichols, PhD, DABCC, FACB8, Michael A. Kohn, MD, MPP9
Author Affiliations

The Diabetes Technology Society Blood Glucose Monitor System (BGMS) Surveillance Program was established because of evidence that cleared BGMSs do not always achieve levels of accuracy matching either: 1) their performance that resulted in becoming cleared by FDA; or 2) international standards of accuracy.  Poor performance of these devices can lead to adverse clinical and economic consequences.  This surveillance program assessed the accuracy of 18 blood glucose monitoring systems (BGMSs) marketed in the USA across a wide range of blood glucose levels in the hands of trained professionals.  These 18 BGMSs represented approximately 90% of the commercially available systems that were used from 2013 to 2015 by diabetes patients and obtained from consumer outlets.

Subjects 18 years and older with type 1 diabetes, type 2 diabetes, pre-diabetes, or no diabetes, participated at 3 clinical sites. Each clinical site conducted 3 consecutive sub-studies.  In each of the sub-studies, subjects' finger-stick blood was assessed on 6 different marketed BGMSs, and a tube of capillary blood was collected, and the plasma prepared from it was frozen and sent to a separate laboratory site to assay on the glucose comparative instrument (YSI).  At each clinical site, 3 separate studies were conducted with a different set of 6 BGMSs in each.  This assured that all 18 BGMS were tested at all 3 clinical sites. The results of first study performed at each of three sites were combined into Study 1, the results of the second study at each site were combined into Study 2, and the results of the third study at each site were combined into Study 3. 

Glycolized blood samples were tested to ascertain the accuracy of BGMSs in the very low blood glucose range. At each of the three sites during each of the three studies, an additional tube of capillary blood was collected from approximately 20 subjects.  This blood was glycolyzed to achieve very low glucose values that could not be safely obtained naturally.  The glycolyzed blood was tested on the oxygen insensitive BGMSs whose labels specified glucose dehydrogenase chemistry, which is oxygen insensitive.  Plasma from each tube was then prepared, frozen, and sent to the research laboratory site for assay on a comparative reference glucose instrument (YSI 2300 STAT PLUS Glucose & L-Lactate Analyzer).  At the research laboratory site, the frozen tubes of plasma were thawed, thoroughly mixed, and assayed on YSI instruments whose accuracy was validated and traceable to a higher order using NIST 965b Standards. The results of these measurements of glycolized specimens were analyzed separately from the results of natural specimens that were used in the pass-fail analysis process.  

This study was triple blinded.  None of the people involved in conducting this study (i.e. neither investigators, laboratory staff, statistician, nor sponsor) had all the information to break the BGMS code until all results were calculated and posted. 

Analyses were performed to determine whether the 18 BGMSs, assayed with natural samples (that is blood samples taken directly from a subject's finger), met pre-determined analytical accuracy criteria agreed upon by the DTS-BGMS Surveillance Committee. The protocol specified that to be compliant a blood glucose value must be within 15% of a reference plasma value for a blood glucose >100 mg/dl and within 15 mg/dl of a reference plasma value for a blood glucose <100 mg/dl. These protocol criteria were similar to: 1) (for all blood glucose levels) the latest ISO standard (ISO15197-2013, "In vitro diagnostic test systems — Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus"), which was in effect when the protocol was developed; and 2) (for a blood glucose >100 mg/dl) the latest FDA guidance (FDA 2016, "Self-Monitoring of Blood Glucose Test Systems for Over-the-Counter Use"), which was released after the protocol was developed. The number of compliant readings needed to pass depended on the number of trials. For a study of 100 trials, at least 91 readings were required to be within 15% or 15 mg/dL of the reference value. A BGMS that passed all 3 studies received the DTS Seal of Approval.

The frequency of outliers for each BGMS was assessed with a modified Bland-Altman analysis along with calculation of bias, Coefficient of Variation, 95% limits of agreement, and the absolute value of the greatest 95% limit of agreement. 

Clinical accuracy was determined by Surveillance Error Grid analysis. The absolute values of clinical risk levels from 0-4 were separated into five bins.  Each data point was assigned to a bin corresponding to the absolute value of its clinical risk.

These three studies were performed on 1035 participating subjects total at three clinical sites: Rainier Clinical Research Center, Inc. (n=353); Diablo Clinical Research (n=335); and AMCR Institute, Inc. (n=347).  Plasma reference laboratory testing was performed at The William Sansum Diabetes Center. Six BGMSs were assayed in each sub-study, assuring that each clinical site assayed all 18 BGMSs.  The results for each BGMS (i.e., approximately one third from each clinical site) were combined in the analysis of overall compliance to produce the overall combined results of 3 repeated studies for each BGMS. A Seal of Approval was awarded to any BGMS that passed all 3 of the 3 studies. The determination of whether to award a Seal of Approval was not based on: overall analytical performance of the 3 studies; variability; clinical accuracy; or accuracy for testing glycolized hypoglycemic plasma specimens. 

The topline results are summarized in the tables below. We tested 18 BGMSs in 3 studies each. The results were that only 6 of the 18 BGMSs passed all 3 times and received the Seal of Approval.

Demographics

Subject Demographic Characteristic N = 1035
Gender
Female 560
Male 471
Not specified 4
Type of Diabetes
Type 1 370
Type 2 470
Pre-diabetes 4
Does not have diabetes 187
Not specified 4
Age (Years)
Range 18-87
Mean 51.5
Race*
White/Caucasian 851
Black/African American 91
Hispanic 156
Native American/Aboriginal 9
Asian 47
Other and not specified 37

*Some subjects identified themselves in more than one category

Main Results

Study 1 Study 2 Study 3 Overall
Brand Blood Glucose Monitor Test Strip N Percentage Compliant Pass/ Fail N Percentage Compliant Pass/ Fail N Percentage Compliant Pass/ Fail Passes out of 3 Studies Seal of Approval? Valid Trials Compliant
N
Compliant
%
Roche ACCU-CHEK AVIVA Plus ACCU-CHEK AVIVA Plus 97 97% PASS 101 100% PASS 113 98% PASS 3 YES 311 306 98%
Roche Accu-Chek Smart View ACCU-CHEK SmartView 108 98% PASS 106 96% PASS 106 92% PASS 3 YES 320 305 95%
Diabetic Supply of Suncoast Advocate Redi-Code + Advocate 102 88% FAIL 114 71% FAIL 103 68% FAIL 0 NO 319 241 76%
Bayer Contour Classic Contour 108 95% PASS 106 85% FAIL 106 86% FAIL 1 NO 320 284 89%
Bayer Contour Next Contour Next 98 99% PASS 101 100% PASS 113 100% PASS 3 YES 312 311 100%
Agamatrix CVS Advanced CVS Advanced 101 96% PASS 114 96% PASS 103 98% PASS 3 YES 318 307 97%
Omnis Health Embrace Embrace No Code 102 87% FAIL 114 93% PASS 103 84% FAIL 1 NO 319 282 88%
Abbott FreeStyle Lite FreeStyle Lite 98 92% PASS 101 96% PASS 113 98% PASS 3 YES 312 298 96%
Philosys, Inc. Gmate Smart Gmate 108 61% FAIL 106 79% FAIL 106 72% FAIL 0 NO 320 226 71%
LifeScan OneTouch Ultra2 OneTouch Ultra 97 92% PASS 101 84% FAIL 113 94% PASS 2 NO 311 280 90%
LifeScan OneTouch Verio OneTouch Verio 108 87% FAIL 106 98% PASS 105 91% PASS 2 NO 319 294 92%
Prodigy Prodigy Auto Code Prodigy No Coding 98 86% FAIL 101 92% PASS 113 93% PASS 2 NO 312 282 90%
BioSense Medical SolusV2 SOLUS 108 56% FAIL 106 84% FAIL 106 89% FAIL 0 NO 320 244 76%
HDI/Nipro True Result TrueResult 101 94% PASS 114 83% FAIL 103 86% FAIL 1 NO 318 279 88%
HDI/Nipro True Track TrueTrack 102 83% FAIL 103 80% FAIL FAIL* 0 NO 205 167 81%
Arkray Walmart ReliOn Prime ReliOn Prime 98 85% FAIL 101 95% PASS 113 96% PASS 2 NO 312 288 92%
Arkray Walmart ReliOn Confirm (Micro) ReliOn Confirm/micro 100 96% PASS 114 96% PASS 103 99% PASS 3 YES 317 307 97%
Abbott Walmart ReliOn Ultima ReliOn Ultima 107 96% PASS 106 97% PASS 106 75% FAIL 2 NO 319 285 89%

*No Study 3 data for HDI/Nipro TrueTrack because of test strip recall.

Within 15% of reference value if ≥ 100 mg/dL or 15 mg/dL of reference value if ≤ 100 mg/dL.

In Studies 1, 2, and 3, each of the 18 blood glucose monitoring systems (BGMSs) was tested at 3 different sites. Over the 3 studies each BGMS was tested once by each site. The number of compliant readings needed to pass depends on the number of trials. For 100 trials, at least 91 readings must be within 15% or 15 mg/dL of the reference value. Full methodology of the study will be published.

Overall Results

Trials Within Protocol Limits
Brand Blood Glucose Monitor Test Strip Valid Trials N %
Roche ACCU-CHEK AVIVA Plus ACCU-CHEK AVIVA Plus 311 306 98%
Roche Accu-Chek Smart View ACCU-CHEK SmartView 320 305 95%
Diabetic Supply of Suncoast Advocate Redi-Code + Advocate 319 241 76%
Bayer Contour Classic Contour 320 284 89%
Bayer Contour Next Contour Next 312 311 100%
Agamatrix CVS Advanced CVS Advanced 318 307 97%
Omnis Health Embrace Embrace No Code 319 282 88%
Abbott FreeStyle Lite FreeStyle Lite 312 298 96%
Philosys, Inc. Gmate Smart Gmate 320 226 71%
LifeScan OneTouch Ultra2 OneTouch Ultra 311 280 90%
LifeScan OneTouch Verio OneTouch Verio 319 294 92%
Prodigy Prodigy Auto Code Prodigy No Coding 312 282 90%
BioSense Medical SolusV2 SOLUS 320 244 76%
HDI/Nipro True Result TrueResult 318 279 88%
HDI/Nipro True Track TrueTrack 205 167 81%
Arkray Walmart ReliOn Prime ReliOn Prime 312 288 92%
Arkray Walmart ReliOn Confirm (Micro) ReliOn Confirm/micro 317 307 97%
Abbott Walmart ReliOn Ultima ReliOn Ultima 319 285 89%
Total Data Pairs 5584

Summary of Modified Bland-Altman Comparison

95% Limits of agreement
Brand Blood Glucose Monitor Test Strip Valid Trials Bias Coefficient of Variation* Lower Limit Upper Limit Larger Absolute Limit Boundary B-A Plot
Roche ACCU-CHEK AVIVA Plus ACCU-CHEK AVIVA Plus 311 -3.4% 6.3% -15% - 9% 15% Bland-Altman Plot
Roche Accu-Chek Smart View ACCU-CHEK SmartView 320 -5.3% 6.5% -17% - 8% 17% Bland-Altman Plot
Diabetic Supply of Suncoast Advocate Redi-Code + Advocate 319 -9.1% 10.5% -26% - 12% 26% Bland-Altman Plot
Bayer Contour Classic Contour 320 -6.5% 9.7% -23% - 13% 23% Bland-Altman Plot
Bayer Contour Next Contour Next 312 -1.2% 5.3% -11% - 10% 11% Bland-Altman Plot
Agamatrix CVS Advanced CVS Advanced 318 -0.3% 7.0% -13% - 14% 14% Bland-Altman Plot
Omnis Health Embrace Embrace No Code 319 0.9% 10.0% -17% - 23% 23% Bland-Altman Plot
Abbott FreeStyle Lite FreeStyle Lite 312 -6.0% 7.4% -19% - 9% 19% Bland-Altman Plot
Philosys, Inc. Gmate Smart Gmate 320 5.7% 15.5% -22% - 43% 43% Bland-Altman Plot
LifeScan OneTouch Ultra2 OneTouch Ultra 311 -3.0% 9.3% -19% - 17% 19% Bland-Altman Plot
LifeScan OneTouch Verio OneTouch Verio 319 5.9% 6.8% -7% - 21% 21% Bland-Altman Plot
Prodigy Prodigy Auto Code Prodigy No Coding 312 1.2% 10.3% -17% - 24% 24% Bland-Altman Plot
BioSense Medical SolusV2 SOLUS 320 -10.1% 8.1% -23% - 5% 23% Bland-Altman Plot
HDI/Nipro True Result TrueResult 318 -8.9% 8.2% -22% - 7% 22% Bland-Altman Plot
HDI/Nipro True Track TrueTrack 205 -6.9% 10.6% -24% - 15% 24% Bland-Altman Plot
Arkray Walmart ReliOn Prime ReliOn Prime 312 0.4% 9.5% -17% - 21% 21% Bland-Altman Plot
Arkray Walmart ReliOn Confirm (Micro) ReliOn Confirm/micro 317 2.4% 6.8% -10% - 17% 17% Bland-Altman Plot
Abbott Walmart ReliOn Ultima ReliOn Ultima 319 3.0% 10.2% -16% - 26% 26% Bland-Altman Plot

*Standard deviation of the difference in the log-transformed measurements, which is essentially the same as the standard deviation of the % difference = (BGMS Reading - Reference Value)/(Reference Value)

95% Limits of Agreement define the range around the reference value containing 95% of the BGMS readings.

Limit boundary with the larger absolute value

Modified Bland-Altman analysis compares the difference (BGMS Reading - Reference Value) to the reference value rather than comparing the difference to the average of the BGMS reading and the reference value. The bias is the average difference as a percent of the reference value. A bias of -5.0% means the BGMS meter reading is, on average, 5% lower than the reference value. The 95% Limits of Agreement define the range around the reference value containing 95% of the BGMS readings.

number and percent of values within specified error limits

Distance from the Reference Value*
Brand Blood Glucose Monitor Test Strip Valid Trials Within +/- 5% Within +/- 10% Within +/- 15% Within +/- 20% >20%
Bayer Contour Next Contour Next 312 212 68% 302 97% 311 100% 311 100% 1 0.3%
Roche ACCU-CHEK AVIVA Plus ACCU-CHEK AVIVA Plus 311 161 52% 272 87% 306 98% 311 100% 0 0.0%
Arkray Walmart ReliOn Confirm (Micro) ReliOn Confirm/micro 317 167 53% 276 87% 307 97% 314 99% 3 0.9%
Agamatrix CVS Advanced CVS Advanced 318 172 54% 272 86% 307 97% 317 100% 1 0.3%
Abbott FreeStyle Lite FreeStyle Lite 312 95 30% 238 76% 298 96% 306 98% 6 1.9%
Roche Accu-Chek Smart View ACCU-CHEK SmartView 320 133 42% 251 78% 305 95% 317 99% 3 0.9%
Arkray Walmart ReliOn Prime ReliOn Prime 312 121 39% 224 72% 288 92% 305 98% 7 2.2%
LifeScan OneTouch Verio OneTouch Verio 319 139 44% 239 75% 294 92% 315 99% 4 1.3%
Prodigy Prodigy Auto Code Prodigy No Coding 312 135 43% 229 73% 282 90% 304 97% 8 2.6%
LifeScan OneTouch Ultra2 OneTouch Ultra 311 127 41% 230 74% 280 90% 302 97% 9 2.9%
Abbott Walmart ReliOn Ultima ReliOn Ultima 319 140 44% 241 76% 285 89% 302 95% 17 5.3%
Bayer Contour Classic Contour 320 109 34% 215 67% 284 89% 313 98% 7 2.2%
Omnis Health Embrace Embrace No Code 319 116 36% 230 72% 282 88% 308 97% 11 3.4%
HDI/Nipro True Result TrueResult 318 81 25% 188 59% 279 88% 311 98% 7 2.2%
HDI/Nipro True Track TrueTrack 205 57 28% 112 55% 167 81% 186 91% 19 9.3%
BioSense Medical SolusV2 SOLUS 320 59 18% 148 46% 244 76% 297 93% 23 7.2%
Diabetic Supply of Suncoast Advocate Redi-Code + Advocate 319 66 21% 148 46% 241 76% 288 90% 31 9.7%
Philosys, Inc. Gmate Smart Gmate 320 82 26% 159 50% 226 71% 267 83% 53 16.6%

*When reference value is <100 mg/dL, then distance is absolute mg/dL rather than %.

The percentage of data points within +/- 15% distance from the Reference Value is the same as the percentage of data points for Trials Within Protocol Limits in the Overall Results Table.

Distance from Comparator

Proportion of blood glucose monitor readings within a given distance from the reference value.

Summary of Surveillance Error Grid Comparison

Risk
Brand Blood Glucose Monitor Test Strip Total
Data Pairs
Percentage with No Risk None
0–0.5
Slight, Lower
>0.5–1.0
Slight, Higher
>1.0–1.5
Moderate, Lower
>1.5–2.0
Moderate, Higher
>2.0–2.5
Severe, Lower
>2.5-3.0
Severe, Higher
>3.0-3.5
Extreme,
>3.5-4.0
Roche ACCU-CHEK AVIVA Plus ACCU-CHEK AVIVA Plus 311 99.7% 310 1
Roche Accu-Chek Smart View ACCU-CHEK SmartView 320 97.8% 313 7
Diabetic Supply of Suncoast Advocate Redi-Code + Advocate 319 87.5% 279 38 2
Bayer Contour Classic Contour 320 93.1% 298 20 2
Bayer Contour Next Contour Next 312 99.4% 310 2
Agamatrix CVS Advanced CVS Advanced 318 98.7% 314 4
Omnis Health Embrace Embrace No Code 319 96.6% 308 8 3
Abbott FreeStyle Lite FreeStyle Lite 312 97.1% 303 8 1
Philosys, Inc. Gmate Smart Gmate 320 89.7% 287 25 4 2 2
LifeScan OneTouch Ultra2 OneTouch Ultra 311 96.8% 301 9 1  
LifeScan OneTouch Verio OneTouch Verio 319 95.6% 305 13 1
Prodigy Prodigy Auto Code Prodigy No Coding 312 95.8% 299 11 1 1
BioSense Medical SolusV2 SOLUS 320 89.4% 286 34
HDI/Nipro True Result TrueResult 318 90.9% 289 26 2 1
HDI/Nipro True Track TrueTrack 205 89.8% 184 19 2
Arkray Walmart ReliOn Prime ReliOn Prime 312 96.2% 300 12
Arkray Walmart ReliOn Confirm (Micro) ReliOn Confirm/micro 317 97.5% 309 8
Abbott Walmart ReliOn Ultima ReliOn Ultima 319 95.0% 303 14 1 1

Glycolized Sample Results

Trials Within Protocol Limits
Brand Blood Glucose Monitor Test Strip Valid Trials N Percent
Roche ACCU-CHEK AVIVA Plus ACCU-CHEK AVIVA Plus 55 55 100%
Roche Accu-Chek Smart View ACCU-CHEK SmartView 58 58 100%
Diabetic Supply of Suncoast Advocate Redi-Code + Advocate NOT TESTED
Bayer Contour Classic Contour 57 56 98%
Bayer Contour Next Contour Next 55 55 100%
Agamatrix CVS Advanced CVS Advanced NOT TESTED
Omnis Health Embrace Embrace No Code NOT TESTED
Abbott FreeStyle Lite FreeStyle Lite 53 53 100%
Philosys, Inc. Gmate Smart Gmate NOT TESTED
LifeScan OneTouch Ultra2 OneTouch Ultra NOT TESTED
LifeScan OneTouch Verio OneTouch Verio 58 58 100%
Prodigy Prodigy Auto Code Prodigy No Coding NOT TESTED
BioSense Medical SolusV2 SOLUS NOT TESTED
HDI/Nipro True Result TrueResult 53 52 98%
HDI/Nipro True Track TrueTrack NOT TESTED
Arkray Walmart ReliOn Prime ReliOn Prime NOT TESTED
Arkray Walmart ReliOn Confirm (Micro) ReliOn Confirm/micro NOT TESTED
Abbott Walmart ReliOn Ultima ReliOn Ultima 47 46 98%
Total Data Pairs 436

BGMSs that use glucose oxidase chemistry are particularly sensitive to oxygen levels. Since glycolysis changes the oxygen concentration of a blood sample, glycolized blood samples were only
assayed on BGMSs that specify glucose dehydrogenase chemistry, which is oxygen insensitive. The glucose levels ranged from 11 mg/dL to 80 mg/dL with mean (SD) 49.1 (15.6) mg/dL.

Study Summary

N Percent Meter
Readings
Readings Per
Reference Value
Non-Glycolized Specimens 1032
Excluded Reference Values* 81 7.8%
Included 951 92.2% 5584 5.9
No Hematocrit Value 19 2.0%
Hematocrit Out of Range (<33%, >52%) 57 6.0%
Hematocrit in Range 875 92.0% 5119 5.9
Seal/No Seal did not change when the analysis was restricted to in-range hematocrit.
Glycolized Specimens 185
Excluded Reference Values 13 7.0%
Included 172 93.0% 436 2.5

*Excluded 63 for > 4% variability between YSI reference runs, 6 for hemolysis, and 12 for other sample problems.

Excluded 7 glycolized specimens for > 4% variability between YSI reference runs, 4 for hemolysis, and 2 for reference values > 80 mg/dL.