Module: 2 independent tests, 2 populations, no gold standard (Program: TAGS)
The TAGS software was developed by R Pouillot and G. Gerbier, from the AFSSA (Agence Française de Sécurité Sanitaire des Aliments), Maisons-Alfort, France.
Pouillot R, Gerbier G, Gardner IA. "TAGS", a program for the evaluation of test accuracy in the absence of a gold standard. Prev Vet Med. 2002 Feb 14;53(1-2):67-81.
The TAGS software can be used to estimate the sensitivity of two or more diagnostic tests in the absence of a gold standard, provided two or more populations with differing prevalences can be cross-classified based on diagnostic test results. The algorithm in the TAGS software follows the frequentist paradigm and utilizes Newton-Raphson and EM algorithms to generate maximum likelihood estimates.
The TAGS software can accomodate not only data of the "2 independent tests, 2 populations"-type, but also higher order combinations of numbers of tests and numbers of populations. Recognizing that in some instances, true prevalence may be known for some populations, TAGS is capable of utilizing "reference population data" (where one or more populations is of known disease status). Parameter estimation using TAGS becomes possible once the number of degrees of freedom given by the data is greater than the number of parameters to be estimated. A goodness-of-fit test and residual correlations, both of which are provided by TAGS output, provide a means of evaluating model adequacy.
The algorithm includes 2 strong assumptions: (i) diagnostic tests are assumed to be conditionally independent, and (ii) test diagnostic values are considered constant when applied to different populations.
TAGS can be implemented in three distinct ways. A user may (i) submit data and its structure over the internet using an HTML inferface to a Rweb platform in Montana State University (Rweb Server), (ii) run TAGS on a PC using R, or (iii) run TAGS on a PC using S-PLUS.
R Version of TAGS through HTML
A version of TAGS has been developed for the R statistical programming environment through an HTML page. Data that is submitted will be loaded using an HTML page implemented in Javascript. A form will then be submitted to an Rweb platform (Copyright Jeff Banfield) located in Montana, US. Within minutes, a new web browser window will appear presenting results of the analysis. Nothing other than a web browser is required. Note that it has not been possible to develop bootstrap confidence interval estimation in HTML version, because the time needed for its calculation was too long and the internet connection mostly failed.
TAGS HTML
TAGS HTML hosted by AFSSA
PC Version of TAGS using R (Windows or Linux)
Another version of TAGS has been developed in R that can be run directly on a PC. The software program "R" can be obtained from the The R Project for Statistical Computing web site. Once you have installed R, following the instructions provided on the R website, it is a simple matter to install TAGS:
• Download "tags.R.zip" into a working directory on your computer and unzip it.
• Run R using "Rgui.exe".
• Load TAGS (File, Source R code, then select "tags.R").
• Type TAGS() in the command windows.
• Follow the instructions
Download "tags.R" from here (6 KB)
PC Version of TAGS using S-PLUS (Windows)
A third Windows version of TAGS has been developed for the S-PLUS 2000 statistical computing environment. To run this version of TAGS, you will need to have access to S-PLUS, available from Insightful (c) Corporation.
Next, you will need to install Prof. Brian Ripley's 'Hessian library' available from: http://www.stats.ox.ac.uk/pub/SWin/. To install the library, extract the file hessian.zip into the directory "\SP2000\library". There is need to open the library; it is automatically done in the function TAGS. It is now a simple matter to install TAGS:
• Download "tags.ssc.zip" to your computer and unzip it.
• Start S-PLUS and source the code, either using the function "source".
• Type source("path_name/tags.ssc") in the command window in S-PLUS.
• Type TAGS() in the command window.
Download "tags.ssc" from here (6 KB)
Dubey JP et al. Am J Vet Res. 1995 Aug;56(8):1030-6 compared 5 serologic tests for the diagnosis of toxoplasmosis in 1000 naturally-exposed sows using bioassay methods as the gold standard (definitive test). Bioassays were done in mice (all sows) and cats (183 sows) using cardiac muscle from sampled sows. Samples in the study were collected in two batches: nos. 1- 463 and 464- 1000.
If Toxoplasma gondii were isolated from either mice or cats, the sow was considered infected. A sow was considered non-infected if the bioassay results were negative. To demonstrate use of TAGS, we will use results of the bioassay test and 2 of the 5 serologic tests: the modified agglutination test (MAT) and the enzyme-linked immunosorbent assay (ELISA). These 2 serologic tests were the most accurate of the tests evaluated and are commonly used in screening of pigs for toxoplasmosis. The MAT was considered positive if the titer was >= 20 and the ELISA was positive if the OD value was > 0.36. The sensitivity (Se) and specificity (Sp) of the MAT using bioassay as the gold standard were calculated to be 82.9% and 90.2%, respectively. This represents the traditional approach that assumes that the combined bioassay (cat + mouse) has Se = Sp = 1.
Calculations:
The TAGS method requires a minimum of 2 populations with 2 conditionally independent tests. Use of the batch data (batch 1 = 463; batch 2 = 537) provides a logical way to create 2 populations of similar size in which the sensitivity and specificity of the tests should be equivalent. Cross-classified test results for the batches are in the following table:
Batch |
MAT+/ Bioassay + |
MAT+/ Bioassay - |
MAT-/ Bioassay + |
MAT-/ Bioassay - |
| 1 |
37 |
55 |
7 |
364 |
| 2 |
104 |
26 |
22 |
385 |
What is the sensitivity and specificity of the MAT using “TAGS” when the MAT and bioassay are the tests under consideration?
| Estimates for the MAT using TAGS are exactly the same as using the traditional approach i.e. Se = 82.9% and Sp = 90.2%. Also, TAGS estimated that the bioassay was perfectly sensitive and specific – exactly the same as the traditional approach. Hence, this provides evidence in support of bioassay as a true gold standard. |
Now let ’s ignore the bioassay results and the use the cross-classified data from MAT and ELISA by batch as follows (1 pig had a missing ELISA value):
Batch |
MAT+/ ELISA + |
MAT+/ ELISA - |
MAT-/ ELISA + |
MAT-/ ELISA - |
1 |
67 |
25 |
41 |
329 |
2 |
97 |
33 |
36 |
371 |
What is the sensitivity and specificity of the MAT now using “TAGS” when the MAT and the ELISA are the tests under comparison?
| MAT is estimated to have a Se = 100% and a Sp = 97.3% which clearly are overestimated compared with the correct values obtained when bioassay is used at the comparison. |
What is the most likely reason for the change in MAT estimates?
| Results of the two tests (MAT and ELISA) are positively correlated (dependent), conditional on infection status (Gardner et al. Conditional dependence between tests affects the diagnosis and surveillance of animal diseases. Prev Vet Med. 2000 May 30;45(1-2):107-22) . This positive dependence between the tests results in overestimation of MAT accuracy with TAGS. The main conclusion is that is inappropriate to use the TAGS software to compare 2 serologic tests without a gold-standard because the tests are likely to be dependent. Other methods (and software) that account for this dependence (2 conditionally dependent tests, 2 populations) are necessary to obtain unbiased estimates. |