Bradley Ferguson and Dr. C. Shane Reese, Statistics

Detection of biological and chemical threats is an important consideration in the modern national defense policy. Much of the testing and evaluation of threat detection technologies is performed without appropriate uncertainty quantification. Under the guidance of Dr. Shane Reese, my ORCA project dealt with developing a more effective and cost-efficient way of testing threat detection technologies. I utilized a Bayesian Gaussian Process model that allows for a more flexible and robust model t. I also developed an adaptive experimental design scheme that provides more information than a typical experimental design by learning the concentration levels that we are more interested in and performing more tests around those locations.

The data that I am working with is success/failure (binary) data at a given set of threat concentration levels. The testing personnel choose a set of concentration levels to test the instruments at and then they perform multiple tests at each concentration and record whether the detection instrument alarmed or not. As a statistician, my goal is use a probability model to capture the underlying relationship between threat concentration and probability of detection.

The first part of my research was learning about flexible probability models that can be used to model binary data. The most common probability model for this data is a logistic model that relates the probability of a successful detection with some parametric function of threat concentration. The logistic model assumes a monotone (increasing or decreasing) relationship between concentration and probability of successful detection. This model is thus very effective if the probabilistic relationship is strictly increasing or decreasing but does a very poor job of modeling a non-monotone relationship. I wanted to use a probability model that is more flexible that allows for the possibility of a non-monotone probability curve so I spent a lot of time researching Bayesian Gaussian process models. These models are powerful because (1) they are Bayesian thus they allow for the incorporation of prior knowledge into the model, and (2) they are non-parametric in nature which allows for non-monotone probability curves. Once I understood the theoretical aspects of the Bayesian Gaussian process models, I coded up an iterative algorithm in R to fit the model to the data. I found immediately that the model t the data essentially the same as the logistic model with the underlying probability curve was monotone and did a much better job at fitting the data when the underlying curve was non-monotone.

The other part of my research was implementing a more efficient experimental design scheme that what is traditionally used. Instead of performing the same number of tests at each concentration level, I wanted to have an experimental design that adapted as more tests were performed and suggested at which concentrations to do the next set of tests. I got many of my ideas from current papers on adaptive clinical trial designs but I had to adjust for that fact that my data was success/failure data and that I would have different criteria for determining where to perform the tests. The adaptive experimental design that I developed chooses to perform tests (with higher probability) at concentrations where there is greater uncertainty in the probability curve and at concentrations that are closer to the C50 level. The C50 level is the concentration at which there is a 50% of detection and is a value that the testing personnel are very interested in. This C50 concentration is not known ahead of time but is hopefully learned as the experiment progresses. Thus, this adaptive design will ultimately perform more tests around the C50 value and where we do not have enough information in the curve.

In order to compare the adaptive design vs. the non-adaptive design, I performed a simulation study. I started with an initial 12 total tests – 2 at each of 6 concentration levels. I then estimated the C50 value and quantifed the uncertainty in the probability curve at each concentration and assigned 6 new tests to be done based on this information. I repeated this until 90 total tests were done. I compared my results to the non-adaptive design with 90 total tests and found that the adaptive design scheme estimated the C50 value much better the variance associated with the C50 value was much smaller.

I have performed more simulation studies and the results are always the same – the adaptive design always does a better job at estimating the C50 value. The next step in my research is to compare my results to the logistic model in a number of different settings. It would be useful to know which model is best in both the adaptive and non-adaptive experimental designs.