Weighting formulas for the least-squares analysis of binding phenomena data

Authors: TELLINGHUISEN, JOEL - VANDERBILT UNIVERSITY; Bolster, Carl

Submitted to: Journal of Physical Chemistry
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 2/20/2009
Publication Date: 4/4/2009
Citation: Tellinghuisen, J., Bolster, C.H. 2009. Weighting formulas for the least-squares analysis of binding phenomena data. Journal of Physical Chemistry. 113:6151-6157.

Interpretive Summary: The Langmuir equation is of the form of a rectangular hyperbola, y = a b x /(1 + b x ), and is widely used for fitting phosphorus sorption data to obtain model soil sorption constants. The most common approach for fitting this model to data is through the use of least-squares analysis. However, it is commonly the case in most sorption studies that the "independent variable" is actually a directly measured quantity, and the dependent variable is simply a computed function of the independent variable. These circumstances violate one of the fundamental tenets of most least-squares methods — that the independent variable be error-free. We treat this problem by deriving weighting formulas for the least-squares analysis of such data for the rectangular hyperbola and all of its common linearized versions — the double reciprocal, y-reciprocal, and x-reciprocal forms. We verify the correctness of these expressions by computing the nonlinear least squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. Our findings will improve the statistical treatment of phosphorus sorption data.

Technical Abstract: The rectangular hyperbola, y = a b x /(1 + b x ), is widely used as a fit model in the analysis of data obtained in studies of complexation, sorption, fluorescence quenching, and enzyme kinetics. Frequently the "independent variable" x is actually a directly measured quantity, and y may be a simply computed function of x, like y = x0 – x. These circumstances violate one of the fundamental tenets of most least squares methods — that the independent variable be error-free — and they lead to fully correlated error in x and y. Using an effective variance approach, we treat this problem to derive weighting formulas for the least-squares analysis of such data by the given equation and by all of its common linearized versions — the double reciprocal, y-reciprocal, and x-reciprocal forms. We verify the correctness of these expressions by computing the nonlinear least squares parameter standard errors for exactly fitting data, and we confirm their utility through Monte Carlo simulations. The latter reveal a problem with the inversion methods when the inverted data are moderately uncertain (~30%), leading to the recommendation that the reciprocal methods not be used for such data. For benchmark tests, results are presented for specific data sets having error in x alone and in both x and x0. The actual estimates of a and b and their standard errors vary somewhat with the choice of fit model, with one important exception: The Deming/Lybanon algorithm treats multiple uncertain variables equivalently and returns a single set of parameters and standard errors independent of the manner in which the fit model is expressed.