Skip to content

NIST certified data

Marcin Wojdyr edited this page Apr 27, 2011 · 9 revisions

Statistical Reference Datasets (StRD) from NIST

National Institute of Standards and Technology (U.S.) provides datasets for assessment of software accuracy for a variety of statistical methods. 27 datasets are provided for nonlinear regression testing: http://www.itl.nist.gov/div898/strd/nls/nls_main.shtml. This page describes how the program was tested and presents the results.

Fityk gives more accurate results than two other programs that we also tested. This is a bit surprising, because, like most of the curve fitting programs, it uses (by default) Levenberg-Marquardt method and double precision for floating point numbers. In all the cases sum of squared residuals (Residual Sum of Squares in StRD) is exactly the same as certified values, so it is not listed below.

Fityk can read the data directly from Misra1a.dat, automatically skipping description, but since the file has y in the first column and x in the second, we need to swap x and y. Also, we ensure that the standard deviation of all y's is 1:

@0 < Misra1a.dat
Y=x, X=y, S=1

After swapping the axes, you may click the Zoom All button in the GUI (or type plot [] in the CLI). Since this dataset has only a few points, it is a good idea to increase the size of the points (in GUI) to make them more visible.

The certified values in the NIST files are presented in the scientific format with 11 decimal places. It is possible to set the same format in Fityk:

set numeric_format='%.10E'

Now let's prepare the model with the initial values from Start 1:

define Misra1a(b1, b2) = b1*(1-exp(-b2*x))
F = Misra1a(~500, ~0.0001)

To fit the data with a very high precision we change the stopping criteria:

=-> set lm_stop_rel_change=1e-20
=-> fit
Fitting 2 (of 2) parameters to 14 points ...
In L-M method: lambda=1e+15 > 1e+15, stopped.
levenberg_marquardt: 100 iterations, 101 evaluations, 0.01 s. of CPU time.
WSSR: 1.2455138894E-01 (-99.9988%)
=-> i errors
Standard errors:
$_1 = 2.3894212918E+02 +- 2.7070075241E+00
$_2 = 5.5015643181E-04 +- 7.2668688436E-06

If Fityk was compiled with the option to use the long double type for calculations (USE_LONG_DOUBLE set to 1 in fityk.h) we get exactly the same numbers as the certified values. With the double precision used (which is the default) the parameters and standard deviations differ at 10th or 11th decimal place. All the results shown below are obtained with Fityk 1.0.0 compiled with (only) double precision.

Parameter Standard Deviation
b1 certified 2.3894212918E+02 2.7070075241E+00
b1 fityk/double 2.3894212923E+02 2.7070075253E+00
b2 certified 5.5015643181E-04 7.2668688436E-06
b2 fityk/double 5.5015643168E-04 7.2668688435E-06
define Chwirut2(b1, b2, b3) = exp(-b1*x)/(b2+b3*x)
F = Chwirut2(~0.1, ~0.01, ~0.02)

Again, we get exactly the certified results using the long double type:

=-> i errors
Standard errors:
$_1 = 1.6657666537E-01 +- 3.8303286810E-02
$_2 = 5.1653291286E-03 +- 6.6621605126E-04
$_3 = 1.2150007096E-02 +- 1.5304234767E-03

And the only difference when using double is in b2 on 11th place.

Parameter Standard Deviation
b1 certified 1.6657666537E-01 3.8303286810E-02
b1 fityk/double 1.6657666537E-01 3.8303286810E-02
b2 certified 5.1653291286E-03 6.6621605126E-04
b2 fityk/double 5.1653291285E-03 6.6621605127E-04
b3 certified 1.2150007096E-02 1.5304234767E-03
b3 fityk/double 1.2150007096E-02 1.5304234767E-03
define Chwirut1(b1, b2, b3) = exp(-b1*x)/(b2+b3*x)
F = Chwirut1(~0.1, ~0.01, ~0.02)
Parameter Standard Deviation
b1 certified 1.9027818370E-01 2.1938557035E-02
b1 fityk/double 1.9027818358E-01 2.1938557029E-02
b2 certified 6.1314004477E-03 3.4500025051E-04
b2 fityk/double 6.1314004468E-03 3.4500025053E-04
b3 certified 1.0530908399E-02 7.9281847748E-04
b3 fityk/double 1.0530908402E-02 7.9281847747E-04
define Lanczos3(b1, b2, b3, b4, b5, b6) = b1*exp(-b2*x) + b3*exp(-b4*x) + b5*exp(-b6*x)
F = Lanczos3(~1.2, ~0.3, ~5.6, ~5.5, ~6.5, ~7.6)
Parameter Standard Deviation
b1 certified 8.6816414977E-02 1.7197908859E-02
b1 fityk/double 8.6816415134E-02 1.7197908832E-02
b2 certified 9.5498101505E-01 9.7041624475E-02
b2 fityk/double 9.5498101593E-01 9.7041624152E-02
b3 certified 8.4400777463E-01 4.1488663282E-02
b3 fityk/double 8.4400777500E-01 4.1488663264E-02
b4 certified 2.9515951832E+00 1.0766312506E-01
b4 fityk/double 2.9515951842E+00 1.0766312488E-01
b5 certified 1.5825685901E+00 5.8371576281E-02
b5 fityk/double 1.5825685896E+00 5.8371576235E-02
b6 certified 4.9863565084E+00 3.4436403035E-02
b6 fityk/double 4.9863565087E+00 3.4436403007E-02
define Gauss1(b1, b2, b3, b4, b5, b6, b7, b8) = b1*exp(-b2*x) + b3*exp(-(x-b4)^2 / b5^2) + b6*exp(-(x-b7)^2 / b8^2)
F = Gauss1(~97.0, ~0.009, ~100.0, ~65.0, ~20.0, ~70.0, ~178.0, ~16.5)

This is the first dataset that gives all 11 digits in all results the same as the NIST certified values.

Parameter Standard Deviation
b1 certified 9.8778210871E+01 5.7527312730E-01
b1 fityk/double 9.8778210871E+01 5.7527312730E-01
b2 certified 1.0497276517E-02 1.1406289017E-04
b2 fityk/double 1.0497276517E-02 1.1406289017E-04
b3 certified 1.0048990633E+02 5.8831775752E-01
b3 fityk/double 1.0048990633E+02 5.8831775752E-01
b4 certified 6.7481111276E+01 1.0460593412E-01
b4 fityk/double 6.7481111276E+01 1.0460593412E-01
b5 certified 2.3129773360E+01 1.7439951146E-01
b5 fityk/double 2.3129773360E+01 1.7439951146E-01
b6 certified 7.1994503004E+01 6.2622793913E-01
b6 fityk/double 7.1994503004E+01 6.2622793913E-01
b7 certified 1.7899805021E+02 1.2436988217E-01
b7 fityk/double 1.7899805021E+02 1.2436988217E-01
b8 certified 1.8389389025E+01 2.0134312832E-01
b8 fityk/double 1.8389389025E+01 2.0134312832E-01
define Gauss2(b1, b2, b3, b4, b5, b6, b7, b8) = b1*exp(-b2*x) + b3*exp(-(x-b4)^2 / b5^2) + b6*exp(-(x-b7)^2 / b8^2)
F = Gauss2(~97.0, ~0.009, ~100.0, ~65.0, ~20.0, ~70.0, ~178.0, ~16.5)
Parameter Standard Deviation
b1 certified 9.9018328406E+01 5.3748766879E-01
b1 fityk/double 9.9018328406E+01 5.3748766879E-01
b2 certified 1.0994945399E-02 1.3335306766E-04
b2 fityk/double 1.0994945399E-02 1.3335306766E-04
b3 certified 1.0188022528E+02 5.9217315772E-01
b3 fityk/double 1.0188022528E+02 5.9217315773E-01
b4 certified 1.0703095519E+02 1.5006798316E-01
b4 fityk/double 1.0703095519E+02 1.5006798316E-01
b5 certified 2.3578584029E+01 2.2695595067E-01
b5 fityk/double 2.3578584028E+01 2.2695595067E-01
b6 certified 7.2045589471E+01 6.1721965884E-01
b6 fityk/double 7.2045589471E+01 6.1721965883E-01
b7 certified 1.5327010194E+02 1.9466674341E-01
b7 fityk/double 1.5327010194E+02 1.9466674342E-01
b8 certified 1.9525972636E+01 2.6416549393E-01
b8 fityk/double 1.9525972637E+01 2.6416549394E-01
define DanWood(b1, b2) = b1*x^b2
F = DanWood(~1, ~5)

All digits the same.

Parameter Standard Deviation
b1 certified 7.6886226176E-01 1.8281973860E-02
b1 fityk/double 7.6886226176E-01 1.8281973860E-02
b2 certified 3.8604055871E+00 5.1726610913E-02
b2 fityk/double 3.8604055871E+00 5.1726610913E-02
define Misra1b(b1, b2) = b1 * (1-(1+b2*x/2)^(-2))
F = Misra1b(~500, ~0.0001)

Again, we get exactly the certified results.

Parameter Standard Deviation
b1 certified 3.3799746163E+02 3.1643950207E+00
b1 fityk/double 3.3799746163E+02 3.1643950207E+00
b2 certified 3.9039091287E-04 4.2547321834E-06
b2 fityk/double 3.9039091287E-04 4.2547321834E-06
define Kirby2(b1, b2, b3, b4, b5) = (b1 + b2*x + b3*x^2) / (1 + b4*x + b5*x^2)
F = Kirby2(~2, ~-0.1, ~0.003, ~-0.001, ~0.00001)
Parameter Standard Deviation
b1 certified 1.6745063063E+00 8.7989634338E-02
b1 fityk/double 1.6745063047E+00 8.7989634326E-02
b2 certified -1.3927397867E-01 4.1182041386E-03
b2 fityk/double -1.3927397859E-01 4.1182041376E-03
b3 certified 2.5961181191E-03 4.1856520458E-05
b3 fityk/double 2.5961181182E-03 4.1856520445E-05
b4 certified -1.7241811870E-03 5.8931897355E-05
b4 fityk/double -1.7241811882E-03 5.8931897332E-05
b5 certified 2.1664802578E-05 2.0129761919E-07
b5 fityk/double 2.1664802573E-05 2.0129761914E-07
define Hahn1(b1, b2, b3, b4, b5, b6, b7) = (b1+b2*x+b3*x^2+b4*x^3) / (1+b5*x+b6*x^2+b7*x^3)
F = Hahn1(~10, ~-1, ~0.05, ~-0.00001, ~-0.05, ~0.001, ~-0.000001)
Parameter Standard Deviation
b1 certified 1.0776351733E+00 1.7070154742E-01
b1 fityk/double 1.0776351730E+00 1.7070154741E-01
b2 certified -1.2269296921E-01 1.2000289189E-02
b2 fityk/double -1.2269296919E-01 1.2000289189E-02
b3 certified 4.0863750610E-03 2.2508314937E-04
b3 fityk/double 4.0863750606E-03 2.2508314935E-04
b4 certified -1.4262662514E-06 2.7578037666E-07
b4 fityk/double -1.4262662508E-06 2.7578037665E-07
b5 certified -5.7609940901E-03 2.4712888219E-04
b5 fityk/double -5.7609940903E-03 2.4712888218E-04
b6 certified 2.4053735503E-04 1.0449373768E-05
b6 fityk/double 2.4053735501E-04 1.0449373767E-05
b7 certified -1.2314450199E-07 1.3027335327E-08
b7 fityk/double -1.2314450196E-07 1.3027335327E-08

Dataset Nelson has 2 predictors (x1 and x2), Fityk does not handle 3D data, we skip this test.

define MGH17(b1, b2, b3, b4, b5) = b1 + b2*exp(-x*b4) + b3*exp(-x*b5)
F = MGH17(~0.5, ~1.5, ~-1, ~0.01, ~0.02)
Parameter Standard Deviation
b1 certified 3.7541005211E-01 2.0723153551E-03
b1 fityk/double 3.7541005211E-01 2.0723153551E-03
b2 certified 1.9358469127E+00 2.2031669222E-01
b2 fityk/double 1.9358469126E+00 2.2031669217E-01
b3 certified -1.4646871366E+00 2.2175707739E-01
b3 fityk/double -1.4646871365E+00 2.2175707733E-01
b4 certified 1.2867534640E-02 4.4861358114E-04
b4 fityk/double 1.2867534640E-02 4.4861358109E-04
b5 certified 2.2122699662E-02 8.9471996575E-04
b5 fityk/double 2.2122699662E-02 8.9471996569E-04
define Lanczos1(b1, b2, b3, b4, b5, b6) = b1*exp(-b2*x) + b3*exp(-b4*x) + b5*exp(-b6*x)
F = Lanczos1(~1.2, ~0.3, ~5.6, ~5.5, ~6.5, ~7.6)
Parameter Standard Deviation
b1 certified 9.5100000027E-02 5.3347304234E-11
b1 fityk/double 9.5100000027E-02 5.3291941815E-11
b2 certified 1.0000000001E+00 2.7473038179E-10
b2 fityk/double 1.0000000001E+00 2.7444527387E-10
b3 certified 8.6070000013E-01 1.3576062225E-10
b3 fityk/double 8.6070000013E-01 1.3561973349E-10
b4 certified 3.0000000002E+00 3.3308253069E-10
b4 fityk/double 3.0000000002E+00 3.3273686649E-10
b5 certified 1.5575999998E+00 1.8815731448E-10
b5 fityk/double 1.5575999998E+00 1.8796204982E-10
b6 certified 5.0000000001E+00 1.1057500538E-10
b6 fityk/double 5.0000000001E+00 1.1046025358E-10
define Lanczos2(b1, b2, b3, b4, b5, b6) = b1*exp(-b2*x) + b3*exp(-b4*x) + b5*exp(-b6*x)
F = Lanczos2(~1.2, ~0.3, ~5.6, ~5.5, ~6.5, ~7.6)
Parameter Standard Deviation
b1 certified 9.6251029939E-02 6.6770575477E-04
b1 fityk/double 9.6251029939E-02 6.6770575433E-04
b2 certified 1.0057332849E+00 3.3989646176E-03
b2 fityk/double 1.0057332849E+00 3.3989646154E-03
b3 certified 8.6424689056E-01 1.7185846685E-03
b3 fityk/double 8.6424689056E-01 1.7185846674E-03
b4 certified 3.0078283915E+00 4.1707005856E-03
b4 fityk/double 3.0078283915E+00 4.1707005829E-03
b5 certified 1.5529016879E+00 2.3744381417E-03
b5 fityk/double 1.5529016879E+00 2.3744381401E-03
b6 certified 5.0028798100E+00 1.3958787284E-03
b6 fityk/double 5.0028798100E+00 1.3958787275E-03
define Gauss3(b1, b2, b3, b4, b5, b6, b7, b8) = b1*exp(-b2*x) + b3*exp(-(x-b4)^2 / b5^2) + b6*exp(-(x-b7)^2 / b8^2)
F = Gauss3(~97.0, ~0.009, ~100.0, ~65.0, ~20.0, ~70.0, ~178.0, ~16.5)
Parameter Standard Deviation
b1 certified 9.8940368970E+01 5.3005192833E-01
b1 fityk/double 9.8940368970E+01 5.3005192832E-01
b2 certified 1.0945879335E-02 1.2554058911E-04
b2 fityk/double 1.0945879335E-02 1.2554058910E-04
b3 certified 1.0069553078E+02 8.1256587317E-01
b3 fityk/double 1.0069553078E+02 8.1256587330E-01
b4 certified 1.1163619459E+02 3.5317859757E-01
b4 fityk/double 1.1163619459E+02 3.5317859756E-01
b5 certified 2.3300500029E+01 3.6584783023E-01
b5 fityk/double 2.3300500027E+01 3.6584783020E-01
b6 certified 7.3705031418E+01 1.2091239082E+00
b6 fityk/double 7.3705031424E+01 1.2091239080E+00
b7 certified 1.4776164251E+02 4.0488183351E-01
b7 fityk/double 1.4776164250E+02 4.0488183353E-01
b8 certified 1.9668221230E+01 3.7806634336E-01
b8 fityk/double 1.9668221231E+01 3.7806634338E-01
define Misra1c(b1, b2) = b1 * (1-(1+2*b2*x)^(-.5))
F = Misra1c(~500, ~0.0001)
Parameter Standard Deviation
b1 certified 6.3642725809E+02 4.6638326572E+00
b1 fityk/double 6.3642725809E+02 4.6638326572E+00
b2 certified 2.0813627256E-04 1.7728423155E-06
b2 fityk/double 2.0813627256E-04 1.7728423155E-06
define Misra1d(b1, b2) = b1*b2*x*((1+b2*x)^(-1))
F = Misra1d(~500, ~0.0001)
Parameter Standard Deviation
b1 certified 4.3736970754E+02 3.6489174345E+00
b1 fityk/double 4.3736970754E+02 3.6489174345E+00
b2 certified 3.0227324449E-04 2.9334354479E-06
b2 fityk/double 3.0227324449E-04 2.9334354479E-06
define Roszman1(b1, b2, b3, b4) = b1 - b2*x - atan(b3/(x-b4))/pi
F = Roszman1(~0.1, ~-0.00001, ~1000, ~-100)
Parameter Standard Deviation
b1 certified 2.0196866396E-01 1.9172666023E-02
b1 fityk/double 2.0196866395E-01 1.9172666023E-02
b2 certified -6.1953516256E-06 3.2058931691E-06
b2 fityk/double -6.1953516245E-06 3.2058931692E-06
b3 certified 1.2044556708E+03 7.4050983057E+01
b3 fityk/double 1.2044556708E+03 7.4050983056E+01
b4 certified -1.8134269537E+02 4.9573513849E+01
b4 fityk/double -1.8134269538E+02 4.9573513852E+01
define Enso(b1, b2, b3, b4, b5, b6, b7, b8, b9) = b1 + b2*cos(t/12) + b3*sin(t/12) + b5*cos(t/b4) + b6*sin(t/b4) + b8*cos(t/b7) + b9*sin(t/b7) where t=2*pi*x
F = Enso(~11.0, ~3.0, ~0.5, ~40.0, ~-0.7, ~-1.3, ~25.0, ~-0.3, ~1.4)
Parameter Standard Deviation
b1 certified 1.0510749193E+01 1.7488832467E-01
b1 fityk/double 1.0510749192E+01 1.7488832468E-01
b2 certified 3.0762128085E+00 2.4310052139E-01
b2 fityk/double 3.0762128089E+00 2.4310052139E-01
b3 certified 5.3280138227E-01 2.4354686618E-01
b3 fityk/double 5.3280138256E-01 2.4354686618E-01
b4 certified 4.4311088700E+01 9.4408025976E-01
b4 fityk/double 4.4311088678E+01 9.4408025781E-01
b5 certified -1.6231428586E+00 2.8078369611E-01
b5 fityk/double -1.6231428614E+00 2.8078369522E-01
b6 certified 5.2554493756E-01 4.8073701119E-01
b6 fityk/double 5.2554493176E-01 4.8073701153E-01
b7 certified 2.6887614440E+01 4.1612939130E-01
b7 fityk/double 2.6887614416E+01 4.1612939012E-01
b8 certified 2.1232288488E-01 5.1460022911E-01
b8 fityk/double 2.1232286294E-01 5.1460022934E-01
b9 certified 1.4966870418E+00 2.5434468893E-01
b9 fityk/double 1.4966870451E+00 2.5434468712E-01
define MGH09(b1, b2, b3, b4) = b1*(x^2+x*b2) / (x^2+x*b3+b4)
F = MGH09(~25, ~39, ~41.5, ~39)
Parameter Standard Deviation
b1 certified 1.9280693458E-01 1.1435312227E-02
b1 fityk/double 1.9280693477E-01 1.1435312166E-02
b2 certified 1.9128232873E-01 1.9633220911E-01
b2 fityk/double 1.9128232455E-01 1.9633220578E-01
b3 certified 1.2305650693E-01 8.0842031232E-02
b3 fityk/double 1.2305650614E-01 8.0842030241E-02
b4 certified 1.3606233068E-01 9.0025542308E-02
b4 fityk/double 1.3606232875E-01 9.0025541048E-02
define Thurber(b1, b2, b3, b4, b5, b6, b7) = (b1 + b2*x + b3*x^2 + b4*x^3) / (1 + b5*x + b6*x^2 + b7*x^3)
F = Thurber(~1000, ~1000, ~400, ~40, ~0.7, ~0.3, ~0.03)
Parameter Standard Deviation
b1 certified 1.2881396800E+03 4.6647963344E+00
b1 fityk/double 1.2881396799E+03 4.6647963334E+00
b2 certified 1.4910792535E+03 3.9571156086E+01
b2 fityk/double 1.4910792548E+03 3.9571155198E+01
b3 certified 5.8323836877E+02 2.8698696102E+01
b3 fityk/double 5.8323836966E+02 2.8698695459E+01
b4 certified 7.5416644291E+01 5.5675370270E+00
b4 fityk/double 7.5416644466E+01 5.5675368991E+00
b5 certified 9.6629502864E-01 3.1333340687E-02
b5 fityk/double 9.6629502951E-01 3.1333340084E-02
b6 certified 3.9797285797E-01 1.4984928198E-02
b6 fityk/double 3.9797285837E-01 1.4984927908E-02
b7 certified 4.9727297349E-02 6.5842344623E-03
b7 fityk/double 4.9727297607E-02 6.5842343110E-03
define BoxBOD(b1, b2) = b1*(1-exp(-b2*x))
F = BoxBOD(~100, ~0.75)
Parameter Standard Deviation
b1 certified 2.1380940889E+02 1.2354515176E+01
b1 fityk/double 2.1380940890E+02 1.2354515178E+01
b2 certified 5.4723748542E-01 1.0455993237E-01
b2 fityk/double 5.4723748534E-01 1.0455993235E-01
define Rat42(b1, b2, b3) = b1 / (1+exp(b2-b3*x))
F = Rat42(~100, ~1, ~0.1)
Parameter Standard Deviation
b1 certified 7.2462237576E+01 1.7340283401E+00
b1 fityk/double 7.2462237576E+01 1.7340283401E+00
b2 certified 2.6180768402E+00 8.8295217536E-02
b2 fityk/double 2.6180768402E+00 8.8295217536E-02
b3 certified 6.7359200066E-02 3.4465663377E-03
b3 fityk/double 6.7359200066E-02 3.4465663377E-03
define MGH10(b1, b2, b3) = b1 * exp(b2/(x+b3))
F = MGH10(~0.02, ~4000, ~250)
Parameter Standard Deviation
b1 certified 5.6096364710E-03 1.5687892471E-04
b1 fityk/double 5.6096364710E-03 1.5687892478E-04
b2 certified 6.1813463463E+03 2.3309021107E+01
b2 fityk/double 6.1813463463E+03 2.3309021118E+01
b3 certified 3.4522363462E+02 7.8486103508E-01
b3 fityk/double 3.4522363462E+02 7.8486103544E-01
define Eckerle4(b1, b2, b3) = b1 * exp(b2/(x+b3))
F = Eckerle4(~1.5, ~5, ~450)
Parameter Standard Deviation
b1 certified 1.5543827178E+00 1.5408051163E-02
b1 fityk/double 1.5543827179E+00 1.5408051165E-02
b2 certified 4.0888321754E+00 4.6803020753E-02
b2 fityk/double 4.0888321764E+00 4.6803020765E-02
b3 certified 4.5154121844E+02 4.6800518816E-02
b3 fityk/double 4.5154121844E+02 4.6800518828E-02
define Rat43(b1, b2, b3, b4) = b1 / ((1+exp(b2-b3*x))^(1/b4))
F = Rat43(~100, ~10, ~1, ~1)
Parameter Standard Deviation
b1 certified 6.9964151270E+02 1.6302297817E+01
b1 fityk/double 6.9964151266E+02 1.6302297809E+01
b2 certified 5.2771253025E+00 2.0828735829E+00
b2 fityk/double 5.2771253072E+00 2.0828735841E+00
b3 certified 7.5962938329E-01 1.9566123451E-01
b3 fityk/double 7.5962938380E-01 1.9566123468E-01
b4 certified 1.2792483859E+00 6.8761936385E-01
b4 fityk/double 1.2792483872E+00 6.8761936428E-01
define Bennett5(b1, b2, b3) = b1 * (b2+x)^(-1/b3)
F = Bennett5(~-2000, ~50, ~0.8)
Parameter Standard Deviation
b1 certified -2.5235058043E+03 2.9715175411E+02
b1 fityk/double -2.5235058043E+03 2.9715175183E+02
b2 certified 4.6736564644E+01 1.2448871856E+00
b2 fityk/double 4.6736564644E+01 1.2448871760E+00
b3 certified 9.3218483193E-01 2.0272299378E-02
b3 fityk/double 9.3218483193E-01 2.0272299222E-02