This is gsl-ref.info, produced by makeinfo version 5.1 from
gsl-ref.texi.

Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 The GSL
Team.

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with the
Invariant Sections being "GNU General Public License" and "Free Software
Needs Free Documentation", the Front-Cover text being "A GNU Manual",
and with the Back-Cover Text being (a) (see below).  A copy of the
license is included in the section entitled "GNU Free Documentation
License".

   (a) The Back-Cover Text is: "You have the freedom to copy and modify
this GNU Manual."
INFO-DIR-SECTION Software libraries
START-INFO-DIR-ENTRY
* gsl-ref: (gsl-ref).                   GNU Scientific Library - Reference
END-INFO-DIR-ENTRY


File: gsl-ref.info,  Node: Root Finding Caveats,  Next: Initializing the Solver,  Prev: Root Finding Overview,  Up: One dimensional Root-Finding

34.2 Caveats
============

Note that root finding functions can only search for one root at a time.
When there are several roots in the search area, the first root to be
found will be returned; however it is difficult to predict which of the
roots this will be.  _In most cases, no error will be reported if you
try to find a root in an area where there is more than one._

   Care must be taken when a function may have a multiple root (such as
f(x) = (x-x_0)^2 or f(x) = (x-x_0)^3).  It is not possible to use
root-bracketing algorithms on even-multiplicity roots.  For these
algorithms the initial interval must contain a zero-crossing, where the
function is negative at one end of the interval and positive at the
other end.  Roots with even-multiplicity do not cross zero, but only
touch it instantaneously.  Algorithms based on root bracketing will
still work for odd-multiplicity roots (e.g.  cubic, quintic, ...).  Root
polishing algorithms generally work with higher multiplicity roots, but
at a reduced rate of convergence.  In these cases the "Steffenson
algorithm" can be used to accelerate the convergence of multiple roots.

   While it is not absolutely required that f have a root within the
search region, numerical root finding functions should not be used
haphazardly to check for the _existence_ of roots.  There are better
ways to do this.  Because it is easy to create situations where
numerical root finders can fail, it is a bad idea to throw a root finder
at a function you do not know much about.  In general it is best to
examine the function visually by plotting before searching for a root.


File: gsl-ref.info,  Node: Initializing the Solver,  Next: Providing the function to solve,  Prev: Root Finding Caveats,  Up: One dimensional Root-Finding

34.3 Initializing the Solver
============================

 -- Function: gsl_root_fsolver * gsl_root_fsolver_alloc (const
          gsl_root_fsolver_type * T)
     This function returns a pointer to a newly allocated instance of a
     solver of type T.  For example, the following code creates an
     instance of a bisection solver,

          const gsl_root_fsolver_type * T
            = gsl_root_fsolver_bisection;
          gsl_root_fsolver * s
            = gsl_root_fsolver_alloc (T);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_root_fdfsolver * gsl_root_fdfsolver_alloc (const
          gsl_root_fdfsolver_type * T)
     This function returns a pointer to a newly allocated instance of a
     derivative-based solver of type T.  For example, the following code
     creates an instance of a Newton-Raphson solver,

          const gsl_root_fdfsolver_type * T
            = gsl_root_fdfsolver_newton;
          gsl_root_fdfsolver * s
            = gsl_root_fdfsolver_alloc (T);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: int gsl_root_fsolver_set (gsl_root_fsolver * S,
          gsl_function * F, double X_LOWER, double X_UPPER)
     This function initializes, or reinitializes, an existing solver S
     to use the function F and the initial search interval [X_LOWER,
     X_UPPER].

 -- Function: int gsl_root_fdfsolver_set (gsl_root_fdfsolver * S,
          gsl_function_fdf * FDF, double ROOT)
     This function initializes, or reinitializes, an existing solver S
     to use the function and derivative FDF and the initial guess ROOT.

 -- Function: void gsl_root_fsolver_free (gsl_root_fsolver * S)
 -- Function: void gsl_root_fdfsolver_free (gsl_root_fdfsolver * S)
     These functions free all the memory associated with the solver S.

 -- Function: const char * gsl_root_fsolver_name (const gsl_root_fsolver
          * S)
 -- Function: const char * gsl_root_fdfsolver_name (const
          gsl_root_fdfsolver * S)
     These functions return a pointer to the name of the solver.  For
     example,

          printf ("s is a '%s' solver\n",
                  gsl_root_fsolver_name (s));

     would print something like 's is a 'bisection' solver'.


File: gsl-ref.info,  Node: Providing the function to solve,  Next: Search Bounds and Guesses,  Prev: Initializing the Solver,  Up: One dimensional Root-Finding

34.4 Providing the function to solve
====================================

You must provide a continuous function of one variable for the root
finders to operate on, and, sometimes, its first derivative.  In order
to allow for general parameters the functions are defined by the
following data types:

 -- Data Type: gsl_function
     This data type defines a general function with parameters.

     'double (* function) (double X, void * PARAMS)'
          this function should return the value f(x,params) for argument
          X and parameters PARAMS

     'void * params'
          a pointer to the parameters of the function

   Here is an example for the general quadratic function,

     f(x) = a x^2 + b x + c

with a = 3, b = 2, c = 1.  The following code defines a 'gsl_function'
'F' which you could pass to a root finder as a function pointer:

     struct my_f_params { double a; double b; double c; };

     double
     my_f (double x, void * p) {
        struct my_f_params * params
          = (struct my_f_params *)p;
        double a = (params->a);
        double b = (params->b);
        double c = (params->c);

        return  (a * x + b) * x + c;
     }

     gsl_function F;
     struct my_f_params params = { 3.0, 2.0, 1.0 };

     F.function = &my_f;
     F.params = &params;

The function f(x) can be evaluated using the macro 'GSL_FN_EVAL(&F,x)'
defined in 'gsl_math.h'.

 -- Data Type: gsl_function_fdf
     This data type defines a general function with parameters and its
     first derivative.

     'double (* f) (double X, void * PARAMS)'
          this function should return the value of f(x,params) for
          argument X and parameters PARAMS

     'double (* df) (double X, void * PARAMS)'
          this function should return the value of the derivative of F
          with respect to X, f'(x,params), for argument X and parameters
          PARAMS

     'void (* fdf) (double X, void * PARAMS, double * F, double * DF)'
          this function should set the values of the function F to
          f(x,params) and its derivative DF to f'(x,params) for argument
          X and parameters PARAMS.  This function provides an
          optimization of the separate functions for f(x) and f'(x)--it
          is always faster to compute the function and its derivative at
          the same time.

     'void * params'
          a pointer to the parameters of the function

   Here is an example where f(x) = 2\exp(2x):

     double
     my_f (double x, void * params)
     {
        return exp (2 * x);
     }

     double
     my_df (double x, void * params)
     {
        return 2 * exp (2 * x);
     }

     void
     my_fdf (double x, void * params,
             double * f, double * df)
     {
        double t = exp (2 * x);

        *f = t;
        *df = 2 * t;   /* uses existing value */
     }

     gsl_function_fdf FDF;

     FDF.f = &my_f;
     FDF.df = &my_df;
     FDF.fdf = &my_fdf;
     FDF.params = 0;

The function f(x) can be evaluated using the macro
'GSL_FN_FDF_EVAL_F(&FDF,x)' and the derivative f'(x) can be evaluated
using the macro 'GSL_FN_FDF_EVAL_DF(&FDF,x)'.  Both the function y =
f(x) and its derivative dy = f'(x) can be evaluated at the same time
using the macro 'GSL_FN_FDF_EVAL_F_DF(&FDF,x,y,dy)'.  The macro stores
f(x) in its Y argument and f'(x) in its DY argument--both of these
should be pointers to 'double'.


File: gsl-ref.info,  Node: Search Bounds and Guesses,  Next: Root Finding Iteration,  Prev: Providing the function to solve,  Up: One dimensional Root-Finding

34.5 Search Bounds and Guesses
==============================

You provide either search bounds or an initial guess; this section
explains how search bounds and guesses work and how function arguments
control them.

   A guess is simply an x value which is iterated until it is within the
desired precision of a root.  It takes the form of a 'double'.

   Search bounds are the endpoints of an interval which is iterated
until the length of the interval is smaller than the requested
precision.  The interval is defined by two values, the lower limit and
the upper limit.  Whether the endpoints are intended to be included in
the interval or not depends on the context in which the interval is
used.


File: gsl-ref.info,  Node: Root Finding Iteration,  Next: Search Stopping Parameters,  Prev: Search Bounds and Guesses,  Up: One dimensional Root-Finding

34.6 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any solver of the
corresponding type.  The same functions work for all solvers so that
different methods can be substituted at runtime without modifications to
the code.

 -- Function: int gsl_root_fsolver_iterate (gsl_root_fsolver * S)
 -- Function: int gsl_root_fdfsolver_iterate (gsl_root_fdfsolver * S)
     These functions perform a single iteration of the solver S.  If the
     iteration encounters an unexpected problem then an error code will
     be returned,

     'GSL_EBADFUNC'
          the iteration encountered a singular point where the function
          or its derivative evaluated to 'Inf' or 'NaN'.

     'GSL_EZERODIV'
          the derivative of the function vanished at the iteration
          point, preventing the algorithm from continuing without a
          division by zero.

   The solver maintains a current best estimate of the root at all
times.  The bracketing solvers also keep track of the current best
interval bounding the root.  This information can be accessed with the
following auxiliary functions,

 -- Function: double gsl_root_fsolver_root (const gsl_root_fsolver * S)
 -- Function: double gsl_root_fdfsolver_root (const gsl_root_fdfsolver *
          S)
     These functions return the current estimate of the root for the
     solver S.

 -- Function: double gsl_root_fsolver_x_lower (const gsl_root_fsolver *
          S)
 -- Function: double gsl_root_fsolver_x_upper (const gsl_root_fsolver *
          S)
     These functions return the current bracketing interval for the
     solver S.


File: gsl-ref.info,  Node: Search Stopping Parameters,  Next: Root Bracketing Algorithms,  Prev: Root Finding Iteration,  Up: One dimensional Root-Finding

34.7 Search Stopping Parameters
===============================

A root finding procedure should stop when one of the following
conditions is true:

   * A root has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result in
several standard ways.

 -- Function: int gsl_root_test_interval (double X_LOWER, double
          X_UPPER, double EPSABS, double EPSREL)
     This function tests for the convergence of the interval [X_LOWER,
     X_UPPER] with absolute error EPSABS and relative error EPSREL.  The
     test returns 'GSL_SUCCESS' if the following condition is achieved,

          |a - b| < epsabs + epsrel min(|a|,|b|)

     when the interval x = [a,b] does not include the origin.  If the
     interval includes the origin then \min(|a|,|b|) is replaced by zero
     (which is the minimum value of |x| over the interval).  This
     ensures that the relative error is accurately estimated for roots
     close to the origin.

     This condition on the interval also implies that any estimate of
     the root r in the interval satisfies the same condition with
     respect to the true root r^*,

          |r - r^*| < epsabs + epsrel r^*

     assuming that the true root r^* is contained within the interval.

 -- Function: int gsl_root_test_delta (double X1, double X0, double
          EPSABS, double EPSREL)

     This function tests for the convergence of the sequence ..., X0, X1
     with absolute error EPSABS and relative error EPSREL.  The test
     returns 'GSL_SUCCESS' if the following condition is achieved,

          |x_1 - x_0| < epsabs + epsrel |x_1|

     and returns 'GSL_CONTINUE' otherwise.

 -- Function: int gsl_root_test_residual (double F, double EPSABS)
     This function tests the residual value F against the absolute error
     bound EPSABS.  The test returns 'GSL_SUCCESS' if the following
     condition is achieved,

          |f| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  This criterion is suitable
     for situations where the precise location of the root, x, is
     unimportant provided a value can be found where the residual,
     |f(x)|, is small enough.


File: gsl-ref.info,  Node: Root Bracketing Algorithms,  Next: Root Finding Algorithms using Derivatives,  Prev: Search Stopping Parameters,  Up: One dimensional Root-Finding

34.8 Root Bracketing Algorithms
===============================

The root bracketing algorithms described in this section require an
initial interval which is guaranteed to contain a root--if a and b are
the endpoints of the interval then f(a) must differ in sign from f(b).
This ensures that the function crosses zero at least once in the
interval.  If a valid initial interval is used then these algorithm
cannot fail, provided the function is well-behaved.

   Note that a bracketing algorithm cannot find roots of even degree,
since these do not cross the x-axis.

 -- Solver: gsl_root_fsolver_bisection

     The "bisection algorithm" is the simplest method of bracketing the
     roots of a function.  It is the slowest algorithm provided by the
     library, with linear convergence.

     On each iteration, the interval is bisected and the value of the
     function at the midpoint is calculated.  The sign of this value is
     used to determine which half of the interval does not contain a
     root.  That half is discarded to give a new, smaller interval
     containing the root.  This procedure can be continued indefinitely
     until the interval is sufficiently small.

     At any time the current estimate of the root is taken as the
     midpoint of the interval.

 -- Solver: gsl_root_fsolver_falsepos

     The "false position algorithm" is a method of finding roots based
     on linear interpolation.  Its convergence is linear, but it is
     usually faster than bisection.

     On each iteration a line is drawn between the endpoints (a,f(a))
     and (b,f(b)) and the point where this line crosses the x-axis taken
     as a "midpoint".  The value of the function at this point is
     calculated and its sign is used to determine which side of the
     interval does not contain a root.  That side is discarded to give a
     new, smaller interval containing the root.  This procedure can be
     continued indefinitely until the interval is sufficiently small.

     The best estimate of the root is taken from the linear
     interpolation of the interval on the current iteration.

 -- Solver: gsl_root_fsolver_brent

     The "Brent-Dekker method" (referred to here as "Brent's method")
     combines an interpolation strategy with the bisection algorithm.
     This produces a fast algorithm which is still robust.

     On each iteration Brent's method approximates the function using an
     interpolating curve.  On the first iteration this is a linear
     interpolation of the two endpoints.  For subsequent iterations the
     algorithm uses an inverse quadratic fit to the last three points,
     for higher accuracy.  The intercept of the interpolating curve with
     the x-axis is taken as a guess for the root.  If it lies within the
     bounds of the current interval then the interpolating point is
     accepted, and used to generate a smaller interval.  If the
     interpolating point is not accepted then the algorithm falls back
     to an ordinary bisection step.

     The best estimate of the root is taken from the most recent
     interpolation or bisection.


File: gsl-ref.info,  Node: Root Finding Algorithms using Derivatives,  Next: Root Finding Examples,  Prev: Root Bracketing Algorithms,  Up: One dimensional Root-Finding

34.9 Root Finding Algorithms using Derivatives
==============================================

The root polishing algorithms described in this section require an
initial guess for the location of the root.  There is no absolute
guarantee of convergence--the function must be suitable for this
technique and the initial guess must be sufficiently close to the root
for it to work.  When these conditions are satisfied then convergence is
quadratic.

   These algorithms make use of both the function and its derivative.

 -- Derivative Solver: gsl_root_fdfsolver_newton

     Newton's Method is the standard root-polishing algorithm.  The
     algorithm begins with an initial guess for the location of the
     root.  On each iteration, a line tangent to the function f is drawn
     at that position.  The point where this line crosses the x-axis
     becomes the new guess.  The iteration is defined by the following
     sequence,

          x_{i+1} = x_i - f(x_i)/f'(x_i)

     Newton's method converges quadratically for single roots, and
     linearly for multiple roots.

 -- Derivative Solver: gsl_root_fdfsolver_secant

     The "secant method" is a simplified version of Newton's method
     which does not require the computation of the derivative on every
     step.

     On its first iteration the algorithm begins with Newton's method,
     using the derivative to compute a first step,

          x_1 = x_0 - f(x_0)/f'(x_0)

     Subsequent iterations avoid the evaluation of the derivative by
     replacing it with a numerical estimate, the slope of the line
     through the previous two points,

          x_{i+1} = x_i f(x_i) / f'_{est} where
           f'_{est} = (f(x_i) - f(x_{i-1})/(x_i - x_{i-1})

     When the derivative does not change significantly in the vicinity
     of the root the secant method gives a useful saving.
     Asymptotically the secant method is faster than Newton's method
     whenever the cost of evaluating the derivative is more than 0.44
     times the cost of evaluating the function itself.  As with all
     methods of computing a numerical derivative the estimate can suffer
     from cancellation errors if the separation of the points becomes
     too small.

     On single roots, the method has a convergence of order (1 + \sqrt
     5)/2 (approximately 1.62).  It converges linearly for multiple
     roots.

 -- Derivative Solver: gsl_root_fdfsolver_steffenson

     The "Steffenson Method"(1) provides the fastest convergence of all
     the routines.  It combines the basic Newton algorithm with an
     Aitken "delta-squared" acceleration.  If the Newton iterates are
     x_i then the acceleration procedure generates a new sequence R_i,

          R_i = x_i - (x_{i+1} - x_i)^2 / (x_{i+2} - 2 x_{i+1} + x_{i})

     which converges faster than the original sequence under reasonable
     conditions.  The new sequence requires three terms before it can
     produce its first value so the method returns accelerated values on
     the second and subsequent iterations.  On the first iteration it
     returns the ordinary Newton estimate.  The Newton iterate is also
     returned if the denominator of the acceleration term ever becomes
     zero.

     As with all acceleration procedures this method can become unstable
     if the function is not well-behaved.

   ---------- Footnotes ----------

   (1) J.F. Steffensen (1873-1961).  The spelling used in the name of
the function is slightly incorrect, but has been preserved to avoid
incompatibility.


File: gsl-ref.info,  Node: Root Finding Examples,  Next: Root Finding References and Further Reading,  Prev: Root Finding Algorithms using Derivatives,  Up: One dimensional Root-Finding

34.10 Examples
==============

For any root finding algorithm we need to prepare the function to be
solved.  For this example we will use the general quadratic equation
described earlier.  We first need a header file ('demo_fn.h') to define
the function parameters,

     struct quadratic_params
       {
         double a, b, c;
       };

     double quadratic (double x, void *params);
     double quadratic_deriv (double x, void *params);
     void quadratic_fdf (double x, void *params,
                         double *y, double *dy);

We place the function definitions in a separate file ('demo_fn.c'),

     double
     quadratic (double x, void *params)
     {
       struct quadratic_params *p
         = (struct quadratic_params *) params;

       double a = p->a;
       double b = p->b;
       double c = p->c;

       return (a * x + b) * x + c;
     }

     double
     quadratic_deriv (double x, void *params)
     {
       struct quadratic_params *p
         = (struct quadratic_params *) params;

       double a = p->a;
       double b = p->b;

       return 2.0 * a * x + b;
     }

     void
     quadratic_fdf (double x, void *params,
                    double *y, double *dy)
     {
       struct quadratic_params *p
         = (struct quadratic_params *) params;

       double a = p->a;
       double b = p->b;
       double c = p->c;

       *y = (a * x + b) * x + c;
       *dy = 2.0 * a * x + b;
     }

The first program uses the function solver 'gsl_root_fsolver_brent' for
Brent's method and the general quadratic defined above to solve the
following equation,

     x^2 - 5 = 0

with solution x = \sqrt 5 = 2.236068...

     #include <stdio.h>
     #include <gsl/gsl_errno.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_roots.h>

     #include "demo_fn.h"
     #include "demo_fn.c"

     int
     main (void)
     {
       int status;
       int iter = 0, max_iter = 100;
       const gsl_root_fsolver_type *T;
       gsl_root_fsolver *s;
       double r = 0, r_expected = sqrt (5.0);
       double x_lo = 0.0, x_hi = 5.0;
       gsl_function F;
       struct quadratic_params params = {1.0, 0.0, -5.0};

       F.function = &quadratic;
       F.params = &params;

       T = gsl_root_fsolver_brent;
       s = gsl_root_fsolver_alloc (T);
       gsl_root_fsolver_set (s, &F, x_lo, x_hi);

       printf ("using %s method\n",
               gsl_root_fsolver_name (s));

       printf ("%5s [%9s, %9s] %9s %10s %9s\n",
               "iter", "lower", "upper", "root",
               "err", "err(est)");

       do
         {
           iter++;
           status = gsl_root_fsolver_iterate (s);
           r = gsl_root_fsolver_root (s);
           x_lo = gsl_root_fsolver_x_lower (s);
           x_hi = gsl_root_fsolver_x_upper (s);
           status = gsl_root_test_interval (x_lo, x_hi,
                                            0, 0.001);

           if (status == GSL_SUCCESS)
             printf ("Converged:\n");

           printf ("%5d [%.7f, %.7f] %.7f %+.7f %.7f\n",
                   iter, x_lo, x_hi,
                   r, r - r_expected,
                   x_hi - x_lo);
         }
       while (status == GSL_CONTINUE && iter < max_iter);

       gsl_root_fsolver_free (s);

       return status;
     }

Here are the results of the iterations,

     $ ./a.out
     using brent method
      iter [    lower,     upper]      root        err  err(est)
         1 [1.0000000, 5.0000000] 1.0000000 -1.2360680 4.0000000
         2 [1.0000000, 3.0000000] 3.0000000 +0.7639320 2.0000000
         3 [2.0000000, 3.0000000] 2.0000000 -0.2360680 1.0000000
         4 [2.2000000, 3.0000000] 2.2000000 -0.0360680 0.8000000
         5 [2.2000000, 2.2366300] 2.2366300 +0.0005621 0.0366300
     Converged:
         6 [2.2360634, 2.2366300] 2.2360634 -0.0000046 0.0005666

If the program is modified to use the bisection solver instead of
Brent's method, by changing 'gsl_root_fsolver_brent' to
'gsl_root_fsolver_bisection' the slower convergence of the Bisection
method can be observed,

     $ ./a.out
     using bisection method
      iter [    lower,     upper]      root        err  err(est)
         1 [0.0000000, 2.5000000] 1.2500000 -0.9860680 2.5000000
         2 [1.2500000, 2.5000000] 1.8750000 -0.3610680 1.2500000
         3 [1.8750000, 2.5000000] 2.1875000 -0.0485680 0.6250000
         4 [2.1875000, 2.5000000] 2.3437500 +0.1076820 0.3125000
         5 [2.1875000, 2.3437500] 2.2656250 +0.0295570 0.1562500
         6 [2.1875000, 2.2656250] 2.2265625 -0.0095055 0.0781250
         7 [2.2265625, 2.2656250] 2.2460938 +0.0100258 0.0390625
         8 [2.2265625, 2.2460938] 2.2363281 +0.0002601 0.0195312
         9 [2.2265625, 2.2363281] 2.2314453 -0.0046227 0.0097656
        10 [2.2314453, 2.2363281] 2.2338867 -0.0021813 0.0048828
        11 [2.2338867, 2.2363281] 2.2351074 -0.0009606 0.0024414
     Converged:
        12 [2.2351074, 2.2363281] 2.2357178 -0.0003502 0.0012207

   The next program solves the same function using a derivative solver
instead.

     #include <stdio.h>
     #include <gsl/gsl_errno.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_roots.h>

     #include "demo_fn.h"
     #include "demo_fn.c"

     int
     main (void)
     {
       int status;
       int iter = 0, max_iter = 100;
       const gsl_root_fdfsolver_type *T;
       gsl_root_fdfsolver *s;
       double x0, x = 5.0, r_expected = sqrt (5.0);
       gsl_function_fdf FDF;
       struct quadratic_params params = {1.0, 0.0, -5.0};

       FDF.f = &quadratic;
       FDF.df = &quadratic_deriv;
       FDF.fdf = &quadratic_fdf;
       FDF.params = &params;

       T = gsl_root_fdfsolver_newton;
       s = gsl_root_fdfsolver_alloc (T);
       gsl_root_fdfsolver_set (s, &FDF, x);

       printf ("using %s method\n",
               gsl_root_fdfsolver_name (s));

       printf ("%-5s %10s %10s %10s\n",
               "iter", "root", "err", "err(est)");
       do
         {
           iter++;
           status = gsl_root_fdfsolver_iterate (s);
           x0 = x;
           x = gsl_root_fdfsolver_root (s);
           status = gsl_root_test_delta (x, x0, 0, 1e-3);

           if (status == GSL_SUCCESS)
             printf ("Converged:\n");

           printf ("%5d %10.7f %+10.7f %10.7f\n",
                   iter, x, x - r_expected, x - x0);
         }
       while (status == GSL_CONTINUE && iter < max_iter);

       gsl_root_fdfsolver_free (s);
       return status;
     }

Here are the results for Newton's method,

     $ ./a.out
     using newton method
     iter        root        err   err(est)
         1  3.0000000 +0.7639320 -2.0000000
         2  2.3333333 +0.0972654 -0.6666667
         3  2.2380952 +0.0020273 -0.0952381
     Converged:
         4  2.2360689 +0.0000009 -0.0020263

Note that the error can be estimated more accurately by taking the
difference between the current iterate and next iterate rather than the
previous iterate.  The other derivative solvers can be investigated by
changing 'gsl_root_fdfsolver_newton' to 'gsl_root_fdfsolver_secant' or
'gsl_root_fdfsolver_steffenson'.


File: gsl-ref.info,  Node: Root Finding References and Further Reading,  Prev: Root Finding Examples,  Up: One dimensional Root-Finding

34.11 References and Further Reading
====================================

For information on the Brent-Dekker algorithm see the following two
papers,

     R. P. Brent, "An algorithm with guaranteed convergence for finding
     a zero of a function", 'Computer Journal', 14 (1971) 422-425

     J. C. P. Bus and T. J. Dekker, "Two Efficient Algorithms with
     Guaranteed Convergence for Finding a Zero of a Function", 'ACM
     Transactions of Mathematical Software', Vol. 1 No. 4 (1975) 330-345


File: gsl-ref.info,  Node: One dimensional Minimization,  Next: Multidimensional Root-Finding,  Prev: One dimensional Root-Finding,  Up: Top

35 One dimensional Minimization
*******************************

This chapter describes routines for finding minima of arbitrary
one-dimensional functions.  The library provides low level components
for a variety of iterative minimizers and convergence tests.  These can
be combined by the user to achieve the desired solution, with full
access to the intermediate steps of the algorithms.  Each class of
methods uses the same framework, so that you can switch between
minimizers at runtime without needing to recompile your program.  Each
instance of a minimizer keeps track of its own state, allowing the
minimizers to be used in multi-threaded programs.

   The header file 'gsl_min.h' contains prototypes for the minimization
functions and related declarations.  To use the minimization algorithms
to find the maximum of a function simply invert its sign.

* Menu:

* Minimization Overview::       
* Minimization Caveats::        
* Initializing the Minimizer::  
* Providing the function to minimize::  
* Minimization Iteration::      
* Minimization Stopping Parameters::  
* Minimization Algorithms::     
* Minimization Examples::       
* Minimization References and Further Reading::  


File: gsl-ref.info,  Node: Minimization Overview,  Next: Minimization Caveats,  Up: One dimensional Minimization

35.1 Overview
=============

The minimization algorithms begin with a bounded region known to contain
a minimum.  The region is described by a lower bound a and an upper
bound b, with an estimate of the location of the minimum x.

The value of the function at x must be less than the value of the
function at the ends of the interval,

     f(a) > f(x) < f(b)

This condition guarantees that a minimum is contained somewhere within
the interval.  On each iteration a new point x' is selected using one of
the available algorithms.  If the new point is a better estimate of the
minimum, i.e. where f(x') < f(x), then the current estimate of the
minimum x is updated.  The new point also allows the size of the bounded
interval to be reduced, by choosing the most compact set of points which
satisfies the constraint f(a) > f(x) < f(b).  The interval is reduced
until it encloses the true minimum to a desired tolerance.  This
provides a best estimate of the location of the minimum and a rigorous
error estimate.

   Several bracketing algorithms are available within a single
framework.  The user provides a high-level driver for the algorithm, and
the library provides the individual functions necessary for each of the
steps.  There are three main phases of the iteration.  The steps are,

   * initialize minimizer state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

The state for the minimizers is held in a 'gsl_min_fminimizer' struct.
The updating procedure uses only function evaluations (not derivatives).


File: gsl-ref.info,  Node: Minimization Caveats,  Next: Initializing the Minimizer,  Prev: Minimization Overview,  Up: One dimensional Minimization

35.2 Caveats
============

Note that minimization functions can only search for one minimum at a
time.  When there are several minima in the search area, the first
minimum to be found will be returned; however it is difficult to predict
which of the minima this will be.  _In most cases, no error will be
reported if you try to find a minimum in an area where there is more
than one._

   With all minimization algorithms it can be difficult to determine the
location of the minimum to full numerical precision.  The behavior of
the function in the region of the minimum x^* can be approximated by a
Taylor expansion,

     y = f(x^*) + (1/2) f''(x^*) (x - x^*)^2

and the second term of this expansion can be lost when added to the
first term at finite precision.  This magnifies the error in locating
x^*, making it proportional to \sqrt \epsilon (where \epsilon is the
relative accuracy of the floating point numbers).  For functions with
higher order minima, such as x^4, the magnification of the error is
correspondingly worse.  The best that can be achieved is to converge to
the limit of numerical accuracy in the function values, rather than the
location of the minimum itself.


File: gsl-ref.info,  Node: Initializing the Minimizer,  Next: Providing the function to minimize,  Prev: Minimization Caveats,  Up: One dimensional Minimization

35.3 Initializing the Minimizer
===============================

 -- Function: gsl_min_fminimizer * gsl_min_fminimizer_alloc (const
          gsl_min_fminimizer_type * T)
     This function returns a pointer to a newly allocated instance of a
     minimizer of type T.  For example, the following code creates an
     instance of a golden section minimizer,

          const gsl_min_fminimizer_type * T
            = gsl_min_fminimizer_goldensection;
          gsl_min_fminimizer * s
            = gsl_min_fminimizer_alloc (T);

     If there is insufficient memory to create the minimizer then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: int gsl_min_fminimizer_set (gsl_min_fminimizer * S,
          gsl_function * F, double X_MINIMUM, double X_LOWER, double
          X_UPPER)
     This function sets, or resets, an existing minimizer S to use the
     function F and the initial search interval [X_LOWER, X_UPPER], with
     a guess for the location of the minimum X_MINIMUM.

     If the interval given does not contain a minimum, then the function
     returns an error code of 'GSL_EINVAL'.

 -- Function: int gsl_min_fminimizer_set_with_values (gsl_min_fminimizer
          * S, gsl_function * F, double X_MINIMUM, double F_MINIMUM,
          double X_LOWER, double F_LOWER, double X_UPPER, double
          F_UPPER)
     This function is equivalent to 'gsl_min_fminimizer_set' but uses
     the values F_MINIMUM, F_LOWER and F_UPPER instead of computing
     'f(x_minimum)', 'f(x_lower)' and 'f(x_upper)'.

 -- Function: void gsl_min_fminimizer_free (gsl_min_fminimizer * S)
     This function frees all the memory associated with the minimizer S.

 -- Function: const char * gsl_min_fminimizer_name (const
          gsl_min_fminimizer * S)
     This function returns a pointer to the name of the minimizer.  For
     example,

          printf ("s is a '%s' minimizer\n",
                  gsl_min_fminimizer_name (s));

     would print something like 's is a 'brent' minimizer'.


File: gsl-ref.info,  Node: Providing the function to minimize,  Next: Minimization Iteration,  Prev: Initializing the Minimizer,  Up: One dimensional Minimization

35.4 Providing the function to minimize
=======================================

You must provide a continuous function of one variable for the
minimizers to operate on.  In order to allow for general parameters the
functions are defined by a 'gsl_function' data type (*note Providing the
function to solve::).


File: gsl-ref.info,  Node: Minimization Iteration,  Next: Minimization Stopping Parameters,  Prev: Providing the function to minimize,  Up: One dimensional Minimization

35.5 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any minimizer of
the corresponding type.  The same functions work for all minimizers so
that different methods can be substituted at runtime without
modifications to the code.

 -- Function: int gsl_min_fminimizer_iterate (gsl_min_fminimizer * S)
     This function performs a single iteration of the minimizer S.  If
     the iteration encounters an unexpected problem then an error code
     will be returned,

     'GSL_EBADFUNC'
          the iteration encountered a singular point where the function
          evaluated to 'Inf' or 'NaN'.

     'GSL_FAILURE'
          the algorithm could not improve the current best approximation
          or bounding interval.

   The minimizer maintains a current best estimate of the position of
the minimum at all times, and the current interval bounding the minimum.
This information can be accessed with the following auxiliary functions,

 -- Function: double gsl_min_fminimizer_x_minimum (const
          gsl_min_fminimizer * S)
     This function returns the current estimate of the position of the
     minimum for the minimizer S.

 -- Function: double gsl_min_fminimizer_x_upper (const
          gsl_min_fminimizer * S)
 -- Function: double gsl_min_fminimizer_x_lower (const
          gsl_min_fminimizer * S)
     These functions return the current upper and lower bound of the
     interval for the minimizer S.

 -- Function: double gsl_min_fminimizer_f_minimum (const
          gsl_min_fminimizer * S)
 -- Function: double gsl_min_fminimizer_f_upper (const
          gsl_min_fminimizer * S)
 -- Function: double gsl_min_fminimizer_f_lower (const
          gsl_min_fminimizer * S)
     These functions return the value of the function at the current
     estimate of the minimum and at the upper and lower bounds of the
     interval for the minimizer S.


File: gsl-ref.info,  Node: Minimization Stopping Parameters,  Next: Minimization Algorithms,  Prev: Minimization Iteration,  Up: One dimensional Minimization

35.6 Stopping Parameters
========================

A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The function
below allows the user to test the precision of the current result.

 -- Function: int gsl_min_test_interval (double X_LOWER, double X_UPPER,
          double EPSABS, double EPSREL)
     This function tests for the convergence of the interval [X_LOWER,
     X_UPPER] with absolute error EPSABS and relative error EPSREL.  The
     test returns 'GSL_SUCCESS' if the following condition is achieved,

          |a - b| < epsabs + epsrel min(|a|,|b|)

     when the interval x = [a,b] does not include the origin.  If the
     interval includes the origin then \min(|a|,|b|) is replaced by zero
     (which is the minimum value of |x| over the interval).  This
     ensures that the relative error is accurately estimated for minima
     close to the origin.

     This condition on the interval also implies that any estimate of
     the minimum x_m in the interval satisfies the same condition with
     respect to the true minimum x_m^*,

          |x_m - x_m^*| < epsabs + epsrel x_m^*

     assuming that the true minimum x_m^* is contained within the
     interval.


File: gsl-ref.info,  Node: Minimization Algorithms,  Next: Minimization Examples,  Prev: Minimization Stopping Parameters,  Up: One dimensional Minimization

35.7 Minimization Algorithms
============================

The minimization algorithms described in this section require an initial
interval which is guaranteed to contain a minimum--if a and b are the
endpoints of the interval and x is an estimate of the minimum then f(a)
> f(x) < f(b).  This ensures that the function has at least one minimum
somewhere in the interval.  If a valid initial interval is used then
these algorithm cannot fail, provided the function is well-behaved.

 -- Minimizer: gsl_min_fminimizer_goldensection

     The "golden section algorithm" is the simplest method of bracketing
     the minimum of a function.  It is the slowest algorithm provided by
     the library, with linear convergence.

     On each iteration, the algorithm first compares the subintervals
     from the endpoints to the current minimum.  The larger subinterval
     is divided in a golden section (using the famous ratio (3-\sqrt
     5)/2 = 0.3189660...) and the value of the function at this new
     point is calculated.  The new value is used with the constraint
     f(a') > f(x') < f(b') to a select new interval containing the
     minimum, by discarding the least useful point.  This procedure can
     be continued indefinitely until the interval is sufficiently small.
     Choosing the golden section as the bisection ratio can be shown to
     provide the fastest convergence for this type of algorithm.

 -- Minimizer: gsl_min_fminimizer_brent

     The "Brent minimization algorithm" combines a parabolic
     interpolation with the golden section algorithm.  This produces a
     fast algorithm which is still robust.

     The outline of the algorithm can be summarized as follows: on each
     iteration Brent's method approximates the function using an
     interpolating parabola through three existing points.  The minimum
     of the parabola is taken as a guess for the minimum.  If it lies
     within the bounds of the current interval then the interpolating
     point is accepted, and used to generate a smaller interval.  If the
     interpolating point is not accepted then the algorithm falls back
     to an ordinary golden section step.  The full details of Brent's
     method include some additional checks to improve convergence.

 -- Minimizer: gsl_min_fminimizer_quad_golden
     This is a variant of Brent's algorithm which uses the safeguarded
     step-length algorithm of Gill and Murray.


File: gsl-ref.info,  Node: Minimization Examples,  Next: Minimization References and Further Reading,  Prev: Minimization Algorithms,  Up: One dimensional Minimization

35.8 Examples
=============

The following program uses the Brent algorithm to find the minimum of
the function f(x) = \cos(x) + 1, which occurs at x = \pi.  The starting
interval is (0,6), with an initial guess for the minimum of 2.

     #include <stdio.h>
     #include <gsl/gsl_errno.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_min.h>

     double fn1 (double x, void * params)
     {
       (void)(params); /* avoid unused parameter warning */
       return cos(x) + 1.0;
     }

     int
     main (void)
     {
       int status;
       int iter = 0, max_iter = 100;
       const gsl_min_fminimizer_type *T;
       gsl_min_fminimizer *s;
       double m = 2.0, m_expected = M_PI;
       double a = 0.0, b = 6.0;
       gsl_function F;

       F.function = &fn1;
       F.params = 0;

       T = gsl_min_fminimizer_brent;
       s = gsl_min_fminimizer_alloc (T);
       gsl_min_fminimizer_set (s, &F, m, a, b);

       printf ("using %s method\n",
               gsl_min_fminimizer_name (s));

       printf ("%5s [%9s, %9s] %9s %10s %9s\n",
               "iter", "lower", "upper", "min",
               "err", "err(est)");

       printf ("%5d [%.7f, %.7f] %.7f %+.7f %.7f\n",
               iter, a, b,
               m, m - m_expected, b - a);

       do
         {
           iter++;
           status = gsl_min_fminimizer_iterate (s);

           m = gsl_min_fminimizer_x_minimum (s);
           a = gsl_min_fminimizer_x_lower (s);
           b = gsl_min_fminimizer_x_upper (s);

           status
             = gsl_min_test_interval (a, b, 0.001, 0.0);

           if (status == GSL_SUCCESS)
             printf ("Converged:\n");

           printf ("%5d [%.7f, %.7f] "
                   "%.7f %+.7f %.7f\n",
                   iter, a, b,
                   m, m - m_expected, b - a);
         }
       while (status == GSL_CONTINUE && iter < max_iter);

       gsl_min_fminimizer_free (s);

       return status;
     }

Here are the results of the minimization procedure.

     $ ./a.out
     using brent method
      iter [    lower,     upper]       min        err  err(est)
         0 [0.0000000, 6.0000000] 2.0000000 -1.1415927 6.0000000
         1 [2.0000000, 6.0000000] 3.5278640 +0.3862713 4.0000000
         2 [2.0000000, 3.5278640] 3.1748217 +0.0332290 1.5278640
         3 [2.0000000, 3.1748217] 3.1264576 -0.0151351 1.1748217
         4 [3.1264576, 3.1748217] 3.1414743 -0.0001183 0.0483641
         5 [3.1414743, 3.1748217] 3.1415930 +0.0000004 0.0333474
     Converged:
         6 [3.1414743, 3.1415930] 3.1415927 +0.0000000 0.0001187


File: gsl-ref.info,  Node: Minimization References and Further Reading,  Prev: Minimization Examples,  Up: One dimensional Minimization

35.9 References and Further Reading
===================================

Further information on Brent's algorithm is available in the following
book,

     Richard Brent, 'Algorithms for minimization without derivatives',
     Prentice-Hall (1973), republished by Dover in paperback (2002),
     ISBN 0-486-41998-3.


File: gsl-ref.info,  Node: Multidimensional Root-Finding,  Next: Multidimensional Minimization,  Prev: One dimensional Minimization,  Up: Top

36 Multidimensional Root-Finding
********************************

This chapter describes functions for multidimensional root-finding
(solving nonlinear systems with n equations in n unknowns).  The library
provides low level components for a variety of iterative solvers and
convergence tests.  These can be combined by the user to achieve the
desired solution, with full access to the intermediate steps of the
iteration.  Each class of methods uses the same framework, so that you
can switch between solvers at runtime without needing to recompile your
program.  Each instance of a solver keeps track of its own state,
allowing the solvers to be used in multi-threaded programs.  The solvers
are based on the original Fortran library MINPACK.

   The header file 'gsl_multiroots.h' contains prototypes for the
multidimensional root finding functions and related declarations.

* Menu:

* Overview of Multidimensional Root Finding::  
* Initializing the Multidimensional Solver::  
* Providing the multidimensional system of equations to solve::  
* Iteration of the multidimensional solver::  
* Search Stopping Parameters for the multidimensional solver::  
* Algorithms using Derivatives::  
* Algorithms without Derivatives::  
* Example programs for Multidimensional Root finding::  
* References and Further Reading for Multidimensional Root Finding::  


File: gsl-ref.info,  Node: Overview of Multidimensional Root Finding,  Next: Initializing the Multidimensional Solver,  Up: Multidimensional Root-Finding

36.1 Overview
=============

The problem of multidimensional root finding requires the simultaneous
solution of n equations, f_i, in n variables, x_i,

     f_i (x_1, ..., x_n) = 0    for i = 1 ... n.

In general there are no bracketing methods available for n dimensional
systems, and no way of knowing whether any solutions exist.  All
algorithms proceed from an initial guess using a variant of the Newton
iteration,

     x -> x' = x - J^{-1} f(x)

where x, f are vector quantities and J is the Jacobian matrix J_{ij} = d
f_i / d x_j.  Additional strategies can be used to enlarge the region of
convergence.  These include requiring a decrease in the norm |f| on each
step proposed by Newton's method, or taking steepest-descent steps in
the direction of the negative gradient of |f|.

   Several root-finding algorithms are available within a single
framework.  The user provides a high-level driver for the algorithms,
and the library provides the individual functions necessary for each of
the steps.  There are three main phases of the iteration.  The steps
are,

   * initialize solver state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

The evaluation of the Jacobian matrix can be problematic, either because
programming the derivatives is intractable or because computation of the
n^2 terms of the matrix becomes too expensive.  For these reasons the
algorithms provided by the library are divided into two classes
according to whether the derivatives are available or not.

   The state for solvers with an analytic Jacobian matrix is held in a
'gsl_multiroot_fdfsolver' struct.  The updating procedure requires both
the function and its derivatives to be supplied by the user.

   The state for solvers which do not use an analytic Jacobian matrix is
held in a 'gsl_multiroot_fsolver' struct.  The updating procedure uses
only function evaluations (not derivatives).  The algorithms estimate
the matrix J or J^{-1} by approximate methods.


File: gsl-ref.info,  Node: Initializing the Multidimensional Solver,  Next: Providing the multidimensional system of equations to solve,  Prev: Overview of Multidimensional Root Finding,  Up: Multidimensional Root-Finding

36.2 Initializing the Solver
============================

The following functions initialize a multidimensional solver, either
with or without derivatives.  The solver itself depends only on the
dimension of the problem and the algorithm and can be reused for
different problems.

 -- Function: gsl_multiroot_fsolver * gsl_multiroot_fsolver_alloc (const
          gsl_multiroot_fsolver_type * T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     solver of type T for a system of N dimensions.  For example, the
     following code creates an instance of a hybrid solver, to solve a
     3-dimensional system of equations.

          const gsl_multiroot_fsolver_type * T
              = gsl_multiroot_fsolver_hybrid;
          gsl_multiroot_fsolver * s
              = gsl_multiroot_fsolver_alloc (T, 3);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_multiroot_fdfsolver * gsl_multiroot_fdfsolver_alloc
          (const gsl_multiroot_fdfsolver_type * T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     derivative solver of type T for a system of N dimensions.  For
     example, the following code creates an instance of a Newton-Raphson
     solver, for a 2-dimensional system of equations.

          const gsl_multiroot_fdfsolver_type * T
              = gsl_multiroot_fdfsolver_newton;
          gsl_multiroot_fdfsolver * s =
              gsl_multiroot_fdfsolver_alloc (T, 2);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: int gsl_multiroot_fsolver_set (gsl_multiroot_fsolver * S,
          gsl_multiroot_function * F, const gsl_vector * X)
 -- Function: int gsl_multiroot_fdfsolver_set (gsl_multiroot_fdfsolver *
          S, gsl_multiroot_function_fdf * FDF, const gsl_vector * X)
     These functions set, or reset, an existing solver S to use the
     function F or function and derivative FDF, and the initial guess X.
     Note that the initial position is copied from X, this argument is
     not modified by subsequent iterations.

 -- Function: void gsl_multiroot_fsolver_free (gsl_multiroot_fsolver *
          S)
 -- Function: void gsl_multiroot_fdfsolver_free (gsl_multiroot_fdfsolver
          * S)
     These functions free all the memory associated with the solver S.

 -- Function: const char * gsl_multiroot_fsolver_name (const
          gsl_multiroot_fsolver * S)
 -- Function: const char * gsl_multiroot_fdfsolver_name (const
          gsl_multiroot_fdfsolver * S)
     These functions return a pointer to the name of the solver.  For
     example,

          printf ("s is a '%s' solver\n",
                  gsl_multiroot_fdfsolver_name (s));

     would print something like 's is a 'newton' solver'.


File: gsl-ref.info,  Node: Providing the multidimensional system of equations to solve,  Next: Iteration of the multidimensional solver,  Prev: Initializing the Multidimensional Solver,  Up: Multidimensional Root-Finding

36.3 Providing the function to solve
====================================

You must provide n functions of n variables for the root finders to
operate on.  In order to allow for general parameters the functions are
defined by the following data types:

 -- Data Type: gsl_multiroot_function
     This data type defines a general system of functions with
     parameters.

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          this function should store the vector result f(x,params) in F
          for argument X and parameters PARAMS, returning an appropriate
          error code if the function cannot be computed.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X and F.

     'void * params'
          a pointer to the parameters of the function.

Here is an example using Powell's test function,

     f_1(x) = A x_0 x_1 - 1,
     f_2(x) = exp(-x_0) + exp(-x_1) - (1 + 1/A)

with A = 10^4.  The following code defines a 'gsl_multiroot_function'
system 'F' which you could pass to a solver:

     struct powell_params { double A; };

     int
     powell (gsl_vector * x, void * p, gsl_vector * f) {
        struct powell_params * params
          = (struct powell_params *)p;
        const double A = (params->A);
        const double x0 = gsl_vector_get(x,0);
        const double x1 = gsl_vector_get(x,1);

        gsl_vector_set (f, 0, A * x0 * x1 - 1);
        gsl_vector_set (f, 1, (exp(-x0) + exp(-x1)
                               - (1.0 + 1.0/A)));
        return GSL_SUCCESS
     }

     gsl_multiroot_function F;
     struct powell_params params = { 10000.0 };

     F.f = &powell;
     F.n = 2;
     F.params = &params;

 -- Data Type: gsl_multiroot_function_fdf
     This data type defines a general system of functions with
     parameters and the corresponding Jacobian matrix of derivatives,

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          this function should store the vector result f(x,params) in F
          for argument X and parameters PARAMS, returning an appropriate
          error code if the function cannot be computed.

     'int (* df) (const gsl_vector * X, void * PARAMS, gsl_matrix * J)'
          this function should store the N-by-N matrix result J_ij = d
          f_i(x,params) / d x_j in J for argument X and parameters
          PARAMS, returning an appropriate error code if the function
          cannot be computed.

     'int (* fdf) (const gsl_vector * X, void * PARAMS, gsl_vector * F, gsl_matrix * J)'
          This function should set the values of the F and J as above,
          for arguments X and parameters PARAMS.  This function provides
          an optimization of the separate functions for f(x) and
          J(x)--it is always faster to compute the function and its
          derivative at the same time.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X and F.

     'void * params'
          a pointer to the parameters of the function.

The example of Powell's test function defined above can be extended to
include analytic derivatives using the following code,

     int
     powell_df (gsl_vector * x, void * p, gsl_matrix * J)
     {
        struct powell_params * params
          = (struct powell_params *)p;
        const double A = (params->A);
        const double x0 = gsl_vector_get(x,0);
        const double x1 = gsl_vector_get(x,1);
        gsl_matrix_set (J, 0, 0, A * x1);
        gsl_matrix_set (J, 0, 1, A * x0);
        gsl_matrix_set (J, 1, 0, -exp(-x0));
        gsl_matrix_set (J, 1, 1, -exp(-x1));
        return GSL_SUCCESS
     }

     int
     powell_fdf (gsl_vector * x, void * p,
                 gsl_matrix * f, gsl_matrix * J) {
        struct powell_params * params
          = (struct powell_params *)p;
        const double A = (params->A);
        const double x0 = gsl_vector_get(x,0);
        const double x1 = gsl_vector_get(x,1);

        const double u0 = exp(-x0);
        const double u1 = exp(-x1);

        gsl_vector_set (f, 0, A * x0 * x1 - 1);
        gsl_vector_set (f, 1, u0 + u1 - (1 + 1/A));

        gsl_matrix_set (J, 0, 0, A * x1);
        gsl_matrix_set (J, 0, 1, A * x0);
        gsl_matrix_set (J, 1, 0, -u0);
        gsl_matrix_set (J, 1, 1, -u1);
        return GSL_SUCCESS
     }

     gsl_multiroot_function_fdf FDF;

     FDF.f = &powell_f;
     FDF.df = &powell_df;
     FDF.fdf = &powell_fdf;
     FDF.n = 2;
     FDF.params = 0;

Note that the function 'powell_fdf' is able to reuse existing terms from
the function when calculating the Jacobian, thus saving time.


File: gsl-ref.info,  Node: Iteration of the multidimensional solver,  Next: Search Stopping Parameters for the multidimensional solver,  Prev: Providing the multidimensional system of equations to solve,  Up: Multidimensional Root-Finding

36.4 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any solver of the
corresponding type.  The same functions work for all solvers so that
different methods can be substituted at runtime without modifications to
the code.

 -- Function: int gsl_multiroot_fsolver_iterate (gsl_multiroot_fsolver *
          S)
 -- Function: int gsl_multiroot_fdfsolver_iterate
          (gsl_multiroot_fdfsolver * S)
     These functions perform a single iteration of the solver S.  If the
     iteration encounters an unexpected problem then an error code will
     be returned,

     'GSL_EBADFUNC'
          the iteration encountered a singular point where the function
          or its derivative evaluated to 'Inf' or 'NaN'.

     'GSL_ENOPROG'
          the iteration is not making any progress, preventing the
          algorithm from continuing.

   The solver maintains a current best estimate of the root 's->x' and
its function value 's->f' at all times.  This information can be
accessed with the following auxiliary functions,

 -- Function: gsl_vector * gsl_multiroot_fsolver_root (const
          gsl_multiroot_fsolver * S)
 -- Function: gsl_vector * gsl_multiroot_fdfsolver_root (const
          gsl_multiroot_fdfsolver * S)
     These functions return the current estimate of the root for the
     solver S, given by 's->x'.

 -- Function: gsl_vector * gsl_multiroot_fsolver_f (const
          gsl_multiroot_fsolver * S)
 -- Function: gsl_vector * gsl_multiroot_fdfsolver_f (const
          gsl_multiroot_fdfsolver * S)
     These functions return the function value f(x) at the current
     estimate of the root for the solver S, given by 's->f'.

 -- Function: gsl_vector * gsl_multiroot_fsolver_dx (const
          gsl_multiroot_fsolver * S)
 -- Function: gsl_vector * gsl_multiroot_fdfsolver_dx (const
          gsl_multiroot_fdfsolver * S)
     These functions return the last step dx taken by the solver S,
     given by 's->dx'.


File: gsl-ref.info,  Node: Search Stopping Parameters for the multidimensional solver,  Next: Algorithms using Derivatives,  Prev: Iteration of the multidimensional solver,  Up: Multidimensional Root-Finding

36.5 Search Stopping Parameters
===============================

A root finding procedure should stop when one of the following
conditions is true:

   * A multidimensional root has been found to within the user-specified
     precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result in
several standard ways.

 -- Function: int gsl_multiroot_test_delta (const gsl_vector * DX, const
          gsl_vector * X, double EPSABS, double EPSREL)

     This function tests for the convergence of the sequence by
     comparing the last step DX with the absolute error EPSABS and
     relative error EPSREL to the current position X.  The test returns
     'GSL_SUCCESS' if the following condition is achieved,

          |dx_i| < epsabs + epsrel |x_i|

     for each component of X and returns 'GSL_CONTINUE' otherwise.

 -- Function: int gsl_multiroot_test_residual (const gsl_vector * F,
          double EPSABS)
     This function tests the residual value F against the absolute error
     bound EPSABS.  The test returns 'GSL_SUCCESS' if the following
     condition is achieved,

          \sum_i |f_i| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  This criterion is suitable
     for situations where the precise location of the root, x, is
     unimportant provided a value can be found where the residual is
     small enough.


File: gsl-ref.info,  Node: Algorithms using Derivatives,  Next: Algorithms without Derivatives,  Prev: Search Stopping Parameters for the multidimensional solver,  Up: Multidimensional Root-Finding

36.6 Algorithms using Derivatives
=================================

The root finding algorithms described in this section make use of both
the function and its derivative.  They require an initial guess for the
location of the root, but there is no absolute guarantee of
convergence--the function must be suitable for this technique and the
initial guess must be sufficiently close to the root for it to work.
When the conditions are satisfied then convergence is quadratic.

 -- Derivative Solver: gsl_multiroot_fdfsolver_hybridsj
     This is a modified version of Powell's Hybrid method as implemented
     in the HYBRJ algorithm in MINPACK.  Minpack was written by Jorge J.
     More', Burton S. Garbow and Kenneth E. Hillstrom.  The Hybrid
     algorithm retains the fast convergence of Newton's method but will
     also reduce the residual when Newton's method is unreliable.

     The algorithm uses a generalized trust region to keep each step
     under control.  In order to be accepted a proposed new position x'
     must satisfy the condition |D (x' - x)| < \delta, where D is a
     diagonal scaling matrix and \delta is the size of the trust region.
     The components of D are computed internally, using the column norms
     of the Jacobian to estimate the sensitivity of the residual to each
     component of x.  This improves the behavior of the algorithm for
     badly scaled functions.

     On each iteration the algorithm first determines the standard
     Newton step by solving the system J dx = - f.  If this step falls
     inside the trust region it is used as a trial step in the next
     stage.  If not, the algorithm uses the linear combination of the
     Newton and gradient directions which is predicted to minimize the
     norm of the function while staying inside the trust region,

          dx = - \alpha J^{-1} f(x) - \beta \nabla |f(x)|^2.

     This combination of Newton and gradient directions is referred to
     as a "dogleg step".

     The proposed step is now tested by evaluating the function at the
     resulting point, x'.  If the step reduces the norm of the function
     sufficiently then it is accepted and size of the trust region is
     increased.  If the proposed step fails to improve the solution then
     the size of the trust region is decreased and another trial step is
     computed.

     The speed of the algorithm is increased by computing the changes to
     the Jacobian approximately, using a rank-1 update.  If two
     successive attempts fail to reduce the residual then the full
     Jacobian is recomputed.  The algorithm also monitors the progress
     of the solution and returns an error if several steps fail to make
     any improvement,

     'GSL_ENOPROG'
          the iteration is not making any progress, preventing the
          algorithm from continuing.

     'GSL_ENOPROGJ'
          re-evaluations of the Jacobian indicate that the iteration is
          not making any progress, preventing the algorithm from
          continuing.

 -- Derivative Solver: gsl_multiroot_fdfsolver_hybridj
     This algorithm is an unscaled version of 'hybridsj'.  The steps are
     controlled by a spherical trust region |x' - x| < \delta, instead
     of a generalized region.  This can be useful if the generalized
     region estimated by 'hybridsj' is inappropriate.

 -- Derivative Solver: gsl_multiroot_fdfsolver_newton

     Newton's Method is the standard root-polishing algorithm.  The
     algorithm begins with an initial guess for the location of the
     solution.  On each iteration a linear approximation to the function
     F is used to estimate the step which will zero all the components
     of the residual.  The iteration is defined by the following
     sequence,

          x -> x' = x - J^{-1} f(x)

     where the Jacobian matrix J is computed from the derivative
     functions provided by F.  The step dx is obtained by solving the
     linear system,

          J dx = - f(x)

     using LU decomposition.  If the Jacobian matrix is singular, an
     error code of 'GSL_EDOM' is returned.

 -- Derivative Solver: gsl_multiroot_fdfsolver_gnewton
     This is a modified version of Newton's method which attempts to
     improve global convergence by requiring every step to reduce the
     Euclidean norm of the residual, |f(x)|.  If the Newton step leads
     to an increase in the norm then a reduced step of relative size,

          t = (\sqrt(1 + 6 r) - 1) / (3 r)

     is proposed, with r being the ratio of norms |f(x')|^2/|f(x)|^2.
     This procedure is repeated until a suitable step size is found.


File: gsl-ref.info,  Node: Algorithms without Derivatives,  Next: Example programs for Multidimensional Root finding,  Prev: Algorithms using Derivatives,  Up: Multidimensional Root-Finding

36.7 Algorithms without Derivatives
===================================

The algorithms described in this section do not require any derivative
information to be supplied by the user.  Any derivatives needed are
approximated by finite differences.  Note that if the
finite-differencing step size chosen by these routines is inappropriate,
an explicit user-supplied numerical derivative can always be used with
the algorithms described in the previous section.

 -- Solver: gsl_multiroot_fsolver_hybrids
     This is a version of the Hybrid algorithm which replaces calls to
     the Jacobian function by its finite difference approximation.  The
     finite difference approximation is computed using
     'gsl_multiroots_fdjac' with a relative step size of
     'GSL_SQRT_DBL_EPSILON'.  Note that this step size will not be
     suitable for all problems.

 -- Solver: gsl_multiroot_fsolver_hybrid
     This is a finite difference version of the Hybrid algorithm without
     internal scaling.

 -- Solver: gsl_multiroot_fsolver_dnewton

     The "discrete Newton algorithm" is the simplest method of solving a
     multidimensional system.  It uses the Newton iteration

          x -> x - J^{-1} f(x)

     where the Jacobian matrix J is approximated by taking finite
     differences of the function F.  The approximation scheme used by
     this implementation is,

          J_{ij} = (f_i(x + \delta_j) - f_i(x)) /  \delta_j

     where \delta_j is a step of size \sqrt\epsilon |x_j| with \epsilon
     being the machine precision (\epsilon \approx 2.22 \times 10^-16).
     The order of convergence of Newton's algorithm is quadratic, but
     the finite differences require n^2 function evaluations on each
     iteration.  The algorithm may become unstable if the finite
     differences are not a good approximation to the true derivatives.

 -- Solver: gsl_multiroot_fsolver_broyden

     The "Broyden algorithm" is a version of the discrete Newton
     algorithm which attempts to avoids the expensive update of the
     Jacobian matrix on each iteration.  The changes to the Jacobian are
     also approximated, using a rank-1 update,

          J^{-1} \to J^{-1} - (J^{-1} df - dx) dx^T J^{-1} / dx^T J^{-1} df

     where the vectors dx and df are the changes in x and f.  On the
     first iteration the inverse Jacobian is estimated using finite
     differences, as in the discrete Newton algorithm.

     This approximation gives a fast update but is unreliable if the
     changes are not small, and the estimate of the inverse Jacobian
     becomes worse as time passes.  The algorithm has a tendency to
     become unstable unless it starts close to the root.  The Jacobian
     is refreshed if this instability is detected (consult the source
     for details).

     This algorithm is included only for demonstration purposes, and is
     not recommended for serious use.


File: gsl-ref.info,  Node: Example programs for Multidimensional Root finding,  Next: References and Further Reading for Multidimensional Root Finding,  Prev: Algorithms without Derivatives,  Up: Multidimensional Root-Finding

36.8 Examples
=============

The multidimensional solvers are used in a similar way to the
one-dimensional root finding algorithms.  This first example
demonstrates the 'hybrids' scaled-hybrid algorithm, which does not
require derivatives.  The program solves the Rosenbrock system of
equations,

     f_1 (x, y) = a (1 - x)
     f_2 (x, y) = b (y - x^2)

with a = 1, b = 10.  The solution of this system lies at (x,y) = (1,1)
in a narrow valley.

   The first stage of the program is to define the system of equations,

     #include <stdlib.h>
     #include <stdio.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_multiroots.h>

     struct rparams
       {
         double a;
         double b;
       };

     int
     rosenbrock_f (const gsl_vector * x, void *params,
                   gsl_vector * f)
     {
       double a = ((struct rparams *) params)->a;
       double b = ((struct rparams *) params)->b;

       const double x0 = gsl_vector_get (x, 0);
       const double x1 = gsl_vector_get (x, 1);

       const double y0 = a * (1 - x0);
       const double y1 = b * (x1 - x0 * x0);

       gsl_vector_set (f, 0, y0);
       gsl_vector_set (f, 1, y1);

       return GSL_SUCCESS;
     }

The main program begins by creating the function object 'f', with the
arguments '(x,y)' and parameters '(a,b)'.  The solver 's' is initialized
to use this function, with the 'hybrids' method.

     int
     main (void)
     {
       const gsl_multiroot_fsolver_type *T;
       gsl_multiroot_fsolver *s;

       int status;
       size_t i, iter = 0;

       const size_t n = 2;
       struct rparams p = {1.0, 10.0};
       gsl_multiroot_function f = {&rosenbrock_f, n, &p};

       double x_init[2] = {-10.0, -5.0};
       gsl_vector *x = gsl_vector_alloc (n);

       gsl_vector_set (x, 0, x_init[0]);
       gsl_vector_set (x, 1, x_init[1]);

       T = gsl_multiroot_fsolver_hybrids;
       s = gsl_multiroot_fsolver_alloc (T, 2);
       gsl_multiroot_fsolver_set (s, &f, x);

       print_state (iter, s);

       do
         {
           iter++;
           status = gsl_multiroot_fsolver_iterate (s);

           print_state (iter, s);

           if (status)   /* check if solver is stuck */
             break;

           status =
             gsl_multiroot_test_residual (s->f, 1e-7);
         }
       while (status == GSL_CONTINUE && iter < 1000);

       printf ("status = %s\n", gsl_strerror (status));

       gsl_multiroot_fsolver_free (s);
       gsl_vector_free (x);
       return 0;
     }

Note that it is important to check the return status of each solver
step, in case the algorithm becomes stuck.  If an error condition is
detected, indicating that the algorithm cannot proceed, then the error
can be reported to the user, a new starting point chosen or a different
algorithm used.

   The intermediate state of the solution is displayed by the following
function.  The solver state contains the vector 's->x' which is the
current position, and the vector 's->f' with corresponding function
values.

     int
     print_state (size_t iter, gsl_multiroot_fsolver * s)
     {
       printf ("iter = %3u x = % .3f % .3f "
               "f(x) = % .3e % .3e\n",
               iter,
               gsl_vector_get (s->x, 0),
               gsl_vector_get (s->x, 1),
               gsl_vector_get (s->f, 0),
               gsl_vector_get (s->f, 1));
     }

Here are the results of running the program.  The algorithm is started
at (-10,-5) far from the solution.  Since the solution is hidden in a
narrow valley the earliest steps follow the gradient of the function
downhill, in an attempt to reduce the large value of the residual.  Once
the root has been approximately located, on iteration 8, the Newton
behavior takes over and convergence is very rapid.

     iter =  0 x = -10.000  -5.000  f(x) = 1.100e+01 -1.050e+03
     iter =  1 x = -10.000  -5.000  f(x) = 1.100e+01 -1.050e+03
     iter =  2 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  3 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  4 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  5 x =  -1.274  -5.680  f(x) = 2.274e+00 -7.302e+01
     iter =  6 x =  -1.274  -5.680  f(x) = 2.274e+00 -7.302e+01
     iter =  7 x =   0.249   0.298  f(x) = 7.511e-01  2.359e+00
     iter =  8 x =   0.249   0.298  f(x) = 7.511e-01  2.359e+00
     iter =  9 x =   1.000   0.878  f(x) = 1.268e-10 -1.218e+00
     iter = 10 x =   1.000   0.989  f(x) = 1.124e-11 -1.080e-01
     iter = 11 x =   1.000   1.000  f(x) = 0.000e+00  0.000e+00
     status = success

Note that the algorithm does not update the location on every iteration.
Some iterations are used to adjust the trust-region parameter, after
trying a step which was found to be divergent, or to recompute the
Jacobian, when poor convergence behavior is detected.

   The next example program adds derivative information, in order to
accelerate the solution.  There are two derivative functions
'rosenbrock_df' and 'rosenbrock_fdf'.  The latter computes both the
function and its derivative simultaneously.  This allows the
optimization of any common terms.  For simplicity we substitute calls to
the separate 'f' and 'df' functions at this point in the code below.

     int
     rosenbrock_df (const gsl_vector * x, void *params,
                    gsl_matrix * J)
     {
       const double a = ((struct rparams *) params)->a;
       const double b = ((struct rparams *) params)->b;

       const double x0 = gsl_vector_get (x, 0);

       const double df00 = -a;
       const double df01 = 0;
       const double df10 = -2 * b  * x0;
       const double df11 = b;

       gsl_matrix_set (J, 0, 0, df00);
       gsl_matrix_set (J, 0, 1, df01);
       gsl_matrix_set (J, 1, 0, df10);
       gsl_matrix_set (J, 1, 1, df11);

       return GSL_SUCCESS;
     }

     int
     rosenbrock_fdf (const gsl_vector * x, void *params,
                     gsl_vector * f, gsl_matrix * J)
     {
       rosenbrock_f (x, params, f);
       rosenbrock_df (x, params, J);

       return GSL_SUCCESS;
     }

The main program now makes calls to the corresponding 'fdfsolver'
versions of the functions,

     int
     main (void)
     {
       const gsl_multiroot_fdfsolver_type *T;
       gsl_multiroot_fdfsolver *s;

       int status;
       size_t i, iter = 0;

       const size_t n = 2;
       struct rparams p = {1.0, 10.0};
       gsl_multiroot_function_fdf f = {&rosenbrock_f,
                                       &rosenbrock_df,
                                       &rosenbrock_fdf,
                                       n, &p};

       double x_init[2] = {-10.0, -5.0};
       gsl_vector *x = gsl_vector_alloc (n);

       gsl_vector_set (x, 0, x_init[0]);
       gsl_vector_set (x, 1, x_init[1]);

       T = gsl_multiroot_fdfsolver_gnewton;
       s = gsl_multiroot_fdfsolver_alloc (T, n);
       gsl_multiroot_fdfsolver_set (s, &f, x);

       print_state (iter, s);

       do
         {
           iter++;

           status = gsl_multiroot_fdfsolver_iterate (s);

           print_state (iter, s);

           if (status)
             break;

           status = gsl_multiroot_test_residual (s->f, 1e-7);
         }
       while (status == GSL_CONTINUE && iter < 1000);

       printf ("status = %s\n", gsl_strerror (status));

       gsl_multiroot_fdfsolver_free (s);
       gsl_vector_free (x);
       return 0;
     }

The addition of derivative information to the 'hybrids' solver does not
make any significant difference to its behavior, since it able to
approximate the Jacobian numerically with sufficient accuracy.  To
illustrate the behavior of a different derivative solver we switch to
'gnewton'.  This is a traditional Newton solver with the constraint that
it scales back its step if the full step would lead "uphill".  Here is
the output for the 'gnewton' algorithm,

     iter = 0 x = -10.000  -5.000 f(x) =  1.100e+01 -1.050e+03
     iter = 1 x =  -4.231 -65.317 f(x) =  5.231e+00 -8.321e+02
     iter = 2 x =   1.000 -26.358 f(x) = -8.882e-16 -2.736e+02
     iter = 3 x =   1.000   1.000 f(x) = -2.220e-16 -4.441e-15
     status = success

The convergence is much more rapid, but takes a wide excursion out to
the point (-4.23,-65.3).  This could cause the algorithm to go astray in
a realistic application.  The hybrid algorithm follows the downhill path
to the solution more reliably.


File: gsl-ref.info,  Node: References and Further Reading for Multidimensional Root Finding,  Prev: Example programs for Multidimensional Root finding,  Up: Multidimensional Root-Finding

36.9 References and Further Reading
===================================

The original version of the Hybrid method is described in the following
articles by Powell,

     M.J.D. Powell, "A Hybrid Method for Nonlinear Equations" (Chap 6, p
     87-114) and "A Fortran Subroutine for Solving systems of Nonlinear
     Algebraic Equations" (Chap 7, p 115-161), in 'Numerical Methods for
     Nonlinear Algebraic Equations', P. Rabinowitz, editor.  Gordon and
     Breach, 1970.

The following papers are also relevant to the algorithms described in
this section,

     J.J. More', M.Y. Cosnard, "Numerical Solution of Nonlinear
     Equations", 'ACM Transactions on Mathematical Software', Vol 5, No
     1, (1979), p 64-85

     C.G. Broyden, "A Class of Methods for Solving Nonlinear
     Simultaneous Equations", 'Mathematics of Computation', Vol 19
     (1965), p 577-593

     J.J. More', B.S. Garbow, K.E. Hillstrom, "Testing Unconstrained
     Optimization Software", ACM Transactions on Mathematical Software,
     Vol 7, No 1 (1981), p 17-41


File: gsl-ref.info,  Node: Multidimensional Minimization,  Next: Least-Squares Fitting,  Prev: Multidimensional Root-Finding,  Up: Top

37 Multidimensional Minimization
********************************

This chapter describes routines for finding minima of arbitrary
multidimensional functions.  The library provides low level components
for a variety of iterative minimizers and convergence tests.  These can
be combined by the user to achieve the desired solution, while providing
full access to the intermediate steps of the algorithms.  Each class of
methods uses the same framework, so that you can switch between
minimizers at runtime without needing to recompile your program.  Each
instance of a minimizer keeps track of its own state, allowing the
minimizers to be used in multi-threaded programs.  The minimization
algorithms can be used to maximize a function by inverting its sign.

   The header file 'gsl_multimin.h' contains prototypes for the
minimization functions and related declarations.

* Menu:

* Multimin Overview::           
* Multimin Caveats::            
* Initializing the Multidimensional Minimizer::  
* Providing a function to minimize::  
* Multimin Iteration::          
* Multimin Stopping Criteria::  
* Multimin Algorithms with Derivatives::  
* Multimin Algorithms without Derivatives::  
* Multimin Examples::           
* Multimin References and Further Reading::  


File: gsl-ref.info,  Node: Multimin Overview,  Next: Multimin Caveats,  Up: Multidimensional Minimization

37.1 Overview
=============

The problem of multidimensional minimization requires finding a point x
such that the scalar function,

     f(x_1, ..., x_n)

takes a value which is lower than at any neighboring point.  For smooth
functions the gradient g = \nabla f vanishes at the minimum.  In general
there are no bracketing methods available for the minimization of
n-dimensional functions.  The algorithms proceed from an initial guess
using a search algorithm which attempts to move in a downhill direction.

   Algorithms making use of the gradient of the function perform a
one-dimensional line minimisation along this direction until the lowest
point is found to a suitable tolerance.  The search direction is then
updated with local information from the function and its derivatives,
and the whole process repeated until the true n-dimensional minimum is
found.

   Algorithms which do not require the gradient of the function use
different strategies.  For example, the Nelder-Mead Simplex algorithm
maintains n+1 trial parameter vectors as the vertices of a n-dimensional
simplex.  On each iteration it tries to improve the worst vertex of the
simplex by geometrical transformations.  The iterations are continued
until the overall size of the simplex has decreased sufficiently.

   Both types of algorithms use a standard framework.  The user provides
a high-level driver for the algorithms, and the library provides the
individual functions necessary for each of the steps.  There are three
main phases of the iteration.  The steps are,

   * initialize minimizer state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

Each iteration step consists either of an improvement to the
line-minimisation in the current direction or an update to the search
direction itself.  The state for the minimizers is held in a
'gsl_multimin_fdfminimizer' struct or a 'gsl_multimin_fminimizer'
struct.


File: gsl-ref.info,  Node: Multimin Caveats,  Next: Initializing the Multidimensional Minimizer,  Prev: Multimin Overview,  Up: Multidimensional Minimization

37.2 Caveats
============

Note that the minimization algorithms can only search for one local
minimum at a time.  When there are several local minima in the search
area, the first minimum to be found will be returned; however it is
difficult to predict which of the minima this will be.  In most cases,
no error will be reported if you try to find a local minimum in an area
where there is more than one.

   It is also important to note that the minimization algorithms find
local minima; there is no way to determine whether a minimum is a global
minimum of the function in question.


File: gsl-ref.info,  Node: Initializing the Multidimensional Minimizer,  Next: Providing a function to minimize,  Prev: Multimin Caveats,  Up: Multidimensional Minimization

37.3 Initializing the Multidimensional Minimizer
================================================

The following function initializes a multidimensional minimizer.  The
minimizer itself depends only on the dimension of the problem and the
algorithm and can be reused for different problems.

 -- Function: gsl_multimin_fdfminimizer *
          gsl_multimin_fdfminimizer_alloc (const
          gsl_multimin_fdfminimizer_type * T, size_t N)
 -- Function: gsl_multimin_fminimizer * gsl_multimin_fminimizer_alloc
          (const gsl_multimin_fminimizer_type * T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     minimizer of type T for an N-dimension function.  If there is
     insufficient memory to create the minimizer then the function
     returns a null pointer and the error handler is invoked with an
     error code of 'GSL_ENOMEM'.

 -- Function: int gsl_multimin_fdfminimizer_set
          (gsl_multimin_fdfminimizer * S, gsl_multimin_function_fdf *
          FDF, const gsl_vector * X, double STEP_SIZE, double TOL)
 -- Function: int gsl_multimin_fminimizer_set (gsl_multimin_fminimizer *
          S, gsl_multimin_function * F, const gsl_vector * X, const
          gsl_vector * STEP_SIZE)
     The function 'gsl_multimin_fdfminimizer_set' initializes the
     minimizer S to minimize the function FDF starting from the initial
     point X.  The size of the first trial step is given by STEP_SIZE.
     The accuracy of the line minimization is specified by TOL.  The
     precise meaning of this parameter depends on the method used.
     Typically the line minimization is considered successful if the
     gradient of the function g is orthogonal to the current search
     direction p to a relative accuracy of TOL, where dot(p,g) < tol |p|
     |g|.  A TOL value of 0.1 is suitable for most purposes, since line
     minimization only needs to be carried out approximately.  Note that
     setting TOL to zero will force the use of "exact" line-searches,
     which are extremely expensive.

     The function 'gsl_multimin_fminimizer_set' initializes the
     minimizer S to minimize the function F, starting from the initial
     point X.  The size of the initial trial steps is given in vector
     STEP_SIZE.  The precise meaning of this parameter depends on the
     method used.

 -- Function: void gsl_multimin_fdfminimizer_free
          (gsl_multimin_fdfminimizer * S)
 -- Function: void gsl_multimin_fminimizer_free (gsl_multimin_fminimizer
          * S)
     This function frees all the memory associated with the minimizer S.

 -- Function: const char * gsl_multimin_fdfminimizer_name (const
          gsl_multimin_fdfminimizer * S)
 -- Function: const char * gsl_multimin_fminimizer_name (const
          gsl_multimin_fminimizer * S)
     This function returns a pointer to the name of the minimizer.  For
     example,

          printf ("s is a '%s' minimizer\n",
                  gsl_multimin_fdfminimizer_name (s));

     would print something like 's is a 'conjugate_pr' minimizer'.


File: gsl-ref.info,  Node: Providing a function to minimize,  Next: Multimin Iteration,  Prev: Initializing the Multidimensional Minimizer,  Up: Multidimensional Minimization

37.4 Providing a function to minimize
=====================================

You must provide a parametric function of n variables for the minimizers
to operate on.  You may also need to provide a routine which calculates
the gradient of the function and a third routine which calculates both
the function value and the gradient together.  In order to allow for
general parameters the functions are defined by the following data
types:

 -- Data Type: gsl_multimin_function_fdf
     This data type defines a general function of n variables with
     parameters and the corresponding gradient vector of derivatives,

     'double (* f) (const gsl_vector * X, void * PARAMS)'
          this function should return the result f(x,params) for
          argument X and parameters PARAMS.  If the function cannot be
          computed, an error value of 'GSL_NAN' should be returned.

     'void (* df) (const gsl_vector * X, void * PARAMS, gsl_vector * G)'
          this function should store the N-dimensional gradient g_i = d
          f(x,params) / d x_i in the vector G for argument X and
          parameters PARAMS, returning an appropriate error code if the
          function cannot be computed.

     'void (* fdf) (const gsl_vector * X, void * PARAMS, double * f, gsl_vector * G)'
          This function should set the values of the F and G as above,
          for arguments X and parameters PARAMS.  This function provides
          an optimization of the separate functions for f(x) and
          g(x)--it is always faster to compute the function and its
          derivative at the same time.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X.

     'void * params'
          a pointer to the parameters of the function.
 -- Data Type: gsl_multimin_function
     This data type defines a general function of n variables with
     parameters,

     'double (* f) (const gsl_vector * X, void * PARAMS)'
          this function should return the result f(x,params) for
          argument X and parameters PARAMS.  If the function cannot be
          computed, an error value of 'GSL_NAN' should be returned.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X.

     'void * params'
          a pointer to the parameters of the function.

The following example function defines a simple two-dimensional
paraboloid with five parameters,

     /* Paraboloid centered on (p[0],p[1]), with
        scale factors (p[2],p[3]) and minimum p[4] */

     double
     my_f (const gsl_vector *v, void *params)
     {
       double x, y;
       double *p = (double *)params;

       x = gsl_vector_get(v, 0);
       y = gsl_vector_get(v, 1);

       return p[2] * (x - p[0]) * (x - p[0]) +
                p[3] * (y - p[1]) * (y - p[1]) + p[4];
     }

     /* The gradient of f, df = (df/dx, df/dy). */
     void
     my_df (const gsl_vector *v, void *params,
            gsl_vector *df)
     {
       double x, y;
       double *p = (double *)params;

       x = gsl_vector_get(v, 0);
       y = gsl_vector_get(v, 1);

       gsl_vector_set(df, 0, 2.0 * p[2] * (x - p[0]));
       gsl_vector_set(df, 1, 2.0 * p[3] * (y - p[1]));
     }

     /* Compute both f and df together. */
     void
     my_fdf (const gsl_vector *x, void *params,
             double *f, gsl_vector *df)
     {
       *f = my_f(x, params);
       my_df(x, params, df);
     }

The function can be initialized using the following code,

     gsl_multimin_function_fdf my_func;

     /* Paraboloid center at (1,2), scale factors (10, 20),
        minimum value 30 */
     double p[5] = { 1.0, 2.0, 10.0, 20.0, 30.0 };

     my_func.n = 2;  /* number of function components */
     my_func.f = &my_f;
     my_func.df = &my_df;
     my_func.fdf = &my_fdf;
     my_func.params = (void *)p;


File: gsl-ref.info,  Node: Multimin Iteration,  Next: Multimin Stopping Criteria,  Prev: Providing a function to minimize,  Up: Multidimensional Minimization

37.5 Iteration
==============

The following function drives the iteration of each algorithm.  The
function performs one iteration to update the state of the minimizer.
The same function works for all minimizers so that different methods can
be substituted at runtime without modifications to the code.

 -- Function: int gsl_multimin_fdfminimizer_iterate
          (gsl_multimin_fdfminimizer * S)
 -- Function: int gsl_multimin_fminimizer_iterate
          (gsl_multimin_fminimizer * S)
     These functions perform a single iteration of the minimizer S.  If
     the iteration encounters an unexpected problem then an error code
     will be returned.  The error code 'GSL_ENOPROG' signifies that the
     minimizer is unable to improve on its current estimate, either due
     to numerical difficulty or because a genuine local minimum has been
     reached.

The minimizer maintains a current best estimate of the minimum at all
times.  This information can be accessed with the following auxiliary
functions,

 -- Function: gsl_vector * gsl_multimin_fdfminimizer_x (const
          gsl_multimin_fdfminimizer * S)
 -- Function: gsl_vector * gsl_multimin_fminimizer_x (const
          gsl_multimin_fminimizer * S)
 -- Function: double gsl_multimin_fdfminimizer_minimum (const
          gsl_multimin_fdfminimizer * S)
 -- Function: double gsl_multimin_fminimizer_minimum (const
          gsl_multimin_fminimizer * S)
 -- Function: gsl_vector * gsl_multimin_fdfminimizer_gradient (const
          gsl_multimin_fdfminimizer * S)
 -- Function: double gsl_multimin_fminimizer_size (const
          gsl_multimin_fminimizer * S)
     These functions return the current best estimate of the location of
     the minimum, the value of the function at that point, its gradient,
     and minimizer specific characteristic size for the minimizer S.

 -- Function: int gsl_multimin_fdfminimizer_restart
          (gsl_multimin_fdfminimizer * S)
     This function resets the minimizer S to use the current point as a
     new starting point.


File: gsl-ref.info,  Node: Multimin Stopping Criteria,  Next: Multimin Algorithms with Derivatives,  Prev: Multimin Iteration,  Up: Multidimensional Minimization

37.6 Stopping Criteria
======================

A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result.

 -- Function: int gsl_multimin_test_gradient (const gsl_vector * G,
          double EPSABS)
     This function tests the norm of the gradient G against the absolute
     tolerance EPSABS.  The gradient of a multidimensional function goes
     to zero at a minimum.  The test returns 'GSL_SUCCESS' if the
     following condition is achieved,

          |g| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  A suitable choice of EPSABS
     can be made from the desired accuracy in the function for small
     variations in x.  The relationship between these quantities is
     given by \delta f = g \delta x.

 -- Function: int gsl_multimin_test_size (const double SIZE, double
          EPSABS)
     This function tests the minimizer specific characteristic size (if
     applicable to the used minimizer) against absolute tolerance
     EPSABS.  The test returns 'GSL_SUCCESS' if the size is smaller than
     tolerance, otherwise 'GSL_CONTINUE' is returned.


File: gsl-ref.info,  Node: Multimin Algorithms with Derivatives,  Next: Multimin Algorithms without Derivatives,  Prev: Multimin Stopping Criteria,  Up: Multidimensional Minimization

37.7 Algorithms with Derivatives
================================

There are several minimization methods available.  The best choice of
algorithm depends on the problem.  The algorithms described in this
section use the value of the function and its gradient at each
evaluation point.

 -- Minimizer: gsl_multimin_fdfminimizer_conjugate_fr
     This is the Fletcher-Reeves conjugate gradient algorithm.  The
     conjugate gradient algorithm proceeds as a succession of line
     minimizations.  The sequence of search directions is used to build
     up an approximation to the curvature of the function in the
     neighborhood of the minimum.

     An initial search direction P is chosen using the gradient, and
     line minimization is carried out in that direction.  The accuracy
     of the line minimization is specified by the parameter TOL.  The
     minimum along this line occurs when the function gradient G and the
     search direction P are orthogonal.  The line minimization
     terminates when dot(p,g) < tol |p| |g|.  The search direction is
     updated using the Fletcher-Reeves formula p' = g' - \beta g where
     \beta=-|g'|^2/|g|^2, and the line minimization is then repeated for
     the new search direction.

 -- Minimizer: gsl_multimin_fdfminimizer_conjugate_pr
     This is the Polak-Ribiere conjugate gradient algorithm.  It is
     similar to the Fletcher-Reeves method, differing only in the choice
     of the coefficient \beta.  Both methods work well when the
     evaluation point is close enough to the minimum of the objective
     function that it is well approximated by a quadratic hypersurface.

 -- Minimizer: gsl_multimin_fdfminimizer_vector_bfgs2
 -- Minimizer: gsl_multimin_fdfminimizer_vector_bfgs
     These methods use the vector Broyden-Fletcher-Goldfarb-Shanno
     (BFGS) algorithm.  This is a quasi-Newton method which builds up an
     approximation to the second derivatives of the function f using the
     difference between successive gradient vectors.  By combining the
     first and second derivatives the algorithm is able to take
     Newton-type steps towards the function minimum, assuming quadratic
     behavior in that region.

     The 'bfgs2' version of this minimizer is the most efficient version
     available, and is a faithful implementation of the line
     minimization scheme described in Fletcher's 'Practical Methods of
     Optimization', Algorithms 2.6.2 and 2.6.4.  It supersedes the
     original 'bfgs' routine and requires substantially fewer function
     and gradient evaluations.  The user-supplied tolerance TOL
     corresponds to the parameter \sigma used by Fletcher.  A value of
     0.1 is recommended for typical use (larger values correspond to
     less accurate line searches).

 -- Minimizer: gsl_multimin_fdfminimizer_steepest_descent
     The steepest descent algorithm follows the downhill gradient of the
     function at each step.  When a downhill step is successful the
     step-size is increased by a factor of two.  If the downhill step
     leads to a higher function value then the algorithm backtracks and
     the step size is decreased using the parameter TOL.  A suitable
     value of TOL for most applications is 0.1.  The steepest descent
     method is inefficient and is included only for demonstration
     purposes.


File: gsl-ref.info,  Node: Multimin Algorithms without Derivatives,  Next: Multimin Examples,  Prev: Multimin Algorithms with Derivatives,  Up: Multidimensional Minimization

37.8 Algorithms without Derivatives
===================================

The algorithms described in this section use only the value of the
function at each evaluation point.

 -- Minimizer: gsl_multimin_fminimizer_nmsimplex2
 -- Minimizer: gsl_multimin_fminimizer_nmsimplex
     These methods use the Simplex algorithm of Nelder and Mead.
     Starting from the initial vector X = p_0, the algorithm constructs
     an additional n vectors p_i using the step size vector s =
     STEP_SIZE as follows:

          p_0 = (x_0, x_1, ... , x_n)
          p_1 = (x_0 + s_0, x_1, ... , x_n)
          p_2 = (x_0, x_1 + s_1, ... , x_n)
          ... = ...
          p_n = (x_0, x_1, ... , x_n + s_n)

     These vectors form the n+1 vertices of a simplex in n dimensions.
     On each iteration the algorithm uses simple geometrical
     transformations to update the vector corresponding to the highest
     function value.  The geometric transformations are reflection,
     reflection followed by expansion, contraction and multiple
     contraction.  Using these transformations the simplex moves through
     the space towards the minimum, where it contracts itself.

     After each iteration, the best vertex is returned.  Note, that due
     to the nature of the algorithm not every step improves the current
     best parameter vector.  Usually several iterations are required.

     The minimizer-specific characteristic size is calculated as the
     average distance from the geometrical center of the simplex to all
     its vertices.  This size can be used as a stopping criteria, as the
     simplex contracts itself near the minimum.  The size is returned by
     the function 'gsl_multimin_fminimizer_size'.

     The 'nmsimplex2' version of this minimiser is a new O(N) operations
     implementation of the earlier O(N^2) operations 'nmsimplex'
     minimiser.  It uses the same underlying algorithm, but the simplex
     updates are computed more efficiently for high-dimensional
     problems.  In addition, the size of simplex is calculated as the
     RMS distance of each vertex from the center rather than the mean
     distance, allowing a linear update of this quantity on each step.
     The memory usage is O(N^2) for both algorithms.

 -- Minimizer: gsl_multimin_fminimizer_nmsimplex2rand
     This method is a variant of 'nmsimplex2' which initialises the
     simplex around the starting point X using a randomly-oriented set
     of basis vectors instead of the fixed coordinate axes.  The final
     dimensions of the simplex are scaled along the coordinate axes by
     the vector STEP_SIZE.  The randomisation uses a simple
     deterministic generator so that repeated calls to
     'gsl_multimin_fminimizer_set' for a given solver object will vary
     the orientation in a well-defined way.


File: gsl-ref.info,  Node: Multimin Examples,  Next: Multimin References and Further Reading,  Prev: Multimin Algorithms without Derivatives,  Up: Multidimensional Minimization

37.9 Examples
=============

This example program finds the minimum of the paraboloid function
defined earlier.  The location of the minimum is offset from the origin
in x and y, and the function value at the minimum is non-zero.  The main
program is given below, it requires the example function given earlier
in this chapter.

     int
     main (void)
     {
       size_t iter = 0;
       int status;

       const gsl_multimin_fdfminimizer_type *T;
       gsl_multimin_fdfminimizer *s;

       /* Position of the minimum (1,2), scale factors
          10,20, height 30. */
       double par[5] = { 1.0, 2.0, 10.0, 20.0, 30.0 };

       gsl_vector *x;
       gsl_multimin_function_fdf my_func;

       my_func.n = 2;
       my_func.f = my_f;
       my_func.df = my_df;
       my_func.fdf = my_fdf;
       my_func.params = par;

       /* Starting point, x = (5,7) */
       x = gsl_vector_alloc (2);
       gsl_vector_set (x, 0, 5.0);
       gsl_vector_set (x, 1, 7.0);

       T = gsl_multimin_fdfminimizer_conjugate_fr;
       s = gsl_multimin_fdfminimizer_alloc (T, 2);

       gsl_multimin_fdfminimizer_set (s, &my_func, x, 0.01, 1e-4);

       do
         {
           iter++;
           status = gsl_multimin_fdfminimizer_iterate (s);

           if (status)
             break;

           status = gsl_multimin_test_gradient (s->gradient, 1e-3);

           if (status == GSL_SUCCESS)
             printf ("Minimum found at:\n");

           printf ("%5d %.5f %.5f %10.5f\n", iter,
                   gsl_vector_get (s->x, 0),
                   gsl_vector_get (s->x, 1),
                   s->f);

         }
       while (status == GSL_CONTINUE && iter < 100);

       gsl_multimin_fdfminimizer_free (s);
       gsl_vector_free (x);

       return 0;
     }

The initial step-size is chosen as 0.01, a conservative estimate in this
case, and the line minimization parameter is set at 0.0001.  The program
terminates when the norm of the gradient has been reduced below 0.001.
The output of the program is shown below,

              x       y         f
         1 4.99629 6.99072  687.84780
         2 4.98886 6.97215  683.55456
         3 4.97400 6.93501  675.01278
         4 4.94429 6.86073  658.10798
         5 4.88487 6.71217  625.01340
         6 4.76602 6.41506  561.68440
         7 4.52833 5.82083  446.46694
         8 4.05295 4.63238  261.79422
         9 3.10219 2.25548   75.49762
        10 2.85185 1.62963   67.03704
        11 2.19088 1.76182   45.31640
        12 0.86892 2.02622   30.18555
     Minimum found at:
        13 1.00000 2.00000   30.00000

Note that the algorithm gradually increases the step size as it
successfully moves downhill, as can be seen by plotting the successive
points.

The conjugate gradient algorithm finds the minimum on its second
direction because the function is purely quadratic.  Additional
iterations would be needed for a more complicated function.

   Here is another example using the Nelder-Mead Simplex algorithm to
minimize the same example object function, as above.

     int
     main(void)
     {
       double par[5] = {1.0, 2.0, 10.0, 20.0, 30.0};

       const gsl_multimin_fminimizer_type *T =
         gsl_multimin_fminimizer_nmsimplex2;
       gsl_multimin_fminimizer *s = NULL;
       gsl_vector *ss, *x;
       gsl_multimin_function minex_func;

       size_t iter = 0;
       int status;
       double size;

       /* Starting point */
       x = gsl_vector_alloc (2);
       gsl_vector_set (x, 0, 5.0);
       gsl_vector_set (x, 1, 7.0);

       /* Set initial step sizes to 1 */
       ss = gsl_vector_alloc (2);
       gsl_vector_set_all (ss, 1.0);

       /* Initialize method and iterate */
       minex_func.n = 2;
       minex_func.f = my_f;
       minex_func.params = par;

       s = gsl_multimin_fminimizer_alloc (T, 2);
       gsl_multimin_fminimizer_set (s, &minex_func, x, ss);

       do
         {
           iter++;
           status = gsl_multimin_fminimizer_iterate(s);

           if (status)
             break;

           size = gsl_multimin_fminimizer_size (s);
           status = gsl_multimin_test_size (size, 1e-2);

           if (status == GSL_SUCCESS)
             {
               printf ("converged to minimum at\n");
             }

           printf ("%5d %10.3e %10.3e f() = %7.3f size = %.3f\n",
                   iter,
                   gsl_vector_get (s->x, 0),
                   gsl_vector_get (s->x, 1),
                   s->fval, size);
         }
       while (status == GSL_CONTINUE && iter < 100);

       gsl_vector_free(x);
       gsl_vector_free(ss);
       gsl_multimin_fminimizer_free (s);

       return status;
     }

The minimum search stops when the Simplex size drops to 0.01.  The
output is shown below.

         1  6.500e+00  5.000e+00 f() = 512.500 size = 1.130
         2  5.250e+00  4.000e+00 f() = 290.625 size = 1.409
         3  5.250e+00  4.000e+00 f() = 290.625 size = 1.409
         4  5.500e+00  1.000e+00 f() = 252.500 size = 1.409
         5  2.625e+00  3.500e+00 f() = 101.406 size = 1.847
         6  2.625e+00  3.500e+00 f() = 101.406 size = 1.847
         7  0.000e+00  3.000e+00 f() =  60.000 size = 1.847
         8  2.094e+00  1.875e+00 f() =  42.275 size = 1.321
         9  2.578e-01  1.906e+00 f() =  35.684 size = 1.069
        10  5.879e-01  2.445e+00 f() =  35.664 size = 0.841
        11  1.258e+00  2.025e+00 f() =  30.680 size = 0.476
        12  1.258e+00  2.025e+00 f() =  30.680 size = 0.367
        13  1.093e+00  1.849e+00 f() =  30.539 size = 0.300
        14  8.830e-01  2.004e+00 f() =  30.137 size = 0.172
        15  8.830e-01  2.004e+00 f() =  30.137 size = 0.126
        16  9.582e-01  2.060e+00 f() =  30.090 size = 0.106
        17  1.022e+00  2.004e+00 f() =  30.005 size = 0.063
        18  1.022e+00  2.004e+00 f() =  30.005 size = 0.043
        19  1.022e+00  2.004e+00 f() =  30.005 size = 0.043
        20  1.022e+00  2.004e+00 f() =  30.005 size = 0.027
        21  1.022e+00  2.004e+00 f() =  30.005 size = 0.022
        22  9.920e-01  1.997e+00 f() =  30.001 size = 0.016
        23  9.920e-01  1.997e+00 f() =  30.001 size = 0.013
     converged to minimum at
        24  9.920e-01  1.997e+00 f() =  30.001 size = 0.008

The simplex size first increases, while the simplex moves towards the
minimum.  After a while the size begins to decrease as the simplex
contracts around the minimum.


File: gsl-ref.info,  Node: Multimin References and Further Reading,  Prev: Multimin Examples,  Up: Multidimensional Minimization

37.10 References and Further Reading
====================================

The conjugate gradient and BFGS methods are described in detail in the
following book,

     R. Fletcher, 'Practical Methods of Optimization (Second Edition)'
     Wiley (1987), ISBN 0471915475.

   A brief description of multidimensional minimization algorithms and
more recent references can be found in,

     C.W. Ueberhuber, 'Numerical Computation (Volume 2)', Chapter 14,
     Section 4.4 "Minimization Methods", p. 325-335, Springer (1997),
     ISBN 3-540-62057-5.

The simplex algorithm is described in the following paper,

     J.A. Nelder and R. Mead, 'A simplex method for function
     minimization', Computer Journal vol. 7 (1965), 308-313.


File: gsl-ref.info,  Node: Least-Squares Fitting,  Next: Nonlinear Least-Squares Fitting,  Prev: Multidimensional Minimization,  Up: Top

38 Least-Squares Fitting
************************

This chapter describes routines for performing least squares fits to
experimental data using linear combinations of functions.  The data may
be weighted or unweighted, i.e.  with known or unknown errors.  For
weighted data the functions compute the best fit parameters and their
associated covariance matrix.  For unweighted data the covariance matrix
is estimated from the scatter of the points, giving a
variance-covariance matrix.

   The functions are divided into separate versions for simple one- or
two-parameter regression and multiple-parameter fits.

* Menu:

* Fitting Overview::            
* Linear regression::           
* Multi-parameter regression::
* Regularized regression::
* Robust linear regression::
* Large Linear Systems::
* Troubleshooting::
* Fitting Examples for linear regression::
* Fitting Examples for multi-parameter linear regression::
* Fitting Examples for regularized linear regression::
* Fitting Examples for robust linear regression::
* Fitting Examples for large linear systems::
* Fitting References and Further Reading::  


File: gsl-ref.info,  Node: Fitting Overview,  Next: Linear regression,  Up: Least-Squares Fitting

38.1 Overview
=============

Least-squares fits are found by minimizing \chi^2 (chi-squared), the
weighted sum of squared residuals over n experimental datapoints (x_i,
y_i) for the model Y(c,x),

     \chi^2 = \sum_i w_i (y_i - Y(c, x_i))^2

The p parameters of the model are c = {c_0, c_1, ...}.  The weight
factors w_i are given by w_i = 1/\sigma_i^2, where \sigma_i is the
experimental error on the data-point y_i.  The errors are assumed to be
Gaussian and uncorrelated.  For unweighted data the chi-squared sum is
computed without any weight factors.

   The fitting routines return the best-fit parameters c and their p
\times p covariance matrix.  The covariance matrix measures the
statistical errors on the best-fit parameters resulting from the errors
on the data, \sigma_i, and is defined as C_{ab} = <\delta c_a \delta
c_b> where < > denotes an average over the Gaussian error distributions
of the underlying datapoints.

   The covariance matrix is calculated by error propagation from the
data errors \sigma_i.  The change in a fitted parameter \delta c_a
caused by a small change in the data \delta y_i is given by

     \delta c_a = \sum_i (dc_a/dy_i) \delta y_i

allowing the covariance matrix to be written in terms of the errors on
the data,

     C_{ab} = \sum_{i,j} (dc_a/dy_i) (dc_b/dy_j) <\delta y_i \delta y_j>

For uncorrelated data the fluctuations of the underlying datapoints
satisfy <\delta y_i \delta y_j> = \sigma_i^2 \delta_{ij}, giving a
corresponding parameter covariance matrix of

     C_{ab} = \sum_i (1/w_i) (dc_a/dy_i) (dc_b/dy_i)

When computing the covariance matrix for unweighted data, i.e.  data
with unknown errors, the weight factors w_i in this sum are replaced by
the single estimate w = 1/\sigma^2, where \sigma^2 is the computed
variance of the residuals about the best-fit model, \sigma^2 = \sum (y_i
- Y(c,x_i))^2 / (n-p).  This is referred to as the "variance-covariance
matrix".

   The standard deviations of the best-fit parameters are given by the
square root of the corresponding diagonal elements of the covariance
matrix, \sigma_{c_a} = \sqrt{C_{aa}}.  The correlation coefficient of
the fit parameters c_a and c_b is given by \rho_{ab} = C_{ab} /
\sqrt{C_{aa} C_{bb}}.


File: gsl-ref.info,  Node: Linear regression,  Next: Multi-parameter regression,  Prev: Fitting Overview,  Up: Least-Squares Fitting

38.2 Linear regression
======================

The functions in this section are used to fit simple one or two
parameter linear regression models.  The functions are declared in the
header file 'gsl_fit.h'.

* Menu:

* Linear regression with a constant term::
* Linear regression without a constant term::


File: gsl-ref.info,  Node: Linear regression with a constant term,  Next: Linear regression without a constant term,  Up: Linear regression

38.2.1 Linear regression with a constant term
---------------------------------------------

The functions described in this section can be used to perform
least-squares fits to a straight line model, Y(c,x) = c_0 + c_1 x.

 -- Function: int gsl_fit_linear (const double * X, const size_t
          XSTRIDE, const double * Y, const size_t YSTRIDE, size_t N,
          double * C0, double * C1, double * COV00, double * COV01,
          double * COV11, double * SUMSQ)
     This function computes the best-fit linear regression coefficients
     (C0,C1) of the model Y = c_0 + c_1 X for the dataset (X, Y), two
     vectors of length N with strides XSTRIDE and YSTRIDE.  The errors
     on Y are assumed unknown so the variance-covariance matrix for the
     parameters (C0, C1) is estimated from the scatter of the points
     around the best-fit line and returned via the parameters (COV00,
     COV01, COV11).  The sum of squares of the residuals from the
     best-fit line is returned in SUMSQ.  Note: the correlation
     coefficient of the data can be computed using
     'gsl_stats_correlation' (*note Correlation::), it does not depend
     on the fit.

 -- Function: int gsl_fit_wlinear (const double * X, const size_t
          XSTRIDE, const double * W, const size_t WSTRIDE, const double
          * Y, const size_t YSTRIDE, size_t N, double * C0, double * C1,
          double * COV00, double * COV01, double * COV11, double *
          CHISQ)
     This function computes the best-fit linear regression coefficients
     (C0,C1) of the model Y = c_0 + c_1 X for the weighted dataset (X,
     Y), two vectors of length N with strides XSTRIDE and YSTRIDE.  The
     vector W, of length N and stride WSTRIDE, specifies the weight of
     each datapoint.  The weight is the reciprocal of the variance for
     each datapoint in Y.

     The covariance matrix for the parameters (C0, C1) is computed using
     the weights and returned via the parameters (COV00, COV01, COV11).
     The weighted sum of squares of the residuals from the best-fit
     line, \chi^2, is returned in CHISQ.

 -- Function: int gsl_fit_linear_est (double X, double C0, double C1,
          double COV00, double COV01, double COV11, double * Y, double *
          Y_ERR)
     This function uses the best-fit linear regression coefficients C0,
     C1 and their covariance COV00, COV01, COV11 to compute the fitted
     function Y and its standard deviation Y_ERR for the model Y = c_0 +
     c_1 X at the point X.


File: gsl-ref.info,  Node: Linear regression without a constant term,  Prev: Linear regression with a constant term,  Up: Linear regression

38.2.2 Linear regression without a constant term
------------------------------------------------

The functions described in this section can be used to perform
least-squares fits to a straight line model without a constant term, Y =
c_1 X.

 -- Function: int gsl_fit_mul (const double * X, const size_t XSTRIDE,
          const double * Y, const size_t YSTRIDE, size_t N, double * C1,
          double * COV11, double * SUMSQ)
     This function computes the best-fit linear regression coefficient
     C1 of the model Y = c_1 X for the datasets (X, Y), two vectors of
     length N with strides XSTRIDE and YSTRIDE.  The errors on Y are
     assumed unknown so the variance of the parameter C1 is estimated
     from the scatter of the points around the best-fit line and
     returned via the parameter COV11.  The sum of squares of the
     residuals from the best-fit line is returned in SUMSQ.

 -- Function: int gsl_fit_wmul (const double * X, const size_t XSTRIDE,
          const double * W, const size_t WSTRIDE, const double * Y,
          const size_t YSTRIDE, size_t N, double * C1, double * COV11,
          double * SUMSQ)
     This function computes the best-fit linear regression coefficient
     C1 of the model Y = c_1 X for the weighted datasets (X, Y), two
     vectors of length N with strides XSTRIDE and YSTRIDE.  The vector
     W, of length N and stride WSTRIDE, specifies the weight of each
     datapoint.  The weight is the reciprocal of the variance for each
     datapoint in Y.

     The variance of the parameter C1 is computed using the weights and
     returned via the parameter COV11.  The weighted sum of squares of
     the residuals from the best-fit line, \chi^2, is returned in CHISQ.

 -- Function: int gsl_fit_mul_est (double X, double C1, double COV11,
          double * Y, double * Y_ERR)
     This function uses the best-fit linear regression coefficient C1
     and its covariance COV11 to compute the fitted function Y and its
     standard deviation Y_ERR for the model Y = c_1 X at the point X.


File: gsl-ref.info,  Node: Multi-parameter regression,  Next: Regularized regression,  Prev: Linear regression,  Up: Least-Squares Fitting

38.3 Multi-parameter regression
===============================

This section describes routines which perform least squares fits to a
linear model by minimizing the cost function
     \chi^2 = \sum_i w_i (y_i - \sum_j X_ij c_j)^2 = || y - Xc ||_W^2
   where y is a vector of n observations, X is an n-by-p matrix of
predictor variables, c is a vector of the p unknown best-fit parameters
to be estimated, and ||r||_W^2 = r^T W r.  The matrix W =
diag(w_1,w_2,...,w_n) defines the weights or uncertainties of the
observation vector.

   This formulation can be used for fits to any number of functions
and/or variables by preparing the n-by-p matrix X appropriately.  For
example, to fit to a p-th order polynomial in X, use the following
matrix,

     X_{ij} = x_i^j

where the index i runs over the observations and the index j runs from 0
to p-1.

   To fit to a set of p sinusoidal functions with fixed frequencies
\omega_1, \omega_2, ..., \omega_p, use,

     X_{ij} = sin(\omega_j x_i)

To fit to p independent variables x_1, x_2, ..., x_p, use,

     X_{ij} = x_j(i)

where x_j(i) is the i-th value of the predictor variable x_j.

   The solution of the general linear least-squares system requires an
additional working space for intermediate results, such as the singular
value decomposition of the matrix X.

   These functions are declared in the header file 'gsl_multifit.h'.

 -- Function: gsl_multifit_linear_workspace * gsl_multifit_linear_alloc
          (const size_t N, const size_t P)
     This function allocates a workspace for fitting a model to a
     maximum of N observations using a maximum of P parameters.  The
     user may later supply a smaller least squares system if desired.
     The size of the workspace is O(np + p^2).

 -- Function: void gsl_multifit_linear_free
          (gsl_multifit_linear_workspace * WORK)
     This function frees the memory associated with the workspace W.

 -- Function: int gsl_multifit_linear_svd (const gsl_matrix * X,
          gsl_multifit_linear_workspace * WORK)
     This function performs a singular value decomposition of the matrix
     X and stores the SVD factors internally in WORK.

 -- Function: int gsl_multifit_linear_bsvd (const gsl_matrix * X,
          gsl_multifit_linear_workspace * WORK)
     This function performs a singular value decomposition of the matrix
     X and stores the SVD factors internally in WORK.  The matrix X is
     first balanced by applying column scaling factors to improve the
     accuracy of the singular values.

 -- Function: int gsl_multifit_linear (const gsl_matrix * X, const
          gsl_vector * Y, gsl_vector * C, gsl_matrix * COV, double *
          CHISQ, gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the model y = X
     c for the observations Y and the matrix of predictor variables X,
     using the preallocated workspace provided in WORK.  The p-by-p
     variance-covariance matrix of the model parameters COV is set to
     \sigma^2 (X^T X)^{-1}, where \sigma is the standard deviation of
     the fit residuals.  The sum of squares of the residuals from the
     best-fit, \chi^2, is returned in CHISQ.  If the coefficient of
     determination is desired, it can be computed from the expression
     R^2 = 1 - \chi^2 / TSS, where the total sum of squares (TSS) of the
     observations Y may be computed from 'gsl_stats_tss'.

     The best-fit is found by singular value decomposition of the matrix
     X using the modified Golub-Reinsch SVD algorithm, with column
     scaling to improve the accuracy of the singular values.  Any
     components which have zero singular value (to machine precision)
     are discarded from the fit.

 -- Function: int gsl_multifit_wlinear (const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_vector * C,
          gsl_matrix * COV, double * CHISQ,
          gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the weighted
     model y = X c for the observations Y with weights W and the matrix
     of predictor variables X, using the preallocated workspace provided
     in WORK.  The p-by-p covariance matrix of the model parameters COV
     is computed as (X^T W X)^{-1}.  The weighted sum of squares of the
     residuals from the best-fit, \chi^2, is returned in CHISQ.  If the
     coefficient of determination is desired, it can be computed from
     the expression R^2 = 1 - \chi^2 / WTSS, where the weighted total
     sum of squares (WTSS) of the observations Y may be computed from
     'gsl_stats_wtss'.

 -- Function: int gsl_multifit_linear_est (const gsl_vector * X, const
          gsl_vector * C, const gsl_matrix * COV, double * Y, double *
          Y_ERR)
     This function uses the best-fit multilinear regression coefficients
     C and their covariance matrix COV to compute the fitted function
     value Y and its standard deviation Y_ERR for the model y = x.c at
     the point X.

 -- Function: int gsl_multifit_linear_residuals (const gsl_matrix * X,
          const gsl_vector * Y, const gsl_vector * C, gsl_vector * R)
     This function computes the vector of residuals r = y - X c for the
     observations Y, coefficients C and matrix of predictor variables X.


File: gsl-ref.info,  Node: Regularized regression,  Next: Robust linear regression,  Prev: Multi-parameter regression,  Up: Least-Squares Fitting

38.4 Regularized regression
===========================

Ordinary weighted least squares models seek a solution vector c which
minimizes the residual
     \chi^2 = || y - Xc ||_W^2
   where y is the n-by-1 observation vector, X is the n-by-p design
matrix, c is the p-by-1 solution vector, W = diag(w_1,...,w_n) is the
data weighting matrix, and ||r||_W^2 = r^T W r.  In cases where the
least squares matrix X is ill-conditioned, small perturbations (ie:
noise) in the observation vector could lead to widely different solution
vectors c.  In these cases it is often advantageous to include a
regularization term in the least squares minimization
     \chi^2 = || y - Xc ||_W^2 + \lambda^2 || L c ||^2
   for a suitably chosen regularization parameter \lambda and matrix L.
This type of regularization is known as Tikhonov, or ridge, regression.
In some applications, L is chosen as the identity matrix, giving
preference to solution vectors c with smaller norms.  Including this
regularization term leads to the explicit "normal equations" solution
     c = ( X^T W X + \lambda^2 L^T L )^-1 X^T W y
   which reduces to the ordinary least squares solution when L = 0.  In
practice, it is often advantageous to transform a regularized least
squares system into the form
     \chi^2 = || y~ - X~ c~ ||^2 + \lambda^2 || c~ ||^2
   This is known as the Tikhonov "standard form" and has the normal
equations solution \tilde{c} = \left( \tilde{X}^T \tilde{X} + \lambda^2
I \right)^{-1} \tilde{X}^T \tilde{y}.  For an m-by-p matrix L which is
full rank and has m >= p (ie: L is square or has more rows than
columns), we can calculate the "thin" QR decomposition of L, and note
that ||L c|| = ||R c|| since the Q factor will not change the norm.
Since R is p-by-p, we can then use the transformation
     X~ = sqrt(W) X R^-1
     y~ = sqrt(W) y
     c~ = R c
   to achieve the standard form.  For a rectangular matrix L with m < p,
a more sophisticated approach is needed (see Hansen 1998, chapter 2.3).
In practice, the normal equations solution above is not desirable due to
numerical instabilities, and so the system is solved using the singular
value decomposition of the matrix \tilde{X}.  The matrix L is often
chosen as the identity matrix, or as a first or second finite difference
operator, to ensure a smoothly varying coefficient vector c, or as a
diagonal matrix to selectively damp each model parameter differently.
If L \ne I, the user must first convert the least squares problem to
standard form using 'gsl_multifit_linear_stdform1' or
'gsl_multifit_linear_stdform2', solve the system, and then backtransform
the solution vector to recover the solution of the original problem (see
'gsl_multifit_linear_genform1' and 'gsl_multifit_linear_genform2').

   In many regularization problems, care must be taken when choosing the
regularization parameter \lambda.  Since both the residual norm ||y - X
c|| and solution norm ||L c|| are being minimized, the parameter \lambda
represents a tradeoff between minimizing either the residuals or the
solution vector.  A common tool for visualizing the comprimise between
the minimization of these two quantities is known as the L-curve.  The
L-curve is a log-log plot of the residual norm ||y - X c|| on the
horizontal axis and the solution norm ||L c|| on the vertical axis.
This curve nearly always as an L shaped appearance, with a distinct
corner separating the horizontal and vertical sections of the curve.
The regularization parameter corresponding to this corner is often
chosen as the optimal value.  GSL provides routines to calculate the
L-curve for all relevant regularization parameters as well as locating
the corner.

For most applications, the steps required to solve a regularized least
squares problem are as follows:

  1. Construct the least squares system (X, y, W, L)

  2. Transform the system to standard form (\tilde{X},\tilde{y}).  This
     step can be skipped if L = I_p and W = I_n.

  3. Calculate the SVD of \tilde{X}.

  4. Determine an appropriate regularization parameter \lambda (using
     for example L-curve analysis).

  5. Solve the standard form system using the chosen \lambda and the SVD
     of \tilde{X}.

  6. Backtransform the standard form solution \tilde{c} to recover the
     original solution vector c.

 -- Function: int gsl_multifit_linear_stdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_multifit_linear_workspace * WORK)
 -- Function: int gsl_multifit_linear_wstdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * W, const gsl_vector *
          Y, gsl_matrix * XS, gsl_vector * YS,
          gsl_multifit_linear_workspace * WORK)
     These functions define a regularization matrix L =
     diag(l_0,l_1,...,l_{p-1}).  The diagonal matrix element l_i is
     provided by the ith element of the input vector L.  The n-by-p
     least squares matrix X and vector Y of length n are then converted
     to standard form as described above and the parameters
     (\tilde{X},\tilde{y}) are stored in XS and YS on output.  XS and YS
     have the same dimensions as X and Y.  Optional data weights may be
     supplied in the vector W of length n.  In order to apply this
     transformation, L^{-1} must exist and so none of the l_i may be
     zero.  After the standard form system has been solved, use
     'gsl_multifit_linear_genform1' to recover the original solution
     vector.  It is allowed to have X = XS and Y = YS for an in-place
     transform.  In order to perform a weighted regularized fit with L =
     I, the user may call 'gsl_multifit_linear_applyW' to convert to
     standard form.

 -- Function: int gsl_multifit_linear_L_decomp (gsl_matrix * L,
          gsl_vector * TAU)
     This function factors the m-by-p regularization matrix L into a
     form needed for the later transformation to standard form.  L may
     have any number of rows m.  If m \ge p the QR decomposition of L is
     computed and stored in L on output.  If m < p, the QR decomposition
     of L^T is computed and stored in L on output.  On output, the
     Householder scalars are stored in the vector TAU of size MIN(m,p).
     These outputs will be used by 'gsl_multifit_linear_wstdform2' to
     complete the transformation to standard form.

 -- Function: int gsl_multifit_linear_stdform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * Y, gsl_matrix * XS, gsl_vector * YS, gsl_matrix *
          M, gsl_multifit_linear_workspace * WORK)
 -- Function: int gsl_multifit_linear_wstdform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_matrix * M, gsl_multifit_linear_workspace
          * WORK)
     These functions convert the least squares system (X,Y,W,L) to
     standard form (\tilde{X},\tilde{y}) which are stored in XS and YS
     respectively.  The m-by-p regularization matrix L is specified by
     the inputs LQR and LTAU, which are outputs from
     'gsl_multifit_linear_L_decomp'.  The dimensions of the standard
     form parameters (\tilde{X},\tilde{y}) depend on whether m is larger
     or less than p.  For m \ge p, XS is n-by-p, YS is n-by-1, and M is
     not used.  For m < p, XS is (n - p + m)-by-m, YS is (n - p +
     m)-by-1, and M is additional n-by-p workspace, which is required to
     recover the original solution vector after the system has been
     solved (see 'gsl_multifit_linear_genform2').  Optional data weights
     may be supplied in the vector W of length n, where W = diag(w).

 -- Function: int gsl_multifit_linear_solve (const double LAMBDA, const
          gsl_matrix * XS, const gsl_vector * YS, gsl_vector * CS,
          double * RNORM, double * SNORM, gsl_multifit_linear_workspace
          * WORK)
     This function computes the regularized best-fit parameters
     \tilde{c} which minimize the cost function \chi^2 = || \tilde{y} -
     \tilde{X} \tilde{c} ||^2 + \lambda^2 || \tilde{c} ||^2 which is in
     standard form.  The least squares system must therefore be
     converted to standard form prior to calling this function.  The
     observation vector \tilde{y} is provided in YS and the matrix of
     predictor variables \tilde{X} in XS.  The solution vector \tilde{c}
     is returned in CS, which has length min(m,p).  The SVD of XS must
     be computed prior to calling this function, using
     'gsl_multifit_linear_svd'.  The regularization parameter \lambda is
     provided in LAMBDA.  The residual norm || \tilde{y} - \tilde{X}
     \tilde{c} || = ||y - X c||_W is returned in RNORM.  The solution
     norm || \tilde{c} || = ||L c|| is returned in SNORM.

 -- Function: int gsl_multifit_linear_genform1 (const gsl_vector * L,
          const gsl_vector * CS, gsl_vector * C,
          gsl_multifit_linear_workspace * WORK)
     After a regularized system has been solved with L =
     diag(\l_0,\l_1,...,\l_{p-1}), this function backtransforms the
     standard form solution vector CS to recover the solution vector of
     the original problem C.  The diagonal matrix elements l_i are
     provided in the vector L.  It is allowed to have C = CS for an
     in-place transform.

 -- Function: int gsl_multifit_linear_genform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * Y, const gsl_vector * CS, const gsl_matrix * M,
          gsl_vector * C, gsl_multifit_linear_workspace * WORK)
 -- Function: int gsl_multifit_linear_wgenform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, const gsl_vector * CS,
          const gsl_matrix * M, gsl_vector * C,
          gsl_multifit_linear_workspace * WORK)
     After a regularized system has been solved with a general
     rectangular matrix L, specified by (LQR,LTAU), this function
     backtransforms the standard form solution CS to recover the
     solution vector of the original problem, which is stored in C, of
     length p.  The original least squares matrix and observation vector
     are provided in X and Y respectively.  M is the matrix computed by
     'gsl_multifit_linear_stdform2'.  For weighted fits, the weight
     vector W must also be supplied.

 -- Function: int gsl_multifit_linear_applyW (const gsl_matrix * X,
          const gsl_vector * W, const gsl_vector * Y, gsl_matrix * WX,
          gsl_vector * WY, gsl_multifit_linear_workspace * WORK)
     For weighted least squares systems with L = I, this function may be
     used to convert the system to standard form by applying the weight
     matrix W = diag(W) to the least squares matrix X and observation
     vector Y.  On output, WX is equal to W^{1/2} X and WY is equal to
     W^{1/2} y.  It is allowed for WX = X and WY = Y for an in-place
     transform.

 -- Function: int gsl_multifit_linear_lcurve (const gsl_vector * Y,
          gsl_vector * REG_PARAM, gsl_vector * RHO, gsl_vector * ETA,
          gsl_multifit_linear_workspace * WORK)
     This function computes the L-curve for a least squares system using
     the right hand side vector Y and the SVD decomposition of the least
     squares matrix X, which must be provided to
     'gsl_multifit_linear_svd' prior to calling this function.  The
     output vectors REG_PARAM, RHO, and ETA must all be the same size,
     and will contain the regularization parameters \lambda_i, residual
     norms ||y - X c_i||, and solution norms || L c_i || which compose
     the L-curve, where c_i is the regularized solution vector
     corresponding to \lambda_i.  The user may determine the number of
     points on the L-curve by adjusting the size of these input arrays.
     The regularization parameters \lambda_i are estimated from the
     singular values of X, and chosen to represent the most relevant
     portion of the L-curve.

 -- Function: int gsl_multifit_linear_lcorner (const gsl_vector * RHO,
          const gsl_vector * ETA, size_t * IDX)
     This function attempts to locate the corner of the L-curve (||y - X
     c||, ||L c||) defined by the RHO and ETA input arrays respectively.
     The corner is defined as the point of maximum curvature of the
     L-curve in log-log scale.  The RHO and ETA arrays can be outputs of
     'gsl_multifit_linear_lcurve'.  The algorithm used simply fits a
     circle to 3 consecutive points on the L-curve and uses the circle's
     radius to determine the curvature at the middle point.  Therefore,
     the input array sizes must be \ge 3.  With more points provided for
     the L-curve, a better estimate of the curvature can be obtained.
     The array index corresponding to maximum curvature (ie: the corner)
     is returned in IDX.  If the input arrays contain colinear points,
     this function could fail and return 'GSL_EINVAL'.

 -- Function: int gsl_multifit_linear_lcorner2 (const gsl_vector *
          REG_PARAM, const gsl_vector * ETA, size_t * IDX)
     This function attempts to locate the corner of an alternate L-curve
     (\lambda^2, ||L c||^2) studied by Rezghi and Hosseini, 2009.  This
     alternate L-curve can provide better estimates of the
     regularization parameter for smooth solution vectors.  The
     regularization parameters \lambda and solution norms ||L c|| are
     provided in the REG_PARAM and ETA input arrays respectively.  The
     corner is defined as the point of maximum curvature of this
     alternate L-curve in linear scale.  The REG_PARAM and ETA arrays
     can be outputs of 'gsl_multifit_linear_lcurve'.  The algorithm used
     simply fits a circle to 3 consecutive points on the L-curve and
     uses the circle's radius to determine the curvature at the middle
     point.  Therefore, the input array sizes must be \ge 3.  With more
     points provided for the L-curve, a better estimate of the curvature
     can be obtained.  The array index corresponding to maximum
     curvature (ie: the corner) is returned in IDX.  If the input arrays
     contain colinear points, this function could fail and return
     'GSL_EINVAL'.

 -- Function: int gsl_multifit_linear_Lk (const size_t P, const size_t
          K, gsl_matrix * L)
     This function computes the discrete approximation to the derivative
     operator L_k of order K on a regular grid of P points and stores it
     in L.  The dimensions of L are (p-k)-by-p.

 -- Function: int gsl_multifit_linear_Lsobolev (const size_t P, const
          size_t KMAX, const gsl_vector * ALPHA, gsl_matrix * L,
          gsl_multifit_linear_workspace * WORK)
     This function computes the regularization matrix L corresponding to
     the weighted Sobolov norm ||L c||^2 = \sum_k \alpha_k^2 ||L_k c||^2
     where L_k approximates the derivative operator of order k.  This
     regularization norm can be useful in applications where it is
     necessary to smooth several derivatives of the solution.  P is the
     number of model parameters, KMAX is the highest derivative to
     include in the summation above, and ALPHA is the vector of weights
     of size KMAX + 1, where ALPHA[k] = \alpha_k is the weight assigned
     to the derivative of order k.  The output matrix L is of size
     P-by-P.

 -- Function: double gsl_multifit_linear_rcond (const
          gsl_multifit_linear_workspace * WORK)
     This function returns the reciprocal condition number of the least
     squares matrix X, defined as the ratio of the smallest and largest
     singular values, rcond = \sigma_{min}/\sigma_{max}.  The routine
     'gsl_multifit_linear_svd' must first be called to compute the SVD
     of X.


File: gsl-ref.info,  Node: Robust linear regression,  Next: Large Linear Systems,  Prev: Regularized regression,  Up: Least-Squares Fitting

38.5 Robust linear regression
=============================

Ordinary least squares (OLS) models are often heavily influenced by the
presence of outliers.  Outliers are data points which do not follow the
general trend of the other observations, although there is strictly no
precise definition of an outlier.  Robust linear regression refers to
regression algorithms which are robust to outliers.  The most common
type of robust regression is M-estimation.  The general M-estimator
minimizes the objective function

     \sum_i \rho(e_i) = \sum_i \rho (y_i - Y(c, x_i))

   where e_i = y_i - Y(c, x_i) is the residual of the ith data point,
and \rho(e_i) is a function which should have the following properties:
     \rho(e) \ge 0
     \rho(0) = 0
     \rho(-e) = \rho(e)
     \rho(e_1) > \rho(e_2) for |e_1| > |e_2|
The special case of ordinary least squares is given by \rho(e_i) =
e_i^2.  Letting \psi = \rho' be the derivative of \rho, differentiating
the objective function with respect to the coefficients c and setting
the partial derivatives to zero produces the system of equations

     \sum_i \psi(e_i) X_i = 0

   where X_i is a vector containing row i of the design matrix X.  Next,
we define a weight function w(e) = \psi(e)/e, and let w_i = w(e_i):

     \sum_i w_i e_i X_i = 0

   This system of equations is equivalent to solving a weighted ordinary
least squares problem, minimizing \chi^2 = \sum_i w_i e_i^2.  The
weights however, depend on the residuals e_i, which depend on the
coefficients c, which depend on the weights.  Therefore, an iterative
solution is used, called Iteratively Reweighted Least Squares (IRLS).
  1. Compute initial estimates of the coefficients c^{(0)} using
     ordinary least squares

  2. For iteration k, form the residuals e_i^{(k)} = (y_i - X_i
     c^{(k-1)})/(t \sigma^{(k)} \sqrt{1 - h_i}), where t is a tuning
     constant depending on the choice of \psi, and h_i are the
     statistical leverages (diagonal elements of the matrix X (X^T
     X)^{-1} X^T).  Including t and h_i in the residual calculation has
     been shown to improve the convergence of the method.  The residual
     standard deviation is approximated as \sigma^{(k)} = MAD / 0.6745,
     where MAD is the Median-Absolute-Deviation of the n-p largest
     residuals from the previous iteration.

  3. Compute new weights w_i^{(k)} = \psi(e_i^{(k)})/e_i^{(k)}.

  4. Compute new coefficients c^{(k)} by solving the weighted least
     squares problem with weights w_i^{(k)}.

  5. Steps 2 through 4 are iterated until the coefficients converge or
     until some maximum iteration limit is reached.  Coefficients are
     tested for convergence using the critera:

          |c_i^(k) - c_i^(k-1)| \le \epsilon \times max(|c_i^(k)|, |c_i^(k-1)|)

     for all 0 \le i < p where \epsilon is a small tolerance factor.
The key to this method lies in selecting the function \psi(e_i) to
assign smaller weights to large residuals, and larger weights to smaller
residuals.  As the iteration proceeds, outliers are assigned smaller and
smaller weights, eventually having very little or no effect on the
fitted model.

 -- Function: gsl_multifit_robust_workspace * gsl_multifit_robust_alloc
          (const gsl_multifit_robust_type * T, const size_t N, const
          size_t P)
     This function allocates a workspace for fitting a model to N
     observations using P parameters.  The size of the workspace is O(np
     + p^2).  The type T specifies the function \psi and can be selected
     from the following choices.
      -- Robust type: gsl_multifit_robust_default
          This specifies the 'gsl_multifit_robust_bisquare' type (see
          below) and is a good general purpose choice for robust
          regression.

      -- Robust type: gsl_multifit_robust_bisquare
          This is Tukey's biweight (bisquare) function and is a good
          general purpose choice for robust regression.  The weight
          function is given by

               w(e) = (1 - e^2)^2

          and the default tuning constant is t = 4.685.

      -- Robust type: gsl_multifit_robust_cauchy
          This is Cauchy's function, also known as the Lorentzian
          function.  This function does not guarantee a unique solution,
          meaning different choices of the coefficient vector C could
          minimize the objective function.  Therefore this option should
          be used with care.  The weight function is given by

               w(e) = 1 / (1 + e^2)

          and the default tuning constant is t = 2.385.

      -- Robust type: gsl_multifit_robust_fair
          This is the fair \rho function, which guarantees a unique
          solution and has continuous derivatives to three orders.  The
          weight function is given by

               w(e) = 1 / (1 + |e|)

          and the default tuning constant is t = 1.400.

      -- Robust type: gsl_multifit_robust_huber
          This specifies Huber's \rho function, which is a parabola in
          the vicinity of zero and increases linearly for a given
          threshold |e| > t.  This function is also considered an
          excellent general purpose robust estimator, however,
          occasional difficulties can be encountered due to the
          discontinuous first derivative of the \psi function.  The
          weight function is given by

               w(e) = 1/max(1,|e|)

          and the default tuning constant is t = 1.345.

      -- Robust type: gsl_multifit_robust_ols
          This specifies the ordinary least squares solution, which can
          be useful for quickly checking the difference between the
          various robust and OLS solutions.  The weight function is
          given by

               w(e) = 1

          and the default tuning constant is t = 1.

      -- Robust type: gsl_multifit_robust_welsch
          This specifies the Welsch function which can perform well in
          cases where the residuals have an exponential distribution.
          The weight function is given by

               w(e) = \exp(-e^2)

          and the default tuning constant is t = 2.985.

 -- Function: void gsl_multifit_robust_free
          (gsl_multifit_robust_workspace * W)
     This function frees the memory associated with the workspace W.

 -- Function: const char * gsl_multifit_robust_name (const
          gsl_multifit_robust_workspace * W)
     This function returns the name of the robust type T specified to
     'gsl_multifit_robust_alloc'.

 -- Function: int gsl_multifit_robust_tune (const double TUNE,
          gsl_multifit_robust_workspace * W)
     This function sets the tuning constant t used to adjust the
     residuals at each iteration to TUNE.  Decreasing the tuning
     constant increases the downweight assigned to large residuals,
     while increasing the tuning constant decreases the downweight
     assigned to large residuals.

 -- Function: int gsl_multifit_robust_maxiter (const size_t MAXITER,
          gsl_multifit_robust_workspace * W)
     This function sets the maximum number of iterations in the
     iteratively reweighted least squares algorithm to MAXITER.  By
     default, this value is set to 100 by 'gsl_multifit_robust_alloc'.

 -- Function: int gsl_multifit_robust_weights (const gsl_vector * R,
          gsl_vector * WTS, gsl_multifit_robust_workspace * W)
     This function assigns weights to the vector WTS using the residual
     vector R and previously specified weighting function.  The output
     weights are given by wts_i = w(r_i / (t \sigma)), where the
     weighting functions w are detailed in 'gsl_multifit_robust_alloc'.
     \sigma is an estimate of the residual standard deviation based on
     the Median-Absolute-Deviation and t is the tuning constant.  This
     function is useful if the user wishes to implement their own robust
     regression rather than using the supplied 'gsl_multifit_robust'
     routine below.

 -- Function: int gsl_multifit_robust (const gsl_matrix * X, const
          gsl_vector * Y, gsl_vector * C, gsl_matrix * COV,
          gsl_multifit_robust_workspace * W)
     This function computes the best-fit parameters C of the model y = X
     c for the observations Y and the matrix of predictor variables X,
     attemping to reduce the influence of outliers using the algorithm
     outlined above.  The p-by-p variance-covariance matrix of the model
     parameters COV is estimated as \sigma^2 (X^T X)^{-1}, where \sigma
     is an approximation of the residual standard deviation using the
     theory of robust regression.  Special care must be taken when
     estimating \sigma and other statistics such as R^2, and so these
     are computed internally and are available by calling the function
     'gsl_multifit_robust_statistics'.

     If the coefficients do not converge within the maximum iteration
     limit, the function returns 'GSL_EMAXITER'.  In this case, the
     current estimates of the coefficients and covariance matrix are
     returned in C and COV and the internal fit statistics are computed
     with these estimates.

 -- Function: int gsl_multifit_robust_est (const gsl_vector * X, const
          gsl_vector * C, const gsl_matrix * COV, double * Y, double *
          Y_ERR)
     This function uses the best-fit robust regression coefficients C
     and their covariance matrix COV to compute the fitted function
     value Y and its standard deviation Y_ERR for the model y = x.c at
     the point X.

 -- Function: int gsl_multifit_robust_residuals (const gsl_matrix * X,
          const gsl_vector * Y, const gsl_vector * C, gsl_vector * R,
          gsl_multifit_robust_workspace * W)
     This function computes the vector of studentized residuals r_i =
     {y_i - (X c)_i \over \sigma \sqrt{1 - h_i}} for the observations Y,
     coefficients C and matrix of predictor variables X.  The routine
     'gsl_multifit_robust' must first be called to compute the
     statisical leverages h_i of the matrix X and residual standard
     deviation estimate \sigma.

 -- Function: gsl_multifit_robust_stats gsl_multifit_robust_statistics
          (const gsl_multifit_robust_workspace * W)
     This function returns a structure containing relevant statistics
     from a robust regression.  The function 'gsl_multifit_robust' must
     be called first to perform the regression and calculate these
     statistics.  The returned 'gsl_multifit_robust_stats' structure
     contains the following fields.
          double 'sigma_ols' This contains the standard deviation of the
          residuals as computed from ordinary least squares (OLS).

          double 'sigma_mad' This contains an estimate of the standard
          deviation of the final residuals using the
          Median-Absolute-Deviation statistic

          double 'sigma_rob' This contains an estimate of the standard
          deviation of the final residuals from the theory of robust
          regression (see Street et al, 1988).

          double 'sigma' This contains an estimate of the standard
          deviation of the final residuals by attemping to reconcile
          'sigma_rob' and 'sigma_ols' in a reasonable way.

          double 'Rsq' This contains the R^2 coefficient of
          determination statistic using the estimate 'sigma'.

          double 'adj_Rsq' This contains the adjusted R^2 coefficient of
          determination statistic using the estimate 'sigma'.

          double 'rmse' This contains the root mean squared error of the
          final residuals

          double 'sse' This contains the residual sum of squares taking
          into account the robust covariance matrix.

          size_t 'dof' This contains the number of degrees of freedom n
          - p

          size_t 'numit' Upon successful convergence, this contains the
          number of iterations performed

          gsl_vector * 'weights' This contains the final weight vector
          of length N

          gsl_vector * 'r' This contains the final residual vector of
          length N, r = y - X c


File: gsl-ref.info,  Node: Large Linear Systems,  Next: Troubleshooting,  Prev: Robust linear regression,  Up: Least-Squares Fitting

38.6 Large linear systems
=========================

This module is concerned with solving large dense least squares systems
X c = y where the n-by-p matrix X has n >> p (ie: many more rows than
columns).  This type of matrix is called a "tall skinny" matrix, and for
some applications, it may not be possible to fit the entire matrix in
memory at once to use the standard SVD approach.  Therefore, the
algorithms in this module are designed to allow the user to construct
smaller blocks of the matrix X and accumulate those blocks into the
larger system one at a time.  The algorithms in this module never need
to store the entire matrix X in memory.  The large linear least squares
routines support data weights and Tikhonov regularization, and are
designed to minimize the residual
     \chi^2 = || y - Xc ||_W^2 + \lambda^2 || L c ||^2
   where y is the n-by-1 observation vector, X is the n-by-p design
matrix, c is the p-by-1 solution vector, W = diag(w_1,...,w_n) is the
data weighting matrix, L is an m-by-p regularization matrix, \lambda is
a regularization parameter, and ||r||_W^2 = r^T W r.  In the discussion
which follows, we will assume that the system has been converted into
Tikhonov standard form,
     \chi^2 = || y~ - X~ c~ ||^2 + \lambda^2 || c~ ||^2
   and we will drop the tilde characters from the various parameters.
For a discussion of the transformation to standard form *note
Regularized regression::.

   The basic idea is to partition the matrix X and observation vector y
as into k blocks, where each block (X_i,y_i) may have any number of
rows, but each X_i has p columns.  The sections below describe the
methods available for solving this partitioned system.  The functions
are declared in the header file 'gsl_multilarge.h'.

* Menu:

* Large Linear Systems Normal Equations::
* Large Linear Systems TSQR::
* Large Linear Systems Solution Steps::
* Large Linear Systems Routines::


File: gsl-ref.info,  Node: Large Linear Systems Normal Equations,  Next: Large Linear Systems TSQR,  Up: Large Linear Systems

38.6.1 Normal Equations Approach
--------------------------------

The normal equations approach to the large linear least squares problem
described above is popular due to its speed and simplicity.  Since the
normal equations solution to the problem is given by
     c = ( X^T X + \lambda^2 I )^-1 X^T y
   only the p-by-p matrix X^T X and p-by-1 vector X^T y need to be
stored.  Using the partition scheme described above, these are given by
     X^T X = \sum_i X_i^T X_i
     X^T y = \sum_i X_i^T y_i
   Since the matrix X^T X is symmetric, only half of it needs to be
calculated.  Once all of the blocks (X_i,y_i) have been accumulated into
the final X^T X and X^T y, the system can be solved with a Cholesky
factorization of the X^T X matrix.  The normal equations approach is the
fastest method for solving the large least squares problem, and is
accurate for well-conditioned matricies X.  However, for ill-conditioned
matrices, as is often the case for large systems, this method suffers
from numerical instabilities (see Trefethen and Bau, 1997).  The number
of operations for this method is O(np^2 + p^3/3).


File: gsl-ref.info,  Node: Large Linear Systems TSQR,  Next: Large Linear Systems Solution Steps,  Prev: Large Linear Systems Normal Equations,  Up: Large Linear Systems

38.6.2 Tall Skinny QR (TSQR) Approach
-------------------------------------

An algorithm which has better numerical stability for ill-conditioned
problems is known as the Tall Skinny QR (TSQR) method.  This method is
based on computing the thin QR decomposition of the least squares matrix
X = Q R, where Q is an n-by-p matrix with orthogonal columns, and R is a
p-by-p upper triangular matrix.  Once these factors are calculated, the
residual becomes
     \chi^2 = || Q^T y - R c ||^2 + \lambda^2 || c ||^2
   which can be written as the matrix equation
     [ R ; \lambda I ] c = [ Q^T b ; 0 ]
   The matrix on the left hand side is now a much smaller 2p-by-p matrix
which can be solved with a standard SVD approach.  The Q matrix is just
as large as the original matrix X, however it does not need to be
explicitly constructed.  The TSQR algorithm computes only the p-by-p
matrix R and the p-by-1 vector Q^T y, and updates these quantities as
new blocks are added to the system.  Each time a new block of rows
(X_i,y_i) is added, the algorithm performs a QR decomposition of the
matrix
     [ R_(i-1) ; X_i ]
   where R_{i-1} is the upper triangular R factor for the matrix
     [ X_1 ; ... ; X_(i-1) ]
   This QR decomposition is done efficiently taking into account the
sparse structure of R_{i-1}.  See Demmel et al, 2008 for more details on
how this is accomplished.  The number of operations for this method is
O(2np^2 - {2 \over 3}p^3).


File: gsl-ref.info,  Node: Large Linear Systems Solution Steps,  Next: Large Linear Systems Routines,  Prev: Large Linear Systems TSQR,  Up: Large Linear Systems

38.6.3 Large Linear Systems Solution Steps
------------------------------------------

The typical steps required to solve large regularized linear least
squares problems are as follows:

  1. Choose the regularization matrix L.

  2. Construct a block of rows of the least squares matrix, right hand
     side vector, and weight vector (X_i, y_i, w_i).

  3. Transform the block to standard form (\tilde{X_i},\tilde{y_i}).
     This step can be skipped if L = I and W = I.

  4. Accumulate the standard form block (\tilde{X_i},\tilde{y_i}) into
     the system.

  5. Repeat steps 2-4 until the entire matrix and right hand side vector
     have been accumulated.

  6. Determine an appropriate regularization parameter \lambda (using
     for example L-curve analysis).

  7. Solve the standard form system using the chosen \lambda.

  8. Backtransform the standard form solution \tilde{c} to recover the
     original solution vector c.


File: gsl-ref.info,  Node: Large Linear Systems Routines,  Prev: Large Linear Systems Solution Steps,  Up: Large Linear Systems

38.6.4 Large Linear Least Squares Routines
------------------------------------------

 -- Function: gsl_multilarge_linear_workspace *
          gsl_multilarge_linear_alloc (const gsl_multilarge_linear_type
          * T, const size_t P)
     This function allocates a workspace for solving large linear least
     squares systems.  The least squares matrix X has P columns, but may
     have any number of rows.  The parameter T specifies the method to
     be used for solving the large least squares system and may be
     selected from the following choices

      -- Multilarge type: gsl_multilarge_linear_normal
          This specifies the normal equations approach for solving the
          least squares system.  This method is suitable in cases where
          performance is critical and it is known that the least squares
          matrix X is well conditioned.  The size of this workspace is
          O(p^2).

      -- Multilarge type: gsl_multilarge_linear_tsqr
          This specifies the sequential Tall Skinny QR (TSQR) approach
          for solving the least squares system.  This method is a good
          general purpose choice for large systems, but requires about
          twice as many operations as the normal equations method for n
          >> p.  The size of this workspace is O(p^2).

 -- Function: void gsl_multilarge_linear_free
          (gsl_multilarge_linear_workspace * W)
     This function frees the memory associated with the workspace W.

 -- Function: const char * gsl_multilarge_linear_name
          (gsl_multilarge_linear_workspace * W)
     This function returns a string pointer to the name of the
     multilarge solver.

 -- Function: int gsl_multilarge_linear_reset
          (gsl_multilarge_linear_workspace * W)
     This function resets the workspace W so it can begin to accumulate
     a new least squares system.

 -- Function: int gsl_multilarge_linear_stdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_multilarge_linear_workspace * WORK)
 -- Function: int gsl_multilarge_linear_wstdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * W, const gsl_vector *
          Y, gsl_matrix * XS, gsl_vector * YS,
          gsl_multilarge_linear_workspace * WORK)
     These functions define a regularization matrix L =
     diag(l_0,l_1,...,l_{p-1}).  The diagonal matrix element l_i is
     provided by the ith element of the input vector L.  The block (X,Y)
     is converted to standard form and the parameters
     (\tilde{X},\tilde{y}) are stored in XS and YS on output.  XS and YS
     have the same dimensions as X and Y.  Optional data weights may be
     supplied in the vector W.  In order to apply this transformation,
     L^{-1} must exist and so none of the l_i may be zero.  After the
     standard form system has been solved, use
     'gsl_multilarge_linear_genform1' to recover the original solution
     vector.  It is allowed to have X = XS and Y = YS for an in-place
     transform.

 -- Function: int gsl_multilarge_linear_L_decomp (gsl_matrix * L,
          gsl_vector * TAU)
     This function calculates the QR decomposition of the m-by-p
     regularization matrix L.  L must have m \ge p.  On output, the
     Householder scalars are stored in the vector TAU of size p.  These
     outputs will be used by 'gsl_multilarge_linear_wstdform2' to
     complete the transformation to standard form.

 -- Function: int gsl_multilarge_linear_stdform2 (const gsl_matrix *
          LQR, const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * Y, gsl_matrix * XS, gsl_vector * YS,
          gsl_multilarge_linear_workspace * WORK)
 -- Function: int gsl_multilarge_linear_wstdform2 (const gsl_matrix *
          LQR, const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_multilarge_linear_workspace * WORK)
     These functions convert a block of rows (X,Y,W) to standard form
     (\tilde{X},\tilde{y}) which are stored in XS and YS respectively.
     X, Y, and W must all have the same number of rows.  The m-by-p
     regularization matrix L is specified by the inputs LQR and LTAU,
     which are outputs from 'gsl_multilarge_linear_L_decomp'.  XS and YS
     have the same dimensions as X and Y.  After the standard form
     system has been solved, use 'gsl_multilarge_linear_genform2' to
     recover the original solution vector.  Optional data weights may be
     supplied in the vector W, where W = diag(w).

 -- Function: int gsl_multilarge_linear_accumulate (gsl_matrix * X,
          gsl_vector * Y, gsl_multilarge_linear_workspace * W)
     This function accumulates the standard form block (X,y) into the
     current least squares system.  X and Y have the same number of
     rows, which can be arbitrary.  X must have p columns.  For the TSQR
     method, X and Y are destroyed on output.  For the normal equations
     method, they are both unchanged.

 -- Function: int gsl_multilarge_linear_solve (const double LAMBDA,
          gsl_vector * C, double * RNORM, double * SNORM,
          gsl_multilarge_linear_workspace * W)
     After all blocks (X_i,y_i) have been accumulated into the large
     least squares system, this function will compute the solution
     vector which is stored in C on output.  The regularization
     parameter \lambda is provided in LAMBDA.  On output, RNORM contains
     the residual norm ||y - X c||_W and SNORM contains the solution
     norm ||L c||.

 -- Function: int gsl_multilarge_linear_genform1 (const gsl_vector * L,
          const gsl_vector * CS, gsl_vector * C,
          gsl_multilarge_linear_workspace * WORK)
     After a regularized system has been solved with L =
     diag(\l_0,\l_1,...,\l_{p-1}), this function backtransforms the
     standard form solution vector CS to recover the solution vector of
     the original problem C.  The diagonal matrix elements l_i are
     provided in the vector L.  It is allowed to have C = CS for an
     in-place transform.

 -- Function: int gsl_multilarge_linear_genform2 (const gsl_matrix *
          LQR, const gsl_vector * LTAU, const gsl_vector * CS,
          gsl_vector * C, gsl_multilarge_linear_workspace * WORK)
     After a regularized system has been solved with a regularization
     matrix L, specified by (LQR,LTAU), this function backtransforms the
     standard form solution CS to recover the solution vector of the
     original problem, which is stored in C, of length p.

 -- Function: int gsl_multilarge_linear_rcond (double * RCOND,
          gsl_multilarge_linear_workspace * WORK)
     This function computes the reciprocal condition number, stored in
     RCOND, of the least squares matrix after it has been accumulated
     into the workspace WORK.  For the TSQR algorithm, this is
     accomplished by calculating the SVD of the R factor, which has the
     same singular values as the matrix X.  This routine is currently
     not implemented for the normal equations method.


File: gsl-ref.info,  Node: Troubleshooting,  Next: Fitting Examples for linear regression,  Prev: Large Linear Systems,  Up: Least-Squares Fitting

38.7 Troubleshooting
====================

When using models based on polynomials, care should be taken when
constructing the design matrix X.  If the x values are large, then the
matrix X could be ill-conditioned since its columns are powers of x,
leading to unstable least-squares solutions.  In this case it can often
help to center and scale the x values using the mean and standard
deviation:

     x' = (x - mu)/sigma

and then construct the X matrix using the transformed values x'.


File: gsl-ref.info,  Node: Fitting Examples for linear regression,  Next: Fitting Examples for multi-parameter linear regression,  Prev: Troubleshooting,  Up: Least-Squares Fitting

38.8 Examples for linear regression
===================================

The following program computes a least squares straight-line fit to a
simple dataset, and outputs the best-fit line and its associated one
standard-deviation error bars.

     #include <stdio.h>
     #include <gsl/gsl_fit.h>

     int
     main (void)
     {
       int i, n = 4;
       double x[4] = { 1970, 1980, 1990, 2000 };
       double y[4] = {   12,   11,   14,   13 };
       double w[4] = {  0.1,  0.2,  0.3,  0.4 };

       double c0, c1, cov00, cov01, cov11, chisq;

       gsl_fit_wlinear (x, 1, w, 1, y, 1, n,
                        &c0, &c1, &cov00, &cov01, &cov11,
                        &chisq);

       printf ("# best fit: Y = %g + %g X\n", c0, c1);
       printf ("# covariance matrix:\n");
       printf ("# [ %g, %g\n#   %g, %g]\n",
               cov00, cov01, cov01, cov11);
       printf ("# chisq = %g\n", chisq);

       for (i = 0; i < n; i++)
         printf ("data: %g %g %g\n",
                        x[i], y[i], 1/sqrt(w[i]));

       printf ("\n");

       for (i = -30; i < 130; i++)
         {
           double xf = x[0] + (i/100.0) * (x[n-1] - x[0]);
           double yf, yf_err;

           gsl_fit_linear_est (xf,
                               c0, c1,
                               cov00, cov01, cov11,
                               &yf, &yf_err);

           printf ("fit: %g %g\n", xf, yf);
           printf ("hi : %g %g\n", xf, yf + yf_err);
           printf ("lo : %g %g\n", xf, yf - yf_err);
         }
       return 0;
     }

The following commands extract the data from the output of the program
and display it using the GNU plotutils 'graph' utility,

     $ ./demo > tmp
     $ more tmp
     # best fit: Y = -106.6 + 0.06 X
     # covariance matrix:
     # [ 39602, -19.9
     #   -19.9, 0.01]
     # chisq = 0.8

     $ for n in data fit hi lo ;
        do
          grep "^$n" tmp | cut -d: -f2 > $n ;
        done
     $ graph -T X -X x -Y y -y 0 20 -m 0 -S 2 -Ie data
          -S 0 -I a -m 1 fit -m 2 hi -m 2 lo


File: gsl-ref.info,  Node: Fitting Examples for multi-parameter linear regression,  Next: Fitting Examples for regularized linear regression,  Prev: Fitting Examples for linear regression,  Up: Least-Squares Fitting

38.9 Fitting Examples for multi-parameter linear regression
===========================================================

The following program performs a quadratic fit y = c_0 + c_1 x + c_2 x^2
to a weighted dataset using the generalised linear fitting function
'gsl_multifit_wlinear'.  The model matrix X for a quadratic fit is given
by,

     X = [ 1   , x_0  , x_0^2 ;
           1   , x_1  , x_1^2 ;
           1   , x_2  , x_2^2 ;
           ... , ...  , ...   ]

where the column of ones corresponds to the constant term c_0.  The two
remaining columns corresponds to the terms c_1 x and c_2 x^2.

   The program reads N lines of data in the format (X, Y, ERR) where ERR
is the error (standard deviation) in the value Y.

     #include <stdio.h>
     #include <gsl/gsl_multifit.h>

     int
     main (int argc, char **argv)
     {
       int i, n;
       double xi, yi, ei, chisq;
       gsl_matrix *X, *cov;
       gsl_vector *y, *w, *c;

       if (argc != 2)
         {
           fprintf (stderr,"usage: fit n < data\n");
           exit (-1);
         }

       n = atoi (argv[1]);

       X = gsl_matrix_alloc (n, 3);
       y = gsl_vector_alloc (n);
       w = gsl_vector_alloc (n);

       c = gsl_vector_alloc (3);
       cov = gsl_matrix_alloc (3, 3);

       for (i = 0; i < n; i++)
         {
           int count = fscanf (stdin, "%lg %lg %lg",
                               &xi, &yi, &ei);

           if (count != 3)
             {
               fprintf (stderr, "error reading file\n");
               exit (-1);
             }

           printf ("%g %g +/- %g\n", xi, yi, ei);

           gsl_matrix_set (X, i, 0, 1.0);
           gsl_matrix_set (X, i, 1, xi);
           gsl_matrix_set (X, i, 2, xi*xi);

           gsl_vector_set (y, i, yi);
           gsl_vector_set (w, i, 1.0/(ei*ei));
         }

       {
         gsl_multifit_linear_workspace * work
           = gsl_multifit_linear_alloc (n, 3);
         gsl_multifit_wlinear (X, w, y, c, cov,
                               &chisq, work);
         gsl_multifit_linear_free (work);
       }

     #define C(i) (gsl_vector_get(c,(i)))
     #define COV(i,j) (gsl_matrix_get(cov,(i),(j)))

       {
         printf ("# best fit: Y = %g + %g X + %g X^2\n",
                 C(0), C(1), C(2));

         printf ("# covariance matrix:\n");
         printf ("[ %+.5e, %+.5e, %+.5e  \n",
                    COV(0,0), COV(0,1), COV(0,2));
         printf ("  %+.5e, %+.5e, %+.5e  \n",
                    COV(1,0), COV(1,1), COV(1,2));
         printf ("  %+.5e, %+.5e, %+.5e ]\n",
                    COV(2,0), COV(2,1), COV(2,2));
         printf ("# chisq = %g\n", chisq);
       }

       gsl_matrix_free (X);
       gsl_vector_free (y);
       gsl_vector_free (w);
       gsl_vector_free (c);
       gsl_matrix_free (cov);

       return 0;
     }

A suitable set of data for fitting can be generated using the following
program.  It outputs a set of points with gaussian errors from the curve
y = e^x in the region 0 < x < 2.

     #include <stdio.h>
     #include <math.h>
     #include <gsl/gsl_randist.h>

     int
     main (void)
     {
       double x;
       const gsl_rng_type * T;
       gsl_rng * r;

       gsl_rng_env_setup ();

       T = gsl_rng_default;
       r = gsl_rng_alloc (T);

       for (x = 0.1; x < 2; x+= 0.1)
         {
           double y0 = exp (x);
           double sigma = 0.1 * y0;
           double dy = gsl_ran_gaussian (r, sigma);

           printf ("%g %g %g\n", x, y0 + dy, sigma);
         }

       gsl_rng_free(r);

       return 0;
     }

The data can be prepared by running the resulting executable program,

     $ GSL_RNG_TYPE=mt19937_1999 ./generate > exp.dat
     $ more exp.dat
     0.1 0.97935 0.110517
     0.2 1.3359 0.12214
     0.3 1.52573 0.134986
     0.4 1.60318 0.149182
     0.5 1.81731 0.164872
     0.6 1.92475 0.182212
     ....

To fit the data use the previous program, with the number of data points
given as the first argument.  In this case there are 19 data points.

     $ ./fit 19 < exp.dat
     0.1 0.97935 +/- 0.110517
     0.2 1.3359 +/- 0.12214
     ...
     # best fit: Y = 1.02318 + 0.956201 X + 0.876796 X^2
     # covariance matrix:
     [ +1.25612e-02, -3.64387e-02, +1.94389e-02
       -3.64387e-02, +1.42339e-01, -8.48761e-02
       +1.94389e-02, -8.48761e-02, +5.60243e-02 ]
     # chisq = 23.0987

The parameters of the quadratic fit match the coefficients of the
expansion of e^x, taking into account the errors on the parameters and
the O(x^3) difference between the exponential and quadratic functions
for the larger values of x.  The errors on the parameters are given by
the square-root of the corresponding diagonal elements of the covariance
matrix.  The chi-squared per degree of freedom is 1.4, indicating a
reasonable fit to the data.


File: gsl-ref.info,  Node: Fitting Examples for regularized linear regression,  Next: Fitting Examples for robust linear regression,  Prev: Fitting Examples for multi-parameter linear regression,  Up: Least-Squares Fitting

38.10 Fitting Examples for regularized linear regression
========================================================

The next program demonstrates the difference between ordinary and
regularized least squares when the design matrix is near-singular.  In
this program, we generate two random normally distributed variables u
and v, with v = u + noise so that u and v are nearly colinear.  We then
set a third dependent variable y = u + v + noise and solve for the
coefficients c_1,c_2 of the model Y(c_1,c_2) = c_1 u + c_2 v.  Since u
\approx v, the design matrix X is nearly singular, leading to unstable
ordinary least squares solutions.

Here is the program output:
     matrix condition number = 1.025113e+04
     === Unregularized fit ===
     best fit: y = -43.6588 u + 45.6636 v
     chisq/dof = 1.00213
     === Regularized fit ===
     optimal lambda: 4.51103
     best fit: y = 1.00113 u + 1.0032 v
     chisq/dof = 1.04499

We see that the regularized method with the optimal \lambda = 4.51103
finds the correct solution c_1 \approx c_2 \approx 1, while the ordinary
least squares solution is completely wrong.  The optimal regularization
parameter \lambda is found by calculating the L-curve and finding its
corner.  The L-curve and its computed corner are plotted below.

The program is given below.
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_matrix.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_multifit.h>

     int
     main()
     {
       const size_t n = 1000; /* number of observations */
       const size_t p = 2;    /* number of model parameters */
       size_t i;
       gsl_rng *r = gsl_rng_alloc(gsl_rng_default);
       gsl_matrix *X = gsl_matrix_alloc(n, p);
       gsl_vector *y = gsl_vector_alloc(n);

       for (i = 0; i < n; ++i)
         {
           /* generate first random variable u */
           double ui = 5.0 * gsl_ran_gaussian(r, 1.0);

           /* set v = u + noise */
           double vi = ui + gsl_ran_gaussian(r, 0.001);

           /* set y = u + v + noise */
           double yi = ui + vi + gsl_ran_gaussian(r, 1.0);

           /* since u =~ v, the matrix X is ill-conditioned */
           gsl_matrix_set(X, i, 0, ui);
           gsl_matrix_set(X, i, 1, vi);

           /* rhs vector */
           gsl_vector_set(y, i, yi);
         }

       {
         const size_t nL = 200;                   /* number of points on L-curve */
         gsl_multifit_linear_workspace *w =
           gsl_multifit_linear_alloc(n, p);
         gsl_vector *c = gsl_vector_alloc(p);     /* OLS solution */
         gsl_vector *c_reg = gsl_vector_alloc(p); /* regularized solution */
         gsl_vector *reg_param = gsl_vector_alloc(nL);
         gsl_vector *rho = gsl_vector_alloc(nL);  /* residual norms */
         gsl_vector *eta = gsl_vector_alloc(nL);  /* solution norms */
         double lambda;                           /* optimal regularization parameter */
         size_t reg_idx;                          /* index of optimal lambda */
         double rcond;                            /* reciprocal condition number of X */
         double chisq, rnorm, snorm;

         /* compute SVD of X */
         gsl_multifit_linear_svd(X, w);

         rcond = gsl_multifit_linear_rcond(w);
         fprintf(stderr, "matrix condition number = %e\n", 1.0 / rcond);

         /* unregularized (standard) least squares fit, lambda = 0 */
         gsl_multifit_linear_solve(0.0, X, y, c, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0);

         fprintf(stderr, "=== Unregularized fit ===\n");
         fprintf(stderr, "best fit: y = %g u + %g v\n",
           gsl_vector_get(c, 0), gsl_vector_get(c, 1));
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* calculate L-curve and find its corner */
         gsl_multifit_linear_lcurve(y, reg_param, rho, eta, w);
         gsl_multifit_linear_lcorner(rho, eta, &reg_idx);

         /* store optimal regularization parameter */
         lambda = gsl_vector_get(reg_param, reg_idx);

         /* output L-curve */
         for (i = 0; i < nL; ++i)
           printf("%f %f\n", gsl_vector_get(rho, i), gsl_vector_get(eta, i));

         /* output L-curve corner point */
         printf("\n\n%f %f\n",
                gsl_vector_get(rho, reg_idx),
                gsl_vector_get(eta, reg_idx));

         /* regularize with lambda */
         gsl_multifit_linear_solve(lambda, X, y, c_reg, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0) + pow(lambda * snorm, 2.0);

         fprintf(stderr, "=== Regularized fit ===\n");
         fprintf(stderr, "optimal lambda: %g\n", lambda);
         fprintf(stderr, "best fit: y = %g u + %g v\n",
                 gsl_vector_get(c_reg, 0), gsl_vector_get(c_reg, 1));
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         gsl_multifit_linear_free(w);
         gsl_vector_free(c);
         gsl_vector_free(c_reg);
         gsl_vector_free(reg_param);
         gsl_vector_free(rho);
         gsl_vector_free(eta);
       }

       gsl_rng_free(r);
       gsl_matrix_free(X);
       gsl_vector_free(y);

       return 0;
     }


File: gsl-ref.info,  Node: Fitting Examples for robust linear regression,  Next: Fitting Examples for large linear systems,  Prev: Fitting Examples for regularized linear regression,  Up: Least-Squares Fitting

38.11 Fitting Examples for robust linear regression
===================================================

The next program demonstrates the advantage of robust least squares on a
dataset with outliers.  The program generates linear (x,y) data pairs on
the line y = 1.45 x + 3.88, adds some random noise, and inserts 3
outliers into the dataset.  Both the robust and ordinary least squares
(OLS) coefficients are computed for comparison.

     #include <stdio.h>
     #include <gsl/gsl_multifit.h>
     #include <gsl/gsl_randist.h>

     int
     dofit(const gsl_multifit_robust_type *T,
           const gsl_matrix *X, const gsl_vector *y,
           gsl_vector *c, gsl_matrix *cov)
     {
       int s;
       gsl_multifit_robust_workspace * work
         = gsl_multifit_robust_alloc (T, X->size1, X->size2);

       s = gsl_multifit_robust (X, y, c, cov, work);
       gsl_multifit_robust_free (work);

       return s;
     }

     int
     main (int argc, char **argv)
     {
       size_t i;
       size_t n;
       const size_t p = 2; /* linear fit */
       gsl_matrix *X, *cov;
       gsl_vector *x, *y, *c, *c_ols;
       const double a = 1.45; /* slope */
       const double b = 3.88; /* intercept */
       gsl_rng *r;

       if (argc != 2)
         {
           fprintf (stderr,"usage: robfit n\n");
           exit (-1);
         }

       n = atoi (argv[1]);

       X = gsl_matrix_alloc (n, p);
       x = gsl_vector_alloc (n);
       y = gsl_vector_alloc (n);

       c = gsl_vector_alloc (p);
       c_ols = gsl_vector_alloc (p);
       cov = gsl_matrix_alloc (p, p);

       r = gsl_rng_alloc(gsl_rng_default);

       /* generate linear dataset */
       for (i = 0; i < n - 3; i++)
         {
           double dx = 10.0 / (n - 1.0);
           double ei = gsl_rng_uniform(r);
           double xi = -5.0 + i * dx;
           double yi = a * xi + b;

           gsl_vector_set (x, i, xi);
           gsl_vector_set (y, i, yi + ei);
         }

       /* add a few outliers */
       gsl_vector_set(x, n - 3, 4.7);
       gsl_vector_set(y, n - 3, -8.3);

       gsl_vector_set(x, n - 2, 3.5);
       gsl_vector_set(y, n - 2, -6.7);

       gsl_vector_set(x, n - 1, 4.1);
       gsl_vector_set(y, n - 1, -6.0);

       /* construct design matrix X for linear fit */
       for (i = 0; i < n; ++i)
         {
           double xi = gsl_vector_get(x, i);

           gsl_matrix_set (X, i, 0, 1.0);
           gsl_matrix_set (X, i, 1, xi);
         }

       /* perform robust and OLS fit */
       dofit(gsl_multifit_robust_ols, X, y, c_ols, cov);
       dofit(gsl_multifit_robust_bisquare, X, y, c, cov);

       /* output data and model */
       for (i = 0; i < n; ++i)
         {
           double xi = gsl_vector_get(x, i);
           double yi = gsl_vector_get(y, i);
           gsl_vector_view v = gsl_matrix_row(X, i);
           double y_ols, y_rob, y_err;

           gsl_multifit_robust_est(&v.vector, c, cov, &y_rob, &y_err);
           gsl_multifit_robust_est(&v.vector, c_ols, cov, &y_ols, &y_err);

           printf("%g %g %g %g\n", xi, yi, y_rob, y_ols);
         }

     #define C(i) (gsl_vector_get(c,(i)))
     #define COV(i,j) (gsl_matrix_get(cov,(i),(j)))

       {
         printf ("# best fit: Y = %g + %g X\n",
                 C(0), C(1));

         printf ("# covariance matrix:\n");
         printf ("# [ %+.5e, %+.5e\n",
                    COV(0,0), COV(0,1));
         printf ("#   %+.5e, %+.5e\n",
                    COV(1,0), COV(1,1));
       }

       gsl_matrix_free (X);
       gsl_vector_free (x);
       gsl_vector_free (y);
       gsl_vector_free (c);
       gsl_vector_free (c_ols);
       gsl_matrix_free (cov);
       gsl_rng_free(r);

       return 0;
     }

   The output from the program is shown in the following plot.


File: gsl-ref.info,  Node: Fitting Examples for large linear systems,  Next: Fitting References and Further Reading,  Prev: Fitting Examples for robust linear regression,  Up: Least-Squares Fitting

38.12 Fitting Examples for large linear systems
===============================================

The following program demostrates the large linear least squares
solvers.  This example is adapted from Trefethen and Bau, and fits the
function f(t) = \exp{(\sin{4(t - 10)}}) on the interval [10,11] with a
degree 14 polynomial.  The program generates n = 50000 equally spaced
points t_i on this interval, calculates the function value and adds
random noise to determine the observation value y_i.  The entries of the
least squares matrix are X_{ij} = t_i^j, representing a polynomial fit.
The matrix is highly ill-conditioned, and we accumulate the matrix into
the least squares system in 5 blocks, each with 10000 rows.  This way
the full matrix X is never stored in memory.  We solve the system with
both the normal equations and TSQR methods.  The results are shown in
the plot below.  In the left plot, we see the unregularized normal
equations solution has large error due to the ill-conditioning of the
matrix, while TSQR provides a good fit.  In the right panel, we add
regularization with \lambda = 1 which allows the normal equations method
to converge to a good solution.

     #include <gsl/gsl_math.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_matrix.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_multifit.h>
     #include <gsl/gsl_multilarge.h>
     #include <gsl/gsl_blas.h>

     /* function to be fitted */
     double
     func(const double t)
     {
       return exp(sin(4.0 * (t - 10.0)));
     }

     /* construct a row of the least squares matrix */
     int
     build_row(const double t, gsl_vector *row)
     {
       const size_t p = row->size;
       double Xj = 1.0;
       size_t j;

       for (j = 0; j < p; ++j)
         {
           gsl_vector_set(row, j, Xj);
           Xj *= t;
         }

       return 0;
     }

     int
     solve_system(const gsl_multilarge_linear_type * T,
                  const double lambda, const size_t n, const size_t p,
                  gsl_vector * c)
     {
       const size_t nblock = 5;         /* number of blocks to accumulate */
       const size_t nrows = n / nblock; /* number of rows per block */
       gsl_multilarge_linear_workspace * w =
         gsl_multilarge_linear_alloc(T, p);
       gsl_matrix *X = gsl_matrix_alloc(nrows, p);
       gsl_vector *y = gsl_vector_alloc(nrows);
       gsl_rng *r = gsl_rng_alloc(gsl_rng_default);
       size_t rowidx = 0;
       double rnorm, snorm, rcond;
       double t = 10.0;
       double dt = 1.0 / (n - 1.0);

       while (rowidx < n)
         {
           size_t nleft = n - rowidx;         /* number of rows left to accumulate */
           size_t nr = GSL_MIN(nrows, nleft); /* number of rows in this block */
           gsl_matrix_view Xv = gsl_matrix_submatrix(X, 0, 0, nr, p);
           gsl_vector_view yv = gsl_vector_subvector(y, 0, nr);
           size_t i;

           /* build (X,y) block with 'nr' rows */
           for (i = 0; i < nr; ++i)
             {
               gsl_vector_view row = gsl_matrix_row(&Xv.matrix, i);
               double yi = func(t);
               double ei = gsl_ran_gaussian (r, 0.3 * yi); /* noise */

               /* construct this row of LS matrix */
               build_row(t, &row.vector);

               /* set right hand side value with added noise */
               gsl_vector_set(&yv.vector, i, yi + ei);

               t += dt;
             }

           /* accumulate (X,y) block into LS system */
           gsl_multilarge_linear_accumulate(&Xv.matrix, &yv.vector, w);

           rowidx += nr;
         }

       /* solve large LS system and store solution in c */
       gsl_multilarge_linear_solve(lambda, c, &rnorm, &snorm, w);

       /* compute reciprocal condition number */
       gsl_multilarge_linear_rcond(&rcond, w);

       fprintf(stderr, "=== Method %s ===\n", gsl_multilarge_linear_name(w));
       if (rcond != 0.0)
         fprintf(stderr, "matrix condition number = %e\n", 1.0 / rcond);
       fprintf(stderr, "residual norm  = %e\n", rnorm);
       fprintf(stderr, "solution norm  = %e\n", snorm);

       gsl_matrix_free(X);
       gsl_vector_free(y);
       gsl_multilarge_linear_free(w);
       gsl_rng_free(r);

       return 0;
     }

     int
     main(int argc, char *argv[])
     {
       const size_t n = 50000;   /* number of observations */
       const size_t p = 15;      /* polynomial order + 1 */
       double lambda = 0.0;      /* regularization parameter */
       gsl_vector *c_tsqr = gsl_vector_alloc(p);
       gsl_vector *c_normal = gsl_vector_alloc(p);

       if (argc > 1)
         lambda = atof(argv[1]);

       /* solve system with TSQR method */
       solve_system(gsl_multilarge_linear_tsqr, lambda, n, p, c_tsqr);

       /* solve system with Normal equations method */
       solve_system(gsl_multilarge_linear_normal, lambda, n, p, c_normal);

       /* output solutions */
       {
         gsl_vector *v = gsl_vector_alloc(p);
         double t;

         for (t = 10.0; t <= 11.0; t += 0.01)
           {
             double f_exact = func(t);
             double f_tsqr, f_normal;

             build_row(t, v);
             gsl_blas_ddot(v, c_tsqr, &f_tsqr);
             gsl_blas_ddot(v, c_normal, &f_normal);

             printf("%f %e %e %e\n", t, f_exact, f_tsqr, f_normal);
           }

         gsl_vector_free(v);
       }

       gsl_vector_free(c_tsqr);
       gsl_vector_free(c_normal);

       return 0;
     }


File: gsl-ref.info,  Node: Fitting References and Further Reading,  Prev: Fitting Examples for large linear systems,  Up: Least-Squares Fitting

38.13 References and Further Reading
====================================

A summary of formulas and techniques for least squares fitting can be
found in the "Statistics" chapter of the Annual Review of Particle
Physics prepared by the Particle Data Group,

     'Review of Particle Properties', R.M. Barnett et al., Physical
     Review D54, 1 (1996) <http://pdg.lbl.gov/>

The Review of Particle Physics is available online at the website given
above.

   The tests used to prepare these routines are based on the NIST
Statistical Reference Datasets.  The datasets and their documentation
are available from NIST at the following website,

           <http://www.nist.gov/itl/div898/strd/index.html>.

More information on Tikhonov regularization can be found in

     Hansen, P. C. (1998), Rank-Deficient and Discrete Ill-Posed
     Problems: Numerical Aspects of Linear Inversion.  SIAM Monogr.  on
     Mathematical Modeling and Computation, Society for Industrial and
     Applied Mathematics

     M. Rezghi and S. M. Hosseini (2009), A new variant of L-curve for
     Tikhonov regularization, Journal of Computational and Applied
     Mathematics, Volume 231, Issue 2, pages 914-924.

The GSL implementation of robust linear regression closely follows the
publications

     DuMouchel, W. and F. O'Brien (1989), "Integrating a robust option
     into a multiple regression computing environment," Computer Science
     and Statistics: Proceedings of the 21st Symposium on the Interface,
     American Statistical Association

     Street, J.O., R.J. Carroll, and D. Ruppert (1988), "A note on
     computing robust regression estimates via iteratively reweighted
     least squares," The American Statistician, v.  42, pp.  152-154.

More information about the normal equations and TSQR approach for
solving large linear least squares systems can be found in the
publications

     Trefethen, L. N. and Bau, D. (1997), "Numerical Linear Algebra",
     SIAM.

     Demmel, J., Grigori, L., Hoemmen, M. F., and Langou, J.
     "Communication-optimal parallel and sequential QR and LU
     factorizations", UCB Technical Report No.  UCB/EECS-2008-89, 2008.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Fitting,  Next: Basis Splines,  Prev: Least-Squares Fitting,  Up: Top

39 Nonlinear Least-Squares Fitting
**********************************

This chapter describes functions for multidimensional nonlinear
least-squares fitting.  The library provides low level components for a
variety of iterative solvers and convergence tests.  These can be
combined by the user to achieve the desired solution, with full access
to the intermediate steps of the iteration.  Each class of methods uses
the same framework, so that you can switch between solvers at runtime
without needing to recompile your program.  Each instance of a solver
keeps track of its own state, allowing the solvers to be used in
multi-threaded programs.

   The header file 'gsl_multifit_nlin.h' contains prototypes for the
multidimensional nonlinear fitting functions and related declarations.

* Menu:

* Overview of Nonlinear Least-Squares Fitting::  
* Overview of Weighted Nonlinear Least-Squares Fitting::  
* Overview of Regularized Nonlinear Least-Squares Fitting::  
* Initializing the Nonlinear Least-Squares Solver::  
* Providing the Function to be Minimized::  
* Finite Difference Jacobian::  
* Iteration of the Minimization Algorithm::  
* Search Stopping Parameters for Minimization Algorithms::  
* High Level Driver::
* Minimization Algorithms using Derivatives::  
* Minimization Algorithms without Derivatives::  
* Computing the covariance matrix of best fit parameters::  
* Troubleshooting Nonlinear Least Squares::
* Example programs for Nonlinear Least-Squares Fitting::  
* References and Further Reading for Nonlinear Least-Squares Fitting::  


File: gsl-ref.info,  Node: Overview of Nonlinear Least-Squares Fitting,  Next: Overview of Weighted Nonlinear Least-Squares Fitting,  Up: Nonlinear Least-Squares Fitting

39.1 Overview
=============

The problem of multidimensional nonlinear least-squares fitting requires
the minimization of the squared residuals of n functions, f_i, in p
parameters, x_i,

     \Phi(x) = (1/2) || f(x) ||^2
             = (1/2) \sum_{i=1}^{n} f_i(x_1, ..., x_p)^2

All algorithms proceed from an initial guess using the linearization,

     \psi(\delta) = || f(x+\delta) || ~=~ || f(x) + J \delta ||

where x is the initial point, \delta is the proposed step and J is the
Jacobian matrix J_{ij} = d f_i / d x_j.  Additional strategies are used
to enlarge the region of convergence.  These include requiring a
decrease in the norm ||f|| on each step or using a trust region to avoid
steps which fall outside the linear regime.

Note that the model parameters are denoted by x in this chapter since
the non-linear least-squares algorithms are described geometrically
(i.e.  finding the minimum of a surface).  The independent variable of
any data to be fitted is denoted by t.


File: gsl-ref.info,  Node: Overview of Weighted Nonlinear Least-Squares Fitting,  Next: Overview of Regularized Nonlinear Least-Squares Fitting,  Prev: Overview of Nonlinear Least-Squares Fitting,  Up: Nonlinear Least-Squares Fitting

39.2 Weighted Nonlinear Least-Squares
=====================================

Weighted nonlinear least-squares fitting minimizes the function

     \Phi(x) = (1/2) || f(x) ||^2
             = (1/2) \sum_{i=1}^{n} f_i(x_1, ..., x_p)^2

   where W = diag(w_1,w_2,...,w_n) is the weighting matrix, and the
weights w_i are commonly defined as w_i = 1/\sigma_i^2, where \sigma_i
is the error in the ith measurement.  A simple change of variables
\tilde{f} = \sqrt{W} f yields \Phi(x) = {1 \over 2} ||\tilde{f}||^2,
which is in the same form as the unweighted case.  The user can either
perform this transform directly on their function residuals and
Jacobian, or use the 'gsl_multifit_fdfsolver_wset' interface which
automatically performs the correct scaling.  To manually perform this
transformation, the residuals and Jacobian should be modified according
to

     f_i = (Y(x, t_i) - y_i) / \sigma_i

   where Y_i = Y(x,t_i).


File: gsl-ref.info,  Node: Overview of Regularized Nonlinear Least-Squares Fitting,  Next: Initializing the Nonlinear Least-Squares Solver,  Prev: Overview of Weighted Nonlinear Least-Squares Fitting,  Up: Nonlinear Least-Squares Fitting

39.3 Regularized Nonlinear Least-Squares
========================================

In cases where the Jacobian J is rank-deficient or singular, standard
nonlinear least squares can sometimes produce undesirable and unstable
solutions.  In these cases, it can help to regularize the problem using
ridge or Tikhonov regularization.  In this method, we introduce a term
in our minimization function which is designed to damp the solution
vector x, or give preference to solutions with smaller norms.

     \Phi(x) = (1/2) || f(x) ||^2 + (1/2) ||Lx||^2

   Here, the regularization matrix L is often set as L = \lambda I, for
a positive scalar \lambda, but can also be a general m-by-p (where m is
any number of rows) matrix depending on the structure of the problem to
be solved.  If we define a new (n+m)-by-1 vector

     f~(x) = [ f(x); Lx ]

   or, in the weighted case,

     f~(x) = [ sqrt(W) f(x); Lx ]

   then

     \Phi(x) = (1/2) || f~(x) ||^2

   which is in the same form as the standard nonlinear least squares
problem.  The corresponding (n+m)-by-p Jacobian matrix is

     F~(x) = [ J; L ]

   or for weighted systems

     F~(x) = [ sqrt(W) J; L ]

   While the user could explicitly form the \tilde{f}(x) vector and
\tilde{J} matrix, the 'fdfridge' interface described below allows the
user to specify the original data vector f(x), Jacobian J,
regularization matrix L, and optional weighting matrix W, and
automatically forms \tilde{f}(x) and \tilde{J} to solve the system.
This allows switching between regularized and non-regularized solutions
with minimal code changes.


File: gsl-ref.info,  Node: Initializing the Nonlinear Least-Squares Solver,  Next: Providing the Function to be Minimized,  Prev: Overview of Regularized Nonlinear Least-Squares Fitting,  Up: Nonlinear Least-Squares Fitting

39.4 Initializing the Solver
============================

 -- Function: gsl_multifit_fsolver * gsl_multifit_fsolver_alloc (const
          gsl_multifit_fsolver_type * T, size_t N, size_t P)
     This function returns a pointer to a newly allocated instance of a
     solver of type T for N observations and P parameters.  The number
     of observations N must be greater than or equal to parameters P.

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_multifit_fdfsolver * gsl_multifit_fdfsolver_alloc
          (const gsl_multifit_fdfsolver_type * T, size_t N, size_t P)
     This function returns a pointer to a newly allocated instance of a
     derivative solver of type T for N observations and P parameters.
     For example, the following code creates an instance of a
     Levenberg-Marquardt solver for 100 data points and 3 parameters,

          const gsl_multifit_fdfsolver_type * T
              = gsl_multifit_fdfsolver_lmder;
          gsl_multifit_fdfsolver * s
              = gsl_multifit_fdfsolver_alloc (T, 100, 3);

     The number of observations N must be greater than or equal to
     parameters P.

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_multifit_fdfridge * gsl_multifit_fdfridge_alloc (const
          gsl_multifit_fdfsolver_type * T, size_t N, size_t P)
     This function returns a pointer to a newly allocated instance of a
     derivative solver of type T for N observations and P parameters.
     The solver will automatically form the augmented system
     \tilde{f}(x) and \tilde{J} for ridge (Tikhonov) regression.  If
     there is insufficient memory to create the solver then the function
     returns a null pointer and the error handler is invoked with an
     error code of 'GSL_ENOMEM'.

 -- Function: int gsl_multifit_fsolver_set (gsl_multifit_fsolver * S,
          gsl_multifit_function * F, const gsl_vector * X)
     This function initializes, or reinitializes, an existing solver S
     to use the function F and the initial guess X.

 -- Function: int gsl_multifit_fdfsolver_set (gsl_multifit_fdfsolver *
          S, gsl_multifit_function_fdf * FDF, const gsl_vector * X)
 -- Function: int gsl_multifit_fdfsolver_wset (gsl_multifit_fdfsolver *
          S, gsl_multifit_function_fdf * FDF, const gsl_vector * X,
          const gsl_vector * WTS)
     These functions initialize, or reinitialize, an existing solver S
     to use the function and derivative FDF and the initial guess X.

     Optionally, a weight vector WTS can be given to perform a weighted
     nonlinear regression.  Here, the weighting matrix is W =
     diag(w_1,w_2,...,w_n).  The WTS vector is referenced throughout the
     iteration so it should not be freed by the caller until the
     iteration terminates.

 -- Function: int gsl_multifit_fdfridge_set (gsl_multifit_fdfridge * S,
          gsl_multifit_function_fdf * FDF, const gsl_vector * X, const
          double LAMBDA)
 -- Function: int gsl_multifit_fdfridge_wset (gsl_multifit_fdfridge * S,
          gsl_multifit_function_fdf * FDF, const gsl_vector * X, const
          double LAMBDA, const gsl_vector * WTS)
     This function initializes, or reinitializes, an existing ridge
     solver S to use the function and derivative FDF and the initial
     guess X.  Here, the regularization matrix is set to L = \lambda I,
     with \lambda specified in LAMBDA.

     Optionally, a weight vector WTS can be given to perform a weighted
     nonlinear regression.  Here, the weighting matrix is W =
     diag(w_1,w_2,...,w_n).  The WTS vector is referenced throughout the
     iteration so it should not be freed by the caller until the
     iteration terminates.

 -- Function: int gsl_multifit_fdfridge_set2 (gsl_multifit_fdfridge * S,
          gsl_multifit_function_fdf * FDF, const gsl_vector * X, const
          gsl_vector * LAMBDA)
 -- Function: int gsl_multifit_fdfridge_wset2 (gsl_multifit_fdfridge *
          S, gsl_multifit_function_fdf * FDF, const gsl_vector * X,
          const gsl_vector * LAMBDA, const gsl_vector * WTS)
     This function initializes, or reinitializes, an existing ridge
     solver S to use the function and derivative FDF and the initial
     guess X.  Here, the regularization matrix is set to L =
     diag(\lambda_1,\lambda_2,...,\lambda_p), where the \lambda_i are
     given in LAMBDA.

     Optionally, a weight vector WTS can be given to perform a weighted
     nonlinear regression.  Here, the weighting matrix is W =
     diag(w_1,w_2,...,w_n).  The WTS vector is referenced throughout the
     iteration so it should not be freed by the caller until the
     iteration terminates.

 -- Function: int gsl_multifit_fdfridge_set3 (gsl_multifit_fdfridge * S,
          gsl_multifit_function_fdf * FDF, const gsl_vector * X, const
          gsl_matrix * L)
 -- Function: int gsl_multifit_fdfridge_wset3 (gsl_multifit_fdfridge *
          S, gsl_multifit_function_fdf * FDF, const gsl_vector * X,
          const gsl_matrix * L, const gsl_vector * WTS)
     This function initializes, or reinitializes, an existing ridge
     solver S to use the function and derivative FDF and the initial
     guess X.  Here, the regularization matrix is set to L, which must
     have p columns but may have any number of rows.

     Optionally, a weight vector WTS can be given to perform a weighted
     nonlinear regression.  Here, the weighting matrix is W =
     diag(w_1,w_2,...,w_n).  The WTS vector is referenced throughout the
     iteration so it should not be freed by the caller until the
     iteration terminates.

 -- Function: void gsl_multifit_fsolver_free (gsl_multifit_fsolver * S)
 -- Function: void gsl_multifit_fdfsolver_free (gsl_multifit_fdfsolver *
          S)
 -- Function: void gsl_multifit_fdfridge_free (gsl_multifit_fdfridge *
          S)
     These functions free all the memory associated with the solver S.

 -- Function: const char * gsl_multifit_fsolver_name (const
          gsl_multifit_fsolver * S)
 -- Function: const char * gsl_multifit_fdfsolver_name (const
          gsl_multifit_fdfsolver * S)
 -- Function: const char * gsl_multifit_fdfridge_name (const
          gsl_multifit_fdfridge * S)
     These functions return a pointer to the name of the solver.  For
     example,

          printf ("s is a '%s' solver\n",
                  gsl_multifit_fdfsolver_name (s));

     would print something like 's is a 'lmder' solver'.


File: gsl-ref.info,  Node: Providing the Function to be Minimized,  Next: Finite Difference Jacobian,  Prev: Initializing the Nonlinear Least-Squares Solver,  Up: Nonlinear Least-Squares Fitting

39.5 Providing the Function to be Minimized
===========================================

You must provide n functions of p variables for the minimization
algorithms to operate on.  In order to allow for arbitrary parameters
the functions are defined by the following data types:

 -- Data Type: gsl_multifit_function
     This data type defines a general system of functions with arbitrary
     parameters.

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          this function should store the vector result f(x,params) in F
          for argument X and arbitrary parameters PARAMS, returning an
          appropriate error code if the function cannot be computed.

     'size_t n'
          the number of functions, i.e.  the number of components of the
          vector F.

     'size_t p'
          the number of independent variables, i.e.  the number of
          components of the vector X.

     'void * params'
          a pointer to the arbitrary parameters of the function.

 -- Data Type: gsl_multifit_function_fdf
     This data type defines a general system of functions with arbitrary
     parameters and the corresponding Jacobian matrix of derivatives,

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          this function should store the vector result f(x,params) in F
          for argument X and arbitrary parameters PARAMS, returning an
          appropriate error code if the function cannot be computed.

     'int (* df) (const gsl_vector * X, void * PARAMS, gsl_matrix * J)'
          this function should store the N-by-P matrix result J_ij = d
          f_i(x,params) / d x_j in J for argument X and arbitrary
          parameters PARAMS, returning an appropriate error code if the
          function cannot be computed.  If an analytic Jacobian is
          unavailable, or too expensive to compute, this function
          pointer may be set to NULL, in which case the Jacobian will be
          internally computed using finite difference approximations of
          the function F.

     'size_t n'
          the number of functions, i.e.  the number of components of the
          vector F.

     'size_t p'
          the number of independent variables, i.e.  the number of
          components of the vector X.

     'void * params'
          a pointer to the arbitrary parameters of the function.

     'size_t nevalf'
          This does not need to be set by the user.  It counts the
          number of function evaluations and is initialized by the
          '_set' function.

     'size_t nevaldf'
          This does not need to be set by the user.  It counts the
          number of Jacobian evaluations and is initialized by the
          '_set' function.

   Note that when fitting a non-linear model against experimental data,
the data is passed to the functions above using the PARAMS argument and
the trial best-fit parameters through the X argument.


File: gsl-ref.info,  Node: Finite Difference Jacobian,  Next: Iteration of the Minimization Algorithm,  Prev: Providing the Function to be Minimized,  Up: Nonlinear Least-Squares Fitting

39.6 Finite Difference Jacobian
===============================

For the algorithms which require a Jacobian matrix of derivatives of the
fit functions, there are times when an analytic Jacobian may be
unavailable or too expensive to compute.  Therefore GSL supports
approximating the Jacobian numerically using finite differences of the
fit functions.  This is typically done by setting the relevant function
pointers of the 'gsl_multifit_function_fdf' data type to NULL, however
the following functions allow the user to access the approximate
Jacobian directly if needed.

 -- Function: int gsl_multifit_fdfsolver_dif_df (const gsl_vector * X,
          const gsl_vector * WTS, gsl_multifit_function_fdf * FDF, const
          gsl_vector * F, gsl_matrix * J)
     This function takes as input the current position X, weight vector
     WTS and function values computed at the current position F, along
     with FDF which specifies the fit function and parameters and
     approximates the N-by-P Jacobian J using forward finite
     differences: J_ij = sqrt(w_i) d f_i(x) / d x_j = sqrt(w_i) (f_i(x +
     h_j e_j) - f_i(x)) / h_j.  where h_j = \epsilon |x_j|, and \epsilon
     is the square root of the machine precision 'GSL_DBL_EPSILON'.

 -- Function: int gsl_multifit_fdfsolver_dif_fdf (const gsl_vector * X,
          gsl_multifit_function_fdf * FDF, gsl_vector * F, gsl_matrix *
          J)
     This function is deprecated and will be removed in a future
     release.


File: gsl-ref.info,  Node: Iteration of the Minimization Algorithm,  Next: Search Stopping Parameters for Minimization Algorithms,  Prev: Finite Difference Jacobian,  Up: Nonlinear Least-Squares Fitting

39.7 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any solver of the
corresponding type.  The same functions work for all solvers so that
different methods can be substituted at runtime without modifications to
the code.

 -- Function: int gsl_multifit_fsolver_iterate (gsl_multifit_fsolver *
          S)
 -- Function: int gsl_multifit_fdfsolver_iterate (gsl_multifit_fdfsolver
          * S)
 -- Function: int gsl_multifit_fdfridge_iterate (gsl_multifit_fdfridge *
          S)
     These functions perform a single iteration of the solver S.  If the
     iteration encounters an unexpected problem then an error code will
     be returned.  The solver maintains a current estimate of the
     best-fit parameters at all times.

   The solver struct S contains the following entries, which can be used
to track the progress of the solution:

'gsl_vector * x'
     The current position.

'gsl_vector * f'
     The function residual vector at the current position f(x).

'gsl_vector * dx'
     The difference between the current position and the previous
     position, i.e.  the last step \delta, taken as a vector.

   The best-fit information also can be accessed with the following
auxiliary functions,

 -- Function: gsl_vector * gsl_multifit_fsolver_position (const
          gsl_multifit_fsolver * S)
 -- Function: gsl_vector * gsl_multifit_fdfsolver_position (const
          gsl_multifit_fdfsolver * S)
 -- Function: gsl_vector * gsl_multifit_fdfridge_position (const
          gsl_multifit_fdfridge * S)
     These functions return the current position x (i.e.  best-fit
     parameters) of the solver S.

 -- Function: gsl_vector * gsl_multifit_fdfsolver_residual (const
          gsl_multifit_fdfsolver * S)
 -- Function: gsl_vector * gsl_multifit_fdfridge_residual (const
          gsl_multifit_fdfridge * S)
     These functions return the current residual vector f of the solver
     S.  For weighted cases, the residual vector includes the weighting
     factor \sqrt{W}.  For ridge regression, the residual vector is the
     augmented vector \tilde{f}.

 -- Function: size_t gsl_multifit_fdfsolver_niter (const
          gsl_multifit_fdfsolver * S)
 -- Function: size_t gsl_multifit_fdfridge_niter (const
          gsl_multifit_fdfridge * S)
     These functions return the number of iterations performed so far.
     The iteration counter is updated on each call to the '_iterate'
     functions above, and reset to 0 in the '_set' functions.


File: gsl-ref.info,  Node: Search Stopping Parameters for Minimization Algorithms,  Next: High Level Driver,  Prev: Iteration of the Minimization Algorithm,  Up: Nonlinear Least-Squares Fitting

39.8 Search Stopping Parameters
===============================

A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the current estimate of the best-fit
parameters in several standard ways.

 -- Function: int gsl_multifit_fdfsolver_test (const
          gsl_multifit_fdfsolver * S, const double XTOL, const double
          GTOL, const double FTOL, int * INFO)
     This function tests for convergence of the minimization method
     using the following criteria:

        * Testing for a small step size relative to the current
          parameter vector
               |\delta_i| <= xtol (|x_i| + xtol)
          for each 0 <= i < p.  Each element of the step vector \delta
          is tested individually in case the different parameters have
          widely different scales.  Adding XTOL to |x_i| helps the test
          avoid breaking down in situations where the true solution
          value x_i = 0.  If this test succeeds, INFO is set to 1 and
          the function returns 'GSL_SUCCESS'.

          A general guideline for selecting the step tolerance is to
          choose xtol = 10^{-d} where d is the number of accurate
          decimal digits desired in the solution x.  See Dennis and
          Schnabel for more information.

        * Testing for a small gradient (g = \nabla \Phi(x) = J^T f)
          indicating a local function minimum:
               ||g||_inf <= gtol
          This expression tests whether the ratio (\nabla \Phi)_i x_i /
          \Phi is small.  Testing this scaled gradient is a better than
          \nabla \Phi alone since it is a dimensionless quantity and so
          independent of the scale of the problem.  The 'max' arguments
          help ensure the test doesn't break down in regions where x_i
          or \Phi(x) are close to 0.  If this test succeeds, INFO is set
          to 2 and the function returns 'GSL_SUCCESS'.

          A general guideline for choosing the gradient tolerance is to
          set 'gtol = GSL_DBL_EPSILON^(1/3)'.  See Dennis and Schnabel
          for more information.

     If none of the tests succeed, INFO is set to 0 and the function
     returns 'GSL_CONTINUE', indicating further iterations are required.

 -- Function: int gsl_multifit_test_delta (const gsl_vector * DX, const
          gsl_vector * X, double EPSABS, double EPSREL)

     This function tests for the convergence of the sequence by
     comparing the last step DX with the absolute error EPSABS and
     relative error EPSREL to the current position X.  The test returns
     'GSL_SUCCESS' if the following condition is achieved,

          |dx_i| < epsabs + epsrel |x_i|

     for each component of X and returns 'GSL_CONTINUE' otherwise.

 -- Function: int gsl_multifit_test_gradient (const gsl_vector * G,
          double EPSABS)
     This function tests the residual gradient G against the absolute
     error bound EPSABS.  Mathematically, the gradient should be exactly
     zero at the minimum.  The test returns 'GSL_SUCCESS' if the
     following condition is achieved,

          \sum_i |g_i| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  This criterion is suitable
     for situations where the precise location of the minimum, x, is
     unimportant provided a value can be found where the gradient is
     small enough.

 -- Function: int gsl_multifit_gradient (const gsl_matrix * J, const
          gsl_vector * F, gsl_vector * G)
     This function computes the gradient G of \Phi(x) = (1/2) ||f(x)||^2
     from the Jacobian matrix J and the function values F, using the
     formula g = J^T f.


File: gsl-ref.info,  Node: High Level Driver,  Next: Minimization Algorithms using Derivatives,  Prev: Search Stopping Parameters for Minimization Algorithms,  Up: Nonlinear Least-Squares Fitting

39.9 High Level Driver
======================

These routines provide a high level wrapper that combine the iteration
and convergence testing for easy use.

 -- Function: int gsl_multifit_fsolver_driver (gsl_multifit_fsolver * S,
          const size_t MAXITER, const double EPSABS, const double
          EPSREL)
     This function iterates the solver S for a maximum of MAXITER
     iterations.  After each iteration, the system is tested for
     convergence using 'gsl_multifit_test_delta' with the error
     tolerances EPSABS and EPSREL.

 -- Function: int gsl_multifit_fdfsolver_driver (gsl_multifit_fdfsolver
          * S, const size_t MAXITER, const double XTOL, const double
          GTOL, const double FTOL, int * INFO)
 -- Function: int gsl_multifit_fdfridge_driver (gsl_multifit_fdfridge *
          S, const size_t MAXITER, const double XTOL, const double GTOL,
          const double FTOL, int * INFO)
     These functions iterate the solver S for a maximum of MAXITER
     iterations.  After each iteration, the system is tested for
     convergence with the error tolerances XTOL, GTOL and FTOL.  Upon
     successful convergence, the function returns 'GSL_SUCCESS' and sets
     INFO to the reason for convergence (see
     'gsl_multifit_fdfsolver_test').  Otherwise, the function returns
     'GSL_EMAXITER' indicating the system did not converge after MAXITER
     iterations.


File: gsl-ref.info,  Node: Minimization Algorithms using Derivatives,  Next: Minimization Algorithms without Derivatives,  Prev: High Level Driver,  Up: Nonlinear Least-Squares Fitting

39.10 Minimization Algorithms using Derivatives
===============================================

The minimization algorithms described in this section make use of both
the function and its derivative.  They require an initial guess for the
location of the minimum.  There is no absolute guarantee of
convergence--the function must be suitable for this technique and the
initial guess must be sufficiently close to the minimum for it to work.

 -- Derivative Solver: gsl_multifit_fdfsolver_lmsder

     This is a robust and efficient version of the Levenberg-Marquardt
     algorithm as implemented in the scaled LMDER routine in MINPACK.
     Minpack was written by Jorge J. More', Burton S. Garbow and Kenneth
     E. Hillstrom.

     The algorithm uses a generalized trust region to keep each step
     under control.  In order to be accepted a proposed new position x'
     must satisfy the condition |D (x' - x)| < \Delta, where D is a
     diagonal scaling matrix and \Delta is the size of the trust region.
     The components of D are computed internally, using the column norms
     of the Jacobian to estimate the sensitivity of the residual to each
     component of x.  This improves the behavior of the algorithm for
     badly scaled functions.

     On each iteration the algorithm attempts to minimize the linear
     system |f + J \delta| subject to the constraint |D \delta| <
     \Delta.  The solution to this constrained linear system is found by
     solving the linear least squares system

          [J; sqrt(mu) D] \delta = - [f; 0]

     where \mu is the Levenberg-Marquardt parameter.  The above system
     is solved using a QR decomposition of J.

     The proposed step \delta is now tested by evaluating the function
     at the resulting point, x'.  If the step reduces the norm of the
     function sufficiently, and follows the predicted behavior of the
     function within the trust region, then it is accepted and the size
     of the trust region is increased.  If the proposed step fails to
     improve the solution, or differs significantly from the expected
     behavior within the trust region, then the size of the trust region
     is decreased and another trial step is computed.

     The algorithm also monitors the progress of the solution and
     returns an error if the changes in the solution are smaller than
     the machine precision.  The possible error codes are,

     'GSL_ETOLF'
          the decrease in the function falls below machine precision

     'GSL_ETOLX'
          the change in the position vector falls below machine
          precision

     'GSL_ETOLG'
          the norm of the gradient, relative to the norm of the
          function, falls below machine precision

     'GSL_ENOPROG'
          the routine has made 10 or more attempts to find a suitable
          trial step without success (but subsequent calls can be made
          to continue the search).(1)

     These error codes indicate that further iterations will be unlikely
     to change the solution from its current value.

 -- Derivative Solver: gsl_multifit_fdfsolver_lmder

     This is an unscaled version of the LMDER algorithm.  The elements
     of the diagonal scaling matrix D are set to 1.  This algorithm may
     be useful in circumstances where the scaled version of LMDER
     converges too slowly, or the function is already scaled
     appropriately.

 -- Derivative Solver: gsl_multifit_fdfsolver_lmniel

     This is a Levenberg-Marquardt solver based on a smoother updating
     procedure for the damping parameter \mu proposed by Nielsen, 1999.
     It does not use a trust region approach and only performs
     rudimentary scaling and is therefore not as robust as 'lmsder'.
     However, on each iteration it solves the normal equation system to
     compute the next step:

          (J^T J + \mu I) \delta = -J^T f

     which makes it a much more practical method for problems with a
     large number of residuals (n >> p), since only the p-by-p matrix
     J^T J is decomposed rather than the full n-by-p Jacobian.  This
     makes a significant difference in efficiency when solving systems
     with large amounts of data.  While not as robust as 'lmsder', this
     algorithm has proven effective on a wide class of problems.

   ---------- Footnotes ----------

   (1) The return code 'GSL_CONTINUE' was used for this case in versions
prior to 1.14.


File: gsl-ref.info,  Node: Minimization Algorithms without Derivatives,  Next: Computing the covariance matrix of best fit parameters,  Prev: Minimization Algorithms using Derivatives,  Up: Nonlinear Least-Squares Fitting

39.11 Minimization Algorithms without Derivatives
=================================================

There are no algorithms implemented in this section at the moment.


File: gsl-ref.info,  Node: Computing the covariance matrix of best fit parameters,  Next: Troubleshooting Nonlinear Least Squares,  Prev: Minimization Algorithms without Derivatives,  Up: Nonlinear Least-Squares Fitting

39.12 Computing the covariance matrix of best fit parameters
============================================================

 -- Function: int gsl_multifit_fdfsolver_jac (gsl_multifit_fdfsolver *
          S, gsl_matrix * J)
     This function stores the n-by-p Jacobian matrix for the current
     iteration of the solver S into the output J.

 -- Function: int gsl_multifit_covar (const gsl_matrix * J, const double
          EPSREL, gsl_matrix * COVAR)
     This function computes the covariance matrix of best-fit parameters
     using the Jacobian matrix J and stores it in COVAR.  The parameter
     EPSREL is used to remove linear-dependent columns when J is rank
     deficient.

     The covariance matrix is given by,

          covar = (J^T J)^{-1}

     or in the weighted case,

          covar = (J^T W J)^{-1}

     and is computed by QR decomposition of J with column-pivoting.  Any
     columns of R which satisfy

          |R_{kk}| <= epsrel |R_{11}|

     are considered linearly-dependent and are excluded from the
     covariance matrix (the corresponding rows and columns of the
     covariance matrix are set to zero).

     If the minimisation uses the weighted least-squares function f_i =
     (Y(x, t_i) - y_i) / \sigma_i then the covariance matrix above gives
     the statistical error on the best-fit parameters resulting from the
     Gaussian errors \sigma_i on the underlying data y_i.  This can be
     verified from the relation \delta f = J \delta c and the fact that
     the fluctuations in f from the data y_i are normalised by \sigma_i
     and so satisfy <\delta f \delta f^T> = I.

     For an unweighted least-squares function f_i = (Y(x, t_i) - y_i)
     the covariance matrix above should be multiplied by the variance of
     the residuals about the best-fit \sigma^2 = \sum (y_i - Y(x,t_i))^2
     / (n-p) to give the variance-covariance matrix \sigma^2 C.  This
     estimates the statistical error on the best-fit parameters from the
     scatter of the underlying data.

     For more information about covariance matrices see *note Fitting
     Overview::.


File: gsl-ref.info,  Node: Troubleshooting Nonlinear Least Squares,  Next: Example programs for Nonlinear Least-Squares Fitting,  Prev: Computing the covariance matrix of best fit parameters,  Up: Nonlinear Least-Squares Fitting

39.13 Troubleshooting
=====================

When developing a code to solve a nonlinear least squares problem, here
are a few considerations to keep in mind.

  1. The most common difficulty is the accurate implementation of the
     Jacobian matrix.  If the analytic Jacobian is not properly provided
     to the solver, this can hinder and many times prevent convergence
     of the method.  When developing a new nonlinear least squares code,
     it often helps to compare the program output with the internally
     computed finite difference Jacobian and the user supplied analytic
     Jacobian.  If there is a large difference in coefficients, it is
     likely the analytic Jacobian is incorrectly implemented.

  2. If your code is having difficulty converging, the next thing to
     check is the starting point provided to the solver.  The methods of
     this chapter are local methods, meaning if you provide a starting
     point far away from the true minimum, the method may converge to a
     local minimum or not converge at all.  Sometimes it is possible to
     solve a linearized approximation to the nonlinear problem, and use
     the linear solution as the starting point to the nonlinear problem.

  3. If the various parameters of the coefficient vector x vary widely
     in magnitude, then the problem is said to be badly scaled.  The
     methods of this chapter do attempt to automatically rescale the
     elements of x to have roughly the same order of magnitude, but in
     extreme cases this could still cause problems for convergence.  In
     these cases it is recommended for the user to scale their parameter
     vector x so that each parameter spans roughly the same range, say
     [-1,1].  The solution vector can be backscaled to recover the
     original units of the problem.


File: gsl-ref.info,  Node: Example programs for Nonlinear Least-Squares Fitting,  Next: References and Further Reading for Nonlinear Least-Squares Fitting,  Prev: Troubleshooting Nonlinear Least Squares,  Up: Nonlinear Least-Squares Fitting

39.14 Examples
==============

The following example program fits a weighted exponential model with
background to experimental data, Y = A \exp(-\lambda t) + b.  The first
part of the program sets up the functions 'expb_f' and 'expb_df' to
calculate the model and its Jacobian.  The appropriate fitting function
is given by,

     f_i = (A \exp(-\lambda t_i) + b) - y_i

where we have chosen t_i = i.  The Jacobian matrix J is the derivative
of these functions with respect to the three parameters (A, \lambda, b).
It is given by,

     J_{ij} = d f_i / d x_j

where x_0 = A, x_1 = \lambda and x_2 = b.  The weights are given by w_i
= 1/\sigma_i^2.

     /* expfit.c -- model functions for exponential + background */

     struct data {
       size_t n;
       double * y;
     };

     int
     expb_f (const gsl_vector * x, void *data,
             gsl_vector * f)
     {
       size_t n = ((struct data *)data)->n;
       double *y = ((struct data *)data)->y;

       double A = gsl_vector_get (x, 0);
       double lambda = gsl_vector_get (x, 1);
       double b = gsl_vector_get (x, 2);

       size_t i;

       for (i = 0; i < n; i++)
         {
           /* Model Yi = A * exp(-lambda * i) + b */
           double t = i;
           double Yi = A * exp (-lambda * t) + b;
           gsl_vector_set (f, i, Yi - y[i]);
         }

       return GSL_SUCCESS;
     }

     int
     expb_df (const gsl_vector * x, void *data,
              gsl_matrix * J)
     {
       size_t n = ((struct data *)data)->n;

       double A = gsl_vector_get (x, 0);
       double lambda = gsl_vector_get (x, 1);

       size_t i;

       for (i = 0; i < n; i++)
         {
           /* Jacobian matrix J(i,j) = dfi / dxj, */
           /* where fi = (Yi - yi)/sigma[i],      */
           /*       Yi = A * exp(-lambda * i) + b  */
           /* and the xj are the parameters (A,lambda,b) */
           double t = i;
           double e = exp(-lambda * t);
           gsl_matrix_set (J, i, 0, e);
           gsl_matrix_set (J, i, 1, -t * A * e);
           gsl_matrix_set (J, i, 2, 1.0);
         }
       return GSL_SUCCESS;
     }

The main part of the program sets up a Levenberg-Marquardt solver and
some simulated random data.  The data uses the known parameters
(5.0,0.1,1.0) combined with Gaussian noise (standard deviation = 0.1)
over a range of 40 timesteps.  The initial guess for the parameters is
chosen as (0.0, 1.0, 0.0).

     #include <stdlib.h>
     #include <stdio.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_blas.h>
     #include <gsl/gsl_multifit_nlin.h>

     #include "expfit.c"

     /* number of data points to fit */
     #define N 40

     int
     main (void)
     {
       const gsl_multifit_fdfsolver_type *T = gsl_multifit_fdfsolver_lmsder;
       gsl_multifit_fdfsolver *s;
       int status, info;
       size_t i;
       const size_t n = N;
       const size_t p = 3;

       gsl_matrix *J = gsl_matrix_alloc(n, p);
       gsl_matrix *covar = gsl_matrix_alloc (p, p);
       double y[N], weights[N];
       struct data d = { n, y };
       gsl_multifit_function_fdf f;
       double x_init[3] = { 1.0, 0.0, 0.0 };
       gsl_vector_view x = gsl_vector_view_array (x_init, p);
       gsl_vector_view w = gsl_vector_view_array(weights, n);
       const gsl_rng_type * type;
       gsl_rng * r;
       gsl_vector *res_f;
       double chi, chi0;

       const double xtol = 1e-8;
       const double gtol = 1e-8;
       const double ftol = 0.0;

       gsl_rng_env_setup();

       type = gsl_rng_default;
       r = gsl_rng_alloc (type);

       f.f = &expb_f;
       f.df = &expb_df;   /* set to NULL for finite-difference Jacobian */
       f.n = n;
       f.p = p;
       f.params = &d;

       /* This is the data to be fitted */

       for (i = 0; i < n; i++)
         {
           double t = i;
           double yi = 1.0 + 5 * exp (-0.1 * t);
           double si = 0.1 * yi;
           double dy = gsl_ran_gaussian(r, si);

           weights[i] = 1.0 / (si * si);
           y[i] = yi + dy;
           printf ("data: %zu %g %g\n", i, y[i], si);
         };

       s = gsl_multifit_fdfsolver_alloc (T, n, p);

       /* initialize solver with starting point and weights */
       gsl_multifit_fdfsolver_wset (s, &f, &x.vector, &w.vector);

       /* compute initial residual norm */
       res_f = gsl_multifit_fdfsolver_residual(s);
       chi0 = gsl_blas_dnrm2(res_f);

       /* solve the system with a maximum of 20 iterations */
       status = gsl_multifit_fdfsolver_driver(s, 20, xtol, gtol, ftol, &info);

       gsl_multifit_fdfsolver_jac(s, J);
       gsl_multifit_covar (J, 0.0, covar);

       /* compute final residual norm */
       chi = gsl_blas_dnrm2(res_f);

     #define FIT(i) gsl_vector_get(s->x, i)
     #define ERR(i) sqrt(gsl_matrix_get(covar,i,i))

       fprintf(stderr, "summary from method '%s'\n",
               gsl_multifit_fdfsolver_name(s));
       fprintf(stderr, "number of iterations: %zu\n",
               gsl_multifit_fdfsolver_niter(s));
       fprintf(stderr, "function evaluations: %zu\n", f.nevalf);
       fprintf(stderr, "Jacobian evaluations: %zu\n", f.nevaldf);
       fprintf(stderr, "reason for stopping: %s\n",
               (info == 1) ? "small step size" : "small gradient");
       fprintf(stderr, "initial |f(x)| = %g\n", chi0);
       fprintf(stderr, "final   |f(x)| = %g\n", chi);

       {
         double dof = n - p;
         double c = GSL_MAX_DBL(1, chi / sqrt(dof));

         fprintf(stderr, "chisq/dof = %g\n",  pow(chi, 2.0) / dof);

         fprintf (stderr, "A      = %.5f +/- %.5f\n", FIT(0), c*ERR(0));
         fprintf (stderr, "lambda = %.5f +/- %.5f\n", FIT(1), c*ERR(1));
         fprintf (stderr, "b      = %.5f +/- %.5f\n", FIT(2), c*ERR(2));
       }

       fprintf (stderr, "status = %s\n", gsl_strerror (status));

       gsl_multifit_fdfsolver_free (s);
       gsl_matrix_free (covar);
       gsl_matrix_free (J);
       gsl_rng_free (r);
       return 0;
     }

The iteration terminates when the relative change in x is smaller than
10^{-8}, or when the magnitude of the gradient falls below 10^{-8}.
Here are the results of running the program:

     summary from method 'lmsder'
     number of iterations: 8
     function evaluations: 11
     Jacobian evaluations: 9
     reason for stopping: small step size
     initial |f(x)| = 31.1919
     final   |f(x)| = 5.45418
     chisq/dof = 0.804002
     A      = 5.17379 +/- 0.27938
     lambda = 0.11104 +/- 0.00817
     b      = 1.05283 +/- 0.05365
     status = success

The approximate values of the parameters are found correctly, and the
chi-squared value indicates a good fit (the chi-squared per degree of
freedom is approximately 1).  In this case the errors on the parameters
can be estimated from the square roots of the diagonal elements of the
covariance matrix.

   If the chi-squared value shows a poor fit (i.e.  chi^2/dof >> 1) then
the error estimates obtained from the covariance matrix will be too
small.  In the example program the error estimates are multiplied by
\sqrt{\chi^2/dof} in this case, a common way of increasing the errors
for a poor fit.  Note that a poor fit will result from the use an
inappropriate model, and the scaled error estimates may then be outside
the range of validity for Gaussian errors.


File: gsl-ref.info,  Node: References and Further Reading for Nonlinear Least-Squares Fitting,  Prev: Example programs for Nonlinear Least-Squares Fitting,  Up: Nonlinear Least-Squares Fitting

39.15 References and Further Reading
====================================

The MINPACK algorithm is described in the following article,

     J.J. More', 'The Levenberg-Marquardt Algorithm: Implementation and
     Theory', Lecture Notes in Mathematics, v630 (1978), ed G. Watson.

The 'lmniel' algorithm closely follows the following publications,

     H. B. Nielsen, "Damping Parameter in Marquardt's Method", IMM
     Department of Mathematical Modeling, DTU, Tech.  Report
     IMM-REP-1999-05 (1999).

     K. Madsen and H. B. Nielsen, "Introduction to Optimization and Data
     Fitting", IMM Department of Mathematical Modeling, DTU, 2010.

The following publications are also relevant to the algorithms described
in this section,

     J. E. Dennis and R. B. Schnabel, Numerical Methods for
     Unconstrained Optimization and Nonlinear Equations, SIAM, 1996.

     J.J. More', B.S. Garbow, K.E. Hillstrom, "Testing Unconstrained
     Optimization Software", ACM Transactions on Mathematical Software,
     Vol 7, No 1 (1981), p 17-41.

     H. B. Nielsen, "UCTP Test Problems for Unconstrained Optimization",
     IMM Department of Mathematical Modeling, DTU, Tech.  Report
     IMM-REP-2000-17 (2000).


File: gsl-ref.info,  Node: Basis Splines,  Next: Sparse Matrices,  Prev: Nonlinear Least-Squares Fitting,  Up: Top

40 Basis Splines
****************

This chapter describes functions for the computation of smoothing basis
splines (B-splines).  A smoothing spline differs from an interpolating
spline in that the resulting curve is not required to pass through each
datapoint.  *Note Interpolation::, for information about interpolating
splines.

   The header file 'gsl_bspline.h' contains the prototypes for the
bspline functions and related declarations.

* Menu:

* Overview of B-splines::
* Initializing the B-splines solver::
* Constructing the knots vector::
* Evaluation of B-spline basis functions::
* Evaluation of B-spline basis function derivatives::
* Working with the Greville abscissae::
* Example programs for B-splines::
* B-Spline References and Further Reading::


File: gsl-ref.info,  Node: Overview of B-splines,  Next: Initializing the B-splines solver,  Up: Basis Splines

40.1 Overview
=============

B-splines are commonly used as basis functions to fit smoothing curves
to large data sets.  To do this, the abscissa axis is broken up into
some number of intervals, where the endpoints of each interval are
called "breakpoints".  These breakpoints are then converted to "knots"
by imposing various continuity and smoothness conditions at each
interface.  Given a nondecreasing knot vector t = {t_0, t_1, ...,
t_{n+k-1}}, the n basis splines of order k are defined by

     B_(i,1)(x) = (1, t_i <= x < t_(i+1)
                  (0, else
     B_(i,k)(x) = [(x - t_i)/(t_(i+k-1) - t_i)] B_(i,k-1)(x)
                   + [(t_(i+k) - x)/(t_(i+k) - t_(i+1))] B_(i+1,k-1)(x)

for i = 0, ..., n-1.  The common case of cubic B-splines is given by k =
4.  The above recurrence relation can be evaluated in a numerically
stable way by the de Boor algorithm.

   If we define appropriate knots on an interval [a,b] then the B-spline
basis functions form a complete set on that interval.  Therefore we can
expand a smoothing function as

     f(x) = \sum_i c_i B_(i,k)(x)

given enough (x_j, f(x_j)) data pairs.  The coefficients c_i can be
readily obtained from a least-squares fit.


File: gsl-ref.info,  Node: Initializing the B-splines solver,  Next: Constructing the knots vector,  Prev: Overview of B-splines,  Up: Basis Splines

40.2 Initializing the B-splines solver
======================================

The computation of B-spline functions requires a preallocated workspace
of type 'gsl_bspline_workspace'.

 -- Function: gsl_bspline_workspace * gsl_bspline_alloc (const size_t K,
          const size_t NBREAK)
     This function allocates a workspace for computing B-splines of
     order K.  The number of breakpoints is given by NBREAK.  This leads
     to n = nbreak + k - 2 basis functions.  Cubic B-splines are
     specified by k = 4.  The size of the workspace is O(2k^2 + 5k +
     nbreak).

 -- Function: void gsl_bspline_free (gsl_bspline_workspace * W)
     This function frees the memory associated with the workspace W.


File: gsl-ref.info,  Node: Constructing the knots vector,  Next: Evaluation of B-spline basis functions,  Prev: Initializing the B-splines solver,  Up: Basis Splines

40.3 Constructing the knots vector
==================================

 -- Function: int gsl_bspline_knots (const gsl_vector * BREAKPTS,
          gsl_bspline_workspace * W)
     This function computes the knots associated with the given
     breakpoints and stores them internally in 'w->knots'.

 -- Function: int gsl_bspline_knots_uniform (const double A, const
          double B, gsl_bspline_workspace * W)
     This function assumes uniformly spaced breakpoints on [a,b] and
     constructs the corresponding knot vector using the previously
     specified NBREAK parameter.  The knots are stored in 'w->knots'.


File: gsl-ref.info,  Node: Evaluation of B-spline basis functions,  Next: Evaluation of B-spline basis function derivatives,  Prev: Constructing the knots vector,  Up: Basis Splines

40.4 Evaluation of B-splines
============================

 -- Function: int gsl_bspline_eval (const double X, gsl_vector * B,
          gsl_bspline_workspace * W)
     This function evaluates all B-spline basis functions at the
     position X and stores them in the vector B, so that the i-th
     element is B_i(x).  The vector B must be of length n = nbreak + k -
     2.  This value may also be obtained by calling
     'gsl_bspline_ncoeffs'.  Computing all the basis functions at once
     is more efficient than computing them individually, due to the
     nature of the defining recurrence relation.

 -- Function: int gsl_bspline_eval_nonzero (const double X, gsl_vector *
          BK, size_t * ISTART, size_t * IEND, gsl_bspline_workspace * W)
     This function evaluates all potentially nonzero B-spline basis
     functions at the position X and stores them in the vector BK, so
     that the i-th element is B_(istart+i)(x).  The last element of BK
     is B_(iend)(x).  The vector BK must be of length k.  By returning
     only the nonzero basis functions, this function allows quantities
     involving linear combinations of the B_i(x) to be computed without
     unnecessary terms (such linear combinations occur, for example,
     when evaluating an interpolated function).

 -- Function: size_t gsl_bspline_ncoeffs (gsl_bspline_workspace * W)
     This function returns the number of B-spline coefficients given by
     n = nbreak + k - 2.


File: gsl-ref.info,  Node: Evaluation of B-spline basis function derivatives,  Next: Working with the Greville abscissae,  Prev: Evaluation of B-spline basis functions,  Up: Basis Splines

40.5 Evaluation of B-spline derivatives
=======================================

 -- Function: int gsl_bspline_deriv_eval (const double X, const size_t
          NDERIV, gsl_matrix * DB, gsl_bspline_workspace * W)
     This function evaluates all B-spline basis function derivatives of
     orders 0 through nderiv (inclusive) at the position X and stores
     them in the matrix DB.  The (i,j)-th element of DB is
     d^jB_i(x)/dx^j.  The matrix DB must be of size n = nbreak + k - 2
     by nderiv + 1.  The value n may also be obtained by calling
     'gsl_bspline_ncoeffs'.  Note that function evaluations are included
     as the zeroth order derivatives in DB.  Computing all the basis
     function derivatives at once is more efficient than computing them
     individually, due to the nature of the defining recurrence
     relation.

 -- Function: int gsl_bspline_deriv_eval_nonzero (const double X, const
          size_t NDERIV, gsl_matrix * DB, size_t * ISTART, size_t *
          IEND, gsl_bspline_workspace * W)
     This function evaluates all potentially nonzero B-spline basis
     function derivatives of orders 0 through nderiv (inclusive) at the
     position X and stores them in the matrix DB.  The (i,j)-th element
     of DB is d^j/dx^j B_(istart+i)(x).  The last row of DB contains
     d^j/dx^j B_(iend)(x).  The matrix DB must be of size k by at least
     nderiv + 1.  Note that function evaluations are included as the
     zeroth order derivatives in DB.  By returning only the nonzero
     basis functions, this function allows quantities involving linear
     combinations of the B_i(x) and their derivatives to be computed
     without unnecessary terms.


File: gsl-ref.info,  Node: Working with the Greville abscissae,  Next: Example programs for B-splines,  Prev: Evaluation of B-spline basis function derivatives,  Up: Basis Splines

40.6 Working with the Greville abscissae
========================================

The Greville abscissae are defined to be the mean location of k-1
consecutive knots in the knot vector for each basis spline function of
order k.  With the first and last knots in the 'gsl_bspline_workspace'
knot vector excluded, there are 'gsl_bspline_ncoeffs' Greville abscissae
for any given B-spline basis.  These values are often used in B-spline
collocation applications and may also be called Marsden-Schoenberg
points.

 -- Function: double gsl_bspline_greville_abscissa (size_t I,
          gsl_bspline_workspace *W);
     Returns the location of the i-th Greville abscissa for the given
     B-spline basis.  For the ill-defined case when k=1, the
     implementation chooses to return breakpoint interval midpoints.


File: gsl-ref.info,  Node: Example programs for B-splines,  Next: B-Spline References and Further Reading,  Prev: Working with the Greville abscissae,  Up: Basis Splines

40.7 Examples
=============

The following program computes a linear least squares fit to data using
cubic B-spline basis functions with uniform breakpoints.  The data is
generated from the curve y(x) = \cos{(x)} \exp{(-x/10)} on the interval
[0, 15] with Gaussian noise added.

     #include <stdio.h>
     #include <stdlib.h>
     #include <math.h>
     #include <gsl/gsl_bspline.h>
     #include <gsl/gsl_multifit.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_statistics.h>

     /* number of data points to fit */
     #define N        200

     /* number of fit coefficients */
     #define NCOEFFS  12

     /* nbreak = ncoeffs + 2 - k = ncoeffs - 2 since k = 4 */
     #define NBREAK   (NCOEFFS - 2)

     int
     main (void)
     {
       const size_t n = N;
       const size_t ncoeffs = NCOEFFS;
       const size_t nbreak = NBREAK;
       size_t i, j;
       gsl_bspline_workspace *bw;
       gsl_vector *B;
       double dy;
       gsl_rng *r;
       gsl_vector *c, *w;
       gsl_vector *x, *y;
       gsl_matrix *X, *cov;
       gsl_multifit_linear_workspace *mw;
       double chisq, Rsq, dof, tss;

       gsl_rng_env_setup();
       r = gsl_rng_alloc(gsl_rng_default);

       /* allocate a cubic bspline workspace (k = 4) */
       bw = gsl_bspline_alloc(4, nbreak);
       B = gsl_vector_alloc(ncoeffs);

       x = gsl_vector_alloc(n);
       y = gsl_vector_alloc(n);
       X = gsl_matrix_alloc(n, ncoeffs);
       c = gsl_vector_alloc(ncoeffs);
       w = gsl_vector_alloc(n);
       cov = gsl_matrix_alloc(ncoeffs, ncoeffs);
       mw = gsl_multifit_linear_alloc(n, ncoeffs);

       printf("#m=0,S=0\n");
       /* this is the data to be fitted */
       for (i = 0; i < n; ++i)
         {
           double sigma;
           double xi = (15.0 / (N - 1)) * i;
           double yi = cos(xi) * exp(-0.1 * xi);

           sigma = 0.1 * yi;
           dy = gsl_ran_gaussian(r, sigma);
           yi += dy;

           gsl_vector_set(x, i, xi);
           gsl_vector_set(y, i, yi);
           gsl_vector_set(w, i, 1.0 / (sigma * sigma));

           printf("%f %f\n", xi, yi);
         }

       /* use uniform breakpoints on [0, 15] */
       gsl_bspline_knots_uniform(0.0, 15.0, bw);

       /* construct the fit matrix X */
       for (i = 0; i < n; ++i)
         {
           double xi = gsl_vector_get(x, i);

           /* compute B_j(xi) for all j */
           gsl_bspline_eval(xi, B, bw);

           /* fill in row i of X */
           for (j = 0; j < ncoeffs; ++j)
             {
               double Bj = gsl_vector_get(B, j);
               gsl_matrix_set(X, i, j, Bj);
             }
         }

       /* do the fit */
       gsl_multifit_wlinear(X, w, y, c, cov, &chisq, mw);

       dof = n - ncoeffs;
       tss = gsl_stats_wtss(w->data, 1, y->data, 1, y->size);
       Rsq = 1.0 - chisq / tss;

       fprintf(stderr, "chisq/dof = %e, Rsq = %f\n",
                        chisq / dof, Rsq);

       /* output the smoothed curve */
       {
         double xi, yi, yerr;

         printf("#m=1,S=0\n");
         for (xi = 0.0; xi < 15.0; xi += 0.1)
           {
             gsl_bspline_eval(xi, B, bw);
             gsl_multifit_linear_est(B, c, cov, &yi, &yerr);
             printf("%f %f\n", xi, yi);
           }
       }

       gsl_rng_free(r);
       gsl_bspline_free(bw);
       gsl_vector_free(B);
       gsl_vector_free(x);
       gsl_vector_free(y);
       gsl_matrix_free(X);
       gsl_vector_free(c);
       gsl_vector_free(w);
       gsl_matrix_free(cov);
       gsl_multifit_linear_free(mw);

       return 0;
     } /* main() */

   The output can be plotted with GNU 'graph'.

     $ ./a.out > bspline.txt
     chisq/dof = 1.118217e+00, Rsq = 0.989771
     $ graph -T ps -X x -Y y -x 0 15 -y -1 1.3 < bspline.txt > bspline.ps


File: gsl-ref.info,  Node: B-Spline References and Further Reading,  Prev: Example programs for B-splines,  Up: Basis Splines

40.8 B-Spline References and Further Reading
============================================

Further information on the algorithms described in this section can be
found in the following book,

     C. de Boor, 'A Practical Guide to Splines' (1978), Springer-Verlag,
     ISBN 0-387-90356-9.

   Further information of Greville abscissae and B-spline collocation
can be found in the following paper,

     Richard W. Johnson, Higher order B-spline collocation at the
     Greville abscissae.  'Applied Numerical Mathematics'.  vol. 52,
     2005, 63-75.

A large collection of B-spline routines is available in the PPPACK
library available at <http://www.netlib.org/pppack>, which is also part
of SLATEC.


File: gsl-ref.info,  Node: Sparse Matrices,  Next: Sparse BLAS Support,  Prev: Basis Splines,  Up: Top

41 Sparse Matrices
******************

This chapter describes functions for the construction and manipulation
of sparse matrices, matrices which are populated primarily with zeros
and contain only a few non-zero elements.  Sparse matrices often appear
in the solution of partial differential equations.  It is beneficial to
use specialized data structures and algorithms for storing and working
with sparse matrices, since dense matrix algorithms and structures can
be very slow and use huge amounts of memory when applied to sparse
matrices.

The header file 'gsl_spmatrix.h' contains the prototypes for the sparse
matrix functions and related declarations.

* Menu:

* Overview of Sparse Matrices::
* Sparse matrix allocation::
* Accessing sparse matrix elements::
* Initializing sparse matrix elements::
* Copying sparse matrices::
* Sparse matrix operations::
* Sparse matrix properties::
* Finding maximum and minimum elements of sparse matrices::
* Sparse matrix compressed format::
* Conversion between sparse and dense matrices::
* Sparse Matrix Examples::
* Sparse Matrix References and Further Reading::


File: gsl-ref.info,  Node: Overview of Sparse Matrices,  Next: Sparse matrix allocation,  Up: Sparse Matrices

41.1 Overview
=============

These routines provide support for constructing and manipulating sparse
matrices in GSL, using an API similar to 'gsl_matrix'.  The basic
structure is called 'gsl_spmatrix'.  There are two supported storage
formats for sparse matrices: the triplet and compressed column storage
(CCS) formats.  The triplet format stores triplets (i,j,x) for each
non-zero element of the matrix.  This notation means that the (i,j)
element of the matrix A is A_{ij} = x.  Compressed column storage stores
each column of non-zero values in the sparse matrix in a continuous
memory block, keeping pointers to the beginning of each column in that
memory block, and storing the row indices of each non-zero element.  The
triplet format is ideal for adding elements to the sparse matrix
structure while it is being constructed, while the compressed column
storage is better suited for matrix-matrix multiplication or linear
solvers.

The 'gsl_spmatrix' structure is defined as

     typedef struct
     {
       size_t size1;
       size_t size2;
       size_t *i;
       double *data;
       size_t *p;
       size_t nzmax;
       size_t nz;
       gsl_spmatrix_tree *tree_data;
       void *work;
       size_t sptype;
     } gsl_spmatrix;

This defines a SIZE1-by-SIZE2 sparse matrix.  The number of non-zero
elements currently in the matrix is given by NZ.  For the triplet
representation, I, P, and DATA are arrays of size NZ which contain the
row indices, column indices, and element value, respectively.  So if
data[k] = A(i,j), then i = i[k] and j = p[k].  For compressed column
storage, I and DATA are arrays of size NZ containing the row indices and
element values, identical to the triplet case.  P is an array of size
size2 + 1 where p[j] points to the index in DATA of the start of column
J.  Thus, if data[k] = A(i,j), then i = i[k] and p[j] <= k < p[j+1].

The parameter TREE_DATA is a binary tree structure used in the triplet
representation, specifically a balanced AVL tree.  This speeds up
element searches and duplicate detection during the matrix assembly
process.  The parameter WORK is additional workspace needed for various
operations like converting from triplet to compressed column storage.
SPTYPE indicates the type of storage format being used (triplet or
compressed column).

The compressed storage format defined above makes it very simple to
interface with sophisticated external linear solver libraries which
accept compressed column storage input.  The user can simply pass the
arrays I, P, and DATA as the compressed column inputs to external
libraries.


File: gsl-ref.info,  Node: Sparse matrix allocation,  Next: Accessing sparse matrix elements,  Prev: Overview of Sparse Matrices,  Up: Sparse Matrices

41.2 Sparse matrix allocation
=============================

The functions for allocating memory for a sparse matrix follow the style
of 'malloc' and 'free'.  They also perform their own error checking.  If
there is insufficient memory available to allocate a matrix then the
functions call the GSL error handler with an error code of 'GSL_ENOMEM'
in addition to returning a null pointer.

 -- Function: gsl_spmatrix * gsl_spmatrix_alloc (const size_t N1, const
          size_t N2)
     This function allocates a sparse matrix of size N1-by-N2 and
     initializes it to all zeros.  If the size of the matrix is not
     known at allocation time, both N1 and N2 may be set to 1, and they
     will automatically grow as elements are added to the matrix.  This
     function sets the matrix to the triplet representation, which is
     the easiest for adding and accessing matrix elements.  This
     function tries to make a reasonable guess for the number of
     non-zero elements (NZMAX) which will be added to the matrix by
     assuming a sparse density of 10\%.  The function
     'gsl_spmatrix_alloc_nzmax' can be used if this number is known more
     accurately.  The workspace is of size O(nzmax).

 -- Function: gsl_spmatrix * gsl_spmatrix_alloc_nzmax (const size_t N1,
          const size_t N2, const size_t NZMAX, const size_t SPTYPE)
     This function allocates a sparse matrix of size N1-by-N2 and
     initializes it to all zeros.  If the size of the matrix is not
     known at allocation time, both N1 and N2 may be set to 1, and they
     will automatically grow as elements are added to the matrix.  The
     parameter NZMAX specifies the maximum number of non-zero elements
     which will be added to the matrix.  It does not need to be
     precisely known in advance, since storage space will automatically
     grow using 'gsl_spmatrix_realloc' if NZMAX is not large enough.
     Accurate knowledge of this parameter reduces the number of
     reallocation calls required.  The parameter SPTYPE specifies the
     storage format of the sparse matrix.  Possible values are
     'GSL_SPMATRIX_TRIPLET'
          This flag specifies triplet storage.

     'GSL_SPMATRIX_CCS'
          This flag specifies compressed column storage.
     The allocated 'gsl_spmatrix' structure is of size O(nzmax).

 -- Function: int gsl_spmatrix_realloc (const size_t NZMAX, gsl_spmatrix
          * M)
     This function reallocates the storage space for M to accomodate
     NZMAX non-zero elements.  It is typically called internally by
     'gsl_spmatrix_set' if the user wants to add more elements to the
     sparse matrix than the previously specified NZMAX.

 -- Function: void gsl_spmatrix_free (gsl_spmatrix * M)
     This function frees the memory associated with the sparse matrix M.


File: gsl-ref.info,  Node: Accessing sparse matrix elements,  Next: Initializing sparse matrix elements,  Prev: Sparse matrix allocation,  Up: Sparse Matrices

41.3 Accessing sparse matrix elements
=====================================

 -- Function: double gsl_spmatrix_get (const gsl_spmatrix * M, const
          size_t I, const size_t J)
     This function returns element (I,J) of the matrix M.  The matrix
     may be in triplet or compressed format.

 -- Function: int gsl_spmatrix_set (gsl_spmatrix * M, const size_t I,
          const size_t J, const double X)
     This function sets element (I,J) of the matrix M to the value X.
     The matrix must be in triplet representation.


File: gsl-ref.info,  Node: Initializing sparse matrix elements,  Next: Copying sparse matrices,  Prev: Accessing sparse matrix elements,  Up: Sparse Matrices

41.4 Initializing sparse matrix elements
========================================

Since the sparse matrix format only stores the non-zero elements, it is
automatically initialized to zero upon allocation.  The function
'gsl_spmatrix_set_zero' may be used to re-initialize a matrix to zero
after elements have been added to it.

 -- Function: int gsl_spmatrix_set_zero (gsl_spmatrix * M)
     This function sets (or resets) all the elements of the matrix M to
     zero.


File: gsl-ref.info,  Node: Copying sparse matrices,  Next: Sparse matrix operations,  Prev: Initializing sparse matrix elements,  Up: Sparse Matrices

41.5 Copying sparse matrices
============================

 -- Function: int gsl_spmatrix_memcpy (gsl_spmatrix * DEST, const
          gsl_spmatrix * SRC)
     This function copies the elements of the sparse matrix SRC into
     DEST.  The two matrices must have the same dimensions and be in the
     same storage format.

 -- Function: int gsl_spmatrix_transpose_memcpy (gsl_spmatrix * DEST,
          const gsl_spmatrix * SRC)
     This function copies the transpose of the sparse matrix SRC into
     DEST.  The dimensions of DEST must match the transpose of the
     matrix SRC.  Also, both matrices must use the same sparse storage
     format.


File: gsl-ref.info,  Node: Sparse matrix operations,  Next: Sparse matrix properties,  Prev: Copying sparse matrices,  Up: Sparse Matrices

41.6 Sparse matrix operations
=============================

 -- Function: int gsl_spmatrix_add (gsl_spmatrix * C, const gsl_spmatrix
          * A, const gsl_spmatrix * B)
     This function computes the sum c = a + b.  The three matrices must
     have the same dimensions and be stored in compressed column format.

 -- Function: int gsl_spmatrix_scale (gsl_spmatrix * M, const double X)
     This function scales all elements of the matrix M by the constant
     factor X.  The result m(i,j) \leftarrow x m(i,j) is stored in M.


File: gsl-ref.info,  Node: Sparse matrix properties,  Next: Finding maximum and minimum elements of sparse matrices,  Prev: Sparse matrix operations,  Up: Sparse Matrices

41.7 Sparse matrix properties
=============================

 -- Function: size_t gsl_spmatrix_nnz (const gsl_spmatrix * M)
     This function returns the number of non-zero elements in M.

 -- Function: int gsl_spmatrix_equal (const gsl_spmatrix * A, const
          gsl_spmatrix * B)
     This function returns 1 if the matrices A and B are equal (by
     comparison of element values) and 0 otherwise.  The matrices A and
     B must be either both triplet format or both compressed format for
     comparison.


File: gsl-ref.info,  Node: Finding maximum and minimum elements of sparse matrices,  Next: Sparse matrix compressed format,  Prev: Sparse matrix properties,  Up: Sparse Matrices

41.8 Finding maximum and minimum elements of sparse matrices
============================================================

 -- Function: int gsl_spmatrix_minmax (const gsl_spmatrix * M, double *
          MIN_OUT, double * MAX_OUT)
     This function returns the minimum and maximum elements of the
     matrix M, storing them in MIN_OUT and MAX_OUT, and searching only
     the non-zero values.


File: gsl-ref.info,  Node: Sparse matrix compressed format,  Next: Conversion between sparse and dense matrices,  Prev: Finding maximum and minimum elements of sparse matrices,  Up: Sparse Matrices

41.9 Sparse matrix compressed format
====================================

GSL supports the compressed column format, in which the non-zero
elements in each column are stored contiguously in memory.

 -- Function: gsl_spmatrix * gsl_spmatrix_compcol (const gsl_spmatrix *
          T)
     This function creates a sparse matrix in compressed column format
     from the input sparse matrix T which must be in triplet format.  A
     pointer to a newly allocated matrix is returned.  The calling
     function should free the newly allocated matrix when it is no
     longer needed.


File: gsl-ref.info,  Node: Conversion between sparse and dense matrices,  Next: Sparse Matrix Examples,  Prev: Sparse matrix compressed format,  Up: Sparse Matrices

41.10 Conversion between sparse and dense matrices
==================================================

The 'gsl_spmatrix' structure can be converted into the dense
'gsl_matrix' format and vice versa with the following routines.

 -- Function: int gsl_spmatrix_d2sp (gsl_spmatrix * S, const gsl_matrix
          * A)
     This function converts the dense matrix A into sparse triplet
     format and stores the result in S.

 -- Function: int gsl_spmatrix_sp2d (gsl_matrix * A, const gsl_spmatrix
          * S)
     This function converts the sparse matrix S into a dense matrix and
     stores the result in A.  S must be in triplet format.


File: gsl-ref.info,  Node: Sparse Matrix Examples,  Next: Sparse Matrix References and Further Reading,  Prev: Conversion between sparse and dense matrices,  Up: Sparse Matrices

41.11 Examples
==============

The following example program builds a 5-by-4 sparse matrix and prints
it in triplet and compressed column format.  The matrix which is
constructed is The output of the program is

     printing all matrix elements:
     A(0,0) = 0
     A(0,1) = 0
     A(0,2) = 3.1
     A(0,3) = 4.6
     A(1,0) = 1
     .
     .
     .
     A(4,0) = 4.1
     A(4,1) = 0
     A(4,2) = 0
     A(4,3) = 0
     matrix in triplet format (i,j,Aij):
     (0, 2, 3.1)
     (0, 3, 4.6)
     (1, 0, 1.0)
     (1, 2, 7.2)
     (3, 0, 2.1)
     (3, 1, 2.9)
     (3, 3, 8.5)
     (4, 0, 4.1)
     matrix in compressed column format:
     i = [ 1, 3, 4, 3, 0, 1, 0, 3, ]
     p = [ 0, 3, 4, 6, 8, ]
     d = [ 1, 2.1, 4.1, 2.9, 3.1, 7.2, 4.6, 8.5, ]
We see in the compressed column output, the data array stores each
column contiguously, the array i stores the row index of the
corresponding data element, and the array p stores the index of the
start of each column in the data array.

     #include <stdio.h>
     #include <stdlib.h>

     #include <gsl/gsl_spmatrix.h>

     int
     main()
     {
       gsl_spmatrix *A = gsl_spmatrix_alloc(5, 4); /* triplet format */
       gsl_spmatrix *C;
       size_t i, j;

       /* build the sparse matrix */
       gsl_spmatrix_set(A, 0, 2, 3.1);
       gsl_spmatrix_set(A, 0, 3, 4.6);
       gsl_spmatrix_set(A, 1, 0, 1.0);
       gsl_spmatrix_set(A, 1, 2, 7.2);
       gsl_spmatrix_set(A, 3, 0, 2.1);
       gsl_spmatrix_set(A, 3, 1, 2.9);
       gsl_spmatrix_set(A, 3, 3, 8.5);
       gsl_spmatrix_set(A, 4, 0, 4.1);

       printf("printing all matrix elements:\n");
       for (i = 0; i < 5; ++i)
         for (j = 0; j < 4; ++j)
           printf("A(%zu,%zu) = %g\n", i, j,
                  gsl_spmatrix_get(A, i, j));

       /* print out elements in triplet format */
       printf("matrix in triplet format (i,j,Aij):\n");
       for (i = 0; i < A->nz; ++i)
         printf("(%zu, %zu, %.1f)\n", A->i[i], A->p[i], A->data[i]);

       /* convert to compressed column format */
       C = gsl_spmatrix_compcol(A);

       printf("matrix in compressed column format:\n");
       printf("i = [ ");
       for (i = 0; i < C->nz; ++i)
         printf("%zu, ", C->i[i]);
       printf("]\n");

       printf("p = [ ");
       for (i = 0; i < C->size2 + 1; ++i)
         printf("%zu, ", C->p[i]);
       printf("]\n");

       printf("d = [ ");
       for (i = 0; i < C->nz; ++i)
         printf("%g, ", C->data[i]);
       printf("]\n");

       gsl_spmatrix_free(A);
       gsl_spmatrix_free(C);

       return 0;
     }


File: gsl-ref.info,  Node: Sparse Matrix References and Further Reading,  Prev: Sparse Matrix Examples,  Up: Sparse Matrices

41.12 References and Further Reading
====================================

The algorithms used by these functions are described in the following
sources:

     T. A. Davis, Direct Methods for Sparse Linear Systems, SIAM, 2006.

     CSparse software library,
     <https://www.cise.ufl.edu/research/sparse/CSparse>


File: gsl-ref.info,  Node: Sparse BLAS Support,  Next: Sparse Linear Algebra,  Prev: Sparse Matrices,  Up: Top

42 Sparse BLAS Support
**********************

The Sparse Basic Linear Algebra Subprograms (BLAS) define a set of
fundamental operations on vectors and sparse matrices which can be used
to create optimized higher-level linear algebra functionality.  GSL
supports a limited number of BLAS operations for sparse matrices.

The header file 'gsl_spblas.h' contains the prototypes for the sparse
BLAS functions and related declarations.

* Menu:

* Sparse BLAS operations::
* Sparse BLAS References and Further Reading::


File: gsl-ref.info,  Node: Sparse BLAS operations,  Next: Sparse BLAS References and Further Reading,  Up: Sparse BLAS Support

42.1 Sparse BLAS operations
===========================

 -- Function: int gsl_spblas_dgemv (const CBLAS_TRANSPOSE_t TransA,
          const double ALPHA, const gsl_spmatrix * A, const gsl_vector *
          X, const double BETA, gsl_vector * Y)
     This function computes the matrix-vector product and sum y
     \leftarrow \alpha op(A) x + \beta y, where op(A) = A, A^T for
     TRANSA = 'CblasNoTrans', 'CblasTrans'.  In-place computations are
     not supported, so X and Y must be distinct vectors.  The matrix A
     may be in triplet or compressed format.

 -- Function: int gsl_spblas_dgemm (const double ALPHA, const
          gsl_spmatrix * A, const gsl_spmatrix * B, gsl_spmatrix * C)
     This function computes the sparse matrix-matrix product C = \alpha
     A B.  The matrices must be in compressed format.


File: gsl-ref.info,  Node: Sparse BLAS References and Further Reading,  Prev: Sparse BLAS operations,  Up: Sparse BLAS Support

42.2 References and Further Reading
===================================

The algorithms used by these functions are described in the following
sources:

     T. A. Davis, Direct Methods for Sparse Linear Systems, SIAM, 2006.

     CSparse software library,
     <https://www.cise.ufl.edu/research/sparse/CSparse>


File: gsl-ref.info,  Node: Sparse Linear Algebra,  Next: Physical Constants,  Prev: Sparse BLAS Support,  Up: Top

43 Sparse Linear Algebra
************************

This chapter describes functions for solving sparse linear systems of
equations.  The library provides linear algebra routines which operate
directly on the 'gsl_spmatrix' and 'gsl_vector' objects.

The functions described in this chapter are declared in the header file
'gsl_splinalg.h'.

* Menu:

* Overview of Sparse Linear Algebra::
* Sparse Iterative Solvers::
* Sparse Linear Algebra Examples::
* Sparse Linear Algebra References and Further Reading::


File: gsl-ref.info,  Node: Overview of Sparse Linear Algebra,  Next: Sparse Iterative Solvers,  Up: Sparse Linear Algebra

43.1 Overview
=============

This chapter is primarily concerned with the solution of the linear
system
     A x = b
   where A is a general square n-by-n non-singular sparse matrix, x is
an unknown n-by-1 vector, and b is a given n-by-1 right hand side
vector.  There exist many methods for solving such sparse linear
systems, which broadly fall into either direct or iterative categories.
Direct methods include LU and QR decompositions, while iterative methods
start with an initial guess for the vector x and update the guess
through iteration until convergence.  GSL does not currently provide any
direct sparse solvers.


File: gsl-ref.info,  Node: Sparse Iterative Solvers,  Next: Sparse Linear Algebra Examples,  Prev: Overview of Sparse Linear Algebra,  Up: Sparse Linear Algebra

43.2 Sparse Iterative Solvers
=============================

* Menu:

* Sparse Iterative Solver Overview::
* Sparse Iterative Solvers Types::
* Iterating the Sparse Linear System::


File: gsl-ref.info,  Node: Sparse Iterative Solver Overview,  Next: Sparse Iterative Solvers Types,  Up: Sparse Iterative Solvers

43.2.1 Overview
---------------

Many practical iterative methods of solving large n-by-n sparse linear
systems involve projecting an approximate solution for X onto a subspace
of {\bf R}^n.  If we define a m-dimensional subspace {\cal K} as the
subspace of approximations to the solution X, then m constraints must be
imposed to determine the next approximation.  These m constraints define
another m-dimensional subspace denoted by {\cal L}.  The subspace
dimension m is typically chosen to be much smaller than n in order to
reduce the computational effort needed to generate the next approximate
solution vector.  The many iterative algorithms which exist differ
mainly in their choice of {\cal K} and {\cal L}.


File: gsl-ref.info,  Node: Sparse Iterative Solvers Types,  Next: Iterating the Sparse Linear System,  Prev: Sparse Iterative Solver Overview,  Up: Sparse Iterative Solvers

43.2.2 Types of Sparse Iterative Solvers
----------------------------------------

The sparse linear algebra library provides the following types of
iterative solvers:

 -- Sparse Iterative Type: gsl_splinalg_itersolve_gmres
     This specifies the Generalized Minimum Residual Method (GMRES).
     This is a projection method using {\cal K} = {\cal K}_m and {\cal
     L} = A {\cal K}_m where {\cal K}_m is the m-th Krylov subspace
          K_m = span( r_0, A r_0, ..., A^(m-1) r_0)
     and r_0 = b - A x_0 is the residual vector of the initial guess
     x_0.  If m is set equal to n, then the Krylov subspace is {\bf R}^n
     and GMRES will provide the exact solution X.  However, the goal is
     for the method to arrive at a very good approximation to X using a
     much smaller subspace {\cal K}_m.  By default, the GMRES method
     selects m = MIN(n,10) but the user may specify a different value
     for m.  The GMRES storage requirements grow as O(n(m+1)) and the
     number of flops grow as O(4 m^2 n - 4 m^3 / 3).

     In the below function 'gsl_splinalg_itersolve_iterate', one GMRES
     iteration is defined as projecting the approximate solution vector
     onto each Krylov subspace {\cal K}_1, ..., {\cal K}_m, and so m can
     be kept smaller by "restarting" the method and calling
     'gsl_splinalg_itersolve_iterate' multiple times, providing the
     updated approximation X to each new call.  If the method is not
     adequately converging, the user may try increasing the parameter m.

     GMRES is considered a robust general purpose iterative solver,
     however there are cases where the method stagnates if the matrix is
     not positive-definite and fails to reduce the residual until the
     very last projection onto the subspace {\cal K}_n = {\bf R}^n.  In
     these cases, preconditioning the linear system can help, but GSL
     does not currently provide any preconditioners.


File: gsl-ref.info,  Node: Iterating the Sparse Linear System,  Prev: Sparse Iterative Solvers Types,  Up: Sparse Iterative Solvers

43.2.3 Iterating the Sparse Linear System
-----------------------------------------

The following functions are provided to allocate storage for the sparse
linear solvers and iterate the system to a solution.

 -- Function: gsl_splinalg_itersolve * gsl_splinalg_itersolve_alloc
          (const gsl_splinalg_itersolve_type * T, const size_t N, const
          size_t M)
     This function allocates a workspace for the iterative solution of
     N-by-N sparse matrix systems.  The iterative solver type is
     specified by T.  The argument M specifies the size of the solution
     candidate subspace {\cal K}_m.  The dimension M may be set to 0 in
     which case a reasonable default value is used.

 -- Function: void gsl_splinalg_itersolve_free (gsl_splinalg_itersolve *
          W)
     This function frees the memory associated with the workspace W.

 -- Function: const char * gsl_splinalg_itersolve_name (const
          gsl_splinalg_itersolve * W)
     This function returns a string pointer to the name of the solver.

 -- Function: int gsl_splinalg_itersolve_iterate (const gsl_spmatrix *A,
          const gsl_vector *B, const double TOL, gsl_vector *X,
          gsl_splinalg_itersolve *W)
     This function performs one iteration of the iterative method for
     the sparse linear system specfied by the matrix A, right hand side
     vector B and solution vector X.  On input, X must be set to an
     initial guess for the solution.  On output, X is updated to give
     the current solution estimate.  The parameter TOL specifies the
     relative tolerance between the residual norm and norm of B in order
     to check for convergence.  When the following condition is
     satisfied:
          || A x - b || <= tol * || b ||
     the method has converged, the function returns 'GSL_SUCCESS' and
     the final solution is provided in X.  Otherwise, the function
     returns 'GSL_CONTINUE' to signal that more iterations are required.
     Here, || \cdot || represents the Euclidean norm.  The input matrix
     A may be in triplet or compressed column format.

 -- Function: double gsl_splinalg_itersolve_normr (const
          gsl_splinalg_itersolve *W)
     This function returns the current residual norm ||r|| = ||A x -
     b||, which is updated after each call to
     'gsl_splinalg_itersolve_iterate'.


File: gsl-ref.info,  Node: Sparse Linear Algebra Examples,  Next: Sparse Linear Algebra References and Further Reading,  Prev: Sparse Iterative Solvers,  Up: Sparse Linear Algebra

43.3 Examples
=============

This example program demonstrates the sparse linear algebra routines on
the solution of a simple 1D Poisson equation on [0,1]:
     u''(x) = f(x) = -\pi^2 \sin(\pi x)
   with boundary conditions u(0) = u(1) = 0.  The analytic solution of
this simple problem is u(x) = \sin{\pi x}.  We will solve this problem
by finite differencing the left hand side to give
     1/h^2 ( u_(i+1) - 2 u_i + u_(i-1) ) = f_i
   Defining a grid of N points, h = 1/(N-1).  In the finite difference
equation above, u_0 = u_{N-1} = 0 are known from the boundary
conditions, so we will only put the equations for i = 1, ..., N-2 into
the matrix system.  The resulting (N-2) \times (N-2) matrix equation is
An example program which constructs and solves this system is given
below.  The system is solved using the iterative GMRES solver.  Here is
the output of the program:

     iter 0 residual = 4.297275996844e-11
     Converged
showing that the method converged in a single iteration.  The calculated
solution is shown in the following plot.

     #include <stdio.h>
     #include <stdlib.h>
     #include <math.h>

     #include <gsl/gsl_math.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_spmatrix.h>
     #include <gsl/gsl_splinalg.h>

     int
     main()
     {
       const size_t N = 100;                       /* number of grid points */
       const size_t n = N - 2;                     /* subtract 2 to exclude boundaries */
       const double h = 1.0 / (N - 1.0);           /* grid spacing */
       gsl_spmatrix *A = gsl_spmatrix_alloc(n ,n); /* triplet format */
       gsl_spmatrix *C;                            /* compressed format */
       gsl_vector *f = gsl_vector_alloc(n);        /* right hand side vector */
       gsl_vector *u = gsl_vector_alloc(n);        /* solution vector */
       size_t i;

       /* construct the sparse matrix for the finite difference equation */

       /* construct first row */
       gsl_spmatrix_set(A, 0, 0, -2.0);
       gsl_spmatrix_set(A, 0, 1, 1.0);

       /* construct rows [1:n-2] */
       for (i = 1; i < n - 1; ++i)
         {
           gsl_spmatrix_set(A, i, i + 1, 1.0);
           gsl_spmatrix_set(A, i, i, -2.0);
           gsl_spmatrix_set(A, i, i - 1, 1.0);
         }

       /* construct last row */
       gsl_spmatrix_set(A, n - 1, n - 1, -2.0);
       gsl_spmatrix_set(A, n - 1, n - 2, 1.0);

       /* scale by h^2 */
       gsl_spmatrix_scale(A, 1.0 / (h * h));

       /* construct right hand side vector */
       for (i = 0; i < n; ++i)
         {
           double xi = (i + 1) * h;
           double fi = -M_PI * M_PI * sin(M_PI * xi);
           gsl_vector_set(f, i, fi);
         }

       /* convert to compressed column format */
       C = gsl_spmatrix_compcol(A);

       /* now solve the system with the GMRES iterative solver */
       {
         const double tol = 1.0e-6;  /* solution relative tolerance */
         const size_t max_iter = 10; /* maximum iterations */
         const gsl_splinalg_itersolve_type *T = gsl_splinalg_itersolve_gmres;
         gsl_splinalg_itersolve *work =
           gsl_splinalg_itersolve_alloc(T, n, 0);
         size_t iter = 0;
         double residual;
         int status;

         /* initial guess u = 0 */
         gsl_vector_set_zero(u);

         /* solve the system A u = f */
         do
           {
             status = gsl_splinalg_itersolve_iterate(C, f, tol, u, work);

             /* print out residual norm ||A*u - f|| */
             residual = gsl_splinalg_itersolve_normr(work);
             fprintf(stderr, "iter %zu residual = %.12e\n", iter, residual);

             if (status == GSL_SUCCESS)
               fprintf(stderr, "Converged\n");
           }
         while (status == GSL_CONTINUE && ++iter < max_iter);

         /* output solution */
         for (i = 0; i < n; ++i)
           {
             double xi = (i + 1) * h;
             double u_exact = sin(M_PI * xi);
             double u_gsl = gsl_vector_get(u, i);

             printf("%f %.12e %.12e\n", xi, u_gsl, u_exact);
           }

         gsl_splinalg_itersolve_free(work);
       }

       gsl_spmatrix_free(A);
       gsl_spmatrix_free(C);
       gsl_vector_free(f);
       gsl_vector_free(u);

       return 0;
     } /* main() */


File: gsl-ref.info,  Node: Sparse Linear Algebra References and Further Reading,  Prev: Sparse Linear Algebra Examples,  Up: Sparse Linear Algebra

43.4 References and Further Reading
===================================

The implementation of the GMRES iterative solver closely follows the
publications

     H. F. Walker, Implementation of the GMRES method using Householder
     transformations, SIAM J. Sci.  Stat.  Comput.  9(1), 1988.

     Y. Saad, Iterative methods for sparse linear systems, 2nd edition,
     SIAM, 2003.


File: gsl-ref.info,  Node: Physical Constants,  Next: IEEE floating-point arithmetic,  Prev: Sparse Linear Algebra,  Up: Top

44 Physical Constants
*********************

This chapter describes macros for the values of physical constants, such
as the speed of light, c, and gravitational constant, G.  The values are
available in different unit systems, including the standard MKSA system
(meters, kilograms, seconds, amperes) and the CGSM system (centimeters,
grams, seconds, gauss), which is commonly used in Astronomy.

   The definitions of constants in the MKSA system are available in the
file 'gsl_const_mksa.h'.  The constants in the CGSM system are defined
in 'gsl_const_cgsm.h'.  Dimensionless constants, such as the fine
structure constant, which are pure numbers are defined in
'gsl_const_num.h'.

* Menu:

* Fundamental Constants::       
* Astronomy and Astrophysics::  
* Atomic and Nuclear Physics::  
* Measurement of Time::         
* Imperial Units ::             
* Speed and Nautical Units::    
* Printers Units::              
* Volume Area and Length::      
* Mass and Weight ::            
* Thermal Energy and Power::    
* Pressure::                    
* Viscosity::                   
* Light and Illumination::      
* Radioactivity::               
* Force and Energy::            
* Prefixes::                    
* Physical Constant Examples::  
* Physical Constant References and Further Reading::  

   The full list of constants is described briefly below.  Consult the
header files themselves for the values of the constants used in the
library.


File: gsl-ref.info,  Node: Fundamental Constants,  Next: Astronomy and Astrophysics,  Up: Physical Constants

44.1 Fundamental Constants
==========================

'GSL_CONST_MKSA_SPEED_OF_LIGHT'
     The speed of light in vacuum, c.

'GSL_CONST_MKSA_VACUUM_PERMEABILITY'
     The permeability of free space, \mu_0.  This constant is defined in
     the MKSA system only.

'GSL_CONST_MKSA_VACUUM_PERMITTIVITY'
     The permittivity of free space, \epsilon_0.  This constant is
     defined in the MKSA system only.

'GSL_CONST_MKSA_PLANCKS_CONSTANT_H'
     Planck's constant, h.

'GSL_CONST_MKSA_PLANCKS_CONSTANT_HBAR'
     Planck's constant divided by 2\pi, \hbar.

'GSL_CONST_NUM_AVOGADRO'
     Avogadro's number, N_a.

'GSL_CONST_MKSA_FARADAY'
     The molar charge of 1 Faraday.

'GSL_CONST_MKSA_BOLTZMANN'
     The Boltzmann constant, k.

'GSL_CONST_MKSA_MOLAR_GAS'
     The molar gas constant, R_0.

'GSL_CONST_MKSA_STANDARD_GAS_VOLUME'
     The standard gas volume, V_0.

'GSL_CONST_MKSA_STEFAN_BOLTZMANN_CONSTANT'
     The Stefan-Boltzmann radiation constant, \sigma.

'GSL_CONST_MKSA_GAUSS'
     The magnetic field of 1 Gauss.


File: gsl-ref.info,  Node: Astronomy and Astrophysics,  Next: Atomic and Nuclear Physics,  Prev: Fundamental Constants,  Up: Physical Constants

44.2 Astronomy and Astrophysics
===============================

'GSL_CONST_MKSA_ASTRONOMICAL_UNIT'
     The length of 1 astronomical unit (mean earth-sun distance), au.

'GSL_CONST_MKSA_GRAVITATIONAL_CONSTANT'
     The gravitational constant, G.

'GSL_CONST_MKSA_LIGHT_YEAR'
     The distance of 1 light-year, ly.

'GSL_CONST_MKSA_PARSEC'
     The distance of 1 parsec, pc.

'GSL_CONST_MKSA_GRAV_ACCEL'
     The standard gravitational acceleration on Earth, g.

'GSL_CONST_MKSA_SOLAR_MASS'
     The mass of the Sun.


File: gsl-ref.info,  Node: Atomic and Nuclear Physics,  Next: Measurement of Time,  Prev: Astronomy and Astrophysics,  Up: Physical Constants

44.3 Atomic and Nuclear Physics
===============================

'GSL_CONST_MKSA_ELECTRON_CHARGE'
     The charge of the electron, e.

'GSL_CONST_MKSA_ELECTRON_VOLT'
     The energy of 1 electron volt, eV.

'GSL_CONST_MKSA_UNIFIED_ATOMIC_MASS'
     The unified atomic mass, amu.

'GSL_CONST_MKSA_MASS_ELECTRON'
     The mass of the electron, m_e.

'GSL_CONST_MKSA_MASS_MUON'
     The mass of the muon, m_\mu.

'GSL_CONST_MKSA_MASS_PROTON'
     The mass of the proton, m_p.

'GSL_CONST_MKSA_MASS_NEUTRON'
     The mass of the neutron, m_n.

'GSL_CONST_NUM_FINE_STRUCTURE'
     The electromagnetic fine structure constant \alpha.

'GSL_CONST_MKSA_RYDBERG'
     The Rydberg constant, Ry, in units of energy.  This is related to
     the Rydberg inverse wavelength R_\infty by Ry = h c R_\infty.

'GSL_CONST_MKSA_BOHR_RADIUS'
     The Bohr radius, a_0.

'GSL_CONST_MKSA_ANGSTROM'
     The length of 1 angstrom.

'GSL_CONST_MKSA_BARN'
     The area of 1 barn.

'GSL_CONST_MKSA_BOHR_MAGNETON'
     The Bohr Magneton, \mu_B.

'GSL_CONST_MKSA_NUCLEAR_MAGNETON'
     The Nuclear Magneton, \mu_N.

'GSL_CONST_MKSA_ELECTRON_MAGNETIC_MOMENT'
     The absolute value of the magnetic moment of the electron, \mu_e.
     The physical magnetic moment of the electron is negative.

'GSL_CONST_MKSA_PROTON_MAGNETIC_MOMENT'
     The magnetic moment of the proton, \mu_p.

'GSL_CONST_MKSA_THOMSON_CROSS_SECTION'
     The Thomson cross section, \sigma_T.

'GSL_CONST_MKSA_DEBYE'
     The electric dipole moment of 1 Debye, D.


File: gsl-ref.info,  Node: Measurement of Time,  Next: Imperial Units,  Prev: Atomic and Nuclear Physics,  Up: Physical Constants

44.4 Measurement of Time
========================

'GSL_CONST_MKSA_MINUTE'
     The number of seconds in 1 minute.

'GSL_CONST_MKSA_HOUR'
     The number of seconds in 1 hour.

'GSL_CONST_MKSA_DAY'
     The number of seconds in 1 day.

'GSL_CONST_MKSA_WEEK'
     The number of seconds in 1 week.


File: gsl-ref.info,  Node: Imperial Units,  Next: Speed and Nautical Units,  Prev: Measurement of Time,  Up: Physical Constants

44.5 Imperial Units
===================

'GSL_CONST_MKSA_INCH'
     The length of 1 inch.

'GSL_CONST_MKSA_FOOT'
     The length of 1 foot.

'GSL_CONST_MKSA_YARD'
     The length of 1 yard.

'GSL_CONST_MKSA_MILE'
     The length of 1 mile.

'GSL_CONST_MKSA_MIL'
     The length of 1 mil (1/1000th of an inch).


File: gsl-ref.info,  Node: Speed and Nautical Units,  Next: Printers Units,  Prev: Imperial Units,  Up: Physical Constants

44.6 Speed and Nautical Units
=============================

'GSL_CONST_MKSA_KILOMETERS_PER_HOUR'
     The speed of 1 kilometer per hour.

'GSL_CONST_MKSA_MILES_PER_HOUR'
     The speed of 1 mile per hour.

'GSL_CONST_MKSA_NAUTICAL_MILE'
     The length of 1 nautical mile.

'GSL_CONST_MKSA_FATHOM'
     The length of 1 fathom.

'GSL_CONST_MKSA_KNOT'
     The speed of 1 knot.


File: gsl-ref.info,  Node: Printers Units,  Next: Volume Area and Length,  Prev: Speed and Nautical Units,  Up: Physical Constants

44.7 Printers Units
===================

'GSL_CONST_MKSA_POINT'
     The length of 1 printer's point (1/72 inch).

'GSL_CONST_MKSA_TEXPOINT'
     The length of 1 TeX point (1/72.27 inch).


File: gsl-ref.info,  Node: Volume Area and Length,  Next: Mass and Weight,  Prev: Printers Units,  Up: Physical Constants

44.8 Volume, Area and Length
============================

'GSL_CONST_MKSA_MICRON'
     The length of 1 micron.

'GSL_CONST_MKSA_HECTARE'
     The area of 1 hectare.

'GSL_CONST_MKSA_ACRE'
     The area of 1 acre.

'GSL_CONST_MKSA_LITER'
     The volume of 1 liter.

'GSL_CONST_MKSA_US_GALLON'
     The volume of 1 US gallon.

'GSL_CONST_MKSA_CANADIAN_GALLON'
     The volume of 1 Canadian gallon.

'GSL_CONST_MKSA_UK_GALLON'
     The volume of 1 UK gallon.

'GSL_CONST_MKSA_QUART'
     The volume of 1 quart.

'GSL_CONST_MKSA_PINT'
     The volume of 1 pint.


File: gsl-ref.info,  Node: Mass and Weight,  Next: Thermal Energy and Power,  Prev: Volume Area and Length,  Up: Physical Constants

44.9 Mass and Weight
====================

'GSL_CONST_MKSA_POUND_MASS'
     The mass of 1 pound.

'GSL_CONST_MKSA_OUNCE_MASS'
     The mass of 1 ounce.

'GSL_CONST_MKSA_TON'
     The mass of 1 ton.

'GSL_CONST_MKSA_METRIC_TON'
     The mass of 1 metric ton (1000 kg).

'GSL_CONST_MKSA_UK_TON'
     The mass of 1 UK ton.

'GSL_CONST_MKSA_TROY_OUNCE'
     The mass of 1 troy ounce.

'GSL_CONST_MKSA_CARAT'
     The mass of 1 carat.

'GSL_CONST_MKSA_GRAM_FORCE'
     The force of 1 gram weight.

'GSL_CONST_MKSA_POUND_FORCE'
     The force of 1 pound weight.

'GSL_CONST_MKSA_KILOPOUND_FORCE'
     The force of 1 kilopound weight.

'GSL_CONST_MKSA_POUNDAL'
     The force of 1 poundal.


File: gsl-ref.info,  Node: Thermal Energy and Power,  Next: Pressure,  Prev: Mass and Weight,  Up: Physical Constants

44.10 Thermal Energy and Power
==============================

'GSL_CONST_MKSA_CALORIE'
     The energy of 1 calorie.

'GSL_CONST_MKSA_BTU'
     The energy of 1 British Thermal Unit, btu.

'GSL_CONST_MKSA_THERM'
     The energy of 1 Therm.

'GSL_CONST_MKSA_HORSEPOWER'
     The power of 1 horsepower.


File: gsl-ref.info,  Node: Pressure,  Next: Viscosity,  Prev: Thermal Energy and Power,  Up: Physical Constants

44.11 Pressure
==============

'GSL_CONST_MKSA_BAR'
     The pressure of 1 bar.

'GSL_CONST_MKSA_STD_ATMOSPHERE'
     The pressure of 1 standard atmosphere.

'GSL_CONST_MKSA_TORR'
     The pressure of 1 torr.

'GSL_CONST_MKSA_METER_OF_MERCURY'
     The pressure of 1 meter of mercury.

'GSL_CONST_MKSA_INCH_OF_MERCURY'
     The pressure of 1 inch of mercury.

'GSL_CONST_MKSA_INCH_OF_WATER'
     The pressure of 1 inch of water.

'GSL_CONST_MKSA_PSI'
     The pressure of 1 pound per square inch.


File: gsl-ref.info,  Node: Viscosity,  Next: Light and Illumination,  Prev: Pressure,  Up: Physical Constants

44.12 Viscosity
===============

'GSL_CONST_MKSA_POISE'
     The dynamic viscosity of 1 poise.

'GSL_CONST_MKSA_STOKES'
     The kinematic viscosity of 1 stokes.


File: gsl-ref.info,  Node: Light and Illumination,  Next: Radioactivity,  Prev: Viscosity,  Up: Physical Constants

44.13 Light and Illumination
============================

'GSL_CONST_MKSA_STILB'
     The luminance of 1 stilb.

'GSL_CONST_MKSA_LUMEN'
     The luminous flux of 1 lumen.

'GSL_CONST_MKSA_LUX'
     The illuminance of 1 lux.

'GSL_CONST_MKSA_PHOT'
     The illuminance of 1 phot.

'GSL_CONST_MKSA_FOOTCANDLE'
     The illuminance of 1 footcandle.

'GSL_CONST_MKSA_LAMBERT'
     The luminance of 1 lambert.

'GSL_CONST_MKSA_FOOTLAMBERT'
     The luminance of 1 footlambert.


File: gsl-ref.info,  Node: Radioactivity,  Next: Force and Energy,  Prev: Light and Illumination,  Up: Physical Constants

44.14 Radioactivity
===================

'GSL_CONST_MKSA_CURIE'
     The activity of 1 curie.

'GSL_CONST_MKSA_ROENTGEN'
     The exposure of 1 roentgen.

'GSL_CONST_MKSA_RAD'
     The absorbed dose of 1 rad.


File: gsl-ref.info,  Node: Force and Energy,  Next: Prefixes,  Prev: Radioactivity,  Up: Physical Constants

44.15 Force and Energy
======================

'GSL_CONST_MKSA_NEWTON'
     The SI unit of force, 1 Newton.

'GSL_CONST_MKSA_DYNE'
     The force of 1 Dyne = 10^-5 Newton.

'GSL_CONST_MKSA_JOULE'
     The SI unit of energy, 1 Joule.

'GSL_CONST_MKSA_ERG'
     The energy 1 erg = 10^-7 Joule.


File: gsl-ref.info,  Node: Prefixes,  Next: Physical Constant Examples,  Prev: Force and Energy,  Up: Physical Constants

44.16 Prefixes
==============

These constants are dimensionless scaling factors.

'GSL_CONST_NUM_YOTTA'
     10^24

'GSL_CONST_NUM_ZETTA'
     10^21

'GSL_CONST_NUM_EXA'
     10^18

'GSL_CONST_NUM_PETA'
     10^15

'GSL_CONST_NUM_TERA'
     10^12

'GSL_CONST_NUM_GIGA'
     10^9

'GSL_CONST_NUM_MEGA'
     10^6

'GSL_CONST_NUM_KILO'
     10^3

'GSL_CONST_NUM_MILLI'
     10^-3

'GSL_CONST_NUM_MICRO'
     10^-6

'GSL_CONST_NUM_NANO'
     10^-9

'GSL_CONST_NUM_PICO'
     10^-12

'GSL_CONST_NUM_FEMTO'
     10^-15

'GSL_CONST_NUM_ATTO'
     10^-18

'GSL_CONST_NUM_ZEPTO'
     10^-21

'GSL_CONST_NUM_YOCTO'
     10^-24


File: gsl-ref.info,  Node: Physical Constant Examples,  Next: Physical Constant References and Further Reading,  Prev: Prefixes,  Up: Physical Constants

44.17 Examples
==============

The following program demonstrates the use of the physical constants in
a calculation.  In this case, the goal is to calculate the range of
light-travel times from Earth to Mars.

   The required data is the average distance of each planet from the Sun
in astronomical units (the eccentricities and inclinations of the orbits
will be neglected for the purposes of this calculation).  The average
radius of the orbit of Mars is 1.52 astronomical units, and for the
orbit of Earth it is 1 astronomical unit (by definition).  These values
are combined with the MKSA values of the constants for the speed of
light and the length of an astronomical unit to produce a result for the
shortest and longest light-travel times in seconds.  The figures are
converted into minutes before being displayed.

     #include <stdio.h>
     #include <gsl/gsl_const_mksa.h>

     int
     main (void)
     {
       double c  = GSL_CONST_MKSA_SPEED_OF_LIGHT;
       double au = GSL_CONST_MKSA_ASTRONOMICAL_UNIT;
       double minutes = GSL_CONST_MKSA_MINUTE;

       /* distance stored in meters */
       double r_earth = 1.00 * au;
       double r_mars  = 1.52 * au;

       double t_min, t_max;

       t_min = (r_mars - r_earth) / c;
       t_max = (r_mars + r_earth) / c;

       printf ("light travel time from Earth to Mars:\n");
       printf ("minimum = %.1f minutes\n", t_min / minutes);
       printf ("maximum = %.1f minutes\n", t_max / minutes);

       return 0;
     }

Here is the output from the program,

     light travel time from Earth to Mars:
     minimum = 4.3 minutes
     maximum = 21.0 minutes


File: gsl-ref.info,  Node: Physical Constant References and Further Reading,  Prev: Physical Constant Examples,  Up: Physical Constants

44.18 References and Further Reading
====================================

The authoritative sources for physical constants are the 2006 CODATA
recommended values, published in the article below.  Further information
on the values of physical constants is also available from the NIST
website.

     P.J. Mohr, B.N. Taylor, D.B. Newell, "CODATA Recommended Values of
     the Fundamental Physical Constants: 2006", Reviews of Modern
     Physics, 80(2), pp.  633-730 (2008).
     <http://www.physics.nist.gov/cuu/Constants/index.html>
     <http://physics.nist.gov/Pubs/SP811/appenB9.html>


File: gsl-ref.info,  Node: IEEE floating-point arithmetic,  Next: Debugging Numerical Programs,  Prev: Physical Constants,  Up: Top

45 IEEE floating-point arithmetic
*********************************

This chapter describes functions for examining the representation of
floating point numbers and controlling the floating point environment of
your program.  The functions described in this chapter are declared in
the header file 'gsl_ieee_utils.h'.

* Menu:

* Representation of floating point numbers::  
* Setting up your IEEE environment::  
* IEEE References and Further Reading::  


File: gsl-ref.info,  Node: Representation of floating point numbers,  Next: Setting up your IEEE environment,  Up: IEEE floating-point arithmetic

45.1 Representation of floating point numbers
=============================================

The IEEE Standard for Binary Floating-Point Arithmetic defines binary
formats for single and double precision numbers.  Each number is
composed of three parts: a "sign bit" (s), an "exponent" (E) and a
"fraction" (f).  The numerical value of the combination (s,E,f) is given
by the following formula,

     (-1)^s (1.fffff...) 2^E

The sign bit is either zero or one.  The exponent ranges from a minimum
value E_min to a maximum value E_max depending on the precision.  The
exponent is converted to an unsigned number e, known as the "biased
exponent", for storage by adding a "bias" parameter, e = E + bias.  The
sequence fffff... represents the digits of the binary fraction f.  The
binary digits are stored in "normalized form", by adjusting the exponent
to give a leading digit of 1.  Since the leading digit is always 1 for
normalized numbers it is assumed implicitly and does not have to be
stored.  Numbers smaller than 2^(E_min) are be stored in "denormalized
form" with a leading zero,

     (-1)^s (0.fffff...) 2^(E_min)

This allows gradual underflow down to 2^(E_min - p) for p bits of
precision.  A zero is encoded with the special exponent of 2^(E_min - 1)
and infinities with the exponent of 2^(E_max + 1).

The format for single precision numbers uses 32 bits divided in the
following way,

     seeeeeeeefffffffffffffffffffffff

     s = sign bit, 1 bit
     e = exponent, 8 bits  (E_min=-126, E_max=127, bias=127)
     f = fraction, 23 bits

The format for double precision numbers uses 64 bits divided in the
following way,

     seeeeeeeeeeeffffffffffffffffffffffffffffffffffffffffffffffffffff

     s = sign bit, 1 bit
     e = exponent, 11 bits  (E_min=-1022, E_max=1023, bias=1023)
     f = fraction, 52 bits

It is often useful to be able to investigate the behavior of a
calculation at the bit-level and the library provides functions for
printing the IEEE representations in a human-readable form.

 -- Function: void gsl_ieee_fprintf_float (FILE * STREAM, const float *
          X)
 -- Function: void gsl_ieee_fprintf_double (FILE * STREAM, const double
          * X)
     These functions output a formatted version of the IEEE
     floating-point number pointed to by X to the stream STREAM.  A
     pointer is used to pass the number indirectly, to avoid any
     undesired promotion from 'float' to 'double'.  The output takes one
     of the following forms,

     'NaN'
          the Not-a-Number symbol

     'Inf, -Inf'
          positive or negative infinity

     '1.fffff...*2^E, -1.fffff...*2^E'
          a normalized floating point number

     '0.fffff...*2^E, -0.fffff...*2^E'
          a denormalized floating point number

     '0, -0'
          positive or negative zero

     The output can be used directly in GNU Emacs Calc mode by preceding
     it with '2#' to indicate binary.

 -- Function: void gsl_ieee_printf_float (const float * X)
 -- Function: void gsl_ieee_printf_double (const double * X)
     These functions output a formatted version of the IEEE
     floating-point number pointed to by X to the stream 'stdout'.

The following program demonstrates the use of the functions by printing
the single and double precision representations of the fraction 1/3.
For comparison the representation of the value promoted from single to
double precision is also printed.

     #include <stdio.h>
     #include <gsl/gsl_ieee_utils.h>

     int
     main (void)
     {
       float f = 1.0/3.0;
       double d = 1.0/3.0;

       double fd = f; /* promote from float to double */

       printf (" f="); gsl_ieee_printf_float(&f);
       printf ("\n");

       printf ("fd="); gsl_ieee_printf_double(&fd);
       printf ("\n");

       printf (" d="); gsl_ieee_printf_double(&d);
       printf ("\n");

       return 0;
     }

The binary representation of 1/3 is 0.01010101... .  The output below
shows that the IEEE format normalizes this fraction to give a leading
digit of 1,

      f= 1.01010101010101010101011*2^-2
     fd= 1.0101010101010101010101100000000000000000000000000000*2^-2
      d= 1.0101010101010101010101010101010101010101010101010101*2^-2

The output also shows that a single-precision number is promoted to
double-precision by adding zeros in the binary representation.


File: gsl-ref.info,  Node: Setting up your IEEE environment,  Next: IEEE References and Further Reading,  Prev: Representation of floating point numbers,  Up: IEEE floating-point arithmetic

45.2 Setting up your IEEE environment
=====================================

The IEEE standard defines several "modes" for controlling the behavior
of floating point operations.  These modes specify the important
properties of computer arithmetic: the direction used for rounding (e.g.
whether numbers should be rounded up, down or to the nearest number),
the rounding precision and how the program should handle arithmetic
exceptions, such as division by zero.

   Many of these features can now be controlled via standard functions
such as 'fpsetround', which should be used whenever they are available.
Unfortunately in the past there has been no universal API for
controlling their behavior--each system has had its own low-level way of
accessing them.  To help you write portable programs GSL allows you to
specify modes in a platform-independent way using the environment
variable 'GSL_IEEE_MODE'.  The library then takes care of all the
necessary machine-specific initializations for you when you call the
function 'gsl_ieee_env_setup'.

 -- Function: void gsl_ieee_env_setup ()
     This function reads the environment variable 'GSL_IEEE_MODE' and
     attempts to set up the corresponding specified IEEE modes.  The
     environment variable should be a list of keywords, separated by
     commas, like this,

          'GSL_IEEE_MODE' = "KEYWORD,KEYWORD,..."

     where KEYWORD is one of the following mode-names,

          'single-precision'
          'double-precision'
          'extended-precision'
          'round-to-nearest'
          'round-down'
          'round-up'
          'round-to-zero'
          'mask-all'
          'mask-invalid'
          'mask-denormalized'
          'mask-division-by-zero'
          'mask-overflow'
          'mask-underflow'
          'trap-inexact'
          'trap-common'

     If 'GSL_IEEE_MODE' is empty or undefined then the function returns
     immediately and no attempt is made to change the system's IEEE
     mode.  When the modes from 'GSL_IEEE_MODE' are turned on the
     function prints a short message showing the new settings to remind
     you that the results of the program will be affected.

     If the requested modes are not supported by the platform being used
     then the function calls the error handler and returns an error code
     of 'GSL_EUNSUP'.

     When options are specified using this method, the resulting mode is
     based on a default setting of the highest available precision
     (double precision or extended precision, depending on the platform)
     in round-to-nearest mode, with all exceptions enabled apart from
     the INEXACT exception.  The INEXACT exception is generated whenever
     rounding occurs, so it must generally be disabled in typical
     scientific calculations.  All other floating-point exceptions are
     enabled by default, including underflows and the use of
     denormalized numbers, for safety.  They can be disabled with the
     individual 'mask-' settings or together using 'mask-all'.

     The following adjusted combination of modes is convenient for many
     purposes,

          GSL_IEEE_MODE="double-precision,"\
                          "mask-underflow,"\
                            "mask-denormalized"

     This choice ignores any errors relating to small numbers (either
     denormalized, or underflowing to zero) but traps overflows,
     division by zero and invalid operations.

     Note that on the x86 series of processors this function sets both
     the original x87 mode and the newer MXCSR mode, which controls SSE
     floating-point operations.  The SSE floating-point units do not
     have a precision-control bit, and always work in double-precision.
     The single-precision and extended-precision keywords have no effect
     in this case.

To demonstrate the effects of different rounding modes consider the
following program which computes e, the base of natural logarithms, by
summing a rapidly-decreasing series,

     e = 1 + 1/2! + 1/3! + 1/4! + ...
       = 2.71828182846...

     #include <stdio.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_ieee_utils.h>

     int
     main (void)
     {
       double x = 1, oldsum = 0, sum = 0;
       int i = 0;

       gsl_ieee_env_setup (); /* read GSL_IEEE_MODE */

       do
         {
           i++;

           oldsum = sum;
           sum += x;
           x = x / i;

           printf ("i=%2d sum=%.18f error=%g\n",
                   i, sum, sum - M_E);

           if (i > 30)
              break;
         }
       while (sum != oldsum);

       return 0;
     }

Here are the results of running the program in 'round-to-nearest' mode.
This is the IEEE default so it isn't really necessary to specify it
here,

     $ GSL_IEEE_MODE="round-to-nearest" ./a.out
     i= 1 sum=1.000000000000000000 error=-1.71828
     i= 2 sum=2.000000000000000000 error=-0.718282
     ....
     i=18 sum=2.718281828459045535 error=4.44089e-16
     i=19 sum=2.718281828459045535 error=4.44089e-16

After nineteen terms the sum converges to within 4 \times 10^-16 of the
correct value.  If we now change the rounding mode to 'round-down' the
final result is less accurate,

     $ GSL_IEEE_MODE="round-down" ./a.out
     i= 1 sum=1.000000000000000000 error=-1.71828
     ....
     i=19 sum=2.718281828459041094 error=-3.9968e-15

The result is about 4 \times 10^-15 below the correct value, an order of
magnitude worse than the result obtained in the 'round-to-nearest' mode.

   If we change to rounding mode to 'round-up' then the final result is
higher than the correct value (when we add each term to the sum the
final result is always rounded up, which increases the sum by at least
one tick until the added term underflows to zero).  To avoid this
problem we would need to use a safer converge criterion, such as 'while
(fabs(sum - oldsum) > epsilon)', with a suitably chosen value of
epsilon.

   Finally we can see the effect of computing the sum using
single-precision rounding, in the default 'round-to-nearest' mode.  In
this case the program thinks it is still using double precision numbers
but the CPU rounds the result of each floating point operation to
single-precision accuracy.  This simulates the effect of writing the
program using single-precision 'float' variables instead of 'double'
variables.  The iteration stops after about half the number of
iterations and the final result is much less accurate,

     $ GSL_IEEE_MODE="single-precision" ./a.out
     ....
     i=12 sum=2.718281984329223633 error=1.5587e-07

with an error of O(10^-7), which corresponds to single precision
accuracy (about 1 part in 10^7).  Continuing the iterations further does
not decrease the error because all the subsequent results are rounded to
the same value.

