The Optimizer Algorithm

This section explains the optimization algorithms included in IC-CAP.

Random Optimization Algorithm

The Random optimizer uses a simplified stochastic gradient algorithm. Initially an update vector is chosen using a spherical Gaussian distribution (that is, one that favors no particular direction in the optimization variable space).

A trial point is calculated by adding the update vector to the current point and then the cost is calculated at the trial point. If the cost is less than the initial cost, the trial point is accepted and the Gaussian distribution is modified to an ellipsoid that favors the update direction. If the cost is not decreased, a new trial point is calculated by subtracting the update direction from the initial point and the cost is recalculated. If the cost decreases, then the trial point is accepted and the Gaussian distribution is modified to favor the update direction. If the cost does not decrease, the trial point is rejected and the Gaussian distribution is modified so that the unsuccessful update direction is less favored. The whole process is then repeated until either the RMS is reduced to the specified value or the number of iterations is specified by the Rand Iter parameter is reached.

If there are several successive successful steps, then the Gaussian distribution becomes elongated in the direction of the jumps. Several successive failures tend to shrink the Gaussian distribution in the direction of the failed jumps. Because of this, with success, the Random optimizer acts more aggressively in its steps and, with failure, it acts more conservatively.

The Rand Reward parameter controls how much the Gaussian distribution is expanded upon a successful step. A large value increases the expansion. A value of zero eliminates any change in the Gaussian distribution. Likewise, the Rand Penalty parameter controls the compression effects of an unsuccessful step.

Hybrid (Random/LM) Optimization Algorithm

The Hybrid (Random/LM) optimizer is a combination of the Random and Levenberg-Marquardt optimizers. It is the Random optimizer modified so that, at each of its error evaluations, it calls the Levenberg-Marquardt optimizer to evaluate the RMS error. The parameters specified in the Options Tables that effect the Levenberg-Marquardt and Random optimizers will have the same effects during Hybrid (Random/LM) optimization.