Levenberg-Marquardt Algorithm

Abstract

This research presents an efficient Levenberg-Marquardt implementation for neural network training in regression, classification, and transfer learning. While Levenberg-Marquardt offers fast convergence and precision in nonlinear least-squares problems, its high memory and computational demands limit its use in large models. This work optimizes Levenberg-Marquardt to improve its practicality across diverse architectures by addressing these constraints.

A key contribution is integrating Levenberg-Marquardt into PyTorch, a widely used deep learning framework. This enables easier adoption, leveraging PyTorch’s GPU acceleration and parallelization for improved efficiency. By minimizing redundant calculations in the Jacobian and Hessian approximations, this implementation significantly reduces memory usage and computational overhead. Instead of merely optimizing storage, it selectively applies Levenberg-Marquardt where needed, balancing second-order precision with resource constraints.

Experiments validate Levenberg-Marquardt’s efficiency on benchmark tasks, including MNIST classification and fine-tuning AlexNet. Comparisons with Adam and SGD show that Levenberg-Marquardt achieves competitive accuracy with fewer epochs, making it a viable alternative in high-precision scenarios. In transfer learning, limiting trainable parameters helps mitigate memory concerns.

This research demonstrates that Levenberg-Marquardt can be an efficient neural network optimizer when resource management is prioritized. By refining its implementation, Levenberg-Marquardt becomes more practical for deep learning, particularly in tasks requiring fast convergence and high accuracy. Future work will explore further memory optimizations and extensions for high-dimensional datasets, broadening Levenberg-Marquardt’s applicability in modern neural network training.