Implement the new interface for objective functions and derivatives

docs/source/how_to/how_to_algorithm_selection.ipynb

@@ -41,12 +41,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "@om.mark.least_squares\n",
     "def sphere(params):\n",
-    "    out = {\n",
-    "        \"value\": params @ params,\n",
-    "        \"root_contributions\": params,\n",
-    "    }\n",
-    "    return out\n",
+    "    return params\n",
     "\n",
     "\n",
     "def sphere_gradient(params):\n",

docs/source/how_to/how_to_criterion_function.ipynb

@@ -0,0 +1,226 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to write objective functions\n",
+    "\n",
+    "optimagic is very flexible when it comes to the objective function and its derivatives. \n",
+    "In this how-to guide we start with simple examples, that would also work with \n",
+    "scipy.optimize before we show advanced options and their advantages. \n",
+    "\n",
+    "## The simplest case\n",
+    "\n",
+    "In the simplest case, `fun` maps a numpy array into a scalar objective value. The name\n",
+    "of first argument of `fun` is arbitrary. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import optimagic as om\n",
+    "\n",
+    "\n",
+    "def sphere(x):\n",
+    "    return x @ x\n",
+    "\n",
+    "\n",
+    "res = om.minimize(\n",
+    "    fun=sphere,\n",
+    "    params=np.arange(3),\n",
+    "    algorithm=\"scipy_lbfgsb\",\n",
+    ")\n",
+    "res.params.round(6)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## More flexible `params`\n",
+    "\n",
+    "In all but the most simple problems, a flat numpy array is not ideal to keep track of \n",
+    "all the different parameters one wants to optimize over. Therefore, optimagic accepts \n",
+    "objective functions that work with other parameter formats. Below we show a simple \n",
+    "example. More examples can be found [here](how_to_start_parameters.md).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def dict_fun(x):\n",
+    "    return x[\"a\"] ** 2 + x[\"b\"] ** 4\n",
+    "\n",
+    "\n",
+    "res = om.minimize(\n",
+    "    fun=dict_fun,\n",
+    "    params={\"a\": 1, \"b\": 2},\n",
+    "    algorithm=\"scipy_lbfgsb\",\n",
+    ")\n",
+    "\n",
+    "res.params"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The important thing is that the `params` provided to `minimize` need to have the format \n",
+    "that is expected by the objective function.\n",
+    "\n",
+    "## Functions with additional arguments\n",
+    "\n",
+    "In many applications, the objective function takes more than `params` as argument. \n",
+    "This can be achieved via `fun_kwargs`. Take the following simplified example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def shifted_sphere(x, offset):\n",
+    "    return (x - offset) @ (x - offset)\n",
+    "\n",
+    "\n",
+    "res = om.minimize(\n",
+    "    fun=shifted_sphere,\n",
+    "    params=np.arange(3),\n",
+    "    algorithm=\"scipy_lbfgsb\",\n",
+    "    fun_kwargs={\"offset\": np.ones(3)},\n",
+    ")\n",
+    "res.params"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`fun_kwargs` is a dictionary with keyword arguments for `fun`. There is no constraint\n",
+    "on the number or names of those arguments."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Least-Squares problems\n",
+    "\n",
+    "Many estimation problems have a least-squares structure. If so, specialized optimizers that exploit this structure can be much faster than standard optimizers. The `sphere` function from above is the simplest possible least-squarse problem you could imagine: the least-squares residuals are just the params. \n",
+    "\n",
+    "To use least-squares optimizers in optimagic, you need to mark your function with \n",
+    "a decorator and return the least-squares residuals instead of the aggregated function value. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@om.mark.least_squares\n",
+    "def ls_sphere(params):\n",
+    "    return params\n",
+    "\n",
+    "\n",
+    "res = om.minimize(\n",
+    "    fun=ls_sphere,\n",
+    "    params=np.arange(3),\n",
+    "    algorithm=\"pounders\",\n",
+    ")\n",
+    "res.params.round(5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Any least-squares optimization problem is also a standard optimization problem. You \n",
+    "can therefore optimize least-squares functions with scalar optimizers as well:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "res = om.minimize(\n",
+    "    fun=ls_sphere,\n",
+    "    params=np.arange(3),\n",
+    "    algorithm=\"scipy_lbfgsb\",\n",
+    ")\n",
+    "res.params.round(5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Returning additional information\n",
+    "\n",
+    "You can return additional information such as intermediate results, debugging information, etc. in your objective function. This information will be stored in a database if you use [logging](how_to_logging.ipynb).\n",
+    "\n",
+    "To do so, you need to return a `FunctionValue` object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def sphere_with_info(x):\n",
+    "    return om.FunctionValue(value=x @ x, info={\"avg\": x.mean()})\n",
+    "\n",
+    "\n",
+    "res = om.minimize(\n",
+    "    fun=sphere_with_info,\n",
+    "    params=np.arange(3),\n",
+    "    algorithm=\"scipy_lbfgsb\",\n",
+    ")\n",
+    "\n",
+    "res.params.round(6)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The `info` can be an arbitrary dictionary. In the oversimplified example we returned the \n",
+    "mean of the parameters, which could have been recovered from the params history that \n",
+    "is collected anyways but in real applications this feature can be helpful. "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "optimagic",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}