==============================
Dictionary Learning Algorithms
==============================


Algorithms
----------
.. automodule:: dictlearn._dictionary_learning
    :members:


Methods
-------

In order to compare all the available methods we propose the following example

.. code-block:: python

        import copy
        import numpy as np
        import matplotlib.pyplot as plt

        from dictlearn import DictionaryLearning
        from sklearn.datasets import make_sparse_coded_signal

        #############################################################################

        n_components = 50      # number of atoms
        n_features = 20        # signal dimension
        n_nonzero_coefs = 3    # sparsity
        n_samples = 100        # number of signals
        n_iterations = 20      # number of dictionary learning iterations

        #############################################################################

        max_iter = 10
        fit_algorithms = ["ksvd", "aksvd", "uaksvd", "sgk", "nsgk", "mod"]
        transform_algorithm = "omp"

        Y, D_origin, X_origin = make_sparse_coded_signal(
            n_samples=n_samples,
            n_components=n_components,
            n_features=n_features,
            n_nonzero_coefs=n_nonzero_coefs,
            random_state=0
        )

        D0 = np.random.randn(
            D_origin.shape[0],
            D_origin.shape[1]
        )

        for fit_algorithm in fit_algorithms:
            dl = DictionaryLearning(
                n_components=n_components,
                max_iter=max_iter,
                fit_algorithm=fit_algorithm,
                transform_algorithm=transform_algorithm,
                n_nonzero_coefs=n_nonzero_coefs,
                code_init=None,
                dict_init=copy.deepcopy(D0),
                verbose=False,
                random_state=None,
                kernel_function=None,
                params=None
            )
            
            dl.fit(Y)
            plt.plot(range(max_iter), dl.error_, label=fit_algorithm)

        plt.legend()
        plt.show()

The error evolution durring training stages is available as following

.. image:: std_dl.png
  :width: 800
  :alt: Standard Dictionary Learning


Standard Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Standard Dictionary Learning consists of optimizing

.. math::

    \begin{align}
    \min _{D, \mathbf{X}} & \|\mathbf{Y}-\mathbf{D} \mathbf{X}\|_{F}^{2} \\
    \text { s.t. } & \left\|\mathbf{x}_{\ell}\right\|_{0} \leq s, \ell=1: N \\
    & \left\|\mathbf{d}_{j}\right\|=1, j=1: n
    \label{eq-dl}
    \end{align}

where :math:`\mathbf{Y} \in \mathbb{R}^{m \times N}` is a given set of signals, represented by :math:`N` signals of size :math:`m`, :math:`\mathbf{D} \in \mathbb{R}^{m \times n}` is the trained dictionary, :math:`\mathbf{X} \in \mathbb{R}^{n \times N}` is the sparse representation and :math:`s` represents the sparsity constraint.

.. automodule:: dictlearn.methods._ksvd
    :members: ksvd
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: aksvd
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: uaksvd
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: sgk
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: nsgk
    :noindex:

.. automodule:: dictlearn.methods._mod
    :members: mod
    :noindex:


Regularized Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Least squares regularization consists of optimizing

.. math::

    \min _{D, \mathbf{X}} \|\mathbf{Y}-\mathbf{D} \mathbf{X}\|_{F}^{2}+\mu\|\mathbf{X}\|_{F}^{2}

where :math:`\mu > 0` is the regularization parameter.

.. automodule:: dictlearn.methods._ksvd
    :members: ksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: aksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: uaksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: sgk_reg
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: nsgk_reg
    :noindex:

.. automodule:: dictlearn.methods._mod
    :members: mod_reg
    :noindex:


Incoherent Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dictionary Learning algorithms that jointly optimize the representation error and the coherence of the dictionary. The objective function consists in

.. math::

    \min _{D, X}\|\mathbf{Y}-\mathbf{D} \mathbf{X}\|_{F}^{2}+\gamma\left\|\mathbf{D}^{T} \mathbf{D}-\mathbf{I}\right\|_{F}^{2}

where :math:`\gamma > 0` is a trade-off parameter.

.. automodule:: dictlearn.methods._ksvd
    :members: ksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: aksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: uaksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: sgk_coh
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: nsgk_coh
    :noindex:


Parallel Standard Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solves the Standard Dictionary Learning problem by using a parallel perspective on the optimization problem. Instead of updating the atoms one by one, the atoms will be changed in parallel.

.. automodule:: dictlearn.methods._ksvd
    :members: pksvd
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: paksvd
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: puaksvd
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: psgk
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: pnsgk
    :noindex:


Parallel Regularized Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solves the Regularized Dictionary Learning problem by using a parallel perspective on the optimization problem. Instead of updating the atoms one by one, the atoms will be changed in parallel.

.. automodule:: dictlearn.methods._ksvd
    :members: pksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: paksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: puaksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: psgk_reg
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: pnsgk_reg
    :noindex:


Parallel Incoherent Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solves the Incoherent Dictionary Learning problem by using a parallel perspective on the optimization problem. Instead of updating the atoms one by one, the atoms will be changed in parallel.

.. automodule:: dictlearn.methods._ksvd
    :members: pksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: paksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: puaksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: psgk_coh
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: pnsgk_coh
    :noindex:


Kernel Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to evade the linear character of the representation, a Kernel version of the Dictionary Learning problem is introduced. Through this method, the space of signals is extended to a nonlinear feature vector space. We associate with each signal :math:`\mathbf{y}` the feature vector :math:`\varphi(\mathbf{y})`, where :math:`\varphi(\mathbf{y})` is a nonlinear function. The dictionary :math:`\mathbf{D}` is also extended to a nonlinear space by :math:`\varphi(\mathbf{y})\mathbf{A}`, where :math:`\mathbf{A}` contains the coefficients of the dictionary. So, the DL problem is transformed into

.. math::

    \begin{align}
    \min _{D, \mathbf{X}} & \|\varphi(\mathbf{Y})-\varphi(\mathbf{Y})\mathbf{A} \mathbf{X}\|_{F}^{2} \\
    \text { s.t. } & \left\|\mathbf{x}_{\ell}\right\|_{0} \leq s, \ell=1: N \\
    & \left\|\mathbf{a}_{j}\right\|=1, j=1: n
    \end{align}


.. automodule:: dictlearn.methods._ksvd
    :members: ker_ksvd
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: ker_aksvd
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: ker_uaksvd
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: ker_sgk
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: ker_nsgk
    :noindex:

.. automodule:: dictlearn.methods._mod
    :members: ker_mod
    :noindex:


Regularized Kernel Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A regularization version for the Kernel Dictionary Learning. The optimization problem consists in

.. math::

    \min _{A, \mathbf{X}} \|\varphi(\mathbf{Y})-\varphi(\mathbf{Y}) \mathbf{A} \mathbf{X}\|_{F}^{2}+\mu\|\mathbf{X}\|_{F}^{2}

where :math:`\mu > 0` is the regularization parameter.

.. automodule:: dictlearn.methods._ksvd
    :members: ker_ksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: ker_aksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: ker_uaksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: ker_sgk_reg
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: ker_nsgk_reg
    :noindex:


Incoherent Kernel Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A incoherent version for the Kernel Dictionary Learning. The optimization problem consists

.. math::

    \min _{A, X}\|\varphi(\mathbf{Y})-\varphi(\mathbf{Y}) \mathbf{A} \mathbf{X}\|_{F}^{2}+\gamma\left\|\mathbf{A}^{T} \varphi(\mathbf{Y})^{T}  \varphi(\mathbf{Y}) \mathbf{A}-\mathbf{I}\right\|_{F}^{2}

where :math:`\gamma > 0` is a trade-off parameter.

.. automodule:: dictlearn.methods._ksvd
    :members: ker_ksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: ker_aksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: ker_uaksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: ker_sgk_coh
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: ker_nsgk_coh
    :noindex:


Parallel Kernel Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solves the Kernel Dictionary Learning problem by using a parallel perspective on the optimization problem. Instead of updating the atoms one by one, the atoms will be changed in parallel.

.. automodule:: dictlearn.methods._ksvd
    :members: ker_pksvd
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: ker_paksvd
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: ker_puaksvd
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: ker_psgk
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: ker_pnsgk
    :noindex:


Parallel Regularized Kernel Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solves the Regularized Kernel Dictionary Learning problem by using a parallel perspective on the optimization problem. Instead of updating the atoms one by one, the atoms will be changed in parallel.

.. automodule:: dictlearn.methods._ksvd
    :members: ker_pksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: ker_paksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: ker_puaksvd_reg
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: ker_psgk_reg
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: ker_pnsgk_reg
    :noindex:


Parallel Incoherent Kernel Dictionary Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solves the Incoherent Kernel Dictionary Learning problem by using a parallel perspective on the optimization problem. Instead of updating the atoms one by one, the atoms will be changed in parallel.

.. automodule:: dictlearn.methods._ksvd
    :members: ker_pksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._aksvd
    :members: ker_paksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._uaksvd
    :members: ker_puaksvd_coh
    :noindex:

.. automodule:: dictlearn.methods._sgk
    :members: ker_psgk_coh
    :noindex:

.. automodule:: dictlearn.methods._nsgk
    :members: ker_pnsgk_coh
    :noindex: