Classifiers

Discriminative Dictionary Learning

class dictlearn.models.DDL(fit_algorithm, transform_algorithm, n_components, n_nonzero_coefs, params, alpha, init_method=0, init_iter=20, gamma=0.001)

Solves the following classification problem

\[\min _{D, W, \boldsymbol{X}}\|\boldsymbol{Y}-\boldsymbol{D} \boldsymbol{X}\|_{F}^{2}+\alpha\|\boldsymbol{H}-\boldsymbol{W} \boldsymbol{X}\|_{F}^{2}\]
Parameters:

  • fit_algorithm ({'ksvd', 'aksvd', 'uaksvd', 'sgk', 'nsgk', 'mod'}, default='aksvd') –

    Algorithm used to update the dictionary:

    • ’ksvd’: uses the ksvd algorithm

    • ’aksvd’: uses the aksvd algorithm

    • ’uaksvd’: uses the uaksvd algorithm

    • ’sgk’: uses the sgk algorithm

    • ’nsgk’: uses the nsgk algorithm

    • ’mod’: uses the mod algorithm

  • transform_algorithm ({'omp'}, default='omp') –

    Algorithm for computing the sparse representation:

    • ’omp’: uses orthogonal matching pursuit to estimate the sparse solution.

  • n_components (int, default=n_features) – Number of dictionary elements to extract.

  • n_nonzero_coefs (int, default=None) – Number of nonzero coefficients to target in each column of the solution. If None, then n_nonzero_coefs=int(n_features / 10).

  • params (None, default=None) – Dictionary with provided parameters for the DL problem. If the params are not given then they will be automatically initialized.

  • alpha (scalar constant) –

  • init_method (scalar constant) –

  • init_iter (int, default=100) – Maximum number of iterations to perform.

  • gamma (scalar constant) –

Example:

import numpy as np

from dictlearn.models import DDL
from dictlearn.datasets import yaleb


# load data
(X_train, y_train), (X_test, y_test) = yaleb.load_data()

alpha = 2               # classification penalty
n_components = 200      # number of atoms
n_nonzero_coefs = 4     # sparsity constraint

# define dl parameters
params = {}

# train model
ddl = DDL('ksvd', 'omp', n_components, n_nonzero_coefs, params, alpha)
ddl.fit(X_train, y_train, max_iter=10, batches=4)

# test model
y_pred = ddl.predict(X_test)
accuracy = np.sum(y_test == y_pred) / len(y_test)
y_proba = ddl.predict_proba(X_test)

References: Q. Zhang, B. Li, Discriminative K-SVD for dictionary learning in face recognition, in Proceedings of IEEE Conf. Computer Vision and Pattern Recognition (2010), pp. 2691–2698

Kernel Discriminative Dictionary Learning

class dictlearn.models.KDDL(transform_algorithm, n_components, n_nonzero_coefs, kernel_function, params, alpha)

Solves the following classification problem

\[\min _{\boldsymbol{A}, \boldsymbol{W}, \boldsymbol{X}}\| \boldsymbol{\varphi}(\boldsymbol{Y})-\boldsymbol{\varphi} (\boldsymbol{Y}) \boldsymbol{A} \boldsymbol{X}\|_{F}^{2}+\alpha \|\boldsymbol{H}-\boldsymbol{W} \boldsymbol{X}\|_{F}^{2}\]
Parameters:

  • transform_algorithm ({'omp'}, default='omp') –

    Algorithm for computing the sparse representation:

    • ’omp’: uses orthogonal matching pursuit to estimate the sparse solution.

  • n_components (int, default=n_features) – Number of dictionary elements to extract.

  • n_nonzero_coefs (int, default=None) – Number of nonzero coefficients to target in each column of the solution. If None, then n_nonzero_coefs=int(n_features / 10).

  • kernel_function (handle function) – Computes the kernel matrix.

  • params (None, default=None) – Dictionary with provided parameters for the DL problem. If the params are not given then they will be automatically initialized.

  • alpha (scalar constant) –

  • init_method (scalar constant) –

  • init_iter (int, default=100) – Maximum number of iterations to perform.

  • gamma (scalar constant) –

Example:

import numpy as np

from dictlearn.models import KDDL
from dictlearn.datasets import yaleb
from dictlearn.kernels import dl_rbf_kernel


# load data
(X_train, y_train), (X_test, y_test) = yaleb.load_data()

alpha = 2               # classification penalty
n_components = 200      # number of atoms
n_nonzero_coefs = 4     # sparsity constraint

# define general parameters
params = {}
params['gamma'] = 1e-5

# train model
kddl = KDDL('omp', n_components, n_nonzero_coefs, dl_rbf_kernel, params, alpha)
kddl.fit(X_train, y_train, max_iter=10, batches=2)

# test model
y_pred = kddl.predict(X_test)
accuracy = np.sum(y_test == y_pred) / len(y_test)
y_proba = kddl.predict_proba(X_test)

Label Consistent Dictionary Learning

class dictlearn.models.LCDL(fit_algorithm, transform_algorithm, n_components, n_nonzero_coefs, params, alpha, beta, init_method=0, init_iter=20, gamma=0.001)

Solves the following classification problem

\[\min _{D, W, \boldsymbol{A}, X}\|\boldsymbol{Y}-\boldsymbol{D} \boldsymbol{X}\|_{F}^{2}+\alpha\|\boldsymbol{H}-\boldsymbol{W} \boldsymbol{X}\|_{F}^{2}+\beta\|\boldsymbol{Q}-\boldsymbol{A} \boldsymbol{X}\|_{F}^{2}\]
Parameters:

  • fit_algorithm ({'ksvd', 'aksvd', 'uaksvd', 'sgk', 'nsgk', 'mod'}, default='aksvd') –

    Algorithm used to update the dictionary:

    • ’ksvd’: uses the ksvd algorithm problem

    • ’aksvd’: uses the aksvd algorithm problem

    • ’uaksvd’: uses the uaksvd algorithm problem

    • ’sgk’: uses the sgk algorithm problem

    • ’nsgk’: uses the nsgk algorithm problem

    • ’mod’: uses the mod algorithm problem

  • transform_algorithm ({'omp'}, default='omp') –

    Algorithm for computing the sparse representation:

    • ’omp’: uses orthogonal matching pursuit to estimate the sparse solution.

  • n_components (int, default=n_features) – Number of dictionary elements to extract.

  • n_nonzero_coefs (int, default=None) – Number of nonzero coefficients to target in each column of the solution. If None, then n_nonzero_coefs=int(n_features / 10).

  • params (None, default=None) – Dictionary with provided parameters for the DL problem. If the params are not given then they will be automatically initialized.

  • alpha (scalar constant) –

  • beta (scalar constant) –

  • init_method (scalar constant) –

  • init_iter (int, default=100) – Maximum number of iterations to perform.

  • gamma (scalar constant) –

Example:

import numpy as np

from dictlearn.models import LCDL
from dictlearn.datasets import yaleb


# load data
(X_train, y_train), (X_test, y_test) = yaleb.load_data()

alpha = 2               # classification penalty
beta = 4                # label consistent penalty
n_components = 5        # number of atoms
n_nonzero_coefs = 4     # sparsity constraint

# define general parameters
params = {}

# train model
lcdl = LCDL('aksvd', 'omp', n_components, n_nonzero_coefs, params, alpha, beta,
            init_method=0)
lcdl.fit(X_train, y_train, max_iter=10)

# test model
y_pred = lcdl.predict(X_test)
accuracy = np.sum(y_test == y_pred) / len(y_test)
y_proba = lcdl.predict_proba(X_test)

References: Z. Jiang, Z. Lin, L.S. Davis, Label consistent K-SVD: learning a discriminative dictionary for recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2651–2664 (2013)

Kernel Label Consistent Dictionary Learning

class dictlearn.models.KLCDL(transform_algorithm, n_components, n_nonzero_coefs, kernel_function, params, alpha, beta)

Solves the following classification problem

\[\min _{D, W, \boldsymbol{A}, X} \|\boldsymbol{\varphi} (\boldsymbol{Y})-\boldsymbol{\varphi}(\boldsymbol{Y}) \boldsymbol{A} \boldsymbol{X}\|_{F}^{2} +\alpha\|\boldsymbol{H}- \boldsymbol{W} \boldsymbol{X}\|_{F}^{2}+\beta\|\boldsymbol{Q}- \boldsymbol{A} \boldsymbol{X}\|_{F}^{2}\]
Parameters:

  • transform_algorithm ({'omp'}, default='omp') –

    AAlgorithm for computing the sparse representation:

    • ’omp’: uses orthogonal matching pursuit to estimate the sparse solution.

  • n_components (int, default=n_features) – Number of dictionary elements to extract.

  • n_nonzero_coefs (int, default=None) – Number of nonzero coefficients to target in each column of the solution. If None, then n_nonzero_coefs=int(n_features / 10).

  • kernel_function (handle function) – Computes the kernel matrix.

  • params (None, default=None) – Dictionary with provided parameters for the DL problem. If the params are not given then they will be automatically initialized.

  • alpha (scalar constant) –

  • beta (scalar constant) –

Example:

import numpy as np

from dictlearn.models import KLCDL
from dictlearn.datasets import yaleb
from dictlearn.kernels import dl_rbf_kernel


# load data
(X_train, y_train), (X_test, y_test) = yaleb.load_data()

alpha = 0.01             # classification penalty
beta = 0.01              # label consistent penalty
n_components = 200       # number of atoms
n_nonzero_coefs = 4      # sparsity constraint

# define general parameters
params = {}
params['sigma'] = 100

# train model
klcdl = KLCDL('omp', n_components, n_nonzero_coefs, dl_rbf_kernel, params,
              alpha, beta)
klcdl.fit(X_train, y_train, max_iter=10, batches=1)

# test model
y_pred = klcdl.predict(X_test)
accuracy = np.sum(y_test == y_pred) / len(y_test)
y_proba = klcdl.predict_proba(X_test)

Dictionary Pair Learning

class dictlearn.models.DPL(n_components, tau, theta, gamma)

Solves the following classification problem

\[\min _{\boldsymbol{P}, \boldsymbol{A}, \boldsymbol{D}} \sum_{k=1}^{K}\left\|\boldsymbol{X}_{i}-\boldsymbol{D}_{k} \boldsymbol{A}_{k}\right\|_{F}^{2}+\tau\left\|\boldsymbol{P}_{k} \boldsymbol{X}_{k}-\boldsymbol{A}_{k}\right\|_{F}^{2}+ \theta\left\|\boldsymbol{P}_{k} \overline{\boldsymbol{X}}_{k} \right\|_{F}^{2}, \quad \text { s.t. }\left\|\boldsymbol{d}_{i} \right\|_{2}^{2} \leq 1\]
Parameters:

  • n_components – dictionary size (number of atoms)

  • tau – scalar constant

  • theta – scalar constant

  • gamma – scalar constant

Example:

import numpy as np

from sklearn import preprocessing
from dictlearn.models import DPL
from dictlearn.datasets import yaleb


# load data
(X_train, y_train), (X_test, y_test) = yaleb.load_data()

# data normalization
X_train = preprocessing.normalize(X_train, norm='l2', axis=0)
X_test = preprocessing.normalize(X_test, norm='l2', axis=0)

# train model
dpl = DPL(n_components=30, tau=0.05, theta=0.003, gamma=0.0001)
dpl.fit(X_train, y_train, max_iter=10)

# test model
y_pred = dpl.predict(X_test)
accuracy = np.sum(y_test == y_pred) / len(y_test)
y_proba = dpl.predict_proba(X_test)

References: Gu, Shuhang, et al. “Projective dictionary pair learning for pattern classification.” Advances in neural information processing systems 27 (2014): 793-801

Kernel Dictionary Pair Learning

class dictlearn.models.KDPL(n_components, tau, theta, gamma, alpha, kernel_function, params)

Solves the following classification problem

\[\begin{split}\begin{array}{r} \displaystyle\min_{\boldsymbol{P}, \boldsymbol{A}, \boldsymbol{D}} \sum_{k=1}^{\mathrm{N}}\left\| \boldsymbol{\Phi}\left(\boldsymbol{X}_{\boldsymbol{k}}\right) -\boldsymbol{\Phi}(\boldsymbol{X}) \boldsymbol{D}_{k} \boldsymbol{A}_{k}\right\|_{F}^{2}+\tau\left\|\boldsymbol{P}_{k} \boldsymbol{\Phi}^{T}(\boldsymbol{X}) \boldsymbol{\Phi} \left(\boldsymbol{X}_{k}\right)-\boldsymbol{A}_{k}\right\|_{F}^{2} \\ +\theta\left\|\boldsymbol{P}_{k} \boldsymbol{\Phi}^{T}(\boldsymbol{X}) \Phi\left(\overline{\boldsymbol{X}}_{k}\right)\right\|_{F}^{2}, \quad \text { s.t. }\left\|\boldsymbol{d}_{i}\right\|_{2}^{2} \leq 1 \end{array}\end{split}\]
Parameters:

  • n_components – dictionary size (number of atoms)

  • tau – scalar constant

  • theta – scalar constant

  • gamma – scalar constant

  • alpha – scalar constant

  • kernel_function – kernel method

  • params – dictionary learning parameters

Example:

import numpy as np

from sklearn import preprocessing
from dictlearn.models import KDPL
from dictlearn.datasets import yaleb
from dictlearn.kernels import dl_rbf_kernel


# load data
(X_train, y_train), (X_test, y_test) = yaleb.load_data()

# data normalization
X_train = preprocessing.normalize(X_train, norm='l2', axis=0)
X_test = preprocessing.normalize(X_test, norm='l2', axis=0)

# define dl parameters
params = {}
params['gamma'] = 0.5

# train model
kdpl = KDPL(n_components=30, tau=0.05, theta=0.003, gamma=1e-4,
            alpha=1e-4, kernel_function=dl_rbf_kernel, params=params)
kdpl.fit(X_train, y_train, max_iter=10)

# test model
y_pred = kdpl.predict(X_test)
accuracy = np.sum(y_test == y_pred) / len(y_test)
y_proba = kdpl.predict_proba(X_test)