Cross Valdiation

Methods to find the optimal number of neurons via cross validation

CausalELM.CrossValidationModule

Methods to perform cross validation and find the optimum number of neurons.

To reduce computation time, the number of neurons is optimized by using cross validation to estimate the validation error on a small subset of the range of possible numbers of neurons. Then, an Extreme Learning Machine is trained to predict validation loss from the given cross validation sets. Finally, the number of neurons is selected that has the smallest predicted loss or the highest classification metric.

source
CausalELM.CrossValidation.recodeFunction
recode(ŷ)

Round predicted values to their predicted class for classification tasks.

If the smallest predicted label is 0, all labels are shifted up 1; if the smallest label is -1, all labels are shifted up 2. Also labels cannot be smaller than -1.

Examples

julia> recode([-0.7, 0.2, 1.1])
3-element Vector{Float64}
1
2
3
julia> recode([0.1, 0.2, 0.3])
3-element Vector{Float64}
1
1
1
julia> recode([1.1, 1.51, 1.8])
3-element Vector{Float64}
1
2
2
source
CausalELM.CrossValidation.traintestFunction
traintest(X, Y, folds)

Create a train-test split.

If an iteration is specified, the train test split will be treated as time series/panel data.

Examples

julia> xtrain, ytrain, xtest, ytest = traintest(zeros(20, 2), zeros(20), 5)
source
traintest(X, Y, folds, iteration)

Create a rolling train-test split for time series/panel data.

An iteration should not be specified for non-time series/panel data.

Examples

julia> xtrain, ytrain, xtest, ytest = traintest(zeros(20, 2), zeros(20), 5, 1)
source
CausalELM.CrossValidation.validateFunction
validate(X, Y, nodes, metric, iteration...; activation, regularized, folds)

Calculate a validation metric for a single fold in k-fold cross validation.

Examples

julia> x = rand(100, 5); y = Float64.(rand(100) .> 0.5)
julia> validate(x, y, 5, accuracy, 3)
0.0
source
CausalELM.CrossValidation.crossvalidateFunction
crossvalidate(X, Y, neurons, metric, activation, regularized, folds)

Calculate a validation metric for k folds using a single set of hyperparameters.

Examples

julia> x = rand(100, 5); y = Float64.(rand(100) .> 0.5)
julia> crossvalidate(x, y, 5, accuracy)
0.0257841765251021
source
CausalELM.CrossValidation.bestsizeFunction
bestsize(X, Y, metric, task, activation, min_neurons, max_neurons, regularized, folds, temporal, 
    iterations, approximator_neurons)

Compute the best number of neurons for an Extreme Learning Machine.

The procedure tests networks with numbers of neurons in a sequence whose length is given by iterations on the interval [minneurons, maxneurons]. Then, it uses the networks sizes and validation errors from the sequence to predict the validation error or metric for every network size between minneurons and maxneurons using the function approximation ability of an Extreme Learning Machine. Finally, it returns the network size with the best predicted validation error or metric.

Examples

julia> bestsize(rand(100, 5), rand(100), mse, "regression")
11
source