Copyright for portions of project CombineML.jl are held by Samuel Jenkins, 2014 as part of project Orchestra.jl. All other copyright for project CombineML.jl are held by Paulito Palmes, 2016.
The CombineML.jl package is licensed under the MIT "Expat" License:
You may also be interested to TSML (Time Series Machine Learning) package.
Julia 1.0/Linux/OSX: 
CombineML is a heterogeneous ensemble learning package for the Julia programming language. It is driven by a uniform machine learner API designed for learner composition.
See the notebook for demo: MetaModeling.ipnyb
We will cover how to predict on a dataset using CombineML.
A tabular dataset will be used to obtain our features and labels.
This will be split it into a training and test set using holdout method.
import CombineML
using CombineML.Util
using CombineML.Transformers
try
import RDatasets
catch
using Pkg
Pkg.add("RDatasets")
import RDatasets
end
# use shorter module names
CU=CombineML.Util
CT=CombineML.Transformers
RD=RDatasets
# Obtain features and labels
dataset = RD.dataset("datasets", "iris")
features = convert(Matrix,dataset[:, 1:(end-1)])
labels = convert(Array,dataset[:, end])
# Split into training and test sets
(train_ind, test_ind) = CU.holdout(size(features, 1), 0.3)A transformer processes features in some form. Coincidentally, a learner is a subtype of transformer.
A transformer can be created by instantiating it, taking an options dictionary as an optional argument.
All transformers, including learners are called in the same way.
# Learner with default settings
learner = CT.PrunedTree()
# Learner with some of the default settings overriden
learner = CT.PrunedTree(Dict(
:impl_options => Dict(
:purity_threshold => 1.0
)
))
# All learners are called in the same way.
learner = CT.StackEnsemble(Dict(
:learners => [
CT.PrunedTree(),
CT.RandomForest(),
CT.DecisionStumpAdaboost()
],
:stacker => CT.RandomForest()
))Normally we may require the use of data pre-processing before the features are passed to the learner.
We shall use a pipeline transformer to chain many transformers in sequence.
In this case we shall one hot encode categorical features, impute NA values and numerically standardize before we call the learner.
# Create pipeline
pipeline = CT.Pipeline(Dict(
:transformers => [
CT.OneHotEncoder(), # Encodes nominal features into numeric
CT.Imputer(), # Imputes NA values
CT.StandardScaler(), # Standardizes features
learner # Predicts labels on features
]
))Training is done via the fit! function, predicton via transform!.
All transformers, provide these two functions. They are always called the same way.
# Train
CT.fit!(pipeline, features[train_ind, :], labels[train_ind])
# Predict
predictions = CT.transform!(pipeline, features[test_ind, :])Finally we assess how well our learner performed.
# Assess predictions
result = CU.score(:accuracy, labels[test_ind], predictions)Outlined are all the transformers currently available via CombineML.
Baseline learner that by default assigns the most frequent label.
try
import StatsBase
catch
using Pkg
Pkg.add("StatsBase")
import StatsBase
end
learner = CT.Baseline(Dict(
# Output to train against
# (:class).
:output => :class,
# Label assignment strategy.
# Function that takes a label vector and returns the required output.
:strategy => StatsBase.mode
))Identity transformer passes the features as is.
transformer = CT.Identity()Set of machine learners that majority vote to decide prediction.
learner = CT.VoteEnsemble(Dict(
# Output to train against
# (:class).
:output => :class,
# Learners in voting committee.
:learners => [CT.PrunedTree(), CT.DecisionStumpAdaboost(), CT.RandomForest()]
))Ensemble where a 'stack' learner learns on a set of learners' predictions.
learner = CT.StackEnsemble(Dict(
# Output to train against
# (:class).
:output => :class,
# Set of learners that produce feature space for stacker.
:learners => [CT.PrunedTree(), CT.DecisionStumpAdaboost(), CT.RandomForest()],
# Machine learner that trains on set of learners' outputs.
:stacker => CT.RandomForest(),
# Proportion of training set left to train stacker itself.
:stacker_training_proportion => 0.3,
# Provide original features on top of learner outputs to stacker.
:keep_original_features => false
))Selects best learner out of set. Will perform a grid search on learners if options grid is provided.
learner = CT.BestLearner(Dict(
# Output to train against
# (:class).
:output => :class,
# Function to return partitions of instance indices.
:partition_generator => (features, labels) -> kfold(size(features, 1), 5),
# Function that selects the best learner by index.
# Arg learner_partition_scores is a (learner, partition) score matrix.
:selection_function => (learner_partition_scores) -> findmax(mean(learner_partition_scores, 2))[2],
# Score type returned by score() using respective output.
:score_type => Real,
# Candidate learners.
:learners => [CT.PrunedTree(), CT.DecisionStumpAdaboost(), CT.RandomForest()],
# Options grid for learners, to search through by BestLearner.
# Format is [learner_1_options, learner_2_options, ...]
# where learner_options is same as a learner's options but
# with a list of values instead of scalar.
:learner_options_grid => nothing
))Transforms nominal features into one-hot form and coerces the instance matrix to be of element type Float64.
transformer = CT.OneHotEncoder(Dict(
# Nominal columns
:nominal_columns => nothing,
# Nominal column values map. Key is column index, value is list of
# possible values for that column.
:nominal_column_values_map => nothing
))Imputes NaN values from Float64 features.
transformer = CT.Imputer(Dict(
# Imputation strategy.
# Statistic that takes a vector such as mean or median.
:strategy => mean
))Chains multiple transformers in sequence.
transformer = CT.Pipeline(Dict(
# Transformers as list to chain in sequence.
:transformers => [CT.OneHotEncoder(), CT.Imputer()],
# Transformer options as list applied to same index transformer.
:transformer_options => nothing
))Wraps around an CombineML transformer.
transformer = Wrapper(Dict(
# Transformer to call.
:transformer => CT.OneHotEncoder(),
# Transformer options.
:transformer_options => nothing
))Pruned CART decision tree.
learner = CT.PrunedTree(Dict(
# Output to train against
# (:class).
:output => :class,
# Options specific to this implementation.
:impl_options => Dict(
# Merge leaves having >= purity_threshold combined purity.
:purity_threshold => 1.0,
# Maximum depth of the decision tree (default: no maximum).
:max_depth => -1,
# Minimum number of samples each leaf needs to have.
:min_samples_leaf => 1,
# Minimum number of samples in needed for a split.
:min_samples_split => 2,
# Minimum purity increase needed for a split.
:min_purity_increase => 0.0
)
))Random forest (CART).
learner = CT.RandomForest(Dict(
# Output to train against
# (:class).
:output => :class,
# Options specific to this implementation.
:impl_options => Dict(
# Number of features to train on with trees (default: 0, keep all).
# Good values are square root or log2 of total number of features, rounded.
# Number of trees in forest.
:num_trees => 10,
# Proportion of trainingset to be used for trees.
:partial_sampling => 0.7,
# Maximum depth of each decision tree (default: no maximum).
:max_depth => -1
)
))Adaboosted decision stumps.
learner = CT.DecisionStumpAdaboost(Dict(
# Output to train against
# (:class).
:output => :class,
# Options specific to this implementation.
:impl_options => Dict(
# Number of boosting iterations.
:num_iterations => 7
)
))Principal Component Analysis rotation on features. Features ordered by maximal variance descending.
Fails if zero-variance feature exists. Based on MultivariateStats PCA
transformer = CT.PCA(Dict(
:pratio => 1.0,
:maxoutdim => 5
))Standardizes each feature using (X - mean) / stddev. Will produce NaN if standard deviation is zero.
transformer = CT.StandardScaler(Dict(
# Center features
:center => true,
# Scale features
:scale => true
))See the scikit-learn API for what options are available per learner.
Wrapper for scikit-learn that provides access to most learners.
Options for the specific scikit-learn learner is to be passed
in options[:impl_options] dictionary.
Available Classifiers:
- AdaBoostClassifier, BaggingClassifier, ExtraTreesClassifier, GradientBoostingClassifier, RandomForestClassifier, LDA, LogisticRegression, PassiveAggressiveClassifier, RidgeClassifier, RidgeClassifierCV, SGDClassifier, KNeighborsClassifier, RadiusNeighborsClassifier, NearestCentroid, QDA, SVC, LinearSVC, NuSVC, DecisionTreeClassifier, GaussianNB, MultinomialNB, ComplementNB, BernoulliNB
Available Regressors:
- SVR, Ridge, RidgeCV, Lasso, ElasticNet, Lars, LassoLars, OrthogonalMatchingPursuit, BayesianRidge, ARDRegression, SGDRegressor, PassiveAggressiveRegressor, KernelRidge, KNeighborsRegressor, RadiusNeighborsRegressor, GaussianProcessRegressor, DecisionTreeRegressor, RandomForestRegressor, ExtraTreesRegressor, AdaBoostRegressor, GradientBoostingRegressor, IsotonicRegression, MLPRegressor
# classifier example
learner = CT.SKLLearner(Dict(
# Output to train against
# (classification).
:output => :class,
:learner => "LinearSVC",
# Options specific to this implementation.
:impl_options => Dict()
))
# regression example
learner = CT.SKLLearner(Dict(
# Output to train against
# (regression).
:output => :reg,
:learner => "GradientBoostingRegressor",
# Options specific to this implementation.
:impl_options => Dict()
))RCall is used to interface with caret learners.
R 'caret' library offers more than 100 learners. See here for more details.
CARET wrapper that provides access to all learners.
Options for the specific CARET learner is to be passed
in options[:impl_options] dictionary.
learner = CT.CRTLearner(Dict(
# Output to train against
# (:class).
:output => :class,
:learner => "svmLinear",
:impl_options => Dict()
))Learners have only been tested on numeric features.
Inconsistencies may result in using nominal features directly without a numeric transformation (i.e. OneHotEncoder).
The links provided below will only work if you are viewing this in the GitHub repository.
See CHANGELOG.yml.
MIT "Expat" License. See LICENSE.md.