-
Notifications
You must be signed in to change notification settings - Fork 168
Quickstart
The BIDMach bundle includes some scripts for loading medium-sized datasets for experimentation. These are written in bash script (for Linux, Mac or Cygwin). Typing:
<BIDMach_dir>/scripts/getdata.sh
will start loading and converting these datasets into binary files in <BIDMach_dir>/data/. The data include the Reuters news dataset (RCV1), three text datasets from the UC Irvine Data repository (NIP, NYTIMES and PUBMED) and a digit recognition dataset from UCI.
Most text data are stored as sparse, single-precision matrices. RCV1 includes a category assignment matrix, stored as an FMat with 0,1 values for each combination of input instance/category indicating whether that instance belongs in the category. Category assignments in RCV1 are one-to-many, i.e. each document may be assigned to multiple categories, although single category assignments are common.
For each major class of model (e.g. Generalized Linear Models, Latent Dirichlet Allocation, etc.), there is a simple learner which use a matrix as input. To perform logistic regression on Reuters data, you can start by loading the matrices:
val a = loadSMat("<BIDMach_dir>/data/rcv1/docs.smat.lz4")
val c = loadFMat("<BIDMach_dir>/data/rcv1/cats.fmat.lz4")
which loads the documents in RCV1 into a as a sparse matrix, and the category assignments into c. Then doing:
val (mm, mopts) = GLM.learner(a, c, 1)
creates a learner which will build multiple binary predictors (an OAA or One-Against-All multiclass classifier) for the 110 categories in RCV1 using logistic regression. Logistic regression is one of several GLM (Generalized Linear Model) models. Linear regression is also a GLM model. Support vector machines (SVM) are not GLM models, but the code for the SVM loss function is similar enough that they are included in the GLM package.
The choice of GLM model is set by the third argument. Currently:
0 = linear regression 1 = logistic regression 2 = logistic regression predictor with likelihood (not log likelihood) loss 3 = Support Vector Machine
Two objects are returned by the call to GLM.learner. The first "mm" is the learner instance itself, which holds the model, datasource etc. An options object "mopts" is returned separately. The options object is a custom class which holds all of the options for this particular combination of model, mixins, datasource and updater. You can inspect and change any of the options in this instance. To see what they are, do:
> mopts.what Option Name Type Value ========== ==== ===== addConstFeat boolean false lrate FMat 1 batchSize int 10000 dim int 256 doubleScore boolean false epsilon float 1.0E-5 evalStep int 11 featType int 1 initsumsq float 1.0E-5 links IMat 1,1,1,1,1,1,1,1,1,1,... mask FMat null npasses int 1 nzPerColumn int 0 pstep float 0.01 putBack int -1 regweight FMat 1.0000e-07 resFile String null rmask FMat null sample float 1.0 sizeMargin float 3.0 startBlock int 8000 targets FMat null targmap FMat null texp FMat 0.50000 useGPU boolean true vexp FMat 0.50000 waitsteps int 2
most of these are for advanced use only, but here are some basic ones:
npasses: number of passes over the dataset batchSize: size (in instances) in each minibatch lrate: the Learning rate
Npasses is intuitive - it is the number of passes over the data. For classical batch algorithms, this was also the number of iterations. But batch algorithms are very slow on large datasets. Minibatch algorithms, which BIDMach favors, perform many model updates in one pass over the dataset. In fact they are often enough to reach an acceptable loss level after a single pass over the dataset. You should tune the other parameters with a goal of achieving a low level of less in as few passes as possible.
batchSize is a very important parameter. Pure stochastic gradient algorithms update the model every instance (i.e. with a miniBatch size of 1), but this is very expensive and lacks parallelism. But you can generally achieve a comparable and sometimes faster rate of convergence using minibatch sizes that are much larger. Minibatch sizes in the hundreds to thousands are common, and in fact RCV1 achieves almost optimal convergence with a miniBatch size of 10,000.
lrate is the basic scaling constant for the gradient. Small values weakend the gradient on updates which generally slow learning down. Higher values accelerate updates, but can also lead to overshooting.
BIDMach will typically run well with the default parameters, but as you use it and get a feel for their effects, you will probably get a lot of value from systematically tuning them.
You start a learning session by calling a learner's "train" method like this:
> mm.train corpus perplexity=5582.125391 pass= 0 2.00%, ll=-0.693, gf=3.745, secs=0.2, GB=0.02, MB/s=92.42, GPUmem=0.83 16.00%, ll=-0.134, gf=10.185, secs=0.9, GB=0.12, MB/s=131.09, GPUmem=0.82 30.00%, ll=-0.123, gf=11.024, secs=1.6, GB=0.22, MB/s=135.70, GPUmem=0.82 44.00%, ll=-0.102, gf=11.353, secs=2.4, GB=0.33, MB/s=138.03, GPUmem=0.82 58.00%, ll=-0.094, gf=11.555, secs=3.1, GB=0.43, MB/s=139.82, GPUmem=0.82 72.00%, ll=-0.074, gf=11.659, secs=3.8, GB=0.53, MB/s=140.06, GPUmem=0.82 87.00%, ll=-0.085, gf=11.733, secs=4.5, GB=0.63, MB/s=140.70, GPUmem=0.82 100.00%, ll=-0.069, gf=11.778, secs=5.2, GB=0.73, MB/s=139.77, GPUmem=0.82 Time=5.1970 secs, gflops=11.78
The learner reports on each line: the percentage of data processed, the log likelihood, net gigaflops so far, seconds so far, Gigabytes processed, Megabytes/second achieved, and (if a GPU is being used) GPU memory used so far.
To avoid overfitting models, BIDMach always holds out a fixed set of miniBatches on each pass to use for scoring. Those "test" minibatches are never used for model training. The log likelihoods it reports come from those test minibatches. This is controlled by the "evalStep" parameter, which defaults to 11. It causes every nth minibatch to be a test minibatch, so the test minibatches are 0, 11, 22, 33,... All other minibatches are used for training.