-
Notifications
You must be signed in to change notification settings - Fork 18
Running Jobs on YARN Clusters
Josh Patterson edited this page May 6, 2014
·
4 revisions
- A CDH4.1 Hadoop cluster with YARN installed
- Iterative Reduce Framework
yarn jar iterativereduce-0.1-SNAPSHOT.jar [app.properties file]
* _Needs https://github.com/emsixteeen/IterativeReduce_
* git clone https://github.com/emsixteeen/IterativeReduce.git
* cd IterativeReduce
* mvn package -DskipTests
We want to send a single file to Knitting Boar (currently we don't have multiple file input working) and also set the "partition size" of the file. The partition size will be the size of each split a worker in Iterative Reduce works on.
hdfs dfs -Ddfs.block.size=bytes -put <src> <dst>
Example of using the 20newsgroups dataset which is around 16MB converted. We want to set the block size to 4MB (4194304 bytes). The command looks like:
hdfs dfs -Ddfs.block.size=4194304 -put /my/local/dir/20news.txt hdfs:///somewhere/in/hdfs/
On a YARN-client execute the following:
# yarn jar iterativereduce-0.1-SNAPSHOT.jar app.properties
app.properties is the local filesystem path that contains your application properties file. If not specified, IterativeReduce looks for a file ./app.properties
14/05/04 21:28:29 INFO client.Client: Using input path: /user/cloudera/metronome/dbn/mnist
14/05/04 21:28:29 INFO yarn.ResourceManagerHandler: Connecting to the resource manager (client) at /0.0.0.0:8032
14/05/04 21:28:29 INFO yarn.ResourceManagerHandler: Got a new application with id=application_1399163811192_0036
14/05/04 21:28:29 INFO client.Client: Got an application, id=application_1399163811192_0036, appName=IR_DBN_MNIST_TWO_WORKER
14/05/04 21:28:30 WARN client.Client: log4j.properties file not found
14/05/04 21:28:30 WARN iterativereduce.Utils: Unable to copy file /tmp/IR_DBN_MNIST_TWO_WORKER/application_1399163811192_0036/log4j.properties: File not found.
14/05/04 21:28:30 INFO yarn.ResourceManagerHandler: Submitting application to ASM
14/05/04 21:28:33 INFO client.Client: IterativeReduce report: appId=36, state: ACCEPTED, Running Time: 0:00:02.127
14/05/04 21:28:35 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:04.135
14/05/04 21:28:37 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:06.137
14/05/04 21:28:39 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:08.138
14/05/04 21:28:41 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:10.140
14/05/04 21:28:43 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:12.144
14/05/04 21:28:45 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:14.146
14/05/04 21:28:47 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:16.147
14/05/04 21:28:49 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:18.149
14/05/04 21:28:51 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:20.154
14/05/04 21:28:53 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:22.155
14/05/04 21:28:55 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:24.158
14/05/04 21:28:57 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:26.169
14/05/04 21:28:59 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:28.181
14/05/04 21:29:01 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:30.183
14/05/04 21:29:03 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:32.195
14/05/04 21:29:05 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:34.199
14/05/04 21:29:07 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:36.207
14/05/04 21:29:09 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:38.210
14/05/04 21:29:11 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:40.221
14/05/04 21:29:13 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:42.223
14/05/04 21:29:15 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:44.227
14/05/04 21:29:17 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:46.231
14/05/04 21:29:19 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:48.233
14/05/04 21:29:21 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:50.235
14/05/04 21:29:23 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:52.237
14/05/04 21:29:25 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:54.240
14/05/04 21:29:27 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:56.242
14/05/04 21:29:29 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:00:58.244
14/05/04 21:29:31 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:00.247
14/05/04 21:29:33 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:02.249
14/05/04 21:29:35 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:04.251
14/05/04 21:29:37 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:06.253
14/05/04 21:29:39 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:08.255
14/05/04 21:29:41 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:10.274
14/05/04 21:29:43 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:12.276
14/05/04 21:29:45 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:14.278
14/05/04 21:29:47 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:16.281
14/05/04 21:29:49 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:18.284
14/05/04 21:29:51 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:20.286
14/05/04 21:29:53 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:22.290
14/05/04 21:29:55 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:24.294
14/05/04 21:29:57 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:26.300
14/05/04 21:29:59 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:28.304
14/05/04 21:30:01 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:30.309
14/05/04 21:30:03 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:32.311
14/05/04 21:30:05 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:34.313
14/05/04 21:30:07 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:36.316
14/05/04 21:30:09 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:38.318
14/05/04 21:30:11 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:40.319
14/05/04 21:30:13 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:42.321
14/05/04 21:30:15 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:44.323
14/05/04 21:30:17 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:46.325
14/05/04 21:30:19 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:48.327
14/05/04 21:30:21 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:50.329
14/05/04 21:30:23 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:52.335
14/05/04 21:30:25 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:54.340
14/05/04 21:30:27 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:56.350
14/05/04 21:30:29 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:01:58.352
14/05/04 21:30:31 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:00.354
14/05/04 21:30:33 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:02.359
14/05/04 21:30:35 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:04.366
14/05/04 21:30:37 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:06.375
14/05/04 21:30:39 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:08.378
14/05/04 21:30:41 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:10.389
14/05/04 21:30:43 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:12.394
14/05/04 21:30:45 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:14.396
14/05/04 21:30:47 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:16.401
14/05/04 21:30:49 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:18.403
14/05/04 21:30:51 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:20.407
14/05/04 21:30:53 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:22.413
14/05/04 21:30:55 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:24.416
14/05/04 21:30:57 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:26.418
14/05/04 21:30:59 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:28.423
14/05/04 21:31:01 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:30.425
14/05/04 21:31:03 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:32.427
14/05/04 21:31:05 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:34.429
14/05/04 21:31:07 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:36.431
14/05/04 21:31:09 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:38.432
14/05/04 21:31:11 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:40.435
14/05/04 21:31:13 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:42.437
14/05/04 21:31:15 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:44.439
14/05/04 21:31:17 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:46.441
14/05/04 21:31:19 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:48.449
14/05/04 21:31:21 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:50.455
14/05/04 21:31:23 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:52.459
14/05/04 21:31:25 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:54.462
14/05/04 21:31:27 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:56.464
14/05/04 21:31:29 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:02:58.466
14/05/04 21:31:31 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:00.468
14/05/04 21:31:33 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:02.470
14/05/04 21:31:35 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:04.472
14/05/04 21:31:37 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:06.474
14/05/04 21:31:39 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:08.476
14/05/04 21:31:41 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:10.478
14/05/04 21:31:43 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:12.480
14/05/04 21:31:45 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:14.482
14/05/04 21:31:47 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:16.485
14/05/04 21:31:49 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:18.486
14/05/04 21:31:51 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:20.488
14/05/04 21:31:53 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:22.490
14/05/04 21:31:55 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:24.494
14/05/04 21:31:57 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:26.505
14/05/04 21:31:59 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:28.512
14/05/04 21:32:01 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:30.523
14/05/04 21:32:03 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:32.531
14/05/04 21:32:05 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:34.535
14/05/04 21:32:07 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:36.549
14/05/04 21:32:09 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:38.551
14/05/04 21:32:11 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:40.554
14/05/04 21:32:13 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:42.556
14/05/04 21:32:15 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:44.559
14/05/04 21:32:17 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:46.560
14/05/04 21:32:19 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:48.562
14/05/04 21:32:21 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:50.564
14/05/04 21:32:23 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:52.566
14/05/04 21:32:25 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:54.568
14/05/04 21:32:27 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:56.570
14/05/04 21:32:29 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:03:58.572
14/05/04 21:32:31 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:00.574
14/05/04 21:32:33 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:02.580
14/05/04 21:32:35 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:04.582
14/05/04 21:32:37 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:06.586
14/05/04 21:32:39 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:08.589
14/05/04 21:32:41 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:10.591
14/05/04 21:32:43 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:12.600
14/05/04 21:32:45 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:14.606
14/05/04 21:32:47 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:16.623
14/05/04 21:32:49 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:18.632
14/05/04 21:32:51 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:20.635
14/05/04 21:32:53 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:22.642
14/05/04 21:32:55 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:24.646
14/05/04 21:32:57 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:26.653
14/05/04 21:32:59 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:28.662
14/05/04 21:33:01 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:30.689
14/05/04 21:33:03 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:32.694
14/05/04 21:33:05 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:34.697
14/05/04 21:33:07 INFO client.Client: IterativeReduce report: appId=36, state: RUNNING, Running Time: 0:04:36.702
14/05/04 21:33:09 INFO client.Client: IterativeReduce report: appId=36, state: FINISHED, Running Time: 0:04:38.703
14/05/04 21:33:09 INFO client.Client: Application finished in 281801ms
14/05/04 21:33:09 INFO client.Client: Application completed succesfully.
# This is the path for the KnittingBoar JAR
iterativereduce.jar.path=iterativereduce-0.1-SNAPSHOT.jar
# Path to your application (which was compiled against KB!)
app.jar.path=Metronome-1.0-SNAPSHOT-jar-with-dependencies.jar
# Comma separated list of other JAR's required for depenedencies
app.lib.jar.path=avro-1.7.1.jar,avro-ipc-1.7.1.jar
# Input file(s) to process
#app.input.path=src/test/resources/data/iris/threeworkers/
app.input.path=src/test/resources/data/MNIST/twolabels/twoworkers/
# Output results to
#app.output.path=hdfs:///user/yarn/kboar/output
app.output.path=file:///tmp/MNIST/dbn.mnist.twoworkers.twolabels.dl_model
# Number of iterations
app.iteration.count=3
app.name=IR_DBN_MNIST_TWO_LABELS
# Requested memory for YARN clients
yarn.memory=512
# The main() class/entry for the AppMaster
yarn.master.main=tv.floe.metronome.deeplearning.dbn.iterativereduce.MasterNode
# Any extra command-line args
yarn.master.args=
# The main() class/entry for the AppWorker
yarn.worker.main=tv.floe.metronome.deeplearning.dbn.iterativereduce.WorkerNode
# Any extra command-line args
yarn.worker.args=
# Any other configuration params, will be pushed down to clients
#tv.floe.metronome.neuralnetwork.conf.RecordFactoryClassname=tv.floe.metronome.io.records.MetronomeRecordFactory
tv.floe.metronome.dbn.conf.LearningRate=0.01
tv.floe.metronome.dbn.conf.hiddenLayerSizes=500,250,100
tv.floe.metronome.dbn.conf.batchSize=20
tv.floe.metronome.evaluate.dataset.path=/tmp/mnist_filtered_conversion_test.metronome
- Go to the cluster web interface
- http://(cluster-address):8088/cluster
- find the job name, something like
- application_1352770589658_0020
- For the Application Master click on "logs"
- Then click on stdout
- The output should look similar to below
-----------------------------------------
# Master Conf #
Number Iterations: 3
-----------------------------------------
Master Compute: SuperStep - Worker Info -----
[Master] WorkerReport[0]: I: 0, IC: 1 Trained Recs: 2850 AvgLogLikelihood: -1.8742067 PercentCorrect: 70.4782
> worker 0 is done with current iteration
[Master] WorkerReport[1]: I: 0, IC: 1 Trained Recs: 2664 AvgLogLikelihood: -1.7493868 PercentCorrect: 71.17317
> worker 1 is done with current iteration
[Master] WorkerReport[2]: I: 0, IC: 1 Trained Recs: 2991 AvgLogLikelihood: -1.7783203 PercentCorrect: 70.35493
> worker 2 is done with current iteration
[Master] WorkerReport[3]: I: 0, IC: 1 Trained Recs: 2809 AvgLogLikelihood: -1.8381199 PercentCorrect: 73.0591
> worker 3 is done with current iteration
Master Compute: SuperStep - Worker Info -----
[Master] WorkerReport[0]: I: 1, IC: 1 Trained Recs: 5700 AvgLogLikelihood: -1.7117476 PercentCorrect: 78.23344
> worker 0 is done with current iteration
[Master] WorkerReport[1]: I: 1, IC: 1 Trained Recs: 5328 AvgLogLikelihood: -1.5498475 PercentCorrect: 81.4208
> worker 1 is done with current iteration
[Master] WorkerReport[2]: I: 1, IC: 1 Trained Recs: 5982 AvgLogLikelihood: -1.6347487 PercentCorrect: 80.863655
> worker 2 is done with current iteration
[Master] WorkerReport[3]: I: 1, IC: 1 Trained Recs: 5618 AvgLogLikelihood: -1.6497002 PercentCorrect: 78.26145
> worker 3 is done with current iteration
Master Compute: SuperStep - Worker Info -----
[Master] WorkerReport[0]: I: 2, IC: 1 Trained Recs: 8550 AvgLogLikelihood: -1.7519032 PercentCorrect: 77.998276
> worker 0 is done with current iteration
[Master] WorkerReport[1]: I: 2, IC: 1 Trained Recs: 7992 AvgLogLikelihood: -1.5914761 PercentCorrect: 80.05547
> worker 1 is done with current iteration
[Master] WorkerReport[2]: I: 2, IC: 1 Trained Recs: 8973 AvgLogLikelihood: -1.6793529 PercentCorrect: 80.11336
> worker 2 is done with current iteration
[Master] WorkerReport[3]: I: 2, IC: 1 Trained Recs: 8427 AvgLogLikelihood: -1.6903981 PercentCorrect: 77.78894
> worker 3 is done with current iteration