A Deep Learning Framework based on OpenCL, written by C++. Supports popular MLP, RNN(LSTM), CNN(ResNet) neural networks. 基于OpenCL的深度学习计算框架，C++开发，支持多层感知器，长短时记忆模型，卷积神经网络。
Progress: Currently clnet can successfully run fully connected neural networks (MLP), CharRNN (LSTM) which uses dynamic computing graph to deal with loops, CNN (LeNet5 on MNIST dataset), WRN (Wide Residual Networks, CIFAR).
Tested on Nvidia GTX1080, AMD R9 295X2, Intel HD Graphics 630 GPU.
Support multiple devices training.
已完成进度：
可成功运行MLP全连接多层感知器，CharRNN（LSTM，基于动态计算图的循环实现），CNN（LeNet5，MNIST），WRN（CIFAR）的训练及推断。
三种模型均在Nvidia GTX1080, AMD R9 295X2, Intel HD Graphics 630以及Intel CPU，AMD CPU/APU上测试通过。
测试通过的编译环境：
Windows 10，MSVS2015；
Linux CentOS 7，g++ 4.8.5，Makefile；
eclipse/CDT，MinGW64 6；
eclipse/CDT，CrossGCC: CodeBench Lite 2014/05（gcc 4.8.3）；
支持多显卡训练。
TODO list：Kernel性能尚待进一步优化，分布式有待开发。
演示例子运行命令行：
全连接MLP：
.\Release\OpenCLNet.exe MLP /ds
charRNN：
.\Release\OpenCLNet.exe charRNN /ds :corpus_file D:\DataSets\charRNN\obama.txt :index_file D:\DataSets\charRNN\obama.index
p
save D:\DataSets\charRNN\epoch520_91%.clnetparams
charRNN推断：
.\Release\OpenCLNet.exe charRNN /p :index_file D:\DataSets\charRNN\obama.index :params_file D:\DataSets\charRNN\epoch520_91%.clnetparams :sample "Now it's time"
obama.txt可从http://data.mxnet.io/mxnet/data/char_lstm.zip下载。
MNIST CNN（训练及预测命令行。预测图片需使用28*28大小的24位BMP格式，黑底白字）：
.\Release\OpenCLNet.exe MNIST_CNN /ds :mnist_folder D:\DataSets\MNIST\
.\Release\OpenCLNet.exe MNIST_CNN /p :params_file D:\DataSets\MNIST_CNN.clnetparams :file D:\9.bmp
D:/DataSets/下需包含MNIST数据集文件train-images.idx3-ubyte，train-labels.idx1-ubyte，t10k-images.idx3-ubyte，t10k-labels.idx1-ubyte。可从http://yann.lecun.com/exdb/mnist/下载。目录名末尾请加上路径分隔符。
如何调试：
“/ds”生成执行树，“/ss”执行到第一个Tensor，停留，等待交互命令：
.\Release\OpenCLNet.exe MLP /ss /ds /0
clnet::type::GeneralInitializer GeneralInitializer
- clnet::type::Weight l0_weight[2,4096]
clnet::type::Bias l0_bias[4096]
clnet::type::Weight l1_weight[4096,1]
clnet::type::Bias l1_bias[1]
clnet::type::IterativeOptimizer IterativeOptimizer[4]
clnet::InstantTensor data_generator
clnet::type::Data X[128,2]
clnet::type::Weight l0_weight[2,4096]
clnet::type::Bias l0_bias[4096]
clnet::type::FullyConnectedLayer FCLayer_0=sigmoid(l0_weight*X+l0_bias)
clnet::type::Output FCLayer_0[128,4096]
clnet::type::Weight l1_weight[4096,1]
clnet::type::Bias l1_bias[1]
clnet::type::FullyConnectedLayer FCLayer_1=softrelu(l1_weight*FCLayer_0+l1_bias)
clnet::type::Output FCLayer_1[128,1]
clnet::type::Data Y[128]
clnet::back::Loss linear_regression(FCLayer_1,Y)
clnet::back::Gradient gradient(FCLayer_1)[128,1]
clnet::back::FullyConnectedLayer back:FCLayer_1=softrelu(l1_weight*FCLayer_0+l1_bias)
clnet::back::Gradient gradient(FCLayer_0)[128,4096]
clnet::back::FullyConnectedLayer back:FCLayer_0=sigmoid(l0_weight*X+l0_bias)
clnet::back::Gradient gradient(l0_weight)[2,4096]
clnet::back::Gradient gradient(l0_bias)[4096]
clnet::back::Gradient gradient(l1_weight)[4096,1]
clnet::back::Gradient gradient(l1_bias)[1]
clnet::type::StochasticGradientDescentUpdater SGD
- clnet::type::Weight l0_weight[2,4096]
clnet::type::Bias l0_bias[4096]
clnet::type::Weight l1_weight[4096,1]
clnet::type::Bias l1_bias[1]
- clnet::InstantTensor MLPMonitor
[1,@2018-06-30 16:24:29] GeForce GTX 1050 Ti (kernels build: 119ms)
[debugger] interactive thread started on device 1.
[debugger] device 1 break on IterativeOptimizer: clnet::type::IterativeOptimizer
执行到SGD（别名为SGD的Tensor）：
g SGD
[debugger] device 1 continue to run. [debugger] device 1 break on SGD: clnet::type::StochasticGradientDescentUpdater
观察输入样本X（别名为X的Tensor）：
d X
this: 0xfa1e60
type: clnet::type::Data
alias: X
volume: 256
dimensions: [128,2]
size: 1024 bytes
pointer: 0xe7aac0
gradient: NULL
inputs:
data_generator[]: clnet::InstantTensor
peers:
FCLayer_0=sigmoid(l0_weight*X+l0_bias)[]: clnet::type::FullyConnectedLayer
X
X[128,2]: clnet::type::Data 0 0: 1.00375,2.69076 1: 1.57991,3.42622 2: 2.75503,2.43962 3: 2.05087,3.68789 4: 3.46852,3.23981 5: 1.52232,3.57683 6: 3.1315,2.5406 7: 1.91198,1.04495 8: 1.27421,2.09336 9: 1.44194,1.4977 ...
观察梯度值：
d l0_weight
this: 0x2b227b0
type: clnet::type::Weight
alias: l0_weight
volume: 8192
dimensions: [2,4096]
size: 32768 bytes
pointer: 0x2b2c900
gradient: gradient(l0_weight)[2,4096]: clnet::back::Gradient
inputs:
peers:
FCLayer_0=sigmoid(l0_weight*X+l0_bias)[]: clnet::type::FullyConnectedLayer
gradient(l0_weight)
gradient(l0_weight)[2,4096]: clnet::back::Gradient 0 0: 0.0932272,-0.00103467,0.616816,0.0487299,0.108153,0.453982,-0.168111,0.00612603,0.0466066,0.0776809,0.480914,0.00167271,-0.0579107,-0.171267,-0.00544866,0.0305377,0.396773,-0.0364095,-0.0105135,-0.244325,0.0070936,-0.0271294,0.0982886,0.000907668,0.0083473,0.000168261,0.038511,-0.00443278,-0.141771,-0.000452508,0.0574187,0.59741,-0.0461692,0.0273872,0.0211383,0.0937608,-0.0543251,-0.0177396,0.0404992,0.244961 ... 1: 0.043596,-0.105236,0.252182,0.0135588,0.0468406,0.208793,-0.0282288,0.0436221,0.0046685,0.0364535,0.231056,0.0131293,-0.0219158,-0.0984129,-0.000470661,0.010817,0.0848113,-0.00210151,-0.00500153,-0.113508,0.00290996,-0.00091675,-0.0437556,0.000426235,0.0348718,6.88916e-005,0.011789,-0.0166271,-0.046225,-0.000272511,0.0210079,0.22276,-0.0209225,0.0109369,0.00923857,0.0413359,0.0153701,0.0267138,0.0193877,0.177686 ...
只看 部分数据：
gradient(l0_weight)[:,0:8]
data[0:2/2,0:8/4096] for gradient(l0_weight)[2,4096]: clnet::back::Gradient 0: 0.0932272,-0.00103467,0.616816,0.0487299,0.108153,0.453982,-0.168111,0.00612603 1: 0.043596,-0.105236,0.252182,0.0135588,0.0468406,0.208793,-0.0282288,0.0436221
l0_weight[:,0:8]
data[0:2/2,0:8/4096] for l0_weight[2,4096]: clnet::type::Weight 0: -0.280252,-1.62137,0.129004,0.495599,0.42723,0.0478061,-0.688217,1.87265 1: 1.73239,0.18904,-0.326688,0.204418,-2.56337,-0.718758,-0.185233,-0.827314
单步模式，执行完SGD，观察参数的变化：
s
[debugger] step into mode activated.
c
[debugger] device 1 continue to run. [debugger] device 1 break on MLPMonitor: clnet::InstantTensor
l0_weight[:,0:8]
data[0:2/2,0:8/4096] for l0_weight[2,4096]: clnet::type::Weight 0: -0.280253,-1.62137,0.128998,0.495598,0.427229,0.0478016,-0.688215,1.87265 1: 1.73239,0.189041,-0.326691,0.204418,-2.56337,-0.71876,-0.185233,-0.827315
修改超参数：
SGD.learning_rate
[debugger] SGD.learning_rate = 1e-005
SGD.learning_rate *= 0.5
[debugger] SGD.learning_rate = 1e-005 [debugger] SGD.learning_rate *= 0.5 [debugger] SGD.learning_rate = 5e-006
执行profile，性能调优：
pf
[debugger] profile mode activated.
g
[debugger] breakpoint removed.
c
[debugger] device 1 continue to run. [1,0,4ms] error rate: 0.331467 [1,2000,39006/s] error rate: 0.00325364 [1,4000,39072/s] error rate: 0.00251041
p
[debugger] breakpoint added on SGD. [debugger] device 1 break on SGD: clnet::type::StochasticGradientDescentUpdater
pf list
back:FCLayer_1=softrelu(l1_weight*FCLayer_0+l1_bias): clnet::back::FullyConnectedLayer: 3s.271ms/20% FCLayer_1=softrelu(l1_weight*FCLayer_0+l1_bias): clnet::type::FullyConnectedLayer: 3s.87ms/19% back:FCLayer_0=sigmoid(l0_weight*X+l0_bias): clnet::back::FullyConnectedLayer: 923ms/5% SGD: clnet::type::StochasticGradientDescentUpdater: 872ms/5% FCLayer_0=sigmoid(l0_weight*X+l0_bias): clnet::type::FullyConnectedLayer: 854ms/5% X: clnet::type::Data: 805ms/4% linear_regression(FCLayer_1,Y): clnet::back::Loss: 641ms/3% Y: clnet::type::Data: 593ms/3% data_generator: clnet::InstantTensor: 507ms/3% MLPMonitor: clnet::InstantTensor: 455ms/2% gradient(FCLayer_0): clnet::back::Gradient: 440ms/2% l0_bias: clnet::type::Bias: 397ms/2% l0_weight: clnet::type::Weight: 395ms/2% gradient(FCLayer_1): clnet::back::Gradient: 363ms/2% FCLayer_1: clnet::type::Output: 355ms/2% l1_bias: clnet::type::Bias: 347ms/2% gradient(l0_weight): clnet::back::Gradient: 335ms/2% FCLayer_0: clnet::type::Output: 334ms/2% gradient(l0_bias): clnet::back::Gradient: 324ms/2% l1_weight: clnet::type::Weight: 289ms/1% gradient(l1_bias): clnet::back::Gradient: 287ms/1% gradient(l1_weight): clnet::back::Gradient: 278ms/1%
一旦找到瓶颈，可以通过修改内置的kernels.cl或者修改Tensor.generate_source_code()加载的其他来源的OpenCL源码，实时重载kernels，测试提升效果：
rk
[debugger] waiting ... [debugger] kernels reloaded.
使用动态执行图，在执行“不等长”的数据如RNN-LSTM上，有性能优势：
.\Release\OpenCLNet.exe charRNN /ds /0 :corpus_file D:\DataSets\charRNN\obama.txt :index_file D:\DataSets\charRNN\obama.index
clnet::type::GeneralInitializer GeneralInitializer
- clnet::type::Weight embedding_matrix[84,256]
clnet::type::Weight lstm_weight_h0[256,1024]
clnet::type::Weight lstm_weight_x0[256,1024]
clnet::type::Bias lstm_bias0[1024]
clnet::type::Weight lstm_weight_h1[256,1024]
clnet::type::Weight lstm_weight_x1[256,1024]
clnet::type::Bias lstm_bias1[1024]
clnet::type::Weight lstm_weight_h2[256,1024]
clnet::type::Weight lstm_weight_x2[256,1024]
clnet::type::Bias lstm_bias2[1024]
clnet::type::Weight class_weight[256,84]
clnet::type::Bias class_bias[84]
clnet::type::IterativeOptimizer IterativeOptimizer[4]
SentenceIterator [8289]
clnet::type::Data data[32,129]
clnet::type::Weight embedding_matrix[84,256]
clnet::type::Embedding Embedding(data)
clnet::type::Output embedding[32,129,256]
clnet::type::LSTMInitializer lstm_initializer
- clnet::type::Output lstm_cell_state0[32,256]
clnet::type::Output lstm_hidden0[32,256]
clnet::type::Output lstm_cell_state1[32,256]
clnet::type::Output lstm_hidden1[32,256]
clnet::type::Output lstm_cell_state2[32,256]
clnet::type::Output lstm_hidden2[32,256]
clnet::type::LSTM LSTM(embedding)
clnet::type::Weight lstm_weight_h2[256,1024]
clnet::type::FullyConnectedLayer lstm_cell2_FC_hidden=lstm_weight_h2*lstm_hidden2
clnet::type::Weight lstm_weight_h1[256,1024]
clnet::type::FullyConnectedLayer lstm_cell1_FC_hidden=lstm_weight_h1*lstm_hidden1
clnet::type::Weight lstm_weight_h0[256,1024]
clnet::type::FullyConnectedLayer lstm_cell0_FC_hidden=lstm_weight_h0*lstm_hidden0
clnet::type::Output lstm_input_timestep[32,256]
clnet::type::Weight lstm_weight_x0[256,1024]
clnet::type::Bias lstm_bias0[1024]
clnet::type::FullyConnectedLayer lstm_cell0_FC_input=lstm_weight_x0*lstm_input_timestep+lstm_bias0
clnet::type::Output lstm_cell0_FC_input[32,1024]
clnet::type::BinaryOperator lstm_cell0_FC_hidden+=lstm_cell0_FC_input
clnet::type::Output lstm_cell0_FC_hidden[32,1024]
clnet::type::LSTMCell lstm_cell0
clnet::Tensor lstm_dropout0_mask[32,256]
clnet::type::DropOut lstm_dropout0
clnet::type::Output lstm_hidden0[32,256]
clnet::type::Weight lstm_weight_x1[256,1024]
clnet::type::Bias lstm_bias1[1024]
clnet::type::FullyConnectedLayer lstm_cell1_FC_input=lstm_weight_x1*lstm_hidden0+lstm_bias1
clnet::type::Output lstm_cell1_FC_input[32,1024]
clnet::type::BinaryOperator lstm_cell1_FC_hidden+=lstm_cell1_FC_input
clnet::type::Output lstm_cell1_FC_hidden[32,1024]
clnet::type::LSTMCell lstm_cell1
clnet::Tensor lstm_dropout1_mask[32,256]
clnet::type::DropOut lstm_dropout1
clnet::type::Output lstm_hidden1[32,256]
clnet::type::Weight lstm_weight_x2[256,1024]
clnet::type::Bias lstm_bias2[1024]
clnet::type::FullyConnectedLayer lstm_cell2_FC_input=lstm_weight_x2*lstm_hidden1+lstm_bias2
clnet::type::Output lstm_cell2_FC_input[32,1024]
clnet::type::BinaryOperator lstm_cell2_FC_hidden+=lstm_cell2_FC_input
clnet::type::Output lstm_cell2_FC_hidden[32,1024]
clnet::type::LSTMCell lstm_cell2
clnet::Tensor lstm_dropout2_mask[32,256]
clnet::type::DropOut lstm_dropout2
clnet::type::Output lstm_hidden2[32,256]
- clnet::Tensor lstm_runtime_cell_no[3]
clnet::type::Output lstm[32,129,256]
clnet::type::Weight class_weight[256,84]
clnet::type::Bias class_bias[84]
clnet::type::FullyConnectedLayer FC=class_weight*lstm+class_bias
clnet::type::Output FC[4128,84]
clnet::type::Data label[32,129]
clnet::back::Loss negative_log_likelihood(softmax(FC),label)
clnet::back::Gradient gradient(FC)[4128,84]
clnet::back::FullyConnectedLayer back:FC=class_weight*lstm+class_bias
clnet::back::Gradient gradient(lstm)[32,129,256]
clnet::type::LSTMInitializer LSTM(embedding)_gradient_initializer
- clnet::back::Gradient gradient(lstm_cell_state0)[32,256]
clnet::back::Gradient gradient(lstm_hidden0)[32,256]
clnet::back::Gradient gradient(lstm_cell_state1)[32,256]
clnet::back::Gradient gradient(lstm_hidden1)[32,256]
clnet::back::Gradient gradient(lstm_cell_state2)[32,256]
clnet::back::Gradient gradient(lstm_hidden2)[32,256]
clnet::back::LSTM back:LSTM(embedding)
clnet::back::DropOut back:lstm_dropout2
clnet::back::Gradient gradient(lstm_hidden2)[32,256]
clnet::back::LSTMCell back:lstm_cell2
clnet::back::Gradient gradient(lstm_cell2_FC_hidden)[32,1024]
clnet::back::BinaryOperator back:lstm_cell2_FC_hidden+=lstm_cell2_FC_input
clnet::back::Gradient gradient(lstm_cell2_FC_input)[32,1024]
clnet::back::FullyConnectedLayer back:lstm_cell2_FC_input=lstm_weight_x2*lstm_hidden1+lstm_bias2
clnet::back::DropOut back:lstm_dropout1
clnet::back::Gradient gradient(lstm_hidden1)[32,256]
clnet::back::LSTMCell back:lstm_cell1
clnet::back::Gradient gradient(lstm_cell1_FC_hidden)[32,1024]
clnet::back::BinaryOperator back:lstm_cell1_FC_hidden+=lstm_cell1_FC_input
clnet::back::Gradient gradient(lstm_cell1_FC_input)[32,1024]
clnet::back::FullyConnectedLayer back:lstm_cell1_FC_input=lstm_weight_x1*lstm_hidden0+lstm_bias1
clnet::back::DropOut back:lstm_dropout0
clnet::back::Gradient gradient(lstm_hidden0)[32,256]
clnet::back::LSTMCell back:lstm_cell0
clnet::back::Gradient gradient(lstm_cell0_FC_hidden)[32,1024]
clnet::back::BinaryOperator back:lstm_cell0_FC_hidden+=lstm_cell0_FC_input
clnet::back::Gradient gradient(lstm_cell0_FC_input)[32,1024]
clnet::back::FullyConnectedLayer back:lstm_cell0_FC_input=lstm_weight_x0*lstm_input_timestep+lstm_bias0
clnet::back::FullyConnectedLayer back:lstm_cell2_FC_hidden=lstm_weight_h2*lstm_hidden2
clnet::back::Gradient gradient(lstm_weight_h2)[256,1024]
clnet::back::FullyConnectedLayer back:lstm_cell1_FC_hidden=lstm_weight_h1*lstm_hidden1
clnet::back::Gradient gradient(lstm_weight_h1)[256,1024]
clnet::back::FullyConnectedLayer back:lstm_cell0_FC_hidden=lstm_weight_h0*lstm_hidden0
clnet::back::Gradient gradient(lstm_weight_h0)[256,1024]
clnet::back::Gradient gradient(lstm_weight_x0)[256,1024]
clnet::back::Gradient gradient(lstm_bias0)[1024]
clnet::back::Gradient gradient(lstm_weight_x1)[256,1024]
clnet::back::Gradient gradient(lstm_bias1)[1024]
clnet::back::Gradient gradient(lstm_weight_x2)[256,1024]
clnet::back::Gradient gradient(lstm_bias2)[1024]
clnet::back::Gradient gradient(lstm_input_timestep)[32,256]
- clnet::back::Gradient gradient(embedding)[32,129,256]
clnet::Tensor lstm_runtime_cell_no[3]
clnet::back::Gradient gradient(embedding)[32,129,256]
clnet::back::Embedding back:Embedding(data)
clnet::back::Gradient gradient(embedding_matrix)[84,256]
clnet::back::Gradient gradient(class_weight)[256,84]
clnet::back::Gradient gradient(class_bias)[84]
clnet::type::StochasticGradientDescentUpdater SGD
- clnet::type::Weight embedding_matrix[84,256]
clnet::type::Weight lstm_weight_h0[256,1024]
clnet::type::Weight lstm_weight_x0[256,1024]
clnet::type::Bias lstm_bias0[1024]
clnet::type::Weight lstm_weight_h1[256,1024]
clnet::type::Weight lstm_weight_x1[256,1024]
clnet::type::Bias lstm_bias1[1024]
clnet::type::Weight lstm_weight_h2[256,1024]
clnet::type::Weight lstm_weight_x2[256,1024]
clnet::type::Bias lstm_bias2[1024]
clnet::type::Weight class_weight[256,84]
clnet::type::Bias class_bias[84]
- clnet::InstantTensor charRNN_monitor
[1,@2018-06-30 16:29:48] GeForce GTX 1050 Ti (kernels build: 297ms)
[debugger] interactive thread started on device 1.
[debugger] device 1 break on IterativeOptimizer: clnet::type::IterativeOptimizer
使用0.0002的学习率，标准的SGD更新（无weight decay，无冲量），运行Lenet-5，可以在第一个epoch达到97%的测试集准确率。测试集准确率最高99.19%。
clnet::type::GeneralInitializer GeneralInitializer
- clnet::type::Weight conv1_weight[20,5,5,1]
clnet::type::Bias conv1_bias[20]
clnet::type::Weight conv2_weight[50,5,5,20]
clnet::type::Bias conv2_bias[50]
clnet::type::Weight feature_weight[2450,480]
clnet::type::Bias feature_bias[2450]
clnet::type::Weight inference_weight[480,10]
clnet::type::Bias inference_bias[480]
clnet::type::IterativeOptimizer IterativeOptimizer[4]
MNISTImageIterator [60001]
- clnet::Tensor train_images[60000,28,28]
clnet::Tensor train_labels[60000]
clnet::Tensor test_images[10016,28,28]
clnet::Tensor test_labels[10016]
clnet::Tensor train_images_data[32,28,28,1]
clnet::type::Weight conv1_weight[20,5,5,1]
clnet::type::Bias conv1_bias[20]
clnet::type::ConvolutionLayer conv1=Convolution:5x5(train_images_data,tanh)
clnet::type::Output conv1[32,28,28,20]
clnet::type::Pooling pool1=Pooling(conv1,max)
clnet::type::Output pool1[32,14,14,20]
clnet::type::Weight conv2_weight[50,5,5,20]
clnet::type::Bias conv2_bias[50]
clnet::type::ConvolutionLayer conv2=Convolution:5x5(pool1,tanh)
clnet::type::Output conv2[32,14,14,50]
clnet::type::Pooling pool2=Pooling(conv2,max)
clnet::type::Output pool2[32,7,7,50]
clnet::type::Reshape reshape[32,2450]
clnet::type::Weight feature_weight[2450,480]
clnet::type::Bias feature_bias[2450]
clnet::type::FullyConnectedLayer feature=tanh(feature_weight*reshape+feature_bias)
clnet::type::Output feature[32,480]
clnet::type::Weight inference_weight[480,10]
clnet::type::Bias inference_bias[480]
clnet::type::FullyConnectedLayer inference=inference_weight*feature+inference_bias
clnet::type::Output inference[32,10]
clnet::Tensor train_images_label[32]
clnet::back::Loss negative_log_likelihood(softmax(inference),train_images_label)
clnet::back::Gradient gradient(inference)[32,10]
clnet::back::FullyConnectedLayer back:inference=inference_weight*feature+inference_bias
clnet::back::Gradient gradient(feature)[32,480]
clnet::back::FullyConnectedLayer back:feature=tanh(feature_weight*reshape+feature_bias)
clnet::back::Reshape gradient(reshape)[32,2450]
clnet::back::Gradient gradient(pool2)[32,7,7,50]
clnet::back::Pooling back:pool2=Pooling(conv2,max)
clnet::back::Gradient gradient(conv2)[32,14,14,50]
clnet::back::ConvolutionLayer back:conv2=Convolution:5x5(pool1,tanh)
clnet::back::Gradient gradient(pool1)[32,14,14,20]
clnet::back::Pooling back:pool1=Pooling(conv1,max)
clnet::back::Gradient gradient(conv1)[32,28,28,20]
clnet::back::ConvolutionLayer back:conv1=Convolution:5x5(train_images_data,tanh)
clnet::back::Gradient gradient(conv1_weight)[20,5,5,1]
clnet::back::Gradient gradient(conv1_bias)[20]
clnet::back::Gradient gradient(conv2_weight)[50,5,5,20]
clnet::back::Gradient gradient(conv2_bias)[50]
clnet::back::Gradient gradient(feature_weight)[2450,480]
clnet::back::Gradient gradient(feature_bias)[2450]
clnet::back::Gradient gradient(inference_weight)[480,10]
clnet::back::Gradient gradient(inference_bias)[480]
clnet::type::StochasticGradientDescentUpdater SGD
- clnet::type::Weight conv1_weight[20,5,5,1]
clnet::type::Bias conv1_bias[20]
clnet::type::Weight conv2_weight[50,5,5,20]
clnet::type::Bias conv2_bias[50]
clnet::type::Weight feature_weight[2450,480]
clnet::type::Bias feature_bias[2450]
clnet::type::Weight inference_weight[480,10]
clnet::type::Bias inference_bias[480]
- clnet::InstantTensor MNIST_CNN_monitor
clnet::InstantTensor MNIST_CNN_validator
MNISTImageIterator [60001]
- clnet::Tensor train_images[60000,28,28]
clnet::Tensor train_labels[60000]
clnet::Tensor test_images[10016,28,28]
clnet::Tensor test_labels[10016]
clnet::Tensor train_images_data[32,28,28,1]
clnet::type::Weight conv1_weight[20,5,5,1]
clnet::type::Bias conv1_bias[20]
clnet::type::ConvolutionLayer conv1=Convolution:5x5(train_images_data,tanh)
clnet::type::Output conv1[32,28,28,20]
clnet::type::Pooling pool1=Pooling(conv1,max)
clnet::type::Output pool1[32,14,14,20]
clnet::type::Weight conv2_weight[50,5,5,20]
clnet::type::Bias conv2_bias[50]
clnet::type::ConvolutionLayer conv2=Convolution:5x5(pool1,tanh)
clnet::type::Output conv2[32,14,14,50]
clnet::type::Pooling pool2=Pooling(conv2,max)
clnet::type::Output pool2[32,7,7,50]
clnet::type::Reshape reshape[32,2450]
clnet::type::Weight feature_weight[2450,480]
clnet::type::Bias feature_bias[2450]
clnet::type::FullyConnectedLayer feature=tanh(feature_weight*reshape+feature_bias)
clnet::type::Output feature[32,480]
clnet::type::Weight inference_weight[480,10]
clnet::type::Bias inference_bias[480]
clnet::type::FullyConnectedLayer inference=inference_weight*feature+inference_bias
clnet::type::Output inference[32,10]
[0,@2018-06-30 20:12:44] GeForce GTX 1080 Ti (kernels build: 190ms)
[debugger] interactive thread started on device 0.
[0,0,28153ms] train accuracy: 0.976492 test set accuracy: 97.08%
[0,1,1999.400146/s] train accuracy: 0.961088 test set accuracy: 97.79%
[0,2,1977.066040/s] train accuracy: 0.969694 test set accuracy: 98.26%
[0,3,1977.131226/s] train accuracy: 0.975035 test set accuracy: 98.5%
[0,4,1975.308594/s] train accuracy: 0.996367 test set accuracy: 98.62%
[0,5,1974.333618/s] train accuracy: 0.836167 test set accuracy: 98.43%
[0,6,1973.879028/s] train accuracy: 0.885966 test set accuracy: 98.78%
[0,7,1973.359619/s] train accuracy: 0.87724 test set accuracy: 98.66%
[0,8,1973.943970/s] train accuracy: 0.994898 test set accuracy: 98.71%
[0,9,1973.489502/s] train accuracy: 0.975384 test set accuracy: 98.73%
[0,10,1973.554321/s] train accuracy: 0.987039 test set accuracy: 98.87%
[0,11,1973.424561/s] train accuracy: 0.989917 test set accuracy: 98.8%
[0,12,1969.925781/s] train accuracy: 0.971296 test set accuracy: 98.82%
[0,13,1965.280029/s] train accuracy: 0.97529 test set accuracy: 98.98%
[0,14,1965.280029/s] train accuracy: 0.996434 test set accuracy: 99.01%
[0,15,1965.215698/s] train accuracy: 0.989955 test set accuracy: 98.93%
[0,16,1965.022583/s] train accuracy: 0.994556 test set accuracy: 99.05%
[0,17,1965.151367/s] train accuracy: 0.995275 test set accuracy: 99.03%
[0,18,1965.022583/s] train accuracy: 0.977169 test set accuracy: 98.96%
[0,19,1964.765259/s] train accuracy: 0.992683 test set accuracy: 99.05%
[0,20,1965.086914/s] train accuracy: 0.994362 test set accuracy: 98.93%
[0,21,1964.700928/s] train accuracy: 0.959518 test set accuracy: 98.96%
[0,22,1964.893921/s] train accuracy: 0.993861 test set accuracy: 98.92%
[0,23,1965.151367/s] train accuracy: 0.970596 test set accuracy: 99.06%
[0,24,1964.958252/s] train accuracy: 0.981751 test set accuracy: 98.95%
[0,25,1964.636597/s] train accuracy: 0.997321 test set accuracy: 99%
[0,26,1964.829590/s] train accuracy: 0.995127 test set accuracy: 99.11%
[0,27,1964.572266/s] train accuracy: 0.998763 test set accuracy: 99.15%
支持宽残差网络Wide residual networks (http://arxiv.org/abs/1605.07146)。命令行中“/dso”生成仅包含运算Tensor的执行树，“/pp”打印参数清单（按大小倒序）：
./build/OpenCLNet CIFAR_WRN :cifar_folder /cifar-10-batches-bin/ :width 1 :N 1 :batch_size 64 /dso /pp /1
clnet::type::GeneralInitializer GeneralInitializer
clnet::type::IterativeOptimizer IterativeOptimizer[4]
CIFARImageIterator [50049]
- clnet::Tensor train_images[50048,32,32,3]
clnet::Tensor train_labels[50048,1]
clnet::Tensor test_images[10048,32,32,3]
clnet::Tensor test_labels[10048,1]
clnet::Tensor train_images_data[64,32,32,3]
clnet::type::ConvolutionLayer conv0=Convolution:3x3(train_images_data)
clnet::type::BatchNormalizedLayer group0_block0_bn0=group0_block0_bn0_gamma*normalize(conv0)+group0_block0_bn0_beta
clnet::type::Activation ReLU(group0_block0_bn0)
clnet::type::ConvolutionLayer group0_block0_conv0=Convolution:3x3(relu(group0_block0_bn0))
clnet::type::BatchNormalizedLayer group0_block0_bn1=group0_block0_bn1_gamma*normalize(group0_block0_conv0)+group0_block0_bn1_beta
clnet::type::Activation ReLU(group0_block0_bn1)
clnet::type::ConvolutionLayer group0_block0_conv1=Convolution:3x3(relu(group0_block0_bn1))
clnet::type::BinaryOperator group0_block0_conv1+conv0
clnet::type::BatchNormalizedLayer group1_block0_bn0=group1_block0_bn0_gamma*normalize((group0_block0_conv1+conv0))+group1_block0_bn0_beta
clnet::type::Activation ReLU(group1_block0_bn0)
clnet::type::ConvolutionLayer group1_block0_conv0=Convolution:3x3(relu(group1_block0_bn0))
clnet::type::BatchNormalizedLayer group1_block0_bn1=group1_block0_bn1_gamma*normalize(group1_block0_conv0)+group1_block0_bn1_beta
clnet::type::Activation ReLU(group1_block0_bn1)
clnet::type::ConvolutionLayer group1_block0_conv1=Convolution:3x3(relu(group1_block0_bn1))
clnet::type::ConvolutionLayer group1_block0_convdim=Convolution:1x1(relu(group1_block0_bn0))
clnet::type::BinaryOperator group1_block0_conv1+group1_block0_convdim
clnet::type::BatchNormalizedLayer group2_block0_bn0=group2_block0_bn0_gamma*normalize((group1_block0_conv1+group1_block0_convdim))+group2_block0_bn0_beta
clnet::type::Activation ReLU(group2_block0_bn0)
clnet::type::ConvolutionLayer group2_block0_conv0=Convolution:3x3(relu(group2_block0_bn0))
clnet::type::BatchNormalizedLayer group2_block0_bn1=group2_block0_bn1_gamma*normalize(group2_block0_conv0)+group2_block0_bn1_beta
clnet::type::Activation ReLU(group2_block0_bn1)
clnet::type::ConvolutionLayer group2_block0_conv1=Convolution:3x3(relu(group2_block0_bn1))
clnet::type::ConvolutionLayer group2_block0_convdim=Convolution:1x1(relu(group2_block0_bn0))
clnet::type::BinaryOperator group2_block0_conv1+group2_block0_convdim
clnet::type::BatchNormalizedLayer bn=bn_gamma*normalize((group2_block0_conv1+group2_block0_convdim))+bn_beta
clnet::type::Activation ReLU(bn)
clnet::type::Pooling pool=Pooling(relu(bn),average)
clnet::type::Reshape reshape[64,64]
clnet::type::FullyConnectedLayer inference=inference_weight*reshape+inference_bias
clnet::Tensor train_images_label[64]
clnet::back::Loss negative_log_likelihood(softmax(inference),train_images_label)
clnet::back::FullyConnectedLayer back:inference=inference_weight*reshape+inference_bias
clnet::back::Reshape gradient(reshape)[64,64]
clnet::back::Pooling back:pool=Pooling(relu(bn),average)
clnet::back::Activation gradient(ReLU(bn))
clnet::back::BatchNormalizedLayer back:bn=bn_gamma*normalize((group2_block0_conv1+group2_block0_convdim))+bn_beta
clnet::back::BinaryOperator back:group2_block0_conv1+group2_block0_convdim
clnet::back::ConvolutionLayer back:group2_block0_convdim=Convolution:1x1(relu(group2_block0_bn0))
clnet::back::ConvolutionLayer back:group2_block0_conv1=Convolution:3x3(relu(group2_block0_bn1))
clnet::back::Activation gradient(ReLU(group2_block0_bn1))
clnet::back::BatchNormalizedLayer back:group2_block0_bn1=group2_block0_bn1_gamma*normalize(group2_block0_conv0)+group2_block0_bn1_beta
clnet::back::ConvolutionLayer back:group2_block0_conv0=Convolution:3x3(relu(group2_block0_bn0))
clnet::back::Activation gradient(ReLU(group2_block0_bn0))
clnet::back::BatchNormalizedLayer back:group2_block0_bn0=group2_block0_bn0_gamma*normalize((group1_block0_conv1+group1_block0_convdim))+group2_block0_bn0_beta
clnet::back::BinaryOperator back:group1_block0_conv1+group1_block0_convdim
clnet::back::ConvolutionLayer back:group1_block0_convdim=Convolution:1x1(relu(group1_block0_bn0))
clnet::back::ConvolutionLayer back:group1_block0_conv1=Convolution:3x3(relu(group1_block0_bn1))
clnet::back::Activation gradient(ReLU(group1_block0_bn1))
clnet::back::BatchNormalizedLayer back:group1_block0_bn1=group1_block0_bn1_gamma*normalize(group1_block0_conv0)+group1_block0_bn1_beta
clnet::back::ConvolutionLayer back:group1_block0_conv0=Convolution:3x3(relu(group1_block0_bn0))
clnet::back::Activation gradient(ReLU(group1_block0_bn0))
clnet::back::BatchNormalizedLayer back:group1_block0_bn0=group1_block0_bn0_gamma*normalize((group0_block0_conv1+conv0))+group1_block0_bn0_beta
clnet::back::BinaryOperator back:group0_block0_conv1+conv0
clnet::back::ConvolutionLayer back:group0_block0_conv1=Convolution:3x3(relu(group0_block0_bn1))
clnet::back::Activation gradient(ReLU(group0_block0_bn1))
clnet::back::BatchNormalizedLayer back:group0_block0_bn1=group0_block0_bn1_gamma*normalize(group0_block0_conv0)+group0_block0_bn1_beta
clnet::back::ConvolutionLayer back:group0_block0_conv0=Convolution:3x3(relu(group0_block0_bn0))
clnet::back::Activation gradient(ReLU(group0_block0_bn0))
clnet::back::BatchNormalizedLayer back:group0_block0_bn0=group0_block0_bn0_gamma*normalize(conv0)+group0_block0_bn0_beta
clnet::back::ConvolutionLayer back:conv0=Convolution:3x3(train_images_data)
clnet::type::StochasticGradientDescentUpdater SGD
- clnet::InstantTensor CIFAR_WRN_monitor
clnet::InstantTensor CIFAR_WRN_validator
CIFARImageIterator [50049]
- clnet::Tensor train_images[50048,32,32,3]
clnet::Tensor train_labels[50048,1]
clnet::Tensor test_images[10048,32,32,3]
clnet::Tensor test_labels[10048,1]
clnet::Tensor train_images_data[64,32,32,3]
clnet::type::ConvolutionLayer conv0=Convolution:3x3(train_images_data)
clnet::type::BatchNormalizedLayer group0_block0_bn0=group0_block0_bn0_gamma*normalize(conv0)+group0_block0_bn0_beta
clnet::type::Activation ReLU(group0_block0_bn0)
clnet::type::ConvolutionLayer group0_block0_conv0=Convolution:3x3(relu(group0_block0_bn0))
clnet::type::BatchNormalizedLayer group0_block0_bn1=group0_block0_bn1_gamma*normalize(group0_block0_conv0)+group0_block0_bn1_beta
clnet::type::Activation ReLU(group0_block0_bn1)
clnet::type::ConvolutionLayer group0_block0_conv1=Convolution:3x3(relu(group0_block0_bn1))
clnet::type::BinaryOperator group0_block0_conv1+conv0
clnet::type::BatchNormalizedLayer group1_block0_bn0=group1_block0_bn0_gamma*normalize((group0_block0_conv1+conv0))+group1_block0_bn0_beta
clnet::type::Activation ReLU(group1_block0_bn0)
clnet::type::ConvolutionLayer group1_block0_conv0=Convolution:3x3(relu(group1_block0_bn0))
clnet::type::BatchNormalizedLayer group1_block0_bn1=group1_block0_bn1_gamma*normalize(group1_block0_conv0)+group1_block0_bn1_beta
clnet::type::Activation ReLU(group1_block0_bn1)
clnet::type::ConvolutionLayer group1_block0_conv1=Convolution:3x3(relu(group1_block0_bn1))
clnet::type::ConvolutionLayer group1_block0_convdim=Convolution:1x1(relu(group1_block0_bn0))
clnet::type::BinaryOperator group1_block0_conv1+group1_block0_convdim
clnet::type::BatchNormalizedLayer group2_block0_bn0=group2_block0_bn0_gamma*normalize((group1_block0_conv1+group1_block0_convdim))+group2_block0_bn0_beta
clnet::type::Activation ReLU(group2_block0_bn0)
clnet::type::ConvolutionLayer group2_block0_conv0=Convolution:3x3(relu(group2_block0_bn0))
clnet::type::BatchNormalizedLayer group2_block0_bn1=group2_block0_bn1_gamma*normalize(group2_block0_conv0)+group2_block0_bn1_beta
clnet::type::Activation ReLU(group2_block0_bn1)
clnet::type::ConvolutionLayer group2_block0_conv1=Convolution:3x3(relu(group2_block0_bn1))
clnet::type::ConvolutionLayer group2_block0_convdim=Convolution:1x1(relu(group2_block0_bn0))
clnet::type::BinaryOperator group2_block0_conv1+group2_block0_convdim
clnet::type::BatchNormalizedLayer bn=bn_gamma*normalize((group2_block0_conv1+group2_block0_convdim))+bn_beta
clnet::type::Activation ReLU(bn)
clnet::type::Pooling pool=Pooling(relu(bn),average)
clnet::type::Reshape reshape[64,64]
clnet::type::FullyConnectedLayer inference=inference_weight*reshape+inference_bias
Total number of parameters: 78,330, trainable: 77,850
group2_block0_conv1_weight[64,3,3,64]: clnet::type::Weight 36,864
group2_block0_conv0_weight[64,3,3,32]: clnet::type::Weight 18,432
group1_block0_conv1_weight[32,3,3,32]: clnet::type::Weight 9,216
group1_block0_conv0_weight[32,3,3,16]: clnet::type::Weight 4,608
group0_block0_conv0_weight[16,3,3,16]: clnet::type::Weight 2,304
group0_block0_conv1_weight[16,3,3,16]: clnet::type::Weight 2,304
group2_block0_convdim_weight[64,1,1,32]: clnet::type::Weight 2,048
inference_weight[64,10]: clnet::type::Weight 640
group1_block0_convdim_weight[32,1,1,16]: clnet::type::Weight 512
conv0_weight[16,3,3,3]: clnet::type::Weight 432
bn_beta[64]: clnet::type::Bias 64
bn_gamma[64]: clnet::type::Weight 64
bn_moving_mean[64]: clnet::type::Parameter 64 -
bn_moving_variance[64]: clnet::type::Parameter 64 -
group2_block0_bn1_beta[64]: clnet::type::Bias 64
group2_block0_bn1_gamma[64]: clnet::type::Weight 64
group2_block0_bn1_moving_mean[64]: clnet::type::Parameter 64 -
group2_block0_bn1_moving_variance[64]: clnet::type::Parameter 64 -
group1_block0_bn1_beta[32]: clnet::type::Bias 32
group1_block0_bn1_gamma[32]: clnet::type::Weight 32
group1_block0_bn1_moving_mean[32]: clnet::type::Parameter 32 -
group1_block0_bn1_moving_variance[32]: clnet::type::Parameter 32 -
group2_block0_bn0_beta[32]: clnet::type::Bias 32
group2_block0_bn0_gamma[32]: clnet::type::Weight 32
group2_block0_bn0_moving_mean[32]: clnet::type::Parameter 32 -
group2_block0_bn0_moving_variance[32]: clnet::type::Parameter 32 -
group0_block0_bn0_beta[16]: clnet::type::Bias 16
group0_block0_bn0_gamma[16]: clnet::type::Weight 16
group0_block0_bn0_moving_mean[16]: clnet::type::Parameter 16 -
group0_block0_bn0_moving_variance[16]: clnet::type::Parameter 16 -
group0_block0_bn1_beta[16]: clnet::type::Bias 16
group0_block0_bn1_gamma[16]: clnet::type::Weight 16
group0_block0_bn1_moving_mean[16]: clnet::type::Parameter 16 -
group0_block0_bn1_moving_variance[16]: clnet::type::Parameter 16 -
group1_block0_bn0_beta[16]: clnet::type::Bias 16
group1_block0_bn0_gamma[16]: clnet::type::Weight 16
group1_block0_bn0_moving_mean[16]: clnet::type::Parameter 16 -
group1_block0_bn0_moving_variance[16]: clnet::type::Parameter 16 -
inference_bias[10]: clnet::type::Bias 10
[0,@2019-05-06 23:02:55] GeForce GTX 1080 Ti (kernels build: 1s.84ms)
[debugger] interactive thread started on device 0.
[0,0,58902ms] train loss: 1.16239 test set accuracy: 57.65%
[0,1,777.589600/s] train loss: 0.818452 test set accuracy: 65.6%
[0,2,772.429138/s] train loss: 0.941452 test set accuracy: 65.57%
[0,3,772.381409/s] train loss: 0.914646 test set accuracy: 70.4%
[0,4,772.333740/s] train loss: 0.890248 test set accuracy: 71.93%
[0,5,772.286072/s] train loss: 0.868742 test set accuracy: 72.64%
[0,6,772.119263/s] train loss: 0.784145 test set accuracy: 73.25%
[0,7,771.952576/s] train loss: 0.50787 test set accuracy: 74.12%
[0,8,771.964478/s] train loss: 0.679059 test set accuracy: 75%
[0,9,771.964478/s] train loss: 0.619829 test set accuracy: 74.54%
[0,10,772.000183/s] train loss: 0.596677 test set accuracy: 76.02%
[0,11,772.000183/s] train loss: 0.847572 test set accuracy: 75.81%
[0,12,771.904907/s] train loss: 0.802736 test set accuracy: 75.48%
[0,13,772.000183/s] train loss: 0.7105 test set accuracy: 77.84%
[0,14,771.857300/s] train loss: 0.682924 test set accuracy: 77%
[0,15,771.833496/s] train loss: 0.487696 test set accuracy: 76.49%
[0,16,771.893005/s] train loss: 0.54608 test set accuracy: 77.13%
[0,17,771.797791/s] train loss: 0.651461 test set accuracy: 76.69%
[0,18,771.773987/s] train loss: 0.709687 test set accuracy: 76.91%
[0,19,771.940674/s] train loss: 0.57213 test set accuracy: 77.01%
[0,20,731.054626/s] train loss: 0.383843 test set accuracy: 78.44%
[0,21,771.643127/s] train loss: 0.45581 test set accuracy: 78.72%
[0,22,771.583618/s] train loss: 0.498909 test set accuracy: 77.12%
[0,23,763.951660/s] train loss: 0.493421 test set accuracy: 77.9%
[0,24,763.765137/s] train loss: 0.554525 test set accuracy: 77.41%
[0,25,763.776733/s] train loss: 0.500107 test set accuracy: 77.98%
[0,26,759.822693/s] train loss: 0.635151 test set accuracy: 78.26%
[0,27,759.476746/s] train loss: 0.598002 test set accuracy: 78.51%
[0,28,759.407593/s] train loss: 0.559696 test set accuracy: 76.91%
[0,29,759.361511/s] train loss: 0.80741 test set accuracy: 78.15%
[0,30,758.970032/s] train loss: 0.607492 test set accuracy: 78.55%
SGD.learning_rate = 0.005
[debugger] SGD.learning_rate = 0.064
[debugger] SGD.learning_rate = 0.005
[0,31,758.567383/s] train loss: 0.320696 test set accuracy: 82.93%
[0,32,758.555908/s] train loss: 0.218599 test set accuracy: 82.99%