Sponsored Link

Monthly Archives: 9月 2014

(27) cuda-convnet2で二値分類器の結果を見る

(26) cuda-convnet2で二値分類器を作ってみる の続き…

(10) cuda-convnetのレポート表示 と同じ手順でレポート表示できる。


(1) loss curveを表示

SAVEDATA=ConvNet__2014-09-15_10.58.02
python shownet.py --load-file $SAVEDATA --show-cost=dce

20140915_losscurve


(2) error curveを表示

SAVEDATA=ConvNet__2014-09-15_10.58.02
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=1

20140915_errorrate


(3) accuracy curveを表示

SAVEDATA=ConvNet__2014-09-15_10.58.02
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=2    # unit#1 precision
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=3    # unit#1 recall
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=4    # unit#2 precision
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=5    # unit#2 recall
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=6    # unit#3 precision
python shownet.py --load-file $SAVEDATA --show-cost=dce --cost-idx=7    # unit#3 recall

20140915_acc2


(4) 全テストデータの出力層出力値を取得

手順1: 学習を実行する。
(26) cuda-convnet2で二値分類器を作ってみる を参照のこと。

手順2: 学習済みデータを指定して test-only で cuda-convnet2を実行する。

BASEPATH=/home/user/cuda/cuda-convnet2
SAVEDATA=$BASEPATH/save/ConvNet__2014-09-15_10.58.02
FEATPATH=$BASEPATH/tmp
TRGLAYER=fcOut
TEST_RANGE=7
python convnet.py --load-file ${SAVEDATA} \
                  --test-only 1 \
                  --test-range $TEST_RANGE \
                  --write-features $TRGLAYER \
                  --feature-path $FEATPATH

手順3: 出力されたデータファイルの中身を見る。

$ls
batches.meta  data_batch_7

pythonでシリアライズしたデータなので、pythonでデシリアイズして中身を見る。

$ python

[‘labels’] には正解値が、[‘data’] には出力値が格納されている。
どちらも 10,000データ x 3ユニット = 30,000個のデータを持つ。

>>> import numpy as np
>>> import cPickle as cp
>>>
>>> da = cp.load(open('data_batch_7'))
>>> da.keys()
['labels', 'data']
>>> data_raw = da['data']     # raw output
>>> data_lbl = da['labels']   # true label
>>> data_raw = np.asarray(data_raw)
>>> data_lbl = np.asarray(data_lbl)
>>> data_raw.shape
(10000, 3)
>>> data_lbl.shape
(3, 10000)
>>> data_lbl = data_lbl.T
>>> data_lbl.shape
(10000, 3)

取得した正解値 data_lbl と出力値 data_raw の先頭10個だけを参照してみる。

>>> data_lbl[0:10,:]
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       [ 0.,  1.,  1.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  1.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  1.]], dtype=float32)
>>> data_raw[0:10,:]
array([[  9.98521745e-01,   1.49850443e-03,   4.11541946e-03],
       [  1.20352823e-02,   9.88052726e-01,   3.22542548e-01],
       [  9.94093001e-01,   5.99678187e-03,   7.45591475e-03],
       [  2.92361937e-02,   9.71927702e-01,   9.64868903e-01],
       [  2.93485690e-02,   9.70762134e-01,   6.91684261e-02],
       [  9.99665618e-01,   3.34766519e-04,   2.66197207e-03],
       [  1.03651799e-01,   8.95208240e-01,   1.80083804e-03],
       [  9.62865233e-01,   3.65868807e-02,   8.87887657e-01],
       [  2.12325543e-01,   8.14015448e-01,   7.86685292e-03],
       [  9.76591468e-01,   2.24058982e-02,   8.57828259e-01]], dtype=float32)

出力値を 閾値0.5で 2値化した結果を data_rth に取得し、これも先頭から10個だけを参照してみる。
そこそこ学習が進んでいるので上記の正解値 data_lbl とほぼ同じ値だ。(1個だけ間違いあり)

>>> data_rth = (data_raw >= 0.5)*1.0
>>> data_rth[0:10,:]
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       [ 0.,  1.,  1.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  1.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  1.]])

ここから一気にトータルのエラー率を算出してみる。
出力ユニットNo.1,2,3の順に不正解数は 289, 293, 537個
出力値の総数は 3ユニット x 10,000データで30,000個だ。

>>> out1_raw = data_raw[:,0]
>>> out2_raw = data_raw[:,1]
>>> out3_raw = data_raw[:,2]
>>>
>>> out1_lbl = data_lbl[:,0]
>>> out2_lbl = data_lbl[:,1]
>>> out3_lbl = data_lbl[:,2]
>>>
>>> out1_rth = (out1_raw >= 0.5) * 1
>>> out2_rth = (out2_raw >= 0.5) * 1
>>> out3_rth = (out3_raw >= 0.5) * 1
>>>
>>> out1_lbl = (out1_lbl >= 0.5) * 1
>>> out2_lbl = (out2_lbl >= 0.5) * 1
>>> out3_lbl = (out3_lbl >= 0.5) * 1
>>>
>>> out1_diff = np.where(out1_rth != out1_lbl)
>>> out2_diff = np.where(out2_rth != out2_lbl)
>>> out3_diff = np.where(out3_rth != out3_lbl)
>>>
>>> diff1 = len(out1_diff[0])
>>> diff2 = len(out2_diff[0])
>>> diff3 = len(out3_diff[0])
>>> diff1
289
>>> diff2
293
>>> diff3
537
>>>
>>> errorRate = float((diff1 + diff2 + diff3)) / (len(out1_lbl) + len(out2_lbl) + len(out3_lbl))
>>> errorRate
0.0373

ここまでの作業で 「生出力値から計算したエラー率」=「学習時に表示されたエラー率」 であることが確認できた。

======================Test output======================
dce:  (crossent) 0.319596, (err) 0.037300, (Godd) 0.965015, 0.978518, (Geven) 0.977137, 0.963053, (Gtri) 0.925373, 0.940106

【余談】 ちょっと息抜きでNoLimits2

(23) GTX760からGTX780に変えてみるGTX780 を使うことになり、まだまだ十分にハイスペックな GTX760 が余ってしまった。ヤフオクで売ってもよいのだが、せっかく手元にある上等なグラボなので Windowsパソコンに入っている GTX650 と換装することにした。

上等なグラボの性能を体感したくて久々に AION や Line2 に入ってみたが、過疎ってて楽しくない…
そこで一人でも楽しめるジェットコースターシミュレーターを動かしてみた。

NoLimits は今から約14年前にWEB上で29ドルぐらいで購入した roller coaster simulator なるソフトだ。
エディターで好きなレールを敷設し、ありえないくらいにスリリングなジェットコースターを作り、しかも乗車することができるという、ジェットコースター好きにはたまらないソフトだ。

現行の最新バージョンは NoLimits 2.0.5.2、進化した地形作成ツールではなんとトンネルも掘れる!

このソフトの大きな楽しみの一つは、海外のローラーコースターマニアの方々が自作の手間暇かけた素晴らしいコースデータをWEB上に公開してくださっていて、これをインポートして遊べること!
自分ですごいコースが作れなくても、世界中のマニアの方々の力作を楽しむことができる。

ちなみに自分のお気に入りの実物ジェットコースターは…
1位 サンダードルフィン(東京ドーム)
2位 ええじゃないか(富士急ハイランド)
3位 バンデット(よみうりランド)

アブダビのフェラーリワールドにあるという世界最速240キロの フォーミュラ・ロッサ に乗ってみたいものだが…
なんとあの ドドンパ(富士急ハイランド) よりも70キロも速い!

20140914_01

(26) cuda-convnet2で二値分類器を作ってみる

cuda-convnet2 では、新たに cross-entropy cost layer が追加された。

これを使うと何がうれしい?

softmax を用いた多値分類器では、出力ユニット間の相対的な大小関係を学習させて
最大出力ユニット番号=付与するラベル」 として使用した。

これに対して2値分類器では、一つの出力ユニットが 「ある条件に対して Yes or No」 だけを学習する。
これを応用し、複数個の出力ユニットに独立した条件判定ロジックを学習させれば、一つの入力データに対して複数種類の判定結果が出力可能なニューラルネットを作れる。


(1) 実験条件

ブログのテーマがMNISTなので、ここでもMNISTを使って実験する。
1) 学習・テスト画像はMNIST数字画像データを使用する。
2) ベースとなるネットワーク構成は (25) cuda-convnet2でMNIST自動認識(その1) で使用したもの。
3) 出力層を detection cross-entropy cost layer に変更する。
4) 出力ユニット数3個を以下のように定義する。

出力ユニットNo.1 奇数フラグ
出力ユニットNo.2 偶数フラグ
出力ユニットNo.3 3の倍数フラグ

このルールに従うと、入力した MNIST数字画像 0~9 に対する出力は次のようになる。

数字 0 1 2 3 4 5 6 7 8 9
出力値(bin) 010 100 010 101 010 100 011 100 010 101

(2) 学習・テストデータ作り

cuda-convnet用MNISTデータを作る(その2), (その3) で作成した cuda-convnet入力データ作成ツールに少々手を入れる必要あり。改造箇所は1点だけ。
1) 教師データはラベル番号ではなく、上記の出力層の出力値(3次元情報)とする。

例) 今までの多値分類では data_batch_N ファイル中の教師データを以下のように設定していた。 (値は適当)

labels = [0,3,2,6,1,3]

今回の2値分類器では、以下のように設定する。(値は適当)

labels = [[0,1,0],[1,0,0],[1,1,0],[1,0,1]]

(3) NN定義ファイル作り

layers-MNIST-tri.cfg
最後のフル接続層の neuron を logistic にしないと実行時にエラーが出る。

[data]
type=data
dataIdx=0

[labels]
type=data
dataIdx=1

[conv1]
type=conv
inputs=data
channels=1
filters=32
padding=0
stride=1
filterSize=5
neuron=tanh[1,1]
initW=0.0001
sumWidth=4
sharedBiases=1
gpu=0

[pool1]
type=pool
pool=max
inputs=conv1
start=0
sizeX=2
stride=2
outputsX=0
channels=32

[conv2]
type=conv
inputs=pool1
filters=32
padding=0
stride=1
filterSize=5
channels=32
neuron=tanh[1,1]
initW=0.01
sumWidth=2
sharedBiases=1

[pool2]
type=pool
pool=avg
inputs=conv2
start=0
sizeX=2
stride=2
outputsX=0
channels=32

[fcOut]
type=fc
outputs=3
inputs=pool2
initW=0.01
initB=0.1
neuron=logistic

[dce]
type=cost.dce
inputs=labels,fcOut
gpu=0

layer-params-MNIST-tri.cfg

[conv1]
epsW=0.01
epsB=0.01
momW=0.9
momB=0.9
wc=0.0001

[conv2]
epsW=0.01
epsB=0.01
momW=0.9
momB=0.9
wc=0.0001

[fcOut]
epsW=0.01
epsB=0.01
momW=0.9
momB=0.9
wc=0.0001

[dce]
coeff=1
topk=1

(4) cuda-convnet2プログラム改造

プログラムソースコードの変更は以下の1点だけ。
1) MNISTデータ用 且つ detection cross-entropy cost layer用 のデータプロバイダーを新規作成する。

ここの作業にはいろいろと悩み、紆余曲折があり、方針決定&作業実施に 5時間を要した。
cuda-convnet2ホームページのこちらの記述 を読むと、cross-entropy cost layer を使いたい場合は、既存の ImageDataProvider class を使えばよいように書いてある。しかし、ImageDataProvider には従来仕様の data_batch_N ファイルが入力できない。具体的には、画像データの代わりにJPEG画像ファイルパスを設定するなど、従来とはデータファイル仕様が異なるためだ。この新仕様に合わせるためには、MNISTの場合は各画像をばらばらに7万個のJPEG画像ファイル化するなどの手間が必要になる。7万個の画像ファイルを持つのもイヤなので、今回は専用の MNIST data provider for the Detection cross-entropy cost layer を自作することにした。


(5) 実行結果

10epochs 流してみた。
今までよりもログ出力される情報量が多いようだ。

2014年  9月 14日 日曜日 02:21:24 JST
Initialized data layer 'data', producing 784 outputs
Initialized data layer 'labels', producing 3 outputs
Initialized convolutional layer 'conv1' on GPUs 0, producing 24x24 32-channel output
Initialized max-pooling layer 'pool1' on GPUs 0, producing 12x12 32-channel output
Initialized convolutional layer 'conv2' on GPUs 0, producing 8x8 32-channel output
Initialized avg-pooling layer 'pool2' on GPUs 0, producing 4x4 32-channel output
Initialized fully-connected layer 'fcOut' on GPUs 0, producing 3 outputs
Initialized detection cross-entropy cost 'dce' on GPUs 0
Initialized neuron layer 'fcOut_neuron' on GPUs 0, producing 3 outputs
Initialized neuron layer 'conv2_neuron' on GPUs 0, producing 2048 outputs
Initialized neuron layer 'conv1_neuron' on GPUs 0, producing 18432 outputs
Layer conv2_neuron using acts from layer conv2
Layer fcOut_neuron using acts from layer fcOut
Layer conv1_neuron using acts from layer conv1
=========================
Importing cudaconvnet._ConvNet C++ module
Fwd terminal: dce
found bwd terminal conv1[0] in passIdx=0
=========================
Training ConvNet
Add PCA noise to color channels with given scale                        : 0   [DEFAULT]
Check gradients and quit?                                               : 0   [DEFAULT]
Conserve GPU memory (slower)?                                           : 0   [DEFAULT]
Convert given conv layers to unshared local                             :
Cropped DP: crop size (0 = don't crop)                                  : 28
Cropped DP: test on multiple patches?                                   : 0   [DEFAULT]
Data batch range: testing                                               : 7-7
Data batch range: training                                              : 1-6
Data path                                                               : /home/user/cuda/cuda-convnet2/data/MNIST-tri/
Data provider                                                           : DCE
Force save before quitting                                              : 0   [DEFAULT]
GPU override                                                            : 0
Layer definition file                                                   : /home/user/cuda/cuda-convnet2/config/MNIST/layers-MNIST-tri.cfg
Layer file path prefix                                                  :     [DEFAULT]
Layer parameter file                                                    : /home/user/cuda/cuda-convnet2/config/MNIST/layer-params-MNIST-tri.cfg
Load file                                                               :     [DEFAULT]
Logreg cost layer name (for --test-out)                                 :     [DEFAULT]
Minibatch size                                                          : 128 [DEFAULT]
Number of epochs                                                        : 10
Output test case predictions to given path                              :     [DEFAULT]
Save file override                                                      :
Save path                                                               : /home/user/cuda/cuda-convnet2/save/MNIST/
Subtract this scalar from image (-1 = don't)                            : -1  [DEFAULT]
Test and quit?                                                          : 0   [DEFAULT]
Test on one batch at a time?                                            : 1   [DEFAULT]
Testing frequency                                                       : 6
Unshare weight matrices in given layers                                 :
Write test data features from given layer                               :     [DEFAULT]
Write test data features to this path (to be used with --write-features):     [DEFAULT]
=========================
Running on CUDA device(s) 0
Current time: Sun Sep 14 02:21:26 2014
Saving checkpoints to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
=========================
1.1 (0.00%)... dce:  (crossent) 1.199113, (err) 0.183633, (Godd) 0.813920, 0.890335, (Geven) 0.848251, 0.845842, (Gtri) 0.706835, 0.683478 (0.563 sec)
1.2 (1.67%)... dce:  (crossent) 0.766782, (err) 0.103000, (Godd) 0.919650, 0.914490, (Geven) 0.908736, 0.915131, (Gtri) 0.832205, 0.823325 (0.253 sec)
1.3 (3.33%)... dce:  (crossent) 0.567953, (err) 0.068433, (Godd) 0.950119, 0.945626, (Geven) 0.944995, 0.949025, (Gtri) 0.883268, 0.860500 (0.256 sec)
1.4 (5.00%)... dce:  (crossent) 0.443588, (err) 0.052900, (Godd) 0.963820, 0.960780, (Geven) 0.958771, 0.963053, (Gtri) 0.897314, 0.897088 (0.251 sec)
1.5 (6.67%)... dce:  (crossent) 0.410039, (err) 0.049000, (Godd) 0.964023, 0.958687, (Geven) 0.958937, 0.964177, (Gtri) 0.914705, 0.908954 (0.256 sec)
1.6 (8.33%)... dce:  (crossent) 0.349523, (err) 0.040433, (Godd) 0.967851, 0.963834, (Geven) 0.963680, 0.966802, (Gtri) 0.938540, 0.928083
======================Test output======================
dce:  (crossent) 0.317917, (err) 0.036367, (Godd) 0.973128, 0.970635, (Geven) 0.969439, 0.972391, (Gtri) 0.944646, 0.922921
----------------------Averages-------------------------
dce:  (crossent) 0.317917, (err) 0.036367, (Godd) 0.973128, 0.970635, (Geven) 0.969439, 0.972391, (Gtri) 0.944646, 0.922921
-------------------------------------------------------
Layer 'conv1' weights[0]: 8.155207e-02 [7.599560e-04] [9.318661e-03]
Layer 'conv1' biases: 2.710742e-03 [2.686247e-05]
Layer 'conv2' weights[0]: 2.011198e-02 [1.423615e-04] [7.078444e-03]
Layer 'conv2' biases: 2.185383e-02 [2.591967e-04]
Layer 'fcOut' weights[0]: 8.837998e-02 [4.182930e-04] [4.732893e-03]
Layer 'fcOut' biases: 1.263399e-01 [7.117551e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.380 sec)
2.1 (10.00%)... dce:  (crossent) 0.309526, (err) 0.035000, (Godd) 0.973669, 0.970020, (Geven) 0.969685, 0.973225, (Gtri) 0.941206, 0.938634 (0.262 sec)
2.2 (11.67%)... dce:  (crossent) 0.305727, (err) 0.035700, (Godd) 0.973188, 0.969046, (Geven) 0.968035, 0.971641, (Gtri) 0.942188, 0.938213 (0.257 sec)
2.3 (13.33%)... dce:  (crossent) 0.301048, (err) 0.033967, (Godd) 0.976615, 0.970843, (Geven) 0.971296, 0.975833, (Gtri) 0.943435, 0.931514 (0.256 sec)
2.4 (15.00%)... dce:  (crossent) 0.269323, (err) 0.030567, (Godd) 0.978870, 0.976941, (Geven) 0.976287, 0.977873, (Gtri) 0.942836, 0.939759 (0.255 sec)
2.5 (16.67%)... dce:  (crossent) 0.262750, (err) 0.030367, (Godd) 0.980865, 0.972722, (Geven) 0.972105, 0.980368, (Gtri) 0.945326, 0.943662 (0.258 sec)
2.6 (18.33%)... dce:  (crossent) 0.232449, (err) 0.027100, (Godd) 0.979010, 0.977075, (Geven) 0.975994, 0.979352, (Gtri) 0.957322, 0.948595
======================Test output======================
dce:  (crossent) 0.245786, (err) 0.028500, (Godd) 0.987382, 0.971620, (Geven) 0.971611, 0.986602, (Gtri) 0.976171, 0.911044
----------------------Averages-------------------------
dce:  (crossent) 0.245786, (err) 0.028500, (Godd) 0.987382, 0.971620, (Geven) 0.971611, 0.986602, (Gtri) 0.976171, 0.911044
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.103154e-01 [6.913404e-04] [6.266943e-03]
Layer 'conv1' biases: 3.618342e-03 [1.293126e-05]
Layer 'conv2' weights[0]: 2.478795e-02 [1.239718e-04] [5.001291e-03]
Layer 'conv2' biases: 2.868914e-02 [1.545805e-04]
Layer 'fcOut' weights[0]: 1.136545e-01 [2.959493e-04] [2.603940e-03]
Layer 'fcOut' biases: 1.387306e-01 [3.311247e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.391 sec)
3.1 (20.00%)... dce:  (crossent) 0.224097, (err) 0.025033, (Godd) 0.981969, 0.977515, (Geven) 0.976974, 0.981136, (Gtri) 0.958230, 0.957516 (0.274 sec)
3.2 (21.67%)... dce:  (crossent) 0.230526, (err) 0.026400, (Godd) 0.979814, 0.976591, (Geven) 0.974845, 0.978679, (Gtri) 0.957795, 0.957320 (0.274 sec)
3.3 (23.33%)... dce:  (crossent) 0.228185, (err) 0.025833, (Godd) 0.981050, 0.979117, (Geven) 0.977935, 0.981113, (Gtri) 0.956699, 0.949204 (0.268 sec)
3.4 (25.00%)... dce:  (crossent) 0.213223, (err) 0.023333, (Godd) 0.984202, 0.982263, (Geven) 0.981969, 0.983963, (Gtri) 0.954568, 0.954568 (0.266 sec)
3.5 (26.67%)... dce:  (crossent) 0.208459, (err) 0.023700, (Godd) 0.983148, 0.980233, (Geven) 0.979625, 0.982797, (Gtri) 0.960953, 0.953219 (0.266 sec)
3.6 (28.33%)... dce:  (crossent) 0.185518, (err) 0.021233, (Godd) 0.983759, 0.981621, (Geven) 0.981018, 0.983401, (Gtri) 0.965438, 0.962016
======================Test output======================
dce:  (crossent) 0.204320, (err) 0.023067, (Godd) 0.990608, 0.976941, (Geven) 0.976972, 0.990459, (Gtri) 0.983051, 0.923427
----------------------Averages-------------------------
dce:  (crossent) 0.204320, (err) 0.023067, (Godd) 0.990608, 0.976941, (Geven) 0.976972, 0.990459, (Gtri) 0.983051, 0.923427
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.302679e-01 [6.364945e-04] [4.886041e-03]
Layer 'conv1' biases: 3.856979e-03 [8.572064e-06]
Layer 'conv2' weights[0]: 2.778544e-02 [1.004645e-04] [3.615725e-03]
Layer 'conv2' biases: 3.361420e-02 [9.831827e-05]
Layer 'fcOut' weights[0]: 1.289217e-01 [2.037354e-04] [1.580304e-03]
Layer 'fcOut' biases: 1.446893e-01 [1.287783e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.397 sec)
4.1 (30.00%)... dce:  (crossent) 0.188140, (err) 0.021433, (Godd) 0.983564, 0.979684, (Geven) 0.978998, 0.983367, (Gtri) 0.967380, 0.965217 (0.270 sec)
4.2 (31.67%)... dce:  (crossent) 0.199008, (err) 0.022367, (Godd) 0.981395, 0.979687, (Geven) 0.978917, 0.980335, (Gtri) 0.968337, 0.963772 (0.274 sec)
4.3 (33.33%)... dce:  (crossent) 0.194720, (err) 0.021700, (Godd) 0.984011, 0.982072, (Geven) 0.981557, 0.983550, (Gtri) 0.962494, 0.959818 (0.273 sec)
4.4 (35.00%)... dce:  (crossent) 0.178280, (err) 0.019867, (Godd) 0.987352, 0.984628, (Geven) 0.984015, 0.987211, (Gtri) 0.962003, 0.959588 (0.274 sec)
4.5 (36.67%)... dce:  (crossent) 0.181196, (err) 0.020500, (Godd) 0.986097, 0.981419, (Geven) 0.981273, 0.986238, (Gtri) 0.967529, 0.959256 (0.266 sec)
4.6 (38.33%)... dce:  (crossent) 0.163559, (err) 0.018567, (Godd) 0.985550, 0.983992, (Geven) 0.983633, 0.985425, (Gtri) 0.967636, 0.969106
======================Test output======================
dce:  (crossent) 0.190747, (err) 0.021300, (Godd) 0.989081, 0.981868, (Geven) 0.981452, 0.988226, (Gtri) 0.986279, 0.926459
----------------------Averages-------------------------
dce:  (crossent) 0.190747, (err) 0.021300, (Godd) 0.989081, 0.981868, (Geven) 0.981452, 0.988226, (Gtri) 0.986279, 0.926459
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.452356e-01 [5.611114e-04] [3.863456e-03]
Layer 'conv1' biases: 4.199057e-03 [9.355912e-06]
Layer 'conv2' weights[0]: 3.022003e-02 [9.920518e-05] [3.282762e-03]
Layer 'conv2' biases: 3.737093e-02 [1.362906e-04]
Layer 'fcOut' weights[0]: 1.403167e-01 [1.990263e-04] [1.418408e-03]
Layer 'fcOut' biases: 1.483610e-01 [2.851311e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.400 sec)
5.1 (40.00%)... dce:  (crossent) 0.172653, (err) 0.019367, (Godd) 0.984570, 0.981657, (Geven) 0.981201, 0.984584, (Gtri) 0.974661, 0.965217 (0.269 sec)
5.2 (41.67%)... dce:  (crossent) 0.177714, (err) 0.019533, (Godd) 0.986410, 0.982975, (Geven) 0.982663, 0.985510, (Gtri) 0.967629, 0.964268 (0.269 sec)
5.3 (43.33%)... dce:  (crossent) 0.170255, (err) 0.018900, (Godd) 0.986751, 0.983058, (Geven) 0.982598, 0.986190, (Gtri) 0.968568, 0.965631 (0.277 sec)
5.4 (45.00%)... dce:  (crossent) 0.163894, (err) 0.018067, (Godd) 0.986985, 0.986401, (Geven) 0.985804, 0.986805, (Gtri) 0.966801, 0.964859 (0.286 sec)
5.5 (46.67%)... dce:  (crossent) 0.162201, (err) 0.018133, (Godd) 0.987108, 0.983791, (Geven) 0.983458, 0.986642, (Gtri) 0.971176, 0.966046 (0.271 sec)
5.6 (48.33%)... dce:  (crossent) 0.148354, (err) 0.016267, (Godd) 0.987532, 0.986166, (Geven) 0.985651, 0.987247, (Gtri) 0.971660, 0.972398
======================Test output======================
dce:  (crossent) 0.172621, (err) 0.019200, (Godd) 0.989861, 0.981277, (Geven) 0.981672, 0.989444, (Gtri) 0.985964, 0.940864
----------------------Averages-------------------------
dce:  (crossent) 0.172621, (err) 0.019200, (Godd) 0.989861, 0.981277, (Geven) 0.981672, 0.989444, (Gtri) 0.985964, 0.940864
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.587927e-01 [5.413409e-04] [3.409105e-03]
Layer 'conv1' biases: 4.576801e-03 [1.049395e-05]
Layer 'conv2' weights[0]: 3.219037e-02 [9.470111e-05] [2.941908e-03]
Layer 'conv2' biases: 4.058194e-02 [1.346725e-04]
Layer 'fcOut' weights[0]: 1.493520e-01 [1.707393e-04] [1.143201e-03]
Layer 'fcOut' biases: 1.511034e-01 [2.599937e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.400 sec)
6.1 (50.00%)... dce:  (crossent) 0.157575, (err) 0.017633, (Godd) 0.985004, 0.984615, (Geven) 0.983783, 0.984381, (Gtri) 0.974807, 0.970932 (0.268 sec)
6.2 (51.67%)... dce:  (crossent) 0.163027, (err) 0.017433, (Godd) 0.988528, 0.983556, (Geven) 0.982485, 0.986959, (Gtri) 0.972865, 0.969727 (0.268 sec)
6.3 (53.33%)... dce:  (crossent) 0.151389, (err) 0.015667, (Godd) 0.988946, 0.986998, (Geven) 0.986426, 0.988830, (Gtri) 0.973838, 0.968916 (0.269 sec)
6.4 (55.00%)... dce:  (crossent) 0.150452, (err) 0.015900, (Godd) 0.988952, 0.987978, (Geven) 0.987830, 0.988632, (Gtri) 0.971270, 0.967369 (0.277 sec)
6.5 (56.67%)... dce:  (crossent) 0.149220, (err) 0.017100, (Godd) 0.986931, 0.985175, (Geven) 0.984848, 0.986642, (Gtri) 0.974411, 0.967304 (0.268 sec)
6.6 (58.33%)... dce:  (crossent) 0.141361, (err) 0.016100, (Godd) 0.986169, 0.986364, (Geven) 0.986829, 0.985830, (Gtri) 0.973178, 0.973917
======================Test output======================
dce:  (crossent) 0.166090, (err) 0.019333, (Godd) 0.984634, 0.985022, (Geven) 0.984169, 0.984369, (Gtri) 0.985000, 0.945919
----------------------Averages-------------------------
dce:  (crossent) 0.166090, (err) 0.019333, (Godd) 0.984634, 0.985022, (Geven) 0.984169, 0.984369, (Gtri) 0.985000, 0.945919
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.691544e-01 [5.411348e-04] [3.199059e-03]
Layer 'conv1' biases: 4.764370e-03 [1.533124e-05]
Layer 'conv2' weights[0]: 3.393728e-02 [1.179916e-04] [3.476754e-03]
Layer 'conv2' biases: 4.355372e-02 [1.882927e-04]
Layer 'fcOut' weights[0]: 1.571101e-01 [2.250832e-04] [1.432646e-03]
Layer 'fcOut' biases: 1.538871e-01 [4.014182e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.395 sec)
7.1 (60.00%)... dce:  (crossent) 0.146865, (err) 0.016633, (Godd) 0.986553, 0.984024, (Geven) 0.983607, 0.985801, (Gtri) 0.977295, 0.973168 (0.270 sec)
7.2 (61.67%)... dce:  (crossent) 0.148073, (err) 0.015967, (Godd) 0.988176, 0.986264, (Geven) 0.985133, 0.987580, (Gtri) 0.975330, 0.971216 (0.269 sec)
7.3 (63.33%)... dce:  (crossent) 0.139719, (err) 0.014200, (Godd) 0.989154, 0.988180, (Geven) 0.987827, 0.988830, (Gtri) 0.976679, 0.973717 (0.275 sec)
7.4 (65.00%)... dce:  (crossent) 0.137491, (err) 0.014633, (Godd) 0.989927, 0.987781, (Geven) 0.987047, 0.990053, (Gtri) 0.974817, 0.971637 (0.280 sec)
7.5 (66.67%)... dce:  (crossent) 0.138026, (err) 0.015767, (Godd) 0.989094, 0.985966, (Geven) 0.985870, 0.988464, (Gtri) 0.974482, 0.970070 (0.283 sec)
7.6 (68.33%)... dce:  (crossent) 0.131545, (err) 0.014533, (Godd) 0.988142, 0.988142, (Geven) 0.987647, 0.987247, (Gtri) 0.974249, 0.977209
======================Test output======================
dce:  (crossent) 0.147773, (err) 0.017033, (Godd) 0.988528, 0.985022, (Geven) 0.984827, 0.988226, (Gtri) 0.981832, 0.956027
----------------------Averages-------------------------
dce:  (crossent) 0.147773, (err) 0.017033, (Godd) 0.988528, 0.985022, (Geven) 0.984827, 0.988226, (Gtri) 0.981832, 0.956027
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.788537e-01 [6.211860e-04] [3.473151e-03]
Layer 'conv1' biases: 5.401216e-03 [1.268923e-05]
Layer 'conv2' weights[0]: 3.544452e-02 [1.119315e-04] [3.157935e-03]
Layer 'conv2' biases: 4.565623e-02 [1.653222e-04]
Layer 'fcOut' weights[0]: 1.637556e-01 [2.176586e-04] [1.329167e-03]
Layer 'fcOut' biases: 1.556429e-01 [3.749391e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.415 sec)
8.1 (70.00%)... dce:  (crossent) 0.135743, (err) 0.015233, (Godd) 0.987367, 0.986588, (Geven) 0.986226, 0.987627, (Gtri) 0.978505, 0.972671 (0.272 sec)
8.2 (71.67%)... dce:  (crossent) 0.142047, (err) 0.015733, (Godd) 0.988936, 0.985684, (Geven) 0.985145, 0.988408, (Gtri) 0.975579, 0.971464 (0.279 sec)
8.3 (73.33%)... dce:  (crossent) 0.132628, (err) 0.015767, (Godd) 0.988544, 0.986013, (Geven) 0.985616, 0.988018, (Gtri) 0.974398, 0.971443 (0.282 sec)
8.4 (75.00%)... dce:  (crossent) 0.134742, (err) 0.014500, (Godd) 0.990713, 0.988175, (Geven) 0.987654, 0.990662, (Gtri) 0.975246, 0.969127 (0.273 sec)
8.5 (76.67%)... dce:  (crossent) 0.128446, (err) 0.014467, (Godd) 0.990871, 0.986954, (Geven) 0.987094, 0.990690, (Gtri) 0.974773, 0.971831 (0.280 sec)
8.6 (78.33%)... dce:  (crossent) 0.122649, (err) 0.014300, (Godd) 0.988338, 0.988142, (Geven) 0.987664, 0.988664, (Gtri) 0.976408, 0.974677
======================Test output======================
dce:  (crossent) 0.151681, (err) 0.018233, (Godd) 0.984077, 0.986598, (Geven) 0.986156, 0.983354, (Gtri) 0.979323, 0.957544
----------------------Averages-------------------------
dce:  (crossent) 0.151681, (err) 0.018233, (Godd) 0.984077, 0.986598, (Geven) 0.986156, 0.983354, (Gtri) 0.979323, 0.957544
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.873097e-01 [5.005563e-04] [2.672345e-03]
Layer 'conv1' biases: 5.523826e-03 [1.317529e-05]
Layer 'conv2' weights[0]: 3.681655e-02 [9.820487e-05] [2.667411e-03]
Layer 'conv2' biases: 4.807946e-02 [1.357576e-04]
Layer 'fcOut' weights[0]: 1.698663e-01 [1.870220e-04] [1.100995e-03]
Layer 'fcOut' biases: 1.567201e-01 [2.885009e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.418 sec)
9.1 (80.00%)... dce:  (crossent) 0.129455, (err) 0.014500, (Godd) 0.988145, 0.986391, (Geven) 0.986232, 0.988032, (Gtri) 0.978120, 0.977391 (0.269 sec)
9.2 (81.67%)... dce:  (crossent) 0.138982, (err) 0.015867, (Godd) 0.989122, 0.985104, (Geven) 0.984130, 0.988408, (Gtri) 0.975124, 0.972705 (0.278 sec)
9.3 (83.33%)... dce:  (crossent) 0.130423, (err) 0.014233, (Godd) 0.988760, 0.987786, (Geven) 0.987419, 0.988221, (Gtri) 0.978664, 0.973717 (0.281 sec)
9.4 (85.00%)... dce:  (crossent) 0.125232, (err) 0.013933, (Godd) 0.990521, 0.988569, (Geven) 0.988450, 0.990256, (Gtri) 0.973427, 0.974649 (0.278 sec)
9.5 (86.67%)... dce:  (crossent) 0.126634, (err) 0.015000, (Godd) 0.989501, 0.987349, (Geven) 0.987278, 0.989476, (Gtri) 0.975696, 0.969316 (0.269 sec)
9.6 (88.33%)... dce:  (crossent) 0.119767, (err) 0.013300, (Godd) 0.990493, 0.988340, (Geven) 0.988083, 0.990283, (Gtri) 0.977665, 0.975437
======================Test output======================
dce:  (crossent) 0.143420, (err) 0.015733, (Godd) 0.987016, 0.988766, (Geven) 0.988611, 0.986805, (Gtri) 0.982656, 0.959313
----------------------Averages-------------------------
dce:  (crossent) 0.143420, (err) 0.015733, (Godd) 0.987016, 0.988766, (Geven) 0.988611, 0.986805, (Gtri) 0.982656, 0.959313
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.955977e-01 [5.039437e-04] [2.576430e-03]
Layer 'conv1' biases: 5.744199e-03 [1.012264e-05]
Layer 'conv2' weights[0]: 3.810624e-02 [1.096704e-04] [2.878017e-03]
Layer 'conv2' biases: 5.125387e-02 [1.558181e-04]
Layer 'fcOut' weights[0]: 1.754434e-01 [2.125424e-04] [1.211459e-03]
Layer 'fcOut' biases: 1.569183e-01 [3.300385e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.410 sec)
10.1 (90.00%)... dce:  (crossent) 0.124347, (err) 0.014267, (Godd) 0.987379, 0.987574, (Geven) 0.987024, 0.987424, (Gtri) 0.979811, 0.976646 (0.293 sec)
10.2 (91.67%)... dce:  (crossent) 0.127718, (err) 0.014533, (Godd) 0.989333, 0.986845, (Geven) 0.985558, 0.988822, (Gtri) 0.978808, 0.974194 (0.268 sec)
10.3 (93.33%)... dce:  (crossent) 0.121530, (err) 0.012833, (Godd) 0.991308, 0.988574, (Geven) 0.988252, 0.990861, (Gtri) 0.979436, 0.974981 (0.269 sec)
10.4 (95.00%)... dce:  (crossent) 0.120518, (err) 0.012667, (Godd) 0.992289, 0.989160, (Geven) 0.988871, 0.992083, (Gtri) 0.977341, 0.974398 (0.270 sec)
10.5 (96.67%)... dce:  (crossent) 0.121691, (err) 0.014867, (Godd) 0.988538, 0.988733, (Geven) 0.988266, 0.988666, (Gtri) 0.973784, 0.971579 (0.263 sec)
10.6 (98.33%)... dce:  (crossent) 0.113142, (err) 0.013233, (Godd) 0.989715, 0.988933, (Geven) 0.988673, 0.989474, (Gtri) 0.977446, 0.976703
======================Test output======================
dce:  (crossent) 0.136332, (err) 0.014800, (Godd) 0.985885, 0.991131, (Geven) 0.990814, 0.985384, (Gtri) 0.979770, 0.966894
----------------------Averages-------------------------
dce:  (crossent) 0.136332, (err) 0.014800, (Godd) 0.985885, 0.991131, (Geven) 0.990814, 0.985384, (Gtri) 0.979770, 0.966894
-------------------------------------------------------
Layer 'conv1' weights[0]: 2.045221e-01 [5.905093e-04] [2.887264e-03]
Layer 'conv1' biases: 6.253900e-03 [9.817431e-06]
Layer 'conv2' weights[0]: 3.927734e-02 [9.667401e-05] [2.461318e-03]
Layer 'conv2' biases: 5.389512e-02 [1.231133e-04]
Layer 'fcOut' weights[0]: 1.806059e-01 [1.755370e-04] [9.719342e-04]
Layer 'fcOut' biases: 1.589827e-01 [2.841916e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-14_02.21.24
======================================================= (0.395 sec)
11.1 (100.00%)... dce:  (crossent) 0.118477, (err) 0.013900, (Godd) 0.988551, 0.987771, (Geven) 0.987231, 0.988032, (Gtri) 0.980289, 0.976149 (0.290 sec)
2014年  9月 14日 日曜日 02:21:45 JST

Test output に表示されているのは以下のもの。

(crossent) 0.136332 損失関数出力値
(err) 0.014800 全テストデータに対するエラー率
※エラー出力数 / (テスト画像枚数 × 出力ユニット数)
(Godd) 0.985885, 0.991131 出力ユニット#1のPositive正解率
左側はPrecision: TruePositive / DeclaredPositive
右側はRecall: TruePositive / Positive
※batches.metaでGoddと命名した
(Geven) 0.990814, 0.985384 出力ユニットNo.2の正解率
※batches.metaでGevenと命名した
(Gtri) 0.979770, 0.966894 出力ユニットNo.3の正解率
※batches.metaでGtriと命名した
TruePositive (正解値, 出力値) = (1, 1)
DeclaredPositive 出力値 = 1
Positive 正解値 = 1
======================Test output======================
dce:  (crossent) 0.136332, (err) 0.014800, (Godd) 0.985885, 0.991131, (Geven) 0.990814, 0.985384, (Gtri) 0.979770, 0.966894

(6) 念のために…

本当に学習が進んでいるのか確認してみたくなったので、上記のプログラムを実行中に
BinomialCrossEntropyCostLayer::fpropActs() で出力層の出力値をモニタリングしてみた。(各バッチデータの先頭3個だけ)

true label 教師データ
prob. 出力層の生出力値
prob.th. 閾値0.5で二値化済みの出力値

1) epoch#1-batch#1
当然だけれどもまだまだ全然ダメダメ…

[src/layer.cu][fpropActs][2117] true label(:,0) [1.000000][0.000000][0.000000]
[src/layer.cu][fpropActs][2118] true label(:,1) [0.000000][1.000000][1.000000]
[src/layer.cu][fpropActs][2119] true label(:,2) [0.000000][1.000000][0.000000]

[src/layer.cu][fpropActs][2123] prob.     (:,0) [0.524935][0.525479][0.525543]
[src/layer.cu][fpropActs][2124] prob.     (:,1) [0.525781][0.525362][0.524247]
[src/layer.cu][fpropActs][2125] prob.     (:,2) [0.525054][0.525753][0.524198]
 ↓  threshold = 0.5
[src/layer.cu][fpropActs][2141] prob.th.  (:,0) [1.000000][1.000000][1.000000]
[src/layer.cu][fpropActs][2142] prob.th.  (:,1) [1.000000][1.000000][1.000000]
[src/layer.cu][fpropActs][2143] prob.th.  (:,2) [1.000000][1.000000][1.000000]

2) epoch#1-batch#2
まだ2バッチ目(=学習済みデータは10,000個) だけれどもここは3/3正解!

[src/layer.cu][fpropActs][2117] true label(:,0) [1.000000][0.000000][1.000000]
[src/layer.cu][fpropActs][2118] true label(:,1) [0.000000][1.000000][0.000000]
[src/layer.cu][fpropActs][2119] true label(:,2) [1.000000][0.000000][0.000000]

[src/layer.cu][fpropActs][2123] prob.     (:,0) [0.995084][0.005174][0.962656]
[src/layer.cu][fpropActs][2124] prob.     (:,1) [0.232831][0.785004][0.025421]
[src/layer.cu][fpropActs][2125] prob.     (:,2) [0.992378][0.008622][0.050015]
 ↓  threshold = 0.5
[src/layer.cu][fpropActs][2141] prob.th.  (:,0) [1.000000][0.000000][1.000000]
[src/layer.cu][fpropActs][2142] prob.th.  (:,1) [0.000000][1.000000][0.000000]
[src/layer.cu][fpropActs][2143] prob.th.  (:,2) [1.000000][0.000000][0.000000]

3) epoch#1-batch#3
まだ3バッチ目(=学習済みデータは20,000個) 惜しいけどちょっと間違えてる。

[src/layer.cu][fpropActs][2117] true label(:,0) [1.000000][0.000000][0.000000]
[src/layer.cu][fpropActs][2118] true label(:,1) [1.000000][0.000000][1.000000]
[src/layer.cu][fpropActs][2119] true label(:,2) [1.000000][0.000000][0.000000]

[src/layer.cu][fpropActs][2123] prob.     (:,0) [0.970718][0.028501][0.343004]
[src/layer.cu][fpropActs][2124] prob.     (:,1) [0.854348][0.155817][0.465876]
[src/layer.cu][fpropActs][2125] prob.     (:,2) [0.881519][0.121382][0.047769]
 ↓  threshold = 0.5
[src/layer.cu][fpropActs][2141] prob.th.  (:,0) [1.000000][0.000000][0.000000]
[src/layer.cu][fpropActs][2142] prob.th.  (:,1) [1.000000][0.000000][0.000000]
[src/layer.cu][fpropActs][2143] prob.th.  (:,2) [1.000000][0.000000][0.000000]

4) epoch#9-batch#2
ちょっと進んで9epochsの2バッチ目(=学習済みデータは490,000個) 出力値はほぼほぼ教師データと同じだ!

[src/layer.cu][fpropActs][2117] true label(:,0) [1.000000][0.000000][1.000000]
[src/layer.cu][fpropActs][2118] true label(:,1) [0.000000][1.000000][0.000000]
[src/layer.cu][fpropActs][2119] true label(:,2) [1.000000][0.000000][0.000000]

[src/layer.cu][fpropActs][2123] prob.     (:,0) [0.999963][0.000038][0.999809]
[src/layer.cu][fpropActs][2124] prob.     (:,1) [0.000016][0.999983][0.000180]
[src/layer.cu][fpropActs][2125] prob.     (:,2) [0.999750][0.000245][0.001855]
 ↓  threshold = 0.5
[src/layer.cu][fpropActs][2141] prob.th.  (:,0) [1.000000][0.000000][1.000000]
[src/layer.cu][fpropActs][2142] prob.th.  (:,1) [0.000000][1.000000][0.000000]
[src/layer.cu][fpropActs][2143] prob.th.  (:,2) [1.000000][0.000000][0.000000]

cuda-convnet2 でいろいろと遊べる幅が広がった(^o^)/

(25) cuda-convnet2でMNIST自動認識(その1)

cuda-convnet2上で (15) cuda-convnetでMNIST自動認識(その2) とほぼ同じネットワーク構成で学習させてみた。

「まったく同じ」 ではなく 「ほぼ同じ」 なのは、同じに出来ない点があったため…

cuda-convnet2 では、convolution層のフィルター数 filters の値が32の倍数でないとエラーになる。
cuda-convnet のときと同じ filters=16 を指定したら、以下のようなエラーが出て止まった。

1.1 (0.00%)...python: src/img_acts.cu:1204: void _imgActs(NVMatrix&, NVMatrix&, NVMatrix&,
int, int, int, int, int, int, int, float, float, bool):
Assertion `numFilters % (32*numGroups) == 0' failed.

準備

(1) 入力データは(14) cuda-convnetでMNIST自動認識(その1) で作成したものをそのまま使用
(2) convnet.pyMNIST用data provider を追加

from convdata import ImageDataProvider, CIFARDataProvider, DummyConvNetLogRegDataProvider, MNISTDataProvider
 :
DataProvider.register_data_provider('MNIST', 'MNIST data provider', MNISTDataProvider)

(3) convdata.py MNIST用data provider を追加実装

class MNISTDataProvider(LabeledDataProvider):
 :

(4) ネットワーク定義ファイルを作成(CIFAR10用のものをコピーして編集)
layers-MNIST.cfg

[data]
type=data
dataIdx=0

[labels]
type=data
dataIdx=1

[conv1]
type=conv
inputs=data
channels=1
filters=32
padding=0
stride=1
filterSize=5
neuron=tanh[1,1]
initW=0.0001
sumWidth=4
sharedBiases=1
gpu=0

[pool1]
type=pool
pool=max
inputs=conv1
start=0
sizeX=2
stride=2
outputsX=0
channels=32

[conv2]
type=conv
inputs=pool1
filters=32
padding=0
stride=1
filterSize=5
channels=32
neuron=tanh[1,1]
initW=0.01
sumWidth=2
sharedBiases=1

[pool2]
type=pool
pool=avg
inputs=conv2
start=0
sizeX=2
stride=2
outputsX=0
channels=32

[fcOut]
type=fc
outputs=10
inputs=pool2
initW=0.01
initB=0.1

[probs]
type=softmax
inputs=fcOut

[logprob]
type=cost.logreg
inputs=labels,probs
gpu=0

layer-params-MNIST.cfg

[conv1]
epsW=0.01
epsB=0.01
momW=0.9
momB=0.9
wc=0.0001

[conv2]
epsW=0.01
epsB=0.01
momW=0.9
momB=0.9
wc=0.0001

[fcOut]
epsW=0.01
epsB=0.01
momW=0.9
momB=0.9
wc=0.0001

[logprob]
coeff=1

結果

10epochs97.0% の正解率だった。

=========================
Running on CUDA device(s) 0
Current time: Thu Sep 11 23:26:07 2014
Saving checkpoints to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
=========================
1.1 (0.00%)... logprob:  1.172366, 0.338300, 0.338300 (0.408 sec)
1.2 (1.67%)... logprob:  0.644550, 0.195300, 0.195300 (0.283 sec)
1.3 (3.33%)... logprob:  0.534813, 0.158900, 0.158900 (0.261 sec)
1.4 (5.00%)... logprob:  0.490603, 0.137000, 0.137000 (0.266 sec)
1.5 (6.67%)... logprob:  0.414907, 0.118000, 0.118000 (0.270 sec)
1.6 (8.33%)... logprob:  0.323441, 0.090700, 0.090700
======================Test output======================
logprob:  0.331363, 0.096700, 0.096700
----------------------Averages-------------------------
logprob:  0.331363, 0.096700, 0.096700
-------------------------------------------------------
Layer 'conv1' weights[0]: 5.603852e-02 [4.257733e-04] [7.597867e-03]
Layer 'conv1' biases: 4.830383e-03 [1.214175e-05]
Layer 'conv2' weights[0]: 1.724966e-02 [6.070063e-05] [3.518947e-03]
Layer 'conv2' biases: 1.526958e-02 [5.254558e-05]
Layer 'fcOut' weights[0]: 5.033513e-02 [1.485135e-04] [2.950494e-03]
Layer 'fcOut' biases: 1.051513e-01 [2.423674e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.386 sec)
2.1 (10.00%)... logprob:  0.318453, 0.086400, 0.086400 (0.262 sec)
2.2 (11.67%)... logprob:  0.310761, 0.089500, 0.089500 (0.272 sec)
2.3 (13.33%)... logprob:  0.286774, 0.080500, 0.080500 (0.277 sec)
2.4 (15.00%)... logprob:  0.278840, 0.075800, 0.075800 (0.268 sec)
2.5 (16.67%)... logprob:  0.254464, 0.074000, 0.074000 (0.260 sec)
2.6 (18.33%)... logprob:  0.205112, 0.055300, 0.055300
======================Test output======================
logprob:  0.212422, 0.060000, 0.060000
----------------------Averages-------------------------
logprob:  0.212422, 0.060000, 0.060000
-------------------------------------------------------
Layer 'conv1' weights[0]: 7.594936e-02 [4.058031e-04] [5.343074e-03]
Layer 'conv1' biases: 6.179797e-03 [1.451384e-05]
Layer 'conv2' weights[0]: 2.133178e-02 [6.337569e-05] [2.970952e-03]
Layer 'conv2' biases: 2.470282e-02 [9.729735e-05]
Layer 'fcOut' weights[0]: 6.420852e-02 [1.101083e-04] [1.714855e-03]
Layer 'fcOut' biases: 1.128721e-01 [1.821996e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.381 sec)
3.1 (20.00%)... logprob:  0.218257, 0.060000, 0.060000 (0.261 sec)
3.2 (21.67%)... logprob:  0.221481, 0.064700, 0.064700 (0.260 sec)
3.3 (23.33%)... logprob:  0.210569, 0.059100, 0.059100 (0.269 sec)
3.4 (25.00%)... logprob:  0.210470, 0.059600, 0.059600 (0.280 sec)
3.5 (26.67%)... logprob:  0.197011, 0.057800, 0.057800 (0.276 sec)
3.6 (28.33%)... logprob:  0.163356, 0.045600, 0.045600
======================Test output======================
logprob:  0.174262, 0.049500, 0.049500
----------------------Averages-------------------------
logprob:  0.174262, 0.049500, 0.049500
-------------------------------------------------------
Layer 'conv1' weights[0]: 8.960275e-02 [4.249032e-04] [4.742078e-03]
Layer 'conv1' biases: 7.505222e-03 [1.565793e-05]
Layer 'conv2' weights[0]: 2.407989e-02 [6.371664e-05] [2.646052e-03]
Layer 'conv2' biases: 3.122083e-02 [1.076589e-04]
Layer 'fcOut' weights[0]: 7.343277e-02 [1.054850e-04] [1.436484e-03]
Layer 'fcOut' biases: 1.184271e-01 [2.027215e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.390 sec)
4.1 (30.00%)... logprob:  0.177056, 0.049900, 0.049900 (0.267 sec)
4.2 (31.67%)... logprob:  0.177898, 0.052100, 0.052100 (0.260 sec)
4.3 (33.33%)... logprob:  0.175247, 0.050700, 0.050700 (0.268 sec)
4.4 (35.00%)... logprob:  0.171205, 0.047300, 0.047300 (0.276 sec)
4.5 (36.67%)... logprob:  0.165775, 0.048300, 0.048300 (0.277 sec)
4.6 (38.33%)... logprob:  0.138243, 0.037900, 0.037900
======================Test output======================
logprob:  0.145980, 0.043900, 0.043900
----------------------Averages-------------------------
logprob:  0.145980, 0.043900, 0.043900
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.005846e-01 [2.952142e-04] [2.934983e-03]
Layer 'conv1' biases: 8.529577e-03 [8.077021e-06]
Layer 'conv2' weights[0]: 2.600699e-02 [4.605853e-05] [1.771006e-03]
Layer 'conv2' biases: 3.634325e-02 [5.782335e-05]
Layer 'fcOut' weights[0]: 8.008937e-02 [8.605960e-05] [1.074545e-03]
Layer 'fcOut' biases: 1.239006e-01 [1.301457e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.394 sec)
5.1 (40.00%)... logprob:  0.154051, 0.043600, 0.043600 (0.265 sec)
5.2 (41.67%)... logprob:  0.152657, 0.042800, 0.042800 (0.260 sec)
5.3 (43.33%)... logprob:  0.157166, 0.045600, 0.045600 (0.269 sec)
5.4 (45.00%)... logprob:  0.156089, 0.043000, 0.043000 (0.275 sec)
5.5 (46.67%)... logprob:  0.151101, 0.044100, 0.044100 (0.279 sec)
5.6 (48.33%)... logprob:  0.127177, 0.034000, 0.034000
======================Test output======================
logprob:  0.134478, 0.040300, 0.040300
----------------------Averages-------------------------
logprob:  0.134478, 0.040300, 0.040300
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.092488e-01 [3.290545e-04] [3.011972e-03]
Layer 'conv1' biases: 9.419627e-03 [9.917402e-06]
Layer 'conv2' weights[0]: 2.765511e-02 [5.169003e-05] [1.869095e-03]
Layer 'conv2' biases: 4.023369e-02 [7.040862e-05]
Layer 'fcOut' weights[0]: 8.545814e-02 [8.551481e-05] [1.000663e-03]
Layer 'fcOut' biases: 1.289145e-01 [1.510616e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.390 sec)
6.1 (50.00%)... logprob:  0.138790, 0.039100, 0.039100 (0.256 sec)
6.2 (51.67%)... logprob:  0.136038, 0.040000, 0.040000 (0.267 sec)
6.3 (53.33%)... logprob:  0.140736, 0.043300, 0.043300 (0.271 sec)
6.4 (55.00%)... logprob:  0.140494, 0.040500, 0.040500 (0.257 sec)
6.5 (56.67%)... logprob:  0.135578, 0.042900, 0.042900 (0.260 sec)
6.6 (58.33%)... logprob:  0.117202, 0.034200, 0.034200
======================Test output======================
logprob:  0.125607, 0.037000, 0.037000
----------------------Averages-------------------------
logprob:  0.125607, 0.037000, 0.037000
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.185013e-01 [3.215680e-04] [2.713624e-03]
Layer 'conv1' biases: 1.045737e-02 [8.683801e-06]
Layer 'conv2' weights[0]: 2.906979e-02 [5.124793e-05] [1.762928e-03]
Layer 'conv2' biases: 4.338235e-02 [6.073256e-05]
Layer 'fcOut' weights[0]: 8.998768e-02 [8.469566e-05] [9.411917e-04]
Layer 'fcOut' biases: 1.330563e-01 [1.375708e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.389 sec)
7.1 (60.00%)... logprob:  0.127772, 0.036500, 0.036500 (0.264 sec)
7.2 (61.67%)... logprob:  0.129048, 0.035300, 0.035300 (0.264 sec)
7.3 (63.33%)... logprob:  0.126389, 0.036900, 0.036900 (0.257 sec)
7.4 (65.00%)... logprob:  0.126924, 0.036500, 0.036500 (0.258 sec)
7.5 (66.67%)... logprob:  0.124740, 0.039200, 0.039200 (0.262 sec)
7.6 (68.33%)... logprob:  0.107535, 0.031300, 0.031300
======================Test output======================
logprob:  0.117275, 0.036500, 0.036500
----------------------Averages-------------------------
logprob:  0.117275, 0.036500, 0.036500
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.253645e-01 [3.182444e-04] [2.538553e-03]
Layer 'conv1' biases: 1.130659e-02 [9.655061e-06]
Layer 'conv2' weights[0]: 3.027309e-02 [5.097299e-05] [1.683772e-03]
Layer 'conv2' biases: 4.638661e-02 [6.874600e-05]
Layer 'fcOut' weights[0]: 9.390774e-02 [8.618318e-05] [9.177431e-04]
Layer 'fcOut' biases: 1.369651e-01 [1.598123e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.395 sec)
8.1 (70.00%)... logprob:  0.118632, 0.034100, 0.034100 (0.279 sec)
8.2 (71.67%)... logprob:  0.116539, 0.033800, 0.033800 (0.284 sec)
8.3 (73.33%)... logprob:  0.119332, 0.033300, 0.033300 (0.263 sec)
8.4 (75.00%)... logprob:  0.117164, 0.035000, 0.035000 (0.259 sec)
8.5 (76.67%)... logprob:  0.115454, 0.034100, 0.034100 (0.265 sec)
8.6 (78.33%)... logprob:  0.098626, 0.027800, 0.027800
======================Test output======================
logprob:  0.111595, 0.035600, 0.035600
----------------------Averages-------------------------
logprob:  0.111595, 0.035600, 0.035600
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.320663e-01 [2.742416e-04] [2.076545e-03]
Layer 'conv1' biases: 1.205624e-02 [9.303420e-06]
Layer 'conv2' weights[0]: 3.131152e-02 [4.480714e-05] [1.431011e-03]
Layer 'conv2' biases: 4.911382e-02 [6.194661e-05]
Layer 'fcOut' weights[0]: 9.736039e-02 [8.148113e-05] [8.369022e-04]
Layer 'fcOut' biases: 1.395094e-01 [1.539289e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.375 sec)
9.1 (80.00%)... logprob:  0.109055, 0.030600, 0.030600 (0.263 sec)
9.2 (81.67%)... logprob:  0.108679, 0.031500, 0.031500 (0.276 sec)
9.3 (83.33%)... logprob:  0.114341, 0.034500, 0.034500 (0.270 sec)
9.4 (85.00%)... logprob:  0.113470, 0.034200, 0.034200 (0.264 sec)
9.5 (86.67%)... logprob:  0.113821, 0.034900, 0.034900 (0.260 sec)
9.6 (88.33%)... logprob:  0.097121, 0.027200, 0.027200
======================Test output======================
logprob:  0.109638, 0.036300, 0.036300
----------------------Averages-------------------------
logprob:  0.109638, 0.036300, 0.036300
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.373190e-01 [3.058636e-04] [2.227395e-03]
Layer 'conv1' biases: 1.302156e-02 [9.149435e-06]
Layer 'conv2' weights[0]: 3.226066e-02 [4.938124e-05] [1.530695e-03]
Layer 'conv2' biases: 5.098233e-02 [6.026183e-05]
Layer 'fcOut' weights[0]: 1.004117e-01 [8.105130e-05] [8.071900e-04]
Layer 'fcOut' biases: 1.419481e-01 [1.483235e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.374 sec)
10.1 (90.00%)... logprob:  0.109652, 0.031600, 0.031600 (0.265 sec)
10.2 (91.67%)... logprob:  0.106537, 0.031300, 0.031300 (0.274 sec)
10.3 (93.33%)... logprob:  0.108742, 0.031500, 0.031500 (0.277 sec)
10.4 (95.00%)... logprob:  0.107402, 0.031800, 0.031800 (0.281 sec)
10.5 (96.67%)... logprob:  0.108577, 0.033200, 0.033200 (0.265 sec)
10.6 (98.33%)... logprob:  0.089522, 0.025200, 0.025200
======================Test output======================
logprob:  0.103471, 0.032400, 0.032400
----------------------Averages-------------------------
logprob:  0.103471, 0.032400, 0.032400
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.426484e-01 [3.130984e-04] [2.194896e-03]
Layer 'conv1' biases: 1.376614e-02 [7.678883e-06]
Layer 'conv2' weights[0]: 3.318742e-02 [4.514968e-05] [1.360446e-03]
Layer 'conv2' biases: 5.307174e-02 [5.835572e-05]
Layer 'fcOut' weights[0]: 1.032455e-01 [6.837120e-05] [6.622197e-04]
Layer 'fcOut' biases: 1.431666e-01 [1.101904e-04]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/MNIST/ConvNet__2014-09-11_23.26.06
======================================================= (0.386 sec)
11.1 (100.00%)... logprob:  0.102062, 0.029500, 0.029500 (0.265 sec)

(24) cuda-convnet2が動いた!

(22) cuda-convnet2はやってみれない (T_T) では、買ったばかりのGTX760のスペック不足で cuda-convnet2が動かせないことが判明し、とってもがっかり…

当時は高額で購入をあきらめていたGTX780だが、最近は値下がりが進んでいることもあり、思い切って購入!
早速 cuda-convnet2にリベンジしてみた。

結果

cuda-convnet2が動いた!

cuda-convnet2に付属の CIFAR-10を 10epochs指定で動かしてみた。
10epochs 68.5% の正解率だった。

=========================
Running on CUDA device(s) 0
Current time: Thu Sep 11 22:46:45 2014
Saving checkpoints to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
=========================
1.1 (0.00%)... logprob:  2.120814, 0.784000, 0.784000 (0.887 sec)
1.2 (2.00%)... logprob:  1.835924, 0.675100, 0.675100 (0.836 sec)
1.3 (4.00%)... logprob:  1.682182, 0.615700, 0.615700 (0.836 sec)
1.4 (6.00%)... logprob:  1.637781, 0.596200, 0.596200 (0.837 sec)
1.5 (8.00%)... logprob:  1.574231, 0.575300, 0.575300
======================Test output======================
logprob:  1.446012, 0.519900, 0.519900
----------------------Averages-------------------------
logprob:  1.446012, 0.519900, 0.519900
-------------------------------------------------------
Layer 'conv1' weights[0]: 7.474001e-03 [5.467787e-05] [7.315743e-03]
Layer 'conv1' biases: 1.841275e-03 [6.501494e-06]
Layer 'conv2' weights[0]: 8.184642e-03 [1.752141e-05] [2.140767e-03]
Layer 'conv2' biases: 2.113849e-03 [7.906822e-06]
Layer 'local3' weights[0]: 3.137425e-02 [2.088059e-06] [6.655327e-05]
Layer 'local3' biases: 2.683400e-04 [9.420025e-07]
Layer 'local4' weights[0]: 3.142803e-02 [2.171371e-06] [6.909025e-05]
Layer 'local4' biases: 6.031835e-04 [1.956280e-06]
Layer 'fc10' weights[0]: 9.292197e-03 [3.430814e-05] [3.692145e-03]
Layer 'fc10' biases: 3.025318e-02 [7.367652e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.218 sec)
2.1 (10.00%)... logprob:  1.506601, 0.543700, 0.543700 (0.843 sec)
2.2 (12.00%)... logprob:  1.475231, 0.522700, 0.522700 (0.842 sec)
2.3 (14.00%)... logprob:  1.398977, 0.501300, 0.501300 (0.843 sec)
2.4 (16.00%)... logprob:  1.388852, 0.493800, 0.493800 (0.840 sec)
2.5 (18.00%)... logprob:  1.347464, 0.480800, 0.480800
======================Test output======================
logprob:  1.258990, 0.445000, 0.445000
----------------------Averages-------------------------
logprob:  1.258990, 0.445000, 0.445000
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.066526e-02 [7.180368e-05] [6.732486e-03]
Layer 'conv1' biases: 3.529629e-03 [4.447127e-06]
Layer 'conv2' weights[0]: 8.477319e-03 [2.198427e-05] [2.593305e-03]
Layer 'conv2' biases: 3.669117e-03 [5.606374e-06]
Layer 'local3' weights[0]: 3.088271e-02 [2.739488e-06] [8.870618e-05]
Layer 'local3' biases: 4.430772e-04 [9.733770e-07]
Layer 'local4' weights[0]: 3.093651e-02 [2.908529e-06] [9.401605e-05]
Layer 'local4' biases: 1.001289e-03 [2.039535e-06]
Layer 'fc10' weights[0]: 1.015366e-02 [4.788852e-05] [4.716380e-03]
Layer 'fc10' biases: 4.879263e-02 [9.227377e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.204 sec)
3.1 (20.00%)... logprob:  1.320204, 0.466300, 0.466300 (0.849 sec)
3.2 (22.00%)... logprob:  1.322136, 0.469200, 0.469200 (0.849 sec)
3.3 (24.00%)... logprob:  1.254244, 0.445500, 0.445500 (0.844 sec)
3.4 (26.00%)... logprob:  1.265054, 0.444200, 0.444200 (0.852 sec)
3.5 (28.00%)... logprob:  1.239677, 0.438100, 0.438100
======================Test output======================
logprob:  1.172489, 0.419600, 0.419600
----------------------Averages-------------------------
logprob:  1.172489, 0.419600, 0.419600
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.303541e-02 [7.437402e-05] [5.705536e-03]
Layer 'conv1' biases: 4.526725e-03 [5.591116e-06]
Layer 'conv2' weights[0]: 8.756692e-03 [2.101911e-05] [2.400348e-03]
Layer 'conv2' biases: 4.467094e-03 [5.962670e-06]
Layer 'local3' weights[0]: 3.039985e-02 [2.865414e-06] [9.425750e-05]
Layer 'local3' biases: 5.414516e-04 [1.031628e-06]
Layer 'local4' weights[0]: 3.045136e-02 [3.177895e-06] [1.043597e-04]
Layer 'local4' biases: 1.232660e-03 [2.390751e-06]
Layer 'fc10' weights[0]: 1.063668e-02 [4.992195e-05] [4.693376e-03]
Layer 'fc10' biases: 5.920979e-02 [9.777919e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.224 sec)
4.1 (30.00%)... logprob:  1.194791, 0.418200, 0.418200 (0.852 sec)
4.2 (32.00%)... logprob:  1.224260, 0.438500, 0.438500 (0.851 sec)
4.3 (34.00%)... logprob:  1.163682, 0.408000, 0.408000 (0.852 sec)
4.4 (36.00%)... logprob:  1.175036, 0.417000, 0.417000 (0.852 sec)
4.5 (38.00%)... logprob:  1.155333, 0.406400, 0.406400
======================Test output======================
logprob:  1.062875, 0.378300, 0.378300
----------------------Averages-------------------------
logprob:  1.062875, 0.378300, 0.378300
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.509653e-02 [6.724540e-05] [4.454362e-03]
Layer 'conv1' biases: 5.414162e-03 [6.911395e-06]
Layer 'conv2' weights[0]: 9.012531e-03 [2.557474e-05] [2.837686e-03]
Layer 'conv2' biases: 5.005291e-03 [6.856871e-06]
Layer 'local3' weights[0]: 2.992433e-02 [3.229656e-06] [1.079274e-04]
Layer 'local3' biases: 6.130654e-04 [1.145043e-06]
Layer 'local4' weights[0]: 2.997499e-02 [3.427017e-06] [1.143292e-04]
Layer 'local4' biases: 1.398424e-03 [2.556241e-06]
Layer 'fc10' weights[0]: 1.108736e-02 [4.734496e-05] [4.270172e-03]
Layer 'fc10' biases: 6.623243e-02 [9.970758e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.210 sec)
5.1 (40.00%)... logprob:  1.112436, 0.392300, 0.392300 (0.854 sec)
5.2 (42.00%)... logprob:  1.132824, 0.396600, 0.396600 (0.853 sec)
5.3 (44.00%)... logprob:  1.092543, 0.382400, 0.382400 (0.854 sec)
5.4 (46.00%)... logprob:  1.111692, 0.393300, 0.393300 (0.854 sec)
5.5 (48.00%)... logprob:  1.080376, 0.383600, 0.383600
======================Test output======================
logprob:  1.014830, 0.362300, 0.362300
----------------------Averages-------------------------
logprob:  1.014830, 0.362300, 0.362300
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.681544e-02 [5.656535e-05] [3.363894e-03]
Layer 'conv1' biases: 6.239711e-03 [6.276071e-06]
Layer 'conv2' weights[0]: 9.254504e-03 [2.295077e-05] [2.479956e-03]
Layer 'conv2' biases: 5.374766e-03 [5.774366e-06]
Layer 'local3' weights[0]: 2.945584e-02 [3.233688e-06] [1.097809e-04]
Layer 'local3' biases: 6.639807e-04 [9.701089e-07]
Layer 'local4' weights[0]: 2.950781e-02 [3.495634e-06] [1.184647e-04]
Layer 'local4' biases: 1.516043e-03 [2.256748e-06]
Layer 'fc10' weights[0]: 1.149275e-02 [4.630167e-05] [4.028772e-03]
Layer 'fc10' biases: 7.100429e-02 [9.157424e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.213 sec)
6.1 (50.00%)... logprob:  1.060685, 0.376300, 0.376300 (0.850 sec)
6.2 (52.00%)... logprob:  1.069578, 0.375200, 0.375200 (0.853 sec)
6.3 (54.00%)... logprob:  1.032226, 0.359800, 0.359800 (0.851 sec)
6.4 (56.00%)... logprob:  1.064357, 0.373500, 0.373500 (0.854 sec)
6.5 (58.00%)... logprob:  1.045632, 0.365300, 0.365300
======================Test output======================
logprob:  0.934916, 0.329700, 0.329700
----------------------Averages-------------------------
logprob:  0.934916, 0.329700, 0.329700
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.835800e-02 [7.618176e-05] [4.149785e-03]
Layer 'conv1' biases: 7.070946e-03 [4.932727e-06]
Layer 'conv2' weights[0]: 9.482100e-03 [2.152068e-05] [2.269611e-03]
Layer 'conv2' biases: 5.701233e-03 [4.869142e-06]
Layer 'local3' weights[0]: 2.899688e-02 [3.212075e-06] [1.107731e-04]
Layer 'local3' biases: 7.021243e-04 [9.119989e-07]
Layer 'local4' weights[0]: 2.904793e-02 [3.381710e-06] [1.164183e-04]
Layer 'local4' biases: 1.607356e-03 [2.076248e-06]
Layer 'fc10' weights[0]: 1.183851e-02 [3.967304e-05] [3.351186e-03]
Layer 'fc10' biases: 7.488154e-02 [7.453511e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.209 sec)
7.1 (60.00%)... logprob:  0.999895, 0.350200, 0.350200 (0.855 sec)
7.2 (62.00%)... logprob:  1.039784, 0.365700, 0.365700 (0.856 sec)
7.3 (64.00%)... logprob:  0.990986, 0.348300, 0.348300 (0.851 sec)
7.4 (66.00%)... logprob:  0.995326, 0.342200, 0.342200 (0.854 sec)
7.5 (68.00%)... logprob:  0.969341, 0.339700, 0.339700
======================Test output======================
logprob:  1.005710, 0.351300, 0.351300
----------------------Averages-------------------------
logprob:  1.005710, 0.351300, 0.351300
-------------------------------------------------------
Layer 'conv1' weights[0]: 1.968702e-02 [7.124672e-05] [3.618968e-03]
Layer 'conv1' biases: 7.793765e-03 [5.323519e-06]
Layer 'conv2' weights[0]: 9.687451e-03 [2.075292e-05] [2.142248e-03]
Layer 'conv2' biases: 5.936751e-03 [4.238163e-06]
Layer 'local3' weights[0]: 2.854378e-02 [3.352751e-06] [1.174600e-04]
Layer 'local3' biases: 7.324723e-04 [9.184912e-07]
Layer 'local4' weights[0]: 2.859599e-02 [3.617547e-06] [1.265054e-04]
Layer 'local4' biases: 1.686072e-03 [2.135282e-06]
Layer 'fc10' weights[0]: 1.215715e-02 [4.211784e-05] [3.464451e-03]
Layer 'fc10' biases: 7.772458e-02 [7.265426e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.214 sec)
8.1 (70.00%)... logprob:  0.970245, 0.340100, 0.340100 (0.858 sec)
8.2 (72.00%)... logprob:  0.988053, 0.348600, 0.348600 (0.853 sec)
8.3 (74.00%)... logprob:  0.949711, 0.330300, 0.330300 (0.861 sec)
8.4 (76.00%)... logprob:  0.964428, 0.331700, 0.331700 (0.853 sec)
8.5 (78.00%)... logprob:  0.951977, 0.332200, 0.332200
======================Test output======================
logprob:  0.902606, 0.315900, 0.315900
----------------------Averages-------------------------
logprob:  0.902606, 0.315900, 0.315900
-------------------------------------------------------
Layer 'conv1' weights[0]: 2.086830e-02 [7.139416e-05] [3.421177e-03]
Layer 'conv1' biases: 8.485496e-03 [4.683458e-06]
Layer 'conv2' weights[0]: 9.879518e-03 [1.769133e-05] [1.790708e-03]
Layer 'conv2' biases: 6.132266e-03 [3.429028e-06]
Layer 'local3' weights[0]: 2.809841e-02 [3.082417e-06] [1.097008e-04]
Layer 'local3' biases: 7.562819e-04 [7.674955e-07]
Layer 'local4' weights[0]: 2.815062e-02 [3.344105e-06] [1.187933e-04]
Layer 'local4' biases: 1.745867e-03 [1.828604e-06]
Layer 'fc10' weights[0]: 1.240342e-02 [3.713923e-05] [2.994273e-03]
Layer 'fc10' biases: 8.005437e-02 [5.533533e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.215 sec)
9.1 (80.00%)... logprob:  0.941026, 0.327300, 0.327300 (0.852 sec)
9.2 (82.00%)... logprob:  0.953385, 0.335800, 0.335800 (0.856 sec)
9.3 (84.00%)... logprob:  0.931941, 0.324100, 0.324100 (0.853 sec)
9.4 (86.00%)... logprob:  0.933768, 0.324900, 0.324900 (0.856 sec)
9.5 (88.00%)... logprob:  0.928577, 0.320800, 0.320800
======================Test output======================
logprob:  0.883051, 0.309200, 0.309200
----------------------Averages-------------------------
logprob:  0.883051, 0.309200, 0.309200
-------------------------------------------------------
Layer 'conv1' weights[0]: 2.197285e-02 [6.753075e-05] [3.073373e-03]
Layer 'conv1' biases: 9.153372e-03 [4.402948e-06]
Layer 'conv2' weights[0]: 1.006375e-02 [1.881227e-05] [1.869310e-03]
Layer 'conv2' biases: 6.313587e-03 [3.937244e-06]
Layer 'local3' weights[0]: 2.766130e-02 [3.224194e-06] [1.165598e-04]
Layer 'local3' biases: 7.823407e-04 [7.771447e-07]
Layer 'local4' weights[0]: 2.771336e-02 [3.435646e-06] [1.239707e-04]
Layer 'local4' biases: 1.799375e-03 [1.811628e-06]
Layer 'fc10' weights[0]: 1.262947e-02 [3.785512e-05] [2.997363e-03]
Layer 'fc10' biases: 8.223019e-02 [5.913675e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.217 sec)
10.1 (90.00%)... logprob:  0.899000, 0.315300, 0.315300 (0.853 sec)
10.2 (92.00%)... logprob:  0.920165, 0.318600, 0.318600 (0.856 sec)
10.3 (94.00%)... logprob:  0.907986, 0.316500, 0.316500 (0.854 sec)
10.4 (96.00%)... logprob:  0.912464, 0.320900, 0.320900 (0.855 sec)
10.5 (98.00%)... logprob:  0.886616, 0.314900, 0.314900
======================Test output======================
logprob:  0.887153, 0.315100, 0.315100
----------------------Averages-------------------------
logprob:  0.887153, 0.315100, 0.315100
-------------------------------------------------------
Layer 'conv1' weights[0]: 2.302468e-02 [6.561136e-05] [2.849610e-03]
Layer 'conv1' biases: 9.744839e-03 [4.328744e-06]
Layer 'conv2' weights[0]: 1.024172e-02 [1.659464e-05] [1.620298e-03]
Layer 'conv2' biases: 6.463769e-03 [3.200243e-06]
Layer 'local3' weights[0]: 2.723007e-02 [3.112101e-06] [1.142891e-04]
Layer 'local3' biases: 8.001459e-04 [6.947578e-07]
Layer 'local4' weights[0]: 2.728442e-02 [3.306802e-06] [1.211974e-04]
Layer 'local4' biases: 1.843459e-03 [1.568608e-06]
Layer 'fc10' weights[0]: 1.286868e-02 [3.347505e-05] [2.601281e-03]
Layer 'fc10' biases: 8.373742e-02 [2.511169e-05]
-------------------------------------------------------
Saved checkpoint to /home/user/cuda/cuda-convnet2/save/cifar-10/ConvNet__2014-09-11_22.46.42
======================================================= (1.207 sec)
11.1 (100.00%)... logprob:  0.880133, 0.312300, 0.312300 (0.855 sec)

(23) GTX760からGTX780に変えてみる

(22) cuda-convnet2はやってみれない (T_T) で残念な思いをしたのは20日ほど前のこと。
ここ最近で1万円近く値段が下がってきたGTX780を4万円少々で衝動買いした。

27cmもある長いボードは窮屈な筐体内で電気ストーブのように高熱を発しており…
しばらくはこまめに温度を確認しながら使うことになりそう。20140911_01

中身をのぞいてみる

deviceQueryの結果は以下の通り。
Compute capability 3.5 がうれしい!
GTX780のcore数は2304個、GTX650 384個の6倍、GTX760 1152個の2倍になった!

Device 0: "GeForce GTX 780"
  CUDA Driver Version / Runtime Version          6.0 / 6.0
  CUDA Capability Major/Minor version number:    3.5
  Total amount of global memory:                 3071 MBytes (3220504576 bytes)
  (12) Multiprocessors, (192) CUDA Cores/MP:     2304 CUDA Cores
  GPU Clock rate:                                1058 MHz (1.06 GHz)
  Memory Clock rate:                             3104 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                 1572864 bytes

cuda-convnetで速度計測

(21) GTX650からGTX760に変えてみる のときと同じく、ちょっとリッチなMNIST学習用ネットワーク100epochsで計測した。

結果1: 最新環境(intel Xeon x5570 x 2, GTX780)
100epochsの処理時間は 100秒 、エラー率は 0.010だった。

START: 2014年  9月 11日 木曜日 20:57:46 JST
======================Test output======================
logprob:  0.034160, 0.010400
END:   2014年  9月 11日 木曜日 20:59:26 JST

結果2: 直前環境(intel Xeon x5570 x 2, GTX760)
100epochsの処理時間は 157秒 、エラー率は 0.010だった。

START: 2014年  8月 18日 月曜日 15:24:10 JST
======================Test output======================
logprob:  0.034883, 0.010300
END:   2014年  8月 18日 月曜日 15:26:47 JST

結果3: 旧マシン(intel core-i3 3220, GTX650)
100epochsの処理時間は 398秒 、エラー率は 0.011だった。

START: 2014年  8月 18日 月曜日 15:23:48 JST
======================Test output======================
logprob:  0.034047, 0.010900
END:   2014年  8月 18日 月曜日 15:30:26 JST