在caffe中用python添加confusion matrix层

confusion matrix(混淆矩阵)在分类问题中能比单纯的准确率更全面地反应模型的好坏，本文主要目的是在caffe中用python添加confusion matrix层。

Step1：准备confusion matrix的python实现代码

完整代码如下：

import caffe
import json
import numpy as np
import sys

import sklearn.metrics

class PythonConfMat(caffe.Layer):
    """ Compute the Accuracy with a Python Layer """

    def setup(self, bottom, top):
        # check input pair
        if len(bottom) != 2:
            raise Exception("Need two inputs.")

        self.num_labels = bottom[0].channels
        params = json.loads(self.param_str)
        self.test_iter = params['test_iter']
        self.conf_matrix = np.zeros((self.num_labels, self.num_labels))
        self.current_iter = 0

    def reshape(self, bottom, top):
        # bottom[0] are the net's outputs
        # bottom[1] are the ground truth labels

        # Net outputs and labels must have the same number of elements
        if bottom[0].num != bottom[1].num:
            raise Exception("Inputs must have the same number of elements.")

        # accuracy output is scalar
        top[0].reshape(1)

    def forward(self, bottom, top):
        self.current_iter += 1

        # predicted outputs
        pred = np.argmax(bottom[0].data, axis=1)
        accuracy = np.sum(pred == bottom[1].data).astype(np.float32) / bottom[0].num
        top[0].data[...] = accuracy

        # compute confusion matrix
        self.conf_matrix += sklearn.metrics.confusion_matrix(bottom[1].data, pred, labels=range(self.num_labels))

        if self.current_iter == self.test_iter:
            self.current_iter = 0
            sys.stdout.write('\nCAUTION!! test_iter = %i. Make sure this is the correct value' % self.test_iter)
            sys.stdout.write('\n"param_str: \'{"test_iter":%i}\'" has been set in the definition of the PythonLayer' % self.test_iter)
            sys.stdout.write('\n\nConfusion Matrix')
            sys.stdout.write('\t'*(self.num_labels-2)+'| Accuracy')
            sys.stdout.write('\n'+'-'*8*(self.num_labels+1))
            sys.stdout.write('\n')
            for i in range(len(self.conf_matrix)):
                for j in range(len(self.conf_matrix[i])):
                    sys.stdout.write(str(self.conf_matrix[i][j].astype(np.int))+'\t')
                sys.stdout.write('| %3.2f %%' % (self.conf_matrix[i][i]*100 / self.conf_matrix[i].sum()))
                sys.stdout.write('\n')
            sys.stdout.write('Number of test samples: %i \n\n' % self.conf_matrix.sum())
            # reset conf_matrix for next test phase
            self.conf_matrix = np.zeros((self.num_labels, self.num_labels))

    def backward(self, top, propagate_down, bottom):
        pass

注意：此python脚本应放置在$caffe_root/python目录下。

Step2：编译用python新添加的层

若caffe还没有编译，可以打开makefile.config中 WITH_PYTHON_LAYER=1的开关，然后make编译，再然后make pycaffe编译。
若caffe已经编译好了，则需要先make clean，然后用WITH_PYTHON_LAYER=1 make all -j&& make pycaffe编译caffe源码。

Step3：在prototxt中使用confusion matrix layer

在测试网络的prototxt中添加把混淆矩阵层添加进去，列子如下

layer {
  type: 'Python'
  name: 'py_accuracy'
  top: 'py_accuracy'
  bottom: 'fc101'
  bottom: 'label'
  python_param {
    # the module name -- usually the filename -- that needs to be in $PYTHONPATH
    module: 'python_confmat'
    # the layer name -- the class name in the module
    layer: 'PythonConfMat'
    # this is the number of test iterations, it must be the same as defined in the solver.
    param_str: '{"test_iter":100}'
  }
  include {
    phase: TEST
  }
}

注意：test_iter乘以batch size应cover所有测试集样本。

Step4：使用/build/tools/caffe test工具计算测试集的confusion matrix

这个需要提前把测试机生成lmdb文件。生成后用如下姿势使用

CAFFE_ROOT=/home/***/caffe                                                                         
MODEL_ROOT=/home/***/work/resnet_18_age_7                                                
$CAFFE_ROOT/build/tools/caffe test --model=$MODEL_ROOT/ResNet-18-test.prototxt \                     
       --weights ./model_lrstep6/resnet18_age_finetue_iter_43883.caffemodel \                        
       --iterations 1392 --gpu=1 

caffe test中不需要solver.prototxt，所以在命令中需要指定--iterations. 这里的iterations需要与PythonConfMat层中的test_iter相同。

注意：调用test命令可能出现import error：no module named python_confmat
解决方案：终端输入export PYTHONPATH=/home/xxx/caffe/python即可。

最后的效果如下

Reference：
http://blog.csdn.net/feynman233/article/details/71430023
http://gcucurull.github.io/caffe/python/deep-learning/2016/06/29/caffe-confusion-matrix/