利用caffe生成 lmdb 格式的文件，并对网络进行FineTuning
数据的组织格式为：
首先，所需要的脚本指令路径为：
/home/wangxiao/Downloads/caffe-master/examples/imagenet/
其中，生成lmdb的文件为： create_imagenet.sh
接下来的主要任务就是修改自己的data的存放路径了。
#!/usr/bin/env sh
# Create the imagenet lmdb inputs
# N.B. set the path to the imagenet train + val data dirs EXAMPLE=../wangxiao
DATA=../fine_tuning_data
TOOLS=../build/tools TRAIN_DATA_ROOT=../fine_tuning_data/training/data/
VAL_DATA_ROOT=../fine_tuning_data/validation/data/ #TRAIN_DATA_ROOT=/media/yukai/247317a3-e6b5-45d4-81d1-/---------------/Attribute reconginition/final_PETA_dataset/whole_benchmark/用于微调网络的数据/training/data/
#VAL_DATA_ROOT=/media/yukai/247317a3-e6b5-45d4-81d1-/---------------/Attribute reconginition/final_PETA_dataset/whole_benchmark/用于微调网络的数据/validation/data/ # Set RESIZE=true to resize the images to 256x256. Leave as false if images have
# already been resized using another tool. # RESIZE=false default parameter and wangxiao modify it in 2015.10. : RESIZE=true
if $RESIZE; then
RESIZE_HEIGHT=
RESIZE_WIDTH=
else
RESIZE_HEIGHT=
RESIZE_WIDTH=
fi if [ ! -d "$TRAIN_DATA_ROOT" ]; then
echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \
"where the ImageNet training data is stored."
exit
fi if [ ! -d "$VAL_DATA_ROOT" ]; then
echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \
"where the ImageNet validation data is stored."
exit
fi echo "Creating train lmdb..." GLOG_logtostderr= $TOOLS/convert_imageset \
--resize_height=$RESIZE_HEIGHT \
--resize_width=$RESIZE_WIDTH \
--shuffle \
$TRAIN_DATA_ROOT \
$DATA/training/final_train_data.txt \
$EXAMPLE/PETA_train_lmdb #echo "Creating val lmdb..." #GLOG_logtostderr= $TOOLS/convert_imageset \
# --resize_height=$RESIZE_HEIGHT \
# --resize_width=$RESIZE_WIDTH \
# --shuffle \
# $VAL_DATA_ROOT \
# $DATA/validation/final_test_data.txt \
# $EXAMPLE/PETA_val_lmdb echo "Done."
都修改完成后，在终端执行：create_imagenet.sh，然后会有如此的提示，表示正在生成lmdb文件：
然后完成后，提示： Done. 然后可以看到已经生成了所需要的文件：
然后利用 make_imagenet_mean.sh 生成所需要的 mean file :
caffe-master$: sh ./make_imagenet_mean.sh
#!/usr/bin/env sh
# Compute the mean image from the imagenet training lmdb
# N.B. this is available in data/ilsvrc12 EXAMPLE=../wangxiao
DATA=./data
TOOLS=../build/tools #echo $TOOLS/compute_image_mean $EXAMPLE/HAT_train_lmdb \
# $DATA/HAT_mean.binaryproto
$TOOLS/compute_image_mean $EXAMPLE/HAT_train_lmdb \
$DATA/HAT_mean.binaryproto
echo "Done."
然后就生成了 HAT_mean.binaryproto
其中，caffe路径下：/home/wangxiao/Downloads/caffe-master/examples/imagenet/readme.md 对这个过程有一个详细的解释。
然后就是查看 solver.prototxt:
net: "models/bvlc_reference_caffenet/train_val.prototxt"
test_iter:
test_interval:
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
stepsize:
display:
max_iter:
momentum: 0.9
weight_decay: 0.0005
snapshot:
snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"
solver_mode: GPU
打开 models/bvlc_reference_caffenet/train_val.prototxt
需要修改的也就到第55行：
name: "CaffeNet"
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: true
crop_size:
mean_file: "wangxiao/HAT_data/HAT_mean.binaryproto"
}
# mean pixel / channel-wise mean instead of mean image
# transform_param {
# crop_size:
# mean_value:
# mean_value:
# mean_value:
# mirror: true
# }
data_param {
source: "wangxiao/HAT_data/HAT_train_lmdb"
batch_size:
backend: LMDB
}
}
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
mirror: false
crop_size:
mean_file: "wangxiao/HAT_data/HAT_mean.binaryproto"
}
# mean pixel / channel-wise mean instead of mean image
# transform_param {
# crop_size:
# mean_value:
# mean_value:
# mean_value:
# mirror: true
# }
data_param {
source: "wangxiao/HAT_data/HAT_val_lmdb"
batch_size:
backend: LMDB
}
}
然后执行：
终端会有显示：
I0103 ::21.027832 net.cpp:] Network initialization done.
I0103 ::21.027839 net.cpp:] Memory required for data:
I0103 ::21.027928 solver.cpp:] Solver scaffolding done.
I0103 ::21.028312 caffe.cpp:] Starting Optimization
I0103 ::21.028326 solver.cpp:] Solving CaffeNet
I0103 ::21.028333 solver.cpp:] Learning Rate Policy: step
I0103 ::22.012593 solver.cpp:] Iteration , loss = 7.52783
I0103 ::22.012660 solver.cpp:] Train net output #: loss = 7.52783 (* = 7.52783 loss)
I0103 ::22.012687 solver.cpp:] Iteration , lr = 0.01
I0103 ::41.812361 solver.cpp:] Iteration , loss = 3.9723
I0103 ::41.812413 solver.cpp:] Train net output #: loss = 3.9723 (* = 3.9723 loss)
I0103 ::41.812428 solver.cpp:] Iteration , lr = 0.01
I0103 ::01.553021 solver.cpp:] Iteration , loss = 2.9715
I0103 ::01.553104 solver.cpp:] Train net output #: loss = 2.9715 (* = 2.9715 loss)
I0103 ::01.553119 solver.cpp:] Iteration , lr = 0.01
I0103 ::21.574745 solver.cpp:] Iteration , loss = 2.91547
I0103 ::21.574798 solver.cpp:] Train net output #: loss = 2.91547 (* = 2.91547 loss)
I0103 ::21.574811 solver.cpp:] Iteration , lr = 0.01
I0103 ::41.247493 solver.cpp:] Iteration , loss = 2.96451
I0103 ::41.247627 solver.cpp:] Train net output #: loss = 2.96451 (* = 2.96451 loss)
I0103 ::41.247642 solver.cpp:] Iteration , lr = 0.01
I0103 ::00.941267 solver.cpp:] Iteration , loss = 2.85887
I0103 ::00.941318 solver.cpp:] Train net output #: loss = 2.85887 (* = 2.85887 loss)
I0103 ::00.941332 solver.cpp:] Iteration , lr = 0.01
I0103 ::20.628329 solver.cpp:] Iteration , loss = 2.91318
I0103 ::20.628463 solver.cpp:] Train net output #: loss = 2.91318 (* = 2.91318 loss)
I0103 ::20.628476 solver.cpp:] Iteration , lr = 0.01
I0103 ::40.621937 solver.cpp:] Iteration , loss = 3.06499
I0103 ::40.621989 solver.cpp:] Train net output #: loss = 3.06499 (* = 3.06499 loss)
I0103 ::40.622004 solver.cpp:] Iteration , lr = 0.01
I0103 ::00.557921 solver.cpp:] Iteration , loss = 2.9818
I0103 ::00.558048 solver.cpp:] Train net output #: loss = 2.9818 (* = 2.9818 loss)
I0103 ::00.558063 solver.cpp:] Iteration , lr = 0.01
因为设置的迭代次数为： 450000次，所以，接下来就是睡觉了。。。O(∩_∩)O~ 感谢木得兄刚刚的帮助。
------ 未完待续------
另外就是，当loss 后期变化不大的时候，可以试着调整学习率， 在Solver.prototext中：
train_net: "models/bvlc_reference_caffenet/train_val.prototxt"
# test_iter:
# test_interval:
base_lr: 0.0001
lr_policy: "step"
gamma: 0.1
stepsize:
display:
max_iter:
momentum: 0.9
weight_decay: 0.0005
snapshot:
snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"
solver_mode: GPU
base_lr: 0.0001 每次可以改为0.1×base_lr， 这里的 0.0001 是我两次调整之后的数值。
然后运行 resume_training.sh
#!/usr/bin/env sh ./build/tools/caffe train \
--solver=models/bvlc_reference_caffenet/solver.prototxt \
--snapshot=models/bvlc_reference_caffenet/caffenet_train_iter_88251.solverstate
将snapshot改为之前中断时的结果即可，即： caffenet_train_iter_88251.solverstate
继续看loss是否降低。。。
--------------------------------- 未完待续 ---------------------------------