mxnet(1)生成RecordIO与lst文件

时间：2023-12-12 13:46:32

(markdown是用jupypter notebook生成)

mxnet为的提高IO效率, 不会直接读取图片文件, 而是先将图片列表和标签转换为RecordIO格式的二进制文件, 训练时就可以顺序读取数据, 大大提高了IO速率.
# 如何将图片列表与标签转换为RecordIO?
mxnet直接提供了mnist与cifar数据集的recordIO格式, 但为了熟悉这个过程, 我决定自己手动来一遍: 将mnist数据的原始二进制格式转换为recordIO格式.
## 如何将mnist ubyte文件转换成image 文件与lst?
从[Yan Lecun](http://yann.lecun.com/exdb/mnist/index.html)网站上下载下来的原始数据由以下四部分组成 
!ls ../dataset/mnist
t10k-images-idx3-ubyte	train-images-idx3-ubyte

t10k-labels-idx1-ubyte	train-labels-idx1-ubyte

怎样读取训练数据?

train images和labels两个文件分别训练数据的图片与标签, 数量为50k. t10k images 和labels则是测试数据, 10k.

先写脚本读取训练数据, 生成im2rec需要的图片文件与lst文件. train-images-idx3-ubyte文件的数据格式为:

# The labels values are 0 to 9.

# TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

# [offset] [type]          [value]          [description]

# 0000     32 bit integer  0x00000803(2051) magic number

# 0004     32 bit integer  60000            number of images

# 0008     32 bit integer  28               number of rows

# 0012     32 bit integer  28               number of columns

# 0016     unsigned byte   ??               pixel

# 0017     unsigned byte   ??               pixel

# ........

# xxxx     unsigned byte   ??               pixel
import sys;

file_path = '../dataset/mnist/';

train_images_ubyte = file_path + 'train-images-idx3-ubyte';

train_labels_ubyte = file_path + 'train-labels-idx1-ubyte';

test_images_ubyte = file_path + 'test-images-idx3-ubyte';

test_labels_ubyte = file_path + 'test-labels-idx1-ubyte';

train_images = [];

train_labels = [];

def readInt(f, n = 4):

    """从mnist二进制文件中读取整数"""

    return int(f.read(n).encode('hex'), 16);

def readImage(f):

    """从mnist二进制文件中读取图片"""

    n = 28*28;

    img = [0]*n;

    for i in xrange(n):#这样一个一个字节地读取会很慢

        img[i] = readInt(f, 1);

    return img;

# 读取图片

with open(train_images_ubyte, 'r') as f:

    magic = readInt(f);

    num_img = readInt(f);

    num_rows = readInt(f);

    num_cols = readInt(f);

    for i in range(num_img):

        train_images.append(readImage(f));

train-labels-idx1-ubyte的数据格式如下:

#  TRAINING SET LABEL FILE (train-labels-idx1-ubyte):

# [offset] [type]          [value]          [description]

# 0000     32 bit integer  0x00000801(2049) magic number (MSB first)

# 0004     32 bit integer  60000            number of items

# 0008     unsigned byte   ??               label

# 0009     unsigned byte   ??               label

# ........

# xxxx     unsigned byte   ??               label

#
def readLabel(f):

    """从mnist二进制文件中读取label"""

    return readInt(f, 1);

# 读取标签

with open(train_labels_ubyte, 'r') as f:

    magic = readInt(f);

    num_items = readInt(f);

    for i in xrange(num_items):

        train_labels.append(readLabel(f));

可视化一下读取出来的数据

没什么实质性的功能, 就为了看看读出来的数据是怎样的.

import matplotlib.pyplot as plt;

import numpy as np;

%matplotlib inline

def list2img(arr):

    return np.reshape(arr, (28, 28));

# 显示几张图片及其对应对手写数字；

num = 10

for i in range(num):

    img = list2img(train_images[i]);

    plt.subplot(1,num,i+1);

    plt.imshow(img, cmap='gray')

    plt.axis('off')

    plt.title(train_labels[i])

# 画一张条形图看看数字的分布

plt.figure()

plt.hist(train_labels, range(11));

mxnet(1)生成RecordIO与lst文件

mxnet(1)生成RecordIO与lst文件

可以看出, 每个数字的分布还是挺均匀的.

导出图片, 并生成lst文件

#导出图片

import cv2

import os

file_dir = '/home/dengdan/dataset/mnist/raw/train/images/';

if not os.path.exists(file_dir) or not os.path.isdir(file_dir):

    os.makedirs(file_dir);

file_name = 'mnist_train_{0}.jpg';

n = len(train_images)

for i in xrange(n):

    img = list2img(train_images[i]);

    #cv2.imwrite(file_dir + file_name.format(i), img) #需要生成图片时将注释去掉

lst 文件用于将图片文件与它的label对应起来. 每一行为一条记录, 格式为:

integer_image_index \t label_index \t path_to_image
895099  464     n04467665_17283.JPEG

10025081        412     ILSVRC2010_val_00025082.JPEG

74181   789     n01915811_2739.JPEG

10035553        859     ILSVRC2010_val_00035554.JPEG

10048727        929     ILSVRC2010_val_00048728.JPEG

94028   924     n01980166_4956.JPEG

1080682 650     n11807979_571.JPEG

972457  633     n07723039_1627.JPEG

7534    11      n01630670_4486.JPEG

1191261 249     n12407079_5106.JPEG
# 生成lst 文件

def rec(idx, label):

    """生成第idx张图片的记录"""

    name = file_name.format(idx);

    return '{0} \t {1} \t {2}'.format(idx, label, name)

lst_path = '/home/dengdan/dataset/mnist/mxnet/'

if not os.path.exists(lst_path) or not os.path.isdir(lst_path):

    os.makedirs(lst_path);

lst = lst_path + 'train.lst';

with open(lst, 'w') as f:

    for i in xrange(len(train_labels)):

        record = rec(i, train_labels[i]);

        f.write(record + '\n');

现在, 图片也有了, lst文件也有了, 可以利用im2rec工具生成recordio文件了

! ~/github/mxnet/bin/im2rec ~/dataset/mnist/mxnet/train.lst  ~/dataset/mnist/raw/train/images/   ~/dataset/mnist/mxnet/train.bin
[16:28:31] tools/im2rec.cc:96: Keep origin image size

[16:28:31] tools/im2rec.cc:107: Encoding is .jpg

[16:28:31] tools/im2rec.cc:153: Write to output: /home/dengdan/dataset/mnist/mxnet/train.bin

...

[16:28:35] tools/im2rec.cc:251: 55000 images processed, 4.04441 sec elapsed

[16:28:35] tools/im2rec.cc:251: 56000 images processed, 4.11869 sec elapsed

[16:28:35] tools/im2rec.cc:251: 57000 images processed, 4.1915 sec elapsed

[16:28:36] tools/im2rec.cc:251: 58000 images processed, 4.26273 sec elapsed

[16:28:36] tools/im2rec.cc:251: 59000 images processed, 4.3346 sec elapsed

[16:28:36] tools/im2rec.cc:251: 60000 images processed, 4.40678 sec elapsed

[16:28:36] tools/im2rec.cc:254: Total: 60000 images processed, 4.40682 sec elapsed

github: https://github.com/dengdan/deep-learning-with-mxnet/tree/master/io/mnist2rec.ipynb

相关文章