如何并行映射以获得更快的文件读取?

I am working through this code and have the mmap working now, but I am wondering if I can use mmap in parallel and if so, how to accomplish it. Suppose I have my data on a parallel file system (GPFS, RAID0, whatever) and I want to read it using n processes.

我正在处理这段代码并让mmap工作，但是我想知道是否可以并行地使用mmap，如果可以，如何实现它。假设我有一个并行文件系统的数据(GPFS、RAID0等等)，我想使用n个进程读取它。

How could I, for example, have each processor read 1/nth contiguous block of the data into memory? Or, alternatively, read every nth memory block (1 B, 1 MB, 100 MB, 1 GB, whatever I choose for optimization) into memory?

例如，如何让每个处理器将数据的第1/n个连续块读入内存?或者，或者，读取每个第n个内存块(1个B, 1 MB, 100 MB, 1 GB，不管我选择什么优化)到内存中?

I am assuming a posix file system here.

我在这里假设一个posix文件系统。

1 个解决方案

#1

Here is my mpi function for parallel reading. It chops up the file into n contiguous pieces based on pagesize and has each process read a separate piece via mmap. Some extra tricks need to be done at the end since process i will (likely) get the first half of a line as it's last line and process i+1 will get the second half of the same line as it's first line.

这是并行读取的mpi函数。它根据pagesize将文件分割成n个连续的片段，并且每个进程通过mmap读取一个单独的片段。一些额外的技巧需要在最后完成，因为我将(很可能)得到一条线的前半部分，因为它的最后一行和过程i+1将得到与它的第一行相同的线的下半部分。

ikind nchars_orig; // how many characters were in the original file
int pagesize = getpagesize();
off_t offset;
struct stat file_stat;
int finp = open(inpfile, O_RDONLY);
int status = fstat(finp, &file_stat);
nchars_orig = file_stat.st_size;

// find out hwich pieces of the file each process should read
ikind nchars_per_proc[nprocs];
for(int ii = 0; ii < nprocs; ii++) {
    nchars_per_proc[ii] = 0;
}   
// start at the second to last proc, so the last proc will get hit first
// we will decrement him at the end, so this will distribute the work more evenly
int jproc = nprocs-2;
ikind nchars_tot = 0;
ikind nchardiff = 0;
for(ikind ic = 0; ic < nchars_orig; ic+= pagesize) {
    jproc += 1;
    nchars_tot += pagesize;
    if(jproc == nprocs) jproc = 0;
    if(nchars_tot > nchars_orig) nchardiff = nchars_tot - nchars_orig;
    nchars_per_proc[jproc] += pagesize;
}   
nchars = nchars_per_proc[iproc];
if( iproc == nprocs-1 ) nchars = nchars - nchardiff;
offset = 0;
for(int ii = 0; ii < nprocs; ii++) {
    if( ii < iproc ) offset += nchars_per_proc[ii];
} 
cs = (char*)mmap(0, nchars, PROT_READ, MAP_PRIVATE, finp, offset);