批量删除大量文件

Everyday user-uploaded files older than a month are deleted from the server. User uploads are stored into directories by the day (eg /var/www/media/2013-03-13) so its easy to identify the files/directory that needse to be deleted.

每天用户上传超过一个月的文件将从服务器中删除。用户上传按日存储在目录中(例如/var/www/media/2013-03-13)，因此很容易识别需要删除的文件/目录。

Problem: Deleting 100,000 files at a time makes the server unresponsive and takes a long time. (Ubuntu 12.04 with 2x2TB ext4 SATA3 hdd in software RAID1). At the moment PHP is doing exec on the command find /path/to/dir -maxdepth 1 -name '*' -delete.

问题:一次删除100,000个文件会使服务器无法响应，并且需要很长时间。(ubuntu12.04和2x2TB ext4 SATA3 hdd在软件RAID1中)。目前PHP正在执行命令find /path/to/dir -maxdepth 1 -name '*' -delete。

How do I split up the files required for deletion? Doing a ls will take really long on those large directories.

如何分割删除所需的文件?在这些大目录上执行ls会花费很长时间。

Solution need not be in PHP. It does not even require splitting the files into smaller batches

解决方案不需要在PHP中。它甚至不需要将文件分割成更小的批

2 个解决方案

#1

Find the directories of the days you want to keep
找到你想保存的日子的目录
Find the directories of all days
找到所有日子的目录
Remove 1. from 2.
删除1。从2。
rm -rf the directories you get from 3.
rm -rf你从3得到的目录。

I'm not sure if this is faster than your method, but it avoids explicitly listing all the files in the directories.

我不确定这是否比您的方法快，但它避免显式列出目录中的所有文件。

#2

The trick is not to read all the files, but to use readdir to get them one by one. See Perl to the rescue: case study of deleting a large directory for a Perl solution.

窍门不是读取所有的文件，而是使用readdir逐个地读取它们。请参阅Perl来拯救:为Perl解决方案删除大目录的案例研究。

#1

Find the directories of the days you want to keep
找到你想保存的日子的目录
Find the directories of all days
找到所有日子的目录
Remove 1. from 2.
删除1。从2。
rm -rf the directories you get from 3.
rm -rf你从3得到的目录。