We have a project page which consists of users' files, multimedia stuff, etc and we want to allow the user to export all this out into a single zip file. We're using unix and mysql to store all of these currently and our primary goal is to minimize load/performance time from all the processing and compiling all the files into a zip file.
我们有一个项目页面,其中包含用户文件,多媒体内容等,我们希望允许用户将所有这些内容导出到单个zip文件中。我们正在使用unix和mysql来存储所有这些,我们的主要目标是最大限度地减少所有处理的负载/性能时间,并将所有文件编译成zip文件。
My idea was to cache the zip file into a temp dir and keep all the CRC checksum for each files in the zip into a separate text file. Each time the user tries to do an export I will first check through each file's CRC and compare it to the list before adding or removing files from the zip file.
我的想法是将zip文件缓存到临时目录中,并将zip中每个文件的所有CRC校验和保存到单独的文本文件中。每次用户尝试进行导出时,我将首先检查每个文件的CRC,并在添加或删除zip文件中的文件之前将其与列表进行比较。
But my other concern is also the space that the zip file will be occupying as we might have a lot of users.
但我的另一个问题也是zip文件占用的空间,因为我们可能有很多用户。
IMHO, this is probably the dumbest way possible to do this, so can any of you guys please suggest a better way to deal with this problem?
恕我直言,这可能是最愚蠢的方式,所以你们中的任何人都可以建议一个更好的方法来解决这个问题吗?
thanks ~codeNoobian
3 个解决方案
#1
2
This reaks of premature optimization, just use a very light compression, aka 'fastest' and worry about the speed if it's actually a problem.
这种过早优化的问题,只需使用非常轻的压缩,即“最快”,如果实际上是一个问题则担心速度。
#2
0
If bandwidth/download speed is not a concern, I recommend you use an uncompressed tar file. TAR is a very simple format, so it will be easy to write code to update sections of it when a few of the files have changed. Also, leaving it uncompressed will be a huge win on server CPU time.
如果不考虑带宽/下载速度,我建议您使用未压缩的tar文件。 TAR是一种非常简单的格式,因此当一些文件发生变化时,可以很容易地编写代码来更新它的各个部分。此外,保持未压缩状态将是服务器CPU时间的巨大胜利。
Of course, leaving it uncompressed will take a lot of storage space on your server. But since it is uncompressed, it might remove the need for you to keep a cache copy of the file at all, if you can build it fast enough you can just build it on the fly as needed. Then you don't have to worry about storing CRCs and updating the TAR, either.
当然,保持未压缩状态会占用服务器上的大量存储空间。但是由于它是未压缩的,因此可能无需保留文件的缓存副本,如果您可以足够快地构建它,您可以根据需要即时构建它。然后,您不必担心存储CRC和更新TAR。
#3
0
Common sound and image files are pretty well compressed to start with, aren't they? It might be worth looking at your payload to see how much you're buying with compression.
常见的声音和图像文件开始时压缩得很好,不是吗?可能值得查看您的有效负载,看看您通过压缩购买了多少。
#1
2
This reaks of premature optimization, just use a very light compression, aka 'fastest' and worry about the speed if it's actually a problem.
这种过早优化的问题,只需使用非常轻的压缩,即“最快”,如果实际上是一个问题则担心速度。
#2
0
If bandwidth/download speed is not a concern, I recommend you use an uncompressed tar file. TAR is a very simple format, so it will be easy to write code to update sections of it when a few of the files have changed. Also, leaving it uncompressed will be a huge win on server CPU time.
如果不考虑带宽/下载速度,我建议您使用未压缩的tar文件。 TAR是一种非常简单的格式,因此当一些文件发生变化时,可以很容易地编写代码来更新它的各个部分。此外,保持未压缩状态将是服务器CPU时间的巨大胜利。
Of course, leaving it uncompressed will take a lot of storage space on your server. But since it is uncompressed, it might remove the need for you to keep a cache copy of the file at all, if you can build it fast enough you can just build it on the fly as needed. Then you don't have to worry about storing CRCs and updating the TAR, either.
当然,保持未压缩状态会占用服务器上的大量存储空间。但是由于它是未压缩的,因此可能无需保留文件的缓存副本,如果您可以足够快地构建它,您可以根据需要即时构建它。然后,您不必担心存储CRC和更新TAR。
#3
0
Common sound and image files are pretty well compressed to start with, aren't they? It might be worth looking at your payload to see how much you're buying with compression.
常见的声音和图像文件开始时压缩得很好,不是吗?可能值得查看您的有效负载,看看您通过压缩购买了多少。