Lustre，Gluster或MogileFS ??用于视频存储，编码和流媒体

So many options and so little time to test them all... I wonder if someone has experiences with distributed file systems for video streaming and storage/encoding.

这么多的选项和很少的时间来测试它们......我想知道是否有人有分布式文件系统的视频流和存储/编码经验。

I have a lot of huge video files (50GB to 250GB) that I need to store somewhere, be able to encode them to mp4 and stream them from several Adobe FMS servers. The only way to handle all this is with a distributed file system but now the question is which one??

我有很多巨大的视频文件(50GB到250GB),我需要存储在某个地方,能够将它们编码为mp4并从几个Adobe FMS服务器流式传输。处理所有这一切的唯一方法是使用分布式文件系统,但现在的问题是哪个?

My research so far tells me:

到目前为止我的研究告诉我:

Lustre: mature proven solution, used by a lot of big companies, best with >10G files is a kernel driver.

Lustre:成熟的解决方案,被许多大公司使用,最好用> 10G文件是内核驱动程序。

Gluster: new, less mature, FUSE based that means easy to install but maybe slower due to FUSE overhead. Better to handle a large number of smaller files ~1GB

Gluster:基于FUSE的新的,不太成熟的,意味着易于安装,但由于FUSE开销可能会更慢。最好处理大量的小文件~1GB

MogileFS: seems to be only for small files ~MB, uses HTTP for access?? possible FUSE binding in the future.

MogileFS:似乎只适用于小文件~MB,使用HTTP进行访问?未来可能的FUSE绑定。

So far Lustre seems the winner but I would like to hear real experiences for the particular application I have.

到目前为止,Lustre似乎是赢家,但我希望听到我所拥有的特定应用程序的真实体验。

Also Hadoop, Redhat GFS, Coda and Windows DFS sound as options so any experiences are welcome. If someone has benchmarks please share.

Hadoop,Redhat GFS,Coda和Windows DFS也可作为选项,因此欢迎任何体验。如果有人有基准,请分享。

After some real experience this is what I have learned:

经过一些实际经验,这是我所学到的:

Luster:
- Performance: Amazingly fast! I can assert that Lustre can serve a lot of streams and that encoding speed is not affected by accessing files via Lustre.
- POXIS compatibility: Very good!. No need to modify applications to use luster.
- Replication, Load Balancing and Fail Over: Very bad!. For replication load balancing we and fail over we need to rely on other software such as virtual IPs and DRDB.
- Installation: The worst!. Impossible to install by mere mortals. Requires a very specific combination of kernel, lustre patches and tweaks to get it working. And current luster patches usually work with old kernels that are incompatible with new hardware/software.

光泽:性能:惊人的快!我可以断言Lustre可以服务很多流,并且通过Lustre访问文件不会影响编码速度。 POXIS兼容性:非常好!无需修改应用程序即可使用光泽。复制,负载平衡和故障转移:非常糟糕!对于复制负载平衡我们和故障转移,我们需要依赖其他软件,如虚拟IP和DRDB。安装:最糟糕的!凡人都无法安装。需要非常具体的内核,光泽补丁和调整组合才能使其正常工作。目前的光泽补丁通常适用于与新硬件/软件不兼容的旧内核。

MogileFS:
- Performance: Good for small files but not usable for medium to large files. This is mostly due to HTTP overhead since all files are send/receive via HTTP requests that encode all data in base64 adding a 33% overhead to each file.
- POXIX compatibility is non existent. All applications require to be modified to use mogilefs that renders it useless for streaming/encoding since most streaming servers and encoding tools do not understand MogileFS protocol.
- Replication and failover out of the box and load balancing can be implemented in the application by accessing more than one tracker at a time.
- Installation is relatively easy and ready to use packages exist in most distributions. The only difficulty I found was setting the database master-slave to eliminate the single point of failure.
  - Gluster:
- Performance: Very bad for streaming. I cannot reach more than a few Mbps in a 10Gbps network. Clients and Server CPU skyrockets on heavy writes. For encoding works because the CPU is saturated before the network and I/O.
- POXIS: Almost compatible. The tools I use can access gluster mounts as normal folders in disk but in some edge cases things start causing problems. Check gluster mailing lists and you will see there are a lot of problems.
- Replication, Failover and Load balancing: The best! if they actually worked. Gluster is very new and it has a lot of bugs and performance problems.
- Installation is too easy. The management command line is amazing and setting replicated, striped and distributed volumes among several servers can not be any easier.

MogileFS:性能:适用于小文件但不适用于中型到大型文件。这主要是由于HTTP开销,因为所有文件都是通过HTTP请求发送/接收的,这些请求对base64中的所有数据进行编码,为每个文件增加了33%的开销。 POXIX兼容性不存在。所有应用程序都需要修改为使用mogilefs,因为大多数流服务器和编码工具都不了解MogileFS协议,因此它无法用于流/编码。通过一次访问多个跟踪器,可以在应用程序中实现开箱即用的复制和故障转移以及负载平衡。安装相对容易,大多数发行版中都有现成的软件包。我发现的唯一困难是设置数据库主从以消除单点故障。 Gluster:性能:非常糟糕的流式传输。在10Gbps网络中,我的速度不能超过几Mbps。客户端和服务器CPU在大量写入时飙升。对于编码工作,因为CPU在网络和I / O之前已经饱和。 POXIS:几乎兼容。我使用的工具可以访问gluster挂载作为磁盘中的普通文件夹,但在某些边缘情况下,事情开始引起问题。检查gluster邮件列表,你会发现有很多问题。复制,故障转移和负载平衡:最好的!如果他们真的有效。 Gluster非常新,它有很多错误和性能问题。安装太简单了。管理命令行非常棒,在多个服务器之间设置复制,条带和分布式卷是不容易的。

Final conclusion:

Unfortunately the conclusion is "No single silver bullet".

不幸的是,结论是“没有单一的银弹”。

Currently we have our media files in Gluster3.2 in a replicated volume for storage and transcoding. As long as you don't have a lot of servers, avoid geo-replication and stripe volumes things work ok.

目前,我们将Gluster3.2中的媒体文件放在复制卷中以进行存储和转码。只要您没有很多服务器,就可以避免地理复制和条带卷工作正常。

When we are going to stream the media files we copy them to a lustre volume that is replicated to a second lustre volume via DR:DB. The wowza server then read the media files from the lustre volumes.

当我们要传输媒体文件时,我们将它们复制到通过DR:DB复制到第二个光泽卷的光泽卷。然后,wowza服务器从光泽卷中读取媒体文件。

And finally we use MogileFS to serve the thumbnails in our web application servers.

最后,我们使用MogileFS来提供Web应用程序服务器中的缩略图。

5 个解决方案

#1

GlusterFS improved themselves a lot up to this date. They are now providing "granular locking" for large files. See here: http://www.gluster.org/community/documentation/index.php/WhatsNew3.3 Also it is quite dependent video frame rates you should work for too. If you will not go up to 4K rates, Gluster can solve the storage problems. If there is a huge demand for speed, therefore Infiniband can come in to play.

到目前为止,GlusterFS已经进行了很多改进。他们现在为大文件提供“粒度锁定”。请参阅此处:http://www.gluster.org/community/documentation/index.php/WhatsNew3.3此外,您应该依赖视频帧速率。如果你不能达到4K的速率,Gluster可以解决存储问题。如果对速度有巨大需求,那么Infiniband就可以参与其中。

#2

Check out Hadoop Filesystem (HDFS). Its focus is on very large files and parallel task computing (with map/reduce), it has a high latency but very high throughput. It is currently used on such large installations as Facebook and amazon.com

查看Hadoop文件系统(HDFS)。它的重点是非常大的文件和并行任务计算(使用map / reduce),它具有高延迟但吞吐量非常高。它目前用于Facebook和amazon.com等大型设备

#3

MogileFS is great for that sort of thing. The client libraries varies a bit in quality, but I'd be surprised if there weren't large-ish scale production sites using just about any language to access it.

MogileFS非常适合这种事情。客户端库的质量有所不同,但如果没有使用任何语言访问它的大型规模生产站点,我会感到惊讶。

HTTP is a good protocol for this stuff actually. Who doesn't have a feature-rich and efficient HTTP client?

HTTP实际上是这个东西的一个很好的协议。谁没有功能丰富且高效的HTTP客户端?

#4

From the named systems the most suitable is MoglieFS.

从命名系统中最合适的是MoglieFS。

But perhaps you can get by w/out any special system at all. Say you have 4 AdobeFMS servers:

但也许你可以通过任何特殊系统来完成任务。假设您有4个AdobeFMS服务器:

{video0.exmple.com,video1.exmple.com,video2.exmple.com,video3.exmple.com}.

You can distribute all your videos among those 4 servers using simple scheme, like

您可以使用简单的方案在这4台服务器中分发所有视频,例如

    /*
     *  pseudo code
     */

    $server_id = get_server_id(filename);
    ...
    ...
    int function get_server_id(filename) 
    { 
       return hash(filename) mod 4;
    }

after you encode videos, your app would

对视频进行编码后,您的应用就可以了

$server_id = get_server_id(file_name)
copy file_name to /mnt/$server_id/

clients will access videos using something like http://videoN.example.com/filename.mp4, where N is calculated from filename using get_server_id() .

客户将使用http://videoN.example.com/filename.mp4之类的内容访问视频,其中N是使用get_server_id()从文件名计算的。

Luster/Gluster is really not what you should be looking for. Luster FS is more mature, but developers ask you to treat files on such FS as "cache", i.e. they can be lost at any time.

Lustre / Gluster真的不是你应该寻找的。 Lustre FS更成熟,但开发人员要求您将此类FS上的文件视为“缓存”,即它们可能随时丢失。

Luster/Gluster are targeted for use in HPC to allow fast access for huge amounts of data w/out single storage server being performance bottle-neck. Another point for those systems is that they are POSIX-complaint. In HPC/Scientific research environment you usually do not have a time to waist for rewriting your apps because you installed new cool and fast FS.

Lustre / Gluster的目标是在HPC中使用,以便快速访问大量数据,而单个存储服务器则是性能瓶颈。这些系统的另一点是它们是POSIX投诉。在HPC /科研环境中,由于您安装了新的酷且快速的FS,因此您通常没有时间来重写您的应用程序。

#5

Map-reduce doesn't help in write/read ratio of 90/10! The constant file size is a good thing and the files are small. So, MogileFS sounds to be good alternative as Luster/Gluster - cache situation is not appropriate.

Map-reduce对写入/读取比率为90/10没有帮助!常量文件大小是一件好事,文件很小。因此,MogileFS听起来是Lustre / Gluster的好选择 - 缓存情况不合适。

#1

#2

#3