How to setup Darwin Streaming Server 6.0.3 on 32 or 64 bit Linux platforms, add custom functionality by developing plugins ("modules" as Apple calls them), and results of some performance and load tests I ran
Darwin Streaming Server 6.0.3 - setup, customization, plugin or module development, performance and load tests on 32 & 64 bit Redhat LinuxHow to setup Darwin Streaming Server 6.0.3 on 32 or 64 bit Linux platforms, add custom functionality by developing plugins ("modules" as Apple calls them), and results of some performance and load tests I ran 补丁下载：http://bbs.rosoo.net/forum.php?mod=viewthread&tid=929 IntroductionDarwin Streaming Server is Apple's open source streaming server for the internet. This article will get you started with it, and if you choose to, take you to a certain depth regarding how to tinker with its code to add what you need. It has a very well supported modular architecture, which means you can add one or more custom module(s) - just dropping the binary in a certain folder is sufficient for the server to pick it up and use it. That's what I call plug an play. This is very useful if you do not want to re-build the server's code itself. This article will demonstrate exactly how to do that. Custom modules can be used to perform custom authentication/ authorization, custom logging, reporting, data collection or whatever you have in mind. This article will also discuss the results of some load tests and performance tests I did on the server, trying to discover its limits. My platform was Redhat Linux, but the steps of setting up DSS are pretty much the same for most Unix platforms. HintingDarwin Streaming Server can stream hinted media files, like mp4 or mov files - Apple formats that can be played back using the Quick Time Player. In fact, it can stream almost any other kind of content, as long as the file is hinted. You can "hint" media files using tools Apple provides. If you need to hint your media files, please research separately. I will assume that you have a catalog of hinted files which your Darwin server will stream to a Quicktime player (or any other player able to play Apple streams) running on some machine which can communicate with your server. The default installation of Darwin already comes with some sample hinted media files. To set up and test, you do not need any more files. Platform and HardwareServer Platform$ cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 6) $ gcc -v Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs Server HardwareSummary: Dell SC1425, 2 x Xeon 2.80GHz, 3.9GB / 4GB 400MHz ClientFor playback, I used a Windows box running Quicktime Player. For load testing, I used a different Linux box with profile similar to that of the server. Steps to download, install and setup DSS 6.0.3The following steps should work on both 32 bit and 64 bit Linux. Steps 3 and 4 use patch files that makes the DSS code build without errors on 64 bit. You can go ahead and apply the patches even if you are working on 32 bit, it is better to have portable code. On 64 bit, however, you must apply the patches, otherwise it won't build. It is possible, however, that Apple guys change the code on macosforge server without changing the version number, and then these patches might not work. I always encourage to use the --dry-run option with the patch command to make sure that the patch will actually go through before patching it without the option. Similarly, step 5 uses an "Install" script that is different from what will come with the download from macosforge server. The new install script has the necessary changes for 64 bit installation to go through nicely. The 3 files that steps 3, 4 and 5 refer to are in the "Patches" directory of the source code.
Before you start writing your own modules to customize DSS...This is the fun part for C++ developers. Before attempting to create your own module, you have to understand the architecture of the server. The architecture is pretty decoupled, as the entire server is already written with various modules strung together - many of the core server functionalities are implemented as modules. The core server code is in the Server.tproj folder. After starting up, the server loads modules. Modules are either static or dynamic. Static modules are built when the server is built. When the server runs, it loads the dynamic modules first. It looks for shared libraries (unix so files, only that they are not named with so extension) under the directory /usr/local/sbin/StreamingServerModules. After that, it loads the static modules. The code for all modules that already come with the server are in APIModules directory. If you just want to add custom functionality without tampering the actual server code, it is easiest to write and build a dynamic module - all you need to do is build the dynamic module as a shared object and place the binary in /usr/local/sbin/StreamingServerModules directory. Restarting the server will result in loading of the dynamic modules from this directory, including yours. RolesYou must understand Roles in order to start creating your own modules. Each module must register itself in one or more "role"-s. There are many roles: Register, Initialize, Filter, Route, RTSP Pre-Processing, RTSP Request, RTSP Post Processing, RTP Roles, RTSP Authentication Role, RTSP AUthorization role, Error Logging Role, etc. Think of roles as the building blocks in a flowchart. The Server first invokes the Register role. Followed by Initialize Role. Then it sits quietly, waiting for requests. Once it receives a RTSP Request, it invokes the Filter Role, and passes it the entire URI for "filtering" - modules subscribed to this Role have the privilege of getting a writable copy of the RTSP request: it can actually change any part of the request! The server then invokes the Route role - modules subscribed in this role gets a writable copy of the location (folder) from where to serve the content, and can effectively change the serving point (thereby re-routing the request to a new point). And so on. The sequence in which the server invokes its roles is fixed (coded in the server). More than one modules can subscribe to a role and a module can subscribe to more than one roles. Some roles are exceptions - only one module can register for them. This "subscription" happens inside the code handler for the Register Role - all modules must register to this role. When called upon in this role, a module tells the server what other roles it is subscribing to. Pre-requisite - QTSSAPIDocs.pdfRead the QTSSAPIDocs.pdf file, however boring it might be. Though it is available in the Documentationdirectory, I have uploaded it along with the source zip just to be handy. It is the single most important document amongst all others in the Documentation directory. It is the bible for module developers. Unfortunately, it leaves many things to be desired. It is organized, but messily. Still, this article cannot substitute reading this document. This article is like a compass - it will guide you, give you a better bigger picture, but the nity gritty of code must refer to this pdf file. Come back and read the rest of this article again once you are done with QTSSAPIDocs.pdf. Let us create a custom Auth ModuleLet us imagine a likely scenario: you want to allow/decline requests based on a logic that takes into account the connection url (which would be the case if you force your users to supply some specific signature as a query string) and client IP address (which would be the case if you want to restrict access to some subnetwork). You would need a custom Auth module. Authorization/ Authentication in DSSHow does DSS handle Authentication and Authorization? And how is Authorization different from Authentication? I chose the example of a custom Auth module because this will bring us to the topic of how DSS handles authorization/ authentication. It does a fairly good job at whatever basic scheme it supports - but that is, at best, basic. It uses a rudimentary mechanism, and it might not suffice for your purpose. Let us review what DSS provides us with. Right after installation, all users can access all movies from the movies directory. But say you want to set up n user accounts in such a way that each user will have access to his or her own content only. See http://dss.macosforge.org/ --> "Administrator’s Guide" to see how to add user accounts. Basically, it boils down to the fact that each user will have a subdirectory under the movies directory (/usr/local/movies - the main movies directory). Each such subdirectory will have a " Now, when a given user wants to play a file, he/she can send user id and password as part of the RTSP request. The process of validating that pair of user-id and password against the master list in Needless to say, the server calls "RTSP Autheticate Role" before "RTSP Authorize role". Interestingly and correctly, DSS enforces that only one module can subscribe for "Authenticate" Role, but unlimited number of modules can subscribe for "Authorize" role. That corroborates our conclusion above: Authentication will fail only under one circumstance: if you are an unknown user. But Authorization may fail under various situations, and there may be multiple modules, each verifying some aspect of Authorization. Thus, when the server calls the module subscribed for "Authenticate" role, it expects that module to validate the user id/ password. If that passes, it then invokes all the modules signed up for "Authorize" role - and each one of them can then apply custom rules to either approve or deny that RTSP request. For example, an Authorize module may deny the request because of too high bandwidth or too many number of open connections. Such custom authorization must be implemented as separate modules subcscribing to the Authorizerole. Implementing a completely different auth schemeThis mechanism of validating users (looking for them in If you see the code of QTSSAccessModule, you will see that it has registered to the Authenticate Role (look in So if we were to married to the idea that our new module must register to Authenticate role, we would be forced to retire the QTSSAccessModule. That may be okay, but it is not safe to write off the core modules of DSS just like that. Who knows what else that module must be doing? Anyway, QTSSAccessModule is a static module, it is not dynamic - so getting rid of it means getting the hands dirty with server code. Alternatively, we can edit QTSSAccessModule and write our code inside that module. Again, something that is not elegant. For the kind of problem we are trying to solve, Authorize Role fits the bill perfectly. One way to suppress whatever QTSSAccessModule does is not to have any Writing the codeA word on your reference copy: the entire code for Building an existing module as dynamic libraryBefore starting to write the code for a new module, you should learn how to build (make) an existing module - how to build only a module (not the whole server) as a shared library so that you can simply drop the module binary into the /usr/local/sbin/StreamingServerModules directory. With some tweaking of the build scripts, I came up with one that I could copy directly to the subdirectory containing the code for the module. Each subdirectory under APIModules directory already contains a Makefile.POSIX - but that cannot be used directly, as it expects some exported variables that the global build script (remember "Buildit"?) populates before invoking it. But to build a module separately, we need a local build script that will enable us to similarly use the local Makefile.POSIX. On top of that, I tweaked it some more to output a shared library, not a static one. The result is "build" - refer source zip/QTSSMyAuthModule. Try it on an existing module first, say QTSSHomeDirectoryModule. Simply copy the "build" script I have uploaded to QTSSHomeDirectoryModule folder. Edit the script's final echo statement (also you might need to run chmod to make it executable) to print out the correct module's name, and you should be done. Run "build clean" first and then run "build", and see the object files and the binary called "QTSSHomeDirectoryModule" getting created. Now copy that binary to /usr/local/sbin/StreamingServerModules. Restart the server. Look at /var/streaming/logs/Error.log at this point, and it should have freshly logged a list of modules the server just loaded. For each module, it specifies whether it is static or dynamic. If you are successful in building QTSSHomeDirectoryModule, it should have it listed as loaded and dynamic. Repeating the experiment by removing the binary from /usr/local/sbin/StreamingServerModules will result in that line of log not appearing. Time to create QTSSMyAuthModuleCopy any existing module (The following tutorial assumes you have used QTSSHomeDirectoryModule) source code in a new subdirectory "QTSSMyAuthModule" and change all references to the old name to new name. There are 2 roles that QTSS Home Directory Module handles which we do not need to: RTSPRoute Role and ClientSessionClosing Role. I am not going to explain all these roles here, please read the bible for that. Remove the code handling those roles. The other 4 (Register, Initialize, RereadPrefs and RTSPAuthorize) we need. And we need an extra role that is not in there: Shutdown Role, so add code for that one (see uploaded source zip). Let me give you a quick idea of the roles we will handle in QTSSMyAuthModule.
Thus, as you see, no article can show you the exact technique you need to know - there is an ocean of information among the code of existing modules and QTSSAPIDocs.pdf, though scattered and difficult to locate. How quickly you find the technique you need from the existing code base and/or the documentation depends how much back you broke. Using your own classesI will refrain from explaining each line of my code of QTSSMyAuthModule - the place where the exact authentication is performed is full of running comments. I separated the functionality of allowing/denying to a separate class - MyAuth.cpp, MyAuth.h. I have seen that I could not use STL classes and methods directly into QTSSMyAuthModule.cpp (the main template file that houses the mandatory methods a module must have) - they conflict with some header file DSS uses. That should be fixable with some effort, but I opted for the easier route: if I follow my scheme of including a separate class, I can use STL in MyAuth.h/cpp without any problem. How configuration works/etc/streaming/streamingserver.xml is precise and clear. It has all the server's settings listed under /CONFIGURATION/SERVER, followed by the settings for all the modules, each with node /CONFIGURATION/MODULE@NAME . Changing these settings do not need a server restart - only sending the HUP signal to the server process (one with the higher pid). For a custom dynamic module to re-load preferences at the same time when the server does (in response to HUP signal), it must implement the RereadPrefs Role. How logging worksError LoggingThough DSS has all modules in the APIModules directory, I found one that is hidden in Server.tproj folder - QTSSErrorLogModule. That is probably because this module is deemed an integral part of the server itself, and becasue it is so important. It is because of this module that any other module and the sever itself can write to the Error.log file located in /var/streaming/logs. Even informational logs can be simply written to the Error.log file with verbosity parameter = qtssDebugVerbosity, and it won't actually show up unless /CONFIGURATION/SERVER/error_logfile_verbosity is set to 4 (qtssDebugVerbosity). The possible values of this configuration setting is: 0 (Fatal Verbosity), 1 (Warning Verbosity), 2 (Message or INFO Verbosity), 3 (Assert Verbosity) and 4 (Debug). Whatever the value is set to, logs written with verbosity parameter equal to or less than that value will actually get written. Writing a log with Fatal Verbosity causes the server to shut down. In order to write to the error log stream from anywhere in a custom module, we need to save the error log stream in a static Stream Request LoggingThe module Server monitoring/ Heath check supportDSS has a cool feature: turn on the entry "enable_monitor_stats_file" in streamingserver.xml, and it will output a stats file to the same folder as Error.log is written to (which is /var/streaming/logs, but you can change it using the "error_logfile_dir" entry in streamingserver.xml). This stats file is refreshed every X seconds, where X is "monitor_stats_file_interval_seconds". Worth mentioning here is my encounters with this monitoring feature: the file has XML format, so I decided to monitor my server using a browser. But somehow, DSS writes some gibberish to the top part of the stats file, which makes the XML malformed, so the browser cannot render it. Also, the permissions of the file is set such that not everybody can read the file. I also wanted to output the file anywhere I wanted (using the "monitor_stats_file_name" entry), but DSS is written to expect only the file name from that entry, not the entire path. My adventures to fix this led me to Server.tproj/RunServer.cpp, Performance and Load testsI am not happy with the load tests I ran on DSS. It cannot take thousands of concurrent users. Someday I might take the time and effort to try to fix (read improve) that in the code, but let me first point you to how I ran these tests. StreamingLoadToolDSS comes with a load test tool, /usr/local/bin/StreamingLoadTool. Pass -v to it to know about the other parameters it takes. I used the -f <configuration file> parameter mostly, and used the config file to specify everything else. The sample config file is located as /etc/streaming/streamingloadtool.conf. I could successfully run load tests from a different client machine - all it takes is to copy the load test binary and the configuration file over. I found that the StreamingLoadTool that comes with DSS 6.0.3 crashes when I run it with runforever=yes and concurrent number of clients=100. Pfft. openRTSPA better tool to load test DSS is openRTSP from www.live555.com. If you want to build it from source, you have to download the entire live555 streaming media source code. I went ahead and followed their instruction, and it went smoothly. You will end up with a "live" directory at the end - it's your choice where to put it, but that contains everything. openRTSP, the command line RTSP client tool, is inside testProgs folder. It has excellent documentation, so I will refrain from explaining how it should be used. I wrote a script that spawned lots of openRTSP sessions, each with -c option (for continuous play), -r (do not render video) and -p 80 (to redirect the video to that port), and I controlled the number and rate of spawning. ResultsThe result is that DSS, with all optimizatoins I could think of (increase the maximum number of open file handles in your system to a gigantic number using /etc/security/limits.conf, set "maximum_bandwidth" and "maximum_connection" in streamingserver.xml to -1, "run_num_threads" to various values, including 0 for auto-detection of number of cores) - DSS cannot handle more than 350 concurrent users.
The bandwidth numbers above is a straightforward calculation considering I was streaming 300 kbps files. DSS sample files also comes with 1000 kbps files and high definition files, I am sure these files will result in much higher bandwidth, but performance and latency using the high bitrate files was not measured in my experiments - I will await feedback from readers on how that went. I did not notice any memory leaks - in stable mode, it ran constantly for weeks, processing 350 continuous and concurrent users, with no leaks perceived. Running multiple DSS servers for scalabilityFor some, that could be enough. I needed more. There is a solution though - even when DSS topples over with 400 users, it uses virtually no CPU & memory (which indicates that the bottleneck is somewhere in the code, and that makes me bolder when I think that it can be fixed). This "leanness" can be effectively used by running multiple DSS servers! While executing the server, we can pass -c <whatever streamingserver.xml we want>. If we create multiple streamingserver.xml files, each with a different port listed in "rtsp_port" entry (like 7070 in one, 554 in another, and so on), and then execute multiple DSS instances each using a separate streamingserver.xml as configuration file, we can achieve this. I have parallelly run 3 instances of the server, each processing 350 concurrent users - that is one way to reach four digits - 1000 users! Only caveat is the url you pass to openRTSP (or whatever client you are using) must access the correct instance of the server using the port number specified using a colon after the hostname. If you are thinking of using this setup in real world, you have to write another application - a software load balancer, that will listen on any given port, and delegate the request to any of the N running DSS instances. Clients then must connect to that port always. I have not yet tried this thing out though. RTSP, RTP, UDP, RUDP, TCP...While dealing with Darwin Streaming Server, you will come across these acronyms. At times you will have to choose one over another. For example, while setting up StreamingLoadTool (the load test tool that comes inherently with DSS), its configuration file has entries where you pick the mode of execution - RUDP? UDP? TCP? You can retain the defaults if you are not sure what to pick. Explaining the differences and meanings of these is beyond the scope of this article. May be in a future article. I mention these here because you need to understand these concepts to get the most out of DSS. If you are only setting up a server for your users to use - you may ask - why do you care whether it is doing RUDP or UDP? At the beginning, you might not need to care. But if lots of users are using your service, you may be asked which one gives higher throughput. They may be using clients of varying capabilities, and ones that gives it users the option to pick a mode. For absolute beginners, just inderstand this much: RTSP is the protocol of delivery (corresponds to HTTP). RTP is the underlying protocol used for packets transfers by RTSP. TCP and UDP are alternate network protocols - TCP is characterized by high reliability but low timeliness, as it involves a lot of handshaking (ACK-s, NACK-s etc). UDP is characterized by low reliability (packets may not make it to the other end) but high timeliness. R-UDP is Apple's proprietary version of UDP that adds some handshaking, thus they call it "Reliable UDP". Here is an article that attempts a better explanation: http://www.vbrick.net/Topics/RTSP.asp. You may ask why does DSS need to deal with these concepts? Any streaming server would, as its sole job is to deliver a streaming media to the client - which, as opposed to playing back a file, is characterized by the fact that nothing is stored on the client side. Bits must be transmitted exactly at the speed that the video being played demands - a lighter scene may need 100,000 bits per second, and a darker scene may need 700,000 bits per second. This number is called the bitrateof the video. Coordinating this transfer, where the speed goes up and down, ensuring reliability and handshaking is the complex domain of streaming servers. And it cannot be solved by HTTP, as HTTP can only fetch the whole file (it can also fetch partial files based on byte range requests) but it cannot fetch a file continuously over the same connection. RTSP, RTMP, etc are protocols used for this purpose. ConclusionApple guys have coded this well. A reviewer, on afterthought, can always point out areas of improvement, but the project is so massive that finally ending up with a coherent, working and mostly bugfree set of files of this magnitude is a commendable achievement. Usage of so many static variables is something I do not like - it exerts pressure on the heap. And amateurishly conceived utility classes like StrPtrLen literally drives me mad - these guys have created a class to hold a pointer to sombody else's data! Please be extremely careful while using this almost criminal class - it invites misuse. Overall, the code could have been neater, but hey, I did not pay for this. I will take what DSS is worth for. LicenseThis article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL) About the AuthorKoushik serves Yahoo! Video Streaming Center from its Dallas location. An Electronics Engineer from Jadavpur University, he is father to twin daughters. Apart from spending time with work and projects, he loves cricket, Kishore Kumar, Bruce Willis, camping and fishing, Mediterranean food, sea beaches and his gas grill. (Koushik Biswas ) from:http://www.rosoo.net/a/201008/10062.html |