I have to download big file (1xx MB) using PHP.
我必须使用PHP下载大文件(1xx MB)。
How can i download this without wasting memory (RAM) for temporary file ?
我如何下载这个而不浪费内存(RAM)临时文件?
When i use
当我使用
$something=file_get_contents('http://somehost.example/file.zip');
file_put_contents($something,'myfile.zip');
I need to have so much memory that size of that file.
我需要有那么大的内存大小的文件。
Maybe it's possible to download it using any other way ?
也许可以使用任何其他方式下载它?
For example in parts (for example 1024b), write to disk, and download another part repeating until file will be fully downloaded ?
例如,在部分(例如1024b)中,写入磁盘,并下载另一部分重复,直到文件完全下载?
2 个解决方案
#1
23
Copy the file one small chunk at a time
一次将文件复制一个小块
/**
* Copy remote file over HTTP one small chunk at a time.
*
* @param $infile The full URL to the remote file
* @param $outfile The path where to save the file
*/
function copyfile_chunked($infile, $outfile) {
$chunksize = 10 * (1024 * 1024); // 10 Megs
/**
* parse_url breaks a part a URL into it's parts, i.e. host, path,
* query string, etc.
*/
$parts = parse_url($infile);
$i_handle = fsockopen($parts['host'], 80, $errstr, $errcode, 5);
$o_handle = fopen($outfile, 'wb');
if ($i_handle == false || $o_handle == false) {
return false;
}
if (!empty($parts['query'])) {
$parts['path'] .= '?' . $parts['query'];
}
/**
* Send the request to the server for the file
*/
$request = "GET {$parts['path']} HTTP/1.1\r\n";
$request .= "Host: {$parts['host']}\r\n";
$request .= "User-Agent: Mozilla/5.0\r\n";
$request .= "Keep-Alive: 115\r\n";
$request .= "Connection: keep-alive\r\n\r\n";
fwrite($i_handle, $request);
/**
* Now read the headers from the remote server. We'll need
* to get the content length.
*/
$headers = array();
while(!feof($i_handle)) {
$line = fgets($i_handle);
if ($line == "\r\n") break;
$headers[] = $line;
}
/**
* Look for the Content-Length header, and get the size
* of the remote file.
*/
$length = 0;
foreach($headers as $header) {
if (stripos($header, 'Content-Length:') === 0) {
$length = (int)str_replace('Content-Length: ', '', $header);
break;
}
}
/**
* Start reading in the remote file, and writing it to the
* local file one chunk at a time.
*/
$cnt = 0;
while(!feof($i_handle)) {
$buf = '';
$buf = fread($i_handle, $chunksize);
$bytes = fwrite($o_handle, $buf);
if ($bytes == false) {
return false;
}
$cnt += $bytes;
/**
* We're done reading when we've reached the conent length
*/
if ($cnt >= $length) break;
}
fclose($i_handle);
fclose($o_handle);
return $cnt;
}
Adjust the $chunksize variable to your needs. This has only been mildly tested. It could easily break for a number of reasons.
根据需要调整$ chunksize变量。这只是经过了温和的测试。由于多种原因,它很容易破裂。
Usage:
用法:
copyfile_chunked('http://somesite.com/somefile.jpg', '/local/path/somefile.jpg');
#2
6
you can shell out to a wget
using exec()
this will result in the lowest memory usage.
你可以使用exec()来外壳到wget,这将导致最低的内存使用量。
<?php
exec("wget -o outputfilename.tar.gz http://pathtofile/file.tar.gz")
?>
You can also try using fopen()
and fread()
and fwrite()
. That way you onlly download x bytes into memory at a time.
您也可以尝试使用fopen()和fread()以及fwrite()。这样你一次只能将x字节下载到内存中。
#1
23
Copy the file one small chunk at a time
一次将文件复制一个小块
/**
* Copy remote file over HTTP one small chunk at a time.
*
* @param $infile The full URL to the remote file
* @param $outfile The path where to save the file
*/
function copyfile_chunked($infile, $outfile) {
$chunksize = 10 * (1024 * 1024); // 10 Megs
/**
* parse_url breaks a part a URL into it's parts, i.e. host, path,
* query string, etc.
*/
$parts = parse_url($infile);
$i_handle = fsockopen($parts['host'], 80, $errstr, $errcode, 5);
$o_handle = fopen($outfile, 'wb');
if ($i_handle == false || $o_handle == false) {
return false;
}
if (!empty($parts['query'])) {
$parts['path'] .= '?' . $parts['query'];
}
/**
* Send the request to the server for the file
*/
$request = "GET {$parts['path']} HTTP/1.1\r\n";
$request .= "Host: {$parts['host']}\r\n";
$request .= "User-Agent: Mozilla/5.0\r\n";
$request .= "Keep-Alive: 115\r\n";
$request .= "Connection: keep-alive\r\n\r\n";
fwrite($i_handle, $request);
/**
* Now read the headers from the remote server. We'll need
* to get the content length.
*/
$headers = array();
while(!feof($i_handle)) {
$line = fgets($i_handle);
if ($line == "\r\n") break;
$headers[] = $line;
}
/**
* Look for the Content-Length header, and get the size
* of the remote file.
*/
$length = 0;
foreach($headers as $header) {
if (stripos($header, 'Content-Length:') === 0) {
$length = (int)str_replace('Content-Length: ', '', $header);
break;
}
}
/**
* Start reading in the remote file, and writing it to the
* local file one chunk at a time.
*/
$cnt = 0;
while(!feof($i_handle)) {
$buf = '';
$buf = fread($i_handle, $chunksize);
$bytes = fwrite($o_handle, $buf);
if ($bytes == false) {
return false;
}
$cnt += $bytes;
/**
* We're done reading when we've reached the conent length
*/
if ($cnt >= $length) break;
}
fclose($i_handle);
fclose($o_handle);
return $cnt;
}
Adjust the $chunksize variable to your needs. This has only been mildly tested. It could easily break for a number of reasons.
根据需要调整$ chunksize变量。这只是经过了温和的测试。由于多种原因,它很容易破裂。
Usage:
用法:
copyfile_chunked('http://somesite.com/somefile.jpg', '/local/path/somefile.jpg');
#2
6
you can shell out to a wget
using exec()
this will result in the lowest memory usage.
你可以使用exec()来外壳到wget,这将导致最低的内存使用量。
<?php
exec("wget -o outputfilename.tar.gz http://pathtofile/file.tar.gz")
?>
You can also try using fopen()
and fread()
and fwrite()
. That way you onlly download x bytes into memory at a time.
您也可以尝试使用fopen()和fread()以及fwrite()。这样你一次只能将x字节下载到内存中。