I would like to download big file from Amazon S3 into RAM. File is bigger then RAM size. Seems, I need to load it by parts. Each part would be return in endpoint. Also I can not use hard drive, to store downloaded file there. I have InputStream
object and I am trying to load object like below:
我想从Amazon S3下载大文件到RAM。文件大于RAM大小。似乎,我需要按部件加载它。每个部分都将在端点返回。另外,我不能使用硬盘,存储下载的文件。我有InputStream对象,我正在尝试加载如下所示的对象:
inputStream.skip(totalBytes);
long downloadedBytesCount = 0;
ByteArrayOutputStream result = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int length;
do {
length = inputStream.read(buffer);
result.write(buffer, 0, length);
downloadedBytesCount += length;
}
while (downloadedBytesCount <= partOfFileSize && (length != -1));
totalBytes += downloadedBytesCount;
but that code contains problems: each new request will start download file from the begin,so last request for downloading (for example 20 MB) will download all the file (for example 1 GB). So, method skip(long)
doesn't work as I expected.
但该代码包含问题:每个新请求将从开始下载文件,因此上次下载请求(例如20 MB)将下载所有文件(例如1 GB)。因此,方法skip(long)不能像我预期的那样工作。
How can I download file from inputStream
by parts? Any suggestions?
如何通过部件从inputStream下载文件?有什么建议么?
1 个解决方案
#1
1
The standard S3 library can transfer whatever parts of the file you want:
标准S3库可以传输所需文件的任何部分:
(taken from the AWS docs)
(摘自AWS文档)
GetObjectRequest rangeObjectRequest = new GetObjectRequest(
bucketName, key);
rangeObjectRequest.setRange(0, 10); // retrieve 1st 11 bytes.
S3Object objectPortion = s3Client.getObject(rangeObjectRequest);
InputStream objectData = objectPortion.getObjectContent();
In your program you could, for example, read 1000 bytes at a time by moving the range.
例如,在程序中,您可以通过移动范围一次读取1000个字节。
#1
1
The standard S3 library can transfer whatever parts of the file you want:
标准S3库可以传输所需文件的任何部分:
(taken from the AWS docs)
(摘自AWS文档)
GetObjectRequest rangeObjectRequest = new GetObjectRequest(
bucketName, key);
rangeObjectRequest.setRange(0, 10); // retrieve 1st 11 bytes.
S3Object objectPortion = s3Client.getObject(rangeObjectRequest);
InputStream objectData = objectPortion.getObjectContent();
In your program you could, for example, read 1000 bytes at a time by moving the range.
例如,在程序中,您可以通过移动范围一次读取1000个字节。