Java：使用SFTP检查文件是否已完全复制

时间：2022-01-07 01:22:29

I have a poller running on a certain directory every 35s. The files are placed in this directory through a SFTP server. The problem is whenever the polling conflicts with the time when a file is being copied. It picks the incomplete file also which is not yet copied completely.

我有一个轮询器每35秒在某个目录上运行。这些文件通过SFTP服务器放在此目录中。问题在于轮询与复制文件的时间冲突。它还会选择尚未完全复制的不完整文件。

Can we know the status of a file whether it is in copying mode or copied mode?

我们可以知道文件的状态，无论是处于复制模式还是复制模式？

2 个解决方案

#1

2  

There are several common strategies for file watchers to "know" a file is completely transferred

文件监视器有几种常见的策略来“知道”文件被完全传输

  1. Poll with time interval, and treat the file to be completely transferred if file size is not changing within an interval. e.g. watch for file existence every 1 minute. Once you see the file exists, monitor its size for every 5 seconds. If file size stays constant for 30 seconds, then treat it as completely transferred.

    轮询时间间隔，如果文件大小在一个间隔内没有变化，则将文件视为完全传输。例如每1分钟观察文件存在。一旦看到文件存在，就每隔5秒监控一次。如果文件大小保持不变30秒，则将其视为完全传输。

  2. Have the transfer process create a tagging file after file transfer. e.g. After it completed transferring the file FOO.txt, create an empty FOO.txt.tag. Your file watcher is going to check for existence of FOO.txt.tag and once it exists, you know FOO.txt has been completely transferred

    传输过程在文件传输后创建标记文件。例如完成传输文件FOO.txt后，创建一个空的FOO.txt.tag。您的文件观察者将检查FOO.txt.tag是否存在，一旦它存在，您就知道FOO.txt已被完全转移

  3. In some special cases that the file is having special format (e.g. a special footer line) then your file watcher can poll the file and see the last lines, and see if they match with the desired pattern

    在某些特殊情况下，文件具有特殊格式（例如，特殊页脚行），然后您的文件观察程序可以轮询文件并查看最后一行，并查看它们是否与所需模式匹配

Each method has its pros and cons:

每种方法都有其优点和缺点：

  1. Method 1 affects the transfer process least. Sometimes files are transferred by 3rd party that you have almost no way to tell them to create the tag file as in method 2. However you can tell this method is not 100% reliable, especially under poor network.
  2. 方法1最少影响转移过程。有时文件是由第三方传输的，你几乎没有办法告诉他们如方法2那样创建标记文件。但是你可以说这种方法不是100％可靠，特别是在网络不佳的情况下。
  3. Method 2 is the most reliable. However, as said before, there are cases that you have no control on the transfer process
  4. 方法2是最可靠的。但是，如前所述，有些情况下您无法控制转移过程
  5. Method 3 is only applicable to special cases
  6. 方法3仅适用于特殊情况

Choose the one that suit your need

选择适合您需求的产品

#2

1  

Have the poller note file sizes. If the size did not change from one round to the next, the file is done downloading.

让轮询器文件大小。如果大小没有从一轮变为下一轮，则文件下载完成。

Can you influence the SFTP server? Can it create a marker file once the download is complete (e.g. '.thisIsAFile.doc.done')?

你能影响SFTP服务器吗？下载完成后可以创建标记文件（例如'.thisIsAFile.doc.done'）吗？

#1

2  

There are several common strategies for file watchers to "know" a file is completely transferred

文件监视器有几种常见的策略来“知道”文件被完全传输

  1. Poll with time interval, and treat the file to be completely transferred if file size is not changing within an interval. e.g. watch for file existence every 1 minute. Once you see the file exists, monitor its size for every 5 seconds. If file size stays constant for 30 seconds, then treat it as completely transferred.

    轮询时间间隔，如果文件大小在一个间隔内没有变化，则将文件视为完全传输。例如每1分钟观察文件存在。一旦看到文件存在，就每隔5秒监控一次。如果文件大小保持不变30秒，则将其视为完全传输。

  2. Have the transfer process create a tagging file after file transfer. e.g. After it completed transferring the file FOO.txt, create an empty FOO.txt.tag. Your file watcher is going to check for existence of FOO.txt.tag and once it exists, you know FOO.txt has been completely transferred

    传输过程在文件传输后创建标记文件。例如完成传输文件FOO.txt后，创建一个空的FOO.txt.tag。您的文件观察者将检查FOO.txt.tag是否存在，一旦它存在，您就知道FOO.txt已被完全转移

  3. In some special cases that the file is having special format (e.g. a special footer line) then your file watcher can poll the file and see the last lines, and see if they match with the desired pattern

    在某些特殊情况下，文件具有特殊格式（例如，特殊页脚行），然后您的文件观察程序可以轮询文件并查看最后一行，并查看它们是否与所需模式匹配

Each method has its pros and cons:

每种方法都有其优点和缺点：

  1. Method 1 affects the transfer process least. Sometimes files are transferred by 3rd party that you have almost no way to tell them to create the tag file as in method 2. However you can tell this method is not 100% reliable, especially under poor network.
  2. 方法1最少影响转移过程。有时文件是由第三方传输的，你几乎没有办法告诉他们如方法2那样创建标记文件。但是你可以说这种方法不是100％可靠，特别是在网络不佳的情况下。
  3. Method 2 is the most reliable. However, as said before, there are cases that you have no control on the transfer process
  4. 方法2是最可靠的。但是，如前所述，有些情况下您无法控制转移过程
  5. Method 3 is only applicable to special cases
  6. 方法3仅适用于特殊情况

Choose the one that suit your need

选择适合您需求的产品

#2

1  

Have the poller note file sizes. If the size did not change from one round to the next, the file is done downloading.

让轮询器文件大小。如果大小没有从一轮变为下一轮，则文件下载完成。

Can you influence the SFTP server? Can it create a marker file once the download is complete (e.g. '.thisIsAFile.doc.done')?

你能影响SFTP服务器吗？下载完成后可以创建标记文件（例如'.thisIsAFile.doc.done'）吗？

标签：

相关文章