We have several cron jobs that ftp proxy logs to a centralized server. These files can be rather large and take some time to transfer. Part of the requirement of this project is to provide a logging mechanism in which we log the success or failure of these transfers. This is simple enough.
我们有几个cron作业,ftp代理日志到*服务器。这些文件可能相当大,需要一些时间来传输。该项目的部分要求是提供一种记录机制,我们记录这些转移的成功或失败。这很简单。
My question is, is there a way to check if a file is currently being written to? My first solution was to just check the file size twice within a given timeframe and check the file size. But a co-worker said that there may be able to hook into the EXT3 file system via python and check the attributes to see if the file is currently being appended to. My Google-Fu came up empty.
我的问题是,有没有办法检查当前是否正在写入文件?我的第一个解决方案是在给定的时间范围内检查文件大小两次并检查文件大小。但是同事说可以通过python挂钩到EXT3文件系统并检查属性以查看当前是否附加了该文件。我的Google-Fu空了。
Is there a module for EXT3 or something else that would allow me to check the state of a file? The server is running Fedora Core 9 with EXT3 file system.
是否有一个EXT3模块或其他可以让我检查文件状态的模块?服务器正在运行带有EXT3文件系统的Fedora Core 9。
2 个解决方案
#1
no need for ext3-specific hooks; just check lsof, or more exactly, /proc/<pid>/fd/* and /proc/<pid>/fdinfo/* (that's where lsof gets it's info, AFAICT). There you can check if the file is open, if it's writeable, and the 'cursor' position.
不需要ext3特定的钩子;只需检查lsof,或者更确切地说,检查/ proc /
That's not the whole picture; but any more is done in processpace by stdlib on the writing process, as most writes are buffered and the kernel only sees bigger chunks of data, so any 'ext3-aware' monitor wouldn't get that either.
那不是全局;但是在写入过程中,stdlib在进程空间中完成了更多,因为大多数写入都是缓冲的,内核只能看到更大的数据块,所以任何“ext3-aware”监视器都不会得到。
#2
There's no ext3 hooks to check what you'd want directly.
没有ext3钩子可以直接检查你想要的东西。
- I suppose you could dig through the source code of Fuser linux command, replicate the part that finds which process owns a file, and watch that resource. When noone longer has the file opened, it's done transferring.
我想您可以挖掘Fuser linux命令的源代码,复制找到哪个进程拥有文件的部分,并观察该资源。当没有人打开文件时,它就完成了转移。
Another approach:
- Your cron jobs should tell that they're finished.
你的cron工作应该告诉他们已经完成了。
We have our cron jobs that transport files just write an empty filename.finished after it's transferred the filename. Another approach is to transfer them to a temporary filename, e.g. filename.part and then rename it to filename Renaming is atomic. In both cases you check repeatedly until the presence of filename or filename.finished
我们有我们的cron作业传输文件只写一个空文件名。在传输文件名后完成。另一种方法是将它们转移到临时文件名,例如filename.part然后将其重命名为filename重命名是原子的。在这两种情况下,您都会反复检查,直到存在filename或filename.finished
#1
no need for ext3-specific hooks; just check lsof, or more exactly, /proc/<pid>/fd/* and /proc/<pid>/fdinfo/* (that's where lsof gets it's info, AFAICT). There you can check if the file is open, if it's writeable, and the 'cursor' position.
不需要ext3特定的钩子;只需检查lsof,或者更确切地说,检查/ proc /
That's not the whole picture; but any more is done in processpace by stdlib on the writing process, as most writes are buffered and the kernel only sees bigger chunks of data, so any 'ext3-aware' monitor wouldn't get that either.
那不是全局;但是在写入过程中,stdlib在进程空间中完成了更多,因为大多数写入都是缓冲的,内核只能看到更大的数据块,所以任何“ext3-aware”监视器都不会得到。
#2
There's no ext3 hooks to check what you'd want directly.
没有ext3钩子可以直接检查你想要的东西。
- I suppose you could dig through the source code of Fuser linux command, replicate the part that finds which process owns a file, and watch that resource. When noone longer has the file opened, it's done transferring.
我想您可以挖掘Fuser linux命令的源代码,复制找到哪个进程拥有文件的部分,并观察该资源。当没有人打开文件时,它就完成了转移。
Another approach:
- Your cron jobs should tell that they're finished.
你的cron工作应该告诉他们已经完成了。
We have our cron jobs that transport files just write an empty filename.finished after it's transferred the filename. Another approach is to transfer them to a temporary filename, e.g. filename.part and then rename it to filename Renaming is atomic. In both cases you check repeatedly until the presence of filename or filename.finished
我们有我们的cron作业传输文件只写一个空文件名。在传输文件名后完成。另一种方法是将它们转移到临时文件名,例如filename.part然后将其重命名为filename重命名是原子的。在这两种情况下,您都会反复检查,直到存在filename或filename.finished