Linux NFS客户端需要很小心地配置,否则在NFS服务器崩溃时,访问NFS的程序会被挂起,用ps查看,进程状态(STAT)处于D,意为(由于IO阻塞而进入)不可中断睡眠(如果是D+,+号表示程序运行于前台进程组)[1]。
为此,最新的Hadoop HA方案Quorum-Journal的作者在设计文档中对NFS共享存储保存NameNode元数据的作法如下诟病[2]:
- Custom hardware - the hardware requirements of a NAS device and remotely controllable PDU can be expensive, and also may be different than the standard deployments used elsewhere within some "filer-free" organizations.
- Complicated deployment - even after HDFS is installed, the administrator must take extra steps to congure NFS mounts, custom fencing scripts, etc. This complicates HA deployment and may even cause unavailability if misconfigured.
- Poor NFS client implementations - in many versions of Linux, NFS client implementations can be buggy, or easy to misconfigure. For example, it is easy for an administrator to misconfigure mount options in such a way that the NameNodes will freeze unrecoverably in some outage scenarios.
此处第3点就是这次需要调研的内容:NFS客户端正确配置以防程序被长时间阻塞。
1. Hard mount vs Soft mount[3][4]
挂载NFS目录时可以使用hard或soft参数指定client如何处理server的异常(server崩溃或网络连接异常);它们的区别是:
- soft:如果一个请求失败,client会将错误返回给发起请求的进程;
- hard:如果一个请求失败,client会在后台无限重试,直到server从异常恢复;发起请求的进程会被阻塞;默认会采用这个做法;
注:soft挂载可能导致隐性的数据损坏,因而应该在客户端响应性比数据完整性更重要的前提下使用该参数;通过增大retrans参数增加重试次数可以削减soft参数带来的风险;
2. timeo and retrans[3]
soft超时失败由timeo和retrans参数决定:
- timeo=n设定一次尝试超时时间为0.n秒,默认60秒;
- retrans=n设定失败后重试n次,默认3次,仅soft时有效;
通过测试(在client挂载时指定参数,然后将server进程关闭,使用ls访问nfs目标),发现:
- 不指定soft或hard,测试后进程阻塞进入D+状态,必须被kill -9杀死;(尝试重启server机器,结果相同)
- 指定hard,与上面結果相同;
- 指定soft,不设定timeo和retrans,与上面結果相同,一分钟无响应;
- 指定soft,设定timeo和retrans比较小的数值,很快返回错误:Input/output error
超时时间测试:
timeo=1 | timeo=2 | timeo=3 | timeo=4 | timeo=5 | |
retrans=1 | 0.3s | 0.85s | 1.2s | 1.88s | 2s |
retrans=2 | 0.3s | 0.85s | 1.2s | 1.88s | 2s |
retrans=3 | 0.85s | 1.8s | 2.8s | 3s | 3.7s |
retrans=4 | 0.85s | 1.8s | 2.8s | 3s | 3.7s |
retrans=5 | 0.85s | 1.8s | 2.8s | 3s | 3.7s |
参考:
[1]man 1 ps:http://linux.die.net/man/1/ps
[2]Quorum-Journal Design:https://issues.apache.org/jira/secure/attachment/12532989/qjournal-design.pdf
[3]man 5 nfs:http://linux.die.net/man/5/nfs
[4]NFS-HOWTO client:http://tldp.org/HOWTO/NFS-HOWTO/client.html