准备环境
(个人使用记录,方便下次使用查阅~~)
首先保证PAI是登陆状态:
进入GitHub项目所在地址:
https://github.com/Microsoft/pai/
然后切换分支到
具体 使用文档再这里:
https://github.com/Microsoft/pai/tree/pai-0.5.y/pai-fs
将代码clone下来后执行:
`python pai-fs.py --config host=xxx.xxx.234.79 port=50070 user=root`(配置Server)
上传数据到HDFS
进到pai-fs.py目录下(也可以直接绝对路径),然后通过 :
python pai-fs.py + 参数 +路径
进行数据上传
具体使用方法:
(Github README.md)
example use:
pai-fs --config host=10.0.3.9 port=50070 user=root -- store hdfs config
pai-fs -ls hdfs:// -- list the contents of a root HDFS directory
pai-fs -ls hdfs:// --host 10.0.3.9 -- list the contents of a root HDFS directory with host specified
pai-fs -ls hdfs:// --host 10.0.3.9 --port 50070 --user root -- list the contents of a root HDFS directory with host, port and user specified
pai-fs -ls -r hdfs:// -- list the contents of a root HDFS directory, recursively
pai-fs -mkdir hdfs://mydir/mysubdir/mysubdir2 -- makes mysubdir2 and all directories along the way
pai-fs -rm hdfs://mydir/mysubdir/myfile -- removes myfile from mysubdir
pai-fs -rm hdfs://mydir/mysubdir -- removes mysubdir and all files and directories in it
pai-fs -cp c:\mylocalfile hdfs://mydir/myremotedir -- copy mylocalfile into myremotedir
pai-fs -cp -r c:\mylocaldir hdfs://mydir/myremotedir -- copy mylocaldir into myremotedir, recursively
pai-fs -cp -r c:\mylocaldir\* hdfs://mydir/myremotedir -- copy mylocaldir's contents into myremotedir, recursively
pai-fs -cp c:\mylocaldir\\a hdfs://mydir/myremotedir/b -- copy file a from mylocaldir to myremotedir and rename to b
pai-fs -cp -r hdfs://mydir/myremotedir c:\mylocaldir -- copy myremotedir into mylocaldir, recursively
pai-fs -cp -r hdfs://mydir/myremotedir/* c:\mylocaldir -- copy myremotedir's contents into mylocaldir, recursively
exit code:
0 -- Success
1 -- An exception happened during the operation including bad connection
2 -- PAI_VC environment variable not set to valid VC or insufficient/invalid command line argument(s)
3 -- Path not found
4 -- Unauthorized access
5 -- Path not empty
6 -- Check failed after operation
100 -- Failed to copy too many times
101 -- Failed to concat chunks into file
使用演示
执行命令后,开始上传