I have a complex qsub command to run remotely.
我有一个复杂的qsub命令可以远程运行。
PROJECT_NAME_TEXT="TEST PROJECT"
PACK_ORGANIZATION="--source-organization \'MY, ORGANIZATION\'"
CONTACT_NAME="--contact-name \'Tom Riddle\'"
PROJECT_NAME_PACK="--project-name \"${PROJECT_NAME_TEXT}\""
INPUTARGS="${PACK_ORGANIZATION} ${CONTACT_NAME} ${PROJECT_NAME_PACK}"
ssh mycluster "qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
The problem is the remote cluster doesn't recognise the qsub command, it always showing incorrect qsub command or simply alway queued on cluster because of input args are wrong.
问题是远程集群不能识别qsub命令，它总是显示不正确的qsub命令，或者仅仅因为输入args而在集群上排队。
It must be the escaping problem, my question is how to escape the command above properly ?
这一定是转义问题，我的问题是如何正确转义上面的命令?
4 个解决方案
#1
2
Try doing this using a here-doc : you have a quote conflict (nested double quotes that is an error):
使用here-doc试试这样做:您有一个引用冲突(嵌套的双引号是错误的):
#!/bin/bash
PROJECT_NAME_TEXT="TEST PROJECT"
PACK_ORGANIZATION="--source-organization \'MY, ORGANIZATION\'"
CONTACT_NAME="--contact-name \'Tom Riddle\'"
PROJECT_NAME_PACK="--project-name \"${PROJECT_NAME_TEXT}\""
INPUTARGS="${PACK_ORGANIZATION} ${CONTACT_NAME} ${PROJECT_NAME_PACK}"
ssh mycluster <<EOF
qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script
EOF
As you can see, here-docs are really helpful for inputs with quotes.
正如您所看到的，这里的文档对于带引号的输入非常有用。
See man bash | less +/'Here Documents'
见man bash |减少+/'Here文档'
Edit
from your comments :
从你的评论:
I used this method but it gives me "Pseudo-terminal will not be allocated because stdin is not a terminal."
我使用了这个方法，但它给了我“不会分配伪终端，因为stdin不是终端”。
You can ignore this warning with
你可以忽略这个警告
ssh mycluster <<EOF 2>/dev/null
(try the -t switch for ssh if needed)
(如果需要，尝试ssh的-t开关)
If you have
如果你有
-bash: line 2: EOF: command not found-bash:第2行:EOF:命令未找到
I think you have a copy paste problem. Try to remove extra spaces on all end lines
我认为你有复制粘贴的问题。尝试删除所有终端线上的额外空格
And it seems this method cannot pass local variable
$INPUTARGSto the remote cluster而且这个方法似乎不能将本地变量$INPUTARGS传递给远程集群
it seems related to your EOF problem.
这似乎和你的问题有关。
$argvreturns nothing on remote cluster$argv在远程集群上不返回任何内容。
What does this means ? $argv is not a pre-defined variable in bash. If you need to list command line arguments, use the pre-defined variable $@
这意味着什么?$argv不是bash中的预定义变量。如果需要列出命令行参数，请使用预定义的变量$@
Last thing : ensure you are using bash
最后一件事:确保您正在使用bash
#2
1
Your problem is not the length, but the nesting of your quotes - in this line, you are trying to use " inside ", which won't work:
你的问题不在于长度，而在于引号的嵌套——在这一行中，你试图使用“inside”，这是行不通的:
ssh mycluster "qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
Bash will see this as "qsub -v argv=" followed by $INPUTARGS (not quoted), followed by " -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script".
Bash会将其视为“qsub -v argv=”，然后是$INPUTARGS(未引用)，然后是“-l walltime=10:00:00 -l vmem=8GB -l node =1:ppn=4 /myscript_path/run.script”。
It's possible that backslash-escaping those inner quotes will have the desired effect, but nesting quotes in bash can get rather confusing. What I often try to do is add an echo at the beginning of the command, to show how the various stages of expansion pan out. e.g.
反斜杠-转义这些内部引号可能会产生预期的效果，但是bash中的嵌套引号可能会变得相当混乱。我经常尝试在命令的开头添加一个echo，以显示扩展的各个阶段是如何实现的。如。
echo 'As expanded locally:'
echo ssh mycluster "qsub -v argv=\"$INPUTARGS\" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
echo 'As expanded remotely:'
ssh mycluster "echo qsub -v argv=\"$INPUTARGS\" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
#3
0
Thanks for all the answers, however their methods will not work on my case. I have to answer this by myself since this problem is pretty complex, I got the clue from existing solutions in *.
谢谢你所有的回答，但是他们的方法对我的案子不起作用。我必须自己回答这个问题，因为这个问题相当复杂，我从*的现有解决方案中得到了线索。
There are 2 problems must be solved in my case.
在我的案件中有两个问题必须解决。
-
Pass local program's parameters to the remote cluster. Here-doc solution doesn't work in this case.
将本地程序的参数传递给远程集群。这里的doc解决方案在这里不起作用。
-
Run qsub on remote cluster with the a long variable as arguments that contain quote symbol.
在远程集群上运行qsub，使用一个长变量作为包含引号符号的参数。
Problem 1.
问题1。
Firstly, I have to introduce my script that runs on local machine takes parameters like this:
首先，我必须介绍我的脚本，它运行在本地机器上，获取如下参数:
scripttoberunoncluster.py --source-organisation "My_organization_my_department" --project-name "MyProjectName" --processes 4 /targetoutputfolder/
The real parameter is far more longer than above, so all the parameter must be sent to remote. They are sent in file like this:
真正的参数比上面的要长得多，所以所有的参数都必须发送到远程。它们被归档如下:
PROJECT_NAME="MyProjectName"
PACK_ORGANIZATION="--source-organization '\\\"My_organization_my_department\\\"'" # multiple layers of escaping, remove all the spaces
PROJECT_NAME_PACK="--project-name '\\\"${PROJECT_NAME}\\\"'"
PROCESSES_="--processes 4"
TARGET_FOLDER_PACK="/targetoutputfolder/"
INPUTARGS="${PACK_ORGANIZATION} ${PROJECT_NAME_PACK} ${PROCESSES} ${TARGET_FOLDER_PACK}"
echo $INPUTARGS > "TempPath/temp.par"
scp "TempPath/temp.par" "remotecluster:/remotepath/"
My solution is sort of compromising. But in this way the remote cluster can run script with arguments contain quote symbol. If you don't put all your variable (as parameters) in a file and transfer it to remote cluster, no matter how you pass them into variable, the quote symbol will be removed.
我的解决办法是妥协。但是，通过这种方式，远程集群可以使用包含引号符号的参数运行脚本。如果不将所有变量(作为参数)放在一个文件中并将其转移到远程集群中，无论如何将它们传递到变量中，引号符号将被删除。
Problem 2.
问题2。
Check how the qsub runs on remote cluster.
检查qsub如何在远程集群上运行。
ssh remotecluster "qsub -v argv=\"`cat /remotepath/temp.par`\" -l walltime=10:00:00 /remotepath/my.script"
And in the my.script:
和my.script:
INPUT_ARGS=`echo $argv`
python "/pythonprogramlocation/scripttoberunoncluster.py" $INPUT_ARGS ; #note: $INPUT_ARGS hasn't quote
#4
0
The described escaping problem consists in the requirement to preserve final quotes around arguments after two evaluation processes, i. e. after two evaluations we should see something like:
所描述的转义问题是要求在两个评估过程之后，在参数周围保留最终的引号，即在两个评估过程之后，我们应该看到如下内容:
--source-organization "My_organization_my_department" --project-name "MyProjectName" --processes 4 /targetoutputfolder/
This can be achieved codewise by first putting each argument in a separate variable and then enclosing the argument with single quotes while making sure that possible single quotes inside the argument string get "escaped" with '\'' (in fact, the argument will be split up into separate strings but, when used, the split-up argument will automatically get re-concatenated by the string evaluation mechanism of UNIX (POSIX?) shells). And this procedure has to be repeated three times.
这可以实现codewise首先将每个参数在一个单独的变量,然后用单引号封闭的观点虽然确保可能单引号内的参数字符串“逃脱”与“\”(事实上,论证将分成了两个字符串,但使用时,分割参数会自动被绳子re-concatenated评价机制的UNIX(POSIX ?)壳)。这个过程必须重复三次。
{
escsquote="'\''"
PROJECT_NAME="MyProjectName"
myorg="My_organization_my_department"
myorg="'${myorg//\'/${escsquote}}'" # bash
myorg="'${myorg//\'/${escsquote}}'"
myorg="'${myorg//\'/${escsquote}}'"
PACK_ORGANIZATION="--source-organization ${myorg}"
pnp="${PROJECT_NAME}"
pnp="'${pnp//\'/${escsquote}}'"
pnp="'${pnp//\'/${escsquote}}'"
pnp="'${pnp//\'/${escsquote}}'"
PROJECT_NAME_PACK="--project-name ${pnp}"
PROCESSES="--processes 4"
TARGET_FOLDER_PACK="/targetoutputfolder/"
INPUTARGS="${PACK_ORGANIZATION} ${PROJECT_NAME_PACK} ${PROCESSES} ${TARGET_FOLDER_PACK}"
echo "$INPUTARGS"
eval echo "$INPUTARGS"
eval eval echo "$INPUTARGS"
echo
ssh -T localhost <<EOF
echo qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script
EOF
}
For further information please see:
有关详情，请参阅:
- Quotes exercise - how to do ssh inside ssh whilst running sql inside second ssh?
- 引用练习——如何在ssh中执行ssh，同时在第二个ssh中运行sql ?
- Quoting in ssh $host $FOO and ssh $host "sudo su user -c $FOO" type constructs.
- 引用ssh $host $FOO和ssh $host“sudo su用户-c $FOO”类型构造。
#1
2
Try doing this using a here-doc : you have a quote conflict (nested double quotes that is an error):
使用here-doc试试这样做:您有一个引用冲突(嵌套的双引号是错误的):
#!/bin/bash
PROJECT_NAME_TEXT="TEST PROJECT"
PACK_ORGANIZATION="--source-organization \'MY, ORGANIZATION\'"
CONTACT_NAME="--contact-name \'Tom Riddle\'"
PROJECT_NAME_PACK="--project-name \"${PROJECT_NAME_TEXT}\""
INPUTARGS="${PACK_ORGANIZATION} ${CONTACT_NAME} ${PROJECT_NAME_PACK}"
ssh mycluster <<EOF
qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script
EOF
As you can see, here-docs are really helpful for inputs with quotes.
正如您所看到的，这里的文档对于带引号的输入非常有用。
See man bash | less +/'Here Documents'
见man bash |减少+/'Here文档'
Edit
from your comments :
从你的评论:
I used this method but it gives me "Pseudo-terminal will not be allocated because stdin is not a terminal."
我使用了这个方法，但它给了我“不会分配伪终端，因为stdin不是终端”。
You can ignore this warning with
你可以忽略这个警告
ssh mycluster <<EOF 2>/dev/null
(try the -t switch for ssh if needed)
(如果需要，尝试ssh的-t开关)
If you have
如果你有
-bash: line 2: EOF: command not found-bash:第2行:EOF:命令未找到
I think you have a copy paste problem. Try to remove extra spaces on all end lines
我认为你有复制粘贴的问题。尝试删除所有终端线上的额外空格
And it seems this method cannot pass local variable
$INPUTARGSto the remote cluster而且这个方法似乎不能将本地变量$INPUTARGS传递给远程集群
it seems related to your EOF problem.
这似乎和你的问题有关。
$argvreturns nothing on remote cluster$argv在远程集群上不返回任何内容。
What does this means ? $argv is not a pre-defined variable in bash. If you need to list command line arguments, use the pre-defined variable $@
这意味着什么?$argv不是bash中的预定义变量。如果需要列出命令行参数，请使用预定义的变量$@
Last thing : ensure you are using bash
最后一件事:确保您正在使用bash
#2
1
Your problem is not the length, but the nesting of your quotes - in this line, you are trying to use " inside ", which won't work:
你的问题不在于长度，而在于引号的嵌套——在这一行中，你试图使用“inside”，这是行不通的:
ssh mycluster "qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
Bash will see this as "qsub -v argv=" followed by $INPUTARGS (not quoted), followed by " -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script".
Bash会将其视为“qsub -v argv=”，然后是$INPUTARGS(未引用)，然后是“-l walltime=10:00:00 -l vmem=8GB -l node =1:ppn=4 /myscript_path/run.script”。
It's possible that backslash-escaping those inner quotes will have the desired effect, but nesting quotes in bash can get rather confusing. What I often try to do is add an echo at the beginning of the command, to show how the various stages of expansion pan out. e.g.
反斜杠-转义这些内部引号可能会产生预期的效果，但是bash中的嵌套引号可能会变得相当混乱。我经常尝试在命令的开头添加一个echo，以显示扩展的各个阶段是如何实现的。如。
echo 'As expanded locally:'
echo ssh mycluster "qsub -v argv=\"$INPUTARGS\" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
echo 'As expanded remotely:'
ssh mycluster "echo qsub -v argv=\"$INPUTARGS\" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script"
#3
0
Thanks for all the answers, however their methods will not work on my case. I have to answer this by myself since this problem is pretty complex, I got the clue from existing solutions in *.
谢谢你所有的回答，但是他们的方法对我的案子不起作用。我必须自己回答这个问题，因为这个问题相当复杂，我从*的现有解决方案中得到了线索。
There are 2 problems must be solved in my case.
在我的案件中有两个问题必须解决。
-
Pass local program's parameters to the remote cluster. Here-doc solution doesn't work in this case.
将本地程序的参数传递给远程集群。这里的doc解决方案在这里不起作用。
-
Run qsub on remote cluster with the a long variable as arguments that contain quote symbol.
在远程集群上运行qsub，使用一个长变量作为包含引号符号的参数。
Problem 1.
问题1。
Firstly, I have to introduce my script that runs on local machine takes parameters like this:
首先，我必须介绍我的脚本，它运行在本地机器上，获取如下参数:
scripttoberunoncluster.py --source-organisation "My_organization_my_department" --project-name "MyProjectName" --processes 4 /targetoutputfolder/
The real parameter is far more longer than above, so all the parameter must be sent to remote. They are sent in file like this:
真正的参数比上面的要长得多，所以所有的参数都必须发送到远程。它们被归档如下:
PROJECT_NAME="MyProjectName"
PACK_ORGANIZATION="--source-organization '\\\"My_organization_my_department\\\"'" # multiple layers of escaping, remove all the spaces
PROJECT_NAME_PACK="--project-name '\\\"${PROJECT_NAME}\\\"'"
PROCESSES_="--processes 4"
TARGET_FOLDER_PACK="/targetoutputfolder/"
INPUTARGS="${PACK_ORGANIZATION} ${PROJECT_NAME_PACK} ${PROCESSES} ${TARGET_FOLDER_PACK}"
echo $INPUTARGS > "TempPath/temp.par"
scp "TempPath/temp.par" "remotecluster:/remotepath/"
My solution is sort of compromising. But in this way the remote cluster can run script with arguments contain quote symbol. If you don't put all your variable (as parameters) in a file and transfer it to remote cluster, no matter how you pass them into variable, the quote symbol will be removed.
我的解决办法是妥协。但是，通过这种方式，远程集群可以使用包含引号符号的参数运行脚本。如果不将所有变量(作为参数)放在一个文件中并将其转移到远程集群中，无论如何将它们传递到变量中，引号符号将被删除。
Problem 2.
问题2。
Check how the qsub runs on remote cluster.
检查qsub如何在远程集群上运行。
ssh remotecluster "qsub -v argv=\"`cat /remotepath/temp.par`\" -l walltime=10:00:00 /remotepath/my.script"
And in the my.script:
和my.script:
INPUT_ARGS=`echo $argv`
python "/pythonprogramlocation/scripttoberunoncluster.py" $INPUT_ARGS ; #note: $INPUT_ARGS hasn't quote
#4
0
The described escaping problem consists in the requirement to preserve final quotes around arguments after two evaluation processes, i. e. after two evaluations we should see something like:
所描述的转义问题是要求在两个评估过程之后，在参数周围保留最终的引号，即在两个评估过程之后，我们应该看到如下内容:
--source-organization "My_organization_my_department" --project-name "MyProjectName" --processes 4 /targetoutputfolder/
This can be achieved codewise by first putting each argument in a separate variable and then enclosing the argument with single quotes while making sure that possible single quotes inside the argument string get "escaped" with '\'' (in fact, the argument will be split up into separate strings but, when used, the split-up argument will automatically get re-concatenated by the string evaluation mechanism of UNIX (POSIX?) shells). And this procedure has to be repeated three times.
这可以实现codewise首先将每个参数在一个单独的变量,然后用单引号封闭的观点虽然确保可能单引号内的参数字符串“逃脱”与“\”(事实上,论证将分成了两个字符串,但使用时,分割参数会自动被绳子re-concatenated评价机制的UNIX(POSIX ?)壳)。这个过程必须重复三次。
{
escsquote="'\''"
PROJECT_NAME="MyProjectName"
myorg="My_organization_my_department"
myorg="'${myorg//\'/${escsquote}}'" # bash
myorg="'${myorg//\'/${escsquote}}'"
myorg="'${myorg//\'/${escsquote}}'"
PACK_ORGANIZATION="--source-organization ${myorg}"
pnp="${PROJECT_NAME}"
pnp="'${pnp//\'/${escsquote}}'"
pnp="'${pnp//\'/${escsquote}}'"
pnp="'${pnp//\'/${escsquote}}'"
PROJECT_NAME_PACK="--project-name ${pnp}"
PROCESSES="--processes 4"
TARGET_FOLDER_PACK="/targetoutputfolder/"
INPUTARGS="${PACK_ORGANIZATION} ${PROJECT_NAME_PACK} ${PROCESSES} ${TARGET_FOLDER_PACK}"
echo "$INPUTARGS"
eval echo "$INPUTARGS"
eval eval echo "$INPUTARGS"
echo
ssh -T localhost <<EOF
echo qsub -v argv="$INPUTARGS" -l walltime=10:00:00 -l vmem=8GB -l nodes=1:ppn=4 /myscript_path/run.script
EOF
}
For further information please see:
有关详情，请参阅:
- Quotes exercise - how to do ssh inside ssh whilst running sql inside second ssh?
- 引用练习——如何在ssh中执行ssh，同时在第二个ssh中运行sql ?
- Quoting in ssh $host $FOO and ssh $host "sudo su user -c $FOO" type constructs.
- 引用ssh $host $FOO和ssh $host“sudo su用户-c $FOO”类型构造。