为多核机器自动设置作业(-j)标志?

I have a Makefile on a machine that has a ton of cores in it, but I always seem to forget to write -jX when compiling my project and it takes way longer than it should.

我有一个机器上有很多内核的Makefile，但是在编译我的项目时，我似乎总是忘记写-jX，而且它花费的时间比应该要长。

Is there some way I can set the -j flag through an environment variable or some other persistent config file so that make will automatically execute multiple jobs in parallel on this machine?

是否有某种方法可以通过环境变量或其他持久配置文件设置-j标志，从而使make能够在这台机器上并行地自动执行多个作业?

10 个解决方案

#1

It appears that the MAKEFLAGS environment variable can pass flags that are part of every make run (at least for GNU make). I haven't had much luck with this myself, but it might be possible to use -l rather than -j to automatically run as many jobs as are appropriate for the number of cores you have available.

看起来MAKEFLAGS环境变量可以传递每个make运行的一部分的标志(至少对于GNU来说)。我自己并没有多少幸运，但是可以使用-l而不是-j来自动运行与您可用的内核数相匹配的作业。

#2

I'm assuming you're using Linux. This is from my ~/.bashrc

我假设你在使用Linux。这是我的~/.bashrc。

# parallel make
export NUMCPUS=`grep -c '^processor' /proc/cpuinfo`
alias pmake='time nice make -j$NUMCPUS --load-average=$NUMCPUS'

sample usage

示例使用

samm@host src> echo $NUMCPUS
8
samm@host src> pmake

becomes time nice make -j8 --load-average=8.

变成时间nice make -j8, load-average=8。

To answer your specific question about putting this into a Makefile, I don't find it practical to sprinkle this logic all over my Makefiles. Putting this into a top level Makefile also isn't a great solution since I often build from sub-directories and wish to build them in parallel as well. However, if you have a fairly flat source hierarchy, it may work for you.

为了回答您关于将其放入Makefile的具体问题，我不认为在我的Makefile中散布这种逻辑是可行的。把它放到一个*的Makefile中也不是一个好的解决方案，因为我经常从子目录构建，并希望并行地构建它们。但是，如果您有一个相当平坦的源代码层次结构，它可能适合您。

#3

As Jeremiah Willcock said, use MAKEFLAGS, but here is how to do it:

正如Jeremiah Willcock所说，使用MAKEFLAGS，但这里是如何做的:

export MAKEFLAGS="-j $(grep -c ^processor /proc/cpuinfo)"

or you could just set a fixed value like this:

或者你可以这样设置一个固定值:

export MAKEFLAGS="-j 8"

If you really want to boost performance you should use ccache by adding something like the following to your Makefile:

如果你真的想提高性能，你应该在Makefile中添加如下内容:

CCACHE_EXISTS := $(shell ccache -V)
ifdef CCACHE_EXISTS
    CC := ccache $(CC)
    CXX := ccache $(CXX)
endif

#4

I usually do this as follows in my bash scripts:

我通常在bash脚本中这样做:

make -j$(nproc)

#5

You can add a line to your Makefile similar to the following:

您可以在Makefile中添加一个类似于以下内容的行:

NUMJOBS=${NUMJOBS:-" -j4 "}

Then add a ${NUMJOBS} line in your rules, or add it into another Makefile var (like MAKEFLAGS). This will use the NUMJOBS envvar, if it exists; if it doesn't, automatically use -j4. You can tune or rename it to your taste.

然后在您的规则中添加一个${NUMJOBS}行，或者将它添加到另一个Makefile var(比如MAKEFLAGS)中。如果存在，这将使用NUMJOBS envvar;如果没有，就自动使用-j4。你可以调整或重命名它到你的口味。

(N.B.: Personally, I'd prefer the default to be -j1 or "", especially if it's going to be distributed to others, because although I have multiple cores also, I compile on many different platforms, and often forget to dis-able the -jX setting.)

(注意::就个人而言，我倾向于默认为-j1或“”，特别是如果它将被分发给其他人，因为虽然我也有多个内核，但我在许多不同的平台上编译，并且经常忘记了-jX设置。

#6

Aliases are not expanded inside scripts. It's better to create a separate make script and place it into one of the $PATH directories:

脚本中不扩展别名。最好创建一个单独的make脚本，并将其放置在$PATH目录中:

#!/bin/sh

if [ -f /proc/cpuinfo ]; then
    CPUS=`grep processor /proc/cpuinfo | wc -l`
else
    CPUS=1
fi
/usr/bin/make -j`expr $CPUS + 1` "$@"

#7

Until sometime in the last year, you could do this in your makefile: (GNU make tested)

直到去年某个时候，您可以在makefile中这样做:(GNU进行测试)

MAKEFLAGS += "-j$(NUM_CORES) -l$(NUM_CORES)

(where NUM_PPROCS is calculated or set according to one of many of the other answers here) And, bam! you have multi-process building going on.

(NUM_PPROCS是根据这里的许多其他答案来计算或设置的)，还有，bam!你正在进行多进程建设。

Given that this has stopped working, the best thing that I could come up with is this, where the makefile calls itself, but with -jX and -lX.

考虑到这已经停止了工作，我能想到的最好的事情是，makefile调用它自己，但是使用-jX和-lX。

# add parallelism equal to number of cores every time.
# it seems that adding -jX to MAKEFLAGS directly doesn't work any more.
# included some "random" strings to ensure uniqueness
ifneq ($(PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB),done)

NUM_CORES ?= $(shell grep -c "vendor_id" /proc/cpuinfo)
MAKEFLAGS +=" -j$(NUM_CORES) -l$(NUM_CORES) "

# for the default target case
parallel_wrapper_default_target_anthsqjkshbeohcbmeuthnoethoaeou:
    $(MAKE) PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB=done

# catches everything else
% :
    $(MAKE) $@ PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB=done

# the match for this else is at the end of the file
else

##### rest of makefile here #####
all: ...
   ...

other_target: ...
    ...

##### etc. #####

endif

#8

On Ubuntu 16.4 using all CPU cores:

使用所有CPU核心的Ubuntu 16.4:

export MAKEFLAGS='-j$(nproc)'

或

export MAKEFLAGS='-j 2'

#9

At the beginning of a Makefile:

在Makefile的开头:

MAKEFLAGS+="j"

It won't take numeric arguments for any version of GNU Make before 4.2.

在4.2之前，它不会对任何版本的GNU进行数字参数。

If you happen to have some jobs running out of memory, you could make them run only one at a time with flock from util-linux-ng. For example, a convert utility from ImageMagick will use all resources it can get when used to optimize images according to Google's recommendations, so there's no point to have it run in parallel.

如果您碰巧有一些内存不足的工作，您可以让它们一次只运行一个，从util-linux-ng开始。例如，来自ImageMagick的转换实用程序将使用它所能得到的所有资源，根据谷歌的建议来优化图像，所以没有必要让它并行运行。

%.min.jpg: %.jpg
    @flock --wait 600 Makefile convert $< -sampling-factor 4:2:0 -strip $@

It is important to set a long wait time because make will still run most of these commands in parallel. Therefore, wait time must be as such as the deepest queue execution time. If you have eight cores, and eight large images take a minute to optimize, you must set the wait time to at least a minute.

设置一个漫长的等待时间很重要，因为make仍然可以并行地运行这些命令。因此，等待时间必须像最深入的队列执行时间一样。如果您有8个核心，8个大型图像需要一分钟来优化，那么您必须将等待时间设置为至少一分钟。

With hundreds of cores and huge images, six hundred seconds set above might not be enough.

有几百个核心和巨大的图像，超过600秒可能还不够。

#10

-1

I would just put

我只会把

alias make='make -j'

in ~/.profile or ~/.bashrc.

在~ /。配置文件或~ / . bashrc。

According to the manual:

根据手册:

If there is nothing looking like an integer after the ‘-j’ option, there is no limit on the number of job slots.

如果在“-j”选项之后没有任何看起来像整数的整数，那么工作槽的数量是没有限制的。

#1

#2