Maintainers:
Rusty Russell <rusty@rustycorp.com.au>
Srivatsa Vaddagiri <vatsa@in.ibm.com>
i386:
Zwane Mwaikambo <zwane@arm.linux.org.uk>
ppc64:
Nathan Lynch <nathanl@austin.ibm.com>
Joel Schopp <jschopp@austin.ibm.com>
ia64/x86_64:
Ashok Raj <ashok.raj@intel.com>
s390:
Heiko Carstens <heiko.carstens@de.ibm.com>
Authors: Ashok Raj <ashok.raj@intel.com>
Lots of feedback: Nathan Lynch <nathanl@austin.ibm.com>,
Joel Schopp <jschopp@austin.ibm.com>
Introduction
Modern advances in system architectures have introduced advanced error
reporting and correction capabilities in processors. CPU architectures permit
partitioning support, where compute resources of a single CPU could be made
available to virtual machine environments. There are couple OEMS that
support NUMA hardware which are hot pluggable as well, where physical
node insertion and removal require support for CPU hotplug.
Such advances require CPUs available to a kernel to be removed either for
provisioning reasons, or for RAS purposes to keep an offending CPU off
system execution path. Hence the need for CPU hotplug support in the
Linux kernel.
A more novel use of CPU-hotplug support is its use today in suspend
resume support for SMP. Dual-core and HT support makes even
a laptop run SMP kernels which didn't support these methods. SMP support
for suspend/resume is a work in progress.
现 在，在一些先进的体系结构中处理器引进了一些先进的错误报告和错误修正能力。CPU体系结构上允许CPU分割，把分割的一部分CPU作为一个可用的虚拟机 器环境。有很多NUAM结构的硬件设备支持设备热插拔功能，比如说NUMA结构的计算机一个物理节点的插入和移出需要CPU热插拔功能的支持。
如此先进的功能需要操作系统能够提供必要条件满足CPU的热插拔功能的实现。或者因为远程访问服务器有意去使某个CPU不在系统的执行路径上（即这个CPU不能再次被使用）。这些都要求Linux操作系统支持CPU的热插拔功能。
现在很多对称多处理器的机器（SMP）使用CPU热插拔功能。双核和高可用技术使得便携式电脑都能使用SMP内核的操作系统，但是不能实现CPU热插拔功能。SMP支持CPU暂停或恢复当作一个任务在系统运行过程中。
General Stuff about CPU Hotplug（关于CPU热插拔常常碰到的一些技术与问题）
--------------------------------
Command Line Switches（命令行开关，就是通过命令行的执行，对一些数据项和操作的开\关设置）
---------------------
maxcpus=n Restrict boot time cpus to n. Say if you have 4 cpus, using
maxcpus=2 will only boot 2. You can choose to bring the
other cpus later online, read FAQ's for more info.
这个数据项用来限制机器启动时启动CPU的数目。如果你的机器有4个CPU，你将maxcpus设置为2，那么在机器启动时只能启动2个CPU。你可以在机器完全启动后，可以将剩下的CPU热添加进系统。
additional_cpus=n (*) Use this to limit hotpluggable cpus. This option sets
cpu_possible_map = cpu_present_map + additional_cpus
使用这个命令行可以限制可以热添加的CPU数目。下面等式表示系统中几个描述CPU位图之间的关系。
系统中可以允许最大数目CPU=系统中目前现有的CPU数目+添加进去的CPU数目
(*) Option valid only for following architectures
- x86_64, ia64（只针对X86_64和ia64体系结构的CPU）
ia64 and x86_64 use the number of disabled local apics in ACPI tables MADT
to determine the number of potentially hot-pluggable cpus. The implementation
should only rely on this to count the # of cpus, but *MUST* not rely on the
apicid values in those tables for disabled apics. In the event BIOS doesn't
mark such hot-pluggable cpus as disabled entries, one could use this
parameter "additional_cpus=x" to represent those cpus in the cpu_possible_map.
ia64和x86_64芯片使用ACPI的MADT表中大量的被禁用的局部APICs（Advanced Programmable Interrupt Controllers，高级可编程中断控制器）决定这些CPU作为潜在的热插拔CPU。
possible_cpus=n [s390 only] use this to set hotpluggable cpus.
This option sets possible_cpus bits in
cpu_possible_map. Thus keeping the numbers of bits set
constant even if the machine gets rebooted.
CPU maps and such
-----------------
[More on cpumaps and primitive to manipulate, please check
cpu_possible_map: Bitmap of possible CPUs that can ever be available in the
system. This is used to allocate some boot time memory for per_cpu variables
that aren't designed to grow/shrink as CPUs are made available or removed.
Once set during boot time discovery phase, the map is static, i.e no bits
are added or removed anytime. Trimming it accurately for your system needs
upfront can save some boot time memory. See below for how we use heuristics
in x86_64 case to keep this under check.
cpu_possible_map： 系统中可能所有可用的CPU的位图结构变量。一般在系统引导启动的时候为每个可能存在的CPU分配内存给per_cpu变量。这些不是用来描述CPU的热 添加或热移出操作。这个位图是个静态变量，在系统引导启动时候生成后，就不能修改了。
cpu_online_map: Bitmap of all CPUs currently online. Its set in __cpu_up()
after a cpu is available for kernel scheduling and ready to receive
interrupts from devices. Its cleared when a cpu is brought down using
__cpu_disable(), before which all OS services including interrupts are
migrated to another target CPU.
cpu_online_map： 系统当前online的CPU。这个位图在函数__cpu_up()中被设置，这个位图描述了系统有哪些CPU可以用来调度执行任务和可以接受设备中断。 当执行函数__cpu_disable()时，清除指定CPU在位图cpu_online_map对应的位，这就意味着指定的CPU不可用了，不过这个函 数执行之前，要将指定CPU上所有的服务和中断迁移到其他CPU上执行。
cpu_present_map: Bitmap of CPUs currently present in the system. Not all
of them may be online. When physical hotplug is processed by the relevant
subsystem (e.g ACPI) can change and new bit either be added or removed
from the map depending on the event is hot-add/hot-remove. There are currently
no locking rules as of now. Typical usage is to init topology during boot,
at which time hotplug is disabled.
cpu_present_map： 当前系统存在的CPU。但是，并不是所有存在的CPU都处于online。在物理上的CPU热插拔依据相关的子系统处理，会改变位图 cpu_present_map相应位的值。该位图经典地用于在系统启动时描述系统CPU拓扑结构，并且那时候CPU的hotplug功能被禁用了。
You really dont need to manipulate any of the system cpu maps. They should
be read-only for most use. When setting up per-cpu resources almost always use
cpu_possible_map/for_each_possible_cpu() to iterate.
你并不需要熟练使用系统地CPU这些位图。因为他们通常只读。通过cpu_possible_map/for_each_possible_cpu()反复使用为每个CPU建立必需资源。
Never use anything other than cpumask_t to represent bitmap of CPUs.
在系统中所有的CPU位图都是cpumask_t类型的数据结构。
#include
for_each_possible_cpu - Iterate over cpu_possible_map
for_each_online_cpu - Iterate over cpu_online_map
for_each_present_cpu - Iterate over cpu_present_map
for_each_cpu_mask(x,mask) - Iterate over some random collection of cpu mask.
#include
get_online_cpus() and put_online_cpus():
The above calls are used to inhibit cpu hotplug operations. While the
cpu_hotplug.refcount is non zero, the cpu_online_map will not change.
If you merely need to avoid cpus going away, you could also use
preempt_disable() and preempt_enable() for those sections.
Just remember the critical section cannot call any
function that can sleep or schedule this process away. The preempt_disable()
will work as long as stop_machine_run() is used to take a cpu down.
上 面提到的所有函数（例程）用于阻止CPU的热插拔操作。当cpu_hotplug的引用计数不为0时，位图cpu_online_map不会发生改变。如 果你仅仅只是需要避免CPU的消失，你同样可以调用preempt_disable()和preempt_enable()。记住在临界区不能调用任何函 数因为那样可能会被睡眠或者中断。当执行stop_machine_run()函数使一个CPU不可用时，一定会调用执行 preempt_disable()。
CPU Hotplug - Frequently Asked Questions.（CPU热插拔常见问题）
Q: How to enable my kernel to support CPU hotplug?该怎样激活我的操作系统内核才能支持CPU热插拔？
A: When doing make defconfig, Enable CPU hotplug support
"Processor type and Features" -> Support for Hotpluggable CPUs
Make sure that you have CONFIG_HOTPLUG, and CONFIG_SMP turned on as well.
You would need to enable CONFIG_HOTPLUG_CPU for SMP suspend/resume support
as well.
在你配置内核选项时，即在终端输入“make menuconfig”，将会出现一个蓝色界面，将需要的选项打开即可。
A: As of 2.6.14, the following architectures support CPU hotplug.
i386 (Intel), ppc, ppc64, parisc, s390, ia64 and x86_64
Q: How to test if hotplug is supported on the newly built kernel?怎么测试最新编译的内核是否支持CPU热插拔操作。
A: You should now notice an entry in sysfs.
Now you should see entries for all present cpu, the following is an example
in a 8-way system.
#pwd
#/sys/devices/system/cpu
#ls -l
total 0
drwxr-xr-x 10 root root 0 Sep 19 07:44 .
drwxr-xr-x 13 root root 0 Sep 19 07:45 ..
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu0
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu1
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu2
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu3
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu4
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu5
drwxr-xr-x 3 root root 0 Sep 19 07:44 cpu6
drwxr-xr-x 3 root root 0 Sep 19 07:48 cpu7
Under each directory you would find an "online" file which is the control
file to logically online/offline a processor.
在每个目录下可以看见一个“online”文件，该文件描述的所属CPU是否online
Q: Does hot-add/hot-remove refer to physical add/remove of cpus?是否CPU热插拔支持物理CPU添加和移出？
A: The usage of hot-add/remove may not be very consistently used in the code.
CONFIG_HOTPLUG_CPU enables logical online/offline capability in the kernel.
To support physical addition/removal, one would need some BIOS hooks and
the platform should have something like an attention button in PCI hotplug.
CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of CPUs.
CPU逻辑上的热插拔处理是在linux内核的代码完成。支持物理上CPU热插拔，需要BIOS支持以及硬件平台的支持比如像PCI设备热插拔以类似的硬件支持
Q: How do i logically offline a CPU?我该怎么将一个CPU逻辑上热移出。
A: Do the following.
#echo 0 > /sys/devices/system/cpu/cpuX/online（在终端里，输本行命令即可）
Once the logical offline is successful, check（检查是否成功将CPU热移出）
#cat /proc/interrupts（在终端里，输本行命令即可）
You should now not see the CPU that you removed. Also online file will report
the state as 0 when a cpu if offline and 1 when its online.
在“online”文件里描述CPU的状态，0代表已经移出，1代表已经添加成功
#To display the current cpu state.
#cat /sys/devices/system/cpu/cpuX/online
Q: Why cant i remove CPU0 on some systems?
A: Some architectures may have some special dependency on a certain CPU.
For e.g in IA64 platforms we have ability to sent platform interrupts to the
OS. a.k.a Corrected Platform Error Interrupts (CPEI). In current ACPI
specifications, we didn't have a way to change the target CPU. Hence if the
current ACPI version doesn't support such re-direction, we disable that CPU
by making it not-removable.
In such cases you will also notice that the online file is missing under cpu0.
Q: How do i find out if a particular CPU is not removable?
A: Depending on the implementation, some architectures may show this by the
absence of the "online" file. This is done if it can be determined ahead of
time that this CPU cannot be removed.
In some situations, this can be a run time check, i.e if you try to remove the
last CPU, this will not be permitted. You can find such failures by
investigating the return value of the "echo" command.
Q: What happens when a CPU is being logically offlined?当一个CPU逻辑热移出时，内核放生了什么？
A: The following happen, listed in no particular order :-)
- A notification is sent to in-kernel registered modules by sending an event
CPU_DOWN_PREPARE or CPU_DOWN_PREPARE_FROZEN, depending on whether or not the
CPU is being offlined while tasks are frozen due to a suspend operation in
progress
将事件CPU_DOWN_PREPARE 或者 CPU_DOWN_PREPARE_FROZEN发送到内核中
- All processes are migrated away from this outgoing CPU to new CPUs.
The new CPU is chosen from each process' current cpuset, which may be
a subset of all online CPUs.
将所有进程从目标CPU迁移到系统其他CPU上运行。
- All interrupts targeted to this CPU is migrated to a new CPU
将所有中断从目标CPU迁移到其他CPU
- timers/bottom half/task lets are also migrated to a new CPU
计时器和下半部迁移到其他CPU
- Once all services are migrated, kernel calls an arch specific routine
__cpu_disable() to perform arch specific cleanup.
当所有服务迁移后，内核调用例程__cpu_disable()执行体系结构特定的清除操作
- Once this is successful, an event for successful cleanup is sent by an event
CPU_DEAD (or CPU_DEAD_FROZEN if tasks are frozen due to a suspend while the
CPU is being offlined).
一旦成功地将CPU热移出，就会发送CPU_DEAD。
"It is expected that each service cleans up when the CPU_DOWN_PREPARE
notifier is called, when CPU_DEAD is called its expected there is nothing
running on behalf of this CPU that was offlined"
希望在执行通知链上CPU_DOWN_PREPARE操作时完成所有服务的清除操作。当执行通知链上CPU_DEAD的操作时，希望没有任何服务在CPU上执行，因为CPU已经成功地被移出。
Q: If i have some kernel code that needs to be aware of CPU arrival and
departure, how to i arrange for proper notification?
A: This is what you would need in your kernel code to receive notifications.
#include
static int __cpuinit foobar_cpu_callback(struct notifier_block *nfb,
unsigned long action, void *hcpu)
{
unsigned int cpu = (unsigned long)hcpu;
switch (action) {
case CPU_ONLINE:
case CPU_ONLINE_FROZEN:
foobar_online_action(cpu);
break;
case CPU_DEAD:
case CPU_DEAD_FROZEN:
foobar_dead_action(cpu);
break;
}
return NOTIFY_OK;
}
static struct notifier_block __cpuinitdata foobar_cpu_notifer =
{
.notifier_call = foobar_cpu_callback,
};
You need to call register_cpu_notifier() from your init function.
Init functions could be of two types:
1. early init (init function called when only the boot processor is online).
2. late init (init function called _after_ all the CPUs are online).
For the first case, you should add the following to your init function
register_cpu_notifier(&foobar_cpu_notifier);
For the second case, you should add the following to your init function
register_hotcpu_notifier(&foobar_cpu_notifier);
You can fail PREPARE notifiers if something doesn't work to prepare resources.
This will stop the activity and send a following CANCELED event back.
CPU_DEAD should not be failed, its just a goodness indication, but bad
things will happen if a notifier in path sent a BAD notify code.
Q: I don't see my action being called for all CPUs already up and running?
A: Yes, CPU notifiers are called only when new CPUs are on-lined or offlined.
If you need to perform some action for each cpu already in the system, then
for_each_online_cpu(i) {
foobar_cpu_callback(&foobar_cpu_notifier, CPU_UP_PREPARE, i);
foobar_cpu_callback(&foobar_cpu_notifier, CPU_ONLINE, i);
}
Q: If i would like to develop cpu hotplug support for a new architecture,
what do i need at a minimum?
A: The following are what is required for CPU hotplug infrastructure to work
correctly.
- Make sure you have an entry in Kconfig to enable CONFIG_HOTPLUG_CPU
- __cpu_up() - Arch interface to bring up a CPU
- __cpu_disable() - Arch interface to shutdown a CPU, no more interrupts
can be handled by the kernel after the routine
returns. Including local APIC timers etc are
shutdown.
- __cpu_die() - This actually supposed to ensure death of the CPU.
Actually look at some example code in other arch
that implement CPU hotplug. The processor is taken
down from the idle() loop for that specific
architecture. __cpu_die() typically waits for some
per_cpu state to be set, to ensure the processor
dead routine is called to be sure positively.
Q: I need to ensure that a particular cpu is not removed when there is some
work specific to this cpu is in progress.
A: First switch the current thread context to preferred cpu
int my_func_on_cpu(int cpu)
{
cpumask_t saved_mask, new_mask = CPU_MASK_NONE;
int curr_cpu, err = 0;
saved_mask = current->cpus_allowed;
cpu_set(cpu, new_mask);
err = set_cpus_allowed(current, new_mask);
if (err)
return err;
/*
* If we got scheduled out just after the return from
* set_cpus_allowed() before running the work, this ensures
* we stay locked.
*/
curr_cpu = get_cpu();
if (curr_cpu != cpu) {
err = -EAGAIN;
goto ret;
} else {
/*
* Do work : But cant sleep, since get_cpu() disables preempt
*/
}
ret:
put_cpu();
set_cpus_allowed(current, saved_mask);
return err;
}
Q: How do we determine how many CPUs are available for hotplug.
A: There is no clear spec defined way from ACPI that can give us that
information today. Based on some input from Natalie of Unisys,
that the ACPI MADT (Multiple APIC Description Tables) marks those possible
CPUs in a system with disabled status.
Andi implemented some simple heuristics that count the number of disabled
CPUs in MADT as hotpluggable CPUS. In the case there are no disabled CPUS
we assume 1/2 the number of CPUs currently present can be hotplugged.
Caveat: Today's ACPI MADT can only provide 256 entries since the apicid field
in MADT is only 8 bits.
User Space Notification
Hotplug support for devices is common in Linux today. Its being used today to
support automatic configuration of network, usb and pci devices. A hotplug
event can be used to invoke an agent script to perform the configuration task.
You can add /etc/hotplug/cpu.agent to handle hotplug notification user space
scripts.
#!/bin/bash
# $Id: cpu.agent
# Kernel hotplug params include:
#ACTION=%s [online or offline]
#DEVPATH=%s
#
cd /etc/hotplug
. ./hotplug.functions
case $ACTION in
online)
echo `date` ":cpu.agent" add cpu >> /tmp/hotplug.txt
;;
offline)
echo `date` ":cpu.agent" remove cpu >>/tmp/hotplug.txt
;;
*)
debug_mesg CPU $ACTION event not supported
exit 1
;;
esac