I'm trying to write some code to determine if clock_gettime used with CLOCK_MONOTONIC_RAW will give me results coming from the same hardware on different cores.
我正在尝试编写一些代码来确定与CLOCK_MONOTONIC_RAW一起使用的clock_gettime是否会给出来自不同内核上相同硬件的结果。
From what I understand it is possible for each core to produce independent results but not always. I was given the task of obtaining timings on all cores with a precision of 40 nanoseconds.
据我所知，每个核心都可能产生独立的结果，但并非总是如此。我被赋予了在所有内核上获得定时的任务，精度为40纳秒。
The reason I'm not using CLOCK_REALTIME is that my program absolutely must not be affected by NTP adjustments.
我没有使用CLOCK_REALTIME的原因是我的程序绝对不能受NTP调整的影响。
Edit:
编辑：
I have found the unsynchronized_tsc function which tries to test whether the TSC is the same on all cores. I am now attempting to find if CLOCK_MONOTONIC_RAW is based on the TSC.
我找到了unsynchronized_tsc函数，它试图测试所有内核上的TSC是否相同。我现在试图找出CLOCK_MONOTONIC_RAW是否基于TSC。
Final edit:
最终编辑：
It turns out that CLOCK_MONOTONIC_RAW is always usable on multi-core systems and does not rely on the TSC even on Intel machines.
事实证明，CLOCK_MONOTONIC_RAW始终可用于多核系统，即使在Intel机器上也不依赖于TSC。
1 个解决方案
#1
4
To do measurements this precisely; you'd need:
准确地进行测量;你需要：
- code that's executed on all CPUs, that reads the CPU's time stamp counter and stores it as soon as "an event" occurs
- 在所有CPU上执行的代码，它读取CPU的时间戳计数器并在“事件”发生时立即存储
- some way to create "an event" that is noticed at the same time by all CPUs
- 某种方式来创建所有CPU同时注意到的“事件”
- some way to prevent timing problems caused by IRQs, task switches, etc.
- 某种方法可以防止由IRQ，任务切换等引起的时序问题。
Various possibilities for the event include:
活动的各种可能性包括：
- polling a memory location in a loop, where one CPU writes a new value and other CPUs stop polling when they see the new value
- 轮询循环中的内存位置，其中一个CPU写入新值，而其他CPU在看到新值时停止轮询
- using the local APIC to broadcast an IPI (inter-processor interrupt) to all CPUs
- 使用本地APIC向所有CPU广播IPI（处理器间中断）
For both of these methods there are delays between the CPUs (especially for larger NUMA systems) - a write to memory (cache) may be visible on the CPU that made the write immediately, and be visible by a CPU on a different physical chip (in a different NUMA domain) later. To avoid this you may need to find the average of initiating the event on all CPUs. E.g. (for 2 CPUs) one CPU initiates and both measure, then the other CPU initiates and both measure, then results are combined to cancel out any "event propagation latency".
对于这两种方法，CPU之间存在延迟（特别是对于较大的NUMA系统） - 写入存储器（高速缓存）可能在CPU上立即显示，并且可由不同物理芯片上的CPU看到（在一个不同的NUMA域）之后。为避免这种情况，您可能需要找到在所有CPU上启动事件的平均值。例如。 （对于2个CPU）一个CPU启动并进行测量，然后另一个CPU启动并进行测量，然后将结果组合以抵消任何“事件传播延迟”。
To fix other timing problems (IRQs, task switches, etc) I'd want to be doing these tests during boot where nothing else can mess things up. Otherwise you either need to prevent the problems (ensure all CPUs are running at the same speed, disable IRQs, disable thread switches, stop any PCI device bus mastering, etc) or cope with problems (e.g. run the same test many times and see if you get similar results most of the time).
为了解决其他时间问题（IRQ，任务切换等），我想在启动期间进行这些测试，其他任何事情都不会搞砸。否则，您需要防止出现问题（确保所有CPU以相同的速度运行，禁用IRQ，禁用线程切换，停止任何PCI设备总线控制等）或应对问题（例如，多次运行相同的测试，看看是否你大部分时间都得到类似的结果）。
Also note that all of the above can only ensure that the time stamp counters were in sync at the time the test was done, and don't guarantee that they won't become out of sync after the test is done. To ensure the CPUs remain in sync you'd need to rely on the CPU's "monotonic clock" guarantees (but older CPUs don't make that guarantee).
另请注意，以上所有内容只能确保时间戳计数器在测试完成时同步，并且不保证在测试完成后它们不会变得不同步。为确保CPU保持同步，您需要依赖CPU的“单调时钟”保证（但较旧的CPU不能保证这一点）。
Finally; if you're attempting to do this in user-space (and not in kernel code); then my advice is to design code in a way that isn't so fragile to begin with. Even if the TSCs on different CPUs are guaranteed to be perfectly in sync at all times, you can't prevent an IRQ from interrupting immediately before or immediately after reading the TSC (and there's no way to atomically do something and read TSC at the same time); and therefore if your code requires such precisely synchronised timing then your code's design is probably flawed.
最后;如果你试图在用户空间（而不是在内核代码中）这样做;那么我的建议是以一种不易开始的方式设计代码。即使不同CPU上的TSC始终保证完全同步，也无法阻止IRQ在读取TSC之前或之后立即中断（并且无法原子地执行某些操作并在同一时间读取TSC）时间）;因此，如果您的代码需要这种精确同步的时序，那么您的代码设计可能存在缺陷。
#1
4
To do measurements this precisely; you'd need:
准确地进行测量;你需要：
- code that's executed on all CPUs, that reads the CPU's time stamp counter and stores it as soon as "an event" occurs
- 在所有CPU上执行的代码，它读取CPU的时间戳计数器并在“事件”发生时立即存储
- some way to create "an event" that is noticed at the same time by all CPUs
- 某种方式来创建所有CPU同时注意到的“事件”
- some way to prevent timing problems caused by IRQs, task switches, etc.
- 某种方法可以防止由IRQ，任务切换等引起的时序问题。
Various possibilities for the event include:
活动的各种可能性包括：
- polling a memory location in a loop, where one CPU writes a new value and other CPUs stop polling when they see the new value
- 轮询循环中的内存位置，其中一个CPU写入新值，而其他CPU在看到新值时停止轮询
- using the local APIC to broadcast an IPI (inter-processor interrupt) to all CPUs
- 使用本地APIC向所有CPU广播IPI（处理器间中断）
For both of these methods there are delays between the CPUs (especially for larger NUMA systems) - a write to memory (cache) may be visible on the CPU that made the write immediately, and be visible by a CPU on a different physical chip (in a different NUMA domain) later. To avoid this you may need to find the average of initiating the event on all CPUs. E.g. (for 2 CPUs) one CPU initiates and both measure, then the other CPU initiates and both measure, then results are combined to cancel out any "event propagation latency".
对于这两种方法，CPU之间存在延迟（特别是对于较大的NUMA系统） - 写入存储器（高速缓存）可能在CPU上立即显示，并且可由不同物理芯片上的CPU看到（在一个不同的NUMA域）之后。为避免这种情况，您可能需要找到在所有CPU上启动事件的平均值。例如。 （对于2个CPU）一个CPU启动并进行测量，然后另一个CPU启动并进行测量，然后将结果组合以抵消任何“事件传播延迟”。
To fix other timing problems (IRQs, task switches, etc) I'd want to be doing these tests during boot where nothing else can mess things up. Otherwise you either need to prevent the problems (ensure all CPUs are running at the same speed, disable IRQs, disable thread switches, stop any PCI device bus mastering, etc) or cope with problems (e.g. run the same test many times and see if you get similar results most of the time).
为了解决其他时间问题（IRQ，任务切换等），我想在启动期间进行这些测试，其他任何事情都不会搞砸。否则，您需要防止出现问题（确保所有CPU以相同的速度运行，禁用IRQ，禁用线程切换，停止任何PCI设备总线控制等）或应对问题（例如，多次运行相同的测试，看看是否你大部分时间都得到类似的结果）。
Also note that all of the above can only ensure that the time stamp counters were in sync at the time the test was done, and don't guarantee that they won't become out of sync after the test is done. To ensure the CPUs remain in sync you'd need to rely on the CPU's "monotonic clock" guarantees (but older CPUs don't make that guarantee).
另请注意，以上所有内容只能确保时间戳计数器在测试完成时同步，并且不保证在测试完成后它们不会变得不同步。为确保CPU保持同步，您需要依赖CPU的“单调时钟”保证（但较旧的CPU不能保证这一点）。
Finally; if you're attempting to do this in user-space (and not in kernel code); then my advice is to design code in a way that isn't so fragile to begin with. Even if the TSCs on different CPUs are guaranteed to be perfectly in sync at all times, you can't prevent an IRQ from interrupting immediately before or immediately after reading the TSC (and there's no way to atomically do something and read TSC at the same time); and therefore if your code requires such precisely synchronised timing then your code's design is probably flawed.
最后;如果你试图在用户空间（而不是在内核代码中）这样做;那么我的建议是以一种不易开始的方式设计代码。即使不同CPU上的TSC始终保证完全同步，也无法阻止IRQ在读取TSC之前或之后立即中断（并且无法原子地执行某些操作并在同一时间读取TSC）时间）;因此，如果您的代码需要这种精确同步的时序，那么您的代码设计可能存在缺陷。