专题：Linux内存管理专题
关键词：OOM、oom_adj、oom_score、badness。
Linux内核为了提高内存的使用效率采用过度分配内存(over-commit memory)的办法，造成物理内存过度紧张进而触发OOM机制来杀死一些进程回收内存。
该机制会监控那些占用内存过大，尤其是瞬间很快消耗大量内存的进程，为了防止内存耗尽会把该进程杀掉。
1. 关于OOM
内核检测到系统内存不足，在内存分配路径上触发out_of_memory，然后调用select_bad_process()选择一个'bad'进程杀掉，判断和选择一个‘bad'进程的过程由oom_badness()决定。
Linux下每个进程都有自己的OOM权重，在/proc/<pid>/oom_adj里面，范围是-17到+15，取值越高，越容易被杀掉。
2. OOM触发路径
在内存分配路径上，当内存不足的时候会触发kswapd、或者内存规整，极端情况会触发OOM，来获取更多内存。
在内存回收失败之后，__alloc_pages_may_oom是OOM的入口，但是主要工作在out_of_memory中进行处理。
由于Linux内存都是以页为单位，所以__alloc_pages_nodemask是必经之处。
alloc_pages ->_alloc_pages ->__alloc_pages_nodemask ->__alloc_pages_slowpath-------------------------此时已经说明内存不够，会触发一些内存回收、内存规整机制，极端情况触发OOM。 ->__alloc_pages_may_oom -----------------------进入OOM的开始，包括一些检查动作。
->out_of_memory------------------------------OOM的核心
->select_bad_process-----------------------选择最'bad'进程
->oom_scan_process_thread ->oom_badness----------------------------计算当前进程有多'badness'
->oom_kill_process-------------------------杀死选中的进程
3. 影响OOM的内核参数
参照Linux内存管理 (23)内存sysfs节点和工具的OOM章节。
4. OOM代码分析
OOM主要代码在mm/oom_kill.c和include/linux/mm.h中。
/* * Details of the page allocation that triggered the oom killer that are used to * determine what should be killed. */ struct oom_control { /* Used to determine cpuset */ struct zonelist *zonelist; /* Used to determine mempolicy */ nodemask_t *nodemask; /* Used to determine cpuset and node locality requirement */ const gfp_t gfp_mask; /* * order == -1 means the oom kill is required by sysrq, otherwise only * for display purposes. */ const int order; };
__alloc_pages_may_oom是内存分配路径上的OOM入口，在进入OOM之前还会检查一些特殊情况。
static inline struct page * __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, const struct alloc_context *ac, unsigned long *did_some_progress) { struct oom_control oc = {-------------------------------------------------------OOM控制参数 .zonelist = ac->zonelist, .nodemask = ac->nodemask, .gfp_mask = gfp_mask, .order = order, }; struct page *page; *did_some_progress = 0; /* * Acquire the oom lock. If that fails, somebody else is * making progress for us. */ if (!mutex_trylock(&oom_lock)) { *did_some_progress = 1; schedule_timeout_uninterruptible(1); return NULL; } /* * Go through the zonelist yet one more time, keep very high watermark * here, this is only to catch a parallel oom killing, we must fail if * we're still under heavy pressure. */ page = get_page_from_freelist(gfp_mask | __GFP_HARDWALL, order,---------------再次使用高水位检查一次，是否需要启动OOM流程。 ALLOC_WMARK_HIGH|ALLOC_CPUSET, ac); if (page) goto out; if (!(gfp_mask & __GFP_NOFAIL)) {---------------------------------------------跳过OOM的特殊情况 /* Coredumps can quickly deplete all memory reserves */ if (current->flags & PF_DUMPCORE) goto out; /* The OOM killer will not help higher order allocs */ if (order > PAGE_ALLOC_COSTLY_ORDER) goto out; /* The OOM killer does not needlessly kill tasks for lowmem */ if (ac->high_zoneidx < ZONE_NORMAL) goto out; /* The OOM killer does not compensate for IO-less reclaim */ if (!(gfp_mask & __GFP_FS)) { /* * XXX: Page reclaim didn't yield anything, * and the OOM killer can't be invoked, but * keep looping as per tradition. */ *did_some_progress = 1; goto out; } if (pm_suspended_storage()) goto out; /* The OOM killer may not free memory on a specific node */ if (gfp_mask & __GFP_THISNODE) goto out; } /* Exhausted what can be done so it's blamo time */ if (out_of_memory(&oc) || WARN_ON_ONCE(gfp_mask & __GFP_NOFAIL))---------------进入OOM主流程 *did_some_progress = 1; out: mutex_unlock(&oom_lock); return page; }
out_of_memory函数是OOM机制的核心，他可以分为两部分。一是调挑选最’bad‘的进程，二是杀死它。
/** * out_of_memory - kill the "best" process when we run out of memory * @oc: pointer to struct oom_control * * If we run out of memory, we have the choice between either * killing a random task (bad), letting the system crash (worse) * OR try to be smart about which process to kill. Note that we * don't have to be perfect here, we just have to be good. */ bool out_of_memory(struct oom_control *oc) { struct task_struct *p; unsigned long totalpages; unsigned long freed = 0; unsigned int uninitialized_var(points); enum oom_constraint constraint = CONSTRAINT_NONE; if (oom_killer_disabled)----------------------------------------------------在freeze_processes会将其置位，即禁止OOM；在thaw_processes会将其清零，即打开OOM。所以，如果在冻结过程，不允许OOM。 return false; blocking_notifier_call_chain(&oom_notify_list, 0, &freed); if (freed > 0) /* Got some memory back in the last second. */ return true; /* * If current has a pending SIGKILL or is exiting, then automatically * select it. The goal is to allow it to allocate so that it may * quickly exit and free its memory. * * But don't select if current has already released its mm and cleared * TIF_MEMDIE flag at exit_mm(), otherwise an OOM livelock may occur. */ if (current->mm && (fatal_signal_pending(current) || task_will_free_mem(current))) { mark_oom_victim(current); return true; } /* * Check if there were limitations on the allocation (only relevant for * NUMA) that may require different handling. */ constraint = constrained_alloc(oc, &totalpages);-----------------------------未定义CONFIG_NUMA返回CONSTRAINT_NONE if (constraint != CONSTRAINT_MEMORY_POLICY) oc->nodemask = NULL; check_panic_on_oom(oc, constraint, NULL);----------------------------------检查sysctl_panic_on_oom设置，以及是否由sysrq触发，来决定是否触发panic。 if (sysctl_oom_kill_allocating_task && current->mm &&----------------------如果设置了sysctl_oom_kill_allocating_task，那么当内存耗尽时，会把当前申请内存分配的进程杀掉。 !oom_unkillable_task(current, NULL, oc->nodemask) && current->signal->oom_score_adj != OOM_SCORE_ADJ_MIN) { get_task_struct(current); oom_kill_process(oc, current, 0, totalpages, NULL, "Out of memory (oom_kill_allocating_task)"); return true; } p = select_bad_process(oc, &points, totalpages);---------------------------遍历所有进程，进程下的线程，查找合适的候选进程。 /* Found nothing?!?! Either we hang forever, or we panic. */ if (!p && !is_sysrq_oom(oc)) {---------------------------------------------如果没有合适候选进程，并且OOM不是由sysrq触发的，进入panic。 dump_header(oc, NULL, NULL); panic("Out of memory and no killable processes...\n"); } if (p && p != (void *)-1UL) { oom_kill_process(oc, p, points, totalpages, NULL, "Out of memory");---------------------------------------------杀死选中的进程。 /* * Give the killed process a good chance to exit before trying * to allocate memory again. */ schedule_timeout_killable(1); } return true; }
select_bad_process通过oom_scan_process_thread检查当前进程各种属性，返回oom_scan_t以决定for流程走向。
在oom_badness总计算当前进程的得分，选取最高分者。返回选中进程的结构体，以及进程得分ppoints。
/* * Simple selection loop. We chose the process with the highest * number of 'points'. Returns -1 on scan abort. */ static struct task_struct *select_bad_process(struct oom_control *oc, unsigned int *ppoints, unsigned long totalpages) { struct task_struct *g, *p; struct task_struct *chosen = NULL; unsigned long chosen_points = 0; rcu_read_lock(); for_each_process_thread(g, p) {-----------------------------------遍历所有进程线程 unsigned int points; switch (oom_scan_process_thread(oc, p, totalpages)) {---------根据oc和p来决定当前scan的流程，返回oom_scan_t值。 case OOM_SCAN_SELECT:-----------------------------------------相应的进程可以被选择 chosen = p; chosen_points = ULONG_MAX; /* fall through */ case OOM_SCAN_CONTINUE:---------------------------------------跳过for中之后部分 continue; case OOM_SCAN_ABORT:------------------------------------------退出整个for循环，并且直接返回。 rcu_read_unlock(); return (struct task_struct *)(-1UL); case OOM_SCAN_OK: break; }; points = oom_badness(p, NULL, oc->nodemask, totalpages);------对每个进程进行打分。 if (!points || points < chosen_points)------------------------这里保证只取最高分的进程，所以分数最高者被选中。 continue; /* Prefer thread group leaders for display purposes */ if (points == chosen_points && thread_group_leader(chosen)) continue; chosen = p; chosen_points = points; } if (chosen) get_task_struct(chosen); rcu_read_unlock(); *ppoints = chosen_points * 1000 / totalpages; return chosen; }
oom_badness是给进程打分的函数，可以说是核心中的核心。最终结果受oom_score_adj和当前进程内存使用量综合影响。
/** * oom_badness - heuristic function to determine which candidate task to kill * @p: task struct of which task we should calculate * @totalpages: total present RAM allowed for page allocation * * The heuristic for determining which task to kill is made to be as simple and * predictable as possible. The goal is to return the highest value for the * task consuming the most memory to avoid subsequent oom failures. */ unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg, const nodemask_t *nodemask, unsigned long totalpages) { long points; long adj; if (oom_unkillable_task(p, memcg, nodemask)) return 0; p = find_lock_task_mm(p); if (!p) return 0; adj = (long)p->signal->oom_score_adj;--------------------------------------获取当前进程的oom_score_adh参数。 if (adj == OOM_SCORE_ADJ_MIN) { task_unlock(p); return 0;--------------------------------------------------------------如果当前进程oom_score_adj为OOM_SCORE_ADJ_MIN的话，就返回0.等于告诉OOM，此进程不参数'bad'评比。 } /* * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) + atomic_long_read(&p->mm->nr_ptes) + mm_nr_pmds(p->mm);-----------------可以看出points综合了内存占用情况，包括RSS部分、swap file或者swap device占用内存、以及页表占用内存。 task_unlock(p); /* * Root processes get 3% bonus, just like the __vm_enough_memory() * implementation used by LSMs. */ if (has_capability_noaudit(p, CAP_SYS_ADMIN))------------------------------如果是root用户，增加3%的使用特权。 points -= (points * 3) / 100; /* Normalize to oom_score_adj units */ adj *= totalpages / 1000;--------------------------------------------------这里可以看出oom_score_adj对最终分数的影响，如果oom_score_adj小于0，则最终points就会变小，进程更加不会被选中。 points += adj;-------------------------------------------------------------将归一化后的adj和points求和，作为当前进程的分数。 /* * Never return 0 for an eligible task regardless of the root bonus and * oom_score_adj (oom_score_adj can't be OOM_SCORE_ADJ_MIN here). */ return points > 0 ? points : 1; }
oom_kill_process用于杀死最高分的进程，包括进程下的线程。
/* * Must be called while holding a reference to p, which will be released upon * returning. */ void oom_kill_process(struct oom_control *oc, struct task_struct *p, unsigned int points, unsigned long totalpages, struct mem_cgroup *memcg, const char *message) { struct task_struct *victim = p; struct task_struct *child; struct task_struct *t; struct mm_struct *mm; unsigned int victim_points = 0; static DEFINE_RATELIMIT_STATE(oom_rs, DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST); /* * If the task is already exiting, don't alarm the sysadmin or kill * its children or threads, just set TIF_MEMDIE so it can die quickly */ task_lock(p); if (p->mm && task_will_free_mem(p)) {---------------------------------对于非coredump正处于退出状态的线程，标注TIF_MEMDIE，然后退出。 mark_oom_victim(p); task_unlock(p); put_task_struct(p); return; } task_unlock(p); if (__ratelimit(&oom_rs)) dump_header(oc, p, memcg); pr_err("%s: Kill process %d (%s) score %u or sacrifice child\n", message, task_pid_nr(p), p->comm, points); /* * If any of p's children has a different mm and is eligible for kill, * the one with the highest oom_badness() score is sacrificed for its * parent. This attempts to lose the minimal amount of work done while * still freeing memory. */ read_lock(&tasklist_lock); for_each_thread(p, t) {-----------------------------------------------遍历进程下的线程 list_for_each_entry(child, &t->children, sibling) { unsigned int child_points; if (process_shares_mm(child, p->mm)) continue; /* * oom_badness() returns 0 if the thread is unkillable */ child_points = oom_badness(child, memcg, oc->nodemask, totalpages);------------------------------计算子线程的得分情况 if (child_points > victim_points) {---------------------------将得分最高者计为victim，得分为victim_points。 put_task_struct(victim); victim = child; victim_points = child_points; get_task_struct(victim); } } } read_unlock(&tasklist_lock); p = find_lock_task_mm(victim); if (!p) { put_task_struct(victim); return; } else if (victim != p) { get_task_struct(p); put_task_struct(victim); victim = p; } /* Get a reference to safely compare mm after task_unlock(victim) */ mm = victim->mm; atomic_inc(&mm->mm_count); /* * We should send SIGKILL before setting TIF_MEMDIE in order to prevent * the OOM victim from depleting the memory reserves from the user * space under its control. */ do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);--------------发送SIGKILL信号给victim进程。 mark_oom_victim(victim);-----------------------------------------------标注TIF_MEMDIE是因为OOM被杀死 pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB\n", task_pid_nr(victim), victim->comm, K(victim->mm->total_vm), K(get_mm_counter(victim->mm, MM_ANONPAGES)), K(get_mm_counter(victim->mm, MM_FILEPAGES))); task_unlock(victim); /* * Kill all user processes sharing victim->mm in other thread groups, if * any. They don't get access to memory reserves, though, to avoid * depletion of all memory. This prevents mm->mmap_sem livelock when an * oom killed thread cannot exit because it requires the semaphore and * its contended by another thread trying to allocate memory itself. * That thread will now get access to memory reserves since it has a * pending fatal signal. */ rcu_read_lock(); for_each_process(p) {--------------------------------------------------继续处理共享内存的相关线程 if (!process_shares_mm(p, mm)) continue; if (same_thread_group(p, victim)) continue; if (unlikely(p->flags & PF_KTHREAD)) continue; if (is_global_init(p)) continue; if (p->signal->oom_score_adj == OOM_SCORE_ADJ_MIN) continue; do_send_sig_info(SIGKILL, SEND_SIG_FORCED, p, true); } rcu_read_unlock(); mmdrop(mm);------------------------------------------------------------释放mm空间的内存。包括申请的页面、mm结构体等。 put_task_struct(victim);-----------------------------------------------释放task_struct占用的内存空间，包括cgroup等等。 }
5. 关于OOM的测试
相关阅读：《Linux OOM机制介绍》、《Linux内核OOM机制的详细分析》、《Linux内核OOM机制分析》