【原创】大叔经验分享(21)yarn中查看每个应用实时占用的内存和cpu资源

时间:2022-11-03 21:05:29

在yarn中的application详情页面

http://resourcemanager/cluster/app/$applicationId

或者通过application命令

yarn application -status $applicationId

只能看到应用启动以来占用的资源*时间统计,比如:

Aggregate Resource Allocation : 3962853 MB-seconds, 1466 vcore-seconds

到处都找不到这个应用当前实时的资源占用情况,比如当前占用了多少内存多少核,跟进yarn代码发现其实是有这个统计的:

org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport

  public static ApplicationResourceUsageReport newInstance(
int numUsedContainers, int numReservedContainers, Resource usedResources,
Resource reservedResources, Resource neededResources, long memorySeconds,
long vcoreSeconds) {
ApplicationResourceUsageReport report =
Records.newRecord(ApplicationResourceUsageReport.class);
report.setNumUsedContainers(numUsedContainers);
report.setNumReservedContainers(numReservedContainers);
report.setUsedResources(usedResources);
report.setReservedResources(reservedResources);
report.setNeededResources(neededResources);
report.setMemorySeconds(memorySeconds);
report.setVcoreSeconds(vcoreSeconds);
return report;
}

其中usedResources就是当前的实时占用资源情况,包括内存和cpu,这个统计是在YarnScheduler的接口中返回:

org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler

  /**
* Get a resource usage report from a given app attempt ID.
* @param appAttemptId the id of the application attempt
* @return resource usage report for this given attempt
*/
@LimitedPrivate("yarn")
@Evolving
ApplicationResourceUsageReport getAppResourceUsageReport(
ApplicationAttemptId appAttemptId);

getAppResourceUsageReport方法被RMAppAttemptImpl.getApplicationResourceUsageReport调用:

org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl

  @Override
public ApplicationResourceUsageReport getApplicationResourceUsageReport() {
this.readLock.lock();
try {
ApplicationResourceUsageReport report =
scheduler.getAppResourceUsageReport(this.getAppAttemptId());
if (report == null) {
report = RMServerUtils.DUMMY_APPLICATION_RESOURCE_USAGE_REPORT;
}
AggregateAppResourceUsage resUsage =
this.attemptMetrics.getAggregateAppResourceUsage();
report.setMemorySeconds(resUsage.getMemorySeconds());
report.setVcoreSeconds(resUsage.getVcoreSeconds());
return report;
} finally {
this.readLock.unlock();
}
}

RMAppAttemptImpl.getApplicationResourceUsageReport被两个地方调用:

第一个调用

org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl

  public ApplicationReport createAndGetApplicationReport(String clientUserName,
boolean allowAccess) {
...
appUsageReport = currentAttempt.getApplicationResourceUsageReport();
...

RMAppImpl.createAndGetApplicationReport会被ClientRMService.getApplications和ClientRMService.getApplicationReport调用,这两个方法分别对应命令

yarn application -list
yarn application -status $applicationId

这两个地方展示信息的时候都没展示usedResources,可能作者觉得这个实时资源占用统计没那么重要。

详见:
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService

第二个调用

org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo

  public AppInfo(RMApp app, Boolean hasAccess, String schemePrefix) {
...
ApplicationResourceUsageReport resourceReport = attempt
.getApplicationResourceUsageReport();
if (resourceReport != null) {
Resource usedResources = resourceReport.getUsedResources();
allocatedMB = usedResources.getMemory();
allocatedVCores = usedResources.getVirtualCores();
runningContainers = resourceReport.getNumUsedContainers();
}
...

这个构造函数会在RMWebServices.getApp和RMWebServices.getApps时被调用,这是个service接口,对应url分别为:

http://resourcemanager/ws/v1/cluster/apps/$applicationId
http://resourcemanager/ws/v1/cluster/apps?state=RUNNING

这两个接口的返回值中有实时资源占用情况如下:

<allocatedMB>56320</allocatedMB>
<allocatedVCores>21</allocatedVCores>

分别对应实时内存占用和实时CPU占用;

详见:
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices

如果你发现spark应用内存的占用比你分配的要多,可以参考这里:https://www.cnblogs.com/barneywill/p/10102353.html