MapReduce源码分析_李孟_新浪博客

2023-09-05 阅读 29 评论 0

摘要：job.waitForCompletion(true);进入源码 submit()-> connect();连接，客户端获取服务端的代理对象 connect()->new Cluster(getConfiguration()); Cluster(jobTrackAddr,conf)->initialize->clientProtocol RPC通信versionID submit() ->submitter.submi

job.waitForCompletion(true);进入源码

submit()-> connect();连接，客户端获取服务端的代理对象

connect()->new Cluster(getConfiguration());

Cluster(jobTrackAddr,conf)->initialize->clientProtocol RPC通信versionID

submit() ->submitter.submitJobInternal(Job.this, cluster):

checkSpecs(job);检查路径

copyAndConfigureFiles(job, submitJobDir);拷贝并且将文件写入到hfds

printTokens(jobId, job.getCredentials());

submitJob（jobId, submitJobDir.toString(), job.getCredentials()）提交job

int maps = writeSplits(job, submitJobDir);job分割切片

writeSplits（）-》maps = writeNewSplits(job, jobSubmitDir);-》 List splits = input.getSplits(job);//FileInputFormat获取切片

List getSplits(JobContext job)-》long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));