Skip to content
This repository has been archived by the owner on Jan 3, 2023. It is now read-only.

YARN and Map-reduce example error #1732

Closed
qiyuangong opened this issue May 11, 2018 · 1 comment
Closed

YARN and Map-reduce example error #1732

qiyuangong opened this issue May 11, 2018 · 1 comment
Assignees

Comments

@qiyuangong
Copy link
Contributor

Reason:

  1. “smart.server.rpc.address” not configured after deploying SSM’s jars. Please check configuration in CDH.
  2. SSM not started after deploying SSM’s jars. We have found key point of this issue, and are working on it. (Will patch to release-1.3.2 and trunk this week).

Command example:

sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name=root.production.streaming 3 3

Running map-reduce example in command line raise errors:

java.net.ConnectException: Call From bb-node3/192.168.1.103 to localhost:7042 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/Co
nnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
        at org.apache.hadoop.ipc.Client.call(Client.java:1506)
        at org.apache.hadoop.ipc.Client.call(Client.java:1439)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
        at com.sun.proxy.$Proxy25.getFileState(Unknown Source)
        at org.smartdata.protocol.protobuffer.ClientProtocolClientSideTranslator.getFileState(ClientProtocolClientSideTranslator.java:67)
        at org.smartdata.client.SmartClient.getFileState(SmartClient.java:95)
        at org.smartdata.hdfs.client.SmartDFSClient.open(SmartDFSClient.java:133)
        at org.smartdata.hadoop.filesystem.SmartFileSystem.open(SmartFileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:784)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:367)
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:648)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:744)
        at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
        at org.apache.hadoop.ipc.Client.call(Client.java:1478)
        ... 24 more
Caused by: Connection refused

@qiyuangong qiyuangong self-assigned this May 11, 2018
qiyuangong added a commit that referenced this issue May 13, 2018
* Ensure that YARN and MapReduce work well even if SSM is shutdown. 
* Suppress unnecessary error logs caused by reportFileAccessEvent.
qiyuangong added a commit that referenced this issue May 14, 2018
* Ensure that YARN and MapReduce work well even if SSM is shutdown. 
* Suppress unnecessary error logs caused by reportFileAccessEvent.
@qiyuangong
Copy link
Contributor Author

Already fixed. Issue closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant