-
Notifications
You must be signed in to change notification settings - Fork 21
Hadoop Gotcha Type mismatch in key from map
Paul Houle edited this page Jun 16, 2014
·
1 revision
Here's an error message that had me scratching my head a bit: (and yes, spelling is "correct")
2013-09-22 22:23:49,910 ERROR com.ontology2.bakemono.pse3.PSE3Mapper: Caught exception in PSE3Mapper
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable, recieved com.ontology2.bakemono.jena.WritableTriple
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1014)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.ontology2.bakemono.abstractions.PrimaryKeyValueAcceptor.write(PrimaryKeyValueAcceptor.java:16)
at com.ontology2.bakemono.pse3.PSE3Mapper.map(PSE3Mapper.java:62)
at com.ontology2.bakemono.pse3.PSE3Mapper.map(PSE3Mapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
This problem had a simple cause: I had removed the code that set the mapOutputKeyClass
and related types from the Tool
class that sets up the job that uses the PSE3Mapper
job.setMapOutputKeyClass(WritableTriple.class);
job.setMapOutputValueClass(LongWritable.class);
job.setOutputKeyClass(Triple.class);
job.setOutputValueClass(LongWritable.class);
when I put this code back in, everything worked. This problem was observed with my 1.1.2 dev cluster and the 1.0.3-derivative EMR system. The cause of the message is likely that the key value defaults to LongWritable.class
This problem can be avoided by developing the habit of always setting the output classes when you set up a job.
Keyphrase: Type mismatch in key from map