Skip to content

Hadoop Gotcha Type mismatch in key from map

Paul Houle edited this page Jun 16, 2014 · 1 revision

Type mismatch in key from map (expected)

Here's an error message that had me scratching my head a bit: (and yes, spelling is "correct")

2013-09-22 22:23:49,910 ERROR com.ontology2.bakemono.pse3.PSE3Mapper: Caught exception in PSE3Mapper
 java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable, recieved com.ontology2.bakemono.jena.WritableTriple
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1014)
 at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
 at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
 at com.ontology2.bakemono.abstractions.PrimaryKeyValueAcceptor.write(PrimaryKeyValueAcceptor.java:16)
 at com.ontology2.bakemono.pse3.PSE3Mapper.map(PSE3Mapper.java:62)
 at com.ontology2.bakemono.pse3.PSE3Mapper.map(PSE3Mapper.java:1)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)

This problem had a simple cause: I had removed the code that set the mapOutputKeyClass and related types from the Tool class that sets up the job that uses the PSE3Mapper

job.setMapOutputKeyClass(WritableTriple.class);
job.setMapOutputValueClass(LongWritable.class);
job.setOutputKeyClass(Triple.class);
job.setOutputValueClass(LongWritable.class);

when I put this code back in, everything worked. This problem was observed with my 1.1.2 dev cluster and the 1.0.3-derivative EMR system. The cause of the message is likely that the key value defaults to LongWritable.class

Be proactive: always set output key classes!

This problem can be avoided by developing the habit of always setting the output classes when you set up a job.

Keyphrase: Type mismatch in key from map

Clone this wiki locally