Friday, November 4, 2016

Pig UDF Error input.get(0) : The type org.apache.hadoop.io.WritableComparable cannot be resolved. It is indirectly referenced from required .class files

Error: The type org.apache.hadoop.io.WritableComparable cannot be resolved. It is indirectly referenced from required .class files. Getting error in Pig UDF when trying to get the value of a field from Tuple.


public String exec(Tuple input) throws IOException {
if(null == input || input.size()==0)return null;try{epochTime = Long.parseLong((String)input.get(0));//this line gives compilation error in eclipse}catch(Exception ex ){throw new IOException("Caught exception processing input "+input, ex);}

Solution: For Maven project, the below dependency needs to be there for the above compilation error.

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.0.0-cdh4.7.1</version>
    <!-- <scope>provided</scope> -->
</dependency>

<dependency>
    <groupId>org.apache.pig</groupId>
    <artifactId>pig</artifactId>
    <version>0.11.0-cdh4.7.1</version>
 <!-- <scope>provided</scope> -->
</dependency> 

Some times the same error message is seen for other missing jars like commons-logging or any other jar (hadoop related). We can find the jar name for the missing class and get the maven repo entry for that jar.