Wednesday 1 March 2017


Oozie workflow shows in running state even after job is completed successfully


Issue:


Oozie workflow shows to be in running state even after the job has completed successfully. oozie job -info <workflow_ID> will show that job has completed successfully. However in web UI it still remains as running.
If we try to kill the workflow in oozie using oozie job -kill <workflow_ID>, it will throw the following error:

Error: E0607 : E0607: Other error in operation [kill], java.io.EOFException

In the oozie.log you can find following exception: (exception captured while trying to suspend the job)

2017-03-01 15:10:59,753  WARN V2JobServlet:523 - SERVER[phpvcoredev03.chicago.local] USER[vcoredevuser] GROUP[-] TOKEN[] APP[Lab-EdwardLabResults] JOB[0000237-160624170341227-oozie-mapr-W] ACTION[] URL[PUT http://phpvcoredev03:11000/oozie/v2/job/0000237-160624170341227-oozie-mapr-W?action=suspend] error[E0607], E0607: Other error in operation [suspend], java.io.EOFException
org.apache.oozie.servlet.XServletException: E0607: Other error in operation [suspend], java.io.EOFException
        at org.apache.oozie.servlet.V1JobServlet.suspendWorkflowJob(V1JobServlet.java:430)
        at org.apache.oozie.servlet.V1JobServlet.suspendJob(V1JobServlet.java:127)
        at org.apache.oozie.servlet.BaseJobServlet.doPut(BaseJobServlet.java:92)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:304)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:604)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:567)
        at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

Resolution:


Delete the job from the oozie database.
The entry must be deleted from both WF_JOBS and WF_ACTIONS table.

No comments:

Post a Comment