SAP Knowledge Base Article - Preview

2819085 - Execute pyspark job on Hadoop cluster errors using python 3 SAP

Symptom

  • Executing a python 3 script which is intended to execute on the Hadoop Cluster as if it were a spark-scala job with the following command:
     
    /opt/spark/bin/spark-submit --master yarn --deploy-mode cluster --num-executors 23 --executor-cores 5 --executor-memory 35GB profiling.py
     
  • The job is submitted but fails with the following error message:

    ERROR org.apache.spark.deploy.yarn.ApplicationMaster (Logging.scala:logError(91)) - User class threw exception: java.io.IOException: Cannot run /usr/miniconda3/bin/python - execve failed (ENOENT - No such file or directory) 


Read more...

Environment

  • SAP Cloud Platform Big Data Services (BDS) 4.3
  • SAP Cloud Platform Big Data Services (BDS) 5.0

Product

SAP BusinessObjects Predictive Analytics 3.0

Keywords

Altiscale, Portal, HDFS

Hive, Spark, Pig, Tez, Oozie, Alation, MapReduce, SparkContext, YARN , KBA , BC-NEO-BDS , HCP Big Data Service , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP ONE Support launchpad (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.