Skip to content

shengjh/pyspark-profile-udf

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pyspark-profile-udf. Add simple udf profiler for pyspark.

  • Requirements

    1. flameprof
    pip install flameprof
  • How to use

    1. Rename this project to pyspark
    2. Compress this project to pyspark.zip: zip -r pyspark pyspark And move it to $SPARK_HOME/python/lib
    3. Enable pyspark profile in your application code, and set the profile dump path
    spark_session = SparkSession \
          .builder \
          .appName("Python Arrow-in-Spark profile") \
          .config("spark.python.profile", "true") \
          .config("spark.python.profile.dump", profile_dump_path) \
          .getOrCreate()

About

Add simple SQL udf profiler for pyspark.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages