Skip to main content
Warning: You are using the test version of PyPI. This is a pre-production deployment of Warehouse. Changes made here affect the production instance of TestPyPI (testpypi.python.org).
Help us improve Python packaging - Donate today!

A low-overhead sampling profiler for PySpark, that outputs Flame Graphs

Project Description
# pyspark-flame
A low-overhead profiler for Spark on Python

Pyspark-flame hooks into Pyspark's existing profiling capabilities to provide a
low-overhead stack-sampling profiler, that outputs performance data in a
format compatible with
[Brendan Gregg's FlameGraph Visualizer](https://github.com/brendangregg/FlameGraph).

Because pyspark-flame hooks into Pyspark's profiling capabilities, it can profile
the entire execution of an RDD, across the whole of the cluster, and provides
RDD-level visibility of performance.

Unlike the cProfile-based profiler included with Pyspark, pyspark-flame uses
stack sampling. It takes stack traces at regular (configurable) intervals,
which allows its overhead to be low and tunable, and doesn't skew results,
making it suitable for use in performance test environments at high volumes.

## Installation

```bash
pip install pyspark-flame
```

## Usage

```python
from pyspark_flame import FlameProfiler
from pyspark import SparkConf, SparkContext

conf = SparkConf().set("spark.python.profile", "true")
conf = conf.set("spark.python.profile.dump", ".") # Optional - if not, dumps to stdout at exit
sc = SparkContext(
'local', 'test', conf=conf, profiler_cls=FlameProfiler,
environment={'pyspark_flame.interval': 0.25} # Optional - default is 0.2 seconds
)
# Do stuff with Spark context...
sc.show_profiles()
# Or maybe
sc.dump_profiles('.')
```

For convenience, flamegraph.pl is vendored in, so you can produce a flame graph
with:

```bash
flamegraph.pl rdd-1.flame > rdd-1.svg
```
Release History

Release History

This version
History Node

0.2.3

History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
pyspark-flame-0.2.3.tar.gz (15.6 kB) Copy SHA256 Checksum SHA256 Source Aug 16, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting