Skip to main content
Warning: You are using the test version of PyPI. This is a pre-production deployment of Warehouse. Changes made here affect the production instance of TestPyPI (testpypi.python.org).
Help us improve Python packaging - Donate today!

Generic package to run SQL/Scala/Python queries/jobs on Databricks using the Databricks REST API

Project Description

DataBricks REST API

This package requires a DataBricks account, to sign up click here https://databricks.com/try-databricks

This package allows the user to perform the following actions on DataBricks using the REST API:

  • Run commands remotely using Scala/Python/Spark SQL
  • Create and Run jobs remotely

The following classes are defined:

  • DatabricksQuerySet: DatabricksQuerySet is a generic class to run SQL/Scala/Python queries on Databricks using the Databricks REST API (Currently this supports Databricks REST API version 1.2).
  • RunJob: Create a job using 2.0 API with a option to run the job.

Authentication and Account

This is required to run the tests and is the recommended way of using this package as it requires authentication

Store password and username as environment variables:

  • export DATABRICKS_PASSWORD=”mypassword”
  • export DATABRICKS_USERNAME=”myemail”

Required DataBricks Account details:

  • export DATABRICKS_DOMAIN=”url-to-databricks-account”
  • export ADMIN_EMAIL=”admin-email” this is the email of the admin to receive notification about job

DataBricks Queries

To use DatabricksQuerySet:

  • Create an instance of DatabricksQuerySet and provide username and password. ie; dq = DatabricksQuerySet(username, password)
  • Run run_databricks_command method to submit a query.

DataBricks Jobs

To use RunJob:

  • Create an instance of RunJob and provide username, password and job name. ie; job = RunJob(username, password, “Test”)
  • Run run_job_now method to execute the job.

Tests

** Requires authentication credentials to be configured as environment variables **

Tests basic functionality of DatabricksQuerySet and RunJob classes:

  • To run tests: $ python src/tests/test_simple.py
Release History

Release History

This version
History Node

1.0.2a0

History Node

1.0.1a0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
databricks-queryset-1.0.2a0.tar.gz (5.9 kB) Copy SHA256 Checksum SHA256 Source Oct 20, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting