Skip to content

🧱 A collection of supplementary utilities and helper notebooks to perform admin tasks on Databricks

License

Notifications You must be signed in to change notification settings

dotlas/databricks_helpers

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

82 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Databricks Helpers 🧱

Databricks Delta

Plotly Pydantic

Easy-to-use Databricks Notebooks for Admin Tasks.
Made with ❤️ by Dotlas Inc

About

This repository contains a directory of Databricks notebooks that assists with administrative tasks for Databricks, or otherwise helps as a supporting utility.

For example, consider the following use-cases:

  • 📆 View a calendar of scheduled jobs to resolve conflicts in Databricks workflows.
  • 🐼 Upload a Pandas DataFrame to Delta Lake
  • 📑 Update Delta Lake table Documentation using Pydantic Models
  • ➿ Migrate Jobs between Databricks workspaces
  • ⚙️ Mass-edit Job Clusters in Existing Jobs

Directory

Notebook Description
Databricks Daily Cost to Slack Schedule a daily cost summary of Databricks bills to notify you on Slack.
Workflow Calendar Visualize scheduled Jobs on a calendar, eyeball conflicts and view historic runs as a Gantt chart
Delta Docs with Pydantic If you have pydantic models with fields containing description and tags that are used as data models, transfer these field descriptions to Delta lake columns as comments and tags.
Pandas to Delta Use databricks-sql-python and SQLAlchemy to upload a Pandas DataFrame to Delta Lake from outside a Databricks environment
Workspace Jobs Migration Migrate Workflows from one Databricks workspace to another
Job Cluster Update Use the Databricks API to mass-update Job and Task configs
Workflow Config Exporter Export existing workflow configuration and save it for future consumption

Discussions

  • Check out the launch discussion on this LinkedIn Release Post, with a highlight from Databricks CEO, Ali Ghodsi.
  • Feel free to raise an issue on this repository to start a discussion about new features, bug fixes or enhancements.
  • See CONTRIBUTING.md for guidelines when adding or modifying notebooks in this repository.