Dagermeister 🤯

Build complex Airflow DAGs using drag-and-drop

---

Data engineering has never been easier.

Dagermeister is powered by Typescript, Pydantic, FastAPI and Airflow's Dagbag. It produces a Python DAG file that can be easily deployed to your Airflow cluster.

# -*- coding: utf-8 -*-
# This file was generated by a DAG factory engine
# Do not edit this file directly. Your changes will be overwritten by any working CI-CD pipeline.

from airflow import DAG
import dagfactory

# declare configuration
configuration = {'dag': {
    'default_args': {
        'owner': 'example_owner',
        'start_date': '2023-01-01',
        'end_date': '2023-01-05',
        'retries': 1,
        'retry_delay_sec': 300,
        },
    'schedule_interval': '0 3 * * *',
    'concurrency': 1,
    'max_active_runs': 1,
    'dagrun_timeout_sec': 60,
    'default_view': 'tree',
    'orientation': 'LR',
    'description': 'Generated by Airflow-UI',
    'tasks': {
        '2': {'operator': 'BashOperator', 'dependencies': ['1']},
        '3': {'operator': 'DummyOperator', 'dependencies': ['1']},
        '4': {
            'operator': 'PythonOperator',
            'dependencies': ['2'],
            'task_id': '',
            'python_callable': "def main():\n  print('idk')",
            'op_kwargs': '',
            },
        'test': {
            'operator': 'PythonOperator',
            'dependencies': ['4'],
            'task_id': '',
            'python_callable': "def main():\n  print('hello world')",
            'op_kwargs': '',
            },
        },
    }}

dag_factory = dagfactory.DagFactory(config=configuration)

dag_factory.clean_dags(globals())
dag_factory.generate_dags(globals())

The DagFactory itself is built on top of @ajbosco's dag-factory.

Important Notes

I have actively been designing the Dagermeister's architecture over the past year and there is still a lot to do. I am open-sourcing this at the moment for feature suggestions from the data engineering community.