Skip to content
This repository has been archived by the owner on Dec 18, 2019. It is now read-only.

Provide a setuptools-based build infrastructure #93

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Provide a setuptools-based build infrastructure #93

wants to merge 3 commits into from

Conversation

languitar
Copy link

Included in this pull request is a complete build system based on setuptools, as I requested in #91. I suspect there will be a lot of questions or doubts about such a big change, but I am happy to answer them. So I'd really be glad if this would be accepted eventually.

commit message

This rather longish commit provides a complete build infrastructure
based on setuptools that allows to install skyline like any other python
application.

For setuptools to work correctly, a few fundamental changes had to be
made apart from minor fixes. The most fundamental changes are:

  • Inside the src directory a dedicated skyline folder / package has been
    created to ensure that the skyline classes do not interfere with other
    python packages.
  • The python scripts called by the bash scripts that encapsulate the
    real functionality of skyline are now generated by setuptools. For
    this to work the scripts had to be converted to real modules, which
    means that a dash in the name was not acceptable. In case anyone has used
    these scripts directly, the name has now changed. It might be possible
    to restore the behavior by adding symlinks or small bash wrapper
    scripts inside the src tree, but this isn't done so far. The general
    bash scripts in bin reflect the changed names.
  • The settings.py.example has been renamed to settings.py so that
    setuptools can directly execute the unit tests without requiring user
    interaction. In the future, providing a settings.py with defaults that do
    not require any user interaction, might make skyline completely usable
    out of the box.

In detail, the following changes have been made:

  • .gitignore: added stuff generated by setuptools, removed settings.py,
    which is now part of the distribution so that unit tests can execute
    without user interaction
  • .travis.yml: updated to new build system. Unit tests now executed
    through setup.py. Calling nose is still posible, though
  • MANIFEST.in: added a setuptools manifest to package the static files
    for the webapp
  • bin/*.d: updated to new script names for in-source use. Moreover, an
    if-else branch was added to decided whether the scripts should call
    the python files directly when called inside the source tree, or the
    generated setuptools wrapper scripts in case of a real installation
  • setup.cfg: contains settings for running the nose tests from within
    setuptools by calling python setup.py test
  • setup.py: The real build system logic. Some values are quite empty
    still, since I do not know the correct contents. Please updated
    accordingly.
  • src: moved all files to skyline package, made every folder a python
    package by adding an ini.py file to be discoverable by setuptools.
  • src//-agent.py: renamed to agent.py to have a python module name so
    that setuptools can use these modules as entry point for console
    scripts. Moreover, executed script contents were moved to a run
    function which is then callable as a setuptools entry point. Module
    main functionality has been preserved by calling the run function inside
    the main block. Several smaller content changes had to be made to
    preserve the functionality after these changes.
  • src/.py, tests/.py: Manual changes to the pythonpath have been removed since
    with a real package structure, everything is now done automatically by
    python and for testing by the setuptools nose integration.
  • webapp.py: added a route to server anomalies.json from any configured
    location to /anomalies.json. This is necessary, since users should not
    modify the contents of the installed python egg just to push the
    updated anomaly detections to a path that is served by the webserver.
  • skyline.js: Fetch anomalies from /anomalies.json as described above.
    This enables the user to specify any location he wants for the
    generated json file.
  • horizon/listen.py: Explicitly added the StringIO import. No idea how
    this could work before.
  • utils/*.py: Updated imports to reflect new package structure

This rather longish commit provides a complete build infrastructure
based on setuptools that allows to install skyline like any other python
application.

For setuptools to work correctly, a few fundamental changes had to be
made apart from minor fixes. The most fundamental changes are:

* Inside the src directory a dedicated skyline folder / package has been
  created to ensure that the skyline classes do not interfere with other
  python packages.
* The python scripts called by the bash scripts that encapsulate the
  real functionality of skyline are now generated by setuptools. For
  this to work the scripts had to be converted to real modules, which
  means that a dash in the name was not acceptable. In case anyone has used
  these scripts directly, the name has now changed. It might be possible
  to restore the behavior by adding symlinks or small bash wrapper
  scripts inside the src tree, but this isn't done so far. The general
  bash scripts in bin reflect the changed names.
* The settings.py.example has been renamed to settings.py so that
  setuptools can directly execute the unit tests without requiring user
  interaction. In the future, providing a settings.py with defaults that do
  not require any user interaction, might make skyline completely usable
  out of the box.

In detail, the following changes have been made:

* .gitignore: added stuff generated by setuptools, removed settings.py,
  which is now part of the distribution so that unit tests can execute
  without user interaction
* .travis.yml: updated to new build system. Unit tests now executed
  through setup.py. Calling nose is still posible, though
* MANIFEST.in: added a setuptools manifest to package the static files
  for the webapp
* bin/*.d: updated to new script names for in-source use. Moreover, an
  if-else branch was added to decided whether the scripts should call
  the python files directly when called inside the source tree, or the
  generated setuptools wrapper scripts in case of a real installation
* setup.cfg: contains settings for running the nose tests from within
  setuptools by calling python setup.py test
* setup.py: The real build system logic. Some values are quite empty
  still, since I do not know the correct contents. Please updated
  accordingly.
* src: moved all files to skyline package, made every folder a python
  package by adding an __ini__.py file to be discoverable by setuptools.
* src/*/*-agent.py: renamed to agent.py to have a python module name so
  that setuptools can use these modules as entry point for console
  scripts. Moreover, executed script contents were moved to a run
  function which is then callable as a setuptools entry point. Module
  main functionality has been preserved by calling the run function inside
  the main block. Several smaller content changes had to be made to
  preserve the functionality after these changes.
* src/*.py, tests/*.py: Manual changes to the pythonpath have been removed since
  with a real package structure, everything is now done automatically by
  python and for testing by the setuptools nose integration.
* webapp.py: added a route to server anomalies.json from any configured
  location to /anomalies.json. This is necessary, since users should not
  modify the contents of the installed python egg just to push the
  updated anomaly detections to a path that is served by the webserver.
* skyline.js: Fetch anomalies from /anomalies.json as described above.
  This enables the user to specify any location he wants for the
  generated json file.
* horizon/listen.py: Explicitly added the StringIO import. No idea how
  this could work before.
* utils/*.py: Updated imports to reflect new package structure
@astanway
Copy link
Contributor

Wow, thanks for all this! Will go through it, hopefully everything checks out.

Oops, seems I missed this change for the initial commit.
@languitar
Copy link
Author

I just noticed I missed a hunk. It is added now to the pull request.

earthgecko added a commit to earthgecko/skyline that referenced this pull request Jul 17, 2016
- Added a variant of @astanway Crucible into the Skyline
- Added a Crucible app so that ad-hoc timeseries can be feed to it and analyzed
- Restructured the Skyline layout to be more inline with a Python package and
  setuptools with a merge of @languitar changes for setuptools and more pythonic
  structure that was submitted as per:
  etsy/skyline#93
  etsy/skyline#91
  Provide a setuptools-based build infrastructure #93 - etsy#91
- Not totally setuptools compliant yet
- Added sphinx docs and documentation build pattern
- More documentation
- Serve sphinx documentation via webapp /static/docs/
- Handle pandas versions changes to - PANDAS_VERSION
- Attempted to handle logging without log overwrites, not pretty but works
- Performance profiled - pprofile, RunSnakeRun, cProfile, vmprof, snakeviz
- deroomba - kill any lingering vacuum processes - ROOMBA_TIMEOUT
- Self monitor Analyzer spin_process threads and terminate if any spin_process
  has run for longer than 180 seconds - MAX_ANALYZER_PROCESS_RUNTIME
- Optimizations to Analyzer workflow logic - RUN_OPTIMIZED_WORKFLOW
- Some general code optimizations based on profiling results
- Addition of algorithm_breakdown metrics - ENABLE_ALGORITHM_RUN_METRICS and
  SKYLINE_TMP_DIR, tmpfs over multiprocessing Value
- Updated current requirements
- Patterned and tested in Python virtualenv for ease of python version, package
  management and dependencies management
- More documentation
- More documentation
- More documentation
- More documentation, docstrings, docstrings, docstrings
- Panorama modules files and related Webapp UI changes
- rebrow (added https://github.com/marians/rebrow)
- Upgraded Webapp UI jquery, dygraph, bootstrap
- Tested all Skyline components on Python 2.7.11 and 2.7.12

Added:
CHANGES.md
bin/analyzer_dev.d
bin/crucible.d
bin/panorama.d
bin/skyline.d
docs/Makefile
docs/_build/html/.buildinfo
docs/_build/html/_images/mirage-1.png
docs/_build/html/_images/nupic.radar.predicted.14.month.requests.png
docs/_build/html/_images/nupic.radar.real.predicted.difference.14.month.requests.overlayed.png
docs/_build/html/_images/nupic.radar.real.predicted.difference.14.month.requests.png
docs/_build/html/_images/panorama.closest.approximation.aggregrated.png
docs/_build/html/_images/radar.real.14.month.requests.png
docs/_build/html/_modules/algorithm_exceptions.html
docs/_build/html/_modules/analyzer/agent.html
docs/_build/html/_modules/analyzer/alerters.html
docs/_build/html/_modules/analyzer/algorithms.html
docs/_build/html/_modules/analyzer/analyzer.html
docs/_build/html/_modules/analyzer_dev/agent.html
docs/_build/html/_modules/analyzer_dev/alerters.html
docs/_build/html/_modules/analyzer_dev/algorithms_dev.html
docs/_build/html/_modules/analyzer_dev/analyzer_dev.html
docs/_build/html/_modules/boundary/agent.html
docs/_build/html/_modules/boundary/boundary.html
docs/_build/html/_modules/boundary/boundary_alerters.html
docs/_build/html/_modules/boundary/boundary_algorithms.html
docs/_build/html/_modules/crucible/agent.html
docs/_build/html/_modules/crucible/crucible.html
docs/_build/html/_modules/crucible/crucible_algorithms.html
docs/_build/html/_modules/horizon/agent.html
docs/_build/html/_modules/horizon/listen.html
docs/_build/html/_modules/horizon/roomba.html
docs/_build/html/_modules/horizon/worker.html
docs/_build/html/_modules/index.html
docs/_build/html/_modules/logging.html
docs/_build/html/_modules/mirage/agent.html
docs/_build/html/_modules/mirage/mirage.html
docs/_build/html/_modules/mirage/mirage_alerters.html
docs/_build/html/_modules/mirage/mirage_algorithms.html
docs/_build/html/_modules/mirage/negaters.html
docs/_build/html/_modules/panorama/agent.html
docs/_build/html/_modules/panorama/panorama.html
docs/_build/html/_modules/skyline_functions.html
docs/_build/html/_modules/webapp/backend.html
docs/_build/html/_modules/webapp/webapp.html
docs/_build/html/_sources/alert-testing.txt
docs/_build/html/_sources/analyzer-optimizations.txt
docs/_build/html/_sources/analyzer.txt
docs/_build/html/_sources/boundary.txt
docs/_build/html/_sources/building-documentation.txt
docs/_build/html/_sources/crucible.txt
docs/_build/html/_sources/debian-and-vagrant-installation-tips.txt
docs/_build/html/_sources/development/index.txt
docs/_build/html/_sources/development/webapp.txt
docs/_build/html/_sources/getting-data-into-skyline.txt
docs/_build/html/_sources/getting-started.txt
docs/_build/html/_sources/horizon.txt
docs/_build/html/_sources/index.txt
docs/_build/html/_sources/installation.txt
docs/_build/html/_sources/logging.txt
docs/_build/html/_sources/mirage.txt
docs/_build/html/_sources/modules.txt
docs/_build/html/_sources/monitoring-skyline.txt
docs/_build/html/_sources/overview.txt
docs/_build/html/_sources/panorama.txt
docs/_build/html/_sources/redis-integration.txt
docs/_build/html/_sources/releases.txt
docs/_build/html/_sources/releases/1_0_0.txt
docs/_build/html/_sources/requirements.txt
docs/_build/html/_sources/roadmap.txt
docs/_build/html/_sources/running-in-python-virtualenv.txt
docs/_build/html/_sources/skyline-and-friends.txt
docs/_build/html/_sources/skyline.analyzer.txt
docs/_build/html/_sources/skyline.analyzer_dev.txt
docs/_build/html/_sources/skyline.boundary.txt
docs/_build/html/_sources/skyline.crucible.txt
docs/_build/html/_sources/skyline.horizon.txt
docs/_build/html/_sources/skyline.mirage.txt
docs/_build/html/_sources/skyline.panorama.txt
docs/_build/html/_sources/skyline.txt
docs/_build/html/_sources/skyline.webapp.txt
docs/_build/html/_sources/tuning-tips.txt
docs/_build/html/_sources/upgrading.txt
docs/_build/html/_sources/webapp.txt
docs/_build/html/_sources/whats-new.txt
docs/_build/html/_static/ajax-loader.gif
docs/_build/html/_static/basic.css
docs/_build/html/_static/comment-bright.png
docs/_build/html/_static/comment-close.png
docs/_build/html/_static/comment.png
docs/_build/html/_static/css/badge_only.css
docs/_build/html/_static/css/theme.css
docs/_build/html/_static/doctools.js
docs/_build/html/_static/down-pressed.png
docs/_build/html/_static/down.png
docs/_build/html/_static/file.png
docs/_build/html/_static/fonts/Inconsolata-Bold.ttf
docs/_build/html/_static/fonts/Inconsolata-Regular.ttf
docs/_build/html/_static/fonts/Lato-Bold.ttf
docs/_build/html/_static/fonts/Lato-Regular.ttf
docs/_build/html/_static/fonts/RobotoSlab-Bold.ttf
docs/_build/html/_static/fonts/RobotoSlab-Regular.ttf
docs/_build/html/_static/fonts/fontawesome-webfont.eot
docs/_build/html/_static/fonts/fontawesome-webfont.svg
docs/_build/html/_static/fonts/fontawesome-webfont.ttf
docs/_build/html/_static/fonts/fontawesome-webfont.woff
docs/_build/html/_static/jquery-1.11.1.js
docs/_build/html/_static/jquery.js
docs/_build/html/_static/js/modernizr.min.js
docs/_build/html/_static/js/theme.js
docs/_build/html/_static/minus.png
docs/_build/html/_static/plus.png
docs/_build/html/_static/pygments.css
docs/_build/html/_static/searchtools.js
docs/_build/html/_static/skyline.styles.css
docs/_build/html/_static/underscore-1.3.1.js
docs/_build/html/_static/underscore.js
docs/_build/html/_static/up-pressed.png
docs/_build/html/_static/up.png
docs/_build/html/_static/websupport.js
docs/_build/html/alert-testing.html
docs/_build/html/analyzer-optimizations.html
docs/_build/html/analyzer.html
docs/_build/html/boundary.html
docs/_build/html/building-documentation.html
docs/_build/html/crucible.html
docs/_build/html/debian-and-vagrant-installation-tips.html
docs/_build/html/development/index.html
docs/_build/html/development/webapp.html
docs/_build/html/genindex.html
docs/_build/html/getting-data-into-skyline.html
docs/_build/html/getting-started.html
docs/_build/html/horizon.html
docs/_build/html/index.html
docs/_build/html/installation.html
docs/_build/html/logging.html
docs/_build/html/mirage-1.hires.png
docs/_build/html/mirage-1.pdf
docs/_build/html/mirage-1.png
docs/_build/html/mirage-1.py
docs/_build/html/mirage.html
docs/_build/html/modules.html
docs/_build/html/monitoring-skyline.html
docs/_build/html/objects.inv
docs/_build/html/overview.html
docs/_build/html/panorama.html
docs/_build/html/py-modindex.html
docs/_build/html/redis-integration.html
docs/_build/html/releases.html
docs/_build/html/releases/1_0_0.html
docs/_build/html/requirements.html
docs/_build/html/roadmap.html
docs/_build/html/running-in-python-virtualenv.html
docs/_build/html/search.html
docs/_build/html/searchindex.js
docs/_build/html/skyline-and-friends.html
docs/_build/html/skyline.analyzer.html
docs/_build/html/skyline.analyzer_dev.html
docs/_build/html/skyline.boundary.html
docs/_build/html/skyline.crucible.html
docs/_build/html/skyline.horizon.html
docs/_build/html/skyline.html
docs/_build/html/skyline.mirage.html
docs/_build/html/skyline.panorama.html
docs/_build/html/skyline.webapp.html
docs/_build/html/tuning-tips.html
docs/_build/html/upgrading.html
docs/_build/html/webapp.html
docs/_build/html/whats-new.html
docs/_build/plot_directive/mirage-1.hires.png
docs/_build/plot_directive/mirage-1.pdf
docs/_build/plot_directive/mirage-1.png
docs/_static/skyline.styles.css
docs/alert-testing.rst
docs/analyzer-optimizations.rst
docs/analyzer.rst
docs/boundary.rst
docs/building-documentation.md
docs/conf.py
docs/crucible.rst
docs/debian-and-vagrant-installation-tips.md
docs/development/index.rst
docs/development/webapp.rst
docs/getting-data-into-skyline.rst
docs/getting-started.rst
docs/horizon.md
docs/images/nupic.radar.predicted.14.month.requests.png
docs/images/nupic.radar.real.predicted.difference.14.month.requests.overlayed.png
docs/images/nupic.radar.real.predicted.difference.14.month.requests.png
docs/images/panorama.closest.approximation.aggregrated.png
docs/images/radar.real.14.month.requests.png
docs/index.rst
docs/installation.rst
docs/logging.md
docs/mirage.rst
docs/modules.rst
docs/monitoring-skyline.md
docs/overview.rst
docs/panorama.rst
docs/redis-integration.md
docs/releases.rst
docs/releases/1_0_0.rst
docs/requirements.rst
docs/roadmap.rst
docs/running-in-python-virtualenv.rst
docs/skyline-and-friends.md
docs/skyline.analyzer.rst
docs/skyline.analyzer_dev.rst
docs/skyline.boundary.rst
docs/skyline.crucible.rst
docs/skyline.horizon.rst
docs/skyline.mirage.rst
docs/skyline.panorama.rst
docs/skyline.rst
docs/skyline.webapp.rst
docs/tuning-tips.md
docs/upgrading.rst
docs/webapp.rst
docs/whats-new.md
etc/skyline.conf
etc/skyline.httpd.conf.d.example
examples/data/Real_time_energy_data_October.csv
skyline.png
skyline/__init__.py
skyline/algorithm_exceptions.py
skyline/analyzer/__init__.py
skyline/analyzer/agent.py
skyline/analyzer/alerters.py
skyline/analyzer/algorithms.py
skyline/analyzer/analyzer.py
skyline/analyzer_dev/__init__.py
skyline/analyzer_dev/agent.py
skyline/analyzer_dev/alerters.py
skyline/analyzer_dev/algorithms_dev.py
skyline/analyzer_dev/analyzer_dev.py
skyline/boundary/LICENSE.md
skyline/boundary/__init__.py
skyline/boundary/agent.py
skyline/boundary/boundary.py
skyline/boundary/boundary_alerters.py
skyline/boundary/boundary_algorithms.py
skyline/crucible/LICENSE.md
skyline/crucible/__init__.py
skyline/crucible/agent.py
skyline/crucible/crucible.py
skyline/crucible/crucible_algorithms.py
skyline/horizon/__init__.py
skyline/horizon/agent.py
skyline/horizon/listen.py
skyline/horizon/roomba.py
skyline/horizon/worker.py
skyline/mirage/LICENSE.md
skyline/mirage/__init__.py
skyline/mirage/agent.py
skyline/mirage/mirage.py
skyline/mirage/mirage_alerters.py
skyline/mirage/mirage_algorithms.py
skyline/mirage/negaters.py
skyline/panorama/LICENSE.md
skyline/panorama/__init__.py
skyline/panorama/agent.py
skyline/panorama/panorama.py
skyline/settings.py
skyline/skyline.sql
skyline/skyline_functions.py
skyline/skyline_version.py
skyline/webapp/__init__.py
skyline/webapp/backend.py
skyline/webapp/gunicorn.py
skyline/webapp/rebrow/LICENSE
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.css
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.css.map
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.min.css
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap-theme.min.css.map
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.css
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.css.map
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.min.css
skyline/webapp/static/bootstrap-3.3.6-dist/css/bootstrap.min.css.map
skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.eot
skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.svg
skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.ttf
skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.woff
skyline/webapp/static/bootstrap-3.3.6-dist/fonts/glyphicons-halflings-regular.woff2
skyline/webapp/static/bootstrap-3.3.6-dist/js/bootstrap.js
skyline/webapp/static/bootstrap-3.3.6-dist/js/bootstrap.min.js
skyline/webapp/static/bootstrap-3.3.6-dist/js/npm.js
skyline/webapp/static/css/skyline.styles.css
skyline/webapp/static/docs
skyline/webapp/static/dump/.gitignore
skyline/webapp/static/dygraph-1.1.1/dygraph-combined.js
skyline/webapp/static/fontawesome-4.6.3/css/font-awesome.css
skyline/webapp/static/fontawesome-4.6.3/css/font-awesome.css.map
skyline/webapp/static/fontawesome-4.6.3/css/font-awesome.min.css
skyline/webapp/static/fontawesome-4.6.3/fonts/FontAwesome.otf
skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.eot
skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.svg
skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.ttf
skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.woff
skyline/webapp/static/fontawesome-4.6.3/fonts/fontawesome-webfont.woff2
skyline/webapp/static/fonts/montserrat-bold.woff
skyline/webapp/static/fonts/montserrat-regular.woff
skyline/webapp/static/images/favicon.ico
skyline/webapp/static/jquery-2.2.4/dist/jquery.js
skyline/webapp/static/jquery-2.2.4/dist/jquery.min.js
skyline/webapp/static/jquery-2.2.4/dist/jquery.min.map
skyline/webapp/static/js/cubism.v1.min.js
skyline/webapp/static/js/mousetrap.min.js
skyline/webapp/static/js/panorama.js
skyline/webapp/static/js/skyline.js
skyline/webapp/static/strftime-0.9.2/strftime-min.js
skyline/webapp/templates/docs.html
skyline/webapp/templates/layout.html
skyline/webapp/templates/now.html
skyline/webapp/templates/panorama.html
skyline/webapp/templates/rebrow_key.html
skyline/webapp/templates/rebrow_keys.html
skyline/webapp/templates/rebrow_login.html
skyline/webapp/templates/rebrow_server_db.html
skyline/webapp/templates/uh_oh.html
skyline/webapp/webapp.py
tests/test_crucible_algorithms.py
tests/test_imports.py
Modified:
bin/analyzer.d
bin/boundary.d
bin/horizon.d
bin/mirage.d
bin/webapp.d
readme.md
requirements.txt
tests/algorithms_test.py
utils/continuity.py
utils/seed_data.py
utils/verify_alerts.py
.gitignore
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants