Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide data files externally to python package #55

Open
DavidLP opened this issue Feb 9, 2017 · 8 comments
Open

Provide data files externally to python package #55

DavidLP opened this issue Feb 9, 2017 · 8 comments
Labels
enhancement help wanted
Projects
First Major release

Comments

@DavidLP
Copy link
Collaborator

DavidLP commented Feb 9, 2017

To run unit tests and provide example data one needs to have rather large data files. These files should and cannot (max. quota ~ 250 MB) be stored in the python package hosted on pypi. I did not find a good solution to do this. The best would be a python package providing autodowload of files on demand for several supported file hosters. Otherwise github could be misused to a certain extend, by adding a data branch and downloading from there...
@DavidLP
Copy link
Collaborator Author

DavidLP commented Feb 17, 2017 •
edited
Loading

We have 150 Mb per branch (because of the data files) and 6 branches = 6 * 150 = 900 Mb. Github allows for 1 Gb before you get a "polite" E-Mail. Thus one more branch is not supported anymore and we have to find a solution asap. I will create a data branch within testbeam analysis and add a downloader tool for the exampels.

@DavidLP DavidLP added this to To Do in First Major release May 18, 2017
@DavidLP
Copy link
Collaborator Author

DavidLP commented May 22, 2017

Did not come up with a good solution yet. Hosting at University is not a trivial task and you do not want to start username/psw access restrictions.

@DavidLP
Copy link
Collaborator Author

DavidLP commented Jul 18, 2017

https://quiltdata.com/ look promising

DavidLP added a commit that referenced this issue Aug 30, 2017
@DavidLP
PRJ: start hosting data on quilt #55
7adb724
DavidLP added a commit that referenced this issue Aug 31, 2017
@DavidLP
PRJ: start hosting data on quilt #55
9d35889
@DavidLP DavidLP mentioned this issue Sep 19, 2017
Merged
@DavidLP DavidLP moved this from To Do to In progress in First Major release Sep 19, 2017
DavidLP added a commit that referenced this issue Sep 19, 2017
@DavidLP
PRJ: start hosting data on quilt #55
23b92ed
@DavidLP
Copy link
Collaborator Author

DavidLP commented Sep 19, 2017

Implemented in development

@DavidLP DavidLP closed this as completed Sep 19, 2017
@DavidLP DavidLP moved this from In progress to Done in First Major release Sep 19, 2017
@DavidLP
Copy link
Collaborator Author

DavidLP commented Dec 4, 2017 •
edited
Loading

The Scibo campuscloud does not seem to be planned as a high availability cloud, frequent downtime of up to days (when it is weekend) are common. Additionally the downtime (an important classifier for an online hoster) seems not to be disclosed. This does not meet our requirementa, since the downtime stalls our tests and builds. We should search for a better service. I reopen this issue.

@DavidLP DavidLP reopened this Dec 4, 2017
@themperek
Copy link
Member

themperek commented Dec 4, 2017 •
edited
Loading

https://help.github.com/articles/distributing-large-binaries/ (seperate repo?)

@DavidLP
Copy link
Collaborator Author

DavidLP commented Dec 5, 2017

This seems to be exactly what we want, even in same repo: https://git-lfs.github.com/

@laborleben
Copy link
Collaborator

This is maybe the only option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted
Projects
First Major release
Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@themperek @DavidLP @laborleben