Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure Funcotator can support NIO for data sources (including for multi-file data sources) #5348

Closed
jonn-smith opened this issue Oct 23, 2018 · 2 comments
Closed

Ensure Funcotator can support NIO for data sources (including for multi-file data sources) #5348

jonn-smith opened this issue Oct 23, 2018 · 2 comments
Assignees
@jonn-smith
Labels
Funcotator
Milestone
Engine-Q42018

Comments

@jonn-smith
Copy link
Collaborator

jonn-smith commented Oct 23, 2018 •
edited
Loading

Feature request

Tool(s) or class(es) involved

Funcotator, DataSourceFuncotationFactory

Description

Funcotator should support NIO for data sources and data sources backing files.
In addition, the data source readers should be updated to support multiple backing files to support the gnomAD case (http://gnomad.broadinstitute.org/downloads).
@jonn-smith jonn-smith self-assigned this Oct 30, 2018
@jonn-smith jonn-smith added this to the Engine-Q42018 milestone Oct 30, 2018
@jonn-smith
Copy link
Collaborator Author

Must update DataSourceUtils::assertPathFilePropertiesField to allow for absolute paths / URI paths for the data sources supporting files.

@jonn-smith
Copy link
Collaborator Author

Must check individual creation paths in DataSourceUtils::createDataSourceFuncotationFactoriesForDataSources

jonn-smith added a commit that referenced this issue Oct 30, 2018
@jonn-smith
Updated data source inputs to accept NIO paths for backing files.
1711e2c 
Now you can specify a URL in the backing file areas of the configuration
files for Funcotator data sources and the backing files will be read by
the FuncotationDataSourceFactories.

This effectively enables use of data sources in the cloud or a mix of
local- and cloud-based data sources through a config file change.

This update will enable gnomAD annotations (once the data sources are
    updated to point at the gnomAD files on Google Cloud).

Fixes #5348
jonn-smith added a commit that referenced this issue Nov 8, 2018
@jonn-smith
Updated data source inputs to accept NIO paths for backing files.
f733733 
Now you can specify a URL in the backing file areas of the configuration
files for Funcotator data sources and the backing files will be read by
the FuncotationDataSourceFactories.

This effectively enables use of data sources in the cloud or a mix of
local- and cloud-based data sources through a config file change.

This update will enable gnomAD annotations (once the data sources are
    updated to point at the gnomAD files on Google Cloud).

Added in cloud data sources to test with.

Minor refactoring of LocatableXsvFuncotationFactory.  Now can only
support one file at a time instead of multiple files for each instance.

Fixes #5348
jonn-smith added a commit that referenced this issue Nov 9, 2018
@jonn-smith
Updated data source inputs to accept NIO paths for backing files.
05bcb69 
Now you can specify a URL in the backing file areas of the configuration
files for Funcotator data sources and the backing files will be read by
the FuncotationDataSourceFactories.

This effectively enables use of data sources in the cloud or a mix of
local- and cloud-based data sources through a config file change.

This update will enable gnomAD annotations (once the data sources are
    updated to point at the gnomAD files on Google Cloud).

Added in cloud data sources to test with.

Minor refactoring of LocatableXsvFuncotationFactory.  Now can only
support one file at a time instead of multiple files for each instance.

Fixes #5348
@jonn-smith jonn-smith mentioned this issue Nov 16, 2018
Merged
jonn-smith added a commit that referenced this issue Nov 19, 2018
@jonn-smith
# This is a combination of 2 commits.
05e0c15 
# This is the 1st commit message:

Updated data source inputs to accept NIO paths for backing files.

Now you can specify a URL in the backing file areas of the configuration
files for Funcotator data sources and the backing files will be read by
the FuncotationDataSourceFactories.

This effectively enables use of data sources in the cloud or a mix of
local- and cloud-based data sources through a config file change.

This update will enable gnomAD annotations (once the data sources are
    updated to point at the gnomAD files on Google Cloud).

Added in cloud data sources to test with.

Minor refactoring of LocatableXsvFuncotationFactory.  Now can only
support one file at a time instead of multiple files for each instance.

Fixes #5348

# This is the commit message #2:

Added in more cloud data sources.

New cloud dataset contains local data sources and a pointer to the
gnomAD google cloud bucket.
jonn-smith added a commit that referenced this issue Nov 29, 2018
@jonn-smith
Add arbitrary NIO support to Funcotator Data sources (#5425)
e258888 
Updated data source inputs to accept NIO paths for backing files.

Now you can specify a URL in the backing file areas of the configuration
files for Funcotator data sources and the backing files will be read by
the FuncotationDataSourceFactories.

This effectively enables use of data sources in the cloud or a mix of
local- and cloud-based data sources through a config file change.

This update will enable gnomAD annotations (once the data sources are
    updated to point at the gnomAD files on Google Cloud).

Added in cloud data sources to test with.

Minor refactoring of LocatableXsvFuncotationFactory.  Now can only
support one file at a time instead of multiple files for each instance.

New cloud dataset contains local data sources and a pointer to the
gnomAD google cloud bucket.

Fixes #5348
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jonn-smith jonn-smith

Labels
Funcotator
Projects
None yet
Milestone
Engine-Q42018
Development

No branches or pull requests
1 participant
@jonn-smith