-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Supporting Arctic Backend Provider & Orderbook, Tick Data Example #744
Conversation
|
@luocy16 Please follow the instructions to format your code. Thanks |
qlib/data/data.py
Outdated
@@ -886,6 +955,7 @@ def dataset( | |||
return_uri=False, | |||
inst_processors=[], | |||
): | |||
#print("disk_cache {}".format(disk_cache)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove the useless code after testing.
qlib/config.py
Outdated
@@ -122,7 +131,7 @@ def set_conf_from_C(self, config_c): | |||
# default 1 hour | |||
"mem_cache_expire": 60 * 60, | |||
# memory cache space limit, default 5GB, only in used client | |||
"mem_cache_space_limit": 1024 * 1024 * 1024 * 5, | |||
"mem_cache_space_limit": 1024 * 1024 * 1024, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it necessary to change the default memory size?
qlib/data/data.py
Outdated
@@ -502,16 +505,18 @@ def parse_fields(fields): | |||
return [ExpressionD.get_expression_instance(f) for f in fields] | |||
|
|||
@staticmethod | |||
def dataset_processor(instruments_d, column_names, start_time, end_time, freq, inst_processors=[]): | |||
def dataset_processor(instruments_d, column_names, start_time, end_time, freq, type="qlib", inst_processors=[]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using constant string will be better.
qlib/data/data.py
Outdated
from .ops import Operators | ||
from .op_arctic import * |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import *
is not encouraged
qlib/data/data.py
Outdated
try: | ||
arctic = Arctic_Connection_List[0] | ||
#print("arctic example: {}, task_index:{}".format(arctic, task_index)) | ||
except Exception: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please catch specific exception instead of except Exception:
qlib/data/ops.py
Outdated
elif self.func == 'ffill': | ||
# TODO: | ||
# Not sure why dropping duplicated element is necessary | ||
series = series[~series.index.duplicated(keep='first')] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@luocy16 It's very kind of you if you could give any hints why ffill
should be processed in such a special approach
…crosoft#744) * change weight_decay & batchsize * del weight_decay * big weight_decay * mid weight_decay * small layer * 2 layer * full layer * no weight decay * divide into two data source * change parse field * delete some debug * add Toperator * new format of arctic * fix cache bug to arctic read * fix connection problem * add some operator * final version for arcitc * clear HZ cache * remove not used function * add topswrappers * successfully import data and run first test * A simpler version to support arctic * Successfully run all high-freq expressions * Black format and fix add docs * Add docs for download and test data * update scripts and docs * Add docs * fix bug * Refine docs * fix test bug * fix CI error * clean code Co-authored-by: bxdd <[email protected]> Co-authored-by: wangwenxi.handsome <[email protected]> Co-authored-by: Young <[email protected]>
Description
Motivation and Context
How Has This Been Tested?
pytest qlib/tests/test_all_pipeline.py
under upper directory ofqlib
.Screenshots of Test Results (if appropriate):
Types of changes