Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSCW-reader reads in trends of all available year rather than only years selected with periods #1247

Open
heikoklein opened this issue Jul 8, 2024 · 1 comment
Labels
bug 🐛 Something isn't working performance Excessive use of time or memory
Milestone

Comments

@heikoklein
Copy link
Member

Describe the bug

  • Pyaerocom version: latest
  • Configuration file (if applicable):
  • Error message (if applicable): On a 2 year = 24 timestep monthly evaluation, 48 timesteps (= all in subfolders) are read
Extracting timeseries data from large array (shape: (48, 520, 1200)). This may take a while...

To Reproduce
Steps to reproduce the behavior:

  1. create a trend-analysis with mscw-reader. Select periods to be smaller than available trend-years.
  2. The mscw-reader will try to read/collocate all available trend-years, not only the necessary once.
  3. pyaerocom takes much longer to evaluation (typically, only every 5th year are selected) and crashes due to timeouts.

Expected behavior
The mscw-reader only reads years selected by periods.

@heikoklein heikoklein added the bug 🐛 Something isn't working label Jul 8, 2024
@heikoklein heikoklein added this to the m2024-08 milestone Jul 8, 2024
@heikoklein heikoklein added the performance Excessive use of time or memory label Jul 8, 2024
@heikoklein heikoklein assigned dulte and unassigned dulte Jul 18, 2024
@thorbjoernl thorbjoernl self-assigned this Aug 8, 2024
@lewisblake
Copy link
Member

lewisblake commented Aug 12, 2024

This is probably worth discussing. It might fall under the class of a "feature" and not a bug. Trends were originally defined to only exit over seven year periods (or more), so this may have been intentional. The statistics become less robust if we consider short time frames, but maybe there is a specific use case you have in mind.

@lewisblake lewisblake modified the milestones: m2024-08, m2024-09 Aug 12, 2024
@thorbjoernl thorbjoernl removed their assignment Aug 16, 2024
@heikoklein heikoklein modified the milestones: m2024-09, m2024-10 Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Something isn't working performance Excessive use of time or memory
Projects
None yet
Development

No branches or pull requests

4 participants