Use int instead of long to keep CSV row number

[aababilov]

Generate TooManyRowsNotice (error) for files that have more than 1 billion rows. Such large files would require too much memory and cause OOM.

This change reduces memory consumption by 4 bytes per row. Large stops_times.txt files may contain 30 M and even 500 M lines, so we are saving from 120 MB to 2 GB.

[isabelle-dr]

Thanks @aababilov, one question: how do we define MAX_INT?

This new rule will have to be added to the documentation. 😊
We deprecated NOTICES.md, now we centralize all the docs in RULES.md.

[asvechnikov2]

Thanks! The change looks good. Could you please add a bit more information in the first comment on what impact the change has (e.g. memory savings, etc)? Also, should we use something more conservative than INT_MAX [0], maybe restricting to 500 million entries in a single file or 1 billion? @isabelle-dr do you have any opinion on this?

[0] INT_MAX is 2'147'483'647

[isabelle-dr]

@asvechnikov2 what is the proportion of datasets than have > 500 million entries?

[asvechnikov2]

@isabelle-dr I think the biggest I saw was under 100 million entries, so it should be safe to assume that there are none for > 500 millions (not even close)

[aababilov]

We deprecated NOTICES.md, now we centralize all the docs in RULES.md.

This is a risky practice. RULES.md already has almost 3 k rows and it has to be maintained manually. Saving this file in IDEA takes several seconds. I would suggest to drop this file completely and generate docs based on comments.

[bdferris-v2]

@aababilov agreed that RULES.md is a pain to maintain. I don't recommend opening it in IDEA but instead a simple text editor. I've explored generating the file from comments directly and have a local patch that automates a big chunk of it, but it's not quite fully there.

[asvechnikov2]

LGTM! @isabelle-dr the proposal is to use 1 billion entries as a limit, this is well beyond of what is available now and won't cause any issues.

[isabelle-dr]

LGTM! Merging this PR. 🥳