How much memory does it use?

[JesseTG]

I need to dedupe line-separated records in a very large file (hundreds of gigs, each record a few KB). This looks like it could be useful for me, but I don't know how much RAM it uses. For stream mode it has to store the actual lines, which is understandable, but what about mmap mode?

[deepinthebuild]

The maximum amount of memory used for mmap mode depends on the number of distinct entries in the input. The program itself only uses 24 bytes of memory for each distinct entry but the peak run time usage will be significantly (2x-8x) higher than that, due to two copies of the internal hash table temporarily existing during reallocations. On the enwiki9 benchmark (1 GB dataset), the master branch hits 600MB peak memory usage for me (the data set just barely triggers a reallocation to the largest size).

I've been planning on implementing a tempfile-backed option on one of the experimental branches. I can prioritize that feature if it would be useful to you.

[JesseTG]

I'd appteciate that a lot, thank you. Also, reallocations? Yikes. Might wanna look at other data structures. I'll shop around.

[JesseTG]

Here's an idea: when dealing with a mmap(2)ed file, use one of these (especially this) to make a first pass to guess the number of unique lines. Then pre-allocate a hash set with space for that many elements and do the actual deduplication. If there's a more efficient hash set out there, that would be great too.