Memory usage could be reduced when using very large dictionary ( >10k words)

[Willmiff]

With a large enough dictionary, the overhead for the output grows very large.

If, instead of creating a large number of empty arrays, and a few arrays that contain the output words, you just create the useful ones, and leave the other ones sparse.

Real world test:

Dictionary of 23818 words
Memory usage of current implementation : 21.3mb
Memory usage by pruning output object: 16.7 mb

[BrunoRB]

Only saw this issue today, I think it got buried in the middle of work stuff (downside of using github at work...), sorry about that.

Anyway, yeah, it makes sense and we could probably do the same on the goto state transition map . But in reality this is such a tiny improvement that I don't see why bother, especially because it'll make the code a bit more opaque. I implemented this to actually use in some real-life cases, but I like the idea of keeping it clean to serve educational purposes. But thanks for pointing it out!

I'll leave the issue open and if more people complain I'll add the optimization (or if you have a situation where this is actually relevant, then please do tell me!).