Use StringDecoder for Buffers in WritableStream

[ajafff]

http://stackoverflow.com/questions/12121775/convert-streamed-buffers-to-utf8-string

[fb55]

Could you add a test case that fails without this?

[ajafff]

"use strict";

var ParserStream = require('htmlparser2').WritableStream;
var Buffer = require('buffer').Buffer;
var assert = require('assert');

var parser = new ParserStream({
    ontext:function(text){
        assert.equal(text, '€');
    }
});

parser.write(new Buffer([0xE2, 0x82]));
parser.write(new Buffer([0xAC]));

Without the fix this should fail AssertionError: "��" == "€"
After applying the fix those 2 Buffers will be concatenated and result in '€'

[fb55]

Hm, AFAICT, this should be fixed when concatinating text events as discussed with @jails in #124.

[ajafff]

I get your point, but this problem is not inherent to text nodes. It could affect everything which contains characters that are not specified in ASCII (attributes, CDATA, comments, ...)
Also I looked at the implementation of high5. The concatenating of text events you mentioned is done by concatenating multiple string chunks to one string "buffer" (this._buffer)
Unfortunately it will not solve this problem, because

new Buffer([0xE2, 0x82]).toString() + new Buffer([0xAC]).toString() !== '€' //results in '���' instead

When working with Buffers in a streaming fashion you have to use StringDecoder to get utf8 right

[philiiiiiipp]

I think @ajafff is very right. I ran into trouble when parsing web pages, especially since the behaviour is quite unpredictable because a cut right between the two bytes happens quite rarely.

I think it would be beneficial to add this information to the /wiki/Parser-options. Would have saved me some troubles at least.

EDIT:
Ok, I am not sure if this is how it is supposed to go, but I just went ahead and wrote the note myself :-).

[fb55]

I forgot about this, sorry. This needs a test case in the test dir (as a new file – have a look at api.js) and quotes have to be double quotes (that's why the tests fail). Looks good otherwise.

[coveralls]

Coverage increased (+0.01%) to 95.673% when pulling d79e36d on ajafff:master into 9770f24 on fb55:master.

[ajafff]

@fb55 changed single quotes to double quotes, added test
PTAL

[fb55]

Awesome, thanks!