-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IRI output #323
Comments
This one's tricky. Historically header values were defined to be ISO-8859-1. But the latest recommendation is this:
And that's how the library we use handles it. It only allows string conversion if it's pure-ASCII. HTTPie decodes it as ISO-8859-1, which is wrong in this case: $ http -h https://wiktionary.org/wiki/Ῥόδος
HTTP/1.1 301 Moved Permanently
[...]
location: https://www.wiktionary.org/wiki/ῬÏÎ´Î¿Ï And that's also how the dev tools of Firefox and Chromium handle it: So I see a few options:
I'm torn between all three, I don't know what's best. (The current behavior deserves a mention on #322.) |
since a header line is expected to end in a newline, to me that puts header values in the category of "text data", rather than "binary data". I think it would ultimately be a net positive to at least try to decode the values as UTF-8. if I visit the original link in a browser: https://wiktionary.org/wiki/Ῥόδος the values are not escaped in the address bar. also especially if Greek was someones native language, they would probably prefer to hand write the URL without encoding, and view it that way as well. so ideally to me this behavior should persist throughout the stack. however as was keenly pointed out, its not required by the spec, so I will respect if the decision is "do nothing". |
We know it's text, but we don't know the text's encoding (maybe ISO-8859-1, maybe UTF-8, maybe even some other ASCII superset), and it can be dangerous to guess. It kind of pains me but I now think that decoding as ISO-8859-1 is the way to go. If you send an HTTP request from a browser then this is the result: > let r = await fetch('https://wiktionary.org/wiki/Ῥόδος', { redirect:'manual' })
> r.headers.get('location')
'https://www.wiktionary.org/wiki/ῬÏ\x8CδοÏ\x82' At my day job I use xh to test APIs for use in the browser. It can be a big headache if something looks fine on one side and falls apart on the other. It's best not to sweep the brokenness under the rug. It would still be nice to somehow separately display the common-sense URL that the redirect points to. But I don't want to mislead people into thinking they can reliably extract it from the header. |
ISO-8859-1 is hands down the worst option of the three. the original representation is lost, and the hex values aren't returned either, making that part of the URL essentially meaningless garbage to a human, and only useful to a computer. its a lose lose situation. please reconsider that opinion. |
The one benefit is that you know what to expect if you try to handle the header programmatically. IMO that's very important, maybe important enough to outweigh everything else. But the current output also hints that something is wrong, so maybe we don't need to go that far. |
not sure if it helps, but I also have my own encoding that might work here: https://github.com/1268/strconv/blob/main/encode.go its similar to percent-encoding, but will only encode instances of the escape character (%) or "binary data" defined (by me) as control characters, excluding whitespace. it has the nice property that it can be decoded by a normal percent-encoding function. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
Maybe we could show both versions if it successfully decodes as UTF-8? Something like location: "https://www.wiktionary.org/wiki/\xe1\xbf\xac\xcf\x8c\xce\xb4\xce\xbf\xcf\x82" (https://www.wiktionary.org/wiki/Ῥόδος) or location: https://www.wiktionary.org/wiki/ῬÏÎ´Î¿Ï (https://www.wiktionary.org/wiki/Ῥόδος) or
Possibly with a warning to explain the issue a little more? |
The third option seems the least ambiguous for users.
If our warning included the header value in both bytes and ISO-8859-1, I wonder if we could then have only the UTF-8 decoded value in the main output. What do you think? Interestingly enough, cURL decodes the header value as UTF-8:
|
I feel like a lot of people would ignore or not notice the warning if they see that the main output looks good. So I'd like to make the mangled/undecoded version as prominent or more prominent than the fixed one. One scenario I'm thinking of is:
I believe it doesn't decode it at all and just sends the bytes straight to your terminal, which then treats them like UTF-8. Notice also that the charset is declared as $ xh https://wiktionary.org/wiki/Ῥόδος
HTTP/2.0 301 Moved Permanently
[...]
<p>The document has moved <a href="https://www.wiktionary.org/wiki/Ῥόδος">here</a>.</p> unless you redirect the output, in which case we also pass through the bytes as is: $ xh https://wiktionary.org/wiki/Ῥόδος | cat
xh: warning: HTTP 301 Moved Permanently
[...]
<p>The document has moved <a href="https://www.wiktionary.org/wiki/Ῥόδος">here</a>.</p> |
I'm now thinking location: https://www.wiktionary.org/wiki/ῬÏÎ´Î¿Ï (UTF-8: https://www.wiktionary.org/wiki/Ῥόδος) with the mangled version colored red. For people with colors enabled (almost everyone?) that'd hopefully make it clear both that something known to be undesirable is going on and that the parenthetical is not part of the header value. We could even turn the text "UTF-8" into a hyperlink to some page that explains this issue. Not all terminals support that but it's not as spammy as a warning. (Cargo has also been using hyperlinks for a while now with |
If I run this command, I get encoded output:
but with this Go code:
I get expected output:
https://wikipedia.org/wiki/Internationalized_Resource_Identifier
The text was updated successfully, but these errors were encountered: