Incorrect 'dump_out.nt' testcase string_multiline_12?

[LewisGaul]

The string_multiline_12 dumping testcase is failing on zig-nestedtext, and I think it's because of a bug in the testcase.

The input JSON contains the following, which looks as expected:

"the LF character \n\tASCII Linefeed (LF)\nthe CR character \r\tASCII Carriage Return (CR)\n"

But the expected output NestedText does not contain the CR - it is replaced with a LF:

$cat -A tests/official_tests/test_cases/string_multiline_12/load_in.nt
>$
> the BS character \^IBackslash (\)$
> the SQ character '^ISingle quote (')$
> the DQ character "^IDouble quote (")$
> the AB character ^G^IASCII Bell (BEL)$
> the BS character ^H^IASCII Backspace (BS)$
> the FF character ^L^IASCII Formfeed (FF)$
> the LF character $
> ^IASCII Linefeed (LF)$
> the CR character $
> ^IASCII Carriage Return (CR)$
> the HT character ^I^IASCII Horizontal Tab (TAB)$
> the VT character ^K^IASCII Vertical Tab (VT)$
> the ES character ^[^IASCII escape character as octal value$
> the ES character ^[^IASCII escape character as hex value$
>

I expected this to be:

$cat -A tests/official_tests/test_cases/string_multiline_12/load_in.nt
...
> the LF character $
> ^IASCII Linefeed (LF)$
> the CR character ^M> ^IASCII Carriage Return (CR)$
...

Admittedly this looks strange, but it can be explained as follows:

• NestedText should be treating CR as a line ending (hence the >_ before the ^I, as with LF above)
• WSL (where I ran this) doesn't treat CR as a line ending so displays it 'incorrectly'

Unless I'm missing something, I don't think the current behaviour can be justified as desirable because it means CR characters cannot be dumped (they're read in as 'generic line ending' and then dumped out as LF).

The way I handle this in zig-nestedtext is to keep the line endings of multiline string lines so that they can be preserved.

[andrewchambers]

I am also confused by this test case.

[erikw]

This is what the reference implementation does:

https://github.com/KenKundert/nestedtext/blob/master/nestedtext.py#L62

So I decided to do this as well for

https://github.com/erikw/nestedtext-ruby

to pass all the official tests.

[LewisGaul]

I still think there's a bug in this testcase, according to the language spec: https://nestedtext.org/en/stable/file_format.html

Quoting (emphasis mine):

Line breaks:
A NestedText document is partitioned into lines where the lines are split by CR LF, CR, or LF where CR and LF are the ASCII carriage return and line feed characters. A single document may employ any or all of these ways of splitting lines.

String items:
If the first non-space character on a line is a greater-than symbol followed immediately by a space (>␣) or a line break, the line is a string item. After comments and blank lines have been removed, adjacent string items with the same indentation level are combined in order into a multiline string. The string value is the multiline string with the tags removed. Any leading white space that follows the tag is retained, as is any trailing white space and all newlines except the last.

[KenKundert]

Sorry, for some reason GitHub is not notifying me of issues being raised in this repository.

Currently the Python version converts both \r\n and \r to \n when dumping. What is the value in retaining the \r? It will be interpreted as \n when the file is read back in.

[LewisGaul]

What is the value in retaining the \r? It will be interpreted as \n when the file is read back in.

Isn't one of the intentions with NestedText that you can represent any string? Converting \r to \n breaks that, breaking round-tripping of that input too.

FWIW I think the approach JSON takes is to escape special characters, so a literal "\r" (\\r) is needed to represent \r. I thought NestedText wouldn't want to go down this route though given the existing handling of whitespace.

[KenKundert]

The goal for NestedText was to represent any string without quoting or escaping. However, that is not possible. So I tend to say that a NestedText string can contain any printing character. But even that is not precisely true as a string may not contain line ending characters (\n and \r). Well, it can contain them, but it cannot distinguish them. That is because of the desire to read files produced on different operating systems, which causes us to interpret \r\n, \r, and \n as a line separators. So the goals of OS independence and allowing line ending characters in strings are in conflict.

As for round-tripping. It is supported for everything except line-ending characters. If we tried to maintain round-tripping on line-ending characters, we would end up with OS dependency issues.

[KenKundert]

One possible resolution to this issue would be to abandon \r as a valid line ending. It is only used by old Macs. I am far from the Apple sphere of influence, so I don't know if that would be a big loss. My belief is that nobody would notice us dropping support for \r as a line ending, and by doing so we would be a bit closer to our ideal of supporting all printing characters in strings. Furthermore, I think in the future most formats will settle on just supporting \r\n and \n as line endings.

[LewisGaul]

The goal for NestedText was to represent any string without quoting or escaping. However, that is not possible.

I'm confused. This seems entirely possible - as far as I'm aware, my Zig implementation does this, and in doing so fails this testcase.

All you have to do is keep the line endings on multi-line strings, that way a \r is represented as follows:
>\r> (the carriage return will be displayed differently depending on editor/platform, so it may just look like two lines containing '>', but the key point is that the line ending of all but the last is kept). A similar thing is done with Zig's multiline string literals I believe, except they use \\ instead of >.

So the goals of OS independence and allowing line ending characters in strings are in conflict.

I'm not seeing this.

[KenKundert]

The testcase was constructed assuming that CR was a line ending. If you are failing the test then you are not treating it as a line ending.

There are a couple of situations where what you are doing differs from the current spec. First, if CR is a line ending, it should terminate end-of-line strings and comments:

key: this is a string that contains a CR (^M), it should raise an error if CR is a line ending.

It should raise an error because the carriage return (^M) terminates the line, leaving an extra line that does not have a key. We probably need to add this case to the official test suite.

The second way it differs is that when treated as a line break, the underlying implementation is free to convert the CR to the natural line ending for the environment its running in. So under Unix it would be converted to a LF and on Windows it would be converted to CR LF. Your approach would keep it as a CR.

So my proposal is similar to your implementation, except that it in my proposal CR is no longer treated as a line ending.

[fidian]

I saw something similar when implementing tests for a new JavaScript version of this library. If the intent is to change all newlines to CR internally, that should be documented. If it is to change it to OS-specific newlines, again that should be documented, though the test files would not support this because they assume the user is running under *nix.

I think this issue also bleeds into string_8 because the input JSON has no way to indicate that the dumped form should have DOS-style newlines. (See #11.)

[KenKundert]

A NestedText document is partitioned into lines where the lines are split by CR LF, CR, or LF where CR and LF are the ASCII carriage return and line feed characters. A single document may employ any or all of these ways of splitting lines.

The intent of this statement is that CR LF, CR, and LF are not distinguished from one another when reading a NestedText document. They all simply introduce a new line. Similarly when generating a NestedText document, any of these forms can be used to separate the lines.

The benefit of this approach is that the interpretation of a NestedText document is independent of the OS used to create it and it is very simple to implement.

The drawback of this approach is that NestedText loses the ability to distinguish between NL and CR. That does not seem like a huge loss to me, but I am open to counter arguments.

An alternative would be to simply treat LF and CR LF as introducing newlines, but not a lone CR. In this way NestedText would distinguish CR from NL, but it would lose compatibility with old Macs. As a Linux user, this also does not seem like a huge lost to me, but I don't feel like I am in a position to know how many of those old systems are still in use.

I can fix the issue with string_8.

As for clarifying the issue with CR, do you still feel that I need add something to the documentation to remove the confusion.

[LewisGaul]

The drawback of this approach is that NestedText loses the ability to distinguish between NL and CR. That does not seem like a huge loss to me, but I am open to counter arguments.

See our discussion near the beginning of this issue's comments - not being able to represent certain characters at all seems like a huge problem to me, and the claim that it's not possible to satisfy this doesn't hold as far as I can tell. Again, see previous comments.

[KenKundert]

If we were to distinguish between CR and LF, how would one distinguish a CR that is acting as data and one that acts as a line separator? The only easy way I see to do this is to simply to not allow CR as line separators. I notice that this is the approach taken by Vim, so I would be comfortable with this approach.