Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@ashelkovnykov
Copy link
Collaborator

@ashelkovnykov ashelkovnykov commented Apr 11, 2023

Updated version of changes to Arvo from #5566

These changes resolve several issues with how JSON is handled and prepare Arvo for JSON encoding/decoding jets.

See #342 for test results.

Resolves:
#1775
#1776
#5456

pkova and others added 30 commits February 28, 2023 11:25
…umber.

Resolves urbit#6377 - stuck %eth-watcher due to get-block-by-number zero-padded number RPC failure
Fix stuck %eth-watcher due to get-block-by-number zero-padded number RPC failure.
This commit refactors `sur/hood.hoon` and changes the signature of the
`+vats` generator thus:

```hoon
|=  $:  [now=@da eny=@uvj bec=beak]
        $@(~ [?(%suspended %running %blocking %nonexistent) ~])
        $:  verb=?
            show-suspended=?
            show-running=?
            show-blocking=?
            show-nonexistent=?
        ==
    ==
```

Called with a single positional argument, `+vats` will show only those
desks fitting the description given, while keyword parameters allow
finer-grained control over which desks are described. The `verb`
parameter determines whether the full load of desk information be shown,
or a subset thereof.

Resolves urbit#6297.
We were retrying failed kelvin upgrades as many times as we had apps
that needed to be suspended, because suspending an app triggers an
attempt to run the next kelvin upgrade.  This suspends all those apps in
one batch move, and then tries the next kelvin upgrade only once at the
end.

Fixes urbit#6407

Partially addresses urbit#6285
…-desk

arvo: add more files to |new-desk generator
…-contact-scry

ames: add last-contact scry endpoint
Merge master back to develop after urbit-os-v2.136 release
Instead of including wrapped-task as-is in most call traces, we now only
include it in traces for crashing (harden task) calls. For everything
else, we include only the tag of the resulting $task.

Closes urbit#6444.
dill: exclude wrapped-task from most +call traces
pkova and others added 2 commits April 11, 2023 17:18
+vats: fix crash on nonexistent desks
behn: propagate errors in deferred moves
@belisarius222
Copy link
Collaborator

@joemfb says +sane is slightly more restrictive about whitespace than standard UTF-8, so we shouldn't change +teff in hoon.hoon. Instead, a modified +teff should be used in Zuse for the JSON functions.

The unit tests for de-json:html and en-json:html in zuse.hoon were
copied from the JSON Test Suite (https://github.com/nst/JSONTestSuite).
While a good test suite if usable as-is, it has a lot of problems with
it, namely: inconsistent test names; repeated tests; and mixing tests
for multiple cases into one file. Since the tests need to be manually
added to Urbit anyway, this commit uses the opportunity to clean up the
test suite by making the following changes:
- Renaming tests to use a consistent naming scheme
- Removing repetetive test cases
- Splitting multi-case tests into multiple single-purpose tests
The JSON standard rejects all un-escaped, un-printable characters EXCEPT
for the delete character (ID 127, i.e. 0x7F).

See: https://www.crockford.com/mckeeman.html
Via: https://www.json.org/json-en.html
JSON supports Unicode as both UTF8 sequences and escaped UTF16. Unicode
points U+10000 to U+10FFFF are encoded as two consecutive escaped UTF16
units known as a surrogate pair. The JSON decoder was previously
treating surrogate pairs as two individual escaped UTF16 units. In
addition, the JSON standard allows hexadecimal letters to be either
uppercase or lowercase, which the parser was not respecting.

See:
- https://www.crockford.com/mckeeman.html
- https://www.json.org/json-en.html
- https://en.wikipedia.org/wiki/UTF-16#Code_points_from_U+010000_to_U+10FFFF

Git issue: urbit#1776
JSON supports Unicode as both UTF8 sequences and escaped UTF16. UTF8 is
encoded as between 2 and 4 bytes, the first byte of which governs the
length. The JSON parser was previously accepting any non-ASCII bytes as
UTF8 in any order.

See:
- https://www.crockford.com/mckeeman.html
- https://www.json.org/json-en.html
- https://en.wikipedia.org/wiki/UTF-8#Encoding
The backspace character was not being escaped like other invisible
characters.
Encode JSON to cord instead of tape. Code copied from @joemfb 's
version.
Copy link
Collaborator

@joemfb joemfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will do

my motivation to not change +teff and +sane is that @t is not exactly utf-8 -- it imposes additional constraints on whitespace (rejecting newlines). that may or may not be a good idea (and it's violated in more than a few places), but it is longstanding and was intentional (i'll see if i can pull up links, maybe from the old urbit-dev list). it's a different kind of change than the others here, and should be decided on for the whole system with context and care. separate tracking issues will be good to have.

@ashelkovnykov
Copy link
Collaborator Author

@joemfb Sounds good - I'll make some additional tracking issues and include what context I have now.

@jalehman @belisarius222 I leave this one for you guys to merge whenever and into where ever it needs to go.

@belisarius222 belisarius222 changed the base branch from develop to next/kelvin/413 April 13, 2023 15:16
@belisarius222 belisarius222 merged commit f3d2df0 into urbit:next/kelvin/413 Apr 13, 2023
belisarius222 pushed a commit to urbit/vere that referenced this pull request Apr 13, 2023
Updated version of changes to Vere from
[#5566](urbit/urbit#5566)

These changes jet the JSON encoding / decoding arms in [this
PR](urbit/urbit#6463).

**TESTS**
- JSON unit tests in Arvo pass with jets enabled
- `bazel test --build_tests_only ...` passes
- Both of the above hold true when `ice` flag disabled
- Wall-clock unit test times from previous tests replicated
- [Previous
tests](urbit/urbit#5566 (comment)) =
`28.9%`
    - Current tests = `27.7%`
    - Let's be pessimistic and say "70% reduction in time to parse JSON"

**TODO**
- [x] Confirm wall-clock test times from previous PR still hold true
- [ ] Redo Memcheck
- [ ] Redo AddressSanitizer check
@ashelkovnykov ashelkovnykov deleted the json-new branch April 6, 2024 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants