Thank you for tackling this challenging problem Kokubun. I think generally speaking your approach looks good though I am not sure how set_write_pos / get_write_pos will work in this new scheme, particularly with the inline/outlined region no longer being contiguous. I wrote some comments/questions.

I pushed 90c7b01 to make --yjit-code-page-size a command-line option so we can experiment with different values later 🔬 :)

I figured out how to fix a few immediate problems, but didn't get to fix everything I encountered next. This PR still needs some more work.

It has been challenging to make codegen work when the assembler attempts to write code but there's no enough space in the current page. I've got a few ideas:

1. Current PR: On asm.compile, attempt to write the current page. If it fails, write from the beginning in the next page.
   • asm.compile should also return CodePtr in case the caller needs to recognize the next page as the starting address.
2. On asm.compile, predict the size of assembled code, and move to the next page as needed.
   • This also requires you to return CodePtr from asm.compile and use it.
   • You need to implement the code size estimation of some Ops.
3. When emitting an insn, predict the size of its assembled code, and generate a jmp instruction to go to the next page as needed.
   • You don't need to return CodePtr from asm.compile. The current callers should work as is.

1 and 2 would have essentially the same code layout, but there's some difference between 1 and 3. While 3 would minimize the code page size, it's more likely that code of a single ISEQ is split into two pages. 1 doesn't have the split page problem, however, it would leave gaps at the end of every code page.

I'm leaning towards having whichever simpler (still thinking about which one actually is) first, but it's worth evaluating the performance difference after that.

1 and 2 would have essentially the same code layout, but there's some difference between 1 and 3. While 3 would minimize the code page size, it's more likely that code of a single ISEQ is split into two pages. 1 doesn't have the split page problem, however, it would leave gaps at the end of every code page.

I'm leaning towards having whichever simpler (still thinking about which one actually is) first, but it's worth evaluating the performance difference after that.

Solution #3 might be better just because, we're likely to end up needing a jump anyway if a block has to be compiled in the next page. Might as well go for the solution that makes things the most transparent and minimizes memory usage?

I think it's ok to keep things simple and assume that all instructions need 64 bytes available, otherwise we need to change page.

The main caveat that I can see is that we could end up in a situation where we insert a jump in the middle of a patchable/rewritable branch, which could be annoying to deal with? Though as long as there is enough bytes left in the page so that we could generate the whole rewritable branch in one block, it shouldn't be a problem.

Solution #3 might be better just because, we're likely to end up needing a jump anyway if a block has to be compiled in the next page. Might as well go for the solution that makes things the most transparent and minimizes memory usage?

👍

Another idea similar to 3 is that we could attempt a write and retry it after next_page per insn instead of predicting the write size. I just pushed its PoC for Arm at 8491112, but I wasn't able to finish it up because I need to go to the airport soon. It seems less buggy than the previous patch, but still crashing.

Debugging the current failure on Arm might be helpful if anybody wants to make a progress on this project this Friday.

we could attempt a write and retry it after next_page per insn instead of predicting the write size.

That might work but we also need to be able to guarantee that we are able to write the largest possible size of a jump to the next page.

There are also potentially edge cases if we switch page in the middle of the branch instruction at the end of a block... Although I think that should work as well... Provided that we always succeed in allocating a next page 🤔

Welcome back to America btw! I'm going to be away at a burst most of next week so still difficult to pair, though at least we will be in closer time zones so we should be able to chat on Slack in real time. Looking forward to pairing the week after :)

I should mention, @XrXr should be available to pair next week. He knows this codebase and CRuby very well so should definitely be able to help.

Current status: Alan and I paired on the immediate issue of this PR. We see a couple of assertion failures on "branch stubs should never enlarge branches" and PC calculation VM_ASSERT(n <= ISEQ_BODY(iseq)->iseq_size);.

Because of the amount of problems I'm facing again and again, it felt like another option could be simpler. I was thinking about choosing one of the following two options, summarizing the current blockers of each option:

1. Current PR: In arm64_emit, try emitting an Insn, jump to the next page if it fails, and retry emitting the same Insn.
   • pros: Minimizes memory usage
   • cons: Deciding whether to generate a jmp or not is tricky. More edge cases than I initially expected. One of the issues is that you sometimes need to guarantee there's an enough space for a jmp before the current Assembler is instantiated (the previous Assembler should leave space), while you also need to guarantee assumptions like "branch stubs should never enlarge branches".
   • blockers:
     • Assertion failure on "branch stubs should never enlarge branches" (probably not difficult to fix)
     • Assertion failure in PC calculation: VM_ASSERT(n <= ISEQ_BODY(iseq)->iseq_size); (not immediately obvious)
2. In compile_with_regs, try arm64_emit, set the position to the next page if it fails, and retry entire arm64_emit.
   • pros: An Assembler-level retry seems kind of simpler than an Insn-level retry.
   • cons: More memory usage. When generated code is too large, you have to give up.
   • blocker: Assertion failure on "invalidation needs the entry_exit field" because the entry address is decided after compile_with_regs, however, it's needed beforehand for invalidation.

Because of the cons and the blockers, I was almost leaving Solution #1, but after revisiting Solution #2, I noticed the thing that I wrote as the blocker. The next step is to explore an idea of possibly using a label to address the blocker of the Solution #2, and then go back to debugging Solution #1 if it doesn't work.

I want to note that even with solution 2, we still need to write out jumps sometimes. In gen_direct_jump() we assume that we can always place blocks next to each other and that's not true anymore once we change the layout.

Thanks for helping with this @XrXr 🙏

Trying to think ahead here. In the future we might want to try generating code for multiple blocks at once, so we can have longer instruction sequences to optimize. If we were doing that, reusing the same Assembler for multiple blocks, then giving up if we can't fit the entire sequence in the current page could be problematic (could waste a lot of space). IMO that suggests that an Insn-level retry is probably more flexible.

cons: Deciding whether to generate a jmp or not is tricky. More edge cases than I initially expected. One of the issues is that you sometimes need to guarantee there's an enough space for a jmp before the current Assembler is instantiated (the previous Assembler should leave space), while you also need to guarantee assumptions like "branch stubs should never enlarge branches".

I'm wondering if there's not a stupid-but-good-enough way to solve this. Something like, we define a MAX_PATCHABLE_BRANCH_SIZE, and if there isn't that much space left for a whole patchable branch, then we just move to the next page. That way, we never end up in a situation where a patchable branch can be split across two pages?

Unless that assertion failure just comes from the fact that the blocks are no longer always next to each other when queued 🤔 ?

While we still didn't reach a point to make it work today, we tried a few ideas that we thought would work more easily. I ran out of time before posting a summary, so I'll summarize our current status tomorrow, discussing your points as well.

At 24a4a42, Solution 1 was passing btest except for test_yjit_30k_ifelse with Assertion Failed: ../vm_backtrace.c:57:calc_pos:n <= ISEQ_BODY(iseq)->iseq_size and test_yjit_30k_methods with panicked at 'branch stubs should never enlarge branches'.

What we tried on it yesterday were:

• First, we removed a workaround that allows the last insn to run out of space in the current page because it wasn't enough to avoid "branch stubs should never enlarge branches". This resulted in making 30k_ifelse fail with "branch stubs should never enlarge branches" as well.
• Tried starting inline/outlined code of a next page from a 20-byte offset so that the first uninitialized code range triggers SIGILL when inline code steps into outlined code and vice versa. As of this revision, SIGILL wasn't triggered, still both "branch stubs should never enlarge branches".
• Added can_change_page flag to Assembler to explicitly disallow going to the next page when necessary. This caused those two tests to fail with Assertion Failed: ../vm_backtrace.c:57:calc_pos:n <= ISEQ_BODY(iseq)->iseq_size or Assertion Failed: ../vm_backtrace.c:58:calc_pos:n >= 0.
• Removed can_change_page, and tweaked the assertion condition of "branch stubs should never enlarge branches" instead. It caused 'verify_ctx: stack value was mapped to self, but values did not match! and Assertion Failed: ../yjit.c:392:rb_iseq_pc_at_idx:insn_idx < iseq->body->iseq_size.
• Tried another design that ends each page with a jump to the next page, letting Assembler generate nops at the page end. This enters an infinite loop or a failure of an assertion !ocb.dropped_bytes that we added.

In the future we might want to try generating code for multiple blocks at once, so we can have longer instruction sequences to optimize.
IMO that suggests that an Insn-level retry is probably more flexible.

Agreed. We should probably stick to the design unless we find a blocker for it.

Something like, we define a MAX_PATCHABLE_BRANCH_SIZE, and if there isn't that much space left for a whole patchable branch, then we just move to the next page. That way, we never end up in a situation where a patchable branch can be split across two pages?

We did work on that kind of idea. I think we'll explore more of that on today's pairing 🤔

Alan and I made a good progress today. We kept improving and debugging this implementation of Solution 1, and with --yjit-call-threshold=1, make btest is finally passing and make test-all is passing except for three tests of TestYJIT with !cb.has_dropped_bytes.

The current idea is:

• Use page_end_offset: 20 to reserve space for a jmp at the end of each page. Temporarily set it to 0 when it needs to jump to the next page.
• When next_page() is called, do not move the cb that did not request it backwards. For example, when cb runs out of space in an old page while regenerate_branch or branch_stub_hit, ocb should not go to the next page because it could repeatedly point to the same old address that is possibly already written.
  • The ideal solution to keep inline and outlined codes closer is to keep track of write_pos in every past page to go back to an unwritten position correctly, but the tradeoff here is that it requires some extra memory for it. Probably not too significant and worth a consideration though.

Happy to see that you and Alan are pairing and that you are making so much progress 🙏 :)

Today I refactored this branch and added initial x86_64 implementation.

arm64 is stable enough to pass btest and test-all once locally, but test-all seems crashing on CI. test-spec is mostly passing but stops on spec/ruby/optional/capi/kernel_spec.rb with Killed: 9.

x86_64 next_page seems to cause problems; the next step is to investigate SEGV on test_yjit_30k_ifelse.rb and test_yjit_30k_methods.rb.

I'll pair with @XrXr on those issues tomorrow.

In the second pairing session with Alan, we were working on the #6406 side (changes are not finished/pushed yet). I think this PR is almost ready behavior-wise, but I need some more time tomorrow to refactor this before marking it ready for reviews.

Wonderful. Great work so far 👌 :)

included the benchmark numbers for the latest revision in the PR description (not changed much from #6460 (comment))