[IVDescriptors] Call getOpcode on demand in getReductionOpChain. nfc #118777

Mel-Chen · 2024-12-05T09:56:23Z

Non-arithmetic reductions do not require the binary opcodes.
As a first step toward removing the dependency of non-arithmetic reductions on getOpcode function, this patch refactors the getReductionOpChain function.

In the future, once all users of getOpcode function are refactored, an assertion can be added to getOpcode function to ensure that only arithmetic reductions rely on it.

llvmbot · 2024-12-05T09:56:58Z

@llvm/pr-subscribers-llvm-analysis

Author: Mel Chen (Mel-Chen)

Changes

Non-arithmetic reductions do not require the use of binary opcodes.
As a first step toward removing the dependency of non-arithmetic reductions on getOpcode function, this patch refactors the getReductionOpChain function.

In the future, once all users of getOpcode function are refactored, an assertion can be added to getOpcode function to ensure that only arithmetic reductions rely on it.

Full diff: https://github.com/llvm/llvm-project/pull/118777.diff

2 Files Affected:

(modified) llvm/include/llvm/Analysis/IVDescriptors.h (+3)
(modified) llvm/lib/Analysis/IVDescriptors.cpp (+10-5)

diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 5d992faf99d27f..913cd8db894cf0 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -213,6 +213,9 @@ class RecurrenceDescriptor {
   /// Returns true if the recurrence kind is a floating point kind.
   static bool isFloatingPointRecurrenceKind(RecurKind Kind);
 
+  /// Returns true if the recurrence kind is arithmetic.
+  static bool isArithmeticRecurrenceKind(RecurKind Kind);
+
   /// Returns true if the recurrence kind is an integer min/max kind.
   static bool isIntMinMaxRecurrenceKind(RecurKind Kind) {
     return Kind == RecurKind::UMin || Kind == RecurKind::UMax ||
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index e1eb219cf977e1..103f1d3149121b 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -60,6 +60,11 @@ bool RecurrenceDescriptor::isFloatingPointRecurrenceKind(RecurKind Kind) {
   return (Kind != RecurKind::None) && !isIntegerRecurrenceKind(Kind);
 }
 
+bool RecurrenceDescriptor::isArithmeticRecurrenceKind(RecurKind Kind) {
+  return (Kind != RecurKind::None) && !isMinMaxRecurrenceKind(Kind) &&
+         !isAnyOfRecurrenceKind(Kind);
+}
+
 /// Determines if Phi may have been type-promoted. If Phi has a single user
 /// that ANDs the Phi with a type mask, return the user. RT is updated to
 /// account for the narrower bit width represented by the mask, and the AND
@@ -1063,7 +1068,7 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
 SmallVector<Instruction *, 4>
 RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
   SmallVector<Instruction *, 4> ReductionOperations;
-  unsigned RedOp = getOpcode();
+  bool IsArithmetic = isArithmeticRecurrenceKind(Kind);
 
   // Search down from the Phi to the LoopExitInstr, looking for instructions
   // with a single user of the correct type for the reduction.
@@ -1081,7 +1086,7 @@ RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
   // more expensive than out-of-loop reductions, and need to be costed more
   // carefully.
   unsigned ExpectedUses = 1;
-  if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp)
+  if (!IsArithmetic)
     ExpectedUses = 2;
 
   auto getNextInstruction = [&](Instruction *Cur) -> Instruction * {
@@ -1089,7 +1094,7 @@ RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
       Instruction *UI = cast<Instruction>(User);
       if (isa<PHINode>(UI))
         continue;
-      if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp) {
+      if (!IsArithmetic) {
         // We are expecting a icmp/select pair, which we go to the next select
         // instruction if we can. We already know that Cur has 2 uses.
         if (isa<SelectInst>(UI))
@@ -1101,7 +1106,7 @@ RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
     return nullptr;
   };
   auto isCorrectOpcode = [&](Instruction *Cur) {
-    if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp) {
+    if (!IsArithmetic) {
       Value *LHS, *RHS;
       return SelectPatternResult::isMinOrMax(
           matchSelectPattern(Cur, LHS, RHS).Flavor);
@@ -1110,7 +1115,7 @@ RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
     if (isFMulAddIntrinsic(Cur))
       return true;
 
-    return Cur->getOpcode() == RedOp;
+    return Cur->getOpcode() == getOpcode();
   };
 
   // Attempt to look through Phis which are part of the reduction chain

artagnon · 2024-12-05T11:31:10Z

llvm/lib/Analysis/IVDescriptors.cpp

-    return Cur->getOpcode() == RedOp;
+    return Cur->getOpcode() == getOpcode();


We're checking for opcode here anyway.

Only arithmetic reductions call getOpcode and check the opcode, non-arithmetic won't reach this check statement.

artagnon

Not sure about this.

artagnon · 2024-12-05T11:31:51Z

llvm/lib/Analysis/IVDescriptors.cpp

  auto isCorrectOpcode = [&](Instruction *Cur) {
-    if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp) {
+    if (!IsArithmetic) {


This function is called isCorrectOpcode(), and it doesn't check opcode anymore?

artagnon · 2024-12-05T11:33:12Z

llvm/lib/Analysis/IVDescriptors.cpp

-  if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp)
+  if (!IsArithmetic)


Sounds like IsArithmetic needs to be inverted, since we're checking the negation everywhere, but then I wouldn't know what to call the new variable.

We can call it as IsNonArithmetic, I think.
82a107d

david-arm · 2024-12-09T14:53:18Z

llvm/lib/Analysis/IVDescriptors.cpp

@@ -1081,15 +1086,15 @@ RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
  // more expensive than out-of-loop reductions, and need to be costed more
  // carefully.
  unsigned ExpectedUses = 1;
-  if (RedOp == Instruction::ICmp || RedOp == Instruction::FCmp)
+  if (IsNonArithmetic)


I'm not sure this patch is really NFC because in theory IsNonArithmetic could be true when Kind is RecurKind::None. Also, even if you remove references to getOpcode for AnyOf recurrence kinds in this function there are still lots of places in LoopVectorize.cpp that call RecurrenceDescriptor::getOpcode(). It's not obvious to me right now that you can avoid calling getOpcode for AnyOf types. Also, if you have to specialise the code everywhere to avoid calling getOpcode for AnyOf kinds wouldn't that make the code even more complicated?

I'm not sure this patch is really NFC because in theory IsNonArithmetic could be true when Kind is RecurKind::None.

This is a great reminder, but I don't think Kind would normally be None in the function getReductionOpChain. I believe using an assertion here would be more appropriate.
8dd7cde

Also, even if you remove references to getOpcode for AnyOf recurrence kinds in this function there are still lots of places in LoopVectorize.cpp that call RecurrenceDescriptor::getOpcode().

Yes, also in SLP that need to be refined. #121549
This is just the first step in the cleanup.

It's not obvious to me right now that you can avoid calling getOpcode for AnyOf types. Also, if you have to specialise the code everywhere to avoid calling getOpcode for AnyOf kinds wouldn't that make the code even more complicated?

The main purpose of avoiding to call getOpcode is to unify AnyOf.
This won't make things more complicated; the design and usage will be similar to llvm::getRecurrenceIdentity.
On the contrary, passing the type into getOpcode would make it more complex #118393 : when an AnyOf recurrence needs to call getOpcode, I would first have to determine whether it is a select(icmp ixx) or select(fcmp float/double...) before calling getOpcode.

david-arm · 2024-12-09T14:56:21Z

llvm/lib/Analysis/IVDescriptors.cpp

@@ -1063,7 +1068,7 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
 SmallVector<Instruction *, 4>
 RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
  SmallVector<Instruction *, 4> ReductionOperations;
-  unsigned RedOp = getOpcode();
+  bool IsNonArithmetic = !isArithmeticRecurrenceKind(Kind);


I think I'd prefer this to be:

bool IsComparison = Instruction::ICmp || RedOp == Instruction::FCmp;

or something like that. It's more precise and we don't have to worry about updating isArithmeticRecurrenceKind every time we add a new recurrence kind. It also explicitly excludes RecurKind::None.

Hmm, I don't want to call getOpcode for every recurrences. How about adding isComparisonRecurrenceKind function?

…:createOp (NFC) (#122239) This patch is one part of unifying IAnyOf and FAnyOf reduction. #118393 The related patch is #118777.

…:createOp (NFC) (llvm#122239) This patch is one part of unifying IAnyOf and FAnyOf reduction. llvm#118393 The related patch is llvm#118777.

Mel-Chen · 2025-04-07T07:42:20Z

ping

Mel-Chen · 2025-04-14T06:18:42Z

ping

Mel-Chen · 2025-04-29T13:05:00Z

Ping

fhahn

LGTM, thanks.

…dency of non-arithmetic reductions on getOpcode.

…lvm#118777) Non-arithmetic reductions do not require the binary opcodes. As a first step toward removing the dependency of non-arithmetic reductions on `getOpcode` function, this patch refactors the `getReductionOpChain` function. In the future, once all users of `getOpcode` function are refactored, an assertion can be added to `getOpcode` function to ensure that only arithmetic reductions rely on it.

Mel-Chen requested review from artagnon, fhahn, ayalz and david-arm December 5, 2024 09:56

llvmbot added the llvm:analysis label Dec 5, 2024

Mel-Chen mentioned this pull request Dec 5, 2024

IVDesc: unify RecurKinds IAnyOf and FAnyOf #118393

Draft

artagnon reviewed Dec 5, 2024

View reviewed changes

Mel-Chen force-pushed the remove-call-getopcode-ivdes branch from 1282bd9 to 82a107d Compare December 6, 2024 07:17

Mel-Chen requested a review from artagnon December 6, 2024 07:19

david-arm reviewed Dec 9, 2024

View reviewed changes

Mel-Chen mentioned this pull request Jan 3, 2025

[SLPVectorizer] Refactor HorizontalReduction::createOp (NFC) #121549

Merged

Mel-Chen force-pushed the remove-call-getopcode-ivdes branch from 82a107d to 8dd7cde Compare January 3, 2025 08:21

Mel-Chen mentioned this pull request Jan 9, 2025

[SLPVectorizer] Refine the scope of RdxOpcode in HorizontalReduction::createOp (NFC) #122239

Merged

Mel-Chen added a commit that referenced this pull request Jan 10, 2025

[SLPVectorizer] Refine the scope of RdxOpcode in HorizontalReduction:…

e0f14e1

…:createOp (NFC) (#122239) This patch is one part of unifying IAnyOf and FAnyOf reduction. #118393 The related patch is #118777.

Mel-Chen mentioned this pull request Mar 19, 2025

[VPlan] Add support for in-loop AnyOf reductions #131830

Open

Mel-Chen force-pushed the remove-call-getopcode-ivdes branch from 8dd7cde to 367cfd1 Compare March 24, 2025 09:47

Mel-Chen changed the title ~~[IVDescriptors][NFC] Refactor getReductionOpChain to remove the dependency of non-arithmetic reductions on getOpcode.~~ [IVDescriptors] Call getOpcode on demand in getReductionOpChain. nfc Mar 24, 2025

fhahn approved these changes Apr 29, 2025

View reviewed changes

Mel-Chen added 2 commits April 30, 2025 00:12

[IVDescriptors][NFC] Refactor getReductionOpChain to remove the depen…

c458e9e

…dency of non-arithmetic reductions on getOpcode.

Rebase, and remove isArithmeticRecurrenceKind

137c6c4

Mel-Chen force-pushed the remove-call-getopcode-ivdes branch from 367cfd1 to 137c6c4 Compare April 30, 2025 07:12

Mel-Chen merged commit 08f0aa4 into llvm:main Apr 30, 2025
11 checks passed

Mel-Chen deleted the remove-call-getopcode-ivdes branch April 30, 2025 09:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IVDescriptors] Call getOpcode on demand in getReductionOpChain. nfc #118777

[IVDescriptors] Call getOpcode on demand in getReductionOpChain. nfc #118777

Mel-Chen commented Dec 5, 2024 •

edited

Loading

llvmbot commented Dec 5, 2024

artagnon Dec 5, 2024

Mel-Chen Dec 6, 2024

artagnon left a comment

artagnon Dec 5, 2024

artagnon Dec 5, 2024

Mel-Chen Dec 6, 2024

david-arm Dec 9, 2024

Mel-Chen Jan 3, 2025

david-arm Dec 9, 2024

Mel-Chen Jan 3, 2025

Mel-Chen commented Apr 7, 2025

Mel-Chen commented Apr 14, 2025

Mel-Chen commented Apr 29, 2025

fhahn left a comment

		return Cur->getOpcode() == RedOp;
		return Cur->getOpcode() == getOpcode();

		if (RedOp == Instruction::ICmp \|\| RedOp == Instruction::FCmp)
		if (!IsArithmetic)

[IVDescriptors] Call getOpcode on demand in getReductionOpChain. nfc #118777

[IVDescriptors] Call getOpcode on demand in getReductionOpChain. nfc #118777

Conversation

Mel-Chen commented Dec 5, 2024 • edited Loading

llvmbot commented Dec 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artagnon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mel-Chen commented Apr 7, 2025

Mel-Chen commented Apr 14, 2025

Mel-Chen commented Apr 29, 2025

fhahn left a comment

Choose a reason for hiding this comment

Mel-Chen commented Dec 5, 2024 •

edited

Loading