[VectorUtils][VPlan] Consolidate VPWidenIntrinsicRecipe::onlyFirstLaneUsed and isVectorIntrinsicWithScalarOpAtArg #137497

lukel97 · 2025-04-27T05:46:32Z

We can reuse isVectorIntrinsicWithScalarOpAtArg in VectorUtils to determine if only the first lane will be used for a VPWidenIntrinsicRecipe, provided that we also move the VP EVL operand check into it.

This was needed by a local patch I was working on that created a VPWidenIntrinsicRecipe with a VP intrinsic, and prevents the need to update the scalar arguments in two places.

llvmbot · 2025-04-27T05:47:05Z

@llvm/pr-subscribers-llvm-transforms
@llvm/pr-subscribers-llvm-analysis

@llvm/pr-subscribers-vectorizers

Author: Luke Lau (lukel97)

Changes

We can reuse isVectorIntrinsicWithScalarOpAtArg in VectorUtils to determine if only the first lane will be used for a VPWidenIntrinsicRecipe, provided that we also move the VP EVL operand check into it.

This was needed by a local patch I was working on that created a VPWidenIntrinsicRecipe with a VP intrinsic, and prevents the need to update the scalar arguments in two places.

Full diff: https://github.com/llvm/llvm-project/pull/137497.diff

2 Files Affected:

(modified) llvm/lib/Analysis/VectorUtils.cpp (+6-1)
(modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+2-4)

diff --git a/llvm/lib/Analysis/VectorUtils.cpp b/llvm/lib/Analysis/VectorUtils.cpp
index 6448c372f5d5d..b614ac50d2f82 100644
--- a/llvm/lib/Analysis/VectorUtils.cpp
+++ b/llvm/lib/Analysis/VectorUtils.cpp
@@ -149,6 +149,11 @@ bool llvm::isVectorIntrinsicWithScalarOpAtArg(Intrinsic::ID ID,
   if (TTI && Intrinsic::isTargetIntrinsic(ID))
     return TTI->isTargetIntrinsicWithScalarOpAtArg(ID, ScalarOpdIdx);
 
+  // Vector predication intrinsics only demand the the first lane the last
+  // operand (the EVL operand).
+  if (VPIntrinsic::getVectorLengthParamPos(ID) == ScalarOpdIdx)
+    return true;
+
   switch (ID) {
   case Intrinsic::abs:
   case Intrinsic::vp_abs:
@@ -166,7 +171,7 @@ bool llvm::isVectorIntrinsicWithScalarOpAtArg(Intrinsic::ID ID,
   case Intrinsic::umul_fix_sat:
     return (ScalarOpdIdx == 2);
   case Intrinsic::experimental_vp_splice:
-    return ScalarOpdIdx == 2 || ScalarOpdIdx == 4 || ScalarOpdIdx == 5;
+    return ScalarOpdIdx == 2 || ScalarOpdIdx == 4;
   default:
     return false;
   }
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 8ec092ce9a905..787b416ee9248 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -1373,10 +1373,8 @@ StringRef VPWidenIntrinsicRecipe::getIntrinsicName() const {
 
 bool VPWidenIntrinsicRecipe::onlyFirstLaneUsed(const VPValue *Op) const {
   assert(is_contained(operands(), Op) && "Op must be an operand of the recipe");
-  // Vector predication intrinsics only demand the the first lane the last
-  // operand (the EVL operand).
-  return VPIntrinsic::isVPIntrinsic(VectorIntrinsicID) &&
-         Op == getOperand(getNumOperands() - 1);
+  unsigned Idx = std::distance(op_begin(), find(operands(), Op));
+  return isVectorIntrinsicWithScalarOpAtArg(VectorIntrinsicID, Idx, nullptr);
 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)

Mel-Chen · 2025-04-29T14:14:18Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

-  // operand (the EVL operand).
-  return VPIntrinsic::isVPIntrinsic(VectorIntrinsicID) &&
-         Op == getOperand(getNumOperands() - 1);
+  unsigned Idx = std::distance(op_begin(), find(operands(), Op));


Is it possible for Op to appear more than once in operands()?

Good point, I think so. But hopefully having multiple uses of a value in both scalar and vector positions never happens.

If so I think that would be an existing problem with onlyFirstLaneUsed. In which case we should probably be passing around the specific operand index for *Op

onlyFirstLaneUsed should only return true if all uses only use the first lane;

Ah that makes more sense, I've handled that in 353adf0

fhahn · 2025-04-29T14:32:16Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

-  return VPIntrinsic::isVPIntrinsic(VectorIntrinsicID) &&
-         Op == getOperand(getNumOperands() - 1);
+  unsigned Idx = std::distance(op_begin(), find(operands(), Op));
+  return isVectorIntrinsicWithScalarOpAtArg(VectorIntrinsicID, Idx, nullptr);


Can you add a test for the new behavior?

It turns out yes, I was able to add a case where we no longer broadcast and extract the scalar operand to powi. It should be precomitted in this PR now

fhahn · 2025-04-29T14:33:06Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

-  // operand (the EVL operand).
-  return VPIntrinsic::isVPIntrinsic(VectorIntrinsicID) &&
-         Op == getOperand(getNumOperands() - 1);
+  unsigned Idx = std::distance(op_begin(), find(operands(), Op));


onlyFirstLaneUsed should only return true if all uses only use the first lane;

…eUsed and isVectorIntrinsicWithScalarOpAtArg We can reuse isVectorIntrinsicWithScalarOpAtArg in VectorUtils to determine if only the first lane will be used for a VPWidenIntrinsicRecipe, provided that we also move the VP EVL operand check into it. This was needed by a local patch I was working on that created a VPWidenIntrinsicRecipe with a VP intrinsic, and prevents the need to update the scalar arguments in two places.

fhahn · 2025-04-29T19:30:40Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+  for (auto [Idx, V] : enumerate(operands())) {
+    if (V != Op)
+      continue;
+    if (!isVectorIntrinsicWithScalarOpAtArg(VectorIntrinsicID, Idx, nullptr))
+      return false;
+  }
+  return true;


can we use any_of/all_of?

Mel-Chen

LG

fhahn

LGTM, thanks

fhahn · 2025-04-30T08:54:59Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

-  // operand (the EVL operand).
-  return VPIntrinsic::isVPIntrinsic(VectorIntrinsicID) &&
-         Op == getOperand(getNumOperands() - 1);
+  return all_of(enumerate(operands()), [this, &Op](auto &&X) {


might be simpler to just use, if it works

Suggested change

return all_of(enumerate(operands()), [this, &Op](auto &&X) {

return all_of(enumerate(operands()), [this, &Op](const auto &X) {

fhahn · 2025-04-30T08:56:31Z

llvm/lib/Analysis/VectorUtils.cpp

+  // Vector predication intrinsics only demand the the first lane the last
+  // operand (the EVL operand).


Suggested change

// Vector predication intrinsics only demand the the first lane the last

// operand (the EVL operand).

// Vector predication intrinsics only demand the first lane of the EVL operand.

There's no reference to the last argument here, as this uses VPIntrinsic::getVectorLengthParamPos, so better to first mention that this is for the EVL operand?

fhahn · 2025-04-30T08:59:16Z

llvm/test/Transforms/LoopVectorize/widen-intrinsic.ll

+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -S | FileCheck %s
+
+; Check that we don't unnecessarily broadcast %pow
+define void @powi(ptr noalias %p, i32 %pow) {


Suggested change

define void @powi(ptr noalias %p, i32 %pow) {

define void @powi_only_first_lane_used_of_second_arg(ptr noalias %p, i32 %pow) {

no alias could also be stripped, as only p is accessed.

…eUsed and isVectorIntrinsicWithScalarOpAtArg (llvm#137497) We can reuse isVectorIntrinsicWithScalarOpAtArg in VectorUtils to determine if only the first lane will be used for a VPWidenIntrinsicRecipe, provided that we also move the VP EVL operand check into it. This was needed by a local patch I was working on that created a VPWidenIntrinsicRecipe with a VP intrinsic, and prevents the need to update the scalar arguments in two places.

lukel97 requested review from fhahn, Mel-Chen and RKSimon April 27, 2025 05:46

llvmbot added vectorizers llvm:analysis llvm:transforms labels Apr 27, 2025

Mel-Chen reviewed Apr 29, 2025

View reviewed changes

fhahn reviewed Apr 29, 2025

View reviewed changes

lukel97 added 3 commits April 29, 2025 23:18

Precommit test

ecc63e7

Check all operands

353adf0

lukel97 force-pushed the loop-vectorize/consolidate-isVectorIntrinsicWithScalarOpAtArg branch from a3475ac to 353adf0 Compare April 29, 2025 15:19

fhahn reviewed Apr 29, 2025

View reviewed changes

Use all_of

e525298

Mel-Chen approved these changes Apr 30, 2025

View reviewed changes

fhahn approved these changes Apr 30, 2025

View reviewed changes

Use const, update EVL comment, remove noalias, rename test

162896e

lukel97 merged commit 2cd829f into llvm:main Apr 30, 2025
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VectorUtils][VPlan] Consolidate VPWidenIntrinsicRecipe::onlyFirstLaneUsed and isVectorIntrinsicWithScalarOpAtArg #137497

[VectorUtils][VPlan] Consolidate VPWidenIntrinsicRecipe::onlyFirstLaneUsed and isVectorIntrinsicWithScalarOpAtArg #137497

lukel97 commented Apr 27, 2025

llvmbot commented Apr 27, 2025 •

edited

Loading

Mel-Chen Apr 29, 2025

lukel97 Apr 29, 2025

fhahn Apr 29, 2025

lukel97 Apr 29, 2025

fhahn Apr 29, 2025

lukel97 Apr 29, 2025

fhahn Apr 29, 2025

fhahn Apr 29, 2025

Mel-Chen left a comment

fhahn left a comment

fhahn Apr 30, 2025

fhahn Apr 30, 2025

fhahn Apr 30, 2025

	return all_of(enumerate(operands()), [this, &Op](auto &&X) {
	return all_of(enumerate(operands()), [this, &Op](const auto &X) {

		// Vector predication intrinsics only demand the the first lane the last
		// operand (the EVL operand).

	define void @powi(ptr noalias %p, i32 %pow) {
	define void @powi_only_first_lane_used_of_second_arg(ptr noalias %p, i32 %pow) {

[VectorUtils][VPlan] Consolidate VPWidenIntrinsicRecipe::onlyFirstLaneUsed and isVectorIntrinsicWithScalarOpAtArg #137497

[VectorUtils][VPlan] Consolidate VPWidenIntrinsicRecipe::onlyFirstLaneUsed and isVectorIntrinsicWithScalarOpAtArg #137497

Conversation

lukel97 commented Apr 27, 2025

llvmbot commented Apr 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mel-Chen left a comment

Choose a reason for hiding this comment

fhahn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llvmbot commented Apr 27, 2025 •

edited

Loading