-
Notifications
You must be signed in to change notification settings - Fork 319
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Steer round to even to Z's hardware operation (for Z), and MLIR/LLVM roundEven for the other platforms. #2970
Merged
AlexandreEichenberger
merged 30 commits into
onnx:main
from
AlexandreEichenberger:round-opt-v2
Oct 10, 2024
Merged
Changes from 25 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
e5d44b9
use one over scale for quantize linear
AlexandreEichenberger dcd0fd4
add krnl.round
AlexandreEichenberger 23beab9
update
AlexandreEichenberger ac301f0
update
AlexandreEichenberger cf2bd9e
fix lit test because of 1/scale usage
AlexandreEichenberger 17cff60
try to use roundEven
AlexandreEichenberger b85672b
update
AlexandreEichenberger dea818e
update
AlexandreEichenberger 5ac6305
first attempt with hw roundeven
AlexandreEichenberger c7a39f5
cleaner interface
AlexandreEichenberger 0c0a908
use shapecast
AlexandreEichenberger b128e6e
added lowering pattern for shapecast
AlexandreEichenberger a022acf
fix merge issue, add reg pressure to quant lin
AlexandreEichenberger e41eb81
update
AlexandreEichenberger e08351f
update
AlexandreEichenberger b8ebc2e
enable scalar fiebr
AlexandreEichenberger 2d25e55
update
AlexandreEichenberger 5f1999c
update
AlexandreEichenberger c6239bd
update
AlexandreEichenberger 9cd309b
cleanup
AlexandreEichenberger 4f92d93
cleanup
AlexandreEichenberger e938dd0
fix lit tests
AlexandreEichenberger 02e2f96
added doc for new Krnl op
AlexandreEichenberger f620eb3
format
AlexandreEichenberger 42040e7
Merge branch 'main' into round-opt-v2
AlexandreEichenberger 887ccd5
update
AlexandreEichenberger 5b6b867
Merge branch 'round-opt-v2' of https://github.com/AlexandreEichenberg…
AlexandreEichenberger 23031aa
respond to comments
AlexandreEichenberger 3977369
reverted to using round even wiht emulation do to a mac os issue
AlexandreEichenberger 9622cee
also reverted the non-z16 tests
AlexandreEichenberger File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,115 @@ | ||
/* | ||
* SPDX-License-Identifier: Apache-2.0 | ||
*/ | ||
|
||
//===------ KrnlRoundEven.cpp - Lower KrnlRoundEvenOp ---------------------===// | ||
// | ||
// Copyright 2019-2024 The IBM Research Authors. | ||
// | ||
// ============================================================================= | ||
// | ||
// This file lowers the KrnlRoundEvenOp operator. | ||
// | ||
// Currently limited to fp32 integers, instructions supports other data types. | ||
//===----------------------------------------------------------------------===// | ||
|
||
#include "mlir/Conversion/LLVMCommon/Pattern.h" | ||
#include "mlir/Conversion/LLVMCommon/TypeConverter.h" | ||
#include "mlir/Dialect/Func/IR/FuncOps.h" | ||
#include "mlir/Dialect/LLVMIR/LLVMDialect.h" | ||
|
||
#include "src/Conversion/KrnlToLLVM/KrnlToLLVMHelper.hpp" | ||
#include "src/Dialect/Krnl/KrnlHelper.hpp" | ||
#include "src/Dialect/Krnl/KrnlOps.hpp" | ||
#include "src/Dialect/Mlir/DialectBuilder.hpp" | ||
#include "llvm/Support/Debug.h" | ||
|
||
#define DEBUG_TYPE "krnl_to_llvm" | ||
|
||
using namespace mlir; | ||
|
||
namespace onnx_mlir { | ||
namespace krnl { | ||
|
||
class KrnlRoundEvenOpLowering : public ConversionPattern { | ||
public: | ||
explicit KrnlRoundEvenOpLowering( | ||
LLVMTypeConverter &typeConverter, MLIRContext *context) | ||
: ConversionPattern( | ||
typeConverter, KrnlRoundEvenOp::getOperationName(), 1, context) {} | ||
LogicalResult matchAndRewrite(Operation *op, ArrayRef<Value> operands, | ||
ConversionPatternRewriter &rewriter) const override { | ||
Location loc = op->getLoc(); | ||
KrnlRoundEvenOp::Adaptor operandAdaptor(operands); | ||
Value input = operandAdaptor.getIn(); | ||
|
||
// Scalar or Vector? | ||
Type inputType = input.getType(); | ||
Type inputElemType = getElementTypeOrSelf(inputType); | ||
assert(mlir::isa<FloatType>(inputElemType) && "expected float"); | ||
int64_t inputBitWidth = inputElemType.getIntOrFloatBitWidth(); | ||
assert(inputBitWidth == 32 && "expected 32bit float"); | ||
VectorType inputVecType = mlir::dyn_cast<VectorType>(inputType); | ||
assert(VectorMachineSupport::requireCustomASM( | ||
GenericOps::roundEvenGop, inputElemType) && | ||
"expected custom requirement"); | ||
// Common between scalar and vector | ||
MultiDialectBuilder<LLVMBuilder> create(rewriter, loc); | ||
Type i32Ty = rewriter.getI32Type(); | ||
Type f32Ty = rewriter.getF32Type(); | ||
|
||
if (inputVecType) { | ||
// Vector of 4 elements. | ||
Type vecTypeI32 = LLVM::getFixedVectorType(i32Ty, 4); | ||
Type vecTypeF32 = LLVM::getFixedVectorType(f32Ty, 4); | ||
// Use integer as container for inputs. | ||
Value inputVecI32 = create.llvm.bitcast(vecTypeI32, input); | ||
SmallVector<Value> asmVals{inputVecI32}; | ||
// SIMD ASM round to nearest even (M5=4) op | ||
const char *asmStr = "VFISB $0,$1,0,4"; | ||
const char *asmConstraints = "=v,v"; | ||
Value outVecI32 = | ||
rewriter | ||
.create<LLVM::InlineAsmOp>(loc, vecTypeI32, | ||
/*operands=*/asmVals, | ||
/*asm_string=*/asmStr, | ||
/*constraints=*/asmConstraints, /*has_side_effects=*/false, | ||
/*is_align_stack=*/false, | ||
/*asm_dialect=*/LLVM::AsmDialectAttr(), | ||
/*operand_attrs=*/ArrayAttr()) | ||
.getResult(0); | ||
// Cast output back to float. | ||
Value outVecF32 = create.llvm.bitcast(vecTypeF32, outVecI32); | ||
rewriter.replaceOp(op, {outVecF32}); | ||
return success(); | ||
} else { | ||
// Scalar types. | ||
Type typeF32 = rewriter.getF32Type(); | ||
SmallVector<Value> asmVals{input}; | ||
// Scalar ASM round to the nearest even (M3=4) op. | ||
const char *asmStr = "FIEBR $0,4,$1"; | ||
const char *asmConstraints = "=f,f"; | ||
Value outF32 = | ||
rewriter | ||
.create<LLVM::InlineAsmOp>(loc, typeF32, | ||
/*operands=*/asmVals, | ||
/*asm_string=*/asmStr, | ||
/*constraints=*/asmConstraints, /*has_side_effects=*/false, | ||
/*is_align_stack=*/false, | ||
/*asm_dialect=*/LLVM::AsmDialectAttr(), | ||
/*operand_attrs=*/ArrayAttr()) | ||
.getResult(0); | ||
rewriter.replaceOp(op, {outF32}); | ||
return success(); | ||
} | ||
llvm_unreachable("not supported"); | ||
} | ||
}; | ||
|
||
void populateLoweringKrnlRoundEvenOpPattern(LLVMTypeConverter &typeConverter, | ||
RewritePatternSet &patterns, MLIRContext *ctx) { | ||
patterns.insert<KrnlRoundEvenOpLowering>(typeConverter, ctx); | ||
} | ||
|
||
} // namespace krnl | ||
} // namespace onnx_mlir |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This
else
looks redundant since we havereturn
inside theif
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tx, will fix