cmd/compile: fix sign/zero-extension removal

When an opcode generates a known high bit state (typically, a sub-word
operation that zeros the high bits), we can remove any subsequent
extension operation that would be a no-op.

x = (OP ...)
y = (ZeroExt32to64 x)

If OP zeros the high 32 bits, then we can replace y with x, as the
zero extension doesn't do anything.

However, x in this situation normally has a sub-word-sized type.  The
semantics of values in registers is typically that the high bits
beyond the value's type size are junk. So although the opcode
generating x *currently* zeros the high bits, after x is rewritten to
another opcode it may not - rewrites of sub-word-sized values can
trash the high bits.

To fix, move the extension-removing rules to late lower. That ensures
that their arguments won't be rewritten to change their high bits.

I am also worried about spilling and restoring. Spilling and restoring
doesn't preserve the high bits, but instead sets them to a known value
(often 0, but in some cases it could be sign-extended).  I am unable
to come up with a case that would cause a problem here, so leaving for
another time.

Fixes #66066

Change-Id: I3b5c091b3b3278ccbb7f11beda8b56f4b6d3fde7
Reviewed-on: https://go-review.googlesource.com/c/go/+/568616
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

1 files changed, 41 insertions, 0 deletions

diff --git a/test/fixedbugs/issue66066.go b/test/fixedbugs/issue66066.go
new file mode 100644
index 0000000000..a674503b47
--- /dev/null
+++ b/test/fixedbugs/issue66066.go
@@ -0,0 +1,41 @@
+// run
+
+// Copyright 2024 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main
+
+import "fmt"
+
+func main() {
+	testMod()
+	testMul()
+}
+
+//go:noinline
+func mod3(x uint32) uint64 {
+	return uint64(x % 3)
+}
+
+func testMod() {
+	got := mod3(1<<32 - 1)
+	want := uint64((1<<32 - 1) % 3)
+	if got != want {
+		fmt.Printf("testMod: got %x want %x\n", got, want)
+	}
+
+}
+
+//go:noinline
+func mul3(a uint32) uint64 {
+	return uint64(a * 3)
+}
+
+func testMul() {
+	got := mul3(1<<32 - 1)
+	want := uint64((1<<32-1)*3 - 2<<32)
+	if got != want {
+		fmt.Printf("testMul: got %x want %x\n", got, want)
+	}
+}