cmd/compile: use desired register only if it satisfies register mask

In the register allocator, if possible, we allocate a value to its
desired register (the ideal register for its next use). In some
cases the desired register does not satisfies the value's output
register mask. We should not use the register in this case.

In the following example, v33 is going to be returned as a
function result, so it is allocated to its desired register AX.
However, its Op cannot use AX as output, causing miscompilation.

v33 = CMOVQEQF <int> v24 v28 v29 : AX (~R0[int])
v35 = MakeResult <int,int,mem> v33 v26 v18
Ret v35

Change-Id: Id0f4f27c4b233ee297e83077e3c8494fe193e664
Reviewed-on: https://go-review.googlesource.com/c/go/+/314630
Trust: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>

1 files changed, 46 insertions, 0 deletions

diff --git a/test/abi/result_regalloc.go b/test/abi/result_regalloc.go
new file mode 100644
index 0000000000..58aecad970
--- /dev/null
+++ b/test/abi/result_regalloc.go
@@ -0,0 +1,46 @@
+// run
+
+// Copyright 2021 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Bug: in (*bb).d, the value to be returned was not allocated to
+// a register that satisfies its register mask.
+
+package main
+
+type bb struct {
+	r float64
+	x []float64
+}
+
+//go:noinline
+func B(r float64, x []float64) I {
+	return bb{r, x}
+}
+
+func (b bb) d() (int, int) {
+	if b.r == 0 {
+		return 0, len(b.x)
+	}
+	return len(b.x), len(b.x)
+}
+
+type I interface { d() (int, int) }
+
+func D(r I) (int, int) { return r.d() }
+
+//go:noinline
+func F() (int, int) {
+	r := float64(1)
+	x := []float64{0, 1, 2}
+	b := B(r, x)
+	return D(b)
+}
+
+func main() {
+	x, y := F()
+	if x != 3 || y != 3 {
+		panic("FAIL")
+	}
+}