cmd/compile: fix deadlock in (*Named).load

For lazy import resolution, there's reentrancy issue with (*Named).load
method, when "t.resolve(t)" can lead us to the same named type, thus
(*Named).load is called recursively, causing the deadlock.

The main problem is that when instantinate a type, we calculate the type
hashing, including TParams. Calling t.TParams().Len() triggers the
reentrancy call to "(*Named).load".

To fix this, just not checking TParams().Len() if we are hashing.

Updates #48185

Change-Id: Ie34842d7b10fad5d11fbcf75bb1c64a89deac6b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/347534
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>

3 files changed, 37 insertions, 0 deletions

diff --git a/test/typeparam/issue48185a.dir/p.go b/test/typeparam/issue48185a.dir/p.go
new file mode 100644
index 0000000000..176c7f4de5
--- /dev/null
+++ b/test/typeparam/issue48185a.dir/p.go
@@ -0,0 +1,19 @@
+// Copyright 2021 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package p
+
+type MarshalOptions struct {
+	Marshalers *Marshalers
+}
+
+type Encoder struct {}
+
+type Marshalers = marshalers[MarshalOptions, Encoder]
+
+type marshalers[Options, Coder any] struct{}
+
+func MarshalFuncV1[T any](fn func(T) ([]byte, error)) *Marshalers {
+	return &Marshalers{}
+}
diff --git a/test/typeparam/issue48185a.dir/p_test.go b/test/typeparam/issue48185a.dir/p_test.go
new file mode 100644
index 0000000000..52c87a7e29
--- /dev/null
+++ b/test/typeparam/issue48185a.dir/p_test.go
@@ -0,0 +1,11 @@
+// Copyright 2021 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main
+
+import "p"
+
+func main() {
+	_ = p.MarshalFuncV1[int](func(int) ([]byte, error) { return nil, nil })
+}
diff --git a/test/typeparam/issue48185a.go b/test/typeparam/issue48185a.go
new file mode 100644
index 0000000000..40df49f83b
--- /dev/null
+++ b/test/typeparam/issue48185a.go
@@ -0,0 +1,7 @@
+// rundir
+
+// Copyright 2021 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package ignored