runtime: fix over-aggressive proportional sweep

Currently, sweeping is performed before allocating a span by charging
for the entire size of the span requested, rather than the number of
bytes actually available for allocation from the returned span. That
is, if the returned span is 8K, but already has 6K in use, the mutator
is charged for 8K of heap allocation even though it can only allocate
2K more from the span. As a result, proportional sweep is
over-aggressive and tends to finish much earlier than it needs to.
This effect is more amplified by fragmented heaps.

Fix this by reimbursing the mutator for the used space in a span once
it has allocated that span. We still have to charge up-front for the
worst-case because we don't know which span the mutator will get, but
at least we can correct the over-charge once it has a span, which will
go toward later span allocations.

This has negligible effect on the throughput of the go1 benchmarks and
the garbage benchmark.

Fixes #12040.

Change-Id: I0e23e7a4ccf126cca000fed5067b20017028dd6b
Reviewed-on: https://go-review.googlesource.com/16515
Reviewed-by: Rick Hudson <rlh@golang.org>

1 files changed, 4 insertions, 0 deletions

diff --git a/src/runtime/mcentral.go b/src/runtime/mcentral.go
index 1e1f6fd13d..a09801a549 100644
--- a/src/runtime/mcentral.go
+++ b/src/runtime/mcentral.go
@@ -101,6 +101,10 @@ havespan:
 	if n == 0 {
 		throw("empty span")
 	}
+	usedBytes := uintptr(s.ref) * s.elemsize
+	if usedBytes > 0 {
+		reimburseSweepCredit(usedBytes)
+	}
 	if s.freelist.ptr() == nil {
 		throw("freelist empty")
 	}