Grow && Copy

在 Go 语言中,slice 的 cap 是不能直接增加的,例如,如果我们想把一个 slice 的 cap 改为之前的 2 倍,需要创建一个新 slice ,然后使新 slice 的 cap 为原 slice 的 2 倍,之后再把原 slice 中的内容 copy 到新 slice 然后再让原 slice 等于新 slice,听起来有些绕,来看一下图解。

image.png
其中 copy 方法会自动判断并选择较小的长度进行复制,比如长度 6 的 slice 如果要 copy 到长度为 3 的 slice 中,那么只会 copy 前三个元素,这部分逻辑在源码 L2 ~ L9 可以清晰的看到,长度默认等于原 slice 长度,如果目标长度小于原 slice 长度,则长度为目标长度。

  1. func slicecopy(toPtr unsafe.Pointer, toLen int, fromPtr unsafe.Pointer, fromLen int, width uintptr) int {
  2. if fromLen == 0 || toLen == 0 {
  3. return 0
  4. }
  5. n := fromLen
  6. if toLen < n {
  7. n = toLen
  8. }
  9. if width == 0 {
  10. return n
  11. }
  12. size := uintptr(n) * width
  13. if raceenabled {
  14. callerpc := getcallerpc()
  15. pc := abi.FuncPCABIInternal(slicecopy)
  16. racereadrangepc(fromPtr, size, callerpc, pc)
  17. racewriterangepc(toPtr, size, callerpc, pc)
  18. }
  19. if msanenabled {
  20. msanread(fromPtr, size)
  21. msanwrite(toPtr, size)
  22. }
  23. if asanenabled {
  24. asanread(fromPtr, size)
  25. asanwrite(toPtr, size)
  26. }
  27. if size == 1 { // common case worth about 2x to do here
  28. // TODO: is this still worth it with new memmove impl?
  29. *(*byte)(toPtr) = *(*byte)(fromPtr) // known to be a byte pointer
  30. } else {
  31. memmove(toPtr, fromPtr, size)
  32. }
  33. return n
  34. }

在 copy 时,如果双方共用一个底层数组,则需要注意一下由指针带来的底层数组变动,比如在下例中,原 slice 在进行 copy 之后被意外的修改。
image.png

Append

Append 方法的作用是向一个 slice 中追加内容,如果目标 slice 剩余 cap 不足以存放追加内容,则会自动扩容,需要注意的是,在扩容时会声明另一个底层数组,并且使 slice 的 ptr 重新指向新的底层数组,反之会继续延用之前的底层数组,图 3 和 图 4 分别对应以上两种情况。
image.png
image.png

Append-grow

Slice 在 append 时会自动扩容,下面我们来看看扩容的规则。

  1. func growslice(et *_type, old slice, cap int) slice {
  2. if raceenabled {
  3. callerpc := getcallerpc()
  4. racereadrangepc(old.array, uintptr(old.len*int(et.size)), callerpc, abi.FuncPCABIInternal(growslice))
  5. }
  6. if msanenabled {
  7. msanread(old.array, uintptr(old.len*int(et.size)))
  8. }
  9. if asanenabled {
  10. asanread(old.array, uintptr(old.len*int(et.size)))
  11. }
  12. if cap < old.cap {
  13. panic(errorString("growslice: cap out of range"))
  14. }
  15. if et.size == 0 {
  16. // append should not create a slice with nil pointer but non-zero len.
  17. // We assume that append doesn't need to preserve old.array in this case.
  18. return slice{unsafe.Pointer(&zerobase), old.len, cap}
  19. }
  20. newcap := old.cap
  21. doublecap := newcap + newcap
  22. if cap > doublecap {
  23. newcap = cap
  24. } else {
  25. const threshold = 256
  26. if old.cap < threshold {
  27. newcap = doublecap
  28. } else {
  29. // Check 0 < newcap to detect overflow
  30. // and prevent an infinite loop.
  31. for 0 < newcap && newcap < cap {
  32. // Transition from growing 2x for small slices
  33. // to growing 1.25x for large slices. This formula
  34. // gives a smooth-ish transition between the two.
  35. newcap += (newcap + 3*threshold) / 4
  36. }
  37. // Set newcap to the requested cap when
  38. // the newcap calculation overflowed.
  39. if newcap <= 0 {
  40. newcap = cap
  41. }
  42. }
  43. }
  44. var overflow bool
  45. var lenmem, newlenmem, capmem uintptr
  46. // Specialize for common values of et.size.
  47. // For 1 we don't need any division/multiplication.
  48. // For goarch.PtrSize, compiler will optimize division/multiplication into a shift by a constant.
  49. // For powers of 2, use a variable shift.
  50. switch {
  51. case et.size == 1:
  52. lenmem = uintptr(old.len)
  53. newlenmem = uintptr(cap)
  54. capmem = roundupsize(uintptr(newcap))
  55. overflow = uintptr(newcap) > maxAlloc
  56. newcap = int(capmem)
  57. case et.size == goarch.PtrSize:
  58. lenmem = uintptr(old.len) * goarch.PtrSize
  59. newlenmem = uintptr(cap) * goarch.PtrSize
  60. capmem = roundupsize(uintptr(newcap) * goarch.PtrSize)
  61. overflow = uintptr(newcap) > maxAlloc/goarch.PtrSize
  62. newcap = int(capmem / goarch.PtrSize)
  63. case isPowerOfTwo(et.size):
  64. var shift uintptr
  65. if goarch.PtrSize == 8 {
  66. // Mask shift for better code generation.
  67. shift = uintptr(sys.Ctz64(uint64(et.size))) & 63
  68. } else {
  69. shift = uintptr(sys.Ctz32(uint32(et.size))) & 31
  70. }
  71. lenmem = uintptr(old.len) << shift
  72. newlenmem = uintptr(cap) << shift
  73. capmem = roundupsize(uintptr(newcap) << shift)
  74. overflow = uintptr(newcap) > (maxAlloc >> shift)
  75. newcap = int(capmem >> shift)
  76. default:
  77. lenmem = uintptr(old.len) * et.size
  78. newlenmem = uintptr(cap) * et.size
  79. capmem, overflow = math.MulUintptr(et.size, uintptr(newcap))
  80. capmem = roundupsize(capmem)
  81. newcap = int(capmem / et.size)
  82. }
  83. // The check of overflow in addition to capmem > maxAlloc is needed
  84. // to prevent an overflow which can be used to trigger a segfault
  85. // on 32bit architectures with this example program:
  86. //
  87. // type T [1<<27 + 1]int64
  88. //
  89. // var d T
  90. // var s []T
  91. //
  92. // func main() {
  93. // s = append(s, d, d, d, d)
  94. // print(len(s), "\n")
  95. // }
  96. if overflow || capmem > maxAlloc {
  97. panic(errorString("growslice: cap out of range"))
  98. }
  99. var p unsafe.Pointer
  100. if et.ptrdata == 0 {
  101. p = mallocgc(capmem, nil, false)
  102. // The append() that calls growslice is going to overwrite from old.len to cap (which will be the new length).
  103. // Only clear the part that will not be overwritten.
  104. memclrNoHeapPointers(add(p, newlenmem), capmem-newlenmem)
  105. } else {
  106. // Note: can't use rawmem (which avoids zeroing of memory), because then GC can scan uninitialized memory.
  107. p = mallocgc(capmem, et, true)
  108. if lenmem > 0 && writeBarrier.enabled {
  109. // Only shade the pointers in old.array since we know the destination slice p
  110. // only contains nil pointers because it has been cleared during alloc.
  111. bulkBarrierPreWriteSrcOnly(uintptr(p), uintptr(old.array), lenmem-et.size+et.ptrdata)
  112. }
  113. }
  114. memmove(p, old.array, lenmem)
  115. return slice{p, old.len, newcap}
  116. }

观察源码,slice 主要的扩容逻辑在 L23 ~ L46 ,主要分为以下三种情况。

  1. 原 slice 容量小于 256,并且最小需求容量小于二倍原 slice 容量。
  2. 原 slice 容量大于 256,并且最小需求容量小于二倍原 slice 容量。
  3. 最小需求容量大于二倍原 slice 容量。

在第一种情况中,容量会直接乘 2,第二种情况中容量的增长量会随着原容量越来越大而过渡到 1.25 倍,注意是过渡,而不是立刻变为 1.25 倍,接下来我们来看一下测试。

  1. func main() {
  2. testing := make([]int, 0)
  3. appendRange := math.MaxInt64
  4. lastCap := 0
  5. for i := 0; i <= appendRange; i++ {
  6. testing = append(testing, i)
  7. nowCap := cap(testing)
  8. if (nowCap != lastCap) {
  9. fmt.Printf(
  10. "append: %9d, addr: %p, len: %9d, cap: %9d -- %9f;\n",
  11. i, testing, len(testing), nowCap,
  12. float32(nowCap) / float32(lastCap)
  13. )
  14. }
  15. lastCap = nowCap
  16. }
  17. }
  18. //append: 0, addr: 0xc0000a6000, len: 1, cap: 1 -- +Inf;
  19. //append: 1, addr: 0xc0000a6020, len: 2, cap: 2 -- 2.000000;
  20. //append: 2, addr: 0xc0000b0000, len: 3, cap: 4 -- 2.000000;
  21. //append: 4, addr: 0xc0000a0040, len: 5, cap: 8 -- 2.000000;
  22. //append: 8, addr: 0xc0000b2000, len: 9, cap: 16 -- 2.000000;
  23. //append: 16, addr: 0xc0000b4000, len: 17, cap: 32 -- 2.000000;
  24. //append: 32, addr: 0xc0000b6000, len: 33, cap: 64 -- 2.000000;
  25. //append: 64, addr: 0xc0000b8000, len: 65, cap: 128 -- 2.000000;
  26. //append: 128, addr: 0xc0000ba000, len: 129, cap: 256 -- 2.000000;
  27. //append: 256, addr: 0xc0000bc000, len: 257, cap: 512 -- 2.000000;
  28. //append: 512, addr: 0xc0000be000, len: 513, cap: 848 -- 1.656250;
  29. //append: 848, addr: 0xc0000c8000, len: 849, cap: 1280 -- 1.509434;
  30. //append: 1280, addr: 0xc0000d2000, len: 1281, cap: 1792 -- 1.400000;
  31. //append: 1792, addr: 0xc0000e0000, len: 1793, cap: 2560 -- 1.428571;
  32. //append: 2560, addr: 0xc0000ea000, len: 2561, cap: 3408 -- 1.331250;
  33. //append: 3408, addr: 0xc000100000, len: 3409, cap: 5120 -- 1.502347;
  34. //append: 5120, addr: 0xc00010a000, len: 5121, cap: 7168 -- 1.400000;
  35. //append: 7168, addr: 0xc000118000, len: 7169, cap: 9216 -- 1.285714;
  36. //append: 9216, addr: 0xc00012a000, len: 9217, cap: 12288 -- 1.333333;
  37. //append: 12288, addr: 0xc000142000, len: 12289, cap: 16384 -- 1.333333;
  38. //append: 16384, addr: 0xc000162000, len: 16385, cap: 21504 -- 1.312500;
  39. //append: 21504, addr: 0xc00018c000, len: 21505, cap: 27648 -- 1.285714;
  40. //append: 27648, addr: 0xc0001c2000, len: 27649, cap: 34816 -- 1.259259;
  41. //append: 34816, addr: 0xc000206000, len: 34817, cap: 44032 -- 1.264706;
  42. //append: 44032, addr: 0xc00025c000, len: 44033, cap: 55296 -- 1.255814;
  43. //append: 55296, addr: 0xc0002c8000, len: 55297, cap: 69632 -- 1.259259;
  44. //append: 69632, addr: 0xc000350000, len: 69633, cap: 88064 -- 1.264706;
  45. //append: 88064, addr: 0xc0003fc000, len: 88065, cap: 110592 -- 1.255814;
  46. //append: 110592, addr: 0xc000100000, len: 110593, cap: 139264 -- 1.259259;
  47. //append: 139264, addr: 0xc000210000, len: 139265, cap: 175104 -- 1.257353;
  48. //append: 175104, addr: 0xc0004d4000, len: 175105, cap: 219136 -- 1.251462;
  49. //append: 219136, addr: 0xc000100000, len: 219137, cap: 274432 -- 1.252337;
  50. //append: 274432, addr: 0xc000680000, len: 274433, cap: 344064 -- 1.253731;
  51. //append: 344064, addr: 0xc000100000, len: 344065, cap: 431104 -- 1.252976;
  52. //append: 431104, addr: 0xc00044a000, len: 431105, cap: 539648 -- 1.251781;
  53. //append: 539648, addr: 0xc000868000, len: 539649, cap: 674816 -- 1.250474;
  54. //append: 674816, addr: 0xc0000b2000, len: 674817, cap: 843776 -- 1.250379;
  55. //append: 843776, addr: 0xc000722000, len: 843777, cap: 1055744 -- 1.251214;
  56. //append: 1055744, addr: 0xc000f30000, len: 1055745, cap: 1319936 -- 1.250242;
  57. //append: 1319936, addr: 0xc0000b2000, len: 1319937, cap: 1650688 -- 1.250582;
  58. //append: 1650688, addr: 0xc000d4a000, len: 1650689, cap: 2064384 -- 1.250620;
  59. //append: 2064384, addr: 0xc001d0a000, len: 2064385, cap: 2581504 -- 1.250496;
  60. //append: 2581504, addr: 0xc0000b2000, len: 2581505, cap: 3227648 -- 1.250298;
  61. //append: 3227648, addr: 0xc001952000, len: 3227649, cap: 4035584 -- 1.250317;
  62. //append: 4035584, addr: 0xc00381c000, len: 4035585, cap: 5045248 -- 1.250190;
  63. //append: 5045248, addr: 0xc0000b2000, len: 5045249, cap: 6306816 -- 1.250051;
  64. //append: 6306816, addr: 0xc0030d0000, len: 6306817, cap: 7883776 -- 1.250041;
  65. //append: 7883776, addr: 0xc006cf6000, len: 7883777, cap: 9854976 -- 1.250032;
  66. //append: 9854976, addr: 0xc0000b2000, len: 9854977, cap: 12319744 -- 1.250104;

以看到 L32 ~ L48 的输出中,cap 的增长在 1.65 ~ 1.26 倍之间,之后才平缓的达到 1.25 倍。打印输出的方式看起来可能不太直观,如果我们使用图表的形式看观察可以很清楚的看到一条由 2 到 1.25“过渡”的曲线。

line-simple.svg

最后就是第三种情况,这种情况下,只看源码的 L25 L26 行容量是等于最小需求容量的,但是在测试中发现,并不是简单的等于关系,Go 在之后的处理中会对计算出的新容量进行内存对齐,对此我们暂时不做深究。

  1. func main() {
  2. for i := 0; i < 200; i++ {
  3. testing := make([]int, 5)
  4. testing = append(testing, make([]int, i)...)
  5. fmt.Printf("number: %d, len: %d, cap: %d; \n", i, len(testing), cap(testing))
  6. }
  7. }
  8. // number: 0, len: 5, cap: 5;
  9. // number: 1, len: 6, cap: 10;
  10. // number: 2, len: 7, cap: 10;
  11. // number: 3, len: 8, cap: 10;
  12. // number: 4, len: 9, cap: 10;
  13. // number: 5, len: 10, cap: 10;
  14. // number: 6, len: 11, cap: 12;
  15. // number: 7, len: 12, cap: 12;
  16. // number: 8, len: 13, cap: 14;
  17. // number: 9, len: 14, cap: 14;
  18. // number: 10, len: 15, cap: 16;
  19. // number: 11, len: 16, cap: 16;
  20. // number: 12, len: 17, cap: 18;
  21. // number: 13, len: 18, cap: 18;
  22. // number: 14, len: 19, cap: 20;
  23. // number: 15, len: 20, cap: 20;
  24. // number: 16, len: 21, cap: 22;
  25. // number: 17, len: 22, cap: 22;
  26. // number: 18, len: 23, cap: 24;
  27. // number: 19, len: 24, cap: 24;
  28. // number: 20, len: 25, cap: 26;
  29. // number: 21, len: 26, cap: 26;
  30. // number: 22, len: 27, cap: 28;
  31. // number: 23, len: 28, cap: 28;
  32. // number: 24, len: 29, cap: 30;
  33. // number: 25, len: 30, cap: 30;
  34. // number: 26, len: 31, cap: 32;
  35. // number: 27, len: 32, cap: 32;
  36. // number: 28, len: 33, cap: 36;
  37. // number: 29, len: 34, cap: 36;
  38. // number: 30, len: 35, cap: 36;
  39. // number: 31, len: 36, cap: 36;
  40. // number: 32, len: 37, cap: 40;
  41. // number: 33, len: 38, cap: 40;
  42. // number: 34, len: 39, cap: 40;
  43. // number: 35, len: 40, cap: 40;
  44. // number: 36, len: 41, cap: 44;
  45. // number: 37, len: 42, cap: 44;
  46. // number: 38, len: 43, cap: 44;
  47. // number: 39, len: 44, cap: 44;