PSA: Go's defer is not block-scoped
Suppose you have a struct that has some shared state and needs to be concurrent-safe:
type Data struct {
Value int
}
func (d *Data) Increment() {
d.Value += 1
fmt.Printf("d.Value: %d\n", d.Value)
}
func main() {
// Initialize our data.
var d Data
// Spin up some goroutines.
var wg sync.WaitGroup
wg.Go(d.Increment)
wg.Go(d.Increment)
wg.Go(d.Increment)
// You probably already know what I'm trying to get at.
wg.Wait()
// Print the result.
fmt.Printf("Done! Final value: %d", d.Value)
}When you run this multiple times, we get different outputs. This is known as a data race.
d.Value: 1
d.Value: 3
d.Value: 2
Done! Final value: 3d.Value: 1
d.Value: 2
d.Value: 3 # It's in order, but we got lucky this time!
Done! Final value: 3d.Value: 1
d.Value: 3
d.Value: 2
Done! Final value: 3To fix this, we can use a mutex.
You can lock a mutex to make sure that nobody else can work on the same thing. If another goroutine tries to lock the mutex when it is already locked, it will wait until it is unlocked.
We can use this to make sure that our shared Value is incremented in the correct order.
Let’s see what that looks like.
type Data struct {
Value int
Mu sync.Mutex // Use a mutex!
}
func (d *Data) Increment() {
// Hold the lock before accessing the data...
d.Mu.Lock()
// ...And when we're done, release it.
defer d.Mu.Unlock()
// We can safely access the data here without any races!
d.Value += 1
fmt.Printf("d.Value: %d\n", d.Value)
}
func main() {
// Initialize our data.
var d Data
// Spin up some goroutines to increment the value.
var wg sync.WaitGroup
wg.Go(d.Increment)
wg.Go(d.Increment)
wg.Go(d.Increment)
wg.Wait()
// Lock the mutex one last time to access the data.
d.Mu.Lock()
defer d.Mu.Unlock()
fmt.Printf("Done! Final value: %d\n", d.Value)
}If we run this multiple times, we get:
d.Value: 1
d.Value: 2
d.Value: 3
Done! Final value: 3d.Value: 1
d.Value: 2
d.Value: 3
Done! Final value: 3d.Value: 1
d.Value: 2
d.Value: 3
Done! Final value: 3Cool! We’ve made Increment concurrent-safe. Now it correctly increments in the order that Increment is called.
Let’s say we need to do something a bit more complex with Value. Maybe Value needs to be multiplied by 2, and then we need to add 1 to it. Multiplication and addition are obviously very expensive operations, so let’s throw in some other work in between while waiting.
If you come from C++ or Rust like me, you might write it this way:
func (d *Data) DoWork() {
{
// Lock and unlock the mutex before mutating state.
d.Mu.Lock()
defer d.Mu.Unlock()
d.Value *= 2
}
// Don't need to lock here. We're not using `d.Value`.
fmt.Println("Value multiplied by 2. Continuing work...")
{
// Lock and unlock again.
d.Mu.Lock()
defer d.Mu.Unlock()
fmt.Printf("Doing some extra work! %d\n", d.Value)
}
// `d.Increment` already locks the mutex,
// so we can just call it normally here.
d.Increment()
}
func main() {
// ...
// Do the work!
var wg sync.WaitGroup
wg.Go(d.DoWork)
wg.Go(d.DoWork)
wg.Go(d.DoWork)
wg.Wait()
// ...
}Let’s see what that prints.
Value multiplied by 2. Continuing work...
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x47e853?, 0x60?)
/usr/local/go-faketime/src/runtime/sema.go:114 +0x2e
sync.(*WaitGroup).Wait(0xc000010060)
/usr/local/go-faketime/src/sync/waitgroup.go:206 +0x85
main.main()
/tmp/sandbox1439469249/prog.go:57 +0x117
goroutine 7 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
/usr/local/go-faketime/src/runtime/sema.go:95 +0x25
# I'll cut this short here...Uh-oh. All goroutines are asleep? But I’m locking and unlocking just fine. Is DoWork only being called the first time and not running the other two times?
I spent over 3 hours figuring this out so you don’t have to!
It’s actually function-scoped
This is what Go’s tutorial says about the defer keyword:
A defer statement defers the execution of a function until the surrounding function returns.
Let’s remove the defer keyword and move these lines around to see what’s actually happening.
func (d *Data) DoWork() {
{
d.Mu.Lock()
// `defer` moved this to the end of the surrounding function.
d.Value *= 2
}
fmt.Println("Value multiplied by 2. Continuing work...")
{
d.Mu.Lock()
// Same here.
fmt.Printf("Doing some extra work! %d\n", d.Value)
}
d.Increment()
// The unlocks actually happen here!
d.Mu.Unlock()
d.Mu.Unlock()
}We’re locking the mutex once, printing something, and then locking it again!
Because of that, everything waits for the mutex to unlock, which it only actually does at the end of the function.
This is the correct way to write it:
func (d *Data) DoWork() {
d.Mu.Lock()
d.Value *= 2
d.Mu.Unlock() // Unlock it now that we're done.
fmt.Println("Value multiplied by 2. Continuing work...")
d.Mu.Lock()
fmt.Printf("Doing some extra work! %d\n", d.Value)
d.Mu.Unlock() // Unlock here too.
d.Increment()
}Value multiplied by 2. Continuing work...
Doing some extra work! 0
d.Value: 1
Value multiplied by 2. Continuing work...
Doing some extra work! 1
d.Value: 2
Value multiplied by 2. Continuing work...
Doing some extra work! 2
d.Value: 3
Done! Final value: 3Very cool! Everything’s working again.
The issue with this approach is that for sufficiently complex code, you can forget to unlock the mutex. Maybe you’re refactoring and changing where you need the locks. It’s easy to forget to check if the unlocks are in the right place too.
Here’s a workaround:
func (d *Data) DoWork() {
// Wrap it in a function.
func() {
d.Mu.Lock()
defer d.Mu.Unlock() // We can defer again!
d.Value *= 2
}()
fmt.Println("Value multiplied by 2. Continuing work...")
func() {
d.Mu.Lock()
defer d.Mu.Unlock() // Here too.
fmt.Printf("Doing some extra work! %d\n", d.Value)
}()
d.Increment()
}Value multiplied by 2. Continuing work...
Doing some extra work! 0
d.Value: 1
Value multiplied by 2. Continuing work...
Doing some extra work! 1
d.Value: 2
Value multiplied by 2. Continuing work...
Doing some extra work! 2
d.Value: 3
Done! Final value: 3You can wrap that piece of code in its own function, and then call it immediately. This is pretty similar to JavaScript’s IIFEs.
This pattern is useful for when you have big chunks of code, and you absolutely need to make sure your cleanup code runs.
My recommendation is not to do this, though.
If you have to lock a mutex several times in a function, just lock and unlock it wherever you need to. Don’t defer it. And if you really need to defer it, just extract it into a separate, standalone function.
Clarity is the most important thing when writing code. The fewer surprises, the clearer your code! ✻