- Feature Name:
capture_disjoint_fields
- Start Date: 2017-11-28
- RFC PR: rust-lang/rfcs#2229
- Rust Issue: rust-lang/rust#53488
Summary
This RFC proposes that closure capturing should be minimal rather than maximal. Conceptually, existing rules regarding borrowing and moving disjoint fields should be applied to capturing. If implemented, the following code examples would become valid:
let a = &mut foo.a;
|| &mut foo.b; // Error! cannot borrow `foo`
somefunc(a);
let a = &mut foo.a;
move || foo.b; // Error! cannot move `foo`
somefunc(a);
Note that some discussion of this has already taken place:
- rust-lang/rust#19004
- Rust internals forum
Motivation
In the rust language today, any variables named within a closure will be fully captured. This was simple to implement but is inconsistent with the rest of the language because rust normally allows simultaneous borrowing of disjoint fields. Remembering this exception adds to the mental burden of the programmer and makes the rules of borrowing and ownership harder to learn.
The following is allowed; why should closures be treated differently?
let _a = &mut foo.a;
loop { &mut foo.b; } // ok!
This is a particularly annoying problem because closures often need to borrow
data from self
:
pub fn update(&mut self) {
// cannot borrow `self` as immutable because `self.list` is also borrowed as mutable
self.list.retain(|i| self.filter.allowed(i));
}
Guide-level explanation
Rust understands structs sufficiently to know that it’s possible to borrow disjoint fields of a struct simultaneously. Structs can also be destructed and moved piece-by-piece. This functionality should be available anywhere, including from within closures:
struct OneOf {
text: String,
of: Vec<String>,
}
impl OneOf {
pub fn matches(self) -> bool {
// Ok! destructure self
self.of.into_iter().any(|s| s == self.text)
}
pub fn filter(&mut self) {
// Ok! mutate and inspect self
self.of.retain(|s| s != &self.text)
}
}
Rust will prevent dangerous double usage:
struct FirstDuplicated(Vec<String>)
impl FirstDuplicated {
pub fn first_count(self) -> usize {
// Error! can't destructure and mutate same data
self.0.into_iter()
.filter(|s| &s == &self.0[0])
.count()
}
pub fn remove_first(&mut self) {
// Error! can't mutate and inspect same data
self.0.retain(|s| s != &self.0[0])
}
}
Reference-level explanation
This RFC does not propose any changes to the borrow checker. Instead, the MIR
generation for closures should be altered to produce the minimal capture.
Additionally, a hidden repr
for closures might be added, which could reduce
closure size through awareness of the new capture rules (see unresolved).
In a sense, when a closure is lowered to MIR, a list of “capture expressions” is
created, which we will call the “capture set”. Each expression is some part of
the closure body which, in order to capture parts of the enclosing scope, must
be pre-evaluated when the closure is created. The output of the expressions,
which we will call “capture data”, is stored in the anonymous struct which
implements the Fn*
traits. If a binding is used within a closure, at least one
capture expression which borrows or moves that binding’s value must exist in the
capture set.
Currently, lowering creates exactly one capture expression for each used binding, which borrows or moves the value in its entirety. This RFC proposes that lowering should instead create the minimal capture, where each expression is as precise as possible.
This minimal set of capture expressions might be created through a sort of iterative refinement. We would start out capturing all of the local variables. Then, each path would be made more precise by adding additional dereferences and path components depending on which paths are used and how. References to structs would be made more precise by reborrowing fields and owned structs would be made more precise by moving fields.
A capture expression is minimal if it produces a value that is used by the closure in its entirety (e.g. is a primitive, is passed outside the closure, etc.) or if making the expression more precise would require one the following.
- a call to an impure function
- an illegal move (for example, out of a
Drop
type)
When generating a capture expression, we must decide if the output should be
owned or if it can be a reference. In a non-move
closure, a capture expression
will only produce owned data if ownership of that data is required by the body
of the closure. A move
closure will always produce owned data unless the
captured binding does not have ownership.
Note that all functions are considered impure (including to overloaded deref
implementations). And, for the sake of capturing, all indexing is considered
impure. It is possible that overloaded Deref::deref
implementations could be
marked as pure by using a new, marker trait (such as DerefPure
) or attribute
(such as #[deref_transparent]
). However, such a solution should be proposed in
a separate RFC. In the meantime, <Box as Deref>::deref
could be a special case
of a pure function (see unresolved).
Also note that, because capture expressions are all subsets of the closure body, this RFC does not change what is executed. It does change the order/number of executions for some operations, but since these must be pure, order/repetition does not matter. Only changes to lifetimes might be breaking. Specifically, the drop order of uncaptured data can be altered.
We might solve this by considering a struct to be minimal if it contains unused
fields that implement Drop
. This would prevent the drop order of those fields
from changing, but feels strange and non-orthogonal (see unresolved).
Encountering this case at all could trigger a warning, so that this extra rule
could exist temporarily but be removed over the next epoc (see unresolved).
Reference Examples
Below are examples of various closures and their capture sets.
let foo = 10;
|| &mut foo;
&mut foo
(primitive, ownership not required, used in entirety)
let a = &mut foo.a;
|| (&mut foo.b, &mut foo.c);
somefunc(a);
&mut foo.b
(ownership not required, used in entirety)&mut foo.c
(ownership not required, used in entirety)
The borrow checker passes because foo.a
, foo.b
, and foo.c
are disjoint.
let a = &mut foo.a;
move || foo.b;
somefunc(a);
foo.b
(ownership available, used in entirety)
The borrow checker passes because foo.a
and foo.b
are disjoint.
let hello = &foo.hello;
move || foo.drop_world.a;
somefunc(hello);
foo.drop_world
(ownership available, can’t be more precise without moving out ofDrop
)
The borrow checker passes because foo.hello
and foo.drop_world
are disjoint.
|| println!("{}", foo.wrapper_thing.a);
&foo.wrapper_thing
(ownership not required, can’t be more precise because overloadedDeref
onwrapper_thing
is impure)
|| foo.list[0];
foo.list
(ownership required, can’t be more precise because indexing is impure)
let bar = (1, 2); // struct
|| myfunc(bar);
bar
(ownership required, used in entirety)
let foo_again = &mut foo;
|| &mut foo.a;
somefunc(foo_again);
&mut foo.a
(ownership not required, used in entirety)
The borrow checker fails because foo_again
and foo.a
intersect.
let _a = foo.a;
|| foo.a;
foo.a
(ownership required, used in entirety)
The borrow checker fails because foo.a
has already been moved.
let a = &drop_foo.a;
move || drop_foo.b;
somefunc(a);
drop_foo
(ownership available, can’t be more precise without moving out ofDrop
)
The borrow checker fails because drop_foo
cannot be moved while borrowed.
|| &box_foo.a;
&<Box<_> as Deref>::deref(&box_foo).b
(ownership not required,Box::deref
is pure)
move || &box_foo.a;
box_foo
(ownership available, can’t be more precise without moving out ofDrop
)
let foo = &mut a;
let other = &mut foo.other;
move || &mut foo.bar;
somefunc(other);
&mut foo.bar
(ownership not available, borrow can be split)
Drawbacks
This RFC does ruin the intuition that all variables named within a closure are completely captured. I argue that that intuition is not common or necessary enough to justify the extra glue code.
Rationale and alternatives
This proposal is purely ergonomic since there is a complete and common workaround. The existing rules could remain in place and rust users could continue to pre-borrow/move fields. However, this workaround results in significant useless glue code when borrowing many but not all of the fields in a struct. It also produces a larger closure than necessary which could make the difference when inlining.
Unresolved questions
-
How to optimize pointers. Can borrows that all reference parts of the same object be stored as a single pointer? How should this optimization be implemented (e.g. a special
repr
, refinement typing)? -
How to signal that a function is pure. Is this even needed/wanted? Any other places where the language could benefit?
-
Should
Box
be special? -
Drop order can change as a result of this RFC, is this a real stability problem? How should this be resolved?