Rust RFCs - RFC Book - Active RFC List
The “RFC” (request for comments) process is intended to provide a consistent and controlled path for changes to Rust (such as new features) so that all stakeholders can be confident about the direction of the project.
Many changes, including bug fixes and documentation improvements can be implemented and reviewed via the normal GitHub pull request workflow.
Some changes though are “substantial”, and we ask that these be put through a bit of a design process and produce a consensus among the Rust community and the sub-teams.
Table of Contents
- Opening
- Table of Contents
- When you need to follow this process
- Sub-team specific guidelines
- Before creating an RFC
- What the process is
- The RFC life-cycle
- Reviewing RFCs
- Implementing an RFC
- RFC Postponement
- Help this is all too informal!
- License
- Contributions
When you need to follow this process
You need to follow this process if you intend to make “substantial” changes to Rust, Cargo, Crates.io, or the RFC process itself. What constitutes a “substantial” change is evolving based on community norms and varies depending on what part of the ecosystem you are proposing to change, but may include the following.
- Any semantic or syntactic change to the language that is not a bugfix.
- Removing language features, including those that are feature-gated.
- Changes to the interface between the compiler and libraries, including lang items and intrinsics.
- Additions to
std
.
Some changes do not require an RFC:
- Rephrasing, reorganizing, refactoring, or otherwise “changing shape does not change meaning”.
- Additions that strictly improve objective, numerical quality criteria (warning removal, speedup, better platform coverage, more parallelism, trap more errors, etc.)
- Additions only likely to be noticed by other developers-of-rust, invisible to users-of-rust.
If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first.
Sub-team specific guidelines
For more details on when an RFC is required for the following areas, please see the Rust community’s sub-team specific guidelines for:
Before creating an RFC
A hastily-proposed RFC can hurt its chances of acceptance. Low quality proposals, proposals for previously-rejected features, or those that don’t fit into the near-term roadmap, may be quickly rejected, which can be demotivating for the unprepared contributor. Laying some groundwork ahead of the RFC can make the process smoother.
Although there is no single way to prepare for submitting an RFC, it is generally a good idea to pursue feedback from other project developers beforehand, to ascertain that the RFC may be desirable; having a consistent impact on the project requires concerted effort toward consensus-building.
The most common preparations for writing and submitting an RFC include talking the idea over on our official Zulip server, discussing the topic on our developer discussion forum, and occasionally posting “pre-RFCs” on the developer forum. You may file issues on this repo for discussion, but these are not actively looked at by the teams.
As a rule of thumb, receiving encouraging feedback from long-standing project developers, and particularly members of the relevant sub-team is a good indication that the RFC is worth pursuing.
What the process is
In short, to get a major feature added to Rust, one must first get the RFC merged into the RFC repository as a markdown file. At that point the RFC is “active” and may be implemented with the goal of eventual inclusion into Rust.
- Fork the RFC repo RFC repository
- Copy
0000-template.md
totext/0000-my-feature.md
(where “my-feature” is descriptive). Don’t assign an RFC number yet; This is going to be the PR number and we’ll rename the file accordingly if the RFC is accepted. - Fill in the RFC. Put care into the details: RFCs that do not present convincing motivation, demonstrate lack of understanding of the design’s impact, or are disingenuous about the drawbacks or alternatives tend to be poorly-received.
- Submit a pull request. As a pull request the RFC will receive design feedback from the larger community, and the author should be prepared to revise it in response.
- Now that your RFC has an open pull request, use the issue number of the PR
to rename the file: update your
0000-
prefix to that number. Also update the “RFC PR” link at the top of the file. - Each pull request will be labeled with the most relevant sub-team, which will lead to its being triaged by that team in a future meeting and assigned to a member of the subteam.
- Build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don’t receive any comments. Feel free to reach out to the RFC assignee in particular to get help identifying stakeholders and obstacles.
- The sub-team will discuss the RFC pull request, as much as possible in the comment thread of the pull request itself. Offline discussion will be summarized on the pull request comment thread.
- RFCs rarely go through this process unchanged, especially as alternatives and drawbacks are shown. You can make edits, big and small, to the RFC to clarify or change the design, but make changes as new commits to the pull request, and leave a comment on the pull request explaining your changes. Specifically, do not squash or rebase commits after they are visible on the pull request.
- At some point, a member of the subteam will propose a “motion for final
comment period” (FCP), along with a disposition for the RFC (merge, close,
or postpone).
- This step is taken when enough of the tradeoffs have been discussed that the subteam is in a position to make a decision. That does not require consensus amongst all participants in the RFC thread (which is usually impossible). However, the argument supporting the disposition on the RFC needs to have already been clearly articulated, and there should not be a strong consensus against that position outside of the subteam. Subteam members use their best judgment in taking this step, and the FCP itself ensures there is ample time and notification for stakeholders to push back if it is made prematurely.
- For RFCs with lengthy discussion, the motion to FCP is usually preceded by a summary comment trying to lay out the current state of the discussion and major tradeoffs/points of disagreement.
- Before actually entering FCP, all members of the subteam must sign off; this is often the point at which many subteam members first review the RFC in full depth.
- The FCP lasts ten calendar days, so that it is open for at least 5 business days. It is also advertised widely, e.g. in This Week in Rust. This way all stakeholders have a chance to lodge any final objections before a decision is reached.
- In most cases, the FCP period is quiet, and the RFC is either merged or closed. However, sometimes substantial new arguments or ideas are raised, the FCP is canceled, and the RFC goes back into development mode.
The RFC life-cycle
Once an RFC becomes “active” then authors may implement it and submit the feature as a pull request to the Rust repo. Being “active” is not a rubber stamp, and in particular still does not mean the feature will ultimately be merged; it does mean that in principle all the major stakeholders have agreed to the feature and are amenable to merging it.
Furthermore, the fact that a given RFC has been accepted and is “active” implies nothing about what priority is assigned to its implementation, nor does it imply anything about whether a Rust developer has been assigned the task of implementing the feature. While it is not necessary that the author of the RFC also write the implementation, it is by far the most effective way to see an RFC through to completion: authors should not expect that other project developers will take on responsibility for implementing their accepted feature.
Modifications to “active” RFCs can be done in follow-up pull requests. We strive to write each RFC in a manner that it will reflect the final design of the feature; but the nature of the process means that we cannot expect every merged RFC to actually reflect what the end result will be at the time of the next major release.
In general, once accepted, RFCs should not be substantially changed. Only very minor changes should be submitted as amendments. More substantial changes should be new RFCs, with a note added to the original RFC. Exactly what counts as a “very minor change” is up to the sub-team to decide; check Sub-team specific guidelines for more details.
Reviewing RFCs
While the RFC pull request is up, the sub-team may schedule meetings with the author and/or relevant stakeholders to discuss the issues in greater detail, and in some cases the topic may be discussed at a sub-team meeting. In either case a summary from the meeting will be posted back to the RFC pull request.
A sub-team makes final decisions about RFCs after the benefits and drawbacks are well understood. These decisions can be made at any time, but the sub-team will regularly issue decisions. When a decision is made, the RFC pull request will either be merged or closed. In either case, if the reasoning is not clear from the discussion in thread, the sub-team will add a comment describing the rationale for the decision.
Implementing an RFC
Some accepted RFCs represent vital features that need to be implemented right away. Other accepted RFCs can represent features that can wait until some arbitrary developer feels like doing the work. Every accepted RFC has an associated issue tracking its implementation in the Rust repository; thus that associated issue can be assigned a priority via the triage process that the team uses for all issues in the Rust repository.
The author of an RFC is not obligated to implement it. Of course, the RFC author (like any other developer) is welcome to post an implementation for review after the RFC has been accepted.
If you are interested in working on the implementation for an “active” RFC, but cannot determine if someone else is already working on it, feel free to ask (e.g. by leaving a comment on the associated issue).
RFC Postponement
Some RFC pull requests are tagged with the “postponed” label when they are closed (as part of the rejection process). An RFC closed with “postponed” is marked as such because we want neither to think about evaluating the proposal nor about implementing the described feature until some time in the future, and we believe that we can afford to wait until then to do so. Historically, “postponed” was used to postpone features until after 1.0. Postponed pull requests may be re-opened when the time is right. We don’t have any formal process for that, you should ask members of the relevant sub-team.
Usually an RFC pull request marked as “postponed” has already passed an informal first round of evaluation, namely the round of “do we think we would ever possibly consider making this change, as outlined in the RFC pull request, or some semi-obvious variation of it.” (When the answer to the latter question is “no”, then the appropriate response is to close the RFC, not postpone it.)
Help this is all too informal!
The process is intended to be as lightweight as reasonable for the present circumstances. As usual, we are trying to let the process be driven by consensus and community norms, not impose more structure than necessary.
License
This repository is currently in the process of being licensed under either of:
- Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)
at your option. Some parts of the repository are already licensed according to those terms. For more see RFC 2044 and its tracking issue.
Contributions
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
RFC policy - the compiler
Compiler RFCs will be managed by the compiler sub-team, and tagged T-compiler
.
The compiler sub-team will do an initial triage of new PRs within a week of
submission. The result of triage will either be that the PR is assigned to a
member of the sub-team for shepherding, the PR is closed because the sub-team
believe it should be done without an RFC, or closed because the sub-team feel it
should clearly not be done and further discussion is not necessary. We’ll follow
the standard procedure for shepherding, final comment period, etc.
Most compiler decisions that go beyond the scope of a simple PR are done using MCPs, not RFCs. It is therefore likely that you should file an MCP instead of an RFC for your problem.
Changes which need an RFC
- Significant user-facing changes to the compiler with a complex design space, especially if they involve other teams as well (for example, path sanitization).
- Any other change which causes significant backwards incompatible changes to stable behaviour of the compiler, language, or libraries
Changes which don’t need an RFC
- Bug fixes, improved error messages, etc.
- Minor refactoring/tidying up
- Large internal refactorings or redesigns of the compiler (needs an MCP)
- Implementing language features which have an accepted RFC.
- New lints (these fall under the lang team). Lints are best first tried out in clippy and then uplifted later.
- Changing the API presented to syntax extensions or other compiler plugins in non-trivial ways
- Adding, removing, or changing a stable compiler flag (needs an FCP somewhere, like on an MCP or just on a PR)
- Adding unstable API for tools (note that all compiler API is currently unstable)
- Adding, removing, or changing an unstable compiler flag (if the compiler flag is widely used there should be at least some discussion on discuss, or an RFC in some cases)
If in doubt it is probably best to just announce the change you want to make to the compiler subteam on Zulip, and see if anyone feels it needs an RFC.
RFC policy - language design
Pretty much every change to the language needs an RFC. Note that new lints (or major changes to an existing lint) are considered changes to the language.
Language RFCs are managed by the language sub-team, and tagged T-lang
. The
language sub-team will do an initial triage of new PRs within a week of
submission. The result of triage will either be that the PR is assigned to a
member of the sub-team for shepherding, the PR is closed as postponed because
the subteam believe it might be a good idea, but is not currently aligned with
Rust’s priorities, or the PR is closed because the sub-team feel it should
clearly not be done and further discussion is not necessary. In the latter two
cases, the sub-team will give a detailed explanation. We’ll follow the standard
procedure for shepherding, final comment period, etc.
Amendments
Sometimes in the implementation of an RFC, changes are required. In general these don’t require an RFC as long as they are very minor and in the spirit of the accepted RFC (essentially bug fixes). In this case implementers should submit an RFC PR which amends the accepted RFC with the new details. Although the RFC repository is not intended as a reference manual, it is preferred that RFCs do reflect what was actually implemented. Amendment RFCs will go through the same process as regular RFCs, but should be less controversial and thus should move more quickly.
When a change is more dramatic, it is better to create a new RFC. The RFC should be standalone and reference the original, rather than modifying the existing RFC. You should add a comment to the original RFC with referencing the new RFC as part of the PR.
Obviously there is some scope for judgment here. As a guideline, if a change affects more than one part of the RFC (i.e., is a non-local change), affects the applicability of the RFC to its motivating use cases, or there are multiple possible new solutions, then the feature is probably not ‘minor’ and should get a new RFC.
RFC guidelines - libraries sub-team
Motivation
-
RFCs are heavyweight:
- RFCs generally take at minimum 2 weeks from posting to land. In practice it can be more on the order of months for particularly controversial changes.
- RFCs are a lot of effort to write; especially for non-native speakers or for members of the community whose strengths are more technical than literary.
- RFCs may involve pre-RFCs and several rewrites to accommodate feedback.
- RFCs require a dedicated shepherd to herd the community and author towards consensus.
- RFCs require review from a majority of the subteam, as well as an official vote.
- RFCs can’t be downgraded based on their complexity. Full process always applies. Easy RFCs may certainly land faster, though.
- RFCs can be very abstract and hard to grok the consequences of (no implementation).
-
PRs are low overhead but potentially expensive nonetheless:
- Easy PRs can get insta-merged by any rust-lang contributor.
- Harder PRs can be easily escalated. You can ping subject-matter experts for second opinions. Ping the whole team!
- Easier to grok the full consequences. Lots of tests and Crater to save the day.
- PRs can be accepted optimistically with bors, buildbot, and the trains to guard us from major mistakes making it into stable. The size of the nightly community at this point in time can still mean major community breakage regardless of trains, however.
- HOWEVER: Big PRs can be a lot of work to make only to have that work rejected for details that could have been hashed out first.
-
RFCs are only meaningful if a significant and diverse portion of the community actively participates in them. The official teams are not sufficiently diverse to establish meaningful community consensus by agreeing amongst themselves.
-
If there are tons of RFCs – especially trivial ones – people are less likely to engage with them. Official team members are super busy. Domain experts and industry professionals are super busy and have no responsibility to engage in RFCs. Since these are exactly the most important people to get involved in the RFC process, it is important that we be maximally friendly towards their needs.
Is an RFC required?
The overarching philosophy is: do whatever is easiest. If an RFC
would be less work than an implementation, that’s a good sign that an RFC is
necessary. That said, if you anticipate controversy, you might want to short-circuit
straight to an RFC. For instance new APIs almost certainly merit an RFC. Especially
as std
has become more conservative in favour of the much more agile cargoverse.
- Submit a PR if the change is a:
- Bugfix
- Docfix
- Obvious API hole patch, such as adding an API from one type to a symmetric type.
e.g.
Vec<T> -> Box<[T]>
clearly motivates addingString -> Box<str>
- Minor tweak to an unstable API (renaming, generalizing)
- Implementing an “obvious” trait like Clone/Debug/etc
- Submit an RFC if the change is a:
- New API
- Semantic Change to a stable API
- Generalization of a stable API (e.g. how we added Pattern or Borrow)
- Deprecation of a stable API
- Nontrivial trait impl (because all trait impls are insta-stable)
- Do the easier thing if uncertain. (choosing a path is not final)
Non-RFC process
-
A (non-RFC) PR is likely to be closed if clearly not acceptable:
- Disproportionate breaking change (small inference breakage may be acceptable)
- Unsound
- Doesn’t fit our general design philosophy around the problem
- Better as a crate
- Too marginal for std
- Significant implementation problems
-
A PR may also be closed because an RFC is appropriate.
-
A (non-RFC) PR may be merged as unstable. In this case, the feature should have a fresh feature gate and an associated tracking issue for stabilisation. Note that trait impls and docs are insta-stable and thus have no tracking issue. This may imply requiring a higher level of scrutiny for such changes.
However, an accepted RFC is not a rubber-stamp for merging an implementation PR. Nor must an implementation PR perfectly match the RFC text. Implementation details may merit deviations, though obviously they should be justified. The RFC may be amended if deviations are substantial, but are not generally necessary. RFCs should favour immutability. The RFC + Issue + PR should form a total explanation of the current implementation.
-
Once something has been merged as unstable, a shepherd should be assigned to promote and obtain feedback on the design.
-
Every time a release cycle ends, the libs teams assesses the current unstable APIs and selects some number of them for potential stabilization during the next cycle. These are announced for FCP at the beginning of the cycle, and (possibly) stabilized just before the beta is cut.
-
After the final comment period, an API should ideally take one of two paths:
- Stabilize if the change is desired, and consensus is reached
- Deprecate is the change is undesired, and consensus is reached
- Extend the FCP is the change cannot meet consensus
- If consensus still can’t be reached, consider requiring a new RFC or just deprecating as “too controversial for std”.
-
If any problems are found with a newly stabilized API during its beta period, strongly favour reverting stability in order to prevent stabilizing a bad API. Due to the speed of the trains, this is not a serious delay (~2-3 months if it’s not a major problem).
- Start Date: 2014-03-11
- RFC PR: rust-lang/rfcs#1
- Rust Issue: rust-lang/rust#8122
Summary
This is an RFC to make all struct fields private by default. This includes both tuple structs and structural structs.
Motivation
Reasons for default private visibility
-
Visibility is often how soundness is achieved for many types in rust. These types are normally wrapping unsafe behavior of an FFI type or some other rust-specific behavior under the hood (such as the standard
Vec
type). Requiring these types to opt-in to being sound is unfortunate. -
Forcing tuple struct fields to have non-overridable public visibility greatly reduces the utility of such types. Tuple structs cannot be used to create abstraction barriers as they can always be easily destructed.
-
Private-by-default is more consistent with the rest of the Rust language. All other aspects of privacy are private-by-default except for enum variants. Enum variants, however, are a special case in that they are inserted into the parent namespace, and hence naturally inherit privacy.
-
Public fields of a
struct
must be considered as part of the API of the type. This means that the exact definition of all structs is by default the API of the type. Structs must opt-out of this behavior if thepriv
keyword is required. By requiring thepub
keyword, structs must opt-in to exposing more surface area to their API.
Reasons for inherited visibility (today’s design)
- Public definitions like
pub struct Point { x: int, y: int }
are concise and easy to read. - Private definitions certainly want private fields (to hide implementation details).
Detailed design
Currently, rustc has two policies for dealing with the privacy of struct fields:
- Tuple structs have public fields by default (including “newtype structs”)
- Fields of structural structs (
struct Foo { ... }
) inherit the same privacy of the enclosing struct.
This RFC is a proposal to unify the privacy of struct fields with the rest of the language by making them private by default. This means that both tuple variants and structural variants of structs would have private fields by default. For example, the program below is accepted today, but would be rejected with this RFC.
mod inner {
pub struct Foo(u64);
pub struct Bar { field: u64 }
}
fn main() {
inner::Foo(10);
inner::Bar { field: 10 };
}
Refinements to structural structs
Public fields are quite a useful feature of the language, so syntax is required
to opt out of the private-by-default semantics. Structural structs already allow
visibility qualifiers on fields, and the pub
qualifier would make the field
public instead of private.
Additionally, the priv
visibility will no longer be allowed to modify struct
fields. Similarly to how a priv fn
is a compiler error, a priv
field will
become a compiler error.
Refinements on tuple structs
As with their structural cousins, it’s useful to have tuple structs with public fields. This RFC will modify the tuple struct grammar to:
tuple_struct := 'struct' ident '(' fields ')' ';'
fields := field | field ',' fields
field := type | visibility type
For example, these definitions will be added to the language:
// a "newtype wrapper" struct with a private field
struct Foo(u64);
// a "newtype wrapper" struct with a public field
struct Bar(pub u64);
// a tuple struct with many fields, only the first and last of which are public
struct Baz(pub u64, u32, f32, pub int);
Public fields on tuple structs will maintain the semantics that they currently have today. The structs can be constructed, destructed, and participate in pattern matches.
Private fields on tuple structs will prevent the following behaviors:
- Private fields cannot be bound in patterns (both in irrefutable and refutable
contexts, i.e.
let
andmatch
statements). - Private fields cannot be specified outside of the defining module when constructing a tuple struct.
These semantics are intended to closely mirror the behavior of private fields for structural structs.
Statistics gathered
A brief survey was performed over the entire mozilla/rust
repository to gather
these statistics. While not representative of all projects, this repository
should give a good indication of what most structs look like in the real world.
The repository has both libraries (libstd
) as well as libraries which don’t
care much about privacy (librustc
).
These numbers tally up all structs from all locations, and only take into account structural structs, not tuple structs.
Inherited privacy | Private-by-default | |
---|---|---|
Private fields | 1418 | 1852 |
Public fields | 2036 | 1602 |
All-private structs | 551 (52.23%) | 671 (63.60%) |
All-public structs | 468 (44.36%) | 352 (33.36%) |
Mixed privacy structs | 36 ( 3.41%) | 32 ( 3.03%) |
The numbers clearly show that the predominant pattern is to have all-private
structs, and that there are many public fields today which can be private (and
perhaps should!). Additionally, there is on the order of 1418 instances of the
word priv
today, when in theory there should be around 1852
. With this RFC,
there would need to be 1602
instances of the word pub
. A very large portion
of structs requiring pub
fields are FFI structs defined in the libc
module.
Impact on enums
This RFC does not impact enum variants in any way. All enum variants will continue to inherit privacy from the outer enum type. This includes both the fields of tuple variants as well as fields of struct variants in enums.
Alternatives
The main alternative to this design is what is currently implemented today, where fields inherit the privacy of the outer structure. The pros and cons of this strategy are discussed above.
Unresolved questions
As the above statistics show, almost all structures are either all public or all
private. This RFC provides an easy method to make struct fields all private, but
it explicitly does not provide a method to make struct fields all public. The
statistics show that pub
will be written less often than priv
is today, and
it’s always possible to add a method to specify a struct as all-public in the
future in a backwards-compatible fashion.
That being said, it’s an open question whether syntax for an “all public struct” is necessary at this time.
- Start Date: 2014-03-11
- RFC PR: rust-lang/rfcs#2, rust-lang/rfcs#6
- Rust Issue: N/A
Summary
The “RFC” (request for comments) process is intended to provide a consistent and controlled path for new features to enter the language and standard libraries, so that all stakeholders can be confident about the direction the language is evolving in.
Motivation
The freewheeling way that we add new features to Rust has been good for early development, but for Rust to become a mature platform we need to develop some more self-discipline when it comes to changing the system. This is a proposal for a more principled RFC process to make it a more integral part of the overall development process, and one that is followed consistently to introduce features to Rust.
Detailed design
Many changes, including bug fixes and documentation improvements can be implemented and reviewed via the normal GitHub pull request workflow.
Some changes though are “substantial”, and we ask that these be put through a bit of a design process and produce a consensus among the Rust community and the core team.
When you need to follow this process
You need to follow this process if you intend to make “substantial” changes to the Rust distribution. What constitutes a “substantial” change is evolving based on community norms, but may include the following.
- Any semantic or syntactic change to the language that is not a bugfix.
- Removing language features, including those that are feature-gated.
- Changes to the interface between the compiler and libraries, including lang items and intrinsics.
- Additions to
std
Some changes do not require an RFC:
- Rephrasing, reorganizing, refactoring, or otherwise “changing shape does not change meaning”.
- Additions that strictly improve objective, numerical quality criteria (warning removal, speedup, better platform coverage, more parallelism, trap more errors, etc.)
- Additions only likely to be noticed by other developers-of-rust, invisible to users-of-rust.
If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first.
What the process is
In short, to get a major feature added to Rust, one must first get the RFC merged into the RFC repo as a markdown file. At that point the RFC is ‘active’ and may be implemented with the goal of eventual inclusion into Rust.
- Fork the RFC repo https://github.com/rust-lang/rfcs
- Copy
0000-template.md
totext/0000-my-feature.md
(where ‘my-feature’ is descriptive. don’t assign an RFC number yet). - Fill in the RFC
- Submit a pull request. The pull request is the time to get review of the design from the larger community.
- Build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don’t receive any comments.
Eventually, somebody on the core team will either accept the RFC by merging the pull request, at which point the RFC is ‘active’, or reject it by closing the pull request.
Whomever merges the RFC should do the following:
- Assign an id, using the PR number of the RFC pull request. (If the RFC has multiple pull requests associated with it, choose one PR number, preferably the minimal one.)
- Add the file in the
text/
directory. - Create a corresponding issue on Rust repo
- Fill in the remaining metadata in the RFC header, including links for the original pull request(s) and the newly created Rust issue.
- Add an entry in the Active RFC List of the root
README.md
. - Commit everything.
Once an RFC becomes active then authors may implement it and submit the feature as a pull request to the Rust repo. An ‘active’ is not a rubber stamp, and in particular still does not mean the feature will ultimately be merged; it does mean that in principle all the major stakeholders have agreed to the feature and are amenable to merging it.
Modifications to active RFC’s can be done in followup PR’s. An RFC that makes it through the entire process to implementation is considered ‘complete’ and is removed from the Active RFC List; an RFC that fails after becoming active is ‘inactive’ and moves to the ‘inactive’ folder.
Alternatives
Retain the current informal RFC process. The newly proposed RFC process is designed to improve over the informal process in the following ways:
- Discourage unactionable or vague RFCs
- Ensure that all serious RFCs are considered equally
- Give confidence to those with a stake in Rust’s development that they understand why new features are being merged
As an alternative, we could adopt an even stricter RFC process than the one proposed here. If desired, we should likely look to Python’s PEP process for inspiration.
Unresolved questions
- Does this RFC strike a favorable balance between formality and agility?
- Does this RFC successfully address the aforementioned issues with the current informal RFC process?
- Should we retain rejected RFCs in the archive?
- Start Date: 2012-03-20
- RFC PR: rust-lang/rfcs#3
- Rust Issue: rust-lang/rust#14373
Summary
Rust currently has an attribute usage lint but it does not work particularly well. This RFC proposes a new implementation strategy that should make it significantly more useful.
Motivation
The current implementation has two major issues:
- There are very limited warnings for valid attributes that end up in the wrong place. Something like this will be silently ignored:
#[deriving(Clone)]; // Shouldn't have put a ; here
struct Foo;
#[ignore(attribute-usage)] // Should have used #[allow(attribute-usage)] instead!
mod bar {
//...
}
ItemDecorators
can now be defined outside of the compiler, and there’s no way to tag them and associated attributes as valid. Something like this requires an#[allow(attribute-usage)]
:
#[feature(phase)];
#[phase(syntax, link)]
extern crate some_orm;
#[ormify]
pub struct Foo {
#[column(foo_)]
#[primary_key]
foo: int
}
Detailed design
The current implementation is implemented as a simple fold over the AST, comparing attributes against a whitelist. Crate-level attributes use a separate whitelist, but no other distinctions are made.
This RFC would change the implementation to actually track which attributes are
used during the compilation process. syntax::ast::Attribute_
would be
modified to add an ID field:
pub struct AttrId(uint);
pub struct Attribute_ {
id: AttrId,
style: AttrStyle,
value: @MetaItem,
is_sugared_doc: bool,
}
syntax::ast::parse::ParseSess
will generate new AttrId
s on demand. I
believe that attributes will only be created during parsing and expansion, and
the ParseSess
is accessible in both.
The AttrId
s will be used to create a side table of used attributes. This will
most likely be a thread local to make it easily accessible during all stages of
compilation by calling a function in syntax::attr
:
fn mark_used(attr: &Attribute) { }
The attribute-usage
lint would run at the end of compilation and warn on all
attributes whose ID does not appear in the side table.
One interesting edge case is attributes like doc
that are used, but not in
the normal compilation process. There could either be a separate fold pass to
mark all doc
attributes as used or doc
could simply be whitelisted in the
attribute-usage
lint.
Attributes in code that has been eliminated with #[cfg()]
will not be linted,
but I feel that this is consistent with the way #[cfg()]
works in general
(e.g. the code won’t be type-checked either).
Alternatives
An alternative would be to rewrite rustc::middle::lint
to robustly check
that attributes are used where they’re supposed to be. This will be fairly
complex and be prone to failure if/when more nodes are added to the AST. This
also doesn’t solve motivation #2, which would require externally loaded lint
support.
Unresolved questions
- This implementation doesn’t allow for a distinction between “unused” and
“unknown” attributes. The
#[phase(syntax)]
crate loading infrastructure could be extended to pull a list of attributes from crates to use in the lint pass, but I’m not sure if the extra complexity is worth it. - The side table could be threaded through all of the compilation stages that need to use it instead of being a thread local. This would probably require significantly more work than the thread local approach, however. The thread local approach should not negatively impact any future parallelization work as each thread can keep its own side table, which can be merged into one for the lint pass.
- Start Date: 2014-03-14
- RFC PR: rust-lang/rfcs#8
- Rust Issue:
Note: this RFC was never implemented and has been retired. The design may still be useful in the future, but before implementing we would prefer to revisit it so as to be sure it is up to date.
Summary
The way our intrinsics work forces them to be wrapped in order to behave like normal functions. As a result, rustc is forced to inline a great number of tiny intrinsic wrappers, which is bad for both compile-time performance and runtime performance without optimizations. This proposal changes the way intrinsics are surfaced in the language so that they behave the same as normal Rust functions by removing the “rust-intrinsic” foreign ABI and reusing the “Rust” ABI.
Motivation
A number of commonly-used intrinsics, including transmute
, forget
,
init
, uninit
, and move_val_init
, are accessed through wrappers
whose only purpose is to present the intrinsics as normal functions.
As a result, rustc is forced to inline a great number of tiny
intrinsic wrappers, which is bad for both compile-time performance and
runtime performance without optimizations.
Intrinsics have a differently-named ABI from Rust functions (“rust-intrinsic” vs. “Rust”) though the actual ABI implementation is identical. As a result one can’t take the value of an intrinsic as a function:
// error: the type of transmute is `extern "rust-intrinsic" fn ...`
let transmute: fn(int) -> uint = intrinsics::transmute;
This incongruity means that we can’t just expose the intrinsics directly as part of the public API.
Detailed design
extern "Rust" fn
is already equivalent to fn
, so if intrinsics
have the “Rust” ABI then the problem is solved.
Under this scheme intrinsics will be declared as extern "Rust"
functions
and identified as intrinsics with the #[lang = "..."]
attribute:
extern "Rust" {
#[lang = "transmute"]
fn transmute<T, U>(T) -> U;
}
The compiler will type check and translate intrinsics the same as today. Additionally, when trans sees a “Rust” extern tagged as an intrinsic it will not emit a function declaration to LLVM bitcode.
Because intrinsics will be lang items, they can no longer be redeclared arbitrary number of times. This will require a small amount of existing library code to be refactored, and all intrinsics to be exposed through public abstractions.
Currently, “Rust” foreign functions may not be generic; this change will require a special case that allows intrinsics to be generic.
Alternatives
-
Instead of making intrinsics lang items we could create a slightly different mechanism, like an
#[intrinsic]
attribute, that would continue letting intrinsics to be redeclared. -
While using lang items to identify intrinsics, intrinsic lang items could be allowed to be redeclared.
-
We could also make “rust-intrinsic” coerce or otherwise be the same as “Rust” externs and normal Rust functions.
Unresolved questions
None.
- Start Date: 2014-03-20
- RFC PR: rust-lang/rfcs#16
- Rust Issue: rust-lang/rust#15701
Summary
Allow attributes on more places inside functions, such as statements, blocks and expressions.
Motivation
One sometimes wishes to annotate things inside functions with, for
example, lint #[allow]
s, conditional compilation #[cfg]
s, and even
extra semantic (or otherwise) annotations for external tools.
For the lints, one can currently only activate lints at the level of the function which is possibly larger than one needs, and so may allow other “bad” things to sneak through accidentally. E.g.
#[allow(uppercase_variable)]
let L = List::new(); // lowercase looks like one or capital i
For the conditional compilation, the work-around is duplicating the
whole containing function with a #[cfg]
, or breaking the conditional
code into a its own function. This does mean that any variables need
to be explicitly passed as arguments.
The sort of things one could do with other arbitrary annotations are
#[allowed_unsafe_actions(ffi)]
#[audited="2014-04-22"]
unsafe { ... }
and then have an external tool that checks that that unsafe
block’s
only unsafe actions are FFI, or a tool that lists blocks that have
been changed since the last audit or haven’t been audited ever.
The minimum useful functionality would be supporting attributes on
blocks and let
statements, since these are flexible enough to allow
for relatively precise attribute handling.
Detailed design
Normal attribute syntax on let
statements, blocks and expressions.
fn foo() {
#[attr1]
let x = 1;
#[attr2]
{
// code
}
#[attr3]
unsafe {
// code
}
#[attr4] foo();
let x = #[attr5] 1;
qux(3 + #[attr6] 2);
foo(x, #[attr7] y, z);
}
Attributes bind tighter than any operator, that is #[attr] x op y
is
always parsed as (#[attr] x) op y
.
cfg
It is definitely an error to place a #[cfg]
attribute on a
non-statement expressions, that is, attr1
–attr4
can possibly be
#[cfg(foo)]
, but attr5
–attr7
cannot, since it makes little
sense to strip code down to let x = ;
.
However, like #ifdef
in C/C++, widespread use of #[cfg]
may be an
antipattern that makes code harder to read. This RFC is just adding
the ability for attributes to be placed in specific places, it is not
mandating that #[cfg]
actually be stripped in those places (although
it should be an error if it is ignored).
Inner attributes
Inner attributes can be placed at the top of blocks (and other structure incorporating a block) and apply to that block.
{
#![attr11]
foo()
}
match bar {
#![attr12]
_ => {}
}
// are the same as
#[attr11]
{
foo()
}
#[attr12]
match bar {
_ => {}
}
if
Attributes would be disallowed on if
for now, because the
interaction with if
/else
chains are funky, and can be simulated in
other ways.
#[cfg(not(foo))]
if cond1 {
} else #[cfg(not(bar))] if cond2 {
} else #[cfg(not(baz))] {
}
There is two possible interpretations of such a piece of code,
depending on if one regards the attributes as attaching to the whole
if ... else
chain (“exterior”) or just to the branch on which they
are placed (“interior”).
--cfg foo
: could be either removing the whole chain (exterior) or equivalent toif cond2 {} else {}
(interior).--cfg bar
: could be eitherif cond1 {}
(e) orif cond1 {} else {}
(i)--cfg baz
: equivalent toif cond1 {} else if cond2 {}
(no subtlety).--cfg foo --cfg bar
: could be removing the whole chain (e) or the twoif
branches (leaving only theelse
branch) (i).
(This applies to any attribute that has some sense of scoping, not
just #[cfg]
, e.g. #[allow]
and #[warn]
for lints.)
As such, to avoid confusion, attributes would not be supported on
if
. Alternatives include using blocks:
#[attr] if cond { ... } else ...
// becomes, for an exterior attribute,
#[attr] {
if cond { ... } else ...
}
// and, for an interior attribute,
if cond {
#[attr] { ... }
} else ...
And, if the attributes are meant to be associated with the actual
branching (e.g. a hypothetical #[cold]
attribute that indicates a
branch is unlikely), one can annotate match
arms:
match cond {
#[attr] true => { ... }
#[attr] false => { ... }
}
Drawbacks
This starts mixing attributes with nearly arbitrary code, possibly
dramatically restricting syntactic changes related to them, for
example, there was some consideration for using @
for attributes,
this change may make this impossible (especially if @
gets reused
for something else, e.g. Python is
using it for matrix multiplication). It
may also make it impossible to use #
for other things.
As stated above, allowing #[cfg]
s everywhere can make code harder to
reason about, but (also stated), this RFC is not for making such
#[cfg]
s be obeyed, it just opens the language syntax to possibly
allow it.
Alternatives
These instances could possibly be approximated with macros and helper
functions, but to a low degree degree (e.g. how would one annotate a
general unsafe
block).
Only allowing attributes on “statement expressions” that is, expressions at the top level of a block, this is slightly limiting; but we can expand to support other contexts backwards compatibly in the future.
The if
/else
issue may be able to be resolved by introducing
explicit “interior” and “exterior” attributes on if
: by having
#[attr] if cond { ...
be an exterior attribute (applying to the
whole if
/else
chain) and if cond #[attr] { ...
be an interior
attribute (applying to only the current if
branch). There is no
difference between interior and exterior for an else {
branch, and
so else #[attr] {
is sufficient.
Unresolved questions
Are the complications of allowing attributes on arbitrary expressions worth the benefits?
- Start Date: 2014-09-18
- RFC PR #: rust-lang/rfcs#19, rust-lang/rfcs#127
- Rust Issue #: rust-lang/rust#13231
Note: The Share
trait described in this RFC was later
renamed to Sync
.
Summary
The high-level idea is to add language features that simultaneously achieve three goals:
- move
Send
andShare
out of the language entirely and into the standard library, providing mechanisms for end users to easily implement and use similar “marker” traits of their own devising; - make “normal” Rust types sendable and sharable by default, without the need for explicit opt-in; and,
- continue to require “unsafe” Rust types (those that manipulate unsafe pointers or implement special abstractions) to “opt-in” to sendability and sharability with an unsafe declaration.
These goals are achieved by two changes:
-
Unsafe traits: An unsafe trait is a trait that is unsafe to implement, because it represents some kind of trusted assertion. Note that unsafe traits are perfectly safe to use.
Send
andShare
are examples of unsafe traits: implementing these traits is effectively an assertion that your type is safe for threading. -
Default and negative impls: A default impl is one that applies to all types, except for those types that explicitly opt out. For example, there would be a default impl for
Send
, indicating that all types areSend
“by default”.To counteract a default impl, one uses a negative impl that explicitly opts out for a given type
T
and any type that containsT
. For example, this RFC proposes that unsafe pointers*T
will opt out ofSend
andShare
. This implies that unsafe pointers cannot be sent or shared between threads by default. It also implies that any structs which contain an unsafe pointer cannot be sent. In all examples encountered thus far, the set of negative impls is fixed and can easily be declared along with the trait itself.Safe wrappers like
Arc
,Atomic
, orMutex
can opt to implementSend
andShare
explicitly. This will then make them be considered sendable (or sharable) even though they contain unsafe pointers etc.
Based on these two mechanisms, we can remove the notion of Send
and
Share
as builtin concepts. Instead, these would become unsafe traits
with default impls (defined purely in the library). The library would
explicitly opt out of Send
/Share
for certain types, like unsafe
pointers (*T
) or interior mutability (Unsafe<T>
). Any type,
therefore, which contains an unsafe pointer would be confined (by
default) to a single thread. Safe wrappers around those types, like
Arc
, Atomic
, or Mutex
, can then opt back in by explicitly
implementing Send
(these impls would have to be designed as unsafe).
Motivation
Since proposing opt-in builtin traits, I have become increasingly
concerned about the notion of having Send
and Share
be strictly
opt-in. There are two main reasons for my concern:
- Rust is very close to being a language where computations can be
parallelized by default. Making
Send
, and especiallyShare
, opt-in makes that harder to achieve. - The model followed by
Send
/Share
cannot easily be extended to other traits in the future nor can it be extended by end-users with their own similar traits. It is worrisome that I have come across several use cases already which might require such extension (described below).
To elaborate on those two points: With respect to parallelization: for
the most part, Rust types are threadsafe “by default”. To make
something non-threadsafe, you must employ unsynchronized interior
mutability (e.g., Cell
, RefCell
) or unsynchronized shared ownership
(Rc
). In both cases, there are also synchronized variants available
(Mutex
, Arc
, etc). This implies that we can make APIs to enable
intra-task parallelism and they will work ubiquitously, so long as
people avoid Cell
and Rc
when not needed. Explicit opt-in
threatens that future, however, because fewer types will implement
Share
, even if they are in fact threadsafe.
With respect to extensibility, it is particularly worrisome that if a
library forgets to implement Send
or Share
, downstream clients are
stuck. They cannot, for example, use a newtype wrapper, because it
would be illegal to implement Send
on the newtype. This implies that
all libraries must be vigilant about implementing Send
and Share
(even more so than with other pervasive traits like Eq
or Ord
).
The current plan is to address this via lints and perhaps some
convenient deriving syntax, which may be adequate for Send
and
Share
. But if we wish to add new “classification” traits in the
future, these new traits won’t have been around from the start, and
hence won’t be implemented by all existing code.
Another concern of mine is that end users cannot define classification traits of their own. For example, one might like to define a trait for “tainted” data, and then test to ensure that tainted data doesn’t pass through some generic routine. There is no particular way to do this today.
More examples of classification traits that have come up recently in various discussions:
Snapshot
(neeFreeze
), which defines logical immutability rather than physical immutability.Rc<int>
, for example, would be consideredSnapshot
.Snapshot
could be useful becauseSnapshot+Clone
indicates a type whose value can be safely “preserved” by cloning it.NoManaged
, a type which does not contain managed data. This might be useful for integrating garbage collection with custom allocators which do not wish to serve as potential roots.NoDrop
, a type which does not contain an explicit destructor. This can be used to avoid nasty GC quandries.
All three of these (Snapshot
, NoManaged
, NoDrop
) can be easily
defined using traits with default impls.
A final, somewhat weaker, motivator is aesthetics. Ownership has allowed
us to move threading almost entirely into libraries. The one exception
is that the Send
and Share
types remain built-in. Opt-in traits
makes them less built-in, but still requires custom logic in the
“impl matching” code as well as special safety checks when
Safe
or Share
are implemented.
After the changes I propose, the only traits which would be
specifically understood by the compiler are Copy
and Sized
. I
consider this acceptable, since those two traits are intimately tied
to the core Rust type system, unlike Send
and Share
.
Detailed design
Unsafe traits
Certain traits like Send
and Share
are critical to memory safety.
Nonetheless, it is not feasible to check the thread-safety of all
types that implement Send
and Share
. Therefore, we introduce a
notion of an unsafe trait – this is a trait that is unsafe to
implement, because implementing it carries semantic guarantees that,
if compromised, threaten memory safety in a deep way.
An unsafe trait is declared like so:
unsafe trait Foo { ... }
To implement an unsafe trait, one must mark the impl as unsafe:
unsafe impl Foo for Bar { ... }
Designating an impl as unsafe does not automatically mean that the body of the methods is an unsafe block. Each method in the trait must also be declared as unsafe if it to be considered unsafe.
Unsafe traits are only unsafe to implement. It is always safe to reference an unsafe trait. For example, the following function is safe:
fn foo<T:Send>(x: T) { ... }
It is also safe to opt out of an unsafe trait (as discussed in the next section).
Default and negative impls
We add a notion of a default impl, written:
impl Trait for .. { }
Default impls are subject to various limitations:
- The default impl must appear in the same module as
Trait
(or a submodule). Trait
must not define any methods.
We further add the notion of a negative impl, written:
impl !Trait for Foo { }
Negative impls are only permitted if Trait
has a default impl.
Negative impls are subject to the usual orphan rules, but they are
permitting to be overlapping. This makes sense because negative impls
are not providing an implementation and hence we are not forced to
select between them. For similar reasons, negative impls never need to
be marked unsafe, even if they reference an unsafe trait.
Intuitively, to check whether a trait Foo
that contains a default
impl is implemented for some type T
, we first check for explicit
(positive) impls that apply to T
. If any are found, then T
implements Foo
. Otherwise, we check for negative impls. If any are
found, then T
does not implement Foo
. If neither positive nor
negative impls were found, we proceed to check the component types of
T
(i.e., the types of a struct’s fields) to determine whether all of
them implement Foo
. If so, then Foo
is considered implemented by
T
.
Oe non-obvious part of the procedure is that, as we recursively
examine the component types of T
, we add to our list of assumptions
that T
implements Foo
. This allows recursive types like
struct List<T> { data: T, next: Option<List<T>> }
to be checked successfully. Otherwise, we would recursive infinitely.
(This procedure is directly analogous to what the existing
TypeContents
code does.)
Note that there exist types that expand to an infinite tree of types. Such types cannot be successfully checked with a recursive impl; they will simply overflow the builtin depth checking. However, such types also break code generation under monomorphization (we cannot create a finite set of LLVM types that correspond to them) and are in general not supported. Here is an example of such a type:
struct Foo<A> {
data: Option<Foo<Vec<A>>>
}
The difference between Foo
and List
above is that Foo<A>
references Foo<Vec<A>>
, which will then in turn reference
Foo<Vec<Vec<A>>>
and so on.
Modeling Send and Share using default traits
The Send
and Share
traits will be modeled entirely in the library
as follows. First, we declare the two traits as follows:
unsafe trait Send { }
unsafe impl Send for .. { }
unsafe trait Share { }
unsafe impl Share for .. { }
Both traits are declared as unsafe because declaring that a type if
Send
and Share
has ramifications for memory safety (and data-race
freedom) that the compiler cannot, itself, check.
Next, we will add opt out impls of Send
and Share
for the
various unsafe types:
impl<T> !Send for *T { }
impl<T> !Share for *T { }
impl<T> !Send for *mut T { }
impl<T> !Share for *mut T { }
impl<T> !Share for Unsafe<T> { }
Note that it is not necessary to write unsafe to opt out of an unsafe trait, as that is the default state.
Finally, we will add opt in impls of Send
and Share
for the
various safe wrapper types as needed. Here I give one example, which
is Mutex
. Mutex
is interesting because it has the property that it
converts a type T
from being Sendable
to something Sharable
:
unsafe impl<T:Send> Send for Mutex<T> { }
unsafe impl<T:Send> Share for Mutex<T> { }
The Copy
and Sized
traits
The final two builtin traits are Copy
and Share
. This RFC does not
propose any changes to those two traits but rather relies on the
specification from the original opt-in RFC.
Controlling copy vs move with the Copy
trait
The Copy
trait is “opt-in” for user-declared structs and enums. A
struct or enum type is considered to implement the Copy
trait only
if it implements the Copy
trait. This means that structs and enums
would move by default unless their type is explicitly declared to be
Copy
. So, for example, the following code would be in error:
struct Point { x: int, y: int }
...
let p = Point { x: 1, y: 2 };
let q = p; // moves p
print(p.x); // ERROR
To allow that example, one would have to impl Copy
for Point
:
struct Point { x: int, y: int }
impl Copy for Point { }
...
let p = Point { x: 1, y: 2 };
let q = p; // copies p, because Point is Pod
print(p.x); // OK
Effectively, there is a three step ladder for types:
- If you do nothing, your type is linear, meaning that it moves from place to place and can never be copied in any way. (We need a better name for that.)
- If you implement
Clone
, your type is cloneable, meaning that it moves from place to place, but it can be explicitly cloned. This is suitable for cases where copying is expensive. - If you implement
Copy
, your type is copyable, meaning that it is just copied by default without the need for an explicit clone. This is suitable for small bits of data like ints or points.
What is nice about this change is that when a type is defined, the user makes an explicit choice between these three options.
Determining whether a type is Sized
Per the DST specification, the array types [T]
and object types like
Trait
are unsized, as are any structs that embed one of those
types. The Sized
trait can never be explicitly implemented and
membership in the trait is always automatically determined.
Matching and coherence for the builtin types Copy
and Sized
In general, determining whether a type implements a builtin trait can follow the existing trait matching algorithm, but it will have to be somewhat specialized. The problem is that we are somewhat limited in the kinds of impls that we can write, so some of the implementations we would want must be “hard-coded”.
Specifically we are limited around tuples, fixed-length array types, proc types, closure types, and trait types:
- Fixed-length arrays: A fixed-length array
[T, ..n]
isCopy
ifT
isCopy
. It is alwaysSized
asT
is required to beSized
. - Tuples: A tuple
(T_0, ..., T_n)
isCopy/Sized
depending if, for alli
,T_i
isCopy/Sized
. - Trait objects (including procs and closures): A trait object type
Trait:K
(assuming DST here ;) is neverCopy
norSized
.
We cannot currently express the above conditions using impls. We may at some point in the future grow the ability to express some of them. For now, though, these “impls” will be hardcoded into the algorithm as if they were written in libstd.
Per the usual coherence rules, since we will have the above impls in
libstd
, and we will have impls for types like tuples and
fixed-length arrays baked in, the only impls that end users are
permitted to write are impls for struct and enum types that they
define themselves. Although this rule is in the general spirit of the
coherence checks, it will have to be written specially.
Design discussion
Why unsafe traits
Without unsafe traits, it would be possible to
create data races without using the unsafe
keyword:
struct MyStruct { foo: Cell<int> }
impl Share for MyStruct { }
Balancing abstraction, safety, and convenience.
In general, the existence of default traits is anti-abstraction, in the sense that it exposes implementation details a library might prefer to hide. Specifically, adding new private fields can cause your types to become non-sendable or non-sharable, which may break downstream clients without your knowing. This is a known challenge with parallelism: knowing whether it is safe to parallelize relies on implementation details we have traditionally tried to keep secret from clients (often it is said that parallelism is “anti-modular” or “anti-compositional” for this reason).
I think this risk must be weighed against the limitations of requiring total opt in. Requiring total opt in not only means that some types will accidentally fail to implement send or share when they could, but it also means that libraries which wish to employ marker traits cannot be composed with other libraries that are not aware of those marker traits. In effect, opt-in is anti-modular in its own way.
To be more specific, imagine that library A wishes to define a
Untainted
trait, and it specifically opts out of Untainted
for
some base set of types. It then wishes to have routines that only
operate on Untainted
data. Now imagine that there is some other
library B that defines a nifty replacement for Vector
,
NiftyVector
. Finally, some library C wishes to use a
NiftyVector<uint>
, which should not be considered tainted, because
it doesn’t reference any tainted strings. However, NiftyVector<uint>
does not implement Untainted
(nor can it, without either library A
or library B knowing about one another). Similar problems arise for any
trait, of course, due to our coherence rules, but often they can be
overcome with new types. Not so with Send
and Share
.
Other use cases
Part of the design involves making space for other use cases. I’d like to sketch out how some of those use cases can be implemented briefly. This is not included in the Detailed design section of the RFC because these traits generally concern other features and would be added under RFCs of their own.
Isolating snapshot types. It is useful to be able to identify
types which, when cloned, result in a logical snapshot. That is, a
value which can never be mutated. Note that there may in fact be
mutation under the covers, but this mutation is not visible to the
user. An example of such a type is Rc<T>
– although the ref count
on the Rc
may change, the user has no direct access and so Rc<T>
is still logically snapshotable. However, not all Rc
instances are
snapshottable – in particular, something like Rc<Cell<int>>
is not.
trait Snapshot { }
impl Snapshot for .. { }
// In general, anything that can reach interior mutability is not
// snapshotable.
impl<T> !Snapshot for Unsafe<T> { }
// But it's ok for Rc<T>.
impl<T:Snapshot> Snapshot for Rc<T> { }
Note that these definitions could all occur in a library. That is, the
Rc
type itself doesn’t need to know about the Snapshot
trait.
Preventing access to managed data. As part of the GC design, we
expect it will be useful to write specialized allocators or smart
pointers that explicitly do not support tracing, so as to avoid any
kind of GC overhead. The general idea is that there should be a bound,
let’s call it NoManaged
, that indicates that a type cannot reach
managed data and hence does not need to be part of the GC’s root
set. This trait could be implemented as follows:
unsafe trait NoManaged { }
unsafe impl NoManaged for .. { }
impl<T> !NoManaged for Gc<T> { }
Preventing access to destructors. It is generally recognized that
allowing destructors to escape into managed data – frequently
referred to as finalizers – is a bad idea. Therefore, we would
generally like to ensure that anything is placed into a managed box
does not implement the drop trait. Instead, we would prefer to regular
the use of drop through a guardian-like API, which basically means
that destructors are not asynchronously executed by the GC, as they
would be in Java, but rather enqueued for the mutator thread to run
synchronously at its leisure. In order to handle this, though, we
presumably need some sort of guardian wrapper types that can take a
value which has a destructor and allow it to be embedded within
managed data. We can summarize this in a trait GcSafe
as follows:
unsafe trait GcSafe { }
unsafe impl GcSafe for .. { }
// By default, anything which has drop trait is not GcSafe.
impl<T:Drop> !GcSafe for T { }
// But guardians are, even if `T` has drop.
impl<T> GcSafe for Guardian<T> { }
Why are Copy
and Sized
different?
The Copy
and Sized
traits remain builtin to the compiler. This
makes sense because they are intimately tied to analyses the compiler
performs. For example, the running of destructors and tracking of
moves requires knowing which types are Copy
. Similarly, the
allocation of stack frames need to know whether types are fully
Sized
. In contrast, sendability and sharability has been fully
exported to libraries at this point.
In addition, opting in to Copy
makes sense for several reasons:
- Experience has shown that “data-like structs”, for which
Copy
is most appropriate, are a very small percentage of the total. - Changing a public API from being copyable to being only movable has
a outsized impact on users of the API. It is common however that as
APIs evolve they will come to require owned data (like a
Vec
), even if they do not initially, and hence will change from being copyable to only movable. Opting in toCopy
is a way of saying that you never foresee this coming to pass. - Often it is useful to create linear “tokens” that do not themselves have data but represent permissions. This can be done today using markers but it is awkward. It becomes much more natural under this proposal.
Drawbacks
API stability. The main drawback of this approach over the
existing opt-in approach seems to be that a type may be “accidentally”
sendable or sharable. I discuss this above under the heading of
“balancing abstraction, safety, and convenience”. One point I would
like to add here, as it specifically pertains to API stability, is
that a library may, if they choose, opt out of Send
and Share
pre-emptively, in order to “reserve the right” to add non-sendable
things in the future.
Alternatives
-
The existing opt-in design is of course an alternative.
-
We could also simply add the notion of
unsafe
traits and not default impls and then allow types to unsafely implementSend
orShare
, bypassing the normal safety guidelines. This gives an escape valve for a downstream client to assert that something is sendable which was not declared as sendable. However, such a solution is deeply unsatisfactory, because it rests on the downstream client making an assertion about the implementation of the library it uses. If that library should be updated, the client’s assumptions could be invalidated, but no compilation errors will result (the impl was already declared as unsafe, after all).
Phasing
Many of the mechanisms described in this RFC are not needed
immediately. Therefore, we would like to implement a minimal
“forwards compatible” set of changes now and then leave the remaining
work for after the 1.0 release. The builtin rules that the compiler
currently implements for send and share are quite close to what is
proposed in this RFC. The major change is that unsafe pointers and the
UnsafeCell
type are currently considered sendable.
Therefore, to be forwards compatible in the short term, we can use the
same hybrid of builtin and explicit impls for Send
and Share
that
we use for Copy
, with the rule that unsafe pointers and UnsafeCell
are not considered sendable. We must also implement the unsafe trait
and unsafe impl
concept.
What this means in practice is that using *const T
, *mut T
, and
UnsafeCell
will make a type T
non-sendable and non-sharable, and
T
must then explicitly implement Send
or Share
.
Unresolved questions
- The terminology of “unsafe trait” seems somewhat misleading, since
it seems to suggest that “using” the trait is unsafe, rather than
implementing it. One suggestion for an alternate keyword was
trusted trait
, which might dovetail with the use oftrusted
to specify a trusted block of code. If we did usetrusted trait
, it seems that all impls would also have to betrusted impl
. - Perhaps we should declare a trait as a “default trait” directly,
rather than using the
impl Drop for ..
syntax. I don’t know precisely what syntax to use, though. - Currently, there are special rules relating to object types and the builtin traits. If the “builtin” traits are no longer builtin, we will have to generalize object types to be simply a set of trait references. This is already planned but merits a second RFC. Note that no changes here are required for the 1.0, since the phasing plan dictates that builtin traits remain special until after 1.0.
- Start Date: 2014-03-31
- RFC PR: rust-lang/rfcs#26
- Rust Issue: rust-lang/rust#13535
Summary
This RFC is a proposal to remove the usage of the keyword priv
from the Rust
language.
Motivation
By removing priv
entirely from the language, it significantly simplifies the
privacy semantics as well as the ability to explain it to newcomers. The one
remaining case, private enum variants, can be rewritten as such:
// pub enum Foo {
// Bar,
// priv Baz,
// }
pub enum Foo {
Bar,
Baz(BazInner)
}
pub struct BazInner(());
// pub enum Foo2 {
// priv Bar2,
// priv Baz2,
// }
pub struct Foo2 {
variant: FooVariant
}
enum FooVariant {
Bar2,
Baz2,
}
Private enum variants are a rarely used feature of the language, and are
generally not regarded as a strong enough feature to justify the priv
keyword
entirely.
Detailed design
There remains only one use case of the priv
visibility qualifier in the Rust
language, which is to make enum variants private. For example, it is possible
today to write a type such as:
pub enum Foo {
Bar,
priv Baz
}
In this example, the variant Bar
is public, while the variant Baz
is
private. This RFC would remove this ability to have private enum variants.
In addition to disallowing the priv
keyword on enum variants, this RFC would
also forbid visibility qualifiers in front of enum variants entirely, as they no
longer serve any purpose.
Status of the identifier priv
This RFC would demote the identifier priv
from being a keyword to being a
reserved keyword (in case we find a use for it in the future).
Alternatives
- Allow private enum variants, as-is today.
- Add a new keyword for
enum
which means “my variants are all private” with controls to make variants public.
Unresolved questions
- Is the assertion that private enum variants are rarely used true? Are there
legitimate use cases for keeping the
priv
keyword?
- Start Date: 2014-04-05
- RFC PR: rust-lang/rfcs#34
- Rust Issue: rust-lang/rust#15759
Summary
Check all types for well-formedness with respect to the bounds of type variables.
Allow bounds on formal type variable in structs and enums. Check these bounds are satisfied wherever the struct or enum is used with actual type parameters.
Motivation
Makes type checking saner. Catches errors earlier in the development process. Matches behaviour with built-in bounds (I think).
Currently formal type variables in traits and functions may have bounds and these bounds are checked whenever the item is used against the actual type variables. Where these type variables are used in types, these types should be checked for well-formedness with respect to the type definitions. E.g.,
trait U {}
trait T<X: U> {}
trait S<Y> {
fn m(x: ~T<Y>) {} // Should be flagged as an error
}
Formal type variables in structs and enums may not have bounds. It is possible to use these type variables in the types of fields, and these types cannot be checked for well-formedness until the struct is instantiated, where each field must be checked.
struct St<X> {
f: ~T<X>, // Cannot be checked
}
Likewise, impls of structs are not checked. E.g.,
impl<X> St<X> { // Cannot be checked
...
}
Here, no struct can exist where X
is replaced by something implementing U
,
so in the impl, X
can be assumed to have the bound U
. But the impl does not
indicate this. Note, this is sound, but does not indicate programmer intent very
well.
Detailed design
Whenever a type is used it must be checked for well-formedness. For polymorphic
types we currently check only that the type exists. I would like to also check
that any actual type parameters are valid. That is, given a type T<U>
where
T
is declared as T<X: B>
, we currently only check that T
does in fact
exist somewhere (I think we also check that the correct number of type
parameters are supplied, in this case one). I would also like to check that U
satisfies the bound B
.
Work on built-in bounds is (I think) in the process of adding this behaviour for built-in bounds. I would like to apply this to user-specified bounds too.
I think no fewer programs can be expressed. That is, any errors we catch with this new check would have been caught later in the existing scheme, where exactly would depend on where the type was used. The only exception would be if the formal type variable was not used.
We would allow bounds on type variable in structs and enums. Wherever a concrete struct or enum type appears, check the actual type variables against the bounds on the formals (the type well-formedness check).
From the above examples:
trait U {}
trait T<X: U> {}
trait S1<Y> {
fn m(x: ~T<Y>) {} //~ ERROR
}
trait S2<Y: U> {
fn m(x: ~T<Y>) {}
}
struct St<X: U> {
f: ~T<X>,
}
impl<X: U> St<X> {
...
}
Alternatives
Keep the status quo.
We could add bounds on structs, etc. But not check them in impls. This is safe since the implementation is more general than the struct. It would mean we allow impls to be un-necessarily general.
Unresolved questions
Do we allow and check bounds in type aliases? We currently do not. We should probably continue not to since these type variables (and indeed the type aliases) are substituted away early in the type checking process. So if we think of type aliases as almost macro-like, then not checking makes sense. OTOH, it is still a little bit inconsistent.
- Start Date: 2014-04-08
- RFC PR: rust-lang/rfcs#40
- Rust Issue: rust-lang/rust#13851
Summary
Split the current libstd into component libraries, rebuild libstd as a facade in front of these component libraries.
Motivation
Rust as a language is ideal for usage in constrained contexts such as embedding in applications, running on bare metal hardware, and building kernels. The standard library, however, is not quite as portable as the language itself yet. The standard library should be as usable as it can be in as many contexts as possible, without compromising its usability in any context.
This RFC is meant to expand the usability of the standard library into these domains where it does not currently operate easily
Detailed design
In summary, the following libraries would make up part of the standard distribution. Each library listed after the colon are the dependent libraries.
- libmini
- liblibc
- liballoc: libmini liblibc
- libcollections: libmini liballoc
- libtext: libmini liballoc libcollections
- librustrt: libmini liballoc liblibc
- libsync: libmini liballoc liblibc librustrt
- libstd: everything above
libmini
Note: The name
libmini
warrants bikeshedding. Please consider it a placeholder for the name of this library.
This library is meant to be the core component of all rust programs in existence. This library has very few external dependencies, and is entirely self contained.
Current modules in std
which would make up libmini would include the list
below. This list was put together by actually stripping down libstd to these
modules, so it is known that it is possible for libmini to compile with these
modules.
atomics
bool
cast
char
clone
cmp
container
default
finally
fmt
intrinsics
io
, stripped down to its coreiter
kinds
mem
num
(and related modules), no float supportops
option
ptr
raw
result
slice
, but without any~[T]
methodstuple
ty
unit
This list may be a bit surprising, and it’s makeup is discussed below. Note that this makeup is selected specifically to eliminate the need for the dreaded “one off extension trait”. This pattern, while possible, is currently viewed as subpar due to reduced documentation benefit and sharding implementation across many locations.
Strings
In a post-DST world, the string type will actually be a library-defined type,
Str
(or similarly named). Strings will no longer be a language feature or a
language-defined type. This implies that any methods on strings must be in the
same crate that defined the Str
type, or done through extension traits.
In the spirit of reducing extension traits, the Str
type and module were left
out of libmini. It’s impossible for libmini to support all methods of Str
, so
it was entirely removed.
This decision does have ramifications on the implementation of libmini
.
-
String literals are an open question. In theory, making a string literal would require the
Str
lang item to be present, but is not present in libmini. That being said, libmini would certainly create many literal strings (for error messages and such). This may be adequately circumvented by having literal strings create a value of type&'static [u8]
if the string lang item is not present. While difficult to work with, this may get us 90% of the way there. -
The
fmt
module must be tweaked for the removal of strings. The only major user-facing detail is that thepad
function onFormatter
would take a byte-slice and a character length, and then not handle the precision (which truncates the byte slice with a number of characters). This may be overcome by possibly having an extension trait could be added for aFormatter
adding a realpad
function that takes strings, or just removing the function altogether in favor ofstr.fmt(formatter)
. -
The
IoError
type suffers from the removal of strings. Currently, this type is inhabited with three fields, an enum, a static description string, and an optionally allocated detail string. Removal of strings would imply theIoError
type would be just the enum itself. This may be an acceptable compromise to make, defining theIoError
type upstream and providing easy constructors from the enum to the struct. Additionally, theOtherIoError
enum variant would be extended with ani32
payload representing the error code (if it came from the OS). -
The
ascii
module is omitted, but it would likely be defined in the crate that definesStr
.
Formatting
While not often thought of as “ultra-core” functionality, this module may be necessary because printing information about types is a fundamental problem that normally requires no dependencies.
Inclusion of this module is the reason why I/O is included in the module as well (or at least a few traits), but the module can otherwise be included with little to no overhead required in terms of dependencies.
Neither print!
nor format!
macros to be a part of this library, but the
write!
macro would be present.
I/O
The primary reason for defining the io
module in the libmini crate would be to
implement the fmt
module. The ramification of removing strings was previously
discussed for IoError
, but there are further modifications that would be
required for the io
module to exist in libmini:
-
The
Buffer
,Listener
,Seek
, andAcceptor
traits would all be defined upstream instead of in libmini. Very little in libstd uses these traits, and nothing in libmini requires them. They are of questionable utility when considering their applicability to all rust code in existence. -
Some extension methods on the
Reader
andWriter
traits would need to be removed. Methods such aspush_exact
,read_exact
,read_to_end
,write_line
, etc., all require owned vectors or similar unimplemented runtime requirements. These can likely be moved to extension traits upstream defined for all readers and writers. Note that this does not apply to the integral reading and writing methods. These are occasionally overwritten for performance, but removal of some extension methods would strongly suggest to me that these methods should be removed. Regardless, the remaining methods could live in essentially any location.
Slices
The only method lost on mutable slices would currently be the sorting method.
This can be circumvented by implementing a sorting algorithm that doesn’t
require allocating a temporary buffer. If intensive use of a sorting algorithm
is required, Rust can provide a libsort
crate with a variety of sorting
algorithms apart from the default sorting algorithm.
FromStr
This trait and module are left out because strings are left out. All types in libmini can have their implementation of FromStr in the crate which implements strings
Floats
This current design excludes floats entirely from libmini (implementations of traits and such). This is another questionable decision, but the current implementation of floats heavily leans on functions defined in libm, so it is unacceptable for these functions to exist in libmini.
Either libstd or a libfloat crate will define floating point traits and such.
Failure
It is unacceptable for Option
to reside outside of libmini, but it is also
also unacceptable for unwrap
to live outside of the Option
type.
Consequently, this means that it must be possible for libmini
to fail.
While impossible for libmini to define failure, it should simply be able to declare failure. While currently not possible today, this extension to the language is possible through “weak lang items”.
Implementation-wise, the failure lang item would have a predefined symbol at which it is defined, and libraries which declare but to not define failure are required to only exist in the rlib format. This implies that libmini can only be built as an rlib. Note that today’s linkage rules do not allow for this (because building a dylib with rlib dependencies is not possible), but the rules could be tweaked to allow for this use case.
tl;dr; The implementation of libmini can use failure, but it does not define failure. All usage of libmini would require an implementation of failure somewhere.
liblibc
This library will exist to provide bindings to libc. This will be a highly platform-specific library, containing an entirely separate api depending on which platform it’s being built for.
This crate will be used to provide bindings to the C language in all forms, and would itself essentially be a giant metadata blob. It conceptually represents the inclusion of all C header files.
Note that the funny name of the library is to allow extern crate libc;
to be
the form of declaration rather than extern crate c;
which is consider to be
too short for its own good.
Note that this crate can only exist in rlib or dylib form.
liballoc
Note: This name
liballoc
is questionable, please consider it a placeholder.
This library would define the allocator traits as well as bind to libc malloc/free (or jemalloc if we decide to include it again). This crate would depend on liblibc and libmini.
Pointers such as ~
and Rc would move into this crate using the default
allocator. The current Gc pointers would move to libgc if possible, or otherwise
librustrt for now (they’re feature gated currently, not super pressing).
Primarily, this library assumes that an allocation failure should trigger a failure. This makes the library not suitable for use in a kernel, but it is suitable essentially everywhere else.
With today’s libstd, this crate would likely mostly be made up by the
global_heap
module. Its purpose is to define the allocation lang items
required by the compiler.
Note that this crate can only exist in rlib form.
libcollections
This crate would not depend on libstd, it would only depend on liballoc and libmini. These two foundational crates should provide all that is necessary to provide a robust set of containers (what you would expect today). Each container would likely have an allocator parameter, and the default would be the default allocator provided by liballoc.
When using the containers from libcollections, it is implicitly assumed that all allocation succeeds, and this will be reflected in the api of each collection.
The contents of this crate would be the entirety of libcollections
as it is
today, as well as the vec
module from the standard library. This would also
implement any relevant traits necessary for ~[T]
.
Note that this crate can only exist in rlib form.
libtext
This crate would define all functionality in rust related to strings. This would
contain the definition of the Str
type, as well as implementations of the
relevant traits from libmini
for the string type.
The crucial assumption of this crate is that allocation does not fail, and the
rest of the string functionality could be built on top of this. Note that this
crate will depend on libcollections
for the Vec
type as the underlying
building block for string buffers and the string type.
This crate would be composed of the str
, ascii
, and unicode
modules which
live in libstd today, but would allow for the extension of other text-related
functionality.
librustrt
This library would be the crate where the rt
module is almost entirely
implemented. It will assume that allocation succeeds, and it will assume a libc
implementation to run on.
The current libstd modules which would be implemented as part of this crate would be:
rt
task
local_data
Note that comm
is not on this list. This crate will additionally define
failure (as unwinding for each task). This crate can exist in both rlib and
dylib form.
libsync
This library will largely remain what it is today, with the exception that the
comm
implementation would move into this crate. The purpose of doing so would
be to consolidate all concurrency-related primitives in this crate, leaving none
out.
This crate would depend on the runtime for task management (scheduling and descheduling).
The libstd
facade
A new standard library would be created that would primarily be a facade which
would expose the underlying crates as a stable API. This library would depend on
all of the above libraries, and would predominately be a grouping of pub use
statements.
This library would also be the library to contain the prelude which would include types from the previous crates. All remaining functionality of the standard library would be filled in as part of this crate.
Note that all rust programs will by default link to libstd
, and hence will
transitively link to all of the upstream crates mentioned above. Many more apis
will be exposed through libstd
directly, however, such as HashMap
, Arc
,
etc.
The exact details of the makeup of this crate will change over time, but it can be considered as “the current libstd plus more”, and this crate will be the source of the “batteries included” aspect of the rust standard library. The API (reexported paths) of the standard library would not change over time. Once a path is reexported and a release is made, all the path will be forced to remain constant over time.
One of the primary reasons for this facade is to provide freedom to restructure the underlying crates. Once a facade is established, it is the only stable API. The actual structure and makeup of all the above crates will be fluid until an acceptable design is settled on. Note that this fluidity does not apply to libstd, only to the structure of the underlying crates.
Updates to rustdoc
With today’s incarnation of rustdoc, the documentation for this libstd facade would not be as high quality as it is today. The facade would just provide hyperlinks back to the original crates, which would have reduced quantities of documentation in terms of navigation, implemented traits, etc. Additionally, these reexports are meant to be implementation details, not facets of the api. For this reason, rustdoc would have to change in how it renders documentation for libstd.
First, rustdoc would consider a cross-crate reexport as inlining of the documentation (similar to how it inlines reexports of private types). This would allow all documentation in libstd to remain in the same location (even the same urls!). This would likely require extensive changes to rustdoc for when entire module trees are reexported.
Secondly, rustdoc will have to be modified to collect implementors of reexported traits all in one location. When libstd reexports trait X, rustdoc will have to search libstd and all its dependencies for implementors of X, listing them out explicitly.
These changes to rustdoc should place it in a much more presentable space, but it is an open question to what degree these modifications will suffice and how much further rustdoc will have to change.
Remaining crates
There are many more crates in the standard distribution of rust, all of which currently depend on libstd. These crates would continue to depend on libstd as most rust libraries would.
A new effort would likely arise to reduce dependence on the standard library by
cutting down to the core dependencies (if necessary). For example, the
libnative
crate currently depend on libstd
, but it in theory doesn’t need to
depend on much other than librustrt
and liblibc
. By cutting out
dependencies, new use cases will likely arise for these crates.
Crates outside of the standard distribution of rust will like to link to the above crates as well (and specifically not libstd). For example, crates which only depend on libmini are likely candidates for being used in kernels, whereas crates only depending on liballoc are good candidates for being embedded into other languages. Having a clear delineation for the usability of a crate in various environments seems beneficial.
Alternatives
-
There are many alternatives to the above sharding of libstd and its dependent crates. The one that is most rigid is likely libmini, but the contents of all other crates are fairly fluid and able to shift around. To this degree, there are quite a few alternatives in how the remaining crates are organized. The ordering proposed is simply one of many.
-
Compilation profiles. Instead of using crate dependencies to encode where a crate can be used, crates could instead be composed of
cfg(foo)
attributes. In theory, there would be onelibstd
crate (in terms of source code), and this crate could be compiled with flags such as--cfg libc
,--cfg malloc
, etc. This route has may have the problem of “multiple standard libraries” in that code compatible with the “libc libstd” is not necessarily compatible with the “no libc libstd”. Asserting that a crate is compatible with multiple profiles would involve requiring multiple compilations. -
Removing libstd entirely. If the standard library is simply a facade, the compiler could theoretically only inject a select number of crates into the prelude, or possibly even omit the prelude altogether. This works towards elimination the question of “does this belong in libstd”, but it would possibly be difficult to juggle the large number of crates to choose from where one could otherwise just look at libstd.
Unresolved questions
-
Compile times. It’s possible that having so many upstream crates for each rust crate will increase compile times through reading metadata and invoking the system linker. Would sharding crates still be worth it? Could possible problems that arise be overcome? Would extra monomorphization in all these crates end up causing more binary bloat?
-
Binary bloat. Another possible side effect of having many upstream crates would be increasing binary bloat of each rust program. Our current linkage model means that if you use anything from a crate that you get everything in that crate (in terms of object code). It is unknown to what degree this will become a concern, and to what degree it can be overcome.
-
Should floats be left out of libmini? This is largely a question of how much runtime support is required for floating point operations. Ideally functionality such as formatting a float would live in libmini, whereas trigonometric functions would live in an external crate with a dependence on libm.
-
Is it acceptable for strings to be left out of libmini? Many common operations on strings don’t require allocation. This is currently done out of necessity of having to define the Str type elsewhere, but this may be seen as too limiting for the scope of libmini.
-
Does liblibc belong so low in the dependency tree? In the proposed design, only the libmini crate doesn’t depend on liblibc. Crates such as libtext and libcollections, however, arguably have no dependence on libc itself, they simply require some form of allocator. Answering this question would be figuring how how to break liballoc’s dependency on liblibc, but it’s an open question as to whether this is worth it or not.
-
Reexporting macros. Currently the standard library defines a number of useful macros which are used throughout the implementation of libstd. There is no way to reexport a macro, so multiple implementations of the same macro would be required for the core libraries to all use the same macro. Is there a better solution to this situation? How much of an impact does this have?
- Start Date: 2014-04-12
- RFC PR: rust-lang/rfcs#42
- Rust Issue: rust-lang/rust#13700
Summary
Add a regexp
crate to the Rust distribution in addition to a small
regexp_macros
crate that provides a syntax extension for compiling regular
expressions during the compilation of a Rust program.
The implementation that supports this RFC is ready to receive feedback: https://github.com/BurntSushi/regexp
Documentation for the crate can be seen here: http://burntsushi.net/rustdoc/regexp/index.html
regex-dna benchmark (vs. Go, Python): https://github.com/BurntSushi/regexp/tree/master/benchmark/regex-dna
Other benchmarks (vs. Go): https://github.com/BurntSushi/regexp/tree/master/benchmark
(Perhaps the links should be removed if the RFC is accepted, since I can’t guarantee they will always exist.)
Motivation
Regular expressions provide a succinct method of matching patterns against search text and are frequently used. For example, many programming languages include some kind of support for regular expressions in its standard library.
The outcome of this RFC is to include a regular expression library in the Rust distribution and resolve issue #3591.
Detailed design
(Note: This is describing an existing design that has been implemented. I have no idea how much of this is appropriate for an RFC.)
The first choice that most regular expression libraries make is whether or not to include backreferences in the supported syntax, as this heavily influences the implementation and the performance characteristics of matching text.
In this RFC, I am proposing a library that closely models Russ Cox’s RE2
(either its C++ or Go variants). This means that features like backreferences
or generalized zero-width assertions are not supported. In return, we get
O(mn)
worst case performance (with m
being the size of the search text and
n
being the number of instructions in the compiled expression).
My implementation currently simulates an NFA using something resembling the Pike VM. Future work could possibly include adding a DFA. (N.B. RE2/C++ includes both an NFA and a DFA, but RE2/Go only implements an NFA.)
The primary reason why I chose RE2 was that it seemed to be a popular choice in issue #3591, and its worst case performance characteristics seemed appealing. I was also drawn to the limited set of syntax supported by RE2 in comparison to other regexp flavors.
With that out of the way, there are other things that inform the design of a regexp library.
Unicode
Given the already existing support for Unicode in Rust, this is a no-brainer. Unicode literals should be allowed in expressions and Unicode character classes should be included (e.g., general categories and scripts).
Case folding is also important for case insensitive matching. Currently, this is implemented by converting characters to their uppercase forms and then comparing them. Future work includes applying at least a simple fold, since folding one Unicode character can produce multiple characters.
Normalization is another thing to consider, but like most other regexp libraries, the one I’m proposing here does not do any normalization. (It seems the recommended practice is to do normalization before matching if it’s needed.)
A nice implementation strategy to support Unicode is to implement a VM that matches characters instead of bytes. Indeed, my implementation does this. However, the public API of a regular expression library should expose byte indices corresponding to match locations (which ought to be guaranteed to be UTF8 codepoint boundaries by construction of the VM). My reason for this is that byte indices result in a lower cost abstraction. If character indices are desired, then a mapping can be maintained by the client at their discretion.
Additionally, this makes it consistent with the std::str
API, which also
exposes byte indices.
Word boundaries, word characters and Unicode
At least Python and D define word characters, word boundaries and space
characters with Unicode character classes. My implementation does the same
by augmenting the standard Perl character classes \d
, \s
and \w
with
corresponding Unicode categories.
Leftmost-first
As of now, my implementation finds the leftmost-first match. This is consistent with PCRE style regular expressions.
I’ve pretty much ignored POSIX, but I think it’s very possible to add leftmost-longest semantics to the existing VM. (RE2 supports this as a parameter, but I believe still does not fully comply with POSIX with respect to picking the correct submatches.)
Public API
There are three main questions that can be asked when searching text:
- Does the string match this expression?
- If so, where?
- Where are its submatches?
In principle, an API could provide a function to only answer (3). The answers to (1) and (2) would immediately follow. However, keeping track of submatches is expensive, so it is useful to implement an optimization that doesn’t keep track of them if it doesn’t have to. For example, submatches do not need to be tracked to answer questions (1) and (2).
The rabbit hole continues: answering (1) can be more efficient than answering (2) because you don’t have to keep track of any capture groups ((2) requires tracking the position of the full match). More importantly, (1) enables early exit from the VM. As soon as a match is found, the VM can quit instead of continuing to search for greedy expressions.
Therefore, it’s worth it to segregate these operations. The performance difference can get even bigger if a DFA were implemented (which can answer (1) and (2) quickly and even help with (3)). Moreover, most other regular expression libraries provide separate facilities for answering these questions separately.
Some libraries (like Python’s re
and RE2/C++) distinguish between matching an
expression against an entire string and matching an expression against part of
the string. My implementation favors simplicity: matching the entirety of a
string requires using the ^
and/or $
anchors. In all cases, an implicit
.*?
is added the beginning and end of each expression evaluated. (Which is
optimized out in the presence of anchors.)
Finally, most regexp libraries provide facilities for splitting and replacing
text, usually making capture group names available with some sort of $var
syntax. My implementation provides this too. (These are a perfect fit for
Rust’s iterators.)
This basically makes up the entirety of the public API, in addition to perhaps
a quote
function that escapes a string so that it may be used as a literal in
an expression.
The regexp!
macro
With syntax extensions, it’s possible to write an regexp!
macro that compiles
an expression when a Rust program is compiled. This includes translating the
matching algorithm to Rust code specific to the expression given. This “ahead
of time” compiling results in a performance increase. Namely, it elides all
heap allocation.
I’ve called these “native” regexps, whereas expressions compiled at runtime are “dynamic” regexps. The public API need not impose this distinction on users, other than requiring the use of a syntax extension to construct a native regexp. For example:
let re = regexp!("a*");
After construction, re
is indistinguishable from an expression created
dynamically:
let re = Regexp::new("a*").unwrap();
In particular, both have the same type. This is accomplished with a representation resembling:
enum MaybeNative {
Dynamic(~[Inst]),
Native(fn(MatchKind, &str, uint, uint) -> ~[Option<uint>]),
}
This syntax extension requires a second crate, regexp_macros
, where the
regexp!
macro is defined. Technically, this could be provided in the regexp
crate, but this would introduce a runtime dependency on libsyntax
for any use
of the regexp
crate.
@alexcrichton remarks that this state of affairs is a wart that will be corrected in the future.
Untrusted input
Given worst case O(mn)
time complexity, I don’t think it’s worth worrying
about unsafe search text.
Untrusted regular expressions are another matter. For example, it’s very easy
to exhaust a system’s resources with nested counted repetitions. For example,
((a{100}){100}){100}
tries to create 100^3
instructions. My current
implementation does nothing to mitigate against this, but I think a simple hard
limit on the number of instructions allowed would work fine. (Should it be
configurable?)
Name
The name of the crate being proposed is regexp
and the type describing a
compiled regular expression is Regexp
. I think an equally good name would be
regex
(and Regex
). Either name seems to be frequently used, e.g., “regexes”
or “regexps” in colloquial use. I chose regexp
over regex
because it
matches the name used for the corresponding package in Go’s standard library.
Other possible names are regexpr
(and Regexpr
) or something with
underscores: reg_exp
(and RegExp
). However, I perceive these to be more
ugly and less commonly used than either regexp
or regex
.
Finally, we could use re
(like Python), but I think the name could be
ambiguous since it’s so short. regexp
(or regex
) unequivocally identifies
the crate as providing regular expressions.
For consistency’s sake, I propose that the syntax extension provided be named
the same as the crate. So in this case, regexp!
.
Summary
My implementation is pretty much a port of most of RE2. The syntax should be identical or almost identical. I think matching an existing (and popular) library has benefits, since it will make it easier for people to pick it up and start using it. There will also be (hopefully) fewer surprises. There is also plenty of room for performance improvement by implementing a DFA.
Alternatives
I think the single biggest alternative is to provide a backtracking
implementation that supports backreferences and generalized zero-width
assertions. I don’t think my implementation precludes this possibility. For
example, a backtracking approach could be implemented and used only when
features like backreferences are invoked in the expression. However, this gives
up the blanket guarantee of worst case O(mn)
time. I don’t think I have the
wisdom required to voice a strong opinion on whether this is a worthwhile
endeavor.
Another alternative is using a binding to an existing regexp library. I think
this was discussed in issue
#3591 and it seems like people
favor a native Rust implementation if it’s to be included in the Rust
distribution. (Does the regexp!
macro require it? If so, that’s a huge
advantage.) Also, a native implementation makes it maximally portable.
Finally, it is always possible to persist without a regexp library.
Unresolved questions
The public API design is fairly simple and straight-forward with no surprises. I think most of the unresolved stuff is how the backend is implemented, which should be changeable without changing the public API (sans adding features to the syntax).
I can’t remember where I read it, but someone had mentioned defining a trait
that declared the API of a regexp engine. That way, anyone could write their
own backend and use the regexp
interface. My initial thoughts are
YAGNI—since requiring different backends seems like a super specialized
case—but I’m just hazarding a guess here. (If we go this route, then we
might want to expose the regexp parser and AST and possibly the
compiler and instruction set to make writing your own backend easier. That
sounds restrictive with respect to making performance improvements in the
future.)
I personally think there’s great value in keeping the standard regexp implementation small, simple and fast. People who have more specialized needs can always pick one of the existing C or C++ libraries.
For now, we could mark the API as #[unstable]
or #[experimental]
.
Future work
I think most of the future work for this crate is to increase the performance,
either by implementing different matching algorithms (e.g., a DFA) or by
improving the code generator that produces native regexps with regexp!
.
If and when a DFA is implemented, care must be taken when creating a code generator, as the size of the code required can grow rapidly.
Other future work (that is probably more important) includes more Unicode support, specifically for simple case folding.
- Start Date: 2014-06-10
- RFC PR: rust-lang/rfcs#48
- Rust Issue: rust-lang/rust#5527
Summary
Cleanup the trait, method, and operator semantics so that they are well-defined and cover more use cases. A high-level summary of the changes is as follows:
- Generalize explicit self types beyond
&self
and&mut self
etc, so that self-type declarations likeself: Rc<Self>
become possible. - Expand coherence rules to operate recursively and distinguish orphans more carefully.
- Revise vtable resolution algorithm to be gradual.
- Revise method resolution algorithm in terms of vtable resolution.
This RFC excludes discussion of associated types and multidimensional type classes, which will be the subject of a follow-up RFC.
Motivation
The current trait system is ill-specified and inadequate. Its implementation dates from a rather different language. It should be put onto a surer footing.
Use cases
Poor interaction with overloadable deref and index
Addressed by: New method resolution algorithm.
The deref operator *
is a flexible one. Imagine a pointer p
of
type ~T
. This same *
operator can be used for three distinct
purposes, depending on context.
- Create an immutable referent to the referent:
&*p
. - Create a mutable reference to the referent:
&mut *p
. - Copy/move the contents of the referent:
consume(*p)
.
Not all of these operations are supported by all types. In fact,
because most smart pointers represent aliasable data, they will only
support the creation of immutable references (e.g., Rc
, Gc
).
Other smart pointers (e.g., the RefMut
type returned by RefCell
)
support mutable or immutable references, but not moves. Finally, a
type that owns its data (like, indeed, ~T
) might support #3.
To reflect this, we use distinct traits for the various operators. (In fact, we don’t currently have a trait for copying/moving the contents, this could be a distinct RFC (ed., I’m still thinking this over myself, there are non-trivial interactions)).
Unfortunately, the method call algorithm can’t really reliably choose
mutable vs immutable deref. The challenge is that the proper choice
will sometimes not be apparent until quite late in the process. For
example, imagine the expression p.foo()
: if foo()
is defined with
&self
, we want an immutable deref, otherwise we want a mutable
deref.
Note that in this RFC I do not completely address this issue. In
particular, in an expression like (*p).foo()
, where the dereference
is explicit and not automatically inserted, the sense of the
dereference is not inferred. For the time being, the sense can be
manually specified by making the receiver type fully explicit: (&mut *p).foo()
vs (&*p).foo()
. I expect in a follow-up RFC to possibly
address this problem, as well as the question of how to handle copies
and moves of the referent (use #3 in my list above).
Lack of backtracking
Addressed by: New method resolution algorithm.
Issue #XYZ. When multiple traits define methods with the same name, it is ambiguous which trait is being used:
trait Foo { fn method(&self); }
trait Bar { fn method(&self); }
In general, so long as a given type only implements Foo
or Bar
,
these ambiguities don’t present a problem (and ultimately Universal
Function Call Syntax or UFCS will present an explicit resolution).
However, this is not guaranteed. Sometimes we see “blanket” impls
like the following:
impl<A:Base> Foo for A { }
This impl basically says “any type T
that implements Base
automatically implements Foo
”. Now, we expect an ambiguity error
if we have a type T
that implements both Base
and Bar
. But in
fact, we’ll get an ambiguity error even if a type only implements
Bar
. The reason for this is that the current method resolution
doesn’t “recurse” and check additional dependencies when deciding if
an impl
is applicable. So it will decide, in this case, that the
type T
could implement Foo
and then record for later that T
must
implement Base
. This will lead to weird errors.
Overly conservative coherence
Addressed by: Expanded coherence rules.
The job of coherence is to ensure that, for any given set of type parameters, a given trait is implemented at most once (it may of course not be implemented at all). Currently, however, coherence is more conservative that it needs to be. This is partly because it doesn’t take into account the very property that it itself is enforcing.
The problems arise due to the “blanket impls” I discussed in the previous section. Consider the following two traits and a blanket impl:
trait Base { }
trait Derived { }
impl<A:Base> Derived for A { }
Here we have two traits Base
and Derived
, and a blanket impl which
implements the Derived
trait for any type A
that also implements
Base
.
This implies that if you implement Base
for a type S
, then S
automatically implements Derived
:
struct S;
impl Base for S { } // Implement Base => Implements Derived
On a related note, it’d be an error to implement both Base
and Derived
for the same type T
:
// Illegal
struct T;
impl Base for T { }
impl Derived for T { }
This is illegal because now there are two implements of Derived
for T
. There is the direct one, but also an indirect one. We do not
assign either higher precedence, we just report it as an error.
So far, all is in agreement with the current rules. However, problems
arise if we imagine a type U
that only implements Derived
:
struct U;
impl Derived for U { } // Should be OK, currently not.
In this scenario, there is only one implementation of Derived
. But
the current coherence rules still report it as an error.
Here is a concrete example where a rule like this would be useful. We
currently have the Copy
trait (aka Pod
), which states that a type
can be memcopied. We also have the Clone
trait, which is a more
heavyweight version for types where copying requires allocation. It’d
be nice if all types that could be copied could also be cloned – it’d
also be nice if we knew for sure that copying a value had the same
semantics as cloning it, in that case. We can guarantee both using a
blanket impl like the following:
impl<T:Copy> Clone for T {
fn clone(&self) -> T {
*self
}
}
Unfortunately, writing such an impl today would imply that no other
types could implement Clone
. Obviously a non-starter.
There is one not especially interesting ramification of
this. Permitting this rule means that adding impls to a type could
cause coherence errors. For example, if I had a type which implements
Copy
, and I add an explicit implementation of Clone
, I’d get an
error due to the blanket impl. This could be seen as undesirable
(perhaps we’d like to preserve that property that one can always add
impls without causing errors).
But of course we already don’t have the property that one can always add impls, since method calls could become ambiguous. And if we were to add “negative bounds”, which might be nice, we’d lose that property. And the popularity and usefulness of blanket impls cannot be denied. Therefore, I think this property (“always being able to add impls”) is not especially useful or important.
Hokey implementation
Addressed by: Gradual vtable resolution algorithm
In an effort to improve inference, the current implementation has a rather ad-hoc two-pass scheme. When performing a method call, it will immediately attempt “early” trait resolution and – if that fails – defer checking until later. This helps with some particular scenarios, such as a trait like:
trait Map<E> {
fn map(&self, op: |&E| -> E) -> Self;
}
Given some higher-order function like:
fn some_mapping<E,V:Map<E>>(v: &V, op: |&E| -> E) { ... }
If we were then to see a call like:
some_mapping(vec, |elem| ...)
the early resolution would be helpful in connecting the type of elem
with the type of vec
. The reason to use two phases is that often we
don’t need to resolve each trait bound to a specific impl, and if we
wait till the end then we will have more type information available.
In my proposed solution, we eliminate the phase distinction. Instead, we simply track pending constraints. We are free to attempt to resolve pending constraints whenever desired. In particular, whenever we find we need more type information to proceed with some type-overloaded operation, rather than reporting an error we can try and resolve pending constraints. If that helps give more information, we can carry on. Once we reach the end of the function, we must then resolve all pending constraints that have not yet been resolved for some other reason.
Note that there is some interaction with the distinction between input
and output type parameters discussed in the previous
example. Specifically, we must never infer the value of the Self
type parameter based on the impls in scope. This is because it would
cause crate concatenation to potentially lead to compilation errors
in the form of inference failure.
Properties
There are important properties I would like to guarantee:
- Coherence or No Overlapping Instances: Given a trait and values for all of its type parameters, there should always be at most one applicable impl. This should remain true even when unknown, additional crates are loaded.
- Crate concatenation: It should always be possible to take two creates and combine them without causing compilation errors. This property
Here are some properties I do not intend to guarantee:
- Crate divisibility: It is not always possible to divide a crate into two crates. Specifically, this may incur coherence violations due to the orphan rules.
- Decidability: Haskell has various sets of rules aimed at
ensuring that the compiler can decide whether a given trait is
implemented for a given type. All of these rules wind up preventing
useful implementations and thus can be turned off with the
undecidable-instances
flag. I don’t think decidability is especially important. The compiler can simply keep a recursion counter and report an error if that level of recursion is exceeded. This counter can be adjusted by the user on a crate-by-crate basis if some bizarre impl pattern happens to require a deeper depth to be resolved.
Detailed design
In general, I won’t give a complete algorithmic specification. Instead, I refer readers to the prototype implementation. I would like to write out a declarative and non-algorithmic specification for the rules too, but that is work in progress and beyond the scope of this RFC. Instead, I’ll try to explain in “plain English”.
Method self-type syntax
Currently methods must be declared using the explicit-self shorthands:
fn foo(self, ...)
fn foo(&self, ...)
fn foo(&mut self, ...)
fn foo(~self, ...)
Under this proposal we would keep these shorthands but also permit any
function in a trait to be used as a method, so long as the type of the
first parameter is either Self
or something derefable Self
:
fn foo(self: Gc<Self>, ...)
fn foo(self: Rc<Self>, ...)
fn foo(self: Self, ...) // equivalent to `fn foo(self, ...)
fn foo(self: &Self, ...) // equivalent to `fn foo(&self, ...)
It would not be required that the first parameter be named self
,
though it seems like it would be useful to permit it. It’s also
possible we can simply make self
not be a keyword (that would be my
personal preference, if we can achieve it).
Coherence
The coherence rules fall into two categories: the orphan restriction and the overlapping implementations restriction.
Orphan check: Every implementation must meet one of the following conditions:
-
The trait being implemented (if any) must be defined in the current crate.
-
The
Self
type parameter must meet the following grammar, whereC
is a struct or enum defined within the current crate:T = C | [T] | [T, ..n] | &T | &mut T | ~T | (..., T, ...) | X<..., T, ...> where X is not bivariant with respect to T
Overlapping instances: No two implementations of the same trait can
be defined for the same type (note that it is only the Self
type
that matters). For this purpose of this check, we will also
recursively check bounds. This check is ultimately defined in terms of
the RESOLVE algorithm discussed in the implementation section below:
it must be able to conclude that the requirements of one impl are
incompatible with the other.
Here is a simple example that is OK:
trait Show { ... }
impl Show for int { ... }
impl Show for uint { ... }
The first impl implements Show for int
and the case implements
Show for uint
. This is ok because the type int
cannot be unified
with uint
.
The following example is NOT OK:
trait Iterator<E> { ... }
impl Iterator<char> for ~str { ... }
impl Iterator<u8> for ~str { ... }
Even though E
is bound to two distinct types, E
is an output type
parameter, and hence we get a coherence violation because the input
type parameters are the same in each case.
Here is a more complex example that is also OK:
trait Clone { ... }
impl<A:Copy> Clone for A { ... }
impl<B:Clone> Clone for ~B { ... }
These two impls are compatible because the resolution algorithm is
able to see that the type ~B
will never implement Copy
, no matter
what B
is. (Note that our ability to do this check relies on the
orphan checks: without those, we’d never know if some other crate
might add an implementation of Copy
for ~B
.)
Since trait resolution is not fully decidable, it is possible to concoct scenarios in which coherence can neither confirm nor deny the possibility that two impls are overlapping. One way for this to happen is when there are two traits which the user knows are mutually exclusive; mutual exclusion is not currently expressible in the type system [7] however, and hence the coherence check will report errors. For example:
trait Even { } // Naturally can't be Even and Odd at once!
trait Odd { }
impl<T:Even> Foo for T { }
impl<T:Odd> Foo for T { }
Another possible scenario is infinite recursion between impls. For example, in the following scenario, the coherence checked would be unable to decide if the following impls overlap:
impl<A:Foo> Bar for A { ... }
impl<A:Bar> Foo for A { ... }
In such cases, the recursion bound is exceeded and an error is conservatively reported. (Note that recursion is not always so easily detected.)
Method resolution
Let us assume the method call is r.m(...)
and the type of the
receiver r
is R
. We will resolve the call in two phases. The first
phase checks for inherent methods [4] and the second phase for
trait methods. Both phases work in a similar way, however. We will
just describe how trait method search works and then express the
inherent method search in terms of traits.
The core method search looks like this:
METHOD-SEARCH(R, m):
let TRAITS = the set consisting of any in-scope trait T where:
1. T has a method m and
2. R implements T<...> for any values of Ty's type parameters
if TRAITS is an empty set:
if RECURSION DEPTH EXCEEDED:
return UNDECIDABLE
if R implements Deref<U> for some U:
return METHOD-SEARCH(U, m)
return NO-MATCH
if TRAITS is the singleton set {T}:
RECONCILE(R, T, m)
return AMBIGUITY(TRAITS)
Basically, we will continuously auto-dereference the receiver type,
searching for some type that implements a trait that offers the method
m
. This gives precedence to implementations that require fewer
autodereferences. (There exists the possibility of a cycle in the
Deref
chain, so we will only autoderef so many times before
reporting an error.)
Receiver reconciliation
Once we find a trait that is implemented for the (adjusted) receiver
type R
and which offers the method m
, we must reconcile the
receiver with the self type declared in m
. Let me explain by
example.
Consider a trait Mob
(anyone who ever hacked on the MUD source code
will surely remember Mobs!):
trait Mob {
fn hit_points(&self) -> int;
fn take_damage(&mut self, damage: int) -> int;
fn move_to_room(self: GC<Self>, room: &Room);
}
Let’s say we have a type Monster
, and Monster
implements Mob
:
struct Monster { ... }
impl Mob for Monster { ... }
And now we see a call to hit_points()
like so:
fn attack(victim: &mut Monster) {
let hp = victim.hit_points();
...
}
Our method search algorithm above will proceed by searching for an
implementation of Mob
for the type &mut Monster
. It won’t find
any. It will auto-deref &mut Monster
to yield the type Monster
and
search again. Now we find a match. Thus far, then, we have a single
autoderef *victims
, yielding the type Monster
– but the method
hit_points()
actually expects a reference (&Monster
) to be given
to it, not a by-value Monster
.
This is where self-type reconciliation steps in. The reconciliation process works by unwinding the adjustments and adding auto-refs:
RECONCILE(R, T, m):
let E = the expected self type of m in trait T;
// Case 1.
if R <: E:
we're done.
// Case 2.
if &R <: E:
add an autoref adjustment, we're done.
// Case 3.
if &mut R <: E:
adjust R for mutable borrow (if not possible, error).
add a mut autoref adjustment, we're done.
// Case 4.
unwind one adjustment to yield R' (if not possible, error).
return RECONCILE(R', T, m)
In this case, the expected self type E
would be &Monster
. We would
first check for case 1: is Monster <: &Monster
? It is not. We would
then proceed to case 2. Is &Monster <: &Monster
? It is, and hence
add an autoref. The final result then is that victim.hit_points()
becomes transformed to the equivalent of (using UFCS notation)
Mob::hit_points(&*victim)
.
To understand case 3, let’s look at a call to take_damage
:
fn attack(victim: &mut Monster) {
let hp = victim.hit_points(); // ...this is what we saw before
let damage = hp / 10; // 1/10 of current HP in damage
victim.take_damage(damage);
...
}
As before, we would auto-deref once to find the type Monster
. This
time, though, the expected self type is &mut Monster
. This means
that both cases 1 and 2 fail and we wind up at case 3, the test for
which succeeds. Now we get to this statement: “adjust R
for mutable
borrow”.
At issue here is the
overloading of the deref operator that was discussed earlier.
In this case, the end result we want is Mob::hit_points(&mut *victim)
, which means that *
is being used for a mutable borrow,
which is indicated by the DerefMut
trait. However, while doing the
autoderef loop, we always searched for impls of the Deref
trait,
since we did not yet know which trait we wanted. [2] We need to
patch this up. So this loop will check whether the type &mut Monster
implements DerefMut
, in addition to just Deref
(it does).
This check for case 3 could fail if, e.g., victim
had a type like
Gc<Monster>
or Rc<Monster>
. You’d get a nice error message like
“the type Rc
does not support mutable borrows, and the method
take_damage()
requires a mutable receiver”.
We still have not seen an example of cases 1 or 4. Let’s use a slightly modified example:
fn flee_if_possible(victim: Gc<Monster>, room: &mut Room) {
match room.find_random_exit() {
None => { }
Some(exit) => {
victim.move_to_room(exit);
}
}
}
As before, we’ll start out with a type of Monster
, but this type the
method move_to_room()
has a receiver type of Gc<Monster>
. This
doesn’t match cases 1, 2, or 3, so we proceed to case 4 and unwind
by one adjustment. Since the most recent adjustment was to deref from
Gc<Monster>
to Monster
, we are left with a type of
Gc<Monster>
. We now search again. This time, we match case 1. So the
final result is Mob::move_to_room(victim, room)
. This last case is
sort of interesting because we had to use the autoderef to find the
method, but once resolution is complete we do not wind up
dereferencing victim
at all.
Finally, let’s see an error involving case 4. Imagine we modified
the type of victim
in our previous example to be &Monster
and
not Gc<Monster>
:
fn flee_if_possible(victim: &Monster, room: &mut Room) {
match room.find_random_exit() {
None => { }
Some(exit) => {
victim.move_to_room(exit);
}
}
}
In this case, we would again unwind an adjustment, going from
Monster
to &Monster
, but at that point we’d be stuck. There are no
more adjustments to unwind and we never found a type
Gc<Monster>
. Therefore, we report an error like “the method
move_to_room()
expects a Gc<Monster>
but was invoked with an
&Monster
”.
Inherent methods
Inherent methods can be “desugared” into traits by assuming a trait
per struct or enum. Each impl like impl Foo
is effectively an
implementation of that trait, and all those traits are assumed to be
imported and in scope.
Differences from today
Today’s algorithm isn’t really formally defined, but it works very differently from this one. For one thing, it is based purely on subtyping checks, and does not rely on the generic trait matching. This is a crucial limitation that prevents cases like those described in lack of backtracking from working. It also results in a lot of code duplication and a general mess.
Interaction with vtables and type inference
One of the goals of this proposal is to remove the hokey distinction between early and late resolution. The way that this will work now is that, as we execute, we’ll accumulate a list of pending trait obligations. Each obligation is the combination of a trait and set of types. It is called an obligation because, for the method to be correctly typed, we must eventually find an implementation of that trait for those types. Due to type inference, though, it may not be possible to do this right away, since some of the types may not yet be fully known.
The semantics of trait resolution mean that, at any point in time, the type checker is free to stop what it’s doing and try to resolve these pending obligations, so long as none of the input type parameters are unresolved (see below). If it is able to definitely match an impl, this may in turn affect some type variables which are output type parameters. The basic idea then is to always defer doing resolution until we either (a) encounter a point where we need more type information to proceed or (b) have finished checking the function. At those times, we can go ahead and try to do resolution. If, after type checking the function in its entirety, there are still obligations that cannot be definitely resolved, that’s an error.
Ensuring crate concatenation
To ensure crate concentanability, we must only consider the Self
type parameter when deciding when a trait has been implemented (more
generally, we must know the precise set of input type parameters; I
will cover an expanded set of rules for this in a subsequent RFC).
To see why this matters, imagine a scenario like this one:
trait Produce<R> {
fn produce(&self: Self) -> R;
}
Now imagine I have two crates, C and D. Crate C defines two types,
Vector
and Real
, and specifies a way to combine them:
struct Vector;
impl Produce<int> for Vector { ... }
Now imagine crate C has some code like:
fn foo() {
let mut v = None;
loop {
if v.is_some() {
let x = v.get().produce(); // (*)
...
} else {
v = Some(Vector);
}
}
}
At the point (*)
of the call to produce()
we do not yet know the
type of the receiver. But the inferencer might conclude that, since it
can only see one impl
of Produce
for Vector
, v
must have type
Vector
and hence x
must have the type int
.
However, then we might find another crate D that adds a new impl:
struct Other;
struct Real;
impl Combine<Real> for Other { ... }
This definition passes the orphan check because at least one of the
types (Real
, in this case) in the impl is local to the current
crate. But what does this mean for our previous inference result? In
general, it looks risky to decide types based on the impls we can see,
since there could always be more impls we can’t actually see.
It seems clear that this aggressive inference breaks the crate concatenation property. If we combined crates C and D into one crate, then inference would fail where it worked before.
If x
were never used in any way that forces it to be an int
, then
it’s even plausible that the type Real
would have been valid in some
sense. So the inferencer is influencing program execution to some
extent.
Implementation details
The “resolve” algorithm
The basis for the coherence check, method lookup, and vtable lookup algorithms is the same function, called RESOLVE. The basic idea is that it takes a set of obligations and tries to resolve them. The result is four sets:
- CONFIRMED: Obligations for which we were able to definitely select a specific impl.
- NO-IMPL: Obligations which we know can NEVER be satisfied, because there is no specific impl. The only reason that we can ever say this for certain is due to the orphan check.
- DEFERRED: Obligations that we could not definitely link to an impl, perhaps because of insufficient type information.
- UNDECIDABLE: Obligations that were not decidable due to excessive recursion.
In general, if we ever encounter a NO-IMPL or UNDECIDABLE, it’s probably an error. DEFERRED obligations are ok until we reach the end of the function. For details, please refer to the prototype.
Alternatives and downsides
Autoderef and ambiguity
The addition of a Deref
trait makes autoderef complicated, because
we may encounter situations where the smart pointer and its
reference both implement a trait, and we cannot know what the user
wanted.
The current rule just decides in favor of the smart pointer; this is somewhat unfortunate because it is likely to not be what the user wanted. It also means that adding methods to smart pointer types is a potentially breaking change. This is particularly problematic because we may want the smart pointer to implement a trait that requires the method in question!
An interesting thought would be to change this rule and say that we
always autoderef first and only resolve the method against the
innermost reference. Note that UFCS provides an explicit “opt-out” if
this is not what was desired. This should also have the (beneficial,
in my mind) effect of quelling the over-eager use of Deref
for types
that are not smart pointers.
This idea appeals to me but I think belongs in a separate RFC. It needs to be evaluated.
Footnotes
Note 1: when combining with DST, the in
keyword goes
first, and then any other qualifiers. For example, in unsized RHS
or
in type RHS
etc. (The precise qualifier in use will depend on the
DST proposal.)
Note 2: Note that the DerefMut<T>
trait extends
Deref<T>
, so if a type supports mutable derefs, it must also support
immutable derefs.
Note 3: The restriction that inputs must precede outputs is not strictly necessary. I added it to keep options open concerning associated types and so forth. See the Alternatives section, specifically the section on associated types.
Note 4: The prioritization of inherent methods could be
reconsidered after DST has been implemented. It is currently needed to
make impls like impl Trait for ~Trait
work.
Note 5: The set of in-scope traits is currently defined as those that are imported by name. PR #37 proposes possible changes to this rule.
Note 6: In the section on autoderef and ambiguity, I
discuss alternate rules that might allow us to lift the requirement
that the receiver be named self
.
Note 7: I am considering introducing mechanisms in a subsequent RFC that could be used to express mutual exclusion of traits.
- Start Date: 2014-03-20
- RFC PR: rust-lang/rfcs#49
- Rust Issue: rust-lang/rust#12812
Summary
Allow attributes on match arms.
Motivation
One sometimes wishes to annotate the arms of match statements with
attributes, for example with conditional compilation #[cfg]
s or
with branch weights (the latter is the most important use).
For the conditional compilation, the work-around is duplicating the
whole containing function with a #[cfg]
. A case study is
sfackler’s bindings to OpenSSL,
where many distributions remove SSLv2 support, and so that portion of
Rust bindings needs to be conditionally disabled. The obvious way to
support the various different SSL versions is an enum
pub enum SslMethod {
#[cfg(sslv2)]
/// Only support the SSLv2 protocol
Sslv2,
/// Only support the SSLv3 protocol
Sslv3,
/// Only support the TLSv1 protocol
Tlsv1,
/// Support the SSLv2, SSLv3 and TLSv1 protocols
Sslv23,
}
However, all match
s can only mention Sslv2
when the cfg
is
active, i.e. the following is invalid:
fn name(method: SslMethod) -> &'static str {
match method {
Sslv2 => "SSLv2",
Sslv3 => "SSLv3",
_ => "..."
}
}
A valid method would be to have two definitions: #[cfg(sslv2)] fn name(...)
and #[cfg(not(sslv2)] fn name(...)
. The former has the
Sslv2
arm, the latter does not. Clearly, this explodes exponentially
for each additional cfg
’d variant in an enum.
Branch weights would allow the careful micro-optimiser to inform the compiler that, for example, a certain match arm is rarely taken:
match foo {
Common => {}
#[cold]
Rare => {}
}
Detailed design
Normal attribute syntax, applied to a whole match arm.
match x {
#[attr]
Thing => {}
#[attr]
Foo | Bar => {}
#[attr]
_ => {}
}
Alternatives
There aren’t really any general alternatives; one could probably hack around matching on conditional enum variants with some macros and helper functions to share as much code as possible; but in general this won’t work.
Unresolved questions
Nothing particularly.
- Start Date: 2014-04-18
- RFC PR: rust-lang/rfcs#50
- Rust Issue: rust-lang/rust#13789
Summary
Asserts are too expensive for release builds and mess up inlining. There must be a way to turn them off. I propose macros debug_assert!
and assert!
. For test cases, assert!
should be used.
Motivation
Asserts are too expensive in release builds.
Detailed design
There should be two macros, debug_assert!(EXPR)
and assert!(EXPR)
. In debug builds (without --cfg ndebug
), debug_assert!()
is the same as assert!()
. In release builds (with --cfg ndebug
), debug_assert!()
compiles away to nothing. The definition of assert!()
is if (!EXPR) { fail!("assertion failed ({}, {}): {}", file!(), line!(), stringify!(expr) }
Alternatives
Other designs that have been considered are using debug_assert!
in test cases and not providing assert!
, but this doesn’t work with separate compilation.
The impact of not doing this is that assert!
will be expensive, prompting people will write their own local debug_assert!
macros, duplicating functionality that should have been in the standard library.
Unresolved questions
None.
- Start Date: 2014-04-30
- RFC PR: rust-lang/rfcs#59
- Rust Issue: rust-lang/rust#13885
Summary
The tilde (~
) operator and type construction do not support allocators and therefore should be removed in favor of the box
keyword and a language item for the type.
Motivation
-
There will be a unique pointer type in the standard library,
Box<T,A>
whereA
is an allocator. The~T
type syntax does not allow for custom allocators. Therefore, in order to keep~T
around while still supporting allocators, we would need to make it an alias forBox<T,Heap>
. In the spirit of having one way to do things, it seems better to remove~
entirely as a type notation. -
~EXPR
andbox EXPR
are duplicate functionality; the former does not support allocators. Again in the spirit of having one and only one way to do things, I would like to remove~EXPR
. -
Some people think
~
is confusing, as it is less self-documenting thanBox
. -
~
can encourage people to blindly add sigils attempting to get their code to compile instead of consulting the library documentation.
Drawbacks
~T
may be seen as convenient sugar for a common pattern in some situations.
Detailed design
The ~EXPR
production is removed from the language, and all such uses are converted into box
.
Add a lang item, box
. That lang item will be defined in liballoc
(NB: not libmetal
/libmini
, for bare-metal programming) as follows:
#[lang="box"]
pub struct Box<T,A=Heap>(*T);
All parts of the compiler treat instances of Box<T>
identically to the way it treats ~T
today.
The destructuring form for Box<T>
will be box PAT
, as follows:
let box(x) = box(10);
println!("{}", x); // prints 10
Alternatives
The other possible design here is to keep ~T
as sugar. The impact of doing this would be that a common pattern would be terser, but I would like to not do this for the reasons stated in “Motivation” above.
Unresolved questions
The allocator design is not yet fully worked out.
It may be possible that unforeseen interactions will appear between the struct nature of Box<T>
and the built-in nature of ~T
when merged.
- Start Date: 2014-04-30
- RFC PR: rust-lang/rfcs#60
- Rust Issue: rust-lang/rust#14312
Summary
StrBuf
should be renamed to String
.
Motivation
Since StrBuf
is so common, it would benefit from a more traditional name.
Drawbacks
It may be that StrBuf
is a better name because it mirrors Java StringBuilder
or C# StringBuffer
. It may also be that String
is confusing because of its similarity to &str
.
Detailed design
Rename StrBuf
to String
.
Alternatives
The impact of not doing this would be that StrBuf
would remain StrBuf
.
Unresolved questions
None.
- Start Date: 2014-05-02
- RFC PR: rust-lang/rfcs#63
- Rust Issue: rust-lang/rust#14180
Summary
The rules about the places mod foo;
can be used are tightened to only permit
its use in a crate root and in mod.rs
files, to ensure a more sane
correspondence between module structure and file system hierarchy. Most
notably, this prevents a common newbie error where a module is loaded multiple
times, leading to surprising incompatibility between them. This proposal does
not take away one’s ability to shoot oneself in the foot should one really
desire to; it just removes almost all of the rope, leaving only mixed
metaphors.
Motivation
It is a common newbie mistake to write things like this:
lib.rs
:
mod foo;
pub mod bar;
foo.rs
:
mod baz;
pub fn foo(_baz: baz::Baz) { }
bar.rs
:
mod baz;
use foo::foo;
pub fn bar(baz: baz::Baz) {
foo(baz)
}
baz.rs
:
pub struct Baz;
This fails to compile because foo::foo()
wants a foo::baz::Baz
, while
bar::bar()
is giving it a bar::baz::Baz
.
Such a situation, importing one file multiple times, is exceedingly rarely what
the user actually wanted to do, but the present design allows it to occur
without warning the user. The alterations contained herein ensure that there is
no situation where such double loading can occur without deliberate intent via
#[path = "….rs"]
.
Drawbacks
None known.
Detailed design
When a mod foo;
statement is used, the compiler attempts to find a suitable
file. At present, it just blindly seeks for foo.rs
or foo/mod.rs
(relative
to the file under parsing).
The new behaviour will only permit mod foo;
if at least one of the following
conditions hold:
-
The file under parsing is the crate root, or
-
The file under parsing is a
mod.rs
, or -
#[path]
is specified, e.g.#[path = "foo.rs"] mod foo;
.
In layman’s terms, the file under parsing must “own” the directory, so to speak.
Alternatives
The rationale is covered in the summary. This is the simplest repair to the current lack of structure; all alternatives would be more complex and invasive.
One non-invasive alternative is a lint which would detect double loads. This is less desirable than the solution discussed in this RFC as it doesn’t fix the underlying problem which can, fortunately, be fairly easily fixed.
Unresolved questions
None.
- Start Date: 2014-05-04
- RFC PR: rust-lang/rfcs#66
- Rust Issue: rust-lang/rust#15023
Summary
Temporaries live for the enclosing block when found in a let-binding. This only holds when the reference to the temporary is taken directly. This logic should be extended to extend the cleanup scope of any temporary whose lifetime ends up in the let-binding.
For example, the following doesn’t work now, but should:
use std::os;
fn main() {
let x = os::args().slice_from(1);
println!("{}", x);
}
Motivation
Temporary lifetimes are a bit confusing right now. Sometimes you can keep
references to them, and sometimes you get the dreaded “borrowed value does not
live long enough” error. Sometimes one operation works but an equivalent
operation errors, e.g. autoref of ~[T]
to &[T]
works but calling
.as_slice()
doesn’t. In general it feels as though the compiler is simply
being overly restrictive when it decides the temporary doesn’t live long
enough.
Drawbacks
I can’t think of any drawbacks.
Detailed design
When a reference to a temporary is passed to a function (either as a regular
argument or as the self
argument of a method), and the function returns a
value with the same lifetime as the temporary reference, the lifetime of the
temporary should be extended the same way it would if the function was not
invoked.
For example, ~[T].as_slice()
takes &'a self
and returns &'a [T]
. Calling
as_slice()
on a temporary of type ~[T]
will implicitly take a reference
&'a ~[T]
and return a value &'a [T]
This return value should be considered
to extend the lifetime of the ~[T]
temporary just as taking an explicit
reference (and skipping the method call) would.
Alternatives
Don’t do this. We live with the surprising borrowck errors and the ugly workarounds that look like
let x = os::args();
let x = x.slice_from(1);
Unresolved questions
None that I know of.
- Start Date: 2014-06-11
- RFC PR: rust-lang/rfcs#68
- Rust Issue: rust-lang/rust#7362
Summary
Rename *T
to *const T
, retain all other semantics of unsafe pointers.
Motivation
Currently the T*
type in C is equivalent to *mut T
in Rust, and the const T*
type in C is equivalent to the *T
type in Rust. Noticeably, the two most
similar types, T*
and *T
have different meanings in Rust and C, frequently
causing confusion and often incorrect declarations of C functions.
If the compiler is ever to take advantage of the guarantees of declaring an FFI
function as taking T*
or const T*
(in C), then it is crucial that the FFI
declarations in Rust are faithful to the declaration in C.
The current difference in Rust unsafe pointers types with C pointers types is proving to be too error prone to realistically enable these optimizations at a future date. By renaming Rust’s unsafe pointers to closely match their C brethren, the likelihood for erroneously transcribing a signature is diminished.
Detailed design
This section will assume that the current unsafe pointer design is forgotten completely, and will explain the unsafe pointer design from scratch.
There are two unsafe pointers in rust, *mut T
and *const T
. These two types
are primarily useful when interacting with foreign functions through a FFI. The
*mut T
type is equivalent to the T*
type in C, and the *const T
type is
equivalent to the const T*
type in C.
The type &mut T
will automatically coerce to *mut T
in the normal locations
that coercion occurs today. It will also be possible to explicitly cast with an
as
expression. Additionally, the &T
type will automatically coerce to
*const T
. Note that &mut T
will not automatically coerce to *const T
.
The two unsafe pointer types will be freely castable among one another via as
expressions, but no coercion will occur between the two. Additionally, values of
type uint
can be casted to unsafe pointers.
When is a coercion valid?
When coercing from &'a T
to *const T
, Rust will guarantee that the memory
will remain valid for the lifetime 'a
and the memory will be immutable up to
memory stored in Unsafe<U>
. It is the responsibility of the code working with
the *const T
that the pointer is only dereferenced in the lifetime 'a
.
When coercing from &'a mut T
to *mut T
, Rust will guarantee that the memory
will stay valid during 'a
and that the memory will not be accessed during
'a
. Additionally, Rust will consume the &'a mut T
during the coercion. It
is the responsibility of the code working with the *mut T
to guarantee that
the unsafe pointer is only dereferenced in the lifetime 'a
, and that the
memory is “valid again” after 'a
.
Note: Rust will consume
&mut T
coercions with both implicit and explicit coercions.
The term “valid again” is used to represent that some types in Rust require
internal invariants, such as Box<T>
never being NULL
. This is often a
per-type invariant, so it is the responsibility of the unsafe code to uphold
these invariants.
When is a safe cast valid?
Unsafe code can convert an unsafe pointer to a safe pointer via dereferencing inside of an unsafe block. This section will discuss when this action is valid.
When converting *mut T
to &'a mut T
, it must be guaranteed that the memory
is initialized to start out with and that nobody will access the memory during
'a
except for the converted pointer.
When converting *const T
to &'a T
, it must be guaranteed that the memory is
initialized to start out with and that nobody will write to the pointer during
'a
except for memory within Unsafe<U>
.
Drawbacks
Today’s unsafe pointers design is consistent with the borrowed pointers types in
Rust, using the mut
qualifier for a mutable pointer, and no qualifier for an
“immutable” pointer. Renaming the pointers would be divergence from this
consistency, and would also introduce a keyword that is not used elsewhere in
the language, const
.
Alternatives
-
The current
*mut T
type could be removed entirely, leaving only one unsafe pointer type,*T
. This will not allow FFI calls to take advantage of theconst T*
optimizations on the caller side of the function. Additionally, this may not accurately express to the programmer what a FFI API is intending to do. Note, however, that other variants of unsafe pointer types could likely be added in the future in a backwards-compatible way. -
More effort could be invested in auto-generating bindings, and hand-generating bindings could be greatly discouraged. This would maintain consistency with Rust pointer types, and it would allow APIs to usually being transcribed accurately by automating the process. It is unknown how realistic this solution is as it is currently not yet implemented. There may still be confusion as well that
*T
is not equivalent to C’sT*
.
Unresolved questions
-
How much can the compiler help out when coercing
&mut T
to*mut T
? As previously stated, the source pointer&mut T
is consumed during the coercion (it’s already a linear type), but this can lead to some unexpected results:extern { fn bar(a: *mut int, b: *mut int); } fn foo(a: &mut int) { unsafe { bar(&mut *a, &mut *a); } }
This code is invalid because it is creating two copies of the same mutable pointer, and the external function is unaware that the two pointers alias. The rule that the programmer has violated is that the pointer
*mut T
is only dereferenced during the lifetime of the&'a mut T
pointer. For example, here are the lifetimes spelled out:fn foo(a: &mut int) { unsafe { bar(&mut *a, &mut *a); // |-----| |-----| // | | // | Lifetime of second argument // Lifetime of first argument } }
Here it can be seen that it is impossible for the C code to safely dereference the pointers passed in because lifetimes don’t extend into the function call itself. The compiler could, in this case, extend the lifetime of a coerced pointer to follow the otherwise applied temporary rules for expressions.
In the example above, the compiler’s temporary lifetime rules would cause the first coercion to last for the entire lifetime of the call to
bar
, thereby disallowing the second reborrow because it has an overlapping lifetime with the first.It is currently an open question how necessary this sort of treatment will be, and this lifetime treatment will likely require a new RFC.
-
Will all pointer types in C need to have their own keyword in Rust for representation in the FFI?
-
To what degree will the compiler emit metadata about FFI function calls in order to take advantage of optimizations on the caller side of a function call? Do the theoretical wins justify the scope of this redesign? There is currently no concrete data measuring what benefits could be gained from informing optimization passes about const vs non-const pointers.
- Start Date: 2014-05-05
- RFC PR: rust-lang/rfcs#69
- Rust Issue: rust-lang/rust#14646
Summary
Add ASCII byte literals and ASCII byte string literals to the language, similar to the existing (Unicode) character and string literals. Before the RFC process was in place, this was discussed in #4334.
Motivation
Programs dealing with text usually should use Unicode,
represented in Rust by the str
and char
types.
In some cases however,
a program may be dealing with bytes that can not be interpreted as Unicode as a whole,
but still contain ASCII compatible bits.
For example, the HTTP protocol was originally defined as Latin-1, but in practice different pieces of the same request or response can use different encodings. The PDF file format is mostly ASCII, but can contain UTF-16 strings and raw binary data.
There is a precedent at least in Python, which has both Unicode and byte strings.
Drawbacks
The language becomes slightly more complex, although that complexity should be limited to the parser.
Detailed design
Using terminology from the Reference Manual:
Extend the syntax of expressions and patterns to add
byte literals of type u8
and
byte string literals of type &'static [u8]
(or [u8]
, post-DST).
They are identical to the existing character and string literals, except that:
- They are prefixed with a
b
(for “binary”), to distinguish them. This is similar to ther
prefix for raw strings. - Unescaped code points in the body must be in the ASCII range: U+0000 to U+007F.
'\x5c' 'u' hex_digit 4
and'\x5c' 'U' hex_digit 8
escapes are not allowed.'\x5c' 'x' hex_digit 2
escapes represent a single byte rather than a code point. (They are the only way to express a non-ASCII byte.)
Examples: b'A' == 65u8
, b'\t' == 9u8
, b'\xFF' == 0xFFu8
,
b"A\t\xFF" == [65u8, 9, 0xFF]
Assuming buffer
of type &[u8]
match buffer[i] {
b'a' .. b'z' => { /* ... */ }
c => { /* ... */ }
}
Alternatives
Status quo: patterns must use numeric literals for ASCII values, or (for a single byte, not a byte string) cast to char
match buffer[i] {
c @ 0x61 .. 0x7A => { /* ... */ }
c => { /* ... */ }
}
match buffer[i] as char {
// `c` is of the wrong type!
c @ 'a' .. 'z' => { /* ... */ }
c => { /* ... */ }
}
Another option is to change the syntax so that macros such as
bytes!()
can be used in patterns, and add a byte!()
macro:
match buffer[i] {
c @ byte!('a') .. byte!('z') => { /* ... */ }
c => { /* ... */ }
}q
This RFC was written to align the syntax with Python,
but there could be many variations such as using a different prefix (maybe a
for ASCII),
or using a suffix instead (maybe u8
, as in integer literals).
The code points from syntax could be encoded as UTF-8 rather than being mapped to bytes of the same value, but assuming UTF-8 is not always appropriate when working with bytes.
See also previous discussion in #4334.
Unresolved questions
Should there be “raw byte string” literals?
E.g. pdf_file.write(rb"<< /Title (FizzBuzz \(Part one\)) >>")
Should control characters (U+0000 to U+001F) be disallowed in syntax? This should be consistent across all kinds of literals.
Should the bytes!()
macro be removed in favor of this?
- Start Date: 2014-05-07
- RFC PR: rust-lang/rfcs#71
- Rust Issue: rust-lang/rust#14181
Summary
Allow block expressions in statics, as long as they only contain items and a trailing const expression.
Example:
static FOO: uint = { 100 };
static BAR: fn() -> int = {
fn hidden() -> int {
42
}
hidden
};
Motivation
This change allows defining items as part of a const expression,
and evaluating to a value using them.
This is mainly useful for macros, as it allows hiding complex machinery behind something
that expands to a value, but also enables using unsafe {}
blocks in a static initializer.
Real life examples include the regex!
macro, which currently expands to a block containing a
function definition and a value, and would be usable in a static with this.
Another example would be to expose a static reference to a fixed memory address by
dereferencing a raw pointer in a const expr, which is useful in
embedded and kernel, but requires a unsafe
block to do.
The outcome of this is that one additional expression type becomes valid as a const expression, with semantics that are a strict subset of its equivalent in a function.
Drawbacks
Block expressions in a function are usually just used to run arbitrary code before evaluating to a value. Allowing them in statics without allowing code execution might be confusing.
Detailed design
A branch implementing this feature can be found at https://github.com/Kimundi/rust/tree/const_block.
It mainly involves the following changes:
- const check now allows block expressions in statics:
- All statements that are not item declarations lead to an compile error.
- trans and const eval are made aware of block expressions:
- A trailing expression gets evaluated as a constant.
- A missing trailing expressions is treated as a unit value.
- trans is made to recurse into static expressions to generate possible items.
Things like privacy/reachability of definitions inside a static block are already handled more generally at other places, as the situation is very similar to a regular function.
The branch also includes tests that show how this feature works in practice.
Alternatives
Because this feature is a straight forward extension of the valid const expressions, it already causes a very minimal impact on the language, with most alternative ways of enabling the same benefits being more complex.
For example, a expression AST node that can include items but is only usable from procedural macros could be added.
Not having this feature would not prevent anything interesting from getting implemented, but it would lead to less nice looking solutions.
For example, a comparison between static-supporting regex!
with and without this feature:
// With this feature, you can just initialize a static:
static R: Regex = regex!("[0-9]");
// Without it, the static needs to be generated by the
// macro itself, alongside all generated items:
regex! {
static R = "[0-9]";
}
Unresolved questions
None so far.
- Start Date: 2014-05-17
- RFC PR: rust-lang/rfcs#79
- Rust Issue: rust-lang/rust#14309
Summary
Leave structs with unspecified layout by default like enums, for
optimisation purposes. Use something like #[repr(C)]
to expose C
compatible layout.
Motivation
The members of a struct are always laid in memory in the order in which they were specified, e.g.
struct A {
x: u8,
y: u64,
z: i8,
w: i64,
}
will put the u8
first in memory, then the u64
, the i8
and lastly
the i64
. Due to the alignment requirements of various types padding
is often required to ensure the members start at an appropriately
aligned byte. Hence the above struct is not 1 + 8 + 1 + 8 == 18
bytes, but rather 1 + 7 + 8 + 1 + 7 + 8 == 32
bytes, since it is
laid out like
#[packed] // no automatically inserted padding
struct AFull {
x: u8,
_padding1: [u8, .. 7],
y: u64,
z: i8,
_padding2: [u8, .. 7],
w: i64
}
If the fields were reordered to
struct B {
y: u64,
w: i64,
x: u8,
i: i8
}
then the struct is (strictly) only 18 bytes (but the alignment
requirements of u64
forces it to take up 24).
Having an undefined layout does allow for possible security
improvements, like randomising struct fields, but this can trivially
be done with a syntax extension that can be attached to a struct to
reorder the fields in the AST itself. That said, there may be benefits
from being able to randomise all structs in a program
automatically/for testing, effectively fuzzing code (especially
unsafe
code).
Notably, Rust’s enum
s already have undefined layout, and provide the
#[repr]
attribute to control layout more precisely (specifically,
selecting the size of the discriminant).
Drawbacks
Forgetting to add #[repr(C)]
for a struct intended for FFI use can
cause surprising bugs and crashes. There is already a lint for FFI use
of enum
s without a #[repr(...)]
attribute, so this can be extended
to include structs.
Having an unspecified (or otherwise non-C-compatible) layout by
default makes interfacing with C slightly harder. A particularly bad
case is passing to C a struct from an upstream library that doesn’t
have a repr(C)
attribute. This situation seems relatively similar to
one where an upstream library type is missing an implementation of a
core trait e.g. Hash
if one wishes to use it as a hashmap key.
It is slightly better if structs had a specified-but-C-incompatible layout, and one has control over the C interface, because then one can manually arrange the fields in the C definition to match the Rust order.
That said, this scenario requires:
- Needing to pass a Rust struct into C/FFI code, where that FFI code actually needs to use things from the struct, rather than just pass it through, e.g., back into a Rust callback.
- The Rust struct is defined upstream & out of your control, and not intended for use with C code.
- The C/FFI code is designed by someone other than that vendor, or otherwise not designed for use with the Rust struct (or else it is a bug in the vendor’s library that the Rust struct can’t be sanely passed to C).
Detailed design
A struct declaration like
struct Foo {
x: T,
y: U,
...
}
has no fixed layout, that is, a compiler can choose whichever order of fields it prefers.
A fixed layout can be selected with the #[repr]
attribute
#[repr(C)]
struct Foo {
x: T,
y: U,
...
}
This will force a struct to be laid out like the equivalent definition in C.
There would be a lint for the use of non-repr(C)
structs in related
FFI definitions, for example:
struct UnspecifiedLayout {
// ...
}
#[repr(C)]
struct CLayout {
// ...
}
extern {
fn foo(x: UnspecifiedLayout); // warning: use of non-FFI-safe struct in extern declaration
fn bar(x: CLayout); // no warning
}
extern "C" fn foo(x: UnspecifiedLayout) { } // warning: use of non-FFI-safe struct in function with C abi.
Alternatives
- Have non-C layouts opt-in, via
#[repr(smallest)]
and#[repr(random)]
(or similar). - Have layout defined, but not declaration order (like Java(?)), for
example, from largest field to smallest, so
u8
fields get placed last, and[u8, .. 1000000]
fields get placed first. The#[repr]
attributes would still allow for selecting declaration-order layout.
Unresolved questions
- How does this interact with binary compatibility of dynamic libraries?
- How does this interact with DST, where some fields have to be at the
end of a struct? (Just always lay-out unsized fields last?
(i.e. after monomorphisation if a field was originally marked
Sized?
then it needs to be last).)
- Start Date: 2014-05-21
- RFC PR: rust-lang/rfcs#85
- Rust Issue: rust-lang/rust#14473
Summary
Allow macro expansion in patterns, i.e.
match x {
my_macro!() => 1,
_ => 2,
}
Motivation
This is consistent with allowing macros in expressions etc. It’s also a year-old open issue.
I have implemented this feature already and I’m using it to condense some ubiquitous patterns in the HTML parser I’m writing. This makes the code more concise and easier to cross-reference with the spec.
Drawbacks / alternatives
A macro invocation in this position:
match x {
my_macro!()
could potentially expand to any of three different syntactic elements:
- A pattern, i.e.
Foo(x)
- The left side of a
match
arm, i.e.Foo(x) | Bar(x) if x > 5
- An entire
match
arm, i.e.Foo(x) | Bar(x) if x > 5 => 1
This RFC proposes only the first of these, but the others would be more useful in some cases. Supporting multiple of the above would be significantly more complex.
Another alternative is to use a macro for the entire match
expression, e.g.
my_match!(x {
my_new_syntax => 1,
_ => 2,
})
This doesn’t involve any language changes, but requires writing a complicated procedural macro. (My sustained attempts to do things like this with MBE macros have all failed.) Perhaps I could alleviate some of the pain with a library for writing match
-like macros, or better use of the existing parser in libsyntax
.
The my_match!
approach is also not very composable.
Another small drawback: rustdoc
can’t document the name of a function argument which is produced by a pattern macro.
Unresolved questions
None, as far as I know.
- Start Date: 2014-05-22
- RFC PR: rust-lang/rfcs#86
- Rust Issue: rust-lang/rust#14637
Summary
Generalize the #[macro_registrar]
feature so it can register other kinds of compiler plugins.
Motivation
I want to implement loadable lints and use them for project-specific static analysis passes in Servo. Landing this first will allow more evolution of the plugin system without breaking source compatibility for existing users.
Detailed design
To register a procedural macro in current Rust:
use syntax::ast::Name;
use syntax::parse::token;
use syntax::ext::base::{SyntaxExtension, BasicMacroExpander, NormalTT};
#[macro_registrar]
pub fn macro_registrar(register: |Name, SyntaxExtension|) {
register(token::intern("named_entities"),
NormalTT(box BasicMacroExpander {
expander: named_entities::expand,
span: None
},
None));
}
I propose an interface like
use syntax::parse::token;
use syntax::ext::base::{BasicMacroExpander, NormalTT};
use rustc::plugin::Registry;
#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
reg.register_macro(token::intern("named_entities"),
NormalTT(box BasicMacroExpander {
expander: named_entities::expand,
span: None
},
None));
}
Then the struct Registry
could provide additional methods such as register_lint
as those features are implemented.
It could also provide convenience methods:
use rustc::plugin::Registry;
#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
reg.register_simple_macro("named_entities", named_entities::expand);
}
phase(syntax)
becomes phase(plugin)
, with the former as a deprecated synonym that warns. This is to avoid silent breakage of the very common #[phase(syntax)] extern crate log
.
We only need one phase of loading plugin crates, even though the plugins we load may be used at different points (or not at all).
Drawbacks
Breaking change for existing procedural macros.
More moving parts.
Registry
is provided by librustc
, because it will have methods for registering lints and other librustc
things. This means that syntax extensions must link librustc
, when before they only needed libsyntax
(but could link librustc
anyway if desired). This was discussed on the RFC PR and the Rust PR and on IRC.
#![feature(macro_registrar)]
becomes unknown, contradicting a comment in feature_gate.rs
:
This list can never shrink, it may only be expanded (in order to prevent old programs from failing to compile)
Since when do we ensure that old programs will compile? ;) The #[macro_registrar]
attribute wouldn’t work anyway.
Alternatives
We could add #[lint_registrar]
etc. alongside #[macro_registrar]
. This seems like it will produce more duplicated effort all around. It doesn’t provide convenience methods, and it won’t support API evolution as well.
We could support the old #[macro_registrar]
by injecting an adapter shim. This is significant extra work to support a feature with no stability guarantee.
Unresolved questions
Naming bikeshed.
What set of convenience methods should we provide?
- Start Date: 2014-05-22
- RFC PR: rust-lang/rfcs#87
- Rust Issue: rust-lang/rust#12778
Summary
Bounds on trait objects should be separated with +
.
Motivation
With DST there is an ambiguity between the following two forms:
trait X {
fn f(foo: b);
}
and
trait X {
fn f(Trait: Share);
}
See Rust issue #12778 for details.
Also, since kinds are now just built-in traits, it makes sense to treat a bounded trait object as just a combination of traits. This could be extended in the future to allow objects consisting of arbitrary trait combinations.
Detailed design
Instead of :
in trait bounds for first-class traits (e.g. &Trait:Share + Send
), we use +
(e.g. &Trait + Share + Send
).
+
will not be permitted in as
without parentheses. This will be done via a special restriction in the type grammar: the special TYPE
production following as
will be the same as the regular TYPE
production, with the exception that it does not accept +
as a binary operator.
Drawbacks
-
It may be that
+
is ugly. -
Adding a restriction complicates the type grammar more than I would prefer, but the community backlash against the previous proposal was overwhelming.
Alternatives
The impact of not doing this is that the inconsistencies and ambiguities above remain.
Unresolved questions
Where does the 'static
bound fit into all this?
- Start Date: 2014-05-23
- RFC PR: rust-lang/rfcs#89
- Rust Issue: rust-lang/rust#14067
Summary
Allow users to load custom lints into rustc
, similar to loadable syntax extensions.
Motivation
There are many possibilities for user-defined static checking:
- Enforcing correct usage of Servo’s JS-managed pointers
- lilyball’s use case: checking that
rust-lua
functions which calllongjmp
never have destructors on stack variables - Enforcing a company or project style guide
- Detecting common misuses of a library, e.g. expensive or non-idiomatic constructs
- In cryptographic code, annotating which variables contain secrets and then forbidding their use in variable-time operations or memory addressing
Existing project-specific static checkers include:
- A Clang plugin that detects misuse of GLib and GObject
- A GCC plugin (written in Python!) that detects misuse of the CPython extension API
- Sparse, which checks Linux kernel code for issues such as mixing up userspace and kernel pointers (often exploitable for privilege escalation)
We should make it easy to build such tools and integrate them with an existing Rust project.
Detailed design
In rustc::lint
(which today is rustc::middle::lint
):
pub struct Lint {
/// An identifier for the lint, written with underscores,
/// e.g. "unused_imports".
pub name: &'static str,
/// Default level for the lint.
pub default_level: Level,
/// Description of the lint or the issue it detects,
/// e.g. "imports that are never used"
pub desc: &'static str,
}
#[macro_export]
macro_rules! declare_lint ( ($name:ident, $level:ident, $desc:expr) => (
static $name: &'static ::rustc::lint::Lint
= &::rustc::lint::Lint {
name: stringify!($name),
default_level: ::rustc::lint::$level,
desc: $desc,
};
))
pub type LintArray = &'static [&'static Lint];
#[macro_export]
macro_rules! lint_array ( ($( $lint:expr ),*) => (
{
static array: LintArray = &[ $( $lint ),* ];
array
}
))
pub trait LintPass {
fn get_lints(&self) -> LintArray;
fn check_item(&mut self, cx: &Context, it: &ast::Item) { }
fn check_expr(&mut self, cx: &Context, e: &ast::Expr) { }
...
}
pub type LintPassObject = Box<LintPass: 'static>;
To define a lint:
#![crate_id="lipogram"]
#![crate_type="dylib"]
#![feature(phase, plugin_registrar)]
extern crate syntax;
// Load rustc as a plugin to get macros
#[phase(plugin, link)]
extern crate rustc;
use syntax::ast;
use syntax::parse::token;
use rustc::lint::{Context, LintPass, LintPassObject, LintArray};
use rustc::plugin::Registry;
declare_lint!(letter_e, Warn, "forbid use of the letter 'e'")
struct Lipogram;
impl LintPass for Lipogram {
fn get_lints(&self) -> LintArray {
lint_array!(letter_e)
}
fn check_item(&mut self, cx: &Context, it: &ast::Item) {
let name = token::get_ident(it.ident);
if name.get().contains_char('e') || name.get().contains_char('E') {
cx.span_lint(letter_e, it.span, "item name contains the letter 'e'");
}
}
}
#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
reg.register_lint_pass(box Lipogram as LintPassObject);
}
A pass which defines multiple lints will have e.g. lint_array!(deprecated, experimental, unstable)
.
To use a lint when compiling another crate:
#![feature(phase)]
#[phase(plugin)]
extern crate lipogram;
fn hello() { }
fn main() { hello() }
And you will get
test.rs:6:1: 6:15 warning: item name contains the letter 'e', #[warn(letter_e)] on by default
test.rs:6 fn hello() { }
^~~~~~~~~~~~~~
Internally, lints are identified by the address of a static Lint
. This has a number of benefits:
- The linker takes care of assigning unique IDs, even with dynamically loaded plugins.
- A typo writing a lint ID is usually a compiler error, unlike with string IDs.
- The ability to output a given lint is controlled by the usual visibility mechanism. Lints defined within
rustc
use the same infrastructure and will simply export theirLint
s if other parts of the compiler need to output those lints. - IDs are small and easy to hash.
- It’s easy to go from an ID to name, description, etc.
User-defined lints are controlled through the usual mechanism of attributes and the -A -W -D -F
flags to rustc
. User-defined lints will show up in -W help
if a crate filename is also provided; otherwise we append a message suggesting to re-run with a crate filename.
See also the full demo.
Drawbacks
This increases the amount of code in rustc
to implement lints, although it makes each individual lint much easier to understand in isolation.
Loadable lints produce more coupling of user code to rustc
internals (with no official stability guarantee, of course).
There’s no scoping / namespacing of the lint name strings used by attributes and compiler flags. Attempting to register a lint with a duplicate name is an error at registration time.
The use of &'static
means that lint plugins can’t dynamically generate the set of lints based on some external resource.
Alternatives
We could provide a more generic mechanism for user-defined AST visitors. This could support other use cases like code transformation. But it would be harder to use, and harder to integrate with the lint infrastructure.
It would be nice to magically find all static Lint
s in a crate, so we don’t need get_lints
. Is this worth adding another attribute and another crate metadata type? The plugin::Registry
mechanism was meant to avoid such a proliferation of metadata types, but it’s not as declarative as I would like.
Unresolved questions
Do we provide guarantees about visit order for a lint, or the order of multiple lints defined in the same crate? Some lints may require multiple passes.
Should we enforce (while running lints) that each lint printed with span_lint
was registered by the corresponding LintPass
? Users who particularly care can already wrap lints in modules and use visibility to enforce this statically.
Should we separate registering a lint pass from initializing / constructing the value implementing LintPass
? This would support a future where a single rustc
invocation can compile multiple crates and needs to reset lint state.
- Start Date: 2014-05-23
- RFC PR: rust-lang/rfcs#90
- Rust Issue: rust-lang/rust#14504
Summary
Simplify Rust’s lexical syntax to make tooling easier to use and easier to define.
Motivation
Rust’s lexer does a lot of work. It un-escapes escape sequences in string and character literals, and parses numeric literals of 4 different bases. It also strips comments, which is sensible, but can be undesirable for pretty printing or syntax highlighting without hacks. Since many characters are allowed in strings both escaped and raw (tabs, newlines, and unicode characters come to mind), after lexing it is impossible to tell if a given character was escaped or unescaped in the source, making the lexer difficult to test against a model.
Detailed design
The following (antlr4) grammar completely describes the proposed lexical syntax:
lexer grammar RustLexer;
/* import Xidstart, Xidcont; */
/* Expression-operator symbols */
EQ : '=' ;
LT : '<' ;
LE : '<=' ;
EQEQ : '==' ;
NE : '!=' ;
GE : '>=' ;
GT : '>' ;
ANDAND : '&&' ;
OROR : '||' ;
NOT : '!' ;
TILDE : '~' ;
PLUS : '+' ;
MINUS : '-' ;
STAR : '*' ;
SLASH : '/' ;
PERCENT : '%' ;
CARET : '^' ;
AND : '&' ;
OR : '|' ;
SHL : '<<' ;
SHR : '>>' ;
BINOP
: PLUS
| MINUS
| STAR
| PERCENT
| CARET
| AND
| OR
| SHL
| SHR
;
BINOPEQ : BINOP EQ ;
/* "Structural symbols" */
AT : '@' ;
DOT : '.' ;
DOTDOT : '..' ;
DOTDOTDOT : '...' ;
COMMA : ',' ;
SEMI : ';' ;
COLON : ':' ;
MOD_SEP : '::' ;
LARROW : '->' ;
FAT_ARROW : '=>' ;
LPAREN : '(' ;
RPAREN : ')' ;
LBRACKET : '[' ;
RBRACKET : ']' ;
LBRACE : '{' ;
RBRACE : '}' ;
POUND : '#';
DOLLAR : '$' ;
UNDERSCORE : '_' ;
KEYWORD : STRICT_KEYWORD | RESERVED_KEYWORD ;
fragment STRICT_KEYWORD
: 'as'
| 'box'
| 'break'
| 'continue'
| 'crate'
| 'else'
| 'enum'
| 'extern'
| 'fn'
| 'for'
| 'if'
| 'impl'
| 'in'
| 'let'
| 'loop'
| 'match'
| 'mod'
| 'mut'
| 'once'
| 'proc'
| 'pub'
| 'ref'
| 'return'
| 'self'
| 'static'
| 'struct'
| 'super'
| 'trait'
| 'true'
| 'type'
| 'unsafe'
| 'use'
| 'virtual'
| 'while'
;
fragment RESERVED_KEYWORD
: 'alignof'
| 'be'
| 'const'
| 'do'
| 'offsetof'
| 'priv'
| 'pure'
| 'sizeof'
| 'typeof'
| 'unsized'
| 'yield'
;
// Literals
fragment HEXIT
: [0-9a-fA-F]
;
fragment CHAR_ESCAPE
: [nrt\\'"0]
| [xX] HEXIT HEXIT
| 'u' HEXIT HEXIT HEXIT HEXIT
| 'U' HEXIT HEXIT HEXIT HEXIT HEXIT HEXIT HEXIT HEXIT
;
LIT_CHAR
: '\'' ( '\\' CHAR_ESCAPE | ~[\\'\n\t\r] ) '\''
;
INT_SUFFIX
: 'i'
| 'i8'
| 'i16'
| 'i32'
| 'i64'
| 'u'
| 'u8'
| 'u16'
| 'u32'
| 'u64'
;
LIT_INTEGER
: [0-9][0-9_]* INT_SUFFIX?
| '0b' [01][01_]* INT_SUFFIX?
| '0o' [0-7][0-7_]* INT_SUFFIX?
| '0x' [0-9a-fA-F][0-9a-fA-F_]* INT_SUFFIX?
;
FLOAT_SUFFIX
: 'f32'
| 'f64'
| 'f128'
;
LIT_FLOAT
: [0-9][0-9_]* ('.' | ('.' [0-9][0-9_]*)? ([eE] [-+]? [0-9][0-9_]*)? FLOAT_SUFFIX?)
;
LIT_STR
: '"' ('\\\n' | '\\\r\n' | '\\' CHAR_ESCAPE | .)*? '"'
;
/* this is a bit messy */
fragment LIT_STR_RAW_INNER
: '"' .*? '"'
| LIT_STR_RAW_INNER2
;
fragment LIT_STR_RAW_INNER2
: POUND LIT_STR_RAW_INNER POUND
;
LIT_STR_RAW
: 'r' LIT_STR_RAW_INNER
;
fragment BLOCK_COMMENT
: '/*' (BLOCK_COMMENT | .)*? '*/'
;
COMMENT
: '//' ~[\r\n]*
| BLOCK_COMMENT
;
IDENT : XID_start XID_continue* ;
LIFETIME : '\'' IDENT ;
WHITESPACE : [ \r\n\t]+ ;
There are a few notable changes from today’s lexical syntax:
- Non-doc comments are not stripped. To compensate, when encountering a
COMMENT token the parser can check itself whether or not it’s a doc comment.
This can be done with a simple regex:
(//(/[^/]|!)|/\*(\*[^*]|!))
. - Numeric literals are not differentiated based on presence of type suffix, nor are they converted from binary/octal/hexadecimal to decimal, nor are underscores stripped. This can be done trivially in the parser.
- Character escapes are not unescaped. That is, if you write ‘\x20’, this
lexer will give you
LIT_CHAR('\x20')
rather thanLIT_CHAR(' ')
. The same applies to string literals.
The output of the lexer then becomes annotated spans – which part of the document corresponds to which token type. Even whitespace is categorized.
Drawbacks
Including comments and whitespace in the token stream is very non-traditional and not strictly necessary.
- Start Date: 2014-06-10
- RFC PR: rust-lang/rfcs#92
- Rust Issue: rust-lang/rust#14803
Summary
Do not identify struct literals by searching for :
. Instead define a sub-
category of expressions which excludes struct literals and re-define for
,
if
, and other expressions which take an expression followed by a block (or
non-terminal which can be replaced by a block) to take this sub-category,
instead of all expressions.
Motivation
Parsing by looking ahead is fragile - it could easily be broken if we allow :
to appear elsewhere in types (e.g., type ascription) or if we change struct
literals to not require the :
(e.g., if we allow empty structs to be written
with braces, or if we allow struct literals to unify field names to local
variable names, as has been suggested in the past and which we currently do for
struct literal patterns). We should also be able to give better error messages
today if users make these mistakes. More worryingly, we might come up with some
language feature in the future which is not predictable now and which breaks
with the current system.
Hopefully, it is pretty rare to use struct literals in these positions, so there should not be much fallout. Any problems can be easily fixed by assigning the struct literal into a variable. However, this is a backwards incompatible change, so it should block 1.0.
Detailed design
Here is a simplified version of a subset of Rust’s abstract syntax:
e ::= x
| e `.` f
| name `{` (x `:` e)+ `}`
| block
| `for` e `in` e block
| `if` e block (`else` block)?
| `|` pattern* `|` e
| ...
block ::= `{` (e;)* e? `}`
Parsing this grammar is ambiguous since x
cannot be distinguished from name
,
so e block
in the for expression is ambiguous with the struct literal
expression. We currently solve this by using lookahead to find a :
token in
the struct literal.
I propose the following adjustment:
e ::= e'
| name `{` (x `:` e)+ `}`
| `|` pattern* `|` e
| ...
e' ::= x
| e `.` f
| block
| `for` e `in` e' block
| `if` e' block (`else` block)?
| `|` pattern* `|` e'
| ...
block ::= `{` (e;)* e? `}`
e'
is just e without struct literal expressions. We use e'
instead of e
wherever e
is followed directly by block or any other non-terminal which may
have block as its first terminal (after any possible expansions).
For any expressions where a sub-expression is the final lexical element
(closures in the subset above, but also unary and binary operations), we require
two versions of the meta-expression - the normal one in e
and a version with
e'
for the final element in e'
.
Implementation would be simpler, we just add a flag to parser::restriction
called RESTRICT_BLOCK
or something, which puts us into a mode which reflects
e'
. We would drop in to this mode when parsing e'
position expressions and
drop out of it for all but the last sub-expression of an expression.
Drawbacks
It makes the formal grammar and parsing a little more complicated (although it is simpler in terms of needing less lookahead and avoiding a special case).
Alternatives
Don’t do this.
Allow all expressions but greedily parse non-terminals in these positions, e.g.,
for N {} {}
would be parsed as for (N {}) {}
. This seems worse because I
believe it will be much rarer to have structs in these positions than to have an
identifier in the first position, followed by two blocks (i.e., parse as (for N {}) {}
).
Unresolved questions
Do we need to expose this distinction anywhere outside of the parser? E.g., macros?
- Start Date: 2014-06-10
- RFC PR: rust-lang/rfcs#93
- Rust Issue: rust-lang/rust#14812
Summary
Remove localization features from format!, and change the set of escapes
accepted by format strings. The plural
and select
methods would be removed,
#
would no longer need to be escaped, and {{
/}}
would become escapes for
{
and }
, respectively.
Motivation
Localization is difficult to implement correctly, and doing so will likely not be done in the standard library, but rather in an external library. After talking with others much more familiar with localization, it has come to light that our ad-hoc “localization support” in our format strings are woefully inadequate for most real use cases of support for localization.
Instead of having a half-baked unused system adding complexity to the compiler and libraries, the support for this functionality would be removed from format strings.
Detailed design
The primary localization features that format!
supports today are the
plural
and select
methods inside of format strings. These methods are
choices made at format-time based on the input arguments of how to format a
string. This functionality would be removed from the compiler entirely.
As fallout of this change, the #
special character, a back reference to the
argument being formatted, would no longer be necessary. In that case, this
character no longer needs an escape sequence.
The new grammar for format strings would be as follows:
format_string := <text> [ format <text> ] *
format := '{' [ argument ] [ ':' format_spec ] '}'
argument := integer | identifier
format_spec := [[fill]align][sign]['#'][0][width]['.' precision][type]
fill := character
align := '<' | '>'
sign := '+' | '-'
width := count
precision := count | '*'
type := identifier | ''
count := parameter | integer
parameter := integer '$'
The current syntax can be found at http://doc.rust-lang.org/std/fmt/#syntax to see the diff between the two
Choosing a new escape sequence
Upon landing, there was a significant amount of discussion about the escape sequence that would be used in format strings. Some context can be found on some old pull requests, and the current escape mechanism has been the source of much confusion. With the removal of localization methods, and namely nested format directives, it is possible to reconsider the choices of escaping again.
The only two characters that need escaping in format strings are {
and }
.
One of the more appealing syntaxes for escaping was to double the character to
represent the character itself. This would mean that {{
is an escape for a {
character, while }}
would be an escape for a }
character.
Adopting this scheme would avoid clashing with Rust’s string literal escapes. There would be no “double escape” problem. More details on this can be found in the comments of an old PR.
Drawbacks
The localization methods of select/plural are genuinely used for applications that do not involve localization. For example, the compiler and rustdoc often use plural to easily create plural messages. Removing this functionality from format strings would impose a burden of likely dynamically allocating a string at runtime or defining two separate format strings.
Additionally, changing the syntax of format strings is quite an invasive change.
Raw string literals serve as a good use case for format strings that must escape
the {
and }
characters. The current system is arguably good enough to pass
with for today.
Alternatives
The major localization approach explored has been l20n, which has shown itself to be fairly incompatible with the way format strings work today. Different localization systems, however, have not been explored. Systems such as gettext would be able to leverage format strings quite well, but it was claimed that gettext for localization is inadequate for modern use-cases.
It is also an unexplored possibility whether the current format string syntax could be leveraged by l20n. It is unlikely that time will be allocated to polish off an localization library before 1.0, and it is currently seen as undesirable to have a half-baked system in the libraries rather than a first-class well designed system.
Unresolved questions
- Should localization support be left in
std::fmt
as a “poor man’s” implementation for those to use as they see fit?
- Start Date: 2014-06-01
- RFC PR: rust-lang/rfcs#100
- Rust Issue: rust-lang/rust#14987
Summary
Add a partial_cmp
method to PartialOrd
, analogous to cmp
in Ord
.
Motivation
The Ord::cmp
method is useful when working with ordered values. When the
exact ordering relationship between two values is required, cmp
is both
potentially more efficient than computing both a > b
and then a < b
and
makes the code clearer as well.
I feel that in the case of partial orderings, an equivalent to cmp
is even
more important. I’ve found that it’s very easy to accidentally make assumptions
that only hold true in the total order case (for example !(a < b) => a >= b
).
Explicitly matching against the possible results of the comparison helps keep
these assumptions from creeping in.
In addition, the current default implementation setup is a bit strange, as
implementations in the partial equality trait assume total equality. This
currently makes it easier to incorrectly implement PartialOrd
for types that
do not have a total ordering, and if PartialOrd
is separated from Ord
in a
way similar to this proposal,
the default implementations for PartialOrd
will need to be removed and an
implementation of the trait will require four repetitive implementations of
the required methods.
Detailed design
Add a method to PartialOrd
, changing the default implementations of the other
methods:
pub trait PartialOrd {
fn partial_cmp(&self, other: &Self) -> Option<Ordering>;
fn lt(&self, other: &Self) -> bool {
match self.partial_cmp(other) {
Some(Less) => true,
_ => false,
}
}
fn le(&self, other: &Self) -> bool {
match self.partial_cmp(other) {
Some(Less) | Some(Equal) => true,
_ => false,
}
}
fn gt(&self, other: &Self) -> bool {
match self.partial_cmp(other) {
Some(Greater) => true,
_ => false,
}
}
fn ge(&self, other: &Self) -> bool {
match self.partial_cmp(other) {
Some(Greater) | Some(Equal) => true,
_ => false,
}
}
}
Since almost all ordered types have a total ordering, the implementation of
partial_cmp
is trivial in most cases:
impl PartialOrd for Foo {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
This can be done automatically if/when RFC #48 or something like it is accepted and implemented.
Drawbacks
This does add some complexity to PartialOrd
. In addition, the more commonly
used methods (lt
, etc) may become more expensive than they would normally be
if their implementations call into partial_ord
.
Alternatives
We could invert the default implementations and have a default implementation
of partial_cmp
in terms of lt
and gt
. This may slightly simplify things
in current Rust, but it makes the default implementation less efficient than it
should be. It would also require more work to implement PartialOrd
once the
currently planned cmp
reform has finished as noted above.
partial_cmp
could just be called cmp
, but it seems like UFCS would need to
be implemented first for that to be workable.
Unresolved questions
We may want to add something similar to PartialEq
as well. I don’t know what
it would be called, though (maybe partial_eq
?):
// I don't feel great about these variant names, but `Equal` is already taken
// by `Ordering` which is in the same module.
pub enum Equality {
AreEqual,
AreUnequal,
}
pub trait PartialEq {
fn partial_eq(&self, other: &Self) -> Option<Equality>;
fn eq(&self, other: &Self) -> bool {
match self.partial_eq(other) {
Some(AreEqual) => true,
_ => false,
}
}
fn neq(&self, other: &Self) -> bool {
match self.partial_eq(other) {
Some(AreUnequal) => true,
_ => false,
}
}
}
- Start Date: 2014-06-05
- RFC PR: rust-lang/rfcs#107
- Rust Issue: rust-lang/rust#15287
Summary
Rust currently forbids pattern guards on match arms with move-bound variables. Allowing them would increase the applicability of pattern guards.
Motivation
Currently, if you attempt to use guards on a match arm with a move-bound variable, e.g.
struct A { a: Box<int> }
fn foo(n: int) {
let x = A { a: box n };
let y = match x {
A { a: v } if *v == 42 => v,
_ => box 0
};
}
you get an error:
test.rs:6:16: 6:17 error: cannot bind by-move into a pattern guard
test.rs:6 A { a: v } if *v == 42 => v,
^
This should be permitted in cases where the guard only accesses the moved value by reference or copies out of derived paths.
This allows for succinct code with less pattern matching duplication and a minimum number of copies at runtime. The lack of this feature was encountered by @kmcallister when developing Servo’s new HTML 5 parser.
Detailed design
This change requires all occurrences of move-bound pattern variables in the guard to be treated as paths to the values being matched before they are moved, rather than the moved values themselves. Any moves of matched values into the bound variables would occur on the control flow edge between the guard and the arm’s expression. There would be no changes to the handling of reference-bound pattern variables.
The arm would be treated as its own nested scope with respect to borrows, so that pattern-bound variables would be able to be borrowed and dereferenced freely in the guard, but these borrows would not be in scope in the arm’s expression. Since the guard dominates the expression and the move into the pattern-bound variable, moves of either the match’s head expression or any pattern-bound variables in the guard would trigger an error.
The following examples would be accepted:
struct A { a: Box<int> }
impl A {
fn get(&self) -> int { *self.a }
}
fn foo(n: int) {
let x = A { a: box n };
let y = match x {
A { a: v } if *v == 42 => v,
_ => box 0
};
}
fn bar(n: int) {
let x = A { a: box n };
let y = match x {
A { a: v } if x.get() == 42 => v,
_ => box 0
};
}
fn baz(n: int) {
let x = A { a: box n };
let y = match x {
A { a: v } if *v.clone() == 42 => v,
_ => box 0
};
}
This example would be rejected, due to a double move of v
:
struct A { a: Box<int> }
fn foo(n: int) {
let x = A { a: box n };
let y = match x {
A { a: v } if { drop(v); true } => v,
_ => box 0
};
}
This example would also be rejected, even though there is no use of the move-bound variable in the first arm’s expression, since the move into the bound variable would be moving the same value a second time:
enum VecWrapper { A(Vec<int>) }
fn foo(x: VecWrapper) -> uint {
match x {
A(v) if { drop(v); false } => 1,
A(v) => v.len()
}
}
There are issues with mutation of the bound values, but that is true without the changes proposed by this RFC, e.g. Rust issue #14684. The general approach to resolving that issue should also work with these proposed changes.
This would be implemented behind a feature(bind_by_move_pattern_guards)
gate
until we have enough experience with the feature to remove the feature gate.
Drawbacks
The current error message makes it more clear what the user is doing wrong, but if this change is made the error message for an invalid use of this feature (even if it were accidental) would indicate a use of a moved value, which might be more confusing.
This might be moderately difficult to implement in rustc
.
Alternatives
As far as I am aware, the only workarounds for the lack of this feature are to manually expand the control flow of the guard (which can quickly get messy) or use unnecessary copies.
Unresolved questions
This has nontrivial interaction with guards in arbitrary patterns as proposed in #99.
- Start Date: 2014-06-24
- RFC PR: rust-lang/rfcs#109
- Rust Issue: rust-lang/rust#14470
Summary
- Remove the
crate_id
attribute and knowledge of versions from rustc. - Add a
#[crate_name]
attribute similar to the old#[crate_id]
attribute - Filenames will no longer have versions, nor will symbols
- A new flag,
--extern
, will be used to override searching for external crates - A new flag,
-C metadata=foo
, used when hashing symbols
Motivation
The intent of CrateId and its support has become unclear over time as the
initial impetus, rustpkg
, has faded over time. With cargo
on the horizon,
doubts have been cast on the compiler’s support for dealing with crate
versions and friends. The goal of this RFC is to simplify the compiler’s
knowledge about the identity of a crate to allow cargo to do all the necessary
heavy lifting.
This new crate identification is designed to not compromise on the usability of the compiler independent of cargo. Additionally, all use cases support today with a CrateId should still be supported.
Detailed design
A new #[crate_name]
attribute will be accepted by the compiler, which is the
equivalent of the old #[crate_id]
attribute, except without the “crate id”
support. This new attribute can have a string value describe a valid crate name.
A crate name must be a valid rust identifier with the exception of allowing the
-
character after the first character.
#![crate_name = "foo"]
#![crate_type = "lib"]
pub fn foo() { /* ... */ }
Naming library filenames
Currently, rustc creates filenames for library following this pattern:
lib<name>-<version>-<hash>.rlib
The current scheme defines <hash>
to be the hash of the CrateId value. This
naming scheme achieves a number of goals:
- Libraries of the same name can exist next to one another if they have different versions.
- Libraries of the same name and version, but from different sources, can exist next to one another due to having different hashes.
- Rust libraries can have very privileged names such as
core
andstd
without worrying about polluting the global namespace of other system libraries.
One drawback of this scheme is that the output filename of the compiler is
unknown due to the <hash>
component. One must query rustc
itself to
determine the name of the library output.
Under this new scheme, the new output filenames by the compiler would be:
lib<name>.rlib
Note that both the <version>
and the <hash>
are missing by default. The
<version>
was removed because the compiler no longer knows about the version,
and the <hash>
was removed to make the output filename predictable.
The three original goals can still be satisfied with this simplified naming
scheme. As explained in the next section, the compiler’s “glob pattern” when
searching for a crate named foo
will be libfoo*.rlib
, which will help
rationalize some of these conclusions.
- Libraries of the same name can exist next to one another because they can be
manually renamed to have extra data after the
libfoo
, such as the version. - Libraries of the same name and version, but different source, can also exist
by modifying what comes after
libfoo
, such as including a hash. - Rust does not need to occupy a privileged namespace as the default rust installation would include hashes in all the filenames as necessary. More on this later.
Additionally, with a predictable filename output external tooling should be easier to write.
Loading crates
The goal of the crate loading phase of the compiler is to map a set of extern crate
statements to (dylib,rlib) pairs that are present on the filesystem. To
do this, the current system matches dependencies via the CrateId syntax:
extern crate json = "super-fast-json#0.1.0";
In today’s compiler, this directive indicates that the a filename of the form
libsuper-fast-json-0.1.0-<hash>.rlib
must be found to be a candidate. Further
checking happens once a candidate is found to ensure that it is indeed a rust
library.
Concerns have been raised that this key point of dependency management is where the compiler is doing work that is not necessarily its prerogative. In a cargo-driven world, versions are primarily managed in an external manifest, in addition to doing other various actions such as renaming packages at compile time.
One solution would be to add more version management to the compiler, but this
is seen as the compiler delving too far outside what it was initially tasked to
do. With this in mind, this is the new proposal for the extern crate
syntax:
extern crate json = "super-fast-json";
Notably, the CrateId is removed entirely, along with the version and path
associated with it. The string value of the extern crate
directive is still
optional (defaulting to the identifier), and the string must be a valid crate
name (as defined above).
The compiler’s searching and file matching logic would be altered to only match
crates based on name. If two versions of a crate are found, the compiler will
unconditionally emit an error. It will be up to the user to move the two
libraries on the filesystem and control the -L
flags to the compiler to enable
disambiguation.
This imples that when the compiler is searching for the crate named foo
, it
will search all of the lookup paths for files which match the pattern
libfoo*.{so,rlib}
. This is likely to return many false positives, but they
will be easily weeded out once the compiler realizes that there is no metadata
in the library.
This scheme is strictly less powerful than the previous, but it moves a good deal of logic from the compiler to cargo.
Manually specifying dependencies
Cargo is often seen as “expert mode” in its usage of the compiler. Cargo will always have prior knowledge about what exact versions of a library will be used for any particular dependency, as well as where the outputs are located.
If the compiler provided no support for loading crates beyond matching
filenames, it would limit many of cargo’s use cases. For example, cargo could
not compile a crate with two different versions of an upstream crate.
Additionally, cargo could not substitute libfast-json
for libslow-json
at
compile time (assuming they have the same API).
To accommodate an “expert mode” in rustc, the compiler will grow a new command line flag of the form:
--extern json=path/to/libjson
This directive will indicate that the library json
can be found at
path/to/libjson
. The file extension is not specified, and it is assume that
the rlib/dylib pair are located next to one another at this location (libjson
is the file stem).
This will enable cargo to drive how the compiler loads crates by manually specifying where files are located and exactly what corresponds to what.
Symbol mangling
Today, mangled symbols contain the version number at the end of the symbol itself. This was originally intended to tie into Linux’s ability to version symbols, but in retrospect this is generally viewed as over-ambitious as the support is not currently there, nor does it work on windows or OSX.
Symbols would no longer contain the version number anywhere within them. The
hash at the end of each symbol would only include the crate name and metadata
from the command line. Metadata from the command line will be passed via a new
command line flag, -C metadata=foo
, which specifies a string to hash.
The standard rust distribution
The standard distribution would continue to put hashes in filenames manually because the libraries are intended to occupy a privileged space on the system. The build system would manually move a file after it was compiled to the correct destination filename.
Drawbacks
-
The compiler is able to operate fairly well independently of cargo today, and this scheme would hamstring the compiler by limiting the number of “it just works” use cases. If cargo is not being used, build systems will likely have to start using
--extern
to specify dependencies if name conflicts or version conflicts arise between crates. -
This scheme still has redundancy in the list of dependencies with the external cargo manifest. The source code would no longer list versions, but the cargo manifest will contain the same identifier for each dependency that the source code will contain.
Alternatives
- The compiler could go in the opposite direction of this proposal, enhancing
extern crate
instead of simplifying it. The compiler could learn about things like version ranges and friends, while still maintaining flags to fine tune its behavior. It is unclear whether this increase in complexity will be paired with a large enough gain in usability of the compiler independent of cargo.
Unresolved questions
-
An implementation for the more advanced features of cargo does not currently exist, to it is unknown whether
--extern
will be powerful enough for cargo to satisfy all its use cases with. -
Are the string literal parts of
extern crate
justified? Allowing a string literal just for the-
character may be overkill.
- Start Date: 2014-06-09
- RFC PR: rust-lang/rfcs#111
- Rust Issue: rust-lang/rust#6515
Summary
Index
should be split into Index
and IndexMut
.
Motivation
Currently, the Index
trait is not suitable for most array indexing tasks. The slice functionality cannot be replicated using it, and as a result the new Vec
has to use .get()
and .get_mut()
methods.
Additionally, this simply follows the Deref
/DerefMut
split that has been implemented for a while.
Detailed design
We split Index
into two traits (borrowed from @nikomatsakis):
// self[element] -- if used as rvalue, implicitly a deref of the result
trait Index<E,R> {
fn index<'a>(&'a self, element: &E) -> &'a R;
}
// &mut self[element] -- when used as a mutable lvalue
trait IndexMut<E,R> {
fn index_mut<'a>(&'a mut self, element: &E) -> &'a mut R;
}
Drawbacks
-
The number of lang. items increases.
-
This design doesn’t support moving out of a vector-like object. This can be added backwards compatibly.
-
This design doesn’t support hash tables because there is no assignment operator. This can be added backwards compatibly.
Alternatives
The impact of not doing this is that the []
notation will not be available to Vec
.
Unresolved questions
None that I’m aware of.
- Start Date: 2014-06-09
- RFC PR: rust-lang/rfcs#112
- Rust Issue: rust-lang/rust#10504
Summary
Remove the coercion from Box<T>
to &mut T
from the language.
Motivation
Currently, the coercion between Box<T>
to &mut T
can be a hazard because it can lead to surprising mutation where it was not expected.
Detailed design
The coercion between Box<T>
and &mut T
should be removed.
Note that methods that take &mut self
can still be called on values of type Box<T>
without any special referencing or dereferencing. That is because the semantics of auto-deref and auto-ref conspire to make it work: the types unify after one autoderef followed by one autoref.
Drawbacks
Borrowing from Box<T>
to &mut T
may be convenient.
Alternatives
An alternative is to remove &T
coercions as well, but this was decided against as they are convenient.
The impact of not doing this is that the coercion will remain.
Unresolved questions
None.
- Start Date: 2014-07-29
- RFC PR: rust-lang/rfcs#114
- Rust Issue: rust-lang/rust#16095
Summary
- Convert function call
a(b, ..., z)
into an overloadable operator via the traitsFn<A,R>
,FnShare<A,R>
, andFnOnce<A,R>
, whereA
is a tuple(B, ..., Z)
of the typesB...Z
of the argumentsb...z
, andR
is the return type. The three traits differ in their self argument (&mut self
vs&self
vsself
). - Remove the
proc
expression form and type. - Remove the closure types (though the form lives on as syntactic sugar, see below).
- Modify closure expressions to permit specifying by-reference vs
by-value capture and the receiver type:
- Specifying by-reference vs by-value closures:
ref |...| expr
indicates a closure that captures upvars from the environment by reference. This is what closures do today and the behavior will remain unchanged, other than requiring an explicit keyword.|...| expr
will therefore indicate a closure that captures upvars from the environment by value. As usual, this is either a copy or move depending on whether the type of the upvar implementsCopy
.
- Specifying receiver mode (orthogonal to capture mode above):
|a, b, c| expr
is equivalent to|&mut: a, b, c| expr
|&mut: ...| expr
indicates that the closure implementsFn
|&: ...| expr
indicates that the closure implementsFnShare
|: a, b, c| expr
indicates that the closure implementsFnOnce
.
- Specifying by-reference vs by-value closures:
- Add syntactic sugar where
|T1, T2| -> R1
is translated to a reference to one of the fn traits as follows:|T1, ..., Tn| -> R
is translated toFn<(T1, ..., Tn), R>
|&mut: T1, ..., Tn| -> R
is translated toFn<(T1, ..., Tn), R>
|&: T1, ..., Tn| -> R
is translated toFnShare<(T1, ..., Tn), R>
|: T1, ..., Tn| -> R
is translated toFnOnce<(T1, ..., Tn), R>
One aspect of closures that this RFC does not describe is that we must permit trait references to be universally quantified over regions as closures are today. A description of this change is described below under Unresolved questions and the details will come in a forthcoming RFC.
Motivation
Over time we have observed a very large number of possible use cases for closures. The goal of this RFC is to create a unified closure model that encompasses all of these use cases.
Specific goals (explained in more detail below):
- Give control over inlining to users.
- Support closures that bind by reference and closures that bind by value.
- Support different means of accessing the closure environment,
corresponding to
self
,&self
, and&mut self
methods.
As a side benefit, though not a direct goal, the RFC reduces the size/complexity of the language’s core type system by unifying closures and traits.
The core idea: unifying closures and traits
The core idea of the RFC is to unify closures, procs, and traits. There are a number of reasons to do this. First, it simplifies the language, because closures, procs, and traits already served similar roles and there was sometimes a lack of clarity about which would be the appropriate choice. However, in addition, the unification offers increased expressiveness and power, because traits are a more generic model that gives users more control over optimization.
The basic idea is that function calls become an overridable operator.
Therefore, an expression like a(...)
will be desugar into an
invocation of one of the following traits:
trait Fn<A,R> {
fn call(&mut self, args: A) -> R;
}
trait FnShare<A,R> {
fn call_share(&self, args: A) -> R;
}
trait FnOnce<A,R> {
fn call_once(self, args: A) -> R;
}
Essentially, a(b, c, d)
becomes sugar for one of the following:
Fn::call(&mut a, (b, c, d))
FnShare::call_share(&a, (b, c, d))
FnOnce::call_once(a, (b, c, d))
To integrate with this, closure expressions are then translated into a fresh struct that implements one of those three traits. The precise trait is currently indicated using explicit syntax but may eventually be inferred.
This change gives user control over virtual vs static dispatch. This works in the same way as generic types today:
fn foo(x: &mut Fn<(int,),int>) -> int {
x(2) // virtual dispatch
}
fn foo<F:Fn<(int,),int>>(x: &mut F) -> int {
x(2) // static dispatch
}
The change also permits returning closures, which is not currently
possible (the example relies on the proposed impl
syntax from
rust-lang/rfcs#105):
fn foo(x: impl Fn<(int,),int>) -> impl Fn<(int,),int> {
|v| x(v * 2)
}
Basically, in this design there is nothing special about a closure.
Closure expressions are simply a convenient way to generate a struct
that implements a suitable Fn
trait.
Bind by reference vs bind by value
When creating a closure, it is now possible to specify whether the
closure should capture variables from its environment (“upvars”) by
reference or by value. The distinction is indicated using the leading
keyword ref
:
|| foo(a, b) // captures `a` and `b` by value
ref || foo(a, b) // captures `a` and `b` by reference, as today
Reasons to bind by value
Bind by value is useful when creating closures that will escape from
the stack frame that created them, such as task bodies (spawn(|| ...)
) or combinators. It is also useful for moving values out of a
closure, though it should be possible to enable that with bind by
reference as well in the future.
Reasons to bind by reference
Bind by reference is useful for any case where the closure is known not to escape the creating stack frame. This frequently occurs when using closures to encapsulate common control-flow patterns:
map.insert_or_update_with(key, value, || ...)
opt_val.unwrap_or_else(|| ...)
In such cases, the closure frequently wishes to read or modify local variables on the enclosing stack frame. Generally speaking, then, such closures should capture variables by-reference – that is, they should store a reference to the variable in the creating stack frame, rather than copying the value out. Using a reference allows the closure to mutate the variables in place and also avoids moving values that are simply read temporarily.
The vast majority of closures in use today are should be “by reference” closures. The only exceptions are those closures that wish to “move out” from an upvar (where we commonly use the so-called “option dance” today). In fact, even those closures could be “by reference” closures, but we will have to extend the inference to selectively identify those variables that must be moved and take those “by value”.
Detailed design
Closure expression syntax
Closure expressions will have the following form (using EBNF notation,
where []
denotes optional things and {}
denotes a comma-separated
list):
CLOSURE = ['ref'] '|' [SELF] {ARG} '|' ['->' TYPE] EXPR
SELF = ':' | '&' ':' | '&' 'mut' ':'
ARG = ID [ ':' TYPE ]
The optional keyword ref
is used to indicate whether this closure
captures by reference or by value.
Closures are always translated into a fresh struct type with one field
per upvar. In a by-value closure, the types of these fields will be
the same as the types of the corresponding upvars (modulo &mut
reborrows, see below). In a by-reference closure, the types of these
fields will be a suitable reference (&
, &mut
, etc) to the
variables being borrowed.
By-value closures
The default form for a closure is by-value. This implies that all
upvars which are referenced are copied/moved into the closure as
appropriate. There is one special case: if the type of the value to be
moved is &mut
, we will “reborrow” the value when it is copied into
the closure. That is, given an upvar x
of type &'a mut T
, the
value which is actually captured will have type &'b mut T
where 'b <= 'a
. This rule is consistent with our general treatment of &mut
,
which is to aggressively reborrow wherever possible; moreover, this
rule cannot introduce additional compilation errors, it can only make
more programs successfully typecheck.
By-reference closures
A by-reference closure is a convenience form in which values used in the closure are converted into references before being captured. By-reference closures are always rewritable into by-value closures if desired, but the rewrite can often be cumbersome and annoying.
Here is a (rather artificial) example of a by-reference closure in use:
let in_vec: Vec<int> = ...;
let mut out_vec: Vec<int> = Vec::new();
let opt_int: Option<int> = ...;
opt_int.map(ref |v| {
out_vec.push(v);
in_vec.fold(v, |a, &b| a + b)
});
This could be rewritten into a by-value closure as follows:
let in_vec: Vec<int> = ...;
let mut out_vec: Vec<int> = Vec::new();
let opt_int: Option<int> = ...;
opt_int.map({
let in_vec = &in_vec;
let out_vec = &mut in_vec;
|v| {
out_vec.push(v);
in_vec.fold(v, |a, &b| a + b)
}
})
In this case, the capture closed over two variables, in_vec
and
out_vec
. As you can see, the compiler automatically infers, for each
variable, how it should be borrowed and inserts the appropriate
capture.
In the body of a ref
closure, the upvars continue to have the same
type as they did in the outer environment. For example, the type of a
reference to in_vec
in the above example is always Vec<int>
,
whether or not it appears as part of a ref
closure. This is not only
convenient, it is required to make it possible to infer whether each
variable is borrowed as an &T
or &mut T
borrow.
Note that there are some cases where the compiler internally employs a
form of borrow that is not available in the core language,
&uniq
. This borrow does not permit aliasing (like &mut
) but does
not require mutability (like &
). This is required to allow
transparent closing over of &mut
pointers as
described in this blog post.
Evolutionary note: It is possible to evolve by-reference closures in the future in a backwards compatible way. The goal would be to cause more programs to type-check by default. Two possible extensions follow:
- Detect when values are moved and hence should be taken by value rather than by reference. (This is only applicable to once closures.)
- Detect when it is only necessary to borrow a sub-path. Imagine a
closure like
ref || use(&context.variable_map)
. Currently, this closure will borrowcontext
, even though it only uses the fieldvariable_map
. As a result, it is sometimes necessary to rewrite the closure to have the form{let v = &context.variable_map; || use(v)}
. In the future, however, we could extend the inference so that rather than borrowingcontext
to create the closure, we would borrowcontext.variable_map
directly.
Closure sugar in trait references
The current type for closures, |T1, T2| -> R
, will be repurposed as
syntactic sugar for a reference to the appropriate Fn
trait. This
shorthand be used any place that a trait reference is appropriate. The
full type will be written as one of the following:
<'a...'z> |T1...Tn|: K -> R
<'a...'z> |&mut: T1...Tn|: K -> R
<'a...'z> |&: T1...Tn|: K -> R
<'a...'z> |: T1...Tn|: K -> R
Each of which would then be translated into the following trait references, respectively:
<'a...'z> Fn<(T1...Tn), R> + K
<'a...'z> Fn<(T1...Tn), R> + K
<'a...'z> FnShare<(T1...Tn), R> + K
<'a...'z> FnOnce<(T1...Tn), R> + K
Note that the bound lifetimes 'a...'z
are not in scope for the bound
K
.
Drawbacks
This model is more complex than the existing model in some respects (but the existing model does not serve the full set of desired use cases).
Alternatives
There is one aspect of the design that is still under active discussion:
Introduce a more generic sugar. It was proposed that we could
introduce Trait(A, B) -> C
as syntactic sugar for Trait<(A,B),C>
rather than retaining the form |A,B| -> C
. This is appealing but
removes the correspondence between the expression form and the
corresponding type. One (somewhat open) question is whether there will
be additional traits that mirror fn types that might benefit from this
more general sugar.
Tweak trait names. In conjunction with the above, there is some
concern that the type name fn(A) -> B
for a bare function with no
environment is too similar to Fn(A) -> B
for a closure. To remedy
that, we could change the name of the trait to something like
Closure(A) -> B
(naturally the other traits would be renamed to
match).
Then there are a large number of permutations and options that were largely rejected:
Only offer by-value closures. We tried this and found it required a lot of painful rewrites of perfectly reasonable code.
Make by-reference closures the default. We felt this was
inconsistent with the language as a whole, which tends to make “by
value” the default (e.g., x
vs ref x
in patterns, x
vs &x
in
expressions, etc.).
Use a capture clause syntax that borrows individual variables. “By
value” closures combined with let
statements already serve this
role. Simply specifying “by-reference closure” also gives us room to
continue improving inference in the future in a backwards compatible
way. Moreover, the syntactic space around closures expressions is
extremely constrained and we were unable to find a satisfactory
syntax, particularly when combined with self-type annotations.
Finally, if we decide we do want the ability to have “mostly
by-value” closures, we can easily extend the current syntax by writing
something like (ref x, ref mut y) || ...
etc.
Retain the proc expression form. It was proposed that we could
retain the proc
expression form to specify a by-value closure and
have ||
expressions be by-reference. Frankly, the main objection to
this is that nobody likes the proc
keyword.
Use variadic generics in place of tuple arguments. While variadic generics are an interesting addition in their own right, we’d prefer not to introduce a dependency between closures and variadic generics. Having all arguments be placed into a tuple is also a simpler model overall. Moreover, native ABIs on platforms of interest treat a structure passed by value identically to distinct arguments. Finally, given that trait calls have the “Rust” ABI, which is not specified, we can always tweak the rules if necessary (though there are advantages for tooling when the Rust ABI closely matches the native ABI).
Use inference to determine the self type of a closure rather than an
annotation. We retain this option for future expansion, but it is
not clear whether we can always infer the self type of a
closure. Moreover, using inference rather a default raises the
question of what to do for a type like |int| -> uint
, where
inference is not possible.
Default to something other than &mut self
. It is our belief that
this is the most common use case for closures.
Transition plan
TBD. pcwalton is working furiously as we speak.
Unresolved questions
What relationship should there be between the closure
traits? On the one hand, there is clearly a relationship between the
traits. For example, given a FnShare
, one can easily implement
Fn
:
impl<A,R,T:FnShare<A,R>> Fn<A,R> for T {
fn call(&mut self, args: A) -> R {
(&*self).call_share(args)
}
}
Similarly, given a Fn
or FnShare
, you can implement FnOnce
. From
this, one might derive a subtrait relationship:
trait FnOnce { ... }
trait Fn : FnOnce { ... }
trait FnShare : Fn { ... }
Employing this relationship, however, would require that any manual
implement of FnShare
or Fn
must implement adapters for the other
two traits, since a subtrait cannot provide a specialized default of
supertrait methods (yet?). On the other hand, having no relationship
between the traits limits reuse, at least without employing explicit
adapters.
Other alternatives that have been proposed to address the problem:
-
Use impls to implement the fn traits in terms of one another, similar to what is shown above. The problem is that we would need to implement
FnOnce
both for allT
whereT:Fn
and for allT
whereT:FnShare
. This will yield coherence errors unless we extend the language with a means to declare traits as mutually exclusive (which might be valuable, but no such system has currently been proposed nor agreed upon). -
Have the compiler implement multiple traits for a single closure. As with supertraits, this would require manual implements to implement multiple traits. It would also require generic users to write
T:Fn+FnMut
or else employ an explicit adapter. On the other hand, it preserves the “one method per trait” rule described below.
Can we optimize away the trait vtable? The runtime representation
of a reference &Trait
to a trait object (and hence, under this
proposal, closures as well) is a pair of pointers (data, vtable)
. It
has been proposed that we might be able to optimize this
representation to (data, fnptr)
so long as Trait
has a single
function. This slightly improves the performance of invoking the
function as one need not indirect through the vtable. The actual
implications of this on performance are unclear, but it might be a
reason to keep the closure traits to a single method.
Closures that are quantified over lifetimes
A separate RFC is needed to describe bound lifetimes in trait
references. For example, today one can write a type like <'a> |&'a A| -> &'a B
, which indicates a closure that takes and returns a
reference with the same lifetime specified by the caller at each
call-site. Note that a trait reference like Fn<(&'a A), &'a B>
,
while syntactically similar, does not have the same meaning because
it lacks the universal quantifier <'a>
. Therefore, in the second
case, 'a
refers to some specific lifetime 'a
, rather than being a
lifetime parameter that is specified at each callsite. The high-level
summary of the change therefore is to permit trait references like
<'a> Fn<(&'a A), &'a B>
; in this case, the value of <'a>
will be
specified each time a method or other member of the trait is accessed.
- Start Date: 2014-06-11
- RFC PR: rust-lang/rfcs#115
- Rust Issue: rust-lang/rust#6023
Summary
Currently we use inference to find the current type of
otherwise-unannotated integer literals, and when that fails the type
defaults to int
. This is often felt to be potentially error-prone
behavior.
This proposal removes the integer inference fallback and strengthens the types required for several language features that interact with integer inference.
Motivation
With the integer fallback, small changes to code can change the inferred type in unexpected ways. It’s not clear how big a problem this is, but previous experiments1 indicate that removing the fallback has a relatively small impact on existing code, so it’s reasonable to back off of this feature in favor of more strict typing.
See also https://github.com/mozilla/rust/issues/6023.
Detailed design
The primary change here is that, when integer type inference fails,
the compiler will emit an error instead of assigning the value the
type int
.
This change alone will cause a fair bit of existing code to be unable to type check because of lack of constraints. To add more constraints and increase likelihood of unification, we ‘tighten’ up what kinds of integers are required in some situations:
- Array repeat counts must be uint (
[expr, .. count]
) - << and >> require uint when shifting integral types
Finally, inference for as
will be modified to track the types
a value is being cast to for cases where the value being cast
is unconstrained, like 0 as u8
.
Treatment of enum discriminants will need to change:
enum Color { Red = 0, Green = 1, Blue = 2 }
Currently, an unsuffixed integer defaults to int
. Instead, we will
only require enum discriminants primitive integers of unspecified
type; assigning an integer to an enum will behave as if casting from
from the type of the integer to an unsigned integer with the size of
the enum discriminant.
Drawbacks
This will force users to type hint somewhat more often. In particular, ranges of unsigned ints may need to be type-hinted:
for _ in range(0u, 10) { }
Alternatives
Do none of this.
Unresolved questions
- If we’re putting new restrictions on shift operators, should we change the traits, or just make the primitives special?
- Start Date: 2014-06-12
- RFC PR #: https://github.com/rust-lang/rfcs/pull/116
- Rust Issue #: https://github.com/rust-lang/rust/issues/16464
Summary
Remove or feature gate the shadowing of view items on the same scope level, in order to have less complicated semantic and be more future proof for module system changes or experiments.
This means the names brought in scope by extern crate
and use
may never collide with
each other, nor with any other item (unless they live in different namespaces).
Eg, this will no longer work:
extern crate foo;
use foo::bar::foo; // ERROR: There is already a module `foo` in scope
Shadowing would still be allowed in case of lexical scoping, so this continues to work:
extern crate foo;
fn bar() {
use foo::bar::foo; // Shadows the outer foo
foo::baz();
}
Definitions
Due to a certain lack of official, clearly defined semantics and terminology, a list of relevant definitions is included:
-
Scope A scope in Rust is basically defined by a block, following the rules of lexical scoping:
scope 1 (visible: scope 1) { scope 1-1 (visible: scope 1, scope 1-1) { scope 1-1-1 (visible: scope 1, scope 1-1, scope 1-1-1) } scope 1-1 { scope 1-1-2 } scope 1-1 } scope 1
Blocks include block expressions,
fn
items andmod
items, but not things likeextern
,enum
orstruct
. Additionally,mod
is special in that it isolates itself from parent scopes. -
Scope Level Anything with the same name in the example above is on the same scope level. In a scope level, all names defined in parent scopes are visible, but can be shadowed by a new definition with the same name, which will be in scope for that scope itself and all its child scopes.
-
Namespace Rust has different namespaces, and the scoping rules apply to each one separately. The exact number of different namespaces is not well defined, but they are roughly
- types (
enum Foo {}
) - modules (
mod foo {}
) - item values (
static FOO: uint = 0;
) - local values (
let foo = 0;
) - lifetimes (
impl<'a> ...
) - macros (
macro_rules! foo {...}
)
- types (
-
Definition Item Declarations that create new entities in a crate are called (by the author) definition items. They include
struct
,enum
,mod
,fn
, etc. Each of them creates a name in the type, module, item value or macro namespace in the same scope level they are written in. -
View Item Declarations that just create aliases to existing declarations in a crate are called view items. They include
use
andextern crate
, and also create a name in the type, module, item value or macro namespace in the same scope level they are written in. -
Item Both definition items and view items together are collectively called items.
-
Shadowing While the principle of shadowing exists in all namespaces, there are different forms of it:
- item-style: Declarations shadow names from outer scopes, and are visible everywhere in their own, including lexically before their own definition. This requires there to be only one definition with the same name and namespace per scope level. Types, modules, item values and lifetimes fall under these rules.
- sequentially: Declarations shadow names that are lexically before them, both in parent scopes and their own. This means you can reuse the same name in the same scope, but a definition will not be visibly before itself. This is how local values and macros work. (Due to sequential code execution and parsing, respectively)
- view item:
A special case exists with view items; In the same scope level,
extern crate
creates entries in the module namespace, which are shadowable by names created withuse
, which are shadowable with any definition item. The singular goal of this RFC is to remove this shadowing behavior of view items
Motivation
As explained above, what is currently visible under which namespace in a given scope is determined by a somewhat complicated three step process:
- First, every
extern crate
item creates a name in the module namespace. - Then, every
use
can create a name in any namespace, shadowing theextern crate
ones. - Lastly, any definition item can shadow any name brought in scope by both
extern crate
anduse
.
These rules have developed mostly in response to the older, more complicated import system, and
the existence of wildcard imports (use foo::*
).
In the case of wildcard imports, this shadowing behavior prevents local code from breaking if the
source module gets updated to include new names that happen to be defined locally.
However, wildcard imports are now feature gated, and name conflicts in general can be resolved by
using the renaming feature of extern crate
and use
, so in the current non-gated state
of the language there is no need for this shadowing behavior.
Gating it off opens the door to remove it altogether in a backwards compatible way, or to re-enable it in case wildcard imports are officially supported again.
It also makes the mental model around items simpler: Any shadowing of items happens through lexical scoping only, and every item can be considered unordered and mutually recursive.
If this RFC gets accepted, a possible next step would be a RFC to lift the ordering restriction
between extern crate
, use
and definition items, which would make them truly behave the same in
regard to shadowing and the ability to be reordered. It would also lift the weirdness of
use foo::bar; mod foo;
.
Implementing this RFC would also not change anything about how name resolution works, as its just a tightening of the existing rules.
Drawbacks
- Feature gating import shadowing might break some code using
#[feature(globs)]
. - The behavior of
libstd
s prelude becomes more magical if it still allows shadowing, but this could be de-magified again by a new feature, see below in unresolved questions. - Or the utility of
libstd
s prelude becomes more restricted if it doesn’t allow shadowing.
Detailed design
A new feature gate import_shadowing
gets created.
During the name resolution phase of compilation, every time the compiler detects a shadowing
between extern crate
, use
and definition items in the same scope level,
it bails out unless the feature gate got enabled. This amounts to two rules:
- Items in the same scope level and either the type, module, item value or lifetime namespace may not shadow each other in the respective namespace.
- Items may shadow names from outer scopes in any namespace.
Just like for the globs
feature, the libstd
prelude import would be preempt from this,
and still be allowed to be shadowed.
Alternatives
The alternative is to do nothing, and risk running into a backwards compatibility hazard, or committing to make a final design decision around the whole module system before 1.0 gets released.
Unresolved questions
-
It is unclear how the
libstd
preludes fits into this.On the one hand, it basically acts like a hidden
use std::prelude::*;
import which ignores theglobs
feature, so it could simply also ignore theimport_shadowing
feature as well, and the rule becomes that the prelude is a magic compiler feature that injects imports into every module but doesn’t prevent the user from taking the same names.On the other hand, it is also thinkable to simply forbid shadowing of prelude items as well, as defining things with the same name as std exports is not recommended anyway, and this would nicely enforce that. It would however mean that the prelude can not change without breaking backwards compatibility, which might be too restricting.
A compromise would be to specialize wildcard imports into a new
prelude use
feature, which has the explicit properties of being shadow-able and using a wildcard import.libstd
s prelude could then simply use that, and users could define and use their own preludes as well. But that’s a somewhat orthogonal feature, and should be discussed in its own RFC. -
Interaction with overlapping imports.
Right now its legal to write this:
fn main() {
use Bar = std::result::Result;
use Bar = std::option::Option;
let x: Bar
where the latter `use` shadows the former. This would have to be forbidden as well,
however the current semantic seems like a accident anyway.
- Start Date: 2014-06-15
- RFC PR #: rust-lang/rfcs#123
- Rust Issue #: rust-lang/rust#16281
Summary
Rename the Share
trait to Sync
Motivation
With interior mutability, the name “immutable pointer” for a value of type &T
is not quite accurate. Instead, the term “shared reference” is becoming popular
to reference values of type &T
. The usage of the term “shared” is in conflict
with the Share
trait, which is intended for types which can be safely shared
concurrently with a shared reference.
Detailed design
Rename the Share
trait in std::kinds
to Sync
. Documentation would
refer to &T
as a shared reference and the notion of “shared” would simply mean
“many references” while Sync
implies that it is safe to share among many
threads.
Drawbacks
The name Sync
may invoke conceptions of “synchronized” from languages such as
Java where locks are used, rather than meaning “safe to access in a shared
fashion across tasks”.
Alternatives
As any bikeshed, there are a number of other names which could be possible for this trait:
Concurrent
Synchronized
Threadsafe
Parallel
Threaded
Atomic
DataRaceFree
ConcurrentlySharable
Unresolved questions
None.
- Start Date: 2014-07-29
- RFC PR: rust-lang/rfcs#130
- Rust Issue: rust-lang/rust#16094
Summary
Remove special treatment of Box<T>
from the borrow checker.
Motivation
Currently the Box<T>
type is special-cased and converted to the old
~T
internally. This is mostly invisible to the user, but it shows up
in some places that give special treatment to Box<T>
. This RFC is
specifically concerned with the fact that the borrow checker has
greater precision when dereferencing Box<T>
vs other smart pointers
that rely on the Deref
traits. Unlike the other kinds of special
treatment, we do not currently have a plan for how to extend this
behavior to all smart pointer types, and hence we would like to remove
it.
Here is an example that illustrates the extra precision afforded to
Box<T>
vs other types that implement the Deref
traits. The
following program, written using the Box
type, compiles
successfully:
struct Pair {
a: uint,
b: uint
}
fn example1(mut smaht: Box<Pair>) {
let a = &mut smaht.a;
let b = &mut smaht.b;
...
}
This program compiles because the type checker can see that
(*smaht).a
and (*smaht).b
are always distinct paths. In contrast,
if I use a smart pointer, I get compilation errors:
fn example2(cell: RefCell<Pair>) {
let mut smaht: RefMut<Pair> = cell.borrow_mut();
let a = &mut smaht.a;
// Error: cannot borrow `smaht` as mutable more than once at a time
let b = &mut smaht.b;
}
To see why this, consider the desugaring:
fn example2(smaht: RefCell<Pair>) {
let mut smaht = smaht.borrow_mut();
let tmp1: &mut Pair = smaht.deref_mut(); // borrows `smaht`
let a = &mut tmp1.a;
let tmp2: &mut Pair = smaht.deref_mut(); // borrows `smaht` again!
let b = &mut tmp2.b;
}
It is a violation of the Rust type system to invoke deref_mut
when
the reference to a
is valid and usable, since deref_mut
requires
&mut self
, which in turn implies no alias to self
or anything
owned by self
.
This desugaring suggests how the problem can be worked around in user code. The idea is to pull the result of the deref into a new temporary:
fn example3(smaht: RefCell<Pair>) {
let mut smaht: RefMut<Pair> = smaht.borrow_mut();
let temp: &mut Pair = &mut *smaht;
let a = &mut temp.a;
let b = &mut temp.b;
}
Detailed design
Removing this treatment from the borrow checker basically means changing the construction of loan paths for unique pointers.
I don’t actually know how best to implement this in the borrow
checker, particularly concerning the desire to keep the ability to
move out of boxes and use them in patterns. This requires some
investigation. The easiest and best way may be to “do it right” and is
probably to handle derefs of Box<T>
in a similar way to how
overloaded derefs are handled, but somewhat differently to account for
the possibility of moving out of them. Some investigation is needed.
Drawbacks
The borrow checker rules are that much more restrictive.
Alternatives
We have ruled out inconsistent behavior between Box
and other smart
pointer types. We considered a number of ways to extend the current
treatment of box to other smart pointer types:
-
Require compiler to introduce deref temporaries automatically where possible. This is plausible as a future extension but requires some thought to work through all cases. It may be surprising. Note that this would be a required optimization because if the optimization is not performed it affects what programs can successfully type check. (Naturally it is also observable.)
-
Some sort of unsafe deref trait that acknowledges possibility of other pointers into the referent. Unappealing because the problem is not that bad as to require unsafety.
-
Determining conditions (perhaps based on parametricity?) where it is provably safe to call deref. It is dubious and unknown if such conditions exist or what that even means. Rust also does not really enjoy parametricity properties due to presence of reflection and unsafe code.
Unresolved questions
Best implementation strategy.
- Start Date: 2014-06-18
- RFC PR: rust-lang/rfcs#131
- Rust Issue: rust-lang/rust#16093
Summary
Note: This RFC discusses the behavior of rustc
, and not any changes to the
language.
Change how target specification is done to be more flexible for unexpected usecases. Additionally, add support for the “unknown” OS in target triples, providing a minimum set of target specifications that is valid for bare-metal situations.
Motivation
One of Rust’s important use cases is embedded, OS, or otherwise “bare metal”
software. At the moment, we still depend on LLVM’s split-stack prologue for
stack safety. In certain situations, it is impossible or undesirable to
support what LLVM requires to enable this (on x86, a certain thread-local
storage setup). Additionally, porting rustc
to a new platform requires
modifying the compiler, adding a new OS manually.
Detailed design
A target triple consists of three strings separated by a hyphen, with a possible fourth string at the end preceded by a hyphen. The first is the architecture, the second is the “vendor”, the third is the OS type, and the optional fourth is environment type. In theory, this specifies precisely what platform the generated binary will be able to run on. All of this is determined not by us but by LLVM and other tools. When on bare metal or a similar environment, there essentially is no OS, and to handle this there is the concept of “unknown” in the target triple. When the OS is “unknown”, no runtime environment is assumed to be present (including things such as dynamic linking, threads/thread-local storage, IO, etc).
Rather than listing specific targets for special treatment, introduce a
general mechanism for specifying certain characteristics of a target triple.
Redesign how targets are handled around this specification, including for the
built-in targets. Extend the --target
flag to accept a file name of a target
specification. A table of the target specification flags and their meaning:
data-layout
: The LLVM data layout to use. Mostly included for completeness; changing this is unlikely to be used.link-args
: Arguments to pass to the linker, unconditionally.cpu
: Default CPU to use for the target, overridable with-C target-cpu
features
: Default target features to enable, augmentable with-C target-features
.dynamic-linking-available
: Whether thedylib
crate type is allowed.split-stacks-supported
: Whether there is runtime support that will allow LLVM’s split stack prologue to function as intended.llvm-target
: What target to pass to LLVM.relocation-model
: What relocation model to use by default.target_endian
,target_word_size
: Specify the strings used for the correspondingcfg
variables.code-model
: Code model to pass to LLVM, overridable with-C code-model
.no-redzone
: Disable use of any stack redzone, overridable with-C no-redzone
Rather than hardcoding a specific set of behaviors per-target, with no recourse for escaping them, the compiler would also use this mechanism when deciding how to build for a given target. The process would look like:
- Look up the target triple in an internal map, and load that configuration
if it exists. If that fails, check if the target name exists as a file, and
try loading that. If the file does not exist, look up
<target>.json
in theRUST_TARGET_PATH
, which is a colon-separated list of directories. - If
-C linker
is specified, use that instead of the target-specified linker. - If
-C link-args
is given, add those to the ones specified by the target. - If
-C target-cpu
is specified, replace the targetcpu
with it. - If
-C target-feature
is specified, add those to the ones specified by the target. - If
-C relocation-model
is specified, replace the targetrelocation-model
with it. - If
-C code-model
is specified, replace the targetcode-model
with it. - If
-C no-redzone
is specified, replace the targetno-redzone
with true.
Then during compilation, this information is used at the proper places rather
than matching against an enum listing the OSes we recognize. The target_os
,
target_family
, and target_arch
cfg
variables would be extracted from the
--target
passed to rustc.
Drawbacks
More complexity. However, this is very flexible and allows one to use Rust on a new or non-standard target incredibly easy, without having to modify the compiler. rustc is the only compiler I know of that would allow that.
Alternatives
A less holistic approach would be to just allow disabling split stacks on a
per-crate basis. Another solution could be adding a family of targets,
<arch>-unknown-unknown
, which omits all of the above complexity but does not
allow extending to new targets easily.
Summary
This RFC describes a variety of extensions to allow any method to be used as first-class functions. The same extensions also allow for trait methods without receivers to be invoked in a more natural fashion.
First, at present, the notation path::method()
can be used to invoke
inherent methods on types. For example, Vec::new()
is used to create
an instance of a vector. This RFC extends that notion to also cover
trait methods, so that something like T::size_of()
or T::default()
is legal.
Second, currently it is permitted to reference so-called “static
methods” from traits using a function-like syntax. For example, one
can write Default::default()
. This RFC extends that notation so it
can be used with any methods, whether or not they are defined with a
receiver. (In fact, the distinction between static methods and other
methods is completely erased, as per the method lookup of RFC PR #48.)
Third, we introduce an unambiguous if verbose notation that permits
one to precisely specify a trait method and its receiver type in one
form. Specifically, the notation <T as TraitRef>::item
can be used
to designate an item item
, defined in a trait TraitRef
, as
implemented by the type T
.
Motivation
There are several motivations:
- There is a need for an unambiguous way to invoke methods. This is typically
a fallback for when the more convenient invocation forms fail:
- For example, when multiple traits are in scope that all define the same method for the same types, there must be a way to disambiguate which method you mean.
- It is sometimes desirable not to have autoderef:
- For methods like
clone()
that apply to almost all types, it is convenient to be more specific about which precise type you want to clone. To get this right with autoderef, one must know the precise rules being used, which is contrary to the “DWIM” intention. - For types that implement
Deref<T>
, UFCS can be used to unambiguously differentiate between methods invoked on the smart pointer itself and methods invoked on its referent.
- For methods like
- There are many methods, such as
SizeOf::size_of()
, that return properties of the type alone and do not naturally take any argument that can be used to decide which trait impl you are referring to.- This proposal introduces a variety of ways to invoke such methods,
varying in the amount of explicit information one includes:
T::size_of()
– shorthand, but only works ifT
is a path<T>::size_of()
– infers the traitSizeOf
based on the traits in scope, just as with a method call<T as SizeOf>::size_of()
– completely unambiguous
- This proposal introduces a variety of ways to invoke such methods,
varying in the amount of explicit information one includes:
Detailed design
Path syntax
The syntax of paths is extended as follows:
PATH = ID_SEGMENT { '::' ID_SEGMENT }
| TYPE_SEGMENT { '::' ID_SEGMENT }
| ASSOC_SEGMENT '::' ID_SEGMENT { '::' ID_SEGMENT }
ID_SEGMENT = ID [ '::' '<' { TYPE ',' TYPE } '>' ]
TYPE_SEGMENT = '<' TYPE '>'
ASSOC_SEGMENT = '<' TYPE 'as' TRAIT_REFERENCE '>'
Examples of valid paths. In these examples, capitalized names refer to types (though this doesn’t affect the grammar).
a::b::c
a::<T1,T2>::b::c
T::size_of
<T>::size_of
<T as SizeOf>::size_of
Eq::eq
Eq::<T>::eq
Zero::zero
Normalization of path that reference types
Whenever a path like ...::a::...
resolves to a type (but not a
trait), it is rewritten (internally) to <...::a>::...
.
Note that there is a subtle distinction between the following paths:
ToStr::to_str
<ToStr>::to_str
In the former, we are selecting the member to_str
from the trait ToStr
.
The result is a function whose type is basically equivalent to:
fn to_str<Self:ToStr>(self: &Self) -> String
In the latter, we are selecting the member to_str
from the type
ToStr
(i.e., an ToStr
object). Resolving type members is
different. In this case, it would yield a function roughly equivalent
to:
fn to_str(self: &ToStr) -> String
This subtle distinction arises from the fact that we pun on the trait name to indicate both a type and a reference to the trait itself. In this case, depending on which interpretation we choose, the path resolution rules differ slightly.
Paths that begin with a TYPE_SEGMENT
When a path begins with a TYPE_SEGMENT, it is a type-relative path. If
this is the complete path (e.g., <int>
), then the path resolves to
the specified type. If the path continues (e.g., <int>::size_of
)
then the next segment is resolved using the following procedure. The
procedure is intended to mimic method lookup, and hence any changes to
method lookup may also change the details of this lookup.
Given a path <T>::m::...
:
- Search for members of inherent impls defined on
T
(if any) with the namem
. If any are found, the path resolves to that item. - Otherwise, let
IN_SCOPE_TRAITS
be the set of traits that are in scope and which contain a member namedm
:- Let
IMPLEMENTED_TRAITS
be those traits fromIN_SCOPE_TRAITS
for which an implementation exists that (may) apply toT
.- There can be ambiguity in the case that
T
contains type inference variables.
- There can be ambiguity in the case that
- If
IMPLEMENTED_TRAITS
is not a singleton set, report an ambiguity error. Otherwise, letTRAIT
be the member ofIMPLEMENTED_TRAITS
. - If
TRAIT
is ambiguously implemented forT
, report an ambiguity error and request further type information. - Otherwise, rewrite the path to
<T as Trait>::m::...
and continue.
- Let
Paths that begin with an ASSOC_SEGMENT
When a path begins with an ASSOC_SEGMENT, it is a reference to an
associated item defined from a trait. Note that such paths must always
have a follow-on member m
(that is, <T as Trait>
is not a complete
path, but <T as Trait>::m
is).
To resolve the path, first search for an applicable implementation of
Trait
for T
. If no implementation can be found – or the result is
ambiguous – then report an error.
Otherwise:
- Determine the types of output type parameters for
Trait
from the implementation. - If output type parameters were specified in the path, ensure that they
are compatible with those specified on the impl.
- For example, if the path were
<int as SomeTrait<uint>>
, and the impl is declared asimpl SomeTrait<char> for int
, then an error would be reported becausechar
anduint
are not compatible.
- For example, if the path were
- Resolve the path to the member of the trait with the substitution composed
of the output type parameters from the impl and
Self => T
.
Alternatives
We have explored a number of syntactic alternatives. This has been selected as being the only one that is simultaneously:
- Tolerable to look at.
- Able to convey all necessary information along with auxiliary information
the user may want to verify:
- Self type, type of trait, name of member, type output parameters
Here are some leading candidates that were considered along with their equivalents in the syntax proposed by this RFC. The reasons for their rejection are listed:
module::type::(Trait::member) <module::type as Trait>::member
--> semantics of parentheses considered too subtle
--> cannot accommodate types that are not paths, like `[int]`
(type: Trait)::member <type as Trait>::member
--> complicated to parse
--> cannot accommodate types that are not paths, like `[int]`
... (I can't remember all the rest)
One variation that is definitely possible is that we could use the :
rather than the keyword as
:
<type: Trait>::member <type as Trait>::member
--> no real objection. `as` was chosen because it mimics the
syntax for constructing a trait object.
Unresolved questions
Is there a better way to disambiguate a reference to a trait item
ToStr::to_str
versus a reference to a member of the object type
<ToStr>::to_str
? I personally do not think so: so long as we pun on
the name of the trait, the potential for confusion will
remain. Therefore, the only two possibilities I could come up with are
to try and change the question:
-
One answer might be that we simply make the second form meaningless by prohibiting inherent impls on object types. But there remains a utility to being able to write something like
<ToStr>::is_sized()
(whereis_sized()
is an example of a trait fn that could apply to both sized and unsized types). Moreover, artificially restricting object types just for this reason doesn’t seem right. -
Another answer is to change the syntax of object types. I have sometimes considered that
impl ToStr
might be better suited as the object type and thenToStr
could be used as syntactic sugar for a type parameter. But there exists a lot of precedent for the current approach and hence I think this is likely a bad idea (not to mention that it’s a drastic change).
- Start Date: 2014-09-30
- RFC PR #: https://github.com/rust-lang/rfcs/pull/135
- Rust Issue #: https://github.com/rust-lang/rust/issues/17657
Summary
Add where
clauses, which provide a more expressive means of
specifying trait parameter bounds. A where
clause comes after a
declaration of a generic item (e.g., an impl or struct definition) and
specifies a list of bounds that must be proven once precise values are
known for the type parameters in question. The existing bounds
notation would remain as syntactic sugar for where clauses.
So, for example, the impl
for HashMap
could be changed from this:
impl<K:Hash+Eq,V> HashMap<K, V>
{
..
}
to the following:
impl<K,V> HashMap<K, V>
where K : Hash + Eq
{
..
}
The full grammar can be found in the detailed design.
Motivation
The high-level bit is that the current bounds syntax does not scale to
complex cases. Introducing where
clauses is a simple extension that
gives us a lot more expressive power. In particular, it will allow us
to refactor the operator traits to be in a convenient, multidispatch
form (e.g., so that user-defined mathematical types can be added to
int
and vice versa). (It’s also worth pointing out that, once #5527
lands at least, implementing where clauses will be very little work.)
Here is a list of limitations with the current bounds syntax that are
overcome with the where
syntax:
-
It cannot express bounds on anything other than type parameters. Therefore, if you have a function generic in
T
, you can writeT:MyTrait
to declare thatT
must implementMyTrait
, but you can’t writeOption<T> : MyTrait
or(int, T) : MyTrait
. These forms are less commonly required but still important. -
It does not work well with associated types. This is because there is no space to specify the value of an associated type. Other languages use
where
clauses (or something analogous) for this purpose. -
It’s just plain hard to read. Experience has shown that as the number of bounds grows, the current syntax becomes hard to read and format.
Let’s examine each case in detail.
Bounds are insufficiently expressive
Currently bounds can only be declared on type parameters. But there are situations where one wants to declare bounds not on the type parameter itself but rather a type that includes the type parameter.
Partially generic types
One situation where this occurs is when you want to write functions where types are partially known and have those interact with other functions that are fully generic. To explain the situation, let’s examine some code adapted from rustc.
Imagine I have a table parameterized by a value type V
and a key
type K
. There are also two traits, Value
and Key
, that describe
the keys and values. Also, each type of key is linked to a specific
value:
struct Table<V:Value, K:Key<V>> { ... }
trait Key<V:Value> { ... }
trait Value { ... }
Now, imagine I want to write some code that operates over all keys
whose value is an Option<T>
for some T
:
fn example<T,K:Key<Option<T>>>(table: &Table<Option<T>, K>) { ... }
This seems reasonable, but this code will not compile. The problem is
that the compiler needs to know that the value type implements
Value
, but here the value type is Option<T>
. So we’d need to
declare Option<T> : Value
, which we cannot do.
There are workarounds. I might write a new trait OptionalValue
:
trait OptionalValue<T> {
fn as_option<'a>(&'a self) -> &'a Option<T>; // identity fn
}
and then I could write my example as:
fn example<T,O:OptionalValue<T>,K:Key<O>>(table: &Table<O, K>) { ... }
But this is making my example function, already a bit complicated, become quite obscure.
Multidispatch traits
Another situation where a similar problem is encountered is multidispatch traits (aka, multiparameter type classes in Haskell). The idea of a multidispatch trait is to be able to choose the impl based not just on one type, as is the most common case, but on multiple types (usually, but not always, two).
Multidispatch is rarely needed because the vast majority of traits
are characterized by a single type. But when you need it, you really
need it. One example that arises in the standard library is the traits
for binary operators like +
. Today, the Add
trait is defined using
only single-dispatch (like so):
pub trait Add<Rhs,Sum> {
fn add(&self, rhs: &Rhs) -> Sum;
}
The expression a + b
is thus sugar for Add::add(&a, &b)
. Because
of how our trait system works, this means that only the type of the
left-hand side (the Self
parameter) will be used to select the
impl. The type for the right-hand side (Rhs
) along with the type of
their sum (Sum
) are defined as trait parameters, which are always
outputs of the trait matching: that is, they are specified by the
impl and are not used to select which impl is used.
This setup means that addition is not as extensible as we would like. For example, the standard library includes implementations of this trait for integers and other built-in types:
impl Add<int,int> for int { ... }
impl Add<f32,f32> for f32 { ... }
The limitations of this setup become apparent when we consider how a
hypothetical user library might integrate. Imagine a library L that
defines a type Complex
representing complex numbers:
struct Complex { ... }
Naturally, it should be possible to add complex numbers and integers.
Since complex number addition is commutative, it should be possible to
write both 1 + c
and c + 1
. Thus one might try the following
impls:
impl Add<int,Complex> for Complex { ... } // 1. Complex + int
impl Add<Complex,Complex> for int { ... } // 2. int + Complex
impl Add<Complex,Complex> for Complex { ... } // 3. Complex + Complex
Due to the coherence rules, however, this setup will not work. There
are in fact three errors. The first is that there are two impls of
Add
defined for Complex
(1 and 3). The second is that there are
two impls of Add
defined for int
(the one from the standard
library and 2). The final error is that impl 2 violates the orphan
rule, since the type int
is not defined in the current crate.
This is not a new problem. Object-oriented languages, with their focus
on single dispatch, have long had trouble dealing with binary
operators. One common solution is double dispatch, an awkward but
effective pattern in which no type ever implements Add
directly. Instead, we introduce “indirection” traits so that, e.g.,
int
is addable to anything that implements AddToInt
and so
on. This is not my preferred solution so I will not describe it in
detail, but rather refer readers to this blog post where I
describe how it works.
An alternative to double dispatch is to define Add
on tuple types
(LHS, RHS)
rather than on a single value. Imagine that the Add
trait were defined as follows:
trait Add<Sum> {
fn add(self) -> Sum;
}
impl Add<int> for (int, int) {
fn add(self) -> int {
let (x, y) = self;
x + y
}
}
Now the expression a + b
would be sugar for Add::add((a, b))
.
This small change has several interesting ramifications. For one
thing, the library L can easily extend Add
to cover complex numbers:
impl Add<Complex> for (Complex, int) { ... }
impl Add<Complex> for (int, Complex) { ... }
impl Add<Complex> for (Complex, Complex) { ... }
These impls do not violate the coherence rules because they are all applied to distinct types. Moreover, none of them violate the orphan rule because each of them is a tuple involving at least one type local to the library.
One downside of this Add
pattern is that there is no way within the
trait definition to refer to the type of the left- or right-hand side
individually; we can only use the type Self
to refer to the tuple of
both types. In the Discussion section below, I will introduce
an extended “multi-dispatch” pattern that addresses this particular
problem.
There is however another problem that where clauses help to address. Imagine that we wish to define a function to increment complex numbers:
fn increment(c: Complex) -> Complex {
1 + c
}
This function is pretty generic, so perhaps we would like to
generalize it to work over anything that can be added to an int. We’ll
use our new version of the Add
trait that is implemented over
tuples:
fn increment<T:...>(c: T) -> T {
1 + c
}
At this point we encounter the problem. What bound should we give for
T
? We’d like to write something like (int, T) : Add<T>
– that
is, Add
is implemented for the tuple (int, T)
with the sum type
T
. But we can’t write that, because the current bounds syntax is too
limited.
Where clauses give us an answer. We can write a generic version of
increment
like so:
fn increment<T>(c: T) -> T
where (int, T) : Add<T>
{
1 + c
}
Associated types
It is unclear exactly what form associated types will have in Rust, but it is well documented that our current design, in which type parameters decorate traits, does not scale particularly well. (For curious readers, there are several blog posts exploring the design space of associated types with respect to Rust in particular.)
The high-level summary of associated types is that we can replace
a generic trait like Iterator
:
trait Iterator<E> {
fn next(&mut self) -> Option<E>;
}
With a version where the type parameter is a “member” of the
Iterator
trait:
trait Iterator {
type E;
fn next(&mut self) -> Option<E>;
}
This syntactic change helps to highlight that, for any given type, the
type E
is fixed by the impl, and hence it can be considered a
member (or output) of the trait. It also scales better as the number
of associated types grows.
One challenge with this design is that it is not clear how to convert a function like the following:
fn sum<I:Iterator<int>>(i: I) -> int {
...
}
With associated types, the reference Iterator<int>
is no longer
valid, since the trait Iterator
doesn’t have type parameters.
The usual solution to this problem is to employ a where clause:
fn sum<I:Iterator>(i: I) -> int
where I::E == int
{
...
}
We can also employ where clauses with object types via a syntax like
&Iterator<where E=int>
(admittedly somewhat wordy)
Readability
When writing very generic code, it is common to have a large number of
parameters with a large number of bounds. Here is some example
function extracted from rustc
:
fn set_var_to_merged_bounds<T:Clone + InferStr + LatticeValue,
V:Clone+Eq+ToStr+Vid+UnifyVid<Bounds<T>>>(
&self,
v_id: V,
a: &Bounds<T>,
b: &Bounds<T>,
rank: uint)
-> ures;
Definitions like this are very difficult to read (it’s hard to even know how to format such a definition).
Using a where
clause allows the bounds to be separated from the list
of type parameters:
fn set_var_to_merged_bounds<T,V>(&self,
v_id: V,
a: &Bounds<T>,
b: &Bounds<T>,
rank: uint)
-> ures
where T:Clone, // it is legal to use individual clauses...
T:InferStr,
T:LatticeValue,
V:Clone+Eq+ToStr+Vid+UnifyVid<Bounds<T>>, // ...or use `+`
{
..
}
This helps to separate out the function signature from the extra requirements that the function places on its types.
If I may step aside from the “impersonal voice” of the RFC for a moment, I personally find that when writing generic code it is helpful to focus on the types and signatures, and come to the bounds later. Where clauses help to separate these distinctions. Naturally, your mileage may vary. - nmatsakis
Detailed design
Where can where clauses appear?
Where clauses can be added to anything that can be parameterized with
type/lifetime parameters with the exception of trait method
definitions: impl
declarations, fn
declarations, and trait
and
struct
definitions. They appear as follows:
impl Foo<A,B>
where ...
{ }
impl Foo<A,B> for C
where ...
{ }
impl Foo<A,B> for C
{
fn foo<A,B> -> C
where ...
{ }
}
fn foo<A,B> -> C
where ...
{ }
struct Foo<A,B>
where ...
{ }
trait Foo<A,B> : C
where ...
{ }
Where clauses cannot (yet) appear on trait methods
Note that trait method definitions were specifically excluded from the list above. The reason is that including where clauses on a trait method raises interesting questions for what it means to implement the trait. Using where clauses it becomes possible to define methods that do not necessarily apply to all implementations. We intend to enable this feature but it merits a second RFC to delve into the details.
Where clause grammar
The grammar for a where
clause would be as follows (BNF):
WHERE = 'where' BOUND { ',' BOUND } [,]
BOUND = TYPE ':' TRAIT { '+' TRAIT } [+]
TRAIT = Id [ '<' [ TYPE { ',' TYPE } [,] ] '>' ]
TYPE = ... (same type grammar as today)
Semantics
The meaning of a where clause is fairly straightforward. Each bound in the where clause must be proven by the caller after substitution of the parameter types.
One interesting case concerns trivial where clauses where the self-type does not refer to any of the type parameters, such as the following:
fn foo()
where int : Eq
{ ... }
Where clauses like these are considered an error. They have no particular meaning, since the callee knows all types involved. This is a conservative choice: if we find that we do desire a particular interpretation for them, we can always make them legal later.
Drawbacks
This RFC introduces two ways to declare a bound.
Alternatives
Remove the existing trait bounds. I decided against this both to avoid breaking lots of existing code and because the existing syntax is convenient much of the time.
Embed where clauses in the type parameter list. One alternative
syntax that was proposed is to embed a where-like clause in the type
parameter list. Thus the increment()
example
fn increment<T>(c: T) -> T
where () : Add<int,T,T>
{
1 + c
}
would become something like:
fn increment<T, ():Add<int,T,T>>(c: T) -> T
{
1 + c
}
This is unfortunately somewhat ambiguous, since a bound like T:Eq
could either be declared a type parameter T
or as a condition that
the (existing) type T
implement Eq
.
Use a colon instead of the keyword. There is some precedent for
this from the type state days. Unfortunately, it doesn’t work with
traits due to the supertrait list, and it also doesn’t look good with
the use of :
as a trait-bound separator:
fn increment<T>(c: T) -> T
: () : Add<int,T,T>
{
1 + c
}
Summary
Require a feature gate to expose private items in public APIs, until we grow the appropriate language features to be able to remove the feature gate and forbid it entirely.
Motivation
Privacy is central to guaranteeing the invariants necessary to write correct code that employs unsafe blocks. Although the current language rules prevent a private item from being directly named from outside the current module, they still permit direct access to private items in some cases. For example, a public function might return a value of private type. A caller from outside the module could then invoke this function and, thanks to type inference, gain access to the private type (though they still could not invoke public methods or access public fields). This access could undermine the reasoning of the author of the module. Fortunately, it is not hard to prevent.
Detailed design
Overview
The general idea is that:
- If an item is declared as public, items referred to in the public-facing parts of that item (e.g. its type) must themselves be declared as public.
Details follow.
The rules
These rules apply as long as the feature gate is not enabled. After the feature gate has been removed, they will apply always.
When is an item “public”?
Items that are explicitly declared as pub
are always public. In
addition, items in the impl
of a trait (not an inherent impl) are
considered public if all of the following conditions are met:
- The trait being implemented is public.
- All input types (currently, the self type) of the impl are public.
- Motivation: If any of the input types or the trait is public, it should be impossible for an outside to access the items defined in the impl. They cannot name the types nor they can get direct access to a value of those types.
What restrictions apply to public items?
The rules for various kinds of public items are as follows:
-
If it is a
static
declaration, items referred to in its type must be public. -
If it is an
fn
declaration, items referred to in its trait bounds, argument types, and return type must be public. -
If it is a
struct
orenum
declaration, items referred to in its trait bounds and in the types of itspub
fields must be public. -
If it is a
type
declaration, items referred to in its definition must be public. -
If it is a
trait
declaration, items referred to in its super-traits, in the trait bounds of its type parameters, and in the signatures of its methods (seefn
case above) must be public.
Examples
Here are some examples to demonstrate the rules.
Struct fields
// A private struct may refer to any type in any field.
struct Priv {
a: Priv,
b: Pub,
pub c: Priv
}
enum Vapor<A> { X, Y, Z } // Note that A is not used
// Public fields of a public struct may only refer to public types.
pub struct Item {
// Private field may reference a private type.
a: Priv,
// Public field must refer to a public type.
pub b: Pub,
// ERROR: Public field refers to a private type.
pub c: Priv,
// ERROR: Public field refers to a private type.
// For the purposes of this test, we do not descend into the type,
// but merely consider the names that appear in type parameters
// on the type, regardless of usage (or lack thereof) within the type
// definition itself.
pub d: Vapor<Priv>,
}
pub struct Pub { ... }
Methods
struct Priv { .. }
pub struct Pub { .. }
pub struct Foo { .. }
impl Foo {
// Illegal: public method with argument of private type.
pub fn foo(&self, p: Priv) { .. }
}
Trait bounds
trait PrivTrait { ... }
// Error: type parameter on public item bounded by a private trait.
pub struct Foo<X: PrivTrait> { ... }
// OK: type parameter on private item.
struct Foo<X: PrivTrait> { ... }
Trait definitions
struct PrivStruct { ... }
pub trait PubTrait {
// Error: private struct referenced from method in public trait
fn method(x: PrivStruct) { ... }
}
trait PrivTrait {
// OK: private struct referenced from method in private trait
fn method(x: PrivStruct) { ... }
}
Implementations
To some extent, implementations are prevented from exposing private types because their types must match the trait. However, that is not true with generics.
pub trait PubTrait<T> {
fn method(t: T);
}
struct PubStruct { ... }
struct PrivStruct { ... }
impl PubTrait<PrivStruct> for PubStruct {
// ^~~~~~~~~~ Error: Private type referenced from impl of
// public trait on a public type. [Note: this is
// an "associated type" here, not an input.]
fn method(t: PrivStruct) {
// ^~~~~~~~~~ Error: Private type in method signature.
//
// Implementation note. It may not be a good idea to report
// an error here; I think private types can only appear in
// an impl by having an associated type bound to a private
// type.
}
}
Type aliases
Note that the path to the public item does not have to be private.
mod impl {
pub struct Foo { ... }
}
pub type Bar = self::impl::Foo;
Negative examples
The following examples should fail to compile under these rules.
Non-public items referenced by a pub use
These examples are illegal because they use a pub use
to re-export
a private item:
struct Item { ... }
pub mod module {
// Error: Item is not declared as public, but is referenced from
// a `pub use`.
pub use Item;
}
struct Foo { ... }
// Error: Non-public item referenced by `pub use`.
pub use Item = Foo;
If it was desired to have a private name that is publicly “renamed” using a pub use, that can be achieved using a module:
mod impl {
pub struct ItemPriv;
}
pub use Item = self::impl::ItemPriv;
Drawbacks
Adds a (temporary) feature gate.
Requires some existing code to opt-in to the feature gate before transitioning to a more explicit alternative.
Requires effort to implement.
Alternatives
If we stick with the status quo, we’ll have to resolve several bizarre questions and keep supporting its behavior indefinitely after 1.0.
Instead of a feature gate, we could just ban these things outright right away, at the cost of temporarily losing some convenience and a small amount of expressiveness before the more principled replacement features are implemented.
We could make an exception for private supertraits, as these are not quite as problematic as the other cases. However, especially given that a more principled alternative is known (private methods), I would rather not make any exceptions.
The original design of this RFC had a stronger notion of “public”
which also considered whether a public path existed to the item. In
other words, a module X
could not refer to a public item Y
from a
submodule Z
, unless X
also exposed a public path to Y
(whether
that be because Z
was public, or via a pub use
). This definition
strengthened the basic guarantee of “private things are only directly
accessible from within the current module” to include the idea that
public functions in outer modules cannot accidentally refer to public
items from inner modules unless there is a public path from the outer
to the inner module. Unfortunately, these rules were complex to state
concisely and also hard to understand in practice; when an error
occurred under these rules, it was very hard to evaluate whether the
error was legitimate. The newer rules are simpler while still
retaining the basic privacy guarantee.
One important advantage of the earlier approach, and a scenario not
directly addressed in this RFC, is that there may be items which are
declared as public by an inner module but still not intended to be
exposed to the world at large (in other words, the items are only
expected to be used within some subtree). A special case of this is
crate-local data. In the older rules, the “intended scope” of privacy
could be somewhat inferred from the existence (or non-existence) of
pub use
declarations. However, in the author’s opinion, this
scenario would be best addressed by making pub
declarations more
expressive so that the intended scope can be stated directly.
- Start Date: 2014-06-25
- RFC PR: rust-lang/rfcs#139
- Rust Issue: rust-lang/rust#10504
Summary
Remove the coercion from Box<T>
to &T
from the language.
Motivation
The coercion between Box<T>
to &T
is not replicable by user-defined smart pointers and has been found to be rarely used 1. We already removed the coercion between Box<T>
and &mut T
in RFC 33.
Detailed design
The coercion between Box<T>
and &T
should be removed.
Note that methods that take &self
can still be called on values of type Box<T>
without any special referencing or dereferencing. That is because the semantics of auto-deref and auto-ref conspire to make it work: the types unify after one autoderef followed by one autoref.
Drawbacks
Borrowing from Box<T>
to &T
may be convenient.
Alternatives
The impact of not doing this is that the coercion will remain.
Unresolved questions
None.
- Start Date: 2014-06-24
- RFC PR: rust-lang/rfcs#141
- Rust Issue: rust-lang/rust#15552
Summary
This RFC proposes to
- Expand the rules for eliding lifetimes in
fn
definitions, and - Follow the same rules in
impl
headers.
By doing so, we can avoid writing lifetime annotations ~87% of the time that they are currently required, based on a survey of the standard library.
Motivation
In today’s Rust, lifetime annotations make code more verbose, both for methods
fn get_mut<'a>(&'a mut self) -> &'a mut T
and for impl
blocks:
impl<'a> Reader for BufReader<'a> { ... }
In the vast majority of cases, however, the lifetimes follow a very simple pattern.
By codifying this pattern into simple rules for filling in elided lifetimes, we can avoid writing any lifetimes in ~87% of the cases where they are currently required.
Doing so is a clear ergonomic win.
Detailed design
Today’s lifetime elision rules
Rust currently supports eliding lifetimes in functions, so that you can write
fn print(s: &str);
fn get_str() -> &str;
instead of
fn print<'a>(s: &'a str);
fn get_str<'a>() -> &'a str;
The elision rules work well for functions that consume references, but not for
functions that produce them. The get_str
signature above, for example,
promises to produce a string slice that lives arbitrarily long, and is
either incorrect or should be replaced by
fn get_str() -> &'static str;
Returning 'static
is relatively rare, and it has been proposed to make leaving
off the lifetime in output position an error for this reason.
Moreover, lifetimes cannot be elided in impl
headers.
The proposed rules
Overview
This RFC proposes two changes to the lifetime elision rules:
-
Since eliding a lifetime in output position is usually wrong or undesirable under today’s elision rules, interpret it in a different and more useful way.
-
Interpret elided lifetimes for
impl
headers analogously tofn
definitions.
Lifetime positions
A lifetime position is anywhere you can write a lifetime in a type:
&'a T
&'a mut T
T<'a>
As with today’s Rust, the proposed elision rules do not distinguish between
different lifetime positions. For example, both &str
and Ref<uint>
have
elided a single lifetime.
Lifetime positions can appear as either “input” or “output”:
-
For
fn
definitions, input refers to the types of the formal arguments in thefn
definition, while output refers to result types. Sofn foo(s: &str) -> (&str, &str)
has elided one lifetime in input position and two lifetimes in output position. Note that the input positions of afn
method definition do not include the lifetimes that occur in the method’simpl
header (nor lifetimes that occur in the trait header, for a default method). -
For
impl
headers, input refers to the lifetimes appears in the type receiving theimpl
, while output refers to the trait, if any. Soimpl<'a> Foo<'a>
has'a
in input position, whileimpl<'a, 'b, 'c> SomeTrait<'b, 'c> for Foo<'a, 'c>
has'a
in input position,'b
in output position, and'c
in both input and output positions.
The rules
-
Each elided lifetime in input position becomes a distinct lifetime parameter. This is the current behavior for
fn
definitions. -
If there is exactly one input lifetime position (elided or not), that lifetime is assigned to all elided output lifetimes.
-
If there are multiple input lifetime positions, but one of them is
&self
or&mut self
, the lifetime ofself
is assigned to all elided output lifetimes. -
Otherwise, it is an error to elide an output lifetime.
Notice that the actual signature of a fn
or impl
is based on the expansion
rules above; the elided form is just a shorthand.
Examples
fn print(s: &str); // elided
fn print<'a>(s: &'a str); // expanded
fn debug(lvl: uint, s: &str); // elided
fn debug<'a>(lvl: uint, s: &'a str); // expanded
fn substr(s: &str, until: uint) -> &str; // elided
fn substr<'a>(s: &'a str, until: uint) -> &'a str; // expanded
fn get_str() -> &str; // ILLEGAL
fn frob(s: &str, t: &str) -> &str; // ILLEGAL
fn get_mut(&mut self) -> &mut T; // elided
fn get_mut<'a>(&'a mut self) -> &'a mut T; // expanded
fn args<T:ToCStr>(&mut self, args: &[T]) -> &mut Command // elided
fn args<'a, 'b, T:ToCStr>(&'a mut self, args: &'b [T]) -> &'a mut Command // expanded
fn new(buf: &mut [u8]) -> BufWriter; // elided
fn new<'a>(buf: &'a mut [u8]) -> BufWriter<'a> // expanded
impl Reader for BufReader { ... } // elided
impl<'a> Reader for BufReader<'a> { .. } // expanded
impl Reader for (&str, &str) { ... } // elided
impl<'a, 'b> Reader for (&'a str, &'b str) { ... } // expanded
impl StrSlice for &str { ... } // elided
impl<'a> StrSlice<'a> for &'a str { ... } // expanded
trait Bar<'a> { fn bound(&'a self) -> &int { ... } fn fresh(&self) -> &int { ... } } // elided
trait Bar<'a> { fn bound(&'a self) -> &'a int { ... } fn fresh<'b>(&'b self) -> &'b int { ... } } // expanded
impl<'a> Bar<'a> for &'a str {
fn bound(&'a self) -> &'a int { ... } fn fresh(&self) -> &int { ... } // elided
}
impl<'a> Bar<'a> for &'a str {
fn bound(&'a self) -> &'a int { ... } fn fresh<'b>(&'b self) -> &'b int { ... } // expanded
}
// Note that when the impl reuses the same signature (with the same elisions)
// from the trait definition, the expanded forms will also match, and thus
// the `impl` will be compatible with the `trait`.
impl Bar for &str { fn bound(&self) -> &int { ... } } // elided
impl<'a> Bar<'a> for &'a str { fn bound<'b>(&'b self) -> &'b int { ... } } // expanded
// Note that the preceding example's expanded methods do not match the
// signatures from the above trait definition for `Bar`; in the general
// case, if the elided signatures between the `impl` and the `trait` do
// not match, an expanded `impl` may not be compatible with the given
// `trait` (and thus would not compile).
impl Bar for &str { fn fresh(&self) -> &int { ... } } // elided
impl<'a> Bar<'a> for &'a str { fn fresh<'b>(&'b self) -> &'b int { ... } } // expanded
impl Bar for &str {
fn bound(&'a self) -> &'a int { ... } fn fresh(&self) -> &int { ... } // ILLEGAL: unbound 'a
}
Error messages
Since the shorthand described above should eliminate most uses of explicit lifetimes, there is a potential “cliff”. When a programmer first encounters a situation that requires explicit annotations, it is important that the compiler gently guide them toward the concept of lifetimes.
An error can arise with the above shorthand only when the program elides an output lifetime and neither of the rules can determine how to annotate it.
For fn
The error message should guide the programmer toward the concept of lifetime by talking about borrowed values:
This function’s return type contains a borrowed value, but the signature does not say which parameter it is borrowed from. It could be one of a, b, or c. Mark the input parameter it borrows from using lifetimes, e.g. [generated example]. See [url] for an introduction to lifetimes.
This message is slightly inaccurate, since the presence of a lifetime parameter does not necessarily imply the presence of a borrowed value, but there are no known use-cases of phantom lifetime parameters.
For impl
The error case on impl
is exceedingly rare: it requires (1) that the impl
is
for a trait with a lifetime argument, which is uncommon, and (2) that the Self
type has multiple lifetime arguments.
Since there are no clear “borrowed values” for an impl
, this error message
speaks directly in terms of lifetimes. This choice seems warranted given that a
programmer implementing a trait with lifetime parameters will almost certainly
already understand lifetimes.
TraitName requires lifetime arguments, and the impl does not say which lifetime parameters of TypeName to use. Mark the parameters explicitly, e.g. [generated example]. See [url] for an introduction to lifetimes.
The impact
To assess the value of the proposed rules, we conducted a survey of the code
defined in libstd
(as opposed to the code it reexports). This corpus is
large and central enough to be representative, but small enough to easily
analyze.
We found that of the 169 lifetimes that currently require annotation for
libstd
, 147 would be elidable under the new rules, or 87%.
Note: this percentage does not include the large number of lifetimes that are already elided with today’s rules.
The detailed data is available at: https://gist.github.com/aturon/da49a6d00099fdb0e861
Drawbacks
Learning lifetimes
The main drawback of this change is pedagogical. If lifetime annotations are rarely used, newcomers may encounter error messages about lifetimes long before encountering lifetimes in signatures, which may be confusing. Counterpoints:
-
This is already the case, to some extent, with the current elision rules.
-
Most existing error messages are geared to talk about specific borrows not living long enough, pinpointing their locations in the source, rather than talking in terms of lifetime annotations. When the errors do mention annotations, it is usually to suggest specific ones.
-
The proposed error messages above will help programmers transition out of the fully elided regime when they first encounter a signature requiring it.
-
When combined with a good tutorial on the borrow/lifetime system (which should be introduced early in the documentation), the above should provide a reasonably gentle path toward using and understanding explicit lifetimes.
Programmers learn lifetimes once, but will use them many times. Better to favor long-term ergonomics, if a simple elision rule can cover 87% of current lifetime uses (let alone the currently elided cases).
Subtlety for non-&
types
While the rules are quite simple and regular, they can be subtle when applied to types with lifetime positions. To determine whether the signature
fn foo(r: Bar) -> Bar
is actually using lifetimes via the elision rules, you have to know whether
Bar
has a lifetime parameter. But this subtlety already exists with the
current elision rules. The benefit is that library types like Ref<'a, T>
get
the same status and ergonomics as built-ins like &'a T
.
Alternatives
-
Do not include output lifetime elision for
impl
. Since traits with lifetime parameters are quite rare, this would not be a great loss, and would simplify the rules somewhat. -
Only add elision rules for
fn
, in keeping with current practice. -
Only add elision for explicit
&
pointers, eliminating one of the drawbacks mentioned above. Doing so would impose an ergonomic penalty on abstractions, though:Ref
would be more painful to use than&
.
Unresolved questions
The fn
and impl
cases tackled above offer the biggest bang for the buck for
lifetime elision. But we may eventually want to consider other opportunities.
Double lifetimes
Another pattern that sometimes arises is types like &'a Foo<'a>
. We could
consider an additional elision rule that expands &Foo
to &'a Foo<'a>
.
However, such a rule could be easily added later, and it is unclear how common the pattern is, so it seems best to leave that for a later RFC.
Lifetime elision in struct
s
We may want to allow lifetime elision in struct
s, but the cost/benefit
analysis is much less clear. In particular, it could require chasing an
arbitrary number of (potentially private) struct
fields to discover the source
of a lifetime parameter for a struct
. There are also some good reasons to
treat elided lifetimes in struct
s as 'static
.
Again, since shorthand can be added backwards-compatibly, it seems best to wait.
- Start Date: 2014-07-02
- RFC PR: rust-lang/rfcs#151
- Rust Issue: rust-lang/rust#12831
Summary
Closures should capture their upvars by value unless the ref
keyword is used.
Motivation
For unboxed closures, we will need to syntactically distinguish between captures by value and captures by reference.
Detailed design
This is a small part of #114, split off to separate it from the rest of the discussion going on in that RFC.
Closures should capture their upvars (closed-over variables) by value unless the ref
keyword precedes the opening |
of the argument list. Thus |x| x + 2
will capture x
by value (and thus, if x
is not Copy
, it will move x
into the closure), but ref |x| x + 2
will capture x
by reference.
In an unboxed-closures world, the immutability/mutability of the borrow (as the case may be) is inferred from the type of the closure: Fn
captures by immutable reference, while FnMut
captures by mutable reference. In a boxed-closures world, the borrows are always mutable.
Drawbacks
It may be that ref
is unwanted complexity; it only changes the semantics of 10%-20% of closures, after all. This does not add any core functionality to the language, as a reference can always be made explicitly and then captured. However, there are a lot of closures, and the workaround to capture a reference by value is painful.
Alternatives
As above, the impact of not doing this is that reference semantics would have to be achieved. However, the diff against current Rust was thousands of lines of pretty ugly code.
Another alternative would be to annotate each individual upvar with its capture semantics, like capture clauses in C++11. This proposal does not preclude adding that functionality should it be deemed useful in the future. Note that C++11 provides a syntax for capturing all upvars by reference, exactly as this proposal does.
Unresolved questions
None.
- Start Date: 2014-07-04
- RFC PR #: rust-lang/rfcs#155
- Rust Issue #: rust-lang/rust#17059
Summary
Require “anonymous traits”, i.e. impl MyStruct
to occur only in the same module that MyStruct
is defined.
Motivation
Before I can explain the motivation for this, I should provide some background
as to how anonymous traits are implemented, and the sorts of bugs we see with
the current behaviour. The conclusion will be that we effectively already only
support impl MyStruct
in the same module that MyStruct
is defined, and
making this a rule will simply give cleaner error messages.
- The compiler first sees
impl MyStruct
during the resolve phase, specifically inResolver::build_reduced_graph()
, called byResolver::resolve()
insrc/librustc/middle/resolve.rs
. This is before any type checking (or type resolution, for that matter) is done, so the compiler trusts for now thatMyStruct
is a valid type. - If
MyStruct
is a path with more than one segment, such asmymod::MyStruct
, it is silently ignored (how was this not flagged when the code was written??), which effectively causes static methods in suchimpl
s to be dropped on the floor. A silver lining here is that nothing is added to the current module namespace, so the shadowing bugs demonstrated in the next bullet point do not apply here. (To locate this bug in the code, find thematch
immediately following theFIXME (#3785)
comment inresolve.rs
.) This leads to the following
mod break1 {
pub struct MyGuy;
impl MyGuy {
pub fn do1() { println!("do 1"); }
}
}
impl break1::MyGuy {
fn do2() { println!("do 2"); }
}
fn main() {
break1::MyGuy::do1();
break1::MyGuy::do2();
}
<anon>:15:5: 15:23 error: unresolved name `break1::MyGuy::do2`.
<anon>:15 break1::MyGuy::do2();
as noticed by @huonw in https://github.com/rust-lang/rust/issues/15060 .
- If one does not exist, the compiler creates a submodule
MyStruct
of the current module, withkind
ImplModuleKind
. Static methods are placed into this module. If such a module already exists, the methods are appended to it, to support multipleimpl MyStruct
blocks within the same module. If a module exists that is notImplModuleKind
, the compiler signals a duplicate module definition error. - Notice at this point that if there is a
use MyStruct
, the compiler will act as though it is unaware of this. This is because imports are not resolved yet (they are inResolver::resolve_imports()
called immediately afterResolver::build_reduced_graph()
is called). In the final resolution step,MyStruct
will be searched in the namespace of the current module, checking imports only as a fallback (and only in some contexts), so theuse MyStruct
is effectively shadowed. If there is animpl MyStruct
in the file being imported from, the user expects that the newimpl MyStruct
will append to that one, same as if they are in the original file. This leads to the original bug report https://github.com/rust-lang/rust/issues/15060 . - In fact, even if no methods from the import are used, the name
MyStruct
will not be associated to a type, so that
trait T {}
impl<U: T> Vec<U> {
fn from_slice<'a>(x: &'a [uint]) -> Vec<uint> {
fail!()
}
}
fn main() { let r = Vec::from_slice(&[1u]); }
error: found module name used as a type: impl Vec<U>::Vec<U> (id=5)
impl<U: T> Vec<U>
which @Ryman noticed in https://github.com/rust-lang/rust/issues/15060 . The
reason for this is that in Resolver::resolve_crate()
, the final step of
Resolver::resolve()
, the type of an anonymous impl
is determined by
NameBindings::def_for_namespace(TypeNS)
. This function searches the namespace
TypeNS
(which is not affected by imports) for a type; failing that it
tries for a module; failing that it returns None
. The result is that when
typeck runs, it sees impl [module name]
instead of impl [type name]
.
The main motivation of this RFC is to clear out these bugs, which do not make sense to a user of the language (and had me confused for quite a while).
A secondary motivation is to enforce consistency in code layout; anonymous traits are used the way that class methods are used in other languages, and the data and methods of a struct should be defined nearby.
Detailed design
I propose three changes to the language:
impl
on multiple-ident paths such asimpl mymod::MyStruct
is disallowed. Since this currently surprises the user by having absolutely no effect for static methods, support for this is already broken.impl MyStruct
must occur in the same module thatMyStruct
is defined. This is to prevent the above problems withimpl
-across-modules. Migration path is for users to just move code between source files.
Drawbacks
Static methods on impl
s-away-from-definition never worked, while non-static
methods can be implemented using non-anonymous traits. So there is no loss in
expressivity. However, using a trait where before there was none may be clumsy,
since it might not have a sensible name, and it must be explicitly imported by
all users of the trait methods.
For example, in the stdlib src/libstd/io/fs.rs
we see the code impl path::Path
to attach (non-static) filesystem-related methods to the Path
type. This would
have to be done via a FsPath
trait which is implemented on Path
and exported
alongside Path
in the prelude.
It is worth noting that this is the only instance of this RFC conflicting with current usage in the stdlib or compiler.
Alternatives
- Leaving this alone and fixing the bugs directly. This is really hard. To do it properly, we would need to seriously refactor resolve.
Unresolved questions
None.
- Start Date: 2014-08-26
- RFC PR #: rust-lang/rfcs#160
- Rust Issue #: rust-lang/rust#16779
Summary
Introduce a new if let PAT = EXPR { BODY }
construct. This allows for refutable pattern matching
without the syntactic and semantic overhead of a full match
, and without the corresponding extra
rightward drift. Informally this is known as an “if-let statement”.
Motivation
Many times in the past, people have proposed various mechanisms for doing a refutable let-binding. None of them went anywhere, largely because the syntax wasn’t great, or because the suggestion introduced runtime failure if the pattern match failed.
This proposal ties the refutable pattern match to the pre-existing conditional construct (i.e. if
statement), which provides a clear and intuitive explanation for why refutable patterns are allowed
here (as opposed to a let
statement which disallows them) and how to behave if the pattern doesn’t
match.
The motivation for having any construct at all for this is to simplify the cases that today call for
a match
statement with a single non-trivial case. This is predominately used for unwrapping
Option<T>
values, but can be used elsewhere.
The idiomatic solution today for testing and unwrapping an Option<T>
looks like
match optVal {
Some(x) => {
doSomethingWith(x);
}
None => {}
}
This is unnecessarily verbose, with the None => {}
(or _ => {}
) case being required, and
introduces unnecessary rightward drift (this introduces two levels of indentation where a normal
conditional would introduce one).
The alternative approach looks like this:
if optVal.is_some() {
let x = optVal.unwrap();
doSomethingWith(x);
}
This is generally considered to be a less idiomatic solution than the match
. It has the benefit of
fixing rightward drift, but it ends up testing the value twice (which should be optimized away, but
semantically speaking still happens), with the second test being a method that potentially
introduces failure. From context, the failure won’t happen, but it still imposes a semantic burden
on the reader. Finally, it requires having a pre-existing let-binding for the optional value; if the
value is a temporary, then a new let-binding in the parent scope is required in order to be able to
test and unwrap in two separate expressions.
The if let
construct solves all of these problems, and looks like this:
if let Some(x) = optVal {
doSomethingWith(x);
}
Detailed design
The if let
construct is based on the precedent set by Swift, which introduced its own if let
statement. In Swift, if let var = expr { ... }
is directly tied to the notion of optional values,
and unwraps the optional value that expr
evaluates to. In this proposal, the equivalent is if let Some(var) = expr { ... }
.
Given the following rough grammar for an if
condition:
if-expr = 'if' if-cond block else-clause?
if-cond = expression
else-clause = 'else' block | 'else' if-expr
The grammar is modified to add the following productions:
if-cond = 'let' pattern '=' expression
The expression
is restricted to disallow a trailing braced block (e.g. for struct literals) the
same way the expression
in the normal if
statement is, to avoid ambiguity with the then-block.
Contrary to a let
statement, the pattern in the if let
expression allows refutable patterns. The
compiler should emit a warning for an if let
expression with an irrefutable pattern, with the
suggestion that this should be turned into a regular let
statement.
Like the for
loop before it, this construct can be transformed in a syntax-lowering pass into the
equivalent match
statement. The expression
is given to match
and the pattern
becomes a match
arm. If there is an else
block, that becomes the body of the _ => {}
arm, otherwise _ => {}
is
provided.
Optionally, one or more else if
(not else if let
) blocks can be placed in the same match
using
pattern guards on _
. This could be done to simplify the code when pretty-printing the expansion
result. Otherwise, this is an unnecessary transformation.
Due to some uncertainty regarding potentially-surprising fallout of AST rewrites, and some worries
about exhaustiveness-checking (e.g. a tautological if let
would be an error, which may be
unexpected), this is put behind a feature gate named if_let
.
Examples
Source:
if let Some(x) = foo() {
doSomethingWith(x)
}
Result:
match foo() {
Some(x) => {
doSomethingWith(x)
}
_ => {}
}
Source:
if let Some(x) = foo() {
doSomethingWith(x)
} else {
defaultBehavior()
}
Result:
match foo() {
Some(x) => {
doSomethingWith(x)
}
_ => {
defaultBehavior()
}
}
Source:
if cond() {
doSomething()
} else if let Some(x) = foo() {
doSomethingWith(x)
} else {
defaultBehavior()
}
Result:
if cond() {
doSomething()
} else {
match foo() {
Some(x) => {
doSomethingWith(x)
}
_ => {
defaultBehavior()
}
}
}
With the optional addition specified above:
if let Some(x) = foo() {
doSomethingWith(x)
} else if cond() {
doSomething()
} else if other_cond() {
doSomethingElse()
}
Result:
match foo() {
Some(x) => {
doSomethingWith(x)
}
_ if cond() => {
doSomething()
}
_ if other_cond() => {
doSomethingElse()
}
_ => {}
}
Drawbacks
It’s one more addition to the grammar.
Alternatives
This could plausibly be done with a macro, but the invoking syntax would be pretty terrible and would largely negate the whole point of having this sugar.
Alternatively, this could not be done at all. We’ve been getting alone just fine without it so far,
but at the cost of making Option
just a bit more annoying to work with.
Unresolved questions
It’s been suggested that alternates or pattern guards should be allowed. I think if you need those
you could just go ahead and use a match
, and that if let
could be extended to support those in
the future if a compelling use-case is found.
I don’t know how many match
statements in our current code base could be replaced with this
syntax. Probably quite a few, but it would be informative to have real data on this.
- Start Date: 2014-07-14
- RFC PR #: rust-lang/rfcs#164
- Rust Issue #: rust-lang/rust#16951
Summary
Rust’s support for pattern matching on slices has grown steadily and incrementally without a lot of oversight.
We have concern that Rust is doing too much here, and that the complexity is not worth it. This RFC proposes
to feature gate multiple-element slice matches in the head and middle positions ([xs.., 0, 0]
and [0, xs.., 0]
).
Motivation
Some general reasons and one specific: first, the implementation of Rust’s match machinery is notoriously complex, and not well-loved. Removing features is seen as a valid way to reduce complexity. Second, slice matching in particular, is difficult to implement, while also being of only moderate utility (there are many types of collections - slices just happen to be built into the language). Finally, the exhaustiveness check is not correct for slice patterns because of their complexity; it’s not known if it can be done correctly, nor whether it is worth the effort to do so.
Detailed design
The advanced_slice_patterns
feature gate will be added. When the compiler encounters slice pattern matches in head or middle position it will emit a warning or error according to the current settings.
Drawbacks
It removes two features that some people like.
Alternatives
Fixing the exhaustiveness check would allow the feature to remain.
Unresolved questions
N/A
- Start Date: 2014-06-06
- RFC PR: rust-lang/rfcs#168
- Rust Issue: rust-lang/rust#15722
- Author: Tommit (edited by nrc)
Summary
Add syntax sugar for importing a module and items in that module in a single view item.
Motivation
Make use clauses more concise.
Detailed design
The mod
keyword may be used in a braced list of modules in a use
item to
mean the prefix module for that list. For example, writing prefix::{mod, foo};
is equivalent to writing
use prefix;
use prefix::foo;
The mod
keyword cannot be used outside of braces, nor can it be used inside
braces which do not have a prefix path. Both of the following examples are
illegal:
use module::mod;
use {mod, foo};
A programmer may write mod
in a module list with only a single item. E.g.,
use prefix::{mod};
, although this is considered poor style and may be forbidden
by a lint. (The preferred version is use prefix;
).
Drawbacks
Another use of the mod
keyword.
We introduce a way (the only way) to have paths in use items which do not
correspond with paths which can be used in the program. For example, with use foo::bar::{mod, baz};
the programmer can use foo::bar::baz
in their program
but not foo::bar::mod
(instead foo::bar
is imported).
Alternatives
Don’t do this.
Unresolved questions
N/A
- Start Date: 2014-07-16
- RFC PR #: #169
- Rust Issue #: https://github.com/rust-lang/rust/issues/16461
Summary
Change the rebinding syntax from use ID = PATH
to use PATH as ID
,
so that paths all line up on the left side, and imported identifiers
are all on the right side. Also modify extern crate
syntax
analogously, for consistency.
Motivation
Currently, the view items at the start of a module look something like this:
mod old_code {
use a::b::c::d::www;
use a::b::c::e::xxx;
use yyy = a::b::yummy;
use a::b::c::g::zzz;
}
This means that if you want to see what identifiers have been
imported, your eyes need to scan back and forth on both the left-hand
side (immediately beside the use
) and the right-hand side (at the
end of each line). In particular, note that yummy
is not in scope
within the body of old_code
This RFC proposes changing the grammar of Rust so that the example above would look like this:
mod new_code {
use a::b::c::d::www;
use a::b::c::e::xxx;
use a::b::yummy as yyy;
use a::b::c::g::zzz;
}
There are two benefits we can see by comparing mod old_code
and mod new_code
:
-
As alluded to above, now all of the imported identifiers are on the right-hand side of the block of view items.
-
Additionally, the left-hand side looks much more regular, since one sees the straight lines of
a::b::
characters all the way down, which makes the actual differences between the different paths more visually apparent.
Detailed design
Currently, the grammar for use statements is something like:
use_decl : "pub" ? "use" [ ident '=' path
| path_glob ] ;
Likewise, the grammar for extern crate declarations is something like:
extern_crate_decl : "extern" "crate" ident [ '(' link_attrs ')' ] ? [ '=' string_lit ] ? ;
This RFC proposes changing the grammar for use statements to something like:
use_decl : "pub" ? "use" [ path "as" ident
| path_glob ] ;
and the grammar for extern crate declarations to something like:
extern_crate_decl : "extern" "crate" [ string_lit "as" ] ? ident [ '(' link_attrs ')' ] ? ;
Both use
and pub use
forms are changed to use path as ident
instead of ident = path
. The form use path as ident
has the same
constraints and meaning that use ident = path
has today.
Nothing about path globs is changed; the view items that use
ident = path
are disjoint from the view items that use path globs,
and that continues to be the case under path as ident
.
The old syntaxes
"use" ident '=' path
and
"extern" "crate" ident '=' string_lit
are removed (or at least deprecated).
Drawbacks
-
pub use export = import_path
may be preferred overpub use import_path as export
since people are used to seeing the name exported by apub
item on the left-hand side of an=
sign. (See “Have distinct rebinding syntaxes foruse
andpub use
” below.) -
The ‘as’ keyword is not currently used for any binding form in Rust. Adopting this RFC would change that precedent. (See “Change the signaling token” below.)
Alternatives
Keep things as they are
This just has the drawbacks outlined in the motivation: the left-hand side of the view items are less regular, and one needs to scan both the left- and right-hand sides to see all the imported identifiers.
Change the signaling token
Go ahead with switch, so imported identifier is on the left-hand side,
but use a different token than as
to signal a rebinding.
For example, we could use @
, as an analogy with its use as a binding
operator in match expressions:
mod new_code {
use a::b::c::d::www;
use a::b::c::e::xxx;
use a::b::yummy @ yyy;
use a::b::c::g::zzz;
}
(I do not object to path @ ident
, though I find it somehow more
“line-noisy” than as
in this context.)
Or, we could use =
:
mod new_code {
use a::b::c::d::www;
use a::b::c::e::xxx;
use a::b::yummy = yyy;
use a::b::c::g::zzz;
}
(I do object to path = ident
, since typically when =
is used to
bind, the identifier being bound occurs on the left-hand side.)
Or, we could use :
, by (weak) analogy with struct pattern syntax:
mod new_code {
use a::b::c::d::www;
use a::b::c::e::xxx;
use a::b::yummy : yyy;
use a::b::c::g::zzz;
}
(I cannot figure out if this is genius or madness. Probably madness,
especially if one is allowed to omit the whitespace around the :
)
Have distinct rebinding syntaxes for use
and pub use
If people really like having ident = path
for pub use
, by the
reasoning presented above that people are used to seeing the name
exported by a pub
item on the left-hand side of an =
sign, then we
could support that by continuing to support pub use ident = path
.
If we were to go down that route, I would prefer to have distinct notions of the exported name and imported name, so that:
pub use a = foo::bar;
would actually import bar
(and a
would
just be visible as an export), and then one could rebind for export
and import simultaneously, like so:
pub use exported_bar = foo::bar as imported_bar;
But really, is pub use foo::bar as a
all that bad?
Allow extern crate ident as ident
As written, this RFC allows for two variants of extern_crate_decl
:
extern crate old_name;
extern crate "old_name" as new_name;
These are just analogous to the current options that use =
instead of as
.
However, the RFC comment dialogue suggested also allowing a renaming form that does not use a string literal:
extern crate old_name as new_name;
I have no opinion on whether this should be added or not. Arguably
this choice is orthogonal to the goals of this RFC (since, if this is a
good idea, it could just as well be implemented with the =
syntax).
Perhaps it should just be filed as a separate RFC on its own.
Unresolved questions
- In the revised
extern crate
form, is it best to put thelink_attrs
after the identifier, as written above? Or would it be better for them to come after thestring_literal
when using theextern crate string_literal as ident
form?
- Start Date: 23-07-2014
- RFC PR: rust-lang/rfcs#179
- Rust Issue: rust-lang/rust#20496
Summary
Change pattern matching on an &mut T
to &mut <pat>
, away from its
current &<pat>
syntax.
Motivation
Pattern matching mirrors construction for almost all types, except
&mut
, which is constructed with &mut <expr>
but destructured with
&<pat>
. This is almost certainly an unnecessary inconsistency.
This can and does lead to confusion, since people expect the pattern
syntax to match construction, but a pattern like &mut (ref mut x, _)
is
actually currently a parse error:
fn main() {
let &mut (ref mut x, _);
}
and-mut-pat.rs:2:10: 2:13 error: expected identifier, found path
and-mut-pat.rs:2 let &mut (ref mut x, _);
^~~
Another (rarer) way it can be confusing is the pattern &mut x
. It is
expected that this binds x
to the contents of &mut T
pointer… which it does, but as a mutable binding (it is parsed as
&(mut x)
), meaning something like
for &mut x in some_iterator_over_and_mut {
println!("{}", x)
}
gives an unused mutability warning. NB. it’s somewhat rare that one
would want to pattern match to directly bind a name to the contents of
a &mut
(since the normal reason to have a &mut
is to mutate the
thing it points at, but this pattern is (byte) copying the data out,
both before and after this change), but can occur if a type only
offers a &mut
iterator, i.e. types for which a &
one is no more
flexible than the &mut
one.
Detailed design
Add <pat> := &mut <pat>
to the pattern grammar, and require that it is used
when matching on a &mut T
.
Drawbacks
It makes matching through a &mut
more verbose: for &mut (ref mut x, p_) in v.mut_iter()
instead of for &(ref mut x, _) in v.mut_iter()
.
Macros wishing to pattern match on either &
or &mut
need to handle
each case, rather than performing both with a single &
. However,
macros handling these types already need special mut
vs. not
handling if they ever name the types, or if they use ref
vs. ref mut
subpatterns.
It also makes obtaining the current behaviour (binding by-value the
contents of a reference to a mutable local) slightly harder. For a
&mut T
the pattern becomes &mut mut x
, and, at the moment, for a
&T
, it must be matched with &x
and then rebound with let mut x = x;
(since disambiguating like &(mut x)
doesn’t yet work). However,
based on some loose grepping of the Rust repo, both of these are very
rare.
Alternatives
None.
Unresolved questions
None.
- Start Date: 2014-07-24
- RFC PR #: https://github.com/rust-lang/rfcs/pull/184
- Rust Issue #: https://github.com/rust-lang/rust/issues/16950
Summary
Add simple syntax for accessing values within tuples and tuple structs behind a feature gate.
Motivation
Right now accessing fields of tuples and tuple structs is incredibly painful—one
must rely on pattern-matching alone to extract values. This became such a
problem that twelve traits were created in the standard library
(core::tuple::Tuple*
) to make tuple value accesses easier, adding .valN()
,
.refN()
, and .mutN()
methods to help this. But this is not a very nice
solution—it requires the traits to be implemented in the standard library, not
the language, and for those traits to be imported on use. On the whole this is
not a problem, because most of the time std::prelude::*
is imported, but this
is still a hack which is not a real solution to the problem at hand. It also
only supports tuples of length up to twelve, which is normally not a problem but
emphasises how bad the current situation is.
Detailed design
Add syntax of the form <expr>.<integer>
for accessing values within tuples and
tuple structs. This (and the functionality it provides) would only be allowed
when the feature gate tuple_indexing
is enabled. This syntax is recognised
wherever an unsuffixed integer literal is found in place of the normal field or
method name expected when accessing fields with .
. Because the parser would be
expecting an integer, not a float, an expression like expr.0.1
would be a
syntax error (because 0.1
would be treated as a single token).
Tuple/tuple struct field access behaves the same way as accessing named fields on normal structs:
// With tuple struct
struct Foo(int, int);
let mut foo = Foo(3, -15);
foo.0 = 5;
assert_eq!(foo.0, 5);
// With normal struct
struct Foo2 { _0: int, _1: int }
let mut foo2 = Foo2 { _0: 3, _1: -15 };
foo2._0 = 5;
assert_eq!(foo2._0, 5);
Effectively, a tuple or tuple struct field is just a normal named field with an integer for a name.
Drawbacks
This adds more complexity that is not strictly necessary.
Alternatives
Stay with the status quo. Either recommend using a struct with named fields or
suggest using pattern-matching to extract values. If extracting individual
fields of tuples is really necessary, the TupleN
traits could be used instead,
and something like #[deriving(Tuple3)]
could possibly be added for tuple
structs.
Unresolved questions
None.
- Start Date: 2014-08-06
- RFC PR: rust-lang/rfcs#192
- Rust Issue: rust-lang/rust#16462
Summary
- Remove the special-case bound
'static
and replace with a generalized lifetime bound that can be used on objects and type parameters. - Remove the rules that aim to prevent references from being stored into objects and replace with a simple lifetime check.
- Tighten up type rules pertaining to reference lifetimes and well-formed types containing references.
- Introduce explicit lifetime bounds (
'a:'b
), with the meaning that the lifetime'a
outlives the lifetime'b
. These exist today but are always inferred; this RFC adds the ability to specify them explicitly, which is sometimes needed in more complex cases.
Motivation
Currently, the type system is not supposed to allow references to escape into object types. However, there are various bugs where it fails to prevent this from happening. Moreover, it is very useful (and frequently necessary) to store a reference into an object. Moreover, the current treatment of generic types is in some cases naive and not obviously sound.
Detailed design
Lifetime bounds on parameters
The heart of the new design is the concept of a lifetime bound. In fact,
this (sort of) exists today in the form of the 'static
bound:
fn foo<A:'static>(x: A) { ... }
Here, the notation 'static
means “all borrowed content within A
outlives the lifetime 'static
”. (Note that when we say that
something outlives a lifetime, we mean that it lives at least that
long. In other words, for any lifetime 'a
, 'a
outlives 'a
. This
is similar to how we say that every type T
is a subtype of itself.)
In the newer design, it is possible to use an arbitrary lifetime as a
bound, and not just 'static
:
fn foo<'a, A:'a>(x: A) { ... }
Explicit lifetime bounds are in fact only rarely necessary, for two reasons:
- The compiler is often able to infer this relationship from the argument and return types. More on this below.
- It is only important to bound the lifetime of a generic type like
A
when one of two things is happening (and both of these are cases where the inference generally is sufficient):- A borrowed pointer to an
A
instance (i.e., value of type&A
) is being consumed or returned. - A value of type
A
is being closed over into an object reference (or closure, which per the unboxed closures RFC is really the same thing).
- A borrowed pointer to an
Note that, per RFC 11, these lifetime bounds may appear in types as well (this is important later on). For example, an iterator might be declared:
struct Items<'a, T:'a> {
v: &'a Collection<T>
}
Here, the constraint T:'a
indicates that the data being iterated
over must live at least as long as the collection (logically enough).
Lifetime bounds on object types
Like parameters, all object types have a lifetime bound. Unlike parameter types, however, object types are required to have exactly one bound. This bound can be either specified explicitly or derived from the traits that appear in the object type. In general, the rule is as follows:
- If an explicit bound is specified, use that.
- Otherwise, let S be the set of lifetime bounds we can derive.
- Otherwise, if S contains ’static, use ’static.
- Otherwise, if S is a singleton set, use that.
- Otherwise, error.
Here are some examples:
trait IsStatic : 'static { }
trait Is<'a> : 'a { }
// Type Bounds
// IsStatic 'static
// Is<'a> 'a
// IsStatic+Is<'a> 'static+'a
// IsStatic+'a 'static+'a
// IsStatic+Is<'a>+'b 'static,'a,'b
Object types must have exactly one bound – zero bounds is not acceptable. Therefore, if an object type with no derivable bounds appears, we will supply a default lifetime using the normal rules:
trait Writer { /* no derivable bounds */ }
struct Foo<'a> {
Box<Writer>, // Error: try Box<Writer+'static> or Box<Writer+'a>
Box<Writer+Send>, // OK: Send implies 'static
&'a Writer, // Error: try &'a (Writer+'a)
}
fn foo(a: Box<Writer>, // OK: Sugar for Box<Writer+'a> where 'a fresh
b: &Writer) // OK: Sugar for &'b (Writer+'c) where 'b, 'c fresh
{ ... }
This kind of annotation can seem a bit tedious when using object types extensively, though type aliases can help quite a bit:
type WriterObj = Box<Writer+'static>;
type WriterRef<'a> = &'a (Writer+'a);
The unresolved questions section discussed possibles ways to lighten the burden.
See Appendix B for the motivation on why object types are permitted to have exactly one lifetime bound.
Specifying relations between lifetimes
Currently, when a type or fn has multiple lifetime parameters, there is no facility to explicitly specify a relationship between them. For example, in a function like this:
fn foo<'a, 'b>(...) { ... }
the lifetimes 'a
and 'b
are declared as independent. In some
cases, though, it can be important that there be a relation between
them. In most cases, these relationships can be inferred (and in fact
are inferred today, see below), but it is useful to be able to state
them explicitly (and necessary in some cases, see below).
A lifetime bound is written 'a:'b
and it means that “'a
outlives
'b
”. For example, if foo
were declared like so:
fn foo<'x, 'y:'x>(...) { ... }
that would indicate that the lifetime ’x
was shorter than (or equal
to) 'y
.
The “type must outlive” and well-formedness relation
Many of the rules to come make use of a “type must outlive” relation,
written T outlives 'a
. This relation means primarily that all
borrowed data in T
is known to have a lifetime of at least ’a
(hence the name). However, the relation also guarantees various basic
lifetime constraints are met. For example, for every reference type
&'b U
that is found within T
, it would be required that U outlives 'b
(and that 'b
outlives 'a
).
In fact, T outlives 'a
is defined on another function WF(T:'a)
,
which yields up a list of lifetime relations that must hold for T
to
be well-formed and to outlive 'a
. It is not necessary to understand
the details of this relation in order to follow the rest of the RFC, I
will defer its precise specification to an appendix below.
For this section, it suffices to give some examples:
// int always outlives any region
WF(int : 'a) = []
// a reference with lifetime 'a outlives 'b if 'a outlives 'b
WF(&'a int : 'b) = ['a : 'b]
// the outer reference must outlive 'c, and the inner reference
// must outlive the outer reference
WF(&'a &'b int : 'c) = ['a : 'c, 'b : 'a]
// Object type with bound 'static
WF(SomeTrait+'static : 'a) = ['static : 'a]
// Object type with bound 'a
WF(SomeTrait+'a : 'b) = ['a : 'b]
Rules for when object closure is legal
Whenever data of type T
is closed over to form an object, the type
checker will require that T outlives 'a
where 'a
is the primary
lifetime bound of the object type.
Rules for types to be well-formed
Currently we do not apply any tests to the types that appear in type
declarations. Per RFC 11, however, this should change, as we intend to
enforce trait bounds on types, wherever those types appear. Similarly,
we should be requiring that types are well-formed with respect to the
WF
function. This means that a type like the following would be
illegal without a lifetime bound on the type parameter T
:
struct Ref<'a, T> { c: &'a T }
This is illegal because the field c
has type &'a T
, which is only
well-formed if T:'a
. Per usual practice, this RFC does not propose
any form of inference on struct declarations and instead requires all
conditions to be spelled out (this is in contrast to fns and methods,
see below).
Rules for expression type validity
We should add the condition that for every expression with lifetime
'e
and type T
, then T outlives 'e
. We already enforce this in
many special cases but not uniformly.
Inference
The compiler will infer lifetime bounds on both type parameters and
region parameters as follows. Within a function or method, we apply
the wellformedness function WF
to each function or parameter type.
This yields up a set of relations that must hold. The idea here is
that the caller could not have type checked unless the types of the
arguments were well-formed, so that implies that the callee can assume
that those well-formedness constraints hold.
As an example, in the following function:
fn foo<'a, A>(x: &'a A) { ... }
the callee here can assume that the type parameter A
outlives the
lifetime 'a
, even though that was not explicitly declared.
Note that the inference also pulls in constraints that were declared
on the types of arguments. So, for example, if there is a type Items
declared as follows:
struct Items<'a, T:'a> { ... }
And a function that takes an argument of type Items
:
fn foo<'a, T>(x: Items<'a, T>) { ... }
The inference rules will conclude that T:'a
because the Items
type
was declared with that bound.
In practice, these inference rules largely remove the need to manually declare lifetime relations on types. When porting the existing library and rustc over to these rules, I had to add explicit lifetime bounds to exactly one function (but several types, almost exclusively iterators).
Note that this sort of inference is already done. This RFC simply
proposes a more extensive version that also includes bounds of the
form X:'a
, where X
is a type parameter.
What does all this mean in practice?
This RFC has a lot of details. The main implications for end users are:
-
Object types must specify a lifetime bound when they appear in a type. This most commonly means changing
Box<Trait>
toBox<Trait+'static>
and&'a Trait
to&'a Trait+'a
. -
For types that contain references to generic types, lifetime bounds are needed in the type definition. This comes up most often in iterators:
struct Items<'a, T:'a> { x: &'a [T] }
Here, the presence of
&'a [T]
within the type definition requires that the type checker can show thatT outlives 'a
which in turn requires the boundT:'a
on the type definition. These bounds are rarely outside of type definitions, because they are almost always implied by the types of the arguments. -
It is sometimes, but rarely, necessary to use lifetime bounds, specifically around double indirections (references to references, often the second reference is contained within a struct). For example:
struct GlobalContext<'global> { arena: &'global Arena } struct LocalContext<'local, 'global:'local> { x: &'local mut Context<'global> }
Here, we must know that the lifetime
'global
outlives'local
in order for this type to be well-formed.
Phasing
Some parts of this RFC require new syntax and thus must be phased in. The current plan is to divide the implementation three parts:
- Implement support for everything in this RFC except for region bounds and requiring that every expression type be well-formed. Enforcing the latter constraint leads to type errors that require lifetime bounds to resolve.
- Implement support for
'a:'b
notation to be parsed under a feature gateissue_5723_bootstrap
. - Implement the final bits of the RFC:
- Bounds on lifetime parameters
- Wellformedness checks on every expression
- Wellformedness checks in type definitions
Parts 1 and 2 can be landed simultaneously, but part 3 requires a snapshot. Parts 1 and 2 have largely been written. Depending on precisely how the timing works out, it might make sense to just merge parts 1 and 3.
Drawbacks / Alternatives
If we do not implement some solution, we could continue with the current approach (but patched to be sound) of banning references from being closed over in object types. I consider this a non-starter.
Unresolved questions
Inferring wellformedness bounds
Under this RFC, it is required to write bounds on struct types which are in principle inferable from their contents. For example, iterators tend to follow a pattern like:
struct Items<'a, T:'a> {
x: &'a [T]
}
Note that T
is bounded by 'a
. It would be possible to infer these
bounds, but I’ve stuck to our current principle that type definitions
are always fully spelled out. The danger of inference is that it
becomes unclear why a particular constraint exists if one must
traverse the type hierarchy deeply to find its origin. This could
potentially be addressed with better error messages, though our track
record for lifetime error messages is not very good so far.
Also, there is a potential interaction between this sort of inference and the description of default trait bounds below.
Default trait bounds
When referencing a trait object, it is almost always the case that one follows certain fixed patterns:
Box<Trait+'static>
Rc<Trait+'static>
(once DST works)&'a (Trait+'a)
- and so on.
You might think that we should simply provide some kind of defaults
that are sensitive to where the Trait
appears. The same is probably
true of struct type parameters (in other words, &'a SomeStruct<'a>
is a very common pattern).
However, there are complications:
-
What about a type like
struct Ref<'a, T:'a> { x: &'a T }
?Ref<'a, Trait>
should really work the same way as&'a Trait
. One way that I can see to do this is to drive the defaulting based on the default trait bounds of theT
type parameter – but if we do that, it is both a non-local default (you have to consult the definition ofRef
) and interacts with the potential inference described in the previous section. -
There are reasons to want a type like
Box<Trait+'a>
. For example, the macro parser includes a function like:fn make_macro_ext<'cx>(cx: &'cx Context, ...) -> Box<MacroExt+'cx>
In other words, this function returns an object that closes over the macro context. In such a case, if
Box<MacroExt>
implies a static bound, then taking ownership of this macro object would require a signature like:fn take_macro_ext<'cx>(b: Box<MacroExt+'cx>) { }
Note that the
'cx
variable is only used in one place. It’s purpose is just to disable the'static
default that would otherwise be inserted.
Appendix: Definition of the outlives relation and well-formedness
To make this more specific, we can “formally” model the Rust type system as:
T = scalar (int, uint, fn(...)) // Boring stuff
| *const T // Unsafe pointer
| *mut T // Unsafe pointer
| Id<P> // Nominal type (struct, enum)
| &'x T // Reference
| &'x mut T // Mutable reference
| {TraitReference<P>}+'x // Object type
| X // Type variable
P = {'x} + {T}
We can define a function WF(T : 'a)
which, given a type T
and
lifetime 'a
yields a list of 'b:'c
or X:'d
pairs. For each pair
'b:'c
, the lifetime 'b
must outlive the lifetime 'c
for the type
T
to be well-formed in a location with lifetime 'a
. For each pair
X:'d
, the type parameter X
must outlive the lifetime 'd
.
WF(int : 'a)
yields an empty listWF(X:'a)
whereX
is a type parameter yields(X:'a)
.WF(Foo<P>:'a)
whereFoo<P>
is an enum or struct type yields:- For each lifetime parameter
'b
that is contravariant or invariant,'b : 'a
. - For each type parameter
T
that is covariant or invariant, the results ofWF(T : 'a)
. - The lifetime bounds declared on
Foo
’s lifetime or type parameters. - The reasoning here is that if we can reach borrowed data with
lifetime
'a
throughFoo<'a>
, then'a
must be contra- or invariant. Covariant lifetimes only occur in “setter” situations. Analogous reasoning applies to the type case.
- For each lifetime parameter
WF(T:'a)
whereT
is an object type:- For the primary bound
'b
,'b : 'a
. - For each derived bound
'c
ofT
,'b : 'c
- Motivation: The primary bound of an object type implies that all other bounds are met. This simplifies some of the other formulations and does not represent a loss of expressiveness.
- For the primary bound
We can then say that T outlives 'a
if all lifetime relations
returned by WF(T:'a)
hold.
Appendix B: Why object types must have exactly one bound
The motivation is that handling multiple bounds is overwhelmingly
complicated to reason about and implement. In various places,
constraints arise of the form all i. exists j. R[i] <= R[j]
, where
R
is a list of lifetimes. This is challenging for lifetime
inference, since there are many options for it to choose from, and
thus inference is no longer a fixed-point iteration. Moreover, it
doesn’t seem to add any particular expressiveness.
The places where this becomes important are:
- Checking lifetime bounds when data is closed over into an object type
- Subtyping between object types, which would most naturally be contravariant in the lifetime bound
Similarly, requiring that the “master” bound on object lifetimes outlives all other bounds also aids inference. Now, given a type like the following:
trait Foo<'a> : 'a { }
trait Bar<'b> : 'b { }
...
let x: Box<Foo<'a>+Bar<'b>>
the inference engine can create a fresh lifetime variable '0
for the
master bound and then say that '0:'a
and '0:'b
. Without the
requirement that '0
be a master bound, it would be somewhat unclear
how '0
relates to 'a
and 'b
(in fact, there would be no
necessary relation). But if there is no necessary relation, then when
closing over data, one would have to ensure that the closed over data
outlives all derivable lifetime bounds, which again creates a
constraint of the form all i. exists j.
.
- Start Date: 2014-08-09
- RFC PR #: rust-lang/rfcs#194
- Rust Issue: rust-lang/rust#17490
Summary
The #[cfg(...)]
attribute provides a mechanism for conditional compilation of
items in a Rust crate. This RFC proposes to change the syntax of #[cfg]
to
make more sense as well as enable expansion of the conditional compilation
system to attributes while maintaining a single syntax.
Motivation
In the current implementation, #[cfg(...)]
takes a comma separated list of
key
, key = "value"
, not(key)
, or not(key = "value")
. An individual
#[cfg(...)]
attribute “matches” if all of the contained cfg patterns match
the compilation environment, and an item preserved if it either has no
#[cfg(...)]
attributes or any of the #[cfg(...)]
attributes present
match.
This is problematic for several reasons:
- It is excessively verbose in certain situations. For example, implementing
the equivalent of
(a AND (b OR c OR d))
requires three separate attributes anda
to be duplicated in each. - It differs from all other attributes in that all
#[cfg(...)]
attributes on an item must be processed together instead of in isolation. This change will move#[cfg(...)]
closer to implementation as a normal syntax extension.
Detailed design
The <p>
inside of #[cfg(<p>)]
will be called a cfg pattern and have a
simple recursive syntax:
key
is a cfg pattern and will match ifkey
is present in the compilation environment.key = "value"
is a cfg pattern and will match if a mapping fromkey
tovalue
is present in the compilation environment. At present, key-value pairs only exist for compiler defined keys such astarget_os
andendian
.not(<p>)
is a cfg pattern if<p>
is and matches if<p>
does not match.all(<p>, ...)
is a cfg pattern if all of the comma-separated<p>
s are cfg patterns and all of them match.any(<p>, ...)
is a cfg pattern if all of the comma-separated<p>
s are cfg patterns and any of them match.
If an item is tagged with #[cfg(<p>)]
, that item will be stripped from the
AST if the cfg pattern <p>
does not match.
One implementation hazard is that the semantics of
#[cfg(a)]
#[cfg(b)]
fn foo() {}
will change from “include foo
if either of a
and b
are present in the
compilation environment” to “include foo
if both of a
and b
are present
in the compilation environment”. To ease the transition, the old semantics of
multiple #[cfg(...)]
attributes will be maintained as a special case, with a
warning. After some reasonable period of time, the special case will be
removed.
In addition, #[cfg(a, b, c)]
will be accepted with a warning and be
equivalent to #[cfg(all(a, b, c))]
. Again, after some reasonable period of
time, this behavior will be removed as well.
The cfg!()
syntax extension will be modified to accept cfg patterns as well.
A #[cfg_attr(<p>, <attr>)]
syntax extension will be added
(PR 16230) which will expand to
#[<attr>]
if the cfg pattern <p>
matches. The test harness’s
#[ignore]
attribute will have its built-in cfg filtering
functionality stripped in favor of #[cfg_attr(<p>, ignore)]
.
Drawbacks
While the implementation of this change in the compiler will be straightforward, the effects on downstream code will be significant, especially in the standard library.
Alternatives
all
and any
could be renamed to and
and or
, though I feel that the
proposed names read better with the function-like syntax and are consistent
with Iterator::all
and Iterator::any
.
Issue #2119 proposed the
addition of ||
and &&
operators and parentheses to the attribute syntax
to result in something like #[cfg(a || (b && c)]
. I don’t favor this proposal
since it would result in a major change to the attribute syntax for relatively
little readability gain.
Unresolved questions
How long should multiple #[cfg(...)]
attributes on a single item be
forbidden? It should probably be at least until after 0.12 releases.
Should we permanently keep the behavior of treating #[cfg(a, b)]
as
#[cfg(all(a, b))]
? It is the common case, and adding this interpretation
can reduce the noise level a bit. On the other hand, it may be a bit confusing
to read as it’s not immediately clear if it will be processed as and(..)
or
all(..)
.
- Start Date: 2014-08-04
- RFC PR #: rust-lang/rfcs#195
- Rust Issue #: rust-lang/rust#17307
Summary
This RFC extends traits with associated items, which make generic programming more convenient, scalable, and powerful. In particular, traits will consist of a set of methods, together with:
- Associated functions (already present as “static” functions)
- Associated consts
- Associated types
- Associated lifetimes
These additions make it much easier to group together a set of related types, functions, and constants into a single package.
This RFC also provides a mechanism for multidispatch traits, where the impl
is selected based on multiple types. The connection to associated items will
become clear in the detailed text below.
Note: This RFC was originally accepted before RFC 246 introduced the distinction between const and static items. The text has been updated to clarify that associated consts will be added rather than statics, and to provide a summary of restrictions on the initial implementation of associated consts. Other than that modification, the proposal has not been changed to reflect newer Rust features or syntax.
Motivation
A typical example where associated items are helpful is data structures like graphs, which involve at least three types: nodes, edges, and the graph itself.
In today’s Rust, to capture graphs as a generic trait, you have to take the additional types associated with a graph as parameters:
trait Graph<N, E> {
fn has_edge(&self, &N, &N) -> bool;
...
}
The fact that the node and edge types are parameters is confusing, since any concrete graph type is associated with a unique node and edge type. It is also inconvenient, because code working with generic graphs is likewise forced to parameterize, even when not all of the types are relevant:
fn distance<N, E, G: Graph<N, E>>(graph: &G, start: &N, end: &N) -> uint { ... }
With associated types, the graph trait can instead make clear that the node and
edge types are determined by any impl
:
trait Graph {
type N;
type E;
fn has_edge(&self, &N, &N) -> bool;
}
and clients can abstract over them all at once, referring to them through the graph type:
fn distance<G: Graph>(graph: &G, start: &G::N, end: &G::N) -> uint { ... }
The following subsections expand on the above benefits of associated items, as well as some others.
Associated types: engineering benefits for generics
As the graph example above illustrates, associated types do not increase the expressiveness of traits per se, because you can always use extra type parameters to a trait instead. However, associated types provide several engineering benefits:
-
Readability and scalability
Associated types make it possible to abstract over a whole family of types at once, without having to separately name each of them. This improves the readability of generic code (like the
distance
function above). It also makes generics more “scalable”: traits can incorporate additional associated types without imposing an extra burden on clients that don’t care about those types.In today’s Rust, by contrast, adding additional generic parameters to a trait often feels like a very “heavyweight” move.
-
Ease of refactoring/evolution
Because users of a trait do not have to separately parameterize over its associated types, new associated types can be added without breaking all existing client code.
In today’s Rust, by contrast, associated types can only be added by adding more type parameters to a trait, which breaks all code mentioning the trait.
Clearer trait matching
Type parameters to traits can either be “inputs” or “outputs”:
-
Inputs. An “input” type parameter is used to determine which
impl
to use. -
Outputs. An “output” type parameter is uniquely determined by the
impl
, but plays no role in selecting theimpl
.
Input and output types play an important role for type inference and trait coherence rules, which is described in more detail later on.
In the vast majority of current libraries, the only input type is the Self
type implementing the trait, and all other trait type parameters are outputs.
For example, the trait Iterator<A>
takes a type parameter A
for the elements
being iterated over, but this type is always determined by the concrete Self
type (e.g. Items<u8>
) implementing the trait: A
is typically an output.
Additional input type parameters are useful for cases like binary operators,
where you may want the impl
to depend on the types of both
arguments. For example, you might want a trait
trait Add<Rhs, Sum> {
fn add(&self, rhs: &Rhs) -> Sum;
}
to view the Self
and Rhs
types as inputs, and the Sum
type as an output
(since it is uniquely determined by the argument types). This would allow
impl
s to vary depending on the Rhs
type, even though the Self
type is the same:
impl Add<int, int> for int { ... }
impl Add<Complex, Complex> for int { ... }
Today’s Rust does not make a clear distinction between input and output type
parameters to traits. If you attempted to provide the two impl
s above, you
would receive an error like:
error: conflicting implementations for trait `Add`
This RFC clarifies trait matching by:
- Treating all trait type parameters as input types, and
- Providing associated types, which are output types.
In this design, the Add
trait would be written and implemented as follows:
// Self and Rhs are *inputs*
trait Add<Rhs> {
type Sum; // Sum is an *output*
fn add(&self, &Rhs) -> Sum;
}
impl Add<int> for int {
type Sum = int;
fn add(&self, rhs: &int) -> int { ... }
}
impl Add<Complex> for int {
type Sum = Complex;
fn add(&self, rhs: &Complex) -> Complex { ... }
}
With this approach, a trait declaration like trait Add<Rhs> { ... }
is really
defining a family of traits, one for each choice of Rhs
. One can then
provide a distinct impl
for every member of this family.
Expressiveness
Associated types, lifetimes, and functions can already be expressed in today’s Rust, though it is unwieldy to do so (as argued above).
But associated consts cannot be expressed using today’s traits.
For example, today’s Rust includes a variety of numeric traits, including
Float
, which must currently expose constants as static functions:
trait Float {
fn nan() -> Self;
fn infinity() -> Self;
fn neg_infinity() -> Self;
fn neg_zero() -> Self;
fn pi() -> Self;
fn two_pi() -> Self;
...
}
Because these functions cannot be used in constant expressions, the modules for float types also export a separate set of constants as consts, not using traits.
Associated constants would allow the consts to live directly on the traits:
trait Float {
const NAN: Self;
const INFINITY: Self;
const NEG_INFINITY: Self;
const NEG_ZERO: Self;
const PI: Self;
const TWO_PI: Self;
...
}
Why now?
The above motivations aside, it may not be obvious why adding associated types now (i.e., pre-1.0) is important. There are essentially two reasons.
First, the design presented here is not backwards compatible, because it re-interprets trait type parameters as inputs for the purposes of trait matching. The input/output distinction has several ramifications on coherence rules, type inference, and resolution, which are all described later on in the RFC.
Of course, it might be possible to give a somewhat less ideal design where associated types can be added later on without changing the interpretation of existing trait type parameters. For example, type parameters could be explicitly marked as inputs, and otherwise assumed to be outputs. That would be unfortunate, since associated types would also be outputs – leaving the language with two ways of specifying output types for traits.
But the second reason is for the library stabilization process:
-
Since most existing uses of trait type parameters are intended as outputs, they should really be associated types instead. Making promises about these APIs as they currently stand risks locking the libraries into a design that will seem obsolete as soon as associated items are added. Again, this risk could probably be mitigated with a different, backwards-compatible associated item design, but at the cost of cruft in the language itself.
-
The binary operator traits (e.g.
Add
) should be multidispatch. It does not seem possible to stabilize them now in a way that will support moving to multidispatch later. -
There are some thorny problems in the current libraries, such as the
_equiv
methods accumulating inHashMap
, that can be solved using associated items. (See “Defaults” below for more on this specific example.) Additional examples include traits for error propagation and for conversion (to be covered in future RFCs). Adding these traits would improve the quality and consistency of our 1.0 library APIs.
Detailed design
Trait headers
Trait headers are written according to the following grammar:
TRAIT_HEADER =
'trait' IDENT [ '<' INPUT_PARAMS '>' ] [ ':' BOUNDS ] [ WHERE_CLAUSE ]
INPUT_PARAMS = INPUT_PARAM { ',' INPUT_PARAM }* [ ',' ]
INPUT_PARAM = IDENT [ ':' BOUNDS ]
BOUNDS = BOUND { '+' BOUND }* [ '+' ]
BOUND = IDENT [ '<' ARGS '>' ]
ARGS = INPUT_ARGS
| OUTPUT_CONSTRAINTS
| INPUT_ARGS ',' OUTPUT_CONSTRAINTS
INPUT_ARGS = TYPE { ',' TYPE }*
OUTPUT_CONSTRAINTS = OUTPUT_CONSTRAINT { ',' OUTPUT_CONSTRAINT }*
OUTPUT_CONSTRAINT = IDENT '=' TYPE
NOTE: The grammar for WHERE_CLAUSE
and BOUND
is explained in detail in
the subsection “Constraining associated types” below.
All type parameters to a trait are considered inputs, and can be used to select
an impl
; conceptually, each distinct instantiation of the types yields a
distinct trait. More details are given in the section “The input/output type
distinction” below.
Trait bodies: defining associated items
Trait bodies are expanded to include three new kinds of items: consts, types, and lifetimes:
TRAIT = TRAIT_HEADER '{' TRAIT_ITEM* '}'
TRAIT_ITEM =
... <existing productions>
| 'const' IDENT ':' TYPE [ '=' CONST_EXP ] ';'
| 'type' IDENT [ ':' BOUNDS ] [ WHERE_CLAUSE ] [ '=' TYPE ] ';'
| 'lifetime' LIFETIME_IDENT ';'
Traits already support associated functions, which had previously been called “static” functions.
The BOUNDS
and WHERE_CLAUSE
on associated types are obligations for the
implementor of the trait, and assumptions for users of the trait:
trait Graph {
type N: Show + Hash;
type E: Show + Hash;
...
}
impl Graph for MyGraph {
// Both MyNode and MyEdge must implement Show and Hash
type N = MyNode;
type E = MyEdge;
...
}
fn print_nodes<G: Graph>(g: &G) {
// here, can assume G::N implements Show
...
}
Namespacing/shadowing for associated types
Associated types may have the same name as existing types in scope, except for type parameters to the trait:
struct Foo { ... }
trait Bar<Input> {
type Foo; // this is allowed
fn into_foo(self) -> Foo; // this refers to the trait's Foo
type Input; // this is NOT allowed
}
By not allowing name clashes between input and output types, keep open the possibility of later allowing syntax like:
Bar<Input=u8, Foo=uint>
where both input and output parameters are constrained by name. And anyway, there is no use for clashing input/output names.
In the case of a name clash like Foo
above, if the trait needs to refer to the
outer Foo
for some reason, it can always do so by using a type
alias
external to the trait.
Defaults
Notice that associated consts and types both permit defaults, just as trait methods and functions can provide defaults.
Defaults are useful both as a code reuse mechanism, and as a way to expand the items included in a trait without breaking all existing implementors of the trait.
Defaults for associated types, however, present an interesting question: can default methods assume the default type? In other words, is the following allowed?
trait ContainerKey : Clone + Hash + Eq {
type Query: Hash = Self;
fn compare(&self, other: &Query) -> bool { self == other }
fn query_to_key(q: &Query) -> Self { q.clone() };
}
impl ContainerKey for String {
type Query = str;
fn compare(&self, other: &str) -> bool {
self.as_slice() == other
}
fn query_to_key(q: &str) -> String {
q.into_string()
}
}
impl<K,V> HashMap<K,V> where K: ContainerKey {
fn find(&self, q: &K::Query) -> &V { ... }
}
In this example, the ContainerKey
trait is used to associate a “Query
” type
(for lookups) with an owned key type. This resolves the thorny “equiv” problem
in HashMap
, where the hash map keys are String
s but you want to index the
hash map with &str
values rather than &String
values, i.e. you want the
following to work:
// H: HashMap<String, SomeType>
H.find("some literal")
rather than having to write
H.find(&"some literal".to_string())`
The current solution involves duplicating the API surface with _equiv
methods
that use the somewhat subtle Equiv
trait, but the associated type approach
makes it easy to provide a simple, single API that covers the same use cases.
The defaults for ContainerKey
just assume that the owned key and lookup key
types are the same, but the default methods have to assume the default
associated types in order to work.
For this to work, it must not be possible for an implementor of ContainerKey
to override the default Query
type while leaving the default methods in place,
since those methods may no longer typecheck.
We deal with this in a very simple way:
-
If a trait implementor overrides any default associated types, they must also override all default functions and methods.
-
Otherwise, a trait implementor can selectively override individual default methods/functions, as they can today.
Trait implementations
Trait impl
syntax is much the same as before, except that const, type, and
lifetime items are allowed:
IMPL_ITEM =
... <existing productions>
| 'const' IDENT ':' TYPE '=' CONST_EXP ';'
| 'type' IDENT' '=' 'TYPE' ';'
| 'lifetime' LIFETIME_IDENT '=' LIFETIME_REFERENCE ';'
Any type
implementation must satisfy all bounds and where clauses in the
corresponding trait item.
Referencing associated items
Associated items are referenced through paths. The expression path grammar was updated as part of UFCS, but to accommodate associated types and lifetimes we need to update the type path grammar as well.
The full grammar is as follows:
EXP_PATH
= EXP_ID_SEGMENT { '::' EXP_ID_SEGMENT }*
| TYPE_SEGMENT { '::' EXP_ID_SEGMENT }+
| IMPL_SEGMENT { '::' EXP_ID_SEGMENT }+
EXP_ID_SEGMENT = ID [ '::' '<' TYPE { ',' TYPE }* '>' ]
TY_PATH
= TY_ID_SEGMENT { '::' TY_ID_SEGMENT }*
| TYPE_SEGMENT { '::' TY_ID_SEGMENT }*
| IMPL_SEGMENT { '::' TY_ID_SEGMENT }+
TYPE_SEGMENT = '<' TYPE '>'
IMPL_SEGMENT = '<' TYPE 'as' TRAIT_REFERENCE '>'
TRAIT_REFERENCE = ID [ '<' TYPE { ',' TYPE * '>' ]
Here are some example paths, along with what they might be referencing
// Expression paths ///////////////////////////////////////////////////////////////
a::b::c // reference to a function `c` in module `a::b`
a::<T1, T2> // the function `a` instantiated with type arguments `T1`, `T2`
Vec::<T>::new // reference to the function `new` associated with `Vec<T>`
<Vec<T> as SomeTrait>::some_fn
// reference to the function `some_fn` associated with `SomeTrait`,
// as implemented by `Vec<T>`
T::size_of // the function `size_of` associated with the type or trait `T`
<T>::size_of // the function `size_of` associated with `T` _viewed as a type_
<T as SizeOf>::size_of
// the function `size_of` associated with `T`'s impl of `SizeOf`
// Type paths /////////////////////////////////////////////////////////////////////
a::b::C // reference to a type `C` in module `a::b`
A<T1, T2> // type A instantiated with type arguments `T1`, `T2`
Vec<T>::Iter // reference to the type `Iter` associated with `Vec<T>
<Vec<T> as SomeTrait>::SomeType
// reference to the type `SomeType` associated with `SomeTrait`,
// as implemented by `Vec<T>`
Ways to reference items
Next, we’ll go into more detail on the meaning of each kind of path.
For the sake of discussion, we’ll suppose we’ve defined a trait like the following:
trait Container {
type E;
fn empty() -> Self;
fn insert(&mut self, E);
fn contains(&self, &E) -> bool where E: PartialEq;
...
}
impl<T> Container for Vec<T> {
type E = T;
fn empty() -> Vec<T> { Vec::new() }
...
}
Via an ID_SEGMENT
prefix
When the prefix resolves to a type
The most common way to get at an associated item is through a type parameter with a trait bound:
fn pick<C: Container>(c: &C) -> Option<&C::E> { ... }
fn mk_with_two<C>() -> C where C: Container, C::E = uint {
let mut cont = C::empty(); // reference to associated function
cont.insert(0);
cont.insert(1);
cont
}
For these references to be valid, the type parameter must be known to implement the relevant trait:
// Knowledge via bounds
fn pick<C: Container>(c: &C) -> Option<&C::E> { ... }
// ... or equivalently, where clause
fn pick<C>(c: &C) -> Option<&C::E> where C: Container { ... }
// Knowledge via ambient constraints
struct TwoContainers<C1: Container, C2: Container>(C1, C2);
impl<C1: Container, C2: Container> TwoContainers<C1, C2> {
fn pick_one(&self) -> Option<&C1::E> { ... }
fn pick_other(&self) -> Option<&C2::E> { ... }
}
Note that Vec<T>::E
and Vec::<T>::empty
are also valid type and function
references, respectively.
For cases like C::E
or Vec<T>::E
, the path begins with an ID_SEGMENT
prefix that itself resolves to a type: both C
and Vec<T>
are types. In
general, a path PREFIX::REST_OF_PATH
where PREFIX
resolves to a type is
equivalent to using a TYPE_SEGMENT
prefix <PREFIX>::REST_OF_PATH
. So, for
example, following are all equivalent:
fn pick<C: Container>(c: &C) -> Option<&C::E> { ... }
fn pick<C: Container>(c: &C) -> Option<&<C>::E> { ... }
fn pick<C: Container>(c: &C) -> Option<&<<C>::E>> { ... }
The behavior of TYPE_SEGMENT
prefixes is described in the next subsection.
When the prefix resolves to a trait
However, it is possible for an ID_SEGMENT
prefix to resolve to a trait,
rather than a type. In this case, the behavior of an ID_SEGMENT
varies from
that of a TYPE_SEGMENT
in the following way:
// a reference Container::insert is roughly equivalent to:
fn trait_insert<C: Container>(c: &C, e: C::E);
// a reference <Container>::insert is roughly equivalent to:
fn object_insert<E>(c: &Container<E=E>, e: E);
That is, if PREFIX
is an ID_SEGMENT
that
resolves to a trait Trait
:
-
A path
PREFIX::REST
resolves to the item/pathREST
defined withinTrait
, while treating the type implementing the trait as a type parameter. -
A path
<PREFIX>::REST
treatsPREFIX
as a (DST-style) type, and is hence usable only with trait objects. See the UFCS RFC for more detail.
Note that a path like Container::E
, while grammatically valid, will fail to
resolve since there is no way to tell which impl
to use. A path like
Container::empty
, however, resolves to a function roughly equivalent to:
fn trait_empty<C: Container>() -> C;
Via a TYPE_SEGMENT
prefix
The following text is slightly changed from the UFCS RFC.
When a path begins with a TYPE_SEGMENT
, it is a type-relative path. If this is
the complete path (e.g., <int>
), then the path resolves to the specified
type. If the path continues (e.g., <int>::size_of
) then the next segment is
resolved using the following procedure. The procedure is intended to mimic
method lookup, and hence any changes to method lookup may also change the
details of this lookup.
Given a path <T>::m::...
:
-
Search for members of inherent impls defined on
T
(if any) with the namem
. If any are found, the path resolves to that item. -
Otherwise, let
IN_SCOPE_TRAITS
be the set of traits that are in scope and which contain a member namedm
:- Let
IMPLEMENTED_TRAITS
be those traits fromIN_SCOPE_TRAITS
for which an implementation exists that (may) apply toT
.- There can be ambiguity in the case that
T
contains type inference variables.
- There can be ambiguity in the case that
- If
IMPLEMENTED_TRAITS
is not a singleton set, report an ambiguity error. Otherwise, letTRAIT
be the member ofIMPLEMENTED_TRAITS
. - If
TRAIT
is ambiguously implemented forT
, report an ambiguity error and request further type information. - Otherwise, rewrite the path to
<T as Trait>::m::...
and continue.
- Let
Via a IMPL_SEGMENT
prefix
The following text is somewhat different from the UFCS RFC.
When a path begins with an IMPL_SEGMENT
, it is a reference to an item defined
from a trait. Note that such paths must always have a follow-on member m
(that
is, <T as Trait>
is not a complete path, but <T as Trait>::m
is).
To resolve the path, first search for an applicable implementation of Trait
for T
. If no implementation can be found – or the result is ambiguous – then
report an error. Note that when T
is a type parameter, a bound T: Trait
guarantees that there is such an implementation, but does not count for
ambiguity purposes.
Otherwise, resolve the path to the member of the trait with the substitution
Self => T
and continue.
This apparently straightforward algorithm has some subtle consequences, as illustrated by the following example:
trait Foo {
type T;
fn as_T(&self) -> &T;
}
// A blanket impl for any Show type T
impl<T: Show> Foo for T {
type T = T;
fn as_T(&self) -> &T { self }
}
fn bounded<U: Foo>(u: U) where U::T: Show {
// Here, we just constrain the associated type directly
println!("{}", u.as_T())
}
fn blanket<U: Show>(u: U) {
// the blanket impl applies to U, so we know that `U: Foo` and
// <U as Foo>::T = U (and, of course, U: Show)
println!("{}", u.as_T())
}
fn not_allowed<U: Foo>(u: U) {
// this will not compile, since <U as Trait>::T is not known to
// implement Show
println!("{}", u.as_T())
}
This example includes three generic functions that make use of an associated type; the first two will typecheck, while the third will not.
-
The first case,
bounded
, places aShow
constraint directly on the otherwise-abstract associated typeU::T
. Hence, it is allowed to assume thatU::T: Show
, even though it does not know the concrete implementation ofFoo
forU
. -
The second case,
blanket
, places aShow
constraint on the typeU
, which means that the blanketimpl
ofFoo
applies even though we do not know the concrete type thatU
will be. That fact means, moreover, that we can compute exactly what the associated typeU::T
will be, and know that it will satisfyShow
. Coherence guarantees that that the blanketimpl
is the only one that could apply toU
. (See the section “Impl specialization” under “Unresolved questions” for a deeper discussion of this point.) -
The third case assumes only that
U: Foo
, and therefore nothing is known about the associated typeU::T
. In particular, the function cannot assume thatU::T: Show
.
The resolution rules also interact with instantiation of type parameters in an intuitive way. For example:
trait Graph {
type N;
type E;
...
}
impl Graph for MyGraph {
type N = MyNode;
type E = MyEdge;
...
}
fn pick_node<G: Graph>(t: &G) -> &G::N {
// the type G::N is abstract here
...
}
let G = MyGraph::new();
...
pick_node(G) // has type: <MyGraph as Graph>::N = MyNode
Assuming there are no blanket implementations of Graph
, the pick_node
function knows nothing about the associated type G::N
. However, a client of
pick_node
that instantiates it with a particular concrete graph type will also
know the concrete type of the value returned from the function – here, MyNode
.
Scoping of trait
and impl
items
Associated types are frequently referred to in the signatures of a trait’s methods and associated functions, and it is natural and convenient to refer to them directly.
In other words, writing this:
trait Graph {
type N;
type E;
fn has_edge(&self, &N, &N) -> bool;
...
}
is more appealing than writing this:
trait Graph {
type N;
type E;
fn has_edge(&self, &Self::N, &Self::N) -> bool;
...
}
This RFC proposes to treat both trait
and impl
bodies (both
inherent and for traits) the same way we treat mod
bodies: all
items being defined are in scope. In particular, methods are in scope
as UFCS-style functions:
trait Foo {
type AssocType;
lifetime 'assoc_lifetime;
const ASSOC_CONST: uint;
fn assoc_fn() -> Self;
// Note: 'assoc_lifetime and AssocType in scope:
fn method(&self, Self) -> &'assoc_lifetime AssocType;
fn default_method(&self) -> uint {
// method in scope UFCS-style, assoc_fn in scope
let _ = method(self, assoc_fn());
ASSOC_CONST // in scope
}
}
// Same scoping rules for impls, including inherent impls:
struct Bar;
impl Bar {
fn foo(&self) { ... }
fn bar(&self) {
foo(self); // foo in scope UFCS-style
...
}
}
Items from super traits are not in scope, however. See the discussion on super traits below for more detail.
These scope rules provide good ergonomics for associated types in
particular, and a consistent scope model for language constructs that
can contain items (like traits, impls, and modules). In the long run,
we should also explore imports for trait items, i.e. use Trait::some_method
, but that is out of scope for this RFC.
Note that, according to this proposal, associated types/lifetimes are not in
scope for the optional where
clause on the trait header. For example:
trait Foo<Input>
// type parameters in scope, but associated types are not:
where Bar<Input, Self::Output>: Encodable {
type Output;
...
}
This setup seems more intuitive than allowing the trait header to refer directly to items defined within the trait body.
It’s also worth noting that trait-level where
clauses are never needed for
constraining associated types anyway, because associated types also have where
clauses. Thus, the above example could (and should) instead be written as
follows:
trait Foo<Input> {
type Output where Bar<Input, Output>: Encodable;
...
}
Constraining associated types
Associated types are not treated as parameters to a trait, but in some cases a
function will want to constrain associated types in some way. For example, as
explained in the Motivation section, the Iterator
trait should treat the
element type as an output:
trait Iterator {
type A;
fn next(&mut self) -> Option<A>;
...
}
For code that works with iterators generically, there is no need to constrain this type:
fn collect_into_vec<I: Iterator>(iter: I) -> Vec<I::A> { ... }
But other code may have requirements for the element type:
- That it implements some traits (bounds).
- That it unifies with a particular type.
These requirements can be imposed via where
clauses:
fn print_iter<I>(iter: I) where I: Iterator, I::A: Show { ... }
fn sum_uints<I>(iter: I) where I: Iterator, I::A = uint { ... }
In addition, there is a shorthand for equality constraints:
fn sum_uints<I: Iterator<A = uint>>(iter: I) { ... }
In general, a trait like:
trait Foo<Input1, Input2> {
type Output1;
type Output2;
lifetime 'a;
const C: bool;
...
}
can be written in a bound like:
T: Foo<I1, I2>
T: Foo<I1, I2, Output1 = O1>
T: Foo<I1, I2, Output2 = O2>
T: Foo<I1, I2, Output1 = O1, Output2 = O2>
T: Foo<I1, I2, Output1 = O1, 'a = 'b, Output2 = O2>
T: Foo<I1, I2, Output1 = O1, 'a = 'b, C = true, Output2 = O2>
The output constraints must come after all input arguments, but can appear in any order.
Note that output constraints are allowed when referencing a trait in a type or
a bound, but not in an IMPL_SEGMENT
path:
- As a type:
fn foo(obj: Box<Iterator<A = uint>>
is allowed. - In a bound:
fn foo<I: Iterator<A = uint>>(iter: I)
is allowed. - In an
IMPL_SEGMENT
:<I as Iterator<A = uint>>::next
is not allowed.
The reason not to allow output constraints in IMPL_SEGMENT
is that such paths
are references to a trait implementation that has already been determined – it
does not make sense to apply additional constraints to the implementation when
referencing it.
Output constraints are a handy shorthand when using trait bounds, but they are a necessity for trait objects, which we discuss next.
Trait objects
When using trait objects, the Self
type is “erased”, so different types
implementing the trait can be used under the same trait object type:
impl Show for Foo { ... }
impl Show for Bar { ... }
fn make_vec() -> Vec<Box<Show>> {
let f = Foo { ... };
let b = Bar { ... };
let mut v = Vec::new();
v.push(box f as Box<Show>);
v.push(box b as Box<Show>);
v
}
One consequence of erasing Self
is that methods using the Self
type as
arguments or return values cannot be used on trait objects, since their types
would differ for different choices of Self
.
In the model presented in this RFC, traits have additional input parameters
beyond Self
, as well as associated types that may vary depending on all of the
input parameters. This raises the question: which of these types, if any, are
erased in trait objects?
The approach we take here is the simplest and most conservative: when using a
trait as a type (i.e., as a trait object), all input and output types must
be provided as part of the type. In other words, only the Self
type is
erased, and all other types are specified statically in the trait object type.
Consider again the following example:
trait Foo<Input1, Input2> {
type Output1;
type Output2;
lifetime 'a;
const C: bool;
...
}
Unlike the case for static trait bounds, which do not have to specify any of the associated types, lifetimes, or consts, (but do have to specify the input types), trait object types must specify all of the types:
fn consume_foo<T: Foo<I1, I2>>(t: T) // this is valid
fn consume_obj(t: Box<Foo<I1, I2>>) // this is NOT valid
// but this IS valid:
fn consume_obj(t: Box<Foo<I1, I2, Output1 = O2, Output2 = O2, 'a = 'static, C = true>>)
With this design, it is clear that none of the non-Self
types are erased as
part of trait objects. But it leaves wiggle room to relax this restriction
later on: trait object types that are not allowed under this design can be given
meaning in some later design.
Inherent associated items
All associated items are also allowed in inherent impl
s, so a definition like
the following is allowed:
struct MyGraph { ... }
struct MyNode { ... }
struct MyEdge { ... }
impl MyGraph {
type N = MyNode;
type E = MyEdge;
// Note: associated types in scope, just as with trait bodies
fn has_edge(&self, &N, &N) -> bool {
...
}
...
}
Inherent associated items are referenced similarly to trait associated items:
fn distance(g: &MyGraph, from: &MyGraph::N, to: &MyGraph::N) -> uint { ... }
Note, however, that output constraints do not make sense for inherent outputs:
// This is *not* a legal type:
MyGraph<N = SomeNodeType>
The input/output type distinction
When designing a trait that references some unknown type, you now have the option of taking that type as an input parameter, or specifying it as an output associated type. What are the ramifications of this decision?
Coherence implications
Input types are used when determining which impl
matches, even for the same
Self
type:
trait Iterable1<A> {
type I: Iterator<A>;
fn iter(self) -> I;
}
// These impls have distinct input types, so are allowed
impl Iterable1<u8> for Foo { ... }
impl Iterable1<char> for Foo { ... }
trait Iterable2 {
type A;
type I: Iterator<A>;
fn iter(self) -> I;
}
// These impls apply to a common input (Foo), so are NOT allowed
impl Iterable2 for Foo { ... }
impl Iterable2 for Foo { ... }
More formally, the coherence property is revised as follows:
- Given a trait and values for all its type parameters (inputs, including
Self
), there is at most one applicableimpl
.
In the trait reform RFC, coherence is guaranteed by maintaining two other key properties, which are revised as follows:
Orphan check: Every implementation must meet one of the following conditions:
-
The trait being implemented (if any) must be defined in the current crate.
-
At least one of the input type parameters (including but not necessarily
Self
) must meet the following grammar, whereC
is a struct or enum defined within the current crate:T = C | [T] | [T, ..n] | &T | &mut T | ~T | (..., T, ...) | X<..., T, ...> where X is not bivariant with respect to T
Overlapping instances: No two implementations can be instantiable with the same set of types for the input type parameters.
See the trait reform RFC for more discussion of these properties.
Type inference implications
Finally, output type parameters can be inferred/resolved as soon as there is
a matching impl
based on the input type parameters. Because of the
coherence property above, there can be at most one.
On the other hand, even if there is only one applicable impl
, type inference
is not allowed to infer the input type parameters from it. This restriction
makes it possible to ensure crate concatenation: adding another crate may add
impl
s for a given trait, and if type inference depended on the absence of such
impl
s, importing a crate could break existing code.
In practice, these inference benefits can be quite valuable. For example, in the
Add
trait given at the beginning of this RFC, the Sum
output type is
immediately known once the input types are known, which can avoid the need for
type annotations.
Limitations
The main limitation of associated items as presented here is about associated types in particular. You might be tempted to write a trait like the following:
trait Iterable {
type A;
type I: Iterator<&'a A>; // what is the lifetime here?
fn iter<'a>(&'a self) -> I; // and how to connect it to self?
}
The problem is that, when implementing this trait, the return type I
of iter
must generally depend on the lifetime of self. For example, the corresponding
method in Vec
looks like the following:
impl<T> Vec<T> {
fn iter(&'a self) -> Items<'a, T> { ... }
}
This means that, given a Vec<T>
, there isn’t a single type Items<T>
for
iteration – rather, there is a family of types, one for each input lifetime.
In other words, the associated type I
in the Iterable
needs to be
“higher-kinded”: not just a single type, but rather a family:
trait Iterable {
type A;
type I<'a>: Iterator<&'a A>;
fn iter<'a>(&self) -> I<'a>;
}
In this case, I
is parameterized by a lifetime, but in other cases (like
map
) an associated type needs to be parameterized by a type.
In general, such higher-kinded types (HKTs) are a much-requested feature for Rust, and they would extend the reach of associated types. But the design and implementation of higher-kinded types is, by itself, a significant investment. The point of view of this RFC is that associated items bring the most important changes needed to stabilize our existing traits (and add a few key others), while HKTs will allow us to define important traits in the future but are not necessary for 1.0.
Encoding higher-kinded types
That said, it’s worth pointing out that variants of higher-kinded types can be encoded in the system being proposed here.
For example, the Iterable
example above can be written in the following
somewhat contorted style:
trait IterableOwned {
type A;
type I: Iterator<A>;
fn iter_owned(self) -> I;
}
trait Iterable {
fn iter<'a>(&'a self) -> <&'a Self>::I where &'a Self: IterableOwned {
IterableOwned::iter_owned(self)
}
}
The idea here is to define a trait that takes, as input type/lifetimes
parameters, the parameters to any HKTs. In this case, the trait is implemented
on the type &'a Self
, which includes the lifetime parameter.
We can in fact generalize this technique to encode arbitrary HKTs:
// The kind * -> *
trait TypeToType<Input> {
type Output;
}
type Apply<Name, Elt> where Name: TypeToType<Elt> = Name::Output;
struct Vec_;
struct DList_;
impl<T> TypeToType<T> for Vec_ {
type Output = Vec<T>;
}
impl<T> TypeToType<T> for DList_ {
type Output = DList<T>;
}
trait Mappable
{
type E;
type HKT where Apply<HKT, E> = Self;
fn map<F>(self, f: E -> F) -> Apply<HKT, F>;
}
While the above demonstrates the versatility of associated types and where
clauses, it is probably too much of a hack to be viable for use in libstd
.
Associated consts in generic code
If the value of an associated const depends on a type parameter (including
Self
), it cannot be used in a constant expression. This restriction will
almost certainly be lifted in the future, but this raises questions outside the
scope of this RFC.
Staging
Associated lifetimes are probably not necessary for the 1.0 timeframe. While we currently have a few traits that are parameterized by lifetimes, most of these can go away once DST lands.
On the other hand, associated lifetimes are probably trivial to implement once associated types have been implemented.
Other interactions
Interaction with implied bounds
As part of the implied bounds idea, it may be desirable for this:
fn pick_node<G>(g: &G) -> &<G as Graph>::N
to be sugar for this:
fn pick_node<G: Graph>(g: &G) -> &<G as Graph>::N
But this feature can easily be added later, as part of a general implied bounds RFC.
Future-proofing: specialization of impl
s
In the future, we may wish to relax the “overlapping instances” rule so that one can provide “blanket” trait implementations and then “specialize” them for particular types. For example:
trait Sliceable {
type Slice;
// note: not using &self here to avoid need for HKT
fn as_slice(self) -> Slice;
}
impl<'a, T> Sliceable for &'a T {
type Slice = &'a T;
fn as_slice(self) -> &'a T { self }
}
impl<'a, T> Sliceable for &'a Vec<T> {
type Slice = &'a [T];
fn as_slice(self) -> &'a [T] { self.as_slice() }
}
But then there’s a difficult question:
fn dice<A>(a: &A) -> &A::Slice where &A: Sliceable {
a // is this allowed?
}
Here, the blanket and specialized implementations provide incompatible associated types. When working with the trait generically, what can we assume about the associated type? If we assume it is the blanket one, the type may change during monomorphization (when specialization takes effect)!
The RFC does allow generic code to “see” associated types provided by blanket implementations, so this is a potential problem.
Our suggested strategy is the following. If at some later point we wish to add specialization, traits would have to opt in explicitly. For such traits, we would not allow generic code to “see” associated types for blanket implementations; instead, output types would only be visible when all input types were concretely known. This approach is backwards-compatible with the RFC, and is probably a good idea in any case.
Alternatives
Multidispatch through tuple types
This RFC clarifies trait matching by making trait type parameters inputs to matching, and associated types outputs.
A more radical alternative would be to remove type parameters from traits, and instead support multiple input types through a separate multidispatch mechanism.
In this design, the Add
trait would be written and implemented as follows:
// Lhs and Rhs are *inputs*
trait Add for (Lhs, Rhs) {
type Sum; // Sum is an *output*
fn add(&Lhs, &Rhs) -> Sum;
}
impl Add for (int, int) {
type Sum = int;
fn add(left: &int, right: &int) -> int { ... }
}
impl Add for (int, Complex) {
type Sum = Complex;
fn add(left: &int, right: &Complex) -> Complex { ... }
}
The for
syntax in the trait definition is used for multidispatch traits, here
saying that impl
s must be for pairs of types which are bound to Lhs
and
Rhs
respectively. The add
function can then be invoked in UFCS style by
writing
Add::add(some_int, some_complex)
Advantages of the tuple approach:
-
It does not force a distinction between
Self
and other input types, which in some cases (including binary operators likeAdd
) can be artificial. -
Makes it possible to specify input types without specifying the trait:
<(A, B)>::Sum
rather than<A as Add<B>>::Sum
.
Disadvantages of the tuple approach:
-
It’s more painful when you do want a method rather than a function.
-
Requires
where
clauses when used in bounds:where (A, B): Trait
rather thanA: Trait<B>
. -
It gives two ways to write single dispatch: either without
for
, or usingfor
with a single-element tuple. -
There’s a somewhat jarring distinction between single/multiple dispatch traits, making the latter feel “bolted on”.
-
The tuple syntax is unusual in acting as a binder of its types, as opposed to the
Trait<A, B>
syntax. -
Relatedly, the generics syntax for traits is immediately understandable (a family of traits) based on other uses of generics in the language, while the tuple notation stands alone.
-
Less clear story for trait objects (although the fact that
Self
is the only erased input type in this RFC may seem somewhat arbitrary).
On balance, the generics-based approach seems like a better fit for the language design, especially in its interaction with methods and the object system.
A backwards-compatible version
Yet another alternative would be to allow trait type parameters to be either
inputs or outputs, marking the inputs with a keyword in
:
trait Add<in Rhs, Sum> {
fn add(&Lhs, &Rhs) -> Sum;
}
This would provide a way of adding multidispatch now, and then adding associated
items later on without breakage. If, in addition, output types had to come after
all input types, it might even be possible to migrate output type parameters
like Sum
above into associated types later.
This is perhaps a reasonable fallback, but it seems better to introduce a clean design with both multidispatch and associated items together.
Unresolved questions
Super traits
This RFC largely ignores super traits.
Currently, the implementation of super traits treats them identically to a
where
clause that bounds Self
, and this RFC does not propose to change
that. However, a follow-up RFC should clarify that this is the intended
semantics for super traits.
Note that this treatment of super traits is, in particular, consistent with the
proposed scoping rules, which do not bring items from super traits into scope in
the body of a subtrait; they must be accessed via Self::item_name
.
Equality constraints in where
clauses
This RFC allows equality constraints on types for associated types, but does not
propose a similar feature for where
clauses. That will be the subject of a
follow-up RFC.
Multiple trait object bounds for the same trait
The design here makes it possible to write bounds or trait objects that mention the same trait, multiple times, with different inputs:
fn mulit_add<T: Add<int> + Add<Complex>>(t: T) -> T { ... }
fn mulit_add_obj(t: Box<Add<int> + Add<Complex>>) -> Box<Add<int> + Add<Complex>> { ... }
This seems like a potentially useful feature, and should be unproblematic for bounds, but may have implications for vtables that make it problematic for trait objects. Whether or not such trait combinations are allowed will likely depend on implementation concerns, which are not yet clear.
Generic associated consts in match patterns
It seems desirable to allow constants that depend on type parameters in match patterns, but it’s not clear how to do so while still checking exhaustiveness and reachability of the match arms. Most likely this requires new forms of where clause, to constrain associated constant values.
For now, we simply defer the question.
Generic associated consts in array sizes
It would be useful to be able to use trait-associated constants in generic code.
// Shouldn't this be OK?
const ALIAS_N: usize = <T>::N;
let x: [u8; <T>::N] = [0u8; ALIAS_N];
// Or...
let x: [u8; T::N + 1] = [0u8; T::N + 1];
However, this causes some problems. What should we do with the following case in
type checking, where we need to prove that a generic is valid for any T
?
let x: [u8; T::N + T::N] = [0u8; 2 * T::N];
We would like to handle at least some obvious cases (e.g. proving that
T::N == T::N
), but without trying to prove arbitrary statements about
arithmetic. The question of how to do this is deferred.
- Start Date: 2014-09-11
- RFC PR #: rust-lang/rfcs#198
- Rust Issue #: rust-lang/rust#17177
Summary
This RFC adds overloaded slice notation:
foo[]
forfoo.as_slice()
foo[n..m]
forfoo.slice(n, m)
foo[n..]
forfoo.slice_from(n)
foo[..m]
forfoo.slice_to(m)
mut
variants of all the above
via two new traits, Slice
and SliceMut
.
It also changes the notation for range match
patterns to ...
, to
signify that they are inclusive whereas ..
in slices are exclusive.
Motivation
There are two primary motivations for introducing this feature.
Ergonomics
Slicing operations, especially as_slice
, are a very common and basic thing to
do with vectors, and potentially many other kinds of containers. We already
have notation for indexing via the Index
trait, and this RFC is essentially a
continuation of that effort.
The as_slice
operator is particularly important. Since we’ve moved away from
auto-slicing in coercions, explicit as_slice
calls have become extremely
common, and are one of the
leading ergonomic/first impression
problems with the language. There are a few other approaches to address this
particular problem, but these alternatives have downsides that are discussed
below (see “Alternatives”).
Error handling conventions
We are gradually moving toward a Python-like world where notation like foo[n]
calls fail!
when n
is out of bounds, while corresponding methods like get
return Option
values rather than failing. By providing similar notation for
slicing, we open the door to following the same convention throughout
vector-like APIs.
Detailed design
The design is a straightforward continuation of the Index
trait design. We
introduce two new traits, for immutable and mutable slicing:
trait Slice<Idx, S> {
fn as_slice<'a>(&'a self) -> &'a S;
fn slice_from(&'a self, from: Idx) -> &'a S;
fn slice_to(&'a self, to: Idx) -> &'a S;
fn slice(&'a self, from: Idx, to: Idx) -> &'a S;
}
trait SliceMut<Idx, S> {
fn as_mut_slice<'a>(&'a mut self) -> &'a mut S;
fn slice_from_mut(&'a mut self, from: Idx) -> &'a mut S;
fn slice_to_mut(&'a mut self, to: Idx) -> &'a mut S;
fn slice_mut(&'a mut self, from: Idx, to: Idx) -> &'a mut S;
}
(Note, the mutable names here are part of likely changes to naming conventions that will be described in a separate RFC).
These traits will be used when interpreting the following notation:
Immutable slicing
foo[]
forfoo.as_slice()
foo[n..m]
forfoo.slice(n, m)
foo[n..]
forfoo.slice_from(n)
foo[..m]
forfoo.slice_to(m)
Mutable slicing
foo[mut]
forfoo.as_mut_slice()
foo[mut n..m]
forfoo.slice_mut(n, m)
foo[mut n..]
forfoo.slice_from_mut(n)
foo[mut ..m]
forfoo.slice_to_mut(m)
Like Index
, uses of this notation will auto-deref just as if they were method
invocations. So if T
implements Slice<uint, [U]>
, and s: Smaht<T>
, then
s[]
compiles and has type &[U]
.
Note that slicing is “exclusive” (so [n..m]
is the interval n <= x < m
), while ..
in match
patterns is “inclusive”. To avoid
confusion, we propose to change the match
notation to ...
to
reflect the distinction. The reason to change the notation, rather
than the interpretation, is that the exclusive (respectively
inclusive) interpretation is the right default for slicing
(respectively matching).
Rationale for the notation
The choice of square brackets for slicing is straightforward: it matches our indexing notation, and slicing and indexing are closely related.
Some other languages (like Python and Go – and Fortran) use :
rather than
..
in slice notation. The choice of ..
here is influenced by its use
elsewhere in Rust, for example for fixed-length array types [T, ..n]
. The ..
for slicing has precedent in Perl and D.
See Wikipedia for more on the history of slice notation in programming languages.
The mut
qualifier
It may be surprising that mut
is used as a qualifier in the proposed
slice notation, but not for the indexing notation. The reason is that
indexing includes an implicit dereference. If v: Vec<Foo>
then
v[n]
has type Foo
, not &Foo
or &mut Foo
. So if you want to get
a mutable reference via indexing, you write &mut v[n]
. More
generally, this allows us to do resolution/typechecking prior to
resolving the mutability.
This treatment of Index
matches the C tradition, and allows us to
write things like v[0] = foo
instead of *v[0] = foo
.
On the other hand, this approach is problematic for slicing, since in general it would yield an unsized type (under DST) – and of course, slicing is meant to give you a fat pointer indicating the size of the slice, which we don’t want to immediately deref. But the consequence is that we need to know the mutability of the slice up front, when we take it, since it determines the type of the expression.
Drawbacks
The main drawback is the increase in complexity of the language syntax. This
seems minor, especially since the notation here is essentially “finishing” what
was started with the Index
trait.
Limitations in the design
Like the Index
trait, this forces the result to be a reference via
&
, which may rule out some generalizations of slicing.
One way of solving this problem is for the slice methods to take
self
(by value) rather than &self
, and in turn to implement the
trait on &T
rather than T
. Whether this approach is viable in the
long run will depend on the final rules for method resolution and
auto-ref.
In general, the trait system works best when traits can be applied to
types T
rather than borrowed types &T
. Ultimately, if Rust gains
higher-kinded types (HKT), we could change the slice type S
in the
trait to be higher-kinded, so that it is a family of types indexed
by lifetime. Then we could replace the &'a S
in the return value
with S<'a>
. It should be possible to transition from the current
Index
and Slice
trait designs to an HKT version in the future
without breaking backwards compatibility by using blanket
implementations of the new traits (say, IndexHKT
) for types that
implement the old ones.
Alternatives
For improving the ergonomics of as_slice
, there are two main alternatives.
Coercions: auto-slicing
One possibility would be re-introducing some kind of coercion that automatically
slices.
We used to have a coercion from (in today’s terms) Vec<T>
to
&[T]
. Since we no longer coerce owned to borrowed values, we’d probably want a
coercion &Vec<T>
to &[T]
now:
fn use_slice(t: &[u8]) { ... }
let v = vec!(0u8, 1, 2);
use_slice(&v) // automatically coerce here
use_slice(v.as_slice()) // equivalent
Unfortunately, adding such a coercion requires choosing between the following:
-
Tie the coercion to
Vec
andString
. This would reintroduce special treatment of these otherwise purely library types, and would mean that other library types that support slicing would not benefit (defeating some of the purpose of DST). -
Make the coercion extensible, via a trait. This is opening pandora’s box, however: the mechanism could likely be (ab)used to run arbitrary code during coercion, so that any invocation
foo(a, b, c)
might involve running code to pre-process each of the arguments. While we may eventually want such user-extensible coercions, it is a big step to take with a lot of potential downside when reasoning about code, so we should pursue more conservative solutions first.
Deref
Another possibility would be to make String
implement Deref<str>
and
Vec<T>
implement Deref<[T]>
, once DST lands. Doing so would allow explicit
coercions like:
fn use_slice(t: &[u8]) { ... }
let v = vec!(0u8, 1, 2);
use_slice(&*v) // take advantage of deref
use_slice(v.as_slice()) // equivalent
There are at least two downsides to doing so, however:
-
It is not clear how the method resolution rules will ultimately interact with
Deref
. In particular, a leading proposal is that for a smart pointers: Smaht<T>
when you invokes.m(...)
only inherent methodsm
are considered forSmaht<T>
; trait methods are only considered for the maximally-derefed value*s
.With such a resolution strategy, implementing
Deref
forVec
would make it impossible to use trait methods on theVec
type except through UFCS, severely limiting the ability of programmers to usefully implement new traits forVec
. -
The idea of
Vec
as a smart pointer around a slice, and the use of&*v
as above, is somewhat counterintuitive, especially for such a basic type.
Ultimately, notation for slicing seems desirable on its own merits anyway, and
if it can eliminate the need to implement Deref
for Vec
and String
, all
the better.
- Start Date: 2014-08-28
- RFC PR #: rust-lang/rfcs#199
- Rust Issue #: rust-lang/rust#16810
Summary
This is a conventions RFC for settling naming conventions when there are by value, by reference, and by mutable reference variants of an operation.
Motivation
Currently the libraries are not terribly consistent about how to
signal mut variants of functions; sometimes it is by a mut_
prefix,
sometimes a _mut
suffix, and occasionally with _mut_
appearing in
the middle. These inconsistencies make APIs difficult to remember.
While there are arguments in favor of each of the positions, we stand to gain a lot by standardizing, and to some degree we just need to make a choice.
Detailed design
Functions often come in multiple variants: immutably borrowed, mutably borrowed, and owned.
The canonical example is iterator methods:
iter
works with immutably borrowed datamut_iter
works with mutably borrowed datamove_iter
works with owned data
For iterators, the “default” (unmarked) variant is immutably borrowed. In other cases, the default is owned.
The proposed rules depend on which variant is the default, but use suffixes to mark variants in all cases.
The rules
Immutably borrowed by default
If foo
uses/produces an immutable borrow by default, use:
- The
_mut
suffix (e.g.foo_mut
) for the mutably borrowed variant. - The
_move
suffix (e.g.foo_move
) for the owned variant.
However, in the case of iterators, the moving variant can also be
understood as an into
conversion, into_iter
, and for x in v.into_iter()
reads arguably better than for x in v.iter_move()
, so the convention is
into_iter
.
NOTE: This convention covers only the method names for iterators, not the names of the iterator types. That will be the subject of a follow up RFC.
Owned by default
If foo
uses/produces owned data by default, use:
- The
_ref
suffix (e.g.foo_ref
) for the immutably borrowed variant. - The
_mut
suffix (e.g.foo_mut
) for the mutably borrowed variant.
Exceptions
For mutably borrowed variants, if the mut
qualifier is part of a
type name (e.g. as_mut_slice
), it should appear as it would appear
in the type.
References to type names
Some places in the current libraries, we say things like as_ref
and
as_mut
, and others we say get_ref
and get_mut_ref
.
Proposal: generally standardize on mut
as a shortening of mut_ref
.
The rationale
Why suffixes?
Using a suffix makes it easier to visually group variants together, especially when sorted alphabetically. It puts the emphasis on the functionality, rather than the qualifier.
Why move
?
Historically, Rust has used move
as a way to signal ownership
transfer and to connect to C++ terminology. The main disadvantage is
that it does not emphasize ownership, which is our current narrative.
On the other hand, in Rust all data is owned, so using _owned
as a
qualifier is a bit strange.
The Copy
trait poses a problem for any terminology about ownership
transfer. The proposed mental model is that with Copy
data you are
“moving a copy”.
See Alternatives for more discussion.
Why mut
rather then mut_ref
?
It’s shorter, and pairs like as_ref
and as_mut
have a pleasant harmony
that doesn’t place emphasis on one kind of reference over the other.
Alternatives
Prefix or mixed qualifiers
Using prefixes for variants is another possibility, but there seems to be little upside.
It’s possible to rationalize our current mix of prefixes and suffixes via grammatical distinctions, but this seems overly subtle and complex, and requires a strong command of English grammar to work well.
No suffix exception
The rules here make an exception when mut
is part of a type name, as
in as_mut_slice
, but we could instead always place the qualifier
as a suffix: as_slice_mut
. This would make APIs more consistent in
some ways, less in others: conversion functions would no longer
consistently use a transcription of their type name.
This is perhaps not so bad, though, because as it is we often abbreviate type names. In any case, we need a convention (separate RFC) for how to refer to type names in methods.
owned
instead of move
The overall narrative about Rust has been evolving to focus on
ownership as the essential concept, with borrowing giving various
lesser forms of ownership, so _owned
would be a reasonable
alternative to _move
.
On the other hand, the ref
variants do not say “borrowed”, so in
some sense this choice is inconsistent. In addition, the terminology
is less familiar to those coming from C++.
val
instead of owned
Another option would be val
or value
instead of owned
. This
suggestion plays into the “by reference” and “by value” distinction,
and so is even more congruent with ref
than move
is. On the other
hand, it’s less clear/evocative than either move
or owned
.
- Start Date: 2014-07-17
- RFC PR #: rust-lang/rfcs#201
- Rust Issue #: rust-lang/rust#17747
Summary
This RFC improves interoperation between APIs with different error types. It proposes to:
-
Increase the flexibility of the
try!
macro for clients of multiple libraries with disparate error types. -
Standardize on basic functionality that any error type should have by introducing an
Error
trait. -
Support easy error chaining when crossing abstraction boundaries.
The proposed changes are all library changes; no language changes are needed – except that this proposal depends on multidispatch happening.
Motivation
Typically, a module (or crate) will define a custom error type encompassing the
possible error outcomes for the operations it provides, along with a custom
Result
instance baking in this type. For example, we have io::IoError
and
io::IoResult<T> = Result<T, io::IoError>
, and similarly for other libraries.
Together with the try!
macro, the story for interacting with errors for a
single library is reasonably good.
However, we lack infrastructure when consuming or building on errors from multiple APIs, or abstracting over errors.
Consuming multiple error types
Our current infrastructure for error handling does not cope well with mixed notions of error.
Abstractly, as described by this issue, we cannot do the following:
fn func() -> Result<T, Error> {
try!(may_return_error_type_A());
try!(may_return_error_type_B());
}
Concretely, imagine a CLI application that interacts both with files
and HTTP servers, using std::io
and an imaginary http
crate:
fn download() -> Result<(), CLIError> {
let contents = try!(http::get(some_url));
let file = try!(File::create(some_path));
try!(file.write_str(contents));
Ok(())
}
The download
function can encounter both io
and http
errors, and
wants to report them both under the common notion of CLIError
. But
the try!
macro only works for a single error type at a time.
There are roughly two scenarios where multiple library error types need to be coalesced into a common type, each with different needs: application error reporting, and library error reporting
Application error reporting: presenting errors to a user
An application is generally the “last stop” for error handling: it’s the point at which remaining errors are presented to the user in some form, when they cannot be handled programmatically.
As such, the data needed for application-level errors is usually
related to human interaction. For a CLI application, a short text
description and longer verbose description are usually all that’s
needed. For GUI applications, richer data is sometimes required, but
usually not a full enum
describing the full range of errors.
Concretely, then, for something like the download
function above,
for a CLI application, one might want CLIError
to roughly be:
struct CLIError<'a> {
description: &'a str,
detail: Option<String>,
... // possibly more fields here; see detailed design
}
Ideally, one could use the try!
macro as in the download
example
to coalesce a variety of error types into this single, simple
struct
.
Library error reporting: abstraction boundaries
When one library builds on others, it needs to translate from their error types to its own. For example, a web server framework may build on a library for accessing a SQL database, and needs some way to “lift” SQL errors to its own notion of error.
In general, a library may not want to reveal the upstream libraries it relies on – these are implementation details which may change over time. Thus, it is critical that the error type of upstream libraries not leak, and “lifting” an error from one library to another is a way of imposing an abstraction boundaries.
In some cases, the right way to lift a given error will depend on the operation and context. In other cases, though, there will be a general way to embed one kind of error in another (usually via a “cause chain”). Both scenarios should be supported by Rust’s error handling infrastructure.
Abstracting over errors
Finally, libraries sometimes need to work with errors in a generic
way. For example, the serialize::Encoder
type takes is generic over
an arbitrary error type E
. At the moment, such types are completely
arbitrary: there is no Error
trait giving common functionality
expected of all errors. Consequently, error-generic code cannot
meaningfully interact with errors.
(See this issue for a concrete case where a bound would be useful; note, however, that the design below does not cover this use-case, as explained in Alternatives.)
Languages that provide exceptions often have standard exception
classes or interfaces that guarantee some basic functionality,
including short and detailed descriptions and “causes”. We should
begin developing similar functionality in libstd
to ensure that we
have an agreed-upon baseline error API.
Detailed design
We can address all of the problems laid out in the Motivation section
by adding some simple library code to libstd
, so this RFC will
actually give a full implementation.
Note, however, that this implementation relies on the multidispatch proposal currently under consideration.
The proposal consists of two pieces: a standardized Error
trait and
extensions to the try!
macro.
The Error
trait
The standard Error
trait follows very the widespread pattern found
in Exception
base classes in many languages:
pub trait Error: Send + Any {
fn description(&self) -> &str;
fn detail(&self) -> Option<&str> { None }
fn cause(&self) -> Option<&Error> { None }
}
Every concrete error type should provide at least a description. By
making this a slice-returning method, it is possible to define
lightweight enum
error types and then implement this method as
returning static string slices depending on the variant.
The cause
method allows for cause-chaining when an error crosses
abstraction boundaries. The cause is recorded as a trait object
implementing Error
, which makes it possible to read off a kind of
abstract backtrace (often more immediately helpful than a full
backtrace).
The Any
bound is needed to allow downcasting of errors. This RFC
stipulates that it must be possible to downcast errors in the style of
the Any
trait, but leaves unspecified the exact implementation
strategy. (If trait object upcasting was available, one could simply
upcast to Any
; otherwise, we will likely need to duplicate the
downcast
APIs as blanket impl
s on Error
objects.)
It’s worth comparing the Error
trait to the most widespread error
type in libstd
, IoError
:
pub struct IoError {
pub kind: IoErrorKind,
pub desc: &'static str,
pub detail: Option<String>,
}
Code that returns or asks for an IoError
explicitly will be able to
access the kind
field and thus react differently to different kinds
of errors. But code that works with a generic Error
(e.g.,
application code) sees only the human-consumable parts of the error.
In particular, application code will often employ Box<Error>
as the
error type when reporting errors to the user. The try!
macro
support, explained below, makes doing so ergonomic.
An extended try!
macro
The other piece to the proposal is a way for try!
to automatically
convert between different types of errors.
The idea is to introduce a trait FromError<E>
that says how to
convert from some lower-level error type E
to Self
. The try!
macro then passes the error it is given through this conversion before
returning:
// E here is an "input" for dispatch, so conversions from multiple error
// types can be provided
pub trait FromError<E> {
fn from_err(err: E) -> Self;
}
impl<E> FromError<E> for E {
fn from_err(err: E) -> E {
err
}
}
impl<E: Error> FromError<E> for Box<Error> {
fn from_err(err: E) -> Box<Error> {
box err as Box<Error>
}
}
macro_rules! try (
($expr:expr) => ({
use error;
match $expr {
Ok(val) => val,
Err(err) => return Err(error::FromError::from_err(err))
}
})
)
This code depends on
multidispatch, because
the conversion depends on both the source and target error types. (In
today’s Rust, the two implementations of FromError
given above would
be considered overlapping.)
Given the blanket impl
of FromError<E>
for E
, all existing uses
of try!
would continue to work as-is.
With this infrastructure in place, application code can generally use
Box<Error>
as its error type, and try!
will take care of the rest:
fn download() -> Result<(), Box<Error>> {
let contents = try!(http::get(some_url));
let file = try!(File::create(some_path));
try!(file.write_str(contents));
Ok(())
}
Library code that defines its own error type can define custom
FromError
implementations for lifting lower-level errors (where the
lifting should also perform cause chaining) – at least when the
lifting is uniform across the library. The effect is that the mapping
from one error type into another only has to be written one, rather
than at every use of try!
:
impl FromError<ErrorA> MyError { ... }
impl FromError<ErrorB> MyError { ... }
fn my_lib_func() -> Result<T, MyError> {
try!(may_return_error_type_A());
try!(may_return_error_type_B());
}
Drawbacks
The main drawback is that the try!
macro is a bit more complicated.
Unresolved questions
Conventions
This RFC does not define any particular conventions around cause chaining or concrete error types. It will likely take some time and experience using the proposed infrastructure before we can settle these conventions.
Extensions
The functionality in the Error
trait is quite minimal, and should
probably grow over time. Some additional functionality might include:
Features on the Error
trait
-
Generic creation of
Error
s. It might be useful for theError
trait to expose an associated constructor. See this issue for an example where this functionality would be useful. -
Mutation of
Error
s. TheError
trait could be expanded to provide setters as well as getters.
The main reason not to include the above two features is so that
Error
can be used with extremely minimal data structures,
e.g. simple enum
s. For such data structures, it’s possible to
produce fixed descriptions, but not mutate descriptions or other error
properties. Allowing generic creation of any Error
-bounded type
would also require these enum
s to include something like a
GenericError
variant, which is unfortunate. So for now, the design
sticks to the least common denominator.
Concrete error types
On the other hand, for code that doesn’t care about the footprint of its error types, it may be useful to provide something like the following generic error type:
pub struct WrappedError<E> {
pub kind: E,
pub description: String,
pub detail: Option<String>,
pub cause: Option<Box<Error>>
}
impl<E: Show> WrappedError<E> {
pub fn new(err: E) {
WrappedErr {
kind: err,
description: err.to_string(),
detail: None,
cause: None
}
}
}
impl<E> Error for WrappedError<E> {
fn description(&self) -> &str {
self.description.as_slice()
}
fn detail(&self) -> Option<&str> {
self.detail.as_ref().map(|s| s.as_slice())
}
fn cause(&self) -> Option<&Error> {
self.cause.as_ref().map(|c| &**c)
}
}
This type can easily be added later, so again this RFC sticks to the minimal functionality for now.
- Start Date: 2014-08-15
- RFC PR: rust-lang/rfcs#202
- Rust Issue: rust-lang/rust#16967
Summary
Change syntax of subslices matching from ..xs
to xs..
to be more consistent with the rest of the language
and allow future backwards compatible improvements.
Small example:
match slice {
[xs.., _] => xs,
[] => fail!()
}
This is basically heavily stripped version of RFC 101.
Motivation
In Rust, symbol after ..
token usually describes number of things,
as in [T, ..N]
type or in [e, ..N]
expression.
But in following pattern: [_, ..xs]
, xs
doesn’t describe any number,
but the whole subslice.
I propose to move dots to the right for several reasons (including one mentioned above):
- Looks more natural (but that might be subjective).
- Consistent with the rest of the language.
- C++ uses
args...
in variadic templates. - It allows extending slice pattern matching as described in RFC 101.
Detailed design
Slice matching grammar would change to (assuming trailing commas; grammar syntax as in Rust manual):
slice_pattern : "[" [[pattern | subslice_pattern] ","]* "]" ;
subslice_pattern : ["mut"? ident]? ".." ["@" slice_pattern]? ;
To compare, currently it looks like:
slice_pattern : "[" [[pattern | subslice_pattern] ","]* "]" ;
subslice_pattern : ".." ["mut"? ident ["@" slice_pattern]?]? ;
Drawbacks
Backward incompatible.
Alternatives
Don’t do it at all.
Unresolved questions
Whether subslice matching combined with @
should be written as xs.. @[1, 2]
or maybe in another way: xs @[1, 2]..
.
- Start Date: 2014-09-03
- RFC PR: rust-lang/rfcs#212
- Rust Issue: rust-lang/rust#16968
Summary
Restore the integer inference fallback that was removed. Integer
literals whose type is unconstrained will default to i32
, unlike the
previous fallback to int
.
Floating point literals will default to f64
.
Motivation
History lesson
Rust has had a long history with integer and floating-point
literals. Initial versions of Rust required all literals to be
explicitly annotated with a suffix (if no suffix is provided, then
int
or float
was used; note that the float
type has since been
removed). This meant that, for example, if one wanted to count up all
the numbers in a list, one would write 0u
and 1u
so as to employ
unsigned integers:
let mut count = 0u; // let `count` be an unsigned integer
while cond() {
...
count += 1u; // `1u` must be used as well
}
This was particularly troublesome with arrays of integer literals, which could be quite hard to read:
let byte_array = [0u8, 33u8, 50u8, ...];
It also meant that code which was very consciously using 32-bit or 64-bit numbers was hard to read.
Therefore, we introduced integer inference: unlabeled integer literals
are not given any particular integral type rather a fresh “integral
type variable” (floating point literals work in an analogous way). The
idea is that the vast majority of literals will eventually interact
with an actual typed variable at some point, and hence we can infer
what type they ought to have. For those cases where the type cannot be
automatically selected, we decided to fallback to our older behavior,
and have integer/float literals be typed as int
/float
(this is also what Haskell
does). Some time later, we did various measurements and found
that in real world code this fallback was rarely used. Therefore, we
decided that to remove the fallback.
Experience with lack of fallback
Unfortunately, when doing the measurements that led us to decide to
remove the int
fallback, we neglected to consider coding “in the
small” (specifically, we did not include tests in the
measurements). It turns out that when writing small programs, which
includes not only “hello world” sort of things but also tests, the
lack of integer inference fallback is quite annoying. This is
particularly troublesome since small program are often people’s first
exposure to Rust. The problems most commonly occur when integers are
“consumed” by printing them out to the screen or by asserting
equality, both of which are very common in small programs and testing.
There are at least three common scenarios where fallback would be beneficial:
Accumulator loops. Here a counter is initialized to 0
and then
incremented by 1
. Eventually it is printed or compared against
a known value.
let mut c = 0;
loop {
...;
c += 1;
}
println!("{}", c); // Does not constrain type of `c`
assert_eq(c, 22);
Calls to range with constant arguments. Here a call to range like
range(0, 10)
is used to execute something 10 times. It is important
that the actual counter is either unused or only used in a print out
or comparison against another literal:
for _ in range(0, 10) {
}
Large constants. In small tests it is convenient to make dummy test data. This frequently takes the form of a vector or map of ints.
let mut m = HashMap::new();
m.insert(1, 2);
m.insert(3, 4);
assert_eq(m.find(&3).map(|&i| i).unwrap(), 4);
Lack of bugs
To our knowledge, there has not been a single bug exposed by removing
the fallback to the int
type. Moreover, such bugs seem to be
extremely unlikely.
The primary reason for this is that, in production code, the i32
fallback is very rarely used. In a sense, the same measurements
that were used to justify removing the int
fallback also justify
keeping it. As the measurements showed, the vast, vast majority of
integer literals wind up with a constrained type, unless they are only
used to print out and do assertions with. Specifically, any integer
that is passed as a parameter, returned from a function, or stored in
a struct or array, must wind up with a specific type.
Rationale for the choice of defaulting to i32
In contrast to the first revision of the RFC, the fallback type
suggested is i32
. This is justified by a case analysis which showed
that there does not exist a compelling reason for having a signed
pointer-sized integer type as the default.
There are reasons for using i32
instead: It’s familiar to programmers
from the C programming language (where the default int type is 32-bit in
the major calling conventions), it’s faster than 64-bit integers in
arithmetic today, and is superior in memory usage while still providing
a reasonable range of possible values.
To expand on the performance argument: i32
obviously uses half of the
memory of i64
meaning half the memory bandwidth used, half as much
cache consumption and twice as much vectorization – additionally
arithmetic (like multiplication and division) is faster on some of the
modern CPUs.
Case analysis
This is an analysis of cases where int
inference might be thought of
as useful:
Indexing into an array with unconstrained integer literal:
let array = [0u8, 1, 2, 3];
let index = 3;
array[index]
In this case, index
is already automatically inferred to be a uint
.
Using a default integer for tests, tutorials, etc.: Examples of this include “The Guide”, the Rust API docs and the Rust standard library unit tests. This is better served by a smaller, faster and platform independent type as default.
Using an integer for an upper bound or for simply printing it: This
is also served very well by i32
.
Counting of loop iterations: This is a part where int
is as badly
suited as i32
, so at least the move to i32
doesn’t create new
hazards (note that the number of elements of a vector might not
necessarily fit into an int
).
In addition to all the points above, having a platform-independent type obviously results in less differences between the platforms in which the programmer “doesn’t care” about the integer type they are using.
Future-proofing for overloaded literals
It is possible that, in the future, we will wish to allow vector and strings literals to be overloaded so that they can be resolved to user-defined types. In that case, for backwards compatibility, it will be necessary for those literals to have some sort of fallback type. (This is a relatively weak consideration.)
Detailed design
Integral literals are currently type-checked by creating a special
class of type variable. These variables are subject to unification as
normal, but can only unify with integral types. This RFC proposes
that, at the end of type inference, when all constraints are known, we
will identify all integral type variables that have not yet been bound
to anything and bind them to i32
. Similarly, floating point literals
will fallback to f64
.
For those who wish to be very careful about which integral types they
employ, a new lint (unconstrained_literal
) will be added which
defaults to allow
. This lint is triggered whenever the type of an
integer or floating point literal is unconstrained.
Downsides
Although there seems to be little motivation for int
to be the
default, there might be use cases where int
is a more correct fallback
than i32
.
Additionally, it might seem weird to some that i32
is a default, when
int
looks like the default from other languages. The name of int
however is not in the scope of this RFC.
Alternatives
-
No fallback. Status quo.
-
Fallback to something else. We could potentially fallback to
int
like the original RFC suggested or some other integral type rather thani32
. -
Fallback in a more narrow range of cases. We could attempt to identify integers that are “only printed” or “only compared”. There is no concrete proposal in this direction and it seems to lead to an overly complicated design.
-
Default type parameters influencing inference. There is a separate, follow-up proposal being prepared that uses default type parameters to influence inference. This would allow some examples, like
range(0, 10)
to work even without integral fallback, because therange
function itself could specify a fallback type. However, this does not help with many other examples.
History
2014-11-07: Changed the suggested fallback from int
to i32
, add
rationale.
- Start Date: 2015-02-04
- RFC PR: rust-lang/rfcs#213
- Rust Issue: rust-lang/rust#27336
Summary
Rust currently includes feature-gated support for type parameters that specify a default value. This feature is not well-specified. The aim of this RFC is to fully specify the behavior of defaulted type parameters:
- Type parameters in any position can specify a default.
- Within fn bodies, defaulted type parameters are used to drive inference.
- Outside of fn bodies, defaulted type parameters supply fixed defaults.
_
can be used to omit the values of type parameters and apply a suitable default:- In a fn body, any type parameter can be omitted in this way, and a suitable type variable will be used.
- Outside of a fn body, only defaulted type parameters can be omitted, and the specified default is then used.
Points 2 and 4 extend the current behavior of type parameter defaults, aiming to address some shortcomings of the current implementation.
This RFC would remove the feature gate on defaulted type parameters.
Motivation
Why defaulted type parameters
Defaulted type parameters are very useful in two main scenarios:
- Extended a type without breaking existing clients.
- Allowing customization in ways that many or most users do not care about.
Often, these two scenarios occur at the same time. A classic
historical example is the HashMap
type from Rust’s standard
library. This type now supports the ability to specify custom
hashers. For most clients, this is not particularly important and this
initial versions of the HashMap
type were not customizable in this
regard. But there are some cases where having the ability to use a
custom hasher can make a huge difference. Having the ability to
specify defaults for type parameters allowed the HashMap
type to add
a new type parameter H
representing the hasher type without breaking
any existing clients and also without forcing all clients to specify
what hasher to use.
However, customization occurs in places other than types. Consider the
function range()
. In early versions of Rust, there was a distinct
range function for each integral type (e.g. uint::range
,
int::range
, etc). These functions were eventually consolidated into
a single range()
function that is defined generically over all
“enumerable” types:
trait Enumerable : Add<Self,Self> + PartialOrd + Clone + One;
pub fn range<A:Enumerable>(start: A, stop: A) -> Range<A> {
Range{state: start, stop: stop, one: One::one()}
}
This version is often more convenient to use, particularly in a generic context.
However, the generic version does have the downside that when the bounds of the range are integral, inference sometimes lacks enough information to select a proper type:
// ERROR -- Type argument unconstrained, what integral type did you want?
for x in range(0, 10) { ... }
Thus users are forced to write:
for x in range(0u, 10u) { ... }
This RFC describes how to integrate default type parameters with
inference such that the type parameter on range
can specify a
default (uint
, for example):
pub fn range<A:Enumerable=uint>(start: A, stop: A) -> Range<A> {
Range{state: start, stop: stop, one: One::one()}
}
Using this definition, a call like range(0, 10)
is perfectly legal.
If it turns out that the type argument is not other constraint, uint
will be used instead.
Extending types without breaking clients.
Without defaults, once a library is released to “the wild”, it is not possible to add type parameters to a type without breaking all existing clients. However, it frequently happens that one wants to take an existing type and make it more flexible that it used to be. This often entails adding a new type parameter so that some type which was hard-coded before can now be customized. Defaults provide a means to do this while having older clients transparently fallback to the older behavior.
Historical example: Extending HashMap to support various hash algorithms.
Detailed Design
Remove feature gate
This RFC would remove the feature gate on defaulted type parameters.
Type parameters with defaults
Defaults can be placed on any type parameter, whether it is declared
on a type definition (struct
, enum
), type alias (type
), trait
definition (trait
), trait implementation (impl
), or a function or
method (fn
).
Once a given type parameter declares a default value, all subsequent type parameters in the list must declare default values as well:
// OK. All defaulted type parameters come at the end.
fn foo<A,B=uint,C=uint>() { .. }
// ERROR. B has a default, but C does not.
fn foo<A,B=uint,C>() { .. }
The default value of a type parameter X
may refer to other type
parameters declared on the same item. However, it may only refer to
type parameters declared before X
in the list of type parameters:
// OK. Default value of `B` refers to `A`, which is not defaulted.
fn foo<A,B=A>() { .. }
// OK. Default value of `C` refers to `B`, which comes before
// `C` in the list of parameters.
fn foo<A,B=uint,C=B>() { .. }
// ERROR. Default value of `B` refers to `C`, which comes AFTER
// `B` in the list of parameters.
fn foo<A,B=C,C=uint>() { .. }
Instantiating defaults
This section specifies how to interpret a reference to a generic type. Rather than writing out a rather tedious (and hard to understand) description of the algorithm, the rules are instead specified by a series of examples. The high-level idea of the rules is as follows:
- Users must always provide some value for non-defaulted type parameters. Defaulted type parameters may be omitted.
- The
_
notation can always be used to explicitly omit the value of a type parameter:- Inside a fn body, any type parameter may be omitted. Inference is used.
- Outside a fn body, only defaulted type parameters may be omitted. The default value is used.
- Motivation: This is consistent with Rust tradition, which
generally requires explicit types or a mechanical defaulting
process outside of
fn
bodies.
References to generic types
We begin with examples of references to the generic type Foo
:
struct Foo<A,B,C=DefaultHasher,D=C> { ... }
Foo
defines four type parameters, the final two of which are
defaulted. First, let us consider what happens outside of a fn
body. It is mandatory to supply explicit values for all non-defaulted
type parameters:
// ERROR: 2 parameters required, 0 provided.
fn f(_: &Foo) { ... }
Defaulted type parameters are filled in based on the defaults given:
// Legal: Equivalent to `Foo<int,uint,DefaultHasher,DefaultHasher>`
fn f(_: &Foo<int,uint>) { ... }
Naturally it is legal to specify explicit values for the defaulted type parameters if desired:
// Legal: Equivalent to `Foo<int,uint,uint,char,u8>`
fn f(_: &Foo<int,uint,char,u8>) { ... }
It is also legal to provide just one of the defaulted type parameters and not the other:
// Legal: Equivalent to `Foo<int,uint,char,char>`
fn f(_: &Foo<int,uint,char>) { ... }
If the user wishes to supply the value of the type parameter D
explicitly, but not C
, then _
can be used to request the default:
// Legal: Equivalent to `Foo<int,uint,DefaultHasher,uint>`
fn f(_: &Foo<int,uint,_,uint>) { ... }
Note that, outside of a fn body, _
can only be used with
defaulted type parameters:
// ERROR: outside of a fn body, `_` cannot be
// used for a non-defaulted type parameter
fn f(_: &Foo<int,_>) { ... }
Inside a fn body, the rules are much the same, except that _
is
legal everywhere. Every reference to _
creates a fresh type
variable $n
. If the type parameter whose value is omitted has an
associate default, that default is used as the fallback for $n
(see the section “Type variables with fallbacks” for more
information). Here are some examples:
fn f() {
// Error: `Foo` requires at least 2 type parameters, 0 supplied.
let x: Foo = ...;
// All of these 4 examples are OK and equivalent. Each
// results in a type `Foo<$0,$1,$2,$3>` and `$0`-`$4` are type
// variables. `$2` has a fallback of `DefaultHasher` and `$3`
// has a fallback of `$2`.
let x: Foo<_,_> = ...;
let x: Foo<_,_,_> = ...;
let x: Foo<_,_,_,_> = ...;
// Results in a type `Foo<int,uint,$0,char>` where `$0`
// has a fallback of `DefaultHasher`.
let x: Foo<int,uint,_,char> = ...;
}
References to generic traits
The rules for traits are the same as the rules for types. Consider a
trait Foo
:
trait Foo<A,B,C=uint,D=C> { ... }
References to this trait can omit values for C
and D
in precisely
the same way as was shown for types:
// All equivalent to Foo<i8,u8,uint,uint>:
fn foo<T:Foo<i8,u8>>() { ... }
fn foo<T:Foo<i8,u8,_>>() { ... }
fn foo<T:Foo<i8,u8,_,_>>() { ... }
// Equivalent to Foo<i8,u8,char,char>:
fn foo<T:Foo<i8,u8,char,_>>() { ... }
References to generic functions
The rules for referencing generic functions are the same as for types,
except that it is legal to omit values for all type parameters if
desired. In that case, the behavior is the same as it would be if _
were used as the value for every type parameter. Note that functions
can only be referenced from within a fn body.
References to generic impls
Users never explicitly “reference” an impl. Rather, the trait matching system implicitly instantiates impls as part of trait matching. This implies that all type parameters are always instantiated with type variables. These type variables are assigned fallbacks according to the defaults given.
Type variables with fallbacks
We extend the inference system so that when a type variable is created, it can optionally have a fallback value, which is another type.
In the type checker, whenever we create a fresh type variable to represent a type parameter with an associated default, we will use that default as the fallback value for this type variable.
Example:
fn foo<A,B=A>(a: A, b: B) { ... }
fn bar() {
// Here, the values of the type parameters are given explicitly.
let f: fn(uint, uint) = foo::<uint, uint>;
// Here the value of the first type parameter is given explicitly,
// but not the second. Because the second specifies a default, this
// is permitted. The type checker will create a fresh variable `$0`
// and attempt to infer the value of this defaulted type parameter.
let g: fn(uint, $0) = foo::<uint>;
// Here, the values of the type parameters are not given explicitly,
// and hence the type checker will create fresh variables
// `$1` and `$2` for both of them.
let h: fn($1, $2) = foo;
}
In this snippet, there are three references to the generic function
foo
, each of which specifies progressively fewer types. As a result,
the type checker winds up creating three type variables, which are
referred to in the example as $0
, $1
, and $2
(not that this $
notation is just for explanatory purposes and is not actual Rust
syntax).
The fallback values of $0
, $1
, and $2
are as follows:
$0
was created to represent the type parameterB
defined onfoo
. This means that$0
will have a fallback value ofuint
, since the type variableA
was specified to beuint
in the expression that created$0
.$1
was created to represent the type parameterA
, which has no default. Therefore$1
has no fallback.$2
was created to represent the type parameterB
. It will have the fallback value of$1
, which was the value ofA
within the expression where$2
was created.
Trait resolution, fallbacking, and inference
Prior to this RFC, type-checking a function body proceeds roughly as follows:
- The function body is analyzed. This results in an accumulated set of type variables, constraints, and trait obligations.
- Those trait obligations are then resolved until a fixed point is reached.
- If any trait obligations remain unresolved, an error is reported.
- If any type variables were never bound to a concrete value, an error is reported.
To accommodate fallback, the new procedure is somewhat different:
- The function body is analyzed. This results in an accumulated set of type variables, constraints, and trait obligations.
- Execute in a loop:
- Run trait resolution until a fixed point is reached.
- Create a (initially empty) set
UB
of unbound type and integral/float variables. This set represents the set of variables for which fallbacks should be applied. - Add all unbound integral and float variables to the set
UB
- For each type variable
X
:- If
X
has no fallback defined, skip. - If
X
is not bound, addX
toUB
- If
X
is bound to an unbound integral variableI
, addX
toUB
and removeI
fromUB
(if present). - If
X
is bound to an unbound float variableF
, addX
toUB
and removeF
fromUB
(if present).
- If
- If
UB
is the empty set, break out of the loop. - For each member of
UB
:- If the member is an integral type variable
I
, setI
toint
. - If the member is a float variable
F
, setI
tof64
. - Otherwise, the member must be a variable
X
with a defined fallback. SetX
to its fallback.- Note that this “set” operations can fail, which indicates conflicting defaults. A suitable error message should be given.
- If the member is an integral type variable
- If any type parameters still have no value assigned to them, report an error.
- If any trait obligations could not be resolved, report an error.
There are some subtle points to this algorithm:
When defaults are to be applied, we first gather up the set of variables that have applicable defaults (step 2.2) and then later unconditionally apply those defaults (step 2.4). In particular, we do not loop over each type variable, check whether it is unbound, and apply the default only if it is unbound. The reason for this is that it can happen that there are contradictory defaults and we want to ensure that this results in an error:
fn foo<F:Default=uint>() -> F { }
fn bar<B=int>(b: B) { }
fn baz() {
// Here, F is instantiated with $0=uint
let x: $0 = foo();
// Here, B is instantiated with $1=uint, and constraint $0 <: $1 is added.
bar(x);
}
In this example, two type variables are created. $0
is the value of
F
in the call to foo()
and $1
is the value of B
in the call to
bar()
. The fact that x
, which has type $0
, is passed as an
argument to bar()
will add the constraint that $0 <: $1
, but at no
point are any concrete types given. Therefore, once type checking is
complete, we will apply defaults. Using the algorithm given above, we
will determine that both $0
and $1
are unbound and have suitable
defaults. We will then unify $0
with uint
. This will succeed and,
because $0 <: $1
, cause $1
to be unified with uint
. Next, we
will try to unify $1
with its default, int
. This will lead to an
error. If we combined the checking of whether $1
was unbound with
the unification with the default, we would have first unified $0
and
then decided that $1
did not require unification.
In the general case, a loop is required to continue resolving traits and applying defaults in sequence. Resolving traits can lead to unifications, so it is clear that we must resolve all traits that we can before we apply any defaults. However, it is also true that adding defaults can create new trait obligations that must be resolved.
Here is an example where processing trait obligations creates defaults, and processing defaults created trait obligations:
trait Foo { }
trait Bar { }
impl<T:Bar=uint> Foo for Vec<T> { } // Impl 1
impl Bar for uint { } // Impl 2
fn takes_foo<F:Foo>(f: F) { }
fn main() {
let x = Vec::new(); // x: Vec<$0>
takes_foo(x); // adds oblig Vec<$0> : Foo
}
When we finish type checking main
, we are left with a variable $0
and a trait obligation Vec<$0> : Foo
. Processing the trait
obligation selects the impl 1 as the way to fulfill this trait
obligation. This results in:
- a new type variable
$1
, which represents the parameterT
on the impl.$1
has a default,uint
. - the constraint that
$0=$1
. - a new trait obligation
$1 : Bar
.
We cannot process the new trait obligation yet because the type
variable $1
is still unbound. (We know that it is equated with $0
,
but we do not have any concrete types yet, just variables.) After
trait resolution reaches a fixed point, defaults are applied. $1
is
equated with uint
which in turn propagates to $0
. At this point,
there is still an outstanding trait obligation uint : Bar
. This
trait obligation can be resolved to impl 2.
The previous example consisted of “1.5” iterations of the loop. That is, although trait resolution runs twice, defaults are only needed one time:
- Trait resolution executed to resolve
Vec<$0> : Foo
. - Defaults were applied to unify
$1 = $0 = uint
. - Trait resolution executed to resolve
uint : Bar
- No more defaults to apply, done.
The next example does 2 full iterations of the loop.
trait Foo { }
trait Bar<U> { }
trait Baz { }
impl<U,T:Bar<U>=Vec<U>> Foo for Vec<T> { } // Impl 1
impl<V=uint> Bar for Vec<V> { } // Impl 2
fn takes_foo<F:Foo>(f: F) { }
fn main() {
let x = Vec::new(); // x: Vec<$0>
takes_foo(x); // adds oblig Vec<$0> : Foo
}
Here the process is as follows:
- Trait resolution executed to resolve
Vec<$0> : Foo
. The result is two fresh variables,$1
(forU
) and$2=Vec<$1>
(for$T
), the constraint that$0=$2
, and the obligation$2 : Bar<$1>
. - Defaults are applied to unify
$2 = $0 = Vec<$1>
. - Trait resolution executed to resolve
$2 : Bar<$1>
. The result is a fresh variable$3=uint
(for$V
) and the constraint that$1=$3
. - Defaults are applied to unify
$3 = $1 = uint
.
It should be clear that one can create examples in this vein so as to require any number of loops.
Interaction with integer/float literal fallback. This RFC gives defaulted type parameters precedence over integer/float literal fallback. This seems preferable because such types can be more specific. Below are some examples. See also the alternatives section.
// Here the type of the integer literal 22 is inferred
// to `int` using literal fallback.
fn foo<T>(t: T) { ... }
foo(22)
// Here the type of the integer literal 22 is inferred
// to `uint` because the default on `T` overrides the
// standard integer literal fallback.
fn foo<T=uint>(t: T) { ... }
foo(22)
// Here the type of the integer literal 22 is inferred
// to `char`, leading to an error. This can be resolved
// by using an explicit suffix like `22i`.
fn foo<T=char>(t: T) { ... }
foo(22)
Termination. Any time that there is a loop, one must inquire after termination. In principle, the loop above could execute indefinitely. This is because trait resolution is not guaranteed to terminate – basically there might be a cycle between impls such that we continue creating new type variables and new obligations forever. The trait matching system already defends against this with a recursion counter. That same recursion counter is sufficient to guarantee termination even when the default mechanism is added to the mix. This is because the default mechanism can never itself create new trait obligations: it can only cause previous ambiguous trait obligations to now be matchable (because unbound variables become bound). But the actual need to iteration through the loop is still caused by trait matching generating recursive obligations, which have an associated depth limit.
Compatibility analysis
One of the major design goals of defaulted type parameters is to permit new parameters to be added to existing types or methods in a backwards compatible way. This remains possible under the current design.
Note though that adding a default to an existing type parameter can lead to type errors in clients. This can occur if clients were already relying on an inference fallback from some other source and there is now an ambiguity. Naturally clients can always fix this error by specifying the value of the type parameter in question manually.
Downsides and alternatives
Avoid inference
Rather than adding the notion of fallbacks to type variables,
defaults could be mechanically added, even within fn bodies, as they
are today. But this is disappointing because it means that examples
like range(0,10)
, where defaults could inform inference, still
require explicit annotation. Without the notion of fallbacks, it is
also difficult to say what defaulted type parameters in methods or
impls should mean.
More advanced interaction between integer literal inference
There were some other proposals to have a more advanced interaction between custom fallbacks and literal inference. For example, it is possible to imagine that we allow literal inference to take precedence over type default fallbacks, unless the fallback is itself integral. The problem is that this is both complicated and possibly not forwards compatible if we opt to allow a more general notion of literal inference in the future (in other words, if integer literals may be mapped to more than just the built-in integral types). Furthermore, these rules would create strictly fewer errors, and hence can be added in the future if desired.
Notation
Allowing _
notation outside of fn body means that it’s meaning
changes somewhat depending on context. However, this is consistent
with the meaning of omitted lifetimes, which also change in the same
way (mechanical default outside of fn body, inference within).
An alternative design is to use the K=V
notation proposed in the
associated items RFC for specify the values of default type
parameters. However, this is somewhat odd, because default type
parameters appear in a positional list, and thus it is surprising that
values for the non-defaulted parameters are given positionally, but
values for the defaulted type parameters are given with labels.
Another alternative would to simply prohibit users from specifying the value of a defaulted type parameter unless values are given for all previous defaulted typed parameters. But this is clearly annoying in those cases where defaulted type parameters represent distinct axes of customization.
Hat Tip
eddyb introduced defaulted type parameters and also opened the first pull request that used them to inform inference.
- Start Date: 2014-08-27
- RFC PR: rust-lang/rfcs#214
- Rust Issue: rust-lang/rust#17687
Summary
Introduce a new while let PAT = EXPR { BODY }
construct. This allows for using a refutable pattern
match (with optional variable binding) as the condition of a loop.
Motivation
Just as if let
was inspired by Swift, it turns out Swift supports while let
as well. This was
not discovered until much too late to include it in the if let
RFC. It turns out that this sort of
looping is actually useful on occasion. For example, the desugaring for
loop is actually a variant
on this; if while let
existed it could have been implemented to map for PAT in EXPR { BODY }
to
// the match here is so `for` can accept an rvalue for the iterator,
// and was used in the "real" desugaring version.
match &mut EXPR {
i => {
while let Some(PAT) = i.next() {
BODY
}
}
}
(note that the non-desugared form of for
is no longer equivalent).
More generally, this construct can be used any time looping + pattern-matching is desired.
This also makes the language a bit more consistent; right now, any condition that can be used with
if
can be used with while
. The new if let
adds a form of if
that doesn’t map to while
.
Supporting while let
restores the equivalence of these two control-flow constructs.
Detailed design
while let
operates similarly to if let
, in that it desugars to existing syntax. Specifically,
the syntax
['ident:] while let PAT = EXPR {
BODY
}
desugars to
['ident:] loop {
match EXPR {
PAT => BODY,
_ => break
}
}
Just as with if let
, an irrefutable pattern given to while let
is considered an error. This is
largely an artifact of the fact that the desugared match
ends up with an unreachable pattern,
and is not actually a goal of this syntax. The error may be suppressed in the future, which would be
a backwards-compatible change.
Just as with if let
, while let
will be introduced under a feature gate (named while_let
).
Drawbacks
Yet another addition to the grammar. Unlike if let
, it’s not obvious how useful this syntax will
be.
Alternatives
As with if let
, this could plausibly be done with a macro, but it would be ugly and produce bad
error messages.
while let
could be extended to support alternative patterns, just as match arms do. This is not
part of the main proposal for the same reason it was left out of if let
, which is that a) it looks
weird, and b) it’s a bit of an odd coupling with the let
keyword as alternatives like this aren’t
going to be introducing variable bindings. However, it would make while let
more general and able
to replace more instances of loop { match { ... } }
than is possible with the main design.
Unresolved questions
None.
- Start Date: 2014-08-28
- RFC PR: (https://github.com/rust-lang/rfcs/pull/216)
- Rust Issue: (https://github.com/rust-lang/rust/issues/17320)
Summary
Add additional iterator-like Entry objects to collections.
Entries provide a composable mechanism for in-place observation and mutation of a
single element in the collection, without having to “re-find” the element multiple times.
This deprecates several “internal mutation” methods like hashmap’s find_or_insert_with
.
Motivation
As we approach 1.0, we’d like to normalize the standard APIs to be consistent, composable, and simple. However, this currently stands in opposition to manipulating the collections in an efficient manner. For instance, if one wishes to build an accumulating map on top of one of the concrete maps, they need to distinguish between the case when the element they’re inserting is already in the map, and when it’s not. One way to do this is the following:
if map.contains_key(&key) {
*map.find_mut(&key).unwrap() += 1;
} else {
map.insert(key, 1);
}
However, searches for key
twice on every operation.
The second search can be squeezed out the update
re-do by matching on the result
of find_mut
, but the insert
case will always require a re-search.
To solve this problem, Rust currently has an ad-hoc mix of “internal mutation” methods which take multiple values or closures for the collection to use contextually. Hashmap in particular has the following methods:
fn find_or_insert<'a>(&'a mut self, k: K, v: V) -> &'a mut V
fn find_or_insert_with<'a>(&'a mut self, k: K, f: |&K| -> V) -> &'a mut V
fn insert_or_update_with<'a>(&'a mut self, k: K, v: V, f: |&K, &mut V|) -> &'a mut V
fn find_with_or_insert_with<'a, A>(&'a mut self, k: K, a: A, found: |&K, &mut V, A|, not_found: |&K, A| -> V) -> &'a mut V
Not only are these methods fairly complex to use, but they’re over-engineered and
combinatorially explosive. They all seem to return a mutable reference to the region
accessed “just in case”, and find_with_or_insert_with
takes a magic argument a
to
try to work around the fact that the two closures it requires can’t both close over
the same value (even though only one will ever be called). find_with_or_insert_with
is also actually performing the role of insert_with_or_update_with
,
suggesting that these aren’t well understood.
Rust has been in this position before: internal iteration. Internal iteration was (author’s note: I’m told) confusing and complicated. However the solution was simple: external iteration. You get all the benefits of internal iteration, but with a much simpler interface, and greater composability. Thus, this RFC proposes the same solution to the internal mutation problem.
Detailed design
A fully tested “proof of concept” draft of this design has been implemented on top of hashmap, as it seems to be the worst offender, while still being easy to work with. It sits as a pull request here.
All the internal mutation methods are replaced with a single method on a collection: entry
.
The signature of entry
will depend on the specific collection, but generally it will be similar to
the signature for searching in that structure. entry
will in turn return an Entry
object, which
captures the state of a completed search, and allows mutation of the area.
For convenience, we will use the hashmap draft as an example.
/// Get an Entry for where the given key would be inserted in the map
pub fn entry<'a>(&'a mut self, key: K) -> Entry<'a, K, V>;
/// A view into a single occupied location in a HashMap
pub struct OccupiedEntry<'a, K, V>{ ... }
/// A view into a single empty location in a HashMap
pub struct VacantEntry<'a, K, V>{ ... }
/// A view into a single location in a HashMap
pub enum Entry<'a, K, V> {
/// An occupied Entry
Occupied(OccupiedEntry<'a, K, V>),
/// A vacant Entry
Vacant(VacantEntry<'a, K, V>),
}
Of course, the real meat of the API is in the Entry’s interface (impl details removed):
impl<'a, K, V> OccupiedEntry<'a, K, V> {
/// Gets a reference to the value of this Entry
pub fn get(&self) -> &V;
/// Gets a mutable reference to the value of this Entry
pub fn get_mut(&mut self) -> &mut V;
/// Converts the entry into a mutable reference to its value
pub fn into_mut(self) -> &'a mut V;
/// Sets the value stored in this Entry
pub fn set(&mut self, value: V) -> V;
/// Takes the value stored in this Entry
pub fn take(self) -> V;
}
impl<'a, K, V> VacantEntry<'a, K, V> {
/// Set the value stored in this Entry, and returns a reference to it
pub fn set(self, value: V) -> &'a mut V;
}
There are definitely some strange things here, so let’s discuss the reasoning!
First, entry
takes a key
by value, because this is the observed behaviour of the internal mutation
methods. Further, taking the key
up-front allows implementations to avoid validating provided keys if
they require an owned key
later for insertion. This key is effectively a guarantor of the entry.
Taking the key by-value might change once collections reform lands, and Borrow and ToOwned are available.
For now, it’s an acceptable solution, because in particular, the primary use case of this functionality
is when you’re not sure if you need to insert, in which case you should be prepared to insert.
Otherwise, find_mut
is likely sufficient.
The result is actually an enum, that will either be Occupied or Vacant. These two variants correspond to concrete types for when the key matched something in the map, and when the key didn’t, respectively.
If there isn’t a match, the user has exactly one option: insert a value using set
, which will also insert
the guarantor, and destroy the Entry. This is to avoid the costs of maintaining the structure, which
otherwise isn’t particularly interesting anymore.
If there is a match, a more robust set of options is provided. get
and get_mut
provide access to the
value found in the location. set
behaves as the vacant variant, but without destroying the entry.
It also yields the old value. take
simply removes the found value, and destroys the entry for similar reasons as set
.
Let’s look at how we one now writes insert_or_update
:
There are two options. We can either do the following:
// cleaner, and more flexible if logic is more complex
let val = match map.entry(key) {
Vacant(entry) => entry.set(0),
Occupied(entry) => entry.into_mut(),
};
*val += 1;
or
// closer to the original, and more compact
match map.entry(key) {
Vacant(entry) => { entry.set(1); },
Occupied(mut entry) => { *entry.get_mut() += 1; },
}
Either way, one can now write something equivalent to the “intuitive” inefficient code, but it is now as efficient as the complex
insert_or_update
methods. In fact, this matches so closely to the inefficient manipulation
that users could reasonable ignore Entries until performance becomes an issue, at which point
it’s an almost trivial migration. Closures also aren’t needed to dance around the fact that one may
want to avoid generating some values unless they have to, because that falls naturally out of
normal control flow.
If you look at the actual patch that does this, you’ll see that Entry itself is exceptionally simple to implement. Most of the logic is trivial. The biggest amount of work was just capturing the search state correctly, and even that was mostly a cut-and-paste job.
With Entries, the gate is also opened for… adaptors!
Really want insert_or_update
back? That can be written on top of this generically with ease.
However, such discussion is out-of-scope for this RFC. Adaptors can
be tackled in a back-compat manner after this has landed, and usage is observed. Also, this
proposal does not provide any generic trait for Entries, preferring concrete implementations for
the time-being.
Drawbacks
-
More structs, and more methods in the short-term
-
More collection manipulation “modes” for the user to think about
-
insert_or_update_with
is kind of convenient for avoiding the kind of boiler-plate found in the examples
Alternatives
-
Just put our foot down, say “no efficient complex manipulations”, and drop all the internal mutation stuff without a replacement.
-
Try to build out saner/standard internal manipulation methods.
-
Try to make this functionality a subset of Cursors, which would be effectively a bi-directional mut_iter where the returned references borrow the cursor preventing aliasing/safety issues, so that mutation can be performed at the location of the cursor. However, preventing invalidation would be more expensive, and it’s not clear that cursor semantics would make sense on e.g. a HashMap, as you can’t insert any key in any location.
-
This RFC originally [proposed a design without enums that was substantially more complex] (https://github.com/Gankro/rust/commit/6d6804a6d16b13d07934f0a217a3562384e55612). However it had some interesting ideas about Key manipulation, so we mention it here for historical purposes.
Unresolved questions
Naming bikesheds!
- Start Date: 2014-08-28
- RFC PR: rust-lang/rfcs#218
- Rust Issue: rust-lang/rust#24266
Summary
When a struct type S
has no fields (a so-called “empty struct”),
allow it to be defined via either struct S;
or struct S {}
.
When defined via struct S;
, allow instances of it to be constructed
and pattern-matched via either S
or S {}
.
When defined via struct S {}
, require instances to be constructed
and pattern-matched solely via S {}
.
Motivation
Today, when writing code, one must treat an empty struct as a special case, distinct from structs that include fields. That is, one must write code like this:
struct S2 { x1: int, x2: int }
struct S0; // kind of different from the above.
let s2 = S2 { x1: 1, x2: 2 };
let s0 = S0; // kind of different from the above.
match (s2, s0) {
(S2 { x1: y1, x2: y2 },
S0) // you can see my pattern here
=> { println!("Hello from S2({}, {}) and S0", y1, y2); }
}
While this yields code that is relatively free of extraneous
curly-braces, this special case handling of empty structs presents
problems for two cases of interest: automatic code generators
(including, but not limited to, Rust macros) and conditionalized code
(i.e. code with cfg
attributes; see the CFG problem appendix).
The heart of the code-generator argument is: Why force all
to-be-written code-generators and macros with special-case handling of
the empty struct case (in terms of whether or not to include the
surrounding braces), especially since that special case is likely to
be forgotten (yielding a latent bug in the code generator).
The special case handling of empty structs is also a problem for programmers who actively add and remove fields from structs during development; such changes cause a struct to switch from being empty and non-empty, and the associated revisions of changing removing and adding curly braces is aggravating (both in effort revising the code, and also in extra noise introduced into commit histories).
This RFC proposes an approach similar to the one we used circa February
2013, when both S0
and S0 { }
were accepted syntaxes for an empty
struct. The parsing ambiguity that motivated removing support for
S0 { }
is no longer present (see the Ancient History appendix).
Supporting empty braces in the syntax for empty structs is easy to do
in the language now.
Detailed design
There are two kinds of empty structs: Braced empty structs and flexible empty structs. Flexible empty structs are a slight generalization of the structs that we have today.
Flexible empty structs are defined via the syntax struct S;
(as today).
Braced empty structs are defined via the syntax struct S { }
(“new”).
Both braced and flexible empty structs can be constructed via the
expression syntax S { }
(“new”). Flexible empty structs, as today,
can also be constructed via the expression syntax S
.
Both braced and flexible empty structs can be pattern-matched via the
pattern syntax S { }
(“new”). Flexible empty structs, as today,
can also be pattern-matched via the pattern syntax S
.
Braced empty struct definitions solely affect the type namespace, just like normal non-empty structs. Flexible empty structs affect both the type and value namespaces.
As a matter of style, using braceless syntax is preferred for constructing and pattern-matching flexible empty structs. For example, pretty-printer tools are encouraged to emit braceless forms if they know that the corresponding struct is a flexible empty struct. (Note that pretty printers that handle incomplete fragments may not have such information available.)
There is no ambiguity introduced by this change, because we have already introduced a restriction to the Rust grammar to force the use of parentheses to disambiguate struct literals in such contexts. (See Rust RFC 25).
The expectation is that when migrating code from a flexible empty struct to a non-empty struct, it can start by first migrating to a braced empty struct (and then have a tool indicate all of the locations where braces need to be added); after that step has been completed, one can then take the next step of adding the actual field.
Drawbacks
Some people like “There is only one way to do it.” But, there is precedent in Rust for violating “one way to do it” in favor of syntactic convenience or regularity; see the Precedent for flexible syntax in Rust appendix. Also, see the Always Require Braces alternative below.
I have attempted to summarize the previous discussion from RFC PR 147 in the Recent History appendix; some of the points there include drawbacks to this approach and to the Always Require Braces alternative.
Alternatives
Always Require Braces
Alternative 1: “Always Require Braces”. Specifically, require empty
curly braces on empty structs. People who like the current syntax of
curly-brace free structs can encode them this way: enum S0 { S0 }
This would address all of the same issues outlined above. (Also, the
author (pnkfelix) would be happy to take this tack.)
The main reason not to take this tack is that some people may like writing empty structs without braces, but do not want to switch to the unary enum version described in the previous paragraph. See “I wouldn’t want to force noisier syntax …” in the Recent History appendix.
Status quo
Alternative 2: Status quo. Macros and code-generators in general will need to handle empty structs as a special case. We may continue hitting bugs like CFG parse bug. Some users will be annoyed but most will probably cope.
Synonymous in all contexts
Alternative 3: An earlier version of this RFC proposed having struct S;
be entirely synonymous with struct S { }
, and the expression
S { }
be synonymous with S
.
This was deemed problematic, since it would mean that S { }
would
put an entry into both the type and value namespaces, while
S { x: int }
would only put an entry into the type namespace.
Thus the current draft of the RFC proposes the “flexible” versus
“braced” distinction for empty structs.
Never synonymous
Alternative 4: Treat struct S;
as requiring S
at the expression
and pattern sites, and struct S { }
as requiring S { }
at the
expression and pattern sites.
This in some ways follows a principle of least surprise, but it also
is really hard to justify having both syntaxes available for empty
structs with no flexibility about how they are used. (Note again that
one would have the option of choosing between
enum S { S }
, struct S;
, or struct S { }
, each with their own
idiosyncrasies about whether you have to write S
or S { }
.)
I would rather adopt “Always Require Braces” than “Never Synonymous”
Empty Tuple Structs
One might say “why are you including support for curly braces, but not parentheses?” Or in other words, “what about empty tuple structs?”
The code-generation argument could be applied to tuple-structs as
well, to claim that we should allow the syntax S0()
. I am less
inclined to add a special case for that; I think tuple-structs are
less frequently used (especially with many fields); they are largely
for ad-hoc data such as newtype wrappers, not for code generators.
Note that we should not attempt to generalize this RFC as proposed to
include tuple structs, i.e. so that given struct S0 {}
, the
expressions T0
, T0 {}
, and T0()
would be synonymous. The reason
is that given a tuple struct struct T2(int, int)
, the identifier
T2
is already bound to a constructor function:
fn main() {
#[deriving(Show)]
struct T2(int, int);
fn foo<S:std::fmt::Show>(f: |int, int| -> S) {
println!("Hello from {} and {}", f(2,3), f(4,5));
}
foo(T2);
}
So if we were to attempt to generalize the leniency of this RFC to
tuple structs, we would be in the unfortunate situation given struct T0();
of trying to treat T0
simultaneously as an instance of the
struct and as a constructor function. So, the handling of empty
structs proposed by this RFC does not generalize to tuple structs.
(Note that if we adopt alternative 1, Always Require Braces, then
the issue of how tuple structs are handled is totally orthogonal – we
could add support for struct T0()
as a distinct type from struct S0 {}
, if we so wished, or leave it aside.)
Unresolved questions
None
Appendices
The CFG problem
A program like this works today:
fn main() {
#[deriving(Show)]
struct Svaries {
x: int,
y: int,
#[cfg(zed)]
z: int,
}
let s = match () {
#[cfg(zed)] _ => Svaries { x: 3, y: 4, z: 5 },
#[cfg(not(zed))] _ => Svaries { x: 3, y: 4 },
};
println!("Hello from {}", s)
}
Observe what happens when one modifies the above just a bit:
struct Svaries {
#[cfg(eks)]
x: int,
#[cfg(why)]
y: int,
#[cfg(zed)]
z: int,
}
Now, certain cfg
settings yield an empty struct, even though it
is surrounded by braces. Today this leads to a CFG parse bug
when one attempts to actually construct such a struct.
If we want to support situations like this properly, we will probably
need to further extend the cfg
attribute so that it can be placed
before individual fields in a struct constructor, like this:
// You cannot do this today,
// but maybe in the future (after a different RFC)
let s = Svaries {
#[cfg(eks)] x: 3,
#[cfg(why)] y: 4,
#[cfg(zed)] z: 5,
};
Supporting such a syntax consistently in the future should start today with allowing empty braces as legal code. (Strictly speaking, it is not necessary that we add support for empty braces at the parsing level to support this feature at the semantic level. But supporting empty-braces in the syntax still seems like the most consistent path to me.)
Ancient History
A parsing ambiguity was the original motivation for disallowing the
syntax S {}
in favor of S
for constructing an instance of
an empty struct. The ambiguity and various options for dealing with it
were well documented on the rust-dev thread.
Both syntaxes were simultaneously supported at the time.
In particular, at the time that mailing list thread was created, the
code match match x {} ...
would be parsed as match (x {}) ...
, not
as (match x {}) ...
(see Rust PR 5137); likewise, if x {}
would
be parsed as an if-expression whose test component is the struct
literal x {}
. Thus, at the time of Rust PR 5137, if the input to
a match
or if
was an identifier expression, one had to put
parentheses around the identifier to force it to be interpreted as
input to the match
/if
, and not as a struct constructor.
Of the options for resolving this discussed on the mailing list
thread, the one selected (removing S {}
construction expressions)
was chosen as the most expedient option.
At that time, the option of “Place a parser restriction on those
contexts where {
terminates the expression and say that struct
literals cannot appear there unless they are in parentheses.” was
explicitly not chosen, in favor of continuing to use the
disambiguation rule in use at the time, namely that the presence of a
label (e.g. S { a_label: ... }
) was the way to distinguish a
struct constructor from an identifier followed by a control block, and
thus, “there must be one label.”
Naturally, if the construction syntax were to be disallowed, it made
sense to also remove the struct S {}
declaration syntax.
Things have changed since the time of that mailing list thread;
namely, we have now adopted the aforementioned parser restriction
Rust RFC 25. (The text of RFC 25 does not explicitly address
match
, but we have effectively expanded it to include a curly-brace
delimited block of match-arms in the definition of “block”.) Today,
one uses parentheses around struct literals in some contexts (such as
for e in (S {x: 3}) { ... }
or match (S {x: 3}) { ... }
Note that there was never an ambiguity for uses of struct S0 { }
in item
position. The issue was solely about expression position prior to the
adoption of Rust RFC 25.
Precedent for flexible syntax in Rust
There is precedent in Rust for violating “one way to do it” in favor of syntactic convenience or regularity.
For example, one can often include an optional trailing comma, for
example in: let x : &[int] = [3, 2, 1, ];
.
One can also include redundant curly braces or parentheses, for example in:
println!("hi: {}", { if { x.len() > 2 } { ("whoa") } else { ("there") } });
One can even mix the two together when delimiting match arms:
let z: int = match x {
[3, 2] => { 3 }
[3, 2, 1] => 2,
_ => { 1 },
};
We do have lints for some style violations (though none catch the cases above), but lints are different from fundamental language restrictions.
Recent history
There was a previous RFC PR that was effectively the same in spirit to this one. It was closed because it was not sufficient well fleshed out for further consideration by the core team. However, to save people the effort of reviewing the comments on that PR (and hopefully stave off potential bikeshedding on this PR), I here summarize the various viewpoints put forward on the comment thread there, and note for each one, whether that viewpoint would be addressed by this RFC (accept both syntaxes), by Always Require Braces, or by Status Quo.
Note that this list of comments is just meant to summarize the list of views; it does not attempt to reflect the number of commenters who agreed or disagreed with a particular point. (But since the RFC process is not a democracy, the number of commenters should not matter anyway.)
- “+1” ==> Favors: This RFC (or potentially Always Require Braces; I think the content of RFC PR 147 shifted over time, so it is hard to interpret the “+1” comments now).
- “I find
let s = S0;
jarring, think its an enum initially.” ==> Favors: Always Require Braces - “Frequently start out with an empty struct and add fields as I need them.” ==> Favors: This RFC or Always Require Braces
- “Foo{} suggests is constructing something that it’s not; all uses of the value
Foo
are indistinguishable from each other” ==> Favors: Status Quo - “I find it strange anyone would prefer
let x = Foo{};
overlet x = Foo;
” ==> Favors Status Quo; strongly opposes Always Require Braces. - “I agree that ‘instantiation-should-follow-declaration’, that is, structs declared
;, (), {}
should only be instantiated [via];, (), { }
respectively” ==> Opposes leniency of this RFC in that it allows expression to use include or omit{}
on an empty struct, regardless of declaration form, and vice-versa. - “The code generation argument is reasonable, but I wouldn’t want to force noisier syntax on all ‘normal’ code just to make macros work better.” ==> Favors: This RFC
- Start Date: 2014-09-23
- RFC PR #: rust-lang/rfcs#221
- Rust Issue #: rust-lang/rust#17489
Summary
Rename “task failure” to “task panic”, and fail!
to panic!
.
Motivation
The current terminology of “task failure” often causes problems when
writing or speaking about code. You often want to talk about the
possibility of an operation that returns a Result
“failing”, but
cannot because of the ambiguity with task failure. Instead, you have
to speak of “the failing case” or “when the operation does not
succeed” or other circumlocutions.
Likewise, we use a “Failure” header in rustdoc to describe when operations may fail the task, but it would often be helpful to separate out a section describing the “Err-producing” case.
We have been steadily moving away from task failure and toward
Result
as an error-handling mechanism, so we should optimize our
terminology accordingly: Result
-producing functions should be easy
to describe.
Detailed design
Not much more to say here than is in the summary: rename “task
failure” to “task panic” in documentation, and fail!
to panic!
in
code.
The choice of panic
emerged from a
discuss thread
and
workweek discussion.
It has precedent in a language setting in Go, and of course goes back
to Kernel panics.
With this choice, we can use “failure” to refer to an operation that
produces Err
or None
, “panic” for unwinding at the task level, and
“abort” for aborting the entire process.
The connotations of panic seem fairly accurate: the process is not immediately ending, but it is rapidly fleeing from some problematic circumstance (by killing off tasks) until a recovery point.
Drawbacks
The term “panic” is a bit informal, which some consider a drawback.
Making this change is likely to be a lot of work.
Alternatives
Other choices include:
-
throw!
orunwind!
. These options reasonably describe the current behavior of task failure, but “throw” suggests general exception handling, and both place the emphasis on the mechanism rather than the policy. We also are considering eventually adding a flag that allowsfail!
to abort the process, which would make these terms misleading. -
abort!
. Ambiguous with process abort. -
die!
. A reasonable choice, but it’s not immediately obvious what is being killed.
- Start Date: 2014-09-16
- RFC PR: rust-lang/rfcs#230
- Rust Issue: rust-lang/rust#17325
Summary
This RFC proposes to remove the runtime system that is currently part of the standard library, which currently allows the standard library to support both native and green threading. In particular:
-
The
libgreen
crate and associated support will be moved out of tree, into a separate Cargo package. -
The
librustrt
(the runtime) crate will be removed entirely. -
The
std::io
implementation will be directly welded to native threads and system calls. -
The
std::io
module will remain completely cross-platform, though separate platform-specific modules may be added at a later time.
Motivation
Background: thread/task models and I/O
Many languages/libraries offer some notion of “task” as a unit of concurrent execution, possibly distinct from native OS threads. The characteristics of tasks vary along several important dimensions:
-
1:1 vs M:N. The most fundamental question is whether a “task” always corresponds to an OS-level thread (the 1:1 model), or whether there is some userspace scheduler that maps tasks onto worker threads (the M:N model). Some kernels – notably, Windows – support a 1:1 model where the scheduling is performed in userspace, which combines some of the advantages of the two models.
In the M:N model, there are various choices about whether and when blocked tasks can migrate between worker threads. One basic downside of the model, however, is that if a task takes a page fault, the entire worker thread is essentially blocked until the fault is serviced. Choosing the optimal number of worker threads is difficult, and some frameworks attempt to do so dynamically, which has costs of its own.
-
Stack management. In the 1:1 model, tasks are threads and therefore must be equipped with their own stacks. In M:N models, tasks may or may not need their own stack, but there are important tradeoffs:
-
Techniques like segmented stacks allow stack size to grow over time, meaning that tasks can be equipped with their own stack but still be lightweight. Unfortunately, segmented stacks come with a significant performance and complexity cost.
-
On the other hand, if tasks are not equipped with their own stack, they either cannot be migrated between underlying worker threads (the case for frameworks like Java’s fork/join), or else must be implemented using continuation-passing style (CPS), where each blocking operation takes a closure representing the work left to do. (CPS essentially moves the needed parts of the stack into the continuation closure.) The upside is that such tasks can be extremely lightweight – essentially just the size of a closure.
-
-
Blocking and I/O support. In the 1:1 model, a task can block freely without any risk for other tasks, since each task is an OS thread. In the M:N model, however, blocking in the OS sense means blocking the worker thread. (The same applies to long-running loops or page faults.)
M:N models can deal with blocking in a couple of ways. The approach taken in Java’s fork/join framework, for example, is to dynamically spin up/down worker threads. Alternatively, special task-aware blocking operations (including I/O) can be provided, which are mapped under the hood to nonblocking operations, allowing the worker thread to continue. Unfortunately, this latter approach helps only with explicit blocking; it does nothing for loops, page faults and the like.
Where Rust is now
Rust has gradually migrated from a “green” threading model toward a native threading model:
-
In Rust’s green threading, tasks are scheduled M:N and are equipped with their own stack. Initially, Rust used segmented stacks to allow growth over time, but that was removed in favor of pre-allocated stacks, which means Rust’s green threads are not “lightweight”. The treatment of blocking is described below.
-
In Rust’s native threading model, tasks are 1:1 with OS threads.
Initially, Rust supported only the green threading model. Later, native threading was added and ultimately became the default.
In today’s Rust, there is a single I/O API – std::io
– that provides
blocking operations only and works with both threading models.
Rust is somewhat unusual in allowing programs to mix native and green threading,
and furthermore allowing some degree of interoperation between the two. This
feat is achieved through the runtime system – librustrt
– which exposes:
-
The
Runtime
trait, which abstracts over the scheduler (via methods likedeschedule
andspawn_sibling
) as well as the entire I/O API (vialocal_io
). -
The
rtio
module, which provides a number of traits that define the standard I/O abstraction. -
The
Task
struct, which includes aRuntime
trait object as the dynamic entry point into the runtime.
In this setup, libstd
works directly against the runtime interface. When
invoking an I/O or scheduling operation, it first finds the current Task
, and
then extracts the Runtime
trait object to actually perform the operation.
On native tasks, blocking operations simply block. On green tasks, blocking operations are routed through the green scheduler and/or underlying event loop and nonblocking I/O.
The actual scheduler and I/O implementations – libgreen
and libnative
–
then live as crates “above” libstd
.
The problems
While the situation described above may sound good in principle, there are several problems in practice.
Forced co-evolution. With today’s design, the green and native threading models must provide the same I/O API at all times. But there is functionality that is only appropriate or efficient in one of the threading models.
For example, the lightest-weight M:N task models are essentially just collections of closures, and do not provide any special I/O support. This style of lightweight tasks is used in Servo, but also shows up in java.util.concurrent’s exectors and Haskell’s par monad, among many others. These lighter weight models do not fit into the current runtime system.
On the other hand, green threading systems designed explicitly to support I/O may also want to provide low-level access to the underlying event loop – an API surface that doesn’t make sense for the native threading model.
Under the native model we want to provide direct non-blocking and/or
asynchronous I/O support – as a systems language, Rust should be able to work
directly with what the OS provides without imposing global abstraction
costs. These APIs may involve some platform-specific abstractions (epoll
,
kqueue
, IOCP) for maximal performance. But integrating them cleanly with a
green threading model may be difficult or impossible – and at the very least,
makes it difficult to add them quickly and seamlessly to the current I/O
system.
In short, the current design couples threading and I/O models together, and thus forces the green and native models to supply a common I/O interface – despite the fact that they are pulling in different directions.
Overhead. The current Rust model allows runtime mixtures of the green and native models. The implementation achieves this flexibility by using trait objects to model the entire I/O API. Unfortunately, this flexibility has several downsides:
-
Binary sizes. A significant overhead caused by the trait object design is that the entire I/O system is included in any binary that statically links to
libstd
. See this comment for more details. -
Task-local storage. The current implementation of task-local storage is designed to work seamlessly across native and green threads, and its performs substantially suffers as a result. While it is feasible to provide a more efficient form of “hybrid” TLS that works across models, doing so is far more difficult than simply using native thread-local storage.
-
Allocation and dynamic dispatch. With the current design, any invocation of I/O involves at least dynamic dispatch, and in many cases allocation, due to the use of trait objects. However, in most cases these costs are trivial when compared to the cost of actually doing the I/O (or even simply making a syscall), so they are not strong arguments against the current design.
Problematic I/O interactions. As the
documentation for libgreen
explains, only some I/O and synchronization methods work seamlessly across
native and green tasks. For example, any invocation of native code that calls
blocking I/O has the potential to block the worker thread running the green
scheduler. In particular, std::io
objects created on a native task cannot
safely be used within a green task. Thus, even though std::io
presents a
unified I/O API for green and native tasks, it is not fully interoperable.
Embedding Rust. When embedding Rust code into other contexts – whether
calling from C code or embedding in high-level languages – there is a fair
amount of setup needed to provide the “runtime” infrastructure that libstd
relies on. If libstd
was instead bound to the native threading and I/O
system, the embedding setup would be much simpler.
Maintenance burden. Finally, libstd
is made somewhat more complex by
providing such a flexible threading model. As this RFC will explain, moving to
a strictly native threading model will allow substantial simplification and
reorganization of the structure of Rust’s libraries.
Detailed design
To mitigate the above problems, this RFC proposes to tie std::io
directly to
the native threading model, while moving libgreen
and its supporting
infrastructure into an external Cargo package with its own I/O API.
The near-term plan
std::io
and native threading
The plan is to entirely remove librustrt
, including all of the traits.
The abstraction layers will then become:
-
Highest level:
libstd
, providing cross-platform, high-level I/O and scheduling abstractions. The crate will depend onlibnative
(the opposite of today’s situation). -
Mid-level:
libnative
, providing a cross-platform Rust interface for I/O and scheduling. The API will be relatively low-level, compared tolibstd
. The crate will depend onlibsys
. -
Low-level:
libsys
(renamed fromliblibc
), providing platform-specific Rust bindings to system C APIs.
In this scheme, the actual API of libstd
will not change significantly. But
its implementation will invoke functions in libnative
directly, rather than
going through a trait object.
A goal of this work is to minimize the complexity of embedding Rust code in other contexts. It is not yet clear what the final embedding API will look like.
Green threading
Despite tying libstd
to native threading, however, libgreen
will still be
supported – at least initially. The infrastructure in libgreen
and friends will
move into its own Cargo package.
Initially, the green threading package will support essentially the same
interface it does today; there are no immediate plans to change its API, since
the focus will be on first improving the native threading API. Note, however,
that the I/O API will be exposed separately within libgreen
, as opposed to the
current exposure through std::io
.
The long-term plan
Ultimately, a large motivation for the proposed refactoring is to allow the APIs for native I/O to grow.
In particular, over time we should expose more of the underlying system
capabilities under the native threading model. Whenever possible, these
capabilities should be provided at the libstd
level – the highest level of
cross-platform abstraction. However, an important goal is also to provide
nonblocking and/or asynchronous I/O, for which system APIs differ greatly. It
may be necessary to provide additional, platform-specific crates to expose this
functionality. Ideally, these crates would interoperate smoothly with libstd
,
so that for example a libposix
crate would allow using an poll
operation
directly against a std::io::fs::File
value, for example.
We also wish to expose “lowering” operations in libstd
– APIs that allow
you to get at the file descriptor underlying a std::io::fs::File
, for example.
On the other hand, we very much want to explore and support truly lightweight M:N task models (that do not require per-task stacks) – supporting efficient data parallelism with work stealing for CPU-bound computations. These lightweight models will not provide any special support for I/O. But they may benefit from a notion of “task-local storage” and interfacing with the task scheduler when explicitly synchronizing between tasks (via channels, for example).
All of the above long-term plans will require substantial new design and implementation work, and the specifics are out of scope for this RFC. The main point, though, is that the refactoring proposed by this RFC will make it much more plausible to carry out such work.
Finally, a guiding principle for the above work is uncompromising support for
native system APIs, in terms of both functionality and performance. For example,
it must be possible to use thread-local storage without significant overhead,
which is very much not the case today. Any abstractions to support M:N threading
models – including the now-external libgreen
package – must respect this
constraint.
Drawbacks
The main drawback of this proposal is that green I/O will be provided by a
forked interface of std::io
. This change makes green threading
“second class”, and means there’s more to learn when using both models
together.
This setup also somewhat increases the risk of invoking native blocking I/O on a green thread – though of course that risk is very much present today. One way of mitigating this risk in general is the Java executor approach, where the native “worker” threads that are executing the green thread scheduler are monitored for blocking, and new worker threads are spun up as needed.
Unresolved questions
There are may unresolved questions about the exact details of the refactoring,
but these are considered implementation details since the libstd
interface
itself will not substantially change as part of this RFC.
- Start Date: 2014-09-09
- RFC PR: rust-lang/rfcs#231
- Rust Issue: rust-lang/rust#16640
Summary
The ||
unboxed closure form should be split into two forms—||
for nonescaping closures and move ||
for escaping closures—and the capture clauses and self type specifiers should be removed.
Motivation
Having to specify ref
and the capture mode for each unboxed closure is inconvenient (see Rust PR rust-lang/rust#16610). It would be more convenient for the programmer if the type of the closure and the modes of the upvars could be inferred. This also eliminates the “line-noise” syntaxes like |&:|
, which are arguably unsightly.
Not all knobs can be removed, however—the programmer must manually specify whether each closure is escaping or nonescaping. To see this, observe that no sensible default for the closure || (*x).clone()
exists: if the function is nonescaping, it’s a closure that returns a copy of x
every time but does not move x
into it; if the function is escaping, it’s a closure that returns a copy of x
and takes ownership of x
.
Therefore, we need two forms: one for nonescaping closures and one for escaping closures. Nonescaping closures are the commonest, so they get the ||
syntax that we have today, and a new move ||
syntax will be introduced for escaping closures.
Detailed design
For unboxed closures specified with ||
, the capture modes of the free variables are determined as follows:
-
Any variable which is closed over and borrowed mutably is by-reference and mutably borrowed.
-
Any variable of a type that does not implement
Copy
which is moved within the closure is captured by value. -
Any other variable which is closed over is by-reference and immutably borrowed.
The trait that the unboxed closure implements is FnOnce
if any variables were moved out of the closure; otherwise FnMut
if there are any variables that are closed over and mutably borrowed; otherwise Fn
.
The ref
prefix for unboxed closures is removed, since it is now essentially implied.
We introduce a new grammar production, move ||
. The value returned by a move ||
implements FnOnce
, FnMut
, or Fn
, as determined above; thus, for example, move |x: int, y| x + y
produces an unboxed closure that implements the Fn(int, int) -> int
trait (and thus the FnOnce(int, int) -> int
trait by inheritance). Free variables referenced by a move ||
closure are always captured by value.
In the trait reference grammar, we will change the |&:|
sugar to Fn()
, the |&mut:|
sugar to FnMut()
, and the |:|
sugar to FnOnce()
. Thus what was before written fn foo<F:|&mut: int, int| -> int>()
will be fn foo<F:FnMut(int, int) -> int>()
.
It is important to note that the trait reference syntax and closure construction syntax are purposefully distinct. This is because either the ||
form or the move ||
form can construct any of FnOnce
, FnMut
, or Fn
closures.
Drawbacks
-
Having two syntaxes for closures could be seen as unfortunate.
-
move
becomes a keyword.
Alternatives
-
Keep the status quo:
|:|
/|&mut:
/|&:|
are the only ways to create unboxed closures, andref
must be used to get by-reference upvars. -
Use some syntax other than
move ||
for escaping closures. -
Keep the
|:|
/|&mut:
/|&:|
syntax only for trait reference sugar. -
Use
fn()
syntax for trait reference sugar.
Unresolved questions
There may be unforeseen complications in doing the inference.
- Start Date: 2014-09-16
- RFC PR #: https://github.com/rust-lang/rfcs/pull/234
- Rust Issue #: https://github.com/rust-lang/rust/issues/17323
Summary
Make enum variants part of both the type and value namespaces.
Motivation
We might, post-1.0, want to allow using enum variants as types. This would be backwards incompatible, because if a module already has a value with the same name as the variant in scope, then there will be a name clash.
Detailed design
Enum variants would always be part of both the type and value namespaces. Variants would not, however, be usable as types - we might want to allow this later, but it is out of scope for this RFC.
Data
Occurrences of name clashes in the Rust repo:
-
Key
inrustrt::local_data
-
InAddr
innative::io::net
-
Ast
inregex::parse
-
Class
inregex::parse
-
Native
inregex::re
-
Dynamic
inregex::re
-
Zero
innum::bigint
-
String
interm::terminfo::parm
-
String
inserialize::json
-
List
inserialize::json
-
Object
inserialize::json
-
Argument
infmt_macros
-
Metadata
inrustc_llvm
-
ObjectFile
inrustc_llvm
-
‘ItemDecorator’ in
syntax::ext::base
-
‘ItemModifier’ in
syntax::ext::base
-
FunctionDebugContext
inrustc::middle::trans::debuginfo
-
AutoDerefRef
inrustc::middle::ty
-
MethodParam
inrustc::middle::typeck
-
MethodObject
inrustc::middle::typeck
That’s a total of 20 in the compiler and libraries.
Drawbacks
Prevents the common-ish idiom of having a struct with the same name as a variant and then having a value of that struct be the variant’s data.
Alternatives
Don’t do it. That would prevent us making changes to the typed-ness of enums in the future. If we accept this RFC, but at some point we decide we never want to do anything with enum variants and types, we could always roll back this change backwards compatibly.
Unresolved questions
N/A
- Start Date: 2014-10-29
- RFC PR #: rust-lang/rfcs#235
- Rust Issue #: rust-lang/rust#18424
Summary
This is a combined conventions and library stabilization RFC. The goal is to
establish a set of naming and signature conventions for std::collections
.
The major components of the RFC include:
-
Removing most of the traits in
collections
. -
A general proposal for solving the “equiv” problem, as well as improving
MaybeOwned
. -
Patterns for overloading on by-need values and predicates.
-
Initial, forwards-compatible steps toward
Iterable
. -
A coherent set of API conventions across the full variety of collections.
A big thank-you to @Gankro, who helped collect API information and worked through an initial pass of some of the proposals here.
Motivation
This RFC aims to improve the design of the std::collections
module in
preparation for API stabilization. There are a number of problems that need to
be addressed, as spelled out in the subsections below.
Collection traits
The collections
module defines several traits:
- Collection
- Mutable
- MutableSeq
- Deque
- Map, MutableMap
- Set, MutableSet
There are several problems with the current trait design:
-
Most important: the traits do not provide iterator methods like
iter
. It is not possible to do so in a clean way without higher-kinded types, as the RFC explains in more detail below. -
The split between mutable and immutable traits is not well-motivated by any of the existing collections.
-
The methods defined in these traits are somewhat anemic compared to the suite of methods provided on the concrete collections that implement them.
Divergent APIs
Despite the current collection traits, the APIs of various concrete collections has diverged; there is not a globally coherent design, and there are many inconsistencies.
One problem in particular is the lack of clear guiding principles for the API design. This RFC proposes a few along the way.
Providing slice APIs on Vec
and String
The String
and Vec
types each provide a limited subset of the methods
provides on string and vector slices, but there is not a clear reason to limit
the API in this way. Today, one has to write things like
some_str.as_slice().contains(...)
, which is not ergonomic or intuitive.
The Equiv
problem
There is a more subtle problem related to slices. It’s common to use a HashMap
with owned String
keys, but then the natural API for things like lookup is not
very usable:
fn find(&self, k: &K) -> Option<&V>
The problem is that, since K
will be String
, the find
function requests a
&String
value – whereas one typically wants to work with the more flexible
&str
slices. In particular, using find
with a literal string requires
something like:
map.find(&"some literal".to_string())
which is unergonomic and requires an extra allocation just to get a borrow that, in some sense, was already available.
The current HashMap
API works around this problem by providing an additional
set of methods that uses a generic notion of “equivalence” of values that have
different types:
pub trait Equiv<T> {
fn equiv(&self, other: &T) -> bool;
}
impl Equiv<str> for String {
fn equiv(&self, other: &str) -> bool {
self.as_slice() == other
}
}
fn find_equiv<Q: Hash<S> + Equiv<K>>(&self, k: &Q) -> Option<&V>
There are a few downsides to this approach:
-
It requires a duplicated
_equiv
variant of each method taking a reference to the key. (This downside could likely be mitigated using multidispatch.) -
Its correctness depends on equivalent values producing the same hash, which is not checked.
-
String
-keyed hash maps are very common, so newcomers are likely to run headlong into the problem. First,find
will fail to work in the expected way. But the signature offind_equiv
is more difficult to understand thanfind
, and it it’s not immediately obvious that it solves the problem. -
It is the right API for
HashMap
, but not helpful for e.g.TreeMap
, which would want an analog forOrd
.
The TreeMap
API currently deals with this problem in an entirely different
way:
/// Returns the value for which f(key) returns Equal.
/// f is invoked with current key and guides tree navigation.
/// That means f should be aware of natural ordering of the tree.
fn find_with(&self, f: |&K| -> Ordering) -> Option<&V>
Besides being less convenient – you cannot write map.find_with("some literal")
–
this function navigates the tree according to an ordering that may have no
relationship to the actual ordering of the tree.
MaybeOwned
Sometimes a function does not know in advance whether it will need or produce an
owned copy of some data, or whether a borrow suffices. A typical example is the
from_utf8_lossy
function:
fn from_utf8_lossy<'a>(v: &'a [u8]) -> MaybeOwned<'a>
This function will return a string slice if the input was correctly utf8 encoded
– without any allocation. But if the input has invalid utf8 characters, the
function allocates a new String
and inserts utf8 “replacement characters”
instead. Hence, the return type is an enum
:
pub enum MaybeOwned<'a> {
Slice(&'a str),
Owned(String),
}
This interface makes it possible to allocate only when necessary, but the
MaybeOwned
type (and connected machinery) are somewhat ad hoc – and
specialized to String
/str
. It would be somewhat more palatable if there were
a single “maybe owned” abstraction usable across a wide range of types.
Iterable
A frequently-requested feature for the collections
module is an Iterable
trait for “values that can be iterated over”. There are two main motivations:
-
Abstraction. Today, you can write a function that takes a single
Iterator
, but you cannot write a function that takes a container and then iterates over it multiple times (perhaps with differing mutability levels). AnIterable
trait could allow that. -
Ergonomics. You’d be able to write
for v in some_vec { ... }
rather than
for v in some_vec.iter() { ... }
and
consume_iter(some_vec)
rather thanconsume_iter(some_vec.iter())
.
Detailed design
The collections today
The concrete collections currently available in std
fall into roughly three categories:
-
Sequences
- Vec
- String
- Slices
- Bitv
- DList
- RingBuf
- PriorityQueue
-
Sets
- HashSet
- TreeSet
- TrieSet
- EnumSet
- BitvSet
-
Maps
- HashMap
- TreeMap
- TrieMap
- LruCache
- SmallIntMap
The primary goal of this RFC is to establish clean and consistent APIs that apply across each group of collections.
Before diving into the details, there is one high-level changes that should be
made to these collections. The PriorityQueue
collection should be renamed to
BinaryHeap
, following the convention that concrete collections are named according
to their implementation strategy, not the abstract semantics they implement. We
may eventually want PriorityQueue
to be a trait that’s implemented by
multiple concrete collections.
The LruCache
could be renamed for a similar reason (it uses a HashMap
in its
implementation), However, the implementation is actually generic with respect to
this underlying map, and so in the long run (with HKT and other language
changes) LruCache
should probably add a type parameter for the underlying map,
defaulted to HashMap
.
Design principles
-
Centering on
Iterator
s. TheIterator
trait is a strength of Rust’s collections library. Because so many APIs can produce iterators, adding an API that consumes one is very powerful – and conversely as well. Moreover, iterators are highly efficient, since you can chain several layers of modification without having to materialize intermediate results. Thus, whenever possible, collection APIs should strive to work with iterators.In particular, some existing convenience methods avoid iterators for either performance or ergonomic reasons. We should instead improve the ergonomics and performance of iterators, so that these extra convenience methods are not necessary and so that all collections can benefit.
-
Minimizing method variants. One problem with some of the current collection APIs is the proliferation of method variants. For example,
HashMap
include seven methods that begin with the namefind
! While each method has a motivation, the API as a whole can be bewildering, especially to newcomers.When possible, we should leverage the trait system, or find other abstractions, to reduce the need for method variants while retaining their ergonomics and power.
-
Conservatism. It is easier to add APIs than to take them away. This RFC takes a fairly conservative stance on what should be included in the collections APIs. In general, APIs should be very clearly motivated by a wide variety of use cases, either for expressiveness, performance, or ergonomics.
Removing the traits
This RFC proposes a somewhat radical step for the collections traits: rather than reform them, we should eliminate them altogether – for now.
Unlike inherent methods, which can easily be added and deprecated over time, a trait is “forever”: there are very few backwards-compatible modifications to traits. Thus, for something as fundamental as collections, it is prudent to take our time to get the traits right.
Lack of iterator methods
In particular, there is one way in which the current traits are clearly wrong:
they do not provide standard methods like iter
, despite these being
fundamental to working with collections in Rust. Sadly, this gap is due to
inexpressiveness in the language, which makes directly defining iterator methods
in a trait impossible:
trait Iter {
type A;
type I: Iterator<&'a A>; // what is the lifetime here?
fn iter<'a>(&'a self) -> I; // and how to connect it to self?
}
The problem is that, when implementing this trait, the return type I
of iter
should depend on the lifetime of self. For example, the corresponding
method in Vec
looks like the following:
impl<T> Vec<T> {
fn iter(&'a self) -> Items<'a, T> { ... }
}
This means that, given a Vec<T>
, there isn’t a single type Items<T>
for
iteration – rather, there is a family of types, one for each input lifetime.
In other words, the associated type I
in the Iter
needs to be
“higher-kinded”: not just a single type, but rather a family:
trait Iter {
type A;
type I<'a>: Iterator<&'a A>;
fn iter<'a>(&self) -> I<'a>;
}
In this case, I
is parameterized by a lifetime, but in other cases (like
map
) an associated type needs to be parameterized by a type.
In general, such higher-kinded types (HKTs) are a much-requested feature for Rust. But the design and implementation of higher-kinded types is, by itself, a significant investment.
HKT would also allow for parameterization over smart pointer types, which has many potential use cases in the context of collections.
Thus, the goal in this RFC is to do the best we can without HKT for now, while allowing a graceful migration if or when HKT is added.
Persistent/immutable collections
Another problem with the current collection traits is the split between immutable and mutable versions. In the long run, we will probably want to provide persistent collections (which allow non-destructive “updates” that create new collections that share most of their data with the old ones).
However, persistent collection APIs have not been thoroughly explored in Rust; it would be hasty to standardize on a set of traits until we have more experience.
Downsides of removal
There are two main downsides to removing the traits without a replacement:
-
It becomes impossible to write code using generics over a “kind” of collection (like
Map
). -
It becomes more difficult to ensure that the collections share a common API.
For point (1), first, if the APIs are sufficiently consistent it should be
possible to transition code from e.g. a TreeMap
to a HashMap
by changing
very few lines of code. Second, generic programming is currently quite limited,
given the inability to iterate. Finally, generic programming over collections is
a large design space (with much precedent in C++, for example), and we should
take our time and gain more experience with a variety of concrete collections
before settling on a design.
For point (2), first, the current traits have failed to keep the APIs in line, as we will see below. Second, this RFC is the antidote: we establish a clear set of conventions and APIs for concrete collections up front, and stabilize on those, which should make it easy to add traits later on.
Why not leave the traits as “experimental”?
An alternative to removal would be to leave the traits intact, but marked as experimental, with the intent to radically change them later.
Such a strategy doesn’t buy much relative to removal (given the arguments above), but risks the traits becoming “de facto” stable if people begin using them en masse.
Solving the _equiv
and MaybeOwned
problems
The basic problem that leads to _equiv
methods is that:
&String
and&str
are not the same type.- The
&str
type is more flexible and hence more widely used. - Code written for a generic type
T
that takes a reference&T
will therefore not be suitable whenT
is instantiated withString
.
A similar story plays out for &Vec<T>
and &[T]
, and with DST and custom
slice types the same problem will arise elsewhere.
The Borrow
trait
This RFC proposes to use a trait, Borrow
to connect borrowed and owned data
in a generic fashion:
/// A trait for borrowing.
trait Borrow<Sized? B> {
/// Immutably borrow from an owned value.
fn borrow(&self) -> &B;
/// Mutably borrow from an owned value.
fn borrow_mut(&mut self) -> &mut B;
}
// The Sized bound means that this impl does not overlap with the impls below.
impl<T: Sized> Borrow<T> for T {
fn borrow(a: &T) -> &T {
a
}
fn borrow_mut(a: &mut T) -> &mut T {
a
}
}
impl Borrow<str> for String {
fn borrow(s: &String) -> &str {
s.as_slice()
}
fn borrow_mut(s: &mut String) -> &mut str {
s.as_mut_slice()
}
}
impl<T> Borrow<[T]> for Vec<T> {
fn borrow(s: &Vec<T>) -> &[T] {
s.as_slice()
}
fn borrow_mut(s: &mut Vec<T>) -> &mut [T] {
s.as_mut_slice()
}
}
(Note: thanks to @epdtry for suggesting this variation! The original proposal is listed in the Alternatives.)
A primary goal of the design is allowing a blanket impl
for non-sliceable
types (the first impl
above). This blanket impl
ensures that all new sized,
cloneable types are automatically borrowable; new impl
s are required only for
new unsized types, which are rare. The Sized
bound on the initial impl means
that we can freely add impls for unsized types (like str
and [T]
) without
running afoul of coherence.
Because of the blanket impl
, the Borrow
trait can largely be ignored except
when it is actually used – which we describe next.
Using Borrow
to replace _equiv
methods
With the Borrow
trait in place, we can eliminate the _equiv
method variants
by asking map keys to be Borrow
:
impl<K,V> HashMap<K,V> where K: Hash + Eq {
fn find<Q>(&self, k: &Q) -> &V where K: Borrow<Q>, Q: Hash + Eq { ... }
fn contains_key<Q>(&self, k: &Q) -> bool where K: Borrow<Q>, Q: Hash + Eq { ... }
fn insert(&mut self, k: K, v: V) -> Option<V> { ... }
...
}
The benefits of this approach over _equiv
are:
-
The
Borrow
trait captures the borrowing relationship between an owned data structure and both references to it and slices from it – once and for all. This means that it can be used anywhere we need to program generically over “borrowed” data. In particular, the single trait works for bothHashMap
andTreeMap
, and should work for other kinds of data structures as well. It also helps generalizeMaybeOwned
, for similar reasons (see below.)A very important consequence is that the map methods using
Borrow
can potentially be put into a commonMap
trait that’s implemented byHashMap
,TreeMap
, and others. While we do not propose to do so now, we definitely want to do so later on. -
When using a
HashMap<String, T>
, all of the basic methods likefind
,contains_key
andinsert
“just work”, without forcing you to think about&String
vs&str
. -
We don’t need separate
_equiv
variants of methods. (However, this could probably be addressed with multidispatch by providing a blanketEquiv
implementation.)
On the other hand, this approach retains some of the downsides of _equiv
:
-
The signature for methods like
find
andcontains_key
is more complex than their current signatures. There are two counterpoints. First, over time theBorrow
trait is likely to become a well-known concept, so the signature will not appear completely alien. Second, what is perhaps more important than the signature is that, when usingfind
onHashMap<String, T>
, various method arguments just work as expected. -
The API does not guarantee “coherence”: the
Hash
andEq
(orOrd
, forTreeMap
) implementations for the owned and borrowed keys might differ, breaking key invariants of the data structure. This is already the case with_equiv
.
The Alternatives section includes a variant of Borrow
that doesn’t suffer from these downsides, but has some downsides of its own.
Clone-on-write (Cow
) pointers
A side-benefit of the Borrow
trait is that we can give a more general version
of the MaybeOwned
as a “clone-on-write” smart pointer:
/// A generalization of Clone.
trait FromBorrow<Sized? B>: Borrow<B> {
fn from_borrow(b: &B) -> Self;
}
/// A clone-on-write smart pointer
pub enum Cow<'a, T, B> where T: FromBorrow<B> {
Shared(&'a B),
Owned(T)
}
impl<'a, T, B> Cow<'a, T, B> where T: FromBorrow<B> {
pub fn new(shared: &'a B) -> Cow<'a, T, B> {
Shared(shared)
}
pub fn new_owned(owned: T) -> Cow<'static, T, B> {
Owned(owned)
}
pub fn is_owned(&self) -> bool {
match *self {
Owned(_) => true,
Shared(_) => false
}
}
pub fn to_owned_mut(&mut self) -> &mut T {
match *self {
Shared(shared) => {
*self = Owned(FromBorrow::from_borrow(shared));
self.to_owned_mut()
}
Owned(ref mut owned) => owned
}
}
pub fn into_owned(self) -> T {
match self {
Shared(shared) => FromBorrow::from_borrow(shared),
Owned(owned) => owned
}
}
}
impl<'a, T, B> Deref<B> for Cow<'a, T, B> where T: FromBorrow<B> {
fn deref(&self) -> &B {
match *self {
Shared(shared) => shared,
Owned(ref owned) => owned.borrow()
}
}
}
impl<'a, T, B> DerefMut<B> for Cow<'a, T, B> where T: FromBorrow<B> {
fn deref_mut(&mut self) -> &mut B {
self.to_owned_mut().borrow_mut()
}
}
The type Cow<'a, String, str>
is roughly equivalent to today’s MaybeOwned<'a>
(and Cow<'a, Vec<T>, [T]>
to MaybeOwnedVector<'a, T>
).
By implementing Deref
and DerefMut
, the Cow
type acts as a smart pointer
– but in particular, the mut
variant actually clones if the pointed-to
value is not currently owned. Hence “clone on write”.
One slight gotcha with the design is that &mut str
is not very useful, while
&mut String
is (since it allows extending the string, for example). On the
other hand, Deref
and DerefMut
must deref to the same underlying type, and
for Deref
to not require cloning, it must yield a &str
value.
Thus, the Cow
pointer offers a separate to_owned_mut
method that yields a
mutable reference to the owned version of the type.
Note that, by not using into_owned
, the Cow
pointer itself may be owned by
some other data structure (perhaps as part of a collection) and will internally
track whether an owned copy is available.
Altogether, this RFC proposes to introduce Borrow
and Cow
as above, and to
deprecate MaybeOwned
and MaybeOwnedVector
. The API changes for the
collections are discussed below.
IntoIterator
(and Iterable
)
As discussed in earlier, some form of an Iterable
trait is
desirable for both expressiveness and ergonomics. Unfortunately, a full
treatment of Iterable
requires HKT for similar reasons to
the collection traits. However, it’s possible to
get some of the way there in a forwards-compatible fashion.
In particular, the following two traits work fine (with associated items):
trait Iterator {
type A;
fn next(&mut self) -> Option<A>;
...
}
trait IntoIterator {
type A;
type I: Iterator<A = A>;
fn into_iter(self) -> I;
}
Because IntoIterator
consumes self
, lifetimes are not an issue.
It’s tempting to also define a trait like:
trait Iterable<'a> {
type A;
type I: Iterator<&'a A>;
fn iter(&'a self) -> I;
}
(along the lines of those proposed by an earlier RFC).
The problem with Iterable
as defined above is that it’s locked to a particular
lifetime up front. But in many cases, the needed lifetime is not even nameable
in advance:
fn iter_through_rc<I>(c: Rc<I>) where I: Iterable<?> {
// the lifetime of the borrow is established here,
// so cannot even be named in the function signature
for x in c.iter() {
// ...
}
}
To make this kind of example work, you’d need to be able to say something like:
where <'a> I: Iterable<'a>
that is, that I
implements Iterable
for every lifetime 'a
. While such a
feature is feasible to add to where
clauses, the HKT solution is undoubtedly
cleaner.
Fortunately, we can have our cake and eat it too. This RFC proposes the
IntoIterator
trait above, together with the following blanket impl
:
impl<I: Iterator> IntoIterator for I {
type A = I::A;
type I = I;
fn into_iter(self) -> I {
self
}
}
which means that taking IntoIterator
is strictly more flexible than taking
Iterator
. Note that in other languages (like Java), iterators are not
iterable because the latter implies an unlimited number of iterations. But
because IntoIterator
consumes self
, it yields only a single iteration, so
all is good.
For individual collections, one can then implement IntoIterator
on both the
collection and borrows of it:
impl<T> IntoIterator for Vec<T> {
type A = T;
type I = MoveItems<T>;
fn into_iter(self) -> MoveItems<T> { ... }
}
impl<'a, T> IntoIterator for &'a Vec<T> {
type A = &'a T;
type I = Items<'a, T>;
fn into_iter(self) -> Items<'a, T> { ... }
}
impl<'a, T> IntoIterator for &'a mut Vec<T> {
type A = &'a mut T;
type I = ItemsMut<'a, T>;
fn into_iter(self) -> ItemsMut<'a, T> { ... }
}
If/when HKT is added later on, we can add an Iterable
trait and a blanket
impl
like the following:
// the HKT version
trait Iterable {
type A;
type I<'a>: Iterator<&'a A>;
fn iter<'a>(&'a self) -> I<'a>;
}
impl<'a, C: Iterable> IntoIterator for &'a C {
type A = &'a C::A;
type I = C::I<'a>;
fn into_iter(self) -> I {
self.iter()
}
}
This gives a clean migration path: once Vec
implements Iterable
, it can drop
the IntoIterator
impl
s for borrowed vectors, since they will be covered by
the blanket implementation. No code should break.
Likewise, if we add a feature like the “universal” where
clause mentioned
above, it can be used to deal with embedded lifetimes as in the
iter_through_rc
example; and if the HKT version of Iterable
is later added,
thanks to the suggested blanket impl
for IntoIterator
that where
clause
could be changed to use Iterable
instead, again without breakage.
Benefits of IntoIterator
What do we gain by incorporating IntoIterator
today?
This RFC proposes that for
loops should use IntoIterator
rather than
Iterator
. With the blanket impl
of IntoIterator
for any Iterator
, this
is not a breaking change. However, given the IntoIterator
impl
s for Vec
above, we would be able to write:
let v: Vec<Foo> = ...
for x in &v { ... } // iterate over &Foo
for x in &mut v { ... } // iterate over &mut Foo
for x in v { ... } // iterate over Foo
Similarly, methods that currently take slices or iterators can be changed to
take IntoIterator
instead, immediately becoming more general and more
ergonomic.
In general, IntoIterator
will allow us to move toward more Iterator
-centric
APIs today, in a way that’s compatible with HKT tomorrow.
Additional methods
Another typical desire for an Iterable
trait is to offer defaulted versions of
methods that basically re-export iterator methods on containers (see
the earlier RFC). Usually these
methods would go through a reference iterator (i.e. the iter
method) rather
than a moving iterator.
It is possible to add such methods using the design proposed above, but there
are some drawbacks. For example, should Vec::map
produce an iterator, or a new
vector? It would be possible to do the latter generically, but only with
HKT. (See
this discussion.)
This RFC only proposes to add the following method via IntoIterator
, as a
convenience for a common pattern:
trait IterCloned {
type A;
type I: Iterator<A>;
fn iter_cloned(self) -> I;
}
impl<'a, T, I: IntoIterator> IterCloned for I where I::A = &'a T {
type A = T;
type I = ClonedItems<I>;
fn into_iter(self) -> I { ... }
}
(The iter_cloned
method will help reduce the number of method variants in
general for collections, as we will see below).
We will leave to later RFCs the incorporation of additional methods. Notice, in
particular, that such methods can wait until we introduce an Iterable
trait
via HKT without breaking backwards compatibility.
Minimizing variants: ByNeed
and Predicate
traits
There are several kinds of methods that, in their most general form take closures, but for which convenience variants taking simpler data are common:
-
Taking values by need. For example, consider the
unwrap_or
andunwrap_or_else
methods inOption
:fn unwrap_or(self, def: T) -> T fn unwrap_or_else(self, f: || -> T) -> T
The
unwrap_or_else
method is the most general: it invokes the closure to compute a default value only whenself
isNone
. When the default value is expensive to compute, this by-need approach helps. But often the default value is cheap, and closures are somewhat annoying to write, sounwrap_or
provides a convenience wrapper. -
Taking predicates. For example, a method like
contains
often shows up (inconsistently!) in two variants:fn contains(&self, elem: &T) -> bool; // where T: PartialEq fn contains_fn(&self, pred: |&T| -> bool) -> bool;
Again, the
contains_fn
version is the more general, but it’s convenient to provide a specialized variant when the element type can be compared for equality, to avoid writing explicit closures.
As it turns out, with multidispatch) it is possible to use a trait to express these variants through overloading:
trait ByNeed<T> {
fn compute(self) -> T;
}
impl<T> ByNeed<T> for T {
fn compute(self) -> T {
self
}
}
// Due to multidispatch, this impl does NOT overlap with the above one
impl<T> ByNeed<T> for || -> T {
fn compute(self) -> T {
self()
}
}
impl<T> Option<T> {
fn unwrap_or<U>(self, def: U) where U: ByNeed<T> { ... }
...
}
trait Predicate<T> {
fn check(&self, &T) -> bool;
}
impl<T: Eq> Predicate<T> for &T {
fn check(&self, t: &T) -> bool {
*self == t
}
}
impl<T> Predicate<T> for |&T| -> bool {
fn check(&self, t: &T) -> bool {
(*self)(t)
}
}
impl<T> Vec<T> {
fn contains<P>(&self, pred: P) where P: Predicate<T> { ... }
...
}
Since these two patterns are particularly common throughout std
, this RFC
proposes adding both of the above traits, and using them to cut down on the
number of method variants.
In particular, some methods on string slices currently work with CharEq
, which
is similar to Predicate<char>
:
pub trait CharEq {
fn matches(&mut self, char) -> bool;
fn only_ascii(&self) -> bool;
}
The difference is the only_ascii
method, which is used to optimize certain
operations when the predicate only holds for characters in the ASCII range.
To keep these optimizations intact while connecting to Predicate
, this RFC
proposes the following restructuring of CharEq
:
pub trait CharPredicate: Predicate<char> {
fn only_ascii(&self) -> bool {
false
}
}
Why not leverage unboxed closures?
A natural question is: why not use the traits for unboxed closures to achieve a
similar effect? For example, you could imagine writing a blanket impl
for
Fn(&T) -> bool
for any T: PartialEq
, which would allow PartialEq
values to
be used anywhere a predicate-like closure was requested.
The problem is that these blanket impl
s will often conflict. In particular,
any type T
could implement Fn() -> T
, and that single blanket impl
would
preclude any others (at least, assuming that unboxed closure traits treat the
argument and return types as associated (output) types).
In addition, the explicit use of traits like Predicate
makes the intended
semantics more clear, and the overloading less surprising.
The APIs
Now we’ll delve into the detailed APIs for the various concrete collections. These APIs will often be given in tabular form, grouping together common APIs across multiple collections. When writing these function signatures:
-
We will assume a type parameter
T
forVec
,BinaryHeap
,DList
andRingBuf
; we will also use this parameter for APIs onString
, where it should be understood aschar
. -
We will assume type parameters
K: Borrow
andV
forHashMap
andTreeMap
; forTrieMap
andSmallIntMap
theK
is assumed to beuint
-
We will assume a type parameter
K: Borrow
forHashSet
andTreeSet
; forBitvSet
it is assumed to beuint
.
We will begin by outlining the most widespread APIs in tables, making it easy to compare names and signatures across different kinds of collections. Then we will focus on some APIs specific to particular classes of collections – e.g. sets and maps. Finally, we will briefly discuss APIs that are specific to a single concrete collection.
Construction
All of the collections should support a static function:
fn new() -> Self
that creates an empty version of the collection; the constructor may take
arguments needed to set up the collection, e.g. the capacity for LruCache
.
Several collections also support separate constructors for providing capacities in advance; these are discussed below.
The FromIterator
trait
All of the collections should implement the FromIterator
trait:
pub trait FromIterator {
type A:
fn from_iter<T>(T) -> Self where T: IntoIterator<A = A>;
}
Note that this varies from today’s FromIterator
by consuming an IntoIterator
rather than Iterator
. As explained above, this
choice is strictly more general and will not break any existing code.
This constructor initializes the collection with the contents of the iterator. For maps, the iterator is over key/value pairs, and the semantics is equivalent to inserting those pairs in order; if keys are repeated, the last value is the one left in the map.
Insertion
The table below gives methods for inserting items into various concrete collections:
Operation | Collections |
---|---|
fn push(&mut self, T) | Vec , BinaryHeap , String |
fn push_front(&mut self, T) | DList , RingBuf |
fn push_back(&mut self, T) | DList , RingBuf |
fn insert(&mut self, uint, T) | Vec , RingBuf , String |
fn insert(&mut self, K::Owned) -> bool | HashSet , TreeSet , TrieSet , BitvSet |
fn insert(&mut self, K::Owned, V) -> Option<V> | HashMap , TreeMap , TrieMap , SmallIntMap |
fn append(&mut self, Self) | DList |
fn prepend(&mut self, Self) | DList |
There are a few changes here from the current state of affairs:
-
The
DList
andRingBuf
data structures no longer providepush
, but ratherpush_front
andpush_back
. This change is based on (1) viewing them as deques and (2) not giving priority to the “front” or the “back”. -
The
insert
method on maps returns the value previously associated with the key, if any. Previously, this functionality was provided by aswap
method, which has been dropped (consolidating needless method variants.)
Aside from these changes, a number of insertion methods will be deprecated
(e.g. the append
and append_one
methods on Vec
). These are discussed
further in the section on “specialized operations”
below.
The Extend
trait (was: Extendable
)
In addition to the standard insertion operations above, all collections will
implement the Extend
trait. This trait was previously called Extendable
, but
in general we
prefer to avoid -able
suffixes and instead name the trait using a verb (or, especially, the key method
offered by the trait.)
The Extend
trait allows data from an arbitrary iterator to be inserted into a
collection, and will be defined as follows:
pub trait Extend: FromIterator {
fn extend<T>(&mut self, T) where T: IntoIterator<A = Self::A>;
}
As with FromIterator
, this trait has been modified to take an IntoIterator
value.
Deletion
The table below gives methods for removing items into various concrete collections:
Operation | Collections |
---|---|
fn clear(&mut self) | all |
fn pop(&mut self) -> Option<T> | Vec , BinaryHeap , String |
fn pop_front(&mut self) -> Option<T> | DList , RingBuf |
fn pop_back(&mut self) -> Option<T> | DList , RingBuf |
fn remove(&mut self, uint) -> Option<T> | Vec , RingBuf , String |
fn remove(&mut self, &K) -> bool | HashSet , TreeSet , TrieSet , BitvSet |
fn remove(&mut self, &K) -> Option<V> | HashMap , TreeMap , TrieMap , SmallIntMap |
fn truncate(&mut self, len: uint) | Vec , String , Bitv , DList , RingBuf |
fn retain<P>(&mut self, f: P) where P: Predicate<T> | Vec , DList , RingBuf |
fn dedup(&mut self) | Vec , DList , RingBuf where T: PartialEq |
As with the insertion methods, there are some differences from today’s API:
-
The
DList
andRingBuf
data structures no longer providepop
, but ratherpop_front
andpop_back
– similarly to thepush
methods. -
The
remove
method on maps returns the value previously associated with the key, if any. Previously, this functionality was provided by a separatepop
method, which has been dropped (consolidating needless method variants.) -
The
retain
method takes aPredicate
. -
The
truncate
,retain
anddedup
methods are offered more widely.
Again, some of the more specialized methods are not discussed here; see “specialized operations” below.
Inspection/mutation
The next table gives methods for inspection and mutation of existing items in collections:
Operation | Collections |
---|---|
fn len(&self) -> uint | all |
fn is_empty(&self) -> bool | all |
fn get(&self, uint) -> Option<&T> | [T] , Vec , RingBuf |
fn get_mut(&mut self, uint) -> Option<&mut T> | [T] , Vec , RingBuf |
fn get(&self, &K) -> Option<&V> | HashMap , TreeMap , TrieMap , SmallIntMap |
fn get_mut(&mut self, &K) -> Option<&mut V> | HashMap , TreeMap , TrieMap , SmallIntMap |
fn contains<P>(&self, P) where P: Predicate<T> | [T] , str , Vec , String , DList , RingBuf , BinaryHeap |
fn contains(&self, &K) -> bool | HashSet , TreeSet , TrieSet , EnumSet |
fn contains_key(&self, &K) -> bool | HashMap , TreeMap , TrieMap , SmallIntMap |
The biggest changes from the current APIs are:
-
The
find
andfind_mut
methods have been renamed toget
andget_mut
. Further, allget
methods returnOption
values and do not invokefail!
. This is part of a general convention described in the next section (on theIndex
traits). -
The
contains
method is offered more widely. -
There is no longer an equivalent of
find_copy
(which should be calledfind_clone
). Instead, we propose to add the following method to theOption<&'a T>
type whereT: Clone
:fn cloned(self) -> Option<T> { self.map(|x| x.clone()) }
so that
some_map.find_copy(key)
will instead be writtensome_map.find(key).cloned()
. This method chain is slightly longer, but is more clear and allows us to drop the_copy
variants. Moreover, all users ofOption
benefit from the new convenience method.
The Index
trait
The Index
and IndexMut
traits provide indexing notation like v[0]
:
pub trait Index {
type Index;
type Result;
fn index(&'a self, index: &Index) -> &'a Result;
}
pub trait IndexMut {
type Index;
type Result;
fn index_mut(&'a mut self, index: &Index) -> &'a mut Result;
}
These traits will be implemented for: [T]
, Vec
, RingBuf
, HashMap
, TreeMap
, TrieMap
, SmallIntMap
.
As a general convention, implementation of the Index
traits will fail the
task if the index is invalid (out of bounds or key not found); they will
therefore return direct references to values. Any collection implementing Index
(resp. IndexMut
) should also provide a get
method (resp. get_mut
) as a
non-failing variant that returns an Option
value.
This allows us to keep indexing notation maximally concise, while still providing convenient non-failing variants (which can be used to provide a check for index validity).
Iteration
Every collection should provide the standard trio of iteration methods:
fn iter(&'a self) -> Items<'a>;
fn iter_mut(&'a mut self) -> ItemsMut<'a>;
fn into_iter(self) -> ItemsMove;
and in particular implement the IntoIterator
trait on both the collection type
and on (mutable) references to it.
Capacity management
many of the collections have some notion of “capacity”, which may be fixed, grow explicitly, or grow implicitly:
- No capacity/fixed capacity:
DList
,TreeMap
,TreeSet
,TrieMap
,TrieSet
, slices,EnumSet
- Explicit growth:
LruCache
- Implicit growth:
Vec
,RingBuf
,HashMap
,HashSet
,BitvSet
,BinaryHeap
Growable collections provide functions for capacity management, as follows.
Explicit growth
For explicitly-grown collections, the normal constructor (new
) takes a
capacity argument. Capacity can later be inspected or updated as follows:
fn capacity(&self) -> uint
fn set_capacity(&mut self, capacity: uint)
(Note, this renames LruCache::change_capacity
to set_capacity
, the
prevailing style for setter method.)
Implicit growth
For implicitly-grown collections, the normal constructor (new
) does not take a
capacity, but there is an explicit with_capacity
constructor, along with other
functions to work with the capacity later on:
fn with_capacity(uint) -> Self
fn capacity(&self) -> uint
fn reserve(&mut self, additional: uint)
fn reserve_exact(&mut self, additional: uint)
fn shrink_to_fit(&mut self)
There are some important changes from the current APIs:
-
The
reserve
andreserve_exact
methods now take as an argument the extra space to reserve, rather than the final desired capacity, as this usage is vastly more common. Thereserve
function may grow the capacity by a larger amount than requested, to ensure amortization, whilereserve_exact
will reserve exactly the requested additional capacity. Thereserve_additional
methods are deprecated. -
The
with_capacity
constructor does not take any additional arguments, for uniformity withnew
. This change affectsBitv
in particular.
Bounded iterators
Some of the maps (e.g. TreeMap
) currently offer specialized iterators over
their entries starting at a given key (called lower_bound
) and above a given
key (called upper_bound
), along with _mut
variants. While the functionality
is worthwhile, the names are not very clear, so this RFC proposes the following
replacement API (thanks to @Gankro for the suggestion):
Bound<T> {
/// An inclusive bound
Included(T),
/// An exclusive bound
Excluded(T),
Unbounded,
}
/// Creates a double-ended iterator over a sub-range of the collection's items,
/// starting at min, and ending at max. If min is `Unbounded`, then it will
/// be treated as "negative infinity", and if max is `Unbounded`, then it will
/// be treated as "positive infinity". Thus range(Unbounded, Unbounded) will yield
/// the whole collection.
fn range(&self, min: Bound<&T>, max: Bound<&T>) -> RangedItems<'a, T>;
fn range_mut(&self, min: Bound<&T>, max: Bound<&T>) -> RangedItemsMut<'a, T>;
These iterators should be provided for any maps over ordered keys (TreeMap
,
TrieMap
and SmallIntMap
).
In addition, analogous methods should be provided for sets over ordered keys
(TreeSet
, TrieSet
, BitvSet
).
Set operations
Comparisons
All sets should offer the following methods, as they do today:
fn is_disjoint(&self, other: &Self) -> bool;
fn is_subset(&self, other: &Self) -> bool;
fn is_superset(&self, other: &Self) -> bool;
Combinations
Sets can also be combined using the standard operations – union, intersection, difference and symmetric difference (exclusive or). Today’s APIs for doing so look like this:
fn union<'a>(&'a self, other: &'a Self) -> I;
fn intersection<'a>(&'a self, other: &'a Self) -> I;
fn difference<'a>(&'a self, other: &'a Self) -> I;
fn symmetric_difference<'a>(&'a self, other: &'a Self) -> I;
where the I
type is an iterator over keys that varies by concrete
set. Working with these iterators avoids materializing intermediate
sets when they’re not needed; the collect
method can be used to
create sets when they are. This RFC proposes to keep these names
intact, following the
RFC on iterator
conventions.
Sets should also implement the BitOr
, BitAnd
, BitXor
and Sub
traits from
std::ops
, allowing overloaded notation |
, &
, |^
and -
to be used with
sets. These are equivalent to invoking the corresponding iter_
method and then
calling collect
, but for some sets (notably BitvSet
) a more efficient direct
implementation is possible.
Unfortunately, we do not yet have a set of traits corresponding to operations
|=
, &=
, etc, but again in some cases doing the update in place may be more
efficient. Right now, BitvSet
is the only concrete set offering such operations:
fn union_with(&mut self, other: &BitvSet)
fn intersect_with(&mut self, other: &BitvSet)
fn difference_with(&mut self, other: &BitvSet)
fn symmetric_difference_with(&mut self, other: &BitvSet)
This RFC punts on the question of naming here: it does not propose a new set
of names. Ideally, we would add operations like |=
in a separate RFC, and use
those conventionally for sets. If not, we will choose fallback names during the
stabilization of BitvSet
.
Map operations
Combined methods
The HashMap
type currently provides a somewhat bewildering set of find
/insert
variants:
fn find_or_insert(&mut self, k: K, v: V) -> &mut V
fn find_or_insert_with<'a>(&'a mut self, k: K, f: |&K| -> V) -> &'a mut V
fn insert_or_update_with<'a>(&'a mut self, k: K, v: V, f: |&K, &mut V|) -> &'a mut V
fn find_with_or_insert_with<'a, A>(&'a mut self, k: K, a: A, found: |&K, &mut V, A|, not_found: |&K, A| -> V) -> &'a mut V
These methods are used to couple together lookup and insertion/update operations, thereby avoiding an extra lookup step. However, the current set of method variants seems overly complex.
There is another RFC already in the queue addressing this problem in a very nice way, and this RFC defers to that one
Key and value iterators
In addition to the standard iterators, maps should provide by-reference convenience iterators over keys and values:
fn keys(&'a self) -> Keys<'a, K>
fn values(&'a self) -> Values<'a, V>
While these iterators are easy to define in terms of the main iter
method,
they are used often enough to warrant including convenience methods.
Specialized operations
Many concrete collections offer specialized operations beyond the ones given above. These will largely be addressed through the API stabilization process (which focuses on local API issues, as opposed to general conventions), but a few broad points are addressed below.
Relating Vec
and String
to slices
One goal of this RFC is to supply all of the methods on (mutable) slices on
Vec
and String
. There are a few ways to achieve this, so concretely the
proposal is for Vec<T>
to implement Deref<[T]>
and DerefMut<[T]>
, and
String
to implement Deref<str>
. This will automatically allow all slice
methods to be invoked from vectors and strings, and will allow writing &*v
rather than v.as_slice()
.
In this scheme, Vec
and String
are really “smart pointers” around the
corresponding slice types. While counterintuitive at first, this perspective
actually makes a fair amount of sense, especially with DST.
(Initially, it was unclear whether this strategy would play well with method resolution, but the planned resolution rules should work fine.)
String
API
One of the key difficulties with the String
API is that strings use utf8
encoding, and some operations are only efficient when working at the byte level
(and thus taking this encoding into account).
As a general principle, we will move the API toward the following convention: index-related operations always work in terms of bytes, other operations deal with chars by default (but can have suffixed variants for working at other granularities when appropriate.)
DList
The DList
type offers a number of specialized methods:
swap_remove, insert_when, insert_ordered, merge, rotate_forward and rotate_backward
Prior to stabilizing the DList
API, we will attempt to simplify its API
surface, possibly by using idea from the
collection views RFC.
Minimizing method variants via iterators
Partitioning via FromIterator
One place we can move toward iterators is functions like partition
and
partitioned
on vectors and slices:
// on Vec<T>
fn partition(self, f: |&T| -> bool) -> (Vec<T>, Vec<T>);
// on [T] where T: Clone
fn partitioned(&self, f: |&T| -> bool) -> (Vec<T>, Vec<T>);
These two functions transform a vector/slice into a pair of vectors, based on a
“partitioning” function that says which of the two vectors to place elements
into. The partition
variant works by moving elements of the vector, while
partitioned
clones elements.
There are a few unfortunate aspects of an API like this one:
-
It’s specific to vectors/slices, although in principle both the source and target containers could be more general.
-
The fact that two variants have to be exposed, for owned versus clones, is somewhat unfortunate.
This RFC proposes the following alternative design:
pub enum Either<T, U> {
pub Left(T),
pub Right(U),
}
impl<A, B> FromIterator for (A, B) where A: Extend, B: Extend {
fn from_iter<I>(mut iter: I) -> (A, B) where I: IntoIterator<Either<T, U>> {
let mut left: A = FromIterator::from_iter(None::<T>);
let mut right: B = FromIterator::from_iter(None::<U>);
for item in iter {
match item {
Left(t) => left.extend(Some(t)),
Right(u) => right.extend(Some(u)),
}
}
(left, right)
}
}
trait Iterator {
...
fn partition(self, |&A| -> bool) -> Partitioned<A> { ... }
}
// where Partitioned<A>: Iterator<A = Either<A, A>>
This design drastically generalizes the partitioning functionality, allowing it be used with arbitrary collections and iterators, while removing the by-reference and by-value distinction.
Using this design, you have:
// The following two lines are equivalent:
let (u, w) = v.partition(f);
let (u, w): (Vec<T>, Vec<T>) = v.into_iter().partition(f).collect();
// The following two lines are equivalent:
let (u, w) = v.as_slice().partitioned(f);
let (u, w): (Vec<T>, Vec<T>) = v.iter_cloned().partition(f).collect();
There is some extra verbosity, mainly due to the type annotations for collect
,
but the API is much more flexible, since the partitioned data can now be
collected into other collections (or even differing collections). In addition,
partitioning is supported for any iterator.
Removing methods like from_elem
, from_fn
, grow
, and grow_fn
Vectors and some other collections offer constructors and growth functions like the following:
fn from_elem(length: uint, value: T) -> Vec<T>
fn from_fn(length: uint, op: |uint| -> T) -> Vec<T>
fn grow(&mut self, n: uint, value: &T)
fn grow_fn(&mut self, n: uint, f: |uint| -> T)
These extra variants can easily be dropped in favor of iterators, and this RFC proposes to do so.
The iter
module already contains a Repeat
iterator; this RFC proposes to add
a free function repeat
to iter
as a convenience for iter::Repeat::new
.
With that in place, we have:
// Equivalent:
let v = Vec::from_elem(n, a);
let v = Vec::from_iter(repeat(a).take(n));
// Equivalent:
let v = Vec::from_fn(n, f);
let v = Vec::from_iter(range(0, n).map(f));
// Equivalent:
v.grow(n, a);
v.extend(repeat(a).take(n));
// Equivalent:
v.grow_fn(n, f);
v.extend(range(0, n).map(f));
While these replacements are slightly longer, an important aspect of ergonomics
is memorability: by placing greater emphasis on iterators, programmers will
quickly learn the iterator APIs and have those at their fingertips, while
remembering ad hoc method variants like grow_fn
is more difficult.
Long-term: removing push_all
and push_all_move
The push_all
and push_all_move
methods on vectors are yet more API variants
that could, in principle, go through iterators:
// The following are *semantically* equivalent
v.push_all(some_slice);
v.extend(some_slice.iter_cloned());
// The following are *semantically* equivalent
v.push_all_move(some_vec);
v.extend(some_vec);
However, currently the push_all
and push_all_move
methods can rely
on the exact size of the container being pushed, in order to elide
bounds checks. We do not currently have a way to “trust” methods like
len
on iterators to elide bounds checks. A separate RFC will
introduce the notion of a “trusted” method which should support such
optimization and allow us to deprecate the push_all
and
push_all_move
variants. (This is unlikely to happen before 1.0, so
the methods will probably still be included with “experimental”
status, and likely with different names.)
Alternatives
Borrow
and the Equiv
problem
Variants of Borrow
The original version of Borrow
was somewhat more subtle:
/// A trait for borrowing.
/// If `T: Borrow` then `&T` represents data borrowed from `T::Owned`.
trait Borrow for Sized? {
/// The type being borrowed from.
type Owned;
/// Immutably borrow from an owned value.
fn borrow(&Owned) -> &Self;
/// Mutably borrow from an owned value.
fn borrow_mut(&mut Owned) -> &mut Self;
}
trait ToOwned: Borrow {
/// Produce a new owned value, usually by cloning.
fn to_owned(&self) -> Owned;
}
impl<A: Sized> Borrow for A {
type Owned = A;
fn borrow(a: &A) -> &A {
a
}
fn borrow_mut(a: &mut A) -> &mut A {
a
}
}
impl<A: Clone> ToOwned for A {
fn to_owned(&self) -> A {
self.clone()
}
}
impl Borrow for str {
type Owned = String;
fn borrow(s: &String) -> &str {
s.as_slice()
}
fn borrow_mut(s: &mut String) -> &mut str {
s.as_mut_slice()
}
}
impl ToOwned for str {
fn to_owned(&self) -> String {
self.to_string()
}
}
impl<T> Borrow for [T] {
type Owned = Vec<T>;
fn borrow(s: &Vec<T>) -> &[T] {
s.as_slice()
}
fn borrow_mut(s: &mut Vec<T>) -> &mut [T] {
s.as_mut_slice()
}
}
impl<T> ToOwned for [T] {
fn to_owned(&self) -> Vec<T> {
self.to_vec()
}
}
impl<K,V> HashMap<K,V> where K: Borrow + Hash + Eq {
fn find(&self, k: &K) -> &V { ... }
fn insert(&mut self, k: K::Owned, v: V) -> Option<V> { ... }
...
}
pub enum Cow<'a, T> where T: ToOwned {
Shared(&'a T),
Owned(T::Owned)
}
This approach ties Borrow
directly to the borrowed data, and uses an
associated type to uniquely determine the corresponding owned data type.
For string keys, we would use HashMap<str, V>
. Then, the find
method would
take an &str
key argument, while insert
would take an owned String
. On the
other hand, for some other type Foo
a HashMap<Foo, V>
would take
&Foo
for find
and Foo
for insert
. (More discussion on the choice of
ownership is given in the alternatives section.
Benefits of this alternative:
-
Unlike the current
_equiv
orfind_with
methods, or the proposal in the RFC, this approach guarantees coherence about hashing or ordering. For example,HashMap
above requires thatK
(the borrowed key type) isHash
, and will produce hashes from owned keys by first borrowing from them. -
Unlike the proposal in this RFC, the signature of the methods for maps is very simple – essentially the same as the current
find
,insert
, etc. -
Like the proposal in this RFC, there is only a single
Borrow
trait, so it would be possible to standardize on aMap
trait later on and include these APIs. The trait could be made somewhat simpler with this alternative form ofBorrow
, but can be provided in either case; see these comments for details. -
The
Cow
data type is simpler than in the RFC’s proposal, since it does not need a type parameter for the owned data.
Drawbacks of this alternative:
-
It’s quite subtle that you want to use
HashMap<str, T>
rather thanHashMap<String, T>
. That is, if you try to use a map in the “obvious way” you will not be able to use string slices for lookup, which is part of what this RFC is trying to achieve. The same applies toCow
. -
The design is somewhat less flexible than the one in the RFC, because (1) there is a fixed choice of owned type corresponding to each borrowed type and (2) you cannot use multiple borrow types for lookups at different types (e.g. using
&String
sometimes and&str
other times). On the other hand, these restrictions guarantee coherence of hashing/equality/comparison. -
This version of
Borrow
, mapping from borrowed to owned data, is somewhat less intuitive.
On the balance, the approach proposed in the RFC seems better, because using the map APIs in the obvious ways works by default.
The HashMapKey
trait and friends
An earlier proposal for solving the _equiv
problem was given in the
associated items RFC):
trait HashMapKey : Clone + Hash + Eq {
type Query: Hash = Self;
fn compare(&self, other: &Query) -> bool { self == other }
fn query_to_key(q: &Query) -> Self { q.clone() };
}
impl HashMapKey for String {
type Query = str;
fn compare(&self, other: &str) -> bool {
self.as_slice() == other
}
fn query_to_key(q: &str) -> String {
q.into_string()
}
}
impl<K,V> HashMap<K,V> where K: HashMapKey {
fn find(&self, q: &K::Query) -> &V { ... }
}
This solution has several drawbacks, however:
-
It requires a separate trait for different kinds of maps – one for
HashMap
, one forTreeMap
, etc. -
It requires that a trait be implemented on a given key without providing a blanket implementation. Since you also need different traits for different maps, it’s easy to imagine cases where a out-of-crate type you want to use as a key doesn’t implement the key trait, forcing you to newtype.
-
It doesn’t help with the
MaybeOwned
problem.
Daniel Micay’s hack
@strcat has a PR that makes it
possible to, for example, coerce a &str
to an &String
value.
This provides some help for the _equiv
problem, since the _equiv
methods
could potentially be dropped. However, there are a few downsides:
-
Using a map with string keys is still a bit more verbose:
map.find("some static string".as_string()) // with the hack map.find("some static string") // with this RFC
-
The solution is specialized to strings and vectors, and does not necessarily support user-defined unsized types or slices.
-
It doesn’t help with the
MaybeOwned
problem. -
It exposes some representation interplay between slices and references to owned values, which we may not want to commit to or reveal.
For IntoIterator
Handling of for
loops
The fact that for x in v
moves elements from v
, while for x in v.iter()
yields references, may be a bit surprising. On the other hand, moving is the
default almost everywhere in Rust, and with the proposed approach you get to use &
and
&mut
to easily select other forms of iteration.
(See @huon’s comment for additional drawbacks.)
Unfortunately, it’s a bit tricky to make for use by-ref iterators instead. The
problem is that an iterator is IntoIterator
, but it is not Iterable
(or
whatever we call the by-reference trait). Why? Because IntoIterator
gives you
an iterator that can be used only once, while Iterable
allows you to ask for
iterators repeatedly.
If for
demanded an Iterable
, then for x in v.iter()
and for x in v.iter_mut()
would cease to work – we’d have to find some other approach. It might be
doable, but it’s not obvious how to do it.
Input versus output type parameters
An important aspect of the IntoIterator
design is that the element type is an
associated type, not an input type.
This is a tradeoff:
-
Making it an associated type means that the
for
examples work, because the type ofSelf
uniquely determines the element type for iteration, aiding type inference. -
Making it an input type would forgo those benefits, but would allow some additional flexibility. For example, you could implement
IntoIterator<A>
for an iterator on&A
whenA
is cloned, therefore implicitly cloning as needed to make the ownership work out (and obviating the need foriter_cloned
). However, we have generally kept away from this kind of implicit magic, especially when it can involve hidden costs like cloning, so the more explicit design given in this RFC seems best.
Downsides
Design tradeoffs were discussed inline.
Unresolved questions
Unresolved conventions/APIs
As mentioned above, this RFC does not resolve the question of what to call set operations that update the set in place.
It likewise does not settle the APIs that appear in only single concrete collections. These will largely be handled through the API stabilization process, unless radical changes are proposed.
Finally, additional methods provided via the IntoIterator
API are left for
future consideration.
Coercions
Using the Borrow
trait, it might be possible to safely add a coercion for auto-slicing:
If T: Borrow:
coerce &'a T::Owned to &'a T
coerce &'a mut T::Owned to &'a mut T
For sized types, this coercion is forced to be trivial, so the only time it would involve running user code is for unsized values.
A general story about such coercions will be left to a follow-up RFC.
- Start Date: 2014-10-30
- RFC PR #: rust-lang/rfcs#236
- Rust Issue #: rust-lang/rust#18466
Summary
This is a conventions RFC for formalizing the basic conventions around error handling in Rust libraries.
The high-level overview is:
-
For catastrophic errors, abort the process or fail the task depending on whether any recovery is possible.
-
For contract violations, fail the task. (Recover from programmer errors at a coarse grain.)
-
For obstructions to the operation, use
Result
(or, less often,Option
). (Recover from obstructions at a fine grain.) -
Prefer liberal function contracts, especially if reporting errors in input values may be useful to a function’s caller.
This RFC follows up on two earlier attempts by giving more leeway in when to fail the task.
Motivation
Rust provides two basic strategies for dealing with errors:
-
Task failure, which unwinds to at least the task boundary, and by default propagates to other tasks through poisoned channels and mutexes. Task failure works well for coarse-grained error handling.
-
The Result type, which allows functions to signal error conditions through the value that they return. Together with a lint and the
try!
macro,Result
works well for fine-grained error handling.
However, while there have been some general trends in the usage of the two handling mechanisms, we need to have formal guidelines in order to ensure consistency as we stabilize library APIs. That is the purpose of this RFC.
For the most part, the RFC proposes guidelines that are already followed today, but it tries to motivate and clarify them.
Detailed design
Errors fall into one of three categories:
- Catastrophic errors, e.g. out-of-memory.
- Contract violations, e.g. wrong input encoding, index out of bounds.
- Obstructions, e.g. file not found, parse error.
The basic principle of the conventions is that:
- Catastrophic errors and programming errors (bugs) can and should only be recovered at a coarse grain, i.e. a task boundary.
- Obstructions preventing an operation should be reported at a maximally fine grain – to the immediate invoker of the operation.
Catastrophic errors
An error is catastrophic if there is no meaningful way for the current task to continue after the error occurs.
Catastrophic errors are extremely rare, especially outside of libstd
.
Canonical examples: out of memory, stack overflow.
For catastrophic errors, fail the task.
For errors like stack overflow, Rust currently aborts the process, but could in principle fail the task, which (in the best case) would allow reporting and recovery from a supervisory task.
Contract violations
An API may define a contract that goes beyond the type checking enforced by the compiler. For example, slices support an indexing operation, with the contract that the supplied index must be in bounds.
Contracts can be complex and involve more than a single function invocation. For
example, the RefCell
type requires that borrow_mut
not be called until all
existing borrows have been relinquished.
For contract violations, fail the task.
A contract violation is always a bug, and for bugs we follow the Erlang philosophy of “let it crash”: we assume that software will have bugs, and we design coarse-grained task boundaries to report, and perhaps recover, from these bugs.
Contract design
One subtle aspect of these guidelines is that the contract for a function is chosen by an API designer – and so the designer also determines what counts as a violation.
This RFC does not attempt to give hard-and-fast rules for designing contracts. However, here are some rough guidelines:
-
Prefer expressing contracts through static types whenever possible.
-
It must be possible to write code that uses the API without violating the contract.
-
Contracts are most justified when violations are inarguably bugs – but this is surprisingly rare.
-
Consider whether the API client could benefit from the contract-checking logic. The checks may be expensive. Or there may be useful programming patterns where the client does not want to check inputs before hand, but would rather attempt the operation and then find out whether the inputs were invalid.
-
When a contract violation is the only kind of error a function may encounter – i.e., there are no obstructions to its success other than “bad” inputs – using
Result
orOption
instead is especially warranted. Clients can then useunwrap
to assert that they have passed valid input, or re-use the error checking done by the API for their own purposes. -
When in doubt, use loose contracts and instead return a
Result
orOption
.
Obstructions
An operation is obstructed if it cannot be completed for some reason, even though the operation’s contract has been satisfied. Obstructed operations may have (documented!) side effects – they are not required to roll back after encountering an obstruction. However, they should leave the data structures in a “coherent” state (satisfying their invariants, continuing to guarantee safety, etc.).
Obstructions may involve external conditions (e.g., I/O), or they may involve aspects of the input that are not covered by the contract.
Canonical examples: file not found, parse error.
For obstructions, use Result
The
Result<T,E>
type
represents either a success (yielding T
) or failure (yielding E
). By
returning a Result
, a function allows its clients to discover and react to
obstructions in a fine-grained way.
What about Option
?
The Option
type should not be used for “obstructed” operations; it
should only be used when a None
return value could be considered a
“successful” execution of the operation.
This is of course a somewhat subjective question, but a good litmus
test is: would a reasonable client ever ignore the result? The
Result
type provides a lint that ensures the result is actually
inspected, while Option
does not, and this difference of behavior
can help when deciding between the two types.
Another litmus test: can the operation be understood as asking a
question (possibly with sideeffects)? Operations like pop
on a
vector can be viewed as asking for the contents of the first element,
with the side effect of removing it if it exists – with an Option
return value.
Do not provide both Result
and fail!
variants.
An API should not provide both Result
-producing and fail
ing versions of an
operation. It should provide just the Result
version, allowing clients to use
try!
or unwrap
instead as needed. This is part of the general pattern of
cutting down on redundant variants by instead using method chaining.
There is one exception to this rule, however. Some APIs are strongly oriented
around failure, in the sense that their functions/methods are explicitly
intended as assertions. If there is no other way to check in advance for the
validity of invoking an operation foo
, however, the API may provide a
foo_catch
variant that returns a Result
.
The main examples in libstd
that currently provide both variants are:
-
Channels, which are the primary point of failure propagation between tasks. As such, calling
recv()
is an assertion that the other end of the channel is still alive, which will propagate failures from the other end of the channel. On the other hand, since there is no separate way to atomically test whether the other end has hung up, channels provide arecv_opt
variant that produces aResult
.Note: the
_opt
suffix would be replaced by a_catch
suffix if this RFC is accepted. -
RefCell
, which provides a dynamic version of the borrowing rules. Calling theborrow()
method is intended as an assertion that the cell is in a borrowable state, and willfail!
otherwise. On the other hand, there is no separate way to check the state of theRefCell
, so the module provides atry_borrow
variant that produces aResult
.Note: the
try_
prefix would be replaced by a_catch
catch if this RFC is accepted.
(Note: it is unclear whether these APIs will continue to provide both variants.)
Drawbacks
The main drawbacks of this proposal are:
-
Task failure remains somewhat of a landmine: one must be sure to document, and be aware of, all relevant function contracts in order to avoid task failure.
-
The choice of what to make part of a function’s contract remains somewhat subjective, so these guidelines cannot be used to decisively resolve disagreements about an API’s design.
The alternatives mentioned below do not suffer from these problems, but have drawbacks of their own.
Alternatives
Two
alternative designs have been
given in earlier RFCs, both of which take a much harder line on using fail!
(or, put differently, do not allow most functions to have contracts).
As was pointed out by @SiegeLord, however, mixing what might be seen as contract violations with obstructions can make it much more difficult to write obstruction-robust code; see the linked comment for more detail.
Naming
There are numerous possible suffixes for a Result
-producing variant:
-
_catch
, as proposed above. As @lilyball points out, this name connotes exception handling, which could be considered misleading. However, since it effectively prevents further unwinding, catching an exception may indeed be the right analogy. -
_result
, which is straightforward but not as informative/suggestive as some of the other proposed variants. -
try_
prefix. Also connotes exception handling, but has an unfortunately overlap with the common use oftry_
for nonblocking variants (which is in play forrecv
in particular).
- Start Date: 2014-10-07
- RFC PR: rust-lang/rfcs#240
- Rust Issue: rust-lang/rust#17863
Summary
This is a conventions RFC for settling the location of unsafe
APIs relative
to the types they work with, as well as the use of raw
submodules.
The brief summary is:
-
Unsafe APIs should be made into methods or static functions in the same cases that safe APIs would be.
-
raw
submodules should be used only to define explicit low-level representations.
Motivation
Many data structures provide unsafe APIs either for avoiding checks or working
directly with their (otherwise private) representation. For example, string
provides:
-
An
as_mut_vec
method onString
that provides aVec<u8>
view of the string. This method makes it easy to work with the byte-based representation of the string, but thereby also allows violation of the utf8 guarantee. -
A
raw
submodule with a number of free functions, likefrom_parts
, that constructs aString
instances from a raw-pointer-based representation, afrom_utf8
variant that does not actually check for utf8 validity, and so on. The unifying theme is that all of these functions avoid checking some key invariant.
The problem is that currently, there is no clear/consistent guideline about
which of these APIs should live as methods/static functions associated with a
type, and which should live in a raw
submodule. Both forms appear throughout
the standard library.
Detailed design
The proposed convention is:
-
When an unsafe function/method is clearly “about” a certain type (as a way of constructing, destructuring, or modifying values of that type), it should be a method or static function on that type. This is the same as the convention for placement of safe functions/methods. So functions like
string::raw::from_parts
would become static functions onString
. -
raw
submodules should only be used to define low-level types/representations (and methods/functions on them). Methods for converting to/from such low-level types should be available directly on the high-level types. Examples:core::raw
,sync::raw
.
The benefits are:
-
Ergonomics. You can gain easy access to unsafe APIs merely by having a value of the type (or, for static functions, importing the type).
-
Consistency and simplicity. The rules for placement of unsafe APIs are the same as those for safe APIs.
The perspective here is that marking APIs unsafe
is enough to deter their use
in ordinary situations; they don’t need to be further distinguished by placement
into a separate module.
There are also some naming conventions to go along with unsafe static functions and methods:
-
When an unsafe function/method is an unchecked variant of an otherwise safe API, it should be marked using an
_unchecked
suffix.For example, the
String
module should provide bothfrom_utf8
andfrom_utf8_unchecked
constructors, where the latter does not actually check the utf8 encoding. Thestring::raw::slice_bytes
andstring::raw::slice_unchecked
functions should be merged into a singleslice_unchecked
method on strings that checks neither bounds nor utf8 boundaries. -
When an unsafe function/method produces or consumes a low-level representation of a data structure, the API should use
raw
in its name. Specifically,from_raw_parts
is the typical name used for constructing a value from e.g. a pointer-based representation. -
Otherwise, consider using a name that suggests why the API is unsafe. In some cases, like
String::as_mut_vec
, other stronger conventions apply, and theunsafe
qualifier on the signature (together with API documentation) is enough.
The unsafe methods and static functions for a given type should be placed in
their own impl
block, at the end of the module defining the type; this will
ensure that they are grouped together in rustdoc. (Thanks @lilyball for the
suggestion.)
Drawbacks
One potential drawback of these conventions is that the documentation for a
module will be cluttered with rarely-used unsafe
APIs, whereas the raw
submodule approach neatly groups these APIs. But rustdoc could easily be
changed to either hide or separate out unsafe
APIs by default, and in the
meantime the impl
block grouping should help.
More specifically, the convention of placing unsafe constructors in raw
makes
them very easy to find. But the usual from_
convention, together with the
naming conventions suggested above, should make it fairly easy to discover such
constructors even when they’re supplied directly as static functions.
More generally, these conventions give unsafe
APIs more equal status with safe
APIs. Whether this is a drawback depends on your philosophy about the status
of unsafe programming. But on a technical level, the key point is that the APIs
are marked unsafe
, so users still have to opt-in to using them. Ed note: from
my perspective, low-level/unsafe programming is important to support, and there
is no reason to penalize its ergonomics given that it’s opt-in anyway.
Alternatives
There are a few alternatives:
-
Rather than providing unsafe APIs directly as methods/static functions, they could be grouped into a single extension trait. For example, the
String
type could be accompanied by aStringRaw
extension trait providing APIs for working with raw string representations. This would allow a clear grouping of unsafe APIs, while still providing them as methods/static functions and allowing them to easily be imported with e.g.use std::string::StringRaw
. On the other hand, it still further penalizes the raw APIs (beyond marking themunsafe
), and given that rustdoc could easily provide API grouping, it’s unclear exactly what the benefit is. -
Use
raw
for functions that construct a value of the type without checking for one or more invariants.The advantage is that it’s easy to find such invariant-ignoring functions. The disadvantage is that their ergonomics is worsened, since they much be separately imported or referenced through a lengthy path:
// Compare the ergonomics: string::raw::slice_unchecked(some_string, start, end) some_string.slice_unchecked(start, end)
-
Another suggestion by @lilyball is to keep the basic structure of
raw
submodules, but use associated types to improve the ergonomics. Details (and discussions of pros/cons) are in this comment. -
Use
raw
submodules to group together all manipulation of low-level representations. No module instd
currently does this; existing modules provide some free functions inraw
, and some unsafe methods, without a clear driving principle. The ergonomics of moving everything into free functions in araw
submodule are quite poor.
Unresolved questions
The core::raw
module provides structs with public representations equivalent
to several built-in and library types (boxes, closures, slices, etc.). It’s not
clear whether the name of this module, or the location of its contents, should
change as a result of this RFC. The module is a special case, because not all of
the types it deals with even have corresponding modules/type declarations – so
it probably suffices to leave decisions about it to the API stabilization
process.
- Start Date: 2014-09-16
- RFC PR: rust-lang/rfcs#241
- Rust Issue: rust-lang/rust#21432
Summary
Add the following coercions:
- From
&T
to&U
whenT: Deref<U>
. - From
&mut T
to&U
whenT: Deref<U>
. - From
&mut T
to&mut U
whenT: DerefMut<U>
These coercions eliminate the need for “cross-borrowing” (things like &**v
)
and calls to as_slice
.
Motivation
Rust currently supports a conservative set of implicit coercions that are used
when matching the types of arguments against those given for a function’s
parameters. For example, if T: Trait
then &T
is implicitly coerced to
&Trait
when used as a function argument:
trait MyTrait { ... }
struct MyStruct { ... }
impl MyTrait for MyStruct { ... }
fn use_trait_obj(t: &MyTrait) { ... }
fn use_struct(s: &MyStruct) {
use_trait_obj(s) // automatically coerced from &MyStruct to &MyTrait
}
In older incarnations of Rust, in which types like vectors were built in to the
language, coercions included things like auto-borrowing (taking T
to &T
),
auto-slicing (taking Vec<T>
to &[T]
) and “cross-borrowing” (taking Box<T>
to &T
). As built-in types migrated to the library, these coercions have
disappeared: none of them apply today. That means that you have to write code
like &**v
to convert &Box<T>
or Rc<RefCell<T>>
to &T
and v.as_slice()
to convert Vec<T>
to &T
.
The ergonomic regression was coupled with a promise that we’d improve things in a more general way later on.
“Later on” has come! The premise of this RFC is that (1) we have learned some valuable lessons in the interim and (2) there is a quite conservative kind of coercion we can add that dramatically improves today’s ergonomic state of affairs.
Detailed design
Design principles
The centrality of ownership and borrowing
As Rust has evolved, a theme has emerged: ownership and borrowing are the focal point of Rust’s design, and the key enablers of much of Rust’s achievements.
As such, reasoning about ownership/borrowing is a central aspect of programming in Rust.
In the old coercion model, borrowing could be done completely implicitly, so an invocation like:
foo(bar, baz, quux)
might move bar
, immutably borrow baz
, and mutably borrow quux
. To
understand the flow of ownership, then, one has to be aware of the details of
all function signatures involved – it is not possible to see ownership at a
glance.
When auto-borrowing was removed, this reasoning difficulty was cited as a major motivator:
Code readability does not necessarily benefit from autoref on arguments:
let a = ~Foo;
foo(a); // reading this code looks like it moves `a`
fn foo(_: &Foo) {} // ah, nevermind, it doesn't move `a`!
let mut a = ~[ ... ];
sort(a); // not only does this not move `a`, but it mutates it!
Having to include an extra &
or &mut
for arguments is a slight
inconvenience, but it makes it much easier to track ownership at a glance.
(Note that ownership is not entirely explicit, due to self
and macros; see
the appendix.)
This RFC takes as a basic principle: Coercions should never implicitly borrow from owned data.
This is a key difference from the cross-borrowing RFC.
Limit implicit execution of arbitrary code
Another positive aspect of Rust’s current design is that a function call like
foo(bar, baz)
does not invoke arbitrary code (general implicit coercions, as
found in e.g. Scala). It simply executes foo
.
The tradeoff here is similar to the ownership tradeoff: allowing arbitrary implicit coercions means that a programmer must understand the types of the arguments given, the types of the parameters, and all applicable coercion code in order to understand what code will be executed. While arbitrary coercions are convenient, they come at a substantial cost in local reasoning about code.
Of course, method dispatch can implicitly execute code via Deref
. But Deref
is a pretty specialized tool:
-
Each type
T
can only deref to one other type.(Note: this restriction is not currently enforced, but will be enforceable once associated types land.)
-
Deref makes all the methods of the target type visible on the source type.
-
The source and target types are both references, limiting what the
deref
code can do.
These characteristics combined make Deref
suitable for smart pointer-like
types and little else. They make Deref
implementations relatively rare. And as
a consequence, you generally know when you’re working with a type implementing
Deref
.
This RFC takes as a basic principle: Coercions should narrowly limit the code they execute.
Coercions through Deref
are considered narrow enough.
The proposal
The idea is to introduce a coercion corresponding to Deref
/DerefMut
, but
only for already-borrowed values:
- From
&T
to&U
whenT: Deref<U>
. - From
&mut T
to&U
whenT: Deref<U>
. - From
&mut T
to&mut U
whenT: DerefMut<U>
These coercions are applied recursively, similarly to auto-deref for method dispatch.
Here is a simple pseudocode algorithm for determining the applicability of
coercions. Let HasBasicCoercion(T, U)
be a procedure for determining whether
T
can be coerced to U
using today’s coercion rules (i.e. without deref).
The general HasCoercion(T, U)
procedure would work as follows:
HasCoercion(T, U):
if HasBasicCoercion(T, U) then
true
else if T = &V and V: Deref<W> then
HasCoercion(&W, U)
else if T = &mut V and V: Deref<W> then
HasCoercion(&W, U)
else if T = &mut V and V: DerefMut<W> then
HasCoercion(&W, U)
else
false
Essentially, the procedure looks for applicable “basic” coercions at increasing levels of deref from the given argument, just as method resolution searches for applicable methods at increasing levels of deref.
Unlike method resolution, however, this coercion does not automatically borrow.
Benefits of the design
Under this coercion design, we’d see the following ergonomic improvements for “cross-borrowing”:
fn use_ref(t: &T) { ... }
fn use_mut(t: &mut T) { ... }
fn use_rc(t: Rc<T>) {
use_ref(&*t); // what you have to write today
use_ref(&t); // what you'd be able to write
}
fn use_mut_box(t: &mut Box<T>) {
use_mut(&mut *t); // what you have to write today
use_mut(t); // what you'd be able to write
use_ref(*t); // what you have to write today
use_ref(t); // what you'd be able to write
}
fn use_nested(t: &Box<T>) {
use_ref(&**t); // what you have to write today
use_ref(t); // what you'd be able to write (note: recursive deref)
}
In addition, if Vec<T>: Deref<[T]>
(as proposed
here), slicing would be automatic:
fn use_slice(s: &[u8]) { ... }
fn use_vec(v: Vec<u8>) {
use_slice(v.as_slice()); // what you have to write today
use_slice(&v); // what you'd be able to write
}
fn use_vec_ref(v: &Vec<u8>) {
use_slice(v.as_slice()); // what you have to write today
use_slice(v); // what you'd be able to write
}
Characteristics of the design
The design satisfies both of the principles laid out in the Motivation:
-
It does not introduce implicit borrows of owned data, since it only applies to already-borrowed data.
-
It only applies to
Deref
types, which means there is only limited potential for implicitly running unknown code; together with the expectation that programmers are generally aware when they are usingDeref
types, this should retain the kind of local reasoning Rust programmers can do about function/method invocations today.
There is a conceptual model implicit in the design here: &
is a “borrow”
operator, and richer coercions are available between borrowed types. This
perspective is in opposition to viewing &
primarily as adding a layer of
indirection – a view that, given compiler optimizations, is often inaccurate
anyway.
Drawbacks
As with any mechanism that implicitly invokes code, deref coercions make it more complex to fully understand what a given piece of code is doing. The RFC argued inline that the design conserves local reasoning in practice.
As mentioned above, this coercion design also changes the mental model
surrounding &
, and in particular somewhat muddies the idea that it creates a
pointer. This change could make Rust more difficult to learn (though note that
it puts more attention on ownership), though it would make it more convenient
to use in the long run.
Alternatives
The main alternative that addresses the same goals as this RFC is the
cross-borrowing RFC, which
proposes a more aggressive form of deref coercion: it would allow converting
e.g. Box<T>
to &T
and Vec<T>
to &[T]
directly. The advantage is even
greater convenience: in many cases, even &
is not necessary. The disadvantage
is the change to local reasoning about ownership:
let v = vec![0u8, 1, 2];
foo(v); // is v moved here?
bar(v); // is v still available?
Knowing whether v
is moved in the call to foo
requires knowing foo
’s
signature, since the coercion would implicitly borrow from the vector.
Appendix: ownership in Rust today
In today’s Rust, ownership transfer/borrowing is explicit for all function/method arguments. It is implicit only for:
-
self
on method invocations. In practice, the name and context of a method invocation is almost always sufficient to infer its move/borrow semantics. -
Macro invocations. Since macros can expand into arbitrary code, macro invocations can appear to move when they actually borrow.
- Feature-gates:
question_mark
,try_catch
- Start Date: 2014-09-16
- RFC PR #: rust-lang/rfcs#243
- Rust Issue #: rust-lang/rust#31436
Summary
Add syntactic sugar for working with the Result
type which models common
exception handling constructs.
The new constructs are:
-
An
?
operator for explicitly propagating “exceptions”. -
A
catch { ... }
expression for conveniently catching and handling “exceptions”.
The idea for the ?
operator originates from RFC PR 204 by
@aturon.
Motivation and overview
Rust currently uses the enum Result
type for error handling. This solution is
simple, well-behaved, and easy to understand, but often gnarly and inconvenient
to work with. We would like to solve the latter problem while retaining the
other nice properties and avoiding duplication of functionality.
We can accomplish this by adding constructs which mimic the exception-handling constructs of other languages in both appearance and behavior, while improving upon them in typically Rustic fashion. Their meaning can be specified by a straightforward source-to-source translation into existing language constructs, plus a very simple and obvious new one. (They may also, but need not necessarily, be implemented in this way.)
These constructs are strict additions to the existing language, and apart from the issue of keywords, the legality and behavior of all currently existing Rust programs is entirely unaffected.
The most important additions are a postfix ?
operator for
propagating “exceptions” and a catch {..}
expression for catching
them. By an “exception”, for now, we essentially just mean the Err
variant of a Result
, though the Unresolved Questions includes some
discussion of extending to other types.
?
operator
The postfix ?
operator can be applied to Result
values and is equivalent to
the current try!()
macro. It either returns the Ok
value directly, or
performs an early exit and propagates the Err
value further out. (So given
my_result: Result<Foo, Bar>
, we have my_result?: Foo
.) This allows it to be
used for e.g. conveniently chaining method calls which may each “throw an
exception”:
foo()?.bar()?.baz()
Naturally, in this case the types of the “exceptions thrown by” foo()
and
bar()
must unify. Like the current try!()
macro, the ?
operator will also
perform an implicit “upcast” on the exception type.
When used outside of a catch
block, the ?
operator propagates the exception to
the caller of the current function, just like the current try!
macro does. (If
the return type of the function isn’t a Result
, then this is a type error.)
When used inside a catch
block, it propagates the exception up to the innermost
catch
block, as one would expect.
Requiring an explicit ?
operator to propagate exceptions strikes a very
pleasing balance between completely automatic exception propagation, which most
languages have, and completely manual propagation, which we’d have apart from
the try!
macro. It means that function calls remain simply function calls
which return a result to their caller, with no magic going on behind the scenes;
and this also increases flexibility, because one gets to choose between
propagation with ?
or consuming the returned Result
directly.
The ?
operator itself is suggestive, syntactically lightweight enough to not
be bothersome, and lets the reader determine at a glance where an exception may
or may not be thrown. It also means that if the signature of a function changes
with respect to exceptions, it will lead to type errors rather than silent
behavior changes, which is a good thing. Finally, because exceptions are tracked
in the type system, and there is no silent propagation of exceptions, and all
points where an exception may be thrown are readily apparent visually, this also
means that we do not have to worry very much about “exception safety”.
Exception type upcasting
In a language with checked exceptions and subtyping, it is clear that if a
function is declared as throwing a particular type, its body should also be able
to throw any of its subtypes. Similarly, in a language with structural sum types
(a.k.a. anonymous enum
s, polymorphic variants), one should be able to throw a
type with fewer cases in a function declaring that it may throw a superset of
those cases. This is essentially what is achieved by the common Rust practice of
declaring a custom error enum
with From
impl
s for each of the upstream
error types which may be propagated:
enum MyError {
IoError(io::Error),
JsonError(json::Error),
OtherError(...)
}
impl From<io::Error> for MyError { ... }
impl From<json::Error> for MyError { ... }
Here io::Error
and json::Error
can be thought of as subtypes of MyError
,
with a clear and direct embedding into the supertype.
The ?
operator should therefore perform such an implicit conversion, in the
nature of a subtype-to-supertype coercion. The present RFC uses the
std::convert::Into
trait for this purpose (which has a blanket impl
forwarding from From
). The precise requirements for a conversion to be “like”
a subtyping coercion are an open question; see the “Unresolved questions”
section.
catch
expressions
This RFC also introduces an expression form catch {..}
, which serves
to “scope” the ?
operator. The catch
operator executes its
associated block. If no exception is thrown, then the result is
Ok(v)
where v
is the value of the block. Otherwise, if an
exception is thrown, then the result is Err(e)
. Note that unlike
other languages, a catch
block always catches all errors, and they
must all be coercible to a single type, as a Result
only has a
single Err
type. This dramatically simplifies thinking about the
behavior of exception-handling code.
Note that catch { foo()? }
is essentially equivalent to foo()
.
catch
can be useful if you want to coalesce multiple potential
exceptions – catch { foo()?.bar()?.baz()? }
– into a single
Result
, which you wish to then e.g. pass on as-is to another
function, rather than analyze yourself. (The last example could also
be expressed using a series of and_then
calls.)
Detailed design
The meaning of the constructs will be specified by a source-to-source
translation. We make use of an “early exit from any block” feature
which doesn’t currently exist in the language, generalizes the current
break
and return
constructs, and is independently useful.
Early exit from any block
The capability can be exposed either by generalizing break
to take an optional
value argument and break out of any block (not just loops), or by generalizing
return
to take an optional lifetime argument and return from any block, not
just the outermost block of the function. This feature is only used in this RFC
as an explanatory device, and implementing the RFC does not require exposing it,
so I am going to arbitrarily choose the break
syntax for the following and
won’t discuss the question further.
So we are extending break
with an optional value argument: break 'a EXPR
.
This is an expression of type !
which causes an early return from the
enclosing block specified by 'a
, which then evaluates to the value EXPR
(of
course, the type of EXPR
must unify with the type of the last expression in
that block). This works for any block, not only loops.
[Note: This was since added in RFC 2046]
A completely artificial example:
'a: {
let my_thing = if have_thing() {
get_thing()
} else {
break 'a None
};
println!("found thing: {}", my_thing);
Some(my_thing)
}
Here if we don’t have a thing, we escape from the block early with None
.
If no value is specified, it defaults to ()
: in other words, the current
behavior. We can also imagine there is a magical lifetime 'fn
which refers to
the lifetime of the whole function: in this case, break 'fn
is equivalent to
return
.
Again, this RFC does not propose generalizing break
in this way at this time:
it is only used as a way to explain the meaning of the constructs it does
propose.
Definition of constructs
Finally we have the definition of the new constructs in terms of a source-to-source translation.
In each case except the first, I will provide two definitions: a single-step “shallow” desugaring which is defined in terms of the previously defined new constructs, and a “deep” one which is “fully expanded”.
Of course, these could be defined in many equivalent ways: the below definitions are merely one way.
-
Construct:
EXPR?
Shallow:
match EXPR { Ok(a) => a, Err(e) => break 'here Err(e.into()) }
Where
'here
refers to the innermost enclosingcatch
block, or to'fn
if there is none.The
?
operator has the same precedence as.
. -
Construct:
catch { foo()?.bar() }
Shallow:
'here: { Ok(foo()?.bar()) }
Deep:
'here: { Ok(match foo() { Ok(a) => a, Err(e) => break 'here Err(e.into()) }.bar()) }
The fully expanded translations get quite gnarly, but that is why it’s good that you don’t have to write them!
In general, the types of the defined constructs should be the same as the types of their definitions.
(As noted earlier, while the behavior of the constructs can be specified using a source-to-source translation in this manner, they need not necessarily be implemented this way.)
As a result of this RFC, both Into
and Result
would have to become lang
items.
Laws
Without any attempt at completeness, here are some things which should be true:
catch { foo() }
=Ok(foo())
catch { Err(e)? }
=Err(e.into())
catch { try_foo()? }
=try_foo().map_err(Into::into)
(In the above, foo()
is a function returning any type, and try_foo()
is a
function returning a Result
.)
Feature gates
The two major features here, the ?
syntax and catch
expressions,
will be tracked by independent feature gates. Each of the features has
a distinct motivation, and we should evaluate them independently.
Unresolved questions
These questions should be satisfactorily resolved before stabilizing the relevant features, at the latest.
Optional match
sugar
Originally, the RFC included the ability to match
the errors caught
by a catch
by writing catch { .. } match { .. }
, which could be translated
as follows:
-
Construct:
catch { foo()?.bar() } match { A(a) => baz(a), B(b) => quux(b) }
Shallow:
match (catch { foo()?.bar() }) { Ok(a) => a, Err(e) => match e { A(a) => baz(a), B(b) => quux(b) } }
Deep:
match ('here: { Ok(match foo() { Ok(a) => a, Err(e) => break 'here Err(e.into()) }.bar()) }) { Ok(a) => a, Err(e) => match e { A(a) => baz(a), B(b) => quux(b) } }
However, it was removed for the following reasons:
- The
catch
(originally:try
) keyword adds the real expressive “step up” here, thematch
(originally:catch
) was just sugar forunwrap_or
. - It would be easy to add further sugar in the future, once we see how
catch
is used (or not used) in practice. - There was some concern about potential user confusion about two aspects:
catch { }
yields aResult<T,E>
butcatch { } match { }
yields justT
;catch { } match { }
handles all kinds of errors, unliketry/catch
in other languages which let you pick and choose.
It may be worth adding such a sugar in the future, or perhaps a
variant that binds irrefutably and does not immediately lead into a
match
block.
Choice of keywords
The RFC to this point uses the keyword catch
, but there are a number
of other possibilities, each with different advantages and drawbacks:
-
try { ... } catch { ... }
-
try { ... } match { ... }
-
try { ... } handle { ... }
-
catch { ... } match { ... }
-
catch { ... } handle { ... }
-
catch ...
(without braces or a second clause)
Among the considerations:
-
Simplicity. Brevity.
-
Following precedent from existing, popular languages, and familiarity with respect to their analogous constructs.
-
Fidelity to the constructs’ actual behavior. For instance, the first clause always catches the “exception”; the second only branches on it.
-
Consistency with the existing
try!()
macro. If the first clause is calledtry
, thentry { }
andtry!()
would have essentially inverse meanings. -
Language-level backwards compatibility when adding new keywords. I’m not sure how this could or should be handled.
Semantics for “upcasting”
What should the contract for a From
/Into
impl
be? Are these even the right
trait
s to use for this feature?
Two obvious, minimal requirements are:
-
It should be pure: no side effects, and no observation of side effects. (The result should depend only on the argument.)
-
It should be total: no panics or other divergence, except perhaps in the case of resource exhaustion (OOM, stack overflow).
The other requirements for an implicit conversion to be well-behaved in the context of this feature should be thought through with care.
Some further thoughts and possibilities on this matter, only as brainstorming:
-
It should be “like a coercion from subtype to supertype”, as described earlier. The precise meaning of this is not obvious.
-
A common condition on subtyping coercions is coherence: if you can compound-coerce to go from
A
toZ
indirectly along multiple different paths, they should all have the same end result. -
It should be lossless, or in other words, injective: it should map each observably-different element of the input type to observably-different elements of the output type. (Observably-different means that it is possible to write a program which behaves differently depending on which one it gets, modulo things that “shouldn’t count” like observing execution time or resource usage.)
-
It should be unambiguous, or preserve the meaning of the input:
impl From<u8> for u32
asx as u32
feels right; as(x as u32) * 12345
feels wrong, even though this is perfectly pure, total, and injective. What this means precisely in the general case is unclear. -
The types converted between should the “same kind of thing”: for instance, the existing
impl From<u32> for Ipv4Addr
feels suspect on this count. (This perhaps ties into the subtyping angle:Ipv4Addr
is clearly not a supertype ofu32
.)
Forwards-compatibility
If we later want to generalize this feature to other types such as Option
, as
described below, will we be able to do so while maintaining backwards-compatibility?
Monadic do notation
There have been many comparisons drawn between this syntax and monadic
do notation. Before stabilizing, we should determine whether we plan
to make changes to better align this feature with a possible do
notation (for example, by removing the implicit Ok
at the end of a
catch
block). Note that such a notation would have to extend the
standard monadic bind to accommodate rich control flow like break
,
continue
, and return
.
Drawbacks
-
Increases the syntactic surface area of the language.
-
No expressivity is added, only convenience. Some object to “there’s more than one way to do it” on principle.
-
If at some future point we were to add higher-kinded types and syntactic sugar for monads, a la Haskell’s
do
or Scala’sfor
, their functionality may overlap and result in redundancy. However, a number of challenges would have to be overcome for a generic monadic sugar to be able to fully supplant these features: the integration of higher-kinded types into Rust’s type system in the first place, the shape of aMonad
trait
in a language with lifetimes and move semantics, interaction between the monadic control flow and Rust’s native control flow (the “ambient monad”), automatic upcasting of exception types viaInto
(the exception (Either
,Result
) monad normally does not do this, and it’s not clear whether it can), and potentially others.
Alternatives
-
Don’t.
-
Only add the
?
operator, but notcatch
expressions. -
Instead of a built-in
catch
construct, attempt to define one using macros. However, this is likely to be awkward because, at least, macros may only have their contents as a single block, rather than two. Furthermore, macros are excellent as a “safety net” for features which we forget to add to the language itself, or which only have specialized use cases; but generally useful control flow constructs still work better as language features. -
Add first-class checked exceptions, which are propagated automatically (without an
?
operator).This has the drawbacks of being a more invasive change and duplicating functionality: each function must choose whether to use checked exceptions via
throws
, or to return aResult
. While the two are isomorphic and converting between them is easy, with this proposal, the issue does not even arise, as exception handling is defined in terms ofResult
. Furthermore, automatic exception propagation raises the specter of “exception safety”: how serious an issue this would actually be in practice, I don’t know - there’s reason to believe that it would be much less of one than in C++. -
Wait (and hope) for HKTs and generic monad sugar.
Future possibilities
Expose a generalized form of break
or return
as described
This RFC doesn’t propose doing so at this time, but as it would be an independently useful feature, it could be added as well.
throw
and throws
It is possible to carry the exception handling analogy further and also add
throw
and throws
constructs.
throw
is very simple: throw EXPR
is essentially the same thing as
Err(EXPR)?
; in other words it throws the exception EXPR
to the innermost
catch
block, or to the function’s caller if there is none.
A throws
clause on a function:
fn foo(arg: Foo) -> Bar throws Baz { ... }
would mean that instead of writing return Ok(foo)
and return Err(bar)
in the
body of the function, one would write return foo
and throw bar
, and these
are implicitly turned into Ok
or Err
for the caller. This removes syntactic
overhead from both “normal” and “throwing” code paths and (apart from ?
to
propagate exceptions) matches what code might look like in a language with
native exceptions.
Generalize over Result
, Option
, and other result-carrying types
Option<T>
is completely equivalent to Result<T, ()>
modulo names, and many
common APIs use the Option
type, so it would be useful to extend all of the
above syntax to Option
, and other (potentially user-defined)
equivalent-to-Result
types, as well.
This can be done by specifying a trait for types which can be used to “carry”
either a normal result or an exception. There are several different, equivalent
ways to formulate it, which differ in the set of methods provided, but the
meaning in any case is essentially just that you can choose some types Normal
and Exception
such that Self
is isomorphic to Result<Normal, Exception>
.
Here is one way:
#[lang(result_carrier)]
trait ResultCarrier {
type Normal;
type Exception;
fn embed_normal(from: Normal) -> Self;
fn embed_exception(from: Exception) -> Self;
fn translate<Other: ResultCarrier<Normal=Normal, Exception=Exception>>(from: Self) -> Other;
}
For greater clarity on how these methods work, see the section on impl
s below.
(For a simpler formulation of the trait using Result
directly, see further
below.)
The translate
method says that it should be possible to translate to any
other ResultCarrier
type which has the same Normal
and Exception
types.
This may not appear to be very useful, but in fact, this is what can be used to
inspect the result, by translating it to a concrete, known type such as
Result<Normal, Exception>
and then, for example, pattern matching on it.
Laws:
- For all
x
,translate(embed_normal(x): A): B
=embed_normal(x): B
. - For all
x
,translate(embed_exception(x): A): B
=embed_exception(x): B
. - For all
carrier
,translate(translate(carrier: A): B): A
=carrier: A
.
Here I’ve used explicit type ascription syntax to make it clear that e.g. the
types of embed_
on the left and right hand sides are different.
The first two laws say that embedding a result x
into one result-carrying type
and then translating it to a second result-carrying type should be the same as
embedding it into the second type directly.
The third law says that translating to a different result-carrying type and then translating back should be a no-op.
impl
s of the trait
impl<T, E> ResultCarrier for Result<T, E> {
type Normal = T;
type Exception = E;
fn embed_normal(a: T) -> Result<T, E> { Ok(a) }
fn embed_exception(e: E) -> Result<T, E> { Err(e) }
fn translate<Other: ResultCarrier<Normal=T, Exception=E>>(result: Result<T, E>) -> Other {
match result {
Ok(a) => Other::embed_normal(a),
Err(e) => Other::embed_exception(e)
}
}
}
As we can see, translate
can be implemented by deconstructing ourself and then
re-embedding the contained value into the other result-carrying type.
impl<T> ResultCarrier for Option<T> {
type Normal = T;
type Exception = ();
fn embed_normal(a: T) -> Option<T> { Some(a) }
fn embed_exception(e: ()) -> Option<T> { None }
fn translate<Other: ResultCarrier<Normal=T, Exception=()>>(option: Option<T>) -> Other {
match option {
Some(a) => Other::embed_normal(a),
None => Other::embed_exception(())
}
}
}
Potentially also:
impl ResultCarrier for bool {
type Normal = ();
type Exception = ();
fn embed_normal(a: ()) -> bool { true }
fn embed_exception(e: ()) -> bool { false }
fn translate<Other: ResultCarrier<Normal=(), Exception=()>>(b: bool) -> Other {
match b {
true => Other::embed_normal(()),
false => Other::embed_exception(())
}
}
}
The laws should be sufficient to rule out any “icky” impls. For example, an impl
for Vec
where an exception is represented as the empty vector, and a normal
result as a single-element vector: here the third law fails, because if the
Vec
has more than one element to begin with, then it’s not possible to
translate to a different result-carrying type and then back without losing
information.
The bool
impl may be surprising, or not useful, but it is well-behaved:
bool
is, after all, isomorphic to Result<(), ()>
.
Other miscellaneous notes about ResultCarrier
-
Our current lint for unused results could be replaced by one which warns for any unused result of a type which implements
ResultCarrier
. -
If there is ever ambiguity due to the result-carrying type being underdetermined (experience should reveal whether this is a problem in practice), we could resolve it by defaulting to
Result
. -
Translating between different result-carrying types with the same
Normal
andException
types should, but may not necessarily currently be, a machine-level no-op most of the time.We could/should make it so that:
- repr(
Option<T>
) = repr(Result<T, ()>
) - repr(
bool
) = repr(Option<()>
) = repr(Result<(), ()>
)
If these hold, then
translate
between these types could in theory be compiled down to just atransmute
. (Whether LLVM is smart enough to do this, I don’t know.) - repr(
-
The
translate()
function smells to me like a natural transformation between functors, but I’m not category theorist enough for it to be obvious.
Alternative formulations of the ResultCarrier
trait
All of these have the form:
trait ResultCarrier {
type Normal;
type Exception;
...methods...
}
and differ only in the methods, which will be given.
Explicit isomorphism with Result
fn from_result(Result<Normal, Exception>) -> Self;
fn to_result(Self) -> Result<Normal, Exception>;
This is, of course, the simplest possible formulation.
The drawbacks are that it, in some sense, privileges Result
over other
potentially equivalent types, and that it may be less efficient for those types:
for any non-Result
type, every operation requires two method calls (one into
Result
, and one out), whereas with the ResultCarrier
trait in the main text,
they only require one.
Laws:
- For all
x
,from_result(to_result(x))
=x
. - For all
x
,to_result(from_result(x))
=x
.
Laws for the remaining formulations below are left as an exercise for the reader.
Avoid privileging Result
, most naive version
fn embed_normal(Normal) -> Self;
fn embed_exception(Exception) -> Self;
fn is_normal(&Self) -> bool;
fn is_exception(&Self) -> bool;
fn assert_normal(Self) -> Normal;
fn assert_exception(Self) -> Exception;
Of course this is horrible.
Destructuring with HOFs (a.k.a. Church/Scott-encoding)
fn embed_normal(Normal) -> Self;
fn embed_exception(Exception) -> Self;
fn match_carrier<T>(Self, FnOnce(Normal) -> T, FnOnce(Exception) -> T) -> T;
This is probably the right approach for Haskell, but not for Rust.
With this formulation, because they each take ownership of them, the two closures may not even close over the same variables!
Destructuring with HOFs, round 2
trait BiOnceFn {
type ArgA;
type ArgB;
type Ret;
fn callA(Self, ArgA) -> Ret;
fn callB(Self, ArgB) -> Ret;
}
trait ResultCarrier {
type Normal;
type Exception;
fn normal(Normal) -> Self;
fn exception(Exception) -> Self;
fn match_carrier<T>(Self, BiOnceFn<ArgA=Normal, ArgB=Exception, Ret=T>) -> T;
}
Here we solve the environment-sharing problem from above: instead of two objects with a single method each, we use a single object with two methods! I believe this is the most flexible and general formulation (which is however a strange thing to believe when they are all equivalent to each other). Of course, it’s even more awkward syntactically.
- Start Date: 2014-08-08
- RFC PR: rust-lang/rfcs#246
- Rust Issue: rust-lang/rust#17718
Summary
Divide global declarations into two categories:
- constants declare constant values. These represent a value,
not a memory address. This is the most common thing one would reach
for and would replace
static
as we know it today in almost all cases. - statics declare global variables. These represent a memory address. They would be rarely used: the primary use cases are global locks, global atomic counters, and interfacing with legacy C libraries.
Motivation
We have been wrestling with the best way to represent globals for some times. There are a number of interrelated issues:
- Significant addresses and inlining: For optimization purposes, it is useful to be able to inline constant values directly into the program. It is even more useful if those constant values do not have known addresses, because that means the compiler is free to replicate them as it wishes. Moreover, if a constant is inlined into downstream crates, then they must be recompiled whenever that constant changes.
- Read-only memory: Whenever possible, we’d like to place large constants into read-only memory. But this means that the data must be truly immutable, or else a segfault will result.
- Global atomic counters and the like: We’d like to make it possible for people to create global locks or atomic counters that can be used without resorting to unsafe code.
- Interfacing with C code: Some C libraries require the use of global, mutable data. Other times it’s just convenient and threading is not a concern.
- Initializer constants: There must be a way to have initializer
constants for things like locks and atomic counters, so that people
can write
static MY_COUNTER: AtomicUint = INIT_ZERO
or some such. It should not be possible to modify these initializer constants.
The current design is that we have only one keyword, static
, which
declares a global variable. By default, global variables do not have
significant addresses and can be inlined into the program. You can make
a global variable have a significant address by marking it
#[inline(never)]
. Furthermore, you can declare a mutable global
using static mut
: all accesses to static mut
variables are
considered unsafe. Because we wish to allow static
values to be
placed in read-only memory, they are forbidden from having a type that
includes interior mutable data (that is, an appearance of UnsafeCell
type).
Some concrete problems with this design are:
- There is no way to have a safe global counter or lock. Those must be
placed in
static mut
variables, which means that access to them is illegal. To resolve this, there is an alternative proposal, according to which, access tostatic mut
is considered safe if the type of the static mut meets theSync
trait. - The significance (no pun intended) of the
#[inline(never)]
annotation is not intuitive. - There is no way to have a generic type constant.
Other less practical and more aesthetic concerns are:
- Although
static
andlet
look and feel analogous, the two behave quite differently. Generally speaking,static
declarations do not declare variables but rather values, which can be inlined and which do not have fixed addresses. You cannot have interior mutability in astatic
variable, but you can in alet
. So thatstatic
variables can appear in patterns, it is illegal to shadow astatic
variable – butlet
variables cannot appear in patterns. Etc. - There are other constructs in the language, such as nullary enum variants and nullary structs, which look like global data but in fact act quite differently. They are actual values which do not have addresses. They are categorized as rvalues and so forth.
Detailed design
Constants
Reintroduce a const
declaration which declares a constant:
const name: type = value;
Constants may be declared in any scope. They cannot be shadowed. Constants are considered rvalues. Therefore, taking the address of a constant actually creates a spot on the local stack – they by definition have no significant addresses. Constants are intended to behave exactly like nullary enum variants.
Possible extension: Generic constants
As a possible extension, it is perfectly reasonable for constants to have generic parameters. For example, the following constant is legal:
struct WrappedOption<T> { value: Option<T> }
const NONE<T> = WrappedOption { value: None }
Note that this makes no sense for a static
variable, which represents
a memory location and hence must have a concrete type.
Possible extension: constant functions
It is possible to imagine constant functions as well. This could help to address the problem of encapsulating initialization. To avoid the need to specify what kinds of code can execute in a constant function, we can limit them syntactically to a single constant expression that can be expanded at compilation time (no recursion).
struct LockedData<T:Send> { lock: Lock, value: T }
const LOCKED<T:Send>(t: T) -> LockedData<T> {
LockedData { lock: INIT_LOCK, value: t }
}
This would allow us to make the value
field on UnsafeCell
private,
among other things.
Static variables
Repurpose the static
declaration to declare static variables
only. Static variables always have single addresses. static
variables can optionally be declared as mut
. The lifetime of a
static
variable is 'static
. It is not legal to move from a static.
Accesses to a static variable generate actual reads and writes: the
value is not inlined (but see “Unresolved Questions” below).
Non-mut
statics must have a type that meets the Sync
bound. All
access to the static is considered safe (that is, reading the variable
and taking its address). If the type of the static does not contain
an UnsafeCell
in its interior, the compiler may place it in
read-only memory, but otherwise it must be placed in mutable memory.
mut
statics may have any type. All access is considered unsafe.
They may not be placed in read-only memory.
Globals referencing Globals
const => const
It is possible to create a const
or a static
which references another
const
or another static
by its address. For example:
struct SomeStruct { x: uint }
const FOO: SomeStruct = SomeStruct { x: 1 };
const BAR: &'static SomeStruct = &FOO;
Constants are generally inlined into the stack frame from which they are referenced, but in a static context there is no stack frame. Instead, the compiler will reinterpret this as if it were written as:
struct SomeStruct { x: uint }
const FOO: SomeStruct = SomeStruct { x: 1 };
const BAR: &'static SomeStruct = {
static TMP: SomeStruct = FOO;
&TMP
};
Here a static
is introduced to be able to give the const
a pointer which
does indeed have the 'static
lifetime. Due to this rewriting, the compiler
will disallow SomeStruct
from containing an UnsafeCell
(interior
mutability). In general, a constant A cannot reference the address of another
constant B if B contains an UnsafeCell
in its interior.
const => static
It is illegal for a constant to refer to another static. A constant represents a constant value while a static represents a memory location, and this sort of reference is difficult to reconcile in light of their definitions.
static => const
If a static
references the address of a const
, then a similar rewriting
happens, but there is no interior mutability restriction (only a Sync
restriction).
static => static
It is illegal for a static
to reference another static
by value. It is
required that all references be borrowed. Additionally, not all kinds of borrows
are allowed, only explicitly taking the address of another static is allowed.
For example, interior borrows of fields and elements or accessing elements of an
array are both disallowed.
If a by-value reference were allowed, then this sort of reference would require that the static being referenced fall into one of two categories:
- It’s an initializer pattern. This is the purpose of
const
, however. - The values are kept in sync. This is currently technically infeasible.
Instead of falling into one of these two categories, the compiler will instead disallow any references to statics by value (from other statics).
Patterns
Today, a static
is allowed to be used in pattern matching. With the
introduction of const
, however, a static
will be forbidden from appearing
in a pattern match, and instead only a const
can appear.
Drawbacks
This RFC introduces two keywords for global data. Global data is kind
of an edge feature so this feels like overkill. (On the other hand,
the only keyword that most Rust programmers should need to know is
const
– I imagine static
variables will be used quite rarely.)
Alternatives
The other design under consideration is to keep the current split but
make access to static mut
be considered safe if the type of the
static mut is Sync
. For the details of this discussion, please see
RFC 177.
One serious concern is with regard to timing. Adding more things to
the Rust 1.0 schedule is inadvisable. Therefore, it would be possible
to take a hybrid approach: keep the current static
rules, or perhaps
the variation where access to static mut
is safe, for the time
being, and create const
declarations after Rust 1.0 is released.
Unresolved questions
-
Should the compiler be allowed to inline the values of
static
variables which are deeply immutable (and thus force recompilation)? -
Should we permit
static
variables whose type is notSync
, but simply make access to them unsafe? -
Should we permit
static
variables whose type is notSync
, but whose initializer value does not actually contain interior mutability? For example, astatic
ofOption<UnsafeCell<uint>>
with the initializer ofNone
is in theory safe. -
How hard are the envisioned extensions to implement? If easy, they would be nice to have. If hard, they can wait.
- Start Date: 2014-09-22
- RFC PR: rust-lang/rfcs#255
- Rust Issue: rust-lang/rust#17670
Summary
Restrict which traits can be used to make trait objects.
Currently, we allow any traits to be used for trait objects, but restrict the methods which can be called on such objects. Here, we propose instead restricting which traits can be used to make objects. Despite being less flexible, this will make for better error messages, less surprising software evolution, and (hopefully) better design. The motivation for the proposed change is stronger due to part of the DST changes.
Motivation
Part of the planned, in progress DST work is to allow trait objects where a trait is expected. Example:
fn foo<Sized? T: SomeTrait>(y: &T) { ... }
fn bar(x: &SomeTrait) {
foo(x)
}
Previous to DST the call to foo
was not expected to work because SomeTrait
was not a type, so it could not instantiate T
. With DST this is possible, and
it makes intuitive sense for this to work (an alternative is to require impl SomeTrait for SomeTrait { ... }
, but that seems weird and confusing and rather
like boilerplate. Note that the precise mechanism here is out of scope for this
RFC).
This is only sound if the trait is object-safe. We say a method m
on trait
T
is object-safe if it is legal (in current Rust) to call x.m(...)
where x
has type &T
, i.e., x
is a trait object. If all methods in T
are object-safe,
then we say T
is object-safe.
If we ignore this restriction we could allow code such as the following:
trait SomeTrait {
fn foo(&self, other: &Self) { ... } // assume self and other have the same concrete type
}
fn bar<Sized? T: SomeTrait>(x: &T, y: &T) {
x.foo(y); // x and y may have different concrete types, pre-DST we could
// assume that x and y had the same concrete types.
}
fn baz(x: &SomeTrait, y: &SomeTrait) {
bar(x, y) // x and y may have different concrete types
}
This RFC proposes enforcing object-safety when trait objects are created, rather than where methods on a trait object are called or where we attempt to match traits. This makes both method call and using trait objects with generic code simpler. The downside is that it makes Rust less flexible, since not all traits can be used to create trait objects.
Software evolution is improved with this proposal: imagine adding a non-object-safe method to a previously object-safe trait. With this proposal, you would then get errors wherever a trait-object is created. The error would explain why the trait object could not be created and point out exactly which method was to blame and why. Without this proposal, the only errors you would get would be where a trait object is used with a generic call and would be something like “type error: SomeTrait does not implement SomeTrait” - no indication that the non-object-safe method were to blame, only a failure in trait matching.
Another advantage of this proposal is that it implies that all method-calls can always be rewritten into an equivalent UFCS call. This simplifies the “core language” and makes method dispatch notation – which involves some non-trivial inference – into a kind of “sugar” for the more explicit UFCS notation.
Detailed design
To be precise about object-safety, an object-safe method must meet one of the following conditions:
- require
Self : Sized
; or, - meet all of the following conditions:
- must not have any type parameters; and,
- must have a receiver that has type
Self
or which dereferences to theSelf
type;- for now, this means
self
,&self
,&mut self
, orself: Box<Self>
, but eventually this should be extended to custom types likeself: Rc<Self>
and so forth.
- for now, this means
- must not use
Self
(in the future, where we allow arbitrary types for the receiver,Self
may only be used for the type of the receiver and only where we allowSized?
types).
A trait is object-safe if all of the following conditions hold:
- all of its methods are object-safe; and,
- the trait does not require that
Self : Sized
(see also RFC 546).
When an expression with pointer-to-concrete type is coerced to a trait object, the compiler will check that the trait is object-safe (in addition to the usual check that the concrete type implements the trait). It is an error for the trait to be non-object-safe.
Note that a trait can be object-safe even if some of its methods use
features that are not supported with an object receiver. This is true
when code that attempted to use those features would only work if the
Self
type is Sized
. This is why all methods that require
Self:Sized
are exempt from the typical rules. This is also why
by-value self methods are permitted, since currently one cannot invoke
pass an unsized type by-value (though we consider that a useful future
extension).
Drawbacks
This is a breaking change and forbids some safe code which is legal
today. This can be addressed in two ways: splitting traits, or adding
where Self:Sized
clauses to methods that cannot not be used with
objects.
Example problem
Here is an example trait that is not object safe:
trait SomeTrait {
fn foo(&self) -> int { ... }
// Object-safe methods may not return `Self`:
fn new() -> Self;
}
Splitting a trait
One option is to split a trait into object-safe and non-object-safe parts. We hope that this will lead to better design. We are not sure how much code this will affect, it would be good to have data about this.
trait SomeTrait {
fn foo(&self) -> int { ... }
}
trait SomeTraitCtor : SomeTrait {
fn new() -> Self;
}
Adding a where-clause
Sometimes adding a second trait feels like overkill. In that case, it
is often an option to simply add a where Self:Sized
clause to the
methods of the trait that would otherwise violate the object safety
rule.
trait SomeTrait {
fn foo(&self) -> int { ... }
fn new() -> Self
where Self : Sized; // this condition is new
}
The reason that this makes sense is that if one were writing a generic
function with a type parameter T
that may range over the trait
object, that type parameter would have to be declared ?Sized
, and
hence would not have access to the new
method:
fn baz<T:?Sized+SomeTrait>(t: &T) {
let v: T = SomeTrait::new(); // illegal because `T : Sized` is not known to hold
}
However, if one writes a function with sized type parameter, which
could never be a trait object, then the new
function becomes
available.
fn baz<T:SomeTrait>(t: &T) {
let v: T = SomeTrait::new(); // OK
}
Alternatives
We could continue to check methods rather than traits are object-safe. When checking the bounds of a type parameter for a function call where the function is called with a trait object, we would check that all methods are object-safe as part of the check that the actual type parameter satisfies the formal bounds. We could probably give a different error message if the bounds are met, but the trait is not object-safe.
We might in the future use finer-grained reasoning to permit more
non-object-safe methods from appearing in the trait. For example, we
might permit fn foo() -> Self
because it (implicitly) requires that
Self
be sized. Similarly, we might permit other tests beyond just
sized-ness. Any such extension would be backwards compatible.
Unresolved questions
N/A
Edits
- 2014-02-09. Edited by Nicholas Matsakis to (1) include the
requirement that object-safe traits do not require
Self:Sized
and (2) specify that methods may includewhere Self:Sized
to overcome object safety restrictions.
- Start Date: 2014-09-19
- RFC PR: rust-lang/rfcs#256
- Rust Issue: https://github.com/rust-lang/rfcs/pull/256
Summary
Remove the reference-counting based Gc<T>
type from the standard
library and its associated support infrastructure from rustc
.
Doing so lays a cleaner foundation upon which to prototype a proper tracing GC, and will avoid people getting incorrect impressions of Rust based on the current reference-counting implementation.
Motivation
Ancient History
Long ago, the Rust language had integrated support for automatically
managed memory with arbitrary graph structure (notably, multiple
references to the same object), via the type constructors @T
and
@mut T
for any T
. The intention was that Rust would provide a
task-local garbage collector as part of the standard runtime for Rust
programs.
As a short-term convenience, @T
and @mut T
were implemented via
reference-counting: each instance of @T
/@mut T
had a reference
count added to it (as well as other meta-data that were again for
implementation convenience). To support this, the rustc
compiler
would emit, for any instruction copying or overwriting an instance of
@T
/@mut T
, code to update the reference count(s) accordingly.
(At the same time, @T
was still considered an instance of Copy
by
the compiler. Maintaining the reference counts of @T
means that you
cannot create copies of a given type implementing Copy
by
memcpy
’ing blindly; one must distinguish so-called “POD” data that
is Copy and contains no
@Tfrom "non-POD"
Copydata that can contain
@T` and thus must be sure to update reference counts when
creating a copy.)
Over time, @T
was replaced with the library type Gc<T>
(and @mut T
was rewritten as Gc<RefCell<T>>
), but the intention was that Rust
would still have integrated support for a garbage collection. To
continue supporting the reference-count updating semantics, the
Gc<T>
type has a lang item, "gc"
. In effect, all of the compiler
support for maintaining the reference-counts from the prior @T
was
still in place; the move to a library type Gc<T>
was just a shift in
perspective from the end-user’s point of view (and that of the
parser).
Recent history: Removing uses of Gc from the compiler
Largely due to the tireless efforts of eddyb
, one of the primary
clients of Gc<T>
, namely the rustc
compiler itself, has little to
no remaining uses of Gc<T>
.
A new hope
This means that we have an opportunity now, to remove the Gc<T>
type
from libstd
, and its associated built-in reference-counting support
from rustc
itself.
I want to distinguish removal of the particular reference counting
Gc<T>
from our compiler and standard library (which is what is being
proposed here), from removing the goal of supporting a garbage
collected Gc<T>
in the future. I (and I think the majority of the
Rust core team) still believe that there are use cases that would be
well handled by a proper tracing garbage collector.
The expected outcome of removing reference-counting Gc<T>
are as follows:
-
A cleaner compiler code base,
-
A cleaner standard library, where
Copy
data can be indeed copied blindly (assuming the source and target types are in agreement, which is required for a tracing GC), -
It would become impossible for users to use
Gc<T>
and then get incorrect impressions about how Rust’s GC would behave in the future. In particular, if we leave the reference-countingGc<T>
in place, then users may end up depending on implementation artifacts that we would be pressured to continue supporting in the future. (Note thatGc<T>
is already marked “experimental”, so this particular motivation is not very strong.)
Detailed design
Remove the std::gc
module. This, I believe, is the extent of the
end-user visible changes proposed by this RFC, at least for users who
are using libstd
(as opposed to implementing their own).
Then remove the rustc
support for Gc<T>
. As part of this, we can
either leave in or remove the "gc"
and "managed_heap"
entries in
the lang items table (in case they could be of use for a future GC
implementation). I propose leaving them, but it does not matter
terribly to me. The important thing is that once std::gc
is gone,
then we can remove the support code associated with those two lang
items, which is the important thing.
Drawbacks
Taking out the reference-counting Gc<T>
now may lead people to think
that Rust will never have a Gc<T>
.
-
In particular, having
Gc<T>
in place now means that it is easier to argue for putting in a tracing collector (since it would be a net win over the status quo, assuming it works).(This sub-bullet is a bit of a straw man argument, as I suspect any community resistance to adding a tracing GC will probably be unaffected by the presence or absence of the reference-counting
Gc<T>
.) -
As another related note, it may confuse people to take out a
Gc<T>
type now only to add another implementation with the same name later. (Of course, is that more or less confusing than just replacing the underlying implementation in such a severe manner.)
Users may be using Gc<T>
today, and they would have to switch to
some other option (such as Rc<T>
, though note that the two are not
100% equivalent; see [Gc versus Rc] appendix).
Alternatives
Keep the Gc<T>
implementation that we have today, and wait until we
have a tracing GC implemented and ready to be deployed before removing
the reference-counting infrastructure that had been put in to support
@T
. (Which may never happen, since adding a tracing GC is only a
goal, not a certainty, and thus we may be stuck supporting the
reference-counting Gc<T>
until we eventually do decide to remove
Gc<T>
in the future. So this RFC is just suggesting we be proactive
and pull that band-aid off now.
Unresolved questions
None yet.
Appendices
Gc versus Rc
There are performance differences between the current ref-counting
Gc<T>
and the library type Rc<T>
, but such differences are beneath
the level of abstraction of interest to this RFC. The main user
observable difference between the ref-counting Gc<T>
and the library
type Rc<T>
is that cyclic structure allocated via Gc<T>
will be
torn down when the task itself terminates successfully or via unwind.
The following program illustrates this difference. If you have a
program that is using Gc
and is relying on this tear-down behavior
at task death, then switching to Rc
will not suffice.
use std::cell::RefCell;
use std::gc::{GC,Gc};
use std::io::timer;
use std::rc::Rc;
use std::time::Duration;
struct AnnounceDrop { name: String }
#[allow(non_snake_case)]
fn AnnounceDrop<S:Str>(s:S) -> AnnounceDrop {
AnnounceDrop { name: s.as_slice().to_string() }
}
impl Drop for AnnounceDrop{
fn drop(&mut self) {
println!("dropping {}", self.name);
}
}
struct RcCyclic<D> { _on_drop: D, recur: Option<Rc<RefCell<RcCyclic<D>>>> }
struct GcCyclic<D> { _on_drop: D, recur: Option<Gc<RefCell<GcCyclic<D>>>> }
type RRRcell<D> = Rc<RefCell<RcCyclic<D>>>;
type GRRcell<D> = Gc<RefCell<GcCyclic<D>>>;
fn make_rc_and_gc<S:Str>(name: S) -> (RRRcell<AnnounceDrop>, GRRcell<AnnounceDrop>) {
let name = name.as_slice().to_string();
let rc_cyclic = Rc::new(RefCell::new(RcCyclic {
_on_drop: AnnounceDrop(name.clone().append("-rc")),
recur: None,
}));
let gc_cyclic = box (GC) RefCell::new(GcCyclic {
_on_drop: AnnounceDrop(name.append("-gc")),
recur: None,
});
(rc_cyclic, gc_cyclic)
}
fn make_proc(name: &str, sleep_time: i64, and_then: proc():Send) -> proc():Send {
let name = name.to_string();
proc() {
let (rc_cyclic, gc_cyclic) = make_rc_and_gc(name);
rc_cyclic.borrow_mut().recur = Some(rc_cyclic.clone());
gc_cyclic.borrow_mut().recur = Some(gc_cyclic);
timer::sleep(Duration::seconds(sleep_time));
and_then();
}
}
fn main() {
let (_rc_noncyclic, _gc_noncyclic) = make_rc_and_gc("main-noncyclic");
spawn(make_proc("success-cyclic", 2, proc () {}));
spawn(make_proc("failure-cyclic", 1, proc () { fail!("Oop"); }));
println!("Hello, world!")
}
The above program produces output as follows:
% rustc gc-vs-rc-sample.rs && ./gc-vs-rc-sample
Hello, world!
dropping main-noncyclic-gc
dropping main-noncyclic-rc
task '<unnamed>' failed at 'Oop', gc-vs-rc-sample.rs:60
dropping failure-cyclic-gc
dropping success-cyclic-gc
This illustrates that both Gc<T>
and Rc<T>
will be reclaimed when
used to represent non-cyclic data (the cases labelled
main-noncyclic-gc
and main-noncyclic-rc
. But when you actually
complete the cyclic structure, then in the tasks that run to
completion (either successfully or unwinding from a failure), we still
manage to drop the Gc<T>
cyclic structures, illustrated by the
printouts from the cases labelled failure-cyclic-gc
and
success-cyclic-gc
.
- Feature Name: (none for the bulk of RFC); unsafe_no_drop_flag
- Start Date: 2014-09-24
- RFC PR: rust-lang/rfcs#320
- Rust Issue: rust-lang/rust#5016
Summary
Remove drop flags from values implementing Drop
, and remove
automatic memory zeroing associated with dropping values.
Keep dynamic drop semantics, by having each function maintain a (potentially empty) set of auto-injected boolean flags for the drop obligations for the function that need to be tracked dynamically (which we will call “dynamic drop obligations”).
Motivation
Currently, implementing Drop
on a struct (or enum) injects a hidden
bit, known as the “drop-flag”, into the struct (and likewise, each of
the enum variants). The drop-flag, in tandem with Rust’s implicit
zeroing of dropped values, tracks whether a value has already been
moved to another owner or been dropped. (See the “How dynamic drop
semantics works” appendix for more
details if you are unfamiliar with this part of Rust’s current
implementation.)
However, the above implementation is sub-optimal; problems include:
-
Most important: implicit memory zeroing is a hidden cost that today all Rust programs pay, in both execution time and code size. With the removal of the drop flag, we can remove implicit memory zeroing (or at least revisit its utility – there may be other motivations for implicit memory zeroing, e.g. to try to keep secret data from being exposed to unsafe code).
-
Hidden bits are bad: Users coming from a C/C++ background expect
struct Foo { x: u32, y: u32 }
to occupy 8 bytes, but ifFoo
implementsDrop
, the hidden drop flag will cause it to double in size (16 bytes). See the [Program illustrating semantic impact of hidden drop flag] appendix for a concrete illustration. Note that whenFoo
implementsDrop
, each instance ofFoo
carries a drop-flag, even in contexts like aVec<Foo>
where a program cannot actually move individual values out of the collection. Thus, the amount of extra memory being used by drop-flags is not bounded by program stack growth; the memory wastage is strewn throughout the heap.
An earlier RFC (the withdrawn RFC PR #210) suggested resolving this problem by switching from a dynamic drop semantics to a “static drop semantics”, which was defined in that RFC as one that performs drop of certain values earlier to ensure that the set of drop-obligations does not differ at any control-flow merge point, i.e. to ensure that the set of values to drop is statically known at compile-time.
However, discussion on the RFC PR #210 comment thread pointed out
its policy for inserting early drops into the code is non-intuitive
(in other words, that the drop policy should either be more
aggressive, a la RFC PR #239, or should stay with the dynamic drop
status quo). Also, the mitigating mechanisms proposed by that RFC
(NoisyDrop
/QuietDrop
) were deemed unacceptable.
So, static drop semantics are a non-starter. Luckily, the above strategy is not the only way to implement dynamic drop semantics. Rather than requiring that the set of drop-obligations be the same at every control-flow merge point, we can do a intra-procedural static analysis to identify the set of drop-obligations that differ at any merge point, and then inject a set of stack-local boolean-valued drop-flags that dynamically track them. That strategy is what this RFC is describing.
The expected outcomes are as follows:
-
We remove the drop-flags from all structs/enums that implement
Drop
. (There are still the injected stack-local drop flags, but those should be cheaper to inject and maintain.) -
Since invoking drop code is now handled by the stack-local drop flags and we have no more drop-flags on the values themselves, we can (and will) remove memory zeroing.
-
Libraries currently relying on drop doing memory zeroing (i.e. libraries that check whether content is zero to decide whether its
fn drop
has been invoked will need to be revised, since we will not have implicit memory zeroing anymore. -
In the common case, most libraries using
Drop
will not need to change at all from today, apart from the caveat in the previous bullet.
Detailed design
Drop obligations
No struct or enum has an implicit drop-flag. When a local variable is
initialized, that establishes a set of “drop obligations”: a set of
structural paths (e.g. a local a
, or a path to a field b.f.y
) that
need to be dropped (or moved away to a new owner).
The drop obligations for a local variable x
of struct-type T
are
computed from analyzing the structure of T
. If T
itself
implements Drop
, then x
is a drop obligation. If T
does not
implement Drop
, then the set of drop obligations is the union of the
drop obligations of the fields of T
.
When a path is moved to a new location, or consumed by a function call, or when control flow reaches the end of its owner’s lexical scope, the path is removed from the set of drop obligations.
At control-flow merge points, e.g. nodes that have predecessor nodes P_1, P_2, …, P_k with drop obligation sets S_1, S_2, … S_k, we
-
First identify the set of drop obligations that differ between the predecessor nodes, i.e. the set:
(S_1 | S_2 | ... | S_k) \ (S_1 & S_2 & ... & S_k)
where
|
denotes set-union,&
denotes set-intersection,\
denotes set-difference. These are the dynamic drop obligations induced by this merge point. Note that ifS_1 = S_2 = ... = S_k
, the above set is empty. -
The set of drop obligations for the merge point itself is the union of the drop-obligations from all predecessor points in the control flow, i.e.
(S_1 | S_2 | ... | S_k)
in the above notation.(One could also just use the intersection here; it actually makes no difference to the static analysis, since all of the elements of the difference
(S_1 | S_2 | ... | S_k) \ (S_1 & S_2 & ... & S_k)
have already been added to the set of dynamic drop obligations. But the overall code transformation is clearer if one keeps the dynamic drop obligations in the set of drop obligations.)
Stack-local drop flags
For every dynamic drop obligation induced by a merge point, the compiler is responsible for ensure that its drop code is run at some point. If necessary, it will inject and maintain boolean flag analogous to
enum NeedsDropFlag { NeedsLocalDrop, DoNotDrop }
Some compiler analysis may be able to identify dynamic drop obligations that do not actually need to be tracked. Therefore, we do not specify the precise set of boolean flags that are injected.
Example of code with dynamic drop obligations
The function f2
below was copied from the static drop RFC PR #210;
it has differing sets of drop obligations at a merge point,
necessitating a potential injection of a NeedsDropFlag
.
fn f2() {
// At the outset, the set of drop obligations is
// just the set of moved input parameters (empty
// in this case).
// DROP OBLIGATIONS
// ------------------------
// { }
let pDD : Pair<D,D> = ...;
pDD.x = ...;
// {pDD.x}
pDD.y = ...;
// {pDD.x, pDD.y}
let pDS : Pair<D,S> = ...;
// {pDD.x, pDD.y, pDS.x}
let some_d : Option<D>;
// {pDD.x, pDD.y, pDS.x}
if test() {
// {pDD.x, pDD.y, pDS.x}
{
let temp = xform(pDD.y);
// {pDD.x, pDS.x, temp}
some_d = Some(temp);
// {pDD.x, pDS.x, temp, some_d}
} // END OF SCOPE for `temp`
// {pDD.x, pDS.x, some_d}
// MERGE POINT PREDECESSOR 1
} else {
{
// {pDD.x, pDD.y, pDS.x}
let z = D;
// {pDD.x, pDD.y, pDS.x, z}
// This drops `pDD.y` before
// moving `pDD.x` there.
pDD.y = pDD.x;
// { pDD.y, pDS.x, z}
some_d = None;
// { pDD.y, pDS.x, z, some_d}
} // END OF SCOPE for `z`
// { pDD.y, pDS.x, some_d}
// MERGE POINT PREDECESSOR 2
}
// MERGE POINT: set of drop obligations do not
// match on all incoming control-flow paths.
//
// Predecessor 1 has drop obligations
// {pDD.x, pDS.x, some_d}
// and Predecessor 2 has drop obligations
// { pDD.y, pDS.x, some_d}.
//
// Therefore, this merge point implies that
// {pDD.x, pDD.y} are dynamic drop obligations,
// while {pDS.x, some_d} are potentially still
// resolvable statically (and thus may not need
// associated boolean flags).
// The resulting drop obligations are the following:
// {pDD.x, pDD.y, pDS.x, some_d}.
// (... some code that does not change drop obligations ...)
// {pDD.x, pDD.y, pDS.x, some_d}.
// END OF SCOPE for `pDD`, `pDS`, `some_d`
}
After the static analysis has identified all of the dynamic drop
obligations, code is injected to maintain the stack-local drop flags
and to do any necessary drops at the appropriate points.
Below is the updated fn f2
with an approximation of the injected code.
Note: we say “approximation”, because one does need to ensure that the
drop flags are updated in a manner that is compatible with potential
task fail!
/panic!
, because stack unwinding must be informed which
state needs to be dropped; i.e. you need to initialize _pDD_dot_x
before you start to evaluate a fallible expression to initialize
pDD.y
.
fn f2_rewritten() {
// At the outset, the set of drop obligations is
// just the set of moved input parameters (empty
// in this case).
// DROP OBLIGATIONS
// ------------------------
// { }
let _drop_pDD_dot_x : NeedsDropFlag;
let _drop_pDD_dot_y : NeedsDropFlag;
_drop_pDD_dot_x = DoNotDrop;
_drop_pDD_dot_y = DoNotDrop;
let pDD : Pair<D,D>;
pDD.x = ...;
_drop_pDD_dot_x = NeedsLocalDrop;
pDD.y = ...;
_drop_pDD_dot_y = NeedsLocalDrop;
// {pDD.x, pDD.y}
let pDS : Pair<D,S> = ...;
// {pDD.x, pDD.y, pDS.x}
let some_d : Option<D>;
// {pDD.x, pDD.y, pDS.x}
if test() {
// {pDD.x, pDD.y, pDS.x}
{
_drop_pDD_dot_y = DoNotDrop;
let temp = xform(pDD.y);
// {pDD.x, pDS.x, temp}
some_d = Some(temp);
// {pDD.x, pDS.x, temp, some_d}
} // END OF SCOPE for `temp`
// {pDD.x, pDS.x, some_d}
// MERGE POINT PREDECESSOR 1
} else {
{
// {pDD.x, pDD.y, pDS.x}
let z = D;
// {pDD.x, pDD.y, pDS.x, z}
// This drops `pDD.y` before
// moving `pDD.x` there.
_drop_pDD_dot_x = DoNotDrop;
pDD.y = pDD.x;
// { pDD.y, pDS.x, z}
some_d = None;
// { pDD.y, pDS.x, z, some_d}
} // END OF SCOPE for `z`
// { pDD.y, pDS.x, some_d}
// MERGE POINT PREDECESSOR 2
}
// MERGE POINT: set of drop obligations do not
// match on all incoming control-flow paths.
//
// Predecessor 1 has drop obligations
// {pDD.x, pDS.x, some_d}
// and Predecessor 2 has drop obligations
// { pDD.y, pDS.x, some_d}.
//
// Therefore, this merge point implies that
// {pDD.x, pDD.y} are dynamic drop obligations,
// while {pDS.x, some_d} are potentially still
// resolvable statically (and thus may not need
// associated boolean flags).
// The resulting drop obligations are the following:
// {pDD.x, pDD.y, pDS.x, some_d}.
// (... some code that does not change drop obligations ...)
// {pDD.x, pDD.y, pDS.x, some_d}.
// END OF SCOPE for `pDD`, `pDS`, `some_d`
// rustc-inserted code (not legal Rust, since `pDD.x` and `pDD.y`
// are inaccessible).
if _drop_pDD_dot_x { mem::drop(pDD.x); }
if _drop_pDD_dot_y { mem::drop(pDD.y); }
}
Note that in a snippet like
_drop_pDD_dot_y = DoNotDrop;
let temp = xform(pDD.y);
this is okay, in part because the evaluating the identifier xform
is
infallible. If instead it were something like:
_drop_pDD_dot_y = DoNotDrop;
let temp = lookup_closure()(pDD.y);
then that would not be correct, because we need to set
_drop_pDD_dot_y
to DoNotDrop
after the lookup_closure()
invocation.
It may probably be more intellectually honest to write the transformation like:
let temp = lookup_closure()({ _drop_pDD_dot_y = DoNotDrop; pDD.y });
Control-flow sensitivity
Note that the dynamic drop obligations are based on a control-flow analysis, not just the lexical nesting structure of the code.
In particular: If control flow splits at a point like an if-expression, but the two arms never meet, then they can have completely sets of drop obligations.
This is important, since in coding patterns like loops, one
often sees different sets of drop obligations prior to a break
compared to a point where the loop repeats, such as a continue
or the end of a loop
block.
// At the outset, the set of drop obligations is
// just the set of moved input parameters (empty
// in this case).
// DROP OBLIGATIONS
// ------------------------
// { }
let mut pDD : Pair<D,D> = mk_dd();
let mut maybe_set : D;
// { pDD.x, pDD.y }
'a: loop {
// MERGE POINT
// { pDD.x, pDD.y }
if test() {
// { pDD.x, pDD.y }
consume(pDD.x);
// { pDD.y }
break 'a;
}
// *not* merge point (only one path, the else branch, flows here)
// { pDD.x, pDD.y }
// never falls through; must merge with 'a loop.
}
// RESUME POINT: break 'a above flows here
// { pDD.y }
// This is the point immediately preceding `'b: loop`; (1.) below.
'b: loop {
// MERGE POINT
//
// There are *three* incoming paths: (1.) the statement
// preceding `'b: loop`, (2.) the `continue 'b;` below, and
// (3.) the end of the loop's block below. The drop
// obligation for `maybe_set` originates from (3.).
// { pDD.y, maybe_set }
consume(pDD.y);
// { , maybe_set }
if test() {
// { , maybe_set }
pDD.x = mk_d();
// { pDD.x , maybe_set }
break 'b;
}
// *not* merge point (only one path flows here)
// { , maybe_set }
if test() {
// { , maybe_set }
pDD.y = mk_d();
// This is (2.) referenced above. { pDD.y, maybe_set }
continue 'b;
}
// *not* merge point (only one path flows here)
// { , maybe_set }
pDD.y = mk_d();
// This is (3.) referenced above. { pDD.y, maybe_set }
maybe_set = mk_d();
g(&maybe_set);
// This is (3.) referenced above. { pDD.y, maybe_set }
}
// RESUME POINT: break 'b above flows here
// { pDD.x , maybe_set }
// when we hit the end of the scope of `maybe_set`;
// check its stack-local flag.
Likewise, a return
statement represents another control flow jump,
to the end of the function.
Remove implicit memory zeroing
With the above in place, the remainder is relatively trivial.
The compiler can be revised to no longer inject a drop flag into
structs and enums that implement Drop
, and likewise memory zeroing can
be removed.
Beyond that, the libraries will obviously need to be audited for dependence on implicit memory zeroing.
Drawbacks
The only reasons not do this are:
-
Some hypothetical reason to continue doing implicit memory zeroing, or
-
We want to abandon dynamic drop semantics.
At this point Felix thinks the Rust community has made a strong argument in favor of keeping dynamic drop semantics.
Alternatives
-
Static drop semantics RFC PR #210 has been referenced frequently in this document.
-
Eager drops RFC PR #239 is the more aggressive semantics that would drop values immediately after their final use. This would probably invalidate a number of RAII style coding patterns.
Optional Extensions
A lint identifying dynamic drop obligations
Add a lint (set by default to allow
) that reports potential dynamic
drop obligations, so that end-user code can opt-in to having them
reported. The expected benefits of this are:
-
developers may have intended for a value to be moved elsewhere on all paths within a function, and,
-
developers may want to know about how many boolean dynamic drop flags are potentially being injected into their code.
Unresolved questions
How to handle moves out of array[index_expr]
Niko pointed out to me today that my prototype was not addressing
moves out of array[index_expr]
properly. I was assuming
that we would just make such an expression illegal (or that they
should already be illegal).
But they are not already illegal, and above assumption that we would make it illegal should have been explicit. That, or we should address the problem in some other way.
To make this concrete, here is some code that runs today:
#[deriving(Show)]
struct AnnounceDrop { name: &'static str }
impl Drop for AnnounceDrop {
fn drop(&mut self) { println!("dropping {}", self.name); }
}
fn foo<A>(a: [A, ..3], i: uint) -> A {
a[i]
}
fn main() {
let a = [AnnounceDrop { name: "fst" },
AnnounceDrop { name: "snd" },
AnnounceDrop { name: "thd" }];
let r = foo(a, 1);
println!("foo returned {}", r);
}
This prints:
dropping fst
dropping thd
foo returned AnnounceDrop { name: snd }
dropping snd
because it first moves the entire array into foo
, and then foo
returns the second element, but still needs to drop the rest of the
array.
Embedded drop flags and zeroing support this seamlessly, of course. But the whole point of this RFC is to get rid of the embedded per-value drop-flags.
If we want to continue supporting moving out of a[i]
(and we
probably do, I have been converted on this point), then the drop flag
needs to handle this case. Our current thinking is that we can
support it by using a single uint
flag (as opposed to the booleans
used elsewhere) for such array that has been moved out of. The uint
flag represents “drop all elements from the array except for the one
listed in the flag.” (If it is only moved out of on one branch and
not another, then we would either use an Option<uint>
, or still use
uint
and just represent unmoved case via some value that is not
valid index, such as the length of the array).
Should we keep #[unsafe_no_drop_flag]
?
Currently there is an unsafe_no_drop_flag
attribute that is used to
indicate that no drop flag should be associated with a struct/enum,
and instead the user-written drop code will be run multiple times (and
thus must internally guard itself from its own side-effects; e.g. do
not attempt to free the backing buffer for a Vec
more than once, by
tracking within the Vec
itself if the buffer was previously freed).
The “obvious” thing to do is to remove unsafe_no_drop_flag
, since
the per-value drop flag is going away. However, we could keep the
attribute, and just repurpose its meaning to instead mean the
following: Never inject a dynamic stack-local drop-flag for this
value. Just run the drop code multiple times, just like today.
In any case, since the semantics of this attribute are unstable, we
will feature-gate it (with feature name unsafe_no_drop_flag
).
Appendices
How dynamic drop semantics works
(This section is just presenting background information on the
semantics of drop
and the drop-flag as it works in Rust today; it
does not contain any discussion of the changes being proposed by this
RFC.)
A struct or enum implementing Drop
will have its drop-flag
automatically set to a non-zero value when it is constructed. When
attempting to drop the struct or enum (i.e. when control reaches the
end of the lexical scope of its owner), the injected glue code will
only execute its associated fn drop
if its drop-flag is non-zero.
In addition, the compiler injects code to ensure that when a value is moved to a new location in memory or dropped, then the original memory is entirely zeroed.
A struct/enum definition implementing Drop
can be tagged with the
attribute #[unsafe_no_drop_flag]
. When so tagged, the struct/enum
will not have a hidden drop flag embedded within it. In this case, the
injected glue code will execute the associated glue code
unconditionally, even though the struct/enum value may have been moved
to a new location in memory or dropped (in either case, the memory
representing the value will have been zeroed).
The above has a number of implications:
-
A program can manually cause the drop code associated with a value to be skipped by first zeroing out its memory.
-
A
Drop
implementation for a struct tagged withunsafe_no_drop_flag
must assume that it will be called more than once. (However, every call todrop
after the first will be given zeroed memory.)
Program illustrating semantic impact of hidden drop flag
#![feature(macro_rules)]
use std::fmt;
use std::mem;
#[deriving(Clone,Show)]
struct S { name: &'static str }
#[deriving(Clone,Show)]
struct Df { name: &'static str }
#[deriving(Clone,Show)]
struct Pair<X,Y>{ x: X, y: Y }
static mut current_indent: uint = 0;
fn indent() -> String {
String::from_char(unsafe { current_indent }, ' ')
}
impl Drop for Df {
fn drop(&mut self) {
println!("{}dropping Df {}", indent(), self.name)
}
}
macro_rules! struct_Dn {
($Dn:ident) => {
#[unsafe_no_drop_flag]
#[deriving(Clone,Show)]
struct $Dn { name: &'static str }
impl Drop for $Dn {
fn drop(&mut self) {
if unsafe { (0,0) == mem::transmute::<_,(uint,uint)>(self.name) } {
println!("{}dropping already-zeroed {}",
indent(), stringify!($Dn));
} else {
println!("{}dropping {} {}",
indent(), stringify!($Dn), self.name)
}
}
}
}
}
struct_Dn!(DnA)
struct_Dn!(DnB)
struct_Dn!(DnC)
fn take_and_pass<T:fmt::Show>(t: T) {
println!("{}t-n-p took and will pass: {}", indent(), &t);
unsafe { current_indent += 4; }
take_and_drop(t);
unsafe { current_indent -= 4; }
}
fn take_and_drop<T:fmt::Show>(t: T) {
println!("{}t-n-d took and will drop: {}", indent(), &t);
}
fn xform(mut input: Df) -> Df {
input.name = "transformed";
input
}
fn foo(b: || -> bool) {
let mut f1 = Df { name: "f1" };
let mut n2 = DnC { name: "n2" };
let f3 = Df { name: "f3" };
let f4 = Df { name: "f4" };
let f5 = Df { name: "f5" };
let f6 = Df { name: "f6" };
let n7 = DnA { name: "n7" };
let _fx = xform(f6); // `f6` consumed by `xform`
let _n9 = DnB { name: "n9" };
let p = Pair { x: f4, y: f5 }; // `f4` and `f5` moved into `p`
let _f10 = Df { name: "f10" };
println!("foo scope start: {}", (&f3, &n7));
unsafe { current_indent += 4; }
if b() {
take_and_pass(p.x); // `p.x` consumed by `take_and_pass`, which drops it
}
if b() {
take_and_pass(n7); // `n7` consumed by `take_and_pass`, which drops it
}
// totally unsafe: manually zero the struct, including its drop flag.
unsafe fn manually_zero<S>(s: &mut S) {
let len = mem::size_of::<S>();
let p : *mut u8 = mem::transmute(s);
for i in range(0, len) {
*p.offset(i as int) = 0;
}
}
unsafe {
manually_zero(&mut f1);
manually_zero(&mut n2);
}
println!("foo scope end");
unsafe { current_indent -= 4; }
// here, we drop each local variable, in reverse order of declaration.
// So we should see the following drop sequence:
// drop(f10), printing "Df f10"
// drop(p)
// ==> drop(p.y), printing "Df f5"
// ==> attempt to drop(and skip) already-dropped p.x, no-op
// drop(_n9), printing "DnB n9"
// drop(_fx), printing "Df transformed"
// attempt to drop already-dropped n7, printing "already-zeroed DnA"
// no drop of `f6` since it was consumed by `xform`
// no drop of `f5` since it was moved into `p`
// no drop of `f4` since it was moved into `p`
// drop(f3), printing "f3"
// attempt to drop manually-zeroed `n2`, printing "already-zeroed DnC"
// attempt to drop manually-zeroed `f1`, no-op.
}
fn main() {
foo(|| true);
}
- Start Date: 2014-09-26
- RFC PR: rust-lang/rfcs#326
- Rust Issue: rust-lang/rust#18062
Summary
In string literal contexts, restrict \xXX
escape sequences to just
the range of ASCII characters, \x00
– \x7F
. \xXX
inputs in
string literals with higher numbers are rejected (with an error
message suggesting that one use an \uNNNN
escape).
Motivation
In a string literal context, the current \xXX
character escape
sequence is potentially confusing when given inputs greater than
0x7F
, because it does not encode that byte literally, but instead
encodes whatever the escape sequence \u00XX
would produce.
Thus, for inputs greater than 0x7F
, \xXX
will encode multiple
bytes into the generated string literal, as illustrated in the
Rust example appendix.
This is different from what C/C++ programmers might expect (see Behavior of xXX in C appendix).
(It would not be legal to encode the single byte literally into the string literal, since then the string would not be well-formed UTF-8.)
It has been suggested that the \xXX
character escape should be
removed entirely (at least from string literal contexts). This RFC is
taking a slightly less aggressive stance: keep \xXX
, but only for
ASCII inputs when it occurs in string literals. This way, people can
continue using this escape format (which shorter than the \uNNNN
format) when it makes sense.
Here are some links to discussions on this topic, including direct comments that suggest exactly the strategy of this RFC.
- https://github.com/rust-lang/rfcs/issues/312
- https://github.com/rust-lang/rust/issues/12769
- https://github.com/rust-lang/rust/issues/2800#issuecomment-31477259
- https://github.com/rust-lang/rfcs/pull/69#issuecomment-43002505
- https://github.com/rust-lang/rust/issues/12769#issuecomment-43574856
- https://github.com/rust-lang/meeting-minutes/blob/master/weekly-meetings/2014-01-21.md#xnn-escapes-in-strings
- https://mail.mozilla.org/pipermail/rust-dev/2012-July/002025.html
Note in particular the meeting minutes bullet, where the team explicitly decided to keep things “as they are”.
However, at the time of that meeting, Rust did not have byte string
literals; people were converting string-literals into byte arrays via
the bytes!
macro. (Likewise, the rust-dev post is also from a time,
summer 2012, when we did not have byte-string literals.)
We are in a different world now. The fact that now \xXX
denotes a
code unit in a byte-string literal, but in a string literal denotes a
codepoint, does not seem elegant; it rather seems like a source of
confusion. (Caveat: While Felix does believe this assertion, this
context-dependent interpretation of \xXX
does have precedent
in both Python and Racket; see Racket example and Python example
appendices.)
By restricting \xXX
to the range 0x00
–0x7F
, we side-step the
question of “is it a code unit or a code point?” entirely (which was
the real context of both the rust-dev thread and the meeting minutes
bullet). This RFC is a far more conservative choice that we can
safely make for the short term (i.e. for the 1.0 release) than it
would have been to switch to a “\xXX
is a code unit” interpretation.
The expected outcome is reduced confusion for C/C++ programmers (which
is, after all, our primary target audience for conversion), and any
other language where \xXX
never results in more than one byte.
The error message will point them to the syntax they need to adopt.
Detailed design
In string literal contexts, \xXX
inputs with XX > 0x7F
are
rejected (with an error message that mentions either, or both, of
\uNNNN
escapes and the byte-string literal format b".."
).
The full byte range remains supported when \xXX
is used in
byte-string literals, b"..."
Raw strings by design do not offer escape sequences, so they are unchanged.
Character and string escaping routines (such as
core::char::escape_unicode
, and such as used by the "{:?}"
formatter) are updated so that string inputs that previously would
previously have printed \xXX
with XX > 0x7F
are updated to use
\uNNNN
escapes instead.
Drawbacks
Some reasons not to do this:
-
we think that the current behavior is intuitive,
-
it is consistent with language X (and thus has precedent),
-
existing libraries are relying on this behavior, or
-
we want to optimize for inputting characters with codepoints in the range above
0x7F
in string-literals, rather than optimizing for ASCII.
The thesis of this RFC is that the first bullet is a falsehood.
While there is some precedent for the “\xXX
is code point”
interpretation in some languages, the majority do seem to favor the
“\xXX
is code unit” point of view. The proposal of this RFC is
side-stepping the distinction by limiting the input range for \xXX
.
The third bullet is a strawman since we have not yet released 1.0, and thus everything is up for change.
This RFC makes no comment on the validity of the fourth bullet.
Alternatives
-
We could remove
\xXX
entirely from string literals. This would require people to use the\uNNNN
escape format even for bytes in the range00
–0x7F
, which seems annoying. -
We could switch
\xXX
from meaning code point to meaning code unit in both string literal and byte-string literal contexts. This was previously considered and explicitly rejected in an earlier meeting, as discussed in the Motivation section.
Unresolved questions
None.
Appendices
Behavior of xXX in C
Here is a C program illustrating how xXX
escape sequences are treated
in string literals in that context:
#include <stdio.h>
int main() {
char *s;
s = "a";
printf("s[0]: %d\n", s[0]);
printf("s[1]: %d\n", s[1]);
s = "\x61";
printf("s[0]: %d\n", s[0]);
printf("s[1]: %d\n", s[1]);
s = "\x7F";
printf("s[0]: %d\n", s[0]);
printf("s[1]: %d\n", s[1]);
s = "\x80";
printf("s[0]: %d\n", s[0]);
printf("s[1]: %d\n", s[1]);
return 0;
}
Its output is the following:
% gcc example.c && ./a.out
s[0]: 97
s[1]: 0
s[0]: 97
s[1]: 0
s[0]: 127
s[1]: 0
s[0]: -128
s[1]: 0
Rust example
Here is a Rust program that explores the various ways \xXX
sequences are
treated in both string literal and byte-string literal contexts.
#![feature(macro_rules)]
fn main() {
macro_rules! print_str {
($r:expr, $e:expr) => { {
println!("{:>20}: \"{}\"",
format!("\"{}\"", $r),
$e.escape_default())
} }
}
macro_rules! print_bstr {
($r:expr, $e:expr) => { {
println!("{:>20}: {}",
format!("b\"{}\"", $r),
$e)
} }
}
macro_rules! print_bytes {
($r:expr, $e:expr) => {
println!("{:>9}.as_bytes(): {}", format!("\"{}\"", $r), $e.as_bytes())
} }
// println!("{}", b"\u0000"); // invalid: \uNNNN is not a byte escape.
print_str!(r"\0", "\0");
print_bstr!(r"\0", b"\0");
print_bstr!(r"\x00", b"\x00");
print_bytes!(r"\x00", "\x00");
print_bytes!(r"\u0000", "\u0000");
println!("");
print_str!(r"\x61", "\x61");
print_bstr!(r"a", b"a");
print_bstr!(r"\x61", b"\x61");
print_bytes!(r"\x61", "\x61");
print_bytes!(r"\u0061", "\u0061");
println!("");
print_str!(r"\x7F", "\x7F");
print_bstr!(r"\x7F", b"\x7F");
print_bytes!(r"\x7F", "\x7F");
print_bytes!(r"\u007F", "\u007F");
println!("");
print_str!(r"\x80", "\x80");
print_bstr!(r"\x80", b"\x80");
print_bytes!(r"\x80", "\x80");
print_bytes!(r"\u0080", "\u0080");
println!("");
print_str!(r"\xFF", "\xFF");
print_bstr!(r"\xFF", b"\xFF");
print_bytes!(r"\xFF", "\xFF");
print_bytes!(r"\u00FF", "\u00FF");
println!("");
print_str!(r"\u0100", "\u0100");
print_bstr!(r"\x01\x00", b"\x01\x00");
print_bytes!(r"\u0100", "\u0100");
}
In current Rust, it generates output as follows:
% rustc --version && echo && rustc example.rs && ./example
rustc 0.12.0-pre (d52d0c836 2014-09-07 03:36:27 +0000)
"\0": "\x00"
b"\0": [0]
b"\x00": [0]
"\x00".as_bytes(): [0]
"\u0000".as_bytes(): [0]
"\x61": "a"
b"a": [97]
b"\x61": [97]
"\x61".as_bytes(): [97]
"\u0061".as_bytes(): [97]
"\x7F": "\x7f"
b"\x7F": [127]
"\x7F".as_bytes(): [127]
"\u007F".as_bytes(): [127]
"\x80": "\x80"
b"\x80": [128]
"\x80".as_bytes(): [194, 128]
"\u0080".as_bytes(): [194, 128]
"\xFF": "\xff"
b"\xFF": [255]
"\xFF".as_bytes(): [195, 191]
"\u00FF".as_bytes(): [195, 191]
"\u0100": "\u0100"
b"\x01\x00": [1, 0]
"\u0100".as_bytes(): [196, 128]
%
Note that the behavior of \xXX
on byte-string literals matches the
expectations established by the C program in Behavior of xXX in C;
that is good. The problem is the behavior of \xXX
for XX > 0x7F
in string-literal contexts, namely in the fourth and fifth examples
where the .as_bytes()
invocations are showing that the underlying
byte array has two elements instead of one.
Racket example
% racket
Welcome to Racket v5.93.
> (define a-string "\xbb\n")
> (display a-string)
»
> (bytes-length (string->bytes/utf-8 a-string))
3
> (define a-byte-string #"\xc2\xbb\n")
> (bytes-length a-byte-string)
3
> (display a-byte-string)
»
> (exit)
%
The above code illustrates that in Racket, the \xXX
escape sequence
denotes a code unit in byte-string context (#".."
in that language),
while it denotes a code point in string context (".."
).
Python example
% python
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a_string = u"\xbb\n";
>>> print a_string
»
>>> len(a_string.encode("utf-8"))
3
>>> a_byte_string = "\xc2\xbb\n";
>>> len(a_byte_string)
3
>>> print a_byte_string
»
>>> exit()
%
The above code illustrates that in Python, the \xXX
escape sequence
denotes a code unit in byte-string context (".."
in that language),
while it denotes a code point in unicode string context (u".."
).
- Start Date: 2014-09-29
- RFC PR: rust-lang/rfcs#339
- Rust Issue: rust-lang/rust#18465
Summary
Change the types of byte string literals to be references to statically sized types. Ensure the same change can be performed backward compatibly for string literals in the future.
Motivation
Currently byte string and string literals have types &'static [u8]
and &'static str
.
Therefore, although the sizes of the literals are known at compile time, they are erased from their types and inaccessible until runtime.
This RFC suggests to change the type of byte string literals to &'static [u8, ..N]
.
In addition this RFC suggest not to introduce any changes to str
or string literals, that would prevent a backward compatible addition of strings of fixed size FixedString<N>
(the name FixedString in this RFC is a placeholder and is open for bikeshedding) and the change of the type of string literals to &'static FixedString<N>
in the future.
FixedString<N>
is essentially a [u8, ..N]
with UTF-8 invariants and additional string methods/traits.
It fills the gap in the vector/string chart:
Vec<T> | String |
---|---|
[T, ..N] | ??? |
&[T] | &str |
Today, given the lack of non-type generic parameters and compile time (function) evaluation (CTE), strings of fixed size are not very useful. But after introduction of CTE the need in compile time string operations will raise rapidly. Even without CTE but with non-type generic parameters alone fixed size strings can be used in runtime for “heapless” string operations, which are useful in constrained environments or for optimization. So the main motivation for changes today is forward compatibility.
Examples of use for new literals, that are not possible with old literals:
// Today: initialize mutable array with byte string literal
let mut arr: [u8, ..3] = *b"abc";
arr[0] = b'd';
// Future with CTE: compile time string concatenation
static LANG_DIR: FixedString<5 /*The size should, probably, be inferred*/> = *"lang/";
static EN_FILE: FixedString<_> = LANG_DIR + *"en"; // FixedString<N> implements Add
static FR_FILE: FixedString<_> = LANG_DIR + *"fr";
// Future without CTE: runtime "heapless" string concatenation
let DE_FILE = LANG_DIR + *"de"; // Performed at runtime if not optimized
Detailed design
Change the type of byte string literals from &'static [u8]
to &'static [u8, ..N]
.
Leave the door open for a backward compatible change of the type of string literals from &'static str
to &'static FixedString<N>
.
Strings of fixed size
If str
is moved to the library today, then strings of fixed size can be implemented like this:
struct str<Sized? T = [u8]>(T);
Then string literals will have types &'static str<[u8, ..N]>
.
Drawbacks of this approach include unnecessary exposition of the implementation - underlying sized or unsized arrays [u8]
/[u8, ..N]
and generic parameter T
.
The key requirement here is the autocoercion from reference to fixed string to string slice an we are unable to meet it now without exposing the implementation.
In the future, after gaining the ability to parameterize on integers, strings of fixed size could be implemented in a better way:
struct __StrImpl<Sized? T>(T); // private
pub type str = __StrImpl<[u8]>; // unsized referent of string slice `&str`, public
pub type FixedString<const N: uint> = __StrImpl<[u8, ..N]>; // string of fixed size, public
// &FixedString<N> -> &str : OK, including &'static FixedString<N> -> &'static str for string literals
So, we don’t propose to make these changes today and suggest to wait until generic parameterization on integers is added to the language.
Precedents
C and C++ string literals are lvalue char
arrays of fixed size with static duration.
C++ library proposal for strings of fixed size (link), the paper also contains some discussion and motivation.
Rejected alternatives and discussion
Array literals
The types of array literals potentially can be changed from [T, ..N]
to &'a [T, ..N]
for consistency with the other literals and ergonomics.
The major blocker for this change is the inability to move out from a dereferenced array literal if T
is not Copy
.
let mut a = *[box 1i, box 2, box 3]; // Wouldn't work without special-casing of array literals with regard to moving out from dereferenced borrowed pointer
Despite that array literals as references have better usability, possible static
ness and consistency with other literals.
Usage statistics for array literals
Array literals can be used both as slices, when a view to array is sufficient to perform the task, and as values when arrays themselves should be copied or modified.
The exact estimation of the frequencies of both uses is problematic, but some regex search in the Rust codebase gives the next statistics:
In approximately 70% of cases array literals are used as slices (explicit &
on array literals, immutable bindings).
In approximately 20% of cases array literals are used as values (initialization of struct fields, mutable bindings, boxes).
In the rest 10% of cases the usage is unclear.
So, in most cases the change to the types of array literals will lead to shorter notation.
Static lifetime
Although all the literals under consideration are similar and are essentially arrays of fixed size, array literals are different from byte string and string literals with regard to lifetimes. While byte string and string literals can always be placed into static memory and have static lifetime, array literals can depend on local variables and can’t have static lifetime in general case. The chosen design potentially allows to trivially enhance some array literals with static lifetime in the future to allow use like
fn f() -> &'static [int] {
[1, 2, 3]
}
Alternatives
The alternative design is to make the literals the values and not the references.
The changes
Keep the types of array literals as [T, ..N]
.
Change the types of byte literals from &'static [u8]
to [u8, ..N]
.
Change the types of string literals form &'static str
to FixedString<N>
.
2)
Introduce the missing family of types - strings of fixed size - FixedString<N>
.
…
3)
Add the autocoercion of array literals (not arrays of fixed size in general) to slices.
Add the autocoercion of new byte literals to slices.
Add the autocoercion of new string literals to slices.
Non-literal arrays and strings do not autocoerce to slices, in accordance with the general agreements on explicitness.
4)
Make string and byte literals lvalues with static lifetime.
Examples of use:
// Today: initialize mutable array with literal
let mut arr: [u8, ..3] = b"abc";
arr[0] = b'd';
// Future with CTE: compile time string concatenation
static LANG_DIR: FixedString<_> = "lang/";
static EN_FILE: FixedString<_> = LANG_DIR + "en"; // FixedString<N> implements Add
static FR_FILE: FixedString<_> = LANG_DIR + "fr";
// Future without CTE: runtime "heapless" string concatenation
let DE_FILE = LANG_DIR + "de"; // Performed at runtime if not optimized
Drawbacks of the alternative design
Special rules about (byte) string literals being static lvalues add a bit of unnecessary complexity to the specification.
In theory let s = "abcd";
copies the string from static memory to stack, but the copy is unobservable an can, probably, be elided in most cases.
The set of additional autocoercions has to exist for ergonomic purpose (and for backward compatibility). Writing something like:
fn f(arg: &str) {}
f("Hello"[]);
f(&"Hello");
for all literals would be just unacceptable.
Minor breakage:
fn main() {
let s = "Hello";
fn f(arg: &str) {}
f(s); // Will require explicit slicing f(s[]) or implicit DST coercion from reference f(&s)
}
Status quo
Status quo (or partial application of the changes) is always an alternative.
Drawbacks of status quo
Examples:
// Today: can't use byte string literals in some cases
let mut arr: [u8, ..3] = [b'a', b'b', b'c']; // Have to use array literals
arr[0] = b'd';
// Future: FixedString<N> is added, CTE is added, but the literal types remain old
let mut arr: [u8, ..3] = b"abc".to_fixed(); // Have to use a conversion method
arr[0] = b'd';
static LANG_DIR: FixedString<_> = "lang/".to_fixed(); // Have to use a conversion method
static EN_FILE: FixedString<_> = LANG_DIR + "en".to_fixed();
static FR_FILE: FixedString<_> = LANG_DIR + "fr".to_fixed();
// Bad future: FixedString<N> is not added
// "Heapless"/compile-time string operations aren't possible, or performed with "magic" like extended concat! or recursive macros.
Note, that in the “Future” scenario the return type of to_fixed
depends on the value of self
, so it requires sufficiently advanced CTE, for example C++14 with its powerful constexpr
machinery still doesn’t allow to write such a function.
Drawbacks
None.
Unresolved questions
None.
- Start Date: 2014-09-30
- RFC PR: rust-lang/rfcs#341
- Rust Issue: rust-lang/rust#17861
Summary
Removes the “virtual struct” (aka struct inheritance) feature, which is currently feature gated.
Motivation
Virtual structs were added experimentally prior to the RFC process as a way of inheriting fields from one struct when defining a new struct.
The feature was introduced and remains behind a feature gate.
The motivations for removing this feature altogether are:
-
The feature is likely to be replaced by a more general mechanism, as part of the need to address hierarchies such as the DOM, ASTs, and so on. See this post for some recent discussion.
-
The implementation is somewhat buggy and incomplete, and the feature is not well-documented.
-
Although it’s behind a feature gate, keeping the feature around is still a maintenance burden.
Detailed design
Remove the implementation and feature gate for virtual structs.
Retain the virtual
keyword as reserved for possible future use.
Drawbacks
The language will no longer offer any built-in mechanism for avoiding repetition of struct fields. Macros offer a reasonable workaround until a more general mechanism is added.
Unresolved questions
None known.
- Start Date: 2014-10-07
- RFC PR: rust-lang/rfcs#342
- Rust Issue: rust-lang/rust#17862
Summary
Reserve abstract
, final
, and override
as possible keywords.
Motivation
We intend to add some mechanism to Rust to support more efficient inheritance
(see, e.g., RFC PRs #245 and #250, and this
thread
on discuss). Although we have not decided how to do this, we do know that we
will. Any implementation is likely to make use of keywords virtual
(already
used, to remain reserved), abstract
, final
, and override
, so it makes
sense to reserve these now to make the eventual implementation as backwards
compatible as possible.
Detailed design
Make abstract
, final
, and override
reserved keywords.
Drawbacks
Takes a few more words out of the possible vocabulary of Rust programmers.
Alternatives
Don’t do this and deal with it when we have an implementation. This would mean bumping the language version, probably.
Unresolved questions
N/A
- Start Date: 2014-10-15
- RFC PR: rust-lang/rfcs#344
- Rust Issue: rust-lang/rust#18074
Summary
This is a conventions RFC for settling a number of remaining naming conventions:
- Referring to types in method names
- Iterator type names
- Additional iterator method names
- Getter/setter APIs
- Associated types
- Trait naming
- Lint naming
- Suffix ordering
- Prelude traits
It also proposes to standardize on lower case error messages within the compiler and standard library.
Motivation
As part of the ongoing API stabilization process, we need to settle naming conventions for public APIs. This RFC is a continuation of that process, addressing a number of smaller but still global naming issues.
Detailed design
The RFC includes a number of unrelated naming conventions, broken down into subsections below.
Referring to types in method names
Function names often involve type names, the most common example being conversions
like as_slice
. If the type has a purely textual name (ignoring parameters), it
is straightforward to convert between type conventions and function conventions:
Type name | Text in methods |
---|---|
String | string |
Vec<T> | vec |
YourType | your_type |
Types that involve notation are less clear, so this RFC proposes some standard conventions for referring to these types. There is some overlap on these rules; apply the most specific applicable rule.
Type name | Text in methods |
---|---|
&str | str |
&[T] | slice |
&mut [T] | mut_slice |
&[u8] | bytes |
&T | ref |
&mut T | mut |
*const T | ptr |
*mut T | mut_ptr |
The only surprise here is the use of mut
rather than mut_ref
for mutable
references. This abbreviation is already a fairly common convention
(e.g. as_ref
and as_mut
methods), and is meant to keep this very common case
short.
Iterator type names
The current convention for iterator type names is the following:
Iterators require introducing and exporting new types. These types should use the following naming convention:
Base name. If the iterator yields something that can be described with a specific noun, the base name should be the pluralization of that noun (e.g. an iterator yielding words is called
Words
). Generic contains use the base nameItems
.Flavor prefix. Iterators often come in multiple flavors, with the default flavor providing immutable references. Other flavors should prefix their name:
- Moving iterators have a prefix of
Move
.- If the default iterator yields an immutable reference, an iterator yielding a mutable reference has a prefix
Mut
.- Reverse iterators have a prefix of
Rev
.
(These conventions were established as part of this PR and later this one.)
These conventions have not yet been updated to reflect the recent change to the iterator method names, in part to allow for a more significant revamp. There are some problems with the current rules:
-
They are fairly loose and therefore not mechanical or predictable. In particular, the choice of noun to use for the base name is completely arbitrary.
-
They are not always applicable. The
iter
module, for example, defines a large number of iterator types for use in the adapter methods onIterator
(e.g.Map
formap
,Filter
forfilter
, etc.) The module does not follow the convention, and it’s not clear how it could do so.
This RFC proposes to instead align the convention with the iter
module: the
name of an iterator type should be the same as the method that produces the
iterator.
For example:
iter
would yield anIter
iter_mut
would yield anIterMut
into_iter
would yield anIntoIter
These type names make the most sense when prefixed with their owning module,
e.g. vec::IntoIter
.
Advantages:
-
The rule is completely mechanical, and therefore highly predictable.
-
The convention can be (almost) universally followed: it applies equally well to
vec
and toiter
.
Disadvantages:
-
IntoIter
is not an ideal name. Note, however, that since we’ve moved tointo_iter
as the method name, the existing convention (MoveItems
) needs to be updated to match, and it’s not clear how to do better thanIntoItems
in any case. -
This naming scheme can result in clashes if multiple containers are defined in the same module. Note that this is already the case with today’s conventions. In most cases, this situation should be taken as an indication that a more refined module hierarchy is called for.
Additional iterator method names
An earlier RFC settled the
conventions for the “standard” iterator methods: iter
, iter_mut
,
into_iter
.
However, there are many cases where you also want “nonstandard” iterator
methods: bytes
and chars
for strings, keys
and values
for maps,
the various adapters for iterators.
This RFC proposes the following convention:
-
Use
iter
(and variants) for data types that can be viewed as containers, and where the iterator provides the “obvious” sequence of contained items. -
If there is no single “obvious” sequence of contained items, or if there are multiple desired views on the container, provide separate methods for these that do not use
iter
in their name. The name should instead directly reflect the view/item type being iterated (likebytes
). -
Likewise, for iterator adapters (
filter
,map
and so on) or other iterator-producing operations (intersection
), use the clearest name to describe the adapter/operation directly, and do not mentioniter
. -
If not otherwise qualified, an iterator-producing method should provide an iterator over immutable references. Use the
_mut
suffix for variants producing mutable references, and theinto_
prefix for variants consuming the data in order to produce owned values.
Getter/setter APIs
Some data structures do not wish to provide direct access to their fields, but instead offer “getter” and “setter” methods for manipulating the field state (often providing checking or other functionality).
The proposed convention for a field foo: T
is:
- A method
foo(&self) -> &T
for getting the current value of the field. - A method
set_foo(&self, val: T)
for setting the field. (Theval
argument here may take&T
or some other type, depending on the context.)
Note that this convention is about getters/setters on ordinary data types, not on builder objects. The naming conventions for builder methods are still open.
Associated types
Unlike type parameters, the names of associated types for a trait are a meaningful part of its public API.
Associated types should be given concise, but meaningful names, generally
following the convention for type names rather than generic. For example, use
Err
rather than E
, and Item
rather than T
.
Trait naming
The wiki guidelines have long suggested naming traits as follows:
Prefer (transitive) verbs, nouns, and then adjectives; avoid grammatical suffixes (like
able
)
Trait names like Copy
, Clone
and Show
follow this convention. The
convention avoids grammatical verbosity and gives Rust code a distinctive flavor
(similar to its short keywords).
This RFC proposes to amend the convention to further say: if there is a single
method that is the dominant functionality of the trait, consider using the same
name for the trait itself. This is already the case for Clone
and ToCStr
,
for example.
According to these rules, Encodable
should probably be Encode
.
There are some open questions about these rules; see Unresolved Questions below.
Lints
Our lint names are not consistent. While this may seem like a minor concern, when we hit 1.0 the lint names will be locked down, so it’s worth trying to clean them up now.
The basic rule is: the lint name should make sense when read as “allow
lint-name” or “allow lint-name items”. For example, “allow
deprecated
items” and “allow dead_code
” makes sense, while “allow
unsafe_block
” is ungrammatical (should be plural).
Specifically, this RFC proposes that:
-
Lint names should state the bad thing being checked for, e.g.
deprecated
, so that#[allow(deprecated)]
(items) reads correctly. Thusctypes
is not an appropriate name;improper_ctypes
is. -
Lints that apply to arbitrary items (like the stability lints) should just mention what they check for: use
deprecated
rather thandeprecated_items
. This keeps lint names short. (Again, think “allow lint-name items”.) -
If a lint applies to a specific grammatical class, mention that class and use the plural form: use
unused_variables
rather thanunused_variable
. This makes#[allow(unused_variables)]
read correctly. -
Lints that catch unnecessary, unused, or useless aspects of code should use the term
unused
, e.g.unused_imports
,unused_typecasts
. -
Use snake case in the same way you would for function names.
Suffix ordering
Very occasionally, conventions will require a method to have multiple suffixes,
for example get_unchecked_mut
. When feasible, design APIs so that this
situation does not arise.
Because it is so rare, it does not make sense to lay out a complete convention for the order in which various suffixes should appear; no one would be able to remember it.
However, the mut suffix is so common, and is now entrenched as showing up in
final position, that this RFC does propose one simple rule: if there are
multiple suffixes including mut
, place mut
last.
Prelude traits
It is not currently possible to define inherent methods directly on basic data
types like char
or slices. Consequently, libcore
and other basic crates
provide one-off traits (like ImmutableSlice
or Char
) that are intended to be
implemented solely by these primitive types, and which are included in the
prelude.
These traits are generally not designed to be used for generic programming, but the fact that they appear in core libraries with such basic names makes it easy to draw the wrong conclusion.
This RFC proposes to use a Prelude
suffix for these basic traits. Since the
traits are, in fact, included in the prelude their names do not generally appear
in Rust programs. Therefore, choosing a longer and clearer name will help avoid
confusion about the intent of these traits, and will avoid namespace pollution.
(There is one important drawback in today’s Rust: associated functions in these traits cannot yet be called directly on the types implementing the traits. These functions are the one case where you would need to mention the trait by name, today. Hopefully, this situation will change before 1.0; otherwise we may need a separate plan for dealing with associated functions.)
Error messages
Error messages – including those produced by fail!
and those placed in the
desc
or detail
fields of e.g. IoError
– should in general be in all lower
case. This applies to both rustc
and std
.
This is already the predominant convention, but there are some inconsistencies.
Alternatives
Iterator type names
The iterator type name convention could instead basically stick with today’s
convention, but using suffixes instead of prefixes, and IntoItems
rather than
MoveItems
.
Unresolved questions
How far should the rules for trait names go? Should we avoid “-er” suffixes,
e.g. have Read
rather than Reader
?
- Start Date: 2014-10-15
- RFC PR: rust-lang/rfcs#356
- Rust Issue: rust-lang/rust#18073
Summary
This is a conventions RFC that proposes that the items exported from a module
should never be prefixed with that module name. For example, we should have
io::Error
, not io::IoError
.
(An alternative design is included that special-cases overlap with the
prelude
.)
Motivation
Currently there is no clear prohibition around including the module’s name as a
prefix on an exported item, and it is sometimes done for type names that are
feared to be “popular” (like Error
and Result
being IoError
and
IoResult
) for clarity.
This RFC include two designs: one that entirely rules out such prefixes, and one that rules it out except for names that overlap with the prelude. Pros/cons are given for each.
Detailed design
The main rule being proposed is very simple: the items exported from a module should never be prefixed with the module’s name.
Rationale:
- Avoids needless stuttering like
io::IoError
. - Any ambiguity can be worked around:
- Either qualify by the module, i.e.
io::Error
, - Or rename on import:
use io::Error as IoError
.
- Either qualify by the module, i.e.
- The rule is extremely simple and clear.
Downsides:
- The name may already exist in the module wanting to export it.
- If that’s due to explicit imports, those imports can be renamed or module-qualified (see above).
- If that’s due to a prelude conflict, however, confusion may arise due to the conventional global meaning of identifiers defined in the prelude (i.e., programmers do not expect prelude imports to be shadowed).
Overall, the RFC author believes that if this convention is adopted, confusion
around redefining prelude names would gradually go away, because (at least for
things like Result
) we would come to expect it.
Alternative design
An alternative rule would be to never prefix an exported item with the module’s name, except for names that are also defined in the prelude, which must be prefixed by the module’s name.
For example, we would have io::Error
and io::IoResult
.
Rationale:
- Largely the same as the above, but less decisively.
- Avoids confusion around prelude-defined names.
Downsides:
- Retains stuttering for some important cases, e.g. custom
Result
types, which are likely to be fairly common. - Makes it even more problematic to expand the prelude in the future.
- Start Date: 2014-09-16
- RFC PR: rust-lang/rfcs#369
- Rust Issue: rust-lang/rust#18640
Summary
This RFC is preparation for API stabilization for the std::num
module. The
proposal is to finish the simplification efforts started in
@bjz’s reversal of the numerics hierarchy.
Broadly, the proposal is to collapse the remaining numeric hierarchy
in std::num
, and to provide only limited support for generic
programming (roughly, only over primitive numeric types that vary
based on size). Traits giving detailed numeric hierarchy can and
should be provided separately through the Cargo ecosystem.
Thus, this RFC proposes to flatten or remove most of the traits
currently provided by std::num
, and generally to simplify the module
as much as possible in preparation for API stabilization.
Motivation
History
Starting in early 2013, there was an effort to design a comprehensive “numeric hierarchy” for Rust: a collection of traits classifying a wide variety of numbers and other algebraic objects. The intent was to allow highly-generic code to be written for algebraic structures and then instantiated to particular types.
This hierarchy covered structures like bigints, but also primitive integer and float types. It was an enormous and long-running community effort.
Later, it was recognized that
building such a hierarchy within libstd
was misguided:
@bjz The API that resulted from #4819 attempted, like Haskell, to blend both the primitive numerics and higher level mathematical concepts into one API. This resulted in an ugly hybrid where neither goal was adequately met. I think the libstd should have a strong focus on implementing fundamental operations for the base numeric types, but no more. Leave the higher level concepts to libnum or future community projects.
The std::num
module has thus been slowly migrating away from a large trait
hierarchy toward a simpler one providing just APIs for primitive data types:
this is
@bjz’s reversal of the numerics hierarchy.
Along side this effort, there are already external numerics packages like @bjz’s num-rs.
But we’re not finished yet.
The current state of affairs
The std::num
module still contains quite a few traits that subdivide out
various features of numbers:
pub trait Zero: Add<Self, Self> {
fn zero() -> Self;
fn is_zero(&self) -> bool;
}
pub trait One: Mul<Self, Self> {
fn one() -> Self;
}
pub trait Signed: Num + Neg<Self> {
fn abs(&self) -> Self;
fn abs_sub(&self, other: &Self) -> Self;
fn signum(&self) -> Self;
fn is_positive(&self) -> bool;
fn is_negative(&self) -> bool;
}
pub trait Unsigned: Num {}
pub trait Bounded {
fn min_value() -> Self;
fn max_value() -> Self;
}
pub trait Primitive: Copy + Clone + Num + NumCast + PartialOrd + Bounded {}
pub trait Num: PartialEq + Zero + One + Neg<Self> + Add<Self,Self> + Sub<Self,Self>
+ Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self> {}
pub trait Int: Primitive + CheckedAdd + CheckedSub + CheckedMul + CheckedDiv
+ Bounded + Not<Self> + BitAnd<Self,Self> + BitOr<Self,Self>
+ BitXor<Self,Self> + Shl<uint,Self> + Shr<uint,Self> {
fn count_ones(self) -> uint;
fn count_zeros(self) -> uint { ... }
fn leading_zeros(self) -> uint;
fn trailing_zeros(self) -> uint;
fn rotate_left(self, n: uint) -> Self;
fn rotate_right(self, n: uint) -> Self;
fn swap_bytes(self) -> Self;
fn from_be(x: Self) -> Self { ... }
fn from_le(x: Self) -> Self { ... }
fn to_be(self) -> Self { ... }
fn to_le(self) -> Self { ... }
}
pub trait FromPrimitive {
fn from_i64(n: i64) -> Option<Self>;
fn from_u64(n: u64) -> Option<Self>;
// many additional defaulted methods
// ...
}
pub trait ToPrimitive {
fn to_i64(&self) -> Option<i64>;
fn to_u64(&self) -> Option<u64>;
// many additional defaulted methods
// ...
}
pub trait NumCast: ToPrimitive {
fn from<T: ToPrimitive>(n: T) -> Option<Self>;
}
pub trait Saturating {
fn saturating_add(self, v: Self) -> Self;
fn saturating_sub(self, v: Self) -> Self;
}
pub trait CheckedAdd: Add<Self, Self> {
fn checked_add(&self, v: &Self) -> Option<Self>;
}
pub trait CheckedSub: Sub<Self, Self> {
fn checked_sub(&self, v: &Self) -> Option<Self>;
}
pub trait CheckedMul: Mul<Self, Self> {
fn checked_mul(&self, v: &Self) -> Option<Self>;
}
pub trait CheckedDiv: Div<Self, Self> {
fn checked_div(&self, v: &Self) -> Option<Self>;
}
pub trait Float: Signed + Primitive {
// a huge collection of static functions (for constants) and methods
...
}
pub trait FloatMath: Float {
// an additional collection of methods
}
The Primitive
traits are intended primarily to support a mechanism,
#[deriving(FromPrimitive)]
, that makes it easy to provide
conversions from numeric types to C-like enum
s.
The Saturating
and Checked
traits provide operations that provide
special handling for overflow and other numeric errors.
Almost all of these traits are currently included in the prelude.
In addition to these traits, the std::num
module includes a couple
dozen free functions, most of which duplicate methods available though
traits.
Where we want to go: a summary
The goal of this RFC is to refactor the std::num
hierarchy with the
following goals in mind:
-
Simplicity.
-
Limited generic programming: being able to work generically over the natural classes of primitive numeric types that vary only by size. There should be enough abstraction to support porting
strconv
, the generic string/number conversion code used instd
. -
Minimizing dependencies for
libcore
. For example, it should not requirecmath
. -
Future-proofing for external numerics packages. The Cargo ecosystem should ultimately provide choices of sophisticated numeric hierarchies, and
std::num
should not get in the way.
Detailed design
Overview: the new hierarchy
This RFC proposes to collapse the trait hierarchy in std::num
to
just the following traits:
Int
, implemented by all primitive integer types (u8
-u64
,i8
-i64
)UnsignedInt
, implemented byu8
-u64
Signed
, implemented by all signed primitive numeric types (i8
-i64
,f32
-f64
)Float
, implemented byf32
andf64
FloatMath
, implemented byf32
andf64
, which provides functionality fromcmath
These traits inherit from all applicable overloaded operator traits
(from core::ops
). They suffice for generic programming over several
basic categories of primitive numeric types.
As designed, these traits include a certain amount of redundancy
between Int
and Float
. The Alternatives section shows how this
could be factored out into a separate Num
trait. But doing so
suggests a level of generic programming that these traits aren’t
intended to support.
The main reason to pull out Signed
into its own trait is so that it
can be added to the prelude. (Further discussion below.)
Detailed definitions
Below is the full definition of these traits. The functionality remains largely as it is today, just organized into fewer traits:
pub trait Int: Copy + Clone + PartialOrd + PartialEq
+ Add<Self,Self> + Sub<Self,Self>
+ Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self>
+ Not<Self> + BitAnd<Self,Self> + BitOr<Self,Self>
+ BitXor<Self,Self> + Shl<uint,Self> + Shr<uint,Self>
{
// Constants
fn zero() -> Self; // These should be associated constants when those are available
fn one() -> Self;
fn min_value() -> Self;
fn max_value() -> Self;
// Deprecated:
// fn is_zero(&self) -> bool;
// Bit twiddling
fn count_ones(self) -> uint;
fn count_zeros(self) -> uint { ... }
fn leading_zeros(self) -> uint;
fn trailing_zeros(self) -> uint;
fn rotate_left(self, n: uint) -> Self;
fn rotate_right(self, n: uint) -> Self;
fn swap_bytes(self) -> Self;
fn from_be(x: Self) -> Self { ... }
fn from_le(x: Self) -> Self { ... }
fn to_be(self) -> Self { ... }
fn to_le(self) -> Self { ... }
// Checked arithmetic
fn checked_add(self, v: Self) -> Option<Self>;
fn checked_sub(self, v: Self) -> Option<Self>;
fn checked_mul(self, v: Self) -> Option<Self>;
fn checked_div(self, v: Self) -> Option<Self>;
fn saturating_add(self, v: Self) -> Self;
fn saturating_sub(self, v: Self) -> Self;
}
pub trait UnsignedInt: Int {
fn is_power_of_two(self) -> bool;
fn checked_next_power_of_two(self) -> Option<Self>;
fn next_power_of_two(self) -> Self;
}
pub trait Signed: Neg<Self> {
fn abs(&self) -> Self;
fn signum(&self) -> Self;
fn is_positive(&self) -> bool;
fn is_negative(&self) -> bool;
// Deprecated:
// fn abs_sub(&self, other: &Self) -> Self;
}
pub trait Float: Copy + Clone + PartialOrd + PartialEq + Signed
+ Add<Self,Self> + Sub<Self,Self>
+ Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self>
{
// Constants
fn zero() -> Self; // These should be associated constants when those are available
fn one() -> Self;
fn min_value() -> Self;
fn max_value() -> Self;
// Classification and decomposition
fn is_nan(self) -> bool;
fn is_infinite(self) -> bool;
fn is_finite(self) -> bool;
fn is_normal(self) -> bool;
fn classify(self) -> FPCategory;
fn integer_decode(self) -> (u64, i16, i8);
// Float intrinsics
fn floor(self) -> Self;
fn ceil(self) -> Self;
fn round(self) -> Self;
fn trunc(self) -> Self;
fn mul_add(self, a: Self, b: Self) -> Self;
fn sqrt(self) -> Self;
fn powi(self, n: i32) -> Self;
fn powf(self, n: Self) -> Self;
fn exp(self) -> Self;
fn exp2(self) -> Self;
fn ln(self) -> Self;
fn log2(self) -> Self;
fn log10(self) -> Self;
// Conveniences
fn fract(self) -> Self;
fn recip(self) -> Self;
fn rsqrt(self) -> Self;
fn to_degrees(self) -> Self;
fn to_radians(self) -> Self;
fn log(self, base: Self) -> Self;
}
// This lives directly in `std::num`, not `core::num`, since it requires `cmath`
pub trait FloatMath: Float {
// Exactly the methods defined in today's version
}
Float constants, float math, and cmath
This RFC proposes to:
-
Remove all float constants from the
Float
trait. These constants are available directly from thef32
andf64
modules, and are not really useful for the kind of generic programming these new traits are intended to allow. -
Continue providing various
cmath
functions as methods in theFloatMath
trait. Putting this in a separate trait means thatlibstd
depends oncmath
butlibcore
does not.
Free functions
All of the free functions defined in std::num
are deprecated.
The prelude
The prelude will only include the Signed
trait, as the operations it
provides are widely expected to be available when they apply.
The reason for removing the rest of the traits is two-fold:
-
The remaining operations are relatively uncommon. Note that various overloaded operators, like
+
, work regardless of this choice. Those doing intensive work with e.g. floats would only need to importFloat
andFloatMath
. -
Keeping this functionality out of the prelude means that the names of methods and associated items remain available for external numerics libraries in the Cargo ecosystem.
strconv
, FromStr
, ToStr
, FromStrRadix
, ToStrRadix
Currently, traits for converting from &str
and to String
are both
included, in their own modules, in libstd
. This is largely due to
the desire to provide impl
s for numeric types, which in turn relies
on std::num::strconv
.
This RFC proposes to:
- Move the
FromStr
trait intocore::str
. - Rename the
ToStr
trait toToString
, and move it tocollections::string
. - Break up and revise
std::num::strconv
into separate, private modules that provide the needed functionality for thefrom_str
andto_string
methods. (Some of this functionality has already migrated tofmt
and been deprecated instrconv
.) - Move the
FromStrRadix
intocore::num
. - Remove
ToStrRadix
, which is already deprecated in favor offmt
.
FromPrimitive
and friends
Ideally, the FromPrimitive
, ToPrimitive
, Primitive
, NumCast
traits would all be removed in favor of a more principled way of
working with C-like enums. However, such a replacement is outside of
the scope of this RFC, so these traits are left (as #[experimental]
)
for now. A follow-up RFC proposing a better solution should appear soon.
In the meantime, see
this proposal and
the discussion on
this issue about
Ordinal
for the rough direction forward.
Drawbacks
This RFC somewhat reduces the potential for writing generic numeric
code with std::num
traits. This is intentional, however: the new
design represents “just enough” generics to cover differently-sized
built-in types, without any attempt at general algebraic abstraction.
Alternatives
The status quo is clearly not ideal, and as explained above there was
a long attempt at providing a more complete numeric hierarchy in std
.
So some collapse of the hierarchy seems desirable.
That said, there are other possible factorings. We could introduce the
following Num
trait to factor out commonalities between Int
and Float
:
pub trait Num: Copy + Clone + PartialOrd + PartialEq
+ Add<Self,Self> + Sub<Self,Self>
+ Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self>
{
fn zero() -> Self; // These should be associated constants when those are available
fn one() -> Self;
fn min_value() -> Self;
fn max_value() -> Self;
}
However, it’s not clear whether this factoring is worth having a more complex hierarchy, especially because the traits are not intended for generic programming at that level (and generic programming across integer and floating-point types is likely to be extremely rare)
The signed and unsigned operations could be offered on more types, allowing removal of more traits but a less clear-cut semantics.
Unresolved questions
This RFC does not propose a replacement for
#[deriving(FromPrimitive)]
, leaving the relevant traits in limbo
status. (See
this proposal and
the discussion on
this issue about
Ordinal
for the rough direction forward.)
- Start Date: 2014-10-09
- RFC PR #: https://github.com/rust-lang/rfcs/pull/378
- Rust Issue #: https://github.com/rust-lang/rust/issues/18635
Summary
Parse macro invocations with parentheses or square brackets as expressions no matter the context, and require curly braces or a semicolon following the invocation to invoke a macro as a statement.
Motivation
Currently, macros that start a statement want to be a whole statement, and so
expressions such as foo!().bar
don’t parse if they start a statement. The
reason for this is because sometimes one wants a macro that expands to an item
or statement (for example, macro_rules!
), and forcing the user to add a
semicolon to the end is annoying and easy to forget for long, multi-line
statements. However, the vast majority of macro invocations are not intended to
expand to an item or statement, leading to frustrating parser errors.
Unfortunately, this is not as easy to resolve as simply checking for an infix operator after every statement-like macro invocation, because there exist operators that are both infix and prefix. For example, consider the following function:
fn frob(x: int) -> int {
maybe_return!(x)
// Provide a default value
-1
}
Today, this parses successfully. However, if a rule were added to the parser
that any macro invocation followed by an infix operator be parsed as a single
expression, this would still parse successfully, but not in the way expected: it
would be parsed as (maybe_return!(x)) - 1
. This is an example of how it is
impossible to resolve this ambiguity properly without breaking compatibility.
Detailed design
Treat all macro invocations with parentheses, ()
, or square brackets, []
, as
expressions, and never attempt to parse them as statements or items in a block
context unless they are followed directly by a semicolon. Require all
item-position macro invocations to be either invoked with curly braces, {}
, or
be followed by a semicolon (for consistency).
This distinction between parentheses and curly braces has precedent in Rust:
tuple structs, which use parentheses, must be followed by a semicolon, while
structs with fields do not need to be followed by a semicolon. Many constructs
like match
and if
, which use curly braces, also do not require semicolons
when they begin a statement.
Drawbacks
- This introduces a difference between different macro invocation delimiters, where previously there was no difference.
- This requires the use of semicolons in a few places where it was not necessary before.
Alternatives
- Require semicolons after all macro invocations that aren’t being used as
expressions. This would have the downside of requiring semicolons after every
macro_rules!
declaration.
Unresolved questions
None.
- Start Date: 2014-10-13
- RFC PR: rust-lang/rfcs#379
- Rust Issue: rust-lang/rust#18046
Summary
- Remove reflection from the compiler
- Remove
libdebug
- Remove the
Poly
format trait as well as the:?
format specifier
Motivation
In ancient Rust, one of the primary methods of printing a value was via the %?
format specifier. This would use reflection at runtime to determine how to print
a type. Metadata generated by the compiler (a TyDesc
) would be generated to
guide the runtime in how to print a type. One of the great parts about
reflection was that it was quite easy to print any type. No extra burden was
required from the programmer to print something.
There are, however, a number of cons to this approach:
- Generating extra metadata for many many types by the compiler can lead to noticeable increases in compile time and binary size.
- This form of formatting is inherently not speedy. Widespread usage of
%?
led to misleading benchmarks about formatting in Rust. - Depending on how metadata is handled, this scheme makes it very difficult to allow recompiling a library without recompiling downstream dependants.
Over time, usage off the ?
formatting has fallen out of fashion for the
following reasons:
- The
deriving
-based infrastructure was improved greatly and has started seeing much more widespread use, especially for traits likeClone
. - The formatting language implementation and syntax has changed. The most common
formatter is now
{}
(an implementation ofShow
), and it is quite common to see an implementation ofShow
on nearly all types (frequently viaderiving
). This form of customizable-per-typformatting largely provides the gap that the original formatting language did not provide, which was limited to only primitives and%?
. - Compiler built-ins, such as
~[T]
and~str
have been removed from the language, and runtime reflection onVec<T>
andString
are far less useful (they just print pointers, not contents).
As a result, the :?
formatting specifier is quite rarely used today, and
when it is used it’s largely for historical purposes and the output is not of
very high quality any more.
The drawbacks and today’s current state of affairs motivate this RFC to recommend removing this infrastructure entirely. It’s possible to add it back in the future with a more modern design reflecting today’s design principles of Rust and the many language changes since the infrastructure was created.
Detailed design
- Remove all reflection infrastructure from the compiler. I am not personally
super familiar with what exists, but at least these concrete actions will be
taken.
- Remove the
visit_glue
function fromTyDesc
. - Remove any form of
visit_glue
generation. - (maybe?) Remove the
name
field ofTyDesc
.
- Remove the
- Remove
core::intrinsics::TyVisitor
- Remove
core::intrinsics::visit_tydesc
- Remove
libdebug
- Remove
std::fmt::Poly
- Remove the
:?
format specifier in the formatting language syntax.
Drawbacks
The current infrastructure for reflection, although outdated, represents a significant investment of work in the past which could be a shame to lose. While present in the git history, this infrastructure has been updated over time, and it will no longer receive this attention.
Additionally, given an arbitrary type T
, it would now be impossible to print
it in literally any situation. Type parameters will now require some bound, such
as Show
, to allow printing a type.
These two drawbacks are currently not seen as large enough to outweigh the gains
from reducing the surface area of the std::fmt
API and reduction in
maintenance load on the compiler.
Alternatives
The primary alternative to outright removing this infrastructure is to preserve
it, but flag it all as #[experimental]
or feature-gated. The compiler could
require the fmt_poly
feature gate to be enabled to enable formatting via :?
in a crate. This would mean that any backwards-incompatible changes could
continue to be made, and any arbitrary type T
could still be printed.
Unresolved questions
- Can
core::intrinsics::TyDesc
be removed entirely?
- Start Date: 2014-11-12
- RFC PR: rust-lang/rfcs#380
- Rust Issue: rust-lang/rust#18904
Summary
Stabilize the std::fmt
module, in addition to the related macros and
formatting language syntax. As a high-level summary:
- Leave the format syntax as-is.
- Remove a number of superfluous formatting traits (renaming a few in the process).
Motivation
This RFC is primarily motivated by the need to stabilize std::fmt
. In the past
stabilization has not required RFCs, but the changes envisioned for this module
are far-reaching and modify some parts of the language (format syntax), leading
to the conclusion that this stabilization effort required an RFC.
Detailed design
The std::fmt
module encompasses more than just the actual
structs/traits/functions/etc defined within it, but also a number of macros and
the formatting language syntax for describing format strings. Each of these
features of the module will be described in turn.
Formatting Language Syntax
The documented syntax will not be changing as-written. All of these features will be accepted wholesale (considered stable):
- Usage of
{}
for “format something here” placeholders {{
as an escape for{
(and vice-versa for}
)- Various format specifiers
- fill character for alignment
- actual alignment, left (
<
), center (^
), and right (>
). - sign to print (
+
or-
) - minimum width for text to be printed
- both a literal count and a runtime argument to the format string
- precision or maximum width
- all of a literal count, a specific runtime argument to the format string, and “the next” runtime argument to the format string.
- “alternate formatting” (
#
) - leading zeroes (
0
)
- Integer specifiers of what to format (
{0}
) - Named arguments (
{foo}
)
Using Format Specifiers
While quite useful occasionally, there is no static guarantee that any implementation of a formatting trait actually respects the format specifiers passed in. For example, this code does not necessarily work as expected:
#[deriving(Show)]
struct A;
format!("{:10}", A);
All of the primitives for rust (strings, integers, etc) have implementations of
Show
which respect these formatting flags, but almost no other implementations
do (notably those generated via deriving
).
This RFC proposes stabilizing the formatting flags, despite this current state of affairs. There are in theory possible alternatives in which there is a static guarantee that a type does indeed respect format specifiers when one is provided, generating a compile-time error when a type doesn’t respect a specifier. These alternatives, however, appear to be too heavyweight and are considered somewhat overkill.
In general it’s trivial to respect format specifiers if an implementation
delegates to a primitive or somehow has a buffer of what’s to be formatted. To
cover these two use cases, the Formatter
structure passed around has helper
methods to assist in formatting these situations. This is, however, quite rare
to fall into one of these two buckets, so the specifiers are largely ignored
(and the formatter is write!
-n to directly).
Named Arguments
Currently Rust does not support named arguments anywhere except for format strings. Format strings can get away with it because they’re all part of a macro invocation (unlike the rest of Rust syntax).
The worry for stabilizing a named argument syntax for the formatting language is that if Rust ever adopts named arguments with a different syntax, it would be quite odd having two systems.
The most recently proposed keyword argument
RFC used :
for the invocation
syntax rather than =
as formatting does today. Additionally, today foo = bar
is a valid expression, having a value of type ()
.
With these worries, there are one of two routes that could be pursued:
- The
expr = expr
syntax could be disallowed on the language level. This could happen both in a total fashion or just allowing the expression appearing as a function argument. For both cases, this will probably be considered a “wart” of Rust’s grammar. - The
foo = bar
syntax could be allowed in the macro with prior knowledge that the default argument syntax for Rust, if one is ever developed, will likely be different. This would mean that thefoo = bar
syntax in formatting macros will likely be considered a wart in the future.
Given these two cases, the clear choice seems to be accepting a wart in the formatting macros themselves. It will likely be possible to extend the macro in the future to support whatever named argument syntax is developed as well, and the old syntax could be accepted for some time.
Formatting Traits
Today there are 16 formatting traits. Each trait represents a “type” of
formatting, corresponding to the [type]
production in the formatting syntax.
As a bit of history, the original intent was for each trait to declare what
specifier it used, allowing users to add more specifiers in newer crates. For
example the time
crate could provide the {:time}
formatting trait. This
design was seen as too complicated, however, so it was not landed. It does,
however, partly motivate why there is one trait per format specifier today.
The 16 formatting traits and their format specifiers are:
- nothing ⇒
Show
d
⇒Signed
i
⇒Signed
u
⇒Unsigned
b
⇒Bool
c
⇒Char
o
⇒Octal
x
⇒LowerHex
X
⇒UpperHex
s
⇒String
p
⇒Pointer
t
⇒Binary
f
⇒Float
e
⇒LowerExp
E
⇒UpperExp
?
⇒Poly
This RFC proposes removing the following traits:
Signed
Unsigned
Bool
Char
String
Float
Note that this RFC would like to remove Poly
, but that is covered by a
separate RFC.
Today by far the most common formatting trait is Show
, and over time the
usefulness of these formatting traits has been reduced. The traits this RFC
proposes to remove are only assertions that the type provided actually
implements the trait, there are few known implementations of the traits which
diverge on how they are implemented.
Additionally, there are a two of oddities inherited from ancient C:
- Both
d
andi
are wired toSigned
- One may reasonable expect the
Binary
trait to useb
as its specifier.
The remaining traits this RFC recommends leaving. The rationale for this is that they represent alternate representations of primitive types in general, and are also quite often expected when coming from other format syntaxes such as C/Python/Ruby/etc.
It would, of course, be possible to re-add any of these traits in a backwards-compatible fashion.
Format type for Binary
With the removal of the Bool
trait, this RFC recommends renaming the specifier
for Binary
to b
instead of t
.
Combining all traits
A possible alternative to having many traits is to instead have one trait, such as:
pub trait Show {
fn fmt(...);
fn hex(...) { fmt(...) }
fn lower_hex(...) { fmt(...) }
...
}
There are a number of pros to this design:
- Instead of having to consider many traits, only one trait needs to be considered.
- All types automatically implement all format types or zero format types.
- In a hypothetical world where a format string could be constructed at runtime,
this would alleviate the signature of such a function. The concrete type taken
for all its arguments would be
&Show
and then if the format string supplied:x
or:o
the runtime would simply delegate to the relevant trait method.
There are also a number of cons to this design, which motivate this RFC recommending the remaining separation of these traits.
- The “static assertion” that a type implements a relevant format trait becomes almost nonexistent because all types either implement none or all formatting traits.
- The documentation for the
Show
trait becomes somewhat overwhelming because it’s no longer immediately clear which method should be overridden for what. - A hypothetical world with runtime format string construction could find a different system for taking arguments.
Method signature
Currently, each formatting trait has a signature as follows:
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result;
This implies that all formatting is considered to be a stream-oriented operation
where f
is a sink to write bytes to. The fmt::Result
type indicates that
some form of “write error” happened, but conveys no extra information.
This API has a number of oddities:
- The type
Formatter
has inherentwrite
andwrite_fmt
methods to be used in conjunction with thewrite!
macro return an instance offmt::Result
. - The
Formatter
type also implements thestd::io::Writer
trait in order to be able to pass around a&mut Writer
. - This relies on the duck-typing of macros and for the inherent
write_fmt
method to trump theWriter
’swrite_fmt
method in order to return an error of the correct type. - The
Result
return type is an enumeration with precisely one variant,FormatError
.
Overall, this signature seems to be appropriate in terms of “give me a sink of
bytes to write myself to, and let me return an error if one happens”. Due to
this, this RFC recommends that all formatting traits be marked #[unstable]
.
Macros
There are a number of prelude macros which interact with the format syntax:
format_args
format_args_method
write
writeln
print
println
format
fail
assert
debug_assert
All of these are macro_rules!
-defined macros, except for format_args
and
format_args_method
.
Common syntax
All of these macros take some form of prefix, while the trailing suffix is
always some instantiation of the formatting syntax. The suffix portion is
recommended to be considered #[stable]
, and the sections below will discuss
each macro in detail with respect to its prefix and semantics.
format_args
The fundamental purpose of this macro is to generate a value of type
&fmt::Arguments
which represents a pending format computation. This structure
can then be passed at some point to the methods in std::fmt
to actually
perform the format.
The prefix of this macro is some “callable thing”, be it a top-level function or
a closure. It cannot invoke a method because foo.bar
is not a “callable thing”
to call the bar
method on foo
.
Ideally, this macro would have no prefix, and would be callable like:
use std::fmt;
let args = format_args!("Hello {}!", "world");
let hello_world = fmt::format(args);
Unfortunately, without an implementation of RFC 31 this is not
possible. As a result, this RFC proposes a #[stable]
consideration of this
macro and its syntax.
format_args_method
The purpose of this macro is to solve the “call this method” case not covered
with the format_args
macro. This macro was introduced fairly late in the game
to solve the problem that &*trait_object
was not allowed. This is currently
allowed, however (due to DST).
This RFC proposes immediately removing this macro. The primary user of this
macro is write!
, meaning that the following code, which compiles today, would
need to be rewritten:
let mut output = std::io::stdout();
// note the lack of `&mut` in front
write!(output, "hello {}", "world");
The write!
macro would be redefined as:
macro_rules! write(
($dst:expr, $($arg:tt)*) => ({
let dst = &mut *$dst;
format_args!(|args| { dst.write_fmt(args) }, $($arg)*)
})
)
The purpose here is to borrow $dst
outside of the closure to ensure that the
closure doesn’t borrow too many of its contents. Otherwise, code such as this
would be disallowed
write!(&mut my_struct.writer, "{}", my_struct.some_other_field);
write/writeln
These two macros take the prefix of “some pointer to a writer” as an argument,
and then format data into the write (returning whatever write_fmt
returns).
These macros were originally designed to require a &mut T
as the first
argument, but today, due to the usage of format_args_method
, they can take any
T
which responds to write_fmt
.
This RFC recommends marking these two macros #[stable]
with the modification
above (removing format_args_method
). The ln
suffix to writeln
will be
discussed shortly.
print/println
These two macros take no prefix, and semantically print to a task-local stdout stream. The purpose of a task-local stream is provide some form of buffering to make stdout printing at all performant.
This RFC recommends marking these two macros a #[stable]
.
The ln
suffix
The name println
is one of the few locations in Rust where a short C-like
abbreviation is accepted rather than the more verbose, but clear, print_line
(for example). Due to the overwhelming precedent of other languages (even Java
uses println
!), this is seen as an acceptable special case to the rule.
format
This macro takes no prefix and returns a String
.
In ancient rust this macro was called its shorter name, fmt
. Additionally, the
name format
is somewhat inconsistent with the module name of fmt
. Despite
this, this RFC recommends considering this macro #[stable]
due to its
delegation to the format
method in the std::fmt
module, similar to how the
write!
macro delegates to the fmt::write
.
fail/assert/debug_assert
The format string portions of these macros are recommended to be considered as
#[stable]
as part of this RFC. The actual stability of the macros is not
considered as part of this RFC.
Freestanding Functions
There are a number of freestanding
functions to consider in
the std::fmt
module for stabilization.
-
fn format(args: &Arguments) -> String
This RFC recommends
#[experimental]
. This method is largely an implementation detail of this module, and should instead be used via:let args: &fmt::Arguments = ...; format!("{}", args)
-
fn write(output: &mut FormatWriter, args: &Arguments) -> Result
This is somewhat surprising in that the argument to this function is not a
Writer
, but rather aFormatWriter
. This is technically speaking due to the core/std separation and how this function is defined in core andWriter
is defined in std.This RFC recommends marking this function
#[experimental]
as thewrite_fmt
exists onWriter
to perform the corresponding operation. Consequently we may wish to remove this function in favor of thewrite_fmt
method onFormatWriter
.Ideally this method would be removed from the public API as it is just an implementation detail of the
write!
macro. -
fn radix<T>(x: T, base: u8) -> RadixFmt<T, Radix>
This function is a bit of an odd-man-out in that it is a constructor, but does not follow the existing conventions of
Type::new
. The purpose of this function is to expose the ability to format a number for any radix. The default format specifiers:o
,:x
, and:t
are essentially shorthands for this function, except that the format types have specialized implementations per radix instead of a generic implementation.This RFC proposes that this function be considered
#[unstable]
as its location and naming are a bit questionable, but the functionality is desired.
Miscellaneous items
-
trait FormatWriter
This trait is currently the actual implementation strategy of formatting, and is defined specially in libcore. It is rarely used outside of libcore. It is recommended to be
#[experimental]
.There are possibilities in moving
Reader
andWriter
to libcore with the error type as an associated item, allowing theFormatWriter
trait to be eliminated entirely. Due to this possibility, the trait will be experimental for now as alternative solutions are explored. -
struct Argument
,mod rt
,fn argument
,fn argumentstr
,fn argumentuint
,Arguments::with_placeholders
,Arguments::new
These are implementation details of the
Arguments
structure as well as the expansion of theformat_args!
macro. It’s recommended to mark these as#[experimental]
and#[doc(hidden)]
. Ideally there would be some form of macro-based privacy hygiene which would allow these to be truly private, but it will likely be the case that these simply become stable and we must live with them forever. -
struct Arguments
This is a representation of a “pending format string” which can be used to safely execute a
Formatter
over it. This RFC recommends#[stable]
. -
struct Formatter
This instance is passed to all formatting trait methods and contains helper methods for respecting formatting flags. This RFC recommends
#[unstable]
.This RFC also recommends deprecating all public fields in favor of accessor methods. This should help provide future extensibility as well as preventing unnecessary mutation in the future.
-
enum FormatError
This enumeration only has one instance,
WriteError
. It is recommended to make this astruct
instead and rename it to justError
. The purpose of this is to signal that an error has occurred as part of formatting, but it does not provide a generic method to transmit any other information other than “an error happened” to maintain the ergonomics of today’s usage. It’s strongly recommended that implementations ofShow
and friends are infallible and only generate an error if the underlyingFormatter
returns an error itself. -
Radix
/RadixFmt
Like the
radix
function, this RFC recommends#[unstable]
for both of these pieces of functionality.
Drawbacks
Today’s macro system necessitates exporting many implementation details of the formatting system, which is unfortunate.
Alternatives
A number of alternatives were laid out in the detailed description for various aspects.
Unresolved questions
- How feasible and/or important is it to construct a format string at runtime given the recommend stability levels in this RFC?
Module system cleanups
- Start Date: 2014-10-10
- RFC PR: rust-lang/rfcs#385
- Rust Issue: rust-lang/rust#18219
Summary
- Lift the hard ordering restriction between
extern crate
,use
and other items. - Allow
pub extern crate
as opposed to only private ones. - Allow
extern crate
in blocks/functions, and not just in modules.
Motivation
The main motivation is consistency and simplicity: None of the changes proposed here change the module system in any meaningful way, they just remove weird forbidden corner cases that are all already possible to express today with workarounds.
Thus, they make it easier to learn the system for beginners, and easier to for developers to evolve their module hierarchies
Lifting the ordering restriction between extern crate
, use
and other items.
Currently, certain items need to be written in a fixed order: First all extern crate
, then all use
and then all other items.
This has historically reasons, due to the older, more complex resolution algorithm, which included that shadowing was allowed between those items in that order,
and usability reasons, as it makes it easy to locate imports and library dependencies.
However, after RFC 50 got accepted there
is only ever one item name in scope from any given source so the historical “hard” reasons loose validity:
Any resolution algorithm that used to first process all extern crate
, then all use
and then all items can still do so, it
just has to filter out the relevant items from the whole module body, rather then from sequential sections of it.
And any usability reasons for keeping the order can be better addressed with conventions and lints, rather than hard parser rules.
(The exception here are the special cased prelude, and globs and macros, which are feature gated and out of scope for this proposal)
As it is, today the ordering rule is a unnecessary complication, as it routinely causes beginner to stumble over things like this:
mod foo;
use foo::bar; // ERROR: Imports have to precede items
In addition, it doesn’t even prevent certain patterns, as it is possible to work around the order restriction by using a submodule:
struct Foo;
// One of many ways to expose the crate out of order:
mod bar { extern crate bar; pub use self::bar::x; pub use self::bar::y; ... }
Which with this RFC implemented would be identical to
struct Foo;
extern crate bar;
Another use case are item macros/attributes that want to automatically include their their crate dependencies. This is possible by having the macro expand to an item that links to the needed crate, eg like this:
#[my_attribute]
struct UserType;
Expands to:
struct UserType;
extern crate "MyCrate" as <gensymb>
impl <gensymb>::MyTrait for UserType { ... }
With the order restriction still in place, this requires the sub module workaround, which is unnecessary verbose.
As an example, gfx-rs currently employs this strategy.
Allow pub extern crate
as opposed to only private ones.
extern crate
semantically is somewhere between use
ing a module, and declaring one with mod
,
and is identical to both as far as as the module path to it is considered.
As such, its surprising that its not possible to declare a extern crate
as public,
even though you can still make it so with an reexport:
mod foo {
extern crate "bar" as bar_;
pub use bar_ as bar;
}
While its generally not necessary to export a extern library directly, the need for it does arise occasionally during refactorings of huge crate collections, generally if a public module gets turned into its own crate.
As an example,the author recalls stumbling over it during a refactoring of gfx-rs.
Allow extern crate
in blocks/functions, and not just in modules.
Similar to the point above, its currently possible to both import and declare a module in a block expression or function body, but not to link to an library:
fn foo() {
let x = {
extern crate qux; // ERROR: Extern crate not allowed here
use bar::baz; // OK
mod bar { ... }; // OK
qux::foo()
};
}
This is again a unnecessary restriction considering that you can declare modules and imports there, and thus can make an extern library reachable at that point:
fn foo() {
let x = {
mod qux { extern crate "qux" as qux_; pub use self::qux_ as qux; }
qux::foo()
};
}
This again benefits macros and gives the developer the power to place external dependencies only needed for a single function lexically near it.
General benefits
In general, the simplification and freedom added by these changes would positively effect the docs of Rusts module system (which is already often regarded as too complex by outsiders), and possibly admit other simplifications or RFCs based on the now-equality of view items and items in the module system.
(As an example, the author is considering an RFC about merging the use
and type
features;
by lifting the ordering restriction they become more similar and thus more redundant)
This also does not have to be a 1.0 feature, as it is entirely backwards compatible to implement, and strictly allows more programs to compile than before. However, as alluded to above it might be a good idea for 1.0 regardless
Detailed design
- Remove the ordering restriction from resolve
- If necessary, change resolve to look in the whole scope block for view items, not just in a prefix of it.
- Make
pub extern crate
parse and teach privacy about it - Allow
extern crate
view items in blocks
Drawbacks
- The source of names in scope might be harder to track down
- Similarly, it might become confusing to see when a library dependency exist.
However, these issues already exist today in one form or another, and can be addressed by proper docs that make library dependencies clear, and by the fact that definitions are generally greppable in a file.
Alternatives
As this just cleans up a few aspects of the module system, there isn’t really an alternative apart from not or only partially implementing it.
By not implementing this proposal, the module system remains more complex for the user than necessary.
Unresolved questions
-
Inner attributes occupy the same syntactic space as items and view items, and are currently also forced into a given order by needing to be written first. This is also potentially confusing or restrictive for the same reasons as for the view items mentioned above, especially in regard to the build-in crate attributes, and has one big issue: It is currently not possible to load a syntax extension that provides an crate-level attribute, as with the current macro system this would have to be written like this:
#[phase(plugin)] extern crate mycrate; #![myattr]
Which is impossible to write due to the ordering restriction. However, as attributes and the macro system are also not finalized, this has not been included in this RFC directly.
-
This RFC does also explicitly not talk about wildcard imports and macros in regard to resolution, as those are feature gated today and likely subject to change. In any case, it seems unlikely that they will conflict with the changes proposed here, as macros would likely follow the same module system rules where possible, and wildcard imports would either be removed, or allowed in a way that doesn’t conflict with explicitly imported names to prevent compilation errors on upstream library changes (new public item may not conflict with downstream items).
- Start Date: 2014-10-10
- RFC PR: rust-lang/rfcs#387
- Rust Issue: rust-lang/rust#18639
Summary
- Add the ability to have trait bounds that are polymorphic over lifetimes.
Motivation
Currently, closure types can be polymorphic over lifetimes. But closure types are deprecated in favor of traits and object types as part of RFC #44 (unboxed closures). We need to close the gap. The canonical example of where you want this is if you would like a closure that accepts a reference with any lifetime. For example, today you might write:
fn with(callback: |&Data|) {
let data = Data { ... };
callback(&data)
}
If we try to write this using unboxed closures today, we have a problem:
fn with<'a, T>(callback: T)
where T : FnMut(&'a Data)
{
let data = Data { ... };
callback(&data)
}
// Note that the `()` syntax is shorthand for the following:
fn with<'a, T>(callback: T)
where T : FnMut<(&'a Data,),()>
{
let data = Data { ... };
callback(&data)
}
The problem is that the argument type &'a Data
must include a
lifetime, and there is no lifetime one could write in the fn sig that
represents “the stack frame of the with
function”. Naturally
we have the same problem if we try to use an FnMut
object (which is
the closer analog to the original closure example):
fn with<'a>(callback: &mut FnMut(&'a Data))
{
let data = Data { ... };
callback(&data)
}
fn with<'a>(callback: &mut FnMut<(&'a Data,),()>)
{
let data = Data { ... };
callback(&data)
}
Under this proposal, you would be able to write this code as follows:
// Using the FnMut(&Data) notation, the &Data is
// in fact referencing an implicit bound lifetime, just
// as with closures today.
fn with<T>(callback: T)
where T : FnMut(&Data)
{
let data = Data { ... };
callback(&data)
}
// If you prefer, you can use an explicit name,
// introduced by the `for<'a>` syntax.
fn with<T>(callback: T)
where T : for<'a> FnMut(&'a Data)
{
let data = Data { ... };
callback(&data)
}
// No sugar at all.
fn with<T>(callback: T)
w