Rust RFCs - RFC Book - Active RFC List

The “RFC” (request for comments) process is intended to provide a consistent and controlled path for changes to Rust (such as new features) so that all stakeholders can be confident about the direction of the project.

Many changes, including bug fixes and documentation improvements can be implemented and reviewed via the normal GitHub pull request workflow.

Some changes though are “substantial”, and we ask that these be put through a bit of a design process and produce a consensus among the Rust community and the sub-teams.

Table of Contents

When you need to follow this process

You need to follow this process if you intend to make “substantial” changes to Rust, Cargo, Crates.io, or the RFC process itself. What constitutes a “substantial” change is evolving based on community norms and varies depending on what part of the ecosystem you are proposing to change, but may include the following.

  • Any semantic or syntactic change to the language that is not a bugfix.
  • Removing language features, including those that are feature-gated.
  • Changes to the interface between the compiler and libraries, including lang items and intrinsics.
  • Additions to std.

Some changes do not require an RFC:

  • Rephrasing, reorganizing, refactoring, or otherwise “changing shape does not change meaning”.
  • Additions that strictly improve objective, numerical quality criteria (warning removal, speedup, better platform coverage, more parallelism, trap more errors, etc.)
  • Additions only likely to be noticed by other developers-of-rust, invisible to users-of-rust.

If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first.

Sub-team specific guidelines

For more details on when an RFC is required for the following areas, please see the Rust community’s sub-team specific guidelines for:

Before creating an RFC

A hastily-proposed RFC can hurt its chances of acceptance. Low quality proposals, proposals for previously-rejected features, or those that don’t fit into the near-term roadmap, may be quickly rejected, which can be demotivating for the unprepared contributor. Laying some groundwork ahead of the RFC can make the process smoother.

Although there is no single way to prepare for submitting an RFC, it is generally a good idea to pursue feedback from other project developers beforehand, to ascertain that the RFC may be desirable; having a consistent impact on the project requires concerted effort toward consensus-building.

The most common preparations for writing and submitting an RFC include talking the idea over on our official Zulip server, discussing the topic on our developer discussion forum, and occasionally posting “pre-RFCs” on the developer forum. You may file issues on this repo for discussion, but these are not actively looked at by the teams.

As a rule of thumb, receiving encouraging feedback from long-standing project developers, and particularly members of the relevant sub-team is a good indication that the RFC is worth pursuing.

What the process is

In short, to get a major feature added to Rust, one must first get the RFC merged into the RFC repository as a markdown file. At that point the RFC is “active” and may be implemented with the goal of eventual inclusion into Rust.

  • Fork the RFC repo RFC repository
  • Copy 0000-template.md to text/0000-my-feature.md (where “my-feature” is descriptive). Don’t assign an RFC number yet; This is going to be the PR number and we’ll rename the file accordingly if the RFC is accepted.
  • Fill in the RFC. Put care into the details: RFCs that do not present convincing motivation, demonstrate lack of understanding of the design’s impact, or are disingenuous about the drawbacks or alternatives tend to be poorly-received.
  • Submit a pull request. As a pull request the RFC will receive design feedback from the larger community, and the author should be prepared to revise it in response.
  • Now that your RFC has an open pull request, use the issue number of the PR to rename the file: update your 0000- prefix to that number. Also update the “RFC PR” link at the top of the file.
  • Each pull request will be labeled with the most relevant sub-team, which will lead to its being triaged by that team in a future meeting and assigned to a member of the subteam.
  • Build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don’t receive any comments. Feel free to reach out to the RFC assignee in particular to get help identifying stakeholders and obstacles.
  • The sub-team will discuss the RFC pull request, as much as possible in the comment thread of the pull request itself. Offline discussion will be summarized on the pull request comment thread.
  • RFCs rarely go through this process unchanged, especially as alternatives and drawbacks are shown. You can make edits, big and small, to the RFC to clarify or change the design, but make changes as new commits to the pull request, and leave a comment on the pull request explaining your changes. Specifically, do not squash or rebase commits after they are visible on the pull request.
  • At some point, a member of the subteam will propose a “motion for final comment period” (FCP), along with a disposition for the RFC (merge, close, or postpone).
    • This step is taken when enough of the tradeoffs have been discussed that the subteam is in a position to make a decision. That does not require consensus amongst all participants in the RFC thread (which is usually impossible). However, the argument supporting the disposition on the RFC needs to have already been clearly articulated, and there should not be a strong consensus against that position outside of the subteam. Subteam members use their best judgment in taking this step, and the FCP itself ensures there is ample time and notification for stakeholders to push back if it is made prematurely.
    • For RFCs with lengthy discussion, the motion to FCP is usually preceded by a summary comment trying to lay out the current state of the discussion and major tradeoffs/points of disagreement.
    • Before actually entering FCP, all members of the subteam must sign off; this is often the point at which many subteam members first review the RFC in full depth.
  • The FCP lasts ten calendar days, so that it is open for at least 5 business days. It is also advertised widely, e.g. in This Week in Rust. This way all stakeholders have a chance to lodge any final objections before a decision is reached.
  • In most cases, the FCP period is quiet, and the RFC is either merged or closed. However, sometimes substantial new arguments or ideas are raised, the FCP is canceled, and the RFC goes back into development mode.

The RFC life-cycle

Once an RFC becomes “active” then authors may implement it and submit the feature as a pull request to the Rust repo. Being “active” is not a rubber stamp, and in particular still does not mean the feature will ultimately be merged; it does mean that in principle all the major stakeholders have agreed to the feature and are amenable to merging it.

Furthermore, the fact that a given RFC has been accepted and is “active” implies nothing about what priority is assigned to its implementation, nor does it imply anything about whether a Rust developer has been assigned the task of implementing the feature. While it is not necessary that the author of the RFC also write the implementation, it is by far the most effective way to see an RFC through to completion: authors should not expect that other project developers will take on responsibility for implementing their accepted feature.

Modifications to “active” RFCs can be done in follow-up pull requests. We strive to write each RFC in a manner that it will reflect the final design of the feature; but the nature of the process means that we cannot expect every merged RFC to actually reflect what the end result will be at the time of the next major release.

In general, once accepted, RFCs should not be substantially changed. Only very minor changes should be submitted as amendments. More substantial changes should be new RFCs, with a note added to the original RFC. Exactly what counts as a “very minor change” is up to the sub-team to decide; check Sub-team specific guidelines for more details.

Reviewing RFCs

While the RFC pull request is up, the sub-team may schedule meetings with the author and/or relevant stakeholders to discuss the issues in greater detail, and in some cases the topic may be discussed at a sub-team meeting. In either case a summary from the meeting will be posted back to the RFC pull request.

A sub-team makes final decisions about RFCs after the benefits and drawbacks are well understood. These decisions can be made at any time, but the sub-team will regularly issue decisions. When a decision is made, the RFC pull request will either be merged or closed. In either case, if the reasoning is not clear from the discussion in thread, the sub-team will add a comment describing the rationale for the decision.

Implementing an RFC

Some accepted RFCs represent vital features that need to be implemented right away. Other accepted RFCs can represent features that can wait until some arbitrary developer feels like doing the work. Every accepted RFC has an associated issue tracking its implementation in the Rust repository; thus that associated issue can be assigned a priority via the triage process that the team uses for all issues in the Rust repository.

The author of an RFC is not obligated to implement it. Of course, the RFC author (like any other developer) is welcome to post an implementation for review after the RFC has been accepted.

If you are interested in working on the implementation for an “active” RFC, but cannot determine if someone else is already working on it, feel free to ask (e.g. by leaving a comment on the associated issue).

RFC Postponement

Some RFC pull requests are tagged with the “postponed” label when they are closed (as part of the rejection process). An RFC closed with “postponed” is marked as such because we want neither to think about evaluating the proposal nor about implementing the described feature until some time in the future, and we believe that we can afford to wait until then to do so. Historically, “postponed” was used to postpone features until after 1.0. Postponed pull requests may be re-opened when the time is right. We don’t have any formal process for that, you should ask members of the relevant sub-team.

Usually an RFC pull request marked as “postponed” has already passed an informal first round of evaluation, namely the round of “do we think we would ever possibly consider making this change, as outlined in the RFC pull request, or some semi-obvious variation of it.” (When the answer to the latter question is “no”, then the appropriate response is to close the RFC, not postpone it.)

Help this is all too informal!

The process is intended to be as lightweight as reasonable for the present circumstances. As usual, we are trying to let the process be driven by consensus and community norms, not impose more structure than necessary.

License

This repository is currently in the process of being licensed under either of:

  • Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
  • MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option. Some parts of the repository are already licensed according to those terms. For more see RFC 2044 and its tracking issue.

Contributions

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

RFC policy - the compiler

Compiler RFCs will be managed by the compiler sub-team, and tagged T-compiler. The compiler sub-team will do an initial triage of new PRs within a week of submission. The result of triage will either be that the PR is assigned to a member of the sub-team for shepherding, the PR is closed because the sub-team believe it should be done without an RFC, or closed because the sub-team feel it should clearly not be done and further discussion is not necessary. We’ll follow the standard procedure for shepherding, final comment period, etc.

Most compiler decisions that go beyond the scope of a simple PR are done using MCPs, not RFCs. It is therefore likely that you should file an MCP instead of an RFC for your problem.

Changes which need an RFC

  • Significant user-facing changes to the compiler with a complex design space, especially if they involve other teams as well (for example, path sanitization).
  • Any other change which causes significant backwards incompatible changes to stable behaviour of the compiler, language, or libraries

Changes which don’t need an RFC

  • Bug fixes, improved error messages, etc.
  • Minor refactoring/tidying up
  • Large internal refactorings or redesigns of the compiler (needs an MCP)
  • Implementing language features which have an accepted RFC.
  • New lints (these fall under the lang team). Lints are best first tried out in clippy and then uplifted later.
  • Changing the API presented to syntax extensions or other compiler plugins in non-trivial ways
  • Adding, removing, or changing a stable compiler flag (needs an FCP somewhere, like on an MCP or just on a PR)
  • Adding unstable API for tools (note that all compiler API is currently unstable)
  • Adding, removing, or changing an unstable compiler flag (if the compiler flag is widely used there should be at least some discussion on discuss, or an RFC in some cases)

If in doubt it is probably best to just announce the change you want to make to the compiler subteam on Zulip, and see if anyone feels it needs an RFC.

RFC policy - language design

Pretty much every change to the language needs an RFC. Note that new lints (or major changes to an existing lint) are considered changes to the language.

Language RFCs are managed by the language sub-team, and tagged T-lang. The language sub-team will do an initial triage of new PRs within a week of submission. The result of triage will either be that the PR is assigned to a member of the sub-team for shepherding, the PR is closed as postponed because the subteam believe it might be a good idea, but is not currently aligned with Rust’s priorities, or the PR is closed because the sub-team feel it should clearly not be done and further discussion is not necessary. In the latter two cases, the sub-team will give a detailed explanation. We’ll follow the standard procedure for shepherding, final comment period, etc.

Amendments

Sometimes in the implementation of an RFC, changes are required. In general these don’t require an RFC as long as they are very minor and in the spirit of the accepted RFC (essentially bug fixes). In this case implementers should submit an RFC PR which amends the accepted RFC with the new details. Although the RFC repository is not intended as a reference manual, it is preferred that RFCs do reflect what was actually implemented. Amendment RFCs will go through the same process as regular RFCs, but should be less controversial and thus should move more quickly.

When a change is more dramatic, it is better to create a new RFC. The RFC should be standalone and reference the original, rather than modifying the existing RFC. You should add a comment to the original RFC with referencing the new RFC as part of the PR.

Obviously there is some scope for judgment here. As a guideline, if a change affects more than one part of the RFC (i.e., is a non-local change), affects the applicability of the RFC to its motivating use cases, or there are multiple possible new solutions, then the feature is probably not ‘minor’ and should get a new RFC.

RFC guidelines - libraries sub-team

Motivation

  • RFCs are heavyweight:

    • RFCs generally take at minimum 2 weeks from posting to land. In practice it can be more on the order of months for particularly controversial changes.
    • RFCs are a lot of effort to write; especially for non-native speakers or for members of the community whose strengths are more technical than literary.
    • RFCs may involve pre-RFCs and several rewrites to accommodate feedback.
    • RFCs require a dedicated shepherd to herd the community and author towards consensus.
    • RFCs require review from a majority of the subteam, as well as an official vote.
    • RFCs can’t be downgraded based on their complexity. Full process always applies. Easy RFCs may certainly land faster, though.
    • RFCs can be very abstract and hard to grok the consequences of (no implementation).
  • PRs are low overhead but potentially expensive nonetheless:

    • Easy PRs can get insta-merged by any rust-lang contributor.
    • Harder PRs can be easily escalated. You can ping subject-matter experts for second opinions. Ping the whole team!
    • Easier to grok the full consequences. Lots of tests and Crater to save the day.
    • PRs can be accepted optimistically with bors, buildbot, and the trains to guard us from major mistakes making it into stable. The size of the nightly community at this point in time can still mean major community breakage regardless of trains, however.
    • HOWEVER: Big PRs can be a lot of work to make only to have that work rejected for details that could have been hashed out first.
  • RFCs are only meaningful if a significant and diverse portion of the community actively participates in them. The official teams are not sufficiently diverse to establish meaningful community consensus by agreeing amongst themselves.

  • If there are tons of RFCs – especially trivial ones – people are less likely to engage with them. Official team members are super busy. Domain experts and industry professionals are super busy and have no responsibility to engage in RFCs. Since these are exactly the most important people to get involved in the RFC process, it is important that we be maximally friendly towards their needs.

Is an RFC required?

The overarching philosophy is: do whatever is easiest. If an RFC would be less work than an implementation, that’s a good sign that an RFC is necessary. That said, if you anticipate controversy, you might want to short-circuit straight to an RFC. For instance new APIs almost certainly merit an RFC. Especially as std has become more conservative in favour of the much more agile cargoverse.

  • Submit a PR if the change is a:
    • Bugfix
    • Docfix
    • Obvious API hole patch, such as adding an API from one type to a symmetric type. e.g. Vec<T> -> Box<[T]> clearly motivates adding String -> Box<str>
    • Minor tweak to an unstable API (renaming, generalizing)
    • Implementing an “obvious” trait like Clone/Debug/etc
  • Submit an RFC if the change is a:
    • New API
    • Semantic Change to a stable API
    • Generalization of a stable API (e.g. how we added Pattern or Borrow)
    • Deprecation of a stable API
    • Nontrivial trait impl (because all trait impls are insta-stable)
  • Do the easier thing if uncertain. (choosing a path is not final)

Non-RFC process

  • A (non-RFC) PR is likely to be closed if clearly not acceptable:

    • Disproportionate breaking change (small inference breakage may be acceptable)
    • Unsound
    • Doesn’t fit our general design philosophy around the problem
    • Better as a crate
    • Too marginal for std
    • Significant implementation problems
  • A PR may also be closed because an RFC is appropriate.

  • A (non-RFC) PR may be merged as unstable. In this case, the feature should have a fresh feature gate and an associated tracking issue for stabilisation. Note that trait impls and docs are insta-stable and thus have no tracking issue. This may imply requiring a higher level of scrutiny for such changes.

However, an accepted RFC is not a rubber-stamp for merging an implementation PR. Nor must an implementation PR perfectly match the RFC text. Implementation details may merit deviations, though obviously they should be justified. The RFC may be amended if deviations are substantial, but are not generally necessary. RFCs should favour immutability. The RFC + Issue + PR should form a total explanation of the current implementation.

  • Once something has been merged as unstable, a shepherd should be assigned to promote and obtain feedback on the design.

  • Every time a release cycle ends, the libs teams assesses the current unstable APIs and selects some number of them for potential stabilization during the next cycle. These are announced for FCP at the beginning of the cycle, and (possibly) stabilized just before the beta is cut.

  • After the final comment period, an API should ideally take one of two paths:

    • Stabilize if the change is desired, and consensus is reached
    • Deprecate is the change is undesired, and consensus is reached
    • Extend the FCP is the change cannot meet consensus
      • If consensus still can’t be reached, consider requiring a new RFC or just deprecating as “too controversial for std”.
  • If any problems are found with a newly stabilized API during its beta period, strongly favour reverting stability in order to prevent stabilizing a bad API. Due to the speed of the trains, this is not a serious delay (~2-3 months if it’s not a major problem).

Summary

This is an RFC to make all struct fields private by default. This includes both tuple structs and structural structs.

Motivation

Reasons for default private visibility

  • Visibility is often how soundness is achieved for many types in rust. These types are normally wrapping unsafe behavior of an FFI type or some other rust-specific behavior under the hood (such as the standard Vec type). Requiring these types to opt-in to being sound is unfortunate.

  • Forcing tuple struct fields to have non-overridable public visibility greatly reduces the utility of such types. Tuple structs cannot be used to create abstraction barriers as they can always be easily destructed.

  • Private-by-default is more consistent with the rest of the Rust language. All other aspects of privacy are private-by-default except for enum variants. Enum variants, however, are a special case in that they are inserted into the parent namespace, and hence naturally inherit privacy.

  • Public fields of a struct must be considered as part of the API of the type. This means that the exact definition of all structs is by default the API of the type. Structs must opt-out of this behavior if the priv keyword is required. By requiring the pub keyword, structs must opt-in to exposing more surface area to their API.

Reasons for inherited visibility (today’s design)

  • Public definitions like pub struct Point { x: int, y: int } are concise and easy to read.
  • Private definitions certainly want private fields (to hide implementation details).

Detailed design

Currently, rustc has two policies for dealing with the privacy of struct fields:

  • Tuple structs have public fields by default (including “newtype structs”)
  • Fields of structural structs (struct Foo { ... }) inherit the same privacy of the enclosing struct.

This RFC is a proposal to unify the privacy of struct fields with the rest of the language by making them private by default. This means that both tuple variants and structural variants of structs would have private fields by default. For example, the program below is accepted today, but would be rejected with this RFC.

mod inner {
    pub struct Foo(u64);
    pub struct Bar { field: u64 }
}

fn main() {
    inner::Foo(10);
    inner::Bar { field: 10 };
}

Refinements to structural structs

Public fields are quite a useful feature of the language, so syntax is required to opt out of the private-by-default semantics. Structural structs already allow visibility qualifiers on fields, and the pub qualifier would make the field public instead of private.

Additionally, the priv visibility will no longer be allowed to modify struct fields. Similarly to how a priv fn is a compiler error, a priv field will become a compiler error.

Refinements on tuple structs

As with their structural cousins, it’s useful to have tuple structs with public fields. This RFC will modify the tuple struct grammar to:

tuple_struct := 'struct' ident '(' fields ')' ';'
fields := field | field ',' fields
field := type | visibility type

For example, these definitions will be added to the language:

// a "newtype wrapper" struct with a private field
struct Foo(u64);

// a "newtype wrapper" struct with a public field
struct Bar(pub u64);

// a tuple struct with many fields, only the first and last of which are public
struct Baz(pub u64, u32, f32, pub int);

Public fields on tuple structs will maintain the semantics that they currently have today. The structs can be constructed, destructed, and participate in pattern matches.

Private fields on tuple structs will prevent the following behaviors:

  • Private fields cannot be bound in patterns (both in irrefutable and refutable contexts, i.e. let and match statements).
  • Private fields cannot be specified outside of the defining module when constructing a tuple struct.

These semantics are intended to closely mirror the behavior of private fields for structural structs.

Statistics gathered

A brief survey was performed over the entire mozilla/rust repository to gather these statistics. While not representative of all projects, this repository should give a good indication of what most structs look like in the real world. The repository has both libraries (libstd) as well as libraries which don’t care much about privacy (librustc).

These numbers tally up all structs from all locations, and only take into account structural structs, not tuple structs.

Inherited privacyPrivate-by-default
Private fields14181852
Public fields20361602
All-private structs551 (52.23%)671 (63.60%)
All-public structs468 (44.36%)352 (33.36%)
Mixed privacy structs36 ( 3.41%)32 ( 3.03%)

The numbers clearly show that the predominant pattern is to have all-private structs, and that there are many public fields today which can be private (and perhaps should!). Additionally, there is on the order of 1418 instances of the word priv today, when in theory there should be around 1852. With this RFC, there would need to be 1602 instances of the word pub. A very large portion of structs requiring pub fields are FFI structs defined in the libc module.

Impact on enums

This RFC does not impact enum variants in any way. All enum variants will continue to inherit privacy from the outer enum type. This includes both the fields of tuple variants as well as fields of struct variants in enums.

Alternatives

The main alternative to this design is what is currently implemented today, where fields inherit the privacy of the outer structure. The pros and cons of this strategy are discussed above.

Unresolved questions

As the above statistics show, almost all structures are either all public or all private. This RFC provides an easy method to make struct fields all private, but it explicitly does not provide a method to make struct fields all public. The statistics show that pub will be written less often than priv is today, and it’s always possible to add a method to specify a struct as all-public in the future in a backwards-compatible fashion.

That being said, it’s an open question whether syntax for an “all public struct” is necessary at this time.

Summary

The “RFC” (request for comments) process is intended to provide a consistent and controlled path for new features to enter the language and standard libraries, so that all stakeholders can be confident about the direction the language is evolving in.

Motivation

The freewheeling way that we add new features to Rust has been good for early development, but for Rust to become a mature platform we need to develop some more self-discipline when it comes to changing the system. This is a proposal for a more principled RFC process to make it a more integral part of the overall development process, and one that is followed consistently to introduce features to Rust.

Detailed design

Many changes, including bug fixes and documentation improvements can be implemented and reviewed via the normal GitHub pull request workflow.

Some changes though are “substantial”, and we ask that these be put through a bit of a design process and produce a consensus among the Rust community and the core team.

When you need to follow this process

You need to follow this process if you intend to make “substantial” changes to the Rust distribution. What constitutes a “substantial” change is evolving based on community norms, but may include the following.

  • Any semantic or syntactic change to the language that is not a bugfix.
  • Removing language features, including those that are feature-gated.
  • Changes to the interface between the compiler and libraries, including lang items and intrinsics.
  • Additions to std

Some changes do not require an RFC:

  • Rephrasing, reorganizing, refactoring, or otherwise “changing shape does not change meaning”.
  • Additions that strictly improve objective, numerical quality criteria (warning removal, speedup, better platform coverage, more parallelism, trap more errors, etc.)
  • Additions only likely to be noticed by other developers-of-rust, invisible to users-of-rust.

If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first.

What the process is

In short, to get a major feature added to Rust, one must first get the RFC merged into the RFC repo as a markdown file. At that point the RFC is ‘active’ and may be implemented with the goal of eventual inclusion into Rust.

  • Fork the RFC repo https://github.com/rust-lang/rfcs
  • Copy 0000-template.md to text/0000-my-feature.md (where ‘my-feature’ is descriptive. don’t assign an RFC number yet).
  • Fill in the RFC
  • Submit a pull request. The pull request is the time to get review of the design from the larger community.
  • Build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don’t receive any comments.

Eventually, somebody on the core team will either accept the RFC by merging the pull request, at which point the RFC is ‘active’, or reject it by closing the pull request.

Whomever merges the RFC should do the following:

  • Assign an id, using the PR number of the RFC pull request. (If the RFC has multiple pull requests associated with it, choose one PR number, preferably the minimal one.)
  • Add the file in the text/ directory.
  • Create a corresponding issue on Rust repo
  • Fill in the remaining metadata in the RFC header, including links for the original pull request(s) and the newly created Rust issue.
  • Add an entry in the Active RFC List of the root README.md.
  • Commit everything.

Once an RFC becomes active then authors may implement it and submit the feature as a pull request to the Rust repo. An ‘active’ is not a rubber stamp, and in particular still does not mean the feature will ultimately be merged; it does mean that in principle all the major stakeholders have agreed to the feature and are amenable to merging it.

Modifications to active RFC’s can be done in followup PR’s. An RFC that makes it through the entire process to implementation is considered ‘complete’ and is removed from the Active RFC List; an RFC that fails after becoming active is ‘inactive’ and moves to the ‘inactive’ folder.

Alternatives

Retain the current informal RFC process. The newly proposed RFC process is designed to improve over the informal process in the following ways:

  • Discourage unactionable or vague RFCs
  • Ensure that all serious RFCs are considered equally
  • Give confidence to those with a stake in Rust’s development that they understand why new features are being merged

As an alternative, we could adopt an even stricter RFC process than the one proposed here. If desired, we should likely look to Python’s PEP process for inspiration.

Unresolved questions

  1. Does this RFC strike a favorable balance between formality and agility?
  2. Does this RFC successfully address the aforementioned issues with the current informal RFC process?
  3. Should we retain rejected RFCs in the archive?

Summary

Rust currently has an attribute usage lint but it does not work particularly well. This RFC proposes a new implementation strategy that should make it significantly more useful.

Motivation

The current implementation has two major issues:

  • There are very limited warnings for valid attributes that end up in the wrong place. Something like this will be silently ignored:
#[deriving(Clone)]; // Shouldn't have put a ; here
struct Foo;

#[ignore(attribute-usage)] // Should have used #[allow(attribute-usage)] instead!
mod bar {
    //...
}
  • ItemDecorators can now be defined outside of the compiler, and there’s no way to tag them and associated attributes as valid. Something like this requires an #[allow(attribute-usage)]:
#[feature(phase)];
#[phase(syntax, link)]
extern crate some_orm;

#[ormify]
pub struct Foo {
    #[column(foo_)]
    #[primary_key]
    foo: int
}

Detailed design

The current implementation is implemented as a simple fold over the AST, comparing attributes against a whitelist. Crate-level attributes use a separate whitelist, but no other distinctions are made.

This RFC would change the implementation to actually track which attributes are used during the compilation process. syntax::ast::Attribute_ would be modified to add an ID field:

pub struct AttrId(uint);

pub struct Attribute_ {
    id: AttrId,
    style: AttrStyle,
    value: @MetaItem,
    is_sugared_doc: bool,
}

syntax::ast::parse::ParseSess will generate new AttrIds on demand. I believe that attributes will only be created during parsing and expansion, and the ParseSess is accessible in both.

The AttrIds will be used to create a side table of used attributes. This will most likely be a thread local to make it easily accessible during all stages of compilation by calling a function in syntax::attr:

fn mark_used(attr: &Attribute) { }

The attribute-usage lint would run at the end of compilation and warn on all attributes whose ID does not appear in the side table.

One interesting edge case is attributes like doc that are used, but not in the normal compilation process. There could either be a separate fold pass to mark all doc attributes as used or doc could simply be whitelisted in the attribute-usage lint.

Attributes in code that has been eliminated with #[cfg()] will not be linted, but I feel that this is consistent with the way #[cfg()] works in general (e.g. the code won’t be type-checked either).

Alternatives

An alternative would be to rewrite rustc::middle::lint to robustly check that attributes are used where they’re supposed to be. This will be fairly complex and be prone to failure if/when more nodes are added to the AST. This also doesn’t solve motivation #2, which would require externally loaded lint support.

Unresolved questions

  • This implementation doesn’t allow for a distinction between “unused” and “unknown” attributes. The #[phase(syntax)] crate loading infrastructure could be extended to pull a list of attributes from crates to use in the lint pass, but I’m not sure if the extra complexity is worth it.
  • The side table could be threaded through all of the compilation stages that need to use it instead of being a thread local. This would probably require significantly more work than the thread local approach, however. The thread local approach should not negatively impact any future parallelization work as each thread can keep its own side table, which can be merged into one for the lint pass.

Note: this RFC was never implemented and has been retired. The design may still be useful in the future, but before implementing we would prefer to revisit it so as to be sure it is up to date.

Summary

The way our intrinsics work forces them to be wrapped in order to behave like normal functions. As a result, rustc is forced to inline a great number of tiny intrinsic wrappers, which is bad for both compile-time performance and runtime performance without optimizations. This proposal changes the way intrinsics are surfaced in the language so that they behave the same as normal Rust functions by removing the “rust-intrinsic” foreign ABI and reusing the “Rust” ABI.

Motivation

A number of commonly-used intrinsics, including transmute, forget, init, uninit, and move_val_init, are accessed through wrappers whose only purpose is to present the intrinsics as normal functions. As a result, rustc is forced to inline a great number of tiny intrinsic wrappers, which is bad for both compile-time performance and runtime performance without optimizations.

Intrinsics have a differently-named ABI from Rust functions (“rust-intrinsic” vs. “Rust”) though the actual ABI implementation is identical. As a result one can’t take the value of an intrinsic as a function:

// error: the type of transmute is `extern "rust-intrinsic" fn ...`
let transmute: fn(int) -> uint = intrinsics::transmute;

This incongruity means that we can’t just expose the intrinsics directly as part of the public API.

Detailed design

extern "Rust" fn is already equivalent to fn, so if intrinsics have the “Rust” ABI then the problem is solved.

Under this scheme intrinsics will be declared as extern "Rust" functions and identified as intrinsics with the #[lang = "..."] attribute:

extern "Rust" {
    #[lang = "transmute"]
    fn transmute<T, U>(T) -> U;
}

The compiler will type check and translate intrinsics the same as today. Additionally, when trans sees a “Rust” extern tagged as an intrinsic it will not emit a function declaration to LLVM bitcode.

Because intrinsics will be lang items, they can no longer be redeclared arbitrary number of times. This will require a small amount of existing library code to be refactored, and all intrinsics to be exposed through public abstractions.

Currently, “Rust” foreign functions may not be generic; this change will require a special case that allows intrinsics to be generic.

Alternatives

  1. Instead of making intrinsics lang items we could create a slightly different mechanism, like an #[intrinsic] attribute, that would continue letting intrinsics to be redeclared.

  2. While using lang items to identify intrinsics, intrinsic lang items could be allowed to be redeclared.

  3. We could also make “rust-intrinsic” coerce or otherwise be the same as “Rust” externs and normal Rust functions.

Unresolved questions

None.

Summary

Allow attributes on more places inside functions, such as statements, blocks and expressions.

Motivation

One sometimes wishes to annotate things inside functions with, for example, lint #[allow]s, conditional compilation #[cfg]s, and even extra semantic (or otherwise) annotations for external tools.

For the lints, one can currently only activate lints at the level of the function which is possibly larger than one needs, and so may allow other “bad” things to sneak through accidentally. E.g.

#[allow(uppercase_variable)]
let L = List::new(); // lowercase looks like one or capital i

For the conditional compilation, the work-around is duplicating the whole containing function with a #[cfg], or breaking the conditional code into a its own function. This does mean that any variables need to be explicitly passed as arguments.

The sort of things one could do with other arbitrary annotations are

#[allowed_unsafe_actions(ffi)]
#[audited="2014-04-22"]
unsafe { ... }

and then have an external tool that checks that that unsafe block’s only unsafe actions are FFI, or a tool that lists blocks that have been changed since the last audit or haven’t been audited ever.

The minimum useful functionality would be supporting attributes on blocks and let statements, since these are flexible enough to allow for relatively precise attribute handling.

Detailed design

Normal attribute syntax on let statements, blocks and expressions.

fn foo() {
    #[attr1]
    let x = 1;

    #[attr2]
    {
        // code
    }

    #[attr3]
    unsafe {
        // code
    }
    #[attr4] foo();

    let x = #[attr5] 1;

    qux(3 + #[attr6] 2);

    foo(x, #[attr7] y, z);
}

Attributes bind tighter than any operator, that is #[attr] x op y is always parsed as (#[attr] x) op y.

cfg

It is definitely an error to place a #[cfg] attribute on a non-statement expressions, that is, attr1attr4 can possibly be #[cfg(foo)], but attr5attr7 cannot, since it makes little sense to strip code down to let x = ;.

However, like #ifdef in C/C++, widespread use of #[cfg] may be an antipattern that makes code harder to read. This RFC is just adding the ability for attributes to be placed in specific places, it is not mandating that #[cfg] actually be stripped in those places (although it should be an error if it is ignored).

Inner attributes

Inner attributes can be placed at the top of blocks (and other structure incorporating a block) and apply to that block.

{
    #![attr11]

    foo()
}

match bar {
    #![attr12]

    _ => {}
}

// are the same as

#[attr11]
{
    foo()
}

#[attr12]
match bar {
    _ => {}
}

if

Attributes would be disallowed on if for now, because the interaction with if/else chains are funky, and can be simulated in other ways.

#[cfg(not(foo))]
if cond1 {
} else #[cfg(not(bar))] if cond2 {
} else #[cfg(not(baz))] {
}

There is two possible interpretations of such a piece of code, depending on if one regards the attributes as attaching to the whole if ... else chain (“exterior”) or just to the branch on which they are placed (“interior”).

  • --cfg foo: could be either removing the whole chain (exterior) or equivalent to if cond2 {} else {} (interior).
  • --cfg bar: could be either if cond1 {} (e) or if cond1 {} else {} (i)
  • --cfg baz: equivalent to if cond1 {} else if cond2 {} (no subtlety).
  • --cfg foo --cfg bar: could be removing the whole chain (e) or the two if branches (leaving only the else branch) (i).

(This applies to any attribute that has some sense of scoping, not just #[cfg], e.g. #[allow] and #[warn] for lints.)

As such, to avoid confusion, attributes would not be supported on if. Alternatives include using blocks:

#[attr] if cond { ... } else ...
// becomes, for an exterior attribute,
#[attr] {
    if cond { ... } else ...
}
// and, for an interior attribute,
if cond {
    #[attr] { ... }
} else ...

And, if the attributes are meant to be associated with the actual branching (e.g. a hypothetical #[cold] attribute that indicates a branch is unlikely), one can annotate match arms:

match cond {
    #[attr] true => { ... }
    #[attr] false => { ... }
}

Drawbacks

This starts mixing attributes with nearly arbitrary code, possibly dramatically restricting syntactic changes related to them, for example, there was some consideration for using @ for attributes, this change may make this impossible (especially if @ gets reused for something else, e.g. Python is using it for matrix multiplication). It may also make it impossible to use # for other things.

As stated above, allowing #[cfg]s everywhere can make code harder to reason about, but (also stated), this RFC is not for making such #[cfg]s be obeyed, it just opens the language syntax to possibly allow it.

Alternatives

These instances could possibly be approximated with macros and helper functions, but to a low degree degree (e.g. how would one annotate a general unsafe block).

Only allowing attributes on “statement expressions” that is, expressions at the top level of a block, this is slightly limiting; but we can expand to support other contexts backwards compatibly in the future.

The if/else issue may be able to be resolved by introducing explicit “interior” and “exterior” attributes on if: by having #[attr] if cond { ... be an exterior attribute (applying to the whole if/else chain) and if cond #[attr] { ... be an interior attribute (applying to only the current if branch). There is no difference between interior and exterior for an else { branch, and so else #[attr] { is sufficient.

Unresolved questions

Are the complications of allowing attributes on arbitrary expressions worth the benefits?

Note: The Share trait described in this RFC was later renamed to Sync.

Summary

The high-level idea is to add language features that simultaneously achieve three goals:

  1. move Send and Share out of the language entirely and into the standard library, providing mechanisms for end users to easily implement and use similar “marker” traits of their own devising;
  2. make “normal” Rust types sendable and sharable by default, without the need for explicit opt-in; and,
  3. continue to require “unsafe” Rust types (those that manipulate unsafe pointers or implement special abstractions) to “opt-in” to sendability and sharability with an unsafe declaration.

These goals are achieved by two changes:

  1. Unsafe traits: An unsafe trait is a trait that is unsafe to implement, because it represents some kind of trusted assertion. Note that unsafe traits are perfectly safe to use. Send and Share are examples of unsafe traits: implementing these traits is effectively an assertion that your type is safe for threading.

  2. Default and negative impls: A default impl is one that applies to all types, except for those types that explicitly opt out. For example, there would be a default impl for Send, indicating that all types are Send “by default”.

    To counteract a default impl, one uses a negative impl that explicitly opts out for a given type T and any type that contains T. For example, this RFC proposes that unsafe pointers *T will opt out of Send and Share. This implies that unsafe pointers cannot be sent or shared between threads by default. It also implies that any structs which contain an unsafe pointer cannot be sent. In all examples encountered thus far, the set of negative impls is fixed and can easily be declared along with the trait itself.

    Safe wrappers like Arc, Atomic, or Mutex can opt to implement Send and Share explicitly. This will then make them be considered sendable (or sharable) even though they contain unsafe pointers etc.

Based on these two mechanisms, we can remove the notion of Send and Share as builtin concepts. Instead, these would become unsafe traits with default impls (defined purely in the library). The library would explicitly opt out of Send/Share for certain types, like unsafe pointers (*T) or interior mutability (Unsafe<T>). Any type, therefore, which contains an unsafe pointer would be confined (by default) to a single thread. Safe wrappers around those types, like Arc, Atomic, or Mutex, can then opt back in by explicitly implementing Send (these impls would have to be designed as unsafe).

Motivation

Since proposing opt-in builtin traits, I have become increasingly concerned about the notion of having Send and Share be strictly opt-in. There are two main reasons for my concern:

  1. Rust is very close to being a language where computations can be parallelized by default. Making Send, and especially Share, opt-in makes that harder to achieve.
  2. The model followed by Send/Share cannot easily be extended to other traits in the future nor can it be extended by end-users with their own similar traits. It is worrisome that I have come across several use cases already which might require such extension (described below).

To elaborate on those two points: With respect to parallelization: for the most part, Rust types are threadsafe “by default”. To make something non-threadsafe, you must employ unsynchronized interior mutability (e.g., Cell, RefCell) or unsynchronized shared ownership (Rc). In both cases, there are also synchronized variants available (Mutex, Arc, etc). This implies that we can make APIs to enable intra-task parallelism and they will work ubiquitously, so long as people avoid Cell and Rc when not needed. Explicit opt-in threatens that future, however, because fewer types will implement Share, even if they are in fact threadsafe.

With respect to extensibility, it is particularly worrisome that if a library forgets to implement Send or Share, downstream clients are stuck. They cannot, for example, use a newtype wrapper, because it would be illegal to implement Send on the newtype. This implies that all libraries must be vigilant about implementing Send and Share (even more so than with other pervasive traits like Eq or Ord). The current plan is to address this via lints and perhaps some convenient deriving syntax, which may be adequate for Send and Share. But if we wish to add new “classification” traits in the future, these new traits won’t have been around from the start, and hence won’t be implemented by all existing code.

Another concern of mine is that end users cannot define classification traits of their own. For example, one might like to define a trait for “tainted” data, and then test to ensure that tainted data doesn’t pass through some generic routine. There is no particular way to do this today.

More examples of classification traits that have come up recently in various discussions:

  • Snapshot (nee Freeze), which defines logical immutability rather than physical immutability. Rc<int>, for example, would be considered Snapshot. Snapshot could be useful because Snapshot+Clone indicates a type whose value can be safely “preserved” by cloning it.
  • NoManaged, a type which does not contain managed data. This might be useful for integrating garbage collection with custom allocators which do not wish to serve as potential roots.
  • NoDrop, a type which does not contain an explicit destructor. This can be used to avoid nasty GC quandries.

All three of these (Snapshot, NoManaged, NoDrop) can be easily defined using traits with default impls.

A final, somewhat weaker, motivator is aesthetics. Ownership has allowed us to move threading almost entirely into libraries. The one exception is that the Send and Share types remain built-in. Opt-in traits makes them less built-in, but still requires custom logic in the “impl matching” code as well as special safety checks when Safe or Share are implemented.

After the changes I propose, the only traits which would be specifically understood by the compiler are Copy and Sized. I consider this acceptable, since those two traits are intimately tied to the core Rust type system, unlike Send and Share.

Detailed design

Unsafe traits

Certain traits like Send and Share are critical to memory safety. Nonetheless, it is not feasible to check the thread-safety of all types that implement Send and Share. Therefore, we introduce a notion of an unsafe trait – this is a trait that is unsafe to implement, because implementing it carries semantic guarantees that, if compromised, threaten memory safety in a deep way.

An unsafe trait is declared like so:

unsafe trait Foo { ... }

To implement an unsafe trait, one must mark the impl as unsafe:

unsafe impl Foo for Bar { ... }

Designating an impl as unsafe does not automatically mean that the body of the methods is an unsafe block. Each method in the trait must also be declared as unsafe if it to be considered unsafe.

Unsafe traits are only unsafe to implement. It is always safe to reference an unsafe trait. For example, the following function is safe:

fn foo<T:Send>(x: T) { ... }

It is also safe to opt out of an unsafe trait (as discussed in the next section).

Default and negative impls

We add a notion of a default impl, written:

impl Trait for .. { }

Default impls are subject to various limitations:

  1. The default impl must appear in the same module as Trait (or a submodule).
  2. Trait must not define any methods.

We further add the notion of a negative impl, written:

impl !Trait for Foo { }

Negative impls are only permitted if Trait has a default impl. Negative impls are subject to the usual orphan rules, but they are permitting to be overlapping. This makes sense because negative impls are not providing an implementation and hence we are not forced to select between them. For similar reasons, negative impls never need to be marked unsafe, even if they reference an unsafe trait.

Intuitively, to check whether a trait Foo that contains a default impl is implemented for some type T, we first check for explicit (positive) impls that apply to T. If any are found, then T implements Foo. Otherwise, we check for negative impls. If any are found, then T does not implement Foo. If neither positive nor negative impls were found, we proceed to check the component types of T (i.e., the types of a struct’s fields) to determine whether all of them implement Foo. If so, then Foo is considered implemented by T.

Oe non-obvious part of the procedure is that, as we recursively examine the component types of T, we add to our list of assumptions that T implements Foo. This allows recursive types like

struct List<T> { data: T, next: Option<List<T>> }

to be checked successfully. Otherwise, we would recursive infinitely. (This procedure is directly analogous to what the existing TypeContents code does.)

Note that there exist types that expand to an infinite tree of types. Such types cannot be successfully checked with a recursive impl; they will simply overflow the builtin depth checking. However, such types also break code generation under monomorphization (we cannot create a finite set of LLVM types that correspond to them) and are in general not supported. Here is an example of such a type:

struct Foo<A> {
    data: Option<Foo<Vec<A>>>
}

The difference between Foo and List above is that Foo<A> references Foo<Vec<A>>, which will then in turn reference Foo<Vec<Vec<A>>> and so on.

Modeling Send and Share using default traits

The Send and Share traits will be modeled entirely in the library as follows. First, we declare the two traits as follows:

unsafe trait Send { }
unsafe impl Send for .. { }

unsafe trait Share { }
unsafe impl Share for .. { }

Both traits are declared as unsafe because declaring that a type if Send and Share has ramifications for memory safety (and data-race freedom) that the compiler cannot, itself, check.

Next, we will add opt out impls of Send and Share for the various unsafe types:

impl<T> !Send for *T { }
impl<T> !Share for *T { }

impl<T> !Send for *mut T { }
impl<T> !Share for *mut T { }

impl<T> !Share for Unsafe<T> { }

Note that it is not necessary to write unsafe to opt out of an unsafe trait, as that is the default state.

Finally, we will add opt in impls of Send and Share for the various safe wrapper types as needed. Here I give one example, which is Mutex. Mutex is interesting because it has the property that it converts a type T from being Sendable to something Sharable:

unsafe impl<T:Send> Send for Mutex<T> { }
unsafe impl<T:Send> Share for Mutex<T> { }

The Copy and Sized traits

The final two builtin traits are Copy and Share. This RFC does not propose any changes to those two traits but rather relies on the specification from the original opt-in RFC.

Controlling copy vs move with the Copy trait

The Copy trait is “opt-in” for user-declared structs and enums. A struct or enum type is considered to implement the Copy trait only if it implements the Copy trait. This means that structs and enums would move by default unless their type is explicitly declared to be Copy. So, for example, the following code would be in error:

struct Point { x: int, y: int }
...
let p = Point { x: 1, y: 2 };
let q = p;  // moves p
print(p.x); // ERROR

To allow that example, one would have to impl Copy for Point:

struct Point { x: int, y: int }
impl Copy for Point { }
...
let p = Point { x: 1, y: 2 };
let q = p;  // copies p, because Point is Pod
print(p.x); // OK

Effectively, there is a three step ladder for types:

  1. If you do nothing, your type is linear, meaning that it moves from place to place and can never be copied in any way. (We need a better name for that.)
  2. If you implement Clone, your type is cloneable, meaning that it moves from place to place, but it can be explicitly cloned. This is suitable for cases where copying is expensive.
  3. If you implement Copy, your type is copyable, meaning that it is just copied by default without the need for an explicit clone. This is suitable for small bits of data like ints or points.

What is nice about this change is that when a type is defined, the user makes an explicit choice between these three options.

Determining whether a type is Sized

Per the DST specification, the array types [T] and object types like Trait are unsized, as are any structs that embed one of those types. The Sized trait can never be explicitly implemented and membership in the trait is always automatically determined.

Matching and coherence for the builtin types Copy and Sized

In general, determining whether a type implements a builtin trait can follow the existing trait matching algorithm, but it will have to be somewhat specialized. The problem is that we are somewhat limited in the kinds of impls that we can write, so some of the implementations we would want must be “hard-coded”.

Specifically we are limited around tuples, fixed-length array types, proc types, closure types, and trait types:

  • Fixed-length arrays: A fixed-length array [T, ..n] is Copy if T is Copy. It is always Sized as T is required to be Sized.
  • Tuples: A tuple (T_0, ..., T_n) is Copy/Sized depending if, for all i, T_i is Copy/Sized.
  • Trait objects (including procs and closures): A trait object type Trait:K (assuming DST here ;) is never Copy nor Sized.

We cannot currently express the above conditions using impls. We may at some point in the future grow the ability to express some of them. For now, though, these “impls” will be hardcoded into the algorithm as if they were written in libstd.

Per the usual coherence rules, since we will have the above impls in libstd, and we will have impls for types like tuples and fixed-length arrays baked in, the only impls that end users are permitted to write are impls for struct and enum types that they define themselves. Although this rule is in the general spirit of the coherence checks, it will have to be written specially.

Design discussion

Why unsafe traits

Without unsafe traits, it would be possible to create data races without using the unsafe keyword:

struct MyStruct { foo: Cell<int> }
impl Share for MyStruct { }

Balancing abstraction, safety, and convenience.

In general, the existence of default traits is anti-abstraction, in the sense that it exposes implementation details a library might prefer to hide. Specifically, adding new private fields can cause your types to become non-sendable or non-sharable, which may break downstream clients without your knowing. This is a known challenge with parallelism: knowing whether it is safe to parallelize relies on implementation details we have traditionally tried to keep secret from clients (often it is said that parallelism is “anti-modular” or “anti-compositional” for this reason).

I think this risk must be weighed against the limitations of requiring total opt in. Requiring total opt in not only means that some types will accidentally fail to implement send or share when they could, but it also means that libraries which wish to employ marker traits cannot be composed with other libraries that are not aware of those marker traits. In effect, opt-in is anti-modular in its own way.

To be more specific, imagine that library A wishes to define a Untainted trait, and it specifically opts out of Untainted for some base set of types. It then wishes to have routines that only operate on Untainted data. Now imagine that there is some other library B that defines a nifty replacement for Vector, NiftyVector. Finally, some library C wishes to use a NiftyVector<uint>, which should not be considered tainted, because it doesn’t reference any tainted strings. However, NiftyVector<uint> does not implement Untainted (nor can it, without either library A or library B knowing about one another). Similar problems arise for any trait, of course, due to our coherence rules, but often they can be overcome with new types. Not so with Send and Share.

Other use cases

Part of the design involves making space for other use cases. I’d like to sketch out how some of those use cases can be implemented briefly. This is not included in the Detailed design section of the RFC because these traits generally concern other features and would be added under RFCs of their own.

Isolating snapshot types. It is useful to be able to identify types which, when cloned, result in a logical snapshot. That is, a value which can never be mutated. Note that there may in fact be mutation under the covers, but this mutation is not visible to the user. An example of such a type is Rc<T> – although the ref count on the Rc may change, the user has no direct access and so Rc<T> is still logically snapshotable. However, not all Rc instances are snapshottable – in particular, something like Rc<Cell<int>> is not.

trait Snapshot { }
impl Snapshot for .. { }

// In general, anything that can reach interior mutability is not
// snapshotable.
impl<T> !Snapshot for Unsafe<T> { }

// But it's ok for Rc<T>.
impl<T:Snapshot> Snapshot for Rc<T> { }

Note that these definitions could all occur in a library. That is, the Rc type itself doesn’t need to know about the Snapshot trait.

Preventing access to managed data. As part of the GC design, we expect it will be useful to write specialized allocators or smart pointers that explicitly do not support tracing, so as to avoid any kind of GC overhead. The general idea is that there should be a bound, let’s call it NoManaged, that indicates that a type cannot reach managed data and hence does not need to be part of the GC’s root set. This trait could be implemented as follows:

unsafe trait NoManaged { }
unsafe impl NoManaged for .. { }
impl<T> !NoManaged for Gc<T> { }

Preventing access to destructors. It is generally recognized that allowing destructors to escape into managed data – frequently referred to as finalizers – is a bad idea. Therefore, we would generally like to ensure that anything is placed into a managed box does not implement the drop trait. Instead, we would prefer to regular the use of drop through a guardian-like API, which basically means that destructors are not asynchronously executed by the GC, as they would be in Java, but rather enqueued for the mutator thread to run synchronously at its leisure. In order to handle this, though, we presumably need some sort of guardian wrapper types that can take a value which has a destructor and allow it to be embedded within managed data. We can summarize this in a trait GcSafe as follows:

unsafe trait GcSafe { }
unsafe impl GcSafe for .. { }

// By default, anything which has drop trait is not GcSafe.
impl<T:Drop> !GcSafe for T { }

// But guardians are, even if `T` has drop.
impl<T> GcSafe for Guardian<T> { }

Why are Copy and Sized different?

The Copy and Sized traits remain builtin to the compiler. This makes sense because they are intimately tied to analyses the compiler performs. For example, the running of destructors and tracking of moves requires knowing which types are Copy. Similarly, the allocation of stack frames need to know whether types are fully Sized. In contrast, sendability and sharability has been fully exported to libraries at this point.

In addition, opting in to Copy makes sense for several reasons:

  • Experience has shown that “data-like structs”, for which Copy is most appropriate, are a very small percentage of the total.
  • Changing a public API from being copyable to being only movable has a outsized impact on users of the API. It is common however that as APIs evolve they will come to require owned data (like a Vec), even if they do not initially, and hence will change from being copyable to only movable. Opting in to Copy is a way of saying that you never foresee this coming to pass.
  • Often it is useful to create linear “tokens” that do not themselves have data but represent permissions. This can be done today using markers but it is awkward. It becomes much more natural under this proposal.

Drawbacks

API stability. The main drawback of this approach over the existing opt-in approach seems to be that a type may be “accidentally” sendable or sharable. I discuss this above under the heading of “balancing abstraction, safety, and convenience”. One point I would like to add here, as it specifically pertains to API stability, is that a library may, if they choose, opt out of Send and Share pre-emptively, in order to “reserve the right” to add non-sendable things in the future.

Alternatives

  • The existing opt-in design is of course an alternative.

  • We could also simply add the notion of unsafe traits and not default impls and then allow types to unsafely implement Send or Share, bypassing the normal safety guidelines. This gives an escape valve for a downstream client to assert that something is sendable which was not declared as sendable. However, such a solution is deeply unsatisfactory, because it rests on the downstream client making an assertion about the implementation of the library it uses. If that library should be updated, the client’s assumptions could be invalidated, but no compilation errors will result (the impl was already declared as unsafe, after all).

Phasing

Many of the mechanisms described in this RFC are not needed immediately. Therefore, we would like to implement a minimal “forwards compatible” set of changes now and then leave the remaining work for after the 1.0 release. The builtin rules that the compiler currently implements for send and share are quite close to what is proposed in this RFC. The major change is that unsafe pointers and the UnsafeCell type are currently considered sendable.

Therefore, to be forwards compatible in the short term, we can use the same hybrid of builtin and explicit impls for Send and Share that we use for Copy, with the rule that unsafe pointers and UnsafeCell are not considered sendable. We must also implement the unsafe trait and unsafe impl concept.

What this means in practice is that using *const T, *mut T, and UnsafeCell will make a type T non-sendable and non-sharable, and T must then explicitly implement Send or Share.

Unresolved questions

  • The terminology of “unsafe trait” seems somewhat misleading, since it seems to suggest that “using” the trait is unsafe, rather than implementing it. One suggestion for an alternate keyword was trusted trait, which might dovetail with the use of trusted to specify a trusted block of code. If we did use trusted trait, it seems that all impls would also have to be trusted impl.
  • Perhaps we should declare a trait as a “default trait” directly, rather than using the impl Drop for .. syntax. I don’t know precisely what syntax to use, though.
  • Currently, there are special rules relating to object types and the builtin traits. If the “builtin” traits are no longer builtin, we will have to generalize object types to be simply a set of trait references. This is already planned but merits a second RFC. Note that no changes here are required for the 1.0, since the phasing plan dictates that builtin traits remain special until after 1.0.

Summary

This RFC is a proposal to remove the usage of the keyword priv from the Rust language.

Motivation

By removing priv entirely from the language, it significantly simplifies the privacy semantics as well as the ability to explain it to newcomers. The one remaining case, private enum variants, can be rewritten as such:

// pub enum Foo {
//     Bar,
//     priv Baz,
// }

pub enum Foo {
    Bar,
    Baz(BazInner)
}

pub struct BazInner(());

// pub enum Foo2 {
//     priv Bar2,
//     priv Baz2,
// }

pub struct Foo2 {
    variant: FooVariant
}

enum FooVariant {
    Bar2,
    Baz2,
}

Private enum variants are a rarely used feature of the language, and are generally not regarded as a strong enough feature to justify the priv keyword entirely.

Detailed design

There remains only one use case of the priv visibility qualifier in the Rust language, which is to make enum variants private. For example, it is possible today to write a type such as:

pub enum Foo {
    Bar,
    priv Baz
}

In this example, the variant Bar is public, while the variant Baz is private. This RFC would remove this ability to have private enum variants.

In addition to disallowing the priv keyword on enum variants, this RFC would also forbid visibility qualifiers in front of enum variants entirely, as they no longer serve any purpose.

Status of the identifier priv

This RFC would demote the identifier priv from being a keyword to being a reserved keyword (in case we find a use for it in the future).

Alternatives

  • Allow private enum variants, as-is today.
  • Add a new keyword for enum which means “my variants are all private” with controls to make variants public.

Unresolved questions

  • Is the assertion that private enum variants are rarely used true? Are there legitimate use cases for keeping the priv keyword?

Summary

Check all types for well-formedness with respect to the bounds of type variables.

Allow bounds on formal type variable in structs and enums. Check these bounds are satisfied wherever the struct or enum is used with actual type parameters.

Motivation

Makes type checking saner. Catches errors earlier in the development process. Matches behaviour with built-in bounds (I think).

Currently formal type variables in traits and functions may have bounds and these bounds are checked whenever the item is used against the actual type variables. Where these type variables are used in types, these types should be checked for well-formedness with respect to the type definitions. E.g.,

trait U {}
trait T<X: U> {}
trait S<Y> {
    fn m(x: ~T<Y>) {}  // Should be flagged as an error
}

Formal type variables in structs and enums may not have bounds. It is possible to use these type variables in the types of fields, and these types cannot be checked for well-formedness until the struct is instantiated, where each field must be checked.

struct St<X> {
    f: ~T<X>, // Cannot be checked
}

Likewise, impls of structs are not checked. E.g.,

impl<X> St<X> {  // Cannot be checked
    ...
}

Here, no struct can exist where X is replaced by something implementing U, so in the impl, X can be assumed to have the bound U. But the impl does not indicate this. Note, this is sound, but does not indicate programmer intent very well.

Detailed design

Whenever a type is used it must be checked for well-formedness. For polymorphic types we currently check only that the type exists. I would like to also check that any actual type parameters are valid. That is, given a type T<U> where T is declared as T<X: B>, we currently only check that T does in fact exist somewhere (I think we also check that the correct number of type parameters are supplied, in this case one). I would also like to check that U satisfies the bound B.

Work on built-in bounds is (I think) in the process of adding this behaviour for built-in bounds. I would like to apply this to user-specified bounds too.

I think no fewer programs can be expressed. That is, any errors we catch with this new check would have been caught later in the existing scheme, where exactly would depend on where the type was used. The only exception would be if the formal type variable was not used.

We would allow bounds on type variable in structs and enums. Wherever a concrete struct or enum type appears, check the actual type variables against the bounds on the formals (the type well-formedness check).

From the above examples:

trait U {}
trait T<X: U> {}
trait S1<Y> {
    fn m(x: ~T<Y>) {}  //~ ERROR
}
trait S2<Y: U> {
    fn m(x: ~T<Y>) {}
}

struct St<X: U> {
    f: ~T<X>,
}

impl<X: U> St<X> {
    ...
}

Alternatives

Keep the status quo.

We could add bounds on structs, etc. But not check them in impls. This is safe since the implementation is more general than the struct. It would mean we allow impls to be un-necessarily general.

Unresolved questions

Do we allow and check bounds in type aliases? We currently do not. We should probably continue not to since these type variables (and indeed the type aliases) are substituted away early in the type checking process. So if we think of type aliases as almost macro-like, then not checking makes sense. OTOH, it is still a little bit inconsistent.

Summary

Split the current libstd into component libraries, rebuild libstd as a facade in front of these component libraries.

Motivation

Rust as a language is ideal for usage in constrained contexts such as embedding in applications, running on bare metal hardware, and building kernels. The standard library, however, is not quite as portable as the language itself yet. The standard library should be as usable as it can be in as many contexts as possible, without compromising its usability in any context.

This RFC is meant to expand the usability of the standard library into these domains where it does not currently operate easily

Detailed design

In summary, the following libraries would make up part of the standard distribution. Each library listed after the colon are the dependent libraries.

  • libmini
  • liblibc
  • liballoc: libmini liblibc
  • libcollections: libmini liballoc
  • libtext: libmini liballoc libcollections
  • librustrt: libmini liballoc liblibc
  • libsync: libmini liballoc liblibc librustrt
  • libstd: everything above

libmini

Note: The name libmini warrants bikeshedding. Please consider it a placeholder for the name of this library.

This library is meant to be the core component of all rust programs in existence. This library has very few external dependencies, and is entirely self contained.

Current modules in std which would make up libmini would include the list below. This list was put together by actually stripping down libstd to these modules, so it is known that it is possible for libmini to compile with these modules.

  • atomics
  • bool
  • cast
  • char
  • clone
  • cmp
  • container
  • default
  • finally
  • fmt
  • intrinsics
  • io, stripped down to its core
  • iter
  • kinds
  • mem
  • num (and related modules), no float support
  • ops
  • option
  • ptr
  • raw
  • result
  • slice, but without any ~[T] methods
  • tuple
  • ty
  • unit

This list may be a bit surprising, and it’s makeup is discussed below. Note that this makeup is selected specifically to eliminate the need for the dreaded “one off extension trait”. This pattern, while possible, is currently viewed as subpar due to reduced documentation benefit and sharding implementation across many locations.

Strings

In a post-DST world, the string type will actually be a library-defined type, Str (or similarly named). Strings will no longer be a language feature or a language-defined type. This implies that any methods on strings must be in the same crate that defined the Str type, or done through extension traits.

In the spirit of reducing extension traits, the Str type and module were left out of libmini. It’s impossible for libmini to support all methods of Str, so it was entirely removed.

This decision does have ramifications on the implementation of libmini.

  • String literals are an open question. In theory, making a string literal would require the Str lang item to be present, but is not present in libmini. That being said, libmini would certainly create many literal strings (for error messages and such). This may be adequately circumvented by having literal strings create a value of type &'static [u8] if the string lang item is not present. While difficult to work with, this may get us 90% of the way there.

  • The fmt module must be tweaked for the removal of strings. The only major user-facing detail is that the pad function on Formatter would take a byte-slice and a character length, and then not handle the precision (which truncates the byte slice with a number of characters). This may be overcome by possibly having an extension trait could be added for a Formatter adding a real pad function that takes strings, or just removing the function altogether in favor of str.fmt(formatter).

  • The IoError type suffers from the removal of strings. Currently, this type is inhabited with three fields, an enum, a static description string, and an optionally allocated detail string. Removal of strings would imply the IoError type would be just the enum itself. This may be an acceptable compromise to make, defining the IoError type upstream and providing easy constructors from the enum to the struct. Additionally, the OtherIoError enum variant would be extended with an i32 payload representing the error code (if it came from the OS).

  • The ascii module is omitted, but it would likely be defined in the crate that defines Str.

Formatting

While not often thought of as “ultra-core” functionality, this module may be necessary because printing information about types is a fundamental problem that normally requires no dependencies.

Inclusion of this module is the reason why I/O is included in the module as well (or at least a few traits), but the module can otherwise be included with little to no overhead required in terms of dependencies.

Neither print! nor format! macros to be a part of this library, but the write! macro would be present.

I/O

The primary reason for defining the io module in the libmini crate would be to implement the fmt module. The ramification of removing strings was previously discussed for IoError, but there are further modifications that would be required for the io module to exist in libmini:

  • The Buffer, Listener, Seek, and Acceptor traits would all be defined upstream instead of in libmini. Very little in libstd uses these traits, and nothing in libmini requires them. They are of questionable utility when considering their applicability to all rust code in existence.

  • Some extension methods on the Reader and Writer traits would need to be removed. Methods such as push_exact, read_exact, read_to_end, write_line, etc., all require owned vectors or similar unimplemented runtime requirements. These can likely be moved to extension traits upstream defined for all readers and writers. Note that this does not apply to the integral reading and writing methods. These are occasionally overwritten for performance, but removal of some extension methods would strongly suggest to me that these methods should be removed. Regardless, the remaining methods could live in essentially any location.

Slices

The only method lost on mutable slices would currently be the sorting method. This can be circumvented by implementing a sorting algorithm that doesn’t require allocating a temporary buffer. If intensive use of a sorting algorithm is required, Rust can provide a libsort crate with a variety of sorting algorithms apart from the default sorting algorithm.

FromStr

This trait and module are left out because strings are left out. All types in libmini can have their implementation of FromStr in the crate which implements strings

Floats

This current design excludes floats entirely from libmini (implementations of traits and such). This is another questionable decision, but the current implementation of floats heavily leans on functions defined in libm, so it is unacceptable for these functions to exist in libmini.

Either libstd or a libfloat crate will define floating point traits and such.

Failure

It is unacceptable for Option to reside outside of libmini, but it is also also unacceptable for unwrap to live outside of the Option type. Consequently, this means that it must be possible for libmini to fail.

While impossible for libmini to define failure, it should simply be able to declare failure. While currently not possible today, this extension to the language is possible through “weak lang items”.

Implementation-wise, the failure lang item would have a predefined symbol at which it is defined, and libraries which declare but to not define failure are required to only exist in the rlib format. This implies that libmini can only be built as an rlib. Note that today’s linkage rules do not allow for this (because building a dylib with rlib dependencies is not possible), but the rules could be tweaked to allow for this use case.

tl;dr; The implementation of libmini can use failure, but it does not define failure. All usage of libmini would require an implementation of failure somewhere.

liblibc

This library will exist to provide bindings to libc. This will be a highly platform-specific library, containing an entirely separate api depending on which platform it’s being built for.

This crate will be used to provide bindings to the C language in all forms, and would itself essentially be a giant metadata blob. It conceptually represents the inclusion of all C header files.

Note that the funny name of the library is to allow extern crate libc; to be the form of declaration rather than extern crate c; which is consider to be too short for its own good.

Note that this crate can only exist in rlib or dylib form.

liballoc

Note: This name liballoc is questionable, please consider it a placeholder.

This library would define the allocator traits as well as bind to libc malloc/free (or jemalloc if we decide to include it again). This crate would depend on liblibc and libmini.

Pointers such as ~ and Rc would move into this crate using the default allocator. The current Gc pointers would move to libgc if possible, or otherwise librustrt for now (they’re feature gated currently, not super pressing).

Primarily, this library assumes that an allocation failure should trigger a failure. This makes the library not suitable for use in a kernel, but it is suitable essentially everywhere else.

With today’s libstd, this crate would likely mostly be made up by the global_heap module. Its purpose is to define the allocation lang items required by the compiler.

Note that this crate can only exist in rlib form.

libcollections

This crate would not depend on libstd, it would only depend on liballoc and libmini. These two foundational crates should provide all that is necessary to provide a robust set of containers (what you would expect today). Each container would likely have an allocator parameter, and the default would be the default allocator provided by liballoc.

When using the containers from libcollections, it is implicitly assumed that all allocation succeeds, and this will be reflected in the api of each collection.

The contents of this crate would be the entirety of libcollections as it is today, as well as the vec module from the standard library. This would also implement any relevant traits necessary for ~[T].

Note that this crate can only exist in rlib form.

libtext

This crate would define all functionality in rust related to strings. This would contain the definition of the Str type, as well as implementations of the relevant traits from libmini for the string type.

The crucial assumption of this crate is that allocation does not fail, and the rest of the string functionality could be built on top of this. Note that this crate will depend on libcollections for the Vec type as the underlying building block for string buffers and the string type.

This crate would be composed of the str, ascii, and unicode modules which live in libstd today, but would allow for the extension of other text-related functionality.

librustrt

This library would be the crate where the rt module is almost entirely implemented. It will assume that allocation succeeds, and it will assume a libc implementation to run on.

The current libstd modules which would be implemented as part of this crate would be:

  • rt
  • task
  • local_data

Note that comm is not on this list. This crate will additionally define failure (as unwinding for each task). This crate can exist in both rlib and dylib form.

libsync

This library will largely remain what it is today, with the exception that the comm implementation would move into this crate. The purpose of doing so would be to consolidate all concurrency-related primitives in this crate, leaving none out.

This crate would depend on the runtime for task management (scheduling and descheduling).

The libstd facade

A new standard library would be created that would primarily be a facade which would expose the underlying crates as a stable API. This library would depend on all of the above libraries, and would predominately be a grouping of pub use statements.

This library would also be the library to contain the prelude which would include types from the previous crates. All remaining functionality of the standard library would be filled in as part of this crate.

Note that all rust programs will by default link to libstd, and hence will transitively link to all of the upstream crates mentioned above. Many more apis will be exposed through libstd directly, however, such as HashMap, Arc, etc.

The exact details of the makeup of this crate will change over time, but it can be considered as “the current libstd plus more”, and this crate will be the source of the “batteries included” aspect of the rust standard library. The API (reexported paths) of the standard library would not change over time. Once a path is reexported and a release is made, all the path will be forced to remain constant over time.

One of the primary reasons for this facade is to provide freedom to restructure the underlying crates. Once a facade is established, it is the only stable API. The actual structure and makeup of all the above crates will be fluid until an acceptable design is settled on. Note that this fluidity does not apply to libstd, only to the structure of the underlying crates.

Updates to rustdoc

With today’s incarnation of rustdoc, the documentation for this libstd facade would not be as high quality as it is today. The facade would just provide hyperlinks back to the original crates, which would have reduced quantities of documentation in terms of navigation, implemented traits, etc. Additionally, these reexports are meant to be implementation details, not facets of the api. For this reason, rustdoc would have to change in how it renders documentation for libstd.

First, rustdoc would consider a cross-crate reexport as inlining of the documentation (similar to how it inlines reexports of private types). This would allow all documentation in libstd to remain in the same location (even the same urls!). This would likely require extensive changes to rustdoc for when entire module trees are reexported.

Secondly, rustdoc will have to be modified to collect implementors of reexported traits all in one location. When libstd reexports trait X, rustdoc will have to search libstd and all its dependencies for implementors of X, listing them out explicitly.

These changes to rustdoc should place it in a much more presentable space, but it is an open question to what degree these modifications will suffice and how much further rustdoc will have to change.

Remaining crates

There are many more crates in the standard distribution of rust, all of which currently depend on libstd. These crates would continue to depend on libstd as most rust libraries would.

A new effort would likely arise to reduce dependence on the standard library by cutting down to the core dependencies (if necessary). For example, the libnative crate currently depend on libstd, but it in theory doesn’t need to depend on much other than librustrt and liblibc. By cutting out dependencies, new use cases will likely arise for these crates.

Crates outside of the standard distribution of rust will like to link to the above crates as well (and specifically not libstd). For example, crates which only depend on libmini are likely candidates for being used in kernels, whereas crates only depending on liballoc are good candidates for being embedded into other languages. Having a clear delineation for the usability of a crate in various environments seems beneficial.

Alternatives

  • There are many alternatives to the above sharding of libstd and its dependent crates. The one that is most rigid is likely libmini, but the contents of all other crates are fairly fluid and able to shift around. To this degree, there are quite a few alternatives in how the remaining crates are organized. The ordering proposed is simply one of many.

  • Compilation profiles. Instead of using crate dependencies to encode where a crate can be used, crates could instead be composed of cfg(foo) attributes. In theory, there would be one libstd crate (in terms of source code), and this crate could be compiled with flags such as --cfg libc, --cfg malloc, etc. This route has may have the problem of “multiple standard libraries” in that code compatible with the “libc libstd” is not necessarily compatible with the “no libc libstd”. Asserting that a crate is compatible with multiple profiles would involve requiring multiple compilations.

  • Removing libstd entirely. If the standard library is simply a facade, the compiler could theoretically only inject a select number of crates into the prelude, or possibly even omit the prelude altogether. This works towards elimination the question of “does this belong in libstd”, but it would possibly be difficult to juggle the large number of crates to choose from where one could otherwise just look at libstd.

Unresolved questions

  • Compile times. It’s possible that having so many upstream crates for each rust crate will increase compile times through reading metadata and invoking the system linker. Would sharding crates still be worth it? Could possible problems that arise be overcome? Would extra monomorphization in all these crates end up causing more binary bloat?

  • Binary bloat. Another possible side effect of having many upstream crates would be increasing binary bloat of each rust program. Our current linkage model means that if you use anything from a crate that you get everything in that crate (in terms of object code). It is unknown to what degree this will become a concern, and to what degree it can be overcome.

  • Should floats be left out of libmini? This is largely a question of how much runtime support is required for floating point operations. Ideally functionality such as formatting a float would live in libmini, whereas trigonometric functions would live in an external crate with a dependence on libm.

  • Is it acceptable for strings to be left out of libmini? Many common operations on strings don’t require allocation. This is currently done out of necessity of having to define the Str type elsewhere, but this may be seen as too limiting for the scope of libmini.

  • Does liblibc belong so low in the dependency tree? In the proposed design, only the libmini crate doesn’t depend on liblibc. Crates such as libtext and libcollections, however, arguably have no dependence on libc itself, they simply require some form of allocator. Answering this question would be figuring how how to break liballoc’s dependency on liblibc, but it’s an open question as to whether this is worth it or not.

  • Reexporting macros. Currently the standard library defines a number of useful macros which are used throughout the implementation of libstd. There is no way to reexport a macro, so multiple implementations of the same macro would be required for the core libraries to all use the same macro. Is there a better solution to this situation? How much of an impact does this have?

Summary

Add a regexp crate to the Rust distribution in addition to a small regexp_macros crate that provides a syntax extension for compiling regular expressions during the compilation of a Rust program.

The implementation that supports this RFC is ready to receive feedback: https://github.com/BurntSushi/regexp

Documentation for the crate can be seen here: http://burntsushi.net/rustdoc/regexp/index.html

regex-dna benchmark (vs. Go, Python): https://github.com/BurntSushi/regexp/tree/master/benchmark/regex-dna

Other benchmarks (vs. Go): https://github.com/BurntSushi/regexp/tree/master/benchmark

(Perhaps the links should be removed if the RFC is accepted, since I can’t guarantee they will always exist.)

Motivation

Regular expressions provide a succinct method of matching patterns against search text and are frequently used. For example, many programming languages include some kind of support for regular expressions in its standard library.

The outcome of this RFC is to include a regular expression library in the Rust distribution and resolve issue #3591.

Detailed design

(Note: This is describing an existing design that has been implemented. I have no idea how much of this is appropriate for an RFC.)

The first choice that most regular expression libraries make is whether or not to include backreferences in the supported syntax, as this heavily influences the implementation and the performance characteristics of matching text.

In this RFC, I am proposing a library that closely models Russ Cox’s RE2 (either its C++ or Go variants). This means that features like backreferences or generalized zero-width assertions are not supported. In return, we get O(mn) worst case performance (with m being the size of the search text and n being the number of instructions in the compiled expression).

My implementation currently simulates an NFA using something resembling the Pike VM. Future work could possibly include adding a DFA. (N.B. RE2/C++ includes both an NFA and a DFA, but RE2/Go only implements an NFA.)

The primary reason why I chose RE2 was that it seemed to be a popular choice in issue #3591, and its worst case performance characteristics seemed appealing. I was also drawn to the limited set of syntax supported by RE2 in comparison to other regexp flavors.

With that out of the way, there are other things that inform the design of a regexp library.

Unicode

Given the already existing support for Unicode in Rust, this is a no-brainer. Unicode literals should be allowed in expressions and Unicode character classes should be included (e.g., general categories and scripts).

Case folding is also important for case insensitive matching. Currently, this is implemented by converting characters to their uppercase forms and then comparing them. Future work includes applying at least a simple fold, since folding one Unicode character can produce multiple characters.

Normalization is another thing to consider, but like most other regexp libraries, the one I’m proposing here does not do any normalization. (It seems the recommended practice is to do normalization before matching if it’s needed.)

A nice implementation strategy to support Unicode is to implement a VM that matches characters instead of bytes. Indeed, my implementation does this. However, the public API of a regular expression library should expose byte indices corresponding to match locations (which ought to be guaranteed to be UTF8 codepoint boundaries by construction of the VM). My reason for this is that byte indices result in a lower cost abstraction. If character indices are desired, then a mapping can be maintained by the client at their discretion.

Additionally, this makes it consistent with the std::str API, which also exposes byte indices.

Word boundaries, word characters and Unicode

At least Python and D define word characters, word boundaries and space characters with Unicode character classes. My implementation does the same by augmenting the standard Perl character classes \d, \s and \w with corresponding Unicode categories.

Leftmost-first

As of now, my implementation finds the leftmost-first match. This is consistent with PCRE style regular expressions.

I’ve pretty much ignored POSIX, but I think it’s very possible to add leftmost-longest semantics to the existing VM. (RE2 supports this as a parameter, but I believe still does not fully comply with POSIX with respect to picking the correct submatches.)

Public API

There are three main questions that can be asked when searching text:

  1. Does the string match this expression?
  2. If so, where?
  3. Where are its submatches?

In principle, an API could provide a function to only answer (3). The answers to (1) and (2) would immediately follow. However, keeping track of submatches is expensive, so it is useful to implement an optimization that doesn’t keep track of them if it doesn’t have to. For example, submatches do not need to be tracked to answer questions (1) and (2).

The rabbit hole continues: answering (1) can be more efficient than answering (2) because you don’t have to keep track of any capture groups ((2) requires tracking the position of the full match). More importantly, (1) enables early exit from the VM. As soon as a match is found, the VM can quit instead of continuing to search for greedy expressions.

Therefore, it’s worth it to segregate these operations. The performance difference can get even bigger if a DFA were implemented (which can answer (1) and (2) quickly and even help with (3)). Moreover, most other regular expression libraries provide separate facilities for answering these questions separately.

Some libraries (like Python’s re and RE2/C++) distinguish between matching an expression against an entire string and matching an expression against part of the string. My implementation favors simplicity: matching the entirety of a string requires using the ^ and/or $ anchors. In all cases, an implicit .*? is added the beginning and end of each expression evaluated. (Which is optimized out in the presence of anchors.)

Finally, most regexp libraries provide facilities for splitting and replacing text, usually making capture group names available with some sort of $var syntax. My implementation provides this too. (These are a perfect fit for Rust’s iterators.)

This basically makes up the entirety of the public API, in addition to perhaps a quote function that escapes a string so that it may be used as a literal in an expression.

The regexp! macro

With syntax extensions, it’s possible to write an regexp! macro that compiles an expression when a Rust program is compiled. This includes translating the matching algorithm to Rust code specific to the expression given. This “ahead of time” compiling results in a performance increase. Namely, it elides all heap allocation.

I’ve called these “native” regexps, whereas expressions compiled at runtime are “dynamic” regexps. The public API need not impose this distinction on users, other than requiring the use of a syntax extension to construct a native regexp. For example:

let re = regexp!("a*");

After construction, re is indistinguishable from an expression created dynamically:

let re = Regexp::new("a*").unwrap();

In particular, both have the same type. This is accomplished with a representation resembling:

enum MaybeNative {
    Dynamic(~[Inst]),
    Native(fn(MatchKind, &str, uint, uint) -> ~[Option<uint>]),
}

This syntax extension requires a second crate, regexp_macros, where the regexp! macro is defined. Technically, this could be provided in the regexp crate, but this would introduce a runtime dependency on libsyntax for any use of the regexp crate.

@alexcrichton remarks that this state of affairs is a wart that will be corrected in the future.

Untrusted input

Given worst case O(mn) time complexity, I don’t think it’s worth worrying about unsafe search text.

Untrusted regular expressions are another matter. For example, it’s very easy to exhaust a system’s resources with nested counted repetitions. For example, ((a{100}){100}){100} tries to create 100^3 instructions. My current implementation does nothing to mitigate against this, but I think a simple hard limit on the number of instructions allowed would work fine. (Should it be configurable?)

Name

The name of the crate being proposed is regexp and the type describing a compiled regular expression is Regexp. I think an equally good name would be regex (and Regex). Either name seems to be frequently used, e.g., “regexes” or “regexps” in colloquial use. I chose regexp over regex because it matches the name used for the corresponding package in Go’s standard library.

Other possible names are regexpr (and Regexpr) or something with underscores: reg_exp (and RegExp). However, I perceive these to be more ugly and less commonly used than either regexp or regex.

Finally, we could use re (like Python), but I think the name could be ambiguous since it’s so short. regexp (or regex) unequivocally identifies the crate as providing regular expressions.

For consistency’s sake, I propose that the syntax extension provided be named the same as the crate. So in this case, regexp!.

Summary

My implementation is pretty much a port of most of RE2. The syntax should be identical or almost identical. I think matching an existing (and popular) library has benefits, since it will make it easier for people to pick it up and start using it. There will also be (hopefully) fewer surprises. There is also plenty of room for performance improvement by implementing a DFA.

Alternatives

I think the single biggest alternative is to provide a backtracking implementation that supports backreferences and generalized zero-width assertions. I don’t think my implementation precludes this possibility. For example, a backtracking approach could be implemented and used only when features like backreferences are invoked in the expression. However, this gives up the blanket guarantee of worst case O(mn) time. I don’t think I have the wisdom required to voice a strong opinion on whether this is a worthwhile endeavor.

Another alternative is using a binding to an existing regexp library. I think this was discussed in issue #3591 and it seems like people favor a native Rust implementation if it’s to be included in the Rust distribution. (Does the regexp! macro require it? If so, that’s a huge advantage.) Also, a native implementation makes it maximally portable.

Finally, it is always possible to persist without a regexp library.

Unresolved questions

The public API design is fairly simple and straight-forward with no surprises. I think most of the unresolved stuff is how the backend is implemented, which should be changeable without changing the public API (sans adding features to the syntax).

I can’t remember where I read it, but someone had mentioned defining a trait that declared the API of a regexp engine. That way, anyone could write their own backend and use the regexp interface. My initial thoughts are YAGNI—since requiring different backends seems like a super specialized case—but I’m just hazarding a guess here. (If we go this route, then we might want to expose the regexp parser and AST and possibly the compiler and instruction set to make writing your own backend easier. That sounds restrictive with respect to making performance improvements in the future.)

I personally think there’s great value in keeping the standard regexp implementation small, simple and fast. People who have more specialized needs can always pick one of the existing C or C++ libraries.

For now, we could mark the API as #[unstable] or #[experimental].

Future work

I think most of the future work for this crate is to increase the performance, either by implementing different matching algorithms (e.g., a DFA) or by improving the code generator that produces native regexps with regexp!.

If and when a DFA is implemented, care must be taken when creating a code generator, as the size of the code required can grow rapidly.

Other future work (that is probably more important) includes more Unicode support, specifically for simple case folding.

Summary

Cleanup the trait, method, and operator semantics so that they are well-defined and cover more use cases. A high-level summary of the changes is as follows:

  1. Generalize explicit self types beyond &self and &mut self etc, so that self-type declarations like self: Rc<Self> become possible.
  2. Expand coherence rules to operate recursively and distinguish orphans more carefully.
  3. Revise vtable resolution algorithm to be gradual.
  4. Revise method resolution algorithm in terms of vtable resolution.

This RFC excludes discussion of associated types and multidimensional type classes, which will be the subject of a follow-up RFC.

Motivation

The current trait system is ill-specified and inadequate. Its implementation dates from a rather different language. It should be put onto a surer footing.

Use cases

Poor interaction with overloadable deref and index

Addressed by: New method resolution algorithm.

The deref operator * is a flexible one. Imagine a pointer p of type ~T. This same * operator can be used for three distinct purposes, depending on context.

  1. Create an immutable referent to the referent: &*p.
  2. Create a mutable reference to the referent: &mut *p.
  3. Copy/move the contents of the referent: consume(*p).

Not all of these operations are supported by all types. In fact, because most smart pointers represent aliasable data, they will only support the creation of immutable references (e.g., Rc, Gc). Other smart pointers (e.g., the RefMut type returned by RefCell) support mutable or immutable references, but not moves. Finally, a type that owns its data (like, indeed, ~T) might support #3.

To reflect this, we use distinct traits for the various operators. (In fact, we don’t currently have a trait for copying/moving the contents, this could be a distinct RFC (ed., I’m still thinking this over myself, there are non-trivial interactions)).

Unfortunately, the method call algorithm can’t really reliably choose mutable vs immutable deref. The challenge is that the proper choice will sometimes not be apparent until quite late in the process. For example, imagine the expression p.foo(): if foo() is defined with &self, we want an immutable deref, otherwise we want a mutable deref.

Note that in this RFC I do not completely address this issue. In particular, in an expression like (*p).foo(), where the dereference is explicit and not automatically inserted, the sense of the dereference is not inferred. For the time being, the sense can be manually specified by making the receiver type fully explicit: (&mut *p).foo() vs (&*p).foo(). I expect in a follow-up RFC to possibly address this problem, as well as the question of how to handle copies and moves of the referent (use #3 in my list above).

Lack of backtracking

Addressed by: New method resolution algorithm.

Issue #XYZ. When multiple traits define methods with the same name, it is ambiguous which trait is being used:

trait Foo { fn method(&self); }
trait Bar { fn method(&self); }

In general, so long as a given type only implements Foo or Bar, these ambiguities don’t present a problem (and ultimately Universal Function Call Syntax or UFCS will present an explicit resolution). However, this is not guaranteed. Sometimes we see “blanket” impls like the following:

impl<A:Base> Foo for A { }

This impl basically says “any type T that implements Base automatically implements Foo”. Now, we expect an ambiguity error if we have a type T that implements both Base and Bar. But in fact, we’ll get an ambiguity error even if a type only implements Bar. The reason for this is that the current method resolution doesn’t “recurse” and check additional dependencies when deciding if an impl is applicable. So it will decide, in this case, that the type T could implement Foo and then record for later that T must implement Base. This will lead to weird errors.

Overly conservative coherence

Addressed by: Expanded coherence rules.

The job of coherence is to ensure that, for any given set of type parameters, a given trait is implemented at most once (it may of course not be implemented at all). Currently, however, coherence is more conservative that it needs to be. This is partly because it doesn’t take into account the very property that it itself is enforcing.

The problems arise due to the “blanket impls” I discussed in the previous section. Consider the following two traits and a blanket impl:

trait Base { }
trait Derived { }
impl<A:Base> Derived for A { }

Here we have two traits Base and Derived, and a blanket impl which implements the Derived trait for any type A that also implements Base.

This implies that if you implement Base for a type S, then S automatically implements Derived:

struct S;
impl Base for S { } // Implement Base => Implements Derived

On a related note, it’d be an error to implement both Base and Derived for the same type T:

// Illegal
struct T;
impl Base for T { }
impl Derived for T { }

This is illegal because now there are two implements of Derived for T. There is the direct one, but also an indirect one. We do not assign either higher precedence, we just report it as an error.

So far, all is in agreement with the current rules. However, problems arise if we imagine a type U that only implements Derived:

struct U;
impl Derived for U { } // Should be OK, currently not.

In this scenario, there is only one implementation of Derived. But the current coherence rules still report it as an error.

Here is a concrete example where a rule like this would be useful. We currently have the Copy trait (aka Pod), which states that a type can be memcopied. We also have the Clone trait, which is a more heavyweight version for types where copying requires allocation. It’d be nice if all types that could be copied could also be cloned – it’d also be nice if we knew for sure that copying a value had the same semantics as cloning it, in that case. We can guarantee both using a blanket impl like the following:

impl<T:Copy> Clone for T {
    fn clone(&self) -> T {
        *self
    }
}

Unfortunately, writing such an impl today would imply that no other types could implement Clone. Obviously a non-starter.

There is one not especially interesting ramification of this. Permitting this rule means that adding impls to a type could cause coherence errors. For example, if I had a type which implements Copy, and I add an explicit implementation of Clone, I’d get an error due to the blanket impl. This could be seen as undesirable (perhaps we’d like to preserve that property that one can always add impls without causing errors).

But of course we already don’t have the property that one can always add impls, since method calls could become ambiguous. And if we were to add “negative bounds”, which might be nice, we’d lose that property. And the popularity and usefulness of blanket impls cannot be denied. Therefore, I think this property (“always being able to add impls”) is not especially useful or important.

Hokey implementation

Addressed by: Gradual vtable resolution algorithm

In an effort to improve inference, the current implementation has a rather ad-hoc two-pass scheme. When performing a method call, it will immediately attempt “early” trait resolution and – if that fails – defer checking until later. This helps with some particular scenarios, such as a trait like:

trait Map<E> {
    fn map(&self, op: |&E| -> E) -> Self;
}

Given some higher-order function like:

fn some_mapping<E,V:Map<E>>(v: &V, op: |&E| -> E) { ... }

If we were then to see a call like:

some_mapping(vec, |elem| ...)

the early resolution would be helpful in connecting the type of elem with the type of vec. The reason to use two phases is that often we don’t need to resolve each trait bound to a specific impl, and if we wait till the end then we will have more type information available.

In my proposed solution, we eliminate the phase distinction. Instead, we simply track pending constraints. We are free to attempt to resolve pending constraints whenever desired. In particular, whenever we find we need more type information to proceed with some type-overloaded operation, rather than reporting an error we can try and resolve pending constraints. If that helps give more information, we can carry on. Once we reach the end of the function, we must then resolve all pending constraints that have not yet been resolved for some other reason.

Note that there is some interaction with the distinction between input and output type parameters discussed in the previous example. Specifically, we must never infer the value of the Self type parameter based on the impls in scope. This is because it would cause crate concatenation to potentially lead to compilation errors in the form of inference failure.

Properties

There are important properties I would like to guarantee:

  • Coherence or No Overlapping Instances: Given a trait and values for all of its type parameters, there should always be at most one applicable impl. This should remain true even when unknown, additional crates are loaded.
  • Crate concatenation: It should always be possible to take two creates and combine them without causing compilation errors. This property

Here are some properties I do not intend to guarantee:

  • Crate divisibility: It is not always possible to divide a crate into two crates. Specifically, this may incur coherence violations due to the orphan rules.
  • Decidability: Haskell has various sets of rules aimed at ensuring that the compiler can decide whether a given trait is implemented for a given type. All of these rules wind up preventing useful implementations and thus can be turned off with the undecidable-instances flag. I don’t think decidability is especially important. The compiler can simply keep a recursion counter and report an error if that level of recursion is exceeded. This counter can be adjusted by the user on a crate-by-crate basis if some bizarre impl pattern happens to require a deeper depth to be resolved.

Detailed design

In general, I won’t give a complete algorithmic specification. Instead, I refer readers to the prototype implementation. I would like to write out a declarative and non-algorithmic specification for the rules too, but that is work in progress and beyond the scope of this RFC. Instead, I’ll try to explain in “plain English”.

Method self-type syntax

Currently methods must be declared using the explicit-self shorthands:

fn foo(self, ...)
fn foo(&self, ...)
fn foo(&mut self, ...)
fn foo(~self, ...)

Under this proposal we would keep these shorthands but also permit any function in a trait to be used as a method, so long as the type of the first parameter is either Self or something derefable Self:

fn foo(self: Gc<Self>, ...)
fn foo(self: Rc<Self>, ...)
fn foo(self: Self, ...)      // equivalent to `fn foo(self, ...)
fn foo(self: &Self, ...)     // equivalent to `fn foo(&self, ...)

It would not be required that the first parameter be named self, though it seems like it would be useful to permit it. It’s also possible we can simply make self not be a keyword (that would be my personal preference, if we can achieve it).

Coherence

The coherence rules fall into two categories: the orphan restriction and the overlapping implementations restriction.

Orphan check: Every implementation must meet one of the following conditions:

  1. The trait being implemented (if any) must be defined in the current crate.

  2. The Self type parameter must meet the following grammar, where C is a struct or enum defined within the current crate:

    T = C
      | [T]
      | [T, ..n]
      | &T
      | &mut T
      | ~T
      | (..., T, ...)
      | X<..., T, ...> where X is not bivariant with respect to T
    

Overlapping instances: No two implementations of the same trait can be defined for the same type (note that it is only the Self type that matters). For this purpose of this check, we will also recursively check bounds. This check is ultimately defined in terms of the RESOLVE algorithm discussed in the implementation section below: it must be able to conclude that the requirements of one impl are incompatible with the other.

Here is a simple example that is OK:

trait Show { ... }
impl Show for int { ... }
impl Show for uint { ... }

The first impl implements Show for int and the case implements Show for uint. This is ok because the type int cannot be unified with uint.

The following example is NOT OK:

trait Iterator<E> { ... }
impl Iterator<char> for ~str  { ... }
impl Iterator<u8> for ~str { ... }

Even though E is bound to two distinct types, E is an output type parameter, and hence we get a coherence violation because the input type parameters are the same in each case.

Here is a more complex example that is also OK:

trait Clone { ... }
impl<A:Copy> Clone for A { ... }
impl<B:Clone> Clone for ~B { ... }

These two impls are compatible because the resolution algorithm is able to see that the type ~B will never implement Copy, no matter what B is. (Note that our ability to do this check relies on the orphan checks: without those, we’d never know if some other crate might add an implementation of Copy for ~B.)

Since trait resolution is not fully decidable, it is possible to concoct scenarios in which coherence can neither confirm nor deny the possibility that two impls are overlapping. One way for this to happen is when there are two traits which the user knows are mutually exclusive; mutual exclusion is not currently expressible in the type system [7] however, and hence the coherence check will report errors. For example:

trait Even { } // Naturally can't be Even and Odd at once!
trait Odd { }
impl<T:Even> Foo for T { }
impl<T:Odd> Foo for T { }

Another possible scenario is infinite recursion between impls. For example, in the following scenario, the coherence checked would be unable to decide if the following impls overlap:

impl<A:Foo> Bar for A { ... }
impl<A:Bar> Foo for A { ... }

In such cases, the recursion bound is exceeded and an error is conservatively reported. (Note that recursion is not always so easily detected.)

Method resolution

Let us assume the method call is r.m(...) and the type of the receiver r is R. We will resolve the call in two phases. The first phase checks for inherent methods [4] and the second phase for trait methods. Both phases work in a similar way, however. We will just describe how trait method search works and then express the inherent method search in terms of traits.

The core method search looks like this:

METHOD-SEARCH(R, m):
    let TRAITS = the set consisting of any in-scope trait T where:
        1. T has a method m and
        2. R implements T<...> for any values of Ty's type parameters

    if TRAITS is an empty set:
        if RECURSION DEPTH EXCEEDED:
            return UNDECIDABLE
        if R implements Deref<U> for some U:
            return METHOD-SEARCH(U, m)
        return NO-MATCH

    if TRAITS is the singleton set {T}:
        RECONCILE(R, T, m)

    return AMBIGUITY(TRAITS)

Basically, we will continuously auto-dereference the receiver type, searching for some type that implements a trait that offers the method m. This gives precedence to implementations that require fewer autodereferences. (There exists the possibility of a cycle in the Deref chain, so we will only autoderef so many times before reporting an error.)

Receiver reconciliation

Once we find a trait that is implemented for the (adjusted) receiver type R and which offers the method m, we must reconcile the receiver with the self type declared in m. Let me explain by example.

Consider a trait Mob (anyone who ever hacked on the MUD source code will surely remember Mobs!):

trait Mob {
    fn hit_points(&self) -> int;
    fn take_damage(&mut self, damage: int) -> int;
    fn move_to_room(self: GC<Self>, room: &Room);
}

Let’s say we have a type Monster, and Monster implements Mob:

struct Monster { ... }
impl Mob for Monster { ... }

And now we see a call to hit_points() like so:

fn attack(victim: &mut Monster) {
    let hp = victim.hit_points();
    ...
}

Our method search algorithm above will proceed by searching for an implementation of Mob for the type &mut Monster. It won’t find any. It will auto-deref &mut Monster to yield the type Monster and search again. Now we find a match. Thus far, then, we have a single autoderef *victims, yielding the type Monster – but the method hit_points() actually expects a reference (&Monster) to be given to it, not a by-value Monster.

This is where self-type reconciliation steps in. The reconciliation process works by unwinding the adjustments and adding auto-refs:

RECONCILE(R, T, m):
    let E = the expected self type of m in trait T;

    // Case 1.
    if R <: E:
      we're done.

    // Case 2.
    if &R <: E:
      add an autoref adjustment, we're done.

    // Case 3.
    if &mut R <: E:
      adjust R for mutable borrow (if not possible, error).
      add a mut autoref adjustment, we're done.

    // Case 4.
    unwind one adjustment to yield R' (if not possible, error).
    return RECONCILE(R', T, m)

In this case, the expected self type E would be &Monster. We would first check for case 1: is Monster <: &Monster? It is not. We would then proceed to case 2. Is &Monster <: &Monster? It is, and hence add an autoref. The final result then is that victim.hit_points() becomes transformed to the equivalent of (using UFCS notation) Mob::hit_points(&*victim).

To understand case 3, let’s look at a call to take_damage:

fn attack(victim: &mut Monster) {
    let hp = victim.hit_points(); // ...this is what we saw before
    let damage = hp / 10;         // 1/10 of current HP in damage
    victim.take_damage(damage);
    ...
}

As before, we would auto-deref once to find the type Monster. This time, though, the expected self type is &mut Monster. This means that both cases 1 and 2 fail and we wind up at case 3, the test for which succeeds. Now we get to this statement: “adjust R for mutable borrow”.

At issue here is the overloading of the deref operator that was discussed earlier. In this case, the end result we want is Mob::hit_points(&mut *victim), which means that * is being used for a mutable borrow, which is indicated by the DerefMut trait. However, while doing the autoderef loop, we always searched for impls of the Deref trait, since we did not yet know which trait we wanted. [2] We need to patch this up. So this loop will check whether the type &mut Monster implements DerefMut, in addition to just Deref (it does).

This check for case 3 could fail if, e.g., victim had a type like Gc<Monster> or Rc<Monster>. You’d get a nice error message like “the type Rc does not support mutable borrows, and the method take_damage() requires a mutable receiver”.

We still have not seen an example of cases 1 or 4. Let’s use a slightly modified example:

fn flee_if_possible(victim: Gc<Monster>, room: &mut Room) {
  match room.find_random_exit() {
    None => { }
    Some(exit) => {
      victim.move_to_room(exit);
    }
  }
}

As before, we’ll start out with a type of Monster, but this type the method move_to_room() has a receiver type of Gc<Monster>. This doesn’t match cases 1, 2, or 3, so we proceed to case 4 and unwind by one adjustment. Since the most recent adjustment was to deref from Gc<Monster> to Monster, we are left with a type of Gc<Monster>. We now search again. This time, we match case 1. So the final result is Mob::move_to_room(victim, room). This last case is sort of interesting because we had to use the autoderef to find the method, but once resolution is complete we do not wind up dereferencing victim at all.

Finally, let’s see an error involving case 4. Imagine we modified the type of victim in our previous example to be &Monster and not Gc<Monster>:

fn flee_if_possible(victim: &Monster, room: &mut Room) {
  match room.find_random_exit() {
    None => { }
    Some(exit) => {
      victim.move_to_room(exit);
    }
  }
}

In this case, we would again unwind an adjustment, going from Monster to &Monster, but at that point we’d be stuck. There are no more adjustments to unwind and we never found a type Gc<Monster>. Therefore, we report an error like “the method move_to_room() expects a Gc<Monster> but was invoked with an &Monster”.

Inherent methods

Inherent methods can be “desugared” into traits by assuming a trait per struct or enum. Each impl like impl Foo is effectively an implementation of that trait, and all those traits are assumed to be imported and in scope.

Differences from today

Today’s algorithm isn’t really formally defined, but it works very differently from this one. For one thing, it is based purely on subtyping checks, and does not rely on the generic trait matching. This is a crucial limitation that prevents cases like those described in lack of backtracking from working. It also results in a lot of code duplication and a general mess.

Interaction with vtables and type inference

One of the goals of this proposal is to remove the hokey distinction between early and late resolution. The way that this will work now is that, as we execute, we’ll accumulate a list of pending trait obligations. Each obligation is the combination of a trait and set of types. It is called an obligation because, for the method to be correctly typed, we must eventually find an implementation of that trait for those types. Due to type inference, though, it may not be possible to do this right away, since some of the types may not yet be fully known.

The semantics of trait resolution mean that, at any point in time, the type checker is free to stop what it’s doing and try to resolve these pending obligations, so long as none of the input type parameters are unresolved (see below). If it is able to definitely match an impl, this may in turn affect some type variables which are output type parameters. The basic idea then is to always defer doing resolution until we either (a) encounter a point where we need more type information to proceed or (b) have finished checking the function. At those times, we can go ahead and try to do resolution. If, after type checking the function in its entirety, there are still obligations that cannot be definitely resolved, that’s an error.

Ensuring crate concatenation

To ensure crate concentanability, we must only consider the Self type parameter when deciding when a trait has been implemented (more generally, we must know the precise set of input type parameters; I will cover an expanded set of rules for this in a subsequent RFC).

To see why this matters, imagine a scenario like this one:

trait Produce<R> {
    fn produce(&self: Self) -> R;
}

Now imagine I have two crates, C and D. Crate C defines two types, Vector and Real, and specifies a way to combine them:

struct Vector;
impl Produce<int> for Vector { ... }

Now imagine crate C has some code like:

fn foo() {
    let mut v = None;
    loop {
        if v.is_some() {
            let x = v.get().produce(); // (*)
            ...
        } else {
            v = Some(Vector);
        }
    }
}

At the point (*) of the call to produce() we do not yet know the type of the receiver. But the inferencer might conclude that, since it can only see one impl of Produce for Vector, v must have type Vector and hence x must have the type int.

However, then we might find another crate D that adds a new impl:

struct Other;
struct Real;
impl Combine<Real> for Other { ... }

This definition passes the orphan check because at least one of the types (Real, in this case) in the impl is local to the current crate. But what does this mean for our previous inference result? In general, it looks risky to decide types based on the impls we can see, since there could always be more impls we can’t actually see.

It seems clear that this aggressive inference breaks the crate concatenation property. If we combined crates C and D into one crate, then inference would fail where it worked before.

If x were never used in any way that forces it to be an int, then it’s even plausible that the type Real would have been valid in some sense. So the inferencer is influencing program execution to some extent.

Implementation details

The “resolve” algorithm

The basis for the coherence check, method lookup, and vtable lookup algorithms is the same function, called RESOLVE. The basic idea is that it takes a set of obligations and tries to resolve them. The result is four sets:

  • CONFIRMED: Obligations for which we were able to definitely select a specific impl.
  • NO-IMPL: Obligations which we know can NEVER be satisfied, because there is no specific impl. The only reason that we can ever say this for certain is due to the orphan check.
  • DEFERRED: Obligations that we could not definitely link to an impl, perhaps because of insufficient type information.
  • UNDECIDABLE: Obligations that were not decidable due to excessive recursion.

In general, if we ever encounter a NO-IMPL or UNDECIDABLE, it’s probably an error. DEFERRED obligations are ok until we reach the end of the function. For details, please refer to the prototype.

Alternatives and downsides

Autoderef and ambiguity

The addition of a Deref trait makes autoderef complicated, because we may encounter situations where the smart pointer and its reference both implement a trait, and we cannot know what the user wanted.

The current rule just decides in favor of the smart pointer; this is somewhat unfortunate because it is likely to not be what the user wanted. It also means that adding methods to smart pointer types is a potentially breaking change. This is particularly problematic because we may want the smart pointer to implement a trait that requires the method in question!

An interesting thought would be to change this rule and say that we always autoderef first and only resolve the method against the innermost reference. Note that UFCS provides an explicit “opt-out” if this is not what was desired. This should also have the (beneficial, in my mind) effect of quelling the over-eager use of Deref for types that are not smart pointers.

This idea appeals to me but I think belongs in a separate RFC. It needs to be evaluated.

Footnotes

Note 1: when combining with DST, the in keyword goes first, and then any other qualifiers. For example, in unsized RHS or in type RHS etc. (The precise qualifier in use will depend on the DST proposal.)

Note 2: Note that the DerefMut<T> trait extends Deref<T>, so if a type supports mutable derefs, it must also support immutable derefs.

Note 3: The restriction that inputs must precede outputs is not strictly necessary. I added it to keep options open concerning associated types and so forth. See the Alternatives section, specifically the section on associated types.

Note 4: The prioritization of inherent methods could be reconsidered after DST has been implemented. It is currently needed to make impls like impl Trait for ~Trait work.

Note 5: The set of in-scope traits is currently defined as those that are imported by name. PR #37 proposes possible changes to this rule.

Note 6: In the section on autoderef and ambiguity, I discuss alternate rules that might allow us to lift the requirement that the receiver be named self.

Note 7: I am considering introducing mechanisms in a subsequent RFC that could be used to express mutual exclusion of traits.

Summary

Allow attributes on match arms.

Motivation

One sometimes wishes to annotate the arms of match statements with attributes, for example with conditional compilation #[cfg]s or with branch weights (the latter is the most important use).

For the conditional compilation, the work-around is duplicating the whole containing function with a #[cfg]. A case study is sfackler’s bindings to OpenSSL, where many distributions remove SSLv2 support, and so that portion of Rust bindings needs to be conditionally disabled. The obvious way to support the various different SSL versions is an enum

pub enum SslMethod {
    #[cfg(sslv2)]
    /// Only support the SSLv2 protocol
    Sslv2,
    /// Only support the SSLv3 protocol
    Sslv3,
    /// Only support the TLSv1 protocol
    Tlsv1,
    /// Support the SSLv2, SSLv3 and TLSv1 protocols
    Sslv23,
}

However, all matchs can only mention Sslv2 when the cfg is active, i.e. the following is invalid:

fn name(method: SslMethod) -> &'static str {
    match method {
        Sslv2 => "SSLv2",
        Sslv3 => "SSLv3",
        _ => "..."
    }
}

A valid method would be to have two definitions: #[cfg(sslv2)] fn name(...) and #[cfg(not(sslv2)] fn name(...). The former has the Sslv2 arm, the latter does not. Clearly, this explodes exponentially for each additional cfg’d variant in an enum.

Branch weights would allow the careful micro-optimiser to inform the compiler that, for example, a certain match arm is rarely taken:

match foo {
    Common => {}
    #[cold]
    Rare => {}
}

Detailed design

Normal attribute syntax, applied to a whole match arm.

match x {
    #[attr]
    Thing => {}

    #[attr]
    Foo | Bar => {}

    #[attr]
    _ => {}
}

Alternatives

There aren’t really any general alternatives; one could probably hack around matching on conditional enum variants with some macros and helper functions to share as much code as possible; but in general this won’t work.

Unresolved questions

Nothing particularly.

Summary

Asserts are too expensive for release builds and mess up inlining. There must be a way to turn them off. I propose macros debug_assert! and assert!. For test cases, assert! should be used.

Motivation

Asserts are too expensive in release builds.

Detailed design

There should be two macros, debug_assert!(EXPR) and assert!(EXPR). In debug builds (without --cfg ndebug), debug_assert!() is the same as assert!(). In release builds (with --cfg ndebug), debug_assert!() compiles away to nothing. The definition of assert!() is if (!EXPR) { fail!("assertion failed ({}, {}): {}", file!(), line!(), stringify!(expr) }

Alternatives

Other designs that have been considered are using debug_assert! in test cases and not providing assert!, but this doesn’t work with separate compilation.

The impact of not doing this is that assert! will be expensive, prompting people will write their own local debug_assert! macros, duplicating functionality that should have been in the standard library.

Unresolved questions

None.

Summary

The tilde (~) operator and type construction do not support allocators and therefore should be removed in favor of the box keyword and a language item for the type.

Motivation

  • There will be a unique pointer type in the standard library, Box<T,A> where A is an allocator. The ~T type syntax does not allow for custom allocators. Therefore, in order to keep ~T around while still supporting allocators, we would need to make it an alias for Box<T,Heap>. In the spirit of having one way to do things, it seems better to remove ~ entirely as a type notation.

  • ~EXPR and box EXPR are duplicate functionality; the former does not support allocators. Again in the spirit of having one and only one way to do things, I would like to remove ~EXPR.

  • Some people think ~ is confusing, as it is less self-documenting than Box.

  • ~ can encourage people to blindly add sigils attempting to get their code to compile instead of consulting the library documentation.

Drawbacks

~T may be seen as convenient sugar for a common pattern in some situations.

Detailed design

The ~EXPR production is removed from the language, and all such uses are converted into box.

Add a lang item, box. That lang item will be defined in liballoc (NB: not libmetal/libmini, for bare-metal programming) as follows:

#[lang="box"]
pub struct Box<T,A=Heap>(*T);

All parts of the compiler treat instances of Box<T> identically to the way it treats ~T today.

The destructuring form for Box<T> will be box PAT, as follows:

let box(x) = box(10);
println!("{}", x); // prints 10

Alternatives

The other possible design here is to keep ~T as sugar. The impact of doing this would be that a common pattern would be terser, but I would like to not do this for the reasons stated in “Motivation” above.

Unresolved questions

The allocator design is not yet fully worked out.

It may be possible that unforeseen interactions will appear between the struct nature of Box<T> and the built-in nature of ~T when merged.

Summary

StrBuf should be renamed to String.

Motivation

Since StrBuf is so common, it would benefit from a more traditional name.

Drawbacks

It may be that StrBuf is a better name because it mirrors Java StringBuilder or C# StringBuffer. It may also be that String is confusing because of its similarity to &str.

Detailed design

Rename StrBuf to String.

Alternatives

The impact of not doing this would be that StrBuf would remain StrBuf.

Unresolved questions

None.

Summary

The rules about the places mod foo; can be used are tightened to only permit its use in a crate root and in mod.rs files, to ensure a more sane correspondence between module structure and file system hierarchy. Most notably, this prevents a common newbie error where a module is loaded multiple times, leading to surprising incompatibility between them. This proposal does not take away one’s ability to shoot oneself in the foot should one really desire to; it just removes almost all of the rope, leaving only mixed metaphors.

Motivation

It is a common newbie mistake to write things like this:

lib.rs:

mod foo;
pub mod bar;

foo.rs:

mod baz;

pub fn foo(_baz: baz::Baz) { }

bar.rs:

mod baz;
use foo::foo;

pub fn bar(baz: baz::Baz) {
    foo(baz)
}

baz.rs:

pub struct Baz;

This fails to compile because foo::foo() wants a foo::baz::Baz, while bar::bar() is giving it a bar::baz::Baz.

Such a situation, importing one file multiple times, is exceedingly rarely what the user actually wanted to do, but the present design allows it to occur without warning the user. The alterations contained herein ensure that there is no situation where such double loading can occur without deliberate intent via #[path = "….rs"].

Drawbacks

None known.

Detailed design

When a mod foo; statement is used, the compiler attempts to find a suitable file. At present, it just blindly seeks for foo.rs or foo/mod.rs (relative to the file under parsing).

The new behaviour will only permit mod foo; if at least one of the following conditions hold:

  • The file under parsing is the crate root, or

  • The file under parsing is a mod.rs, or

  • #[path] is specified, e.g. #[path = "foo.rs"] mod foo;.

In layman’s terms, the file under parsing must “own” the directory, so to speak.

Alternatives

The rationale is covered in the summary. This is the simplest repair to the current lack of structure; all alternatives would be more complex and invasive.

One non-invasive alternative is a lint which would detect double loads. This is less desirable than the solution discussed in this RFC as it doesn’t fix the underlying problem which can, fortunately, be fairly easily fixed.

Unresolved questions

None.

Summary

Temporaries live for the enclosing block when found in a let-binding. This only holds when the reference to the temporary is taken directly. This logic should be extended to extend the cleanup scope of any temporary whose lifetime ends up in the let-binding.

For example, the following doesn’t work now, but should:

use std::os;

fn main() {
	let x = os::args().slice_from(1);
	println!("{}", x);
}

Motivation

Temporary lifetimes are a bit confusing right now. Sometimes you can keep references to them, and sometimes you get the dreaded “borrowed value does not live long enough” error. Sometimes one operation works but an equivalent operation errors, e.g. autoref of ~[T] to &[T] works but calling .as_slice() doesn’t. In general it feels as though the compiler is simply being overly restrictive when it decides the temporary doesn’t live long enough.

Drawbacks

I can’t think of any drawbacks.

Detailed design

When a reference to a temporary is passed to a function (either as a regular argument or as the self argument of a method), and the function returns a value with the same lifetime as the temporary reference, the lifetime of the temporary should be extended the same way it would if the function was not invoked.

For example, ~[T].as_slice() takes &'a self and returns &'a [T]. Calling as_slice() on a temporary of type ~[T] will implicitly take a reference &'a ~[T] and return a value &'a [T] This return value should be considered to extend the lifetime of the ~[T] temporary just as taking an explicit reference (and skipping the method call) would.

Alternatives

Don’t do this. We live with the surprising borrowck errors and the ugly workarounds that look like

let x = os::args();
let x = x.slice_from(1);

Unresolved questions

None that I know of.

Summary

Rename *T to *const T, retain all other semantics of unsafe pointers.

Motivation

Currently the T* type in C is equivalent to *mut T in Rust, and the const T* type in C is equivalent to the *T type in Rust. Noticeably, the two most similar types, T* and *T have different meanings in Rust and C, frequently causing confusion and often incorrect declarations of C functions.

If the compiler is ever to take advantage of the guarantees of declaring an FFI function as taking T* or const T* (in C), then it is crucial that the FFI declarations in Rust are faithful to the declaration in C.

The current difference in Rust unsafe pointers types with C pointers types is proving to be too error prone to realistically enable these optimizations at a future date. By renaming Rust’s unsafe pointers to closely match their C brethren, the likelihood for erroneously transcribing a signature is diminished.

Detailed design

This section will assume that the current unsafe pointer design is forgotten completely, and will explain the unsafe pointer design from scratch.

There are two unsafe pointers in rust, *mut T and *const T. These two types are primarily useful when interacting with foreign functions through a FFI. The *mut T type is equivalent to the T* type in C, and the *const T type is equivalent to the const T* type in C.

The type &mut T will automatically coerce to *mut T in the normal locations that coercion occurs today. It will also be possible to explicitly cast with an as expression. Additionally, the &T type will automatically coerce to *const T. Note that &mut T will not automatically coerce to *const T.

The two unsafe pointer types will be freely castable among one another via as expressions, but no coercion will occur between the two. Additionally, values of type uint can be casted to unsafe pointers.

When is a coercion valid?

When coercing from &'a T to *const T, Rust will guarantee that the memory will remain valid for the lifetime 'a and the memory will be immutable up to memory stored in Unsafe<U>. It is the responsibility of the code working with the *const T that the pointer is only dereferenced in the lifetime 'a.

When coercing from &'a mut T to *mut T, Rust will guarantee that the memory will stay valid during 'a and that the memory will not be accessed during 'a. Additionally, Rust will consume the &'a mut T during the coercion. It is the responsibility of the code working with the *mut T to guarantee that the unsafe pointer is only dereferenced in the lifetime 'a, and that the memory is “valid again” after 'a.

Note: Rust will consume &mut T coercions with both implicit and explicit coercions.

The term “valid again” is used to represent that some types in Rust require internal invariants, such as Box<T> never being NULL. This is often a per-type invariant, so it is the responsibility of the unsafe code to uphold these invariants.

When is a safe cast valid?

Unsafe code can convert an unsafe pointer to a safe pointer via dereferencing inside of an unsafe block. This section will discuss when this action is valid.

When converting *mut T to &'a mut T, it must be guaranteed that the memory is initialized to start out with and that nobody will access the memory during 'a except for the converted pointer.

When converting *const T to &'a T, it must be guaranteed that the memory is initialized to start out with and that nobody will write to the pointer during 'a except for memory within Unsafe<U>.

Drawbacks

Today’s unsafe pointers design is consistent with the borrowed pointers types in Rust, using the mut qualifier for a mutable pointer, and no qualifier for an “immutable” pointer. Renaming the pointers would be divergence from this consistency, and would also introduce a keyword that is not used elsewhere in the language, const.

Alternatives

  • The current *mut T type could be removed entirely, leaving only one unsafe pointer type, *T. This will not allow FFI calls to take advantage of the const T* optimizations on the caller side of the function. Additionally, this may not accurately express to the programmer what a FFI API is intending to do. Note, however, that other variants of unsafe pointer types could likely be added in the future in a backwards-compatible way.

  • More effort could be invested in auto-generating bindings, and hand-generating bindings could be greatly discouraged. This would maintain consistency with Rust pointer types, and it would allow APIs to usually being transcribed accurately by automating the process. It is unknown how realistic this solution is as it is currently not yet implemented. There may still be confusion as well that *T is not equivalent to C’s T*.

Unresolved questions

  • How much can the compiler help out when coercing &mut T to *mut T? As previously stated, the source pointer &mut T is consumed during the coercion (it’s already a linear type), but this can lead to some unexpected results:

    extern {
        fn bar(a: *mut int, b: *mut int);
    }
    
    fn foo(a: &mut int) {
        unsafe {
            bar(&mut *a, &mut *a);
        }
    }
    

    This code is invalid because it is creating two copies of the same mutable pointer, and the external function is unaware that the two pointers alias. The rule that the programmer has violated is that the pointer *mut T is only dereferenced during the lifetime of the &'a mut T pointer. For example, here are the lifetimes spelled out:

    fn foo(a: &mut int) {
        unsafe {
            bar(&mut *a, &mut *a);
    //          |-----|  |-----|
    //             |        |
    //             |       Lifetime of second argument
    //            Lifetime of first argument
        }
    }
    

    Here it can be seen that it is impossible for the C code to safely dereference the pointers passed in because lifetimes don’t extend into the function call itself. The compiler could, in this case, extend the lifetime of a coerced pointer to follow the otherwise applied temporary rules for expressions.

    In the example above, the compiler’s temporary lifetime rules would cause the first coercion to last for the entire lifetime of the call to bar, thereby disallowing the second reborrow because it has an overlapping lifetime with the first.

    It is currently an open question how necessary this sort of treatment will be, and this lifetime treatment will likely require a new RFC.

  • Will all pointer types in C need to have their own keyword in Rust for representation in the FFI?

  • To what degree will the compiler emit metadata about FFI function calls in order to take advantage of optimizations on the caller side of a function call? Do the theoretical wins justify the scope of this redesign? There is currently no concrete data measuring what benefits could be gained from informing optimization passes about const vs non-const pointers.

Summary

Add ASCII byte literals and ASCII byte string literals to the language, similar to the existing (Unicode) character and string literals. Before the RFC process was in place, this was discussed in #4334.

Motivation

Programs dealing with text usually should use Unicode, represented in Rust by the str and char types. In some cases however, a program may be dealing with bytes that can not be interpreted as Unicode as a whole, but still contain ASCII compatible bits.

For example, the HTTP protocol was originally defined as Latin-1, but in practice different pieces of the same request or response can use different encodings. The PDF file format is mostly ASCII, but can contain UTF-16 strings and raw binary data.

There is a precedent at least in Python, which has both Unicode and byte strings.

Drawbacks

The language becomes slightly more complex, although that complexity should be limited to the parser.

Detailed design

Using terminology from the Reference Manual:

Extend the syntax of expressions and patterns to add byte literals of type u8 and byte string literals of type &'static [u8] (or [u8], post-DST). They are identical to the existing character and string literals, except that:

  • They are prefixed with a b (for “binary”), to distinguish them. This is similar to the r prefix for raw strings.
  • Unescaped code points in the body must be in the ASCII range: U+0000 to U+007F.
  • '\x5c' 'u' hex_digit 4 and '\x5c' 'U' hex_digit 8 escapes are not allowed.
  • '\x5c' 'x' hex_digit 2 escapes represent a single byte rather than a code point. (They are the only way to express a non-ASCII byte.)

Examples: b'A' == 65u8, b'\t' == 9u8, b'\xFF' == 0xFFu8, b"A\t\xFF" == [65u8, 9, 0xFF]

Assuming buffer of type &[u8]

match buffer[i] {
    b'a' .. b'z' => { /* ... */ }
    c => { /* ... */ }
}

Alternatives

Status quo: patterns must use numeric literals for ASCII values, or (for a single byte, not a byte string) cast to char

match buffer[i] {
    c @ 0x61 .. 0x7A => { /* ... */ }
    c => { /* ... */ }
}
match buffer[i] as char {
    // `c` is of the wrong type!
    c @ 'a' .. 'z' => { /* ... */ }
    c => { /* ... */ }
}

Another option is to change the syntax so that macros such as bytes!() can be used in patterns, and add a byte!() macro:

match buffer[i] {
    c @ byte!('a') .. byte!('z') => { /* ... */ }
    c => { /* ... */ }
}q

This RFC was written to align the syntax with Python, but there could be many variations such as using a different prefix (maybe a for ASCII), or using a suffix instead (maybe u8, as in integer literals).

The code points from syntax could be encoded as UTF-8 rather than being mapped to bytes of the same value, but assuming UTF-8 is not always appropriate when working with bytes.

See also previous discussion in #4334.

Unresolved questions

Should there be “raw byte string” literals? E.g. pdf_file.write(rb"<< /Title (FizzBuzz \(Part one\)) >>")

Should control characters (U+0000 to U+001F) be disallowed in syntax? This should be consistent across all kinds of literals.

Should the bytes!() macro be removed in favor of this?

Summary

Allow block expressions in statics, as long as they only contain items and a trailing const expression.

Example:

static FOO: uint = { 100 };
static BAR: fn() -> int = {
    fn hidden() -> int {
        42
    }
    hidden
};

Motivation

This change allows defining items as part of a const expression, and evaluating to a value using them. This is mainly useful for macros, as it allows hiding complex machinery behind something that expands to a value, but also enables using unsafe {} blocks in a static initializer.

Real life examples include the regex! macro, which currently expands to a block containing a function definition and a value, and would be usable in a static with this.

Another example would be to expose a static reference to a fixed memory address by dereferencing a raw pointer in a const expr, which is useful in embedded and kernel, but requires a unsafe block to do.

The outcome of this is that one additional expression type becomes valid as a const expression, with semantics that are a strict subset of its equivalent in a function.

Drawbacks

Block expressions in a function are usually just used to run arbitrary code before evaluating to a value. Allowing them in statics without allowing code execution might be confusing.

Detailed design

A branch implementing this feature can be found at https://github.com/Kimundi/rust/tree/const_block.

It mainly involves the following changes:

  • const check now allows block expressions in statics:
    • All statements that are not item declarations lead to an compile error.
  • trans and const eval are made aware of block expressions:
    • A trailing expression gets evaluated as a constant.
    • A missing trailing expressions is treated as a unit value.
  • trans is made to recurse into static expressions to generate possible items.

Things like privacy/reachability of definitions inside a static block are already handled more generally at other places, as the situation is very similar to a regular function.

The branch also includes tests that show how this feature works in practice.

Alternatives

Because this feature is a straight forward extension of the valid const expressions, it already causes a very minimal impact on the language, with most alternative ways of enabling the same benefits being more complex.

For example, a expression AST node that can include items but is only usable from procedural macros could be added.

Not having this feature would not prevent anything interesting from getting implemented, but it would lead to less nice looking solutions.

For example, a comparison between static-supporting regex! with and without this feature:

// With this feature, you can just initialize a static:
static R: Regex = regex!("[0-9]");

// Without it, the static needs to be generated by the
// macro itself, alongside all generated items:
regex! {
    static R = "[0-9]";
}

Unresolved questions

None so far.

Summary

Leave structs with unspecified layout by default like enums, for optimisation purposes. Use something like #[repr(C)] to expose C compatible layout.

Motivation

The members of a struct are always laid in memory in the order in which they were specified, e.g.

struct A {
    x: u8,
    y: u64,
    z: i8,
    w: i64,
}

will put the u8 first in memory, then the u64, the i8 and lastly the i64. Due to the alignment requirements of various types padding is often required to ensure the members start at an appropriately aligned byte. Hence the above struct is not 1 + 8 + 1 + 8 == 18 bytes, but rather 1 + 7 + 8 + 1 + 7 + 8 == 32 bytes, since it is laid out like

#[packed] // no automatically inserted padding
struct AFull {
    x: u8,
    _padding1: [u8, .. 7],
    y: u64,
    z: i8,
    _padding2: [u8, .. 7],
    w: i64
}

If the fields were reordered to

struct B {
    y: u64,
    w: i64,

    x: u8,
    i: i8
}

then the struct is (strictly) only 18 bytes (but the alignment requirements of u64 forces it to take up 24).

Having an undefined layout does allow for possible security improvements, like randomising struct fields, but this can trivially be done with a syntax extension that can be attached to a struct to reorder the fields in the AST itself. That said, there may be benefits from being able to randomise all structs in a program automatically/for testing, effectively fuzzing code (especially unsafe code).

Notably, Rust’s enums already have undefined layout, and provide the #[repr] attribute to control layout more precisely (specifically, selecting the size of the discriminant).

Drawbacks

Forgetting to add #[repr(C)] for a struct intended for FFI use can cause surprising bugs and crashes. There is already a lint for FFI use of enums without a #[repr(...)] attribute, so this can be extended to include structs.

Having an unspecified (or otherwise non-C-compatible) layout by default makes interfacing with C slightly harder. A particularly bad case is passing to C a struct from an upstream library that doesn’t have a repr(C) attribute. This situation seems relatively similar to one where an upstream library type is missing an implementation of a core trait e.g. Hash if one wishes to use it as a hashmap key.

It is slightly better if structs had a specified-but-C-incompatible layout, and one has control over the C interface, because then one can manually arrange the fields in the C definition to match the Rust order.

That said, this scenario requires:

  • Needing to pass a Rust struct into C/FFI code, where that FFI code actually needs to use things from the struct, rather than just pass it through, e.g., back into a Rust callback.
  • The Rust struct is defined upstream & out of your control, and not intended for use with C code.
  • The C/FFI code is designed by someone other than that vendor, or otherwise not designed for use with the Rust struct (or else it is a bug in the vendor’s library that the Rust struct can’t be sanely passed to C).

Detailed design

A struct declaration like

struct Foo {
    x: T,
    y: U,
    ...
}

has no fixed layout, that is, a compiler can choose whichever order of fields it prefers.

A fixed layout can be selected with the #[repr] attribute

#[repr(C)]
struct Foo {
    x: T,
    y: U,
    ...
}

This will force a struct to be laid out like the equivalent definition in C.

There would be a lint for the use of non-repr(C) structs in related FFI definitions, for example:

struct UnspecifiedLayout {
   // ...
}

#[repr(C)]
struct CLayout {
   // ...
}


extern {
    fn foo(x: UnspecifiedLayout); // warning: use of non-FFI-safe struct in extern declaration

    fn bar(x: CLayout); // no warning
}

extern "C" fn foo(x: UnspecifiedLayout) { } // warning: use of non-FFI-safe struct in function with C abi.

Alternatives

  • Have non-C layouts opt-in, via #[repr(smallest)] and #[repr(random)] (or similar).
  • Have layout defined, but not declaration order (like Java(?)), for example, from largest field to smallest, so u8 fields get placed last, and [u8, .. 1000000] fields get placed first. The #[repr] attributes would still allow for selecting declaration-order layout.

Unresolved questions

  • How does this interact with binary compatibility of dynamic libraries?
  • How does this interact with DST, where some fields have to be at the end of a struct? (Just always lay-out unsized fields last? (i.e. after monomorphisation if a field was originally marked Sized? then it needs to be last).)

Summary

Allow macro expansion in patterns, i.e.

match x {
    my_macro!() => 1,
    _ => 2,
}

Motivation

This is consistent with allowing macros in expressions etc. It’s also a year-old open issue.

I have implemented this feature already and I’m using it to condense some ubiquitous patterns in the HTML parser I’m writing. This makes the code more concise and easier to cross-reference with the spec.

Drawbacks / alternatives

A macro invocation in this position:

match x {
    my_macro!()

could potentially expand to any of three different syntactic elements:

  • A pattern, i.e. Foo(x)
  • The left side of a match arm, i.e. Foo(x) | Bar(x) if x > 5
  • An entire match arm, i.e. Foo(x) | Bar(x) if x > 5 => 1

This RFC proposes only the first of these, but the others would be more useful in some cases. Supporting multiple of the above would be significantly more complex.

Another alternative is to use a macro for the entire match expression, e.g.

my_match!(x {
    my_new_syntax => 1,
    _ => 2,
})

This doesn’t involve any language changes, but requires writing a complicated procedural macro. (My sustained attempts to do things like this with MBE macros have all failed.) Perhaps I could alleviate some of the pain with a library for writing match-like macros, or better use of the existing parser in libsyntax.

The my_match! approach is also not very composable.

Another small drawback: rustdoc can’t document the name of a function argument which is produced by a pattern macro.

Unresolved questions

None, as far as I know.

Summary

Generalize the #[macro_registrar] feature so it can register other kinds of compiler plugins.

Motivation

I want to implement loadable lints and use them for project-specific static analysis passes in Servo. Landing this first will allow more evolution of the plugin system without breaking source compatibility for existing users.

Detailed design

To register a procedural macro in current Rust:

use syntax::ast::Name;
use syntax::parse::token;
use syntax::ext::base::{SyntaxExtension, BasicMacroExpander, NormalTT};

#[macro_registrar]
pub fn macro_registrar(register: |Name, SyntaxExtension|) {
    register(token::intern("named_entities"),
        NormalTT(box BasicMacroExpander {
            expander: named_entities::expand,
            span: None
        },
        None));
}

I propose an interface like

use syntax::parse::token;
use syntax::ext::base::{BasicMacroExpander, NormalTT};

use rustc::plugin::Registry;

#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
    reg.register_macro(token::intern("named_entities"),
        NormalTT(box BasicMacroExpander {
            expander: named_entities::expand,
            span: None
        },
        None));
}

Then the struct Registry could provide additional methods such as register_lint as those features are implemented.

It could also provide convenience methods:

use rustc::plugin::Registry;

#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
    reg.register_simple_macro("named_entities", named_entities::expand);
}

phase(syntax) becomes phase(plugin), with the former as a deprecated synonym that warns. This is to avoid silent breakage of the very common #[phase(syntax)] extern crate log.

We only need one phase of loading plugin crates, even though the plugins we load may be used at different points (or not at all).

Drawbacks

Breaking change for existing procedural macros.

More moving parts.

Registry is provided by librustc, because it will have methods for registering lints and other librustc things. This means that syntax extensions must link librustc, when before they only needed libsyntax (but could link librustc anyway if desired). This was discussed on the RFC PR and the Rust PR and on IRC.

#![feature(macro_registrar)] becomes unknown, contradicting a comment in feature_gate.rs:

This list can never shrink, it may only be expanded (in order to prevent old programs from failing to compile)

Since when do we ensure that old programs will compile? ;) The #[macro_registrar] attribute wouldn’t work anyway.

Alternatives

We could add #[lint_registrar] etc. alongside #[macro_registrar]. This seems like it will produce more duplicated effort all around. It doesn’t provide convenience methods, and it won’t support API evolution as well.

We could support the old #[macro_registrar] by injecting an adapter shim. This is significant extra work to support a feature with no stability guarantee.

Unresolved questions

Naming bikeshed.

What set of convenience methods should we provide?

Summary

Bounds on trait objects should be separated with +.

Motivation

With DST there is an ambiguity between the following two forms:

trait X {
    fn f(foo: b);
}

and

trait X {
    fn f(Trait: Share);
}

See Rust issue #12778 for details.

Also, since kinds are now just built-in traits, it makes sense to treat a bounded trait object as just a combination of traits. This could be extended in the future to allow objects consisting of arbitrary trait combinations.

Detailed design

Instead of : in trait bounds for first-class traits (e.g. &Trait:Share + Send), we use + (e.g. &Trait + Share + Send).

+ will not be permitted in as without parentheses. This will be done via a special restriction in the type grammar: the special TYPE production following as will be the same as the regular TYPE production, with the exception that it does not accept + as a binary operator.

Drawbacks

  • It may be that + is ugly.

  • Adding a restriction complicates the type grammar more than I would prefer, but the community backlash against the previous proposal was overwhelming.

Alternatives

The impact of not doing this is that the inconsistencies and ambiguities above remain.

Unresolved questions

Where does the 'static bound fit into all this?

Summary

Allow users to load custom lints into rustc, similar to loadable syntax extensions.

Motivation

There are many possibilities for user-defined static checking:

  • Enforcing correct usage of Servo’s JS-managed pointers
  • lilyball’s use case: checking that rust-lua functions which call longjmp never have destructors on stack variables
  • Enforcing a company or project style guide
  • Detecting common misuses of a library, e.g. expensive or non-idiomatic constructs
  • In cryptographic code, annotating which variables contain secrets and then forbidding their use in variable-time operations or memory addressing

Existing project-specific static checkers include:

  • A Clang plugin that detects misuse of GLib and GObject
  • A GCC plugin (written in Python!) that detects misuse of the CPython extension API
  • Sparse, which checks Linux kernel code for issues such as mixing up userspace and kernel pointers (often exploitable for privilege escalation)

We should make it easy to build such tools and integrate them with an existing Rust project.

Detailed design

In rustc::lint (which today is rustc::middle::lint):

pub struct Lint {
    /// An identifier for the lint, written with underscores,
    /// e.g. "unused_imports".
    pub name: &'static str,

    /// Default level for the lint.
    pub default_level: Level,

    /// Description of the lint or the issue it detects,
    /// e.g. "imports that are never used"
    pub desc: &'static str,
}

#[macro_export]
macro_rules! declare_lint ( ($name:ident, $level:ident, $desc:expr) => (
    static $name: &'static ::rustc::lint::Lint
        = &::rustc::lint::Lint {
            name: stringify!($name),
            default_level: ::rustc::lint::$level,
            desc: $desc,
        };
))

pub type LintArray = &'static [&'static Lint];

#[macro_export]
macro_rules! lint_array ( ($( $lint:expr ),*) => (
    {
        static array: LintArray = &[ $( $lint ),* ];
        array
    }
))

pub trait LintPass {
    fn get_lints(&self) -> LintArray;

    fn check_item(&mut self, cx: &Context, it: &ast::Item) { }
    fn check_expr(&mut self, cx: &Context, e: &ast::Expr) { }
    ...
}

pub type LintPassObject = Box<LintPass: 'static>;

To define a lint:

#![crate_id="lipogram"]
#![crate_type="dylib"]
#![feature(phase, plugin_registrar)]

extern crate syntax;

// Load rustc as a plugin to get macros
#[phase(plugin, link)]
extern crate rustc;

use syntax::ast;
use syntax::parse::token;
use rustc::lint::{Context, LintPass, LintPassObject, LintArray};
use rustc::plugin::Registry;

declare_lint!(letter_e, Warn, "forbid use of the letter 'e'")

struct Lipogram;

impl LintPass for Lipogram {
    fn get_lints(&self) -> LintArray {
        lint_array!(letter_e)
    }

    fn check_item(&mut self, cx: &Context, it: &ast::Item) {
        let name = token::get_ident(it.ident);
        if name.get().contains_char('e') || name.get().contains_char('E') {
            cx.span_lint(letter_e, it.span, "item name contains the letter 'e'");
        }
    }
}

#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
    reg.register_lint_pass(box Lipogram as LintPassObject);
}

A pass which defines multiple lints will have e.g. lint_array!(deprecated, experimental, unstable).

To use a lint when compiling another crate:

#![feature(phase)]

#[phase(plugin)]
extern crate lipogram;

fn hello() { }

fn main() { hello() }

And you will get

test.rs:6:1: 6:15 warning: item name contains the letter 'e', #[warn(letter_e)] on by default
test.rs:6 fn hello() { }
          ^~~~~~~~~~~~~~

Internally, lints are identified by the address of a static Lint. This has a number of benefits:

  • The linker takes care of assigning unique IDs, even with dynamically loaded plugins.
  • A typo writing a lint ID is usually a compiler error, unlike with string IDs.
  • The ability to output a given lint is controlled by the usual visibility mechanism. Lints defined within rustc use the same infrastructure and will simply export their Lints if other parts of the compiler need to output those lints.
  • IDs are small and easy to hash.
  • It’s easy to go from an ID to name, description, etc.

User-defined lints are controlled through the usual mechanism of attributes and the -A -W -D -F flags to rustc. User-defined lints will show up in -W help if a crate filename is also provided; otherwise we append a message suggesting to re-run with a crate filename.

See also the full demo.

Drawbacks

This increases the amount of code in rustc to implement lints, although it makes each individual lint much easier to understand in isolation.

Loadable lints produce more coupling of user code to rustc internals (with no official stability guarantee, of course).

There’s no scoping / namespacing of the lint name strings used by attributes and compiler flags. Attempting to register a lint with a duplicate name is an error at registration time.

The use of &'static means that lint plugins can’t dynamically generate the set of lints based on some external resource.

Alternatives

We could provide a more generic mechanism for user-defined AST visitors. This could support other use cases like code transformation. But it would be harder to use, and harder to integrate with the lint infrastructure.

It would be nice to magically find all static Lints in a crate, so we don’t need get_lints. Is this worth adding another attribute and another crate metadata type? The plugin::Registry mechanism was meant to avoid such a proliferation of metadata types, but it’s not as declarative as I would like.

Unresolved questions

Do we provide guarantees about visit order for a lint, or the order of multiple lints defined in the same crate? Some lints may require multiple passes.

Should we enforce (while running lints) that each lint printed with span_lint was registered by the corresponding LintPass? Users who particularly care can already wrap lints in modules and use visibility to enforce this statically.

Should we separate registering a lint pass from initializing / constructing the value implementing LintPass? This would support a future where a single rustc invocation can compile multiple crates and needs to reset lint state.

Summary

Simplify Rust’s lexical syntax to make tooling easier to use and easier to define.

Motivation

Rust’s lexer does a lot of work. It un-escapes escape sequences in string and character literals, and parses numeric literals of 4 different bases. It also strips comments, which is sensible, but can be undesirable for pretty printing or syntax highlighting without hacks. Since many characters are allowed in strings both escaped and raw (tabs, newlines, and unicode characters come to mind), after lexing it is impossible to tell if a given character was escaped or unescaped in the source, making the lexer difficult to test against a model.

Detailed design

The following (antlr4) grammar completely describes the proposed lexical syntax:

lexer grammar RustLexer;

/* import Xidstart, Xidcont; */

/* Expression-operator symbols */

EQ      : '=' ;
LT      : '<' ;
LE      : '<=' ;
EQEQ    : '==' ;
NE      : '!=' ;
GE      : '>=' ;
GT      : '>' ;
ANDAND  : '&&' ;
OROR    : '||' ;
NOT     : '!' ;
TILDE   : '~' ;
PLUS    : '+' ;
MINUS   : '-' ;
STAR    : '*' ;
SLASH   : '/' ;
PERCENT : '%' ;
CARET   : '^' ;
AND     : '&' ;
OR      : '|' ;
SHL     : '<<' ;
SHR     : '>>' ;

BINOP
    : PLUS
    | MINUS
    | STAR
    | PERCENT
    | CARET
    | AND
    | OR
    | SHL
    | SHR
    ;

BINOPEQ : BINOP EQ ;

/* "Structural symbols" */

AT         : '@' ;
DOT        : '.' ;
DOTDOT     : '..' ;
DOTDOTDOT  : '...' ;
COMMA      : ',' ;
SEMI       : ';' ;
COLON      : ':' ;
MOD_SEP    : '::' ;
LARROW     : '->' ;
FAT_ARROW  : '=>' ;
LPAREN     : '(' ;
RPAREN     : ')' ;
LBRACKET   : '[' ;
RBRACKET   : ']' ;
LBRACE     : '{' ;
RBRACE     : '}' ;
POUND      : '#';
DOLLAR     : '$' ;
UNDERSCORE : '_' ;

KEYWORD : STRICT_KEYWORD | RESERVED_KEYWORD ;

fragment STRICT_KEYWORD
  : 'as'
  | 'box'
  | 'break'
  | 'continue'
  | 'crate'
  | 'else'
  | 'enum'
  | 'extern'
  | 'fn'
  | 'for'
  | 'if'
  | 'impl'
  | 'in'
  | 'let'
  | 'loop'
  | 'match'
  | 'mod'
  | 'mut'
  | 'once'
  | 'proc'
  | 'pub'
  | 'ref'
  | 'return'
  | 'self'
  | 'static'
  | 'struct'
  | 'super'
  | 'trait'
  | 'true'
  | 'type'
  | 'unsafe'
  | 'use'
  | 'virtual'
  | 'while'
  ;

fragment RESERVED_KEYWORD
  : 'alignof'
  | 'be'
  | 'const'
  | 'do'
  | 'offsetof'
  | 'priv'
  | 'pure'
  | 'sizeof'
  | 'typeof'
  | 'unsized'
  | 'yield'
  ;

// Literals

fragment HEXIT
  : [0-9a-fA-F]
  ;

fragment CHAR_ESCAPE
  : [nrt\\'"0]
  | [xX] HEXIT HEXIT
  | 'u' HEXIT HEXIT HEXIT HEXIT
  | 'U' HEXIT HEXIT HEXIT HEXIT HEXIT HEXIT HEXIT HEXIT
  ;

LIT_CHAR
  : '\'' ( '\\' CHAR_ESCAPE | ~[\\'\n\t\r] ) '\''
  ;

INT_SUFFIX
  : 'i'
  | 'i8'
  | 'i16'
  | 'i32'
  | 'i64'
  | 'u'
  | 'u8'
  | 'u16'
  | 'u32'
  | 'u64'
  ;

LIT_INTEGER
  : [0-9][0-9_]* INT_SUFFIX?
  | '0b' [01][01_]* INT_SUFFIX?
  | '0o' [0-7][0-7_]* INT_SUFFIX?
  | '0x' [0-9a-fA-F][0-9a-fA-F_]* INT_SUFFIX?
  ;

FLOAT_SUFFIX
  : 'f32'
  | 'f64'
  | 'f128'
  ;

LIT_FLOAT
  : [0-9][0-9_]* ('.' | ('.' [0-9][0-9_]*)? ([eE] [-+]? [0-9][0-9_]*)? FLOAT_SUFFIX?)
  ;

LIT_STR
  : '"' ('\\\n' | '\\\r\n' | '\\' CHAR_ESCAPE | .)*? '"'
  ;

/* this is a bit messy */

fragment LIT_STR_RAW_INNER
  : '"' .*? '"'
  | LIT_STR_RAW_INNER2
  ;

fragment LIT_STR_RAW_INNER2
  : POUND LIT_STR_RAW_INNER POUND
  ;

LIT_STR_RAW
  : 'r' LIT_STR_RAW_INNER
  ;

fragment BLOCK_COMMENT
  : '/*' (BLOCK_COMMENT | .)*? '*/'
  ;

COMMENT
  : '//' ~[\r\n]*
  | BLOCK_COMMENT
  ;

IDENT : XID_start XID_continue* ;

LIFETIME : '\'' IDENT ;

WHITESPACE : [ \r\n\t]+ ;

There are a few notable changes from today’s lexical syntax:

  • Non-doc comments are not stripped. To compensate, when encountering a COMMENT token the parser can check itself whether or not it’s a doc comment. This can be done with a simple regex: (//(/[^/]|!)|/\*(\*[^*]|!)).
  • Numeric literals are not differentiated based on presence of type suffix, nor are they converted from binary/octal/hexadecimal to decimal, nor are underscores stripped. This can be done trivially in the parser.
  • Character escapes are not unescaped. That is, if you write ‘\x20’, this lexer will give you LIT_CHAR('\x20') rather than LIT_CHAR(' '). The same applies to string literals.

The output of the lexer then becomes annotated spans – which part of the document corresponds to which token type. Even whitespace is categorized.

Drawbacks

Including comments and whitespace in the token stream is very non-traditional and not strictly necessary.

Summary

Do not identify struct literals by searching for :. Instead define a sub- category of expressions which excludes struct literals and re-define for, if, and other expressions which take an expression followed by a block (or non-terminal which can be replaced by a block) to take this sub-category, instead of all expressions.

Motivation

Parsing by looking ahead is fragile - it could easily be broken if we allow : to appear elsewhere in types (e.g., type ascription) or if we change struct literals to not require the : (e.g., if we allow empty structs to be written with braces, or if we allow struct literals to unify field names to local variable names, as has been suggested in the past and which we currently do for struct literal patterns). We should also be able to give better error messages today if users make these mistakes. More worryingly, we might come up with some language feature in the future which is not predictable now and which breaks with the current system.

Hopefully, it is pretty rare to use struct literals in these positions, so there should not be much fallout. Any problems can be easily fixed by assigning the struct literal into a variable. However, this is a backwards incompatible change, so it should block 1.0.

Detailed design

Here is a simplified version of a subset of Rust’s abstract syntax:

e      ::= x
         | e `.` f
         | name `{` (x `:` e)+ `}`
         | block
         | `for` e `in` e block
         | `if` e block (`else` block)?
         | `|` pattern* `|` e
         | ...
block  ::=  `{` (e;)* e? `}`

Parsing this grammar is ambiguous since x cannot be distinguished from name, so e block in the for expression is ambiguous with the struct literal expression. We currently solve this by using lookahead to find a : token in the struct literal.

I propose the following adjustment:

e      ::= e'
         | name `{` (x `:` e)+ `}`
         | `|` pattern* `|` e
         | ...
e'     ::= x
         | e `.` f
         | block
         | `for` e `in` e' block
         | `if` e' block (`else` block)?
         | `|` pattern* `|` e'
         | ...
block  ::=  `{` (e;)* e? `}`

e' is just e without struct literal expressions. We use e' instead of e wherever e is followed directly by block or any other non-terminal which may have block as its first terminal (after any possible expansions).

For any expressions where a sub-expression is the final lexical element (closures in the subset above, but also unary and binary operations), we require two versions of the meta-expression - the normal one in e and a version with e' for the final element in e'.

Implementation would be simpler, we just add a flag to parser::restriction called RESTRICT_BLOCK or something, which puts us into a mode which reflects e'. We would drop in to this mode when parsing e' position expressions and drop out of it for all but the last sub-expression of an expression.

Drawbacks

It makes the formal grammar and parsing a little more complicated (although it is simpler in terms of needing less lookahead and avoiding a special case).

Alternatives

Don’t do this.

Allow all expressions but greedily parse non-terminals in these positions, e.g., for N {} {} would be parsed as for (N {}) {}. This seems worse because I believe it will be much rarer to have structs in these positions than to have an identifier in the first position, followed by two blocks (i.e., parse as (for N {}) {}).

Unresolved questions

Do we need to expose this distinction anywhere outside of the parser? E.g., macros?

Summary

Remove localization features from format!, and change the set of escapes accepted by format strings. The plural and select methods would be removed, # would no longer need to be escaped, and {{/}} would become escapes for { and }, respectively.

Motivation

Localization is difficult to implement correctly, and doing so will likely not be done in the standard library, but rather in an external library. After talking with others much more familiar with localization, it has come to light that our ad-hoc “localization support” in our format strings are woefully inadequate for most real use cases of support for localization.

Instead of having a half-baked unused system adding complexity to the compiler and libraries, the support for this functionality would be removed from format strings.

Detailed design

The primary localization features that format! supports today are the plural and select methods inside of format strings. These methods are choices made at format-time based on the input arguments of how to format a string. This functionality would be removed from the compiler entirely.

As fallout of this change, the # special character, a back reference to the argument being formatted, would no longer be necessary. In that case, this character no longer needs an escape sequence.

The new grammar for format strings would be as follows:

format_string := <text> [ format <text> ] *
format := '{' [ argument ] [ ':' format_spec ] '}'
argument := integer | identifier

format_spec := [[fill]align][sign]['#'][0][width]['.' precision][type]
fill := character
align := '<' | '>'
sign := '+' | '-'
width := count
precision := count | '*'
type := identifier | ''
count := parameter | integer
parameter := integer '$'

The current syntax can be found at http://doc.rust-lang.org/std/fmt/#syntax to see the diff between the two

Choosing a new escape sequence

Upon landing, there was a significant amount of discussion about the escape sequence that would be used in format strings. Some context can be found on some old pull requests, and the current escape mechanism has been the source of much confusion. With the removal of localization methods, and namely nested format directives, it is possible to reconsider the choices of escaping again.

The only two characters that need escaping in format strings are { and }. One of the more appealing syntaxes for escaping was to double the character to represent the character itself. This would mean that {{ is an escape for a { character, while }} would be an escape for a } character.

Adopting this scheme would avoid clashing with Rust’s string literal escapes. There would be no “double escape” problem. More details on this can be found in the comments of an old PR.

Drawbacks

The localization methods of select/plural are genuinely used for applications that do not involve localization. For example, the compiler and rustdoc often use plural to easily create plural messages. Removing this functionality from format strings would impose a burden of likely dynamically allocating a string at runtime or defining two separate format strings.

Additionally, changing the syntax of format strings is quite an invasive change. Raw string literals serve as a good use case for format strings that must escape the { and } characters. The current system is arguably good enough to pass with for today.

Alternatives

The major localization approach explored has been l20n, which has shown itself to be fairly incompatible with the way format strings work today. Different localization systems, however, have not been explored. Systems such as gettext would be able to leverage format strings quite well, but it was claimed that gettext for localization is inadequate for modern use-cases.

It is also an unexplored possibility whether the current format string syntax could be leveraged by l20n. It is unlikely that time will be allocated to polish off an localization library before 1.0, and it is currently seen as undesirable to have a half-baked system in the libraries rather than a first-class well designed system.

Unresolved questions

  • Should localization support be left in std::fmt as a “poor man’s” implementation for those to use as they see fit?

Summary

Add a partial_cmp method to PartialOrd, analogous to cmp in Ord.

Motivation

The Ord::cmp method is useful when working with ordered values. When the exact ordering relationship between two values is required, cmp is both potentially more efficient than computing both a > b and then a < b and makes the code clearer as well.

I feel that in the case of partial orderings, an equivalent to cmp is even more important. I’ve found that it’s very easy to accidentally make assumptions that only hold true in the total order case (for example !(a < b) => a >= b). Explicitly matching against the possible results of the comparison helps keep these assumptions from creeping in.

In addition, the current default implementation setup is a bit strange, as implementations in the partial equality trait assume total equality. This currently makes it easier to incorrectly implement PartialOrd for types that do not have a total ordering, and if PartialOrd is separated from Ord in a way similar to this proposal, the default implementations for PartialOrd will need to be removed and an implementation of the trait will require four repetitive implementations of the required methods.

Detailed design

Add a method to PartialOrd, changing the default implementations of the other methods:

pub trait PartialOrd {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering>;

    fn lt(&self, other: &Self) -> bool {
        match self.partial_cmp(other) {
            Some(Less) => true,
            _ => false,
        }
    }

    fn le(&self, other: &Self) -> bool {
        match self.partial_cmp(other) {
            Some(Less) | Some(Equal) => true,
            _ => false,
        }
    }

    fn gt(&self, other: &Self) -> bool {
        match self.partial_cmp(other) {
            Some(Greater) => true,
            _ => false,
        }
    }

    fn ge(&self, other: &Self) -> bool {
        match self.partial_cmp(other) {
            Some(Greater) | Some(Equal) => true,
            _ => false,
        }
    }
}

Since almost all ordered types have a total ordering, the implementation of partial_cmp is trivial in most cases:

impl PartialOrd for Foo {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        Some(self.cmp(other))
    }
}

This can be done automatically if/when RFC #48 or something like it is accepted and implemented.

Drawbacks

This does add some complexity to PartialOrd. In addition, the more commonly used methods (lt, etc) may become more expensive than they would normally be if their implementations call into partial_ord.

Alternatives

We could invert the default implementations and have a default implementation of partial_cmp in terms of lt and gt. This may slightly simplify things in current Rust, but it makes the default implementation less efficient than it should be. It would also require more work to implement PartialOrd once the currently planned cmp reform has finished as noted above.

partial_cmp could just be called cmp, but it seems like UFCS would need to be implemented first for that to be workable.

Unresolved questions

We may want to add something similar to PartialEq as well. I don’t know what it would be called, though (maybe partial_eq?):

// I don't feel great about these variant names, but `Equal` is already taken
// by `Ordering` which is in the same module.
pub enum Equality {
    AreEqual,
    AreUnequal,
}

pub trait PartialEq {
    fn partial_eq(&self, other: &Self) -> Option<Equality>;

    fn eq(&self, other: &Self) -> bool {
        match self.partial_eq(other) {
            Some(AreEqual) => true,
            _ => false,
        }
    }

    fn neq(&self, other: &Self) -> bool {
        match self.partial_eq(other) {
            Some(AreUnequal) => true,
            _ => false,
        }
    }
}

Summary

Rust currently forbids pattern guards on match arms with move-bound variables. Allowing them would increase the applicability of pattern guards.

Motivation

Currently, if you attempt to use guards on a match arm with a move-bound variable, e.g.

struct A { a: Box<int> }

fn foo(n: int) {
    let x = A { a: box n };
    let y = match x {
        A { a: v } if *v == 42 => v,
        _ => box 0
    };
}

you get an error:

test.rs:6:16: 6:17 error: cannot bind by-move into a pattern guard
test.rs:6         A { a: v } if *v == 42 => v,
                         ^

This should be permitted in cases where the guard only accesses the moved value by reference or copies out of derived paths.

This allows for succinct code with less pattern matching duplication and a minimum number of copies at runtime. The lack of this feature was encountered by @kmcallister when developing Servo’s new HTML 5 parser.

Detailed design

This change requires all occurrences of move-bound pattern variables in the guard to be treated as paths to the values being matched before they are moved, rather than the moved values themselves. Any moves of matched values into the bound variables would occur on the control flow edge between the guard and the arm’s expression. There would be no changes to the handling of reference-bound pattern variables.

The arm would be treated as its own nested scope with respect to borrows, so that pattern-bound variables would be able to be borrowed and dereferenced freely in the guard, but these borrows would not be in scope in the arm’s expression. Since the guard dominates the expression and the move into the pattern-bound variable, moves of either the match’s head expression or any pattern-bound variables in the guard would trigger an error.

The following examples would be accepted:

struct A { a: Box<int> }

impl A {
    fn get(&self) -> int { *self.a }
}

fn foo(n: int) {
    let x = A { a: box n };
    let y = match x {
        A { a: v } if *v == 42 => v,
        _ => box 0
    };
}

fn bar(n: int) {
    let x = A { a: box n };
    let y = match x {
        A { a: v } if x.get() == 42 => v,
        _ => box 0
    };
}

fn baz(n: int) {
    let x = A { a: box n };
    let y = match x {
        A { a: v } if *v.clone() == 42 => v,
        _ => box 0
    };
}

This example would be rejected, due to a double move of v:

struct A { a: Box<int> }

fn foo(n: int) {
    let x = A { a: box n };
    let y = match x {
        A { a: v } if { drop(v); true } => v,
        _ => box 0
    };
}

This example would also be rejected, even though there is no use of the move-bound variable in the first arm’s expression, since the move into the bound variable would be moving the same value a second time:

enum VecWrapper { A(Vec<int>) }

fn foo(x: VecWrapper) -> uint {
    match x {
        A(v) if { drop(v); false } => 1,
        A(v) => v.len()
    }
}

There are issues with mutation of the bound values, but that is true without the changes proposed by this RFC, e.g. Rust issue #14684. The general approach to resolving that issue should also work with these proposed changes.

This would be implemented behind a feature(bind_by_move_pattern_guards) gate until we have enough experience with the feature to remove the feature gate.

Drawbacks

The current error message makes it more clear what the user is doing wrong, but if this change is made the error message for an invalid use of this feature (even if it were accidental) would indicate a use of a moved value, which might be more confusing.

This might be moderately difficult to implement in rustc.

Alternatives

As far as I am aware, the only workarounds for the lack of this feature are to manually expand the control flow of the guard (which can quickly get messy) or use unnecessary copies.

Unresolved questions

This has nontrivial interaction with guards in arbitrary patterns as proposed in #99.

Summary

  • Remove the crate_id attribute and knowledge of versions from rustc.
  • Add a #[crate_name] attribute similar to the old #[crate_id] attribute
  • Filenames will no longer have versions, nor will symbols
  • A new flag, --extern, will be used to override searching for external crates
  • A new flag, -C metadata=foo, used when hashing symbols

Motivation

The intent of CrateId and its support has become unclear over time as the initial impetus, rustpkg, has faded over time. With cargo on the horizon, doubts have been cast on the compiler’s support for dealing with crate versions and friends. The goal of this RFC is to simplify the compiler’s knowledge about the identity of a crate to allow cargo to do all the necessary heavy lifting.

This new crate identification is designed to not compromise on the usability of the compiler independent of cargo. Additionally, all use cases support today with a CrateId should still be supported.

Detailed design

A new #[crate_name] attribute will be accepted by the compiler, which is the equivalent of the old #[crate_id] attribute, except without the “crate id” support. This new attribute can have a string value describe a valid crate name.

A crate name must be a valid rust identifier with the exception of allowing the - character after the first character.

#![crate_name = "foo"]
#![crate_type = "lib"]

pub fn foo() { /* ... */ }

Naming library filenames

Currently, rustc creates filenames for library following this pattern:

lib<name>-<version>-<hash>.rlib

The current scheme defines <hash> to be the hash of the CrateId value. This naming scheme achieves a number of goals:

  • Libraries of the same name can exist next to one another if they have different versions.
  • Libraries of the same name and version, but from different sources, can exist next to one another due to having different hashes.
  • Rust libraries can have very privileged names such as core and std without worrying about polluting the global namespace of other system libraries.

One drawback of this scheme is that the output filename of the compiler is unknown due to the <hash> component. One must query rustc itself to determine the name of the library output.

Under this new scheme, the new output filenames by the compiler would be:

lib<name>.rlib

Note that both the <version> and the <hash> are missing by default. The <version> was removed because the compiler no longer knows about the version, and the <hash> was removed to make the output filename predictable.

The three original goals can still be satisfied with this simplified naming scheme. As explained in the next section, the compiler’s “glob pattern” when searching for a crate named foo will be libfoo*.rlib, which will help rationalize some of these conclusions.

  • Libraries of the same name can exist next to one another because they can be manually renamed to have extra data after the libfoo, such as the version.
  • Libraries of the same name and version, but different source, can also exist by modifying what comes after libfoo, such as including a hash.
  • Rust does not need to occupy a privileged namespace as the default rust installation would include hashes in all the filenames as necessary. More on this later.

Additionally, with a predictable filename output external tooling should be easier to write.

Loading crates

The goal of the crate loading phase of the compiler is to map a set of extern crate statements to (dylib,rlib) pairs that are present on the filesystem. To do this, the current system matches dependencies via the CrateId syntax:

extern crate json = "super-fast-json#0.1.0";

In today’s compiler, this directive indicates that the a filename of the form libsuper-fast-json-0.1.0-<hash>.rlib must be found to be a candidate. Further checking happens once a candidate is found to ensure that it is indeed a rust library.

Concerns have been raised that this key point of dependency management is where the compiler is doing work that is not necessarily its prerogative. In a cargo-driven world, versions are primarily managed in an external manifest, in addition to doing other various actions such as renaming packages at compile time.

One solution would be to add more version management to the compiler, but this is seen as the compiler delving too far outside what it was initially tasked to do. With this in mind, this is the new proposal for the extern crate syntax:

extern crate json = "super-fast-json";

Notably, the CrateId is removed entirely, along with the version and path associated with it. The string value of the extern crate directive is still optional (defaulting to the identifier), and the string must be a valid crate name (as defined above).

The compiler’s searching and file matching logic would be altered to only match crates based on name. If two versions of a crate are found, the compiler will unconditionally emit an error. It will be up to the user to move the two libraries on the filesystem and control the -L flags to the compiler to enable disambiguation.

This imples that when the compiler is searching for the crate named foo, it will search all of the lookup paths for files which match the pattern libfoo*.{so,rlib}. This is likely to return many false positives, but they will be easily weeded out once the compiler realizes that there is no metadata in the library.

This scheme is strictly less powerful than the previous, but it moves a good deal of logic from the compiler to cargo.

Manually specifying dependencies

Cargo is often seen as “expert mode” in its usage of the compiler. Cargo will always have prior knowledge about what exact versions of a library will be used for any particular dependency, as well as where the outputs are located.

If the compiler provided no support for loading crates beyond matching filenames, it would limit many of cargo’s use cases. For example, cargo could not compile a crate with two different versions of an upstream crate. Additionally, cargo could not substitute libfast-json for libslow-json at compile time (assuming they have the same API).

To accommodate an “expert mode” in rustc, the compiler will grow a new command line flag of the form:

--extern json=path/to/libjson

This directive will indicate that the library json can be found at path/to/libjson. The file extension is not specified, and it is assume that the rlib/dylib pair are located next to one another at this location (libjson is the file stem).

This will enable cargo to drive how the compiler loads crates by manually specifying where files are located and exactly what corresponds to what.

Symbol mangling

Today, mangled symbols contain the version number at the end of the symbol itself. This was originally intended to tie into Linux’s ability to version symbols, but in retrospect this is generally viewed as over-ambitious as the support is not currently there, nor does it work on windows or OSX.

Symbols would no longer contain the version number anywhere within them. The hash at the end of each symbol would only include the crate name and metadata from the command line. Metadata from the command line will be passed via a new command line flag, -C metadata=foo, which specifies a string to hash.

The standard rust distribution

The standard distribution would continue to put hashes in filenames manually because the libraries are intended to occupy a privileged space on the system. The build system would manually move a file after it was compiled to the correct destination filename.

Drawbacks

  • The compiler is able to operate fairly well independently of cargo today, and this scheme would hamstring the compiler by limiting the number of “it just works” use cases. If cargo is not being used, build systems will likely have to start using --extern to specify dependencies if name conflicts or version conflicts arise between crates.

  • This scheme still has redundancy in the list of dependencies with the external cargo manifest. The source code would no longer list versions, but the cargo manifest will contain the same identifier for each dependency that the source code will contain.

Alternatives

  • The compiler could go in the opposite direction of this proposal, enhancing extern crate instead of simplifying it. The compiler could learn about things like version ranges and friends, while still maintaining flags to fine tune its behavior. It is unclear whether this increase in complexity will be paired with a large enough gain in usability of the compiler independent of cargo.

Unresolved questions

  • An implementation for the more advanced features of cargo does not currently exist, to it is unknown whether --extern will be powerful enough for cargo to satisfy all its use cases with.

  • Are the string literal parts of extern crate justified? Allowing a string literal just for the - character may be overkill.

Summary

Index should be split into Index and IndexMut.

Motivation

Currently, the Index trait is not suitable for most array indexing tasks. The slice functionality cannot be replicated using it, and as a result the new Vec has to use .get() and .get_mut() methods.

Additionally, this simply follows the Deref/DerefMut split that has been implemented for a while.

Detailed design

We split Index into two traits (borrowed from @nikomatsakis):

// self[element] -- if used as rvalue, implicitly a deref of the result
trait Index<E,R> {
    fn index<'a>(&'a self, element: &E) -> &'a R;
}

// &mut self[element] -- when used as a mutable lvalue
trait IndexMut<E,R> {
    fn index_mut<'a>(&'a mut self, element: &E) -> &'a mut R;
}

Drawbacks

  • The number of lang. items increases.

  • This design doesn’t support moving out of a vector-like object. This can be added backwards compatibly.

  • This design doesn’t support hash tables because there is no assignment operator. This can be added backwards compatibly.

Alternatives

The impact of not doing this is that the [] notation will not be available to Vec.

Unresolved questions

None that I’m aware of.

Summary

Remove the coercion from Box<T> to &mut T from the language.

Motivation

Currently, the coercion between Box<T> to &mut T can be a hazard because it can lead to surprising mutation where it was not expected.

Detailed design

The coercion between Box<T> and &mut T should be removed.

Note that methods that take &mut self can still be called on values of type Box<T> without any special referencing or dereferencing. That is because the semantics of auto-deref and auto-ref conspire to make it work: the types unify after one autoderef followed by one autoref.

Drawbacks

Borrowing from Box<T> to &mut T may be convenient.

Alternatives

An alternative is to remove &T coercions as well, but this was decided against as they are convenient.

The impact of not doing this is that the coercion will remain.

Unresolved questions

None.

Summary

  • Convert function call a(b, ..., z) into an overloadable operator via the traits Fn<A,R>, FnShare<A,R>, and FnOnce<A,R>, where A is a tuple (B, ..., Z) of the types B...Z of the arguments b...z, and R is the return type. The three traits differ in their self argument (&mut self vs &self vs self).
  • Remove the proc expression form and type.
  • Remove the closure types (though the form lives on as syntactic sugar, see below).
  • Modify closure expressions to permit specifying by-reference vs by-value capture and the receiver type:
    • Specifying by-reference vs by-value closures:
      • ref |...| expr indicates a closure that captures upvars from the environment by reference. This is what closures do today and the behavior will remain unchanged, other than requiring an explicit keyword.
      • |...| expr will therefore indicate a closure that captures upvars from the environment by value. As usual, this is either a copy or move depending on whether the type of the upvar implements Copy.
    • Specifying receiver mode (orthogonal to capture mode above):
      • |a, b, c| expr is equivalent to |&mut: a, b, c| expr
      • |&mut: ...| expr indicates that the closure implements Fn
      • |&: ...| expr indicates that the closure implements FnShare
      • |: a, b, c| expr indicates that the closure implements FnOnce.
  • Add syntactic sugar where |T1, T2| -> R1 is translated to a reference to one of the fn traits as follows:
    • |T1, ..., Tn| -> R is translated to Fn<(T1, ..., Tn), R>
    • |&mut: T1, ..., Tn| -> R is translated to Fn<(T1, ..., Tn), R>
    • |&: T1, ..., Tn| -> R is translated to FnShare<(T1, ..., Tn), R>
    • |: T1, ..., Tn| -> R is translated to FnOnce<(T1, ..., Tn), R>

One aspect of closures that this RFC does not describe is that we must permit trait references to be universally quantified over regions as closures are today. A description of this change is described below under Unresolved questions and the details will come in a forthcoming RFC.

Motivation

Over time we have observed a very large number of possible use cases for closures. The goal of this RFC is to create a unified closure model that encompasses all of these use cases.

Specific goals (explained in more detail below):

  1. Give control over inlining to users.
  2. Support closures that bind by reference and closures that bind by value.
  3. Support different means of accessing the closure environment, corresponding to self, &self, and &mut self methods.

As a side benefit, though not a direct goal, the RFC reduces the size/complexity of the language’s core type system by unifying closures and traits.

The core idea: unifying closures and traits

The core idea of the RFC is to unify closures, procs, and traits. There are a number of reasons to do this. First, it simplifies the language, because closures, procs, and traits already served similar roles and there was sometimes a lack of clarity about which would be the appropriate choice. However, in addition, the unification offers increased expressiveness and power, because traits are a more generic model that gives users more control over optimization.

The basic idea is that function calls become an overridable operator. Therefore, an expression like a(...) will be desugar into an invocation of one of the following traits:

trait Fn<A,R> {
    fn call(&mut self, args: A) -> R;
}

trait FnShare<A,R> {
    fn call_share(&self, args: A) -> R;
}

trait FnOnce<A,R> {
    fn call_once(self, args: A) -> R;
}

Essentially, a(b, c, d) becomes sugar for one of the following:

Fn::call(&mut a, (b, c, d))
FnShare::call_share(&a, (b, c, d))
FnOnce::call_once(a, (b, c, d))

To integrate with this, closure expressions are then translated into a fresh struct that implements one of those three traits. The precise trait is currently indicated using explicit syntax but may eventually be inferred.

This change gives user control over virtual vs static dispatch. This works in the same way as generic types today:

fn foo(x: &mut Fn<(int,),int>) -> int {
    x(2) // virtual dispatch
}

fn foo<F:Fn<(int,),int>>(x: &mut F) -> int {
    x(2) // static dispatch
}

The change also permits returning closures, which is not currently possible (the example relies on the proposed impl syntax from rust-lang/rfcs#105):

fn foo(x: impl Fn<(int,),int>) -> impl Fn<(int,),int> {
    |v| x(v * 2)
}

Basically, in this design there is nothing special about a closure. Closure expressions are simply a convenient way to generate a struct that implements a suitable Fn trait.

Bind by reference vs bind by value

When creating a closure, it is now possible to specify whether the closure should capture variables from its environment (“upvars”) by reference or by value. The distinction is indicated using the leading keyword ref:

|| foo(a, b)      // captures `a` and `b` by value

ref || foo(a, b)  // captures `a` and `b` by reference, as today

Reasons to bind by value

Bind by value is useful when creating closures that will escape from the stack frame that created them, such as task bodies (spawn(|| ...)) or combinators. It is also useful for moving values out of a closure, though it should be possible to enable that with bind by reference as well in the future.

Reasons to bind by reference

Bind by reference is useful for any case where the closure is known not to escape the creating stack frame. This frequently occurs when using closures to encapsulate common control-flow patterns:

map.insert_or_update_with(key, value, || ...)
opt_val.unwrap_or_else(|| ...)

In such cases, the closure frequently wishes to read or modify local variables on the enclosing stack frame. Generally speaking, then, such closures should capture variables by-reference – that is, they should store a reference to the variable in the creating stack frame, rather than copying the value out. Using a reference allows the closure to mutate the variables in place and also avoids moving values that are simply read temporarily.

The vast majority of closures in use today are should be “by reference” closures. The only exceptions are those closures that wish to “move out” from an upvar (where we commonly use the so-called “option dance” today). In fact, even those closures could be “by reference” closures, but we will have to extend the inference to selectively identify those variables that must be moved and take those “by value”.

Detailed design

Closure expression syntax

Closure expressions will have the following form (using EBNF notation, where [] denotes optional things and {} denotes a comma-separated list):

CLOSURE = ['ref'] '|' [SELF] {ARG} '|' ['->' TYPE] EXPR
SELF    =  ':' | '&' ':' | '&' 'mut' ':'
ARG     = ID [ ':' TYPE ]

The optional keyword ref is used to indicate whether this closure captures by reference or by value.

Closures are always translated into a fresh struct type with one field per upvar. In a by-value closure, the types of these fields will be the same as the types of the corresponding upvars (modulo &mut reborrows, see below). In a by-reference closure, the types of these fields will be a suitable reference (&, &mut, etc) to the variables being borrowed.

By-value closures

The default form for a closure is by-value. This implies that all upvars which are referenced are copied/moved into the closure as appropriate. There is one special case: if the type of the value to be moved is &mut, we will “reborrow” the value when it is copied into the closure. That is, given an upvar x of type &'a mut T, the value which is actually captured will have type &'b mut T where 'b <= 'a. This rule is consistent with our general treatment of &mut, which is to aggressively reborrow wherever possible; moreover, this rule cannot introduce additional compilation errors, it can only make more programs successfully typecheck.

By-reference closures

A by-reference closure is a convenience form in which values used in the closure are converted into references before being captured. By-reference closures are always rewritable into by-value closures if desired, but the rewrite can often be cumbersome and annoying.

Here is a (rather artificial) example of a by-reference closure in use:

let in_vec: Vec<int> = ...;
let mut out_vec: Vec<int> = Vec::new();
let opt_int: Option<int> = ...;

opt_int.map(ref |v| {
    out_vec.push(v);
    in_vec.fold(v, |a, &b| a + b)
});

This could be rewritten into a by-value closure as follows:

let in_vec: Vec<int> = ...;
let mut out_vec: Vec<int> = Vec::new();
let opt_int: Option<int> = ...;

opt_int.map({
    let in_vec = &in_vec;
    let out_vec = &mut in_vec;
    |v| {
        out_vec.push(v);
        in_vec.fold(v, |a, &b| a + b)
    }
})

In this case, the capture closed over two variables, in_vec and out_vec. As you can see, the compiler automatically infers, for each variable, how it should be borrowed and inserts the appropriate capture.

In the body of a ref closure, the upvars continue to have the same type as they did in the outer environment. For example, the type of a reference to in_vec in the above example is always Vec<int>, whether or not it appears as part of a ref closure. This is not only convenient, it is required to make it possible to infer whether each variable is borrowed as an &T or &mut T borrow.

Note that there are some cases where the compiler internally employs a form of borrow that is not available in the core language, &uniq. This borrow does not permit aliasing (like &mut) but does not require mutability (like &). This is required to allow transparent closing over of &mut pointers as described in this blog post.

Evolutionary note: It is possible to evolve by-reference closures in the future in a backwards compatible way. The goal would be to cause more programs to type-check by default. Two possible extensions follow:

  • Detect when values are moved and hence should be taken by value rather than by reference. (This is only applicable to once closures.)
  • Detect when it is only necessary to borrow a sub-path. Imagine a closure like ref || use(&context.variable_map). Currently, this closure will borrow context, even though it only uses the field variable_map. As a result, it is sometimes necessary to rewrite the closure to have the form {let v = &context.variable_map; || use(v)}. In the future, however, we could extend the inference so that rather than borrowing context to create the closure, we would borrow context.variable_map directly.

Closure sugar in trait references

The current type for closures, |T1, T2| -> R, will be repurposed as syntactic sugar for a reference to the appropriate Fn trait. This shorthand be used any place that a trait reference is appropriate. The full type will be written as one of the following:

<'a...'z> |T1...Tn|: K -> R
<'a...'z> |&mut: T1...Tn|: K -> R
<'a...'z> |&: T1...Tn|: K -> R
<'a...'z> |: T1...Tn|: K -> R

Each of which would then be translated into the following trait references, respectively:

<'a...'z> Fn<(T1...Tn), R> + K
<'a...'z> Fn<(T1...Tn), R> + K
<'a...'z> FnShare<(T1...Tn), R> + K
<'a...'z> FnOnce<(T1...Tn), R> + K

Note that the bound lifetimes 'a...'z are not in scope for the bound K.

Drawbacks

This model is more complex than the existing model in some respects (but the existing model does not serve the full set of desired use cases).

Alternatives

There is one aspect of the design that is still under active discussion:

Introduce a more generic sugar. It was proposed that we could introduce Trait(A, B) -> C as syntactic sugar for Trait<(A,B),C> rather than retaining the form |A,B| -> C. This is appealing but removes the correspondence between the expression form and the corresponding type. One (somewhat open) question is whether there will be additional traits that mirror fn types that might benefit from this more general sugar.

Tweak trait names. In conjunction with the above, there is some concern that the type name fn(A) -> B for a bare function with no environment is too similar to Fn(A) -> B for a closure. To remedy that, we could change the name of the trait to something like Closure(A) -> B (naturally the other traits would be renamed to match).

Then there are a large number of permutations and options that were largely rejected:

Only offer by-value closures. We tried this and found it required a lot of painful rewrites of perfectly reasonable code.

Make by-reference closures the default. We felt this was inconsistent with the language as a whole, which tends to make “by value” the default (e.g., x vs ref x in patterns, x vs &x in expressions, etc.).

Use a capture clause syntax that borrows individual variables. “By value” closures combined with let statements already serve this role. Simply specifying “by-reference closure” also gives us room to continue improving inference in the future in a backwards compatible way. Moreover, the syntactic space around closures expressions is extremely constrained and we were unable to find a satisfactory syntax, particularly when combined with self-type annotations. Finally, if we decide we do want the ability to have “mostly by-value” closures, we can easily extend the current syntax by writing something like (ref x, ref mut y) || ... etc.

Retain the proc expression form. It was proposed that we could retain the proc expression form to specify a by-value closure and have || expressions be by-reference. Frankly, the main objection to this is that nobody likes the proc keyword.

Use variadic generics in place of tuple arguments. While variadic generics are an interesting addition in their own right, we’d prefer not to introduce a dependency between closures and variadic generics. Having all arguments be placed into a tuple is also a simpler model overall. Moreover, native ABIs on platforms of interest treat a structure passed by value identically to distinct arguments. Finally, given that trait calls have the “Rust” ABI, which is not specified, we can always tweak the rules if necessary (though there are advantages for tooling when the Rust ABI closely matches the native ABI).

Use inference to determine the self type of a closure rather than an annotation. We retain this option for future expansion, but it is not clear whether we can always infer the self type of a closure. Moreover, using inference rather a default raises the question of what to do for a type like |int| -> uint, where inference is not possible.

Default to something other than &mut self. It is our belief that this is the most common use case for closures.

Transition plan

TBD. pcwalton is working furiously as we speak.

Unresolved questions

What relationship should there be between the closure traits? On the one hand, there is clearly a relationship between the traits. For example, given a FnShare, one can easily implement Fn:

impl<A,R,T:FnShare<A,R>> Fn<A,R> for T {
    fn call(&mut self, args: A) -> R {
        (&*self).call_share(args)
    }
}

Similarly, given a Fn or FnShare, you can implement FnOnce. From this, one might derive a subtrait relationship:

trait FnOnce { ... }
trait Fn : FnOnce { ... }
trait FnShare : Fn { ... }

Employing this relationship, however, would require that any manual implement of FnShare or Fn must implement adapters for the other two traits, since a subtrait cannot provide a specialized default of supertrait methods (yet?). On the other hand, having no relationship between the traits limits reuse, at least without employing explicit adapters.

Other alternatives that have been proposed to address the problem:

  • Use impls to implement the fn traits in terms of one another, similar to what is shown above. The problem is that we would need to implement FnOnce both for all T where T:Fn and for all T where T:FnShare. This will yield coherence errors unless we extend the language with a means to declare traits as mutually exclusive (which might be valuable, but no such system has currently been proposed nor agreed upon).

  • Have the compiler implement multiple traits for a single closure. As with supertraits, this would require manual implements to implement multiple traits. It would also require generic users to write T:Fn+FnMut or else employ an explicit adapter. On the other hand, it preserves the “one method per trait” rule described below.

Can we optimize away the trait vtable? The runtime representation of a reference &Trait to a trait object (and hence, under this proposal, closures as well) is a pair of pointers (data, vtable). It has been proposed that we might be able to optimize this representation to (data, fnptr) so long as Trait has a single function. This slightly improves the performance of invoking the function as one need not indirect through the vtable. The actual implications of this on performance are unclear, but it might be a reason to keep the closure traits to a single method.

Closures that are quantified over lifetimes

A separate RFC is needed to describe bound lifetimes in trait references. For example, today one can write a type like <'a> |&'a A| -> &'a B, which indicates a closure that takes and returns a reference with the same lifetime specified by the caller at each call-site. Note that a trait reference like Fn<(&'a A), &'a B>, while syntactically similar, does not have the same meaning because it lacks the universal quantifier <'a>. Therefore, in the second case, 'a refers to some specific lifetime 'a, rather than being a lifetime parameter that is specified at each callsite. The high-level summary of the change therefore is to permit trait references like <'a> Fn<(&'a A), &'a B>; in this case, the value of <'a> will be specified each time a method or other member of the trait is accessed.

Summary

Currently we use inference to find the current type of otherwise-unannotated integer literals, and when that fails the type defaults to int. This is often felt to be potentially error-prone behavior.

This proposal removes the integer inference fallback and strengthens the types required for several language features that interact with integer inference.

Motivation

With the integer fallback, small changes to code can change the inferred type in unexpected ways. It’s not clear how big a problem this is, but previous experiments1 indicate that removing the fallback has a relatively small impact on existing code, so it’s reasonable to back off of this feature in favor of more strict typing.

See also https://github.com/mozilla/rust/issues/6023.

Detailed design

The primary change here is that, when integer type inference fails, the compiler will emit an error instead of assigning the value the type int.

This change alone will cause a fair bit of existing code to be unable to type check because of lack of constraints. To add more constraints and increase likelihood of unification, we ‘tighten’ up what kinds of integers are required in some situations:

  • Array repeat counts must be uint ([expr, .. count])
  • << and >> require uint when shifting integral types

Finally, inference for as will be modified to track the types a value is being cast to for cases where the value being cast is unconstrained, like 0 as u8.

Treatment of enum discriminants will need to change:

enum Color { Red = 0, Green = 1, Blue = 2 }

Currently, an unsuffixed integer defaults to int. Instead, we will only require enum discriminants primitive integers of unspecified type; assigning an integer to an enum will behave as if casting from from the type of the integer to an unsigned integer with the size of the enum discriminant.

Drawbacks

This will force users to type hint somewhat more often. In particular, ranges of unsigned ints may need to be type-hinted:

for _ in range(0u, 10) { }

Alternatives

Do none of this.

Unresolved questions

  • If we’re putting new restrictions on shift operators, should we change the traits, or just make the primitives special?
  • Start Date: 2014-06-12
  • RFC PR #: https://github.com/rust-lang/rfcs/pull/116
  • Rust Issue #: https://github.com/rust-lang/rust/issues/16464

Summary

Remove or feature gate the shadowing of view items on the same scope level, in order to have less complicated semantic and be more future proof for module system changes or experiments.

This means the names brought in scope by extern crate and use may never collide with each other, nor with any other item (unless they live in different namespaces). Eg, this will no longer work:

extern crate foo;
use foo::bar::foo; // ERROR: There is already a module `foo` in scope

Shadowing would still be allowed in case of lexical scoping, so this continues to work:

extern crate foo;

fn bar() {
    use foo::bar::foo; // Shadows the outer foo

    foo::baz();
}

Definitions

Due to a certain lack of official, clearly defined semantics and terminology, a list of relevant definitions is included:

  • Scope A scope in Rust is basically defined by a block, following the rules of lexical scoping:

    scope 1 (visible: scope 1)
    {
          scope 1-1 (visible: scope 1, scope 1-1)
          {
              scope 1-1-1 (visible: scope 1, scope 1-1, scope 1-1-1)
          }
          scope 1-1
          {
              scope 1-1-2
          }
          scope 1-1
    }
    scope 1
    

    Blocks include block expressions, fn items and mod items, but not things like extern, enum or struct. Additionally, mod is special in that it isolates itself from parent scopes.

  • Scope Level Anything with the same name in the example above is on the same scope level. In a scope level, all names defined in parent scopes are visible, but can be shadowed by a new definition with the same name, which will be in scope for that scope itself and all its child scopes.

  • Namespace Rust has different namespaces, and the scoping rules apply to each one separately. The exact number of different namespaces is not well defined, but they are roughly

    • types (enum Foo {})
    • modules (mod foo {})
    • item values (static FOO: uint = 0;)
    • local values (let foo = 0;)
    • lifetimes (impl<'a> ...)
    • macros (macro_rules! foo {...})
  • Definition Item Declarations that create new entities in a crate are called (by the author) definition items. They include struct, enum, mod, fn, etc. Each of them creates a name in the type, module, item value or macro namespace in the same scope level they are written in.

  • View Item Declarations that just create aliases to existing declarations in a crate are called view items. They include use and extern crate, and also create a name in the type, module, item value or macro namespace in the same scope level they are written in.

  • Item Both definition items and view items together are collectively called items.

  • Shadowing While the principle of shadowing exists in all namespaces, there are different forms of it:

    • item-style: Declarations shadow names from outer scopes, and are visible everywhere in their own, including lexically before their own definition. This requires there to be only one definition with the same name and namespace per scope level. Types, modules, item values and lifetimes fall under these rules.
    • sequentially: Declarations shadow names that are lexically before them, both in parent scopes and their own. This means you can reuse the same name in the same scope, but a definition will not be visibly before itself. This is how local values and macros work. (Due to sequential code execution and parsing, respectively)
    • view item: A special case exists with view items; In the same scope level, extern crate creates entries in the module namespace, which are shadowable by names created with use, which are shadowable with any definition item. The singular goal of this RFC is to remove this shadowing behavior of view items

Motivation

As explained above, what is currently visible under which namespace in a given scope is determined by a somewhat complicated three step process:

  1. First, every extern crate item creates a name in the module namespace.
  2. Then, every use can create a name in any namespace, shadowing the extern crate ones.
  3. Lastly, any definition item can shadow any name brought in scope by both extern crate and use.

These rules have developed mostly in response to the older, more complicated import system, and the existence of wildcard imports (use foo::*). In the case of wildcard imports, this shadowing behavior prevents local code from breaking if the source module gets updated to include new names that happen to be defined locally.

However, wildcard imports are now feature gated, and name conflicts in general can be resolved by using the renaming feature of extern crate and use, so in the current non-gated state of the language there is no need for this shadowing behavior.

Gating it off opens the door to remove it altogether in a backwards compatible way, or to re-enable it in case wildcard imports are officially supported again.

It also makes the mental model around items simpler: Any shadowing of items happens through lexical scoping only, and every item can be considered unordered and mutually recursive.

If this RFC gets accepted, a possible next step would be a RFC to lift the ordering restriction between extern crate, use and definition items, which would make them truly behave the same in regard to shadowing and the ability to be reordered. It would also lift the weirdness of use foo::bar; mod foo;.

Implementing this RFC would also not change anything about how name resolution works, as its just a tightening of the existing rules.

Drawbacks

  • Feature gating import shadowing might break some code using #[feature(globs)].
  • The behavior of libstds prelude becomes more magical if it still allows shadowing, but this could be de-magified again by a new feature, see below in unresolved questions.
  • Or the utility of libstds prelude becomes more restricted if it doesn’t allow shadowing.

Detailed design

A new feature gate import_shadowing gets created.

During the name resolution phase of compilation, every time the compiler detects a shadowing between extern crate, use and definition items in the same scope level, it bails out unless the feature gate got enabled. This amounts to two rules:

  • Items in the same scope level and either the type, module, item value or lifetime namespace may not shadow each other in the respective namespace.
  • Items may shadow names from outer scopes in any namespace.

Just like for the globs feature, the libstd prelude import would be preempt from this, and still be allowed to be shadowed.

Alternatives

The alternative is to do nothing, and risk running into a backwards compatibility hazard, or committing to make a final design decision around the whole module system before 1.0 gets released.

Unresolved questions

  • It is unclear how the libstd preludes fits into this.

    On the one hand, it basically acts like a hidden use std::prelude::*; import which ignores the globs feature, so it could simply also ignore the import_shadowing feature as well, and the rule becomes that the prelude is a magic compiler feature that injects imports into every module but doesn’t prevent the user from taking the same names.

    On the other hand, it is also thinkable to simply forbid shadowing of prelude items as well, as defining things with the same name as std exports is not recommended anyway, and this would nicely enforce that. It would however mean that the prelude can not change without breaking backwards compatibility, which might be too restricting.

    A compromise would be to specialize wildcard imports into a new prelude use feature, which has the explicit properties of being shadow-able and using a wildcard import. libstds prelude could then simply use that, and users could define and use their own preludes as well. But that’s a somewhat orthogonal feature, and should be discussed in its own RFC.

  • Interaction with overlapping imports.

    Right now its legal to write this:

fn main() { use Bar = std::result::Result; use Bar = std::option::Option; let x: Bar = None; }

where the latter `use` shadows the former. This would have to be forbidden as well,
however the current semantic seems like a accident anyway.

Summary

Rename the Share trait to Sync

Motivation

With interior mutability, the name “immutable pointer” for a value of type &T is not quite accurate. Instead, the term “shared reference” is becoming popular to reference values of type &T. The usage of the term “shared” is in conflict with the Share trait, which is intended for types which can be safely shared concurrently with a shared reference.

Detailed design

Rename the Share trait in std::kinds to Sync. Documentation would refer to &T as a shared reference and the notion of “shared” would simply mean “many references” while Sync implies that it is safe to share among many threads.

Drawbacks

The name Sync may invoke conceptions of “synchronized” from languages such as Java where locks are used, rather than meaning “safe to access in a shared fashion across tasks”.

Alternatives

As any bikeshed, there are a number of other names which could be possible for this trait:

  • Concurrent
  • Synchronized
  • Threadsafe
  • Parallel
  • Threaded
  • Atomic
  • DataRaceFree
  • ConcurrentlySharable

Unresolved questions

None.

Summary

Remove special treatment of Box<T> from the borrow checker.

Motivation

Currently the Box<T> type is special-cased and converted to the old ~T internally. This is mostly invisible to the user, but it shows up in some places that give special treatment to Box<T>. This RFC is specifically concerned with the fact that the borrow checker has greater precision when dereferencing Box<T> vs other smart pointers that rely on the Deref traits. Unlike the other kinds of special treatment, we do not currently have a plan for how to extend this behavior to all smart pointer types, and hence we would like to remove it.

Here is an example that illustrates the extra precision afforded to Box<T> vs other types that implement the Deref traits. The following program, written using the Box type, compiles successfully:

struct Pair {
    a: uint,
    b: uint
}

fn example1(mut smaht: Box<Pair>) {
    let a = &mut smaht.a;
    let b = &mut smaht.b;
    ...
}

This program compiles because the type checker can see that (*smaht).a and (*smaht).b are always distinct paths. In contrast, if I use a smart pointer, I get compilation errors:

fn example2(cell: RefCell<Pair>) {
    let mut smaht: RefMut<Pair> = cell.borrow_mut();
    let a = &mut smaht.a;
    
    // Error: cannot borrow `smaht` as mutable more than once at a time
    let b = &mut smaht.b;
}

To see why this, consider the desugaring:

fn example2(smaht: RefCell<Pair>) {
    let mut smaht = smaht.borrow_mut();
    
    let tmp1: &mut Pair = smaht.deref_mut(); // borrows `smaht`
    let a = &mut tmp1.a;
    
    let tmp2: &mut Pair = smaht.deref_mut(); // borrows `smaht` again!
    let b = &mut tmp2.b;
}

It is a violation of the Rust type system to invoke deref_mut when the reference to a is valid and usable, since deref_mut requires &mut self, which in turn implies no alias to self or anything owned by self.

This desugaring suggests how the problem can be worked around in user code. The idea is to pull the result of the deref into a new temporary:

fn example3(smaht: RefCell<Pair>) {
    let mut smaht: RefMut<Pair> = smaht.borrow_mut();
    let temp: &mut Pair = &mut *smaht;
    let a = &mut temp.a;
    let b = &mut temp.b;
}

Detailed design

Removing this treatment from the borrow checker basically means changing the construction of loan paths for unique pointers.

I don’t actually know how best to implement this in the borrow checker, particularly concerning the desire to keep the ability to move out of boxes and use them in patterns. This requires some investigation. The easiest and best way may be to “do it right” and is probably to handle derefs of Box<T> in a similar way to how overloaded derefs are handled, but somewhat differently to account for the possibility of moving out of them. Some investigation is needed.

Drawbacks

The borrow checker rules are that much more restrictive.

Alternatives

We have ruled out inconsistent behavior between Box and other smart pointer types. We considered a number of ways to extend the current treatment of box to other smart pointer types:

  1. Require compiler to introduce deref temporaries automatically where possible. This is plausible as a future extension but requires some thought to work through all cases. It may be surprising. Note that this would be a required optimization because if the optimization is not performed it affects what programs can successfully type check. (Naturally it is also observable.)

  2. Some sort of unsafe deref trait that acknowledges possibility of other pointers into the referent. Unappealing because the problem is not that bad as to require unsafety.

  3. Determining conditions (perhaps based on parametricity?) where it is provably safe to call deref. It is dubious and unknown if such conditions exist or what that even means. Rust also does not really enjoy parametricity properties due to presence of reflection and unsafe code.

Unresolved questions

Best implementation strategy.

Summary

Note: This RFC discusses the behavior of rustc, and not any changes to the language.

Change how target specification is done to be more flexible for unexpected usecases. Additionally, add support for the “unknown” OS in target triples, providing a minimum set of target specifications that is valid for bare-metal situations.

Motivation

One of Rust’s important use cases is embedded, OS, or otherwise “bare metal” software. At the moment, we still depend on LLVM’s split-stack prologue for stack safety. In certain situations, it is impossible or undesirable to support what LLVM requires to enable this (on x86, a certain thread-local storage setup). Additionally, porting rustc to a new platform requires modifying the compiler, adding a new OS manually.

Detailed design

A target triple consists of three strings separated by a hyphen, with a possible fourth string at the end preceded by a hyphen. The first is the architecture, the second is the “vendor”, the third is the OS type, and the optional fourth is environment type. In theory, this specifies precisely what platform the generated binary will be able to run on. All of this is determined not by us but by LLVM and other tools. When on bare metal or a similar environment, there essentially is no OS, and to handle this there is the concept of “unknown” in the target triple. When the OS is “unknown”, no runtime environment is assumed to be present (including things such as dynamic linking, threads/thread-local storage, IO, etc).

Rather than listing specific targets for special treatment, introduce a general mechanism for specifying certain characteristics of a target triple. Redesign how targets are handled around this specification, including for the built-in targets. Extend the --target flag to accept a file name of a target specification. A table of the target specification flags and their meaning:

  • data-layout: The LLVM data layout to use. Mostly included for completeness; changing this is unlikely to be used.
  • link-args: Arguments to pass to the linker, unconditionally.
  • cpu: Default CPU to use for the target, overridable with -C target-cpu
  • features: Default target features to enable, augmentable with -C target-features.
  • dynamic-linking-available: Whether the dylib crate type is allowed.
  • split-stacks-supported: Whether there is runtime support that will allow LLVM’s split stack prologue to function as intended.
  • llvm-target: What target to pass to LLVM.
  • relocation-model: What relocation model to use by default.
  • target_endian, target_word_size: Specify the strings used for the corresponding cfg variables.
  • code-model: Code model to pass to LLVM, overridable with -C code-model.
  • no-redzone: Disable use of any stack redzone, overridable with -C no-redzone

Rather than hardcoding a specific set of behaviors per-target, with no recourse for escaping them, the compiler would also use this mechanism when deciding how to build for a given target. The process would look like:

  1. Look up the target triple in an internal map, and load that configuration if it exists. If that fails, check if the target name exists as a file, and try loading that. If the file does not exist, look up <target>.json in the RUST_TARGET_PATH, which is a colon-separated list of directories.
  2. If -C linker is specified, use that instead of the target-specified linker.
  3. If -C link-args is given, add those to the ones specified by the target.
  4. If -C target-cpu is specified, replace the target cpu with it.
  5. If -C target-feature is specified, add those to the ones specified by the target.
  6. If -C relocation-model is specified, replace the target relocation-model with it.
  7. If -C code-model is specified, replace the target code-model with it.
  8. If -C no-redzone is specified, replace the target no-redzone with true.

Then during compilation, this information is used at the proper places rather than matching against an enum listing the OSes we recognize. The target_os, target_family, and target_arch cfg variables would be extracted from the --target passed to rustc.

Drawbacks

More complexity. However, this is very flexible and allows one to use Rust on a new or non-standard target incredibly easy, without having to modify the compiler. rustc is the only compiler I know of that would allow that.

Alternatives

A less holistic approach would be to just allow disabling split stacks on a per-crate basis. Another solution could be adding a family of targets, <arch>-unknown-unknown, which omits all of the above complexity but does not allow extending to new targets easily.

  • Start Date: 2014-03-17
  • RFC PR #: #132
  • Rust Issue #: #16293

Summary

This RFC describes a variety of extensions to allow any method to be used as first-class functions. The same extensions also allow for trait methods without receivers to be invoked in a more natural fashion.

First, at present, the notation path::method() can be used to invoke inherent methods on types. For example, Vec::new() is used to create an instance of a vector. This RFC extends that notion to also cover trait methods, so that something like T::size_of() or T::default() is legal.

Second, currently it is permitted to reference so-called “static methods” from traits using a function-like syntax. For example, one can write Default::default(). This RFC extends that notation so it can be used with any methods, whether or not they are defined with a receiver. (In fact, the distinction between static methods and other methods is completely erased, as per the method lookup of RFC PR #48.)

Third, we introduce an unambiguous if verbose notation that permits one to precisely specify a trait method and its receiver type in one form. Specifically, the notation <T as TraitRef>::item can be used to designate an item item, defined in a trait TraitRef, as implemented by the type T.

Motivation

There are several motivations:

  • There is a need for an unambiguous way to invoke methods. This is typically a fallback for when the more convenient invocation forms fail:
    • For example, when multiple traits are in scope that all define the same method for the same types, there must be a way to disambiguate which method you mean.
    • It is sometimes desirable not to have autoderef:
      • For methods like clone() that apply to almost all types, it is convenient to be more specific about which precise type you want to clone. To get this right with autoderef, one must know the precise rules being used, which is contrary to the “DWIM” intention.
      • For types that implement Deref<T>, UFCS can be used to unambiguously differentiate between methods invoked on the smart pointer itself and methods invoked on its referent.
  • There are many methods, such as SizeOf::size_of(), that return properties of the type alone and do not naturally take any argument that can be used to decide which trait impl you are referring to.
    • This proposal introduces a variety of ways to invoke such methods, varying in the amount of explicit information one includes:
      • T::size_of() – shorthand, but only works if T is a path
      • <T>::size_of() – infers the trait SizeOf based on the traits in scope, just as with a method call
      • <T as SizeOf>::size_of() – completely unambiguous

Detailed design

Path syntax

The syntax of paths is extended as follows:

PATH = ID_SEGMENT { '::' ID_SEGMENT }
     | TYPE_SEGMENT { '::' ID_SEGMENT }
     | ASSOC_SEGMENT '::' ID_SEGMENT { '::' ID_SEGMENT }
ID_SEGMENT   = ID [ '::' '<' { TYPE ',' TYPE } '>' ]
TYPE_SEGMENT = '<' TYPE '>'
ASSOC_SEGMENT = '<' TYPE 'as' TRAIT_REFERENCE '>'

Examples of valid paths. In these examples, capitalized names refer to types (though this doesn’t affect the grammar).

a::b::c
a::<T1,T2>::b::c
T::size_of
<T>::size_of
<T as SizeOf>::size_of
Eq::eq
Eq::<T>::eq
Zero::zero

Normalization of path that reference types

Whenever a path like ...::a::... resolves to a type (but not a trait), it is rewritten (internally) to <...::a>::....

Note that there is a subtle distinction between the following paths:

ToStr::to_str
<ToStr>::to_str

In the former, we are selecting the member to_str from the trait ToStr. The result is a function whose type is basically equivalent to:

fn to_str<Self:ToStr>(self: &Self) -> String

In the latter, we are selecting the member to_str from the type ToStr (i.e., an ToStr object). Resolving type members is different. In this case, it would yield a function roughly equivalent to:

fn to_str(self: &ToStr) -> String

This subtle distinction arises from the fact that we pun on the trait name to indicate both a type and a reference to the trait itself. In this case, depending on which interpretation we choose, the path resolution rules differ slightly.

Paths that begin with a TYPE_SEGMENT

When a path begins with a TYPE_SEGMENT, it is a type-relative path. If this is the complete path (e.g., <int>), then the path resolves to the specified type. If the path continues (e.g., <int>::size_of) then the next segment is resolved using the following procedure. The procedure is intended to mimic method lookup, and hence any changes to method lookup may also change the details of this lookup.

Given a path <T>::m::...:

  1. Search for members of inherent impls defined on T (if any) with the name m. If any are found, the path resolves to that item.
  2. Otherwise, let IN_SCOPE_TRAITS be the set of traits that are in scope and which contain a member named m:
    • Let IMPLEMENTED_TRAITS be those traits from IN_SCOPE_TRAITS for which an implementation exists that (may) apply to T.
      • There can be ambiguity in the case that T contains type inference variables.
    • If IMPLEMENTED_TRAITS is not a singleton set, report an ambiguity error. Otherwise, let TRAIT be the member of IMPLEMENTED_TRAITS.
    • If TRAIT is ambiguously implemented for T, report an ambiguity error and request further type information.
    • Otherwise, rewrite the path to <T as Trait>::m::... and continue.

Paths that begin with an ASSOC_SEGMENT

When a path begins with an ASSOC_SEGMENT, it is a reference to an associated item defined from a trait. Note that such paths must always have a follow-on member m (that is, <T as Trait> is not a complete path, but <T as Trait>::m is).

To resolve the path, first search for an applicable implementation of Trait for T. If no implementation can be found – or the result is ambiguous – then report an error.

Otherwise:

  • Determine the types of output type parameters for Trait from the implementation.
  • If output type parameters were specified in the path, ensure that they are compatible with those specified on the impl.
    • For example, if the path were <int as SomeTrait<uint>>, and the impl is declared as impl SomeTrait<char> for int, then an error would be reported because char and uint are not compatible.
  • Resolve the path to the member of the trait with the substitution composed of the output type parameters from the impl and Self => T.

Alternatives

We have explored a number of syntactic alternatives. This has been selected as being the only one that is simultaneously:

  • Tolerable to look at.
  • Able to convey all necessary information along with auxiliary information the user may want to verify:
    • Self type, type of trait, name of member, type output parameters

Here are some leading candidates that were considered along with their equivalents in the syntax proposed by this RFC. The reasons for their rejection are listed:

module::type::(Trait::member)    <module::type as Trait>::member
--> semantics of parentheses considered too subtle
--> cannot accommodate types that are not paths, like `[int]`

(type: Trait)::member            <type as Trait>::member
--> complicated to parse
--> cannot accommodate types that are not paths, like `[int]`

... (I can't remember all the rest)

One variation that is definitely possible is that we could use the : rather than the keyword as:

<type: Trait>::member            <type as Trait>::member
--> no real objection. `as` was chosen because it mimics the
    syntax for constructing a trait object.

Unresolved questions

Is there a better way to disambiguate a reference to a trait item ToStr::to_str versus a reference to a member of the object type <ToStr>::to_str? I personally do not think so: so long as we pun on the name of the trait, the potential for confusion will remain. Therefore, the only two possibilities I could come up with are to try and change the question:

  • One answer might be that we simply make the second form meaningless by prohibiting inherent impls on object types. But there remains a utility to being able to write something like <ToStr>::is_sized() (where is_sized() is an example of a trait fn that could apply to both sized and unsized types). Moreover, artificially restricting object types just for this reason doesn’t seem right.

  • Another answer is to change the syntax of object types. I have sometimes considered that impl ToStr might be better suited as the object type and then ToStr could be used as syntactic sugar for a type parameter. But there exists a lot of precedent for the current approach and hence I think this is likely a bad idea (not to mention that it’s a drastic change).

  • Start Date: 2014-09-30
  • RFC PR #: https://github.com/rust-lang/rfcs/pull/135
  • Rust Issue #: https://github.com/rust-lang/rust/issues/17657

Summary

Add where clauses, which provide a more expressive means of specifying trait parameter bounds. A where clause comes after a declaration of a generic item (e.g., an impl or struct definition) and specifies a list of bounds that must be proven once precise values are known for the type parameters in question. The existing bounds notation would remain as syntactic sugar for where clauses.

So, for example, the impl for HashMap could be changed from this:

impl<K:Hash+Eq,V> HashMap<K, V>
{
    ..
}

to the following:

impl<K,V> HashMap<K, V>
    where K : Hash + Eq
{
    ..
}

The full grammar can be found in the detailed design.

Motivation

The high-level bit is that the current bounds syntax does not scale to complex cases. Introducing where clauses is a simple extension that gives us a lot more expressive power. In particular, it will allow us to refactor the operator traits to be in a convenient, multidispatch form (e.g., so that user-defined mathematical types can be added to int and vice versa). (It’s also worth pointing out that, once #5527 lands at least, implementing where clauses will be very little work.)

Here is a list of limitations with the current bounds syntax that are overcome with the where syntax:

  • It cannot express bounds on anything other than type parameters. Therefore, if you have a function generic in T, you can write T:MyTrait to declare that T must implement MyTrait, but you can’t write Option<T> : MyTrait or (int, T) : MyTrait. These forms are less commonly required but still important.

  • It does not work well with associated types. This is because there is no space to specify the value of an associated type. Other languages use where clauses (or something analogous) for this purpose.

  • It’s just plain hard to read. Experience has shown that as the number of bounds grows, the current syntax becomes hard to read and format.

Let’s examine each case in detail.

Bounds are insufficiently expressive

Currently bounds can only be declared on type parameters. But there are situations where one wants to declare bounds not on the type parameter itself but rather a type that includes the type parameter.

Partially generic types

One situation where this occurs is when you want to write functions where types are partially known and have those interact with other functions that are fully generic. To explain the situation, let’s examine some code adapted from rustc.

Imagine I have a table parameterized by a value type V and a key type K. There are also two traits, Value and Key, that describe the keys and values. Also, each type of key is linked to a specific value:

struct Table<V:Value, K:Key<V>> { ... }
trait Key<V:Value> { ... }
trait Value { ... }

Now, imagine I want to write some code that operates over all keys whose value is an Option<T> for some T:

fn example<T,K:Key<Option<T>>>(table: &Table<Option<T>, K>) { ... }

This seems reasonable, but this code will not compile. The problem is that the compiler needs to know that the value type implements Value, but here the value type is Option<T>. So we’d need to declare Option<T> : Value, which we cannot do.

There are workarounds. I might write a new trait OptionalValue:

trait OptionalValue<T> {
    fn as_option<'a>(&'a self) -> &'a Option<T>; // identity fn
}

and then I could write my example as:

fn example<T,O:OptionalValue<T>,K:Key<O>>(table: &Table<O, K>) { ... }

But this is making my example function, already a bit complicated, become quite obscure.

Multidispatch traits

Another situation where a similar problem is encountered is multidispatch traits (aka, multiparameter type classes in Haskell). The idea of a multidispatch trait is to be able to choose the impl based not just on one type, as is the most common case, but on multiple types (usually, but not always, two).

Multidispatch is rarely needed because the vast majority of traits are characterized by a single type. But when you need it, you really need it. One example that arises in the standard library is the traits for binary operators like +. Today, the Add trait is defined using only single-dispatch (like so):

pub trait Add<Rhs,Sum> {
    fn add(&self, rhs: &Rhs) -> Sum;
}

The expression a + b is thus sugar for Add::add(&a, &b). Because of how our trait system works, this means that only the type of the left-hand side (the Self parameter) will be used to select the impl. The type for the right-hand side (Rhs) along with the type of their sum (Sum) are defined as trait parameters, which are always outputs of the trait matching: that is, they are specified by the impl and are not used to select which impl is used.

This setup means that addition is not as extensible as we would like. For example, the standard library includes implementations of this trait for integers and other built-in types:

impl Add<int,int> for int { ... }
impl Add<f32,f32> for f32 { ... }

The limitations of this setup become apparent when we consider how a hypothetical user library might integrate. Imagine a library L that defines a type Complex representing complex numbers:

struct Complex { ... }

Naturally, it should be possible to add complex numbers and integers. Since complex number addition is commutative, it should be possible to write both 1 + c and c + 1. Thus one might try the following impls:

impl Add<int,Complex> for Complex { ... }     // 1. Complex + int
impl Add<Complex,Complex> for int { ... }     // 2. int + Complex
impl Add<Complex,Complex> for Complex { ... } // 3. Complex + Complex

Due to the coherence rules, however, this setup will not work. There are in fact three errors. The first is that there are two impls of Add defined for Complex (1 and 3). The second is that there are two impls of Add defined for int (the one from the standard library and 2). The final error is that impl 2 violates the orphan rule, since the type int is not defined in the current crate.

This is not a new problem. Object-oriented languages, with their focus on single dispatch, have long had trouble dealing with binary operators. One common solution is double dispatch, an awkward but effective pattern in which no type ever implements Add directly. Instead, we introduce “indirection” traits so that, e.g., int is addable to anything that implements AddToInt and so on. This is not my preferred solution so I will not describe it in detail, but rather refer readers to this blog post where I describe how it works.

An alternative to double dispatch is to define Add on tuple types (LHS, RHS) rather than on a single value. Imagine that the Add trait were defined as follows:

trait Add<Sum> {
    fn add(self) -> Sum;
}

impl Add<int> for (int, int) {
    fn add(self) -> int {
        let (x, y) = self;
        x + y
    }
}

Now the expression a + b would be sugar for Add::add((a, b)). This small change has several interesting ramifications. For one thing, the library L can easily extend Add to cover complex numbers:

impl Add<Complex> for (Complex, int)     { ... }
impl Add<Complex> for (int, Complex)     { ... }
impl Add<Complex> for (Complex, Complex) { ... }

These impls do not violate the coherence rules because they are all applied to distinct types. Moreover, none of them violate the orphan rule because each of them is a tuple involving at least one type local to the library.

One downside of this Add pattern is that there is no way within the trait definition to refer to the type of the left- or right-hand side individually; we can only use the type Self to refer to the tuple of both types. In the Discussion section below, I will introduce an extended “multi-dispatch” pattern that addresses this particular problem.

There is however another problem that where clauses help to address. Imagine that we wish to define a function to increment complex numbers:

fn increment(c: Complex) -> Complex {
    1 + c
}

This function is pretty generic, so perhaps we would like to generalize it to work over anything that can be added to an int. We’ll use our new version of the Add trait that is implemented over tuples:

fn increment<T:...>(c: T) -> T {
    1 + c
}

At this point we encounter the problem. What bound should we give for T? We’d like to write something like (int, T) : Add<T> – that is, Add is implemented for the tuple (int, T) with the sum type T. But we can’t write that, because the current bounds syntax is too limited.

Where clauses give us an answer. We can write a generic version of increment like so:

fn increment<T>(c: T) -> T
    where (int, T) : Add<T>
{
    1 + c
}

Associated types

It is unclear exactly what form associated types will have in Rust, but it is well documented that our current design, in which type parameters decorate traits, does not scale particularly well. (For curious readers, there are several blog posts exploring the design space of associated types with respect to Rust in particular.)

The high-level summary of associated types is that we can replace a generic trait like Iterator:

trait Iterator<E> {
    fn next(&mut self) -> Option<E>;
}

With a version where the type parameter is a “member” of the Iterator trait:

trait Iterator {
    type E;
    
    fn next(&mut self) -> Option<E>;
}

This syntactic change helps to highlight that, for any given type, the type E is fixed by the impl, and hence it can be considered a member (or output) of the trait. It also scales better as the number of associated types grows.

One challenge with this design is that it is not clear how to convert a function like the following:

fn sum<I:Iterator<int>>(i: I) -> int {
    ...    
}

With associated types, the reference Iterator<int> is no longer valid, since the trait Iterator doesn’t have type parameters.

The usual solution to this problem is to employ a where clause:

fn sum<I:Iterator>(i: I) -> int
  where I::E == int
{
    ...    
}

We can also employ where clauses with object types via a syntax like &Iterator<where E=int> (admittedly somewhat wordy)

Readability

When writing very generic code, it is common to have a large number of parameters with a large number of bounds. Here is some example function extracted from rustc:

fn set_var_to_merged_bounds<T:Clone + InferStr + LatticeValue,
                            V:Clone+Eq+ToStr+Vid+UnifyVid<Bounds<T>>>(
                            &self,
                            v_id: V,
                            a: &Bounds<T>,
                            b: &Bounds<T>,
                            rank: uint)
                            -> ures;

Definitions like this are very difficult to read (it’s hard to even know how to format such a definition).

Using a where clause allows the bounds to be separated from the list of type parameters:

fn set_var_to_merged_bounds<T,V>(&self,
                                 v_id: V,
                                 a: &Bounds<T>,
                                 b: &Bounds<T>,
                                 rank: uint)
                                 -> ures
    where T:Clone,         // it is legal to use individual clauses...
          T:InferStr,
          T:LatticeValue,
          V:Clone+Eq+ToStr+Vid+UnifyVid<Bounds<T>>, // ...or use `+`
{                                     
    ..
}

This helps to separate out the function signature from the extra requirements that the function places on its types.

If I may step aside from the “impersonal voice” of the RFC for a moment, I personally find that when writing generic code it is helpful to focus on the types and signatures, and come to the bounds later. Where clauses help to separate these distinctions. Naturally, your mileage may vary. - nmatsakis

Detailed design

Where can where clauses appear?

Where clauses can be added to anything that can be parameterized with type/lifetime parameters with the exception of trait method definitions: impl declarations, fn declarations, and trait and struct definitions. They appear as follows:

impl Foo<A,B>
    where ...
{ }

impl Foo<A,B> for C
    where ...
{ }

impl Foo<A,B> for C
{
    fn foo<A,B> -> C
        where ...
    { }
}

fn foo<A,B> -> C
    where ...
{ }

struct Foo<A,B>
    where ...
{ }

trait Foo<A,B> : C
    where ...
{ }

Where clauses cannot (yet) appear on trait methods

Note that trait method definitions were specifically excluded from the list above. The reason is that including where clauses on a trait method raises interesting questions for what it means to implement the trait. Using where clauses it becomes possible to define methods that do not necessarily apply to all implementations. We intend to enable this feature but it merits a second RFC to delve into the details.

Where clause grammar

The grammar for a where clause would be as follows (BNF):

WHERE = 'where' BOUND { ',' BOUND } [,]
BOUND = TYPE ':' TRAIT { '+' TRAIT } [+]
TRAIT = Id [ '<' [ TYPE { ',' TYPE } [,] ] '>' ]
TYPE  = ... (same type grammar as today)

Semantics

The meaning of a where clause is fairly straightforward. Each bound in the where clause must be proven by the caller after substitution of the parameter types.

One interesting case concerns trivial where clauses where the self-type does not refer to any of the type parameters, such as the following:

fn foo()
    where int : Eq
{ ... }

Where clauses like these are considered an error. They have no particular meaning, since the callee knows all types involved. This is a conservative choice: if we find that we do desire a particular interpretation for them, we can always make them legal later.

Drawbacks

This RFC introduces two ways to declare a bound.

Alternatives

Remove the existing trait bounds. I decided against this both to avoid breaking lots of existing code and because the existing syntax is convenient much of the time.

Embed where clauses in the type parameter list. One alternative syntax that was proposed is to embed a where-like clause in the type parameter list. Thus the increment() example

fn increment<T>(c: T) -> T
    where () : Add<int,T,T>
{
    1 + c
}

would become something like:

fn increment<T, ():Add<int,T,T>>(c: T) -> T
{
    1 + c
}

This is unfortunately somewhat ambiguous, since a bound like T:Eq could either be declared a type parameter T or as a condition that the (existing) type T implement Eq.

Use a colon instead of the keyword. There is some precedent for this from the type state days. Unfortunately, it doesn’t work with traits due to the supertrait list, and it also doesn’t look good with the use of : as a trait-bound separator:

fn increment<T>(c: T) -> T
    : () : Add<int,T,T>
{
    1 + c
}
  • Start Date: 2014-06-24
  • RFC PR #: #136
  • Rust Issue #: #16463

Summary

Require a feature gate to expose private items in public APIs, until we grow the appropriate language features to be able to remove the feature gate and forbid it entirely.

Motivation

Privacy is central to guaranteeing the invariants necessary to write correct code that employs unsafe blocks. Although the current language rules prevent a private item from being directly named from outside the current module, they still permit direct access to private items in some cases. For example, a public function might return a value of private type. A caller from outside the module could then invoke this function and, thanks to type inference, gain access to the private type (though they still could not invoke public methods or access public fields). This access could undermine the reasoning of the author of the module. Fortunately, it is not hard to prevent.

Detailed design

Overview

The general idea is that:

  • If an item is declared as public, items referred to in the public-facing parts of that item (e.g. its type) must themselves be declared as public.

Details follow.

The rules

These rules apply as long as the feature gate is not enabled. After the feature gate has been removed, they will apply always.

When is an item “public”?

Items that are explicitly declared as pub are always public. In addition, items in the impl of a trait (not an inherent impl) are considered public if all of the following conditions are met:

  • The trait being implemented is public.
  • All input types (currently, the self type) of the impl are public.
  • Motivation: If any of the input types or the trait is public, it should be impossible for an outside to access the items defined in the impl. They cannot name the types nor they can get direct access to a value of those types.

What restrictions apply to public items?

The rules for various kinds of public items are as follows:

  • If it is a static declaration, items referred to in its type must be public.

  • If it is an fn declaration, items referred to in its trait bounds, argument types, and return type must be public.

  • If it is a struct or enum declaration, items referred to in its trait bounds and in the types of its pub fields must be public.

  • If it is a type declaration, items referred to in its definition must be public.

  • If it is a trait declaration, items referred to in its super-traits, in the trait bounds of its type parameters, and in the signatures of its methods (see fn case above) must be public.

Examples

Here are some examples to demonstrate the rules.

Struct fields

// A private struct may refer to any type in any field.
struct Priv {
    a: Priv,
    b: Pub,
    pub c: Priv
}

enum Vapor<A> { X, Y, Z } // Note that A is not used

// Public fields of a public struct may only refer to public types.
pub struct Item {
    // Private field may reference a private type.
    a: Priv,
    
    // Public field must refer to a public type.
    pub b: Pub,

    // ERROR: Public field refers to a private type.
    pub c: Priv,
    
    // ERROR: Public field refers to a private type.
    // For the purposes of this test, we do not descend into the type,
    // but merely consider the names that appear in type parameters
    // on the type, regardless of usage (or lack thereof) within the type
    // definition itself.
    pub d: Vapor<Priv>,
}

pub struct Pub { ... }

Methods

struct Priv { .. }
pub struct Pub { .. }
pub struct Foo { .. }

impl Foo {
    // Illegal: public method with argument of private type.
    pub fn foo(&self, p: Priv) { .. }
}

Trait bounds

trait PrivTrait { ... }

// Error: type parameter on public item bounded by a private trait.
pub struct Foo<X: PrivTrait> { ... }

// OK: type parameter on private item.
struct Foo<X: PrivTrait> { ... }

Trait definitions

struct PrivStruct { ... }

pub trait PubTrait {
    // Error: private struct referenced from method in public trait
    fn method(x: PrivStruct) { ... }
}

trait PrivTrait {
    // OK: private struct referenced from method in private trait 
    fn method(x: PrivStruct) { ... }
}

Implementations

To some extent, implementations are prevented from exposing private types because their types must match the trait. However, that is not true with generics.

pub trait PubTrait<T> {
    fn method(t: T);
}

struct PubStruct { ... }

struct PrivStruct { ... }

impl PubTrait<PrivStruct> for PubStruct {
           // ^~~~~~~~~~ Error: Private type referenced from impl of
           //            public trait on a public type. [Note: this is
           //            an "associated type" here, not an input.]

    fn method(t: PrivStruct) {
              // ^~~~~~~~~~ Error: Private type in method signature.
              //
              // Implementation note. It may not be a good idea to report
              // an error here; I think private types can only appear in
              // an impl by having an associated type bound to a private
              // type.
    }
}

Type aliases

Note that the path to the public item does not have to be private.

mod impl {
    pub struct Foo { ... }
}
pub type Bar = self::impl::Foo;

Negative examples

The following examples should fail to compile under these rules.

Non-public items referenced by a pub use

These examples are illegal because they use a pub use to re-export a private item:

struct Item { ... }
pub mod module {
    // Error: Item is not declared as public, but is referenced from
    // a `pub use`.
    pub use Item;
}
struct Foo { ... }
// Error: Non-public item referenced by `pub use`.
pub use Item = Foo;

If it was desired to have a private name that is publicly “renamed” using a pub use, that can be achieved using a module:

mod impl {
    pub struct ItemPriv;
}
pub use Item = self::impl::ItemPriv;

Drawbacks

Adds a (temporary) feature gate.

Requires some existing code to opt-in to the feature gate before transitioning to a more explicit alternative.

Requires effort to implement.

Alternatives

If we stick with the status quo, we’ll have to resolve several bizarre questions and keep supporting its behavior indefinitely after 1.0.

Instead of a feature gate, we could just ban these things outright right away, at the cost of temporarily losing some convenience and a small amount of expressiveness before the more principled replacement features are implemented.

We could make an exception for private supertraits, as these are not quite as problematic as the other cases. However, especially given that a more principled alternative is known (private methods), I would rather not make any exceptions.

The original design of this RFC had a stronger notion of “public” which also considered whether a public path existed to the item. In other words, a module X could not refer to a public item Y from a submodule Z, unless X also exposed a public path to Y (whether that be because Z was public, or via a pub use). This definition strengthened the basic guarantee of “private things are only directly accessible from within the current module” to include the idea that public functions in outer modules cannot accidentally refer to public items from inner modules unless there is a public path from the outer to the inner module. Unfortunately, these rules were complex to state concisely and also hard to understand in practice; when an error occurred under these rules, it was very hard to evaluate whether the error was legitimate. The newer rules are simpler while still retaining the basic privacy guarantee.

One important advantage of the earlier approach, and a scenario not directly addressed in this RFC, is that there may be items which are declared as public by an inner module but still not intended to be exposed to the world at large (in other words, the items are only expected to be used within some subtree). A special case of this is crate-local data. In the older rules, the “intended scope” of privacy could be somewhat inferred from the existence (or non-existence) of pub use declarations. However, in the author’s opinion, this scenario would be best addressed by making pub declarations more expressive so that the intended scope can be stated directly.

Summary

Remove the coercion from Box<T> to &T from the language.

Motivation

The coercion between Box<T> to &T is not replicable by user-defined smart pointers and has been found to be rarely used 1. We already removed the coercion between Box<T> and &mut T in RFC 33.

Detailed design

The coercion between Box<T> and &T should be removed.

Note that methods that take &self can still be called on values of type Box<T> without any special referencing or dereferencing. That is because the semantics of auto-deref and auto-ref conspire to make it work: the types unify after one autoderef followed by one autoref.

Drawbacks

Borrowing from Box<T> to &T may be convenient.

Alternatives

The impact of not doing this is that the coercion will remain.

Unresolved questions

None.

Summary

This RFC proposes to

  1. Expand the rules for eliding lifetimes in fn definitions, and
  2. Follow the same rules in impl headers.

By doing so, we can avoid writing lifetime annotations ~87% of the time that they are currently required, based on a survey of the standard library.

Motivation

In today’s Rust, lifetime annotations make code more verbose, both for methods

fn get_mut<'a>(&'a mut self) -> &'a mut T

and for impl blocks:

impl<'a> Reader for BufReader<'a> { ... }

In the vast majority of cases, however, the lifetimes follow a very simple pattern.

By codifying this pattern into simple rules for filling in elided lifetimes, we can avoid writing any lifetimes in ~87% of the cases where they are currently required.

Doing so is a clear ergonomic win.

Detailed design

Today’s lifetime elision rules

Rust currently supports eliding lifetimes in functions, so that you can write

fn print(s: &str);
fn get_str() -> &str;

instead of

fn print<'a>(s: &'a str);
fn get_str<'a>() -> &'a str;

The elision rules work well for functions that consume references, but not for functions that produce them. The get_str signature above, for example, promises to produce a string slice that lives arbitrarily long, and is either incorrect or should be replaced by

fn get_str() -> &'static str;

Returning 'static is relatively rare, and it has been proposed to make leaving off the lifetime in output position an error for this reason.

Moreover, lifetimes cannot be elided in impl headers.

The proposed rules

Overview

This RFC proposes two changes to the lifetime elision rules:

  1. Since eliding a lifetime in output position is usually wrong or undesirable under today’s elision rules, interpret it in a different and more useful way.

  2. Interpret elided lifetimes for impl headers analogously to fn definitions.

Lifetime positions

A lifetime position is anywhere you can write a lifetime in a type:

&'a T
&'a mut T
T<'a>

As with today’s Rust, the proposed elision rules do not distinguish between different lifetime positions. For example, both &str and Ref<uint> have elided a single lifetime.

Lifetime positions can appear as either “input” or “output”:

  • For fn definitions, input refers to the types of the formal arguments in the fn definition, while output refers to result types. So fn foo(s: &str) -> (&str, &str) has elided one lifetime in input position and two lifetimes in output position. Note that the input positions of a fn method definition do not include the lifetimes that occur in the method’s impl header (nor lifetimes that occur in the trait header, for a default method).

  • For impl headers, input refers to the lifetimes appears in the type receiving the impl, while output refers to the trait, if any. So impl<'a> Foo<'a> has 'a in input position, while impl<'a, 'b, 'c> SomeTrait<'b, 'c> for Foo<'a, 'c> has 'a in input position, 'b in output position, and 'c in both input and output positions.

The rules

  • Each elided lifetime in input position becomes a distinct lifetime parameter. This is the current behavior for fn definitions.

  • If there is exactly one input lifetime position (elided or not), that lifetime is assigned to all elided output lifetimes.

  • If there are multiple input lifetime positions, but one of them is &self or &mut self, the lifetime of self is assigned to all elided output lifetimes.

  • Otherwise, it is an error to elide an output lifetime.

Notice that the actual signature of a fn or impl is based on the expansion rules above; the elided form is just a shorthand.

Examples

fn print(s: &str);                                      // elided
fn print<'a>(s: &'a str);                               // expanded

fn debug(lvl: uint, s: &str);                           // elided
fn debug<'a>(lvl: uint, s: &'a str);                    // expanded

fn substr(s: &str, until: uint) -> &str;                // elided
fn substr<'a>(s: &'a str, until: uint) -> &'a str;      // expanded

fn get_str() -> &str;                                   // ILLEGAL

fn frob(s: &str, t: &str) -> &str;                      // ILLEGAL

fn get_mut(&mut self) -> &mut T;                        // elided
fn get_mut<'a>(&'a mut self) -> &'a mut T;              // expanded

fn args<T:ToCStr>(&mut self, args: &[T]) -> &mut Command                  // elided
fn args<'a, 'b, T:ToCStr>(&'a mut self, args: &'b [T]) -> &'a mut Command // expanded

fn new(buf: &mut [u8]) -> BufWriter;                    // elided
fn new<'a>(buf: &'a mut [u8]) -> BufWriter<'a>          // expanded

impl Reader for BufReader { ... }                       // elided
impl<'a> Reader for BufReader<'a> { .. }                // expanded

impl Reader for (&str, &str) { ... }                    // elided
impl<'a, 'b> Reader for (&'a str, &'b str) { ... }      // expanded

impl StrSlice for &str { ... }                          // elided
impl<'a> StrSlice<'a> for &'a str { ... }               // expanded

trait Bar<'a> { fn bound(&'a self) -> &int { ... }    fn fresh(&self) -> &int { ... } }           // elided
trait Bar<'a> { fn bound(&'a self) -> &'a int { ... } fn fresh<'b>(&'b self) -> &'b int { ... } } // expanded

impl<'a> Bar<'a> for &'a str {
  fn bound(&'a self) -> &'a int { ... } fn fresh(&self) -> &int { ... }              // elided
}
impl<'a> Bar<'a> for &'a str {
  fn bound(&'a self) -> &'a int { ... } fn fresh<'b>(&'b self) -> &'b int { ... }    // expanded
}

// Note that when the impl reuses the same signature (with the same elisions)
// from the trait definition, the expanded forms will also match, and thus
// the `impl` will be compatible with the `trait`.

impl Bar for &str            { fn bound(&self) -> &int { ... } }           // elided
impl<'a> Bar<'a> for &'a str { fn bound<'b>(&'b self) -> &'b int { ... } } // expanded

// Note that the preceding example's expanded methods do not match the
// signatures from the above trait definition for `Bar`; in the general
// case, if the elided signatures between the `impl` and the `trait` do
// not match, an expanded `impl` may not be compatible with the given
// `trait` (and thus would not compile).

impl Bar for &str            { fn fresh(&self) -> &int { ... } }           // elided
impl<'a> Bar<'a> for &'a str { fn fresh<'b>(&'b self) -> &'b int { ... } } // expanded

impl Bar for &str {
  fn bound(&'a self) -> &'a int { ... } fn fresh(&self) -> &int { ... }    // ILLEGAL: unbound 'a
}

Error messages

Since the shorthand described above should eliminate most uses of explicit lifetimes, there is a potential “cliff”. When a programmer first encounters a situation that requires explicit annotations, it is important that the compiler gently guide them toward the concept of lifetimes.

An error can arise with the above shorthand only when the program elides an output lifetime and neither of the rules can determine how to annotate it.

For fn

The error message should guide the programmer toward the concept of lifetime by talking about borrowed values:

This function’s return type contains a borrowed value, but the signature does not say which parameter it is borrowed from. It could be one of a, b, or c. Mark the input parameter it borrows from using lifetimes, e.g. [generated example]. See [url] for an introduction to lifetimes.

This message is slightly inaccurate, since the presence of a lifetime parameter does not necessarily imply the presence of a borrowed value, but there are no known use-cases of phantom lifetime parameters.

For impl

The error case on impl is exceedingly rare: it requires (1) that the impl is for a trait with a lifetime argument, which is uncommon, and (2) that the Self type has multiple lifetime arguments.

Since there are no clear “borrowed values” for an impl, this error message speaks directly in terms of lifetimes. This choice seems warranted given that a programmer implementing a trait with lifetime parameters will almost certainly already understand lifetimes.

TraitName requires lifetime arguments, and the impl does not say which lifetime parameters of TypeName to use. Mark the parameters explicitly, e.g. [generated example]. See [url] for an introduction to lifetimes.

The impact

To assess the value of the proposed rules, we conducted a survey of the code defined in libstd (as opposed to the code it reexports). This corpus is large and central enough to be representative, but small enough to easily analyze.

We found that of the 169 lifetimes that currently require annotation for libstd, 147 would be elidable under the new rules, or 87%.

Note: this percentage does not include the large number of lifetimes that are already elided with today’s rules.

The detailed data is available at: https://gist.github.com/aturon/da49a6d00099fdb0e861

Drawbacks

Learning lifetimes

The main drawback of this change is pedagogical. If lifetime annotations are rarely used, newcomers may encounter error messages about lifetimes long before encountering lifetimes in signatures, which may be confusing. Counterpoints:

  • This is already the case, to some extent, with the current elision rules.

  • Most existing error messages are geared to talk about specific borrows not living long enough, pinpointing their locations in the source, rather than talking in terms of lifetime annotations. When the errors do mention annotations, it is usually to suggest specific ones.

  • The proposed error messages above will help programmers transition out of the fully elided regime when they first encounter a signature requiring it.

  • When combined with a good tutorial on the borrow/lifetime system (which should be introduced early in the documentation), the above should provide a reasonably gentle path toward using and understanding explicit lifetimes.

Programmers learn lifetimes once, but will use them many times. Better to favor long-term ergonomics, if a simple elision rule can cover 87% of current lifetime uses (let alone the currently elided cases).

Subtlety for non-& types

While the rules are quite simple and regular, they can be subtle when applied to types with lifetime positions. To determine whether the signature

fn foo(r: Bar) -> Bar

is actually using lifetimes via the elision rules, you have to know whether Bar has a lifetime parameter. But this subtlety already exists with the current elision rules. The benefit is that library types like Ref<'a, T> get the same status and ergonomics as built-ins like &'a T.

Alternatives

  • Do not include output lifetime elision for impl. Since traits with lifetime parameters are quite rare, this would not be a great loss, and would simplify the rules somewhat.

  • Only add elision rules for fn, in keeping with current practice.

  • Only add elision for explicit & pointers, eliminating one of the drawbacks mentioned above. Doing so would impose an ergonomic penalty on abstractions, though: Ref would be more painful to use than &.

Unresolved questions

The fn and impl cases tackled above offer the biggest bang for the buck for lifetime elision. But we may eventually want to consider other opportunities.

Double lifetimes

Another pattern that sometimes arises is types like &'a Foo<'a>. We could consider an additional elision rule that expands &Foo to &'a Foo<'a>.

However, such a rule could be easily added later, and it is unclear how common the pattern is, so it seems best to leave that for a later RFC.

Lifetime elision in structs

We may want to allow lifetime elision in structs, but the cost/benefit analysis is much less clear. In particular, it could require chasing an arbitrary number of (potentially private) struct fields to discover the source of a lifetime parameter for a struct. There are also some good reasons to treat elided lifetimes in structs as 'static.

Again, since shorthand can be added backwards-compatibly, it seems best to wait.

Summary

Closures should capture their upvars by value unless the ref keyword is used.

Motivation

For unboxed closures, we will need to syntactically distinguish between captures by value and captures by reference.

Detailed design

This is a small part of #114, split off to separate it from the rest of the discussion going on in that RFC.

Closures should capture their upvars (closed-over variables) by value unless the ref keyword precedes the opening | of the argument list. Thus |x| x + 2 will capture x by value (and thus, if x is not Copy, it will move x into the closure), but ref |x| x + 2 will capture x by reference.

In an unboxed-closures world, the immutability/mutability of the borrow (as the case may be) is inferred from the type of the closure: Fn captures by immutable reference, while FnMut captures by mutable reference. In a boxed-closures world, the borrows are always mutable.

Drawbacks

It may be that ref is unwanted complexity; it only changes the semantics of 10%-20% of closures, after all. This does not add any core functionality to the language, as a reference can always be made explicitly and then captured. However, there are a lot of closures, and the workaround to capture a reference by value is painful.

Alternatives

As above, the impact of not doing this is that reference semantics would have to be achieved. However, the diff against current Rust was thousands of lines of pretty ugly code.

Another alternative would be to annotate each individual upvar with its capture semantics, like capture clauses in C++11. This proposal does not preclude adding that functionality should it be deemed useful in the future. Note that C++11 provides a syntax for capturing all upvars by reference, exactly as this proposal does.

Unresolved questions

None.

Summary

Require “anonymous traits”, i.e. impl MyStruct to occur only in the same module that MyStruct is defined.

Motivation

Before I can explain the motivation for this, I should provide some background as to how anonymous traits are implemented, and the sorts of bugs we see with the current behaviour. The conclusion will be that we effectively already only support impl MyStruct in the same module that MyStruct is defined, and making this a rule will simply give cleaner error messages.

  • The compiler first sees impl MyStruct during the resolve phase, specifically in Resolver::build_reduced_graph(), called by Resolver::resolve() in src/librustc/middle/resolve.rs. This is before any type checking (or type resolution, for that matter) is done, so the compiler trusts for now that MyStruct is a valid type.
  • If MyStruct is a path with more than one segment, such as mymod::MyStruct, it is silently ignored (how was this not flagged when the code was written??), which effectively causes static methods in such impls to be dropped on the floor. A silver lining here is that nothing is added to the current module namespace, so the shadowing bugs demonstrated in the next bullet point do not apply here. (To locate this bug in the code, find the match immediately following the FIXME (#3785) comment in resolve.rs.) This leads to the following
mod break1 {
    pub struct MyGuy;

    impl MyGuy {
        pub fn do1() { println!("do 1"); }
    }
}

impl break1::MyGuy {
    fn do2() { println!("do 2"); }
}

fn main() {
    break1::MyGuy::do1();
    break1::MyGuy::do2();
}
<anon>:15:5: 15:23 error: unresolved name `break1::MyGuy::do2`.
<anon>:15     break1::MyGuy::do2();

as noticed by @huonw in https://github.com/rust-lang/rust/issues/15060 .

  • If one does not exist, the compiler creates a submodule MyStruct of the current module, with kind ImplModuleKind. Static methods are placed into this module. If such a module already exists, the methods are appended to it, to support multiple impl MyStruct blocks within the same module. If a module exists that is not ImplModuleKind, the compiler signals a duplicate module definition error.
  • Notice at this point that if there is a use MyStruct, the compiler will act as though it is unaware of this. This is because imports are not resolved yet (they are in Resolver::resolve_imports() called immediately after Resolver::build_reduced_graph() is called). In the final resolution step, MyStruct will be searched in the namespace of the current module, checking imports only as a fallback (and only in some contexts), so the use MyStruct is effectively shadowed. If there is an impl MyStruct in the file being imported from, the user expects that the new impl MyStruct will append to that one, same as if they are in the original file. This leads to the original bug report https://github.com/rust-lang/rust/issues/15060 .
  • In fact, even if no methods from the import are used, the name MyStruct will not be associated to a type, so that
trait T {}
impl<U: T> Vec<U> {
    fn from_slice<'a>(x: &'a [uint]) -> Vec<uint> {
        fail!()
    }
}
fn main() { let r = Vec::from_slice(&[1u]); }
error: found module name used as a type: impl Vec<U>::Vec<U> (id=5)
impl<U: T> Vec<U>

which @Ryman noticed in https://github.com/rust-lang/rust/issues/15060 . The reason for this is that in Resolver::resolve_crate(), the final step of Resolver::resolve(), the type of an anonymous impl is determined by NameBindings::def_for_namespace(TypeNS). This function searches the namespace TypeNS (which is not affected by imports) for a type; failing that it tries for a module; failing that it returns None. The result is that when typeck runs, it sees impl [module name] instead of impl [type name].

The main motivation of this RFC is to clear out these bugs, which do not make sense to a user of the language (and had me confused for quite a while).

A secondary motivation is to enforce consistency in code layout; anonymous traits are used the way that class methods are used in other languages, and the data and methods of a struct should be defined nearby.

Detailed design

I propose three changes to the language:

  • impl on multiple-ident paths such as impl mymod::MyStruct is disallowed. Since this currently surprises the user by having absolutely no effect for static methods, support for this is already broken.
  • impl MyStruct must occur in the same module that MyStruct is defined. This is to prevent the above problems with impl-across-modules. Migration path is for users to just move code between source files.

Drawbacks

Static methods on impls-away-from-definition never worked, while non-static methods can be implemented using non-anonymous traits. So there is no loss in expressivity. However, using a trait where before there was none may be clumsy, since it might not have a sensible name, and it must be explicitly imported by all users of the trait methods.

For example, in the stdlib src/libstd/io/fs.rs we see the code impl path::Path to attach (non-static) filesystem-related methods to the Path type. This would have to be done via a FsPath trait which is implemented on Path and exported alongside Path in the prelude.

It is worth noting that this is the only instance of this RFC conflicting with current usage in the stdlib or compiler.

Alternatives

  • Leaving this alone and fixing the bugs directly. This is really hard. To do it properly, we would need to seriously refactor resolve.

Unresolved questions

None.

Summary

Introduce a new if let PAT = EXPR { BODY } construct. This allows for refutable pattern matching without the syntactic and semantic overhead of a full match, and without the corresponding extra rightward drift. Informally this is known as an “if-let statement”.

Motivation

Many times in the past, people have proposed various mechanisms for doing a refutable let-binding. None of them went anywhere, largely because the syntax wasn’t great, or because the suggestion introduced runtime failure if the pattern match failed.

This proposal ties the refutable pattern match to the pre-existing conditional construct (i.e. if statement), which provides a clear and intuitive explanation for why refutable patterns are allowed here (as opposed to a let statement which disallows them) and how to behave if the pattern doesn’t match.

The motivation for having any construct at all for this is to simplify the cases that today call for a match statement with a single non-trivial case. This is predominately used for unwrapping Option<T> values, but can be used elsewhere.

The idiomatic solution today for testing and unwrapping an Option<T> looks like

match optVal {
    Some(x) => {
        doSomethingWith(x);
    }
    None => {}
}

This is unnecessarily verbose, with the None => {} (or _ => {}) case being required, and introduces unnecessary rightward drift (this introduces two levels of indentation where a normal conditional would introduce one).

The alternative approach looks like this:

if optVal.is_some() {
    let x = optVal.unwrap();
    doSomethingWith(x);
}

This is generally considered to be a less idiomatic solution than the match. It has the benefit of fixing rightward drift, but it ends up testing the value twice (which should be optimized away, but semantically speaking still happens), with the second test being a method that potentially introduces failure. From context, the failure won’t happen, but it still imposes a semantic burden on the reader. Finally, it requires having a pre-existing let-binding for the optional value; if the value is a temporary, then a new let-binding in the parent scope is required in order to be able to test and unwrap in two separate expressions.

The if let construct solves all of these problems, and looks like this:

if let Some(x) = optVal {
    doSomethingWith(x);
}

Detailed design

The if let construct is based on the precedent set by Swift, which introduced its own if let statement. In Swift, if let var = expr { ... } is directly tied to the notion of optional values, and unwraps the optional value that expr evaluates to. In this proposal, the equivalent is if let Some(var) = expr { ... }.

Given the following rough grammar for an if condition:

if-expr     = 'if' if-cond block else-clause?
if-cond     = expression
else-clause = 'else' block | 'else' if-expr

The grammar is modified to add the following productions:

if-cond = 'let' pattern '=' expression

The expression is restricted to disallow a trailing braced block (e.g. for struct literals) the same way the expression in the normal if statement is, to avoid ambiguity with the then-block.

Contrary to a let statement, the pattern in the if let expression allows refutable patterns. The compiler should emit a warning for an if let expression with an irrefutable pattern, with the suggestion that this should be turned into a regular let statement.

Like the for loop before it, this construct can be transformed in a syntax-lowering pass into the equivalent match statement. The expression is given to match and the pattern becomes a match arm. If there is an else block, that becomes the body of the _ => {} arm, otherwise _ => {} is provided.

Optionally, one or more else if (not else if let) blocks can be placed in the same match using pattern guards on _. This could be done to simplify the code when pretty-printing the expansion result. Otherwise, this is an unnecessary transformation.

Due to some uncertainty regarding potentially-surprising fallout of AST rewrites, and some worries about exhaustiveness-checking (e.g. a tautological if let would be an error, which may be unexpected), this is put behind a feature gate named if_let.

Examples

Source:

if let Some(x) = foo() {
    doSomethingWith(x)
}

Result:

match foo() {
    Some(x) => {
        doSomethingWith(x)
    }
    _ => {}
}

Source:

if let Some(x) = foo() {
    doSomethingWith(x)
} else {
    defaultBehavior()
}

Result:

match foo() {
    Some(x) => {
        doSomethingWith(x)
    }
    _ => {
        defaultBehavior()
    }
}

Source:

if cond() {
    doSomething()
} else if let Some(x) = foo() {
    doSomethingWith(x)
} else {
    defaultBehavior()
}

Result:

if cond() {
    doSomething()
} else {
    match foo() {
        Some(x) => {
            doSomethingWith(x)
        }
        _ => {
            defaultBehavior()
        }
    }
}

With the optional addition specified above:

if let Some(x) = foo() {
    doSomethingWith(x)
} else if cond() {
    doSomething()
} else if other_cond() {
    doSomethingElse()
}

Result:

match foo() {
    Some(x) => {
        doSomethingWith(x)
    }
    _ if cond() => {
        doSomething()
    }
    _ if other_cond() => {
        doSomethingElse()
    }
    _ => {}
}

Drawbacks

It’s one more addition to the grammar.

Alternatives

This could plausibly be done with a macro, but the invoking syntax would be pretty terrible and would largely negate the whole point of having this sugar.

Alternatively, this could not be done at all. We’ve been getting alone just fine without it so far, but at the cost of making Option just a bit more annoying to work with.

Unresolved questions

It’s been suggested that alternates or pattern guards should be allowed. I think if you need those you could just go ahead and use a match, and that if let could be extended to support those in the future if a compelling use-case is found.

I don’t know how many match statements in our current code base could be replaced with this syntax. Probably quite a few, but it would be informative to have real data on this.

Summary

Rust’s support for pattern matching on slices has grown steadily and incrementally without a lot of oversight. We have concern that Rust is doing too much here, and that the complexity is not worth it. This RFC proposes to feature gate multiple-element slice matches in the head and middle positions ([xs.., 0, 0] and [0, xs.., 0]).

Motivation

Some general reasons and one specific: first, the implementation of Rust’s match machinery is notoriously complex, and not well-loved. Removing features is seen as a valid way to reduce complexity. Second, slice matching in particular, is difficult to implement, while also being of only moderate utility (there are many types of collections - slices just happen to be built into the language). Finally, the exhaustiveness check is not correct for slice patterns because of their complexity; it’s not known if it can be done correctly, nor whether it is worth the effort to do so.

Detailed design

The advanced_slice_patterns feature gate will be added. When the compiler encounters slice pattern matches in head or middle position it will emit a warning or error according to the current settings.

Drawbacks

It removes two features that some people like.

Alternatives

Fixing the exhaustiveness check would allow the feature to remain.

Unresolved questions

N/A

Summary

Add syntax sugar for importing a module and items in that module in a single view item.

Motivation

Make use clauses more concise.

Detailed design

The mod keyword may be used in a braced list of modules in a use item to mean the prefix module for that list. For example, writing prefix::{mod, foo}; is equivalent to writing

use prefix;
use prefix::foo;

The mod keyword cannot be used outside of braces, nor can it be used inside braces which do not have a prefix path. Both of the following examples are illegal:

use module::mod;
use {mod, foo};

A programmer may write mod in a module list with only a single item. E.g., use prefix::{mod};, although this is considered poor style and may be forbidden by a lint. (The preferred version is use prefix;).

Drawbacks

Another use of the mod keyword.

We introduce a way (the only way) to have paths in use items which do not correspond with paths which can be used in the program. For example, with use foo::bar::{mod, baz}; the programmer can use foo::bar::baz in their program but not foo::bar::mod (instead foo::bar is imported).

Alternatives

Don’t do this.

Unresolved questions

N/A

  • Start Date: 2014-07-16
  • RFC PR #: #169
  • Rust Issue #: https://github.com/rust-lang/rust/issues/16461

Summary

Change the rebinding syntax from use ID = PATH to use PATH as ID, so that paths all line up on the left side, and imported identifiers are all on the right side. Also modify extern crate syntax analogously, for consistency.

Motivation

Currently, the view items at the start of a module look something like this:

mod old_code {
  use a::b::c::d::www;
  use a::b::c::e::xxx;
  use yyy = a::b::yummy;
  use a::b::c::g::zzz;
}

This means that if you want to see what identifiers have been imported, your eyes need to scan back and forth on both the left-hand side (immediately beside the use) and the right-hand side (at the end of each line). In particular, note that yummy is not in scope within the body of old_code

This RFC proposes changing the grammar of Rust so that the example above would look like this:

mod new_code {
  use a::b::c::d::www;
  use a::b::c::e::xxx;
  use a::b::yummy as yyy;
  use a::b::c::g::zzz;
}

There are two benefits we can see by comparing mod old_code and mod new_code:

  • As alluded to above, now all of the imported identifiers are on the right-hand side of the block of view items.

  • Additionally, the left-hand side looks much more regular, since one sees the straight lines of a::b:: characters all the way down, which makes the actual differences between the different paths more visually apparent.

Detailed design

Currently, the grammar for use statements is something like:

  use_decl : "pub" ? "use" [ ident '=' path
                            | path_glob ] ;

Likewise, the grammar for extern crate declarations is something like:

  extern_crate_decl : "extern" "crate" ident [ '(' link_attrs ')' ] ? [ '=' string_lit ] ? ;

This RFC proposes changing the grammar for use statements to something like:

  use_decl : "pub" ? "use" [ path "as" ident
                            | path_glob ] ;

and the grammar for extern crate declarations to something like:

  extern_crate_decl : "extern" "crate" [ string_lit "as" ] ? ident [ '(' link_attrs ')' ] ? ;

Both use and pub use forms are changed to use path as ident instead of ident = path. The form use path as ident has the same constraints and meaning that use ident = path has today.

Nothing about path globs is changed; the view items that use ident = path are disjoint from the view items that use path globs, and that continues to be the case under path as ident.

The old syntaxes "use" ident '=' path and "extern" "crate" ident '=' string_lit are removed (or at least deprecated).

Drawbacks

  • pub use export = import_path may be preferred over pub use import_path as export since people are used to seeing the name exported by a pub item on the left-hand side of an = sign. (See “Have distinct rebinding syntaxes for use and pub use” below.)

  • The ‘as’ keyword is not currently used for any binding form in Rust. Adopting this RFC would change that precedent. (See “Change the signaling token” below.)

Alternatives

Keep things as they are

This just has the drawbacks outlined in the motivation: the left-hand side of the view items are less regular, and one needs to scan both the left- and right-hand sides to see all the imported identifiers.

Change the signaling token

Go ahead with switch, so imported identifier is on the left-hand side, but use a different token than as to signal a rebinding.

For example, we could use @, as an analogy with its use as a binding operator in match expressions:

mod new_code {
  use a::b::c::d::www;
  use a::b::c::e::xxx;
  use a::b::yummy @ yyy;
  use a::b::c::g::zzz;
}

(I do not object to path @ ident, though I find it somehow more “line-noisy” than as in this context.)

Or, we could use =:

mod new_code {
  use a::b::c::d::www;
  use a::b::c::e::xxx;
  use a::b::yummy = yyy;
  use a::b::c::g::zzz;
}

(I do object to path = ident, since typically when = is used to bind, the identifier being bound occurs on the left-hand side.)

Or, we could use :, by (weak) analogy with struct pattern syntax:

mod new_code {
  use a::b::c::d::www;
  use a::b::c::e::xxx;
  use a::b::yummy : yyy;
  use a::b::c::g::zzz;
}

(I cannot figure out if this is genius or madness. Probably madness, especially if one is allowed to omit the whitespace around the :)

Have distinct rebinding syntaxes for use and pub use

If people really like having ident = path for pub use, by the reasoning presented above that people are used to seeing the name exported by a pub item on the left-hand side of an = sign, then we could support that by continuing to support pub use ident = path.

If we were to go down that route, I would prefer to have distinct notions of the exported name and imported name, so that:

pub use a = foo::bar; would actually import bar (and a would just be visible as an export), and then one could rebind for export and import simultaneously, like so: pub use exported_bar = foo::bar as imported_bar;

But really, is pub use foo::bar as a all that bad?

Allow extern crate ident as ident

As written, this RFC allows for two variants of extern_crate_decl:

extern crate old_name;
extern crate "old_name" as new_name;

These are just analogous to the current options that use = instead of as.

However, the RFC comment dialogue suggested also allowing a renaming form that does not use a string literal:

extern crate old_name as new_name;

I have no opinion on whether this should be added or not. Arguably this choice is orthogonal to the goals of this RFC (since, if this is a good idea, it could just as well be implemented with the = syntax). Perhaps it should just be filed as a separate RFC on its own.

Unresolved questions

  • In the revised extern crate form, is it best to put the link_attrs after the identifier, as written above? Or would it be better for them to come after the string_literal when using the extern crate string_literal as ident form?

Summary

Change pattern matching on an &mut T to &mut <pat>, away from its current &<pat> syntax.

Motivation

Pattern matching mirrors construction for almost all types, except &mut, which is constructed with &mut <expr> but destructured with &<pat>. This is almost certainly an unnecessary inconsistency.

This can and does lead to confusion, since people expect the pattern syntax to match construction, but a pattern like &mut (ref mut x, _) is actually currently a parse error:

fn main() {
    let &mut (ref mut x, _);
}
and-mut-pat.rs:2:10: 2:13 error: expected identifier, found path
and-mut-pat.rs:2     let &mut (ref mut x, _);
                          ^~~

Another (rarer) way it can be confusing is the pattern &mut x. It is expected that this binds x to the contents of &mut T pointer… which it does, but as a mutable binding (it is parsed as &(mut x)), meaning something like

for &mut x in some_iterator_over_and_mut {
    println!("{}", x)
}

gives an unused mutability warning. NB. it’s somewhat rare that one would want to pattern match to directly bind a name to the contents of a &mut (since the normal reason to have a &mut is to mutate the thing it points at, but this pattern is (byte) copying the data out, both before and after this change), but can occur if a type only offers a &mut iterator, i.e. types for which a & one is no more flexible than the &mut one.

Detailed design

Add <pat> := &mut <pat> to the pattern grammar, and require that it is used when matching on a &mut T.

Drawbacks

It makes matching through a &mut more verbose: for &mut (ref mut x, p_) in v.mut_iter() instead of for &(ref mut x, _) in v.mut_iter().

Macros wishing to pattern match on either & or &mut need to handle each case, rather than performing both with a single &. However, macros handling these types already need special mut vs. not handling if they ever name the types, or if they use ref vs. ref mut subpatterns.

It also makes obtaining the current behaviour (binding by-value the contents of a reference to a mutable local) slightly harder. For a &mut T the pattern becomes &mut mut x, and, at the moment, for a &T, it must be matched with &x and then rebound with let mut x = x; (since disambiguating like &(mut x) doesn’t yet work). However, based on some loose grepping of the Rust repo, both of these are very rare.

Alternatives

None.

Unresolved questions

None.

  • Start Date: 2014-07-24
  • RFC PR #: https://github.com/rust-lang/rfcs/pull/184
  • Rust Issue #: https://github.com/rust-lang/rust/issues/16950

Summary

Add simple syntax for accessing values within tuples and tuple structs behind a feature gate.

Motivation

Right now accessing fields of tuples and tuple structs is incredibly painful—one must rely on pattern-matching alone to extract values. This became such a problem that twelve traits were created in the standard library (core::tuple::Tuple*) to make tuple value accesses easier, adding .valN(), .refN(), and .mutN() methods to help this. But this is not a very nice solution—it requires the traits to be implemented in the standard library, not the language, and for those traits to be imported on use. On the whole this is not a problem, because most of the time std::prelude::* is imported, but this is still a hack which is not a real solution to the problem at hand. It also only supports tuples of length up to twelve, which is normally not a problem but emphasises how bad the current situation is.

Detailed design

Add syntax of the form <expr>.<integer> for accessing values within tuples and tuple structs. This (and the functionality it provides) would only be allowed when the feature gate tuple_indexing is enabled. This syntax is recognised wherever an unsuffixed integer literal is found in place of the normal field or method name expected when accessing fields with .. Because the parser would be expecting an integer, not a float, an expression like expr.0.1 would be a syntax error (because 0.1 would be treated as a single token).

Tuple/tuple struct field access behaves the same way as accessing named fields on normal structs:

// With tuple struct
struct Foo(int, int);
let mut foo = Foo(3, -15);
foo.0 = 5;
assert_eq!(foo.0, 5);

// With normal struct
struct Foo2 { _0: int, _1: int }
let mut foo2 = Foo2 { _0: 3, _1: -15 };
foo2._0 = 5;
assert_eq!(foo2._0, 5);

Effectively, a tuple or tuple struct field is just a normal named field with an integer for a name.

Drawbacks

This adds more complexity that is not strictly necessary.

Alternatives

Stay with the status quo. Either recommend using a struct with named fields or suggest using pattern-matching to extract values. If extracting individual fields of tuples is really necessary, the TupleN traits could be used instead, and something like #[deriving(Tuple3)] could possibly be added for tuple structs.

Unresolved questions

None.

Summary

  • Remove the special-case bound 'static and replace with a generalized lifetime bound that can be used on objects and type parameters.
  • Remove the rules that aim to prevent references from being stored into objects and replace with a simple lifetime check.
  • Tighten up type rules pertaining to reference lifetimes and well-formed types containing references.
  • Introduce explicit lifetime bounds ('a:'b), with the meaning that the lifetime 'a outlives the lifetime 'b. These exist today but are always inferred; this RFC adds the ability to specify them explicitly, which is sometimes needed in more complex cases.

Motivation

Currently, the type system is not supposed to allow references to escape into object types. However, there are various bugs where it fails to prevent this from happening. Moreover, it is very useful (and frequently necessary) to store a reference into an object. Moreover, the current treatment of generic types is in some cases naive and not obviously sound.

Detailed design

Lifetime bounds on parameters

The heart of the new design is the concept of a lifetime bound. In fact, this (sort of) exists today in the form of the 'static bound:

 fn foo<A:'static>(x: A) { ... }

Here, the notation 'static means “all borrowed content within A outlives the lifetime 'static”. (Note that when we say that something outlives a lifetime, we mean that it lives at least that long. In other words, for any lifetime 'a, 'a outlives 'a. This is similar to how we say that every type T is a subtype of itself.)

In the newer design, it is possible to use an arbitrary lifetime as a bound, and not just 'static:

 fn foo<'a, A:'a>(x: A) { ... }

Explicit lifetime bounds are in fact only rarely necessary, for two reasons:

  1. The compiler is often able to infer this relationship from the argument and return types. More on this below.
  2. It is only important to bound the lifetime of a generic type like A when one of two things is happening (and both of these are cases where the inference generally is sufficient):
    • A borrowed pointer to an A instance (i.e., value of type &A) is being consumed or returned.
    • A value of type A is being closed over into an object reference (or closure, which per the unboxed closures RFC is really the same thing).

Note that, per RFC 11, these lifetime bounds may appear in types as well (this is important later on). For example, an iterator might be declared:

struct Items<'a, T:'a> {
    v: &'a Collection<T>
}

Here, the constraint T:'a indicates that the data being iterated over must live at least as long as the collection (logically enough).

Lifetime bounds on object types

Like parameters, all object types have a lifetime bound. Unlike parameter types, however, object types are required to have exactly one bound. This bound can be either specified explicitly or derived from the traits that appear in the object type. In general, the rule is as follows:

  • If an explicit bound is specified, use that.
  • Otherwise, let S be the set of lifetime bounds we can derive.
  • Otherwise, if S contains ’static, use ’static.
  • Otherwise, if S is a singleton set, use that.
  • Otherwise, error.

Here are some examples:

trait IsStatic : 'static { }
trait Is<'a> : 'a { }

// Type               Bounds
// IsStatic           'static
// Is<'a>             'a
// IsStatic+Is<'a>    'static+'a
// IsStatic+'a        'static+'a
// IsStatic+Is<'a>+'b 'static,'a,'b

Object types must have exactly one bound – zero bounds is not acceptable. Therefore, if an object type with no derivable bounds appears, we will supply a default lifetime using the normal rules:

trait Writer { /* no derivable bounds */ }
struct Foo<'a> {
    Box<Writer>,      // Error: try Box<Writer+'static> or Box<Writer+'a>
    Box<Writer+Send>, // OK: Send implies 'static
    &'a Writer,       // Error: try &'a (Writer+'a)
}

fn foo(a: Box<Writer>, // OK: Sugar for Box<Writer+'a> where 'a fresh
       b: &Writer)     // OK: Sugar for &'b (Writer+'c) where 'b, 'c fresh
{ ... }

This kind of annotation can seem a bit tedious when using object types extensively, though type aliases can help quite a bit:

type WriterObj = Box<Writer+'static>;
type WriterRef<'a> = &'a (Writer+'a);

The unresolved questions section discussed possibles ways to lighten the burden.

See Appendix B for the motivation on why object types are permitted to have exactly one lifetime bound.

Specifying relations between lifetimes

Currently, when a type or fn has multiple lifetime parameters, there is no facility to explicitly specify a relationship between them. For example, in a function like this:

fn foo<'a, 'b>(...) { ... }

the lifetimes 'a and 'b are declared as independent. In some cases, though, it can be important that there be a relation between them. In most cases, these relationships can be inferred (and in fact are inferred today, see below), but it is useful to be able to state them explicitly (and necessary in some cases, see below).

A lifetime bound is written 'a:'b and it means that “'a outlives 'b”. For example, if foo were declared like so:

fn foo<'x, 'y:'x>(...) { ... }

that would indicate that the lifetime ’x was shorter than (or equal to) 'y.

The “type must outlive” and well-formedness relation

Many of the rules to come make use of a “type must outlive” relation, written T outlives 'a. This relation means primarily that all borrowed data in T is known to have a lifetime of at least ’a (hence the name). However, the relation also guarantees various basic lifetime constraints are met. For example, for every reference type &'b U that is found within T, it would be required that U outlives 'b (and that 'b outlives 'a).

In fact, T outlives 'a is defined on another function WF(T:'a), which yields up a list of lifetime relations that must hold for T to be well-formed and to outlive 'a. It is not necessary to understand the details of this relation in order to follow the rest of the RFC, I will defer its precise specification to an appendix below.

For this section, it suffices to give some examples:

// int always outlives any region
WF(int : 'a) = []

// a reference with lifetime 'a outlives 'b if 'a outlives 'b
WF(&'a int : 'b) = ['a : 'b]

// the outer reference must outlive 'c, and the inner reference
// must outlive the outer reference
WF(&'a &'b int : 'c) = ['a : 'c, 'b : 'a]

// Object type with bound 'static
WF(SomeTrait+'static : 'a) = ['static : 'a]

// Object type with bound 'a 
WF(SomeTrait+'a : 'b) = ['a : 'b]

Whenever data of type T is closed over to form an object, the type checker will require that T outlives 'a where 'a is the primary lifetime bound of the object type.

Rules for types to be well-formed

Currently we do not apply any tests to the types that appear in type declarations. Per RFC 11, however, this should change, as we intend to enforce trait bounds on types, wherever those types appear. Similarly, we should be requiring that types are well-formed with respect to the WF function. This means that a type like the following would be illegal without a lifetime bound on the type parameter T:

struct Ref<'a, T> { c: &'a T }

This is illegal because the field c has type &'a T, which is only well-formed if T:'a. Per usual practice, this RFC does not propose any form of inference on struct declarations and instead requires all conditions to be spelled out (this is in contrast to fns and methods, see below).

Rules for expression type validity

We should add the condition that for every expression with lifetime 'e and type T, then T outlives 'e. We already enforce this in many special cases but not uniformly.

Inference

The compiler will infer lifetime bounds on both type parameters and region parameters as follows. Within a function or method, we apply the wellformedness function WF to each function or parameter type. This yields up a set of relations that must hold. The idea here is that the caller could not have type checked unless the types of the arguments were well-formed, so that implies that the callee can assume that those well-formedness constraints hold.

As an example, in the following function:

fn foo<'a, A>(x: &'a A) { ... }

the callee here can assume that the type parameter A outlives the lifetime 'a, even though that was not explicitly declared.

Note that the inference also pulls in constraints that were declared on the types of arguments. So, for example, if there is a type Items declared as follows:

struct Items<'a, T:'a> { ... }

And a function that takes an argument of type Items:

fn foo<'a, T>(x: Items<'a, T>) { ... }

The inference rules will conclude that T:'a because the Items type was declared with that bound.

In practice, these inference rules largely remove the need to manually declare lifetime relations on types. When porting the existing library and rustc over to these rules, I had to add explicit lifetime bounds to exactly one function (but several types, almost exclusively iterators).

Note that this sort of inference is already done. This RFC simply proposes a more extensive version that also includes bounds of the form X:'a, where X is a type parameter.

What does all this mean in practice?

This RFC has a lot of details. The main implications for end users are:

  1. Object types must specify a lifetime bound when they appear in a type. This most commonly means changing Box<Trait> to Box<Trait+'static> and &'a Trait to &'a Trait+'a.

  2. For types that contain references to generic types, lifetime bounds are needed in the type definition. This comes up most often in iterators:

    struct Items<'a, T:'a> {
        x: &'a [T]
    }
    

    Here, the presence of &'a [T] within the type definition requires that the type checker can show that T outlives 'a which in turn requires the bound T:'a on the type definition. These bounds are rarely outside of type definitions, because they are almost always implied by the types of the arguments.

  3. It is sometimes, but rarely, necessary to use lifetime bounds, specifically around double indirections (references to references, often the second reference is contained within a struct). For example:

    struct GlobalContext<'global> {
        arena: &'global Arena
    }
    
    struct LocalContext<'local, 'global:'local> {
        x: &'local mut Context<'global>
    }
    

    Here, we must know that the lifetime 'global outlives 'local in order for this type to be well-formed.

Phasing

Some parts of this RFC require new syntax and thus must be phased in. The current plan is to divide the implementation three parts:

  1. Implement support for everything in this RFC except for region bounds and requiring that every expression type be well-formed. Enforcing the latter constraint leads to type errors that require lifetime bounds to resolve.
  2. Implement support for 'a:'b notation to be parsed under a feature gate issue_5723_bootstrap.
  3. Implement the final bits of the RFC:
    • Bounds on lifetime parameters
    • Wellformedness checks on every expression
    • Wellformedness checks in type definitions

Parts 1 and 2 can be landed simultaneously, but part 3 requires a snapshot. Parts 1 and 2 have largely been written. Depending on precisely how the timing works out, it might make sense to just merge parts 1 and 3.

Drawbacks / Alternatives

If we do not implement some solution, we could continue with the current approach (but patched to be sound) of banning references from being closed over in object types. I consider this a non-starter.

Unresolved questions

Inferring wellformedness bounds

Under this RFC, it is required to write bounds on struct types which are in principle inferable from their contents. For example, iterators tend to follow a pattern like:

struct Items<'a, T:'a> {
    x: &'a [T]
}

Note that T is bounded by 'a. It would be possible to infer these bounds, but I’ve stuck to our current principle that type definitions are always fully spelled out. The danger of inference is that it becomes unclear why a particular constraint exists if one must traverse the type hierarchy deeply to find its origin. This could potentially be addressed with better error messages, though our track record for lifetime error messages is not very good so far.

Also, there is a potential interaction between this sort of inference and the description of default trait bounds below.

Default trait bounds

When referencing a trait object, it is almost always the case that one follows certain fixed patterns:

  • Box<Trait+'static>
  • Rc<Trait+'static> (once DST works)
  • &'a (Trait+'a)
  • and so on.

You might think that we should simply provide some kind of defaults that are sensitive to where the Trait appears. The same is probably true of struct type parameters (in other words, &'a SomeStruct<'a> is a very common pattern).

However, there are complications:

  • What about a type like struct Ref<'a, T:'a> { x: &'a T }? Ref<'a, Trait> should really work the same way as &'a Trait. One way that I can see to do this is to drive the defaulting based on the default trait bounds of the T type parameter – but if we do that, it is both a non-local default (you have to consult the definition of Ref) and interacts with the potential inference described in the previous section.

  • There are reasons to want a type like Box<Trait+'a>. For example, the macro parser includes a function like:

    fn make_macro_ext<'cx>(cx: &'cx Context, ...) -> Box<MacroExt+'cx>
    

    In other words, this function returns an object that closes over the macro context. In such a case, if Box<MacroExt> implies a static bound, then taking ownership of this macro object would require a signature like:

    fn take_macro_ext<'cx>(b: Box<MacroExt+'cx>) {  }
    

    Note that the 'cx variable is only used in one place. It’s purpose is just to disable the 'static default that would otherwise be inserted.

Appendix: Definition of the outlives relation and well-formedness

To make this more specific, we can “formally” model the Rust type system as:

T = scalar (int, uint, fn(...))   // Boring stuff
  | *const T                      // Unsafe pointer
  | *mut T                        // Unsafe pointer
  | Id<P>                         // Nominal type (struct, enum)
  | &'x T                         // Reference
  | &'x mut T                     // Mutable reference
  | {TraitReference<P>}+'x        // Object type
  | X                             // Type variable
P = {'x} + {T}

We can define a function WF(T : 'a) which, given a type T and lifetime 'a yields a list of 'b:'c or X:'d pairs. For each pair 'b:'c, the lifetime 'b must outlive the lifetime 'c for the type T to be well-formed in a location with lifetime 'a. For each pair X:'d, the type parameter X must outlive the lifetime 'd.

  • WF(int : 'a) yields an empty list
  • WF(X:'a) where X is a type parameter yields (X:'a).
  • WF(Foo<P>:'a) where Foo<P> is an enum or struct type yields:
    • For each lifetime parameter 'b that is contravariant or invariant, 'b : 'a.
    • For each type parameter T that is covariant or invariant, the results of WF(T : 'a).
    • The lifetime bounds declared on Foo’s lifetime or type parameters.
    • The reasoning here is that if we can reach borrowed data with lifetime 'a through Foo<'a>, then 'a must be contra- or invariant. Covariant lifetimes only occur in “setter” situations. Analogous reasoning applies to the type case.
  • WF(T:'a) where T is an object type:
    • For the primary bound 'b, 'b : 'a.
    • For each derived bound 'c of T, 'b : 'c
      • Motivation: The primary bound of an object type implies that all other bounds are met. This simplifies some of the other formulations and does not represent a loss of expressiveness.

We can then say that T outlives 'a if all lifetime relations returned by WF(T:'a) hold.

Appendix B: Why object types must have exactly one bound

The motivation is that handling multiple bounds is overwhelmingly complicated to reason about and implement. In various places, constraints arise of the form all i. exists j. R[i] <= R[j], where R is a list of lifetimes. This is challenging for lifetime inference, since there are many options for it to choose from, and thus inference is no longer a fixed-point iteration. Moreover, it doesn’t seem to add any particular expressiveness.

The places where this becomes important are:

  • Checking lifetime bounds when data is closed over into an object type
  • Subtyping between object types, which would most naturally be contravariant in the lifetime bound

Similarly, requiring that the “master” bound on object lifetimes outlives all other bounds also aids inference. Now, given a type like the following:

trait Foo<'a> : 'a { }
trait Bar<'b> : 'b { }

...

let x: Box<Foo<'a>+Bar<'b>>

the inference engine can create a fresh lifetime variable '0 for the master bound and then say that '0:'a and '0:'b. Without the requirement that '0 be a master bound, it would be somewhat unclear how '0 relates to 'a and 'b (in fact, there would be no necessary relation). But if there is no necessary relation, then when closing over data, one would have to ensure that the closed over data outlives all derivable lifetime bounds, which again creates a constraint of the form all i. exists j..

Summary

The #[cfg(...)] attribute provides a mechanism for conditional compilation of items in a Rust crate. This RFC proposes to change the syntax of #[cfg] to make more sense as well as enable expansion of the conditional compilation system to attributes while maintaining a single syntax.

Motivation

In the current implementation, #[cfg(...)] takes a comma separated list of key, key = "value", not(key), or not(key = "value"). An individual #[cfg(...)] attribute “matches” if all of the contained cfg patterns match the compilation environment, and an item preserved if it either has no #[cfg(...)] attributes or any of the #[cfg(...)] attributes present match.

This is problematic for several reasons:

  • It is excessively verbose in certain situations. For example, implementing the equivalent of (a AND (b OR c OR d)) requires three separate attributes and a to be duplicated in each.
  • It differs from all other attributes in that all #[cfg(...)] attributes on an item must be processed together instead of in isolation. This change will move #[cfg(...)] closer to implementation as a normal syntax extension.

Detailed design

The <p> inside of #[cfg(<p>)] will be called a cfg pattern and have a simple recursive syntax:

  • key is a cfg pattern and will match if key is present in the compilation environment.
  • key = "value" is a cfg pattern and will match if a mapping from key to value is present in the compilation environment. At present, key-value pairs only exist for compiler defined keys such as target_os and endian.
  • not(<p>) is a cfg pattern if <p> is and matches if <p> does not match.
  • all(<p>, ...) is a cfg pattern if all of the comma-separated <p>s are cfg patterns and all of them match.
  • any(<p>, ...) is a cfg pattern if all of the comma-separated <p>s are cfg patterns and any of them match.

If an item is tagged with #[cfg(<p>)], that item will be stripped from the AST if the cfg pattern <p> does not match.

One implementation hazard is that the semantics of

#[cfg(a)]
#[cfg(b)]
fn foo() {}

will change from “include foo if either of a and b are present in the compilation environment” to “include foo if both of a and b are present in the compilation environment”. To ease the transition, the old semantics of multiple #[cfg(...)] attributes will be maintained as a special case, with a warning. After some reasonable period of time, the special case will be removed.

In addition, #[cfg(a, b, c)] will be accepted with a warning and be equivalent to #[cfg(all(a, b, c))]. Again, after some reasonable period of time, this behavior will be removed as well.

The cfg!() syntax extension will be modified to accept cfg patterns as well. A #[cfg_attr(<p>, <attr>)] syntax extension will be added (PR 16230) which will expand to #[<attr>] if the cfg pattern <p> matches. The test harness’s #[ignore] attribute will have its built-in cfg filtering functionality stripped in favor of #[cfg_attr(<p>, ignore)].

Drawbacks

While the implementation of this change in the compiler will be straightforward, the effects on downstream code will be significant, especially in the standard library.

Alternatives

all and any could be renamed to and and or, though I feel that the proposed names read better with the function-like syntax and are consistent with Iterator::all and Iterator::any.

Issue #2119 proposed the addition of || and && operators and parentheses to the attribute syntax to result in something like #[cfg(a || (b && c)]. I don’t favor this proposal since it would result in a major change to the attribute syntax for relatively little readability gain.

Unresolved questions

How long should multiple #[cfg(...)] attributes on a single item be forbidden? It should probably be at least until after 0.12 releases.

Should we permanently keep the behavior of treating #[cfg(a, b)] as #[cfg(all(a, b))]? It is the common case, and adding this interpretation can reduce the noise level a bit. On the other hand, it may be a bit confusing to read as it’s not immediately clear if it will be processed as and(..) or all(..).

Summary

This RFC extends traits with associated items, which make generic programming more convenient, scalable, and powerful. In particular, traits will consist of a set of methods, together with:

  • Associated functions (already present as “static” functions)
  • Associated consts
  • Associated types
  • Associated lifetimes

These additions make it much easier to group together a set of related types, functions, and constants into a single package.

This RFC also provides a mechanism for multidispatch traits, where the impl is selected based on multiple types. The connection to associated items will become clear in the detailed text below.

Note: This RFC was originally accepted before RFC 246 introduced the distinction between const and static items. The text has been updated to clarify that associated consts will be added rather than statics, and to provide a summary of restrictions on the initial implementation of associated consts. Other than that modification, the proposal has not been changed to reflect newer Rust features or syntax.

Motivation

A typical example where associated items are helpful is data structures like graphs, which involve at least three types: nodes, edges, and the graph itself.

In today’s Rust, to capture graphs as a generic trait, you have to take the additional types associated with a graph as parameters:

trait Graph<N, E> {
    fn has_edge(&self, &N, &N) -> bool;
    ...
}

The fact that the node and edge types are parameters is confusing, since any concrete graph type is associated with a unique node and edge type. It is also inconvenient, because code working with generic graphs is likewise forced to parameterize, even when not all of the types are relevant:

fn distance<N, E, G: Graph<N, E>>(graph: &G, start: &N, end: &N) -> uint { ... }

With associated types, the graph trait can instead make clear that the node and edge types are determined by any impl:

trait Graph {
    type N;
    type E;
    fn has_edge(&self, &N, &N) -> bool;
}

and clients can abstract over them all at once, referring to them through the graph type:

fn distance<G: Graph>(graph: &G, start: &G::N, end: &G::N) -> uint { ... }

The following subsections expand on the above benefits of associated items, as well as some others.

Associated types: engineering benefits for generics

As the graph example above illustrates, associated types do not increase the expressiveness of traits per se, because you can always use extra type parameters to a trait instead. However, associated types provide several engineering benefits:

  • Readability and scalability

    Associated types make it possible to abstract over a whole family of types at once, without having to separately name each of them. This improves the readability of generic code (like the distance function above). It also makes generics more “scalable”: traits can incorporate additional associated types without imposing an extra burden on clients that don’t care about those types.

    In today’s Rust, by contrast, adding additional generic parameters to a trait often feels like a very “heavyweight” move.

  • Ease of refactoring/evolution

    Because users of a trait do not have to separately parameterize over its associated types, new associated types can be added without breaking all existing client code.

    In today’s Rust, by contrast, associated types can only be added by adding more type parameters to a trait, which breaks all code mentioning the trait.

Clearer trait matching

Type parameters to traits can either be “inputs” or “outputs”:

  • Inputs. An “input” type parameter is used to determine which impl to use.

  • Outputs. An “output” type parameter is uniquely determined by the impl, but plays no role in selecting the impl.

Input and output types play an important role for type inference and trait coherence rules, which is described in more detail later on.

In the vast majority of current libraries, the only input type is the Self type implementing the trait, and all other trait type parameters are outputs. For example, the trait Iterator<A> takes a type parameter A for the elements being iterated over, but this type is always determined by the concrete Self type (e.g. Items<u8>) implementing the trait: A is typically an output.

Additional input type parameters are useful for cases like binary operators, where you may want the impl to depend on the types of both arguments. For example, you might want a trait

trait Add<Rhs, Sum> {
    fn add(&self, rhs: &Rhs) -> Sum;
}

to view the Self and Rhs types as inputs, and the Sum type as an output (since it is uniquely determined by the argument types). This would allow impls to vary depending on the Rhs type, even though the Self type is the same:

impl Add<int, int> for int { ... }
impl Add<Complex, Complex> for int { ... }

Today’s Rust does not make a clear distinction between input and output type parameters to traits. If you attempted to provide the two impls above, you would receive an error like:

error: conflicting implementations for trait `Add`

This RFC clarifies trait matching by:

  • Treating all trait type parameters as input types, and
  • Providing associated types, which are output types.

In this design, the Add trait would be written and implemented as follows:

// Self and Rhs are *inputs*
trait Add<Rhs> {
    type Sum; // Sum is an *output*
    fn add(&self, &Rhs) -> Sum;
}

impl Add<int> for int {
    type Sum = int;
    fn add(&self, rhs: &int) -> int { ... }
}

impl Add<Complex> for int {
    type Sum = Complex;
    fn add(&self, rhs: &Complex) -> Complex { ... }
}

With this approach, a trait declaration like trait Add<Rhs> { ... } is really defining a family of traits, one for each choice of Rhs. One can then provide a distinct impl for every member of this family.

Expressiveness

Associated types, lifetimes, and functions can already be expressed in today’s Rust, though it is unwieldy to do so (as argued above).

But associated consts cannot be expressed using today’s traits.

For example, today’s Rust includes a variety of numeric traits, including Float, which must currently expose constants as static functions:

trait Float {
    fn nan() -> Self;
    fn infinity() -> Self;
    fn neg_infinity() -> Self;
    fn neg_zero() -> Self;
    fn pi() -> Self;
    fn two_pi() -> Self;
    ...
}

Because these functions cannot be used in constant expressions, the modules for float types also export a separate set of constants as consts, not using traits.

Associated constants would allow the consts to live directly on the traits:

trait Float {
    const NAN: Self;
    const INFINITY: Self;
    const NEG_INFINITY: Self;
    const NEG_ZERO: Self;
    const PI: Self;
    const TWO_PI: Self;
    ...
}

Why now?

The above motivations aside, it may not be obvious why adding associated types now (i.e., pre-1.0) is important. There are essentially two reasons.

First, the design presented here is not backwards compatible, because it re-interprets trait type parameters as inputs for the purposes of trait matching. The input/output distinction has several ramifications on coherence rules, type inference, and resolution, which are all described later on in the RFC.

Of course, it might be possible to give a somewhat less ideal design where associated types can be added later on without changing the interpretation of existing trait type parameters. For example, type parameters could be explicitly marked as inputs, and otherwise assumed to be outputs. That would be unfortunate, since associated types would also be outputs – leaving the language with two ways of specifying output types for traits.

But the second reason is for the library stabilization process:

  • Since most existing uses of trait type parameters are intended as outputs, they should really be associated types instead. Making promises about these APIs as they currently stand risks locking the libraries into a design that will seem obsolete as soon as associated items are added. Again, this risk could probably be mitigated with a different, backwards-compatible associated item design, but at the cost of cruft in the language itself.

  • The binary operator traits (e.g. Add) should be multidispatch. It does not seem possible to stabilize them now in a way that will support moving to multidispatch later.

  • There are some thorny problems in the current libraries, such as the _equiv methods accumulating in HashMap, that can be solved using associated items. (See “Defaults” below for more on this specific example.) Additional examples include traits for error propagation and for conversion (to be covered in future RFCs). Adding these traits would improve the quality and consistency of our 1.0 library APIs.

Detailed design

Trait headers

Trait headers are written according to the following grammar:

TRAIT_HEADER =
  'trait' IDENT [ '<' INPUT_PARAMS '>' ] [ ':' BOUNDS ] [ WHERE_CLAUSE ]

INPUT_PARAMS = INPUT_PARAM { ',' INPUT_PARAM }* [ ',' ]
INPUT_PARAM  = IDENT [ ':' BOUNDS ]

BOUNDS = BOUND { '+' BOUND }* [ '+' ]
BOUND  = IDENT [ '<' ARGS '>' ]

ARGS   = INPUT_ARGS
       | OUTPUT_CONSTRAINTS
       | INPUT_ARGS ',' OUTPUT_CONSTRAINTS

INPUT_ARGS = TYPE { ',' TYPE }*

OUTPUT_CONSTRAINTS = OUTPUT_CONSTRAINT { ',' OUTPUT_CONSTRAINT }*
OUTPUT_CONSTRAINT  = IDENT '=' TYPE

NOTE: The grammar for WHERE_CLAUSE and BOUND is explained in detail in the subsection “Constraining associated types” below.

All type parameters to a trait are considered inputs, and can be used to select an impl; conceptually, each distinct instantiation of the types yields a distinct trait. More details are given in the section “The input/output type distinction” below.

Trait bodies: defining associated items

Trait bodies are expanded to include three new kinds of items: consts, types, and lifetimes:

TRAIT = TRAIT_HEADER '{' TRAIT_ITEM* '}'
TRAIT_ITEM =
  ... <existing productions>
  | 'const' IDENT ':' TYPE [ '=' CONST_EXP ] ';'
  | 'type' IDENT [ ':' BOUNDS ] [ WHERE_CLAUSE ] [ '=' TYPE ] ';'
  | 'lifetime' LIFETIME_IDENT ';'

Traits already support associated functions, which had previously been called “static” functions.

The BOUNDS and WHERE_CLAUSE on associated types are obligations for the implementor of the trait, and assumptions for users of the trait:

trait Graph {
    type N: Show + Hash;
    type E: Show + Hash;
    ...
}

impl Graph for MyGraph {
    // Both MyNode and MyEdge must implement Show and Hash
    type N = MyNode;
    type E = MyEdge;
    ...
}

fn print_nodes<G: Graph>(g: &G) {
    // here, can assume G::N implements Show
    ...
}

Namespacing/shadowing for associated types

Associated types may have the same name as existing types in scope, except for type parameters to the trait:

struct Foo { ... }

trait Bar<Input> {
    type Foo; // this is allowed
    fn into_foo(self) -> Foo; // this refers to the trait's Foo

    type Input; // this is NOT allowed
}

By not allowing name clashes between input and output types, keep open the possibility of later allowing syntax like:

Bar<Input=u8, Foo=uint>

where both input and output parameters are constrained by name. And anyway, there is no use for clashing input/output names.

In the case of a name clash like Foo above, if the trait needs to refer to the outer Foo for some reason, it can always do so by using a type alias external to the trait.

Defaults

Notice that associated consts and types both permit defaults, just as trait methods and functions can provide defaults.

Defaults are useful both as a code reuse mechanism, and as a way to expand the items included in a trait without breaking all existing implementors of the trait.

Defaults for associated types, however, present an interesting question: can default methods assume the default type? In other words, is the following allowed?

trait ContainerKey : Clone + Hash + Eq {
    type Query: Hash = Self;
    fn compare(&self, other: &Query) -> bool { self == other }
    fn query_to_key(q: &Query) -> Self { q.clone() };
}

impl ContainerKey for String {
    type Query = str;
    fn compare(&self, other: &str) -> bool {
        self.as_slice() == other
    }
    fn query_to_key(q: &str) -> String {
        q.into_string()
    }
}

impl<K,V> HashMap<K,V> where K: ContainerKey {
    fn find(&self, q: &K::Query) -> &V { ... }
}

In this example, the ContainerKey trait is used to associate a “Query” type (for lookups) with an owned key type. This resolves the thorny “equiv” problem in HashMap, where the hash map keys are Strings but you want to index the hash map with &str values rather than &String values, i.e. you want the following to work:

// H: HashMap<String, SomeType>
H.find("some literal")

rather than having to write

H.find(&"some literal".to_string())`

The current solution involves duplicating the API surface with _equiv methods that use the somewhat subtle Equiv trait, but the associated type approach makes it easy to provide a simple, single API that covers the same use cases.

The defaults for ContainerKey just assume that the owned key and lookup key types are the same, but the default methods have to assume the default associated types in order to work.

For this to work, it must not be possible for an implementor of ContainerKey to override the default Query type while leaving the default methods in place, since those methods may no longer typecheck.

We deal with this in a very simple way:

  • If a trait implementor overrides any default associated types, they must also override all default functions and methods.

  • Otherwise, a trait implementor can selectively override individual default methods/functions, as they can today.

Trait implementations

Trait impl syntax is much the same as before, except that const, type, and lifetime items are allowed:

IMPL_ITEM =
  ... <existing productions>
  | 'const' IDENT ':' TYPE '=' CONST_EXP ';'
  | 'type' IDENT' '=' 'TYPE' ';'
  | 'lifetime' LIFETIME_IDENT '=' LIFETIME_REFERENCE ';'

Any type implementation must satisfy all bounds and where clauses in the corresponding trait item.

Referencing associated items

Associated items are referenced through paths. The expression path grammar was updated as part of UFCS, but to accommodate associated types and lifetimes we need to update the type path grammar as well.

The full grammar is as follows:

EXP_PATH
  = EXP_ID_SEGMENT { '::' EXP_ID_SEGMENT }*
  | TYPE_SEGMENT { '::' EXP_ID_SEGMENT }+
  | IMPL_SEGMENT { '::' EXP_ID_SEGMENT }+
EXP_ID_SEGMENT   = ID [ '::' '<' TYPE { ',' TYPE }* '>' ]

TY_PATH
  = TY_ID_SEGMENT { '::' TY_ID_SEGMENT }*
  | TYPE_SEGMENT { '::' TY_ID_SEGMENT }*
  | IMPL_SEGMENT { '::' TY_ID_SEGMENT }+

TYPE_SEGMENT = '<' TYPE '>'
IMPL_SEGMENT = '<' TYPE 'as' TRAIT_REFERENCE '>'
TRAIT_REFERENCE = ID [ '<' TYPE { ',' TYPE * '>' ]

Here are some example paths, along with what they might be referencing

// Expression paths ///////////////////////////////////////////////////////////////

a::b::c         // reference to a function `c` in module `a::b`
a::<T1, T2>     // the function `a` instantiated with type arguments `T1`, `T2`
Vec::<T>::new   // reference to the function `new` associated with `Vec<T>`
<Vec<T> as SomeTrait>::some_fn
                // reference to the function `some_fn` associated with `SomeTrait`,
                //   as implemented by `Vec<T>`
T::size_of      // the function `size_of` associated with the type or trait `T`
<T>::size_of    // the function `size_of` associated with `T` _viewed as a type_
<T as SizeOf>::size_of
                // the function `size_of` associated with `T`'s impl of `SizeOf`

// Type paths /////////////////////////////////////////////////////////////////////

a::b::C         // reference to a type `C` in module `a::b`
A<T1, T2>       // type A instantiated with type arguments `T1`, `T2`
Vec<T>::Iter    // reference to the type `Iter` associated with `Vec<T>
<Vec<T> as SomeTrait>::SomeType
                // reference to the type `SomeType` associated with `SomeTrait`,
                //   as implemented by `Vec<T>`

Ways to reference items

Next, we’ll go into more detail on the meaning of each kind of path.

For the sake of discussion, we’ll suppose we’ve defined a trait like the following:

trait Container {
    type E;
    fn empty() -> Self;
    fn insert(&mut self, E);
    fn contains(&self, &E) -> bool where E: PartialEq;
    ...
}

impl<T> Container for Vec<T> {
    type E = T;
    fn empty() -> Vec<T> { Vec::new() }
    ...
}

Via an ID_SEGMENT prefix

When the prefix resolves to a type

The most common way to get at an associated item is through a type parameter with a trait bound:

fn pick<C: Container>(c: &C) -> Option<&C::E> { ... }

fn mk_with_two<C>() -> C where C: Container, C::E = uint {
    let mut cont = C::empty();  // reference to associated function
    cont.insert(0);
    cont.insert(1);
    cont
}

For these references to be valid, the type parameter must be known to implement the relevant trait:

// Knowledge via bounds
fn pick<C: Container>(c: &C) -> Option<&C::E> { ... }

// ... or equivalently,  where clause
fn pick<C>(c: &C) -> Option<&C::E> where C: Container { ... }

// Knowledge via ambient constraints
struct TwoContainers<C1: Container, C2: Container>(C1, C2);
impl<C1: Container, C2: Container> TwoContainers<C1, C2> {
    fn pick_one(&self) -> Option<&C1::E> { ... }
    fn pick_other(&self) -> Option<&C2::E> { ... }
}

Note that Vec<T>::E and Vec::<T>::empty are also valid type and function references, respectively.

For cases like C::E or Vec<T>::E, the path begins with an ID_SEGMENT prefix that itself resolves to a type: both C and Vec<T> are types. In general, a path PREFIX::REST_OF_PATH where PREFIX resolves to a type is equivalent to using a TYPE_SEGMENT prefix <PREFIX>::REST_OF_PATH. So, for example, following are all equivalent:

fn pick<C: Container>(c: &C) -> Option<&C::E> { ... }
fn pick<C: Container>(c: &C) -> Option<&<C>::E> { ... }
fn pick<C: Container>(c: &C) -> Option<&<<C>::E>> { ... }

The behavior of TYPE_SEGMENT prefixes is described in the next subsection.

When the prefix resolves to a trait

However, it is possible for an ID_SEGMENT prefix to resolve to a trait, rather than a type. In this case, the behavior of an ID_SEGMENT varies from that of a TYPE_SEGMENT in the following way:

// a reference Container::insert is roughly equivalent to:
fn trait_insert<C: Container>(c: &C, e: C::E);

// a reference <Container>::insert is roughly equivalent to:
fn object_insert<E>(c: &Container<E=E>, e: E);

That is, if PREFIX is an ID_SEGMENT that resolves to a trait Trait:

  • A path PREFIX::REST resolves to the item/path REST defined within Trait, while treating the type implementing the trait as a type parameter.

  • A path <PREFIX>::REST treats PREFIX as a (DST-style) type, and is hence usable only with trait objects. See the UFCS RFC for more detail.

Note that a path like Container::E, while grammatically valid, will fail to resolve since there is no way to tell which impl to use. A path like Container::empty, however, resolves to a function roughly equivalent to:

fn trait_empty<C: Container>() -> C;

Via a TYPE_SEGMENT prefix

The following text is slightly changed from the UFCS RFC.

When a path begins with a TYPE_SEGMENT, it is a type-relative path. If this is the complete path (e.g., <int>), then the path resolves to the specified type. If the path continues (e.g., <int>::size_of) then the next segment is resolved using the following procedure. The procedure is intended to mimic method lookup, and hence any changes to method lookup may also change the details of this lookup.

Given a path <T>::m::...:

  1. Search for members of inherent impls defined on T (if any) with the name m. If any are found, the path resolves to that item.

  2. Otherwise, let IN_SCOPE_TRAITS be the set of traits that are in scope and which contain a member named m:

    • Let IMPLEMENTED_TRAITS be those traits from IN_SCOPE_TRAITS for which an implementation exists that (may) apply to T.
      • There can be ambiguity in the case that T contains type inference variables.
    • If IMPLEMENTED_TRAITS is not a singleton set, report an ambiguity error. Otherwise, let TRAIT be the member of IMPLEMENTED_TRAITS.
    • If TRAIT is ambiguously implemented for T, report an ambiguity error and request further type information.
    • Otherwise, rewrite the path to <T as Trait>::m::... and continue.

Via a IMPL_SEGMENT prefix

The following text is somewhat different from the UFCS RFC.

When a path begins with an IMPL_SEGMENT, it is a reference to an item defined from a trait. Note that such paths must always have a follow-on member m (that is, <T as Trait> is not a complete path, but <T as Trait>::m is).

To resolve the path, first search for an applicable implementation of Trait for T. If no implementation can be found – or the result is ambiguous – then report an error. Note that when T is a type parameter, a bound T: Trait guarantees that there is such an implementation, but does not count for ambiguity purposes.

Otherwise, resolve the path to the member of the trait with the substitution Self => T and continue.

This apparently straightforward algorithm has some subtle consequences, as illustrated by the following example:

trait Foo {
    type T;
    fn as_T(&self) -> &T;
}

// A blanket impl for any Show type T
impl<T: Show> Foo for T {
    type T = T;
    fn as_T(&self) -> &T { self }
}

fn bounded<U: Foo>(u: U) where U::T: Show {
    // Here, we just constrain the associated type directly
    println!("{}", u.as_T())
}

fn blanket<U: Show>(u: U) {
    // the blanket impl applies to U, so we know that `U: Foo` and
    // <U as Foo>::T = U (and, of course, U: Show)
    println!("{}", u.as_T())
}

fn not_allowed<U: Foo>(u: U) {
    // this will not compile, since <U as Trait>::T is not known to
    // implement Show
    println!("{}", u.as_T())
}

This example includes three generic functions that make use of an associated type; the first two will typecheck, while the third will not.

  • The first case, bounded, places a Show constraint directly on the otherwise-abstract associated type U::T. Hence, it is allowed to assume that U::T: Show, even though it does not know the concrete implementation of Foo for U.

  • The second case, blanket, places a Show constraint on the type U, which means that the blanket impl of Foo applies even though we do not know the concrete type that U will be. That fact means, moreover, that we can compute exactly what the associated type U::T will be, and know that it will satisfy Show. Coherence guarantees that that the blanket impl is the only one that could apply to U. (See the section “Impl specialization” under “Unresolved questions” for a deeper discussion of this point.)

  • The third case assumes only that U: Foo, and therefore nothing is known about the associated type U::T. In particular, the function cannot assume that U::T: Show.

The resolution rules also interact with instantiation of type parameters in an intuitive way. For example:

trait Graph {
    type N;
    type E;
    ...
}

impl Graph for MyGraph {
    type N = MyNode;
    type E = MyEdge;
    ...
}

fn pick_node<G: Graph>(t: &G) -> &G::N {
    // the type G::N is abstract here
    ...
}

let G = MyGraph::new();
...
pick_node(G) // has type: <MyGraph as Graph>::N = MyNode

Assuming there are no blanket implementations of Graph, the pick_node function knows nothing about the associated type G::N. However, a client of pick_node that instantiates it with a particular concrete graph type will also know the concrete type of the value returned from the function – here, MyNode.

Scoping of trait and impl items

Associated types are frequently referred to in the signatures of a trait’s methods and associated functions, and it is natural and convenient to refer to them directly.

In other words, writing this:

trait Graph {
    type N;
    type E;
    fn has_edge(&self, &N, &N) -> bool;
    ...
}

is more appealing than writing this:

trait Graph {
    type N;
    type E;
    fn has_edge(&self, &Self::N, &Self::N) -> bool;
    ...
}

This RFC proposes to treat both trait and impl bodies (both inherent and for traits) the same way we treat mod bodies: all items being defined are in scope. In particular, methods are in scope as UFCS-style functions:

trait Foo {
    type AssocType;
    lifetime 'assoc_lifetime;
    const ASSOC_CONST: uint;
    fn assoc_fn() -> Self;

    // Note: 'assoc_lifetime and AssocType in scope:
    fn method(&self, Self) -> &'assoc_lifetime AssocType;

    fn default_method(&self) -> uint {
        // method in scope UFCS-style, assoc_fn in scope
        let _ = method(self, assoc_fn());
        ASSOC_CONST // in scope
    }
}

// Same scoping rules for impls, including inherent impls:
struct Bar;
impl Bar {
    fn foo(&self) { ... }
    fn bar(&self) {
        foo(self); // foo in scope UFCS-style
        ...
    }
}

Items from super traits are not in scope, however. See the discussion on super traits below for more detail.

These scope rules provide good ergonomics for associated types in particular, and a consistent scope model for language constructs that can contain items (like traits, impls, and modules). In the long run, we should also explore imports for trait items, i.e. use Trait::some_method, but that is out of scope for this RFC.

Note that, according to this proposal, associated types/lifetimes are not in scope for the optional where clause on the trait header. For example:

trait Foo<Input>
    // type parameters in scope, but associated types are not:
    where Bar<Input, Self::Output>: Encodable {

    type Output;
    ...
}

This setup seems more intuitive than allowing the trait header to refer directly to items defined within the trait body.

It’s also worth noting that trait-level where clauses are never needed for constraining associated types anyway, because associated types also have where clauses. Thus, the above example could (and should) instead be written as follows:

trait Foo<Input> {
    type Output where Bar<Input, Output>: Encodable;
    ...
}

Constraining associated types

Associated types are not treated as parameters to a trait, but in some cases a function will want to constrain associated types in some way. For example, as explained in the Motivation section, the Iterator trait should treat the element type as an output:

trait Iterator {
    type A;
    fn next(&mut self) -> Option<A>;
    ...
}

For code that works with iterators generically, there is no need to constrain this type:

fn collect_into_vec<I: Iterator>(iter: I) -> Vec<I::A> { ... }

But other code may have requirements for the element type:

  • That it implements some traits (bounds).
  • That it unifies with a particular type.

These requirements can be imposed via where clauses:

fn print_iter<I>(iter: I) where I: Iterator, I::A: Show { ... }
fn sum_uints<I>(iter: I) where I: Iterator, I::A = uint { ... }

In addition, there is a shorthand for equality constraints:

fn sum_uints<I: Iterator<A = uint>>(iter: I) { ... }

In general, a trait like:

trait Foo<Input1, Input2> {
    type Output1;
    type Output2;
    lifetime 'a;
    const C: bool;
    ...
}

can be written in a bound like:

T: Foo<I1, I2>
T: Foo<I1, I2, Output1 = O1>
T: Foo<I1, I2, Output2 = O2>
T: Foo<I1, I2, Output1 = O1, Output2 = O2>
T: Foo<I1, I2, Output1 = O1, 'a = 'b, Output2 = O2>
T: Foo<I1, I2, Output1 = O1, 'a = 'b, C = true, Output2 = O2>

The output constraints must come after all input arguments, but can appear in any order.

Note that output constraints are allowed when referencing a trait in a type or a bound, but not in an IMPL_SEGMENT path:

  • As a type: fn foo(obj: Box<Iterator<A = uint>> is allowed.
  • In a bound: fn foo<I: Iterator<A = uint>>(iter: I) is allowed.
  • In an IMPL_SEGMENT: <I as Iterator<A = uint>>::next is not allowed.

The reason not to allow output constraints in IMPL_SEGMENT is that such paths are references to a trait implementation that has already been determined – it does not make sense to apply additional constraints to the implementation when referencing it.

Output constraints are a handy shorthand when using trait bounds, but they are a necessity for trait objects, which we discuss next.

Trait objects

When using trait objects, the Self type is “erased”, so different types implementing the trait can be used under the same trait object type:

impl Show for Foo { ... }
impl Show for Bar { ... }

fn make_vec() -> Vec<Box<Show>> {
    let f = Foo { ... };
    let b = Bar { ... };
    let mut v = Vec::new();
    v.push(box f as Box<Show>);
    v.push(box b as Box<Show>);
    v
}

One consequence of erasing Self is that methods using the Self type as arguments or return values cannot be used on trait objects, since their types would differ for different choices of Self.

In the model presented in this RFC, traits have additional input parameters beyond Self, as well as associated types that may vary depending on all of the input parameters. This raises the question: which of these types, if any, are erased in trait objects?

The approach we take here is the simplest and most conservative: when using a trait as a type (i.e., as a trait object), all input and output types must be provided as part of the type. In other words, only the Self type is erased, and all other types are specified statically in the trait object type.

Consider again the following example:

trait Foo<Input1, Input2> {
    type Output1;
    type Output2;
    lifetime 'a;
    const C: bool;
    ...
}

Unlike the case for static trait bounds, which do not have to specify any of the associated types, lifetimes, or consts, (but do have to specify the input types), trait object types must specify all of the types:

fn consume_foo<T: Foo<I1, I2>>(t: T) // this is valid
fn consume_obj(t: Box<Foo<I1, I2>>)  // this is NOT valid

// but this IS valid:
fn consume_obj(t: Box<Foo<I1, I2, Output1 = O2, Output2 = O2, 'a = 'static, C = true>>)

With this design, it is clear that none of the non-Self types are erased as part of trait objects. But it leaves wiggle room to relax this restriction later on: trait object types that are not allowed under this design can be given meaning in some later design.

Inherent associated items

All associated items are also allowed in inherent impls, so a definition like the following is allowed:

struct MyGraph { ... }
struct MyNode { ... }
struct MyEdge { ... }

impl MyGraph {
    type N = MyNode;
    type E = MyEdge;

    // Note: associated types in scope, just as with trait bodies
    fn has_edge(&self, &N, &N) -> bool {
        ...
    }

    ...
}

Inherent associated items are referenced similarly to trait associated items:

fn distance(g: &MyGraph, from: &MyGraph::N, to: &MyGraph::N) -> uint { ... }

Note, however, that output constraints do not make sense for inherent outputs:

// This is *not* a legal type:
MyGraph<N = SomeNodeType>

The input/output type distinction

When designing a trait that references some unknown type, you now have the option of taking that type as an input parameter, or specifying it as an output associated type. What are the ramifications of this decision?

Coherence implications

Input types are used when determining which impl matches, even for the same Self type:

trait Iterable1<A> {
    type I: Iterator<A>;
    fn iter(self) -> I;
}

// These impls have distinct input types, so are allowed
impl Iterable1<u8> for Foo { ... }
impl Iterable1<char> for Foo { ... }

trait Iterable2 {
    type A;
    type I: Iterator<A>;
    fn iter(self) -> I;
}

// These impls apply to a common input (Foo), so are NOT allowed
impl Iterable2 for Foo { ... }
impl Iterable2 for Foo { ... }

More formally, the coherence property is revised as follows:

  • Given a trait and values for all its type parameters (inputs, including Self), there is at most one applicable impl.

In the trait reform RFC, coherence is guaranteed by maintaining two other key properties, which are revised as follows:

Orphan check: Every implementation must meet one of the following conditions:

  1. The trait being implemented (if any) must be defined in the current crate.

  2. At least one of the input type parameters (including but not necessarily Self) must meet the following grammar, where C is a struct or enum defined within the current crate:

    T = C
      | [T]
      | [T, ..n]
      | &T
      | &mut T
      | ~T
      | (..., T, ...)
      | X<..., T, ...> where X is not bivariant with respect to T
    

Overlapping instances: No two implementations can be instantiable with the same set of types for the input type parameters.

See the trait reform RFC for more discussion of these properties.

Type inference implications

Finally, output type parameters can be inferred/resolved as soon as there is a matching impl based on the input type parameters. Because of the coherence property above, there can be at most one.

On the other hand, even if there is only one applicable impl, type inference is not allowed to infer the input type parameters from it. This restriction makes it possible to ensure crate concatenation: adding another crate may add impls for a given trait, and if type inference depended on the absence of such impls, importing a crate could break existing code.

In practice, these inference benefits can be quite valuable. For example, in the Add trait given at the beginning of this RFC, the Sum output type is immediately known once the input types are known, which can avoid the need for type annotations.

Limitations

The main limitation of associated items as presented here is about associated types in particular. You might be tempted to write a trait like the following:

trait Iterable {
    type A;
    type I: Iterator<&'a A>; // what is the lifetime here?
    fn iter<'a>(&'a self) -> I;  // and how to connect it to self?
}

The problem is that, when implementing this trait, the return type I of iter must generally depend on the lifetime of self. For example, the corresponding method in Vec looks like the following:

impl<T> Vec<T> {
    fn iter(&'a self) -> Items<'a, T> { ... }
}

This means that, given a Vec<T>, there isn’t a single type Items<T> for iteration – rather, there is a family of types, one for each input lifetime. In other words, the associated type I in the Iterable needs to be “higher-kinded”: not just a single type, but rather a family:

trait Iterable {
    type A;
    type I<'a>: Iterator<&'a A>;
    fn iter<'a>(&self) -> I<'a>;
}

In this case, I is parameterized by a lifetime, but in other cases (like map) an associated type needs to be parameterized by a type.

In general, such higher-kinded types (HKTs) are a much-requested feature for Rust, and they would extend the reach of associated types. But the design and implementation of higher-kinded types is, by itself, a significant investment. The point of view of this RFC is that associated items bring the most important changes needed to stabilize our existing traits (and add a few key others), while HKTs will allow us to define important traits in the future but are not necessary for 1.0.

Encoding higher-kinded types

That said, it’s worth pointing out that variants of higher-kinded types can be encoded in the system being proposed here.

For example, the Iterable example above can be written in the following somewhat contorted style:

trait IterableOwned {
    type A;
    type I: Iterator<A>;
    fn iter_owned(self) -> I;
}

trait Iterable {
    fn iter<'a>(&'a self) -> <&'a Self>::I where &'a Self: IterableOwned {
        IterableOwned::iter_owned(self)
    }
}

The idea here is to define a trait that takes, as input type/lifetimes parameters, the parameters to any HKTs. In this case, the trait is implemented on the type &'a Self, which includes the lifetime parameter.

We can in fact generalize this technique to encode arbitrary HKTs:

// The kind * -> *
trait TypeToType<Input> {
    type Output;
}
type Apply<Name, Elt> where Name: TypeToType<Elt> = Name::Output;

struct Vec_;
struct DList_;

impl<T> TypeToType<T> for Vec_ {
    type Output = Vec<T>;
}

impl<T> TypeToType<T> for DList_ {
    type Output = DList<T>;
}

trait Mappable
{
    type E;
    type HKT where Apply<HKT, E> = Self;

    fn map<F>(self, f: E -> F) -> Apply<HKT, F>;
}

While the above demonstrates the versatility of associated types and where clauses, it is probably too much of a hack to be viable for use in libstd.

Associated consts in generic code

If the value of an associated const depends on a type parameter (including Self), it cannot be used in a constant expression. This restriction will almost certainly be lifted in the future, but this raises questions outside the scope of this RFC.

Staging

Associated lifetimes are probably not necessary for the 1.0 timeframe. While we currently have a few traits that are parameterized by lifetimes, most of these can go away once DST lands.

On the other hand, associated lifetimes are probably trivial to implement once associated types have been implemented.

Other interactions

Interaction with implied bounds

As part of the implied bounds idea, it may be desirable for this:

fn pick_node<G>(g: &G) -> &<G as Graph>::N

to be sugar for this:

fn pick_node<G: Graph>(g: &G) -> &<G as Graph>::N

But this feature can easily be added later, as part of a general implied bounds RFC.

Future-proofing: specialization of impls

In the future, we may wish to relax the “overlapping instances” rule so that one can provide “blanket” trait implementations and then “specialize” them for particular types. For example:

trait Sliceable {
    type Slice;
    // note: not using &self here to avoid need for HKT
    fn as_slice(self) -> Slice;
}

impl<'a, T> Sliceable for &'a T {
    type Slice = &'a T;
    fn as_slice(self) -> &'a T { self }
}

impl<'a, T> Sliceable for &'a Vec<T> {
    type Slice = &'a [T];
    fn as_slice(self) -> &'a [T] { self.as_slice() }
}

But then there’s a difficult question:

fn dice<A>(a: &A) -> &A::Slice where &A: Sliceable {
    a // is this allowed?
}

Here, the blanket and specialized implementations provide incompatible associated types. When working with the trait generically, what can we assume about the associated type? If we assume it is the blanket one, the type may change during monomorphization (when specialization takes effect)!

The RFC does allow generic code to “see” associated types provided by blanket implementations, so this is a potential problem.

Our suggested strategy is the following. If at some later point we wish to add specialization, traits would have to opt in explicitly. For such traits, we would not allow generic code to “see” associated types for blanket implementations; instead, output types would only be visible when all input types were concretely known. This approach is backwards-compatible with the RFC, and is probably a good idea in any case.

Alternatives

Multidispatch through tuple types

This RFC clarifies trait matching by making trait type parameters inputs to matching, and associated types outputs.

A more radical alternative would be to remove type parameters from traits, and instead support multiple input types through a separate multidispatch mechanism.

In this design, the Add trait would be written and implemented as follows:

// Lhs and Rhs are *inputs*
trait Add for (Lhs, Rhs) {
    type Sum; // Sum is an *output*
    fn add(&Lhs, &Rhs) -> Sum;
}

impl Add for (int, int) {
    type Sum = int;
    fn add(left: &int, right: &int) -> int { ... }
}

impl Add for (int, Complex) {
    type Sum = Complex;
    fn add(left: &int, right: &Complex) -> Complex { ... }
}

The for syntax in the trait definition is used for multidispatch traits, here saying that impls must be for pairs of types which are bound to Lhs and Rhs respectively. The add function can then be invoked in UFCS style by writing

Add::add(some_int, some_complex)

Advantages of the tuple approach:

  • It does not force a distinction between Self and other input types, which in some cases (including binary operators like Add) can be artificial.

  • Makes it possible to specify input types without specifying the trait: <(A, B)>::Sum rather than <A as Add<B>>::Sum.

Disadvantages of the tuple approach:

  • It’s more painful when you do want a method rather than a function.

  • Requires where clauses when used in bounds: where (A, B): Trait rather than A: Trait<B>.

  • It gives two ways to write single dispatch: either without for, or using for with a single-element tuple.

  • There’s a somewhat jarring distinction between single/multiple dispatch traits, making the latter feel “bolted on”.

  • The tuple syntax is unusual in acting as a binder of its types, as opposed to the Trait<A, B> syntax.

  • Relatedly, the generics syntax for traits is immediately understandable (a family of traits) based on other uses of generics in the language, while the tuple notation stands alone.

  • Less clear story for trait objects (although the fact that Self is the only erased input type in this RFC may seem somewhat arbitrary).

On balance, the generics-based approach seems like a better fit for the language design, especially in its interaction with methods and the object system.

A backwards-compatible version

Yet another alternative would be to allow trait type parameters to be either inputs or outputs, marking the inputs with a keyword in:

trait Add<in Rhs, Sum> {
    fn add(&Lhs, &Rhs) -> Sum;
}

This would provide a way of adding multidispatch now, and then adding associated items later on without breakage. If, in addition, output types had to come after all input types, it might even be possible to migrate output type parameters like Sum above into associated types later.

This is perhaps a reasonable fallback, but it seems better to introduce a clean design with both multidispatch and associated items together.

Unresolved questions

Super traits

This RFC largely ignores super traits.

Currently, the implementation of super traits treats them identically to a where clause that bounds Self, and this RFC does not propose to change that. However, a follow-up RFC should clarify that this is the intended semantics for super traits.

Note that this treatment of super traits is, in particular, consistent with the proposed scoping rules, which do not bring items from super traits into scope in the body of a subtrait; they must be accessed via Self::item_name.

Equality constraints in where clauses

This RFC allows equality constraints on types for associated types, but does not propose a similar feature for where clauses. That will be the subject of a follow-up RFC.

Multiple trait object bounds for the same trait

The design here makes it possible to write bounds or trait objects that mention the same trait, multiple times, with different inputs:

fn mulit_add<T: Add<int> + Add<Complex>>(t: T) -> T { ... }
fn mulit_add_obj(t: Box<Add<int> + Add<Complex>>) -> Box<Add<int> + Add<Complex>> { ... }

This seems like a potentially useful feature, and should be unproblematic for bounds, but may have implications for vtables that make it problematic for trait objects. Whether or not such trait combinations are allowed will likely depend on implementation concerns, which are not yet clear.

Generic associated consts in match patterns

It seems desirable to allow constants that depend on type parameters in match patterns, but it’s not clear how to do so while still checking exhaustiveness and reachability of the match arms. Most likely this requires new forms of where clause, to constrain associated constant values.

For now, we simply defer the question.

Generic associated consts in array sizes

It would be useful to be able to use trait-associated constants in generic code.

// Shouldn't this be OK?
const ALIAS_N: usize = <T>::N;
let x: [u8; <T>::N] = [0u8; ALIAS_N];
// Or...
let x: [u8; T::N + 1] = [0u8; T::N + 1];

However, this causes some problems. What should we do with the following case in type checking, where we need to prove that a generic is valid for any T?

let x: [u8; T::N + T::N] = [0u8; 2 * T::N];

We would like to handle at least some obvious cases (e.g. proving that T::N == T::N), but without trying to prove arbitrary statements about arithmetic. The question of how to do this is deferred.

Summary

This RFC adds overloaded slice notation:

  • foo[] for foo.as_slice()
  • foo[n..m] for foo.slice(n, m)
  • foo[n..] for foo.slice_from(n)
  • foo[..m] for foo.slice_to(m)
  • mut variants of all the above

via two new traits, Slice and SliceMut.

It also changes the notation for range match patterns to ..., to signify that they are inclusive whereas .. in slices are exclusive.

Motivation

There are two primary motivations for introducing this feature.

Ergonomics

Slicing operations, especially as_slice, are a very common and basic thing to do with vectors, and potentially many other kinds of containers. We already have notation for indexing via the Index trait, and this RFC is essentially a continuation of that effort.

The as_slice operator is particularly important. Since we’ve moved away from auto-slicing in coercions, explicit as_slice calls have become extremely common, and are one of the leading ergonomic/first impression problems with the language. There are a few other approaches to address this particular problem, but these alternatives have downsides that are discussed below (see “Alternatives”).

Error handling conventions

We are gradually moving toward a Python-like world where notation like foo[n] calls fail! when n is out of bounds, while corresponding methods like get return Option values rather than failing. By providing similar notation for slicing, we open the door to following the same convention throughout vector-like APIs.

Detailed design

The design is a straightforward continuation of the Index trait design. We introduce two new traits, for immutable and mutable slicing:

trait Slice<Idx, S> {
    fn as_slice<'a>(&'a self) -> &'a S;
    fn slice_from(&'a self, from: Idx) -> &'a S;
    fn slice_to(&'a self, to: Idx) -> &'a S;
    fn slice(&'a self, from: Idx, to: Idx) -> &'a S;
}

trait SliceMut<Idx, S> {
    fn as_mut_slice<'a>(&'a mut self) -> &'a mut S;
    fn slice_from_mut(&'a mut self, from: Idx) -> &'a mut S;
    fn slice_to_mut(&'a mut self, to: Idx) -> &'a mut S;
    fn slice_mut(&'a mut self, from: Idx, to: Idx) -> &'a mut S;
}

(Note, the mutable names here are part of likely changes to naming conventions that will be described in a separate RFC).

These traits will be used when interpreting the following notation:

Immutable slicing

  • foo[] for foo.as_slice()
  • foo[n..m] for foo.slice(n, m)
  • foo[n..] for foo.slice_from(n)
  • foo[..m] for foo.slice_to(m)

Mutable slicing

  • foo[mut] for foo.as_mut_slice()
  • foo[mut n..m] for foo.slice_mut(n, m)
  • foo[mut n..] for foo.slice_from_mut(n)
  • foo[mut ..m] for foo.slice_to_mut(m)

Like Index, uses of this notation will auto-deref just as if they were method invocations. So if T implements Slice<uint, [U]>, and s: Smaht<T>, then s[] compiles and has type &[U].

Note that slicing is “exclusive” (so [n..m] is the interval n <= x < m), while .. in match patterns is “inclusive”. To avoid confusion, we propose to change the match notation to ... to reflect the distinction. The reason to change the notation, rather than the interpretation, is that the exclusive (respectively inclusive) interpretation is the right default for slicing (respectively matching).

Rationale for the notation

The choice of square brackets for slicing is straightforward: it matches our indexing notation, and slicing and indexing are closely related.

Some other languages (like Python and Go – and Fortran) use : rather than .. in slice notation. The choice of .. here is influenced by its use elsewhere in Rust, for example for fixed-length array types [T, ..n]. The .. for slicing has precedent in Perl and D.

See Wikipedia for more on the history of slice notation in programming languages.

The mut qualifier

It may be surprising that mut is used as a qualifier in the proposed slice notation, but not for the indexing notation. The reason is that indexing includes an implicit dereference. If v: Vec<Foo> then v[n] has type Foo, not &Foo or &mut Foo. So if you want to get a mutable reference via indexing, you write &mut v[n]. More generally, this allows us to do resolution/typechecking prior to resolving the mutability.

This treatment of Index matches the C tradition, and allows us to write things like v[0] = foo instead of *v[0] = foo.

On the other hand, this approach is problematic for slicing, since in general it would yield an unsized type (under DST) – and of course, slicing is meant to give you a fat pointer indicating the size of the slice, which we don’t want to immediately deref. But the consequence is that we need to know the mutability of the slice up front, when we take it, since it determines the type of the expression.

Drawbacks

The main drawback is the increase in complexity of the language syntax. This seems minor, especially since the notation here is essentially “finishing” what was started with the Index trait.

Limitations in the design

Like the Index trait, this forces the result to be a reference via &, which may rule out some generalizations of slicing.

One way of solving this problem is for the slice methods to take self (by value) rather than &self, and in turn to implement the trait on &T rather than T. Whether this approach is viable in the long run will depend on the final rules for method resolution and auto-ref.

In general, the trait system works best when traits can be applied to types T rather than borrowed types &T. Ultimately, if Rust gains higher-kinded types (HKT), we could change the slice type S in the trait to be higher-kinded, so that it is a family of types indexed by lifetime. Then we could replace the &'a S in the return value with S<'a>. It should be possible to transition from the current Index and Slice trait designs to an HKT version in the future without breaking backwards compatibility by using blanket implementations of the new traits (say, IndexHKT) for types that implement the old ones.

Alternatives

For improving the ergonomics of as_slice, there are two main alternatives.

Coercions: auto-slicing

One possibility would be re-introducing some kind of coercion that automatically slices. We used to have a coercion from (in today’s terms) Vec<T> to &[T]. Since we no longer coerce owned to borrowed values, we’d probably want a coercion &Vec<T> to &[T] now:

fn use_slice(t: &[u8]) { ... }

let v = vec!(0u8, 1, 2);
use_slice(&v)           // automatically coerce here
use_slice(v.as_slice()) // equivalent

Unfortunately, adding such a coercion requires choosing between the following:

  • Tie the coercion to Vec and String. This would reintroduce special treatment of these otherwise purely library types, and would mean that other library types that support slicing would not benefit (defeating some of the purpose of DST).

  • Make the coercion extensible, via a trait. This is opening pandora’s box, however: the mechanism could likely be (ab)used to run arbitrary code during coercion, so that any invocation foo(a, b, c) might involve running code to pre-process each of the arguments. While we may eventually want such user-extensible coercions, it is a big step to take with a lot of potential downside when reasoning about code, so we should pursue more conservative solutions first.

Deref

Another possibility would be to make String implement Deref<str> and Vec<T> implement Deref<[T]>, once DST lands. Doing so would allow explicit coercions like:

fn use_slice(t: &[u8]) { ... }

let v = vec!(0u8, 1, 2);
use_slice(&*v)          // take advantage of deref
use_slice(v.as_slice()) // equivalent

There are at least two downsides to doing so, however:

  • It is not clear how the method resolution rules will ultimately interact with Deref. In particular, a leading proposal is that for a smart pointer s: Smaht<T> when you invoke s.m(...) only inherent methods m are considered for Smaht<T>; trait methods are only considered for the maximally-derefed value *s.

    With such a resolution strategy, implementing Deref for Vec would make it impossible to use trait methods on the Vec type except through UFCS, severely limiting the ability of programmers to usefully implement new traits for Vec.

  • The idea of Vec as a smart pointer around a slice, and the use of &*v as above, is somewhat counterintuitive, especially for such a basic type.

Ultimately, notation for slicing seems desirable on its own merits anyway, and if it can eliminate the need to implement Deref for Vec and String, all the better.

Summary

This is a conventions RFC for settling naming conventions when there are by value, by reference, and by mutable reference variants of an operation.

Motivation

Currently the libraries are not terribly consistent about how to signal mut variants of functions; sometimes it is by a mut_ prefix, sometimes a _mut suffix, and occasionally with _mut_ appearing in the middle. These inconsistencies make APIs difficult to remember.

While there are arguments in favor of each of the positions, we stand to gain a lot by standardizing, and to some degree we just need to make a choice.

Detailed design

Functions often come in multiple variants: immutably borrowed, mutably borrowed, and owned.

The canonical example is iterator methods:

  • iter works with immutably borrowed data
  • mut_iter works with mutably borrowed data
  • move_iter works with owned data

For iterators, the “default” (unmarked) variant is immutably borrowed. In other cases, the default is owned.

The proposed rules depend on which variant is the default, but use suffixes to mark variants in all cases.

The rules

Immutably borrowed by default

If foo uses/produces an immutable borrow by default, use:

  • The _mut suffix (e.g. foo_mut) for the mutably borrowed variant.
  • The _move suffix (e.g. foo_move) for the owned variant.

However, in the case of iterators, the moving variant can also be understood as an into conversion, into_iter, and for x in v.into_iter() reads arguably better than for x in v.iter_move(), so the convention is into_iter.

NOTE: This convention covers only the method names for iterators, not the names of the iterator types. That will be the subject of a follow up RFC.

Owned by default

If foo uses/produces owned data by default, use:

  • The _ref suffix (e.g. foo_ref) for the immutably borrowed variant.
  • The _mut suffix (e.g. foo_mut) for the mutably borrowed variant.

Exceptions

For mutably borrowed variants, if the mut qualifier is part of a type name (e.g. as_mut_slice), it should appear as it would appear in the type.

References to type names

Some places in the current libraries, we say things like as_ref and as_mut, and others we say get_ref and get_mut_ref.

Proposal: generally standardize on mut as a shortening of mut_ref.

The rationale

Why suffixes?

Using a suffix makes it easier to visually group variants together, especially when sorted alphabetically. It puts the emphasis on the functionality, rather than the qualifier.

Why move?

Historically, Rust has used move as a way to signal ownership transfer and to connect to C++ terminology. The main disadvantage is that it does not emphasize ownership, which is our current narrative. On the other hand, in Rust all data is owned, so using _owned as a qualifier is a bit strange.

The Copy trait poses a problem for any terminology about ownership transfer. The proposed mental model is that with Copy data you are “moving a copy”.

See Alternatives for more discussion.

Why mut rather then mut_ref?

It’s shorter, and pairs like as_ref and as_mut have a pleasant harmony that doesn’t place emphasis on one kind of reference over the other.

Alternatives

Prefix or mixed qualifiers

Using prefixes for variants is another possibility, but there seems to be little upside.

It’s possible to rationalize our current mix of prefixes and suffixes via grammatical distinctions, but this seems overly subtle and complex, and requires a strong command of English grammar to work well.

No suffix exception

The rules here make an exception when mut is part of a type name, as in as_mut_slice, but we could instead always place the qualifier as a suffix: as_slice_mut. This would make APIs more consistent in some ways, less in others: conversion functions would no longer consistently use a transcription of their type name.

This is perhaps not so bad, though, because as it is we often abbreviate type names. In any case, we need a convention (separate RFC) for how to refer to type names in methods.

owned instead of move

The overall narrative about Rust has been evolving to focus on ownership as the essential concept, with borrowing giving various lesser forms of ownership, so _owned would be a reasonable alternative to _move.

On the other hand, the ref variants do not say “borrowed”, so in some sense this choice is inconsistent. In addition, the terminology is less familiar to those coming from C++.

val instead of owned

Another option would be val or value instead of owned. This suggestion plays into the “by reference” and “by value” distinction, and so is even more congruent with ref than move is. On the other hand, it’s less clear/evocative than either move or owned.

Summary

This RFC improves interoperation between APIs with different error types. It proposes to:

  • Increase the flexibility of the try! macro for clients of multiple libraries with disparate error types.

  • Standardize on basic functionality that any error type should have by introducing an Error trait.

  • Support easy error chaining when crossing abstraction boundaries.

The proposed changes are all library changes; no language changes are needed – except that this proposal depends on multidispatch happening.

Motivation

Typically, a module (or crate) will define a custom error type encompassing the possible error outcomes for the operations it provides, along with a custom Result instance baking in this type. For example, we have io::IoError and io::IoResult<T> = Result<T, io::IoError>, and similarly for other libraries. Together with the try! macro, the story for interacting with errors for a single library is reasonably good.

However, we lack infrastructure when consuming or building on errors from multiple APIs, or abstracting over errors.

Consuming multiple error types

Our current infrastructure for error handling does not cope well with mixed notions of error.

Abstractly, as described by this issue, we cannot do the following:

fn func() -> Result<T, Error> {
    try!(may_return_error_type_A());
    try!(may_return_error_type_B());
}

Concretely, imagine a CLI application that interacts both with files and HTTP servers, using std::io and an imaginary http crate:

fn download() -> Result<(), CLIError> {
    let contents = try!(http::get(some_url));
    let file = try!(File::create(some_path));
    try!(file.write_str(contents));
    Ok(())
}

The download function can encounter both io and http errors, and wants to report them both under the common notion of CLIError. But the try! macro only works for a single error type at a time.

There are roughly two scenarios where multiple library error types need to be coalesced into a common type, each with different needs: application error reporting, and library error reporting

Application error reporting: presenting errors to a user

An application is generally the “last stop” for error handling: it’s the point at which remaining errors are presented to the user in some form, when they cannot be handled programmatically.

As such, the data needed for application-level errors is usually related to human interaction. For a CLI application, a short text description and longer verbose description are usually all that’s needed. For GUI applications, richer data is sometimes required, but usually not a full enum describing the full range of errors.

Concretely, then, for something like the download function above, for a CLI application, one might want CLIError to roughly be:

struct CLIError<'a> {
    description: &'a str,
    detail: Option<String>,
    ... // possibly more fields here; see detailed design
}

Ideally, one could use the try! macro as in the download example to coalesce a variety of error types into this single, simple struct.

Library error reporting: abstraction boundaries

When one library builds on others, it needs to translate from their error types to its own. For example, a web server framework may build on a library for accessing a SQL database, and needs some way to “lift” SQL errors to its own notion of error.

In general, a library may not want to reveal the upstream libraries it relies on – these are implementation details which may change over time. Thus, it is critical that the error type of upstream libraries not leak, and “lifting” an error from one library to another is a way of imposing an abstraction boundaries.

In some cases, the right way to lift a given error will depend on the operation and context. In other cases, though, there will be a general way to embed one kind of error in another (usually via a “cause chain”). Both scenarios should be supported by Rust’s error handling infrastructure.

Abstracting over errors

Finally, libraries sometimes need to work with errors in a generic way. For example, the serialize::Encoder type takes is generic over an arbitrary error type E. At the moment, such types are completely arbitrary: there is no Error trait giving common functionality expected of all errors. Consequently, error-generic code cannot meaningfully interact with errors.

(See this issue for a concrete case where a bound would be useful; note, however, that the design below does not cover this use-case, as explained in Alternatives.)

Languages that provide exceptions often have standard exception classes or interfaces that guarantee some basic functionality, including short and detailed descriptions and “causes”. We should begin developing similar functionality in libstd to ensure that we have an agreed-upon baseline error API.

Detailed design

We can address all of the problems laid out in the Motivation section by adding some simple library code to libstd, so this RFC will actually give a full implementation.

Note, however, that this implementation relies on the multidispatch proposal currently under consideration.

The proposal consists of two pieces: a standardized Error trait and extensions to the try! macro.

The Error trait

The standard Error trait follows very the widespread pattern found in Exception base classes in many languages:

pub trait Error: Send + Any {
    fn description(&self) -> &str;

    fn detail(&self) -> Option<&str> { None }
    fn cause(&self) -> Option<&Error> { None }
}

Every concrete error type should provide at least a description. By making this a slice-returning method, it is possible to define lightweight enum error types and then implement this method as returning static string slices depending on the variant.

The cause method allows for cause-chaining when an error crosses abstraction boundaries. The cause is recorded as a trait object implementing Error, which makes it possible to read off a kind of abstract backtrace (often more immediately helpful than a full backtrace).

The Any bound is needed to allow downcasting of errors. This RFC stipulates that it must be possible to downcast errors in the style of the Any trait, but leaves unspecified the exact implementation strategy. (If trait object upcasting was available, one could simply upcast to Any; otherwise, we will likely need to duplicate the downcast APIs as blanket impls on Error objects.)

It’s worth comparing the Error trait to the most widespread error type in libstd, IoError:

pub struct IoError {
    pub kind: IoErrorKind,
    pub desc: &'static str,
    pub detail: Option<String>,
}

Code that returns or asks for an IoError explicitly will be able to access the kind field and thus react differently to different kinds of errors. But code that works with a generic Error (e.g., application code) sees only the human-consumable parts of the error. In particular, application code will often employ Box<Error> as the error type when reporting errors to the user. The try! macro support, explained below, makes doing so ergonomic.

An extended try! macro

The other piece to the proposal is a way for try! to automatically convert between different types of errors.

The idea is to introduce a trait FromError<E> that says how to convert from some lower-level error type E to Self. The try! macro then passes the error it is given through this conversion before returning:

// E here is an "input" for dispatch, so conversions from multiple error
// types can be provided
pub trait FromError<E> {
    fn from_err(err: E) -> Self;
}

impl<E> FromError<E> for E {
    fn from_err(err: E) -> E {
        err
    }
}

impl<E: Error> FromError<E> for Box<Error> {
    fn from_err(err: E) -> Box<Error> {
        box err as Box<Error>
    }
}

macro_rules! try (
    ($expr:expr) => ({
        use error;
        match $expr {
            Ok(val) => val,
            Err(err) => return Err(error::FromError::from_err(err))
        }
    })
)

This code depends on multidispatch, because the conversion depends on both the source and target error types. (In today’s Rust, the two implementations of FromError given above would be considered overlapping.)

Given the blanket impl of FromError<E> for E, all existing uses of try! would continue to work as-is.

With this infrastructure in place, application code can generally use Box<Error> as its error type, and try! will take care of the rest:

fn download() -> Result<(), Box<Error>> {
    let contents = try!(http::get(some_url));
    let file = try!(File::create(some_path));
    try!(file.write_str(contents));
    Ok(())
}

Library code that defines its own error type can define custom FromError implementations for lifting lower-level errors (where the lifting should also perform cause chaining) – at least when the lifting is uniform across the library. The effect is that the mapping from one error type into another only has to be written one, rather than at every use of try!:

impl FromError<ErrorA> MyError { ... }
impl FromError<ErrorB> MyError { ... }

fn my_lib_func() -> Result<T, MyError> {
    try!(may_return_error_type_A());
    try!(may_return_error_type_B());
}

Drawbacks

The main drawback is that the try! macro is a bit more complicated.

Unresolved questions

Conventions

This RFC does not define any particular conventions around cause chaining or concrete error types. It will likely take some time and experience using the proposed infrastructure before we can settle these conventions.

Extensions

The functionality in the Error trait is quite minimal, and should probably grow over time. Some additional functionality might include:

Features on the Error trait

  • Generic creation of Errors. It might be useful for the Error trait to expose an associated constructor. See this issue for an example where this functionality would be useful.

  • Mutation of Errors. The Error trait could be expanded to provide setters as well as getters.

The main reason not to include the above two features is so that Error can be used with extremely minimal data structures, e.g. simple enums. For such data structures, it’s possible to produce fixed descriptions, but not mutate descriptions or other error properties. Allowing generic creation of any Error-bounded type would also require these enums to include something like a GenericError variant, which is unfortunate. So for now, the design sticks to the least common denominator.

Concrete error types

On the other hand, for code that doesn’t care about the footprint of its error types, it may be useful to provide something like the following generic error type:

pub struct WrappedError<E> {
    pub kind: E,
    pub description: String,
    pub detail: Option<String>,
    pub cause: Option<Box<Error>>
}

impl<E: Show> WrappedError<E> {
    pub fn new(err: E) {
        WrappedErr {
            kind: err,
            description: err.to_string(),
            detail: None,
            cause: None
        }
    }
}

impl<E> Error for WrappedError<E> {
    fn description(&self) -> &str {
        self.description.as_slice()
    }
    fn detail(&self) -> Option<&str> {
        self.detail.as_ref().map(|s| s.as_slice())
    }
    fn cause(&self) -> Option<&Error> {
        self.cause.as_ref().map(|c| &**c)
    }
}

This type can easily be added later, so again this RFC sticks to the minimal functionality for now.

Summary

Change syntax of subslices matching from ..xs to xs.. to be more consistent with the rest of the language and allow future backwards compatible improvements.

Small example:

match slice {
    [xs.., _] => xs,
    [] => fail!()
}

This is basically heavily stripped version of RFC 101.

Motivation

In Rust, symbol after .. token usually describes number of things, as in [T, ..N] type or in [e, ..N] expression. But in following pattern: [_, ..xs], xs doesn’t describe any number, but the whole subslice.

I propose to move dots to the right for several reasons (including one mentioned above):

  1. Looks more natural (but that might be subjective).
  2. Consistent with the rest of the language.
  3. C++ uses args... in variadic templates.
  4. It allows extending slice pattern matching as described in RFC 101.

Detailed design

Slice matching grammar would change to (assuming trailing commas; grammar syntax as in Rust manual):

slice_pattern : "[" [[pattern | subslice_pattern] ","]* "]" ;
subslice_pattern : ["mut"? ident]? ".." ["@" slice_pattern]? ;

To compare, currently it looks like:

slice_pattern : "[" [[pattern | subslice_pattern] ","]* "]" ;
subslice_pattern : ".." ["mut"? ident ["@" slice_pattern]?]? ;

Drawbacks

Backward incompatible.

Alternatives

Don’t do it at all.

Unresolved questions

Whether subslice matching combined with @ should be written as xs.. @[1, 2] or maybe in another way: xs @[1, 2]...

Summary

Restore the integer inference fallback that was removed. Integer literals whose type is unconstrained will default to i32, unlike the previous fallback to int. Floating point literals will default to f64.

Motivation

History lesson

Rust has had a long history with integer and floating-point literals. Initial versions of Rust required all literals to be explicitly annotated with a suffix (if no suffix is provided, then int or float was used; note that the float type has since been removed). This meant that, for example, if one wanted to count up all the numbers in a list, one would write 0u and 1u so as to employ unsigned integers:

let mut count = 0u; // let `count` be an unsigned integer
while cond() {
    ...
    count += 1u;    // `1u` must be used as well
}

This was particularly troublesome with arrays of integer literals, which could be quite hard to read:

let byte_array = [0u8, 33u8, 50u8, ...];

It also meant that code which was very consciously using 32-bit or 64-bit numbers was hard to read.

Therefore, we introduced integer inference: unlabeled integer literals are not given any particular integral type rather a fresh “integral type variable” (floating point literals work in an analogous way). The idea is that the vast majority of literals will eventually interact with an actual typed variable at some point, and hence we can infer what type they ought to have. For those cases where the type cannot be automatically selected, we decided to fallback to our older behavior, and have integer/float literals be typed as int/float (this is also what Haskell does). Some time later, we did various measurements and found that in real world code this fallback was rarely used. Therefore, we decided that to remove the fallback.

Experience with lack of fallback

Unfortunately, when doing the measurements that led us to decide to remove the int fallback, we neglected to consider coding “in the small” (specifically, we did not include tests in the measurements). It turns out that when writing small programs, which includes not only “hello world” sort of things but also tests, the lack of integer inference fallback is quite annoying. This is particularly troublesome since small program are often people’s first exposure to Rust. The problems most commonly occur when integers are “consumed” by printing them out to the screen or by asserting equality, both of which are very common in small programs and testing.

There are at least three common scenarios where fallback would be beneficial:

Accumulator loops. Here a counter is initialized to 0 and then incremented by 1. Eventually it is printed or compared against a known value.

let mut c = 0;
loop {
    ...;
    c += 1;
}
println!("{}", c); // Does not constrain type of `c`
assert_eq(c, 22);

Calls to range with constant arguments. Here a call to range like range(0, 10) is used to execute something 10 times. It is important that the actual counter is either unused or only used in a print out or comparison against another literal:

for _ in range(0, 10) {
}

Large constants. In small tests it is convenient to make dummy test data. This frequently takes the form of a vector or map of ints.

let mut m = HashMap::new();
m.insert(1, 2);
m.insert(3, 4);
assert_eq(m.find(&3).map(|&i| i).unwrap(), 4);

Lack of bugs

To our knowledge, there has not been a single bug exposed by removing the fallback to the int type. Moreover, such bugs seem to be extremely unlikely.

The primary reason for this is that, in production code, the i32 fallback is very rarely used. In a sense, the same measurements that were used to justify removing the int fallback also justify keeping it. As the measurements showed, the vast, vast majority of integer literals wind up with a constrained type, unless they are only used to print out and do assertions with. Specifically, any integer that is passed as a parameter, returned from a function, or stored in a struct or array, must wind up with a specific type.

Rationale for the choice of defaulting to i32

In contrast to the first revision of the RFC, the fallback type suggested is i32. This is justified by a case analysis which showed that there does not exist a compelling reason for having a signed pointer-sized integer type as the default.

There are reasons for using i32 instead: It’s familiar to programmers from the C programming language (where the default int type is 32-bit in the major calling conventions), it’s faster than 64-bit integers in arithmetic today, and is superior in memory usage while still providing a reasonable range of possible values.

To expand on the performance argument: i32 obviously uses half of the memory of i64 meaning half the memory bandwidth used, half as much cache consumption and twice as much vectorization – additionally arithmetic (like multiplication and division) is faster on some of the modern CPUs.

Case analysis

This is an analysis of cases where int inference might be thought of as useful:

Indexing into an array with unconstrained integer literal:

let array = [0u8, 1, 2, 3];
let index = 3;
array[index]

In this case, index is already automatically inferred to be a uint.

Using a default integer for tests, tutorials, etc.: Examples of this include “The Guide”, the Rust API docs and the Rust standard library unit tests. This is better served by a smaller, faster and platform independent type as default.

Using an integer for an upper bound or for simply printing it: This is also served very well by i32.

Counting of loop iterations: This is a part where int is as badly suited as i32, so at least the move to i32 doesn’t create new hazards (note that the number of elements of a vector might not necessarily fit into an int).

In addition to all the points above, having a platform-independent type obviously results in less differences between the platforms in which the programmer “doesn’t care” about the integer type they are using.

Future-proofing for overloaded literals

It is possible that, in the future, we will wish to allow vector and strings literals to be overloaded so that they can be resolved to user-defined types. In that case, for backwards compatibility, it will be necessary for those literals to have some sort of fallback type. (This is a relatively weak consideration.)

Detailed design

Integral literals are currently type-checked by creating a special class of type variable. These variables are subject to unification as normal, but can only unify with integral types. This RFC proposes that, at the end of type inference, when all constraints are known, we will identify all integral type variables that have not yet been bound to anything and bind them to i32. Similarly, floating point literals will fallback to f64.

For those who wish to be very careful about which integral types they employ, a new lint (unconstrained_literal) will be added which defaults to allow. This lint is triggered whenever the type of an integer or floating point literal is unconstrained.

Downsides

Although there seems to be little motivation for int to be the default, there might be use cases where int is a more correct fallback than i32.

Additionally, it might seem weird to some that i32 is a default, when int looks like the default from other languages. The name of int however is not in the scope of this RFC.

Alternatives

  • No fallback. Status quo.

  • Fallback to something else. We could potentially fallback to int like the original RFC suggested or some other integral type rather than i32.

  • Fallback in a more narrow range of cases. We could attempt to identify integers that are “only printed” or “only compared”. There is no concrete proposal in this direction and it seems to lead to an overly complicated design.

  • Default type parameters influencing inference. There is a separate, follow-up proposal being prepared that uses default type parameters to influence inference. This would allow some examples, like range(0, 10) to work even without integral fallback, because the range function itself could specify a fallback type. However, this does not help with many other examples.

History

2014-11-07: Changed the suggested fallback from int to i32, add rationale.

Summary

Rust currently includes feature-gated support for type parameters that specify a default value. This feature is not well-specified. The aim of this RFC is to fully specify the behavior of defaulted type parameters:

  1. Type parameters in any position can specify a default.
  2. Within fn bodies, defaulted type parameters are used to drive inference.
  3. Outside of fn bodies, defaulted type parameters supply fixed defaults.
  4. _ can be used to omit the values of type parameters and apply a suitable default:
    • In a fn body, any type parameter can be omitted in this way, and a suitable type variable will be used.
    • Outside of a fn body, only defaulted type parameters can be omitted, and the specified default is then used.

Points 2 and 4 extend the current behavior of type parameter defaults, aiming to address some shortcomings of the current implementation.

This RFC would remove the feature gate on defaulted type parameters.

Motivation

Why defaulted type parameters

Defaulted type parameters are very useful in two main scenarios:

  1. Extended a type without breaking existing clients.
  2. Allowing customization in ways that many or most users do not care about.

Often, these two scenarios occur at the same time. A classic historical example is the HashMap type from Rust’s standard library. This type now supports the ability to specify custom hashers. For most clients, this is not particularly important and this initial versions of the HashMap type were not customizable in this regard. But there are some cases where having the ability to use a custom hasher can make a huge difference. Having the ability to specify defaults for type parameters allowed the HashMap type to add a new type parameter H representing the hasher type without breaking any existing clients and also without forcing all clients to specify what hasher to use.

However, customization occurs in places other than types. Consider the function range(). In early versions of Rust, there was a distinct range function for each integral type (e.g. uint::range, int::range, etc). These functions were eventually consolidated into a single range() function that is defined generically over all “enumerable” types:

trait Enumerable : Add<Self,Self> + PartialOrd + Clone + One;
pub fn range<A:Enumerable>(start: A, stop: A) -> Range<A> {
    Range{state: start, stop: stop, one: One::one()}
}

This version is often more convenient to use, particularly in a generic context.

However, the generic version does have the downside that when the bounds of the range are integral, inference sometimes lacks enough information to select a proper type:

// ERROR -- Type argument unconstrained, what integral type did you want?
for x in range(0, 10) { ... }

Thus users are forced to write:

for x in range(0u, 10u) { ... }

This RFC describes how to integrate default type parameters with inference such that the type parameter on range can specify a default (uint, for example):

pub fn range<A:Enumerable=uint>(start: A, stop: A) -> Range<A> {
    Range{state: start, stop: stop, one: One::one()}
}

Using this definition, a call like range(0, 10) is perfectly legal. If it turns out that the type argument is not other constraint, uint will be used instead.

Extending types without breaking clients.

Without defaults, once a library is released to “the wild”, it is not possible to add type parameters to a type without breaking all existing clients. However, it frequently happens that one wants to take an existing type and make it more flexible that it used to be. This often entails adding a new type parameter so that some type which was hard-coded before can now be customized. Defaults provide a means to do this while having older clients transparently fallback to the older behavior.

Historical example: Extending HashMap to support various hash algorithms.

Detailed Design

Remove feature gate

This RFC would remove the feature gate on defaulted type parameters.

Type parameters with defaults

Defaults can be placed on any type parameter, whether it is declared on a type definition (struct, enum), type alias (type), trait definition (trait), trait implementation (impl), or a function or method (fn).

Once a given type parameter declares a default value, all subsequent type parameters in the list must declare default values as well:

// OK. All defaulted type parameters come at the end.
fn foo<A,B=uint,C=uint>() { .. }

// ERROR. B has a default, but C does not.
fn foo<A,B=uint,C>() { .. }

The default value of a type parameter X may refer to other type parameters declared on the same item. However, it may only refer to type parameters declared before X in the list of type parameters:

// OK. Default value of `B` refers to `A`, which is not defaulted.
fn foo<A,B=A>() { .. }

// OK. Default value of `C` refers to `B`, which comes before
// `C` in the list of parameters.
fn foo<A,B=uint,C=B>() { .. }

// ERROR. Default value of `B` refers to `C`, which comes AFTER
// `B` in the list of parameters.
fn foo<A,B=C,C=uint>() { .. }

Instantiating defaults

This section specifies how to interpret a reference to a generic type. Rather than writing out a rather tedious (and hard to understand) description of the algorithm, the rules are instead specified by a series of examples. The high-level idea of the rules is as follows:

  • Users must always provide some value for non-defaulted type parameters. Defaulted type parameters may be omitted.
  • The _ notation can always be used to explicitly omit the value of a type parameter:
    • Inside a fn body, any type parameter may be omitted. Inference is used.
    • Outside a fn body, only defaulted type parameters may be omitted. The default value is used.
    • Motivation: This is consistent with Rust tradition, which generally requires explicit types or a mechanical defaulting process outside of fn bodies.

References to generic types

We begin with examples of references to the generic type Foo:

struct Foo<A,B,C=DefaultHasher,D=C> { ... }

Foo defines four type parameters, the final two of which are defaulted. First, let us consider what happens outside of a fn body. It is mandatory to supply explicit values for all non-defaulted type parameters:

// ERROR: 2 parameters required, 0 provided.
fn f(_: &Foo) { ... }

Defaulted type parameters are filled in based on the defaults given:

// Legal: Equivalent to `Foo<int,uint,DefaultHasher,DefaultHasher>`
fn f(_: &Foo<int,uint>) { ... }

Naturally it is legal to specify explicit values for the defaulted type parameters if desired:

// Legal: Equivalent to `Foo<int,uint,uint,char,u8>`
fn f(_: &Foo<int,uint,char,u8>) { ... }

It is also legal to provide just one of the defaulted type parameters and not the other:

// Legal: Equivalent to `Foo<int,uint,char,char>`
fn f(_: &Foo<int,uint,char>) { ... }

If the user wishes to supply the value of the type parameter D explicitly, but not C, then _ can be used to request the default:

// Legal: Equivalent to `Foo<int,uint,DefaultHasher,uint>`
fn f(_: &Foo<int,uint,_,uint>) { ... }

Note that, outside of a fn body, _ can only be used with defaulted type parameters:

// ERROR: outside of a fn body, `_` cannot be
// used for a non-defaulted type parameter
fn f(_: &Foo<int,_>) { ... }

Inside a fn body, the rules are much the same, except that _ is legal everywhere. Every reference to _ creates a fresh type variable $n. If the type parameter whose value is omitted has an associate default, that default is used as the fallback for $n (see the section “Type variables with fallbacks” for more information). Here are some examples:

fn f() {
    // Error: `Foo` requires at least 2 type parameters, 0 supplied.
    let x: Foo = ...;

    // All of these 4 examples are OK and equivalent. Each
    // results in a type `Foo<$0,$1,$2,$3>` and `$0`-`$4` are type
    // variables. `$2` has a fallback of `DefaultHasher` and `$3`
    // has a fallback of `$2`.
    let x: Foo<_,_> = ...;
    let x: Foo<_,_,_> = ...;
    let x: Foo<_,_,_,_> = ...;

    // Results in a type `Foo<int,uint,$0,char>` where `$0`
    // has a fallback of `DefaultHasher`.
    let x: Foo<int,uint,_,char> = ...;
}

References to generic traits

The rules for traits are the same as the rules for types. Consider a trait Foo:

trait Foo<A,B,C=uint,D=C> { ... }

References to this trait can omit values for C and D in precisely the same way as was shown for types:

// All equivalent to Foo<i8,u8,uint,uint>:
fn foo<T:Foo<i8,u8>>() { ... }
fn foo<T:Foo<i8,u8,_>>() { ... }
fn foo<T:Foo<i8,u8,_,_>>() { ... }

// Equivalent to Foo<i8,u8,char,char>:
fn foo<T:Foo<i8,u8,char,_>>() { ... }

References to generic functions

The rules for referencing generic functions are the same as for types, except that it is legal to omit values for all type parameters if desired. In that case, the behavior is the same as it would be if _ were used as the value for every type parameter. Note that functions can only be referenced from within a fn body.

References to generic impls

Users never explicitly “reference” an impl. Rather, the trait matching system implicitly instantiates impls as part of trait matching. This implies that all type parameters are always instantiated with type variables. These type variables are assigned fallbacks according to the defaults given.

Type variables with fallbacks

We extend the inference system so that when a type variable is created, it can optionally have a fallback value, which is another type.

In the type checker, whenever we create a fresh type variable to represent a type parameter with an associated default, we will use that default as the fallback value for this type variable.

Example:

fn foo<A,B=A>(a: A, b: B) { ... }

fn bar() {
    // Here, the values of the type parameters are given explicitly.
    let f: fn(uint, uint) = foo::<uint, uint>;

    // Here the value of the first type parameter is given explicitly,
    // but not the second. Because the second specifies a default, this
    // is permitted. The type checker will create a fresh variable `$0`
    // and attempt to infer the value of this defaulted type parameter.
    let g: fn(uint, $0) = foo::<uint>;

    // Here, the values of the type parameters are not given explicitly,
    // and hence the type checker will create fresh variables
    // `$1` and `$2` for both of them.
    let h: fn($1, $2) = foo;
}

In this snippet, there are three references to the generic function foo, each of which specifies progressively fewer types. As a result, the type checker winds up creating three type variables, which are referred to in the example as $0, $1, and $2 (not that this $ notation is just for explanatory purposes and is not actual Rust syntax).

The fallback values of $0, $1, and $2 are as follows:

  • $0 was created to represent the type parameter B defined on foo. This means that $0 will have a fallback value of uint, since the type variable A was specified to be uint in the expression that created $0.
  • $1 was created to represent the type parameter A, which has no default. Therefore $1 has no fallback.
  • $2 was created to represent the type parameter B. It will have the fallback value of $1, which was the value of A within the expression where $2 was created.

Trait resolution, fallbacking, and inference

Prior to this RFC, type-checking a function body proceeds roughly as follows:

  1. The function body is analyzed. This results in an accumulated set of type variables, constraints, and trait obligations.
  2. Those trait obligations are then resolved until a fixed point is reached.
  3. If any trait obligations remain unresolved, an error is reported.
  4. If any type variables were never bound to a concrete value, an error is reported.

To accommodate fallback, the new procedure is somewhat different:

  1. The function body is analyzed. This results in an accumulated set of type variables, constraints, and trait obligations.
  2. Execute in a loop:
  3. Run trait resolution until a fixed point is reached.
  4. Create a (initially empty) set UB of unbound type and integral/float variables. This set represents the set of variables for which fallbacks should be applied.
  5. Add all unbound integral and float variables to the set UB
  6. For each type variable X:
    • If X has no fallback defined, skip.
    • If X is not bound, add X to UB
    • If X is bound to an unbound integral variable I, add X to UB and remove I from UB (if present).
    • If X is bound to an unbound float variable F, add X to UB and remove F from UB (if present).
  7. If UB is the empty set, break out of the loop.
  8. For each member of UB:
    • If the member is an integral type variable I, set I to int.
    • If the member is a float variable F, set I to f64.
    • Otherwise, the member must be a variable X with a defined fallback. Set X to its fallback.
      • Note that this “set” operations can fail, which indicates conflicting defaults. A suitable error message should be given.
  9. If any type parameters still have no value assigned to them, report an error.
  10. If any trait obligations could not be resolved, report an error.

There are some subtle points to this algorithm:

When defaults are to be applied, we first gather up the set of variables that have applicable defaults (step 2.2) and then later unconditionally apply those defaults (step 2.4). In particular, we do not loop over each type variable, check whether it is unbound, and apply the default only if it is unbound. The reason for this is that it can happen that there are contradictory defaults and we want to ensure that this results in an error:

fn foo<F:Default=uint>() -> F { }
fn bar<B=int>(b: B) { }
fn baz() {
    // Here, F is instantiated with $0=uint
    let x: $0 = foo();

    // Here, B is instantiated with $1=uint, and constraint $0 <: $1 is added.
    bar(x);
}

In this example, two type variables are created. $0 is the value of F in the call to foo() and $1 is the value of B in the call to bar(). The fact that x, which has type $0, is passed as an argument to bar() will add the constraint that $0 <: $1, but at no point are any concrete types given. Therefore, once type checking is complete, we will apply defaults. Using the algorithm given above, we will determine that both $0 and $1 are unbound and have suitable defaults. We will then unify $0 with uint. This will succeed and, because $0 <: $1, cause $1 to be unified with uint. Next, we will try to unify $1 with its default, int. This will lead to an error. If we combined the checking of whether $1 was unbound with the unification with the default, we would have first unified $0 and then decided that $1 did not require unification.

In the general case, a loop is required to continue resolving traits and applying defaults in sequence. Resolving traits can lead to unifications, so it is clear that we must resolve all traits that we can before we apply any defaults. However, it is also true that adding defaults can create new trait obligations that must be resolved.

Here is an example where processing trait obligations creates defaults, and processing defaults created trait obligations:

trait Foo { }
trait Bar { }

impl<T:Bar=uint> Foo for Vec<T> { } // Impl 1
impl Bar for uint { } // Impl 2

fn takes_foo<F:Foo>(f: F) { }

fn main() {
    let x = Vec::new(); // x: Vec<$0>
    takes_foo(x); // adds oblig Vec<$0> : Foo
}

When we finish type checking main, we are left with a variable $0 and a trait obligation Vec<$0> : Foo. Processing the trait obligation selects the impl 1 as the way to fulfill this trait obligation. This results in:

  1. a new type variable $1, which represents the parameter T on the impl. $1 has a default, uint.
  2. the constraint that $0=$1.
  3. a new trait obligation $1 : Bar.

We cannot process the new trait obligation yet because the type variable $1 is still unbound. (We know that it is equated with $0, but we do not have any concrete types yet, just variables.) After trait resolution reaches a fixed point, defaults are applied. $1 is equated with uint which in turn propagates to $0. At this point, there is still an outstanding trait obligation uint : Bar. This trait obligation can be resolved to impl 2.

The previous example consisted of “1.5” iterations of the loop. That is, although trait resolution runs twice, defaults are only needed one time:

  1. Trait resolution executed to resolve Vec<$0> : Foo.
  2. Defaults were applied to unify $1 = $0 = uint.
  3. Trait resolution executed to resolve uint : Bar
  4. No more defaults to apply, done.

The next example does 2 full iterations of the loop.

trait Foo { }
trait Bar<U> { }
trait Baz { }

impl<U,T:Bar<U>=Vec<U>> Foo for Vec<T> { } // Impl 1
impl<V=uint> Bar for Vec<V> { } // Impl 2

fn takes_foo<F:Foo>(f: F) { }

fn main() {
    let x = Vec::new(); // x: Vec<$0>
    takes_foo(x); // adds oblig Vec<$0> : Foo
}

Here the process is as follows:

  1. Trait resolution executed to resolve Vec<$0> : Foo. The result is two fresh variables, $1 (for U) and $2=Vec<$1> (for $T), the constraint that $0=$2, and the obligation $2 : Bar<$1>.
  2. Defaults are applied to unify $2 = $0 = Vec<$1>.
  3. Trait resolution executed to resolve $2 : Bar<$1>. The result is a fresh variable $3=uint (for $V) and the constraint that $1=$3.
  4. Defaults are applied to unify $3 = $1 = uint.

It should be clear that one can create examples in this vein so as to require any number of loops.

Interaction with integer/float literal fallback. This RFC gives defaulted type parameters precedence over integer/float literal fallback. This seems preferable because such types can be more specific. Below are some examples. See also the alternatives section.

// Here the type of the integer literal 22 is inferred
// to `int` using literal fallback.
fn foo<T>(t: T) { ... }
foo(22)
// Here the type of the integer literal 22 is inferred
// to `uint` because the default on `T` overrides the
// standard integer literal fallback.
fn foo<T=uint>(t: T) { ... }
foo(22)
// Here the type of the integer literal 22 is inferred
// to `char`, leading to an error. This can be resolved
// by using an explicit suffix like `22i`.
fn foo<T=char>(t: T) { ... }
foo(22)

Termination. Any time that there is a loop, one must inquire after termination. In principle, the loop above could execute indefinitely. This is because trait resolution is not guaranteed to terminate – basically there might be a cycle between impls such that we continue creating new type variables and new obligations forever. The trait matching system already defends against this with a recursion counter. That same recursion counter is sufficient to guarantee termination even when the default mechanism is added to the mix. This is because the default mechanism can never itself create new trait obligations: it can only cause previous ambiguous trait obligations to now be matchable (because unbound variables become bound). But the actual need to iteration through the loop is still caused by trait matching generating recursive obligations, which have an associated depth limit.

Compatibility analysis

One of the major design goals of defaulted type parameters is to permit new parameters to be added to existing types or methods in a backwards compatible way. This remains possible under the current design.

Note though that adding a default to an existing type parameter can lead to type errors in clients. This can occur if clients were already relying on an inference fallback from some other source and there is now an ambiguity. Naturally clients can always fix this error by specifying the value of the type parameter in question manually.

Downsides and alternatives

Avoid inference

Rather than adding the notion of fallbacks to type variables, defaults could be mechanically added, even within fn bodies, as they are today. But this is disappointing because it means that examples like range(0,10), where defaults could inform inference, still require explicit annotation. Without the notion of fallbacks, it is also difficult to say what defaulted type parameters in methods or impls should mean.

More advanced interaction between integer literal inference

There were some other proposals to have a more advanced interaction between custom fallbacks and literal inference. For example, it is possible to imagine that we allow literal inference to take precedence over type default fallbacks, unless the fallback is itself integral. The problem is that this is both complicated and possibly not forwards compatible if we opt to allow a more general notion of literal inference in the future (in other words, if integer literals may be mapped to more than just the built-in integral types). Furthermore, these rules would create strictly fewer errors, and hence can be added in the future if desired.

Notation

Allowing _ notation outside of fn body means that it’s meaning changes somewhat depending on context. However, this is consistent with the meaning of omitted lifetimes, which also change in the same way (mechanical default outside of fn body, inference within).

An alternative design is to use the K=V notation proposed in the associated items RFC for specify the values of default type parameters. However, this is somewhat odd, because default type parameters appear in a positional list, and thus it is surprising that values for the non-defaulted parameters are given positionally, but values for the defaulted type parameters are given with labels.

Another alternative would to simply prohibit users from specifying the value of a defaulted type parameter unless values are given for all previous defaulted typed parameters. But this is clearly annoying in those cases where defaulted type parameters represent distinct axes of customization.

Hat Tip

eddyb introduced defaulted type parameters and also opened the first pull request that used them to inform inference.

Summary

Introduce a new while let PAT = EXPR { BODY } construct. This allows for using a refutable pattern match (with optional variable binding) as the condition of a loop.

Motivation

Just as if let was inspired by Swift, it turns out Swift supports while let as well. This was not discovered until much too late to include it in the if let RFC. It turns out that this sort of looping is actually useful on occasion. For example, the desugaring for loop is actually a variant on this; if while let existed it could have been implemented to map for PAT in EXPR { BODY } to

// the match here is so `for` can accept an rvalue for the iterator,
// and was used in the "real" desugaring version.
match &mut EXPR {
    i => {
        while let Some(PAT) = i.next() {
            BODY
        }
    }
}

(note that the non-desugared form of for is no longer equivalent).

More generally, this construct can be used any time looping + pattern-matching is desired.

This also makes the language a bit more consistent; right now, any condition that can be used with if can be used with while. The new if let adds a form of if that doesn’t map to while. Supporting while let restores the equivalence of these two control-flow constructs.

Detailed design

while let operates similarly to if let, in that it desugars to existing syntax. Specifically, the syntax

['ident:] while let PAT = EXPR {
    BODY
}

desugars to

['ident:] loop {
    match EXPR {
        PAT => BODY,
        _ => break
    }
}

Just as with if let, an irrefutable pattern given to while let is considered an error. This is largely an artifact of the fact that the desugared match ends up with an unreachable pattern, and is not actually a goal of this syntax. The error may be suppressed in the future, which would be a backwards-compatible change.

Just as with if let, while let will be introduced under a feature gate (named while_let).

Drawbacks

Yet another addition to the grammar. Unlike if let, it’s not obvious how useful this syntax will be.

Alternatives

As with if let, this could plausibly be done with a macro, but it would be ugly and produce bad error messages.

while let could be extended to support alternative patterns, just as match arms do. This is not part of the main proposal for the same reason it was left out of if let, which is that a) it looks weird, and b) it’s a bit of an odd coupling with the let keyword as alternatives like this aren’t going to be introducing variable bindings. However, it would make while let more general and able to replace more instances of loop { match { ... } } than is possible with the main design.

Unresolved questions

None.

  • Start Date: 2014-08-28
  • RFC PR: (https://github.com/rust-lang/rfcs/pull/216)
  • Rust Issue: (https://github.com/rust-lang/rust/issues/17320)

Summary

Add additional iterator-like Entry objects to collections. Entries provide a composable mechanism for in-place observation and mutation of a single element in the collection, without having to “re-find” the element multiple times. This deprecates several “internal mutation” methods like hashmap’s find_or_insert_with.

Motivation

As we approach 1.0, we’d like to normalize the standard APIs to be consistent, composable, and simple. However, this currently stands in opposition to manipulating the collections in an efficient manner. For instance, if one wishes to build an accumulating map on top of one of the concrete maps, they need to distinguish between the case when the element they’re inserting is already in the map, and when it’s not. One way to do this is the following:

if map.contains_key(&key) {
    *map.find_mut(&key).unwrap() += 1;
} else {
    map.insert(key, 1);
}

However, searches for key twice on every operation. The second search can be squeezed out the update re-do by matching on the result of find_mut, but the insert case will always require a re-search.

To solve this problem, Rust currently has an ad-hoc mix of “internal mutation” methods which take multiple values or closures for the collection to use contextually. Hashmap in particular has the following methods:

fn find_or_insert<'a>(&'a mut self, k: K, v: V) -> &'a mut V
fn find_or_insert_with<'a>(&'a mut self, k: K, f: |&K| -> V) -> &'a mut V
fn insert_or_update_with<'a>(&'a mut self, k: K, v: V, f: |&K, &mut V|) -> &'a mut V
fn find_with_or_insert_with<'a, A>(&'a mut self, k: K, a: A, found: |&K, &mut V, A|, not_found: |&K, A| -> V) -> &'a mut V

Not only are these methods fairly complex to use, but they’re over-engineered and combinatorially explosive. They all seem to return a mutable reference to the region accessed “just in case”, and find_with_or_insert_with takes a magic argument a to try to work around the fact that the two closures it requires can’t both close over the same value (even though only one will ever be called). find_with_or_insert_with is also actually performing the role of insert_with_or_update_with, suggesting that these aren’t well understood.

Rust has been in this position before: internal iteration. Internal iteration was (author’s note: I’m told) confusing and complicated. However the solution was simple: external iteration. You get all the benefits of internal iteration, but with a much simpler interface, and greater composability. Thus, this RFC proposes the same solution to the internal mutation problem.

Detailed design

A fully tested “proof of concept” draft of this design has been implemented on top of hashmap, as it seems to be the worst offender, while still being easy to work with. It sits as a pull request here.

All the internal mutation methods are replaced with a single method on a collection: entry. The signature of entry will depend on the specific collection, but generally it will be similar to the signature for searching in that structure. entry will in turn return an Entry object, which captures the state of a completed search, and allows mutation of the area.

For convenience, we will use the hashmap draft as an example.

/// Get an Entry for where the given key would be inserted in the map
pub fn entry<'a>(&'a mut self, key: K) -> Entry<'a, K, V>;

/// A view into a single occupied location in a HashMap
pub struct OccupiedEntry<'a, K, V>{ ... }

/// A view into a single empty location in a HashMap
pub struct VacantEntry<'a, K, V>{ ... }

/// A view into a single location in a HashMap
pub enum Entry<'a, K, V> {
    /// An occupied Entry
    Occupied(OccupiedEntry<'a, K, V>),
    /// A vacant Entry
    Vacant(VacantEntry<'a, K, V>),
}

Of course, the real meat of the API is in the Entry’s interface (impl details removed):

impl<'a, K, V> OccupiedEntry<'a, K, V> {
    /// Gets a reference to the value of this Entry
    pub fn get(&self) -> &V;

    /// Gets a mutable reference to the value of this Entry
    pub fn get_mut(&mut self) -> &mut V;

    /// Converts the entry into a mutable reference to its value
    pub fn into_mut(self) -> &'a mut V;

    /// Sets the value stored in this Entry
    pub fn set(&mut self, value: V) -> V;

    /// Takes the value stored in this Entry
    pub fn take(self) -> V;
}

impl<'a, K, V> VacantEntry<'a, K, V> {
    /// Set the value stored in this Entry, and returns a reference to it
    pub fn set(self, value: V) -> &'a mut V;
}

There are definitely some strange things here, so let’s discuss the reasoning!

First, entry takes a key by value, because this is the observed behaviour of the internal mutation methods. Further, taking the key up-front allows implementations to avoid validating provided keys if they require an owned key later for insertion. This key is effectively a guarantor of the entry.

Taking the key by-value might change once collections reform lands, and Borrow and ToOwned are available. For now, it’s an acceptable solution, because in particular, the primary use case of this functionality is when you’re not sure if you need to insert, in which case you should be prepared to insert. Otherwise, find_mut is likely sufficient.

The result is actually an enum, that will either be Occupied or Vacant. These two variants correspond to concrete types for when the key matched something in the map, and when the key didn’t, respectively.

If there isn’t a match, the user has exactly one option: insert a value using set, which will also insert the guarantor, and destroy the Entry. This is to avoid the costs of maintaining the structure, which otherwise isn’t particularly interesting anymore.

If there is a match, a more robust set of options is provided. get and get_mut provide access to the value found in the location. set behaves as the vacant variant, but without destroying the entry. It also yields the old value. take simply removes the found value, and destroys the entry for similar reasons as set.

Let’s look at how we one now writes insert_or_update:

There are two options. We can either do the following:

// cleaner, and more flexible if logic is more complex
let val = match map.entry(key) {
    Vacant(entry) => entry.set(0),
    Occupied(entry) => entry.into_mut(),
};
*val += 1;

or

// closer to the original, and more compact
match map.entry(key) {
    Vacant(entry) => { entry.set(1); },
    Occupied(mut entry) => { *entry.get_mut() += 1; },
}

Either way, one can now write something equivalent to the “intuitive” inefficient code, but it is now as efficient as the complex insert_or_update methods. In fact, this matches so closely to the inefficient manipulation that users could reasonable ignore Entries until performance becomes an issue, at which point it’s an almost trivial migration. Closures also aren’t needed to dance around the fact that one may want to avoid generating some values unless they have to, because that falls naturally out of normal control flow.

If you look at the actual patch that does this, you’ll see that Entry itself is exceptionally simple to implement. Most of the logic is trivial. The biggest amount of work was just capturing the search state correctly, and even that was mostly a cut-and-paste job.

With Entries, the gate is also opened for… adaptors! Really want insert_or_update back? That can be written on top of this generically with ease. However, such discussion is out-of-scope for this RFC. Adaptors can be tackled in a back-compat manner after this has landed, and usage is observed. Also, this proposal does not provide any generic trait for Entries, preferring concrete implementations for the time-being.

Drawbacks

  • More structs, and more methods in the short-term

  • More collection manipulation “modes” for the user to think about

  • insert_or_update_with is kind of convenient for avoiding the kind of boiler-plate found in the examples

Alternatives

  • Just put our foot down, say “no efficient complex manipulations”, and drop all the internal mutation stuff without a replacement.

  • Try to build out saner/standard internal manipulation methods.

  • Try to make this functionality a subset of Cursors, which would be effectively a bi-directional mut_iter where the returned references borrow the cursor preventing aliasing/safety issues, so that mutation can be performed at the location of the cursor. However, preventing invalidation would be more expensive, and it’s not clear that cursor semantics would make sense on e.g. a HashMap, as you can’t insert any key in any location.

  • This RFC originally [proposed a design without enums that was substantially more complex] (https://github.com/Gankro/rust/commit/6d6804a6d16b13d07934f0a217a3562384e55612). However it had some interesting ideas about Key manipulation, so we mention it here for historical purposes.

Unresolved questions

Naming bikesheds!

Summary

When a struct type S has no fields (a so-called “empty struct”), allow it to be defined via either struct S; or struct S {}. When defined via struct S;, allow instances of it to be constructed and pattern-matched via either S or S {}. When defined via struct S {}, require instances to be constructed and pattern-matched solely via S {}.

Motivation

Today, when writing code, one must treat an empty struct as a special case, distinct from structs that include fields. That is, one must write code like this:

struct S2 { x1: int, x2: int }
struct S0; // kind of different from the above.

let s2 = S2 { x1: 1, x2: 2 };
let s0 = S0; // kind of different from the above.

match (s2, s0) {
    (S2 { x1: y1, x2: y2 },
     S0) // you can see my pattern here
     => { println!("Hello from S2({}, {}) and S0", y1, y2); }
}

While this yields code that is relatively free of extraneous curly-braces, this special case handling of empty structs presents problems for two cases of interest: automatic code generators (including, but not limited to, Rust macros) and conditionalized code (i.e. code with cfg attributes; see the CFG problem appendix). The heart of the code-generator argument is: Why force all to-be-written code-generators and macros with special-case handling of the empty struct case (in terms of whether or not to include the surrounding braces), especially since that special case is likely to be forgotten (yielding a latent bug in the code generator).

The special case handling of empty structs is also a problem for programmers who actively add and remove fields from structs during development; such changes cause a struct to switch from being empty and non-empty, and the associated revisions of changing removing and adding curly braces is aggravating (both in effort revising the code, and also in extra noise introduced into commit histories).

This RFC proposes an approach similar to the one we used circa February 2013, when both S0 and S0 { } were accepted syntaxes for an empty struct. The parsing ambiguity that motivated removing support for S0 { } is no longer present (see the Ancient History appendix). Supporting empty braces in the syntax for empty structs is easy to do in the language now.

Detailed design

There are two kinds of empty structs: Braced empty structs and flexible empty structs. Flexible empty structs are a slight generalization of the structs that we have today.

Flexible empty structs are defined via the syntax struct S; (as today).

Braced empty structs are defined via the syntax struct S { } (“new”).

Both braced and flexible empty structs can be constructed via the expression syntax S { } (“new”). Flexible empty structs, as today, can also be constructed via the expression syntax S.

Both braced and flexible empty structs can be pattern-matched via the pattern syntax S { } (“new”). Flexible empty structs, as today, can also be pattern-matched via the pattern syntax S.

Braced empty struct definitions solely affect the type namespace, just like normal non-empty structs. Flexible empty structs affect both the type and value namespaces.

As a matter of style, using braceless syntax is preferred for constructing and pattern-matching flexible empty structs. For example, pretty-printer tools are encouraged to emit braceless forms if they know that the corresponding struct is a flexible empty struct. (Note that pretty printers that handle incomplete fragments may not have such information available.)

There is no ambiguity introduced by this change, because we have already introduced a restriction to the Rust grammar to force the use of parentheses to disambiguate struct literals in such contexts. (See Rust RFC 25).

The expectation is that when migrating code from a flexible empty struct to a non-empty struct, it can start by first migrating to a braced empty struct (and then have a tool indicate all of the locations where braces need to be added); after that step has been completed, one can then take the next step of adding the actual field.

Drawbacks

Some people like “There is only one way to do it.” But, there is precedent in Rust for violating “one way to do it” in favor of syntactic convenience or regularity; see the Precedent for flexible syntax in Rust appendix. Also, see the Always Require Braces alternative below.

I have attempted to summarize the previous discussion from RFC PR 147 in the Recent History appendix; some of the points there include drawbacks to this approach and to the Always Require Braces alternative.

Alternatives

Always Require Braces

Alternative 1: “Always Require Braces”. Specifically, require empty curly braces on empty structs. People who like the current syntax of curly-brace free structs can encode them this way: enum S0 { S0 } This would address all of the same issues outlined above. (Also, the author (pnkfelix) would be happy to take this tack.)

The main reason not to take this tack is that some people may like writing empty structs without braces, but do not want to switch to the unary enum version described in the previous paragraph. See “I wouldn’t want to force noisier syntax …” in the Recent History appendix.

Status quo

Alternative 2: Status quo. Macros and code-generators in general will need to handle empty structs as a special case. We may continue hitting bugs like CFG parse bug. Some users will be annoyed but most will probably cope.

Synonymous in all contexts

Alternative 3: An earlier version of this RFC proposed having struct S; be entirely synonymous with struct S { }, and the expression S { } be synonymous with S.

This was deemed problematic, since it would mean that S { } would put an entry into both the type and value namespaces, while S { x: int } would only put an entry into the type namespace. Thus the current draft of the RFC proposes the “flexible” versus “braced” distinction for empty structs.

Never synonymous

Alternative 4: Treat struct S; as requiring S at the expression and pattern sites, and struct S { } as requiring S { } at the expression and pattern sites.

This in some ways follows a principle of least surprise, but it also is really hard to justify having both syntaxes available for empty structs with no flexibility about how they are used. (Note again that one would have the option of choosing between enum S { S }, struct S;, or struct S { }, each with their own idiosyncrasies about whether you have to write S or S { }.) I would rather adopt “Always Require Braces” than “Never Synonymous”

Empty Tuple Structs

One might say “why are you including support for curly braces, but not parentheses?” Or in other words, “what about empty tuple structs?”

The code-generation argument could be applied to tuple-structs as well, to claim that we should allow the syntax S0(). I am less inclined to add a special case for that; I think tuple-structs are less frequently used (especially with many fields); they are largely for ad-hoc data such as newtype wrappers, not for code generators.

Note that we should not attempt to generalize this RFC as proposed to include tuple structs, i.e. so that given struct S0 {}, the expressions T0, T0 {}, and T0() would be synonymous. The reason is that given a tuple struct struct T2(int, int), the identifier T2 is already bound to a constructor function:

fn main() {
    #[deriving(Show)]
    struct T2(int, int);

    fn foo<S:std::fmt::Show>(f: |int, int| -> S) {
        println!("Hello from {} and {}", f(2,3), f(4,5));
    }
    foo(T2);
}

So if we were to attempt to generalize the leniency of this RFC to tuple structs, we would be in the unfortunate situation given struct T0(); of trying to treat T0 simultaneously as an instance of the struct and as a constructor function. So, the handling of empty structs proposed by this RFC does not generalize to tuple structs.

(Note that if we adopt alternative 1, Always Require Braces, then the issue of how tuple structs are handled is totally orthogonal – we could add support for struct T0() as a distinct type from struct S0 {}, if we so wished, or leave it aside.)

Unresolved questions

None

Appendices

The CFG problem

A program like this works today:

fn main() {
    #[deriving(Show)]
    struct Svaries {
        x: int,
        y: int,

        #[cfg(zed)]
        z: int,
    }

    let s = match () {
        #[cfg(zed)]      _ => Svaries { x: 3, y: 4, z: 5 },
        #[cfg(not(zed))] _ => Svaries { x: 3, y: 4 },
    };
    println!("Hello from {}", s)
}

Observe what happens when one modifies the above just a bit:

    struct Svaries {
        #[cfg(eks)]
        x: int,
        #[cfg(why)]
        y: int,

        #[cfg(zed)]
        z: int,
    }

Now, certain cfg settings yield an empty struct, even though it is surrounded by braces. Today this leads to a CFG parse bug when one attempts to actually construct such a struct.

If we want to support situations like this properly, we will probably need to further extend the cfg attribute so that it can be placed before individual fields in a struct constructor, like this:

// You cannot do this today,
// but maybe in the future (after a different RFC)
let s = Svaries {
    #[cfg(eks)] x: 3,
    #[cfg(why)] y: 4,
    #[cfg(zed)] z: 5,
};

Supporting such a syntax consistently in the future should start today with allowing empty braces as legal code. (Strictly speaking, it is not necessary that we add support for empty braces at the parsing level to support this feature at the semantic level. But supporting empty-braces in the syntax still seems like the most consistent path to me.)

Ancient History

A parsing ambiguity was the original motivation for disallowing the syntax S {} in favor of S for constructing an instance of an empty struct. The ambiguity and various options for dealing with it were well documented on the rust-dev thread. Both syntaxes were simultaneously supported at the time.

In particular, at the time that mailing list thread was created, the code match match x {} ... would be parsed as match (x {}) ..., not as (match x {}) ... (see Rust PR 5137); likewise, if x {} would be parsed as an if-expression whose test component is the struct literal x {}. Thus, at the time of Rust PR 5137, if the input to a match or if was an identifier expression, one had to put parentheses around the identifier to force it to be interpreted as input to the match/if, and not as a struct constructor.

Of the options for resolving this discussed on the mailing list thread, the one selected (removing S {} construction expressions) was chosen as the most expedient option.

At that time, the option of “Place a parser restriction on those contexts where { terminates the expression and say that struct literals cannot appear there unless they are in parentheses.” was explicitly not chosen, in favor of continuing to use the disambiguation rule in use at the time, namely that the presence of a label (e.g. S { a_label: ... }) was the way to distinguish a struct constructor from an identifier followed by a control block, and thus, “there must be one label.”

Naturally, if the construction syntax were to be disallowed, it made sense to also remove the struct S {} declaration syntax.

Things have changed since the time of that mailing list thread; namely, we have now adopted the aforementioned parser restriction Rust RFC 25. (The text of RFC 25 does not explicitly address match, but we have effectively expanded it to include a curly-brace delimited block of match-arms in the definition of “block”.) Today, one uses parentheses around struct literals in some contexts (such as for e in (S {x: 3}) { ... } or match (S {x: 3}) { ... }

Note that there was never an ambiguity for uses of struct S0 { } in item position. The issue was solely about expression position prior to the adoption of Rust RFC 25.

Precedent for flexible syntax in Rust

There is precedent in Rust for violating “one way to do it” in favor of syntactic convenience or regularity.

For example, one can often include an optional trailing comma, for example in: let x : &[int] = [3, 2, 1, ];.

One can also include redundant curly braces or parentheses, for example in:

println!("hi: {}", { if { x.len() > 2 } { ("whoa") } else { ("there") } });

One can even mix the two together when delimiting match arms:

    let z: int = match x {
        [3, 2] => { 3 }
        [3, 2, 1] => 2,
        _ => { 1 },
    };

We do have lints for some style violations (though none catch the cases above), but lints are different from fundamental language restrictions.

Recent history

There was a previous RFC PR that was effectively the same in spirit to this one. It was closed because it was not sufficient well fleshed out for further consideration by the core team. However, to save people the effort of reviewing the comments on that PR (and hopefully stave off potential bikeshedding on this PR), I here summarize the various viewpoints put forward on the comment thread there, and note for each one, whether that viewpoint would be addressed by this RFC (accept both syntaxes), by Always Require Braces, or by Status Quo.

Note that this list of comments is just meant to summarize the list of views; it does not attempt to reflect the number of commenters who agreed or disagreed with a particular point. (But since the RFC process is not a democracy, the number of commenters should not matter anyway.)

  • “+1” ==> Favors: This RFC (or potentially Always Require Braces; I think the content of RFC PR 147 shifted over time, so it is hard to interpret the “+1” comments now).
  • “I find let s = S0; jarring, think its an enum initially.” ==> Favors: Always Require Braces
  • “Frequently start out with an empty struct and add fields as I need them.” ==> Favors: This RFC or Always Require Braces
  • “Foo{} suggests is constructing something that it’s not; all uses of the value Foo are indistinguishable from each other” ==> Favors: Status Quo
  • “I find it strange anyone would prefer let x = Foo{}; over let x = Foo;” ==> Favors Status Quo; strongly opposes Always Require Braces.
  • “I agree that ‘instantiation-should-follow-declaration’, that is, structs declared ;, (), {} should only be instantiated [via] ;, (), { } respectively” ==> Opposes leniency of this RFC in that it allows expression to use include or omit {} on an empty struct, regardless of declaration form, and vice-versa.
  • “The code generation argument is reasonable, but I wouldn’t want to force noisier syntax on all ‘normal’ code just to make macros work better.” ==> Favors: This RFC

Summary

Rename “task failure” to “task panic”, and fail! to panic!.

Motivation

The current terminology of “task failure” often causes problems when writing or speaking about code. You often want to talk about the possibility of an operation that returns a Result “failing”, but cannot because of the ambiguity with task failure. Instead, you have to speak of “the failing case” or “when the operation does not succeed” or other circumlocutions.

Likewise, we use a “Failure” header in rustdoc to describe when operations may fail the task, but it would often be helpful to separate out a section describing the “Err-producing” case.

We have been steadily moving away from task failure and toward Result as an error-handling mechanism, so we should optimize our terminology accordingly: Result-producing functions should be easy to describe.

Detailed design

Not much more to say here than is in the summary: rename “task failure” to “task panic” in documentation, and fail! to panic! in code.

The choice of panic emerged from a discuss thread and workweek discussion. It has precedent in a language setting in Go, and of course goes back to Kernel panics.

With this choice, we can use “failure” to refer to an operation that produces Err or None, “panic” for unwinding at the task level, and “abort” for aborting the entire process.

The connotations of panic seem fairly accurate: the process is not immediately ending, but it is rapidly fleeing from some problematic circumstance (by killing off tasks) until a recovery point.

Drawbacks

The term “panic” is a bit informal, which some consider a drawback.

Making this change is likely to be a lot of work.

Alternatives

Other choices include:

  • throw! or unwind!. These options reasonably describe the current behavior of task failure, but “throw” suggests general exception handling, and both place the emphasis on the mechanism rather than the policy. We also are considering eventually adding a flag that allows fail! to abort the process, which would make these terms misleading.

  • abort!. Ambiguous with process abort.

  • die!. A reasonable choice, but it’s not immediately obvious what is being killed.

Summary

This RFC proposes to remove the runtime system that is currently part of the standard library, which currently allows the standard library to support both native and green threading. In particular:

  • The libgreen crate and associated support will be moved out of tree, into a separate Cargo package.

  • The librustrt (the runtime) crate will be removed entirely.

  • The std::io implementation will be directly welded to native threads and system calls.

  • The std::io module will remain completely cross-platform, though separate platform-specific modules may be added at a later time.

Motivation

Background: thread/task models and I/O

Many languages/libraries offer some notion of “task” as a unit of concurrent execution, possibly distinct from native OS threads. The characteristics of tasks vary along several important dimensions:

  • 1:1 vs M:N. The most fundamental question is whether a “task” always corresponds to an OS-level thread (the 1:1 model), or whether there is some userspace scheduler that maps tasks onto worker threads (the M:N model). Some kernels – notably, Windows – support a 1:1 model where the scheduling is performed in userspace, which combines some of the advantages of the two models.

    In the M:N model, there are various choices about whether and when blocked tasks can migrate between worker threads. One basic downside of the model, however, is that if a task takes a page fault, the entire worker thread is essentially blocked until the fault is serviced. Choosing the optimal number of worker threads is difficult, and some frameworks attempt to do so dynamically, which has costs of its own.

  • Stack management. In the 1:1 model, tasks are threads and therefore must be equipped with their own stacks. In M:N models, tasks may or may not need their own stack, but there are important tradeoffs:

    • Techniques like segmented stacks allow stack size to grow over time, meaning that tasks can be equipped with their own stack but still be lightweight. Unfortunately, segmented stacks come with a significant performance and complexity cost.

    • On the other hand, if tasks are not equipped with their own stack, they either cannot be migrated between underlying worker threads (the case for frameworks like Java’s fork/join), or else must be implemented using continuation-passing style (CPS), where each blocking operation takes a closure representing the work left to do. (CPS essentially moves the needed parts of the stack into the continuation closure.) The upside is that such tasks can be extremely lightweight – essentially just the size of a closure.

  • Blocking and I/O support. In the 1:1 model, a task can block freely without any risk for other tasks, since each task is an OS thread. In the M:N model, however, blocking in the OS sense means blocking the worker thread. (The same applies to long-running loops or page faults.)

    M:N models can deal with blocking in a couple of ways. The approach taken in Java’s fork/join framework, for example, is to dynamically spin up/down worker threads. Alternatively, special task-aware blocking operations (including I/O) can be provided, which are mapped under the hood to nonblocking operations, allowing the worker thread to continue. Unfortunately, this latter approach helps only with explicit blocking; it does nothing for loops, page faults and the like.

Where Rust is now

Rust has gradually migrated from a “green” threading model toward a native threading model:

  • In Rust’s green threading, tasks are scheduled M:N and are equipped with their own stack. Initially, Rust used segmented stacks to allow growth over time, but that was removed in favor of pre-allocated stacks, which means Rust’s green threads are not “lightweight”. The treatment of blocking is described below.

  • In Rust’s native threading model, tasks are 1:1 with OS threads.

Initially, Rust supported only the green threading model. Later, native threading was added and ultimately became the default.

In today’s Rust, there is a single I/O API – std::io – that provides blocking operations only and works with both threading models. Rust is somewhat unusual in allowing programs to mix native and green threading, and furthermore allowing some degree of interoperation between the two. This feat is achieved through the runtime system – librustrt – which exposes:

  • The Runtime trait, which abstracts over the scheduler (via methods like deschedule and spawn_sibling) as well as the entire I/O API (via local_io).

  • The rtio module, which provides a number of traits that define the standard I/O abstraction.

  • The Task struct, which includes a Runtime trait object as the dynamic entry point into the runtime.

In this setup, libstd works directly against the runtime interface. When invoking an I/O or scheduling operation, it first finds the current Task, and then extracts the Runtime trait object to actually perform the operation.

On native tasks, blocking operations simply block. On green tasks, blocking operations are routed through the green scheduler and/or underlying event loop and nonblocking I/O.

The actual scheduler and I/O implementations – libgreen and libnative – then live as crates “above” libstd.

The problems

While the situation described above may sound good in principle, there are several problems in practice.

Forced co-evolution. With today’s design, the green and native threading models must provide the same I/O API at all times. But there is functionality that is only appropriate or efficient in one of the threading models.

For example, the lightest-weight M:N task models are essentially just collections of closures, and do not provide any special I/O support. This style of lightweight tasks is used in Servo, but also shows up in java.util.concurrent’s exectors and Haskell’s par monad, among many others. These lighter weight models do not fit into the current runtime system.

On the other hand, green threading systems designed explicitly to support I/O may also want to provide low-level access to the underlying event loop – an API surface that doesn’t make sense for the native threading model.

Under the native model we want to provide direct non-blocking and/or asynchronous I/O support – as a systems language, Rust should be able to work directly with what the OS provides without imposing global abstraction costs. These APIs may involve some platform-specific abstractions (epoll, kqueue, IOCP) for maximal performance. But integrating them cleanly with a green threading model may be difficult or impossible – and at the very least, makes it difficult to add them quickly and seamlessly to the current I/O system.

In short, the current design couples threading and I/O models together, and thus forces the green and native models to supply a common I/O interface – despite the fact that they are pulling in different directions.

Overhead. The current Rust model allows runtime mixtures of the green and native models. The implementation achieves this flexibility by using trait objects to model the entire I/O API. Unfortunately, this flexibility has several downsides:

  • Binary sizes. A significant overhead caused by the trait object design is that the entire I/O system is included in any binary that statically links to libstd. See this comment for more details.

  • Task-local storage. The current implementation of task-local storage is designed to work seamlessly across native and green threads, and its performs substantially suffers as a result. While it is feasible to provide a more efficient form of “hybrid” TLS that works across models, doing so is far more difficult than simply using native thread-local storage.

  • Allocation and dynamic dispatch. With the current design, any invocation of I/O involves at least dynamic dispatch, and in many cases allocation, due to the use of trait objects. However, in most cases these costs are trivial when compared to the cost of actually doing the I/O (or even simply making a syscall), so they are not strong arguments against the current design.

Problematic I/O interactions. As the documentation for libgreen explains, only some I/O and synchronization methods work seamlessly across native and green tasks. For example, any invocation of native code that calls blocking I/O has the potential to block the worker thread running the green scheduler. In particular, std::io objects created on a native task cannot safely be used within a green task. Thus, even though std::io presents a unified I/O API for green and native tasks, it is not fully interoperable.

Embedding Rust. When embedding Rust code into other contexts – whether calling from C code or embedding in high-level languages – there is a fair amount of setup needed to provide the “runtime” infrastructure that libstd relies on. If libstd was instead bound to the native threading and I/O system, the embedding setup would be much simpler.

Maintenance burden. Finally, libstd is made somewhat more complex by providing such a flexible threading model. As this RFC will explain, moving to a strictly native threading model will allow substantial simplification and reorganization of the structure of Rust’s libraries.

Detailed design

To mitigate the above problems, this RFC proposes to tie std::io directly to the native threading model, while moving libgreen and its supporting infrastructure into an external Cargo package with its own I/O API.

The near-term plan

std::io and native threading

The plan is to entirely remove librustrt, including all of the traits. The abstraction layers will then become:

  • Highest level: libstd, providing cross-platform, high-level I/O and scheduling abstractions. The crate will depend on libnative (the opposite of today’s situation).

  • Mid-level: libnative, providing a cross-platform Rust interface for I/O and scheduling. The API will be relatively low-level, compared to libstd. The crate will depend on libsys.

  • Low-level: libsys (renamed from liblibc), providing platform-specific Rust bindings to system C APIs.

In this scheme, the actual API of libstd will not change significantly. But its implementation will invoke functions in libnative directly, rather than going through a trait object.

A goal of this work is to minimize the complexity of embedding Rust code in other contexts. It is not yet clear what the final embedding API will look like.

Green threading

Despite tying libstd to native threading, however, libgreen will still be supported – at least initially. The infrastructure in libgreen and friends will move into its own Cargo package.

Initially, the green threading package will support essentially the same interface it does today; there are no immediate plans to change its API, since the focus will be on first improving the native threading API. Note, however, that the I/O API will be exposed separately within libgreen, as opposed to the current exposure through std::io.

The long-term plan

Ultimately, a large motivation for the proposed refactoring is to allow the APIs for native I/O to grow.

In particular, over time we should expose more of the underlying system capabilities under the native threading model. Whenever possible, these capabilities should be provided at the libstd level – the highest level of cross-platform abstraction. However, an important goal is also to provide nonblocking and/or asynchronous I/O, for which system APIs differ greatly. It may be necessary to provide additional, platform-specific crates to expose this functionality. Ideally, these crates would interoperate smoothly with libstd, so that for example a libposix crate would allow using an poll operation directly against a std::io::fs::File value, for example.

We also wish to expose “lowering” operations in libstd – APIs that allow you to get at the file descriptor underlying a std::io::fs::File, for example.

On the other hand, we very much want to explore and support truly lightweight M:N task models (that do not require per-task stacks) – supporting efficient data parallelism with work stealing for CPU-bound computations. These lightweight models will not provide any special support for I/O. But they may benefit from a notion of “task-local storage” and interfacing with the task scheduler when explicitly synchronizing between tasks (via channels, for example).

All of the above long-term plans will require substantial new design and implementation work, and the specifics are out of scope for this RFC. The main point, though, is that the refactoring proposed by this RFC will make it much more plausible to carry out such work.

Finally, a guiding principle for the above work is uncompromising support for native system APIs, in terms of both functionality and performance. For example, it must be possible to use thread-local storage without significant overhead, which is very much not the case today. Any abstractions to support M:N threading models – including the now-external libgreen package – must respect this constraint.

Drawbacks

The main drawback of this proposal is that green I/O will be provided by a forked interface of std::io. This change makes green threading “second class”, and means there’s more to learn when using both models together.

This setup also somewhat increases the risk of invoking native blocking I/O on a green thread – though of course that risk is very much present today. One way of mitigating this risk in general is the Java executor approach, where the native “worker” threads that are executing the green thread scheduler are monitored for blocking, and new worker threads are spun up as needed.

Unresolved questions

There are may unresolved questions about the exact details of the refactoring, but these are considered implementation details since the libstd interface itself will not substantially change as part of this RFC.

Summary

The || unboxed closure form should be split into two forms—|| for nonescaping closures and move || for escaping closures—and the capture clauses and self type specifiers should be removed.

Motivation

Having to specify ref and the capture mode for each unboxed closure is inconvenient (see Rust PR rust-lang/rust#16610). It would be more convenient for the programmer if the type of the closure and the modes of the upvars could be inferred. This also eliminates the “line-noise” syntaxes like |&:|, which are arguably unsightly.

Not all knobs can be removed, however—the programmer must manually specify whether each closure is escaping or nonescaping. To see this, observe that no sensible default for the closure || (*x).clone() exists: if the function is nonescaping, it’s a closure that returns a copy of x every time but does not move x into it; if the function is escaping, it’s a closure that returns a copy of x and takes ownership of x.

Therefore, we need two forms: one for nonescaping closures and one for escaping closures. Nonescaping closures are the commonest, so they get the || syntax that we have today, and a new move || syntax will be introduced for escaping closures.

Detailed design

For unboxed closures specified with ||, the capture modes of the free variables are determined as follows:

  1. Any variable which is closed over and borrowed mutably is by-reference and mutably borrowed.

  2. Any variable of a type that does not implement Copy which is moved within the closure is captured by value.

  3. Any other variable which is closed over is by-reference and immutably borrowed.

The trait that the unboxed closure implements is FnOnce if any variables were moved out of the closure; otherwise FnMut if there are any variables that are closed over and mutably borrowed; otherwise Fn.

The ref prefix for unboxed closures is removed, since it is now essentially implied.

We introduce a new grammar production, move ||. The value returned by a move || implements FnOnce, FnMut, or Fn, as determined above; thus, for example, move |x: int, y| x + y produces an unboxed closure that implements the Fn(int, int) -> int trait (and thus the FnOnce(int, int) -> int trait by inheritance). Free variables referenced by a move || closure are always captured by value.

In the trait reference grammar, we will change the |&:| sugar to Fn(), the |&mut:| sugar to FnMut(), and the |:| sugar to FnOnce(). Thus what was before written fn foo<F:|&mut: int, int| -> int>() will be fn foo<F:FnMut(int, int) -> int>().

It is important to note that the trait reference syntax and closure construction syntax are purposefully distinct. This is because either the || form or the move || form can construct any of FnOnce, FnMut, or Fn closures.

Drawbacks

  1. Having two syntaxes for closures could be seen as unfortunate.

  2. move becomes a keyword.

Alternatives

  1. Keep the status quo: |:|/|&mut:/|&:| are the only ways to create unboxed closures, and ref must be used to get by-reference upvars.

  2. Use some syntax other than move || for escaping closures.

  3. Keep the |:|/|&mut:/|&:| syntax only for trait reference sugar.

  4. Use fn() syntax for trait reference sugar.

Unresolved questions

There may be unforeseen complications in doing the inference.

  • Start Date: 2014-09-16
  • RFC PR #: https://github.com/rust-lang/rfcs/pull/234
  • Rust Issue #: https://github.com/rust-lang/rust/issues/17323

Summary

Make enum variants part of both the type and value namespaces.

Motivation

We might, post-1.0, want to allow using enum variants as types. This would be backwards incompatible, because if a module already has a value with the same name as the variant in scope, then there will be a name clash.

Detailed design

Enum variants would always be part of both the type and value namespaces. Variants would not, however, be usable as types - we might want to allow this later, but it is out of scope for this RFC.

Data

Occurrences of name clashes in the Rust repo:

  • Key in rustrt::local_data

  • InAddr in native::io::net

  • Ast in regex::parse

  • Class in regex::parse

  • Native in regex::re

  • Dynamic in regex::re

  • Zero in num::bigint

  • String in term::terminfo::parm

  • String in serialize::json

  • List in serialize::json

  • Object in serialize::json

  • Argument in fmt_macros

  • Metadata in rustc_llvm

  • ObjectFile in rustc_llvm

  • ‘ItemDecorator’ in syntax::ext::base

  • ‘ItemModifier’ in syntax::ext::base

  • FunctionDebugContext in rustc::middle::trans::debuginfo

  • AutoDerefRef in rustc::middle::ty

  • MethodParam in rustc::middle::typeck

  • MethodObject in rustc::middle::typeck

That’s a total of 20 in the compiler and libraries.

Drawbacks

Prevents the common-ish idiom of having a struct with the same name as a variant and then having a value of that struct be the variant’s data.

Alternatives

Don’t do it. That would prevent us making changes to the typed-ness of enums in the future. If we accept this RFC, but at some point we decide we never want to do anything with enum variants and types, we could always roll back this change backwards compatibly.

Unresolved questions

N/A

Summary

This is a combined conventions and library stabilization RFC. The goal is to establish a set of naming and signature conventions for std::collections.

The major components of the RFC include:

  • Removing most of the traits in collections.

  • A general proposal for solving the “equiv” problem, as well as improving MaybeOwned.

  • Patterns for overloading on by-need values and predicates.

  • Initial, forwards-compatible steps toward Iterable.

  • A coherent set of API conventions across the full variety of collections.

A big thank-you to @Gankro, who helped collect API information and worked through an initial pass of some of the proposals here.

Motivation

This RFC aims to improve the design of the std::collections module in preparation for API stabilization. There are a number of problems that need to be addressed, as spelled out in the subsections below.

Collection traits

The collections module defines several traits:

  • Collection
  • Mutable
  • MutableSeq
  • Deque
  • Map, MutableMap
  • Set, MutableSet

There are several problems with the current trait design:

  • Most important: the traits do not provide iterator methods like iter. It is not possible to do so in a clean way without higher-kinded types, as the RFC explains in more detail below.

  • The split between mutable and immutable traits is not well-motivated by any of the existing collections.

  • The methods defined in these traits are somewhat anemic compared to the suite of methods provided on the concrete collections that implement them.

Divergent APIs

Despite the current collection traits, the APIs of various concrete collections has diverged; there is not a globally coherent design, and there are many inconsistencies.

One problem in particular is the lack of clear guiding principles for the API design. This RFC proposes a few along the way.

Providing slice APIs on Vec and String

The String and Vec types each provide a limited subset of the methods provides on string and vector slices, but there is not a clear reason to limit the API in this way. Today, one has to write things like some_str.as_slice().contains(...), which is not ergonomic or intuitive.

The Equiv problem

There is a more subtle problem related to slices. It’s common to use a HashMap with owned String keys, but then the natural API for things like lookup is not very usable:

fn find(&self, k: &K) -> Option<&V>

The problem is that, since K will be String, the find function requests a &String value – whereas one typically wants to work with the more flexible &str slices. In particular, using find with a literal string requires something like:

map.find(&"some literal".to_string())

which is unergonomic and requires an extra allocation just to get a borrow that, in some sense, was already available.

The current HashMap API works around this problem by providing an additional set of methods that uses a generic notion of “equivalence” of values that have different types:

pub trait Equiv<T> {
    fn equiv(&self, other: &T) -> bool;
}

impl Equiv<str> for String {
    fn equiv(&self, other: &str) -> bool {
        self.as_slice() == other
    }
}

fn find_equiv<Q: Hash<S> + Equiv<K>>(&self, k: &Q) -> Option<&V>

There are a few downsides to this approach:

  • It requires a duplicated _equiv variant of each method taking a reference to the key. (This downside could likely be mitigated using multidispatch.)

  • Its correctness depends on equivalent values producing the same hash, which is not checked.

  • String-keyed hash maps are very common, so newcomers are likely to run headlong into the problem. First, find will fail to work in the expected way. But the signature of find_equiv is more difficult to understand than find, and it it’s not immediately obvious that it solves the problem.

  • It is the right API for HashMap, but not helpful for e.g. TreeMap, which would want an analog for Ord.

The TreeMap API currently deals with this problem in an entirely different way:

/// Returns the value for which f(key) returns Equal.
/// f is invoked with current key and guides tree navigation.
/// That means f should be aware of natural ordering of the tree.
fn find_with(&self, f: |&K| -> Ordering) -> Option<&V>

Besides being less convenient – you cannot write map.find_with("some literal") – this function navigates the tree according to an ordering that may have no relationship to the actual ordering of the tree.

MaybeOwned

Sometimes a function does not know in advance whether it will need or produce an owned copy of some data, or whether a borrow suffices. A typical example is the from_utf8_lossy function:

fn from_utf8_lossy<'a>(v: &'a [u8]) -> MaybeOwned<'a>

This function will return a string slice if the input was correctly utf8 encoded – without any allocation. But if the input has invalid utf8 characters, the function allocates a new String and inserts utf8 “replacement characters” instead. Hence, the return type is an enum:

pub enum MaybeOwned<'a> {
    Slice(&'a str),
    Owned(String),
}

This interface makes it possible to allocate only when necessary, but the MaybeOwned type (and connected machinery) are somewhat ad hoc – and specialized to String/str. It would be somewhat more palatable if there were a single “maybe owned” abstraction usable across a wide range of types.

Iterable

A frequently-requested feature for the collections module is an Iterable trait for “values that can be iterated over”. There are two main motivations:

  • Abstraction. Today, you can write a function that takes a single Iterator, but you cannot write a function that takes a container and then iterates over it multiple times (perhaps with differing mutability levels). An Iterable trait could allow that.

  • Ergonomics. You’d be able to write

    for v in some_vec { ... }

    rather than

    for v in some_vec.iter() { ... }

    and consume_iter(some_vec) rather than consume_iter(some_vec.iter()).

Detailed design

The collections today

The concrete collections currently available in std fall into roughly three categories:

  • Sequences

    • Vec
    • String
    • Slices
    • Bitv
    • DList
    • RingBuf
    • PriorityQueue
  • Sets

    • HashSet
    • TreeSet
    • TrieSet
    • EnumSet
    • BitvSet
  • Maps

    • HashMap
    • TreeMap
    • TrieMap
    • LruCache
    • SmallIntMap

The primary goal of this RFC is to establish clean and consistent APIs that apply across each group of collections.

Before diving into the details, there is one high-level changes that should be made to these collections. The PriorityQueue collection should be renamed to BinaryHeap, following the convention that concrete collections are named according to their implementation strategy, not the abstract semantics they implement. We may eventually want PriorityQueue to be a trait that’s implemented by multiple concrete collections.

The LruCache could be renamed for a similar reason (it uses a HashMap in its implementation), However, the implementation is actually generic with respect to this underlying map, and so in the long run (with HKT and other language changes) LruCache should probably add a type parameter for the underlying map, defaulted to HashMap.

Design principles

  • Centering on Iterators. The Iterator trait is a strength of Rust’s collections library. Because so many APIs can produce iterators, adding an API that consumes one is very powerful – and conversely as well. Moreover, iterators are highly efficient, since you can chain several layers of modification without having to materialize intermediate results. Thus, whenever possible, collection APIs should strive to work with iterators.

    In particular, some existing convenience methods avoid iterators for either performance or ergonomic reasons. We should instead improve the ergonomics and performance of iterators, so that these extra convenience methods are not necessary and so that all collections can benefit.

  • Minimizing method variants. One problem with some of the current collection APIs is the proliferation of method variants. For example, HashMap include seven methods that begin with the name find! While each method has a motivation, the API as a whole can be bewildering, especially to newcomers.

    When possible, we should leverage the trait system, or find other abstractions, to reduce the need for method variants while retaining their ergonomics and power.

  • Conservatism. It is easier to add APIs than to take them away. This RFC takes a fairly conservative stance on what should be included in the collections APIs. In general, APIs should be very clearly motivated by a wide variety of use cases, either for expressiveness, performance, or ergonomics.

Removing the traits

This RFC proposes a somewhat radical step for the collections traits: rather than reform them, we should eliminate them altogether – for now.

Unlike inherent methods, which can easily be added and deprecated over time, a trait is “forever”: there are very few backwards-compatible modifications to traits. Thus, for something as fundamental as collections, it is prudent to take our time to get the traits right.

Lack of iterator methods

In particular, there is one way in which the current traits are clearly wrong: they do not provide standard methods like iter, despite these being fundamental to working with collections in Rust. Sadly, this gap is due to inexpressiveness in the language, which makes directly defining iterator methods in a trait impossible:

trait Iter {
    type A;
    type I: Iterator<&'a A>;    // what is the lifetime here?
    fn iter<'a>(&'a self) -> I; // and how to connect it to self?
}

The problem is that, when implementing this trait, the return type I of iter should depend on the lifetime of self. For example, the corresponding method in Vec looks like the following:

impl<T> Vec<T> {
    fn iter(&'a self) -> Items<'a, T> { ... }
}

This means that, given a Vec<T>, there isn’t a single type Items<T> for iteration – rather, there is a family of types, one for each input lifetime. In other words, the associated type I in the Iter needs to be “higher-kinded”: not just a single type, but rather a family:

trait Iter {
    type A;
    type I<'a>: Iterator<&'a A>;
    fn iter<'a>(&self) -> I<'a>;
}

In this case, I is parameterized by a lifetime, but in other cases (like map) an associated type needs to be parameterized by a type.

In general, such higher-kinded types (HKTs) are a much-requested feature for Rust. But the design and implementation of higher-kinded types is, by itself, a significant investment.

HKT would also allow for parameterization over smart pointer types, which has many potential use cases in the context of collections.

Thus, the goal in this RFC is to do the best we can without HKT for now, while allowing a graceful migration if or when HKT is added.

Persistent/immutable collections

Another problem with the current collection traits is the split between immutable and mutable versions. In the long run, we will probably want to provide persistent collections (which allow non-destructive “updates” that create new collections that share most of their data with the old ones).

However, persistent collection APIs have not been thoroughly explored in Rust; it would be hasty to standardize on a set of traits until we have more experience.

Downsides of removal

There are two main downsides to removing the traits without a replacement:

  1. It becomes impossible to write code using generics over a “kind” of collection (like Map).

  2. It becomes more difficult to ensure that the collections share a common API.

For point (1), first, if the APIs are sufficiently consistent it should be possible to transition code from e.g. a TreeMap to a HashMap by changing very few lines of code. Second, generic programming is currently quite limited, given the inability to iterate. Finally, generic programming over collections is a large design space (with much precedent in C++, for example), and we should take our time and gain more experience with a variety of concrete collections before settling on a design.

For point (2), first, the current traits have failed to keep the APIs in line, as we will see below. Second, this RFC is the antidote: we establish a clear set of conventions and APIs for concrete collections up front, and stabilize on those, which should make it easy to add traits later on.

Why not leave the traits as “experimental”?

An alternative to removal would be to leave the traits intact, but marked as experimental, with the intent to radically change them later.

Such a strategy doesn’t buy much relative to removal (given the arguments above), but risks the traits becoming “de facto” stable if people begin using them en masse.

Solving the _equiv and MaybeOwned problems

The basic problem that leads to _equiv methods is that:

  • &String and &str are not the same type.
  • The &str type is more flexible and hence more widely used.
  • Code written for a generic type T that takes a reference &T will therefore not be suitable when T is instantiated with String.

A similar story plays out for &Vec<T> and &[T], and with DST and custom slice types the same problem will arise elsewhere.

The Borrow trait

This RFC proposes to use a trait, Borrow to connect borrowed and owned data in a generic fashion:

/// A trait for borrowing.
trait Borrow<Sized? B> {
    /// Immutably borrow from an owned value.
    fn borrow(&self) -> &B;

    /// Mutably borrow from an owned value.
    fn borrow_mut(&mut self) -> &mut B;
}

// The Sized bound means that this impl does not overlap with the impls below.
impl<T: Sized> Borrow<T> for T {
    fn borrow(a: &T) -> &T {
        a
    }
    fn borrow_mut(a: &mut T) -> &mut T {
        a
    }
}

impl Borrow<str> for String {
    fn borrow(s: &String) -> &str {
        s.as_slice()
    }
    fn borrow_mut(s: &mut String) -> &mut str {
        s.as_mut_slice()
    }
}

impl<T> Borrow<[T]> for Vec<T> {
    fn borrow(s: &Vec<T>) -> &[T] {
        s.as_slice()
    }
    fn borrow_mut(s: &mut Vec<T>) -> &mut [T] {
        s.as_mut_slice()
    }
}

(Note: thanks to @epdtry for suggesting this variation! The original proposal is listed in the Alternatives.)

A primary goal of the design is allowing a blanket impl for non-sliceable types (the first impl above). This blanket impl ensures that all new sized, cloneable types are automatically borrowable; new impls are required only for new unsized types, which are rare. The Sized bound on the initial impl means that we can freely add impls for unsized types (like str and [T]) without running afoul of coherence.

Because of the blanket impl, the Borrow trait can largely be ignored except when it is actually used – which we describe next.

Using Borrow to replace _equiv methods

With the Borrow trait in place, we can eliminate the _equiv method variants by asking map keys to be Borrow:

impl<K,V> HashMap<K,V> where K: Hash + Eq {
    fn find<Q>(&self, k: &Q) -> &V where K: Borrow<Q>, Q: Hash + Eq { ... }
    fn contains_key<Q>(&self, k: &Q) -> bool where K: Borrow<Q>, Q: Hash + Eq { ... }
    fn insert(&mut self, k: K, v: V) -> Option<V> { ... }

    ...
}

The benefits of this approach over _equiv are:

  • The Borrow trait captures the borrowing relationship between an owned data structure and both references to it and slices from it – once and for all. This means that it can be used anywhere we need to program generically over “borrowed” data. In particular, the single trait works for both HashMap and TreeMap, and should work for other kinds of data structures as well. It also helps generalize MaybeOwned, for similar reasons (see below.)

    A very important consequence is that the map methods using Borrow can potentially be put into a common Map trait that’s implemented by HashMap, TreeMap, and others. While we do not propose to do so now, we definitely want to do so later on.

  • When using a HashMap<String, T>, all of the basic methods like find, contains_key and insert “just work”, without forcing you to think about &String vs &str.

  • We don’t need separate _equiv variants of methods. (However, this could probably be addressed with multidispatch by providing a blanket Equiv implementation.)

On the other hand, this approach retains some of the downsides of _equiv:

  • The signature for methods like find and contains_key is more complex than their current signatures. There are two counterpoints. First, over time the Borrow trait is likely to become a well-known concept, so the signature will not appear completely alien. Second, what is perhaps more important than the signature is that, when using find on HashMap<String, T>, various method arguments just work as expected.

  • The API does not guarantee “coherence”: the Hash and Eq (or Ord, for TreeMap) implementations for the owned and borrowed keys might differ, breaking key invariants of the data structure. This is already the case with _equiv.

The Alternatives section includes a variant of Borrow that doesn’t suffer from these downsides, but has some downsides of its own.

Clone-on-write (Cow) pointers

A side-benefit of the Borrow trait is that we can give a more general version of the MaybeOwned as a “clone-on-write” smart pointer:

/// A generalization of Clone.
trait FromBorrow<Sized? B>: Borrow<B> {
    fn from_borrow(b: &B) -> Self;
}

/// A clone-on-write smart pointer
pub enum Cow<'a, T, B> where T: FromBorrow<B> {
    Shared(&'a B),
    Owned(T)
}

impl<'a, T, B> Cow<'a, T, B> where T: FromBorrow<B> {
    pub fn new(shared: &'a B) -> Cow<'a, T, B> {
        Shared(shared)
    }

    pub fn new_owned(owned: T) -> Cow<'static, T, B> {
        Owned(owned)
    }

    pub fn is_owned(&self) -> bool {
        match *self {
            Owned(_) => true,
            Shared(_) => false
        }
    }

    pub fn to_owned_mut(&mut self) -> &mut T {
        match *self {
            Shared(shared) => {
                *self = Owned(FromBorrow::from_borrow(shared));
                self.to_owned_mut()
            }
            Owned(ref mut owned) => owned
        }
    }

    pub fn into_owned(self) -> T {
        match self {
            Shared(shared) => FromBorrow::from_borrow(shared),
            Owned(owned) => owned
        }
    }
}

impl<'a, T, B> Deref<B> for Cow<'a, T, B> where T: FromBorrow<B>  {
    fn deref(&self) -> &B {
        match *self {
            Shared(shared) => shared,
            Owned(ref owned) => owned.borrow()
        }
    }
}

impl<'a, T, B> DerefMut<B> for Cow<'a, T, B> where T: FromBorrow<B> {
    fn deref_mut(&mut self) -> &mut B {
        self.to_owned_mut().borrow_mut()
    }
}

The type Cow<'a, String, str> is roughly equivalent to today’s MaybeOwned<'a> (and Cow<'a, Vec<T>, [T]> to MaybeOwnedVector<'a, T>).

By implementing Deref and DerefMut, the Cow type acts as a smart pointer – but in particular, the mut variant actually clones if the pointed-to value is not currently owned. Hence “clone on write”.

One slight gotcha with the design is that &mut str is not very useful, while &mut String is (since it allows extending the string, for example). On the other hand, Deref and DerefMut must deref to the same underlying type, and for Deref to not require cloning, it must yield a &str value.

Thus, the Cow pointer offers a separate to_owned_mut method that yields a mutable reference to the owned version of the type.

Note that, by not using into_owned, the Cow pointer itself may be owned by some other data structure (perhaps as part of a collection) and will internally track whether an owned copy is available.

Altogether, this RFC proposes to introduce Borrow and Cow as above, and to deprecate MaybeOwned and MaybeOwnedVector. The API changes for the collections are discussed below.

IntoIterator (and Iterable)

As discussed in earlier, some form of an Iterable trait is desirable for both expressiveness and ergonomics. Unfortunately, a full treatment of Iterable requires HKT for similar reasons to the collection traits. However, it’s possible to get some of the way there in a forwards-compatible fashion.

In particular, the following two traits work fine (with associated items):

trait Iterator {
    type A;
    fn next(&mut self) -> Option<A>;
    ...
}

trait IntoIterator {
    type A;
    type I: Iterator<A = A>;

    fn into_iter(self) -> I;
}

Because IntoIterator consumes self, lifetimes are not an issue.

It’s tempting to also define a trait like:

trait Iterable<'a> {
    type A;
    type I: Iterator<&'a A>;

    fn iter(&'a self) -> I;
}

(along the lines of those proposed by an earlier RFC).

The problem with Iterable as defined above is that it’s locked to a particular lifetime up front. But in many cases, the needed lifetime is not even nameable in advance:

fn iter_through_rc<I>(c: Rc<I>) where I: Iterable<?> {
    // the lifetime of the borrow is established here,
    // so cannot even be named in the function signature
    for x in c.iter() {
        // ...
    }
}

To make this kind of example work, you’d need to be able to say something like:

where <'a> I: Iterable<'a>

that is, that I implements Iterable for every lifetime 'a. While such a feature is feasible to add to where clauses, the HKT solution is undoubtedly cleaner.

Fortunately, we can have our cake and eat it too. This RFC proposes the IntoIterator trait above, together with the following blanket impl:

impl<I: Iterator> IntoIterator for I {
    type A = I::A;
    type I = I;
    fn into_iter(self) -> I {
        self
    }
}

which means that taking IntoIterator is strictly more flexible than taking Iterator. Note that in other languages (like Java), iterators are not iterable because the latter implies an unlimited number of iterations. But because IntoIterator consumes self, it yields only a single iteration, so all is good.

For individual collections, one can then implement IntoIterator on both the collection and borrows of it:

impl<T> IntoIterator for Vec<T> {
    type A = T;
    type I = MoveItems<T>;
    fn into_iter(self) -> MoveItems<T> { ... }
}

impl<'a, T> IntoIterator for &'a Vec<T> {
    type A = &'a T;
    type I = Items<'a, T>;
    fn into_iter(self) -> Items<'a, T> { ... }
}

impl<'a, T> IntoIterator for &'a mut Vec<T> {
    type A = &'a mut T;
    type I = ItemsMut<'a, T>;
    fn into_iter(self) -> ItemsMut<'a, T> { ... }
}

If/when HKT is added later on, we can add an Iterable trait and a blanket impl like the following:

// the HKT version
trait Iterable {
    type A;
    type I<'a>: Iterator<&'a A>;
    fn iter<'a>(&'a self) -> I<'a>;
}

impl<'a, C: Iterable> IntoIterator for &'a C {
    type A = &'a C::A;
    type I = C::I<'a>;
    fn into_iter(self) -> I {
        self.iter()
    }
}

This gives a clean migration path: once Vec implements Iterable, it can drop the IntoIterator impls for borrowed vectors, since they will be covered by the blanket implementation. No code should break.

Likewise, if we add a feature like the “universal” where clause mentioned above, it can be used to deal with embedded lifetimes as in the iter_through_rc example; and if the HKT version of Iterable is later added, thanks to the suggested blanket impl for IntoIterator that where clause could be changed to use Iterable instead, again without breakage.

Benefits of IntoIterator

What do we gain by incorporating IntoIterator today?

This RFC proposes that for loops should use IntoIterator rather than Iterator. With the blanket impl of IntoIterator for any Iterator, this is not a breaking change. However, given the IntoIterator impls for Vec above, we would be able to write:

let v: Vec<Foo> = ...

for x in &v { ... }     // iterate over &Foo
for x in &mut v { ... } // iterate over &mut Foo
for x in v { ... }      // iterate over Foo

Similarly, methods that currently take slices or iterators can be changed to take IntoIterator instead, immediately becoming more general and more ergonomic.

In general, IntoIterator will allow us to move toward more Iterator-centric APIs today, in a way that’s compatible with HKT tomorrow.

Additional methods

Another typical desire for an Iterable trait is to offer defaulted versions of methods that basically re-export iterator methods on containers (see the earlier RFC). Usually these methods would go through a reference iterator (i.e. the iter method) rather than a moving iterator.

It is possible to add such methods using the design proposed above, but there are some drawbacks. For example, should Vec::map produce an iterator, or a new vector? It would be possible to do the latter generically, but only with HKT. (See this discussion.)

This RFC only proposes to add the following method via IntoIterator, as a convenience for a common pattern:

trait IterCloned {
    type A;
    type I: Iterator<A>;
    fn iter_cloned(self) -> I;
}

impl<'a, T, I: IntoIterator> IterCloned for I where I::A = &'a T {
    type A = T;
    type I = ClonedItems<I>;
    fn into_iter(self) -> I { ... }
}

(The iter_cloned method will help reduce the number of method variants in general for collections, as we will see below).

We will leave to later RFCs the incorporation of additional methods. Notice, in particular, that such methods can wait until we introduce an Iterable trait via HKT without breaking backwards compatibility.

Minimizing variants: ByNeed and Predicate traits

There are several kinds of methods that, in their most general form take closures, but for which convenience variants taking simpler data are common:

  • Taking values by need. For example, consider the unwrap_or and unwrap_or_else methods in Option:

    fn unwrap_or(self, def: T) -> T
    fn unwrap_or_else(self, f: || -> T) -> T

    The unwrap_or_else method is the most general: it invokes the closure to compute a default value only when self is None. When the default value is expensive to compute, this by-need approach helps. But often the default value is cheap, and closures are somewhat annoying to write, so unwrap_or provides a convenience wrapper.

  • Taking predicates. For example, a method like contains often shows up (inconsistently!) in two variants:

    fn contains(&self, elem: &T) -> bool; // where T: PartialEq
    fn contains_fn(&self, pred: |&T| -> bool) -> bool;

    Again, the contains_fn version is the more general, but it’s convenient to provide a specialized variant when the element type can be compared for equality, to avoid writing explicit closures.

As it turns out, with multidispatch) it is possible to use a trait to express these variants through overloading:

trait ByNeed<T> {
    fn compute(self) -> T;
}

impl<T> ByNeed<T> for T {
    fn compute(self) -> T {
        self
    }
}

// Due to multidispatch, this impl does NOT overlap with the above one
impl<T> ByNeed<T> for || -> T {
    fn compute(self) -> T {
        self()
    }
}

impl<T> Option<T> {
    fn unwrap_or<U>(self, def: U) where U: ByNeed<T> { ... }
    ...
}
trait Predicate<T> {
    fn check(&self, &T) -> bool;
}

impl<T: Eq> Predicate<T> for &T {
    fn check(&self, t: &T) -> bool {
        *self == t
    }
}

impl<T> Predicate<T> for |&T| -> bool {
    fn check(&self, t: &T) -> bool {
        (*self)(t)
    }
}

impl<T> Vec<T> {
    fn contains<P>(&self, pred: P) where P: Predicate<T> { ... }
    ...
}

Since these two patterns are particularly common throughout std, this RFC proposes adding both of the above traits, and using them to cut down on the number of method variants.

In particular, some methods on string slices currently work with CharEq, which is similar to Predicate<char>:

pub trait CharEq {
    fn matches(&mut self, char) -> bool;
    fn only_ascii(&self) -> bool;
}

The difference is the only_ascii method, which is used to optimize certain operations when the predicate only holds for characters in the ASCII range.

To keep these optimizations intact while connecting to Predicate, this RFC proposes the following restructuring of CharEq:

pub trait CharPredicate: Predicate<char> {
    fn only_ascii(&self) -> bool {
        false
    }
}

Why not leverage unboxed closures?

A natural question is: why not use the traits for unboxed closures to achieve a similar effect? For example, you could imagine writing a blanket impl for Fn(&T) -> bool for any T: PartialEq, which would allow PartialEq values to be used anywhere a predicate-like closure was requested.

The problem is that these blanket impls will often conflict. In particular, any type T could implement Fn() -> T, and that single blanket impl would preclude any others (at least, assuming that unboxed closure traits treat the argument and return types as associated (output) types).

In addition, the explicit use of traits like Predicate makes the intended semantics more clear, and the overloading less surprising.

The APIs

Now we’ll delve into the detailed APIs for the various concrete collections. These APIs will often be given in tabular form, grouping together common APIs across multiple collections. When writing these function signatures:

  • We will assume a type parameter T for Vec, BinaryHeap, DList and RingBuf; we will also use this parameter for APIs on String, where it should be understood as char.

  • We will assume type parameters K: Borrow and V for HashMap and TreeMap; for TrieMap and SmallIntMap the K is assumed to be uint

  • We will assume a type parameter K: Borrow for HashSet and TreeSet; for BitvSet it is assumed to be uint.

We will begin by outlining the most widespread APIs in tables, making it easy to compare names and signatures across different kinds of collections. Then we will focus on some APIs specific to particular classes of collections – e.g. sets and maps. Finally, we will briefly discuss APIs that are specific to a single concrete collection.

Construction

All of the collections should support a static function:

fn new() -> Self

that creates an empty version of the collection; the constructor may take arguments needed to set up the collection, e.g. the capacity for LruCache.

Several collections also support separate constructors for providing capacities in advance; these are discussed below.

The FromIterator trait

All of the collections should implement the FromIterator trait:

pub trait FromIterator {
    type A:
    fn from_iter<T>(T) -> Self where T: IntoIterator<A = A>;
}

Note that this varies from today’s FromIterator by consuming an IntoIterator rather than Iterator. As explained above, this choice is strictly more general and will not break any existing code.

This constructor initializes the collection with the contents of the iterator. For maps, the iterator is over key/value pairs, and the semantics is equivalent to inserting those pairs in order; if keys are repeated, the last value is the one left in the map.

Insertion

The table below gives methods for inserting items into various concrete collections:

OperationCollections
fn push(&mut self, T)Vec, BinaryHeap, String
fn push_front(&mut self, T)DList, RingBuf
fn push_back(&mut self, T)DList, RingBuf
fn insert(&mut self, uint, T)Vec, RingBuf, String
fn insert(&mut self, K::Owned) -> boolHashSet, TreeSet, TrieSet, BitvSet
fn insert(&mut self, K::Owned, V) -> Option<V>HashMap, TreeMap, TrieMap, SmallIntMap
fn append(&mut self, Self)DList
fn prepend(&mut self, Self)DList

There are a few changes here from the current state of affairs:

  • The DList and RingBuf data structures no longer provide push, but rather push_front and push_back. This change is based on (1) viewing them as deques and (2) not giving priority to the “front” or the “back”.

  • The insert method on maps returns the value previously associated with the key, if any. Previously, this functionality was provided by a swap method, which has been dropped (consolidating needless method variants.)

Aside from these changes, a number of insertion methods will be deprecated (e.g. the append and append_one methods on Vec). These are discussed further in the section on “specialized operations” below.

The Extend trait (was: Extendable)

In addition to the standard insertion operations above, all collections will implement the Extend trait. This trait was previously called Extendable, but in general we prefer to avoid -able suffixes and instead name the trait using a verb (or, especially, the key method offered by the trait.)

The Extend trait allows data from an arbitrary iterator to be inserted into a collection, and will be defined as follows:

pub trait Extend: FromIterator {
    fn extend<T>(&mut self, T) where T: IntoIterator<A = Self::A>;
}

As with FromIterator, this trait has been modified to take an IntoIterator value.

Deletion

The table below gives methods for removing items into various concrete collections:

OperationCollections
fn clear(&mut self)all
fn pop(&mut self) -> Option<T>Vec, BinaryHeap, String
fn pop_front(&mut self) -> Option<T>DList, RingBuf
fn pop_back(&mut self) -> Option<T>DList, RingBuf
fn remove(&mut self, uint) -> Option<T>Vec, RingBuf, String
fn remove(&mut self, &K) -> boolHashSet, TreeSet, TrieSet, BitvSet
fn remove(&mut self, &K) -> Option<V>HashMap, TreeMap, TrieMap, SmallIntMap
fn truncate(&mut self, len: uint)Vec, String, Bitv, DList, RingBuf
fn retain<P>(&mut self, f: P) where P: Predicate<T>Vec, DList, RingBuf
fn dedup(&mut self)Vec, DList, RingBuf where T: PartialEq

As with the insertion methods, there are some differences from today’s API:

  • The DList and RingBuf data structures no longer provide pop, but rather pop_front and pop_back – similarly to the push methods.

  • The remove method on maps returns the value previously associated with the key, if any. Previously, this functionality was provided by a separate pop method, which has been dropped (consolidating needless method variants.)

  • The retain method takes a Predicate.

  • The truncate, retain and dedup methods are offered more widely.

Again, some of the more specialized methods are not discussed here; see “specialized operations” below.

Inspection/mutation

The next table gives methods for inspection and mutation of existing items in collections:

OperationCollections
fn len(&self) -> uintall
fn is_empty(&self) -> boolall
fn get(&self, uint) -> Option<&T>[T], Vec, RingBuf
fn get_mut(&mut self, uint) -> Option<&mut T>[T], Vec, RingBuf
fn get(&self, &K) -> Option<&V>HashMap, TreeMap, TrieMap, SmallIntMap
fn get_mut(&mut self, &K) -> Option<&mut V>HashMap, TreeMap, TrieMap, SmallIntMap
fn contains<P>(&self, P) where P: Predicate<T>[T], str, Vec, String, DList, RingBuf, BinaryHeap
fn contains(&self, &K) -> boolHashSet, TreeSet, TrieSet, EnumSet
fn contains_key(&self, &K) -> boolHashMap, TreeMap, TrieMap, SmallIntMap

The biggest changes from the current APIs are:

  • The find and find_mut methods have been renamed to get and get_mut. Further, all get methods return Option values and do not invoke fail!. This is part of a general convention described in the next section (on the Index traits).

  • The contains method is offered more widely.

  • There is no longer an equivalent of find_copy (which should be called find_clone). Instead, we propose to add the following method to the Option<&'a T> type where T: Clone:

    fn cloned(self) -> Option<T> {
        self.map(|x| x.clone())
    }

    so that some_map.find_copy(key) will instead be written some_map.find(key).cloned(). This method chain is slightly longer, but is more clear and allows us to drop the _copy variants. Moreover, all users of Option benefit from the new convenience method.

The Index trait

The Index and IndexMut traits provide indexing notation like v[0]:

pub trait Index {
    type Index;
    type Result;
    fn index(&'a self, index: &Index) -> &'a Result;
}

pub trait IndexMut {
    type Index;
    type Result;
    fn index_mut(&'a mut self, index: &Index) -> &'a mut Result;
}

These traits will be implemented for: [T], Vec, RingBuf, HashMap, TreeMap, TrieMap, SmallIntMap.

As a general convention, implementation of the Index traits will fail the task if the index is invalid (out of bounds or key not found); they will therefore return direct references to values. Any collection implementing Index (resp. IndexMut) should also provide a get method (resp. get_mut) as a non-failing variant that returns an Option value.

This allows us to keep indexing notation maximally concise, while still providing convenient non-failing variants (which can be used to provide a check for index validity).

Iteration

Every collection should provide the standard trio of iteration methods:

fn iter(&'a self) -> Items<'a>;
fn iter_mut(&'a mut self) -> ItemsMut<'a>;
fn into_iter(self) -> ItemsMove;

and in particular implement the IntoIterator trait on both the collection type and on (mutable) references to it.

Capacity management

many of the collections have some notion of “capacity”, which may be fixed, grow explicitly, or grow implicitly:

  • No capacity/fixed capacity: DList, TreeMap, TreeSet, TrieMap, TrieSet, slices, EnumSet
  • Explicit growth: LruCache
  • Implicit growth: Vec, RingBuf, HashMap, HashSet, BitvSet, BinaryHeap

Growable collections provide functions for capacity management, as follows.

Explicit growth

For explicitly-grown collections, the normal constructor (new) takes a capacity argument. Capacity can later be inspected or updated as follows:

fn capacity(&self) -> uint
fn set_capacity(&mut self, capacity: uint)

(Note, this renames LruCache::change_capacity to set_capacity, the prevailing style for setter method.)

Implicit growth

For implicitly-grown collections, the normal constructor (new) does not take a capacity, but there is an explicit with_capacity constructor, along with other functions to work with the capacity later on:

fn with_capacity(uint) -> Self
fn capacity(&self) -> uint
fn reserve(&mut self, additional: uint)
fn reserve_exact(&mut self, additional: uint)
fn shrink_to_fit(&mut self)

There are some important changes from the current APIs:

  • The reserve and reserve_exact methods now take as an argument the extra space to reserve, rather than the final desired capacity, as this usage is vastly more common. The reserve function may grow the capacity by a larger amount than requested, to ensure amortization, while reserve_exact will reserve exactly the requested additional capacity. The reserve_additional methods are deprecated.

  • The with_capacity constructor does not take any additional arguments, for uniformity with new. This change affects Bitv in particular.

Bounded iterators

Some of the maps (e.g. TreeMap) currently offer specialized iterators over their entries starting at a given key (called lower_bound) and above a given key (called upper_bound), along with _mut variants. While the functionality is worthwhile, the names are not very clear, so this RFC proposes the following replacement API (thanks to @Gankro for the suggestion):

Bound<T> {
    /// An inclusive bound
    Included(T),

    /// An exclusive bound
    Excluded(T),

    Unbounded,
}

/// Creates a double-ended iterator over a sub-range of the collection's items,
/// starting at min, and ending at max. If min is `Unbounded`, then it will
/// be treated as "negative infinity", and if max is `Unbounded`, then it will
/// be treated as "positive infinity". Thus range(Unbounded, Unbounded) will yield
/// the whole collection.
fn range(&self, min: Bound<&T>, max: Bound<&T>) -> RangedItems<'a, T>;

fn range_mut(&self, min: Bound<&T>, max: Bound<&T>) -> RangedItemsMut<'a, T>;

These iterators should be provided for any maps over ordered keys (TreeMap, TrieMap and SmallIntMap).

In addition, analogous methods should be provided for sets over ordered keys (TreeSet, TrieSet, BitvSet).

Set operations

Comparisons

All sets should offer the following methods, as they do today:

fn is_disjoint(&self, other: &Self) -> bool;
fn is_subset(&self, other: &Self) -> bool;
fn is_superset(&self, other: &Self) -> bool;

Combinations

Sets can also be combined using the standard operations – union, intersection, difference and symmetric difference (exclusive or). Today’s APIs for doing so look like this:

fn union<'a>(&'a self, other: &'a Self) -> I;
fn intersection<'a>(&'a self, other: &'a Self) -> I;
fn difference<'a>(&'a self, other: &'a Self) -> I;
fn symmetric_difference<'a>(&'a self, other: &'a Self) -> I;

where the I type is an iterator over keys that varies by concrete set. Working with these iterators avoids materializing intermediate sets when they’re not needed; the collect method can be used to create sets when they are. This RFC proposes to keep these names intact, following the RFC on iterator conventions.

Sets should also implement the BitOr, BitAnd, BitXor and Sub traits from std::ops, allowing overloaded notation |, &, |^ and - to be used with sets. These are equivalent to invoking the corresponding iter_ method and then calling collect, but for some sets (notably BitvSet) a more efficient direct implementation is possible.

Unfortunately, we do not yet have a set of traits corresponding to operations |=, &=, etc, but again in some cases doing the update in place may be more efficient. Right now, BitvSet is the only concrete set offering such operations:

fn union_with(&mut self, other: &BitvSet)
fn intersect_with(&mut self, other: &BitvSet)
fn difference_with(&mut self, other: &BitvSet)
fn symmetric_difference_with(&mut self, other: &BitvSet)

This RFC punts on the question of naming here: it does not propose a new set of names. Ideally, we would add operations like |= in a separate RFC, and use those conventionally for sets. If not, we will choose fallback names during the stabilization of BitvSet.

Map operations

Combined methods

The HashMap type currently provides a somewhat bewildering set of find/insert variants:

fn find_or_insert(&mut self, k: K, v: V) -> &mut V
fn find_or_insert_with<'a>(&'a mut self, k: K, f: |&K| -> V) -> &'a mut V
fn insert_or_update_with<'a>(&'a mut self, k: K, v: V, f: |&K, &mut V|) -> &'a mut V
fn find_with_or_insert_with<'a, A>(&'a mut self, k: K, a: A, found: |&K, &mut V, A|, not_found: |&K, A| -> V) -> &'a mut V

These methods are used to couple together lookup and insertion/update operations, thereby avoiding an extra lookup step. However, the current set of method variants seems overly complex.

There is another RFC already in the queue addressing this problem in a very nice way, and this RFC defers to that one

Key and value iterators

In addition to the standard iterators, maps should provide by-reference convenience iterators over keys and values:

fn keys(&'a self) -> Keys<'a, K>
fn values(&'a self) -> Values<'a, V>

While these iterators are easy to define in terms of the main iter method, they are used often enough to warrant including convenience methods.

Specialized operations

Many concrete collections offer specialized operations beyond the ones given above. These will largely be addressed through the API stabilization process (which focuses on local API issues, as opposed to general conventions), but a few broad points are addressed below.

Relating Vec and String to slices

One goal of this RFC is to supply all of the methods on (mutable) slices on Vec and String. There are a few ways to achieve this, so concretely the proposal is for Vec<T> to implement Deref<[T]> and DerefMut<[T]>, and String to implement Deref<str>. This will automatically allow all slice methods to be invoked from vectors and strings, and will allow writing &*v rather than v.as_slice().

In this scheme, Vec and String are really “smart pointers” around the corresponding slice types. While counterintuitive at first, this perspective actually makes a fair amount of sense, especially with DST.

(Initially, it was unclear whether this strategy would play well with method resolution, but the planned resolution rules should work fine.)

String API

One of the key difficulties with the String API is that strings use utf8 encoding, and some operations are only efficient when working at the byte level (and thus taking this encoding into account).

As a general principle, we will move the API toward the following convention: index-related operations always work in terms of bytes, other operations deal with chars by default (but can have suffixed variants for working at other granularities when appropriate.)

DList

The DList type offers a number of specialized methods:

swap_remove, insert_when, insert_ordered, merge, rotate_forward and rotate_backward

Prior to stabilizing the DList API, we will attempt to simplify its API surface, possibly by using idea from the collection views RFC.

Minimizing method variants via iterators

Partitioning via FromIterator

One place we can move toward iterators is functions like partition and partitioned on vectors and slices:

// on Vec<T>
fn partition(self, f: |&T| -> bool) -> (Vec<T>, Vec<T>);

// on [T] where T: Clone
fn partitioned(&self, f: |&T| -> bool) -> (Vec<T>, Vec<T>);

These two functions transform a vector/slice into a pair of vectors, based on a “partitioning” function that says which of the two vectors to place elements into. The partition variant works by moving elements of the vector, while partitioned clones elements.

There are a few unfortunate aspects of an API like this one:

  • It’s specific to vectors/slices, although in principle both the source and target containers could be more general.

  • The fact that two variants have to be exposed, for owned versus clones, is somewhat unfortunate.

This RFC proposes the following alternative design:

pub enum Either<T, U> {
    pub Left(T),
    pub Right(U),
}

impl<A, B> FromIterator for (A, B) where A: Extend, B: Extend {
    fn from_iter<I>(mut iter: I) -> (A, B) where I: IntoIterator<Either<T, U>> {
        let mut left: A = FromIterator::from_iter(None::<T>);
        let mut right: B = FromIterator::from_iter(None::<U>);

        for item in iter {
            match item {
                Left(t) => left.extend(Some(t)),
                Right(u) => right.extend(Some(u)),
            }
        }

        (left, right)
    }
}

trait Iterator {
    ...
    fn partition(self, |&A| -> bool) -> Partitioned<A> { ... }
}

// where Partitioned<A>: Iterator<A = Either<A, A>>

This design drastically generalizes the partitioning functionality, allowing it be used with arbitrary collections and iterators, while removing the by-reference and by-value distinction.

Using this design, you have:

// The following two lines are equivalent:
let (u, w) = v.partition(f);
let (u, w): (Vec<T>, Vec<T>) = v.into_iter().partition(f).collect();

// The following two lines are equivalent:
let (u, w) = v.as_slice().partitioned(f);
let (u, w): (Vec<T>, Vec<T>) = v.iter_cloned().partition(f).collect();

There is some extra verbosity, mainly due to the type annotations for collect, but the API is much more flexible, since the partitioned data can now be collected into other collections (or even differing collections). In addition, partitioning is supported for any iterator.

Removing methods like from_elem, from_fn, grow, and grow_fn

Vectors and some other collections offer constructors and growth functions like the following:

fn from_elem(length: uint, value: T) -> Vec<T>
fn from_fn(length: uint, op: |uint| -> T) -> Vec<T>
fn grow(&mut self, n: uint, value: &T)
fn grow_fn(&mut self, n: uint, f: |uint| -> T)

These extra variants can easily be dropped in favor of iterators, and this RFC proposes to do so.

The iter module already contains a Repeat iterator; this RFC proposes to add a free function repeat to iter as a convenience for iter::Repeat::new.

With that in place, we have:

// Equivalent:
let v = Vec::from_elem(n, a);
let v = Vec::from_iter(repeat(a).take(n));

// Equivalent:
let v = Vec::from_fn(n, f);
let v = Vec::from_iter(range(0, n).map(f));

// Equivalent:
v.grow(n, a);
v.extend(repeat(a).take(n));

// Equivalent:
v.grow_fn(n, f);
v.extend(range(0, n).map(f));

While these replacements are slightly longer, an important aspect of ergonomics is memorability: by placing greater emphasis on iterators, programmers will quickly learn the iterator APIs and have those at their fingertips, while remembering ad hoc method variants like grow_fn is more difficult.

Long-term: removing push_all and push_all_move

The push_all and push_all_move methods on vectors are yet more API variants that could, in principle, go through iterators:

// The following are *semantically* equivalent
v.push_all(some_slice);
v.extend(some_slice.iter_cloned());

// The following are *semantically* equivalent
v.push_all_move(some_vec);
v.extend(some_vec);

However, currently the push_all and push_all_move methods can rely on the exact size of the container being pushed, in order to elide bounds checks. We do not currently have a way to “trust” methods like len on iterators to elide bounds checks. A separate RFC will introduce the notion of a “trusted” method which should support such optimization and allow us to deprecate the push_all and push_all_move variants. (This is unlikely to happen before 1.0, so the methods will probably still be included with “experimental” status, and likely with different names.)

Alternatives

Borrow and the Equiv problem

Variants of Borrow

The original version of Borrow was somewhat more subtle:

/// A trait for borrowing.
/// If `T: Borrow` then `&T` represents data borrowed from `T::Owned`.
trait Borrow for Sized? {
    /// The type being borrowed from.
    type Owned;

    /// Immutably borrow from an owned value.
    fn borrow(&Owned) -> &Self;

    /// Mutably borrow from an owned value.
    fn borrow_mut(&mut Owned) -> &mut Self;
}

trait ToOwned: Borrow {
    /// Produce a new owned value, usually by cloning.
    fn to_owned(&self) -> Owned;
}

impl<A: Sized> Borrow for A {
    type Owned = A;
    fn borrow(a: &A) -> &A {
        a
    }
    fn borrow_mut(a: &mut A) -> &mut A {
        a
    }
}

impl<A: Clone> ToOwned for A {
    fn to_owned(&self) -> A {
        self.clone()
    }
}

impl Borrow for str {
    type Owned = String;
    fn borrow(s: &String) -> &str {
        s.as_slice()
    }
    fn borrow_mut(s: &mut String) -> &mut str {
        s.as_mut_slice()
    }
}

impl ToOwned for str {
    fn to_owned(&self) -> String {
        self.to_string()
    }
}

impl<T> Borrow for [T] {
    type Owned = Vec<T>;
    fn borrow(s: &Vec<T>) -> &[T] {
        s.as_slice()
    }
    fn borrow_mut(s: &mut Vec<T>) -> &mut [T] {
        s.as_mut_slice()
    }
}

impl<T> ToOwned for [T] {
    fn to_owned(&self) -> Vec<T> {
        self.to_vec()
    }
}

impl<K,V> HashMap<K,V> where K: Borrow + Hash + Eq {
    fn find(&self, k: &K) -> &V { ... }
    fn insert(&mut self, k: K::Owned, v: V) -> Option<V> { ... }
    ...
}

pub enum Cow<'a, T> where T: ToOwned {
    Shared(&'a T),
    Owned(T::Owned)
}

This approach ties Borrow directly to the borrowed data, and uses an associated type to uniquely determine the corresponding owned data type.

For string keys, we would use HashMap<str, V>. Then, the find method would take an &str key argument, while insert would take an owned String. On the other hand, for some other type Foo a HashMap<Foo, V> would take &Foo for find and Foo for insert. (More discussion on the choice of ownership is given in the alternatives section.

Benefits of this alternative:

  • Unlike the current _equiv or find_with methods, or the proposal in the RFC, this approach guarantees coherence about hashing or ordering. For example, HashMap above requires that K (the borrowed key type) is Hash, and will produce hashes from owned keys by first borrowing from them.

  • Unlike the proposal in this RFC, the signature of the methods for maps is very simple – essentially the same as the current find, insert, etc.

  • Like the proposal in this RFC, there is only a single Borrow trait, so it would be possible to standardize on a Map trait later on and include these APIs. The trait could be made somewhat simpler with this alternative form of Borrow, but can be provided in either case; see these comments for details.

  • The Cow data type is simpler than in the RFC’s proposal, since it does not need a type parameter for the owned data.

Drawbacks of this alternative:

  • It’s quite subtle that you want to use HashMap<str, T> rather than HashMap<String, T>. That is, if you try to use a map in the “obvious way” you will not be able to use string slices for lookup, which is part of what this RFC is trying to achieve. The same applies to Cow.

  • The design is somewhat less flexible than the one in the RFC, because (1) there is a fixed choice of owned type corresponding to each borrowed type and (2) you cannot use multiple borrow types for lookups at different types (e.g. using &String sometimes and &str other times). On the other hand, these restrictions guarantee coherence of hashing/equality/comparison.

  • This version of Borrow, mapping from borrowed to owned data, is somewhat less intuitive.

On the balance, the approach proposed in the RFC seems better, because using the map APIs in the obvious ways works by default.

The HashMapKey trait and friends

An earlier proposal for solving the _equiv problem was given in the associated items RFC):

trait HashMapKey : Clone + Hash + Eq {
    type Query: Hash = Self;
    fn compare(&self, other: &Query) -> bool { self == other }
    fn query_to_key(q: &Query) -> Self { q.clone() };
}

impl HashMapKey for String {
    type Query = str;
    fn compare(&self, other: &str) -> bool {
        self.as_slice() == other
    }
    fn query_to_key(q: &str) -> String {
        q.into_string()
    }
}

impl<K,V> HashMap<K,V> where K: HashMapKey {
    fn find(&self, q: &K::Query) -> &V { ... }
}

This solution has several drawbacks, however:

  • It requires a separate trait for different kinds of maps – one for HashMap, one for TreeMap, etc.

  • It requires that a trait be implemented on a given key without providing a blanket implementation. Since you also need different traits for different maps, it’s easy to imagine cases where a out-of-crate type you want to use as a key doesn’t implement the key trait, forcing you to newtype.

  • It doesn’t help with the MaybeOwned problem.

Daniel Micay’s hack

@strcat has a PR that makes it possible to, for example, coerce a &str to an &String value.

This provides some help for the _equiv problem, since the _equiv methods could potentially be dropped. However, there are a few downsides:

  • Using a map with string keys is still a bit more verbose:

    map.find("some static string".as_string()) // with the hack
    map.find("some static string")             // with this RFC
  • The solution is specialized to strings and vectors, and does not necessarily support user-defined unsized types or slices.

  • It doesn’t help with the MaybeOwned problem.

  • It exposes some representation interplay between slices and references to owned values, which we may not want to commit to or reveal.

For IntoIterator

Handling of for loops

The fact that for x in v moves elements from v, while for x in v.iter() yields references, may be a bit surprising. On the other hand, moving is the default almost everywhere in Rust, and with the proposed approach you get to use & and &mut to easily select other forms of iteration.

(See @huon’s comment for additional drawbacks.)

Unfortunately, it’s a bit tricky to make for use by-ref iterators instead. The problem is that an iterator is IntoIterator, but it is not Iterable (or whatever we call the by-reference trait). Why? Because IntoIterator gives you an iterator that can be used only once, while Iterable allows you to ask for iterators repeatedly.

If for demanded an Iterable, then for x in v.iter() and for x in v.iter_mut() would cease to work – we’d have to find some other approach. It might be doable, but it’s not obvious how to do it.

Input versus output type parameters

An important aspect of the IntoIterator design is that the element type is an associated type, not an input type.

This is a tradeoff:

  • Making it an associated type means that the for examples work, because the type of Self uniquely determines the element type for iteration, aiding type inference.

  • Making it an input type would forgo those benefits, but would allow some additional flexibility. For example, you could implement IntoIterator<A> for an iterator on &A when A is cloned, therefore implicitly cloning as needed to make the ownership work out (and obviating the need for iter_cloned). However, we have generally kept away from this kind of implicit magic, especially when it can involve hidden costs like cloning, so the more explicit design given in this RFC seems best.

Downsides

Design tradeoffs were discussed inline.

Unresolved questions

Unresolved conventions/APIs

As mentioned above, this RFC does not resolve the question of what to call set operations that update the set in place.

It likewise does not settle the APIs that appear in only single concrete collections. These will largely be handled through the API stabilization process, unless radical changes are proposed.

Finally, additional methods provided via the IntoIterator API are left for future consideration.

Coercions

Using the Borrow trait, it might be possible to safely add a coercion for auto-slicing:

  If T: Borrow:
    coerce  &'a T::Owned      to  &'a T
    coerce  &'a mut T::Owned  to  &'a mut T

For sized types, this coercion is forced to be trivial, so the only time it would involve running user code is for unsized values.

A general story about such coercions will be left to a follow-up RFC.

Summary

This is a conventions RFC for formalizing the basic conventions around error handling in Rust libraries.

The high-level overview is:

  • For catastrophic errors, abort the process or fail the task depending on whether any recovery is possible.

  • For contract violations, fail the task. (Recover from programmer errors at a coarse grain.)

  • For obstructions to the operation, use Result (or, less often, Option). (Recover from obstructions at a fine grain.)

  • Prefer liberal function contracts, especially if reporting errors in input values may be useful to a function’s caller.

This RFC follows up on two earlier attempts by giving more leeway in when to fail the task.

Motivation

Rust provides two basic strategies for dealing with errors:

  • Task failure, which unwinds to at least the task boundary, and by default propagates to other tasks through poisoned channels and mutexes. Task failure works well for coarse-grained error handling.

  • The Result type, which allows functions to signal error conditions through the value that they return. Together with a lint and the try! macro, Result works well for fine-grained error handling.

However, while there have been some general trends in the usage of the two handling mechanisms, we need to have formal guidelines in order to ensure consistency as we stabilize library APIs. That is the purpose of this RFC.

For the most part, the RFC proposes guidelines that are already followed today, but it tries to motivate and clarify them.

Detailed design

Errors fall into one of three categories:

  • Catastrophic errors, e.g. out-of-memory.
  • Contract violations, e.g. wrong input encoding, index out of bounds.
  • Obstructions, e.g. file not found, parse error.

The basic principle of the conventions is that:

  • Catastrophic errors and programming errors (bugs) can and should only be recovered at a coarse grain, i.e. a task boundary.
  • Obstructions preventing an operation should be reported at a maximally fine grain – to the immediate invoker of the operation.

Catastrophic errors

An error is catastrophic if there is no meaningful way for the current task to continue after the error occurs.

Catastrophic errors are extremely rare, especially outside of libstd.

Canonical examples: out of memory, stack overflow.

For catastrophic errors, fail the task.

For errors like stack overflow, Rust currently aborts the process, but could in principle fail the task, which (in the best case) would allow reporting and recovery from a supervisory task.

Contract violations

An API may define a contract that goes beyond the type checking enforced by the compiler. For example, slices support an indexing operation, with the contract that the supplied index must be in bounds.

Contracts can be complex and involve more than a single function invocation. For example, the RefCell type requires that borrow_mut not be called until all existing borrows have been relinquished.

For contract violations, fail the task.

A contract violation is always a bug, and for bugs we follow the Erlang philosophy of “let it crash”: we assume that software will have bugs, and we design coarse-grained task boundaries to report, and perhaps recover, from these bugs.

Contract design

One subtle aspect of these guidelines is that the contract for a function is chosen by an API designer – and so the designer also determines what counts as a violation.

This RFC does not attempt to give hard-and-fast rules for designing contracts. However, here are some rough guidelines:

  • Prefer expressing contracts through static types whenever possible.

  • It must be possible to write code that uses the API without violating the contract.

  • Contracts are most justified when violations are inarguably bugs – but this is surprisingly rare.

  • Consider whether the API client could benefit from the contract-checking logic. The checks may be expensive. Or there may be useful programming patterns where the client does not want to check inputs before hand, but would rather attempt the operation and then find out whether the inputs were invalid.

  • When a contract violation is the only kind of error a function may encounter – i.e., there are no obstructions to its success other than “bad” inputs – using Result or Option instead is especially warranted. Clients can then use unwrap to assert that they have passed valid input, or re-use the error checking done by the API for their own purposes.

  • When in doubt, use loose contracts and instead return a Result or Option.

Obstructions

An operation is obstructed if it cannot be completed for some reason, even though the operation’s contract has been satisfied. Obstructed operations may have (documented!) side effects – they are not required to roll back after encountering an obstruction. However, they should leave the data structures in a “coherent” state (satisfying their invariants, continuing to guarantee safety, etc.).

Obstructions may involve external conditions (e.g., I/O), or they may involve aspects of the input that are not covered by the contract.

Canonical examples: file not found, parse error.

For obstructions, use Result

The Result<T,E> type represents either a success (yielding T) or failure (yielding E). By returning a Result, a function allows its clients to discover and react to obstructions in a fine-grained way.

What about Option?

The Option type should not be used for “obstructed” operations; it should only be used when a None return value could be considered a “successful” execution of the operation.

This is of course a somewhat subjective question, but a good litmus test is: would a reasonable client ever ignore the result? The Result type provides a lint that ensures the result is actually inspected, while Option does not, and this difference of behavior can help when deciding between the two types.

Another litmus test: can the operation be understood as asking a question (possibly with sideeffects)? Operations like pop on a vector can be viewed as asking for the contents of the first element, with the side effect of removing it if it exists – with an Option return value.

Do not provide both Result and fail! variants.

An API should not provide both Result-producing and failing versions of an operation. It should provide just the Result version, allowing clients to use try! or unwrap instead as needed. This is part of the general pattern of cutting down on redundant variants by instead using method chaining.

There is one exception to this rule, however. Some APIs are strongly oriented around failure, in the sense that their functions/methods are explicitly intended as assertions. If there is no other way to check in advance for the validity of invoking an operation foo, however, the API may provide a foo_catch variant that returns a Result.

The main examples in libstd that currently provide both variants are:

  • Channels, which are the primary point of failure propagation between tasks. As such, calling recv() is an assertion that the other end of the channel is still alive, which will propagate failures from the other end of the channel. On the other hand, since there is no separate way to atomically test whether the other end has hung up, channels provide a recv_opt variant that produces a Result.

    Note: the _opt suffix would be replaced by a _catch suffix if this RFC is accepted.

  • RefCell, which provides a dynamic version of the borrowing rules. Calling the borrow() method is intended as an assertion that the cell is in a borrowable state, and will fail! otherwise. On the other hand, there is no separate way to check the state of the RefCell, so the module provides a try_borrow variant that produces a Result.

    Note: the try_ prefix would be replaced by a _catch catch if this RFC is accepted.

(Note: it is unclear whether these APIs will continue to provide both variants.)

Drawbacks

The main drawbacks of this proposal are:

  • Task failure remains somewhat of a landmine: one must be sure to document, and be aware of, all relevant function contracts in order to avoid task failure.

  • The choice of what to make part of a function’s contract remains somewhat subjective, so these guidelines cannot be used to decisively resolve disagreements about an API’s design.

The alternatives mentioned below do not suffer from these problems, but have drawbacks of their own.

Alternatives

Two alternative designs have been given in earlier RFCs, both of which take a much harder line on using fail! (or, put differently, do not allow most functions to have contracts).

As was pointed out by @SiegeLord, however, mixing what might be seen as contract violations with obstructions can make it much more difficult to write obstruction-robust code; see the linked comment for more detail.

Naming

There are numerous possible suffixes for a Result-producing variant:

  • _catch, as proposed above. As @lilyball points out, this name connotes exception handling, which could be considered misleading. However, since it effectively prevents further unwinding, catching an exception may indeed be the right analogy.

  • _result, which is straightforward but not as informative/suggestive as some of the other proposed variants.

  • try_ prefix. Also connotes exception handling, but has an unfortunately overlap with the common use of try_ for nonblocking variants (which is in play for recv in particular).

Summary

This is a conventions RFC for settling the location of unsafe APIs relative to the types they work with, as well as the use of raw submodules.

The brief summary is:

  • Unsafe APIs should be made into methods or static functions in the same cases that safe APIs would be.

  • raw submodules should be used only to define explicit low-level representations.

Motivation

Many data structures provide unsafe APIs either for avoiding checks or working directly with their (otherwise private) representation. For example, string provides:

  • An as_mut_vec method on String that provides a Vec<u8> view of the string. This method makes it easy to work with the byte-based representation of the string, but thereby also allows violation of the utf8 guarantee.

  • A raw submodule with a number of free functions, like from_parts, that constructs a String instances from a raw-pointer-based representation, a from_utf8 variant that does not actually check for utf8 validity, and so on. The unifying theme is that all of these functions avoid checking some key invariant.

The problem is that currently, there is no clear/consistent guideline about which of these APIs should live as methods/static functions associated with a type, and which should live in a raw submodule. Both forms appear throughout the standard library.

Detailed design

The proposed convention is:

  • When an unsafe function/method is clearly “about” a certain type (as a way of constructing, destructuring, or modifying values of that type), it should be a method or static function on that type. This is the same as the convention for placement of safe functions/methods. So functions like string::raw::from_parts would become static functions on String.

  • raw submodules should only be used to define low-level types/representations (and methods/functions on them). Methods for converting to/from such low-level types should be available directly on the high-level types. Examples: core::raw, sync::raw.

The benefits are:

  • Ergonomics. You can gain easy access to unsafe APIs merely by having a value of the type (or, for static functions, importing the type).

  • Consistency and simplicity. The rules for placement of unsafe APIs are the same as those for safe APIs.

The perspective here is that marking APIs unsafe is enough to deter their use in ordinary situations; they don’t need to be further distinguished by placement into a separate module.

There are also some naming conventions to go along with unsafe static functions and methods:

  • When an unsafe function/method is an unchecked variant of an otherwise safe API, it should be marked using an _unchecked suffix.

    For example, the String module should provide both from_utf8 and from_utf8_unchecked constructors, where the latter does not actually check the utf8 encoding. The string::raw::slice_bytes and string::raw::slice_unchecked functions should be merged into a single slice_unchecked method on strings that checks neither bounds nor utf8 boundaries.

  • When an unsafe function/method produces or consumes a low-level representation of a data structure, the API should use raw in its name. Specifically, from_raw_parts is the typical name used for constructing a value from e.g. a pointer-based representation.

  • Otherwise, consider using a name that suggests why the API is unsafe. In some cases, like String::as_mut_vec, other stronger conventions apply, and the unsafe qualifier on the signature (together with API documentation) is enough.

The unsafe methods and static functions for a given type should be placed in their own impl block, at the end of the module defining the type; this will ensure that they are grouped together in rustdoc. (Thanks @lilyball for the suggestion.)

Drawbacks

One potential drawback of these conventions is that the documentation for a module will be cluttered with rarely-used unsafe APIs, whereas the raw submodule approach neatly groups these APIs. But rustdoc could easily be changed to either hide or separate out unsafe APIs by default, and in the meantime the impl block grouping should help.

More specifically, the convention of placing unsafe constructors in raw makes them very easy to find. But the usual from_ convention, together with the naming conventions suggested above, should make it fairly easy to discover such constructors even when they’re supplied directly as static functions.

More generally, these conventions give unsafe APIs more equal status with safe APIs. Whether this is a drawback depends on your philosophy about the status of unsafe programming. But on a technical level, the key point is that the APIs are marked unsafe, so users still have to opt-in to using them. Ed note: from my perspective, low-level/unsafe programming is important to support, and there is no reason to penalize its ergonomics given that it’s opt-in anyway.

Alternatives

There are a few alternatives:

  • Rather than providing unsafe APIs directly as methods/static functions, they could be grouped into a single extension trait. For example, the String type could be accompanied by a StringRaw extension trait providing APIs for working with raw string representations. This would allow a clear grouping of unsafe APIs, while still providing them as methods/static functions and allowing them to easily be imported with e.g. use std::string::StringRaw. On the other hand, it still further penalizes the raw APIs (beyond marking them unsafe), and given that rustdoc could easily provide API grouping, it’s unclear exactly what the benefit is.

  • (Suggested by @lilyball):

    Use raw for functions that construct a value of the type without checking for one or more invariants.

    The advantage is that it’s easy to find such invariant-ignoring functions. The disadvantage is that their ergonomics is worsened, since they much be separately imported or referenced through a lengthy path:

    // Compare the ergonomics:
    string::raw::slice_unchecked(some_string, start, end)
    some_string.slice_unchecked(start, end)
  • Another suggestion by @lilyball is to keep the basic structure of raw submodules, but use associated types to improve the ergonomics. Details (and discussions of pros/cons) are in this comment.

  • Use raw submodules to group together all manipulation of low-level representations. No module in std currently does this; existing modules provide some free functions in raw, and some unsafe methods, without a clear driving principle. The ergonomics of moving everything into free functions in a raw submodule are quite poor.

Unresolved questions

The core::raw module provides structs with public representations equivalent to several built-in and library types (boxes, closures, slices, etc.). It’s not clear whether the name of this module, or the location of its contents, should change as a result of this RFC. The module is a special case, because not all of the types it deals with even have corresponding modules/type declarations – so it probably suffices to leave decisions about it to the API stabilization process.

Summary

Add the following coercions:

  • From &T to &U when T: Deref<U>.
  • From &mut T to &U when T: Deref<U>.
  • From &mut T to &mut U when T: DerefMut<U>

These coercions eliminate the need for “cross-borrowing” (things like &**v) and calls to as_slice.

Motivation

Rust currently supports a conservative set of implicit coercions that are used when matching the types of arguments against those given for a function’s parameters. For example, if T: Trait then &T is implicitly coerced to &Trait when used as a function argument:

trait MyTrait { ... }
struct MyStruct { ... }
impl MyTrait for MyStruct { ... }

fn use_trait_obj(t: &MyTrait) { ... }
fn use_struct(s: &MyStruct) {
    use_trait_obj(s)    // automatically coerced from &MyStruct to &MyTrait
}

In older incarnations of Rust, in which types like vectors were built in to the language, coercions included things like auto-borrowing (taking T to &T), auto-slicing (taking Vec<T> to &[T]) and “cross-borrowing” (taking Box<T> to &T). As built-in types migrated to the library, these coercions have disappeared: none of them apply today. That means that you have to write code like &**v to convert &Box<T> or Rc<RefCell<T>> to &T and v.as_slice() to convert Vec<T> to &T.

The ergonomic regression was coupled with a promise that we’d improve things in a more general way later on.

“Later on” has come! The premise of this RFC is that (1) we have learned some valuable lessons in the interim and (2) there is a quite conservative kind of coercion we can add that dramatically improves today’s ergonomic state of affairs.

Detailed design

Design principles

The centrality of ownership and borrowing

As Rust has evolved, a theme has emerged: ownership and borrowing are the focal point of Rust’s design, and the key enablers of much of Rust’s achievements.

As such, reasoning about ownership/borrowing is a central aspect of programming in Rust.

In the old coercion model, borrowing could be done completely implicitly, so an invocation like:

foo(bar, baz, quux)

might move bar, immutably borrow baz, and mutably borrow quux. To understand the flow of ownership, then, one has to be aware of the details of all function signatures involved – it is not possible to see ownership at a glance.

When auto-borrowing was removed, this reasoning difficulty was cited as a major motivator:

Code readability does not necessarily benefit from autoref on arguments:

let a = ~Foo;
foo(a); // reading this code looks like it moves `a`
fn foo(_: &Foo) {} // ah, nevermind, it doesn't move `a`!

let mut a = ~[ ... ];
sort(a); // not only does this not move `a`, but it mutates it!

Having to include an extra & or &mut for arguments is a slight inconvenience, but it makes it much easier to track ownership at a glance. (Note that ownership is not entirely explicit, due to self and macros; see the appendix.)

This RFC takes as a basic principle: Coercions should never implicitly borrow from owned data.

This is a key difference from the cross-borrowing RFC.

Limit implicit execution of arbitrary code

Another positive aspect of Rust’s current design is that a function call like foo(bar, baz) does not invoke arbitrary code (general implicit coercions, as found in e.g. Scala). It simply executes foo.

The tradeoff here is similar to the ownership tradeoff: allowing arbitrary implicit coercions means that a programmer must understand the types of the arguments given, the types of the parameters, and all applicable coercion code in order to understand what code will be executed. While arbitrary coercions are convenient, they come at a substantial cost in local reasoning about code.

Of course, method dispatch can implicitly execute code via Deref. But Deref is a pretty specialized tool:

  • Each type T can only deref to one other type.

    (Note: this restriction is not currently enforced, but will be enforceable once associated types land.)

  • Deref makes all the methods of the target type visible on the source type.

  • The source and target types are both references, limiting what the deref code can do.

These characteristics combined make Deref suitable for smart pointer-like types and little else. They make Deref implementations relatively rare. And as a consequence, you generally know when you’re working with a type implementing Deref.

This RFC takes as a basic principle: Coercions should narrowly limit the code they execute.

Coercions through Deref are considered narrow enough.

The proposal

The idea is to introduce a coercion corresponding to Deref/DerefMut, but only for already-borrowed values:

  • From &T to &U when T: Deref<U>.
  • From &mut T to &U when T: Deref<U>.
  • From &mut T to &mut U when T: DerefMut<U>

These coercions are applied recursively, similarly to auto-deref for method dispatch.

Here is a simple pseudocode algorithm for determining the applicability of coercions. Let HasBasicCoercion(T, U) be a procedure for determining whether T can be coerced to U using today’s coercion rules (i.e. without deref). The general HasCoercion(T, U) procedure would work as follows:

HasCoercion(T, U):

  if HasBasicCoercion(T, U) then
      true
  else if T = &V and V: Deref<W> then
      HasCoercion(&W, U)
  else if T = &mut V and V: Deref<W> then
      HasCoercion(&W, U)
  else if T = &mut V and V: DerefMut<W> then
      HasCoercion(&W, U)
  else
      false

Essentially, the procedure looks for applicable “basic” coercions at increasing levels of deref from the given argument, just as method resolution searches for applicable methods at increasing levels of deref.

Unlike method resolution, however, this coercion does not automatically borrow.

Benefits of the design

Under this coercion design, we’d see the following ergonomic improvements for “cross-borrowing”:

fn use_ref(t: &T) { ... }
fn use_mut(t: &mut T) { ... }

fn use_rc(t: Rc<T>) {
    use_ref(&*t);  // what you have to write today
    use_ref(&t);   // what you'd be able to write
}

fn use_mut_box(t: &mut Box<T>) {
    use_mut(&mut *t); // what you have to write today
    use_mut(t);       // what you'd be able to write

    use_ref(*t);      // what you have to write today
    use_ref(t);       // what you'd be able to write
}

fn use_nested(t: &Box<T>) {
    use_ref(&**t);  // what you have to write today
    use_ref(t);     // what you'd be able to write (note: recursive deref)
}

In addition, if Vec<T>: Deref<[T]> (as proposed here), slicing would be automatic:

fn use_slice(s: &[u8]) { ... }

fn use_vec(v: Vec<u8>) {
    use_slice(v.as_slice());    // what you have to write today
    use_slice(&v);              // what you'd be able to write
}

fn use_vec_ref(v: &Vec<u8>) {
    use_slice(v.as_slice());    // what you have to write today
    use_slice(v);               // what you'd be able to write
}

Characteristics of the design

The design satisfies both of the principles laid out in the Motivation:

  • It does not introduce implicit borrows of owned data, since it only applies to already-borrowed data.

  • It only applies to Deref types, which means there is only limited potential for implicitly running unknown code; together with the expectation that programmers are generally aware when they are using Deref types, this should retain the kind of local reasoning Rust programmers can do about function/method invocations today.

There is a conceptual model implicit in the design here: & is a “borrow” operator, and richer coercions are available between borrowed types. This perspective is in opposition to viewing & primarily as adding a layer of indirection – a view that, given compiler optimizations, is often inaccurate anyway.

Drawbacks

As with any mechanism that implicitly invokes code, deref coercions make it more complex to fully understand what a given piece of code is doing. The RFC argued inline that the design conserves local reasoning in practice.

As mentioned above, this coercion design also changes the mental model surrounding &, and in particular somewhat muddies the idea that it creates a pointer. This change could make Rust more difficult to learn (though note that it puts more attention on ownership), though it would make it more convenient to use in the long run.

Alternatives

The main alternative that addresses the same goals as this RFC is the cross-borrowing RFC, which proposes a more aggressive form of deref coercion: it would allow converting e.g. Box<T> to &T and Vec<T> to &[T] directly. The advantage is even greater convenience: in many cases, even & is not necessary. The disadvantage is the change to local reasoning about ownership:

let v = vec![0u8, 1, 2];
foo(v); // is v moved here?
bar(v); // is v still available?

Knowing whether v is moved in the call to foo requires knowing foo’s signature, since the coercion would implicitly borrow from the vector.

Appendix: ownership in Rust today

In today’s Rust, ownership transfer/borrowing is explicit for all function/method arguments. It is implicit only for:

  • self on method invocations. In practice, the name and context of a method invocation is almost always sufficient to infer its move/borrow semantics.

  • Macro invocations. Since macros can expand into arbitrary code, macro invocations can appear to move when they actually borrow.

Summary

Add syntactic sugar for working with the Result type which models common exception handling constructs.

The new constructs are:

  • An ? operator for explicitly propagating “exceptions”.

  • A catch { ... } expression for conveniently catching and handling “exceptions”.

The idea for the ? operator originates from RFC PR 204 by @aturon.

Motivation and overview

Rust currently uses the enum Result type for error handling. This solution is simple, well-behaved, and easy to understand, but often gnarly and inconvenient to work with. We would like to solve the latter problem while retaining the other nice properties and avoiding duplication of functionality.

We can accomplish this by adding constructs which mimic the exception-handling constructs of other languages in both appearance and behavior, while improving upon them in typically Rustic fashion. Their meaning can be specified by a straightforward source-to-source translation into existing language constructs, plus a very simple and obvious new one. (They may also, but need not necessarily, be implemented in this way.)

These constructs are strict additions to the existing language, and apart from the issue of keywords, the legality and behavior of all currently existing Rust programs is entirely unaffected.

The most important additions are a postfix ? operator for propagating “exceptions” and a catch {..} expression for catching them. By an “exception”, for now, we essentially just mean the Err variant of a Result, though the Unresolved Questions includes some discussion of extending to other types.

? operator

The postfix ? operator can be applied to Result values and is equivalent to the current try!() macro. It either returns the Ok value directly, or performs an early exit and propagates the Err value further out. (So given my_result: Result<Foo, Bar>, we have my_result?: Foo.) This allows it to be used for e.g. conveniently chaining method calls which may each “throw an exception”:

foo()?.bar()?.baz()

Naturally, in this case the types of the “exceptions thrown by” foo() and bar() must unify. Like the current try!() macro, the ? operator will also perform an implicit “upcast” on the exception type.

When used outside of a catch block, the ? operator propagates the exception to the caller of the current function, just like the current try! macro does. (If the return type of the function isn’t a Result, then this is a type error.) When used inside a catch block, it propagates the exception up to the innermost catch block, as one would expect.

Requiring an explicit ? operator to propagate exceptions strikes a very pleasing balance between completely automatic exception propagation, which most languages have, and completely manual propagation, which we’d have apart from the try! macro. It means that function calls remain simply function calls which return a result to their caller, with no magic going on behind the scenes; and this also increases flexibility, because one gets to choose between propagation with ? or consuming the returned Result directly.

The ? operator itself is suggestive, syntactically lightweight enough to not be bothersome, and lets the reader determine at a glance where an exception may or may not be thrown. It also means that if the signature of a function changes with respect to exceptions, it will lead to type errors rather than silent behavior changes, which is a good thing. Finally, because exceptions are tracked in the type system, and there is no silent propagation of exceptions, and all points where an exception may be thrown are readily apparent visually, this also means that we do not have to worry very much about “exception safety”.

Exception type upcasting

In a language with checked exceptions and subtyping, it is clear that if a function is declared as throwing a particular type, its body should also be able to throw any of its subtypes. Similarly, in a language with structural sum types (a.k.a. anonymous enums, polymorphic variants), one should be able to throw a type with fewer cases in a function declaring that it may throw a superset of those cases. This is essentially what is achieved by the common Rust practice of declaring a custom error enum with From impls for each of the upstream error types which may be propagated:

enum MyError {
    IoError(io::Error),
    JsonError(json::Error),
    OtherError(...)
}

impl From<io::Error> for MyError { ... }
impl From<json::Error> for MyError { ... }

Here io::Error and json::Error can be thought of as subtypes of MyError, with a clear and direct embedding into the supertype.

The ? operator should therefore perform such an implicit conversion, in the nature of a subtype-to-supertype coercion. The present RFC uses the std::convert::Into trait for this purpose (which has a blanket impl forwarding from From). The precise requirements for a conversion to be “like” a subtyping coercion are an open question; see the “Unresolved questions” section.

catch expressions

This RFC also introduces an expression form catch {..}, which serves to “scope” the ? operator. The catch operator executes its associated block. If no exception is thrown, then the result is Ok(v) where v is the value of the block. Otherwise, if an exception is thrown, then the result is Err(e). Note that unlike other languages, a catch block always catches all errors, and they must all be coercible to a single type, as a Result only has a single Err type. This dramatically simplifies thinking about the behavior of exception-handling code.

Note that catch { foo()? } is essentially equivalent to foo(). catch can be useful if you want to coalesce multiple potential exceptions – catch { foo()?.bar()?.baz()? } – into a single Result, which you wish to then e.g. pass on as-is to another function, rather than analyze yourself. (The last example could also be expressed using a series of and_then calls.)

Detailed design

The meaning of the constructs will be specified by a source-to-source translation. We make use of an “early exit from any block” feature which doesn’t currently exist in the language, generalizes the current break and return constructs, and is independently useful.

Early exit from any block

The capability can be exposed either by generalizing break to take an optional value argument and break out of any block (not just loops), or by generalizing return to take an optional lifetime argument and return from any block, not just the outermost block of the function. This feature is only used in this RFC as an explanatory device, and implementing the RFC does not require exposing it, so I am going to arbitrarily choose the break syntax for the following and won’t discuss the question further.

So we are extending break with an optional value argument: break 'a EXPR. This is an expression of type ! which causes an early return from the enclosing block specified by 'a, which then evaluates to the value EXPR (of course, the type of EXPR must unify with the type of the last expression in that block). This works for any block, not only loops.

[Note: This was since added in RFC 2046]

A completely artificial example:

'a: {
    let my_thing = if have_thing() {
        get_thing()
    } else {
        break 'a None
    };
    println!("found thing: {}", my_thing);
    Some(my_thing)
}

Here if we don’t have a thing, we escape from the block early with None.

If no value is specified, it defaults to (): in other words, the current behavior. We can also imagine there is a magical lifetime 'fn which refers to the lifetime of the whole function: in this case, break 'fn is equivalent to return.

Again, this RFC does not propose generalizing break in this way at this time: it is only used as a way to explain the meaning of the constructs it does propose.

Definition of constructs

Finally we have the definition of the new constructs in terms of a source-to-source translation.

In each case except the first, I will provide two definitions: a single-step “shallow” desugaring which is defined in terms of the previously defined new constructs, and a “deep” one which is “fully expanded”.

Of course, these could be defined in many equivalent ways: the below definitions are merely one way.

  • Construct:

     EXPR?
    

    Shallow:

    match EXPR {
        Ok(a)  => a,
        Err(e) => break 'here Err(e.into())
    }

    Where 'here refers to the innermost enclosing catch block, or to 'fn if there is none.

    The ? operator has the same precedence as ..

  • Construct:

    catch {
        foo()?.bar()
    }

    Shallow:

    'here: {
        Ok(foo()?.bar())
    }

    Deep:

    'here: {
        Ok(match foo() {
            Ok(a) => a,
            Err(e) => break 'here Err(e.into())
        }.bar())
    }

The fully expanded translations get quite gnarly, but that is why it’s good that you don’t have to write them!

In general, the types of the defined constructs should be the same as the types of their definitions.

(As noted earlier, while the behavior of the constructs can be specified using a source-to-source translation in this manner, they need not necessarily be implemented this way.)

As a result of this RFC, both Into and Result would have to become lang items.

Laws

Without any attempt at completeness, here are some things which should be true:

  • catch { foo() } = Ok(foo())
  • catch { Err(e)? } = Err(e.into())
  • catch { try_foo()? } = try_foo().map_err(Into::into)

(In the above, foo() is a function returning any type, and try_foo() is a function returning a Result.)

Feature gates

The two major features here, the ? syntax and catch expressions, will be tracked by independent feature gates. Each of the features has a distinct motivation, and we should evaluate them independently.

Unresolved questions

These questions should be satisfactorily resolved before stabilizing the relevant features, at the latest.

Optional match sugar

Originally, the RFC included the ability to match the errors caught by a catch by writing catch { .. } match { .. }, which could be translated as follows:

  • Construct:

    catch {
        foo()?.bar()
    } match {
        A(a) => baz(a),
        B(b) => quux(b)
    }

    Shallow:

    match (catch {
        foo()?.bar()
    }) {
        Ok(a) => a,
        Err(e) => match e {
            A(a) => baz(a),
            B(b) => quux(b)
        }
    }

    Deep:

    match ('here: {
        Ok(match foo() {
            Ok(a) => a,
            Err(e) => break 'here Err(e.into())
        }.bar())
    }) {
        Ok(a) => a,
        Err(e) => match e {
            A(a) => baz(a),
            B(b) => quux(b)
        }
    }

However, it was removed for the following reasons:

  • The catch (originally: try) keyword adds the real expressive “step up” here, the match (originally: catch) was just sugar for unwrap_or.
  • It would be easy to add further sugar in the future, once we see how catch is used (or not used) in practice.
  • There was some concern about potential user confusion about two aspects:
    • catch { } yields a Result<T,E> but catch { } match { } yields just T;
    • catch { } match { } handles all kinds of errors, unlike try/catch in other languages which let you pick and choose.

It may be worth adding such a sugar in the future, or perhaps a variant that binds irrefutably and does not immediately lead into a match block.

Choice of keywords

The RFC to this point uses the keyword catch, but there are a number of other possibilities, each with different advantages and drawbacks:

  • try { ... } catch { ... }

  • try { ... } match { ... }

  • try { ... } handle { ... }

  • catch { ... } match { ... }

  • catch { ... } handle { ... }

  • catch ... (without braces or a second clause)

Among the considerations:

  • Simplicity. Brevity.

  • Following precedent from existing, popular languages, and familiarity with respect to their analogous constructs.

  • Fidelity to the constructs’ actual behavior. For instance, the first clause always catches the “exception”; the second only branches on it.

  • Consistency with the existing try!() macro. If the first clause is called try, then try { } and try!() would have essentially inverse meanings.

  • Language-level backwards compatibility when adding new keywords. I’m not sure how this could or should be handled.

Semantics for “upcasting”

What should the contract for a From/Into impl be? Are these even the right traits to use for this feature?

Two obvious, minimal requirements are:

  • It should be pure: no side effects, and no observation of side effects. (The result should depend only on the argument.)

  • It should be total: no panics or other divergence, except perhaps in the case of resource exhaustion (OOM, stack overflow).

The other requirements for an implicit conversion to be well-behaved in the context of this feature should be thought through with care.

Some further thoughts and possibilities on this matter, only as brainstorming:

  • It should be “like a coercion from subtype to supertype”, as described earlier. The precise meaning of this is not obvious.

  • A common condition on subtyping coercions is coherence: if you can compound-coerce to go from A to Z indirectly along multiple different paths, they should all have the same end result.

  • It should be lossless, or in other words, injective: it should map each observably-different element of the input type to observably-different elements of the output type. (Observably-different means that it is possible to write a program which behaves differently depending on which one it gets, modulo things that “shouldn’t count” like observing execution time or resource usage.)

  • It should be unambiguous, or preserve the meaning of the input: impl From<u8> for u32 as x as u32 feels right; as (x as u32) * 12345 feels wrong, even though this is perfectly pure, total, and injective. What this means precisely in the general case is unclear.

  • The types converted between should the “same kind of thing”: for instance, the existing impl From<u32> for Ipv4Addr feels suspect on this count. (This perhaps ties into the subtyping angle: Ipv4Addr is clearly not a supertype of u32.)

Forwards-compatibility

If we later want to generalize this feature to other types such as Option, as described below, will we be able to do so while maintaining backwards-compatibility?

Monadic do notation

There have been many comparisons drawn between this syntax and monadic do notation. Before stabilizing, we should determine whether we plan to make changes to better align this feature with a possible do notation (for example, by removing the implicit Ok at the end of a catch block). Note that such a notation would have to extend the standard monadic bind to accommodate rich control flow like break, continue, and return.

Drawbacks

  • Increases the syntactic surface area of the language.

  • No expressivity is added, only convenience. Some object to “there’s more than one way to do it” on principle.

  • If at some future point we were to add higher-kinded types and syntactic sugar for monads, a la Haskell’s do or Scala’s for, their functionality may overlap and result in redundancy. However, a number of challenges would have to be overcome for a generic monadic sugar to be able to fully supplant these features: the integration of higher-kinded types into Rust’s type system in the first place, the shape of a Monad trait in a language with lifetimes and move semantics, interaction between the monadic control flow and Rust’s native control flow (the “ambient monad”), automatic upcasting of exception types via Into (the exception (Either, Result) monad normally does not do this, and it’s not clear whether it can), and potentially others.

Alternatives

  • Don’t.

  • Only add the ? operator, but not catch expressions.

  • Instead of a built-in catch construct, attempt to define one using macros. However, this is likely to be awkward because, at least, macros may only have their contents as a single block, rather than two. Furthermore, macros are excellent as a “safety net” for features which we forget to add to the language itself, or which only have specialized use cases; but generally useful control flow constructs still work better as language features.

  • Add first-class checked exceptions, which are propagated automatically (without an ? operator).

    This has the drawbacks of being a more invasive change and duplicating functionality: each function must choose whether to use checked exceptions via throws, or to return a Result. While the two are isomorphic and converting between them is easy, with this proposal, the issue does not even arise, as exception handling is defined in terms of Result. Furthermore, automatic exception propagation raises the specter of “exception safety”: how serious an issue this would actually be in practice, I don’t know - there’s reason to believe that it would be much less of one than in C++.

  • Wait (and hope) for HKTs and generic monad sugar.

Future possibilities

Expose a generalized form of break or return as described

This RFC doesn’t propose doing so at this time, but as it would be an independently useful feature, it could be added as well.

throw and throws

It is possible to carry the exception handling analogy further and also add throw and throws constructs.

throw is very simple: throw EXPR is essentially the same thing as Err(EXPR)?; in other words it throws the exception EXPR to the innermost catch block, or to the function’s caller if there is none.

A throws clause on a function:

fn foo(arg: Foo) -> Bar throws Baz { ... }

would mean that instead of writing return Ok(foo) and return Err(bar) in the body of the function, one would write return foo and throw bar, and these are implicitly turned into Ok or Err for the caller. This removes syntactic overhead from both “normal” and “throwing” code paths and (apart from ? to propagate exceptions) matches what code might look like in a language with native exceptions.

Generalize over Result, Option, and other result-carrying types

Option<T> is completely equivalent to Result<T, ()> modulo names, and many common APIs use the Option type, so it would be useful to extend all of the above syntax to Option, and other (potentially user-defined) equivalent-to-Result types, as well.

This can be done by specifying a trait for types which can be used to “carry” either a normal result or an exception. There are several different, equivalent ways to formulate it, which differ in the set of methods provided, but the meaning in any case is essentially just that you can choose some types Normal and Exception such that Self is isomorphic to Result<Normal, Exception>.

Here is one way:

#[lang(result_carrier)]
trait ResultCarrier {
    type Normal;
    type Exception;
    fn embed_normal(from: Normal) -> Self;
    fn embed_exception(from: Exception) -> Self;
    fn translate<Other: ResultCarrier<Normal=Normal, Exception=Exception>>(from: Self) -> Other;
}

For greater clarity on how these methods work, see the section on impls below. (For a simpler formulation of the trait using Result directly, see further below.)

The translate method says that it should be possible to translate to any other ResultCarrier type which has the same Normal and Exception types. This may not appear to be very useful, but in fact, this is what can be used to inspect the result, by translating it to a concrete, known type such as Result<Normal, Exception> and then, for example, pattern matching on it.

Laws:

  1. For all x, translate(embed_normal(x): A): B = embed_normal(x): B.
  2. For all x, translate(embed_exception(x): A): B = embed_exception(x): B.
  3. For all carrier, translate(translate(carrier: A): B): A = carrier: A.

Here I’ve used explicit type ascription syntax to make it clear that e.g. the types of embed_ on the left and right hand sides are different.

The first two laws say that embedding a result x into one result-carrying type and then translating it to a second result-carrying type should be the same as embedding it into the second type directly.

The third law says that translating to a different result-carrying type and then translating back should be a no-op.

impls of the trait

impl<T, E> ResultCarrier for Result<T, E> {
    type Normal = T;
    type Exception = E;
    fn embed_normal(a: T) -> Result<T, E> { Ok(a) }
    fn embed_exception(e: E) -> Result<T, E> { Err(e) }
    fn translate<Other: ResultCarrier<Normal=T, Exception=E>>(result: Result<T, E>) -> Other {
        match result {
            Ok(a)  => Other::embed_normal(a),
            Err(e) => Other::embed_exception(e)
        }
    }
}

As we can see, translate can be implemented by deconstructing ourself and then re-embedding the contained value into the other result-carrying type.

impl<T> ResultCarrier for Option<T> {
    type Normal = T;
    type Exception = ();
    fn embed_normal(a: T) -> Option<T> { Some(a) }
    fn embed_exception(e: ()) -> Option<T> { None }
    fn translate<Other: ResultCarrier<Normal=T, Exception=()>>(option: Option<T>) -> Other {
        match option {
            Some(a) => Other::embed_normal(a),
            None    => Other::embed_exception(())
        }
    }
}

Potentially also:

impl ResultCarrier for bool {
    type Normal = ();
    type Exception = ();
    fn embed_normal(a: ()) -> bool { true }
    fn embed_exception(e: ()) -> bool { false }
    fn translate<Other: ResultCarrier<Normal=(), Exception=()>>(b: bool) -> Other {
        match b {
            true  => Other::embed_normal(()),
            false => Other::embed_exception(())
        }
    }
}

The laws should be sufficient to rule out any “icky” impls. For example, an impl for Vec where an exception is represented as the empty vector, and a normal result as a single-element vector: here the third law fails, because if the Vec has more than one element to begin with, then it’s not possible to translate to a different result-carrying type and then back without losing information.

The bool impl may be surprising, or not useful, but it is well-behaved: bool is, after all, isomorphic to Result<(), ()>.

Other miscellaneous notes about ResultCarrier

  • Our current lint for unused results could be replaced by one which warns for any unused result of a type which implements ResultCarrier.

  • If there is ever ambiguity due to the result-carrying type being underdetermined (experience should reveal whether this is a problem in practice), we could resolve it by defaulting to Result.

  • Translating between different result-carrying types with the same Normal and Exception types should, but may not necessarily currently be, a machine-level no-op most of the time.

    We could/should make it so that:

    • repr(Option<T>) = repr(Result<T, ()>)
    • repr(bool) = repr(Option<()>) = repr(Result<(), ()>)

    If these hold, then translate between these types could in theory be compiled down to just a transmute. (Whether LLVM is smart enough to do this, I don’t know.)

  • The translate() function smells to me like a natural transformation between functors, but I’m not category theorist enough for it to be obvious.

Alternative formulations of the ResultCarrier trait

All of these have the form:

trait ResultCarrier {
    type Normal;
    type Exception;
    ...methods...
}

and differ only in the methods, which will be given.

Explicit isomorphism with Result

fn from_result(Result<Normal, Exception>) -> Self;
fn to_result(Self) -> Result<Normal, Exception>;

This is, of course, the simplest possible formulation.

The drawbacks are that it, in some sense, privileges Result over other potentially equivalent types, and that it may be less efficient for those types: for any non-Result type, every operation requires two method calls (one into Result, and one out), whereas with the ResultCarrier trait in the main text, they only require one.

Laws:

  • For all x, from_result(to_result(x)) = x.
  • For all x, to_result(from_result(x)) = x.

Laws for the remaining formulations below are left as an exercise for the reader.

Avoid privileging Result, most naive version

fn embed_normal(Normal) -> Self;
fn embed_exception(Exception) -> Self;
fn is_normal(&Self) -> bool;
fn is_exception(&Self) -> bool;
fn assert_normal(Self) -> Normal;
fn assert_exception(Self) -> Exception;

Of course this is horrible.

Destructuring with HOFs (a.k.a. Church/Scott-encoding)

fn embed_normal(Normal) -> Self;
fn embed_exception(Exception) -> Self;
fn match_carrier<T>(Self, FnOnce(Normal) -> T, FnOnce(Exception) -> T) -> T;

This is probably the right approach for Haskell, but not for Rust.

With this formulation, because they each take ownership of them, the two closures may not even close over the same variables!

Destructuring with HOFs, round 2

trait BiOnceFn {
    type ArgA;
    type ArgB;
    type Ret;
    fn callA(Self, ArgA) -> Ret;
    fn callB(Self, ArgB) -> Ret;
}

trait ResultCarrier {
    type Normal;
    type Exception;
    fn normal(Normal) -> Self;
    fn exception(Exception) -> Self;
    fn match_carrier<T>(Self, BiOnceFn<ArgA=Normal, ArgB=Exception, Ret=T>) -> T;
}

Here we solve the environment-sharing problem from above: instead of two objects with a single method each, we use a single object with two methods! I believe this is the most flexible and general formulation (which is however a strange thing to believe when they are all equivalent to each other). Of course, it’s even more awkward syntactically.

Summary

Divide global declarations into two categories:

  • constants declare constant values. These represent a value, not a memory address. This is the most common thing one would reach for and would replace static as we know it today in almost all cases.
  • statics declare global variables. These represent a memory address. They would be rarely used: the primary use cases are global locks, global atomic counters, and interfacing with legacy C libraries.

Motivation

We have been wrestling with the best way to represent globals for some times. There are a number of interrelated issues:

  • Significant addresses and inlining: For optimization purposes, it is useful to be able to inline constant values directly into the program. It is even more useful if those constant values do not have known addresses, because that means the compiler is free to replicate them as it wishes. Moreover, if a constant is inlined into downstream crates, then they must be recompiled whenever that constant changes.
  • Read-only memory: Whenever possible, we’d like to place large constants into read-only memory. But this means that the data must be truly immutable, or else a segfault will result.
  • Global atomic counters and the like: We’d like to make it possible for people to create global locks or atomic counters that can be used without resorting to unsafe code.
  • Interfacing with C code: Some C libraries require the use of global, mutable data. Other times it’s just convenient and threading is not a concern.
  • Initializer constants: There must be a way to have initializer constants for things like locks and atomic counters, so that people can write static MY_COUNTER: AtomicUint = INIT_ZERO or some such. It should not be possible to modify these initializer constants.

The current design is that we have only one keyword, static, which declares a global variable. By default, global variables do not have significant addresses and can be inlined into the program. You can make a global variable have a significant address by marking it #[inline(never)]. Furthermore, you can declare a mutable global using static mut: all accesses to static mut variables are considered unsafe. Because we wish to allow static values to be placed in read-only memory, they are forbidden from having a type that includes interior mutable data (that is, an appearance of UnsafeCell type).

Some concrete problems with this design are:

  • There is no way to have a safe global counter or lock. Those must be placed in static mut variables, which means that access to them is illegal. To resolve this, there is an alternative proposal, according to which, access to static mut is considered safe if the type of the static mut meets the Sync trait.
  • The significance (no pun intended) of the #[inline(never)] annotation is not intuitive.
  • There is no way to have a generic type constant.

Other less practical and more aesthetic concerns are:

  • Although static and let look and feel analogous, the two behave quite differently. Generally speaking, static declarations do not declare variables but rather values, which can be inlined and which do not have fixed addresses. You cannot have interior mutability in a static variable, but you can in a let. So that static variables can appear in patterns, it is illegal to shadow a static variable – but let variables cannot appear in patterns. Etc.
  • There are other constructs in the language, such as nullary enum variants and nullary structs, which look like global data but in fact act quite differently. They are actual values which do not have addresses. They are categorized as rvalues and so forth.

Detailed design

Constants

Reintroduce a const declaration which declares a constant:

const name: type = value;

Constants may be declared in any scope. They cannot be shadowed. Constants are considered rvalues. Therefore, taking the address of a constant actually creates a spot on the local stack – they by definition have no significant addresses. Constants are intended to behave exactly like nullary enum variants.

Possible extension: Generic constants

As a possible extension, it is perfectly reasonable for constants to have generic parameters. For example, the following constant is legal:

struct WrappedOption<T> { value: Option<T> }
const NONE<T> = WrappedOption { value: None }

Note that this makes no sense for a static variable, which represents a memory location and hence must have a concrete type.

Possible extension: constant functions

It is possible to imagine constant functions as well. This could help to address the problem of encapsulating initialization. To avoid the need to specify what kinds of code can execute in a constant function, we can limit them syntactically to a single constant expression that can be expanded at compilation time (no recursion).

struct LockedData<T:Send> { lock: Lock, value: T }

const LOCKED<T:Send>(t: T) -> LockedData<T> {
    LockedData { lock: INIT_LOCK, value: t }
}

This would allow us to make the value field on UnsafeCell private, among other things.

Static variables

Repurpose the static declaration to declare static variables only. Static variables always have single addresses. static variables can optionally be declared as mut. The lifetime of a static variable is 'static. It is not legal to move from a static. Accesses to a static variable generate actual reads and writes: the value is not inlined (but see “Unresolved Questions” below).

Non-mut statics must have a type that meets the Sync bound. All access to the static is considered safe (that is, reading the variable and taking its address). If the type of the static does not contain an UnsafeCell in its interior, the compiler may place it in read-only memory, but otherwise it must be placed in mutable memory.

mut statics may have any type. All access is considered unsafe. They may not be placed in read-only memory.

Globals referencing Globals

const => const

It is possible to create a const or a static which references another const or another static by its address. For example:

struct SomeStruct { x: uint }
const FOO: SomeStruct = SomeStruct { x: 1 };
const BAR: &'static SomeStruct = &FOO;

Constants are generally inlined into the stack frame from which they are referenced, but in a static context there is no stack frame. Instead, the compiler will reinterpret this as if it were written as:

struct SomeStruct { x: uint }
const FOO: SomeStruct = SomeStruct { x: 1 };
const BAR: &'static SomeStruct = {
    static TMP: SomeStruct = FOO;
    &TMP
};

Here a static is introduced to be able to give the const a pointer which does indeed have the 'static lifetime. Due to this rewriting, the compiler will disallow SomeStruct from containing an UnsafeCell (interior mutability). In general, a constant A cannot reference the address of another constant B if B contains an UnsafeCell in its interior.

const => static

It is illegal for a constant to refer to another static. A constant represents a constant value while a static represents a memory location, and this sort of reference is difficult to reconcile in light of their definitions.

static => const

If a static references the address of a const, then a similar rewriting happens, but there is no interior mutability restriction (only a Sync restriction).

static => static

It is illegal for a static to reference another static by value. It is required that all references be borrowed. Additionally, not all kinds of borrows are allowed, only explicitly taking the address of another static is allowed. For example, interior borrows of fields and elements or accessing elements of an array are both disallowed.

If a by-value reference were allowed, then this sort of reference would require that the static being referenced fall into one of two categories:

  1. It’s an initializer pattern. This is the purpose of const, however.
  2. The values are kept in sync. This is currently technically infeasible.

Instead of falling into one of these two categories, the compiler will instead disallow any references to statics by value (from other statics).

Patterns

Today, a static is allowed to be used in pattern matching. With the introduction of const, however, a static will be forbidden from appearing in a pattern match, and instead only a const can appear.

Drawbacks

This RFC introduces two keywords for global data. Global data is kind of an edge feature so this feels like overkill. (On the other hand, the only keyword that most Rust programmers should need to know is const – I imagine static variables will be used quite rarely.)

Alternatives

The other design under consideration is to keep the current split but make access to static mut be considered safe if the type of the static mut is Sync. For the details of this discussion, please see RFC 177.

One serious concern is with regard to timing. Adding more things to the Rust 1.0 schedule is inadvisable. Therefore, it would be possible to take a hybrid approach: keep the current static rules, or perhaps the variation where access to static mut is safe, for the time being, and create const declarations after Rust 1.0 is released.

Unresolved questions

  • Should the compiler be allowed to inline the values of static variables which are deeply immutable (and thus force recompilation)?

  • Should we permit static variables whose type is not Sync, but simply make access to them unsafe?

  • Should we permit static variables whose type is not Sync, but whose initializer value does not actually contain interior mutability? For example, a static of Option<UnsafeCell<uint>> with the initializer of None is in theory safe.

  • How hard are the envisioned extensions to implement? If easy, they would be nice to have. If hard, they can wait.

Summary

Restrict which traits can be used to make trait objects.

Currently, we allow any traits to be used for trait objects, but restrict the methods which can be called on such objects. Here, we propose instead restricting which traits can be used to make objects. Despite being less flexible, this will make for better error messages, less surprising software evolution, and (hopefully) better design. The motivation for the proposed change is stronger due to part of the DST changes.

Motivation

Part of the planned, in progress DST work is to allow trait objects where a trait is expected. Example:

fn foo<Sized? T: SomeTrait>(y: &T) { ... }

fn bar(x: &SomeTrait) {
    foo(x)
}

Previous to DST the call to foo was not expected to work because SomeTrait was not a type, so it could not instantiate T. With DST this is possible, and it makes intuitive sense for this to work (an alternative is to require impl SomeTrait for SomeTrait { ... }, but that seems weird and confusing and rather like boilerplate. Note that the precise mechanism here is out of scope for this RFC).

This is only sound if the trait is object-safe. We say a method m on trait T is object-safe if it is legal (in current Rust) to call x.m(...) where x has type &T, i.e., x is a trait object. If all methods in T are object-safe, then we say T is object-safe.

If we ignore this restriction we could allow code such as the following:

trait SomeTrait {
    fn foo(&self, other: &Self) { ... } // assume self and other have the same concrete type
}

fn bar<Sized? T: SomeTrait>(x: &T, y: &T) {
    x.foo(y); // x and y may have different concrete types, pre-DST we could
        // assume that x and y had the same concrete types.
}

fn baz(x: &SomeTrait, y: &SomeTrait) {
    bar(x, y) // x and y may have different concrete types
}

This RFC proposes enforcing object-safety when trait objects are created, rather than where methods on a trait object are called or where we attempt to match traits. This makes both method call and using trait objects with generic code simpler. The downside is that it makes Rust less flexible, since not all traits can be used to create trait objects.

Software evolution is improved with this proposal: imagine adding a non-object-safe method to a previously object-safe trait. With this proposal, you would then get errors wherever a trait-object is created. The error would explain why the trait object could not be created and point out exactly which method was to blame and why. Without this proposal, the only errors you would get would be where a trait object is used with a generic call and would be something like “type error: SomeTrait does not implement SomeTrait” - no indication that the non-object-safe method were to blame, only a failure in trait matching.

Another advantage of this proposal is that it implies that all method-calls can always be rewritten into an equivalent UFCS call. This simplifies the “core language” and makes method dispatch notation – which involves some non-trivial inference – into a kind of “sugar” for the more explicit UFCS notation.

Detailed design

To be precise about object-safety, an object-safe method must meet one of the following conditions:

  • require Self : Sized; or,
  • meet all of the following conditions:
    • must not have any type parameters; and,
    • must have a receiver that has type Self or which dereferences to the Self type;
      • for now, this means self, &self, &mut self, or self: Box<Self>, but eventually this should be extended to custom types like self: Rc<Self> and so forth.
    • must not use Self (in the future, where we allow arbitrary types for the receiver, Self may only be used for the type of the receiver and only where we allow Sized? types).

A trait is object-safe if all of the following conditions hold:

  • all of its methods are object-safe; and,
  • the trait does not require that Self : Sized (see also RFC 546).

When an expression with pointer-to-concrete type is coerced to a trait object, the compiler will check that the trait is object-safe (in addition to the usual check that the concrete type implements the trait). It is an error for the trait to be non-object-safe.

Note that a trait can be object-safe even if some of its methods use features that are not supported with an object receiver. This is true when code that attempted to use those features would only work if the Self type is Sized. This is why all methods that require Self:Sized are exempt from the typical rules. This is also why by-value self methods are permitted, since currently one cannot invoke pass an unsized type by-value (though we consider that a useful future extension).

Drawbacks

This is a breaking change and forbids some safe code which is legal today. This can be addressed in two ways: splitting traits, or adding where Self:Sized clauses to methods that cannot not be used with objects.

Example problem

Here is an example trait that is not object safe:

trait SomeTrait {
    fn foo(&self) -> int { ... }
    
    // Object-safe methods may not return `Self`:
    fn new() -> Self;
}

Splitting a trait

One option is to split a trait into object-safe and non-object-safe parts. We hope that this will lead to better design. We are not sure how much code this will affect, it would be good to have data about this.

trait SomeTrait {
    fn foo(&self) -> int { ... }
}

trait SomeTraitCtor : SomeTrait {
    fn new() -> Self;
}

Adding a where-clause

Sometimes adding a second trait feels like overkill. In that case, it is often an option to simply add a where Self:Sized clause to the methods of the trait that would otherwise violate the object safety rule.

trait SomeTrait {
    fn foo(&self) -> int { ... }
    
    fn new() -> Self
        where Self : Sized; // this condition is new
}

The reason that this makes sense is that if one were writing a generic function with a type parameter T that may range over the trait object, that type parameter would have to be declared ?Sized, and hence would not have access to the new method:

fn baz<T:?Sized+SomeTrait>(t: &T) {
    let v: T = SomeTrait::new(); // illegal because `T : Sized` is not known to hold
}

However, if one writes a function with sized type parameter, which could never be a trait object, then the new function becomes available.

fn baz<T:SomeTrait>(t: &T) {
    let v: T = SomeTrait::new(); // OK
}

Alternatives

We could continue to check methods rather than traits are object-safe. When checking the bounds of a type parameter for a function call where the function is called with a trait object, we would check that all methods are object-safe as part of the check that the actual type parameter satisfies the formal bounds. We could probably give a different error message if the bounds are met, but the trait is not object-safe.

We might in the future use finer-grained reasoning to permit more non-object-safe methods from appearing in the trait. For example, we might permit fn foo() -> Self because it (implicitly) requires that Self be sized. Similarly, we might permit other tests beyond just sized-ness. Any such extension would be backwards compatible.

Unresolved questions

N/A

Edits

  • 2014-02-09. Edited by Nicholas Matsakis to (1) include the requirement that object-safe traits do not require Self:Sized and (2) specify that methods may include where Self:Sized to overcome object safety restrictions.
  • Start Date: 2014-09-19
  • RFC PR: rust-lang/rfcs#256
  • Rust Issue: https://github.com/rust-lang/rfcs/pull/256

Summary

Remove the reference-counting based Gc<T> type from the standard library and its associated support infrastructure from rustc.

Doing so lays a cleaner foundation upon which to prototype a proper tracing GC, and will avoid people getting incorrect impressions of Rust based on the current reference-counting implementation.

Motivation

Ancient History

Long ago, the Rust language had integrated support for automatically managed memory with arbitrary graph structure (notably, multiple references to the same object), via the type constructors @T and @mut T for any T. The intention was that Rust would provide a task-local garbage collector as part of the standard runtime for Rust programs.

As a short-term convenience, @T and @mut T were implemented via reference-counting: each instance of @T/@mut T had a reference count added to it (as well as other meta-data that were again for implementation convenience). To support this, the rustc compiler would emit, for any instruction copying or overwriting an instance of @T/@mut T, code to update the reference count(s) accordingly.

(At the same time, @T was still considered an instance of Copy by the compiler. Maintaining the reference counts of @T means that you cannot create copies of a given type implementing Copy by memcpy’ing blindly; one must distinguish so-called “POD” data that is Copy and contains no @Tfrom "non-POD"Copydata that can contain@T` and thus must be sure to update reference counts when creating a copy.)

Over time, @T was replaced with the library type Gc<T> (and @mut T was rewritten as Gc<RefCell<T>>), but the intention was that Rust would still have integrated support for a garbage collection. To continue supporting the reference-count updating semantics, the Gc<T> type has a lang item, "gc". In effect, all of the compiler support for maintaining the reference-counts from the prior @T was still in place; the move to a library type Gc<T> was just a shift in perspective from the end-user’s point of view (and that of the parser).

Recent history: Removing uses of Gc from the compiler

Largely due to the tireless efforts of eddyb, one of the primary clients of Gc<T>, namely the rustc compiler itself, has little to no remaining uses of Gc<T>.

A new hope

This means that we have an opportunity now, to remove the Gc<T> type from libstd, and its associated built-in reference-counting support from rustc itself.

I want to distinguish removal of the particular reference counting Gc<T> from our compiler and standard library (which is what is being proposed here), from removing the goal of supporting a garbage collected Gc<T> in the future. I (and I think the majority of the Rust core team) still believe that there are use cases that would be well handled by a proper tracing garbage collector.

The expected outcome of removing reference-counting Gc<T> are as follows:

  • A cleaner compiler code base,

  • A cleaner standard library, where Copy data can be indeed copied blindly (assuming the source and target types are in agreement, which is required for a tracing GC),

  • It would become impossible for users to use Gc<T> and then get incorrect impressions about how Rust’s GC would behave in the future. In particular, if we leave the reference-counting Gc<T> in place, then users may end up depending on implementation artifacts that we would be pressured to continue supporting in the future. (Note that Gc<T> is already marked “experimental”, so this particular motivation is not very strong.)

Detailed design

Remove the std::gc module. This, I believe, is the extent of the end-user visible changes proposed by this RFC, at least for users who are using libstd (as opposed to implementing their own).

Then remove the rustc support for Gc<T>. As part of this, we can either leave in or remove the "gc" and "managed_heap" entries in the lang items table (in case they could be of use for a future GC implementation). I propose leaving them, but it does not matter terribly to me. The important thing is that once std::gc is gone, then we can remove the support code associated with those two lang items, which is the important thing.

Drawbacks

Taking out the reference-counting Gc<T> now may lead people to think that Rust will never have a Gc<T>.

  • In particular, having Gc<T> in place now means that it is easier to argue for putting in a tracing collector (since it would be a net win over the status quo, assuming it works).

    (This sub-bullet is a bit of a straw man argument, as I suspect any community resistance to adding a tracing GC will probably be unaffected by the presence or absence of the reference-counting Gc<T>.)

  • As another related note, it may confuse people to take out a Gc<T> type now only to add another implementation with the same name later. (Of course, is that more or less confusing than just replacing the underlying implementation in such a severe manner.)

Users may be using Gc<T> today, and they would have to switch to some other option (such as Rc<T>, though note that the two are not 100% equivalent; see [Gc versus Rc] appendix).

Alternatives

Keep the Gc<T> implementation that we have today, and wait until we have a tracing GC implemented and ready to be deployed before removing the reference-counting infrastructure that had been put in to support @T. (Which may never happen, since adding a tracing GC is only a goal, not a certainty, and thus we may be stuck supporting the reference-counting Gc<T> until we eventually do decide to remove Gc<T> in the future. So this RFC is just suggesting we be proactive and pull that band-aid off now.

Unresolved questions

None yet.

Appendices

Gc versus Rc

There are performance differences between the current ref-counting Gc<T> and the library type Rc<T>, but such differences are beneath the level of abstraction of interest to this RFC. The main user observable difference between the ref-counting Gc<T> and the library type Rc<T> is that cyclic structure allocated via Gc<T> will be torn down when the task itself terminates successfully or via unwind.

The following program illustrates this difference. If you have a program that is using Gc and is relying on this tear-down behavior at task death, then switching to Rc will not suffice.

use std::cell::RefCell;
use std::gc::{GC,Gc};
use std::io::timer;
use std::rc::Rc;
use std::time::Duration;

struct AnnounceDrop { name: String }

#[allow(non_snake_case)]
fn AnnounceDrop<S:Str>(s:S) -> AnnounceDrop {
    AnnounceDrop { name: s.as_slice().to_string() }
}

impl Drop for AnnounceDrop{ 
    fn drop(&mut self) {
       println!("dropping {}", self.name);
    }
}

struct RcCyclic<D> { _on_drop: D, recur: Option<Rc<RefCell<RcCyclic<D>>>> }
struct GcCyclic<D> { _on_drop: D, recur: Option<Gc<RefCell<GcCyclic<D>>>> }

type RRRcell<D> = Rc<RefCell<RcCyclic<D>>>;
type GRRcell<D> = Gc<RefCell<GcCyclic<D>>>;

fn make_rc_and_gc<S:Str>(name: S) -> (RRRcell<AnnounceDrop>, GRRcell<AnnounceDrop>) {
    let name = name.as_slice().to_string();
    let rc_cyclic = Rc::new(RefCell::new(RcCyclic {
        _on_drop: AnnounceDrop(name.clone().append("-rc")),
        recur: None,
    }));

    let gc_cyclic = box (GC) RefCell::new(GcCyclic {
        _on_drop: AnnounceDrop(name.append("-gc")),
        recur: None,
    });

    (rc_cyclic, gc_cyclic)
}

fn make_proc(name: &str, sleep_time: i64, and_then: proc():Send) -> proc():Send {
    let name = name.to_string();
    proc() {
        let (rc_cyclic, gc_cyclic) = make_rc_and_gc(name);

        rc_cyclic.borrow_mut().recur = Some(rc_cyclic.clone());
        gc_cyclic.borrow_mut().recur = Some(gc_cyclic);

        timer::sleep(Duration::seconds(sleep_time));

        and_then();
    }
}

fn main() {
    let (_rc_noncyclic, _gc_noncyclic) = make_rc_and_gc("main-noncyclic");

    spawn(make_proc("success-cyclic", 2, proc () {}));

    spawn(make_proc("failure-cyclic", 1, proc () { fail!("Oop"); }));

    println!("Hello, world!")
}

The above program produces output as follows:

% rustc gc-vs-rc-sample.rs && ./gc-vs-rc-sample
Hello, world!
dropping main-noncyclic-gc
dropping main-noncyclic-rc
task '<unnamed>' failed at 'Oop', gc-vs-rc-sample.rs:60
dropping failure-cyclic-gc
dropping success-cyclic-gc

This illustrates that both Gc<T> and Rc<T> will be reclaimed when used to represent non-cyclic data (the cases labelled main-noncyclic-gc and main-noncyclic-rc. But when you actually complete the cyclic structure, then in the tasks that run to completion (either successfully or unwinding from a failure), we still manage to drop the Gc<T> cyclic structures, illustrated by the printouts from the cases labelled failure-cyclic-gc and success-cyclic-gc.

Summary

Remove drop flags from values implementing Drop, and remove automatic memory zeroing associated with dropping values.

Keep dynamic drop semantics, by having each function maintain a (potentially empty) set of auto-injected boolean flags for the drop obligations for the function that need to be tracked dynamically (which we will call “dynamic drop obligations”).

Motivation

Currently, implementing Drop on a struct (or enum) injects a hidden bit, known as the “drop-flag”, into the struct (and likewise, each of the enum variants). The drop-flag, in tandem with Rust’s implicit zeroing of dropped values, tracks whether a value has already been moved to another owner or been dropped. (See the “How dynamic drop semantics works” appendix for more details if you are unfamiliar with this part of Rust’s current implementation.)

However, the above implementation is sub-optimal; problems include:

  • Most important: implicit memory zeroing is a hidden cost that today all Rust programs pay, in both execution time and code size. With the removal of the drop flag, we can remove implicit memory zeroing (or at least revisit its utility – there may be other motivations for implicit memory zeroing, e.g. to try to keep secret data from being exposed to unsafe code).

  • Hidden bits are bad: Users coming from a C/C++ background expect struct Foo { x: u32, y: u32 } to occupy 8 bytes, but if Foo implements Drop, the hidden drop flag will cause it to double in size (16 bytes). See the [Program illustrating semantic impact of hidden drop flag] appendix for a concrete illustration. Note that when Foo implements Drop, each instance of Foo carries a drop-flag, even in contexts like a Vec<Foo> where a program cannot actually move individual values out of the collection. Thus, the amount of extra memory being used by drop-flags is not bounded by program stack growth; the memory wastage is strewn throughout the heap.

An earlier RFC (the withdrawn RFC PR #210) suggested resolving this problem by switching from a dynamic drop semantics to a “static drop semantics”, which was defined in that RFC as one that performs drop of certain values earlier to ensure that the set of drop-obligations does not differ at any control-flow merge point, i.e. to ensure that the set of values to drop is statically known at compile-time.

However, discussion on the RFC PR #210 comment thread pointed out its policy for inserting early drops into the code is non-intuitive (in other words, that the drop policy should either be more aggressive, a la RFC PR #239, or should stay with the dynamic drop status quo). Also, the mitigating mechanisms proposed by that RFC (NoisyDrop/QuietDrop) were deemed unacceptable.

So, static drop semantics are a non-starter. Luckily, the above strategy is not the only way to implement dynamic drop semantics. Rather than requiring that the set of drop-obligations be the same at every control-flow merge point, we can do a intra-procedural static analysis to identify the set of drop-obligations that differ at any merge point, and then inject a set of stack-local boolean-valued drop-flags that dynamically track them. That strategy is what this RFC is describing.

The expected outcomes are as follows:

  • We remove the drop-flags from all structs/enums that implement Drop. (There are still the injected stack-local drop flags, but those should be cheaper to inject and maintain.)

  • Since invoking drop code is now handled by the stack-local drop flags and we have no more drop-flags on the values themselves, we can (and will) remove memory zeroing.

  • Libraries currently relying on drop doing memory zeroing (i.e. libraries that check whether content is zero to decide whether its fn drop has been invoked will need to be revised, since we will not have implicit memory zeroing anymore.

  • In the common case, most libraries using Drop will not need to change at all from today, apart from the caveat in the previous bullet.

Detailed design

Drop obligations

No struct or enum has an implicit drop-flag. When a local variable is initialized, that establishes a set of “drop obligations”: a set of structural paths (e.g. a local a, or a path to a field b.f.y) that need to be dropped (or moved away to a new owner).

The drop obligations for a local variable x of struct-type T are computed from analyzing the structure of T. If T itself implements Drop, then x is a drop obligation. If T does not implement Drop, then the set of drop obligations is the union of the drop obligations of the fields of T.

When a path is moved to a new location, or consumed by a function call, or when control flow reaches the end of its owner’s lexical scope, the path is removed from the set of drop obligations.

At control-flow merge points, e.g. nodes that have predecessor nodes P_1, P_2, …, P_k with drop obligation sets S_1, S_2, … S_k, we

  • First identify the set of drop obligations that differ between the predecessor nodes, i.e. the set:

    (S_1 | S_2 | ... | S_k) \ (S_1 & S_2 & ... & S_k)

    where | denotes set-union, & denotes set-intersection, \ denotes set-difference. These are the dynamic drop obligations induced by this merge point. Note that if S_1 = S_2 = ... = S_k, the above set is empty.

  • The set of drop obligations for the merge point itself is the union of the drop-obligations from all predecessor points in the control flow, i.e. (S_1 | S_2 | ... | S_k) in the above notation.

    (One could also just use the intersection here; it actually makes no difference to the static analysis, since all of the elements of the difference

    (S_1 | S_2 | ... | S_k) \ (S_1 & S_2 & ... & S_k)

    have already been added to the set of dynamic drop obligations. But the overall code transformation is clearer if one keeps the dynamic drop obligations in the set of drop obligations.)

Stack-local drop flags

For every dynamic drop obligation induced by a merge point, the compiler is responsible for ensure that its drop code is run at some point. If necessary, it will inject and maintain boolean flag analogous to

enum NeedsDropFlag { NeedsLocalDrop, DoNotDrop }

Some compiler analysis may be able to identify dynamic drop obligations that do not actually need to be tracked. Therefore, we do not specify the precise set of boolean flags that are injected.

Example of code with dynamic drop obligations

The function f2 below was copied from the static drop RFC PR #210; it has differing sets of drop obligations at a merge point, necessitating a potential injection of a NeedsDropFlag.

fn f2() {

    // At the outset, the set of drop obligations is
    // just the set of moved input parameters (empty
    // in this case).

    //                                      DROP OBLIGATIONS
    //                                  ------------------------
    //                                  {  }
    let pDD : Pair<D,D> = ...;
    pDD.x = ...;
    //                                  {pDD.x}
    pDD.y = ...;
    //                                  {pDD.x, pDD.y}
    let pDS : Pair<D,S> = ...;
    //                                  {pDD.x, pDD.y, pDS.x}
    let some_d : Option<D>;
    //                                  {pDD.x, pDD.y, pDS.x}
    if test() {
        //                                  {pDD.x, pDD.y, pDS.x}
        {
            let temp = xform(pDD.y);
            //                              {pDD.x,        pDS.x, temp}
            some_d = Some(temp);
            //                              {pDD.x,        pDS.x, temp, some_d}
        } // END OF SCOPE for `temp`
        //                                  {pDD.x,        pDS.x, some_d}

        // MERGE POINT PREDECESSOR 1

    } else {
        {
            //                              {pDD.x, pDD.y, pDS.x}
            let z = D;
            //                              {pDD.x, pDD.y, pDS.x, z}

            // This drops `pDD.y` before
            // moving `pDD.x` there.
            pDD.y = pDD.x;

            //                              {       pDD.y, pDS.x, z}
            some_d = None;
            //                              {       pDD.y, pDS.x, z, some_d}
        } // END OF SCOPE for `z`
        //                                  {       pDD.y, pDS.x, some_d}

        // MERGE POINT PREDECESSOR 2

    }

    // MERGE POINT: set of drop obligations do not
    // match on all incoming control-flow paths.
    //
    // Predecessor 1 has drop obligations
    // {pDD.x,        pDS.x, some_d}
    // and Predecessor 2 has drop obligations
    // {       pDD.y, pDS.x, some_d}.
    //
    // Therefore, this merge point implies that
    // {pDD.x, pDD.y} are dynamic drop obligations,
    // while {pDS.x, some_d} are potentially still
    // resolvable statically (and thus may not need
    // associated boolean flags).

    // The resulting drop obligations are the following:

    //                                  {pDD.x, pDD.y, pDS.x, some_d}.

    // (... some code that does not change drop obligations ...)

    //                                  {pDD.x, pDD.y, pDS.x, some_d}.

    // END OF SCOPE for `pDD`, `pDS`, `some_d`
}

After the static analysis has identified all of the dynamic drop obligations, code is injected to maintain the stack-local drop flags and to do any necessary drops at the appropriate points. Below is the updated fn f2 with an approximation of the injected code.

Note: we say “approximation”, because one does need to ensure that the drop flags are updated in a manner that is compatible with potential task fail!/panic!, because stack unwinding must be informed which state needs to be dropped; i.e. you need to initialize _pDD_dot_x before you start to evaluate a fallible expression to initialize pDD.y.

fn f2_rewritten() {

    // At the outset, the set of drop obligations is
    // just the set of moved input parameters (empty
    // in this case).

    //                                      DROP OBLIGATIONS
    //                                  ------------------------
    //                                  {  }
    let _drop_pDD_dot_x : NeedsDropFlag;
    let _drop_pDD_dot_y : NeedsDropFlag;

    _drop_pDD_dot_x = DoNotDrop;
    _drop_pDD_dot_y = DoNotDrop;

    let pDD : Pair<D,D>;
    pDD.x = ...;
    _drop_pDD_dot_x = NeedsLocalDrop;
    pDD.y = ...;
    _drop_pDD_dot_y = NeedsLocalDrop;

    //                                  {pDD.x, pDD.y}
    let pDS : Pair<D,S> = ...;
    //                                  {pDD.x, pDD.y, pDS.x}
    let some_d : Option<D>;
    //                                  {pDD.x, pDD.y, pDS.x}
    if test() {
        //                                  {pDD.x, pDD.y, pDS.x}
        {
            _drop_pDD_dot_y = DoNotDrop;
            let temp = xform(pDD.y);
            //                              {pDD.x,        pDS.x, temp}
            some_d = Some(temp);
            //                              {pDD.x,        pDS.x, temp, some_d}
        } // END OF SCOPE for `temp`
        //                                  {pDD.x,        pDS.x, some_d}

        // MERGE POINT PREDECESSOR 1

    } else {
        {
            //                              {pDD.x, pDD.y, pDS.x}
            let z = D;
            //                              {pDD.x, pDD.y, pDS.x, z}

            // This drops `pDD.y` before
            // moving `pDD.x` there.
            _drop_pDD_dot_x = DoNotDrop;
            pDD.y = pDD.x;

            //                              {       pDD.y, pDS.x, z}
            some_d = None;
            //                              {       pDD.y, pDS.x, z, some_d}
        } // END OF SCOPE for `z`
        //                                  {       pDD.y, pDS.x, some_d}

        // MERGE POINT PREDECESSOR 2

    }

    // MERGE POINT: set of drop obligations do not
    // match on all incoming control-flow paths.
    //
    // Predecessor 1 has drop obligations
    // {pDD.x,        pDS.x, some_d}
    // and Predecessor 2 has drop obligations
    // {       pDD.y, pDS.x, some_d}.
    //
    // Therefore, this merge point implies that
    // {pDD.x, pDD.y} are dynamic drop obligations,
    // while {pDS.x, some_d} are potentially still
    // resolvable statically (and thus may not need
    // associated boolean flags).

    // The resulting drop obligations are the following:

    //                                  {pDD.x, pDD.y, pDS.x, some_d}.

    // (... some code that does not change drop obligations ...)

    //                                  {pDD.x, pDD.y, pDS.x, some_d}.

    // END OF SCOPE for `pDD`, `pDS`, `some_d`

    // rustc-inserted code (not legal Rust, since `pDD.x` and `pDD.y`
    // are inaccessible).

    if _drop_pDD_dot_x { mem::drop(pDD.x); }
    if _drop_pDD_dot_y { mem::drop(pDD.y); }
}

Note that in a snippet like

       _drop_pDD_dot_y = DoNotDrop;
       let temp = xform(pDD.y);

this is okay, in part because the evaluating the identifier xform is infallible. If instead it were something like:

       _drop_pDD_dot_y = DoNotDrop;
       let temp = lookup_closure()(pDD.y);

then that would not be correct, because we need to set _drop_pDD_dot_y to DoNotDrop after the lookup_closure() invocation.

It may probably be more intellectually honest to write the transformation like:

       let temp = lookup_closure()({ _drop_pDD_dot_y = DoNotDrop; pDD.y });

Control-flow sensitivity

Note that the dynamic drop obligations are based on a control-flow analysis, not just the lexical nesting structure of the code.

In particular: If control flow splits at a point like an if-expression, but the two arms never meet, then they can have completely sets of drop obligations.

This is important, since in coding patterns like loops, one often sees different sets of drop obligations prior to a break compared to a point where the loop repeats, such as a continue or the end of a loop block.

    // At the outset, the set of drop obligations is
    // just the set of moved input parameters (empty
    // in this case).

    //                                      DROP OBLIGATIONS
    //                                  ------------------------
    //                                  {  }
    let mut pDD : Pair<D,D> = mk_dd();
    let mut maybe_set : D;

    //                                  {         pDD.x, pDD.y }
    'a: loop {
        // MERGE POINT

        //                                  {     pDD.x, pDD.y }
        if test() {
            //                                  { pDD.x, pDD.y }
            consume(pDD.x);
            //                                  {        pDD.y }
            break 'a;
        }
        // *not* merge point (only one path, the else branch, flows here)

        //                                  {     pDD.x, pDD.y }

        // never falls through; must merge with 'a loop.
    }

    // RESUME POINT: break 'a above flows here

    //                                  {                pDD.y }

    // This is the point immediately preceding `'b: loop`; (1.) below.

    'b: loop {
        // MERGE POINT
        //
        // There are *three* incoming paths: (1.) the statement
        // preceding `'b: loop`, (2.) the `continue 'b;` below, and
        // (3.) the end of the loop's block below.  The drop
        // obligation for `maybe_set` originates from (3.).

        //                                  {            pDD.y, maybe_set }

        consume(pDD.y);

        //                                  {                 , maybe_set }

        if test() {
            //                                  {             , maybe_set }
            pDD.x = mk_d();
            //                                  { pDD.x       , maybe_set }
            break 'b;
        }

        // *not* merge point (only one path flows here)

        //                                  {                 , maybe_set }

        if test() {
            //                                  {             , maybe_set }
            pDD.y = mk_d();

            // This is (2.) referenced above.   {        pDD.y, maybe_set }
            continue 'b;
        }
        // *not* merge point (only one path flows here)

        //                                  {                 , maybe_set }

        pDD.y = mk_d();
        // This is (3.) referenced above.   {            pDD.y, maybe_set }

        maybe_set = mk_d();
        g(&maybe_set);

        // This is (3.) referenced above.   {            pDD.y, maybe_set }
    }

    // RESUME POINT: break 'b above flows here

    //                                  {         pDD.x       , maybe_set }

    // when we hit the end of the scope of `maybe_set`;
    // check its stack-local flag.

Likewise, a return statement represents another control flow jump, to the end of the function.

Remove implicit memory zeroing

With the above in place, the remainder is relatively trivial. The compiler can be revised to no longer inject a drop flag into structs and enums that implement Drop, and likewise memory zeroing can be removed.

Beyond that, the libraries will obviously need to be audited for dependence on implicit memory zeroing.

Drawbacks

The only reasons not do this are:

  1. Some hypothetical reason to continue doing implicit memory zeroing, or

  2. We want to abandon dynamic drop semantics.

At this point Felix thinks the Rust community has made a strong argument in favor of keeping dynamic drop semantics.

Alternatives

  • Static drop semantics RFC PR #210 has been referenced frequently in this document.

  • Eager drops RFC PR #239 is the more aggressive semantics that would drop values immediately after their final use. This would probably invalidate a number of RAII style coding patterns.

Optional Extensions

A lint identifying dynamic drop obligations

Add a lint (set by default to allow) that reports potential dynamic drop obligations, so that end-user code can opt-in to having them reported. The expected benefits of this are:

  1. developers may have intended for a value to be moved elsewhere on all paths within a function, and,

  2. developers may want to know about how many boolean dynamic drop flags are potentially being injected into their code.

Unresolved questions

How to handle moves out of array[index_expr]

Niko pointed out to me today that my prototype was not addressing moves out of array[index_expr] properly. I was assuming that we would just make such an expression illegal (or that they should already be illegal).

But they are not already illegal, and above assumption that we would make it illegal should have been explicit. That, or we should address the problem in some other way.

To make this concrete, here is some code that runs today:

#[deriving(Show)]
struct AnnounceDrop { name: &'static str }

impl Drop for AnnounceDrop {
    fn drop(&mut self) { println!("dropping {}", self.name); }
}

fn foo<A>(a: [A, ..3], i: uint) -> A {
    a[i]
}

fn main() {
    let a = [AnnounceDrop { name: "fst" },
             AnnounceDrop { name: "snd" },
             AnnounceDrop { name: "thd" }];
    let r = foo(a, 1);
    println!("foo returned {}", r);
}

This prints:

dropping fst
dropping thd
foo returned AnnounceDrop { name: snd }
dropping snd

because it first moves the entire array into foo, and then foo returns the second element, but still needs to drop the rest of the array.

Embedded drop flags and zeroing support this seamlessly, of course. But the whole point of this RFC is to get rid of the embedded per-value drop-flags.

If we want to continue supporting moving out of a[i] (and we probably do, I have been converted on this point), then the drop flag needs to handle this case. Our current thinking is that we can support it by using a single uint flag (as opposed to the booleans used elsewhere) for such array that has been moved out of. The uint flag represents “drop all elements from the array except for the one listed in the flag.” (If it is only moved out of on one branch and not another, then we would either use an Option<uint>, or still use uint and just represent unmoved case via some value that is not valid index, such as the length of the array).

Should we keep #[unsafe_no_drop_flag] ?

Currently there is an unsafe_no_drop_flag attribute that is used to indicate that no drop flag should be associated with a struct/enum, and instead the user-written drop code will be run multiple times (and thus must internally guard itself from its own side-effects; e.g. do not attempt to free the backing buffer for a Vec more than once, by tracking within the Vec itself if the buffer was previously freed).

The “obvious” thing to do is to remove unsafe_no_drop_flag, since the per-value drop flag is going away. However, we could keep the attribute, and just repurpose its meaning to instead mean the following: Never inject a dynamic stack-local drop-flag for this value. Just run the drop code multiple times, just like today.

In any case, since the semantics of this attribute are unstable, we will feature-gate it (with feature name unsafe_no_drop_flag).

Appendices

How dynamic drop semantics works

(This section is just presenting background information on the semantics of drop and the drop-flag as it works in Rust today; it does not contain any discussion of the changes being proposed by this RFC.)

A struct or enum implementing Drop will have its drop-flag automatically set to a non-zero value when it is constructed. When attempting to drop the struct or enum (i.e. when control reaches the end of the lexical scope of its owner), the injected glue code will only execute its associated fn drop if its drop-flag is non-zero.

In addition, the compiler injects code to ensure that when a value is moved to a new location in memory or dropped, then the original memory is entirely zeroed.

A struct/enum definition implementing Drop can be tagged with the attribute #[unsafe_no_drop_flag]. When so tagged, the struct/enum will not have a hidden drop flag embedded within it. In this case, the injected glue code will execute the associated glue code unconditionally, even though the struct/enum value may have been moved to a new location in memory or dropped (in either case, the memory representing the value will have been zeroed).

The above has a number of implications:

  • A program can manually cause the drop code associated with a value to be skipped by first zeroing out its memory.

  • A Drop implementation for a struct tagged with unsafe_no_drop_flag must assume that it will be called more than once. (However, every call to drop after the first will be given zeroed memory.)

Program illustrating semantic impact of hidden drop flag

#![feature(macro_rules)]

use std::fmt;
use std::mem;

#[deriving(Clone,Show)]
struct S {  name: &'static str }

#[deriving(Clone,Show)]
struct Df { name: &'static str }

#[deriving(Clone,Show)]
struct Pair<X,Y>{ x: X, y: Y }

static mut current_indent: uint = 0;

fn indent() -> String {
    String::from_char(unsafe { current_indent }, ' ')
}

impl Drop for Df {
    fn drop(&mut self) {
        println!("{}dropping Df {}", indent(), self.name)
    }
}

macro_rules! struct_Dn {
    ($Dn:ident) => {

        #[unsafe_no_drop_flag]
        #[deriving(Clone,Show)]
        struct $Dn { name: &'static str }

        impl Drop for $Dn {
            fn drop(&mut self) {
                if unsafe { (0,0) == mem::transmute::<_,(uint,uint)>(self.name) } {
                    println!("{}dropping already-zeroed {}",
                             indent(), stringify!($Dn));
                } else {
                    println!("{}dropping {} {}",
                             indent(), stringify!($Dn), self.name)
                }
            }
        }
    }
}

struct_Dn!(DnA)
struct_Dn!(DnB)
struct_Dn!(DnC)

fn take_and_pass<T:fmt::Show>(t: T) {
    println!("{}t-n-p took and will pass: {}", indent(), &t);
    unsafe { current_indent += 4; }
    take_and_drop(t);
    unsafe { current_indent -= 4; }
}

fn take_and_drop<T:fmt::Show>(t: T) {
    println!("{}t-n-d took and will drop: {}", indent(), &t);
}

fn xform(mut input: Df) -> Df {
    input.name = "transformed";
    input
}

fn foo(b: || -> bool) {
    let mut f1 = Df  { name: "f1" };
    let mut n2 = DnC { name: "n2" };
    let f3 = Df  { name: "f3" };
    let f4 = Df  { name: "f4" };
    let f5 = Df  { name: "f5" };
    let f6 = Df  { name: "f6" };
    let n7 = DnA { name: "n7" };
    let _fx = xform(f6);           // `f6` consumed by `xform`
    let _n9 = DnB { name: "n9" };
    let p = Pair { x: f4, y: f5 }; // `f4` and `f5` moved into `p`
    let _f10 = Df { name: "f10" };

    println!("foo scope start: {}", (&f3, &n7));
    unsafe { current_indent += 4; }
    if b() {
        take_and_pass(p.x); // `p.x` consumed by `take_and_pass`, which drops it
    }
    if b() {
        take_and_pass(n7); // `n7` consumed by `take_and_pass`, which drops it
    }
    
    // totally unsafe: manually zero the struct, including its drop flag.
    unsafe fn manually_zero<S>(s: &mut S) {
        let len = mem::size_of::<S>();
        let p : *mut u8 = mem::transmute(s);
        for i in range(0, len) {
            *p.offset(i as int) = 0;
        }
    }
    unsafe {
        manually_zero(&mut f1);
        manually_zero(&mut n2);
    }
    println!("foo scope end");
    unsafe { current_indent -= 4; }

    // here, we drop each local variable, in reverse order of declaration.
    // So we should see the following drop sequence:
    // drop(f10), printing "Df f10"
    // drop(p)
    //   ==> drop(p.y), printing "Df f5"
    //   ==> attempt to drop(and skip) already-dropped p.x, no-op
    // drop(_n9), printing "DnB n9"
    // drop(_fx), printing "Df transformed"
    // attempt to drop already-dropped n7, printing "already-zeroed DnA"
    // no drop of `f6` since it was consumed by `xform`
    // no drop of `f5` since it was moved into `p`
    // no drop of `f4` since it was moved into `p`
    // drop(f3), printing "f3"
    // attempt to drop manually-zeroed `n2`, printing "already-zeroed DnC"
    // attempt to drop manually-zeroed `f1`, no-op.
}

fn main() {
    foo(|| true);
}

Summary

In string literal contexts, restrict \xXX escape sequences to just the range of ASCII characters, \x00\x7F. \xXX inputs in string literals with higher numbers are rejected (with an error message suggesting that one use an \uNNNN escape).

Motivation

In a string literal context, the current \xXX character escape sequence is potentially confusing when given inputs greater than 0x7F, because it does not encode that byte literally, but instead encodes whatever the escape sequence \u00XX would produce.

Thus, for inputs greater than 0x7F, \xXX will encode multiple bytes into the generated string literal, as illustrated in the Rust example appendix.

This is different from what C/C++ programmers might expect (see Behavior of xXX in C appendix).

(It would not be legal to encode the single byte literally into the string literal, since then the string would not be well-formed UTF-8.)

It has been suggested that the \xXX character escape should be removed entirely (at least from string literal contexts). This RFC is taking a slightly less aggressive stance: keep \xXX, but only for ASCII inputs when it occurs in string literals. This way, people can continue using this escape format (which shorter than the \uNNNN format) when it makes sense.

Here are some links to discussions on this topic, including direct comments that suggest exactly the strategy of this RFC.

  • https://github.com/rust-lang/rfcs/issues/312
  • https://github.com/rust-lang/rust/issues/12769
  • https://github.com/rust-lang/rust/issues/2800#issuecomment-31477259
  • https://github.com/rust-lang/rfcs/pull/69#issuecomment-43002505
  • https://github.com/rust-lang/rust/issues/12769#issuecomment-43574856
  • https://github.com/rust-lang/meeting-minutes/blob/master/weekly-meetings/2014-01-21.md#xnn-escapes-in-strings
  • https://mail.mozilla.org/pipermail/rust-dev/2012-July/002025.html

Note in particular the meeting minutes bullet, where the team explicitly decided to keep things “as they are”.

However, at the time of that meeting, Rust did not have byte string literals; people were converting string-literals into byte arrays via the bytes! macro. (Likewise, the rust-dev post is also from a time, summer 2012, when we did not have byte-string literals.)

We are in a different world now. The fact that now \xXX denotes a code unit in a byte-string literal, but in a string literal denotes a codepoint, does not seem elegant; it rather seems like a source of confusion. (Caveat: While Felix does believe this assertion, this context-dependent interpretation of \xXX does have precedent in both Python and Racket; see Racket example and Python example appendices.)

By restricting \xXX to the range 0x000x7F, we side-step the question of “is it a code unit or a code point?” entirely (which was the real context of both the rust-dev thread and the meeting minutes bullet). This RFC is a far more conservative choice that we can safely make for the short term (i.e. for the 1.0 release) than it would have been to switch to a “\xXX is a code unit” interpretation.

The expected outcome is reduced confusion for C/C++ programmers (which is, after all, our primary target audience for conversion), and any other language where \xXX never results in more than one byte. The error message will point them to the syntax they need to adopt.

Detailed design

In string literal contexts, \xXX inputs with XX > 0x7F are rejected (with an error message that mentions either, or both, of \uNNNN escapes and the byte-string literal format b"..").

The full byte range remains supported when \xXX is used in byte-string literals, b"..."

Raw strings by design do not offer escape sequences, so they are unchanged.

Character and string escaping routines (such as core::char::escape_unicode, and such as used by the "{:?}" formatter) are updated so that string inputs that previously would previously have printed \xXX with XX > 0x7F are updated to use \uNNNN escapes instead.

Drawbacks

Some reasons not to do this:

  • we think that the current behavior is intuitive,

  • it is consistent with language X (and thus has precedent),

  • existing libraries are relying on this behavior, or

  • we want to optimize for inputting characters with codepoints in the range above 0x7F in string-literals, rather than optimizing for ASCII.

The thesis of this RFC is that the first bullet is a falsehood.

While there is some precedent for the “\xXX is code point” interpretation in some languages, the majority do seem to favor the “\xXX is code unit” point of view. The proposal of this RFC is side-stepping the distinction by limiting the input range for \xXX.

The third bullet is a strawman since we have not yet released 1.0, and thus everything is up for change.

This RFC makes no comment on the validity of the fourth bullet.

Alternatives

  • We could remove \xXX entirely from string literals. This would require people to use the \uNNNN escape format even for bytes in the range 000x7F, which seems annoying.

  • We could switch \xXX from meaning code point to meaning code unit in both string literal and byte-string literal contexts. This was previously considered and explicitly rejected in an earlier meeting, as discussed in the Motivation section.

Unresolved questions

None.

Appendices

Behavior of xXX in C

Here is a C program illustrating how xXX escape sequences are treated in string literals in that context:

#include <stdio.h>

int main() {
    char *s;

    s = "a";
    printf("s[0]: %d\n", s[0]);
    printf("s[1]: %d\n", s[1]);

    s = "\x61";
    printf("s[0]: %d\n", s[0]);
    printf("s[1]: %d\n", s[1]);

    s = "\x7F";
    printf("s[0]: %d\n", s[0]);
    printf("s[1]: %d\n", s[1]);

    s = "\x80";
    printf("s[0]: %d\n", s[0]);
    printf("s[1]: %d\n", s[1]);
    return 0;
}

Its output is the following:

% gcc example.c && ./a.out
s[0]: 97
s[1]: 0
s[0]: 97
s[1]: 0
s[0]: 127
s[1]: 0
s[0]: -128
s[1]: 0

Rust example

Here is a Rust program that explores the various ways \xXX sequences are treated in both string literal and byte-string literal contexts.

 #![feature(macro_rules)]

fn main() {
    macro_rules! print_str {
        ($r:expr, $e:expr) => { {
            println!("{:>20}: \"{}\"",
                     format!("\"{}\"", $r),
                     $e.escape_default())
        } }
    }

    macro_rules! print_bstr {
        ($r:expr, $e:expr) => { {
            println!("{:>20}: {}",
                     format!("b\"{}\"", $r),
                     $e)
        } }
    }

    macro_rules! print_bytes {
        ($r:expr, $e:expr) => {
            println!("{:>9}.as_bytes(): {}", format!("\"{}\"", $r), $e.as_bytes())
        } }

    // println!("{}", b"\u0000"); // invalid: \uNNNN is not a byte escape.
    print_str!(r"\0", "\0");
    print_bstr!(r"\0", b"\0");
    print_bstr!(r"\x00", b"\x00");
    print_bytes!(r"\x00", "\x00");
    print_bytes!(r"\u0000", "\u0000");
    println!("");
    print_str!(r"\x61", "\x61");
    print_bstr!(r"a", b"a");
    print_bstr!(r"\x61", b"\x61");
    print_bytes!(r"\x61", "\x61");
    print_bytes!(r"\u0061", "\u0061");
    println!("");
    print_str!(r"\x7F", "\x7F");
    print_bstr!(r"\x7F", b"\x7F");
    print_bytes!(r"\x7F", "\x7F");
    print_bytes!(r"\u007F", "\u007F");
    println!("");
    print_str!(r"\x80", "\x80");
    print_bstr!(r"\x80", b"\x80");
    print_bytes!(r"\x80", "\x80");
    print_bytes!(r"\u0080", "\u0080");
    println!("");
    print_str!(r"\xFF", "\xFF");
    print_bstr!(r"\xFF", b"\xFF");
    print_bytes!(r"\xFF", "\xFF");
    print_bytes!(r"\u00FF", "\u00FF");
    println!("");
    print_str!(r"\u0100", "\u0100");
    print_bstr!(r"\x01\x00", b"\x01\x00");
    print_bytes!(r"\u0100", "\u0100");
}

In current Rust, it generates output as follows:

% rustc --version && echo && rustc example.rs && ./example
rustc 0.12.0-pre (d52d0c836 2014-09-07 03:36:27 +0000)

                "\0": "\x00"
               b"\0": [0]
             b"\x00": [0]
   "\x00".as_bytes(): [0]
 "\u0000".as_bytes(): [0]

              "\x61": "a"
                b"a": [97]
             b"\x61": [97]
   "\x61".as_bytes(): [97]
 "\u0061".as_bytes(): [97]

              "\x7F": "\x7f"
             b"\x7F": [127]
   "\x7F".as_bytes(): [127]
 "\u007F".as_bytes(): [127]

              "\x80": "\x80"
             b"\x80": [128]
   "\x80".as_bytes(): [194, 128]
 "\u0080".as_bytes(): [194, 128]

              "\xFF": "\xff"
             b"\xFF": [255]
   "\xFF".as_bytes(): [195, 191]
 "\u00FF".as_bytes(): [195, 191]

            "\u0100": "\u0100"
         b"\x01\x00": [1, 0]
 "\u0100".as_bytes(): [196, 128]
%

Note that the behavior of \xXX on byte-string literals matches the expectations established by the C program in Behavior of xXX in C; that is good. The problem is the behavior of \xXX for XX > 0x7F in string-literal contexts, namely in the fourth and fifth examples where the .as_bytes() invocations are showing that the underlying byte array has two elements instead of one.

Racket example

% racket
Welcome to Racket v5.93.
> (define a-string "\xbb\n")
> (display a-string)
»
> (bytes-length (string->bytes/utf-8 a-string))
3
> (define a-byte-string #"\xc2\xbb\n")
> (bytes-length a-byte-string)
3
> (display a-byte-string)
»
> (exit)
%

The above code illustrates that in Racket, the \xXX escape sequence denotes a code unit in byte-string context (#".." in that language), while it denotes a code point in string context ("..").

Python example

% python
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a_string = u"\xbb\n";
>>> print a_string
»

>>> len(a_string.encode("utf-8"))
3
>>> a_byte_string = "\xc2\xbb\n";
>>> len(a_byte_string)
3
>>> print a_byte_string
»

>>> exit()
%

The above code illustrates that in Python, the \xXX escape sequence denotes a code unit in byte-string context (".." in that language), while it denotes a code point in unicode string context (u"..").

Summary

Change the types of byte string literals to be references to statically sized types. Ensure the same change can be performed backward compatibly for string literals in the future.

Motivation

Currently byte string and string literals have types &'static [u8] and &'static str. Therefore, although the sizes of the literals are known at compile time, they are erased from their types and inaccessible until runtime. This RFC suggests to change the type of byte string literals to &'static [u8, ..N]. In addition this RFC suggest not to introduce any changes to str or string literals, that would prevent a backward compatible addition of strings of fixed size FixedString<N> (the name FixedString in this RFC is a placeholder and is open for bikeshedding) and the change of the type of string literals to &'static FixedString<N> in the future.

FixedString<N> is essentially a [u8, ..N] with UTF-8 invariants and additional string methods/traits. It fills the gap in the vector/string chart:

Vec<T>String
[T, ..N]???
&[T]&str

Today, given the lack of non-type generic parameters and compile time (function) evaluation (CTE), strings of fixed size are not very useful. But after introduction of CTE the need in compile time string operations will raise rapidly. Even without CTE but with non-type generic parameters alone fixed size strings can be used in runtime for “heapless” string operations, which are useful in constrained environments or for optimization. So the main motivation for changes today is forward compatibility.

Examples of use for new literals, that are not possible with old literals:

// Today: initialize mutable array with byte string literal
let mut arr: [u8, ..3] = *b"abc";
arr[0] = b'd';

// Future with CTE: compile time string concatenation
static LANG_DIR: FixedString<5 /*The size should, probably, be inferred*/> = *"lang/";
static EN_FILE: FixedString<_> = LANG_DIR + *"en"; // FixedString<N> implements Add
static FR_FILE: FixedString<_> = LANG_DIR + *"fr";

// Future without CTE: runtime "heapless" string concatenation
let DE_FILE = LANG_DIR + *"de"; // Performed at runtime if not optimized

Detailed design

Change the type of byte string literals from &'static [u8] to &'static [u8, ..N]. Leave the door open for a backward compatible change of the type of string literals from &'static str to &'static FixedString<N>.

Strings of fixed size

If str is moved to the library today, then strings of fixed size can be implemented like this:

struct str<Sized? T = [u8]>(T);

Then string literals will have types &'static str<[u8, ..N]>.

Drawbacks of this approach include unnecessary exposition of the implementation - underlying sized or unsized arrays [u8]/[u8, ..N] and generic parameter T. The key requirement here is the autocoercion from reference to fixed string to string slice an we are unable to meet it now without exposing the implementation.

In the future, after gaining the ability to parameterize on integers, strings of fixed size could be implemented in a better way:

struct __StrImpl<Sized? T>(T); // private

pub type str = __StrImpl<[u8]>; // unsized referent of string slice `&str`, public
pub type FixedString<const N: uint> = __StrImpl<[u8, ..N]>; // string of fixed size, public

// &FixedString<N> -> &str : OK, including &'static FixedString<N> -> &'static str for string literals

So, we don’t propose to make these changes today and suggest to wait until generic parameterization on integers is added to the language.

Precedents

C and C++ string literals are lvalue char arrays of fixed size with static duration. C++ library proposal for strings of fixed size (link), the paper also contains some discussion and motivation.

Rejected alternatives and discussion

Array literals

The types of array literals potentially can be changed from [T, ..N] to &'a [T, ..N] for consistency with the other literals and ergonomics. The major blocker for this change is the inability to move out from a dereferenced array literal if T is not Copy.

let mut a = *[box 1i, box 2, box 3]; // Wouldn't work without special-casing of array literals with regard to moving out from dereferenced borrowed pointer

Despite that array literals as references have better usability, possible staticness and consistency with other literals.

Usage statistics for array literals

Array literals can be used both as slices, when a view to array is sufficient to perform the task, and as values when arrays themselves should be copied or modified. The exact estimation of the frequencies of both uses is problematic, but some regex search in the Rust codebase gives the next statistics: In approximately 70% of cases array literals are used as slices (explicit & on array literals, immutable bindings). In approximately 20% of cases array literals are used as values (initialization of struct fields, mutable bindings, boxes). In the rest 10% of cases the usage is unclear.

So, in most cases the change to the types of array literals will lead to shorter notation.

Static lifetime

Although all the literals under consideration are similar and are essentially arrays of fixed size, array literals are different from byte string and string literals with regard to lifetimes. While byte string and string literals can always be placed into static memory and have static lifetime, array literals can depend on local variables and can’t have static lifetime in general case. The chosen design potentially allows to trivially enhance some array literals with static lifetime in the future to allow use like

fn f() -> &'static [int] {
    [1, 2, 3]
}

Alternatives

The alternative design is to make the literals the values and not the references.

The changes

Keep the types of array literals as [T, ..N]. Change the types of byte literals from &'static [u8] to [u8, ..N]. Change the types of string literals form &'static str to FixedString<N>. 2) Introduce the missing family of types - strings of fixed size - FixedString<N>. … 3) Add the autocoercion of array literals (not arrays of fixed size in general) to slices. Add the autocoercion of new byte literals to slices. Add the autocoercion of new string literals to slices. Non-literal arrays and strings do not autocoerce to slices, in accordance with the general agreements on explicitness. 4) Make string and byte literals lvalues with static lifetime.

Examples of use:

// Today: initialize mutable array with literal
let mut arr: [u8, ..3] = b"abc";
arr[0] = b'd';

// Future with CTE: compile time string concatenation
static LANG_DIR: FixedString<_> = "lang/";
static EN_FILE: FixedString<_> = LANG_DIR + "en"; // FixedString<N> implements Add
static FR_FILE: FixedString<_> = LANG_DIR + "fr";

// Future without CTE: runtime "heapless" string concatenation
let DE_FILE = LANG_DIR + "de"; // Performed at runtime if not optimized

Drawbacks of the alternative design

Special rules about (byte) string literals being static lvalues add a bit of unnecessary complexity to the specification.

In theory let s = "abcd"; copies the string from static memory to stack, but the copy is unobservable an can, probably, be elided in most cases.

The set of additional autocoercions has to exist for ergonomic purpose (and for backward compatibility). Writing something like:

fn f(arg: &str) {}
f("Hello"[]);
f(&"Hello");

for all literals would be just unacceptable.

Minor breakage:

fn main() {
    let s = "Hello";
    fn f(arg: &str) {}
    f(s); // Will require explicit slicing f(s[]) or implicit DST coercion from reference f(&s)
}

Status quo

Status quo (or partial application of the changes) is always an alternative.

Drawbacks of status quo

Examples:

// Today: can't use byte string literals in some cases
let mut arr: [u8, ..3] = [b'a', b'b', b'c']; // Have to use array literals
arr[0] = b'd';

// Future: FixedString<N> is added, CTE is added, but the literal types remain old
let mut arr: [u8, ..3] = b"abc".to_fixed(); // Have to use a conversion method
arr[0] = b'd';

static LANG_DIR: FixedString<_> = "lang/".to_fixed(); // Have to use a conversion method
static EN_FILE: FixedString<_> = LANG_DIR + "en".to_fixed();
static FR_FILE: FixedString<_> = LANG_DIR + "fr".to_fixed();

// Bad future: FixedString<N> is not added
// "Heapless"/compile-time string operations aren't possible, or performed with "magic" like extended concat! or recursive macros.

Note, that in the “Future” scenario the return type of to_fixed depends on the value of self, so it requires sufficiently advanced CTE, for example C++14 with its powerful constexpr machinery still doesn’t allow to write such a function.

Drawbacks

None.

Unresolved questions

None.

Summary

Removes the “virtual struct” (aka struct inheritance) feature, which is currently feature gated.

Motivation

Virtual structs were added experimentally prior to the RFC process as a way of inheriting fields from one struct when defining a new struct.

The feature was introduced and remains behind a feature gate.

The motivations for removing this feature altogether are:

  1. The feature is likely to be replaced by a more general mechanism, as part of the need to address hierarchies such as the DOM, ASTs, and so on. See this post for some recent discussion.

  2. The implementation is somewhat buggy and incomplete, and the feature is not well-documented.

  3. Although it’s behind a feature gate, keeping the feature around is still a maintenance burden.

Detailed design

Remove the implementation and feature gate for virtual structs.

Retain the virtual keyword as reserved for possible future use.

Drawbacks

The language will no longer offer any built-in mechanism for avoiding repetition of struct fields. Macros offer a reasonable workaround until a more general mechanism is added.

Unresolved questions

None known.

Summary

Reserve abstract, final, and override as possible keywords.

Motivation

We intend to add some mechanism to Rust to support more efficient inheritance (see, e.g., RFC PRs #245 and #250, and this thread on discuss). Although we have not decided how to do this, we do know that we will. Any implementation is likely to make use of keywords virtual (already used, to remain reserved), abstract, final, and override, so it makes sense to reserve these now to make the eventual implementation as backwards compatible as possible.

Detailed design

Make abstract, final, and override reserved keywords.

Drawbacks

Takes a few more words out of the possible vocabulary of Rust programmers.

Alternatives

Don’t do this and deal with it when we have an implementation. This would mean bumping the language version, probably.

Unresolved questions

N/A

Summary

This is a conventions RFC for settling a number of remaining naming conventions:

  • Referring to types in method names
  • Iterator type names
  • Additional iterator method names
  • Getter/setter APIs
  • Associated types
  • Trait naming
  • Lint naming
  • Suffix ordering
  • Prelude traits

It also proposes to standardize on lower case error messages within the compiler and standard library.

Motivation

As part of the ongoing API stabilization process, we need to settle naming conventions for public APIs. This RFC is a continuation of that process, addressing a number of smaller but still global naming issues.

Detailed design

The RFC includes a number of unrelated naming conventions, broken down into subsections below.

Referring to types in method names

Function names often involve type names, the most common example being conversions like as_slice. If the type has a purely textual name (ignoring parameters), it is straightforward to convert between type conventions and function conventions:

Type nameText in methods
Stringstring
Vec<T>vec
YourTypeyour_type

Types that involve notation are less clear, so this RFC proposes some standard conventions for referring to these types. There is some overlap on these rules; apply the most specific applicable rule.

Type nameText in methods
&strstr
&[T]slice
&mut [T]mut_slice
&[u8]bytes
&Tref
&mut Tmut
*const Tptr
*mut Tmut_ptr

The only surprise here is the use of mut rather than mut_ref for mutable references. This abbreviation is already a fairly common convention (e.g. as_ref and as_mut methods), and is meant to keep this very common case short.

Iterator type names

The current convention for iterator type names is the following:

Iterators require introducing and exporting new types. These types should use the following naming convention:

  • Base name. If the iterator yields something that can be described with a specific noun, the base name should be the pluralization of that noun (e.g. an iterator yielding words is called Words). Generic contains use the base name Items.

  • Flavor prefix. Iterators often come in multiple flavors, with the default flavor providing immutable references. Other flavors should prefix their name:

    • Moving iterators have a prefix of Move.
    • If the default iterator yields an immutable reference, an iterator yielding a mutable reference has a prefix Mut.
    • Reverse iterators have a prefix of Rev.

(These conventions were established as part of this PR and later this one.)

These conventions have not yet been updated to reflect the recent change to the iterator method names, in part to allow for a more significant revamp. There are some problems with the current rules:

  • They are fairly loose and therefore not mechanical or predictable. In particular, the choice of noun to use for the base name is completely arbitrary.

  • They are not always applicable. The iter module, for example, defines a large number of iterator types for use in the adapter methods on Iterator (e.g. Map for map, Filter for filter, etc.) The module does not follow the convention, and it’s not clear how it could do so.

This RFC proposes to instead align the convention with the iter module: the name of an iterator type should be the same as the method that produces the iterator.

For example:

  • iter would yield an Iter
  • iter_mut would yield an IterMut
  • into_iter would yield an IntoIter

These type names make the most sense when prefixed with their owning module, e.g. vec::IntoIter.

Advantages:

  • The rule is completely mechanical, and therefore highly predictable.

  • The convention can be (almost) universally followed: it applies equally well to vec and to iter.

Disadvantages:

  • IntoIter is not an ideal name. Note, however, that since we’ve moved to into_iter as the method name, the existing convention (MoveItems) needs to be updated to match, and it’s not clear how to do better than IntoItems in any case.

  • This naming scheme can result in clashes if multiple containers are defined in the same module. Note that this is already the case with today’s conventions. In most cases, this situation should be taken as an indication that a more refined module hierarchy is called for.

Additional iterator method names

An earlier RFC settled the conventions for the “standard” iterator methods: iter, iter_mut, into_iter.

However, there are many cases where you also want “nonstandard” iterator methods: bytes and chars for strings, keys and values for maps, the various adapters for iterators.

This RFC proposes the following convention:

  • Use iter (and variants) for data types that can be viewed as containers, and where the iterator provides the “obvious” sequence of contained items.

  • If there is no single “obvious” sequence of contained items, or if there are multiple desired views on the container, provide separate methods for these that do not use iter in their name. The name should instead directly reflect the view/item type being iterated (like bytes).

  • Likewise, for iterator adapters (filter, map and so on) or other iterator-producing operations (intersection), use the clearest name to describe the adapter/operation directly, and do not mention iter.

  • If not otherwise qualified, an iterator-producing method should provide an iterator over immutable references. Use the _mut suffix for variants producing mutable references, and the into_ prefix for variants consuming the data in order to produce owned values.

Getter/setter APIs

Some data structures do not wish to provide direct access to their fields, but instead offer “getter” and “setter” methods for manipulating the field state (often providing checking or other functionality).

The proposed convention for a field foo: T is:

  • A method foo(&self) -> &T for getting the current value of the field.
  • A method set_foo(&self, val: T) for setting the field. (The val argument here may take &T or some other type, depending on the context.)

Note that this convention is about getters/setters on ordinary data types, not on builder objects. The naming conventions for builder methods are still open.

Associated types

Unlike type parameters, the names of associated types for a trait are a meaningful part of its public API.

Associated types should be given concise, but meaningful names, generally following the convention for type names rather than generic. For example, use Err rather than E, and Item rather than T.

Trait naming

The wiki guidelines have long suggested naming traits as follows:

Prefer (transitive) verbs, nouns, and then adjectives; avoid grammatical suffixes (like able)

Trait names like Copy, Clone and Show follow this convention. The convention avoids grammatical verbosity and gives Rust code a distinctive flavor (similar to its short keywords).

This RFC proposes to amend the convention to further say: if there is a single method that is the dominant functionality of the trait, consider using the same name for the trait itself. This is already the case for Clone and ToCStr, for example.

According to these rules, Encodable should probably be Encode.

There are some open questions about these rules; see Unresolved Questions below.

Lints

Our lint names are not consistent. While this may seem like a minor concern, when we hit 1.0 the lint names will be locked down, so it’s worth trying to clean them up now.

The basic rule is: the lint name should make sense when read as “allow lint-name” or “allow lint-name items”. For example, “allow deprecated items” and “allow dead_code” makes sense, while “allow unsafe_block” is ungrammatical (should be plural).

Specifically, this RFC proposes that:

  • Lint names should state the bad thing being checked for, e.g. deprecated, so that #[allow(deprecated)] (items) reads correctly. Thus ctypes is not an appropriate name; improper_ctypes is.

  • Lints that apply to arbitrary items (like the stability lints) should just mention what they check for: use deprecated rather than deprecated_items. This keeps lint names short. (Again, think “allow lint-name items”.)

  • If a lint applies to a specific grammatical class, mention that class and use the plural form: use unused_variables rather than unused_variable. This makes #[allow(unused_variables)] read correctly.

  • Lints that catch unnecessary, unused, or useless aspects of code should use the term unused, e.g. unused_imports, unused_typecasts.

  • Use snake case in the same way you would for function names.

Suffix ordering

Very occasionally, conventions will require a method to have multiple suffixes, for example get_unchecked_mut. When feasible, design APIs so that this situation does not arise.

Because it is so rare, it does not make sense to lay out a complete convention for the order in which various suffixes should appear; no one would be able to remember it.

However, the mut suffix is so common, and is now entrenched as showing up in final position, that this RFC does propose one simple rule: if there are multiple suffixes including mut, place mut last.

Prelude traits

It is not currently possible to define inherent methods directly on basic data types like char or slices. Consequently, libcore and other basic crates provide one-off traits (like ImmutableSlice or Char) that are intended to be implemented solely by these primitive types, and which are included in the prelude.

These traits are generally not designed to be used for generic programming, but the fact that they appear in core libraries with such basic names makes it easy to draw the wrong conclusion.

This RFC proposes to use a Prelude suffix for these basic traits. Since the traits are, in fact, included in the prelude their names do not generally appear in Rust programs. Therefore, choosing a longer and clearer name will help avoid confusion about the intent of these traits, and will avoid namespace pollution.

(There is one important drawback in today’s Rust: associated functions in these traits cannot yet be called directly on the types implementing the traits. These functions are the one case where you would need to mention the trait by name, today. Hopefully, this situation will change before 1.0; otherwise we may need a separate plan for dealing with associated functions.)

Error messages

Error messages – including those produced by fail! and those placed in the desc or detail fields of e.g. IoError – should in general be in all lower case. This applies to both rustc and std.

This is already the predominant convention, but there are some inconsistencies.

Alternatives

Iterator type names

The iterator type name convention could instead basically stick with today’s convention, but using suffixes instead of prefixes, and IntoItems rather than MoveItems.

Unresolved questions

How far should the rules for trait names go? Should we avoid “-er” suffixes, e.g. have Read rather than Reader?

Summary

This is a conventions RFC that proposes that the items exported from a module should never be prefixed with that module name. For example, we should have io::Error, not io::IoError.

(An alternative design is included that special-cases overlap with the prelude.)

Motivation

Currently there is no clear prohibition around including the module’s name as a prefix on an exported item, and it is sometimes done for type names that are feared to be “popular” (like Error and Result being IoError and IoResult) for clarity.

This RFC include two designs: one that entirely rules out such prefixes, and one that rules it out except for names that overlap with the prelude. Pros/cons are given for each.

Detailed design

The main rule being proposed is very simple: the items exported from a module should never be prefixed with the module’s name.

Rationale:

  • Avoids needless stuttering like io::IoError.
  • Any ambiguity can be worked around:
    • Either qualify by the module, i.e. io::Error,
    • Or rename on import: use io::Error as IoError.
  • The rule is extremely simple and clear.

Downsides:

  • The name may already exist in the module wanting to export it.
    • If that’s due to explicit imports, those imports can be renamed or module-qualified (see above).
    • If that’s due to a prelude conflict, however, confusion may arise due to the conventional global meaning of identifiers defined in the prelude (i.e., programmers do not expect prelude imports to be shadowed).

Overall, the RFC author believes that if this convention is adopted, confusion around redefining prelude names would gradually go away, because (at least for things like Result) we would come to expect it.

Alternative design

An alternative rule would be to never prefix an exported item with the module’s name, except for names that are also defined in the prelude, which must be prefixed by the module’s name.

For example, we would have io::Error and io::IoResult.

Rationale:

  • Largely the same as the above, but less decisively.
  • Avoids confusion around prelude-defined names.

Downsides:

  • Retains stuttering for some important cases, e.g. custom Result types, which are likely to be fairly common.
  • Makes it even more problematic to expand the prelude in the future.

Summary

This RFC is preparation for API stabilization for the std::num module. The proposal is to finish the simplification efforts started in @bjz’s reversal of the numerics hierarchy.

Broadly, the proposal is to collapse the remaining numeric hierarchy in std::num, and to provide only limited support for generic programming (roughly, only over primitive numeric types that vary based on size). Traits giving detailed numeric hierarchy can and should be provided separately through the Cargo ecosystem.

Thus, this RFC proposes to flatten or remove most of the traits currently provided by std::num, and generally to simplify the module as much as possible in preparation for API stabilization.

Motivation

History

Starting in early 2013, there was an effort to design a comprehensive “numeric hierarchy” for Rust: a collection of traits classifying a wide variety of numbers and other algebraic objects. The intent was to allow highly-generic code to be written for algebraic structures and then instantiated to particular types.

This hierarchy covered structures like bigints, but also primitive integer and float types. It was an enormous and long-running community effort.

Later, it was recognized that building such a hierarchy within libstd was misguided:

@bjz The API that resulted from #4819 attempted, like Haskell, to blend both the primitive numerics and higher level mathematical concepts into one API. This resulted in an ugly hybrid where neither goal was adequately met. I think the libstd should have a strong focus on implementing fundamental operations for the base numeric types, but no more. Leave the higher level concepts to libnum or future community projects.

The std::num module has thus been slowly migrating away from a large trait hierarchy toward a simpler one providing just APIs for primitive data types: this is @bjz’s reversal of the numerics hierarchy.

Along side this effort, there are already external numerics packages like @bjz’s num-rs.

But we’re not finished yet.

The current state of affairs

The std::num module still contains quite a few traits that subdivide out various features of numbers:

pub trait Zero: Add<Self, Self> {
    fn zero() -> Self;
    fn is_zero(&self) -> bool;
}

pub trait One: Mul<Self, Self> {
    fn one() -> Self;
}

pub trait Signed: Num + Neg<Self> {
    fn abs(&self) -> Self;
    fn abs_sub(&self, other: &Self) -> Self;
    fn signum(&self) -> Self;
    fn is_positive(&self) -> bool;
    fn is_negative(&self) -> bool;
}

pub trait Unsigned: Num {}

pub trait Bounded {
    fn min_value() -> Self;
    fn max_value() -> Self;
}

pub trait Primitive: Copy + Clone + Num + NumCast + PartialOrd + Bounded {}

pub trait Num: PartialEq + Zero + One + Neg<Self> + Add<Self,Self> + Sub<Self,Self>
             + Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self> {}

pub trait Int: Primitive + CheckedAdd + CheckedSub + CheckedMul + CheckedDiv
             + Bounded + Not<Self> + BitAnd<Self,Self> + BitOr<Self,Self>
             + BitXor<Self,Self> + Shl<uint,Self> + Shr<uint,Self> {
    fn count_ones(self) -> uint;
    fn count_zeros(self) -> uint { ... }
    fn leading_zeros(self) -> uint;
    fn trailing_zeros(self) -> uint;
    fn rotate_left(self, n: uint) -> Self;
    fn rotate_right(self, n: uint) -> Self;
    fn swap_bytes(self) -> Self;
    fn from_be(x: Self) -> Self { ... }
    fn from_le(x: Self) -> Self { ... }
    fn to_be(self) -> Self { ... }
    fn to_le(self) -> Self { ... }
}

pub trait FromPrimitive {
    fn from_i64(n: i64) -> Option<Self>;
    fn from_u64(n: u64) -> Option<Self>;

    // many additional defaulted methods
    // ...
}

pub trait ToPrimitive {
    fn to_i64(&self) -> Option<i64>;
    fn to_u64(&self) -> Option<u64>;

    // many additional defaulted methods
    // ...
}

pub trait NumCast: ToPrimitive {
    fn from<T: ToPrimitive>(n: T) -> Option<Self>;
}

pub trait Saturating {
    fn saturating_add(self, v: Self) -> Self;
    fn saturating_sub(self, v: Self) -> Self;
}

pub trait CheckedAdd: Add<Self, Self> {
    fn checked_add(&self, v: &Self) -> Option<Self>;
}

pub trait CheckedSub: Sub<Self, Self> {
    fn checked_sub(&self, v: &Self) -> Option<Self>;
}

pub trait CheckedMul: Mul<Self, Self> {
    fn checked_mul(&self, v: &Self) -> Option<Self>;
}

pub trait CheckedDiv: Div<Self, Self> {
    fn checked_div(&self, v: &Self) -> Option<Self>;
}

pub trait Float: Signed + Primitive {
    // a huge collection of static functions (for constants) and methods
    ...
}

pub trait FloatMath: Float {
    // an additional collection of methods
}

The Primitive traits are intended primarily to support a mechanism, #[deriving(FromPrimitive)], that makes it easy to provide conversions from numeric types to C-like enums.

The Saturating and Checked traits provide operations that provide special handling for overflow and other numeric errors.

Almost all of these traits are currently included in the prelude.

In addition to these traits, the std::num module includes a couple dozen free functions, most of which duplicate methods available though traits.

Where we want to go: a summary

The goal of this RFC is to refactor the std::num hierarchy with the following goals in mind:

  • Simplicity.

  • Limited generic programming: being able to work generically over the natural classes of primitive numeric types that vary only by size. There should be enough abstraction to support porting strconv, the generic string/number conversion code used in std.

  • Minimizing dependencies for libcore. For example, it should not require cmath.

  • Future-proofing for external numerics packages. The Cargo ecosystem should ultimately provide choices of sophisticated numeric hierarchies, and std::num should not get in the way.

Detailed design

Overview: the new hierarchy

This RFC proposes to collapse the trait hierarchy in std::num to just the following traits:

  • Int, implemented by all primitive integer types (u8 - u64, i8-i64)
    • UnsignedInt, implemented by u8 - u64
  • Signed, implemented by all signed primitive numeric types (i8-i64, f32-f64)
  • Float, implemented by f32 and f64
    • FloatMath, implemented by f32 and f64, which provides functionality from cmath

These traits inherit from all applicable overloaded operator traits (from core::ops). They suffice for generic programming over several basic categories of primitive numeric types.

As designed, these traits include a certain amount of redundancy between Int and Float. The Alternatives section shows how this could be factored out into a separate Num trait. But doing so suggests a level of generic programming that these traits aren’t intended to support.

The main reason to pull out Signed into its own trait is so that it can be added to the prelude. (Further discussion below.)

Detailed definitions

Below is the full definition of these traits. The functionality remains largely as it is today, just organized into fewer traits:

pub trait Int: Copy + Clone + PartialOrd + PartialEq
             + Add<Self,Self> + Sub<Self,Self>
             + Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self>
             + Not<Self> + BitAnd<Self,Self> + BitOr<Self,Self>
             + BitXor<Self,Self> + Shl<uint,Self> + Shr<uint,Self>
{
    // Constants
    fn zero() -> Self;  // These should be associated constants when those are available
    fn one() -> Self;
    fn min_value() -> Self;
    fn max_value() -> Self;

    // Deprecated:
    // fn is_zero(&self) -> bool;

    // Bit twiddling
    fn count_ones(self) -> uint;
    fn count_zeros(self) -> uint { ... }
    fn leading_zeros(self) -> uint;
    fn trailing_zeros(self) -> uint;
    fn rotate_left(self, n: uint) -> Self;
    fn rotate_right(self, n: uint) -> Self;
    fn swap_bytes(self) -> Self;
    fn from_be(x: Self) -> Self { ... }
    fn from_le(x: Self) -> Self { ... }
    fn to_be(self) -> Self { ... }
    fn to_le(self) -> Self { ... }

    // Checked arithmetic
    fn checked_add(self, v: Self) -> Option<Self>;
    fn checked_sub(self, v: Self) -> Option<Self>;
    fn checked_mul(self, v: Self) -> Option<Self>;
    fn checked_div(self, v: Self) -> Option<Self>;
    fn saturating_add(self, v: Self) -> Self;
    fn saturating_sub(self, v: Self) -> Self;
}

pub trait UnsignedInt: Int {
    fn is_power_of_two(self) -> bool;
    fn checked_next_power_of_two(self) -> Option<Self>;
    fn next_power_of_two(self) -> Self;
}

pub trait Signed: Neg<Self> {
    fn abs(&self) -> Self;
    fn signum(&self) -> Self;
    fn is_positive(&self) -> bool;
    fn is_negative(&self) -> bool;

    // Deprecated:
    // fn abs_sub(&self, other: &Self) -> Self;
}

pub trait Float: Copy + Clone + PartialOrd + PartialEq + Signed
               + Add<Self,Self> + Sub<Self,Self>
               + Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self>
{
    // Constants
    fn zero() -> Self;  // These should be associated constants when those are available
    fn one() -> Self;
    fn min_value() -> Self;
    fn max_value() -> Self;

    // Classification and decomposition
    fn is_nan(self) -> bool;
    fn is_infinite(self) -> bool;
    fn is_finite(self) -> bool;
    fn is_normal(self) -> bool;
    fn classify(self) -> FPCategory;
    fn integer_decode(self) -> (u64, i16, i8);

    // Float intrinsics
    fn floor(self) -> Self;
    fn ceil(self) -> Self;
    fn round(self) -> Self;
    fn trunc(self) -> Self;
    fn mul_add(self, a: Self, b: Self) -> Self;
    fn sqrt(self) -> Self;
    fn powi(self, n: i32) -> Self;
    fn powf(self, n: Self) -> Self;
    fn exp(self) -> Self;
    fn exp2(self) -> Self;
    fn ln(self) -> Self;
    fn log2(self) -> Self;
    fn log10(self) -> Self;

    // Conveniences
    fn fract(self) -> Self;
    fn recip(self) -> Self;
    fn rsqrt(self) -> Self;
    fn to_degrees(self) -> Self;
    fn to_radians(self) -> Self;
    fn log(self, base: Self) -> Self;
}

// This lives directly in `std::num`, not `core::num`, since it requires `cmath`
pub trait FloatMath: Float {
    // Exactly the methods defined in today's version
}

Float constants, float math, and cmath

This RFC proposes to:

  • Remove all float constants from the Float trait. These constants are available directly from the f32 and f64 modules, and are not really useful for the kind of generic programming these new traits are intended to allow.

  • Continue providing various cmath functions as methods in the FloatMath trait. Putting this in a separate trait means that libstd depends on cmath but libcore does not.

Free functions

All of the free functions defined in std::num are deprecated.

The prelude

The prelude will only include the Signed trait, as the operations it provides are widely expected to be available when they apply.

The reason for removing the rest of the traits is two-fold:

  • The remaining operations are relatively uncommon. Note that various overloaded operators, like +, work regardless of this choice. Those doing intensive work with e.g. floats would only need to import Float and FloatMath.

  • Keeping this functionality out of the prelude means that the names of methods and associated items remain available for external numerics libraries in the Cargo ecosystem.

strconv, FromStr, ToStr, FromStrRadix, ToStrRadix

Currently, traits for converting from &str and to String are both included, in their own modules, in libstd. This is largely due to the desire to provide impls for numeric types, which in turn relies on std::num::strconv.

This RFC proposes to:

  • Move the FromStr trait into core::str.
  • Rename the ToStr trait to ToString, and move it to collections::string.
  • Break up and revise std::num::strconv into separate, private modules that provide the needed functionality for the from_str and to_string methods. (Some of this functionality has already migrated to fmt and been deprecated in strconv.)
  • Move the FromStrRadix into core::num.
  • Remove ToStrRadix, which is already deprecated in favor of fmt.

FromPrimitive and friends

Ideally, the FromPrimitive, ToPrimitive, Primitive, NumCast traits would all be removed in favor of a more principled way of working with C-like enums. However, such a replacement is outside of the scope of this RFC, so these traits are left (as #[experimental]) for now. A follow-up RFC proposing a better solution should appear soon.

In the meantime, see this proposal and the discussion on this issue about Ordinal for the rough direction forward.

Drawbacks

This RFC somewhat reduces the potential for writing generic numeric code with std::num traits. This is intentional, however: the new design represents “just enough” generics to cover differently-sized built-in types, without any attempt at general algebraic abstraction.

Alternatives

The status quo is clearly not ideal, and as explained above there was a long attempt at providing a more complete numeric hierarchy in std. So some collapse of the hierarchy seems desirable.

That said, there are other possible factorings. We could introduce the following Num trait to factor out commonalities between Int and Float:

pub trait Num: Copy + Clone + PartialOrd + PartialEq
             + Add<Self,Self> + Sub<Self,Self>
             + Mul<Self,Self> + Div<Self,Self> + Rem<Self,Self>
{
    fn zero() -> Self;  // These should be associated constants when those are available
    fn one() -> Self;
    fn min_value() -> Self;
    fn max_value() -> Self;
}

However, it’s not clear whether this factoring is worth having a more complex hierarchy, especially because the traits are not intended for generic programming at that level (and generic programming across integer and floating-point types is likely to be extremely rare)

The signed and unsigned operations could be offered on more types, allowing removal of more traits but a less clear-cut semantics.

Unresolved questions

This RFC does not propose a replacement for #[deriving(FromPrimitive)], leaving the relevant traits in limbo status. (See this proposal and the discussion on this issue about Ordinal for the rough direction forward.)

  • Start Date: 2014-10-09
  • RFC PR #: https://github.com/rust-lang/rfcs/pull/378
  • Rust Issue #: https://github.com/rust-lang/rust/issues/18635

Summary

Parse macro invocations with parentheses or square brackets as expressions no matter the context, and require curly braces or a semicolon following the invocation to invoke a macro as a statement.

Motivation

Currently, macros that start a statement want to be a whole statement, and so expressions such as foo!().bar don’t parse if they start a statement. The reason for this is because sometimes one wants a macro that expands to an item or statement (for example, macro_rules!), and forcing the user to add a semicolon to the end is annoying and easy to forget for long, multi-line statements. However, the vast majority of macro invocations are not intended to expand to an item or statement, leading to frustrating parser errors.

Unfortunately, this is not as easy to resolve as simply checking for an infix operator after every statement-like macro invocation, because there exist operators that are both infix and prefix. For example, consider the following function:

fn frob(x: int) -> int {
    maybe_return!(x)
    // Provide a default value
    -1
}

Today, this parses successfully. However, if a rule were added to the parser that any macro invocation followed by an infix operator be parsed as a single expression, this would still parse successfully, but not in the way expected: it would be parsed as (maybe_return!(x)) - 1. This is an example of how it is impossible to resolve this ambiguity properly without breaking compatibility.

Detailed design

Treat all macro invocations with parentheses, (), or square brackets, [], as expressions, and never attempt to parse them as statements or items in a block context unless they are followed directly by a semicolon. Require all item-position macro invocations to be either invoked with curly braces, {}, or be followed by a semicolon (for consistency).

This distinction between parentheses and curly braces has precedent in Rust: tuple structs, which use parentheses, must be followed by a semicolon, while structs with fields do not need to be followed by a semicolon. Many constructs like match and if, which use curly braces, also do not require semicolons when they begin a statement.

Drawbacks

  • This introduces a difference between different macro invocation delimiters, where previously there was no difference.
  • This requires the use of semicolons in a few places where it was not necessary before.

Alternatives

  • Require semicolons after all macro invocations that aren’t being used as expressions. This would have the downside of requiring semicolons after every macro_rules! declaration.

Unresolved questions

None.

Summary

  • Remove reflection from the compiler
  • Remove libdebug
  • Remove the Poly format trait as well as the :? format specifier

Motivation

In ancient Rust, one of the primary methods of printing a value was via the %? format specifier. This would use reflection at runtime to determine how to print a type. Metadata generated by the compiler (a TyDesc) would be generated to guide the runtime in how to print a type. One of the great parts about reflection was that it was quite easy to print any type. No extra burden was required from the programmer to print something.

There are, however, a number of cons to this approach:

  • Generating extra metadata for many many types by the compiler can lead to noticeable increases in compile time and binary size.
  • This form of formatting is inherently not speedy. Widespread usage of %? led to misleading benchmarks about formatting in Rust.
  • Depending on how metadata is handled, this scheme makes it very difficult to allow recompiling a library without recompiling downstream dependants.

Over time, usage off the ? formatting has fallen out of fashion for the following reasons:

  • The deriving-based infrastructure was improved greatly and has started seeing much more widespread use, especially for traits like Clone.
  • The formatting language implementation and syntax has changed. The most common formatter is now {} (an implementation of Show), and it is quite common to see an implementation of Show on nearly all types (frequently via deriving). This form of customizable-per-typformatting largely provides the gap that the original formatting language did not provide, which was limited to only primitives and %?.
  • Compiler built-ins, such as ~[T] and ~str have been removed from the language, and runtime reflection on Vec<T> and String are far less useful (they just print pointers, not contents).

As a result, the :? formatting specifier is quite rarely used today, and when it is used it’s largely for historical purposes and the output is not of very high quality any more.

The drawbacks and today’s current state of affairs motivate this RFC to recommend removing this infrastructure entirely. It’s possible to add it back in the future with a more modern design reflecting today’s design principles of Rust and the many language changes since the infrastructure was created.

Detailed design

  • Remove all reflection infrastructure from the compiler. I am not personally super familiar with what exists, but at least these concrete actions will be taken.
    • Remove the visit_glue function from TyDesc.
    • Remove any form of visit_glue generation.
    • (maybe?) Remove the name field of TyDesc.
  • Remove core::intrinsics::TyVisitor
  • Remove core::intrinsics::visit_tydesc
  • Remove libdebug
  • Remove std::fmt::Poly
  • Remove the :? format specifier in the formatting language syntax.

Drawbacks

The current infrastructure for reflection, although outdated, represents a significant investment of work in the past which could be a shame to lose. While present in the git history, this infrastructure has been updated over time, and it will no longer receive this attention.

Additionally, given an arbitrary type T, it would now be impossible to print it in literally any situation. Type parameters will now require some bound, such as Show, to allow printing a type.

These two drawbacks are currently not seen as large enough to outweigh the gains from reducing the surface area of the std::fmt API and reduction in maintenance load on the compiler.

Alternatives

The primary alternative to outright removing this infrastructure is to preserve it, but flag it all as #[experimental] or feature-gated. The compiler could require the fmt_poly feature gate to be enabled to enable formatting via :? in a crate. This would mean that any backwards-incompatible changes could continue to be made, and any arbitrary type T could still be printed.

Unresolved questions

  • Can core::intrinsics::TyDesc be removed entirely?

Summary

Stabilize the std::fmt module, in addition to the related macros and formatting language syntax. As a high-level summary:

  • Leave the format syntax as-is.
  • Remove a number of superfluous formatting traits (renaming a few in the process).

Motivation

This RFC is primarily motivated by the need to stabilize std::fmt. In the past stabilization has not required RFCs, but the changes envisioned for this module are far-reaching and modify some parts of the language (format syntax), leading to the conclusion that this stabilization effort required an RFC.

Detailed design

The std::fmt module encompasses more than just the actual structs/traits/functions/etc defined within it, but also a number of macros and the formatting language syntax for describing format strings. Each of these features of the module will be described in turn.

Formatting Language Syntax

The documented syntax will not be changing as-written. All of these features will be accepted wholesale (considered stable):

  • Usage of {} for “format something here” placeholders
  • {{ as an escape for { (and vice-versa for })
  • Various format specifiers
    • fill character for alignment
    • actual alignment, left (<), center (^), and right (>).
    • sign to print (+ or -)
    • minimum width for text to be printed
      • both a literal count and a runtime argument to the format string
    • precision or maximum width
      • all of a literal count, a specific runtime argument to the format string, and “the next” runtime argument to the format string.
    • “alternate formatting” (#)
    • leading zeroes (0)
  • Integer specifiers of what to format ({0})
  • Named arguments ({foo})

Using Format Specifiers

While quite useful occasionally, there is no static guarantee that any implementation of a formatting trait actually respects the format specifiers passed in. For example, this code does not necessarily work as expected:

#[deriving(Show)]
struct A;

format!("{:10}", A);

All of the primitives for rust (strings, integers, etc) have implementations of Show which respect these formatting flags, but almost no other implementations do (notably those generated via deriving).

This RFC proposes stabilizing the formatting flags, despite this current state of affairs. There are in theory possible alternatives in which there is a static guarantee that a type does indeed respect format specifiers when one is provided, generating a compile-time error when a type doesn’t respect a specifier. These alternatives, however, appear to be too heavyweight and are considered somewhat overkill.

In general it’s trivial to respect format specifiers if an implementation delegates to a primitive or somehow has a buffer of what’s to be formatted. To cover these two use cases, the Formatter structure passed around has helper methods to assist in formatting these situations. This is, however, quite rare to fall into one of these two buckets, so the specifiers are largely ignored (and the formatter is write!-n to directly).

Named Arguments

Currently Rust does not support named arguments anywhere except for format strings. Format strings can get away with it because they’re all part of a macro invocation (unlike the rest of Rust syntax).

The worry for stabilizing a named argument syntax for the formatting language is that if Rust ever adopts named arguments with a different syntax, it would be quite odd having two systems.

The most recently proposed keyword argument RFC used : for the invocation syntax rather than = as formatting does today. Additionally, today foo = bar is a valid expression, having a value of type ().

With these worries, there are one of two routes that could be pursued:

  1. The expr = expr syntax could be disallowed on the language level. This could happen both in a total fashion or just allowing the expression appearing as a function argument. For both cases, this will probably be considered a “wart” of Rust’s grammar.
  2. The foo = bar syntax could be allowed in the macro with prior knowledge that the default argument syntax for Rust, if one is ever developed, will likely be different. This would mean that the foo = bar syntax in formatting macros will likely be considered a wart in the future.

Given these two cases, the clear choice seems to be accepting a wart in the formatting macros themselves. It will likely be possible to extend the macro in the future to support whatever named argument syntax is developed as well, and the old syntax could be accepted for some time.

Formatting Traits

Today there are 16 formatting traits. Each trait represents a “type” of formatting, corresponding to the [type] production in the formatting syntax. As a bit of history, the original intent was for each trait to declare what specifier it used, allowing users to add more specifiers in newer crates. For example the time crate could provide the {:time} formatting trait. This design was seen as too complicated, however, so it was not landed. It does, however, partly motivate why there is one trait per format specifier today.

The 16 formatting traits and their format specifiers are:

  • nothingShow
  • dSigned
  • iSigned
  • uUnsigned
  • bBool
  • cChar
  • oOctal
  • xLowerHex
  • XUpperHex
  • sString
  • pPointer
  • tBinary
  • fFloat
  • eLowerExp
  • EUpperExp
  • ?Poly

This RFC proposes removing the following traits:

  • Signed
  • Unsigned
  • Bool
  • Char
  • String
  • Float

Note that this RFC would like to remove Poly, but that is covered by a separate RFC.

Today by far the most common formatting trait is Show, and over time the usefulness of these formatting traits has been reduced. The traits this RFC proposes to remove are only assertions that the type provided actually implements the trait, there are few known implementations of the traits which diverge on how they are implemented.

Additionally, there are a two of oddities inherited from ancient C:

  • Both d and i are wired to Signed
  • One may reasonable expect the Binary trait to use b as its specifier.

The remaining traits this RFC recommends leaving. The rationale for this is that they represent alternate representations of primitive types in general, and are also quite often expected when coming from other format syntaxes such as C/Python/Ruby/etc.

It would, of course, be possible to re-add any of these traits in a backwards-compatible fashion.

Format type for Binary

With the removal of the Bool trait, this RFC recommends renaming the specifier for Binary to b instead of t.

Combining all traits

A possible alternative to having many traits is to instead have one trait, such as:

pub trait Show {
    fn fmt(...);
    fn hex(...) { fmt(...) }
    fn lower_hex(...) { fmt(...) }
    ...
}

There are a number of pros to this design:

  • Instead of having to consider many traits, only one trait needs to be considered.
  • All types automatically implement all format types or zero format types.
  • In a hypothetical world where a format string could be constructed at runtime, this would alleviate the signature of such a function. The concrete type taken for all its arguments would be &Show and then if the format string supplied :x or :o the runtime would simply delegate to the relevant trait method.

There are also a number of cons to this design, which motivate this RFC recommending the remaining separation of these traits.

  • The “static assertion” that a type implements a relevant format trait becomes almost nonexistent because all types either implement none or all formatting traits.
  • The documentation for the Show trait becomes somewhat overwhelming because it’s no longer immediately clear which method should be overridden for what.
  • A hypothetical world with runtime format string construction could find a different system for taking arguments.

Method signature

Currently, each formatting trait has a signature as follows:

fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result;

This implies that all formatting is considered to be a stream-oriented operation where f is a sink to write bytes to. The fmt::Result type indicates that some form of “write error” happened, but conveys no extra information.

This API has a number of oddities:

  • The type Formatter has inherent write and write_fmt methods to be used in conjunction with the write! macro return an instance of fmt::Result.
  • The Formatter type also implements the std::io::Writer trait in order to be able to pass around a &mut Writer.
  • This relies on the duck-typing of macros and for the inherent write_fmt method to trump the Writer’s write_fmt method in order to return an error of the correct type.
  • The Result return type is an enumeration with precisely one variant, FormatError.

Overall, this signature seems to be appropriate in terms of “give me a sink of bytes to write myself to, and let me return an error if one happens”. Due to this, this RFC recommends that all formatting traits be marked #[unstable].

Macros

There are a number of prelude macros which interact with the format syntax:

  • format_args
  • format_args_method
  • write
  • writeln
  • print
  • println
  • format
  • fail
  • assert
  • debug_assert

All of these are macro_rules!-defined macros, except for format_args and format_args_method.

Common syntax

All of these macros take some form of prefix, while the trailing suffix is always some instantiation of the formatting syntax. The suffix portion is recommended to be considered #[stable], and the sections below will discuss each macro in detail with respect to its prefix and semantics.

format_args

The fundamental purpose of this macro is to generate a value of type &fmt::Arguments which represents a pending format computation. This structure can then be passed at some point to the methods in std::fmt to actually perform the format.

The prefix of this macro is some “callable thing”, be it a top-level function or a closure. It cannot invoke a method because foo.bar is not a “callable thing” to call the bar method on foo.

Ideally, this macro would have no prefix, and would be callable like:

use std::fmt;

let args = format_args!("Hello {}!", "world");
let hello_world = fmt::format(args);

Unfortunately, without an implementation of RFC 31 this is not possible. As a result, this RFC proposes a #[stable] consideration of this macro and its syntax.

format_args_method

The purpose of this macro is to solve the “call this method” case not covered with the format_args macro. This macro was introduced fairly late in the game to solve the problem that &*trait_object was not allowed. This is currently allowed, however (due to DST).

This RFC proposes immediately removing this macro. The primary user of this macro is write!, meaning that the following code, which compiles today, would need to be rewritten:

let mut output = std::io::stdout();
// note the lack of `&mut` in front
write!(output, "hello {}", "world");

The write! macro would be redefined as:

macro_rules! write(
    ($dst:expr, $($arg:tt)*) => ({
        let dst = &mut *$dst;
        format_args!(|args| { dst.write_fmt(args) }, $($arg)*)
    })
)

The purpose here is to borrow $dst outside of the closure to ensure that the closure doesn’t borrow too many of its contents. Otherwise, code such as this would be disallowed

write!(&mut my_struct.writer, "{}", my_struct.some_other_field);

write/writeln

These two macros take the prefix of “some pointer to a writer” as an argument, and then format data into the write (returning whatever write_fmt returns). These macros were originally designed to require a &mut T as the first argument, but today, due to the usage of format_args_method, they can take any T which responds to write_fmt.

This RFC recommends marking these two macros #[stable] with the modification above (removing format_args_method). The ln suffix to writeln will be discussed shortly.

print/println

These two macros take no prefix, and semantically print to a task-local stdout stream. The purpose of a task-local stream is provide some form of buffering to make stdout printing at all performant.

This RFC recommends marking these two macros a #[stable].

The ln suffix

The name println is one of the few locations in Rust where a short C-like abbreviation is accepted rather than the more verbose, but clear, print_line (for example). Due to the overwhelming precedent of other languages (even Java uses println!), this is seen as an acceptable special case to the rule.

format

This macro takes no prefix and returns a String.

In ancient rust this macro was called its shorter name, fmt. Additionally, the name format is somewhat inconsistent with the module name of fmt. Despite this, this RFC recommends considering this macro #[stable] due to its delegation to the format method in the std::fmt module, similar to how the write! macro delegates to the fmt::write.

fail/assert/debug_assert

The format string portions of these macros are recommended to be considered as #[stable] as part of this RFC. The actual stability of the macros is not considered as part of this RFC.

Freestanding Functions

There are a number of freestanding functions to consider in the std::fmt module for stabilization.

  • fn format(args: &Arguments) -> String

    This RFC recommends #[experimental]. This method is largely an implementation detail of this module, and should instead be used via:

    let args: &fmt::Arguments = ...;
    format!("{}", args)
  • fn write(output: &mut FormatWriter, args: &Arguments) -> Result

    This is somewhat surprising in that the argument to this function is not a Writer, but rather a FormatWriter. This is technically speaking due to the core/std separation and how this function is defined in core and Writer is defined in std.

    This RFC recommends marking this function #[experimental] as the write_fmt exists on Writer to perform the corresponding operation. Consequently we may wish to remove this function in favor of the write_fmt method on FormatWriter.

    Ideally this method would be removed from the public API as it is just an implementation detail of the write! macro.

  • fn radix<T>(x: T, base: u8) -> RadixFmt<T, Radix>

    This function is a bit of an odd-man-out in that it is a constructor, but does not follow the existing conventions of Type::new. The purpose of this function is to expose the ability to format a number for any radix. The default format specifiers :o, :x, and :t are essentially shorthands for this function, except that the format types have specialized implementations per radix instead of a generic implementation.

    This RFC proposes that this function be considered #[unstable] as its location and naming are a bit questionable, but the functionality is desired.

Miscellaneous items

  • trait FormatWriter

    This trait is currently the actual implementation strategy of formatting, and is defined specially in libcore. It is rarely used outside of libcore. It is recommended to be #[experimental].

    There are possibilities in moving Reader and Writer to libcore with the error type as an associated item, allowing the FormatWriter trait to be eliminated entirely. Due to this possibility, the trait will be experimental for now as alternative solutions are explored.

  • struct Argument, mod rt, fn argument, fn argumentstr, fn argumentuint, Arguments::with_placeholders, Arguments::new

    These are implementation details of the Arguments structure as well as the expansion of the format_args! macro. It’s recommended to mark these as #[experimental] and #[doc(hidden)]. Ideally there would be some form of macro-based privacy hygiene which would allow these to be truly private, but it will likely be the case that these simply become stable and we must live with them forever.

  • struct Arguments

    This is a representation of a “pending format string” which can be used to safely execute a Formatter over it. This RFC recommends #[stable].

  • struct Formatter

    This instance is passed to all formatting trait methods and contains helper methods for respecting formatting flags. This RFC recommends #[unstable].

    This RFC also recommends deprecating all public fields in favor of accessor methods. This should help provide future extensibility as well as preventing unnecessary mutation in the future.

  • enum FormatError

    This enumeration only has one instance, WriteError. It is recommended to make this a struct instead and rename it to just Error. The purpose of this is to signal that an error has occurred as part of formatting, but it does not provide a generic method to transmit any other information other than “an error happened” to maintain the ergonomics of today’s usage. It’s strongly recommended that implementations of Show and friends are infallible and only generate an error if the underlying Formatter returns an error itself.

  • Radix/RadixFmt

    Like the radix function, this RFC recommends #[unstable] for both of these pieces of functionality.

Drawbacks

Today’s macro system necessitates exporting many implementation details of the formatting system, which is unfortunate.

Alternatives

A number of alternatives were laid out in the detailed description for various aspects.

Unresolved questions

  • How feasible and/or important is it to construct a format string at runtime given the recommend stability levels in this RFC?

Module system cleanups

Summary

  • Lift the hard ordering restriction between extern crate, use and other items.
  • Allow pub extern crate as opposed to only private ones.
  • Allow extern crate in blocks/functions, and not just in modules.

Motivation

The main motivation is consistency and simplicity: None of the changes proposed here change the module system in any meaningful way, they just remove weird forbidden corner cases that are all already possible to express today with workarounds.

Thus, they make it easier to learn the system for beginners, and easier to for developers to evolve their module hierarchies

Lifting the ordering restriction between extern crate, use and other items.

Currently, certain items need to be written in a fixed order: First all extern crate, then all use and then all other items. This has historically reasons, due to the older, more complex resolution algorithm, which included that shadowing was allowed between those items in that order, and usability reasons, as it makes it easy to locate imports and library dependencies.

However, after RFC 50 got accepted there is only ever one item name in scope from any given source so the historical “hard” reasons loose validity: Any resolution algorithm that used to first process all extern crate, then all use and then all items can still do so, it just has to filter out the relevant items from the whole module body, rather then from sequential sections of it. And any usability reasons for keeping the order can be better addressed with conventions and lints, rather than hard parser rules.

(The exception here are the special cased prelude, and globs and macros, which are feature gated and out of scope for this proposal)

As it is, today the ordering rule is a unnecessary complication, as it routinely causes beginner to stumble over things like this:

mod foo;
use foo::bar; // ERROR: Imports have to precede items

In addition, it doesn’t even prevent certain patterns, as it is possible to work around the order restriction by using a submodule:

struct Foo;
// One of many ways to expose the crate out of order:
mod bar { extern crate bar; pub use self::bar::x; pub use self::bar::y; ... }

Which with this RFC implemented would be identical to

struct Foo;
extern crate bar;

Another use case are item macros/attributes that want to automatically include their their crate dependencies. This is possible by having the macro expand to an item that links to the needed crate, eg like this:

#[my_attribute]
struct UserType;

Expands to:

struct UserType;
extern crate "MyCrate" as <gensymb>
impl <gensymb>::MyTrait for UserType { ... }

With the order restriction still in place, this requires the sub module workaround, which is unnecessary verbose.

As an example, gfx-rs currently employs this strategy.

Allow pub extern crate as opposed to only private ones.

extern crate semantically is somewhere between useing a module, and declaring one with mod, and is identical to both as far as as the module path to it is considered. As such, its surprising that its not possible to declare a extern crate as public, even though you can still make it so with an reexport:


mod foo {
    extern crate "bar" as bar_;
    pub use bar_ as bar;
}

While its generally not necessary to export a extern library directly, the need for it does arise occasionally during refactorings of huge crate collections, generally if a public module gets turned into its own crate.

As an example,the author recalls stumbling over it during a refactoring of gfx-rs.

Allow extern crate in blocks/functions, and not just in modules.

Similar to the point above, its currently possible to both import and declare a module in a block expression or function body, but not to link to an library:

fn foo() {
    let x = {
        extern crate qux; // ERROR: Extern crate not allowed here
        use bar::baz;     // OK
        mod bar { ... };  // OK
        qux::foo()
    };
}

This is again a unnecessary restriction considering that you can declare modules and imports there, and thus can make an extern library reachable at that point:

fn foo() {
    let x = {
        mod qux { extern crate "qux" as qux_; pub use self::qux_ as qux; }
        qux::foo()
    };
}

This again benefits macros and gives the developer the power to place external dependencies only needed for a single function lexically near it.

General benefits

In general, the simplification and freedom added by these changes would positively effect the docs of Rusts module system (which is already often regarded as too complex by outsiders), and possibly admit other simplifications or RFCs based on the now-equality of view items and items in the module system.

(As an example, the author is considering an RFC about merging the use and type features; by lifting the ordering restriction they become more similar and thus more redundant)

This also does not have to be a 1.0 feature, as it is entirely backwards compatible to implement, and strictly allows more programs to compile than before. However, as alluded to above it might be a good idea for 1.0 regardless

Detailed design

  • Remove the ordering restriction from resolve
  • If necessary, change resolve to look in the whole scope block for view items, not just in a prefix of it.
  • Make pub extern crate parse and teach privacy about it
  • Allow extern crate view items in blocks

Drawbacks

  • The source of names in scope might be harder to track down
  • Similarly, it might become confusing to see when a library dependency exist.

However, these issues already exist today in one form or another, and can be addressed by proper docs that make library dependencies clear, and by the fact that definitions are generally greppable in a file.

Alternatives

As this just cleans up a few aspects of the module system, there isn’t really an alternative apart from not or only partially implementing it.

By not implementing this proposal, the module system remains more complex for the user than necessary.

Unresolved questions

  • Inner attributes occupy the same syntactic space as items and view items, and are currently also forced into a given order by needing to be written first. This is also potentially confusing or restrictive for the same reasons as for the view items mentioned above, especially in regard to the build-in crate attributes, and has one big issue: It is currently not possible to load a syntax extension that provides an crate-level attribute, as with the current macro system this would have to be written like this:

    #[phase(plugin)]
    extern crate mycrate;
    #![myattr]
    

    Which is impossible to write due to the ordering restriction. However, as attributes and the macro system are also not finalized, this has not been included in this RFC directly.

  • This RFC does also explicitly not talk about wildcard imports and macros in regard to resolution, as those are feature gated today and likely subject to change. In any case, it seems unlikely that they will conflict with the changes proposed here, as macros would likely follow the same module system rules where possible, and wildcard imports would either be removed, or allowed in a way that doesn’t conflict with explicitly imported names to prevent compilation errors on upstream library changes (new public item may not conflict with downstream items).

Summary

  • Add the ability to have trait bounds that are polymorphic over lifetimes.

Motivation

Currently, closure types can be polymorphic over lifetimes. But closure types are deprecated in favor of traits and object types as part of RFC #44 (unboxed closures). We need to close the gap. The canonical example of where you want this is if you would like a closure that accepts a reference with any lifetime. For example, today you might write:

fn with(callback: |&Data|) {
    let data = Data { ... };
    callback(&data)
}

If we try to write this using unboxed closures today, we have a problem:

fn with<'a, T>(callback: T)
    where T : FnMut(&'a Data)
{
    let data = Data { ... };
    callback(&data)
}

// Note that the `()` syntax is shorthand for the following:
fn with<'a, T>(callback: T)
    where T : FnMut<(&'a Data,),()>
{
    let data = Data { ... };
    callback(&data)
}

The problem is that the argument type &'a Data must include a lifetime, and there is no lifetime one could write in the fn sig that represents “the stack frame of the with function”. Naturally we have the same problem if we try to use an FnMut object (which is the closer analog to the original closure example):

fn with<'a>(callback: &mut FnMut(&'a Data))
{
    let data = Data { ... };
    callback(&data)
}

fn with<'a>(callback: &mut FnMut<(&'a Data,),()>)
{
    let data = Data { ... };
    callback(&data)
}

Under this proposal, you would be able to write this code as follows:

// Using the FnMut(&Data) notation, the &Data is
// in fact referencing an implicit bound lifetime, just
// as with closures today.
fn with<T>(callback: T)
    where T : FnMut(&Data)
{
    let data = Data { ... };
    callback(&data)
}

// If you prefer, you can use an explicit name,
// introduced by the `for<'a>` syntax.
fn with<T>(callback: T)
    where T : for<'a> FnMut(&'a Data)
{
    let data = Data { ... };
    callback(&data)
}

// No sugar at all.
fn with<T>(callback: T)
    w