Skip to content

rustc_parser incorrectly parses groups with Delimiter::None #67062

@petrochenkov

Description

@petrochenkov

Such groups can currently be created by proc macros.

Reproduction, proc macro crate:

#[proc_macro]
pub fn add_mul(_: TokenStream) -> TokenStream {
    // ⟪ 2 + 2 ⟫ * 2
    vec![
        TokenTree::from(Group::new(Delimiter::None, "2 + 2".parse().unwrap())),
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ].into_iter().collect()
}

User crate:

fn main() {
    let x = add_mul!();
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).

However, this will became a huge issue when interpolated tokens like $e: expr migrate from AST pieces to token streams, because in the token stream model they are represented exactly like groups with Delimiter::None delimiters (proc macros already see them in this way).

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-macrosArea: All kinds of macros (custom derive, macro_rules!, proc macros, ..)A-parserArea: The lexing & parsing of Rust source code to an ASTC-bugCategory: This is a bug.T-langRelevant to the language team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      pFad - Phonifier reborn

      Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

      Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


      Alternative Proxies:

      Alternative Proxy

      pFad Proxy

      pFad v3 Proxy

      pFad v4 Proxy