Object to tokenize a line of F# source code, starting with the given lexState. The lexState should be FSharpTokenizerLexState.Initial for the first line of text. Returns an array of ranges of the text and two enumerations categorizing the tokens and characters covered by that range, i.e. FSharpTokenColorKind and FSharpTokenCharKind. The enumerations are somewhat adhoc but useful enough to give good colorization options to the user in an IDE. A new lexState is also returned. An IDE-plugin should in general cache the lexState values for each line of the edited code.

Tokenizer for a source file. Holds some expensive-to-compute resources at the scope of the file.

Gives an indication of the class to assign to the characters of the token an IDE

Gives an indication of the color class to assign to the token an IDE

Information about a particular token from the tokenizer

Represents stable information for the state of the lexing engine at the end of a line

Represents encoded information for the end-of-line continuation of lexing

Some of the values in the field FSharpTokenInfo.Tag

Gives an indication of what should happen when the token is typed in an IDE