lexer

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2026 License: MIT Imports: 5 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Graphemes 

func Graphemes(input string) []string

Graphemes splits input into Unicode grapheme clusters.

func IsControl 

func IsControl(r rune) bool

IsControl returns true for control characters except newline and tab.

func IsZeroWidth 

func IsZeroWidth(r rune) bool

IsZeroWidth returns true for invisible zero-width runes.

Types

type Token 

type Token struct {
	Type  TokenType
	Value string

	Start int
	End   int
}

Token represents a lexical unit with byte offsets in the original content.

func Classify 

func Classify(tokens []Token) []Token

Classify upgrades lexical tokens into semantic token classes.

func Lex 

func Lex(input string) []Token

Lex tokenizes input in one pass while preserving byte offsets.

type TokenType 

type TokenType uint8

TokenType identifies lexical and semantic token categories. 

const (
	TokenUnknown TokenType = iota

	TokenWord
	TokenNumber
	TokenWhitespace
	TokenNewline
	TokenSymbol

	TokenURL
	TokenEmail
	TokenPath

	TokenString
	TokenCodeBlock

	TokenPlaceholder
	TokenBase64
	TokenShellCommand

	TokenZeroWidth
	TokenControlChar
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.