M.V. Hutz
da3da70855
## Description Both the `saccharine` and `lambda` packages need tokenizing and parsing primitives. This PR extracts shared token infrastructure into a new `pkg/token` package, then wires both languages up to use it. - Add `pkg/token` with a generic `Token[T]` type, `Scan`, `ScanAtom`, `ScanRune`, `ScanCharacter`, `IsVariable`, `ParseRawToken`, and `ParseList`. - Refactor `pkg/saccharine` to delegate to `pkg/token`, removing duplicated scanning and parsing helpers. - Implement `Codec.Decode` for `pkg/lambda` (scanner + parser) using the shared token package. - Add `iterator.While` for predicate-driven iteration. - Rename `iterator.Do` to `iterator.Try` to better describe its rollback semantics. ### Decisions - The `Type` constraint (`comparable` + `Name() string`) keeps the generic token flexible while ensuring every token type can produce readable error messages. - `iterator.Do` was renamed to `iterator.Try` since it describes a try/rollback operation, not a side-effecting "do". ## Benefits - Eliminates duplicated token, scanning, and parsing code between languages. - Enables the `lambda` package to decode (parse) lambda calculus strings, which was previously unimplemented. - Makes it straightforward to add new languages by reusing `pkg/token` primitives. ## Checklist - [x] Code follows conventional commit format. - [x] Branch follows naming convention (`<type>/<description>`). Always use underscores. - [x] Tests pass (if applicable). - [ ] Documentation updated (if applicable). Reviewed-on: #46 Co-authored-by: M.V. Hutz <git@maximhutz.me> Co-committed-by: M.V. Hutz <git@maximhutz.me>
37 lines
1.2 KiB
Go
37 lines
1.2 KiB
Go
// Package token provides generic token types and scanning/parsing primitives
|
|
// for building language-specific lexers and parsers.
|
|
package token
|
|
|
|
// A Type is a constraint for language-specific token type enums.
|
|
// It must be comparable (for equality checks) and must have a Name method
|
|
// that returns a human-readable string for error messages.
|
|
type Type interface {
|
|
comparable
|
|
// Name returns a human-readable name for this token type.
|
|
Name() string
|
|
}
|
|
|
|
// A Token is a lexical unit in a source language.
|
|
type Token[T Type] struct {
|
|
Column int // Where the token begins in the source text.
|
|
Type T // What type the token is.
|
|
Value string // The value of the token.
|
|
}
|
|
|
|
// New creates a Token of the given type at the given column.
|
|
// The token's value is derived from its type's Name method.
|
|
func New[T Type](typ T, column int) *Token[T] {
|
|
return &Token[T]{Type: typ, Column: column, Value: typ.Name()}
|
|
}
|
|
|
|
// NewAtom creates a Token of the given type with a custom value at the given
|
|
// column.
|
|
func NewAtom[T Type](typ T, name string, column int) *Token[T] {
|
|
return &Token[T]{Type: typ, Column: column, Value: name}
|
|
}
|
|
|
|
// Name returns the type of the Token, as a string.
|
|
func (t Token[T]) Name() string {
|
|
return t.Type.Name()
|
|
}
|