M.V. Hutz
da3da70855
## Description Both the `saccharine` and `lambda` packages need tokenizing and parsing primitives. This PR extracts shared token infrastructure into a new `pkg/token` package, then wires both languages up to use it. - Add `pkg/token` with a generic `Token[T]` type, `Scan`, `ScanAtom`, `ScanRune`, `ScanCharacter`, `IsVariable`, `ParseRawToken`, and `ParseList`. - Refactor `pkg/saccharine` to delegate to `pkg/token`, removing duplicated scanning and parsing helpers. - Implement `Codec.Decode` for `pkg/lambda` (scanner + parser) using the shared token package. - Add `iterator.While` for predicate-driven iteration. - Rename `iterator.Do` to `iterator.Try` to better describe its rollback semantics. ### Decisions - The `Type` constraint (`comparable` + `Name() string`) keeps the generic token flexible while ensuring every token type can produce readable error messages. - `iterator.Do` was renamed to `iterator.Try` since it describes a try/rollback operation, not a side-effecting "do". ## Benefits - Eliminates duplicated token, scanning, and parsing code between languages. - Enables the `lambda` package to decode (parse) lambda calculus strings, which was previously unimplemented. - Makes it straightforward to add new languages by reusing `pkg/token` primitives. ## Checklist - [x] Code follows conventional commit format. - [x] Branch follows naming convention (`<type>/<description>`). Always use underscores. - [x] Tests pass (if applicable). - [ ] Documentation updated (if applicable). Reviewed-on: #46 Co-authored-by: M.V. Hutz <git@maximhutz.me> Co-committed-by: M.V. Hutz <git@maximhutz.me>
66 lines
1.4 KiB
Go
66 lines
1.4 KiB
Go
package saccharine
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"git.maximhutz.com/max/lambda/pkg/token"
|
|
)
|
|
|
|
// A TokenType is an identifier for any token in the Saccharine language.
|
|
type TokenType int
|
|
|
|
// All official tokens of the Saccharine language.
|
|
const (
|
|
// TokenOpenParen denotes the '(' token.
|
|
TokenOpenParen TokenType = iota
|
|
// TokenCloseParen denotes the ')' token.
|
|
TokenCloseParen
|
|
// TokenOpenBrace denotes the '{' token.
|
|
TokenOpenBrace
|
|
// TokenCloseBrace denotes the '}' token.
|
|
TokenCloseBrace
|
|
// TokenHardBreak denotes the ';' token.
|
|
TokenHardBreak
|
|
// TokenAssign denotes the ':=' token.
|
|
TokenAssign
|
|
// TokenAtom denotes an alpha-numeric variable.
|
|
TokenAtom
|
|
// TokenSlash denotes the '\\' token.
|
|
TokenSlash
|
|
// TokenDot denotes the '.' token.
|
|
TokenDot
|
|
// TokenSoftBreak denotes a new-line.
|
|
TokenSoftBreak
|
|
)
|
|
|
|
// Name returns the type of the TokenType, as a string.
|
|
func (t TokenType) Name() string {
|
|
switch t {
|
|
case TokenOpenParen:
|
|
return "("
|
|
case TokenCloseParen:
|
|
return ")"
|
|
case TokenOpenBrace:
|
|
return "{"
|
|
case TokenCloseBrace:
|
|
return "}"
|
|
case TokenHardBreak:
|
|
return ";"
|
|
case TokenAssign:
|
|
return ":="
|
|
case TokenAtom:
|
|
return "ATOM"
|
|
case TokenSlash:
|
|
return "\\"
|
|
case TokenDot:
|
|
return "."
|
|
case TokenSoftBreak:
|
|
return "\\n"
|
|
default:
|
|
panic(fmt.Errorf("unknown token type %v", t))
|
|
}
|
|
}
|
|
|
|
// Token is the concrete token type for the Saccharine language.
|
|
type Token = token.Token[TokenType]
|