refactor: extract shared token package #46

mvhutz · 2026-02-12T00:17:08Z

mvhutz commented

2026-02-12 00:17:08 +00:00

Description

Both the saccharine and lambda packages need tokenizing and parsing primitives.
This PR extracts shared token infrastructure into a new pkg/token package, then wires both languages up to use it.

Add pkg/token with a generic Token[T] type, Scan, ScanAtom, ScanRune, ScanCharacter, IsVariable, ParseRawToken, and ParseList.
Refactor pkg/saccharine to delegate to pkg/token, removing duplicated scanning and parsing helpers.
Implement Codec.Decode for pkg/lambda (scanner + parser) using the shared token package.
Add iterator.While for predicate-driven iteration.
Rename iterator.Do to iterator.Try to better describe its rollback semantics.

Decisions

The Type constraint (comparable + Name() string) keeps the generic token flexible while ensuring every token type can produce readable error messages.
iterator.Do was renamed to iterator.Try since it describes a try/rollback operation, not a side-effecting "do".

Benefits

Eliminates duplicated token, scanning, and parsing code between languages.
Enables the lambda package to decode (parse) lambda calculus strings, which was previously unimplemented.
Makes it straightforward to add new languages by reusing pkg/token primitives.

Checklist

Code follows conventional commit format.
Branch follows naming convention (<type>/<description>). Always use underscores.
Tests pass (if applicable).
Documentation updated (if applicable).

## Description Both the `saccharine` and `lambda` packages need tokenizing and parsing primitives. This PR extracts shared token infrastructure into a new `pkg/token` package, then wires both languages up to use it. - Add `pkg/token` with a generic `Token[T]` type, `Scan`, `ScanAtom`, `ScanRune`, `ScanCharacter`, `IsVariable`, `ParseRawToken`, and `ParseList`. - Refactor `pkg/saccharine` to delegate to `pkg/token`, removing duplicated scanning and parsing helpers. - Implement `Codec.Decode` for `pkg/lambda` (scanner + parser) using the shared token package. - Add `iterator.While` for predicate-driven iteration. - Rename `iterator.Do` to `iterator.Try` to better describe its rollback semantics. ### Decisions - The `Type` constraint (`comparable` + `Name() string`) keeps the generic token flexible while ensuring every token type can produce readable error messages. - `iterator.Do` was renamed to `iterator.Try` since it describes a try/rollback operation, not a side-effecting "do". ## Benefits - Eliminates duplicated token, scanning, and parsing code between languages. - Enables the `lambda` package to decode (parse) lambda calculus strings, which was previously unimplemented. - Makes it straightforward to add new languages by reusing `pkg/token` primitives. ## Checklist - [x] Code follows conventional commit format. - [x] Branch follows naming convention (`<type>/<description>`). Always use underscores. - [x] Tests pass (if applicable). - [ ] Documentation updated (if applicable).

mvhutz added 2 commits 2026-02-12 00:17:08 +00:00

feat: new parser strategy 87f2370d81

feat: iterator.While, slimmed comments in saccharine e1ec3a3c09

mvhutz added 1 commit 2026-02-12 00:20:39 +00:00

fix: remove out.klc 654c38cc9c

mvhutz added 1 commit 2026-02-12 00:24:39 +00:00

feat: update mustget behavior a230ff9bc7

mvhutz merged commit da3da70855 into main

2026-02-12 00:25:19 +00:00

mvhutz deleted branch refactor/shared-token-package

2026-02-12 00:25:19 +00:00

mvhutz referenced this issue from a commit