refactor: extract shared token package (#46)

## Description

Both the `saccharine` and `lambda` packages need tokenizing and parsing primitives.
This PR extracts shared token infrastructure into a new `pkg/token` package, then wires both languages up to use it.

- Add `pkg/token` with a generic `Token[T]` type, `Scan`, `ScanAtom`, `ScanRune`, `ScanCharacter`, `IsVariable`, `ParseRawToken`, and `ParseList`.
- Refactor `pkg/saccharine` to delegate to `pkg/token`, removing duplicated scanning and parsing helpers.
- Implement `Codec.Decode` for `pkg/lambda` (scanner + parser) using the shared token package.
- Add `iterator.While` for predicate-driven iteration.
- Rename `iterator.Do` to `iterator.Try` to better describe its rollback semantics.

### Decisions

- The `Type` constraint (`comparable` + `Name() string`) keeps the generic token flexible while ensuring every token type can produce readable error messages.
- `iterator.Do` was renamed to `iterator.Try` since it describes a try/rollback operation, not a side-effecting "do".

## Benefits

- Eliminates duplicated token, scanning, and parsing code between languages.
- Enables the `lambda` package to decode (parse) lambda calculus strings, which was previously unimplemented.
- Makes it straightforward to add new languages by reusing `pkg/token` primitives.

## Checklist

- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`). Always use underscores.
- [x] Tests pass (if applicable).
- [ ] Documentation updated (if applicable).

Reviewed-on: #46
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
This commit was merged in pull request #46.
This commit is contained in:
2026-02-12 00:25:18 +00:00
committed by Maxim Hutz
parent 361f529bdc
commit da3da70855
11 changed files with 392 additions and 160 deletions

View File

@@ -43,12 +43,12 @@ func (i Iterator[T]) Get() (T, error) {
// MustGet is a version of Get, that panics if the datum cannot be returned.
func (i Iterator[T]) MustGet() T {
var null T
if i.Done() {
return null
t, err := i.Get()
if err != nil {
panic(fmt.Errorf("cannot get current token: %w", err))
}
return i.items[i.index]
return t
}
// Forward increments the iterator if the iterator is not yet at the end of the
@@ -81,10 +81,21 @@ func (i Iterator[T]) Done() bool {
return i.index == len(i.items)
}
// Do attempts to perform an operation using the iterator. If the operation
// While increments the iterator as long as the current item satisfies the
// predicate. The first item that does not match is left unconsumed.
func (i *Iterator[T]) While(fn func(T) bool) {
for !i.Done() {
if !fn(i.MustGet()) {
return
}
i.Forward()
}
}
// Try attempts to perform an operation using the iterator. If the operation
// succeeds, the iterator is updated. If the operation fails, the iterator is
// rolled back, and an error is returned.
func Do[T any, U any](i *Iterator[T], fn func(i *Iterator[T]) (U, error)) (U, error) {
func Try[T any, U any](i *Iterator[T], fn func(i *Iterator[T]) (U, error)) (U, error) {
i2 := i.Copy()
out, err := fn(i2)