## Description
Both the `saccharine` and `lambda` packages need tokenizing and parsing primitives.
This PR extracts shared token infrastructure into a new `pkg/token` package, then wires both languages up to use it.
- Add `pkg/token` with a generic `Token[T]` type, `Scan`, `ScanAtom`, `ScanRune`, `ScanCharacter`, `IsVariable`, `ParseRawToken`, and `ParseList`.
- Refactor `pkg/saccharine` to delegate to `pkg/token`, removing duplicated scanning and parsing helpers.
- Implement `Codec.Decode` for `pkg/lambda` (scanner + parser) using the shared token package.
- Add `iterator.While` for predicate-driven iteration.
- Rename `iterator.Do` to `iterator.Try` to better describe its rollback semantics.
### Decisions
- The `Type` constraint (`comparable` + `Name() string`) keeps the generic token flexible while ensuring every token type can produce readable error messages.
- `iterator.Do` was renamed to `iterator.Try` since it describes a try/rollback operation, not a side-effecting "do".
## Benefits
- Eliminates duplicated token, scanning, and parsing code between languages.
- Enables the `lambda` package to decode (parse) lambda calculus strings, which was previously unimplemented.
- Makes it straightforward to add new languages by reusing `pkg/token` primitives.
## Checklist
- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`). Always use underscores.
- [x] Tests pass (if applicable).
- [ ] Documentation updated (if applicable).
Reviewed-on: #46
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
## Description
The old architecture used a monolithic `main()` with a custom arg parser, an event-emitter-based runtime, and a plugin system for optional features.
This PR rewrites the CLI and internal architecture to be modular, extensible, and built around a registry of interchangeable components.
- Replace custom CLI arg parsing with Cobra subcommands (`convert`, `reduce`, `engine list`).
- Introduce a registry system (`internal/registry`) for marshalers, codecs, and engines, with BFS-based conversion path resolution.
- Add type-erased adapter layer (`internal/cli`) with `Repr`, `Engine`, `Process`, `Marshaler`, and `Conversion` interfaces wrapping generic `pkg/` types.
- Replace the event-emitter-based `Runtime` with a simpler `Engine`/`Process` model (`pkg/engine`).
- Add generic `Codec[T, U]` and `Marshaler[T]` interfaces (`pkg/codec`).
- Merge `saccharine/token` sub-package into `saccharine` and rename scanner functions from `parse*` to `scan*`.
- Make saccharine-to-lambda conversion bidirectional (encode and decode).
- Add `lambda.Marshaler` and `saccharine.Marshaler` implementing `codec.Marshaler`.
- Remove old infrastructure: `pkg/runtime`, `pkg/expr`, `internal/plugins`, `internal/statistics`.
- Add `make lint` target and update golangci-lint config.
### Decisions
- Cobra was chosen for the CLI framework to support nested subcommands and standard flag handling.
- The registry uses BFS to find conversion paths between representations, allowing multi-hop conversions without hardcoding routes.
- Type erasure via `cli.Repr` (wrapping `any`) enables the registry to work with heterogeneous types while keeping `pkg/` generics type-safe.
- The old plugin/event system was removed entirely rather than adapted, since the new `Process` model can support hooks differently in the future.
## Benefits
- Subcommands make the CLI self-documenting and easier to extend with new functionality.
- The registry pattern decouples representations, conversions, and engines, making it trivial to add new ones.
- BFS conversion routing means adding a single codec automatically enables transitive conversions.
- Simpler `Engine`/`Process` model reduces complexity compared to the event-emitter runtime.
- Consolidating the `token` sub-package reduces import depth and package sprawl.
## Checklist
- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`). Always use underscores.
- [ ] Tests pass (if applicable).
- [ ] Documentation updated (if applicable).
Reviewed-on: #41
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
