Simplify Try to save/restore the index directly instead of
copying and syncing the entire iterator. Remove the now-unused
Copy and Sync methods.
Rewrite ScanRune and ParseRawToken as peek-then-advance so they
no longer need Try at all. Remove redundant Try wrappers from
parse functions that are already disambiguated by their callers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Description
Both the `saccharine` and `lambda` packages need tokenizing and parsing primitives.
This PR extracts shared token infrastructure into a new `pkg/token` package, then wires both languages up to use it.
- Add `pkg/token` with a generic `Token[T]` type, `Scan`, `ScanAtom`, `ScanRune`, `ScanCharacter`, `IsVariable`, `ParseRawToken`, and `ParseList`.
- Refactor `pkg/saccharine` to delegate to `pkg/token`, removing duplicated scanning and parsing helpers.
- Implement `Codec.Decode` for `pkg/lambda` (scanner + parser) using the shared token package.
- Add `iterator.While` for predicate-driven iteration.
- Rename `iterator.Do` to `iterator.Try` to better describe its rollback semantics.
### Decisions
- The `Type` constraint (`comparable` + `Name() string`) keeps the generic token flexible while ensuring every token type can produce readable error messages.
- `iterator.Do` was renamed to `iterator.Try` since it describes a try/rollback operation, not a side-effecting "do".
## Benefits
- Eliminates duplicated token, scanning, and parsing code between languages.
- Enables the `lambda` package to decode (parse) lambda calculus strings, which was previously unimplemented.
- Makes it straightforward to add new languages by reusing `pkg/token` primitives.
## Checklist
- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`). Always use underscores.
- [x] Tests pass (if applicable).
- [ ] Documentation updated (if applicable).
Reviewed-on: #46
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
## Description
The `internal/cli` package had grown to contain both CLI utilities (source/destination I/O) and registry-level abstractions (repr, conversion, engine, marshaler).
This PR separates concerns by moving registry types into `internal/registry` and keeping only CLI I/O types in `internal/cli`.
It also simplifies several core abstractions and aligns naming conventions.
- Move `Source`, `Destination` from `internal/config` to `internal/cli`.
- Move `Repr`, `Conversion`, `Engine`, `Process`, `Codec` from `internal/cli` to `internal/registry`.
- Rename "marshalers" to "codecs" throughout the codebase.
- Simplify `codec.Codec[T, U]` to `codec.Codec[T]` (string-based marshaling only).
- Add `codec.Conversion[T, U]` as a function type alias.
- Change `engine.Engine[T]` from an interface to a function type.
- Merge `Engine.Load()` + `Process.Set()` into a single `Engine.Load(Repr)` call.
- Convert `Saccharine2Lambda` from a struct to standalone conversion functions.
- Replace registry methods (`MustAddMarshaler`, `MustAddEngine`, `MustAddConversions`) with generic free functions (`RegisterCodec`, `RegisterEngine`, `RegisterConversion`).
- Remove unused `internal/config` package (`Config`, `GetLogger`, `ParseFromArgs`).
- Remove unused `pkg/emitter` package.
- Rename `Id()` to `ID()` per Go conventions.
- Add documentation comments and enable `checkPublicInterface` lint rule.
- Rename `reduce_one.go` to `reduce_once.go`.
### Decisions
- `Engine[T]` is now a function type (`func(T) (Process[T], error)`) rather than an interface, since the only method was `Load`.
- `Codec[T, U]` was split into `Codec[T]` (string marshaling) and `Conversion[T, U]` (type-to-type conversion function), which better reflects how they are actually used.
- Registration uses free generic functions (`RegisterCodec`, `RegisterEngine`, `RegisterConversion`) instead of methods on `Registry`, enabling type inference at the call site.
## Benefits
- Clearer separation of concerns between CLI I/O and the registry's internal type system.
- Simpler abstractions: fewer interfaces, fewer wrapper types, fewer indirections.
- Removing unused packages (`config`, `emitter`) reduces maintenance burden.
- Naming conventions (`ID`, codecs, `reduce_once`) are more idiomatic.
## Checklist
- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`).
- [x] Tests pass (if applicable).
- [x] Documentation updated (if applicable).
Reviewed-on: #43
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
## Description
The old architecture used a monolithic `main()` with a custom arg parser, an event-emitter-based runtime, and a plugin system for optional features.
This PR rewrites the CLI and internal architecture to be modular, extensible, and built around a registry of interchangeable components.
- Replace custom CLI arg parsing with Cobra subcommands (`convert`, `reduce`, `engine list`).
- Introduce a registry system (`internal/registry`) for marshalers, codecs, and engines, with BFS-based conversion path resolution.
- Add type-erased adapter layer (`internal/cli`) with `Repr`, `Engine`, `Process`, `Marshaler`, and `Conversion` interfaces wrapping generic `pkg/` types.
- Replace the event-emitter-based `Runtime` with a simpler `Engine`/`Process` model (`pkg/engine`).
- Add generic `Codec[T, U]` and `Marshaler[T]` interfaces (`pkg/codec`).
- Merge `saccharine/token` sub-package into `saccharine` and rename scanner functions from `parse*` to `scan*`.
- Make saccharine-to-lambda conversion bidirectional (encode and decode).
- Add `lambda.Marshaler` and `saccharine.Marshaler` implementing `codec.Marshaler`.
- Remove old infrastructure: `pkg/runtime`, `pkg/expr`, `internal/plugins`, `internal/statistics`.
- Add `make lint` target and update golangci-lint config.
### Decisions
- Cobra was chosen for the CLI framework to support nested subcommands and standard flag handling.
- The registry uses BFS to find conversion paths between representations, allowing multi-hop conversions without hardcoding routes.
- Type erasure via `cli.Repr` (wrapping `any`) enables the registry to work with heterogeneous types while keeping `pkg/` generics type-safe.
- The old plugin/event system was removed entirely rather than adapted, since the new `Process` model can support hooks differently in the future.
## Benefits
- Subcommands make the CLI self-documenting and easier to extend with new functionality.
- The registry pattern decouples representations, conversions, and engines, making it trivial to add new ones.
- BFS conversion routing means adding a single codec automatically enables transitive conversions.
- Simpler `Engine`/`Process` model reduces complexity compared to the event-emitter runtime.
- Consolidating the `token` sub-package reduces import depth and package sprawl.
## Checklist
- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`). Always use underscores.
- [ ] Tests pass (if applicable).
- [ ] Documentation updated (if applicable).
Reviewed-on: #41
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
## Description
The saccharine language previously lacked comment support, preventing proper code documentation.
This PR implements '#' comment syntax similar to Python.
Comments can appear on their own line or at the end of a line, with all content after '#' ignored until the next newline or EOF.
The tokenizer now detects '#' and skips characters appropriately without creating tokens.
### Decisions
Comments are silently consumed during tokenization rather than being preserved as tokens, keeping the token stream clean for the parser.
The implementation preserves newlines after comments by using the iterator's Back() method, allowing them to be processed as soft breaks.
## Benefits
Developers can now document their saccharine code with inline and full-line comments.
The implementation is minimal and efficient, adding no overhead to the token stream.
Tests verify that comments work correctly in various positions without breaking code execution.
## Checklist
- [x] Code follows conventional commit format.
- [x] Branch follows naming convention (`<type>/<description>`). Always use underscores.
- [x] Tests pass (if applicable).
- [x] Documentation updated (if applicable).
Closes#24
Reviewed-on: #25
Co-authored-by: M.V. Hutz <git@maximhutz.me>
Co-committed-by: M.V. Hutz <git@maximhutz.me>
