3 Commits

Author	SHA1	Message	Date
jon-tow	4c846d7114	fix: make eos_token/pad_token overridable and add pickle support	2024-01-22 23:45:51 -05:00
Jonathan Tow	a8f2f2862b	feat(tokenizer): expose merge ranks and special tokens for GGUF	2024-01-19 18:22:13 +00:00
jon-tow	3aeae29673	init: release	2024-01-18 23:46:52 +00:00