Commit Graph

2 Commits

Author SHA1 Message Date
Raven Scott
6efd069b5d counting tokens properly 2023-08-14 23:00:23 -04:00
Raven Scott
10473ef702 beginnings of using a tokenizer to check CTX length, remove context if needed 2023-06-12 18:15:11 +02:00