compactlm-5m / tokenizer.json

Commit History

28 days ago

  1. Add BPE tokenizer (8192 vocab)

    Verified signature
    a50c5c3

29 days ago

  1. Add gollem_eval byte-level BPE tokenizer (12,288 vocab) (#2)⁠

    e1fdac6