Content-Length: 276915 | pFad | http://github.com/twinnydotdev/toxe

3D GitHub - twinnydotdev/toxe: SentencePiece tokenizer for cross-encoders
Skip to content

twinnydotdev/toxe

Repository files navigation

toxe

Install

npm i toxe

Usage

import { Toxe } from 'toxe';

const toxe = new Toxe('./spm.model', {
  bos: 1,
  eos: 2
});

const ids = await toxe.encode("a", [
  "a b",
  "a b c",
]);

Credits

https://github.com/JanKaul/sentencepiece









ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/twinnydotdev/toxe

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy