gm.text.Gemma2Tokenizer

class gemma.gm.text.Gemma2Tokenizer(

path: str | os.PathLike = 'gs://gemma-data/tokenizers/tokenizer_gemma2.model',

*,

custom_tokens: dict[int,

str] = <factory>,

)

Bases: gemma.gm.text._tokenizer.Tokenizer

Gemma 2 的分词器。

path: str | os.PathLike = 'gs://gemma-data/tokenizers/tokenizer_gemma2.model'

special_tokens

别名： gemma.gm.text._tokenizer._Gemma2SpecialTokens

VERSION: ClassVar[int] = 2