gm.text.Gemma3Tokenizer

class gemma.gm.text.Gemma3Tokenizer(

path: str | os.PathLike = 'gs://gemma-data/tokenizers/tokenizer_gemma3.model',

*,

custom_tokens: dict[int,

str] = <factory>,

)

基类： gemma.gm.text._tokenizer.Tokenizer

Gemma 3 的分词器。

path: str | os.PathLike = 'gs://gemma-data/tokenizers/tokenizer_gemma3.model'

special_tokens

别名： gemma.gm.text._tokenizer._Gemma3SpecialTokens

FORBIDDEN_TOKENS: ClassVar[tuple[int, ...]] = (_Gemma3SpecialTokens.START_OF_IMAGE, _Gemma3SpecialTokens.END_OF_IMAGE)

VERSION: ClassVar[int] = 3