Skip to content

fix: resolve AttributeError in Encoding.is_special_token()#537

Open
HrushiYadav wants to merge 1 commit intoopenai:mainfrom
HrushiYadav:fix/is-special-token-attribute-error
Open

fix: resolve AttributeError in Encoding.is_special_token()#537
HrushiYadav wants to merge 1 commit intoopenai:mainfrom
HrushiYadav:fix/is-special-token-attribute-error

Conversation

@HrushiYadav
Copy link
Copy Markdown

Summary

Fixes #536

  • is_special_token() references self._special_token_values which is never defined, causing AttributeError on every call
  • Changed to self._special_tokens.values() which correctly checks if the token id exists among the special token values
  • Added test for is_special_token with both special and regular tokens

Verification

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

# Before fix: AttributeError
# After fix:
eot = enc.encode_single_token("<|endoftext|>")
enc.is_special_token(eot)  # True

hello = enc.encode("hello")[0]
enc.is_special_token(hello)  # False

Test plan

  • Verified bug exists on installed tiktoken (AttributeError)
  • Verified fix returns correct results (True for special, False for regular)
  • Added test in test_encoding.py

Generated with Claude Code

is_special_token() references self._special_token_values which is never
defined in __init__. Any call to enc.is_special_token(token_id) raises
AttributeError. Changed to self._special_tokens.values() which correctly
checks if the token id exists among the special token values.

Added test for is_special_token with both special and regular tokens.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Encoding.is_special_token() raises AttributeError - undefined attribute _special_token_values

1 participant