NativeElixirPdfUtilities.Tokenizer (native_elixir_pdf_utilities v0.6.0)

View Source

A PDF tokenizer in pure Elixir.

Produces a stream of tokens from a PDF byte stream. It skips whitespace and comments. It recognizes:

  • numbers (integers and reals)
  • keywords: obj, endobj, stream, endstream, xref, trailer, startxref
  • booleans true/false, null
  • names /Name (with #xx hex escapes decoded)
  • strings: literal ( ... ) with escapes and nesting; hex <...>
  • punctuation: [ ] << >> and R
  • operators: any other keyword returned as {:op, word}

Streams:

  • After emitting :stream, the next token is {:stream_data, binary} either read by the preceding dictionary /Length (if an integer) or by scanning until endstream.
  • The tokenizer does not resolve indirect /Length references.

Summary

Types

t()
token()
tokenizer_error()

Functions

new(bin)

Initialize the tokenizer from a binary.

next(st)

Return the next token and updated tokenizer state.

next_with_span(st)

Return the next token along with its byte span in the original binary.

peek(st)

Look at the next token without advancing the tokenizer state.

pending_stream_length(st)

If called after a :stream token, returns a hint for the stream length

tokenize_all(st)

Tokenize the entire binary, returning a list of tokens.

tokenize_all_with_spans(st)

Tokenize the entire binary, returning a list of {token, span}.

Types

t()

@type t() :: %NativeElixirPdfUtilities.Tokenizer{
  bin: binary(),
  dict_depth: term(),
  in_stream: term(),
  last_length: term(),
  last_length_ref: term(),
  last_name: term(),
  pending_length_first: term(),
  pending_length_second: term(),
  pos: non_neg_integer(),
  size: non_neg_integer()
}

token()

@type token() ::
  {:int, integer()}
  | {:real, float()}
  | {:name, binary()}
  | {:string, binary()}
  | {:hex_string, binary()}
  | {:stream_data, binary()}
  | {:op, binary()}
  | :null
  | true
  | false
  | :lbracket
  | :rbracket
  | :dict_start
  | :dict_end
  | :obj
  | :endobj
  | :stream
  | :endstream
  | :xref
  | :trailer
  | :startxref
  | :R
  | {:error, tokenizer_error()}
  | {:eof, nil}

tokenizer_error()

@type tokenizer_error() ::
  {:unexpected_char, byte(), non_neg_integer()}
  | {:unexpected_gt, non_neg_integer()}
  | {:not_a_number, binary()}
  | {:unterminated_literal_string, non_neg_integer()}
  | {:unterminated_hex_string, non_neg_integer()}

Functions

new(bin)

@spec new(binary()) :: t()

Initialize the tokenizer from a binary.

next(st)

@spec next(t()) :: {token(), t()}

Return the next token and updated tokenizer state.

next_with_span(st)

@spec next_with_span(t()) ::
  {{token(),
    %{
      from: non_neg_integer(),
      to: non_neg_integer(),
      stream_mode?: atom() | nil
    }}, t()}

Return the next token along with its byte span in the original binary.

The span is a map with :from and :to (exclusive end index). For {:stream_data, _} tokens, from excludes the EOL after the stream keyword.

peek(st)

@spec peek(t()) :: token()

Look at the next token without advancing the tokenizer state.

pending_stream_length(st)

@spec pending_stream_length(t()) ::
  {:direct, non_neg_integer()} | {:indirect, {integer(), integer()}} | :unknown

If called after a :stream token, returns a hint for the stream length:

  • {:direct, int} when /Length was a direct integer
  • {:indirect, {obj, gen}} when /Length was an indirect reference
  • :unknown when no hint is available

tokenize_all(st)

@spec tokenize_all(t()) :: [token()]

Tokenize the entire binary, returning a list of tokens.

tokenize_all_with_spans(st)

@spec tokenize_all_with_spans(t()) :: [
  {token(),
   %{from: non_neg_integer(), to: non_neg_integer(), stream_mode?: atom() | nil}}
]

Tokenize the entire binary, returning a list of {token, span}.