
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/advanced/custom_lexer.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_advanced_custom_lexer.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_advanced_custom_lexer.py:


Custom lexer
============

Demonstrates using a custom lexer to parse a non-textual stream of data

You can use a custom lexer to tokenize text when the lexers offered by Lark
are too slow, or not flexible enough.

You can also use it (as shown in this example) to tokenize streams of objects.

.. GENERATED FROM PYTHON SOURCE LINES 12-58

.. code-block:: Python

    from lark import Lark, Transformer, v_args
    from lark.lexer import Lexer, Token

    class TypeLexer(Lexer):
        def __init__(self, lexer_conf):
            pass

        def lex(self, data):
            for obj in data:
                if isinstance(obj, int):
                    yield Token('INT', obj)
                elif isinstance(obj, (type(''), type(u''))):
                    yield Token('STR', obj)
                else:
                    raise TypeError(obj)

    parser = Lark("""
            start: data_item+
            data_item: STR INT*

            %declare STR INT
            """, parser='lalr', lexer=TypeLexer)


    class ParseToDict(Transformer):
        @v_args(inline=True)
        def data_item(self, name, *numbers):
            return name.value, [n.value for n in numbers]

        start = dict


    def test():
        data = ['alice', 1, 27, 3, 'bob', 4, 'carrie', 'dan', 8, 6]

        print(data)

        tree = parser.parse(data)
        res = ParseToDict().transform(tree)

        print('-->')
        print(res) # prints {'alice': [1, 27, 3], 'bob': [4], 'carrie': [], 'dan': [8, 6]}


    if __name__ == '__main__':
        test()


.. _sphx_glr_download_examples_advanced_custom_lexer.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: custom_lexer.ipynb <custom_lexer.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: custom_lexer.py <custom_lexer.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: custom_lexer.zip <custom_lexer.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
