This mechanism applies only to the ``python`` expression type, and by
derivation ``string``.
.. _tales_built_in_names:
``python``
^^^^^^^^^^
Evaluates a Python expression.
Syntax
~~~~~~
Python expression syntax::
Any valid Python language expression
Description
~~~~~~~~~~~
Python expressions are executed natively within the translated
template source code. There is no built-in security apparatus.
``string``
^^^^^^^^^^
Syntax
~~~~~~
String expression syntax::
string_expression ::= ( plain_string | [ varsub ] )*
varsub ::= ( '$' Variable ) | ( '${ Expression }' )
plain_string ::= ( '$$' | non_dollar )*
non_dollar ::= any character except '$'
Description
~~~~~~~~~~~
String expressions interpret the expression string as text. If no
expression string is supplied the resulting string is *empty*. The
string can contain variable substitutions of the form ``$name`` or
``${expression}``, where ``name`` is a variable name, and ``expression`` is a TALES-expression. The escaped string value of the expression is inserted into the string.
.. note:: To prevent a ``$`` from being interpreted this
way, it must be escaped as ``$$``. Using a backslash-escape
is not supported.
Examples
~~~~~~~~
Basic string formatting::
Spam and Eggs
total: 12
Including a dollar sign::
cost: $42.00
.. _import-expression:
``import``
^^^^^^^^^^
Imports a module global.
.. _structure-expression:
``structure``
^^^^^^^^^^^^^
Wraps the expression result as *structure*: The replacement text is
inserted into the document without escaping, allowing HTML/XML markup
to be inserted. This can break your page if the text contains
unanticipated markup (eg. text submitted via a web form), which is
the reason that it is not the default.
.. _load-expression:
``load``
^^^^^^^^
Loads a template instance.
Syntax
~~~~~~
Load expression syntax::
Relative or absolute file path
Description
~~~~~~~~~~~
The template will be loaded using the same template class as the
calling template.
Examples
~~~~~~~~
Loading a template and using it as a macro::
Built-in names
--------------
These are the names always available in the TALES expression namespace:
- ``default`` - special value used to specify that existing text or attributes should not be replaced. See the documentation for individual TAL statements for details on how they interpret *default*.
- ``repeat`` - the *repeat* variables; see :ref:`tal_repeat` for more
information.
- ``template`` - reference to the template which was first called; this symbol is carried over when using macros.
- ``macros`` - reference to the macros dictionary that corresponds to the current template.
.. _metal:
Macros (METAL)
##############
The *Macro Expansion Template Attribute Language* (METAL) standard is
a facility for HTML/XML macro preprocessing. It can be used in
conjunction with or independently of TAL and TALES.
Macros provide a way to define a chunk of presentation in one
template, and share it in others, so that changes to the macro are
immediately reflected in all of the places that share it.
Additionally, macros are always fully expanded, even in a template's
source text, so that the template appears very similar to its final
rendering.
A single Page Template can accomodate multiple macros.
Namespace
---------
The METAL namespace URI and recommended alias are currently defined
as::
xmlns:metal="http://xml.zope.org/namespaces/metal"
Just like the TAL namespace URI, this URI is not attached to a web
page; it's just a unique identifier. This identifier must be used in
all templates which use METAL.
Note that elements that appear in a template with the METAL namespace
are omitted from the output where they appear. This is useful when
defining a macro::
...
In the example above the element is named `block` but any name can be
used to the same effect as long as it is qualified with the METAL
namespace.
Statements
----------
METAL defines a number of statements:
* ``metal:define-macro`` Define a macro.
* ``metal:use-macro`` Use a macro.
* ``metal:extend-macro`` Extend a macro.
* ``metal:define-slot`` Define a macro customization point.
* ``metal:fill-slot`` Customize a macro.
Although METAL does not define the syntax of expression non-terminals,
leaving that up to the implementation, a canonical expression syntax
for use in METAL arguments is described in TALES Specification.
``define-macro``
^^^^^^^^^^^^^^^^
Defines a macro.
Syntax
~~~~~~
``metal:define-macro`` syntax::
argument ::= Name
Description
~~~~~~~~~~~
The ``metal:define-macro`` statement defines a macro. The macro is named
by the statement expression, and is defined as the element and its
sub-tree.
Examples
~~~~~~~~
Simple macro definition::
Copyright 2011, Foobar Inc.
``define-slot``
^^^^^^^^^^^^^^^
Defines a macro customization point.
Syntax
~~~~~~
``metal:define-slot`` syntax::
argument ::= Name
Description
~~~~~~~~~~~
The ``metal:define-slot`` statement defines a macro customization
point or *slot*. When a macro is used, its slots can be replaced, in
order to customize the macro. Slot definitions provide default content
for the slot. You will get the default slot contents if you decide not
to customize the macro when using it.
The ``metal:define-slot`` statement must be used inside a
``metal:define-macro`` statement.
Slot names must be unique within a macro.
Examples
~~~~~~~~
Simple macro with slot::
Hello World
This example defines a macro with one slot named ``name``. When you use
this macro you can customize the ``b`` element by filling the ``name``
slot.
``fill-slot``
^^^^^^^^^^^^^
Customize a macro.
Syntax
~~~~~~
``metal:fill-slot`` syntax::
argument ::= Name
Description
~~~~~~~~~~~
The ``metal:fill-slot`` statement customizes a macro by replacing a
*slot* in the macro with the statement element (and its content).
The ``metal:fill-slot`` statement must be used inside a
``metal:use-macro`` statement.
Slot names must be unique within a macro.
If the named slot does not exist within the macro, the slot
contents will be silently dropped.
Examples
~~~~~~~~
Given this macro::
Hello World
You can fill the ``name`` slot like so::
Hello Kevin Bacon
``use-macro``
^^^^^^^^^^^^^
Use a macro.
Syntax
~~~~~~
``metal:use-macro`` syntax::
argument ::= expression
Description
~~~~~~~~~~~
The ``metal:use-macro`` statement replaces the statement element with
a macro. The statement expression describes a macro definition. The
``macroname`` variable will be bound to the defined name of the macro
being used.
.. note:: In Chameleon the expression may point to a template instance; in this case it will be rendered in its entirety.
``extend-macro``
^^^^^^^^^^^^^^^^
Extends a macro.
Syntax
~~~~~~
``metal:extend-macro`` syntax::
argument ::= expression
Description
~~~~~~~~~~~
To extend an existing macro, choose a name for the macro and add a
define-macro attribute to a document element with the name as the
argument. Add an extend-macro attribute to the document element with
an expression referencing the base macro as the argument. The
extend-macro must be used in conjunction with define-macro, and must
not be used with use-macro. The element's subtree is the macro
body.
Examples
~~~~~~~~
::
.. _i18n:
Translation (I18N)
##################
Translation of template contents and attributes is supported via the
``i18n`` namespace and message objects.
Messages
--------
The translation machinery defines a message as *any object* which is
not a string or a number and which does not provide an ``__html__``
method.
When any such object is inserted into the template, the translate
function is invoked first to see if it needs translation. The result
is always coerced to a native string before it's inserted into the
template.
Translation function
--------------------
The simplest way to hook into the translation machinery is to provide
a translation function to the template constructor or at
render-time. In either case it should be passed as the keyword
argument ``translate``.
The function has the following signature:
.. code-block:: python
def translate(msgid, domain=None, mapping=None, context=None, target_language=None, default=None):
...
The result should be a string or ``None``. If another type of object
is returned, it's automatically coerced into a string.
If `zope.i18n
`_ is available,
the translation machinery defaults to using its translation
function. Note that this function requires messages to conform to the
message class from `zope.i18nmessageid
`_; specifically,
messages must have attributes ``domain``, ``mapping`` and
``default``. Example use:
.. code-block:: python
from zope.i18nmessageid import MessageFactory
_ = MessageFactory("food")
apple = _(u"Apple")
There's currently no further support for other translation frameworks.
Using Zope's translation framework
-----------------------------------
The translation function from ``zope.i18n`` relies on *translation
domains* to provide translations.
These are components that are registered for some translation domain
identifier and which implement a ``translate`` method that translates
messages for that domain.
.. note:: To register translation domain components, the Zope Component Architecture must be used (see `zope.component `_).
The easiest way to configure translation domains is to use the the
``registerTranslations`` ZCML-directive; this requires the use of the
`zope.configuration `_
package. This will set up translation domains and gettext catalogs
automatically:
.. code-block:: xml
The ``./locales`` directory must follow a particular directory
structure:
.. code-block:: bash
./locales/en/LC_MESSAGES
./locales/de/LC_MESSAGES
...
In each of the ``LC_MESSAGES`` directories, one `GNU gettext
`_ file in the ``.po``
format must be present per translation domain:
.. code-block:: po
# ./locales/de/LC_MESSAGES/food.po
msgid ""
msgstr ""
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
msgid "Apple"
msgstr "Apfel"
It may be necessary to compile the message catalog using the
``msgfmt`` utility. This will produce a ``.mo`` file.
Translation domains without gettext
-----------------------------------
The following example demonstrates how to manually set up and
configure a translation domain for which messages are provided
directly::
from zope import component
from zope.i18n.simpletranslationdomain import SimpleTranslationDomain
food = SimpleTranslationDomain("food", {
('de', u'Apple'): u'Apfel',
})
component.provideUtility(food, food.domain)
An example of a custom translation domain class::
from zope import interface
class TranslationDomain(object):
interface.implements(ITranslationDomain)
def translate(self, msgid, mapping=None, context=None,
target_language=None, default=None):
...
component.provideUtility(TranslationDomain(), name="custom")
This approach can be used to integrate other translation catalog
implementations.
.. highlight:: xml
Namespace
---------
The ``i18n`` namespace URI and recommended prefix are currently
defined as::
xmlns:i18n="http://xml.zope.org/namespaces/i18n"
This is not a URL, but merely a unique identifier. Do not expect a
browser to resolve it successfully.
Statements
----------
The allowable ``i18n`` statements are:
- ``i18n:translate``
- ``i18n:domain``
- ``i18n:context``
- ``i18n:source``
- ``i18n:target``
- ``i18n:name``
- ``i18n:attributes``
- ``i18n:data``
- ``i18n:comment``
- ``i18n:ignore``
- ``i18n:ignore-attributes``
``i18n:translate``
^^^^^^^^^^^^^^^^^^
This attribute is used to mark units of text for translation. If this
attribute is specified with an empty string as the value, the message
ID is computed from the content of the element bearing this attribute.
Otherwise, the value of the element gives the message ID.
``i18n:domain``
^^^^^^^^^^^^^^^
The ``i18n:domain`` attribute is used to specify the domain to be used
to get the translation. If not specified, the translation services
will use a default domain. The value of the attribute is used
directly; it is not a TALES expression.
``i18n:context``
^^^^^^^^^^^^^^^^
The ``i18n:context`` attribute is used to specify the context to be
used to get the translation. If not specified, the translation
services will use a default context. The context is generally use to
distinguish identical texts in different context (because in a
translation this may not be the case.) The value of the attribute is
used literally; it is not an expression.
``i18n:source``
^^^^^^^^^^^^^^^
The ``i18n:source`` attribute specifies the language of the text to be
translated. The default is ``nothing``, which means we don't provide
this information to the translation services.
``i18n:target``
^^^^^^^^^^^^^^^
The ``i18n:target`` attribute specifies the language of the
translation we want to get. If the value is ``default``, the language
negotiation services will be used to choose the destination language.
If the value is ``nothing``, no translation will be performed; this
can be used to suppress translation within a larger translated unit.
Any other value must be a language code.
The attribute value is a TALES expression; the result of evaluating
the expression is the language code or one of the reserved values.
.. note:: ``i18n:target`` is primarily used for hints to text
extraction tools and translation teams. If you had some text that
should only be translated to e.g. German, then it probably
shouldn't be wrapped in an ``i18n:translate`` span.
``i18n:name``
^^^^^^^^^^^^^
Name the content of the current element for use in interpolation
within translated content. This allows a replaceable component in
content to be re-ordered by translation. For example::
was born in
.
would cause this text to be passed to the translation service::
"${name} was born in ${country}."
``i18n:attributes``
^^^^^^^^^^^^^^^^^^^
This attribute will allow us to translate attributes of HTML tags,
such as the ``alt`` attribute in the ``img`` tag. The
``i18n:attributes`` attribute specifies a list of attributes to be
translated with optional message IDs for each; if multiple attribute
names are given, they must be separated by semicolons. Message IDs
used in this context must not include whitespace.
Note that the value of the particular attributes come either from the
HTML attribute value itself or from the data inserted by
``tal:attributes``.
If an attibute is to be both computed using ``tal:attributes`` and
translated, the translation service is passed the result of the TALES
expression for that attribute.
An example::
In this example, we let ``tal:attributes`` set the value of the ``alt``
attribute to the text "Stop by for a visit!". This text will be
passed to the translation service, which uses the result of language
negotiation to translate "Stop by for a visit!" into the requested
language. The example text in the template, "Visit us", will simply
be discarded.
Another example, with explicit message IDs::
Here, the message ID ``up-arrow-icon`` will be used to generate the
link to an icon image file, and the message ID 'up-arrow-alttext' will
be used for the "alt" text.
``i18n:data``
^^^^^^^^^^^^^
Since TAL always returns strings, we need a way in ZPT to translate
objects, one of the most obvious cases being ``datetime`` objects. The
``data`` attribute will allow us to specify such an object, and
``i18n:translate`` will provide us with a legal format string for that
object. If ``data`` is used, ``i18n:translate`` must be used to give
an explicit message ID, rather than relying on a message ID computed
from the content.
``i18n:comment``
^^^^^^^^^^^^^^^^
The ``i18n:comment`` attribute can be used to add extra comments for
translators. It is not used by Chameleon for processing, but will be
picked up by tools like `lingua `_.
An example:
News
``i18n:ignore``
^^^^^^^^^^^^^^^
The ``i18n:ignore`` attribute can be used to inform translation extraction tools
like `i18ndude `_ to not give a
warning/error on the given tag if there is no ``i18n:translate`` attribute.
An example:
News
``i18n:ignore-attributes``
^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``i18n:ignore-attributes``, just like ``i18n:ignore`` is expected to be
used by translation extraction tools like `i18ndude `_.
If ``i18n:ignore`` makes text within a tag to be ignored, ``i18n:ignore-attributes``
marks the given attributes as ignored.
An example:
Python website
Relation with TAL processing
----------------------------
The attributes defined in the ``i18n`` namespace modify the behavior
of the TAL interpreter for the ``tal:attributes``, ``tal:content``,
``tal:repeat``, and ``tal:replace`` attributes, but otherwise do not
affect TAL processing.
Since these attributes only affect TAL processing by causing
translations to occur at specific times, using these with a TAL
processor which does not support the ``i18n`` namespace degrades well;
the structural expectations for a template which uses the ``i18n``
support is no different from those for a page which does not. The
only difference is that translations will not be performed in a legacy
processor.
Relation with METAL processing
-------------------------------
When using translation with METAL macros, the internationalization
context is considered part of the specific documents that page
components are retrieved from rather than part of the combined page.
This makes the internationalization context lexical rather than
dynamic, making it easier for a site builder to understand the
behavior of each element with respect to internationalization.
Let's look at an example to see what this means::
-
Note heading goes here
Some longer explanation for the note goes here.
And the macro source::
January
Place for the application to add additional notes if desired.
Note that the macro is using a different domain than the application
(which it should be). With lexical scoping, no special markup needs
to be applied to cause the slot-filler in the application to be part
of the same domain as the rest of the application's page components.
If dynamic scoping were used, the internationalization context would
need to be re-established in the slot-filler.
Extracting translatable message
-------------------------------
Translators use `PO files
`_
when translating messages. To create and update PO files you need to
do two things: *extract* all messages from python and templates files
and store them in a ``.pot`` file, and for each language *update* its
``.po`` file. Chameleon facilitates this by providing extractors for
`Babel `_. To use this you need modify
``setup.py``. For example:
.. code-block:: python
from setuptools import setup
setup(name="mypackage",
install_requires = [
"Babel",
],
message_extractors = { "src": [
("**.py", "chameleon_python", None ),
("**.pt", "chameleon_xml", None ),
]},
)
This tells Babel to scan the ``src`` directory while using the
``chameleon_python`` extractor for all ``.py`` files and the
``chameleon_xml`` extractor for all ``.pt`` files.
You can now use Babel to manage your PO files:
.. code-block:: bash
python setup.py extract_messages --output-file=i18n/mydomain.pot
python setup.py update_catalog \
-l nl \
-i i18n/mydomain.pot \
-o i18n/nl/LC_MESSAGES/mydomain.po
python setup.py compile_catalog \
--directory i18n --locale nl
You can also configure default options in a ``setup.cfg`` file. For example::
[compile_catalog]
domain = mydomain
directory = i18n
[extract_messages]
copyright_holder = Acme Inc.
output_file = i18n/mydomain.pot
charset = UTF-8
[init_catalog]
domain = mydomain
input_file = i18n/mydomain.pot
output_dir = i18n
[update_catalog]
domain = mydomain
input_file = i18n/mydomain.pot
output_dir = i18n
previous = true
You can now use the Babel commands directly::
python setup.py extract_messages
python setup.py update_catalog
python setup.py compile_catalog
${...} operator
###############
The ``${...}`` notation is short-hand for text insertion. The
Python-expression inside the braces is evaluated and the result
included in the output (all inserted text is escaped by default):
.. code-block:: html
${content}
To escape this behavior, prefix the notation with a backslash
character: ``\${...}``.
Note that if an object implements the ``__html__`` method, the result
of this method will be inserted as-is (without XML escaping).
Code blocks
###########
The ```` notation allows you to embed Python code in
templates:
.. code-block:: html
Please input a number from the range ${", ".join(numbers)}.
The scope of name assignments is up to the nearest macro definition,
or the template, if macros are not used.
Note that code blocks can span multiple line and start on the next
line of where the processing instruction begins:
.. code-block:: html
You can use this to debug templates:
.. code-block:: html
Markup comments
###############
You can apply the "!" and "?" modifiers to change how comments are
processed:
Drop
````
Verbatim
````
That is, evaluation of ``${...}`` expressions is disabled if the
comment opens with the "?" character.
.. _new-features:
Language extensions
###################
Chameleon extends the *page template* language with a new expression
types and language features. Some take inspiration from `Genshi
`_.
*New expression types*
The :ref:`structure ` expression wraps an
expression result as *structure*::
${structure: body.text}
The :ref:`import ` expression imports module globals::
...
The :ref:`load ` expression loads templates
relative to the current template::
...
*Tuple unpacking*
The ``tal:define`` and ``tal:repeat`` statements support tuple
unpacking::
tal:define="(a, b, c) [1, 2, 3]"
Extended `iterable unpacking
`_ using the asterisk
character is not currently supported (even for versions of
Python that support it natively).
*Dictionary lookup as fallback after attribute error*
If attribute lookup (using the ``obj.`` syntax) raises an
``AttributeError`` exception, a secondary lookup is attempted
using dictionary lookup --- ``obj['']``.
Behind the scenes, this is done by rewriting all
attribute-lookups to a custom lookup call:
.. code-block:: python
def lookup_attr(obj, key):
try:
return getattr(obj, key)
except AttributeError as exc:
try:
get = obj.__getitem__
except AttributeError:
raise exc
try:
return get(key)
except KeyError:
raise exc
*Inline string substitution*
In element attributes and in the text or tail of an element,
string expression interpolation is available using the
``${...}`` syntax::
${title or item_id}
*Code blocks*
Using ```` notation, you can embed Python
statements in your templates:
.. code-block:: html
Please input a number from the range ${", ".join(numbers)}.
*Literal content*
While the ``tal:content`` and ``tal:repeat`` attributes both
support the ``structure`` keyword which inserts the content as
a literal (without XML-escape), an object may also provide an
``__html__`` method to the same effect.
The result of the method will be inserted as *structure*.
This is particularly useful for content which is substituted
using the expression operator: ``"${...}"`` since the
``structure`` keyword is not allowed here.
*Switch statement*
Two new attributes have been added: ``tal:switch`` and
``tal:case``. A case attribute works like a condition and only
allows content if the value matches that of the nearest parent
switch value.
Incompatibilities and differences
#################################
There are a number of incompatibilities and differences between the
Chameleon language implementation and the Zope reference
implementation (ZPT):
*Default expression*
The default expression type is Python.
*Template arguments*
Arguments passed by keyword to the render- or call method are
inserted directly into the template execution namespace. This is
different from ZPT where these are only available through the
``options`` dictionary.
Zope::
Chameleon::
*Special symbols*
The ``CONTEXTS`` symbol is not available.
The `z3c.pt `_ package works as a
compatibility layer. The template classes in this package provide a
implementation which is fully compatible with ZPT.
chameleon-3.8.1/docs/requirements.txt 0000664 0000000 0000000 00000000017 13700675072 0017677 0 ustar 00root root 0000000 0000000 Chameleon>=3.3
chameleon-3.8.1/setup.py 0000664 0000000 0000000 00000004106 13700675072 0015200 0 ustar 00root root 0000000 0000000 __version__ = '3.8.1'
import os
from setuptools import setup, find_packages
from setuptools.command.test import test
here = os.path.abspath(os.path.dirname(__file__))
try:
README = open(os.path.join(here, 'README.rst')).read()
CHANGES = open(os.path.join(here, 'CHANGES.rst')).read()
except: # doesn't work under tox/pip
README = ''
CHANGES = ''
install_requires = []
class Benchmark(test):
description = "Run benchmarks"
def finalize_options(self):
self.distribution.tests_require = [
'zope.pagetemplate',
'zope.component',
'zope.i18n',
'zope.testing']
test.finalize_options(self)
def run_tests(self):
from chameleon import benchmark
print("running benchmark...")
benchmark.start()
setup(
name="Chameleon",
version=__version__,
description="Fast HTML/XML Template Compiler.",
long_description="\n\n".join((README, CHANGES)),
classifiers=[
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
],
author="Malthe Borch",
author_email="mborch@gmail.com",
url="https://chameleon.readthedocs.io",
license='BSD-like (http://repoze.org/license.html)',
packages=find_packages('src'),
package_dir = {'': 'src'},
include_package_data=True,
install_requires=install_requires,
zip_safe=False,
test_suite="chameleon.tests",
cmdclass={
'benchmark': Benchmark,
}
)
chameleon-3.8.1/src/ 0000775 0000000 0000000 00000000000 13700675072 0014254 5 ustar 00root root 0000000 0000000 chameleon-3.8.1/src/chameleon/ 0000775 0000000 0000000 00000000000 13700675072 0016207 5 ustar 00root root 0000000 0000000 chameleon-3.8.1/src/chameleon/__init__.py 0000664 0000000 0000000 00000000410 13700675072 0020313 0 ustar 00root root 0000000 0000000 from .zpt.template import PageTemplate
from .zpt.template import PageTemplateFile
from .zpt.template import PageTextTemplate
from .zpt.template import PageTextTemplateFile
from .zpt.loader import TemplateLoader as PageTemplateLoader
from .exc import TemplateError
chameleon-3.8.1/src/chameleon/ast25.py 0000664 0000000 0000000 00000010237 13700675072 0017522 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
#
# Copyright 2008 by Armin Ronacher.
# License: Python License.
#
import _ast
from _ast import *
def fix_missing_locations(node):
"""
When you compile a node tree with compile(), the compiler expects lineno and
col_offset attributes for every node that supports them. This is rather
tedious to fill in for generated nodes, so this helper adds these attributes
recursively where not already set, by setting them to the values of the
parent node. It works recursively starting at *node*.
"""
def _fix(node, lineno, col_offset):
if 'lineno' in node._attributes:
if not hasattr(node, 'lineno'):
node.lineno = lineno
else:
lineno = node.lineno
if 'col_offset' in node._attributes:
if not hasattr(node, 'col_offset'):
node.col_offset = col_offset
else:
col_offset = node.col_offset
for child in iter_child_nodes(node):
_fix(child, lineno, col_offset)
_fix(node, 1, 0)
return node
def iter_child_nodes(node):
"""
Yield all direct child nodes of *node*, that is, all fields that are nodes
and all items of fields that are lists of nodes.
"""
for name, field in iter_fields(node):
if isinstance(field, (AST, _ast.AST)):
yield field
elif isinstance(field, list):
for item in field:
if isinstance(item, (AST, _ast.AST)):
yield item
def iter_fields(node):
"""
Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
that is present on *node*.
"""
for field in node._fields or ():
try:
yield field, getattr(node, field)
except AttributeError:
pass
def walk(node):
"""
Recursively yield all child nodes of *node*, in no specified order. This is
useful if you only want to modify nodes in place and don't care about the
context.
"""
from collections import deque
todo = deque([node])
while todo:
node = todo.popleft()
todo.extend(iter_child_nodes(node))
yield node
class NodeVisitor(object):
"""
A node visitor base class that walks the abstract syntax tree and calls a
visitor function for every node found. This function may return a value
which is forwarded by the `visit` method.
This class is meant to be subclassed, with the subclass adding visitor
methods.
Per default the visitor functions for the nodes are ``'visit_'`` +
class name of the node. So a `TryFinally` node visit function would
be `visit_TryFinally`. This behavior can be changed by overriding
the `visit` method. If no visitor function exists for a node
(return value `None`) the `generic_visit` visitor is used instead.
Don't use the `NodeVisitor` if you want to apply changes to nodes during
traversing. For this a special visitor exists (`NodeTransformer`) that
allows modifications.
"""
def visit(self, node):
"""Visit a node."""
method = 'visit_' + node.__class__.__name__
visitor = getattr(self, method, self.generic_visit)
return visitor(node)
def generic_visit(self, node):
"""Called if no explicit visitor function exists for a node."""
for field, value in iter_fields(node):
if isinstance(value, list):
for item in value:
if isinstance(item, (AST, _ast.AST)):
self.visit(item)
elif isinstance(value, (AST, _ast.AST)):
self.visit(value)
class AST(object):
_fields = ()
_attributes = 'lineno', 'col_offset'
def __init__(self, *args, **kwargs):
self.__dict__.update(kwargs)
self._fields = self._fields or ()
for name, value in zip(self._fields, args):
setattr(self, name, value)
for name, cls in _ast.__dict__.items():
if isinstance(cls, type) and issubclass(cls, _ast.AST):
try:
cls.__bases__ = (AST, ) + cls.__bases__
except TypeError:
pass
class ExceptHandler(AST):
_fields = "type", "name", "body"
chameleon-3.8.1/src/chameleon/astutil.py 0000664 0000000 0000000 00000073661 13700675072 0020263 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
#
# Copyright (C) 2008-2009 Edgewall Software
# All rights reserved.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at http://genshi.edgewall.org/wiki/License.
#
# This software consists of voluntary contributions made by many
# individuals. For the exact contribution history, see the revision
# history and logs, available at http://genshi.edgewall.org/log/.
"""Support classes for generating code from abstract syntax trees."""
try:
import ast
except ImportError:
from chameleon import ast25 as ast
import sys
import logging
import weakref
import collections
AST_NONE = ast.Name(id='None', ctx=ast.Load())
node_annotations = weakref.WeakKeyDictionary()
try:
node_annotations[ast.Name()] = None
except TypeError:
logging.debug(
"Unable to create weak references to AST nodes. " \
"A lock will be used around compilation loop."
)
node_annotations = {}
__docformat__ = 'restructuredtext en'
def annotated(value):
node = load("annotation")
node_annotations[node] = value
return node
def parse(source, mode='eval'):
return compile(source, '', mode, ast.PyCF_ONLY_AST)
def load(name):
return ast.Name(id=name, ctx=ast.Load())
def store(name):
return ast.Name(id=name, ctx=ast.Store())
def param(name):
return ast.Name(id=name, ctx=ast.Param())
def delete(name):
return ast.Name(id=name, ctx=ast.Del())
def subscript(name, value, ctx):
return ast.Subscript(
value=value,
slice=ast.Index(value=ast.Str(s=name)),
ctx=ctx,
)
def walk_names(target, mode):
for node in ast.walk(target):
if isinstance(node, ast.Name) and \
isinstance(node.ctx, mode):
yield node.id
def iter_fields(node):
"""
Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
that is present on *node*.
"""
for field in node._fields:
try:
yield field, getattr(node, field)
except AttributeError:
pass
def iter_child_nodes(node):
"""
Yield all direct child nodes of *node*, that is, all fields that are nodes
and all items of fields that are lists of nodes.
"""
for name, field in iter_fields(node):
if isinstance(field, Node):
yield field
elif isinstance(field, list):
for item in field:
if isinstance(item, Node):
yield item
def walk(node):
"""
Recursively yield all descendant nodes in the tree starting at *node*
(including *node* itself), in no specified order. This is useful if you
only want to modify nodes in place and don't care about the context.
"""
todo = collections.deque([node])
while todo:
node = todo.popleft()
todo.extend(iter_child_nodes(node))
yield node
def copy(source, target):
target.__class__ = source.__class__
target.__dict__ = source.__dict__
def swap(body, replacement, name):
root = ast.Expression(body=body)
for node in ast.walk(root):
if (isinstance(node, ast.Name) and
isinstance(node.ctx, ast.Load) and
node.id == name):
assert hasattr(replacement, '_fields')
node_annotations.setdefault(node, replacement)
def marker(name):
return ast.Str(s="__%s" % name)
class Node(object):
"""AST baseclass that gives us a convenient initialization
method. We explicitly declare and use the ``_fields`` attribute."""
_fields = ()
def __init__(self, *args, **kwargs):
assert isinstance(self._fields, tuple)
self.__dict__.update(kwargs)
for name, value in zip(self._fields, args):
setattr(self, name, value)
def __repr__(self):
"""Poor man's single-line pretty printer."""
name = type(self).__name__
return '<%s%s at %x>' % (
name,
"".join(" %s=%r" % (name, getattr(self, name, "\"?\""))
for name in self._fields),
id(self)
)
def extract(self, condition):
result = []
for node in walk(self):
if condition(node):
result.append(node)
return result
class Builtin(Node):
"""Represents a Python builtin.
Used when a builtin is used internally by the compiler, to avoid
clashing with a user assignment (e.g. ``help`` is a builtin, but
also commonly assigned in templates).
"""
_fields = "id", "ctx"
ctx = ast.Load()
class Symbol(Node):
"""Represents an importable symbol."""
_fields = "value",
class Static(Node):
"""Represents a static value."""
_fields = "value", "name"
name = None
class Comment(Node):
_fields = "text", "space", "stmt"
stmt = None
space = ""
class TokenRef(Node):
"""Represents a source-code token reference."""
_fields = "pos", "length"
class ASTCodeGenerator(object):
"""General purpose base class for AST transformations.
Every visitor method can be overridden to return an AST node that has been
altered or replaced in some way.
"""
def __init__(self, tree):
self.lines_info = []
self.line_info = []
self.lines = []
self.line = ""
self.last = None
self.indent = 0
self.blame_stack = []
self.visit(tree)
if self.line.strip():
self._new_line()
self.line = None
self.line_info = None
# strip trivial lines
self.code = "\n".join(
line.strip() and line or ""
for line in self.lines
)
def _change_indent(self, delta):
self.indent += delta
def _new_line(self):
if self.line is not None:
self.lines.append(self.line)
self.lines_info.append(self.line_info)
self.line = ' ' * 4 * self.indent
if len(self.blame_stack) == 0:
self.line_info = []
self.last = None
else:
self.line_info = [(0, self.blame_stack[-1],)]
self.last = self.blame_stack[-1]
def _write(self, s):
if len(s) == 0:
return
if len(self.blame_stack) == 0:
if self.last is not None:
self.last = None
self.line_info.append((len(self.line), self.last))
else:
if self.last != self.blame_stack[-1]:
self.last = self.blame_stack[-1]
self.line_info.append((len(self.line), self.last))
self.line += s
def flush(self):
if self.line:
self._new_line()
def visit(self, node):
if node is None:
return None
if type(node) is tuple:
return tuple([self.visit(n) for n in node])
try:
self.blame_stack.append((node.lineno, node.col_offset,))
info = True
except AttributeError:
info = False
visitor = getattr(self, 'visit_%s' % node.__class__.__name__, None)
if visitor is None:
raise Exception('No handler for ``%s`` (%s).' % (
node.__class__.__name__, repr(node)))
ret = visitor(node)
if info:
self.blame_stack.pop()
return ret
def visit_Module(self, node):
for n in node.body:
self.visit(n)
visit_Interactive = visit_Module
visit_Suite = visit_Module
def visit_Expression(self, node):
return self.visit(node.body)
# arguments = (expr* args, identifier? vararg,
# identifier? kwarg, expr* defaults)
def visit_arguments(self, node):
first = True
no_default_count = len(node.args) - len(node.defaults)
for i, arg in enumerate(node.args):
if not first:
self._write(', ')
else:
first = False
self.visit(arg)
if i >= no_default_count:
self._write('=')
self.visit(node.defaults[i - no_default_count])
if getattr(node, 'vararg', None):
if not first:
self._write(', ')
else:
first = False
self._write('*' + node.vararg)
if getattr(node, 'kwarg', None):
if not first:
self._write(', ')
else:
first = False
self._write('**' + node.kwarg)
def visit_arg(self, node):
self._write(node.arg)
# FunctionDef(identifier name, arguments args,
# stmt* body, expr* decorators)
def visit_FunctionDef(self, node):
self._new_line()
for decorator in getattr(node, 'decorator_list', ()):
self._new_line()
self._write('@')
self.visit(decorator)
self._new_line()
self._write('def ' + node.name + '(')
self.visit(node.args)
self._write('):')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
# ClassDef(identifier name, expr* bases, stmt* body)
def visit_ClassDef(self, node):
self._new_line()
self._write('class ' + node.name)
if node.bases:
self._write('(')
self.visit(node.bases[0])
for base in node.bases[1:]:
self._write(', ')
self.visit(base)
self._write(')')
self._write(':')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
# Return(expr? value)
def visit_Return(self, node):
self._new_line()
self._write('return')
if getattr(node, 'value', None):
self._write(' ')
self.visit(node.value)
# Delete(expr* targets)
def visit_Delete(self, node):
self._new_line()
self._write('del ')
self.visit(node.targets[0])
for target in node.targets[1:]:
self._write(', ')
self.visit(target)
# Assign(expr* targets, expr value)
def visit_Assign(self, node):
self._new_line()
for target in node.targets:
self.visit(target)
self._write(' = ')
self.visit(node.value)
# AugAssign(expr target, operator op, expr value)
def visit_AugAssign(self, node):
self._new_line()
self.visit(node.target)
self._write(' ' + self.binary_operators[node.op.__class__] + '= ')
self.visit(node.value)
# JoinedStr(expr* values)
def visit_JoinedStr(self, node):
if node.values:
self._write('"".join((')
for value in node.values:
self.visit(value)
self._write(',')
self._write('))')
else:
self._write('""')
# FormattedValue(expr value)
def visit_FormattedValue(self, node):
if node.conversion == ord('r'):
self._write('repr')
elif node.conversion == ord('a'):
self._write('ascii')
else:
self._write('str')
self._write('(')
self.visit(node.value)
if node.format_spec is not None:
self._write(').__format__(')
self.visit(node.format_spec)
self._write(')')
# Print(expr? dest, expr* values, bool nl)
def visit_Print(self, node):
self._new_line()
self._write('print')
if getattr(node, 'dest', None):
self._write(' >> ')
self.visit(node.dest)
if getattr(node, 'values', None):
self._write(', ')
else:
self._write(' ')
if getattr(node, 'values', None):
self.visit(node.values[0])
for value in node.values[1:]:
self._write(', ')
self.visit(value)
if not node.nl:
self._write(',')
# For(expr target, expr iter, stmt* body, stmt* orelse)
def visit_For(self, node):
self._new_line()
self._write('for ')
self.visit(node.target)
self._write(' in ')
self.visit(node.iter)
self._write(':')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'orelse', None):
self._new_line()
self._write('else:')
self._change_indent(1)
for statement in node.orelse:
self.visit(statement)
self._change_indent(-1)
# While(expr test, stmt* body, stmt* orelse)
def visit_While(self, node):
self._new_line()
self._write('while ')
self.visit(node.test)
self._write(':')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'orelse', None):
self._new_line()
self._write('else:')
self._change_indent(1)
for statement in node.orelse:
self.visit(statement)
self._change_indent(-1)
# If(expr test, stmt* body, stmt* orelse)
def visit_If(self, node):
self._new_line()
self._write('if ')
self.visit(node.test)
self._write(':')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'orelse', None):
self._new_line()
self._write('else:')
self._change_indent(1)
for statement in node.orelse:
self.visit(statement)
self._change_indent(-1)
# With(expr context_expr, expr? optional_vars, stmt* body)
def visit_With(self, node):
self._new_line()
self._write('with ')
self.visit(node.context_expr)
if getattr(node, 'optional_vars', None):
self._write(' as ')
self.visit(node.optional_vars)
self._write(':')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
# Raise(expr? type, expr? inst, expr? tback)
def visit_Raise(self, node):
self._new_line()
self._write('raise')
if not getattr(node, "type", None):
exc = getattr(node, "exc", None)
if exc is None:
return
self._write(' ')
return self.visit(exc)
self._write(' ')
self.visit(node.type)
if not node.inst:
return
self._write(', ')
self.visit(node.inst)
if not node.tback:
return
self._write(', ')
self.visit(node.tback)
# Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
def visit_Try(self, node):
self._new_line()
self._write('try:')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'handlers', None):
for handler in node.handlers:
self.visit(handler)
self._new_line()
if getattr(node, 'orelse', None):
self._write('else:')
self._change_indent(1)
for statement in node.orelse:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'finalbody', None):
self._new_line()
self._write('finally:')
self._change_indent(1)
for statement in node.finalbody:
self.visit(statement)
self._change_indent(-1)
# TryExcept(stmt* body, excepthandler* handlers, stmt* orelse)
def visit_TryExcept(self, node):
self._new_line()
self._write('try:')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'handlers', None):
for handler in node.handlers:
self.visit(handler)
self._new_line()
if getattr(node, 'orelse', None):
self._write('else:')
self._change_indent(1)
for statement in node.orelse:
self.visit(statement)
self._change_indent(-1)
# excepthandler = (expr? type, expr? name, stmt* body)
def visit_ExceptHandler(self, node):
self._new_line()
self._write('except')
if getattr(node, 'type', None):
self._write(' ')
self.visit(node.type)
if getattr(node, 'name', None):
if sys.version_info[0] == 2:
assert getattr(node, 'type', None)
self._write(', ')
else:
self._write(' as ')
self.visit(node.name)
self._write(':')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
visit_excepthandler = visit_ExceptHandler
# TryFinally(stmt* body, stmt* finalbody)
def visit_TryFinally(self, node):
self._new_line()
self._write('try:')
self._change_indent(1)
for statement in node.body:
self.visit(statement)
self._change_indent(-1)
if getattr(node, 'finalbody', None):
self._new_line()
self._write('finally:')
self._change_indent(1)
for statement in node.finalbody:
self.visit(statement)
self._change_indent(-1)
# Assert(expr test, expr? msg)
def visit_Assert(self, node):
self._new_line()
self._write('assert ')
self.visit(node.test)
if getattr(node, 'msg', None):
self._write(', ')
self.visit(node.msg)
def visit_alias(self, node):
self._write(node.name)
if getattr(node, 'asname', None):
self._write(' as ')
self._write(node.asname)
# Import(alias* names)
def visit_Import(self, node):
self._new_line()
self._write('import ')
self.visit(node.names[0])
for name in node.names[1:]:
self._write(', ')
self.visit(name)
# ImportFrom(identifier module, alias* names, int? level)
def visit_ImportFrom(self, node):
self._new_line()
self._write('from ')
if node.level:
self._write('.' * node.level)
self._write(node.module)
self._write(' import ')
self.visit(node.names[0])
for name in node.names[1:]:
self._write(', ')
self.visit(name)
# Exec(expr body, expr? globals, expr? locals)
def visit_Exec(self, node):
self._new_line()
self._write('exec ')
self.visit(node.body)
if not node.globals:
return
self._write(', ')
self.visit(node.globals)
if not node.locals:
return
self._write(', ')
self.visit(node.locals)
# Global(identifier* names)
def visit_Global(self, node):
self._new_line()
self._write('global ')
self.visit(node.names[0])
for name in node.names[1:]:
self._write(', ')
self.visit(name)
# Expr(expr value)
def visit_Expr(self, node):
self._new_line()
self.visit(node.value)
# Pass
def visit_Pass(self, node):
self._new_line()
self._write('pass')
# Break
def visit_Break(self, node):
self._new_line()
self._write('break')
# Continue
def visit_Continue(self, node):
self._new_line()
self._write('continue')
### EXPRESSIONS
def with_parens(f):
def _f(self, node):
self._write('(')
f(self, node)
self._write(')')
return _f
bool_operators = {ast.And: 'and', ast.Or: 'or'}
# BoolOp(boolop op, expr* values)
@with_parens
def visit_BoolOp(self, node):
joiner = ' ' + self.bool_operators[node.op.__class__] + ' '
self.visit(node.values[0])
for value in node.values[1:]:
self._write(joiner)
self.visit(value)
binary_operators = {
ast.Add: '+',
ast.Sub: '-',
ast.Mult: '*',
ast.Div: '/',
ast.Mod: '%',
ast.Pow: '**',
ast.LShift: '<<',
ast.RShift: '>>',
ast.BitOr: '|',
ast.BitXor: '^',
ast.BitAnd: '&',
ast.FloorDiv: '//'
}
# BinOp(expr left, operator op, expr right)
@with_parens
def visit_BinOp(self, node):
self.visit(node.left)
self._write(' ' + self.binary_operators[node.op.__class__] + ' ')
self.visit(node.right)
unary_operators = {
ast.Invert: '~',
ast.Not: 'not',
ast.UAdd: '+',
ast.USub: '-',
}
# UnaryOp(unaryop op, expr operand)
def visit_UnaryOp(self, node):
self._write(self.unary_operators[node.op.__class__] + ' ')
self.visit(node.operand)
# Lambda(arguments args, expr body)
@with_parens
def visit_Lambda(self, node):
self._write('lambda ')
self.visit(node.args)
self._write(': ')
self.visit(node.body)
# IfExp(expr test, expr body, expr orelse)
@with_parens
def visit_IfExp(self, node):
self.visit(node.body)
self._write(' if ')
self.visit(node.test)
self._write(' else ')
self.visit(node.orelse)
# Dict(expr* keys, expr* values)
def visit_Dict(self, node):
self._write('{')
for key, value in zip(node.keys, node.values):
self.visit(key)
self._write(': ')
self.visit(value)
self._write(', ')
self._write('}')
def visit_Set(self, node):
self._write('{')
elts = list(node.elts)
last = elts.pop()
for elt in elts:
self.visit(elt)
self._write(', ')
self.visit(last)
self._write('}')
# ListComp(expr elt, comprehension* generators)
def visit_ListComp(self, node):
self._write('[')
self.visit(node.elt)
for generator in node.generators:
# comprehension = (expr target, expr iter, expr* ifs)
self._write(' for ')
self.visit(generator.target)
self._write(' in ')
self.visit(generator.iter)
for ifexpr in generator.ifs:
self._write(' if ')
self.visit(ifexpr)
self._write(']')
# GeneratorExp(expr elt, comprehension* generators)
def visit_GeneratorExp(self, node):
self._write('(')
self.visit(node.elt)
for generator in node.generators:
# comprehension = (expr target, expr iter, expr* ifs)
self._write(' for ')
self.visit(generator.target)
self._write(' in ')
self.visit(generator.iter)
for ifexpr in generator.ifs:
self._write(' if ')
self.visit(ifexpr)
self._write(')')
# Yield(expr? value)
def visit_Yield(self, node):
self._write('yield')
if getattr(node, 'value', None):
self._write(' ')
self.visit(node.value)
comparison_operators = {
ast.Eq: '==',
ast.NotEq: '!=',
ast.Lt: '<',
ast.LtE: '<=',
ast.Gt: '>',
ast.GtE: '>=',
ast.Is: 'is',
ast.IsNot: 'is not',
ast.In: 'in',
ast.NotIn: 'not in',
}
# Compare(expr left, cmpop* ops, expr* comparators)
@with_parens
def visit_Compare(self, node):
self.visit(node.left)
for op, comparator in zip(node.ops, node.comparators):
self._write(' ' + self.comparison_operators[op.__class__] + ' ')
self.visit(comparator)
# Call(expr func, expr* args, keyword* keywords,
# expr? starargs, expr? kwargs)
def visit_Call(self, node):
self.visit(node.func)
self._write('(')
first = True
for arg in node.args:
if not first:
self._write(', ')
first = False
self.visit(arg)
for keyword in node.keywords:
if not first:
self._write(', ')
first = False
# keyword = (identifier arg, expr value)
if keyword.arg is not None:
self._write(keyword.arg)
self._write('=')
else:
self._write('**')
self.visit(keyword.value)
# Attribute removed in Python 3.5
if getattr(node, 'starargs', None):
if not first:
self._write(', ')
first = False
self._write('*')
self.visit(node.starargs)
# Attribute removed in Python 3.5
if getattr(node, 'kwargs', None):
if not first:
self._write(', ')
first = False
self._write('**')
self.visit(node.kwargs)
self._write(')')
# Repr(expr value)
def visit_Repr(self, node):
self._write('`')
self.visit(node.value)
self._write('`')
# Constant(object value)
def visit_Constant(self, node):
if node.value is Ellipsis:
self._write('...')
else:
self._write(repr(node.value))
# Num(object n)
def visit_Num(self, node):
self._write(repr(node.n))
# Str(string s)
def visit_Str(self, node):
self._write(repr(node.s))
def visit_Ellipsis(self, node):
self._write('...')
# Attribute(expr value, identifier attr, expr_context ctx)
def visit_Attribute(self, node):
self.visit(node.value)
self._write('.')
self._write(node.attr)
# Subscript(expr value, slice slice, expr_context ctx)
def visit_Subscript(self, node):
self.visit(node.value)
self._write('[')
if isinstance(node.slice, ast.Tuple) and node.slice.elts:
self.visit(node.slice.elts[0])
if len(node.slice.elts) == 1:
self._write(', ')
else:
for dim in node.slice.elts[1:]:
self._write(', ')
self.visit(dim)
elif isinstance(node.slice, ast.Slice):
self.visit_Slice(node.slice, True)
else:
self.visit(node.slice)
self._write(']')
# Slice(expr? lower, expr? upper, expr? step)
def visit_Slice(self, node, subscription=False):
if subscription:
if getattr(node, 'lower', None) is not None:
self.visit(node.lower)
self._write(':')
if getattr(node, 'upper', None) is not None:
self.visit(node.upper)
if getattr(node, 'step', None) is not None:
self._write(':')
self.visit(node.step)
else:
self._write('slice(')
self.visit(getattr(node, "lower", None) or AST_NONE)
self._write(', ')
self.visit(getattr(node, "upper", None) or AST_NONE)
self._write(', ')
self.visit(getattr(node, "step", None) or AST_NONE)
self._write(')')
# Index(expr value)
def visit_Index(self, node):
self.visit(node.value)
# ExtSlice(slice* dims)
def visit_ExtSlice(self, node):
self.visit(node.dims[0])
if len(node.dims) == 1:
self._write(', ')
else:
for dim in node.dims[1:]:
self._write(', ')
self.visit(dim)
# Starred(expr value, expr_context ctx)
def visit_Starred(self, node):
self._write('*')
self.visit(node.value)
# Name(identifier id, expr_context ctx)
def visit_Name(self, node):
self._write(node.id)
# List(expr* elts, expr_context ctx)
def visit_List(self, node):
self._write('[')
for elt in node.elts:
self.visit(elt)
self._write(', ')
self._write(']')
# Tuple(expr *elts, expr_context ctx)
def visit_Tuple(self, node):
self._write('(')
for elt in node.elts:
self.visit(elt)
self._write(', ')
self._write(')')
# NameConstant(singleton value)
def visit_NameConstant(self, node):
self._write(str(node.value))
class AnnotationAwareVisitor(ast.NodeVisitor):
def visit(self, node):
annotation = node_annotations.get(node)
if annotation is not None:
assert hasattr(annotation, '_fields')
node = annotation
super(AnnotationAwareVisitor, self).visit(node)
def apply_transform(self, node):
if node not in node_annotations:
result = self.transform(node)
if result is not None and result is not node:
node_annotations[node] = result
class NameLookupRewriteVisitor(AnnotationAwareVisitor):
def __init__(self, transform):
self.transform = transform
self.transformed = set()
self.scopes = [set()]
def __call__(self, node):
self.visit(node)
return self.transformed
def visit_arg(self, node):
scope = self.scopes[-1]
scope.add(node.arg)
def visit_Name(self, node):
scope = self.scopes[-1]
if isinstance(node.ctx, ast.Param):
scope.add(node.id)
elif node.id not in scope:
self.transformed.add(node.id)
self.apply_transform(node)
def visit_FunctionDef(self, node):
self.scopes[-1].add(node.name)
def visit_alias(self, node):
name = node.asname if node.asname is not None else node.name
self.scopes[-1].add(name)
def visit_Lambda(self, node):
self.scopes.append(set())
try:
self.visit(node.args)
self.visit(node.body)
finally:
self.scopes.pop()
class ItemLookupOnAttributeErrorVisitor(AnnotationAwareVisitor):
def __init__(self, transform):
self.transform = transform
def visit_Attribute(self, node):
self.generic_visit(node)
self.apply_transform(node)
chameleon-3.8.1/src/chameleon/benchmark.py 0000664 0000000 0000000 00000035442 13700675072 0020523 0 ustar 00root root 0000000 0000000 import unittest
import time
import os
import re
from .utils import text_
re_amp = re.compile(r'&(?!([A-Za-z]+|#[0-9]+);)')
BIGTABLE_ZPT = """\
"""
MANY_STRINGS_ZPT = """\
"""
HELLO_WORLD_ZPT = """\
Hello, world!
"""
I18N_ZPT = """\
Hello world!
Hello world!
Hello world!
"""
def benchmark(title):
def decorator(f):
def wrapper(*args):
print(
"==========================\n " \
"%s\n==========================" % \
title)
return f(*args)
return wrapper
return decorator
def timing(func, *args, **kwargs):
t1 = t2 = time.time()
i = 0
while t2 - t1 < 3:
func(**kwargs)
func(**kwargs)
func(**kwargs)
func(**kwargs)
i += 4
t2 = time.time()
return float(10 * (t2 - t1)) / i
START = 0
END = 1
TAG = 2
def yield_tokens(table=None):
index = []
tag = index.append
_re_amp = re_amp
tag(START)
yield "<", "html", "", ">\n"
for r in table:
tag(START)
yield "<", "tr", "", ">\n"
for c in r.values():
d = c + 1
tag(START)
yield "<", "td", "", ">\n"
_tmp5 = d
if not isinstance(_tmp5, unicode):
_tmp5 = str(_tmp5)
if ('&' in _tmp5):
if (';' in _tmp5):
_tmp5 = _re_amp.sub('&', _tmp5)
else:
_tmp5 = _tmp5.replace('&', '&')
if ('<' in _tmp5):
_tmp5 = _tmp5.replace('<', '<')
if ('>' in _tmp5):
_tmp5 = _tmp5.replace('>', '>')
if ('"' in _tmp5):
_tmp5 = _tmp5.replace('"', '"')
_tmp5 = "column-%s" % _tmp5
_tmp = d
if (_tmp.__class__ not in (str, unicode, int, float, )):
raise
if (_tmp is not None):
if not isinstance(_tmp, unicode):
_tmp = str(_tmp)
if ('&' in _tmp):
if (';' in _tmp):
_tmp = _re_amp.sub('&', _tmp)
else:
_tmp = _tmp.replace('&', '&')
if ('<' in _tmp):
_tmp = _tmp.replace('<', '<')
if ('>' in _tmp):
_tmp = _tmp.replace('>', '>')
tag(START)
t = ["classicism"]
yield "<", "span", " ", t[0], '="', _tmp5, '"', ">\n"
tag(END)
yield "", "span", ">\n"
tag(END)
yield "", "td", ">\n"
tag(END)
yield "", "tr", ">\n"
tag(END)
yield "", "html", ">\n"
def yield_tokens_dict_version(**kwargs):
index = []
tag = index.append
_re_amp = re_amp
tag(START)
yield "<", "html", "", ">\n"
for r in kwargs['table']:
kwargs['r'] = r
tag(START)
yield "<", "tr", "", ">\n"
for c in kwargs['r'].values():
kwargs['d'] = c + 1
tag(START)
yield "<", "td", "", ">\n"
_tmp5 = kwargs['d']
if not isinstance(_tmp5, unicode):
_tmp5 = str(_tmp5)
if ('&' in _tmp5):
if (';' in _tmp5):
_tmp5 = _re_amp.sub('&', _tmp5)
else:
_tmp5 = _tmp5.replace('&', '&')
if ('<' in _tmp5):
_tmp5 = _tmp5.replace('<', '<')
if ('>' in _tmp5):
_tmp5 = _tmp5.replace('>', '>')
if ('"' in _tmp5):
_tmp5 = _tmp5.replace('"', '"')
_tmp5 = "column-%s" % _tmp5
_tmp = kwargs['d']
if (_tmp.__class__ not in (str, unicode, int, float, )):
raise
if (_tmp is not None):
if not isinstance(_tmp, unicode):
_tmp = str(_tmp)
if ('&' in _tmp):
if (';' in _tmp):
_tmp = _re_amp.sub('&', _tmp)
else:
_tmp = _tmp.replace('&', '&')
if ('<' in _tmp):
_tmp = _tmp.replace('<', '<')
if ('>' in _tmp):
_tmp = _tmp.replace('>', '>')
tag(START)
t = ["classicism"]
yield "<", "span", " ", t[0], '="', _tmp5, '"', ">\n"
tag(END)
yield "", "span", ">\n"
tag(END)
yield "", "td", ">\n"
tag(END)
yield "", "tr", ">\n"
tag(END)
yield "", "html", ">\n"
def yield_stream(table=None):
_re_amp = re_amp
yield START, ("html", "", "\n"), None
for r in table:
yield START, ("tr", "", "\n"), None
for c in r.values():
d = c + 1
yield START, ("td", "", "\n"), None
_tmp5 = d
if not isinstance(_tmp5, unicode):
_tmp5 = str(_tmp5)
if ('&' in _tmp5):
if (';' in _tmp5):
_tmp5 = _re_amp.sub('&', _tmp5)
else:
_tmp5 = _tmp5.replace('&', '&')
if ('<' in _tmp5):
_tmp5 = _tmp5.replace('<', '<')
if ('>' in _tmp5):
_tmp5 = _tmp5.replace('>', '>')
if ('"' in _tmp5):
_tmp5 = _tmp5.replace('"', '"')
_tmp5 = "column-%s" % _tmp5
_tmp = d
if (_tmp.__class__ not in (str, unicode, int, float, )):
raise
if (_tmp is not None):
if not isinstance(_tmp, unicode):
_tmp = str(_tmp)
if ('&' in _tmp):
if (';' in _tmp):
_tmp = _re_amp.sub('&', _tmp)
else:
_tmp = _tmp.replace('&', '&')
if ('<' in _tmp):
_tmp = _tmp.replace('<', '<')
if ('>' in _tmp):
_tmp = _tmp.replace('>', '>')
yield START, ("span", "", _tmp, " ", "class", _tmp5), None
yield END, ("span", "", "\n"), None
yield END, ("td", "", "\n"), None
yield END, ("tr", "", "\n"), None
yield END, ("html", "", "\n"), None
from itertools import chain
def bigtable_python_tokens(table=None, renderer=None):
iterable = renderer(table=table)
stream = chain(*iterable)
return "".join(stream)
def bigtable_python_stream(table=None, renderer=None):
stream = renderer(table=table)
return "".join(stream_output(stream))
def bigtable_python_stream_with_filter(table=None, renderer=None):
stream = renderer(table=table)
return "".join(stream_output(uppercase_filter(stream)))
def uppercase_filter(stream):
for kind, data, pos in stream:
if kind is START:
data = (data[0], data[1], data[2].upper(),) + data[3:]
elif kind is END:
data = (data[0], data[1], data[2].upper())
elif kind is TAG:
raise NotImplemented
yield kind, data, pos
def stream_output(stream):
for kind, data, pos in stream:
if kind is START:
tag = data[0]
yield "<%s" % tag
l = len(data)
# optimize for common cases
if l == 3:
pass
elif l == 6:
yield '%s%s="%s"' % (data[3], data[4], data[5])
else:
i = 3
while i < l:
yield '%s%s="%s"' % (data[i], data[i + 1], data[i + 2])
i += 3
yield "%s>%s" % (data[1], data[2])
elif kind is END:
yield "%s%s>%s" % data
elif kind is TAG:
raise NotImplemented
class Benchmarks(unittest.TestCase):
table = [dict(a=1, b=2, c=3, d=4, e=5, f=6, g=7, h=8, i=9, j=10) \
for x in range(1000)]
def setUp(self):
# set up i18n component
from zope.i18n import translate
from zope.i18n.interfaces import INegotiator
from zope.i18n.interfaces import ITranslationDomain
from zope.i18n.negotiator import Negotiator
from zope.i18n.simpletranslationdomain import SimpleTranslationDomain
from zope.i18n.tests.test_negotiator import Env
from zope.tales.tales import Context
self.env = Env(('klingon', 'da', 'en', 'fr', 'no'))
class ZopeI18NContext(Context):
def translate(self, msgid, domain=None, context=None,
mapping=None, default=None):
context = self.vars['options']['env']
return translate(msgid, domain, mapping,
context=context, default=default)
def _getContext(self, contexts=None, **kwcontexts):
if contexts is not None:
if kwcontexts:
kwcontexts.update(contexts)
else:
kwcontexts = contexts
return ZopeI18NContext(self, kwcontexts)
def _pt_getEngineContext(namespace):
self = namespace['template']
engine = self.pt_getEngine()
return _getContext(engine, namespace)
import zope.component
zope.component.provideUtility(Negotiator(), INegotiator)
catalog = SimpleTranslationDomain('domain')
zope.component.provideUtility(catalog, ITranslationDomain, 'domain')
self.files = os.path.abspath(os.path.join(__file__, '..', 'input'))
@staticmethod
def _chameleon(body, **kwargs):
from .zpt.template import PageTemplate
return PageTemplate(body, **kwargs)
@staticmethod
def _zope(body):
from zope.pagetemplate.pagetemplatefile import PageTemplate
template = PageTemplate()
template.pt_edit(body, 'text/xhtml')
return template
@benchmark(text_("BIGTABLE [python]"))
def test_bigtable(self):
options = {'table': self.table}
t_chameleon = timing(self._chameleon(BIGTABLE_ZPT), options=options)
print("chameleon: %7.2f" % t_chameleon)
t_chameleon_utf8 = timing(
self._chameleon(BIGTABLE_ZPT, encoding='utf-8'), options=options)
print("chameleon (utf-8): %7.2f" % t_chameleon_utf8)
t_tokens = timing(
bigtable_python_tokens, table=self.table, renderer=yield_tokens)
print("token: %7.2f" % t_tokens)
t_tokens_dict_version = timing(
bigtable_python_tokens, table=self.table,
renderer=yield_tokens_dict_version)
print("token (dict): %7.2f" % t_tokens_dict_version)
t_stream = timing(
bigtable_python_stream, table=self.table, renderer=yield_stream)
print("stream: %7.2f" % t_stream)
t_zope = timing(self._zope(BIGTABLE_ZPT), table=self.table)
print("zope.pagetemplate: %7.2f" % t_zope)
print(" %7.1fX" % (t_zope / t_chameleon))
print("--------------------------")
print("check: %d vs %d" % (
len(self._chameleon(BIGTABLE_ZPT)(options=options)),
len(self._zope(BIGTABLE_ZPT)(table=self.table))))
print("--------------------------")
@benchmark(text_("MANY STRINGS [python]"))
def test_many_strings(self):
t_chameleon = timing(self._chameleon(MANY_STRINGS_ZPT))
print("chameleon: %7.2f" % t_chameleon)
t_zope = timing(self._zope(MANY_STRINGS_ZPT))
print("zope.pagetemplate: %7.2f" % t_zope)
print(" %7.1fX" % (t_zope / t_chameleon))
print("--------------------------")
print("check: %d vs %d" % (
len(self._chameleon(MANY_STRINGS_ZPT)()),
len(self._zope(MANY_STRINGS_ZPT)())))
print("--------------------------")
@benchmark(text_("HELLO WORLD"))
def test_hello_world(self):
t_chameleon = timing(self._chameleon(HELLO_WORLD_ZPT)) * 1000
print("chameleon: %7.2f" % t_chameleon)
t_zope = timing(self._zope(HELLO_WORLD_ZPT)) * 1000
print("zope.pagetemplate: %7.2f" % t_zope)
print(" %7.1fX" % (t_zope / t_chameleon))
print("--------------------------")
print("check: %d vs %d" % (
len(self._chameleon(HELLO_WORLD_ZPT)()),
len(self._zope(HELLO_WORLD_ZPT)())))
print("--------------------------")
@benchmark(text_("I18N"))
def test_i18n(self):
from zope.i18n import translate
t_chameleon = timing(
self._chameleon(I18N_ZPT),
translate=translate,
language="klingon") * 1000
print("chameleon: %7.2f" % t_chameleon)
t_zope = timing(self._zope(I18N_ZPT), env=self.env) * 1000
print("zope.pagetemplate: %7.2f" % t_zope)
print(" %7.1fX" % (t_zope / t_chameleon))
@benchmark(text_("COMPILATION"))
def test_compilation(self):
template = self._chameleon(HELLO_WORLD_ZPT)
def chameleon_cook_and_render(template=template):
template.cook(HELLO_WORLD_ZPT)
template()
t_chameleon = timing(chameleon_cook_and_render) * 1000
print("chameleon: %7.2f" % t_chameleon)
template = self._zope(HELLO_WORLD_ZPT)
def zope_cook_and_render(templte=template):
template._cook()
template()
t_zope = timing(zope_cook_and_render) * 1000
print("zope.pagetemplate: %7.2f" % t_zope)
print(" %0.3fX" % (t_zope / t_chameleon))
def start():
result = unittest.TestResult()
test = unittest.makeSuite(Benchmarks)
test.run(result)
for error in result.errors:
print("Error in %s...\n" % error[0])
print(error[1])
for failure in result.failures:
print("Failure in %s...\n" % failure[0])
print(failure[1])
chameleon-3.8.1/src/chameleon/codegen.py 0000664 0000000 0000000 00000014640 13700675072 0020172 0 ustar 00root root 0000000 0000000 try:
import ast
except ImportError:
from chameleon import ast25 as ast
import inspect
import textwrap
import types
import copy
try:
import __builtin__ as builtins
except ImportError:
import builtins
reverse_builtin_map = {}
for name, value in builtins.__dict__.items():
try:
hash(value)
except TypeError:
continue
reverse_builtin_map[value] = name
try:
basestring
except NameError:
basestring = str
from .astutil import ASTCodeGenerator
from .astutil import load
from .astutil import store
from .astutil import parse
from .astutil import Builtin
from .astutil import Symbol
from .astutil import node_annotations
from .exc import CompilationError
try:
NATIVE_NUMBERS = int, float, long, bool
except NameError:
NATIVE_NUMBERS = int, float, bool
def template(source, mode='exec', is_func=False, func_args=(), func_defaults=(), **kw):
def wrapper(*vargs, **kwargs):
symbols = dict(zip(args, vargs + defaults))
symbols.update(kwargs)
class Visitor(ast.NodeVisitor):
def visit_FunctionDef(self, node):
self.generic_visit(node)
name = symbols.get(node.name, self)
if name is not self:
node_annotations[node] = ast.FunctionDef(
name=name,
args=node.args,
body=node.body,
decorator_list=getattr(node, "decorator_list", []),
)
def visit_Name(self, node):
value = symbols.get(node.id, self)
if value is not self:
if isinstance(value, basestring):
value = load(value)
if isinstance(value, type) or value in reverse_builtin_map:
name = reverse_builtin_map.get(value)
if name is not None:
value = Builtin(name)
else:
value = Symbol(value)
assert node not in node_annotations
assert hasattr(value, '_fields')
node_annotations[node] = value
expr = parse(textwrap.dedent(source), mode=mode)
Visitor().visit(expr)
return expr.body
assert isinstance(source, basestring)
defaults = func_defaults
args = func_args
if is_func:
return wrapper
else:
return wrapper(**kw)
class TemplateCodeGenerator(ASTCodeGenerator):
"""Extends the standard Python code generator class with handlers
for the helper node classes:
- Symbol (an importable value)
- Static (value that can be made global)
- Builtin (from the builtins module)
- Marker (short-hand for a unique static object)
"""
names = ()
def __init__(self, tree, source=None):
self.imports = {}
self.defines = {}
self.markers = {}
self.source = source
self.tokens = []
# Generate code
super(TemplateCodeGenerator, self).__init__(tree)
def visit_Module(self, node):
super(TemplateCodeGenerator, self).visit_Module(node)
# Make sure we terminate the line printer
self.flush()
# Clear lines array for import visits
body = self.lines
self.lines = []
while self.defines:
name, node = self.defines.popitem()
assignment = ast.Assign(targets=[store(name)], value=node)
self.visit(assignment)
# Make sure we terminate the line printer
self.flush()
# Clear lines array for import visits
defines = self.lines
self.lines = []
while self.imports:
value, node = self.imports.popitem()
if isinstance(value, types.ModuleType):
stmt = ast.Import(
names=[ast.alias(name=value.__name__, asname=node.id)])
elif hasattr(value, '__name__'):
path = reverse_builtin_map.get(value)
if path is None:
path = value.__module__
name = value.__name__
stmt = ast.ImportFrom(
module=path,
names=[ast.alias(name=name, asname=node.id)],
level=0,
)
else:
raise TypeError(value)
self.visit(stmt)
# Clear last import
self.flush()
# Stich together lines
self.lines += defines + body
def define(self, name, node):
assert node is not None
value = self.defines.get(name)
if value is node:
pass
elif value is None:
self.defines[name] = node
else:
raise CompilationError(
"Duplicate symbol name for define.", name)
return load(name)
def require(self, value):
if value is None:
return load("None")
if isinstance(value, NATIVE_NUMBERS):
return ast.Num(value)
node = self.imports.get(value)
if node is None:
# we come up with a unique symbol based on the class name
name = "_%s" % getattr(value, '__name__', str(value)).\
rsplit('.', 1)[-1]
node = load(name)
self.imports[value] = store(node.id)
return node
def visit(self, node):
annotation = node_annotations.get(node)
if annotation is None:
super(TemplateCodeGenerator, self).visit(node)
else:
self.visit(annotation)
def visit_Comment(self, node):
if node.stmt is None:
self._new_line()
else:
self.visit(node.stmt)
for line in node.text.replace('\r', '\n').split('\n'):
self._new_line()
self._write("%s#%s" % (node.space, line))
def visit_Builtin(self, node):
name = load(node.id)
self.visit(name)
def visit_Symbol(self, node):
node = self.require(node.value)
self.visit(node)
def visit_Static(self, node):
if node.name is None:
name = "_static_%s" % str(id(node.value)).replace('-', '_')
else:
name = node.name
node = self.define(name, node.value)
self.visit(node)
def visit_TokenRef(self, node):
self.tokens.append((node.pos, node.length))
super(TemplateCodeGenerator, self).visit(ast.Num(n=node.pos))
chameleon-3.8.1/src/chameleon/compiler.py 0000664 0000000 0000000 00000155317 13700675072 0020407 0 ustar 00root root 0000000 0000000 import re
import sys
import itertools
import logging
import threading
import functools
import collections
import pickle
import textwrap
from .astutil import load
from .astutil import store
from .astutil import param
from .astutil import swap
from .astutil import subscript
from .astutil import node_annotations
from .astutil import annotated
from .astutil import NameLookupRewriteVisitor
from .astutil import Comment
from .astutil import Symbol
from .astutil import Builtin
from .astutil import Static
from .astutil import TokenRef
from .astutil import Node
from .codegen import TemplateCodeGenerator
from .codegen import template
from .tal import ErrorInfo
from .tal import NAME
from .i18n import simple_translate
from .nodes import Text
from .nodes import Value
from .nodes import Substitution
from .nodes import Assignment
from .nodes import Module
from .nodes import Context
from .nodes import Is
from .nodes import IsNot
from .nodes import Equals
from .nodes import Logical
from .nodes import And
from .tokenize import Token
from .config import DEBUG_MODE
from .exc import TranslationError
from .exc import ExpressionError
from .parser import groupdict
from .utils import DebuggingOutputStream
from .utils import char2entity
from .utils import ListDictProxy
from .utils import native_string
from .utils import byte_string
from .utils import string_type
from .utils import unicode_string
from .utils import version
from .utils import ast
from .utils import safe_native
from .utils import builtins
from .utils import decode_htmlentities
from .utils import join
if version >= (3, 0, 0):
long = int
log = logging.getLogger('chameleon.compiler')
COMPILER_INTERNALS_OR_DISALLOWED = set([
"econtext",
"rcontext",
"str",
"int",
"float",
"long",
"len",
"None",
"True",
"False",
"RuntimeError",
])
RE_MANGLE = re.compile(r'[^\w_]')
RE_NAME = re.compile('^%s$' % NAME)
if DEBUG_MODE:
LIST = template("cls()", cls=DebuggingOutputStream, mode="eval")
else:
LIST = template("[]", mode="eval")
def identifier(prefix, suffix=None):
return "__%s_%s" % (prefix, mangle(suffix or id(prefix)))
def mangle(string):
return RE_MANGLE.sub(
'_', unicode_string(string)
).replace('\n', '').replace('-', '_')
def load_econtext(name):
return template("getitem(KEY)", KEY=ast.Str(s=name), mode="eval")
def store_econtext(name):
name = native_string(name)
return subscript(name, load("econtext"), ast.Store())
def store_rcontext(name):
name = native_string(name)
return subscript(name, load("rcontext"), ast.Store())
def set_token(stmts, token):
pos = getattr(token, "pos", 0)
body = template("__token = pos", pos=TokenRef(pos, len(token)))
return body + stmts
def eval_token(token):
try:
line, column = token.location
filename = token.filename
except AttributeError:
line, column = 0, 0
filename = ""
string = safe_native(token)
return template(
"(string, line, col)",
string=ast.Str(s=string),
line=ast.Num(n=line),
col=ast.Num(n=column),
mode="eval"
)
emit_node_if_non_trivial = template(is_func=True, func_args=('node',),
source=r"""
if node is not None:
__append(node)
""")
emit_bool = template(is_func=True,
func_args=('target', 's', 'default_marker', 'default'),
func_defaults=(None, None), source=r"""
if target is default_marker:
target = default
elif target:
target = s
else:
target = None""")
emit_convert = template(is_func=True,
func_args=('target', 'encoded', 'str', 'long', 'type',
'default_marker', 'default'),
func_defaults=(byte_string, unicode_string, long, type,
None),
source=r"""
if target is None:
pass
elif target is default_marker:
target = default
else:
__tt = type(target)
if __tt is int or __tt is float or __tt is long:
target = str(target)
elif __tt is encoded:
target = decode(target)
elif __tt is not str:
try:
target = target.__html__
except AttributeError:
__converted = convert(target)
target = str(target) if target is __converted else __converted
else:
target = target()""")
emit_func_convert = template(is_func=True,
func_args=('func', 'encoded', 'str','long','type'),
func_defaults=(byte_string, unicode_string, long,
type),
source=r"""
def func(target):
if target is None:
return
__tt = type(target)
if __tt is int or __tt is float or __tt is long:
target = str(target)
elif __tt is encoded:
target = decode(target)
elif __tt is not str:
try:
target = target.__html__
except AttributeError:
__converted = convert(target)
target = str(target) if target is __converted else __converted
else:
target = target()
return target""")
emit_translate = template(is_func=True,
func_args=('target', 'msgid', 'target_language',
'default'),
func_defaults=(None,),
source=r"""
target = translate(msgid, default=default, domain=__i18n_domain,
context=__i18n_context,
target_language=target_language)""")
emit_func_convert_and_escape = template(
is_func=True,
func_args=('func', 'str', 'long', 'type', 'encoded'),
func_defaults=(unicode_string, long, type, byte_string,),
source=r"""
def func(target, quote, quote_entity, default, default_marker):
if target is None:
return
if target is default_marker:
return default
__tt = type(target)
if __tt is int or __tt is float or __tt is long:
target = str(target)
else:
if __tt is encoded:
target = decode(target)
elif __tt is not str:
try:
target = target.__html__
except:
__converted = convert(target)
target = str(target) if target is __converted \
else __converted
else:
return target()
if target is not None:
try:
escape = __re_needs_escape(target) is not None
except TypeError:
pass
else:
if escape:
# Character escape
if '&' in target:
target = target.replace('&', '&')
if '<' in target:
target = target.replace('<', '<')
if '>' in target:
target = target.replace('>', '>')
if quote is not None and quote in target:
target = target.replace(quote, quote_entity)
return target""")
class EmitText(Node):
"""Append text to output."""
_fields = "s",
class Scope(Node):
""""Set a local output scope."""
_fields = "body", "append", "stream"
body = None
append = None
stream = None
class Interpolator(object):
braces_required_regex = re.compile(
r'(\$)?\$({(?P.*)})',
re.DOTALL)
braces_optional_regex = re.compile(
r'(\$)?\$({(?P.*)}|(?P[A-Za-z][A-Za-z0-9_]*))',
re.DOTALL)
def __init__(self, expression, braces_required, translate=False,
decode_htmlentities=False):
self.expression = expression
self.regex = self.braces_required_regex if braces_required else \
self.braces_optional_regex
self.translate = translate
self.decode_htmlentities = decode_htmlentities
def __call__(self, name, engine):
"""The strategy is to find possible expression strings and
call the ``validate`` function of the parser to validate.
For every possible starting point, the longest possible
expression is tried first, then the second longest and so
forth.
Example 1:
${'expressions use the ${} format'}
The entire expression is attempted first and it is also the
only one that validates.
Example 2:
${'Hello'} ${'world!'}
Validation of the longest possible expression (the entire
string) will fail, while the second round of attempts,
``${'Hello'}`` and ``${'world!'}`` respectively, validate.
"""
body = []
nodes = []
text = self.expression
expr_map = {}
translate = self.translate
while text:
matched = text
m = self.regex.search(matched)
if m is None:
text = text.replace('$$', '$')
nodes.append(ast.Str(s=text))
break
part = text[:m.start()]
text = text[m.start():]
skip = text.startswith('$$')
if skip:
part = part + '$'
if part:
part = part.replace('$$', '$')
node = ast.Str(s=part)
nodes.append(node)
if skip:
text = text[2:]
continue
if not body:
target = name
else:
target = store("%s_%d" % (name.id, text.pos))
while True:
d = groupdict(m, matched)
string = d["expression"] or d.get("variable") or ""
if self.decode_htmlentities:
string = decode_htmlentities(string)
if string:
try:
compiler = engine.parse(string)
body += compiler.assign_text(target)
except ExpressionError:
matched = matched[m.start():m.end() - 1]
m = self.regex.search(matched)
if m is None:
raise
continue
else:
s = m.group()
assign = ast.Assign(targets=[target], value=ast.Str(s=s))
body += [assign]
break
# If one or more expressions are not simple names, we
# disable translation.
if RE_NAME.match(string) is None:
translate = False
# if this is the first expression, use the provided
# assignment name; otherwise, generate one (here based
# on the string position)
node = load(target.id)
nodes.append(node)
expr_map[node] = safe_native(string)
text = text[len(m.group()):]
if len(nodes) == 1:
target = nodes[0]
if translate and isinstance(target, ast.Str):
target = template(
"translate(msgid, domain=__i18n_domain, context=__i18n_context, target_language=target_language)",
msgid=target, mode="eval",
target_language=load("target_language"),
)
else:
if translate:
formatting_string = ""
keys = []
values = []
for node in nodes:
if isinstance(node, ast.Str):
formatting_string += node.s
else:
string = expr_map[node]
formatting_string += "${%s}" % string
keys.append(ast.Str(s=string))
values.append(node)
target = template(
"translate(msgid, mapping=mapping, domain=__i18n_domain, context=__i18n_context, target_language=target_language)",
msgid=ast.Str(s=formatting_string),
target_language=load("target_language"),
mapping=ast.Dict(keys=keys, values=values),
mode="eval"
)
else:
nodes = [
node if isinstance(node, ast.Str) else
template(
"NODE if NODE is not None else ''",
NODE=node, mode="eval"
)
for node in nodes
]
target = ast.BinOp(
left=ast.Str(s="%s" * len(nodes)),
op=ast.Mod(),
right=ast.Tuple(elts=nodes, ctx=ast.Load()))
body += [ast.Assign(targets=[name], value=target)]
return body
class ExpressionEngine(object):
"""Expression engine.
This test demonstrates how to configure and invoke the engine.
>>> from chameleon import tales
>>> parser = tales.ExpressionParser({
... 'python': tales.PythonExpr,
... 'not': tales.NotExpr,
... 'exists': tales.ExistsExpr,
... 'string': tales.StringExpr,
... }, 'python')
>>> engine = ExpressionEngine(parser)
An expression evaluation function:
>>> eval = lambda expression: tales.test(
... tales.IdentityExpr(expression), engine)
We have provided 'python' as the default expression type. This
means that when no prefix is given, the expression is evaluated as
a Python expression:
>>> eval('not False')
True
Note that the ``type`` prefixes bind left. If ``not`` and
``exits`` are two expression type prefixes, consider the
following::
>>> eval('not: exists: int(None)')
True
The pipe operator binds right. In the following example, but
arguments are evaluated against ``not: exists: ``.
>>> eval('not: exists: help')
False
"""
supported_char_escape_set = set(('&', '<', '>'))
def __init__(self, parser, char_escape=(),
default=None, default_marker=None):
self._parser = parser
self._char_escape = char_escape
self._default = default
self._default_marker = default_marker
def __call__(self, string, target):
# BBB: This method is deprecated. Instead, a call should first
# be made to ``parse`` and then one of the assignment methods
# ("value" or "text").
compiler = self.parse(string)
return compiler(string, target)
def parse(self, string, handle_errors=True, char_escape=None):
expression = self._parser(string)
compiler = self.get_compiler(expression, string, handle_errors, char_escape)
return ExpressionCompiler(compiler, self)
def get_compiler(self, expression, string, handle_errors, char_escape):
if char_escape is None:
char_escape = self._char_escape
def compiler(target, engine, result_type=None, *args):
stmts = expression(target, engine)
if result_type is not None:
method = getattr(self, '_convert_%s' % result_type)
steps = method(target, char_escape, *args)
stmts.extend(steps)
if handle_errors:
return set_token(stmts, string.strip())
return stmts
return compiler
def _convert_bool(self, target, char_escape, s):
"""Converts value given by ``target`` to a string ``s`` if the
target is a true value, otherwise ``None``.
"""
return emit_bool(
target, ast.Str(s=s),
default=self._default,
default_marker=self._default_marker
)
def _convert_structure(self, target, char_escape):
"""Converts value given by ``target`` to structure output."""
return emit_convert(
target,
default=self._default,
default_marker=self._default_marker,
)
def _convert_text(self, target, char_escape):
"""Converts value given by ``target`` to text."""
if not char_escape:
return self._convert_structure(target, char_escape)
# This is a cop-out - we really only support a very select
# set of escape characters
other = set(char_escape) - self.supported_char_escape_set
if other:
for supported in '"', '\'', '':
if supported in char_escape:
quote = supported
break
else:
raise RuntimeError(
"Unsupported escape set: %s." % repr(char_escape)
)
else:
quote = '\0'
entity = char2entity(quote or '\0')
return template(
"TARGET = __quote(TARGET, QUOTE, Q_ENTITY, DEFAULT, MARKER)",
TARGET=target,
QUOTE=ast.Str(s=quote),
Q_ENTITY=ast.Str(s=entity),
DEFAULT=self._default,
MARKER=self._default_marker,
)
class ExpressionCompiler(object):
def __init__(self, compiler, engine):
self.compiler = compiler
self.engine = engine
def assign_bool(self, target, s):
return self.compiler(target, self.engine, "bool", s)
def assign_text(self, target):
return self.compiler(target, self.engine, "text")
def assign_value(self, target):
return self.compiler(target, self.engine)
class ExpressionEvaluator(object):
"""Evaluates dynamic expression.
This is not particularly efficient, but supported for legacy
applications.
>>> from chameleon import tales
>>> parser = tales.ExpressionParser({'python': tales.PythonExpr}, 'python')
>>> engine = functools.partial(ExpressionEngine, parser)
>>> evaluate = ExpressionEvaluator(engine, {
... 'foo': 'bar',
... })
The evaluation function is passed the local and remote context,
the expression type and finally the expression.
>>> evaluate({'boo': 'baz'}, {}, 'python', 'foo + boo')
'barbaz'
The cache is now primed:
>>> evaluate({'boo': 'baz'}, {}, 'python', 'foo + boo')
'barbaz'
Note that the call method supports currying of the expression
argument:
>>> python = evaluate({'boo': 'baz'}, {}, 'python')
>>> python('foo + boo')
'barbaz'
"""
__slots__ = "_engine", "_cache", "_names", "_builtins"
def __init__(self, engine, builtins):
self._engine = engine
self._names, self._builtins = zip(*builtins.items())
self._cache = {}
def __call__(self, econtext, rcontext, expression_type, string=None):
if string is None:
return functools.partial(
self.__call__, econtext, rcontext, expression_type
)
expression = "%s:%s" % (expression_type, string)
try:
evaluate = self._cache[expression]
except KeyError:
assignment = Assignment(["_result"], expression, True)
module = Module("evaluate", Context(assignment))
compiler = Compiler(
self._engine, module, "", string,
('econtext', 'rcontext') + self._names
)
env = {}
exec(compiler.code, env)
evaluate = self._cache[expression] = env["evaluate"]
evaluate(econtext, rcontext, *self._builtins)
return econtext['_result']
class NameTransform(object):
"""
>>> nt = NameTransform(
... set(('foo', 'bar', )), {'boo': 'boz'},
... ('econtext', ),
... )
>>> def test(node):
... rewritten = nt(node)
... module = ast.Module([ast.fix_missing_locations(rewritten)])
... codegen = TemplateCodeGenerator(module)
... return codegen.code
Any odd name:
>>> test(load('frobnitz'))
"getitem('frobnitz')"
A 'builtin' name will first be looked up via ``get`` allowing fall
back to the global builtin value:
>>> test(load('foo'))
"get('foo', foo)"
Internal names (with two leading underscores) are left alone:
>>> test(load('__internal'))
'__internal'
Compiler internals or disallowed names:
>>> test(load('econtext'))
'econtext'
Aliased names:
>>> test(load('boo'))
'boz'
"""
def __init__(self, builtins, aliases, internals):
self.builtins = builtins
self.aliases = aliases
self.internals = internals
def __call__(self, node):
name = node.id
# Don't rewrite names that begin with an underscore; they are
# internal and can be assumed to be locally defined. This
# policy really should be part of the template program, not
# defined here in the compiler.
if name.startswith('__') or name in self.internals:
return node
if isinstance(node.ctx, ast.Store):
return store_econtext(name)
aliased = self.aliases.get(name)
if aliased is not None:
return load(aliased)
# If the name is a Python global, first try acquiring it from
# the dynamic context, then fall back to the global.
if name in self.builtins:
return template(
"get(key, name)",
mode="eval",
key=ast.Str(s=name),
name=load(name),
)
# Otherwise, simply acquire it from the dynamic context.
return load_econtext(name)
class ExpressionTransform(object):
"""Internal wrapper to transform expression nodes into assignment
statements.
The node input may use the provided expression engine, but other
expression node types are supported such as ``Builtin`` which
simply resolves a built-in name.
Used internally be the compiler.
"""
loads_symbol = Symbol(pickle.loads)
def __init__(self, engine_factory, cache, visitor, strict=True):
self.engine_factory = engine_factory
self.cache = cache
self.strict = strict
self.visitor = visitor
def __call__(self, expression, target):
if isinstance(target, string_type):
target = store(target)
try:
stmts = self.translate(expression, target)
except ExpressionError:
if self.strict:
raise
exc = sys.exc_info()[1]
p = pickle.dumps(exc, -1)
stmts = template(
"__exc = loads(p)", loads=self.loads_symbol, p=ast.Str(s=p)
)
stmts += set_token([ast.Raise(exc=load("__exc"))], exc.token)
# Apply visitor to each statement
for stmt in stmts:
self.visitor(stmt)
return stmts
def translate(self, expression, target):
if isinstance(target, string_type):
target = store(target)
cached = self.cache.get(expression)
if cached is not None:
stmts = [ast.Assign(targets=[target], value=cached)]
elif isinstance(expression, ast.expr):
stmts = [ast.Assign(targets=[target], value=expression)]
else:
# The engine interface supports simple strings, which
# default to expression nodes
if isinstance(expression, string_type):
expression = Value(expression, True)
kind = type(expression).__name__
visitor = getattr(self, "visit_%s" % kind)
stmts = visitor(expression, target)
# Add comment
target_id = getattr(target, "id", target)
comment = Comment(" %r -> %s" % (expression, target_id))
stmts.insert(0, comment)
return stmts
def visit_Value(self, node, target):
engine = self.engine_factory(
default=node.default,
default_marker=node.default_marker
)
compiler = engine.parse(node.value)
return compiler.assign_value(target)
def visit_Copy(self, node, target):
return self.translate(node.expression, target)
def visit_Substitution(self, node, target):
engine = self.engine_factory(
default=node.default,
default_marker=node.default_marker
)
compiler = engine.parse(node.value, char_escape=node.char_escape)
return compiler.assign_text(target)
def visit_Negate(self, node, target):
return self.translate(node.value, target) + \
template("TARGET = not TARGET", TARGET=target)
def visit_BinOp(self, node, target):
expression = self.translate(node.left, "__expression")
value = self.translate(node.right, "__value")
op = {
Is: "is",
IsNot: "is not",
Equals: "==",
}[node.op]
return expression + value + \
template("TARGET = __expression %s __value" % op, TARGET=target)
def visit_Boolean(self, node, target):
engine = self.engine_factory(
default=node.default,
default_marker=node.default_marker,
)
compiler = engine.parse(node.value)
return compiler.assign_bool(target, node.s)
def visit_Interpolation(self, node, target):
expr = node.value
if isinstance(expr, Substitution):
engine = self.engine_factory(
char_escape=expr.char_escape,
default=expr.default,
default_marker=expr.default_marker
)
elif isinstance(expr, Value):
engine = self.engine_factory(
default=expr.default,
default_marker=expr.default_marker
)
else:
raise RuntimeError("Bad value: %r." % node.value)
interpolator = Interpolator(
expr.value, node.braces_required,
translate=node.translation,
decode_htmlentities=True
)
compiler = engine.get_compiler(
interpolator, expr.value, True, ()
)
return compiler(target, engine, "text")
def visit_Translate(self, node, target):
if node.msgid is not None:
msgid = ast.Str(s=node.msgid)
else:
msgid = target
return self.translate(node.node, target) + \
emit_translate(
target, msgid, "target_language",
default=target
)
def visit_Static(self, node, target):
value = annotated(node)
return [ast.Assign(targets=[target], value=value)]
def visit_Builtin(self, node, target):
value = annotated(node)
return [ast.Assign(targets=[target], value=value)]
def visit_Symbol(self, node, target):
value = annotated(node)
return template("TARGET = SYMBOL", TARGET=target, SYMBOL=node)
class Compiler(object):
"""Generic compiler class.
Iterates through nodes and yields Python statements which form a
template program.
"""
exceptions = NameError, \
ValueError, \
AttributeError, \
LookupError, \
TypeError
defaults = {
'translate': Symbol(simple_translate),
'decode': Builtin("str"),
'convert': Builtin("str"),
'on_error_handler': Builtin("str")
}
lock = threading.Lock()
global_builtins = set(builtins.__dict__)
def __init__(self, engine_factory, node, filename, source,
builtins={}, strict=True):
self._scopes = [set()]
self._expression_cache = {}
self._translations = []
self._builtins = builtins
self._aliases = [{}]
self._macros = []
self._current_slot = []
internals = COMPILER_INTERNALS_OR_DISALLOWED | \
set(self.defaults)
transform = NameTransform(
self.global_builtins | set(builtins),
ListDictProxy(self._aliases),
internals,
)
self._visitor = visitor = NameLookupRewriteVisitor(transform)
self._engine = ExpressionTransform(
engine_factory,
self._expression_cache,
visitor,
strict=strict,
)
if isinstance(node_annotations, dict):
self.lock.acquire()
backup = node_annotations.copy()
else:
backup = None
try:
module = ast.Module([])
module.body += self.visit(node)
ast.fix_missing_locations(module)
class Generator(TemplateCodeGenerator):
scopes = [Scope()]
def visit_EmitText(self, node):
append = load(self.scopes[-1].append or "__append")
for node in template("append(s)", append=append, s=ast.Str(s=node.s)):
self.visit(node)
def visit_Scope(self, node):
self.scopes.append(node)
body = list(node.body)
swap(body, load(node.append), "__append")
if node.stream:
swap(body, load(node.stream), "__stream")
for node in body:
self.visit(node)
self.scopes.pop()
generator = Generator(module, source)
tokens = [
Token(source[pos:pos + length], pos, source)
for pos, length in generator.tokens
]
token_map_def = "__tokens = {" + ", ".join("%d: %r" % (
token.pos,
(token, ) + token.location
) for token in tokens) + "}"
finally:
if backup is not None:
node_annotations.clear()
node_annotations.update(backup)
self.lock.release()
self.code = "\n".join((
"__filename = %r\n" % filename,
token_map_def,
generator.code
))
def visit(self, node):
if node is None:
return ()
kind = type(node).__name__
visitor = getattr(self, "visit_%s" % kind)
iterator = visitor(node)
result = []
for key, group in itertools.groupby(iterator, lambda node: node.__class__):
nodes = list(group)
if key is EmitText:
text = join(node.s for node in nodes)
nodes = [EmitText(text)]
result.extend(nodes)
return result
def visit_Sequence(self, node):
for item in node.items:
for stmt in self.visit(item):
yield stmt
def visit_Element(self, node):
for stmt in self.visit(node.start):
yield stmt
for stmt in self.visit(node.content):
yield stmt
if node.end is not None:
for stmt in self.visit(node.end):
yield stmt
def visit_Module(self, node):
body = []
body += template("import re")
body += template("import functools")
body += template("from itertools import chain as __chain")
if version < (3, 0, 0):
body += template("from sys import exc_clear as __exc_clear")
else:
body += template("from sys import intern")
body += template("__default = intern('__default__')")
body += template("__marker = object()")
body += template(
r"g_re_amp = re.compile(r'&(?!([A-Za-z]+|#[0-9]+);)')"
)
body += template(
r"g_re_needs_escape = re.compile(r'[&<>\"\']').search")
body += template(
r"__re_whitespace = "
r"functools.partial(re.compile('\\s+').sub, ' ')",
)
# Visit module content
program = self.visit(node.program)
body += [ast.FunctionDef(
name=node.name, args=ast.arguments(
args=[param(b) for b in self._builtins],
defaults=(),
),
body=program
)]
return body
def visit_MacroProgram(self, node):
functions = []
# Visit defined macros
macros = getattr(node, "macros", ())
names = []
for macro in macros:
stmts = self.visit(macro)
function = stmts[-1]
names.append(function.name)
functions += stmts
# Return function dictionary
functions += [ast.Return(value=ast.Dict(
keys=[ast.Str(s=name) for name in names],
values=[load(name) for name in names],
))]
return functions
def visit_Context(self, node):
return template("getitem = econtext.__getitem__") + \
template("get = econtext.get") + \
self.visit(node.node)
def visit_Macro(self, node):
body = []
# Initialization
body += template("__append = __stream.append")
body += template("__re_amp = g_re_amp")
body += template("__token = None")
body += template("__re_needs_escape = g_re_needs_escape")
body += emit_func_convert("__convert")
body += emit_func_convert_and_escape("__quote")
# Resolve defaults
for name in self.defaults:
body += template(
"NAME = econtext[KEY]",
NAME=name, KEY=ast.Str(s="__" + name)
)
# Internal set of defined slots
self._slots = set()
# Visit macro body
nodes = itertools.chain(*tuple(map(self.visit, node.body)))
# Slot resolution
for name in self._slots:
body += template(
"try: NAME = econtext[KEY].pop()\n"
"except: NAME = None",
KEY=ast.Str(s=name), NAME=store(name))
exc = template(
"exc_info()[1]", exc_info=Symbol(sys.exc_info), mode="eval"
)
exc_handler = template(
"if pos is not None: rcontext.setdefault('__error__', [])."
"append(token + (__filename, exc, ))",
exc=exc,
token=template("__tokens[pos]", pos="__token", mode="eval"),
pos="__token"
) + template("raise")
# Wrap visited nodes in try-except error handler.
body += [
ast.TryExcept(
body=nodes,
handlers=[ast.ExceptHandler(body=exc_handler)]
)
]
function_name = "render" if node.name is None else \
"render_%s" % mangle(node.name)
function = ast.FunctionDef(
name=function_name, args=ast.arguments(
args=[
param("__stream"),
param("econtext"),
param("rcontext"),
param("__i18n_domain"),
param("__i18n_context"),
],
defaults=[load("None"), load("None")],
),
body=body
)
yield function
def visit_Text(self, node):
yield EmitText(node.value)
def visit_Domain(self, node):
backup = "__previous_i18n_domain_%s" % mangle(id(node))
return template("BACKUP = __i18n_domain", BACKUP=backup) + \
template("__i18n_domain = NAME", NAME=ast.Str(s=node.name)) + \
self.visit(node.node) + \
template("__i18n_domain = BACKUP", BACKUP=backup)
def visit_TxContext(self, node):
backup = "__previous_i18n_context_%s" % mangle(id(node))
return template("BACKUP = __i18n_context", BACKUP=backup) + \
template("__i18n_context = NAME", NAME=ast.Str(s=node.name)) + \
self.visit(node.node) + \
template("__i18n_context = BACKUP", BACKUP=backup)
def visit_OnError(self, node):
body = []
fallback = identifier("__fallback")
body += template("fallback = len(__stream)", fallback=fallback)
self._enter_assignment((node.name, ))
fallback_body = self.visit(node.fallback)
self._leave_assignment((node.name, ))
error_assignment = template(
"econtext[key] = cls(__exc, __tokens[__token][1:3])\n"
"if handler is not None: handler(__exc)",
cls=ErrorInfo,
handler=load("on_error_handler"),
key=ast.Str(s=node.name),
)
body += [ast.TryExcept(
body=self.visit(node.node),
handlers=[ast.ExceptHandler(
type=ast.Tuple(elts=[Builtin("Exception")], ctx=ast.Load()),
name=store("__exc"),
body=(error_assignment + \
template("del __stream[fallback:]", fallback=fallback) + \
fallback_body
),
)]
)]
return body
def visit_Content(self, node):
name = "__content"
body = self._engine(node.expression, store(name))
if node.translate:
body += emit_translate(
name, name, load_econtext("target_language")
)
if node.char_escape:
body += template(
"NAME=__quote(NAME, None, '\255', None, None)",
NAME=name,
)
else:
body += template("NAME = __convert(NAME)", NAME=name)
body += template("if NAME is not None: __append(NAME)", NAME=name)
return body
def visit_Interpolation(self, node):
name = identifier("content")
return self._engine(node, name) + \
emit_node_if_non_trivial(name)
def visit_Alias(self, node):
assert len(node.names) == 1
name = node.names[0]
target = self._aliases[-1][name] = identifier(name, id(node))
return self._engine(node.expression, target)
def visit_Assignment(self, node):
for name in node.names:
if name in COMPILER_INTERNALS_OR_DISALLOWED:
raise TranslationError(
"Name disallowed by compiler.", name
)
if name.startswith('__'):
raise TranslationError(
"Name disallowed by compiler (double underscore).",
name
)
assignment = self._engine(node.expression, store("__value"))
if len(node.names) != 1:
target = ast.Tuple(
elts=[store_econtext(name) for name in node.names],
ctx=ast.Store(),
)
else:
target = store_econtext(node.names[0])
assignment.append(ast.Assign(targets=[target], value=load("__value")))
for name in node.names:
if not node.local:
assignment += template(
"rcontext[KEY] = __value", KEY=ast.Str(s=native_string(name))
)
return assignment
def visit_Define(self, node):
scope = set(self._scopes[-1])
self._scopes.append(scope)
self._aliases.append(self._aliases[-1].copy())
for assignment in node.assignments:
if assignment.local:
for stmt in self._enter_assignment(assignment.names):
yield stmt
for stmt in self.visit(assignment):
yield stmt
for stmt in self.visit(node.node):
yield stmt
for assignment in node.assignments:
if assignment.local:
for stmt in self._leave_assignment(assignment.names):
yield stmt
self._scopes.pop()
self._aliases.pop()
def visit_Omit(self, node):
return self.visit_Condition(node)
def visit_Condition(self, node):
target = "__condition"
def step(expressions, body, condition):
for i, expression in enumerate(reversed(expressions)):
stmts = evaluate(expression, body)
if i > 0:
stmts.append(
ast.If(
ast.Compare(
left=load(target),
ops=[ast.Is()],
comparators=[load(str(condition))]
),
body,
None
)
)
body = stmts
return body
def evaluate(node, body=None):
if isinstance(node, Logical):
condition = isinstance(node, And)
return step(node.expressions, body, condition)
return self._engine(node, target)
body = evaluate(node.expression)
orelse = getattr(node, "orelse", None)
body.append(
ast.If(
test=load(target),
body=self.visit(node.node) or [ast.Pass()],
orelse=self.visit(orelse) if orelse else None,
)
)
return body
def visit_Translate(self, node):
"""Translation.
Visit items and assign output to a default value.
Finally, compile a translation expression and use either
result or default.
"""
body = []
# Track the blocks of this translation
self._translations.append(set())
# Prepare new stream
append = identifier("append", id(node))
stream = identifier("stream", id(node))
body += template("s = new_list", s=stream, new_list=LIST) + \
template("a = s.append", a=append, s=stream)
# Visit body to generate the message body
code = self.visit(node.node)
body.append(Scope(code, append, stream))
# Reduce white space and assign as message id
msgid = identifier("msgid", id(node))
body += template(
"msgid = __re_whitespace(''.join(stream)).strip()",
msgid=msgid, stream=stream
)
default = msgid
# Compute translation block mapping if applicable
names = self._translations[-1]
if names:
keys = []
values = []
for name in names:
stream, append = self._get_translation_identifiers(name)
keys.append(ast.Str(s=name))
values.append(load(stream))
# Initialize value
body.insert(
0, ast.Assign(
targets=[store(stream)],
value=ast.Str(s=native_string(""))))
mapping = ast.Dict(keys=keys, values=values)
else:
mapping = None
# if this translation node has a name, use it as the message id
if node.msgid:
msgid = ast.Str(s=node.msgid)
# emit the translation expression
body += template(
"if msgid: __append(translate("
"msgid, mapping=mapping, default=default, domain=__i18n_domain, context=__i18n_context, target_language=target_language))",
msgid=msgid, default=default, mapping=mapping,
target_language=load_econtext("target_language")
)
# pop away translation block reference
self._translations.pop()
return body
def visit_Start(self, node):
try:
line, column = node.prefix.location
except AttributeError:
line, column = 0, 0
yield Comment(
" %s%s ... (%d:%d)\n"
" --------------------------------------------------------" % (
node.prefix, node.name, line, column))
if node.attributes:
yield EmitText(node.prefix + node.name)
for stmt in self.visit(node.attributes):
yield stmt
yield EmitText(node.suffix)
else:
yield EmitText(node.prefix + node.name + node.suffix)
def visit_End(self, node):
yield EmitText(node.prefix + node.name + node.space + node.suffix)
def visit_Attribute(self, node):
attr_format = (node.space + node.name + node.eq +
node.quote + "%s" + node.quote)
filter_args = list(map(self._engine.cache.get, node.filters))
filter_condition = template(
"NAME not in CHAIN",
NAME=ast.Str(s=node.name),
CHAIN=ast.Call(
func=load("__chain"),
args=filter_args,
keywords=[],
starargs=None,
kwargs=None,
),
mode="eval"
)
# Static attributes are just outputted directly
if isinstance(node.expression, ast.Str):
s = attr_format % node.expression.s
if node.filters:
return template(
"if C: __append(S)", C=filter_condition, S=ast.Str(s=s)
)
else:
return [EmitText(s)]
target = identifier("attr", node.name)
body = self._engine(node.expression, store(target))
condition = template("TARGET is not None", TARGET=target, mode="eval")
if node.filters:
condition = ast.BoolOp(
values=[condition, filter_condition],
op=ast.And(),
)
return body + template(
"if CONDITION: __append(FORMAT % TARGET)",
FORMAT=ast.Str(s=attr_format),
TARGET=target,
CONDITION=condition,
)
def visit_DictAttributes(self, node):
target = identifier("attr", id(node))
body = self._engine(node.expression, store(target))
exclude = Static(template(
"set(LIST)", LIST=ast.List(
elts=[ast.Str(s=name) for name in node.exclude],
ctx=ast.Load(),
), mode="eval"
))
body += template(
"for name, value in TARGET.items():\n "
"if name not in EXCLUDE and value is not None: __append("
"' ' + name + '=' + QUOTE + "
"QUOTE_FUNC(value, QUOTE, QUOTE_ENTITY, None, None) + QUOTE"
")",
TARGET=target,
EXCLUDE=exclude,
QUOTE_FUNC="__quote",
QUOTE=ast.Str(s=node.quote),
QUOTE_ENTITY=ast.Str(s=char2entity(node.quote or '\0')),
)
return body
def visit_Cache(self, node):
body = []
for expression in node.expressions:
# Skip re-evaluation
if self._expression_cache.get(expression):
continue
name = identifier("cache", id(expression))
target = store(name)
body += self._engine(expression, target)
self._expression_cache[expression] = target
body += self.visit(node.node)
return body
def visit_Cancel(self, node):
body = []
for expression in node.expressions:
assert self._expression_cache.get(expression) is not None
name = identifier("cache", id(expression))
target = store(name)
body += self._engine(node.value, target)
body += self.visit(node.node)
return body
def visit_UseInternalMacro(self, node):
if node.name is None:
render = "render"
else:
render = "render_%s" % mangle(node.name)
token_reset = template("__token = None")
return token_reset + template(
"f(__stream, econtext.copy(), rcontext, __i18n_domain)",
f=render) + \
template("econtext.update(rcontext)")
def visit_DefineSlot(self, node):
name = "__slot_%s" % mangle(node.name)
body = self.visit(node.node)
self._slots.add(name)
orelse = template(
"SLOT(__stream, econtext.copy(), rcontext)",
SLOT=name)
test = ast.Compare(
left=load(name),
ops=[ast.Is()],
comparators=[load("None")]
)
return [
ast.If(test=test, body=body or [ast.Pass()], orelse=orelse)
]
def visit_Name(self, node):
"""Translation name."""
if not self._translations:
raise TranslationError(
"Not allowed outside of translation.", node.name)
if node.name in self._translations[-1]:
raise TranslationError(
"Duplicate translation name: %s.", node.name)
self._translations[-1].add(node.name)
body = []
# prepare new stream
stream, append = self._get_translation_identifiers(node.name)
body += template("s = new_list", s=stream, new_list=LIST) + \
template("a = s.append", a=append, s=stream)
# generate code
code = self.visit(node.node)
body.append(Scope(code, append))
# output msgid
text = Text('${%s}' % node.name)
body += self.visit(text)
# Concatenate stream
body += template("stream = ''.join(stream)", stream=stream)
return body
def visit_CodeBlock(self, node):
stmts = template(textwrap.dedent(node.source.strip('\n')))
for stmt in stmts:
self._visitor(stmt)
return set_token(stmts, node.source)
def visit_UseExternalMacro(self, node):
self._macros.append(node.extend)
callbacks = []
for slot in node.slots:
key = "__slot_%s" % mangle(slot.name)
fun = "__fill_%s" % mangle(slot.name)
self._current_slot.append(slot.name)
body = template("getitem = econtext.__getitem__") + \
template("get = econtext.get") + \
self.visit(slot.node)
assert self._current_slot.pop() == slot.name
callbacks.append(
ast.FunctionDef(
name=fun,
args=ast.arguments(
args=[
param("__stream"),
param("econtext"),
param("rcontext"),
param("__i18n_domain"),
param("__i18n_context"),
],
defaults=[load("__i18n_domain"), load("__i18n_context")],
),
body=body or [ast.Pass()],
))
key = ast.Str(s=key)
assignment = template(
"_slots = econtext[KEY] = DEQUE((NAME,))",
KEY=key, NAME=fun, DEQUE=Symbol(collections.deque),
)
if node.extend:
append = template("_slots.appendleft(NAME)", NAME=fun)
assignment = [ast.TryExcept(
body=template("_slots = getitem(KEY)", KEY=key),
handlers=[ast.ExceptHandler(body=assignment)],
orelse=append,
)]
callbacks.extend(assignment)
assert self._macros.pop() == node.extend
assignment = self._engine(node.expression, store("__macro"))
return (
callbacks +
assignment +
set_token(
template("__m = __macro.include"),
node.expression.value
) +
template(
"__m(__stream, econtext.copy(), "
"rcontext, __i18n_domain)"
) +
template("econtext.update(rcontext)")
)
def visit_Repeat(self, node):
# Used for loop variable definition and restore
self._scopes.append(set())
# Variable assignment and repeat key for single- and
# multi-variable repeat clause
if node.local:
contexts = "econtext",
else:
contexts = "econtext", "rcontext"
for name in node.names:
if name in COMPILER_INTERNALS_OR_DISALLOWED:
raise TranslationError(
"Name disallowed by compiler.", name
)
if len(node.names) > 1:
targets = [
ast.Tuple(elts=[
subscript(native_string(name), load(context), ast.Store())
for name in node.names], ctx=ast.Store())
for context in contexts
]
key = ast.Tuple(
elts=[ast.Str(s=name) for name in node.names],
ctx=ast.Load())
else:
name = node.names[0]
targets = [
subscript(native_string(name), load(context), ast.Store())
for context in contexts
]
key = ast.Str(s=node.names[0])
index = identifier("__index", id(node))
assignment = [ast.Assign(targets=targets, value=load("__item"))]
# Make repeat assignment in outer loop
names = node.names
local = node.local
outer = self._engine(node.expression, store("__iterator"))
if local:
outer[:] = list(self._enter_assignment(names)) + outer
outer += template(
"__iterator, INDEX = getitem('repeat')(key, __iterator)",
key=key, INDEX=index
)
# Set a trivial default value for each name assigned to make
# sure we assign a value even if the iteration is empty
outer += [ast.Assign(
targets=[store_econtext(name)
for name in node.names],
value=load("None"))
]
# Compute inner body
inner = self.visit(node.node)
# After each iteration, decrease the index
inner += template("index -= 1", index=index)
# For items up to N - 1, emit repeat whitespace
inner += template(
"if INDEX > 0: __append(WHITESPACE)",
INDEX=index, WHITESPACE=ast.Str(s=node.whitespace)
)
# Main repeat loop
outer += [ast.For(
target=store("__item"),
iter=load("__iterator"),
body=assignment + inner,
)]
# Finally, clean up assignment if it's local
if outer:
outer += self._leave_assignment(names)
self._scopes.pop()
return outer
def _get_translation_identifiers(self, name):
assert self._translations
prefix = str(id(self._translations[-1])).replace('-', '_')
stream = identifier("stream_%s" % prefix, name)
append = identifier("append_%s" % prefix, name)
return stream, append
def _enter_assignment(self, names):
for name in names:
for stmt in template(
"BACKUP = get(KEY, __marker)",
BACKUP=identifier("backup_%s" % name, id(names)),
KEY=ast.Str(s=native_string(name)),
):
yield stmt
def _leave_assignment(self, names):
for name in names:
for stmt in template(
"if BACKUP is __marker: del econtext[KEY]\n"
"else: econtext[KEY] = BACKUP",
BACKUP=identifier("backup_%s" % name, id(names)),
KEY=ast.Str(s=native_string(name)),
):
yield stmt
chameleon-3.8.1/src/chameleon/config.py 0000664 0000000 0000000 00000003534 13700675072 0020033 0 ustar 00root root 0000000 0000000 import os
import logging
log = logging.getLogger('chameleon.config')
environment = dict(
(k[10:], v) for (k, v) in (
((j.lower(), x) for (j, x) in os.environ.items()))
if k.startswith('chameleon_')
)
# Define which values are read as true
TRUE = ('y', 'yes', 't', 'true', 'on', '1')
# If eager parsing is enabled, templates are parsed upon
# instantiation, rather than when first called upon; this mode is
# useful for verifying validity of templates across a project
EAGER_PARSING = environment.pop('eager', 'false')
EAGER_PARSING = EAGER_PARSING.lower() in TRUE
# Debug mode is mostly useful for debugging the template engine
# itself. When enabled, generated source code is written to disk to
# ease step-debugging and some log levels are lowered to increase
# output. Also, the generated source code is available in the
# ``source`` attribute of the template instance if compilation
# succeeded.
DEBUG_MODE = environment.pop('debug', 'false')
DEBUG_MODE = DEBUG_MODE.lower() in TRUE
# If a cache directory is specified, template source code will be
# persisted on disk and reloaded between sessions
path = environment.pop('cache', None)
if path is not None:
CACHE_DIRECTORY = os.path.abspath(path)
if not os.path.exists(CACHE_DIRECTORY):
raise ValueError(
"Cache directory does not exist: %s." % CACHE_DIRECTORY
)
log.info("directory cache: %s." % CACHE_DIRECTORY)
else:
CACHE_DIRECTORY = None
# When auto-reload is enabled, templates are reloaded on file change.
AUTO_RELOAD = environment.pop('reload', 'false')
AUTO_RELOAD = AUTO_RELOAD.lower() in TRUE
for key in environment:
log.warning(
"unknown environment variable set: \"CHAMELEON_%s\"." % key.upper()
)
# This is the slice length of the expression displayed in the
# formatted exception string
SOURCE_EXPRESSION_MARKER_LENGTH = 60
chameleon-3.8.1/src/chameleon/exc.py 0000664 0000000 0000000 00000021053 13700675072 0017341 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
import traceback
from .utils import create_formatted_exception
from .utils import safe_native
from .tokenize import Token
from .config import SOURCE_EXPRESSION_MARKER_LENGTH as LENGTH
def compute_source_marker(line, column, expression, size):
"""Computes source marker location string.
>>> def test(l, c, e, s):
... s, marker = compute_source_marker(l, c, e, s)
... out = s + '\\n' + marker
...
... # Replace dot with middle-dot to work around doctest ellipsis
... print(out.replace('...', '···'))
>>> test('foo bar', 4, 'bar', 7)
foo bar
^^^
>>> test('foo ${bar}', 4, 'bar', 10)
foo ${bar}
^^^
>>> test(' foo bar', 6, 'bar', 6)
··· oo bar
^^^
>>> test(' foo bar baz ', 6, 'bar', 6)
··· o bar ···
^^^
The entire expression is always shown, even if ``size`` does not
accomodate for it.
>>> test(' foo bar baz ', 6, 'bar baz', 10)
··· oo bar baz
^^^^^^^
>>> test(' foo bar', 10, 'bar', 5)
··· o bar
^^^
>>> test(' foo bar', 10, 'boo', 5)
··· o bar
^
"""
s = line.lstrip()
column -= len(line) - len(s)
s = s.rstrip()
try:
i = s[column:].index(expression)
except ValueError:
# If we can't find the expression
# (this shouldn't happen), simply
# use a standard size marker
marker = "^"
else:
column += i
marker = "^" * len(expression)
if len(expression) > size:
offset = column
size = len(expression)
else:
window = (size - len(expression)) / 2.0
offset = column - window
offset -= min(3, max(0, column + window + len(expression) - len(s)))
offset = int(offset)
if offset > 0:
s = s[offset:]
r = s.lstrip()
d = len(s) - len(r)
s = "... " + r
column += 4 - d
column -= offset
# This also adds to the displayed length
size += 4
if len(s) > size:
s = s[:size].rstrip() + " ..."
return s, column * " " + marker
def iter_source_marker_lines(source, expression, line, column):
for i, l in enumerate(source):
if i + 1 != line:
continue
s, marker = compute_source_marker(
l, column, expression, LENGTH
)
yield " - Source: %s" % s
yield " %s" % marker
break
def ellipsify(string, limit):
if len(string) > limit:
return "... " + string[-(limit - 4):]
return string
class RenderError(Exception):
"""An error raised during rendering.
This class is used as a mixin which is added to the original
exception.
"""
class TemplateError(Exception):
"""An error raised by Chameleon.
>>> from chameleon.tokenize import Token
>>> token = Token('token')
>>> message = 'message'
Make sure the exceptions can be copied:
>>> from copy import copy
>>> copy(TemplateError(message, token))
TemplateError('message', 'token')
And pickle/unpickled:
>>> from pickle import dumps, loads
>>> loads(dumps(TemplateError(message, token), -1))
TemplateError('message', 'token')
"""
def __init__(self, msg, token):
if not isinstance(token, Token):
token = Token(token, 0)
Exception.__init__(self, msg, token)
def __copy__(self):
inst = Exception.__new__(type(self))
inst.args = self.args
return inst
def __str__(self):
text = "%s\n\n" % self.args[0]
text += " - String: \"%s\"" % safe_native(self.token)
if self.filename:
text += "\n"
text += " - Filename: %s" % self.filename
line, column = self.location
text += "\n"
text += " - Location: (line %d: col %d)" % (line, column)
if line and column:
if self.token.source:
lines = iter_source_marker_lines(
self.token.source.splitlines(),
self.token, line, column
)
elif self.filename and not self.filename.startswith('<'):
try:
f = open(self.filename, 'r')
except IOError:
pass
else:
it = iter_source_marker_lines(
iter(f), self.token, line, column
)
try:
lines = list(lines)
finally:
f.close()
else:
lines = ()
# Prepend newlines.
for line in lines:
text += "\n" + line
return text
def __repr__(self):
try:
return "%s('%s', '%s')" % (
self.__class__.__name__, self.args[0], safe_native(self.token)
)
except AttributeError:
return object.__repr__(self)
@property
def token(self):
return self.args[1]
@property
def filename(self):
return self.token.filename
@property
def location(self):
return self.token.location
@property
def offset(self):
return getattr(self.token, "pos", 0)
class ParseError(TemplateError):
"""An error occurred during parsing.
Indicates an error on the structural level.
"""
class CompilationError(TemplateError):
"""An error occurred during compilation.
Indicates a general compilation error.
"""
class TranslationError(TemplateError):
"""An error occurred during translation.
Indicates a general translation error.
"""
class LanguageError(CompilationError):
"""Language syntax error.
Indicates a syntactical error due to incorrect usage of the
template language.
"""
class ExpressionError(LanguageError):
"""An error occurred compiling an expression.
Indicates a syntactical error in an expression.
"""
class ExceptionFormatter(object):
def __init__(self, errors, econtext, rcontext, value_repr):
kwargs = rcontext.copy()
kwargs.update(econtext)
for name in tuple(kwargs):
if name.startswith('__'):
del kwargs[name]
self._errors = errors
self._kwargs = kwargs
self._value_repr = value_repr
def __call__(self):
# Format keyword arguments; consecutive arguments are indented
# for readability
formatted = [
"%s: %s" % (name, self._value_repr(value))
for name, value in self._kwargs.items()
]
for index, string in enumerate(formatted[1:]):
formatted[index + 1] = " " * 15 + string
out = []
for error in self._errors:
expression, line, column, filename, exc = error
if isinstance(exc, UnicodeDecodeError):
string = safe_native(exc.object)
s, marker = compute_source_marker(
string, exc.start, string[exc.start:exc.end], LENGTH
)
out.append(" - Stream: %s" % s)
out.append(" %s" % marker)
_filename = ellipsify(filename, 60) if filename else ""
out.append(" - Expression: \"%s\"" % expression)
out.append(" - Filename: %s" % _filename)
out.append(" - Location: (line %d: col %d)" % (line, column))
if filename and not filename.startswith('<') and line and column:
try:
f = open(filename, 'r')
except IOError:
pass
else:
lines = iter_source_marker_lines(
iter(f), expression, line, column
)
try:
out.extend(lines)
finally:
f.close()
out.append(" - Arguments: %s" % "\n".join(formatted))
if isinstance(exc.__str__, ExceptionFormatter):
# This is a nested error that has already been wrapped
# We must unwrap it before trying to format it to prevent
# recursion
exc = create_formatted_exception(exc, type(exc), exc._original__str__)
formatted = traceback.format_exception_only(type(exc), exc)[-1]
formatted_class = "%s:" % type(exc).__name__
if formatted.startswith(formatted_class):
formatted = formatted[len(formatted_class):].lstrip()
return "\n".join(map(safe_native, [formatted] + out))
chameleon-3.8.1/src/chameleon/i18n.py 0000664 0000000 0000000 00000007304 13700675072 0017344 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2001, 2002 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
import re
from .exc import CompilationError
from .utils import unicode_string
NAME_RE = r"[a-zA-Z][-a-zA-Z0-9_]*"
WHITELIST = frozenset([
"translate",
"domain",
"context",
"target",
"source",
"attributes",
"data",
"name",
"mode",
"xmlns",
"xml",
"comment",
"ignore",
"ignore-attributes",
])
_interp_regex = re.compile(r'(?