Nuitka-0.5.18.1/0000755000175000017500000000000012651072347013440 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/LICENSE.txt0000644000175000017500000002613612651070406015265 0ustar hayenhayen00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Nuitka-0.5.18.1/misc/0000755000175000017500000000000012651072347014373 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/misc/sort-imports.sh0000755000175000017500000000171612651071040017406 0ustar hayenhayen00000000000000#!/bin/bash # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # for filename in $(find nuitka -name \*.py -a \! -path \*inline_copy\*; find bin -name \*.py; echo bin/nuitka nuitka/build/SingleExe.scons); do echo isort -ot -m3 $filename done Nuitka-0.5.18.1/misc/print-all-sources.sh0000755000175000017500000000207512651071040020306 0ustar hayenhayen00000000000000#!/bin/sh # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # cd `dirname $0`/.. find nuitka -name \*.py -a \! -path *inline_copy* find bin -name \*.py find nuitka/build/static_src -name \*.cpp find nuitka/build/include -name \*.hpp find nuitka/build/ -name \*.scons find misc -name \*.sh find bin -name \*.sh echo Developer_Manual.rst echo Changelog.rst Nuitka-0.5.18.1/misc/nuitka.bat0000644000175000017500000000160712651071040016347 0ustar hayenhayen00000000000000@echo off rem Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com rem rem Part of "Nuitka", an optimizing Python compiler that is compatible and rem integrates with CPython, but also works on its own. rem rem Licensed under the Apache License, Version 2.0 (the "License"); rem you may not use this file except in compliance with the License. rem You may obtain a copy of the License at rem rem http://www.apache.org/licenses/LICENSE-2.0 rem rem Unless required by applicable law or agreed to in writing, software rem distributed under the License is distributed on an "AS IS" BASIS, rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. rem See the License for the specific language governing permissions and rem limitations under the License. rem setlocal "%~dp0..\python" "%~dp0nuitka" %* endlocal Nuitka-0.5.18.1/misc/check-with-pylint0000755000175000017500000001626112651072046017666 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Disabled globally: # # W0232: Class has no __init__ method # Who cares, we are using overrides that don't need to change object init a lot # and rarely ever made a mistake with forgetting to call used __init__ of the # parent. # # I0011: Locally disabling W.... # Strange one anyway, we want to locally disable stuff. And that just makes it # a different warning. Amazing. Luckily we can decide to ignore that globally # then. # # I0012: Locally enabling W.... # Sure, we disabled it for a block, and re-enabled it then. # # C0326: No space allowed... # Our spaces before keyword argument calls are not allowed, and this is # not possible to distinguish. # # C0330: Wrong hanging line indentation # No it's not wrong. # # C1001: Old-style class defined. # Yes, so what, why care. # # E1120 / E1123: .... # Constructor call checks frequently fail miserably, so this is full of # mysterious false alarms, while it's unlikely to help much. # # E1103: Instance of 'x' has no 'y' member but some types could not be inferred # Rarely is this any help, but it's full of false alarms. # # W0632: Possible unbalanced tuple unpacking with sequence defined at ... # It's not really good at guessing these things. # # W1504: Using type() instead of isinstance() for a typecheck. # Nuitka is all about exact type checks, so this doesn't apply # # C0123: Using type() instead of isinstance() for a typecheck. # Nuitka is all about exact type checks, so this doesn't apply # # C0413: Import "..." should be placed at the top of the module # There is no harm to this and imports are deal with by isort binary. # # C0411: external import "external" comes before "local" # There is no harm to this and imports are deal with by isort binary. # # R0204: Redefinition of var type from x to y # I do this all the time, e.g. to convert str to unicode, or list to string. from __future__ import print_function import sys, os, subprocess # Go its own directory, to have it easy with path knowledge. os.chdir(os.path.dirname(os.path.abspath(__file__))) os.chdir("..") pylint_options = """\ --rcfile=/dev/null --disable=I0011,I0012,W0232,C0326,C0330,C1001,E1103,W0632,W1504,C0123,C0413,C0411,R0204 --msg-template="{path}:{line} {msg_id} {obj} {msg}" --reports=no --persistent=no --method-rgx=[a-z_][a-zA-Z0-9_]{2,40}$ --module-rgx=.* --function-rgx=.* --variable-rgx=.* --argument-rgx=.* --dummy-variables-rgx=_.*|constraint_collection --const-rgx=.* --max-line-length=120 --no-docstring-rgx=.* --max-module-lines=5000 --min-public-methods=0 --max-public-methods=100 --max-args=10 --max-parents=10 --max-nested-blocks=10 --max-bool-expr=10 --enable=useless-suppression""".split('\n') from optparse import OptionParser parser = OptionParser() parser.add_option( "--hide-todos", "--no-todos", action = "store_true", dest = "no_todos", default = False, help = """\ Default is %default.""" ) parser.add_option( "--verbose", action = "store_true", dest = "verbose", default = False, help = """\ Default is %default.""" ) options, positional_args = parser.parse_args() if os.environ.get("TODO", 0) or options.no_todos: pylint_options.append("--notes=") blacklist = ( "oset.py", "odict.py", "SyntaxHighlighting.py", ) our_exit_code = 0 def executePyLint(filename): if options.verbose: print("Checking", filename, "...") global our_exit_code extra_options = os.environ.get("PYLINT_EXTRA_OPTIONS", "").split() if "" in extra_options: extra_options.remove("") command = ["pylint"] + pylint_options + extra_options + [filename] process = subprocess.Popen( args = command, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, shell = False ) stdout, _stderr = process.communicate() exit_code = process.returncode assert not _stderr if stdout: stdout = stdout.replace("\r\n", '\n') # Remove hard to disable error line given under Windows. lines = stdout.split(b"\n") try: error_line = lines.index( b"No config file found, using default configuration" ) del lines[error_line] del lines[error_line] except ValueError: pass lines = [ line.decode() for line in lines ] lines = [ line for line in lines if "Unable to import 'resource'" not in line ] # If we filtered everything away, remove the leading file name report. if len(lines) == 1: assert lines[0].startswith("*****") lines = [] for line in lines: print(line) if lines: our_exit_code = 1 sys.stdout.flush() if "PYTHONPATH" not in os.environ: os.environ["PYTHONPATH"] = '.' # For Windows, add this to the PATH, so pip installed PyLint will be found # near the Python executing this script. os.environ["PATH"] = os.environ["PATH"] + os.pathsep + \ os.path.join(os.path.dirname(sys.executable),"scripts") def addFromDirectory(path): for dirpath, dirnames, filenames in os.walk(path): dirnames.sort() if "inline_copy" in dirnames: dirnames.remove("inline_copy") filenames.sort() for filename in filenames: if not filename.endswith(".py"): continue # Skip temporary files from flymake mode of Emacs. if filename.endswith("_flymake.py"): continue # Skip temporary files from unsaved files of Emacs. if filename.startswith(".#"): continue if filename in blacklist: continue check_filenames.append( os.path.join(dirpath, filename) ) if not positional_args: positional_args = ["bin/nuitka", "nuitka"] check_filenames = [] for positional_arg in positional_args: if os.path.isdir(positional_arg): addFromDirectory(positional_arg) else: check_filenames.append(positional_arg) pylint_version = subprocess.check_output( ["pylint", "--version"], stderr = open(os.devnull, 'w') ) pylint_version = pylint_version.split(b"\n")[0].split()[-1] if pylint_version < b"1.4.4": sys.exit("Error, needs PyLint 1.4.4 or higher.") for check_filename in check_filenames: executePyLint(check_filename) sys.exit(our_exit_code) Nuitka-0.5.18.1/misc/Logo/0000755000175000017500000000000012651072347015273 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/misc/Logo/Nuitka-Logo-Symbol.svg0000644000175000017500000001635112651070406021410 0ustar hayenhayen00000000000000 image/svg+xml Nuitka-0.5.18.1/misc/Logo/Nuitka-Logo-Vertical.svg0000644000175000017500000004277512651070406021725 0ustar hayenhayen00000000000000 image/svg+xml Nuitka Nuitka-0.5.18.1/misc/Logo/Nuitka-Logo-Horizontal.svg0000644000175000017500000001623112651070406022271 0ustar hayenhayen00000000000000 image/svg+xml Nuitka Nuitka-0.5.18.1/misc/nuitka-run.bat0000644000175000017500000000161312651071040017146 0ustar hayenhayen00000000000000@echo off rem Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com rem rem Part of "Nuitka", an optimizing Python compiler that is compatible and rem integrates with CPython, but also works on its own. rem rem Licensed under the Apache License, Version 2.0 (the "License"); rem you may not use this file except in compliance with the License. rem You may obtain a copy of the License at rem rem http://www.apache.org/licenses/LICENSE-2.0 rem rem Unless required by applicable law or agreed to in writing, software rem distributed under the License is distributed on an "AS IS" BASIS, rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. rem See the License for the specific language governing permissions and rem limitations under the License. rem setlocal "%~dp0..\python" "%~dp0nuitka-run" %* endlocal Nuitka-0.5.18.1/Developer_Manual.rst0000644000175000017500000033144712651070406017422 0ustar hayenhayen00000000000000Nuitka Developer Manual ~~~~~~~~~~~~~~~~~~~~~~~ .. image:: images/Nuitka-Logo-Symbol.png .. contents:: .. raw:: pdf PageBreak oneColumn SetPageCounter 1 The purpose of this developer manual is to present the current design of Nuitka, the project rules, and the motivations for choices made. It is intended to be a guide to the source code, and to give explanations that don't fit into the source code in comments form. It should be used as a reference for the process of planning and documenting decisions we made. Therefore we are e.g. presenting here the type inference plans before implementing them. And we update them as we proceed. It grows out of discussions and presentations made at conferences as well as private conversations or discussions on the mailing list or bug tracker. Milestones ========== 1. Feature parity with CPython, understand all the language construct and behave absolutely compatible. Feature parity has been reached for CPython 2.6 and 2.7. We do not target any older CPython release. For CPython 3.2 to 3.5 it also has been reached. We do not target the older and practically unused CPython 3.1 and 3.0 releases. This milestone was reached. Dropping support for Python 2.6 and 3.2 is an option, should this prove to be any benefit. Currently it is not, as it extends the test coverage only. 2. Create the most efficient native code from this. This means to be fast with the basic Python object handling. This milestone was reached, although of course, micro optimizations to this are happening all the time. 3. Then do constant propagation, determine as many values and useful constraints as possible at compile time and create more efficient code. This milestone is considered almost reached. We continue to discover new things, but the infrastructure is there, and these are easy to add. 4. Type inference, detect and special case the handling of strings, integers, lists in the program. This milestone is considered in progress 5. Add interfacing to C code, so Nuitka can turn a ``ctypes`` binding into an efficient binding as written with C. This milestone is planned only. 6. Add hints module with a useful Python implementation that the compiler can use to learn about types from the programmer. This milestone is planned only. Version Numbers =============== For Nuitka we use a defensive version numbering system to indicate that it is not yet ready for everything. We have defined milestones and the version numbers should express which of these, we consider done. - So far: Before milestone 1, we used "0.1.x" version numbers. After reaching it, we used "0.2.x" version numbers. Before milestone 2 and 3, we used "0.3.x" version numbers. After almost reaching 3, and beginning with 4, we use "0.4.x" version numbers. Due to an interface change, "0.5.x" version numbers are being used. - Future: When we start to have sufficient amount of type inference in a stable release, that will be "0.6.x" version numbers. With ``ctypes`` bindings in a sufficient state it will be "0.7.x". - Final: We will then round it up and call it "Nuitka 1.0" when this works as expected for a bunch of people. The plan is to reach this goal during 2016. This is based on positive assumptions that may not hold up though. Of course, all of this may be subject to change. Current State ============= Nuitka top level works like this: - ``nuitka.tree.Building`` outputs node tree - ``nuitka.optimization`` enhances it as best as it can - ``nuitka.finalization`` marks the tree for code generation - ``nuitka.codegen.CodeGeneration`` orchestrates the creation of code snippets - ``nuitka.codegen.*Codes`` knows how specific code kinds are created - ``nuitka.MainControl`` keeps it all together This design is intended to last. Regarding Types, the state is: - Types are always ``PyObject *``, implicitly. - The only more specific use of type is "compile time constant", which can be used to predict some operations, conditions, etc. - Every operation is expected to have ``PyObject *`` as result, if it is not a constant, then we know nothing about it. For some interfaces, e.g. iteration, there are initial attempts at abstracting it. The limitation to only ``PyObject *`` will go away. Coding Rules ============ These rules should generally be adhered when working on Nuitka code. It's not library code and it's optimized for readability, and avoids all performance optimization for itself. Tool to format -------------- There is a tool ``misc/autoformat.py`` which is to apply automatic formatting to code as much as possible. Line Length ----------- No more than 120 characters. Screens are wider these days, but most of the code aims at keeping the lines below 80. Long lines are also a sign of writing incomprehensible code. Indentation ----------- No tabs, 4 spaces, no trailing white space. Identifiers ----------- Classes are camel case with leading upper case. Methods are with leading verb in lower case, but also camel case. Around braces there are no spaces, but after comma, there is spaces for better readability. Variables and arguments are lower case with ``_`` as a separator. .. code-block:: python class SomeClass: def doSomething(some_parameter): some_var = ("foo", "bar") Base classes that are abstract have their name end with ``Base``, so that a meta class can use that convention, and readers immediately know. Function calls use keyword argument preferably. These are slower in CPython, but more readable: .. code-block:: python return Generator.getSequenceCreationCode( sequence_kind = sequence_kind, element_identifiers = identifiers, context = context ) The ``=`` are all aligned to the longest parameter names without extra spaces for it. When the names don't add much value, sequential calls should be done, but ideally with one value per line: .. code-block:: python context.setLoopContinueTarget( handler_start_target, continue_name ) Here, ``setLoopContinueTarget`` will be so well known that the reader is expected to know the argument names and their meaning, but it would be still better to add them. Contractions should span across multiple lines for increased readability: .. code-block:: python result = [ "PyObject *decorator_%d" % (d + 1) for d in range(decorator_count) ] Module/Package Names -------------------- Normal modules are named in camel case with leading upper case, because their of role as singleton classes. The difference between a module and a class is small enough and in the source code they are also used similarly. For the packages, no real code is allowed in them and they must be lower case, like e.g. ``nuitka`` or ``codegen``. This is to distinguish them from the modules. Packages shall only be used to group packages. In ``nuitka.codegen`` the code generation packages are located, while the main interface is ``nuitka.codegen.CodeGeneration`` and may then use most of the entries as local imports. There is no code in packages. The use of a global package ``nuitka``, originally introduced by Nicolas, makes the packaging of Nuitka with ``distutils`` etc. easier and lowers the requirements on changes to the ``sys.path`` if necessary. .. note:: There are not yet enough packages inside Nuitka, feel free to propose changes as you see fit. Names of modules should be plurals if they contain classes. Example is ``Nodes`` contains ``Node`` class. Prefer list contractions over built-ins --------------------------------------- This concerns ``map``, ``filter``, and ``apply``. Usage of these built-ins is highly discouraged within Nuitka source code. Using them is considered worth a warning by "PyLint" e.g. "Used built-in function 'map'". We should use list contractions instead, because they are more readable. List contractions are a generalization for all of them. We love readability and with Nuitka as a compiler will there won't be any performance difference at all. There are cases where a list contraction are faster because you can avoid to make a function call. And there may be cases, where map is faster, if a function must be called. These calls can be very expensive in CPython, and if you introduce a function, just for ``map``, then it might be slower. But of course, Nuitka is the project to free us from what is faster and to allow us to use what is more readable, so whatever is faster, we don't care. We make all options equally fast and let people choose. For Nuitka the choice is list contractions as these are more easily changed and readable. Look at this code examples from Python: .. code-block:: python class A: def getX(self): return 1 x = property(getX) class B(A): def getX(self): return 2 A().x == 1 # True B().x == 1 # True (!) This pretty much is what makes properties bad. One would hope ``B().x`` to be ``2``, but instead it's not changed. Because of the way properties take the functions and not members, and because they then are not part of the class, they cannot be overloaded without re-declaring them. Overloading is then not at all obvious anymore. Now imagine having a setter and only overloading the getter. How to update the property easily? So, that's not likeable about them. And then we are also for clarity in these internal APIs too. Properties try and hide the fact that code needs to run and may do things. So lets not use them. For an external API you may exactly want to hide things, but internally that has no use, and in Nuitka, every API is internal API. One exception may be the ``hints`` module, which will gladly use such tricks for an easier write syntax. The "git flow" model ==================== * The flow was is used for releases and occasionally subsequent hot fixes. A few feature branches were used so far. It allows for quick delivery of fixes to both the stable and the development version, supported by a git plug-in, that can be installed via "apt-get install git-flow" on latest Debian Testing at least. * Stable (master branch) The stable version, is expected to pass all the tests at all times and is fully supported. As soon as bugs are discovered, they are fixed as hot fixes, and then merged to develop by the "git flow" automatically. * Development (develop branch) The future release, supposedly in almost ready for release state at nearly all times, but this is as strict. It is not officially supported, and may have problems and at times inconsistencies. Normally this branch is supposed to not be rebased. For severe problems it may be done though. * Factory (default feature branch) Code under construction. We publish commits there, that may not hold up in testing, and before it enters develop branch. Factory may have severe regressions frequently, and commits become **rebased all the time**, so do not base your patches on it, please prefer the ``develop`` branch for that, unless of course, it's about factory code itself. * Feature Branches On these long lived developments that extend for multiple release cycles or contain changes that break Nuitka temporarily. They need not be functional at all, and may lag behind main branches. Checking the Source =================== The static checking for errors is currently done with "PyLint". In the future, Nuitka itself will gain the ability to present its findings in a similar way, but this is not a priority, and we are not there yet. So, we currently use "PyLint" with options defined in a script. .. code-block:: sh ./misc/check-with-pylint --hide-todos Ideally the above command gives no warnings. This is currently the case. If you submit a patch, it would be good if you checked that it doesn't introduce new warnings, but that is not strictly required. it will happen before release, and that is considered enough. You probably are already aware of the beneficial effects. Running the Tests ================= This section describes how to run Nuitka tests. Running all Tests ----------------- The top level access to the tests is as simple as this: .. code-block:: sh ./tests/run-tests For fine grained control, it has the following options:: --skip-basic-tests The basic tests, execute these to check if Nuitka is healthy. Default is True. --skip-syntax-tests The syntax tests, execute these to check if Nuitka handles Syntax errors fine. Default is True. --skip-program-tests The programs tests, execute these to check if Nuitka handles programs, e.g. import recursions, etc. fine. Default is True. --skip-package-tests The packages tests, execute these to check if Nuitka handles packages, e.g. import recursions, etc. fine. Default is True. --skip-optimizations-tests The optimization tests, execute these to check if Nuitka does optimize certain constructs fully away. Default is True. --skip-standalone-tests The standalone tests, execute these to check if Nuitka standalone mode, e.g. not referring to outside, important 3rd library packages like PyQt fine. Default is True. --skip-reflection-test The reflection test compiles Nuitka with Nuitka, and then Nuitka with the compile Nuitka and compares the outputs. Default is True. --skip-cpython26-tests The standard CPython2.6 test suite. Execute this for all corner cases to be covered. With Python 2.7 this covers exception behavior quite well. Default is True. --skip-cpython27-tests The standard CPython2.7 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is True. --skip-cpython32-tests The standard CPython3.2 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is True. --skip-cpython33-tests The standard CPython3.3 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is True. --skip-cpython34-tests The standard CPython3.4 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is True. --no-python2.6 Do not use Python2.6 even if available on the system. Default is False. --no-python2.7 Do not use Python2.7 even if available on the system. Default is False. --no-python3.2 Do not use Python3.2 even if available on the system. Default is False. --no-python3.3 Do not use Python3.3 even if available on the system. Default is False. --no-python3.4 Do not use Python3.4 even if available on the system. Default is False. --coverage Make a coverage analysis, that does not really check. Default is False. You will only run the CPython test suites, if you have the submodules of the Nuitka git repository checked out. Otherwise, these will be skipped with a warning that they are not available. The policy is generally, that ``./test/run-tests`` running and passing all the tests on Linux and Windows shall be considered sufficient for a release, but of course, depending on changes going on, that might have to be expanded. Basic Tests ----------- You can run the "basic" tests like this: .. code-block:: sh ./tests/basics/run_all.py search These tests normally give sufficient coverage to assume that a change is correct, if these "basic" tests pass. The most important constructs and built-ins are excercised. To control the Python version used for testing, you can set the ``PYTHON`` environment variable to e.g. ``python3.2`` (can only be a path), or execute the ``run_all.py`` script directly with the intended version, it is portable across all supported Python versions. Syntax Tests ------------ Then there are "syntax" tests, i.e. language constructs that need to give a syntax error. It sometimes so happens that Nuitka must do this itself, because the ``ast.parse`` doesn't see the problem and raises no ``SyntaxError`` of its own. Using the ``global`` statement on a function argument is an example of this. These tests make sure that the errors of Nuitka and CPython are totally the same for this: .. code-block:: sh ./tests/syntax/run_all.py search Program Tests ------------- Then there are small "programs" tests, that e.g. exercise many kinds of import tricks and are designed to reveal problems with inter-module behavior. These can be run like this: .. code-block:: sh ./tests/programs/run_all.py search Compile Nuitka with Nuitka -------------------------- And there is the "compile itself" or "reflected" test. This test makes Nuitka compile itself and compare the resulting C++ when running compiled to non-compiled, which helps to find in-determinism. The test compiles every module of Nuitka into an extension module and all of Nuitka into a single binary. That test case also gives good coverage of the ``import`` mechanisms, because Nuitka uses a lot of packages and imports between them. .. code-block:: sh ./tests/reflected/compile_itself.py Design Descriptions =================== These should be a lot more and contain graphics from presentations given. It will be filled in, but not now. Nuitka Logo ----------- The logo was submitted by "dr. Equivalent". It's source is contained in ``misc/Logo`` where 3 variants of the logo in SVG are placed. * Symbol only (symbol) .. image:: images/Nuitka-Logo-Symbol.png * Text next to symbol (horizontal) .. image:: images/Nuitka-Logo-Horizontal.png * Text beneath symbol (vertical) .. image:: images/Nuitka-Logo-Vertical.png From these logos, PNG images, and "favicons", and are derived. The exact ImageMagick commands are in ``misc/make-doc.py``, but are not executed each time, the commands are also replicated here: .. code-block:: sh convert -background none misc/Logo/Nuitka-Logo-Symbol.svg images/Nuitka-Logo-Symbol.png convert -background none misc/Logo/Nuitka-Logo-Vertical.svg images/Nuitka-Logo-Vertical.png convert -background none misc/Logo/Nuitka-Logo-Horizontal.svg images/Nuitka-Logo-Horizontal.png optipng -o2 images/Nuitka-Logo-Symbol.png optipng -o2 images/Nuitka-Logo-Vertical.png optipng -o2 images/Nuitka-Logo-Horizontal.png Choice of the Target Language ----------------------------- * Choosing the target language, is an important decision * The portability of Nuitka is decided here * Other factors: * How difficult is it to generate the code? * Does the Python C-API have bindings? * Is that language known? * Does the language aid to find bugs? * These candidates were considered * C++03, C++11, Ada, C .. table:: Requirement to Language matrix: ===================== ===== ====== ========= ========= Requirement\\Language C C++03 C++11 Ada ===================== ===== ====== ========= ========= Portable Yes Yes No [1]_ Yes --------------------- ----- ------ --------- --------- Knowledge Yes Yes No [2]_ Yes --------------------- ----- ------ --------- --------- Python C-API Yes Yes Yes No [3]_ --------------------- ----- ------ --------- --------- Runtime checks No No No Yes [4]_ --------------------- ----- ------ --------- --------- Code Generation Tough Hard Easy Harder ===================== ===== ====== ========= ========= _`1`:: C++11 is not fully supported by all compilers. _`2`:: Not a whole lot of people have C++11 knowledge. My *only* C++11 code was that in Nuitka. _`3`:: The Python C-API for Ada would have to be created by us, possible just big project by itself. _`4`:: Run time checks exist only for Ada in that quality. I miss automatic ``CONSTRAINT_ERROR`` exceptions, for data structures with validity indicators, where in other languages, I need to check myself. The *decision for C* is ultimately: * for portability * for language knowledge * for control over created code. All of these are important advantages. For C++11 initially spoke easy code generation: * variadic templates * raw strings Yet, as it turns out, variadic templates do not help at all with evaluation order, so that code that used it, needed to be changed to generating instances of their code. And raw strings turned out to be not as perfect as one wants to be, and solving the problem with C++03 is feasible too, even if not pretty. For C++03 initially spoke less explicit code generation: * Destructors can ensure cleanups happen * Local objects could e.g. repair the stack frames For Ada would have spoken the time savings through run time checks, which would have shortened some debugging sessions quite some. But building the Python C-API bindings on our own, and potentially incorrectly, would have eaten that up. Later, it was found that using C++ for exceptions is tremendously inefficient, and must be avoided. In order to do this, a more C style code generation is needed, where even less things are done with C++, e.g. the cleanup of temporary variables inside a statement will be done manually instead. The current status is C-ish. That is, with very few classes remaining, the syntax used is C++ still, but we are approaching being pure C. Use of Scons internally ----------------------- Nuitka does not involve Scons in its user interface at all; Scons is purely used internally. Nuitka itself, being pure Python, will run without any build process just fine. Nuitka simply prepares ``.build`` folders with lots of files and tasks scons to execute the final build. .. note:: When we speak of "standalone" mode, this is handled outside of Scons, and after it, creating the ".dist" folder. This is done in ``nuitka.MainControl`` module. For interfacing to Scons, there is the module ``nuitka.build.SconsInterface`` that will support calling ``scons`` - potentially from an inline copy, mainly on Windows or when using source releases - and passing arguments to it. These arguments are passed as ``key=value``, and decoded in the scons file of Nuitka. The scons file is named ``SingleExe.scons`` for lack of better name. It's really wrong now, but we have yet to find a better name. It once expressed the intention to be used to create executables, but the same works for modules too, as in terms of building, and to Scons, things really are the same. The scons file supports operation in multiple modes for many things, and modules is just one of them. It runs outside of Nuitka process scope, even with a different Python version potentially, so all the information must be passed on the command line. What follows is the (lengthy) list of arguments that the scons file processes: * ``source_dir`` Where is the generated C++ source code. Scons will just compile everything it finds there. No list of files is passed. * ``nuitka_src`` Where do the include files and static C++ parts of Nuitka live. These provide e.g. the implementation of compiled function, generators, and other helper codes, this will point to where ``nuitka.build`` package lives normally. * ``result_base`` This is not a full name, merely the basename for the result to be produced, but with path included, and the suffix comes from module or executable mode. * ``module_mode`` Build a module instead of a program. * ``debug_mode`` Enable debug mode, which is a mode, where Nuitka tries to help identify errors in itself, and will generate less optimal code. This also asks for warnings, and makes the build fail if there are any. * ``python_debug`` Compile and link against Python debug mode, which does assertions and extra checks, to identify errors, mostly related to reference counting. May make the build fail, if no debug build library of CPython is available. On Windows it typically is not installed. * ``optimize_mode`` Optimization mode, enable as much as currently possible. This refers to building the binary. * ``full_compat_mode`` Full compatibility, even where it's stupid, i.e. do not provide information, even if available, in order to assert maximum compatibility. Intended to control level of compatibility to absurd. * ``experimental_mode`` Do things that are not yet accepted to be safe. * ``lto_mode`` Make use of link time optimization of g++ compiler if available and known good with the compiler in question. So far, this was not found to make major differences. * ``win_disable_console`` Windows subsystem mode: Disable console for windows builds. * ``unstriped_mode`` Unstriped mode: Do not remove debug symbols. * ``clang_mode`` Clang compiler mode, default on MacOS X and FreeBSD, optional on Linux. * ``mingw_mode`` MinGW compiler mode, optional and interesting to Windows only. * ``standalone_mode`` Building a standalone distribution for the binary. * ``show_scons`` Show scons mode, output information about Scons operation. This will e.g. also output the actual compiler used, output from compilation process, and generally debug information relating to be build process. * ``python_prefix`` Home of Python to be compiled against, used to locate headers and libraries. * ``target_arch`` Target architecture to build. * ``icon_path`` The icon to use for Windows programs if given. Locating Modules and Packages ----------------------------- The search for of modules used is driven by ``nuitka.Importing`` module. * From the module documentation The actual import of a module may already execute code that changes things. Imagine a module that does ``os.system()``, it would be done during compilation. People often connect to databases, and these kind of things, at import time. Therefore CPython exhibits the interfaces in an ``imp`` module in standard library, which one can use those to know ahead of time, what file import would load. For us unfortunately there is nothing in CPython that is easily accessible and gives us this functionality for packages and search paths exactly like CPython does, so we implement here a multi step search process that is compatible. This approach is much safer of course and there is no loss. To determine if it's from the standard library, one can abuse the attribute ``__file__`` of the ``os`` module like it's done in ``isStandardLibraryPath`` of this module. * Role This module serves the recursion into modules and analysis if a module is a known one. It will give warnings for modules attempted to be located, but not found. These warnings are controlled by a while list inside the module. The decision making and caching are located in the ``nuitka.tree`` package, in modules ``nuitka.tree.Recursion`` and ``nuitka.tree.ImportCache``. Each module is only imported once (cache), and we need to obey lots of user choices, e.g. to compile standard library or not. Hooking for module ``import`` process ------------------------------------- Currently, in generated code, for every ``import`` a normal ``__import__()`` built-in call is executed. The ``nuitka/build/static_src/ModuleUnfreezer.cpp`` file provides the implementation of a ``sys.meta_path`` hook. This meta path based importer allows us to have the Nuitka provided module imported even when imported by non-compiled code. .. note:: Of course it would make sense to compile time detect which module it is that is being imported and then to make it directly. At this time, we don't have this inter-module optimization yet, mid-term it should become easy to add. Supporting ``__class__`` of Python3 ----------------------------------- In Python3 the handling of ``__class__`` and ``super`` is different from Python2. It used to be a normal variable, and now the following things have changed. * The use of the ``super`` variable name triggers the addition of a closure variable ``__class__``, as can be witnessed by the following code: .. code-block:: python class X: def f1(self): print( locals() ) def f2(self): print( locals() ) super # Just using the name, not even calling it. x = X() x.f1() x.f2() .. code-block:: python {'self': <__main__.X object at 0x7f1773762390>} {'self': <__main__.X object at 0x7f1773762390>, '__class__': } * This value of ``__class__`` is also available in the child functions. * The parser marks up code objects usage of "super". It doesn't have to be a call, it can also be a local variable. If the ``super`` built-in is assigned to another name and that is used without arguments, it won't work unless ``__class__`` is taken as a closure variable. * As can be seen in the CPython3.2 code, the closure value is added after the class creation is performed. * It appears, that only functions locally defined to the class are affected and take the closure. This left Nuitka with the strange problem, of how to emulate that. The solution is this: * Under Python3, usage of ``__class__`` as a reference in a function body that is not a class dictionary creation, marks it up via ``markAsClassClosureTaker``. * Functions that are marked up, will be forced to reference variable to ``__class__``. .. note:: This one should be optimized away later if not used. Currently we have "no unused closure variable" detection, but it would cover it. * When recognizing calls to ``super`` without arguments, make the arguments into variable reference to ``__class__`` and potentially ``self`` (actually first argument name). * Class dictionary definitions are added. These are special direct function calls, ready to propagate also "bases" and "metaclass" values, which need to be calculated outside. The function bodies used for classes will automatically store ``__class__`` as a shared local variable, if anything uses it. And if it's not assigned by user code, it doesn't show up in the "locals()" used for dictionary creation. Existing ``__class__`` local variable values are in fact provided as closure, and overridden with the built class , but they should be used for the closure giving, before the class is finished. So ``__class__`` will be local variable of the class body, until the class is built, then it will be the ``__class__`` itself. Frame Stack ----------- In Python, every function, class, and module has a frame. It creates created when the scope it entered, and there is a stack of these at run time, which becomes visible in tracebacks in case of exceptions. The choice of Nuitka is to make this an explicit element of the node tree, that are as such subject to optimization. In cases, where they are not needed, they may be removed. Consider the following code. .. code-block:: python def f(): if someNotRaisingCall(): return somePotentiallyRaisingCall() else: return None In this example, the frame is not needed for all the code, because the condition checked wouldn't possibly raise at all. The idea is the make the frame guard explicit and then to reduce its scope whenever possible. So we start out with code like this one: .. code-block:: python def f(): with frame_guard( "f" ): if someNotRaisingCall(): return somePotentiallyRaisingCall() else: return None This is to be optimized into: .. code-block:: python def f(): if someNotRaisingCall(): with frame_guard( "f" ): return somePotentiallyRaisingCall() else: return None Notice how the frame guard taking is limited and may be avoided, or in best cases, it might be removed completely. Also this will play a role when in-lining function. The frame stack entry will then be automatically preserved without extra care. .. note:: In the actual code, ``nuitka.nodes.FrameNodes.StatementsFrame`` is represents this as a set of statements to be guarded by a frame presence. Parameter Parsing ----------------- The parsing of parameters is very convoluted in Python, and doing it in an compatible way is not that easy. This is a description of the required process, for easier overview. Input +++++ The input is an argument ``tuple`` (the type is fixed), which contains the positional arguments, and potentially an argument ``dict`` (type is fixed as well, but could also be ``NULL``, indicating that there are no keyword arguments. Keyword dictionary ++++++++++++++++++ The keyword argument dictionary is checked first. Anything in there, that cannot be associated, either raises an error, or is added to a potentially given star dict argument. So there are two major cases. * No star dict argument: Iterate over dictionary, and assign or raise errors. This check covers extra arguments given. * With star dict argument: Iterate over dictionary, and assign or raise errors. Interesting case for optimization are no positional arguments, then no check is needed, and the keyword argument dictionary could be used as the star argument. Should it change, a copy is needed though. What's noteworthy here, is that in comparison of the keywords, we can hope that they are the same value as we use. The interning of strings increases chances for non-compiled code to do that, esp. for short names. We then can do a simple ``is`` comparison and only fall back to real string `==` comparisons, after all of these failed. That means more code, but also a lot faster code in the positive case. Argument tuple ++++++++++++++ After this completed, the argument tuple is up for processing. The first thing it needs to do is to check if it's too many of them, and then to complain. For arguments in Python2, there is the possibility of them being nested, in which case they cannot be provided in the keyword dictionary, and merely should get picked from the argument tuple. Otherwise, the length of the argument tuple should be checked against its position and if possible, values should be taken from there. If it's already set (from the keyword dictionary), raise an error instead. Code Generation towards C ------------------------- Currently, Nuitka uses C++ as a glorified C, it will tend to use less and less actual C++ patterns. Nuitka needs to control the order of releases fully, and may have objects becoming illegal references by e.g. in-place operations. Exceptions ++++++++++ To handle and work with exceptions, every construct that can raise has either a ``bool`` or ``int`` return code or ``PyObject *`` with ``NULL`` return value. This is very much in line with that the Python C-API does. Every helper function that contains code that might raise needs these variables. After a failed call, our variant of ``PyErr_Fetch`` called ``FETCH_ERROR_OCCURRED`` must be used to catch the defined error, unless some quick exception cases apply. The quick exception means, ``NULL`` return from C-API without a set exception means means e.g. ``StopIteration``. As an optimization, functions that raise exceptions, but are known not to do so, for whatever reason, could only be asserted to not do so. Statement Temporary Variables +++++++++++++++++++++++++++++ For statements and larger constructs the context object track temporary values, that represent references. For some, these should be released at the end of the statement, or they represent a leak. The larger scope temporary variables, are tracked in the function or module context, where they are supposed to have explicit ``del`` to release their references. Local Variables Storage +++++++++++++++++++++++ Closure variables taken are to be released when the function object is later destroyed. For in-lined calls, variables are just passed, and it does not become an issue to release anything. For function exit, owned variables, local or shared to other functions, must be released. This cannot be a ``del`` operation, as it also involves setting a value, which would be wrong for shared variables (and wasteful to local variables, as that would be its last usage). Therefore we need a special operation that simply releases the reference to the cell or object variable. Exit Targets ++++++++++++ Each error or other exit releases statement temporary values and then executes a ``goto`` to the exit target. These targets need to be setup. The ``try``/``except`` will e.g. catch error exits. Other exits are ``continue``, ``break``, and ``return`` exits. They all work alike. Generally, the exits stack of with constructs that need to register themselves for some exit types. A loop e.g. registers the ``continue`` exit, and a contained ``try``/``finally`` too, so it can execute the final code should it be needed. Frames ++++++ Frames are containers for variable declarations and cleanups. As such, frames provide error exits and success exits, which remove the frame from the frame stack, and then proceed to the parent exit. Abortive Statements +++++++++++++++++++ The way ``try``/``finally`` is handled, copies of the ``finally`` block are made, and optimized independently for each abort method. The ones there are of course, ``return``, ``continue``, and ``break``, but also implicit and explicit ``raise`` of an exception. Code trailing an abortive statement can be discarded, and the control flow will follow these "exits". Constant Preparation -------------------- Early versions of Nuitka, created all constants for the whole program for ready access to generated code, before the program launches. It did so in a single file, but that approach didn't scale well. Problems were * Even unused code contributed to start-up time, this can become a lot for large programs, especially in standalone mode. * The massive amount of constant creation codes gave backend compilers a much harder time than necessary to analyse it all at once. The current approach is as follows. Code generation detects constants used in only one module, and declared ``static`` there, if the module is the only user, or ``extern`` if it is not. Some values or forced to be global, as they are used pre-main or in helpers. These ``extern`` values are globally created before anything is used. The ``static`` values are created when the module is loaded, i.e. something did import it. We trace used constants per module, and for nested ones, we also associate them. The global constants code is special in that it can only use ``static`` for nested values it exclusively uses, and has to export values that others use. Language Conversions to make things simpler ------------------------------------------- There are some cases, where the Python language has things that can in fact be expressed in a simpler or more general way, and where we choose to do that at either tree building or optimization time. The ``assert`` statement ++++++++++++++++++++++++ The ``assert`` statement is a special statement in Python, allowed by the syntax. It has two forms, with and without a second argument. The later is probably less known, as is the fact that raise statements can have multiple arguments too. The handling in Nuitka is: .. code-block:: python assert value # Absolutely the same as: if not value: raise AssertionError .. code-block:: python assert value, raise_arg # Absolutely the same as: if not value: raise AssertionError, raise_arg This makes assertions absolutely the same as a raise exception in a conditional statement. This transformation is performed at tree building already, so Nuitka never knows about ``assert`` as an element and standard optimizations apply. If e.g. the truth value of the assertion can be predicted, the conditional statement will have the branch statically executed or removed. The "comparison chain" expressions ++++++++++++++++++++++++++++++++++ In Nuitka we have the concept of an outline, and therefore we can make the following re-formulation instead: .. code-block:: python a < b() > c < d def _comparison_chain(): # So called "outline" function tmp_a = a tmp_b = b() tmp = tmp_a < tmp_b if not tmp: return tmp del tmp_a tmp_c = c tmp = tmp_b > tmp_c if not tmp: return tmp del tmp_b return tmp_c < d _comparison_chain() This transformation is performed at tree building already. The temporary variables keep the value for the use of the same expression. Only the last expression needs no temporary variable to keep it. What we got from this, is making the checks of the comparison chain explicit and comparisons in Nuitka to be internally always about two operands only. The ``execfile`` built-in +++++++++++++++++++++++++ Handling is: .. code-block:: python execfile(filename) # Basically the same as: exec compile(open(filename).read()), filename, "exec" .. note:: This allows optimizations to discover the file opening nature easily and apply file embedding or whatever we will have there one day. This transformation is performed when the ``execfile`` built-in is detected as such during optimization. Generator expressions with ``yield`` ++++++++++++++++++++++++++++++++++++ These are converted at tree building time into a generator function body that yields from the iterator given, which is the put into a for loop to iterate, created a lambda function of and then called with the first iterator. That eliminates the generator expression for this case. It's a bizarre construct and with this trick needs no special code generation. This is a complex example, demonstrating multiple cases of yield in unexpected cases: .. code-block:: python x = ((yield i) for i in (1,2) if not (yield)) # Basically the same as: def x(): for i in (1,2): if not (yield): yield(yield i) Function Decorators +++++++++++++++++++ When one learns about decorators, you see that: .. code-block:: python @decorator def function(): pass # Is basically the same as: def function(): pass function = decorator( function ) The only difference is the assignment to function. In the ``@decorator`` case, if the decorator fails with an exception, the name ``function`` is not assigned yet, but kept in a temporary variable. Therefore in Nuitka this assignment is more similar to that of a lambda expression, where the assignment to the name is only at the end, which also has the extra benefit of not treating real function and lambda functions any different. This removes the need for optimization and code generation to support decorators at all. And it should make the two variants optimize equally well. Functions nested arguments ++++++++++++++++++++++++++ Nested arguments are a Python2 only feature supported by Nuitka. Consider this example: .. code-block:: python def function(a,(b,c)): return a, b, c We solve this, by kind of wrapping the function with another function that does the unpacking and gives the errors that come from this: .. code-block:: python def function(a,".1"): def _tmp(a, b, c): return a, b, c a, b = ".1" return _tmp(a, b, c) The ``".1"`` is the variable name used by CPython internally, and actually works if you use keyword arguments via star dictionary. So this is very compatible and actually the right kind of re-formulation, but it removes the need from the code that does parameter parsing to deal with these. Obviously, there is no frame for ``_tmp``, just one for ``function`` and we do not use local variables, but temporary functions. In-place Assignments ++++++++++++++++++++ In-place assignments are re-formulated to an expression using temporary variables. These are not as much a reformulation of ``+=`` to ``+``, but instead one which makes it explicit that the assign target may change its value. .. code-block:: python a += b .. code-block:: python _tmp = a.__iadd__( b ) if a is not _tmp: a = _tmp Using ``__iadd__`` here to express that for the ``+``, the in-place variant ``iadd`` is used instead. The ``is`` check may be optimized away depending on type and value knowledge later on. Complex Assignments +++++++++++++++++++ Complex assignments are defined as those with multiple targets to assign from a single source and are re-formulated to such using a temporary variable and multiple simple assignments instead. .. code-block:: python a = b = c .. code-block:: python _tmp = c b = _tmp a = _tmp del _tmp This is possible, because in Python, if one assignment fails, it can just be interrupted, so in fact, they are sequential, and all that is required is to not calculate ``c`` twice, which the temporary variable takes care of. Were ``a`` a more complex expression, e.g. ``a.some_attribute`` that might raise an exception, ``b`` would still be assigned. Unpacking Assignments +++++++++++++++++++++ Unpacking assignments are re-formulated to use temporary variables as well. .. code-block:: python a, b.attr, c[ind] = d = e, f, g = h() Becomes this: .. code-block:: python _tmp = h() _iter1 = iter(_tmp) _tmp1 = unpack(_iter1, 3) _tmp2 = unpack(_iter1, 3) _tmp3 = unpack(_iter1, 3) unpack_check(_iter1) a = _tmp1 b.attr = _tmp2 c[ind] = _tmp3 d = _tmp _iter2 = iter(_tmp) _tmp4 = unpack(_iter2, 3) _tmp5 = unpack(_iter2, 3) _tmp6 = unpack(_iter2, 3) unpack_check(_iter1) e = _tmp4 f = _tmp5 g = _tmp6 That way, the unpacking is decomposed into multiple simple statements. It will be the job of optimizations to try and remove unnecessary unpacking, in case e.g. the source is a known tuple or list creation. .. note:: The ``unpack`` is a special node which is a form of ``next`` that will raise a ``ValueError`` when it cannot get the next value, rather than a ``StopIteration``. The message text contains the number of values to unpack, therefore the integer argument. .. note:: The ``unpack_check`` is a special node that raises a ``ValueError`` exception if the iterator is not finished, i.e. there are more values to unpack. Again the number of values to unpack is provided to construct the error message. With Statements +++++++++++++++ The ``with`` statements are re-formulated to use temporary variables as well. The taking and calling of ``__enter__`` and ``__exit__`` with arguments, is presented with standard operations instead. The promise to call ``__exit__`` is fulfilled by ``try``/``except`` clause instead. .. code-block:: python with some_context as x: something( x ) .. code-block:: python tmp_source = some_context # Actually it needs to be "special look-up" for Python2.7, so attribute # look-up won't be exactly what is there. tmp_exit = tmp_source.__exit__ # This one must be held for the whole with statement, it may be assigned # or not, in our example it is. If an exception occurs when calling # ``__enter__``, the ``__exit__`` should not be called. tmp_enter_result = tmp_source.__enter__() # Indicator variable to know if "tmp_exit" has been called. tmp_indicator = False try: # Now the assignment is to be done, if there is any name for the # manager given, this may become multiple assignment statements and # even unpacking ones. x = tmp_enter_result # Then the code of the "with" block. something( x ) except Exception: # Note: This part of the code must not set line numbers, which we # indicate with special source code references, which we call "internal". # Otherwise the line of the frame would get corrupted. tmp_indicator = True if not tmp_exit(*sys.exc_info()): raise finally: if not tmp_indicator # Call the exit if no exception occurred with all arguments # as "None". tmp_exit(None, None, None) .. note:: We don't refer really to ``sys.exc_info()`` at all, instead, we have fast references to the current exception type, value and trace, taken directly from the caught exception object on the C level. If we had the ability to optimize ``sys.exc_info()`` to do that, we could use the same transformation, but right now we don't have it. For Loops +++++++++ The ``for`` loops use normal assignments and handle the iterator that is implicit in the code explicitly. .. code-block:: python for x, y in iterable: if something( x ): break else: otherwise() This is roughly equivalent to the following code: .. code-block:: python _iter = iter(iterable) _no_break_indicator = False while 1: try: _tmp_value = next(_iter) except StopIteration: # Set the indicator that the else branch may be executed. _no_break_indicator = True # Optimization should be able to tell that the else branch is run # only once. break # Normal assignment re-formulation applies to this assignment of course. x, y = _tmp_value del _tmp_value if something(x): break if _no_break_indicator: otherwise() .. note:: The ``_iter`` temporary variable is of course also in a ``try/finally`` construct, to make sure it releases after its used. The ``x, y`` assignment is of course subject to unpacking re-formulation. The ``try``/``except`` is detected to allow to use a variant of ``next`` that does not raise an exception, but to be fast check about the ``NULL`` return from ``next`` built-in. So no actual exception handling is happening in this case. While Loops +++++++++++ Loops in Nuitka have no condition attached anymore, so while loops are re-formulated like this: .. code-block:: python while condition: something() .. code-block:: python while 1: if not condition: break something() This is to totally remove the specialization of loops, with the condition moved to the loop body in an initial conditional statement, which contains a ``break`` statement. That achieves, that only ``break`` statements exit the loop, and allow for optimization to remove always true loop conditions, without concerning code generation about it, and to detect such a situation, consider e.g. endless loops. .. note:: Loop analysis (not yet done) can then work on a reduced problem (which ``break`` statements are executed under what conditions) and is then automatically very general. The fact that the loop body may not be entered at all, is still optimized, but also in the general sense. Explicit breaks at the loop start and loop conditions are the same. Exception Handlers ++++++++++++++++++ Exception handlers in Python may assign the caught exception value to a variable in the handler definition. And the different handlers are represented as conditional checks on the result of comparison operations. .. code-block:: python try: block() except A as e: handlerA(e) except B as e: handlerB(e) else: handlerElse() .. code-block:: python try: block() except: # These are special nodes that access the exception, and don't really # use the "sys" module. tmp_exc_type = sys.exc_info()[0] tmp_exc_value = sys.exc_info()[1] # exception_matches is a comparison operation, also a special node. if exception_matches(tmp_exc_type, (A,)): e = tmp_exc_value handlerA(e) elif exception_matches(tmp_exc_type, (B,)): e = tmp_exc_value handlerB(e) else: handlerElse() For Python3, the assigned ``e`` variables get deleted at the end of the handler block. Should that value be already deleted, that ``del`` does not raise, therefore it's tolerant. This has to be done in any case, so for Python3 it is even more complex. .. code-block:: python try: block() except: # These are special nodes that access the exception, and don't really # use the "sys" module. tmp_exc_type = sys.exc_info()[0] tmp_exc_value = sys.exc_info()[1] # exception_matches is a comparison operation, also a special node. if exception_matches(tmp_exc_type, (A,)): try: e = tmp_exc_value handlerA(e) finally: del e elif exception_matches(tmp_exc_type, (B,)): try: e = tmp_exc_value handlerB(e) finally: del e else: handlerElse() Should there be no ``else:`` branch, a default re-raise statement is used instead. And of course, the values of the current exception type and value, both use special references, that access the C++ and don't go via ``sys.exc_info`` at all, nodes called ``CaughtExceptionTypeRef`` and ``CaughtExceptionValueRef``. This means, that the different handlers and their catching run time behavior are all explicit and reduced the branches. Statement ``try``/``except`` with ``else`` ++++++++++++++++++++++++++++++++++++++++++ Much like ``else`` branches of loops, an indicator variable is used to indicate the entry into any of the exception handlers. Therefore, the ``else`` becomes a real conditional statement in the node tree, checking the indicator variable and guarding the execution of the ``else`` branch. Class Creation (Python2) ++++++++++++++++++++++++ Classes in Python2 have a body that only serves to build the class dictionary and is a normal function otherwise. This is expressed with the following re-formulation: .. code-block:: python # in module "SomeModule" # ... class SomeClass(SomeBase, AnotherBase) """ This is the class documentation. """ some_member = 3 .. code-block:: python def _makeSomeClass: # The module name becomes a normal local variable too. __module__ = "SomeModule" # The doc string becomes a normal local variable. __doc__ = """ This is the class documentation. """ some_member = 3 return locals() # force locals to be a writable dictionary, will be optimized away, but # that property will stick. This is only to express, that locals(), where # used will be writable to. exec "" SomeClass = make_class("SomeClass", (SomeBase, AnotherBase), _makeSomeClass()) That is roughly the same, except that ``_makeSomeClass`` is *not* visible to its child functions when it comes to closure taking, which we cannot express in Python language at all. Therefore, class bodies are just special function bodies that create a dictionary for use in class creation. They don't really appear after the tree building stage anymore. The type inference will of course have to become able to understand ``make_class`` quite well, so it can recognize the created class again. Class Creation (Python3) ++++++++++++++++++++++++ In Python3, classes are a complicated way to write a function call, that can interact with its body. The body starts with a dictionary provided by the metaclass, so that is different, because it can ``__prepare__`` a non-empty locals for it, which is hidden away in "prepare_class_dict" below. What's noteworthy, is that this dictionary, could e.g. be a ``OrderDict``. I am not sure, what ``__prepare__`` is allowed to return. .. code-block:: python # in module "SomeModule" # ... class SomeClass(SomeBase, AnotherBase, metaclass = SomeMetaClass) """ This is the class documentation. """ some_member = 3 .. code-block:: python # Non-keyword arguments, need to be evaluated first. tmp_bases = ( SomeBase, AnotherBase ) # Keyword arguments go next, __metaclass__ is just one of them. In principle # we need to forward the others as well, but this is ignored for the sake of # brevity. tmp_metaclass = select_metaclass(tmp_bases, SomeMetaClass ) tmp_prepared = tmp_metaclass.__prepare__("SomeClass", tmp_bases) # The function that creates the class dictionary. Receives temporary variables # to work with. def _makeSomeClass: # This has effect, currently I don't know how to force that in Python3 # syntax, but we will use something that ensures it. locals() = tmp_prepared # The module name becomes a normal local variable too. __module__ = "SomeModule" # The doc string becomes a normal local variable. __doc__ = """ This is the class documentation. """ some_member = 3 # Create the class, share the potential closure variable "__class__" # with others. __class__ = tmp_metaclass("SomeClass", tmp_bases, locals()) return __class__ # Build and assign the class. SomeClass = _makeSomeClass() Generator Expressions +++++++++++++++++++++ There are re-formulated as functions. Generally they are turned into calls of function bodies with (potentially nested) for loops: .. code-block:: python gen = (x*2 for x in range(8) if cond()) .. code-block:: python def _gen_helper(__iterator): for x in __iterator: if cond(): yield x*2 gen = _gen_helper(range(8)) List Contractions +++++++++++++++++ The list contractions of Python2 are different from those of Python3, in that they don't actually do any closure variable taking, and that no function object ever exists. .. code-block:: python list_value = [x*2 for x in range(8) if cond()] .. code-block:: python def _listcontr_helper(__iterator): result = [] for x in __iterator: if cond(): result.append(x*2) return result list_value = _listcontr_helper(range(8)) The difference is that with Python3, the function "_listcontr_helper" is really there and named ````, whereas with Python2 the function is only an outline, so it can readily access the name space. Set Contractions ++++++++++++++++ The set contractions of Python2.7 are like list contractions in Python3, in that they produce an actual helper function: .. code-block:: python set_value = {x*2 for x in range(8) if cond()} .. code-block:: python def _setcontr_helper(__iterator): result = set() for x in __iterator: if cond(): result.add(x*2) return result set_value = _setcontr_helper( range(8) ) Dictionary Contractions +++++++++++++++++++++++ The dictionary contractions of are like list contractions in Python3, in that they produce an actual helper function: .. code-block:: python dict_value = {x: x*2 for x in range(8) if cond()} .. code-block:: python def _dictcontr_helper(__iterator): result = {} for x in __iterator: if cond(): result[x] = x*2 return result set_value = _dictcontr_helper( range(8) ) Boolean expressions ``and`` and ``or`` ++++++++++++++++++++++++++++++++++++++ The short circuit operators ``or`` and ``and`` tend to be only less general that the ``if``/``else`` expressions, but have dedicated nodes. We used to have a re-formulation towards those, but we now do these via dedicated nodes too. These new nodes, present the evaluation of the left value, checking for its truth value, and depending on it, to pick it, or use the right value. Simple Calls ++++++++++++ As seen below, even complex calls are simple calls. In simple calls of Python there is still some hidden semantic going on, that we expose. .. code-block:: python func(arg1, arg2, named1 = arg3, named2 = arg4) On the C-API level there is a tuple and dictionary built. This one is exposed: .. code-block:: python func(*(arg1, arg2), **{"named1" : arg3, "named2" : arg4}) A called function will access this tuple and the dictionary to parse the arguments, once that is also re-formulated (argument parsing), it can then lead to simple in-lining. This way calls only have 2 arguments with constant semantics, that fits perfectly with the C-API where it is the same, so it is actually easier for code generation. Although the above looks like a complex call, it actually is not. No checks are needed for the types of the star arguments and it's directly translated to ``PyObject_Call``. Complex Calls +++++++++++++ The call operator in Python allows to provide arguments in 4 forms. * Positional (or normal) arguments * Named (or keyword) arguments * Star list arguments * Star dictionary arguments The evaluation order is precisely that. An example would be: .. code-block:: python something(pos1, pos2, name1 = named1, name2 = named2, *star_list, **star_dict) The task here is that first all the arguments are evaluated, left to right, and then they are merged into only two, that is positional and named arguments only. for this, the star list argument and the star dictionary arguments, are merged with the positional and named arguments. What's peculiar, is that if both the star list and dictionary arguments are present, the merging is first done for star dictionary, and only after that for the star list argument. This makes a difference, because in case of an error, the star argument raises first. .. code-block:: python something(*1, **2) This raises "TypeError: something() argument after ** must be a mapping, not int" as opposed to a possibly more expected "TypeError: something() argument after * must be a sequence, not int." That doesn't matter much though, because the value is to be evaluated first anyway, and the check is only performed afterwards. If the star list argument calculation gives an error, this one is raised before checking the star dictionary argument. So, what we do, is we convert complex calls by the way of special functions, which handle the dirty work for us. The optimization is then tasked to do the difficult stuff. Our example becomes this: .. code-block:: python def _complex_call(called, pos, kw, star_list_arg, star_dict_arg): # Raises errors in case of duplicate arguments or tmp_star_dict not # being a mapping. tmp_merged_dict = merge_star_dict_arguments( called, tmp_named, mapping_check( called, tmp_star_dict ) ) # Raises an error if tmp_star_list is not a sequence. tmp_pos_merged = merge_pos_arguments( called, tmp_pos, tmp_star_list ) # On the C-API level, this is what it looks like. return called( *tmp_pos_merged, **tmp_merged_dict ) returned = _complex_call( called = something, pos = (pos1, pos2), named = { "name1" : named1, "name2" = named2 }, star_list_arg = star_list, star_list_arg = star_dict ) The call to ``_complex_call`` is be a direct function call with no parameter parsing overhead. And the call in its end, is a special call operation, which relates to the "PyObject_Call" C-API. Print statements ++++++++++++++++ The ``print`` statement exists only in Python2. It implicitly converts its arguments to strings before printing them. In order to make this accessible and compile time optimized, this is made visible in the node tree. .. code-block:: python print arg1, "1", 1 .. code-block:: python print str(arg1), "1", str(1) Only string objects are spared from the ``str`` built-in wrapper, because that would only cause noise in optimization stage. Additionally, each ``print`` may have a target, and multiple arguments, which we break down as well for dumber code generation. The target is evaluated first and should be a file, kept referenced throughout the whole print statement. .. code-block:: python print >>target_file, str(arg1), "1", str(1) This is being reformulated to: try: tmp_target = target_file print >>tmp_target, str(arg1), print >>tmp_target, "1", print >>tmp_target, str(1), print >>tmp_target finally: del tmp_target This allows code generation to not deal with arbitrary amount of arguments to ``print``. It also separates the newline indicator from the rest of things, which makes sense too, having it as a special node, as it's behavior with regards to soft-space is different of course. And finally, for ``print`` without a target, we still assume that a target was given, which would be ``sys.stdout`` in a rather hard-coded way (no variable look-ups involved). Call to ``dir`` without arguments --------------------------------- This expression is reformulated to ``locals().keys()`` for Python2, and ``list(locals.keys())``. Nodes that serve special purposes --------------------------------- Try statements ++++++++++++++ In Python, there is ``try``/``except`` and ``try``/``finally``. In Nuitka there is only a ``try``, which then has blocks to handle exceptions, ``continue``, or ``break``, or ``return``. There is no ``else`` to this node type. This is more low level and universal. Code for the different handlers can be different. User provided ``finally`` blocks become copied into the different handlers. Releases ++++++++ When a function exits, the local variables are to be released. The same applies to temporary variables used in re-formulations. These releases cause a reference to the object to the released, but no value change. They are typically the last use of the object in the function. The are similar to ``del``, but make no value change. For shared variables this effect is most visible. Side Effects ++++++++++++ When an exception is bound to occur, and this can be determined at compile time, Nuitka will not generate the code the leads to the exception, but directly just raise it. But not in all cases, this is the full thing. Consider this code: .. code-block:: python f(a(), 1 / 0) The second argument will create a ``ZeroDivisionError`` exception, but before that ``a()`` must be executed, but the call to ``f`` will never happen and no code is needed for that, but the name look-up must still succeed. This then leads to code that is internally like this: .. code-block:: python f(a(), raise ZeroDivisionError) which is then modeled as: .. code-block:: python side_effect(a(), f, raise ZeroDivisionError) where we can consider "side_effect" to be a function that returns the last expression. Of course, if this is not part of another expression, but close to statement level, side effects, can be converted to multiple statements simply. Another use case, is that the value of an expression can be predicted, but that the language still requires things to happen, consider this: .. code-block:: python a = len( ( f(), g() ) ) We can tell that ``a`` will be 2, but the call to ``f`` and ``g`` must still be performed, so it becomes: .. code-block:: python a = side_effects(f(), g(), 2) Modelling side effects explicitely has the advantage of recognizing them easily and allowing to drop the call to the tuple building and checking its length, only to release it. Caught Exception Type/Value References ++++++++++++++++++++++++++++++++++++++ When catching an exception, these are not directly put to ``sys.exc_info()``, but remain as mere C variables. From there, they can be accessed with these nodes, or if published then from the thread state. Hard Module Imports +++++++++++++++++++ These are module look-ups that don't depend on any local variable for the module to be looked up, but with hard-coded names. These may be the result of optimization gaining such level of certainty. Currently they are used to represent ``sys.stdout`` usage for ``print`` statements, but other usages will follow. Locals Dict Update Statement ++++++++++++++++++++++++++++ For the ``exec`` re-formulation, we apply an explicit sync back to locals as an explicit node. It helps us to tell the affected local variable traces that they might be affected. It represents the bit of ``exec`` in Python2, that treats ``None`` as the locals argument as an indication to copy back. Plan to replace "python-qt" for the GUI ======================================= Porting the tree inspector available with ``--dump-gui`` to "wxWindows" is very much welcome as the "python-qt4" bindings are severely under documented. Plan to add "ctypes" support ============================ Add interfacing to C code, so Nuitka can turn a ``ctypes`` binding into an efficient binding as if it were written manually with Python C-API or better. Goals/Allowances to the task ---------------------------- 1. Goal: Must not use any pre-existing C/C++ language file headers, only generate declarations in generated C code ourselves. We would rather write or use tools that turn an existing a C header to ``ctypes`` declarations if it needs to be, but not mix and use declarations from existing header code. 2. Allowance: May use ``ctypes`` module at compile time to ask things about ``ctypes`` and its types. 3. Goal: Should make use of ``ctypes``, to e.g. not hard code in Nuitka what ``ctypes.c_int()`` gives on the current platform, unless there is a specific benefit. 4. Allowance: Not all ``ctypes`` usages must be supported immediately. 5. Goal: Try and be as general as possible. For the compiler, ``ctypes`` support should be hidden behind a generic interface of some sort. Supporting ``math`` module should be the same thing. Type Inference - The Discussion ------------------------------- Main initial goal is to forward value knowledge. When you have ``a = b``, that means that a and b now "alias". And if you know the value of ``b`` you can assume to know the value of ``a``. This is called "aliasing". When assigning ``a`` to something new, that won't change ``b`` at all. But when an attribute is set, a method called of it, that impacts the actual value, referenced by both. We need to understand mutable vs. immutable though. .. code-block:: python a = 3 b = 3 b += 4 # a is not changed a = [ 3 ] b = a b += [ 4 ] # a is changed If we cannot tell, we must assume that ``a`` might be changed. It's either ``b`` or what ``a`` was before. If the type is not mutable, we can assume the aliasing to be broken up, and if it is, we can assume both to be the same value still. When that value is a compile time constant, we will want to push it forward, and we do that with "(Constant) Value Propagation", which is implemented already. We avoid too large constants, and we properly trace value assignments, but not yet aliases. In order to fully benefit from type knowledge, the new type system must be able to be fully friends with existing built-in types. The behavior of a type ``long``, ``str``, etc. ought to be implemented as far as possible with the built-in ``long``, ``str`` at compiled time as well. .. note:: This "use the real thing" concept extends beyond builtin types, e.g. ``ctypes.c_int()`` should also be used, but we must be aware of platform dependencies. The maximum size of ``ctypes.c_int`` values would be an example of that. Of course that may not be possible for everything. This approach has well proven itself with built-in functions already, where we use real built-ins where possible to make computations. We have the problem though that built-ins may have problems to execute everything with reasonable compile time cost. Another example, consider the following code: .. code-block:: python len("a" * 1000000000000) To predict this code, calculating it at compile time using constant operations, while feasible, puts an unacceptable burden on the compilation. Esp. we wouldn't want to produce such a huge constant and stream it, the C++ code would become too huge. So, we need to stop the ``*`` operator from being used at compile time and cope with reduced knowledge, already here: .. code-block:: python "a" * 10000000000000 Instead, we would probably say that for this expression: - The result is a ``str`` or ``PyStringObject``. - We know its length exactly, it's ``10000000000000``. - Can predict every of its elements when sub-scripted, sliced, etc., if need be, with a function we may create. Similar is true for this horrible thing: .. code-block:: python range( 10000000000000 ) So it's a rather general problem, this time we know: - The result is a ``list`` or ``PyListObject`` - We know its length exactly, ``10000000000000`` - Can predict every of its elements when index, sliced, etc., if need be, with a function. Again, we wouldn't want to create the list. Therefore Nuitka avoids executing these calculation, when they result in constants larger than a threshold of e.g. 256 elements. This concept has to be also applied to large integers and more CPU and memory traps. Now lets look at a more complete use case: .. code-block:: python for x in range( 10000000000000 ): doSomething() Looking at this example, one traditional way to look at it, would be to turn ``range`` into ``xrange``, and to note that ``x`` is unused. That would already perform better. But really better is to notice that ``range()`` generated values are not used at all, but only the length of the expression matters. And even if ``x`` were used, only the ability to predict the value from a function would be interesting, so we would use that computation function instead of having an iteration source. Being able to predict from a function could mean to have Python code to do it, as well as C code to do it. Then code for the loop can be generated without any CPython library usage at all. .. note:: Of course, it would only make sense where such calculations are "O(1)" complexity, i.e. do not require recursion like "n!" does. The other thing is that CPython appears to at - run time - take length hints from objects for some operations, and there it would help too, to track length of objects, and provide it, to outside code. Back to the original example: .. code-block:: python len("a" * 1000000000000) The theme here, is that when we can't compute all intermediate expressions, and we sure can't do it in the general case. But we can still, predict some of properties of an expression result, more or less. Here we have ``len`` to look at an argument that we know the size of. Great. We need to ask if there are any side effects, and if there are, we need to maintain them of course. This is already done by existing optimization if an operation generates an exception. .. note:: The optimization of ``len`` has been implemented and works for all kinds of container creation and ranges. Applying this to "ctypes" ------------------------- The not so specific problem to be solved to understand ``ctypes`` declarations is maybe as follows: .. code-block:: python import ctypes This leads to Nuitka in its tree to have an assignment from a ``__import__`` expression to the variable ``ctypes``. It can be predicted by default to be a module object, and even better, it can be known as ``ctypes`` from standard library with more or less certainty. See the section about "Importing". So that part is "easy", and it's what will happen. During optimization, when the module ``__import__`` expression is examined, it should say: - ``ctypes`` is a module - ``ctypes`` is from standard library (if it is, may not be true) - ``ctypes`` then has code behind it, called ``ModuleFriend`` that knows things about it attributes, that should be asked. The later is the generic interface, and the optimization should connect the two, of course via package and module full names. It will need a ``ModuleFriendRegistry``, from which it can be pulled. It would be nice if we can avoid ``ctypes`` to be loaded into Nuitka unless necessary, so these need to be more like a plug-in, loaded only if necessary, i.e. the user code actually uses ``ctypes``. Coming back to the original expression, it also contains an assignment expression, because it re-formulated to be more like this: .. code-block:: python ctypes = __import__("ctypes") The assigned to object, simply gets the type inferred propagated as part of an SSA form. Ideally, we could be sure that nothing in the program changes the variable, and therefore have only one version of that variable. For module variables, when the execution leaves the module to unknown code, or unclear code, it might change the variable. Therefore, likely we will often only assume that it could still be ``ctypes``, but also something else. Depending on how well we control module variable assignment, we can decide this more of less quickly. With "compiled modules" types, the expectation is that it's merely a quick C `==` comparison check. The module friend should offer code to allow a check if it applies, for uncertain cases. Then when we come to uses of it: .. code-block:: python ctypes.c_int() At this point, using SSA, we are more of less sure, that ``ctypes`` is at that point the module, and that we know what it's ``c_int`` attribute is, at compile time, and what it's call result is. We will use the module friend to help with that. It will attach knowledge about the result of that expression during the SSA collection process. This is more like a value forward propagation than anything else. In fact, constant propagation should only be the special case of it, and one design goal of Nuitka was always to cover these two cases with the same code. Excursion to Functions ---------------------- In order to decide what this means to functions and their call boundaries, if we propagate forward, how to handle this: .. code-block:: python def my_append(a, b): a.append(b) return a We would annotate that ``a`` is first a "unknown but defined parameter object", then later on something that definitely has an ``append`` attribute, when returned. Otherwise an exception occurs. The type of ``a`` changes to that after ``a.append`` look-up succeeds. It might be many kinds of an object, but e.g. it could have a higher probability of being a ``PyListObject``. And we would know it cannot be a ``PyStringObject``, as that one has no ``append`` method, and would have raised an exception therefore. .. note:: If classes, i.e. other types in the program, have an ``append`` attribute, it should play a role too, there needs to be a way to plug-in to this decisions. .. note:: On the other hand, types without ``append`` attribute can be eliminated. Therefore, functions through SSA provide an automatic analysis on their return state, or return value types, or a quick way to predict return value properties, based on input value knowledge. So this could work: .. code-block:: python b = my_append([], 3) assert b == [3] # Could be decided now Goal: The structure we use makes it easy to tell what ``my_append`` may be. So, there should be a means to ask it about call results with given type/value information. We need to be able to tell, if evaluating ``my_append`` makes sense with given parameters or not, if it does impact the return value. We should e.g. be able to make ``my_append`` tell, one or more of these: - Returns the first parameter value as return value (unless it raises an exception). - The return value has the same type as ``a`` (unless it raises an exception). - The return value has an ``append`` attribute. - The return value might be a ``list`` object. - The return value may not be a ``str`` object. - The function will raise if first argument has no ``append`` attribute. The exactness of statements may vary. But some things may be more interesting. If e.g. the aliasing of a parameter value to the return value is known exactly, then information about it need to all be given up, but some can survive. It would be nice, if ``my_append`` had sufficient information, so we could specialize with ``list`` and ``int`` from the parameters, and then e.g. know at least some things that it does in that case. Such specialization would have to be decided if it makes sense. In the alternative, it could be done for each variant anyway, as there won't be that many of them. Doing this "forward" analysis appears to be best suited for functions and therefore long term. We will try it that way. Excursion to Loops ------------------ .. code-block:: python a = 1 while 1: # think loop: here b = a + 1 a = b if cond(): break print a The handling of loops (both ``for`` and ``while`` are re-formulated to this kind of loops with ``break`` statements) has its own problem. The loop start and may have an assumption from before it started, that ``a`` is constant, but that is only true for the first iteration. So, we can't pass knowledge from outside loop forward directly into the for loop body. So the collection for loops needs to be two pass for loops. First, to collect assignments, and merge these into the start state, before entering the loop body. The need to make two passes is special to loops. For a start, it is done like this. At loop entry, all knowledge is removed about everything assigned or changed in the loop. From that basis, the ``break`` exits are analysed, and merged. Then, coming back, ``continue`` exits of the loop resulting from this, can replace the initial wide reaching removal of knowledge with a far smaller one. Merging this will state before loop, we got close to no degradation of knowledge, except what's necessary. Excursion to Conditions ----------------------- .. code-block:: python if cond: x = 1 else: x = 2 b = x < 3 The above code contains a condition, and these have the problem, that when exiting the conditional block, a merge must be done, of the ``x`` versions. It could be either one. The merge may trace the condition under which a choice is taken. That way, we could decide pairs of traces under the same condition. These merges of SSA variable "versions", represent alternative values. They pose difficulties, and might have to be reduced to commonality. In the above example, the ``<`` operator will have to check for each version, and then to decide that both indeed give the same result. The constraint collection tracks variable changes in conditional branches, and then merges the existing state at conditional statement exits. .. note:: A branch is considered "exiting" if it is not abortive. Should it end in a ``raise``, ``break``, ``continue``, or ``return``, there is no need to merge that branch, as execution of that branch is terminated. Should both branches be abortive, that makes things really simple, as there is no need to even continue. Should only one branch exist, but be abortive, then no merge is needed, and the collection can assume after the conditional statement, that the branch was not taken, and continue. When exiting both the branches, these branches must both be merged, with their new information. In the above case: - The "yes" branch knows variable ``x`` is an ``int`` of constant value ``1`` - The "no" branch knows variable ``x`` is an ``int`` of constant value ``2`` That might be collapsed to: - The variable ``x`` is an integer of value in ``(1,2)`` Given this, we then should be able to precompute the value of this: .. code-block:: python b = x < 3 The comparison operator can therefore decide and tell: - The variable ``b`` is a boolean of constant value ``True``. Were it unable to decide, it would still be able to say: - The variable ``b`` is a boolean. For conditional statements optimization, it's also noteworthy, that the condition is known to pass or not pass the truth check, inside branches, and in the case of non-exiting single branches, after the statement it's not true. We may want to take advantage of it. Consider e.g. .. code-block:: python if type( a ) is list: a.append( x ) else: a += ( x, ) In this case, the knowledge that ``a`` is a list, could be used to generate better code and with the definite knowledge that ``a`` is of type list. With that knowledge the ``append`` attribute call will become the ``list`` built-in type operation. Excursion to ``return`` statements ---------------------------------- The ``return`` statement (like ``break``, ``continue``, ``raise``) is "aborting" to control flow. It is always the last statement of inspected block. When there statements to follow it, optimization will remove it as "dead code". If all branches of a conditional statement are "aborting", the statement is decided "aborting" too. If a loop doesn't abort with a break, it should be considered "aborting" too. Excursion to ``yield`` expressions ---------------------------------- The ``yield`` expression can be treated like a normal function call, and as such invalidates some known constraints just as much as they do. It executes outside code for an unknown amount of time, and then returns, with little about the outside world known anymore, if it's accessible from there. Mixed Types ----------- Consider the following inside a function or module: .. code-block:: python if cond is not None: a = [x for x in something() if cond(x)] else: a = () A programmer will often not make a difference between ``list`` and ``tuple``. In fact, using a ``tuple`` is a good way to express that something won't be changed later, as these are mutable. .. note:: Better programming style, would be to use this: .. code-block:: python if cond is not None: a = tuple(x for x in something() if cond(x)) else: a = () People don't do it, because they dislike the performance hit encountered by the generator expression being used to initialize the tuple. But it would be more consistent, and so Nuitka is using it, and of course one day Nuitka ought to be able to make no difference in performance for it. To Nuitka though this means, that if ``cond`` is not predictable, after the conditional statement we may either have a ``tuple`` or a ``list`` type object in ``a``. In order to represent that without resorting to "I know nothing about it", we need a kind of ``min``/``max`` operating mechanism that is capable of say what is common with multiple alternative values. .. note:: At this time, we don't really have that mechanism to find the commonality between values. Back to "ctypes" ---------------- .. code-block:: python v = ctypes.c_int() Coming back to this example, we needed to propagate ``ctypes``, then we can propagate "something" from ``ctypes.int`` and then known what this gives with a call and no arguments, so the walk of the nodes, and diverse operations should be addressed by a module friend. In case a module friend doesn't know what to do, it needs to say so by default. This should be enforced by a base class and give a warning or note. Now to the interface -------------------- The following is the intended interface: - Iteration with node methods ``computeStatement`` and ``computeExpression``. These traverse modules and functions (i.e. scopes) and visit everything in the order that Python executes it. The visiting object is ``ConstraintCollection`` and pass forward. Some node types, e.g. ``StatementConditional`` new create branch constraint collections and handle the SSA merging at exit. - Replacing nodes during the visit. Both ``computeStatement`` and ``computeExpression`` are tasked to return potential replacements of themselves, together with "tags" (meaningless now), and a "message", used for verbose tracing. The replacement node of ``+`` operator, may e.g. be the pre-computed constant result, wrapped in side effects of the node, or the expression raised, again wrapped in side effects. - Assignments and references affect SSA. The SSA tree is initialized every time a scope is visited. Then during traversal, traces are built up. Every assignment and merge starts a new trace for that matter. References to a given variable version are traced that way. - Value escapes are traced too. When an operation hands over a value to outside code, it indicates so to the constraint collection. This is for it to know, when e.g. a constant value, might be mutated meanwhile. - Nodes can be queried about their properties. The node base classes offers methods that allow to check if certain operations are supported or not. These can always return ``True`` (yes), ``False`` (no), and ``None`` (cannot decide). In the case of the later, optimizations may not be able do much about it. Lets call these values "tri-state". The default implementation will be very pessimistic. Specific node types may then declare, that they e.g. have no side effects, do no raise, have a know truth value, have a known iteration length, can predict their iteration values, etc. - Nodes are linked to certain states. During the collect, a variable reference, is linked to a certain trace state, and that can be used by parent operations. .. code-block:: python a = 1 b = a + a In this example, the references to "a", can look-up the "1" in the trace, and base their responses to "+" on it. It will ask "isCompileTimeConstant()" and both nodes will respond "True", then "getCompileTimeConstant()" will return "1", which will be computed. Then "extractSideEffects()" will return "()" and therefore, the result "2" will not be wrapped. - Class for module import expression ``ExpressionImportModule``. This one just knows that something is imported, but not how or what it is assigned to. It will be able in a recursive compile, to provide the module as an assignment source, or the module variables or submodules as an attribute source when referenced from a variable trace or in an expression. - Base class for module friend ``ModuleFriendBase``. This is intended to provide something to overload, which e.g. can handle ``math`` in a better way. - Module ``ModuleFriendRegistry`` Provides a register function with ``name`` and instances of ``ValueFriendModuleBase`` to be registered. Recursed to modules should integrate with that too. The registry could well be done with a metaclass approach. - The module friends should each live in a module of their own. With a naming policy to be determined. These modules should add themselves via above mechanism to ``ModuleFriendRegistry`` and all shall be imported and register. Importing of e.g. ``ctypes`` should be delayed to when the friend is actually used. A meta class should aid this task. The delay will avoid unnecessary blot of the compiler at run time, if no such module is used. For "qt" and other complex stuff, this will be a must. - The walk should initially be single pass, and not maintain history. Instead optimization that needs to look at multiple things, e.g. "unused assignment", will look at the whole SSA collection afterwards. Discussing with examples ------------------------ The following examples: .. code-block:: python # Assignment, the source decides the type of the assigned expression a = b # Operator "attribute look-up", the looked up expression "ctypes" decides # via its trace. ctypes.c_int # Call operator, the called expressions decides with help of arguments, # which have been walked, before the call itself. called_expression_of_any_complexity() # import gives a module any case, and the "ModuleRegistry" may say more. import ctypes # From import need not give module, "x" decides what it is. from x import y # Operations are decided by arguments, and CPython operator rules between # argument states. a + b The optimization is mostly performed by walking of the tree and performing constraint collection. When it encounters assignments and references to them, it considers current state of traces and uses it for ``computeExpression``. .. note:: Assignments to attributes, indexes, slices, etc. will also need to follow the flow of ``append``, so it cannot escape attention that a list may be modified. Usages of ``append`` that we cannot be sure about, must be traced to exist, and disallow the list to be considered known value again. Code Generation Impact ---------------------- Right now, code generation assumes that everything is a ``PyObject *``, i.e. a Python object, and does not take knowledge of ``int`` or other types into consideration at all, and it should remain like that for some time to come. Instead, ``ctypes`` value friend will be asked give ``Identifiers``, like other codes do too. And these need to be able to convert themselves to objects to work with the other things. But Code Generation should no longer require that operations must be performed on that level. Imagine e.g. the following calls: .. code-block:: python c_call( other_c_call() ) Value returned by "other_c_call()" of say ``c_int`` type, should be possible to be fed directly into another call. That should be easy by having a ``asIntC()`` in the identifier classes, which the ``ctypes`` Identifiers handle without conversions. Code Generation should one day also become able to tell that all uses of a variable have only ``c_int`` value, and use ``int`` instead of ``PyObjectLocalVariable`` more or less directly. We could consider ``PyIntLocalVariable`` of similar complexity as ``int`` after the C++ compiler performed its in-lining. Such decisions would be prepared by finalization, which then would track the history of values throughout a function or part of it. Initial Implementation ---------------------- The basic interface will be added to *all* expressions and a node may override it, potentially using constraint collection state, as attached during "computeExpression". Goal 1 ++++++ Initially most things will only be able to give up on about anything. And it will be little more than a tool to do simple look-ups in a general form. It will then be the first goal to turn the following code into better performing one: .. code-block:: python a = 3 b = 7 c = a / b return c to: .. code-block:: python a = 3 b = 7 c = 3 / 7 return c and then: .. code-block:: python a = 3 b = 7 c = 0 return c and then: .. code-block:: python a = 3 b = 7 c = 0 return 0 This depends on SSA form to be able to tell us the values of ``a``, ``b``, and ``c`` to be written to by constants, which can be forward propagated at no cost. Goal 2 ++++++ The assignments to ``a``, ``b``, and ``c`` shall all become prey to "unused" assignment analysis in the next step. They are all only assigned to, and the assignment source has no effect, so they can be simply dropped. .. code-block:: python return 0 In the SSA form, these are then assignments without references. These assignments, can be removed if the assignment source has no side effect. Or at least they could be made "anonymous", i.e. use a temporary variable instead of the named one. That would have to take into account though, that the old version still needs a release. The most general form would first merely remove assignments that have no impact, and leave the value as a side effect, so we arrive at this first: .. code-block:: python 3 7 0 return 0 When applying the removal of expression only statements without effect, this gives us: .. code-block:: python return 0 which is the perfect result. Doing it in one step would only be an optimization. In order to be able to manipulate nodes related to a variable trace, we need to attach the nodes that did it. Consider this: .. code-block:: python if cond(): x = 1 elif other(): x = 3 # Not using "x". return 0 In the above case, the merge of the value friends, should say that ``x`` may be undefined, or one of ``1`` or ``3``, but since ``x`` is not used, apply the "dead value" trick to each branch. The removal of the "merge" of the 3 ``x`` versions, should exhibit that the other versions are also only assigned to, and can be removed. These merges of course appear as usages of the ``x`` versions. Goal 3 ++++++ Then third goal is to understand all of this: .. code-block:: python def f(): a = [] print a for i in range(1000): print a a.append( i ) return len( a ) .. note:: There are many operations in this, and all of them should be properly handled, or at least ignored in safe way. The first goal code gave us that the ``list`` has an annotation from the assignment of ``[]`` and that it will be copied to ``a`` until the for loop in encountered. Then it must be removed, because the ``for`` loop somehow says so. The ``a`` may change its value, due to the unknown attribute look-up of it already, not even the call. The for loop must be able to say "may change value" due to that, of course also due to the call of that attribute too. The code should therefore become equivalent to: .. code-block:: python def f(): a = [] print [] for i in range(1000): print a a.append( i ) return len( a ) But no other changes must occur, especially not to the "return" statement, it must not assume "a" to be constant "[]" but an unknown "a" instead. With that, we would handle this code correctly and have some form constant value propagation in place, handle loops at least correctly, and while it is not much, it is important demonstration of the concept. Goal 4 ++++++ The fourth goal is to understand the following: .. code-block:: python def f(cond): y = 3 if cond: x = 1 else: x = 2 return x < y In this we have a branch, and we will be required to keep track of both the branches separately, and then to merge with the original knowledge. After the conditional statement we will know that "x" is an "int" with possible values in "(1,2)", which can be used to predict that the return value is always "True". The forth goal will therefore be that the "ValueFriendConstantList" knows that append changes "a" value, but it remains a list, and that the size increases by one. It should provide an other value friend "ValueFriendList" for "a" due to that. In order to do that, such code must be considered: .. code-block:: python a = [] a.append( 1 ) a.append( 2 ) print len( a ) It will be good, if ``len`` still knows that "a" is a list, but not the constant list anymore. From here, work should be done to demonstrate the correctness of it with the basic tests applied to discover undetected issues. Fifth and optional goal: Extra bonus points for being able to track and predict "append" to update the constant list in a known way. Using "list.append" that should be done and lead to a constant result of "len" being used. The sixth and challenging goal will be to make the code generation be impacted by the value friends types. It should have a knowledge that "PyList_Append" does the job of append and use "PyList_Size" for "len". The "ValueFriends" should aid the code generation too. Last and right now optional goal will be to make "range" have a value friend, that can interact with iteration of the for loop, and "append" of the "list" value friend, so it knows it's possible to iterate 5000 times, and that "a" has then after the "loop" this size, so "len( a )" could be predicted. For during the loop, about a the range of its length should be known to be less than 5000. That would make the code of goal 2 completely analyzed at compile time. Limitations for now ------------------- - Aim only for limited examples. For ``ctypes`` that means to compile time evaluate: .. code-block:: python print ctypes.c_int( 17 ) + ctypes.c_long( 19 ) Later then call to "libc" or something else universally available, e.g. "strlen()" or "strcmp()" from full blown declarations of the callable. - We won't have the ability to test that optimization are actually performed, we will check the generated code by hand. With time, we will add XML based checks with "xpath" queries, expressed as hints, but that is some work that will be based on this work here. The "hints" fits into the "ValueFriends" concept nicely or so the hope is. - No inter-function optimization functions yet Of course, once in place, it will make the ``ctypes`` annotation even more usable. Using ``ctypes`` objects inside functions, while creating them on the module level, is therefore not immediately going to work. - No loops yet Loops break value propagation. For the ``ctypes`` use case, this won't be much of a difficulty. Due to the strangeness of the task, it should be tackled later on at a higher priority. - Not too much. Try and get simple things to work now. We shall see, what kinds of constraints really make the most sense. Understanding ``list`` subscript/slice values e.g. is not strictly useful for much code and should not block us. .. note:: This design is not likely to be the final one. .. raw:: pdf PageBreak Idea Bin ======== This an area where to drop random ideas on our minds, to later sort it out, and out it into action, which could be code changes, plan changes, issues created, etc. * Make "SELECT_METACLASS" meta class selection transparent. Looking at the "SELECT_METACLASS" it should become an anonymous helper function. In that way, the optimization process can remove choices at compile time, and e.g. in-line the effect of a meta class, if it is known. This of course makes most sense, if we have the optimizations in place that will allow this to actually happen. * Keeping track of iterations The constraint collection trace should become the place, where variables or values track their use state. The iterator should keep track of the "next()" calls made to it, so it can tell which value to given in that case. That would solve the "iteration of constants" as a side effect and it would allow to tell that they can be removed. That would mean to go back in the tree and modify it long after. .. code-block:: python a = iter( ( 2, 3 ) ) b = next( a ) c = next( a ) del a It would be sweet if we could recognize that: .. code-block:: python a = iter( ( 2, 3 ) ) b = side_effect( next( a ), 2 ) c = side_effect( next( a ), 3 ) del a That trivially becomes: .. code-block:: python a = iter( ( 2, 3 ) ) next( a ) b = 2 next( a ) c = 3 del a When the "del a" is examined at the end of scope, or due to another assignment to the same variable, ending the trace, we would have to consider of the "next" uses, and retrofit the information that they had no effect. .. code-block:: python a = iter( ( 2, 3 ) ) b = 2 b = 3 del a * Aliasing Each time an assignment is made, an alias is created. A value may have different names. .. code-block:: python a = iter( range(9 )) b = a c = next(b) d = next(a) If we fail to detect the aliasing nature, we will calculate "d" wrongly. We may incref and decref values to trace it. Aliasing is automatically traced already in SSA form. The "b" is assigned to version of "a". So, that should allow to replace it with this: .. code-block:: python a = iter( range(9 )) c = next(a) d = next(a) Which then will be properly handled. * Shelve for caching If we ever came to the conclusion to want and cache complex results of analysis, we could do so with the shelve module. We would have to implement ``__deepcopy__`` and then could store in there optimized node structures from start values after parsing. * Tail recursion optimization. Functions that return the results of calls, can be optimized. The Stackless Python does it already. * Integrate with "upx" compression. Calling "upx" on the created binaries, would be easy. * In-lining constant "exec" and "eval". It should be possible to re-formulate at least cases without "locals" or "globals" given. .. code-block:: python def f(): a = 1 b = 2 exec( """a+=b;c=1""" ) return a, c Should become this here: .. code-block:: python def f(): a = 1 b = 2 a+=b # c=1 # MaybeLocalVariables for everything except known local ones. return a, c If this holds up, inlining ``exec`` should be relatively easy. * Original and overloaded built-ins This is about making things visible in the node tree. In Nuitka things that are not visible in the node tree tend to be wrong. We already pushed around information to the node tree a lot. Later versions, Nuitka will become able to determine it has to be the original built-in at compile time, then a condition that checks will be optimized away, together with the slow path. Or the other path, if it won't be. Then it will be optimized away, or if doubt exists, it will be correct. That is the goal. Right now, the change would mean to effectively disable all built-in call optimization, which is why we don't immediately do it. Making the compatible version, will also require a full listing of all built-ins, which is typing work merely, but not needed now. And a way to stop built-in optimization from optimizing built-in calls that it used in a wrap. Probably just some flag to indicate it when it visits it to skip it. That's for later. But should we have that both, I figure, we could not raise a ``RuntimeError`` error, but just do the correct thing, in all cases. An earlier step may raise ``RuntimeError`` error, when built-in module values are written to, that we don't support. .. raw:: pdf PageBreak * SSA form for Nuitka nodes * Assignments collect a counter from the variable, which becomes the variable version. This happens during tree building phase. * References need to back track to the last assignment on their path, which may be a merge. Constraint collection can do that. * Data structures Every constraint collection has these: * variable_actives Dictionary, where per "variable" the currently used version is. Used to track situations changes in branches. This is the main input for merge process. * variable_traces Dictionary, where "variable" and "version" form the key. The values are objects with or without an assignment, and a list of usages, which starts out empty. These objects have usages appended to them. In "onVariableSet", a new version is allocated, which gives a new object for the dictionary, with an empty usages list, because each write starts a new version. In "onVariableUsage" the version is detected from the current version. It may be not set yet, which means, it's a read of an undefined value (local variable, not a parameter name), or unknown in case of global variable. These objects may be told that their value has escaped. This should influence the value friend they attached to the initial assignment. Each usage may have a current value friend state that is different. * When merging branches of conditional statements, the merge shall apply as follows. * Branches have their own collection, with deviating sets of "variable_actives". These are children of an outer collections * Case a) One branch only. For that branch a collection is performed. As usual new assignments generate a new version making it "active", references then related to these "active" versions. Then, when the branch is merged, for all "active" variables, it is considered, if that is a change related to before the branch. If it's not the same, a merge trace with the branch condition is created with the one active in the collection before that statement. * Case b) Two branches. When there are two branches, they both as are treated as above, except for the merge. When merging, a difference in active variables between the two branches creates the merge trace. .. note:: For conditional expressions, there are always only two branches. Even if you think you have more than one branch, you do not. It's always nested branches, already when it comes out of the parser. * Trace structure * Initial write of the version There may be a initial write for each version. It can only occur at the start of it, but not later, and there is only one. The "value friend" of it. * Merge of other one or two other versions One could be empty, i.e. the variable would not be assigned. This is kind of the initial write, and the merge references one or multiple "value friends", which are optional. * Bunch of read usages. They may allow escape of the value or not. When they do, it's a change. The value friend must be informed of it. If it's a real escape, usage is not known. If it's merely an alias, e.g. the value is now in another variable trace, they could be linked. Otherwise the "value friend" must be demoted immediately to one that gives more vague information. This should be reflected in a class "VariableTrace". * Recursion checks are expensive. If the "caller" or the "called" can declare that it cannot be called by itself, we could leave it out. TODO: Are they really that expensive? Unnecessary yes, but expensive may not be true. .. raw:: pdf PageBreak Updates for this Manual ======================= This document is written in REST. That is an ASCII format which is readable as ASCII, but used to generate PDF or HTML documents. You will find the current source under: http://nuitka.net/gitweb/?p=Nuitka.git;a=blob_plain;f=Developer_Manual.rst And the current PDF under: http://nuitka.net/doc/Developer_Manual.pdf Nuitka-0.5.18.1/PKG-INFO0000644000175000017500000000050112651072347014531 0ustar hayenhayen00000000000000Metadata-Version: 1.0 Name: Nuitka Version: 0.5.18.1 Summary: Python compiler with full language support and CPython compatibility Home-page: http://nuitka.net Author: Kay Hayen Author-email: Kay.Hayen@gmail.com License: Apache License, Version 2.0 Description: UNKNOWN Keywords: compiler,python,nuitka Platform: UNKNOWN Nuitka-0.5.18.1/MANIFEST.in0000644000175000017500000000211012651070406015162 0ustar hayenhayen00000000000000include LICENSE.txt include MANIFEST.in include README.rst README.pdf include Changelog.rst Changelog.pdf include Developer_Manual.rst Developer_Manual.pdf include doc/*.1 include bin/compare_with_cpython include bin/compare_with_xml include misc/*.sh include misc/*.bat include misc/check-with-pylint include tests/run-tests # Logo with source include misc/Logo/Nuitka-Logo-Symbol.svg include misc/Logo/Nuitka-Logo-Vertical.svg include misc/Logo/Nuitka-Logo-Horizontal.svg include images/Nuitka-Logo-Symbol.png include images/Nuitka-Logo-Vertical.png include images/Nuitka-Logo-Horizontal.png recursive-include lib *.py # Core tests are included along with Nuitka itself. recursive-include tests/basics *.py recursive-include tests/syntax *.py recursive-include tests/packages *.py recursive-include tests/programs *.py recursive-include tests/optimizations *.py recursive-include tests/standalone *.py recursive-include tests/reflected *.py include tests/test_common.py # Bnechmarks are included too, but will be removed for Debian. recursive-include tests/benchmarks *.py LICENSE README Nuitka-0.5.18.1/README.rst0000644000175000017500000007122412651072132015125 0ustar hayenhayen00000000000000Nuitka User Manual ~~~~~~~~~~~~~~~~~~ .. image:: images/Nuitka-Logo-Symbol.png .. contents:: .. raw:: pdf PageBreak oneColumn SetPageCounter 1 Overview ======== This document is the recommended first read if you are interested in using Nuitka, understand its use cases, check what you can expect, license, requirements, credits, etc. Nuitka is **the** Python compiler. It is a seamless replacement or extension to the Python interpreter and compiles **every** construct that CPython 2.6, 2.7, 3.2, 3.3, 3.4, and 3.5 have. It then executed uncompiled code, and compiled code together in an extremely compatible manner. You can use all Python library modules or and all extension modules freely. It translates the Python into a C level program that then uses "libpython" to execute in the same way as CPython does. All optimization is aimed at avoiding overhead, where it's unnecessary. None is aimed at removing compatibility, although there is an "improved" mode, where not every bug of standard Python is emulated, e.g. more complete error messages are given. Usage ===== Requirements ------------ - C++ Compiler: You need a compiler with support for C++03 [#]_ Currently this means, you need to use either of these compilers: * GNU g++ compiler of at least version 4.4 * The clang compiler on MacOS X or FreeBSD, based on LLVM version 3.2 or higher. * The MinGW64 [#]_ compiler on Windows * Visual Studio 2015 or higher on Windows [#]_ - Python: Version 2.6, 2.7 or 3.2, 3.3, 3.4, 3.5 (yes, but read below) .. admonition:: Python3, yes but Python2 *compile time* dependency For Python3 you *need* a Python2, but only during the compile time only, and that is for Scons (which orchestrates the C++ compilation), and is not yet ported. So for Python 3.x, there is currently a requirement to also have a Python 2.x installed. Nuitka itself is fully Python3 compatible except for Scons. .. admonition:: Moving to other machines The created binaries can be made executable independent of the Python installation, with ``--standalone`` option. .. admonition:: Binary filename suffix ".exe" even on Linux The created binaries have an ".exe" suffix, that you are free to remove and yes, they are still Linux binaries. The suffix is just to be sure that the original script name and the binary name do not collide. .. admonition:: It has to be CPython, maybe WinPython or AnaConda You need the standard Python implementation, called "CPython", to execute Nuitka, because it is closely tied to using it. On Windows, the so called "WinPython" and "AnaConda" distributions but will cause issues for acceleration mode. Standalone and creating extension modules or packages will also work. For acceleration mode, you need to copy the "PythonXX.DLL" alongside of it. - Operating System: Linux, FreeBSD, NetBSD, MacOS X, and Windows (32/64 bits). Others may work as well. The portability is expected to be generally good, but the e.g. Scons usage may have to be adapted. - Architectures: x86, x86_64 (amd64), and arm, likely more Other architectures are expected to also work, out of the box, as Nuitka is generally not using any hardware specifics. These are just the ones tested and known to be good. Feedback is welcome. Generally the architectures that Debian supports can be considered good and tested too. .. [#] Support for this C++03 language standard is practically a given on any C++ compiler you encounter. Nuitka used to have higher requirements in the past, but it changed. .. [#] Download MinGW64 from here http://mingw-w64.org/ and choose 64 or 32 bits matching your Python. Use both MinGW64 and 64 bits Python if you have the choice of which Python to use. .. [#] Download for free from http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx (the Express editions will normally work just fine). Command Line ------------ No environment variable changes are needed, you can call the ``nuitka`` and ``nuitka-run`` scripts directly without any changes to the environment. You may want to add the ``bin`` directory to your ``PATH`` for your convenience, but that step is optional. Nuitka has a ``--help`` option to output what it can do: .. code-block:: bash nuitka --help The ``nuitka-run`` command is the same as ``nuitka``, but with different default. It tries to compile *and* directly execute a Python script: .. code-block:: bash nuitka-run --help These option that is different is ``--run``, and passing on arguments after the first non-option to the created binary, so it is somewhat more similar to what plain ``python`` will do. License ------- Nuitka is licensed under the Apache License, Version 2.0; you may not use it except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Use Cases ========= Use Case 1 - Program compilation with all modules embedded ---------------------------------------------------------- If you want to compile a whole program recursively, and not only the single file that is the main program, do it like this: .. code-block:: bash nuitka --recurse-all program.py .. note:: There are more fine grained controls than ``--recurse-all`` available. Consider the output of ``nuitka --help``. In case you have a plugin directory, i.e. one which cannot be found by recursing after normal import statements via the ``PYTHONPATH`` (which would be recommended way), you can always require that a given directory shall also be included in the executable: .. code-block:: bash nuitka --recurse-all --recurse-directory=plugin_dir program.py .. note:: If you don't do any dynamic imports, simply setting your ``PYTHONPATH`` at compilation time will be sufficient for all your needs normally. Use ``--recurse-directory`` only if you make ``__import__()`` calls that Nuitka cannot predict, because they e.g. depend on command line parameters. Nuitka also warns about these, and point to the option. .. note:: The resulting binary still depends on CPython and used C extension modules being installed. If you want to be able to copy it to another machine, use ``--standalone`` and copy the created ``program.dist`` directory and execute the ``program.exe`` put inside. .. note:: The resulting filename will be ``program.exe`` on all platforms, that doesn't mean it doesn't run on non-Windows! But if you compile ``program`` we wouldn't want to overwrite it, or be unsure which one is the compiled form, and which one is not. Use Case 2 - Extension Module compilation ----------------------------------------- If you want to compile a single extension module, all you have to do is this: .. code-block:: bash nuitka --module some_module.py The resulting file "some_module.so" can then be used instead of "some_module.py". It's left as an exercise to the reader, what happens if both are present. .. note:: The option ``--recurse-all`` and other variants work as well. Use Case 3 - Package compilation -------------------------------- If you need to compile a whole package and embedded all modules, that is also feasible, use Nuitka like this: .. code-block:: bash nuitka --module some_package --recurse-directory=some_package .. note:: The recursion into the package directory needs to be provided manually, otherwise the package is empty. Data files located inside the package will not be embedded yet. Where to go next ================ Remember, this project is not completed yet. Although the CPython test suite works near perfect, there is still more work needed, to make it do more optimization. Try it out. Subscribe to its mailing lists ------------------------------ Please visit the `mailing list page `__ in order to subscribe the relatively low volume mailing list. All Nuitka issues can be discussed there. Report issues or bugs --------------------- Should you encounter any issues, bugs, or ideas, please visit the `Nuitka bug tracker `__ and report them. Best practices for reporting bugs: - Please aways include the following information in your report, for the underlying Python version. You can easily copy&paste this into your report. .. code-block:: sh nuitka --version - Try to make your example minimal. That is, try to remove code that does not contribute to the issue as much as possible. Ideally come up with a small reproducing program that illustrates the issue, using ``print`` with different results when that programs runs compiled or native. - If the problem occurs spuriously (i.e. not each time), try to set the environment variable ``PYTHONHASHSEED`` to ``0``, disabling hash randomization. If that makes the problem go away, try increasing in steps of 1 to a hash seed value that makes it happen every time. - Do not include the created code in your report. Given proper input, it's redundant, and it's not likely that I will look at it without the ability to change the Python or Nuitka source and re-run it. Contact me via email with your questions ---------------------------------------- You are welcome to `contact me via email `__ with your questions. But it is increasingly true that for user questions the mailing list is the best place to go. Word of Warning --------------- Consider using this software with caution. Your feedback and patches to Nuitka are very welcome. Especially report it please, if you find that anything doesn't work, because the project is now at the stage that this should not happen and most definitely will mean you encountered an unknown bug. Join Nuitka =========== You are more than welcome to join Nuitka development and help to complete the project in all minor and major ways. The development of Nuitka occurs in git. We currently have these 3 branches: - `master `__: This branch contains the stable release to which only hotfixes for bugs will be done. It is supposed to work at all times and is supported. - `develop `__: This branch contains the ongoing development. It may at times contain little regressions, but also new features. On this branch the integration work is done, whereas new features might be developed on feature branches. - `factory `__: This branch contains unfinished and incomplete work. It is very frequently subject ``git rebase`` and the public staging ground, where my work for develop branch lives first. It is intended for testing only and recommended to base any of your own development on. When updating it, you very often will get merge conflicts. Simply resolve those by doing ``git reset --hard origin/factory`` and switch to the latest version. .. note:: I accept patch files, git formatted patch queues (use ``git format-patch origin`` command), or if you prefer git pull on the social code platforms. I will do the integration work. If you base your work on "master" or "develop" at any given time, I will do any re-basing required and keep your authorship intact. .. note:: The `Developer Manual `__ explains the coding rules, branching model used, with feature branches and hotfix releases, the Nuitka design and much more. Consider reading it to become a contributor. This document is intended for Nuitka users. Donations ========= Should you feel that you cannot help Nuitka directly, but still want to support, please consider `making a donation `__ and help this way. Unsupported functionality ========================= The ``co_code`` attribute of code objects ----------------------------------------- The code objects are empty for for native compiled functions. There is no bytecode with Nuitka's compiled function objects, so there is no way to provide it. Optimization ============ Constant Folding ---------------- The most important form of optimization is the constant folding. This is when an operation can be fully predicted at compile time. Currently Nuitka does these for some built-ins (but not all yet, somebody to look at this more closely will be very welcome!), and it does it e.g. for binary/unary operations and comparisons. Constants currently recognized: .. code-block:: python 5 + 6 # binary operations not 7 # unary operations 5 < 6 # comparisons range(3) # built-ins Literals are the one obvious source of constants, but also most likely other optimization steps like constant propagation or function inlining will be. So this one should not be underestimated and a very important step of successful optimizations. Every option to produce a constant may impact the generated code quality a lot. .. admonition:: Status The folding of constants is considered implemented, but it might be incomplete in that not all possible cases are caught. Please report it as a bug when you find an operation in Nuitka that has only constants as input and is not folded. Constant Propagation -------------------- At the core of optimizations there is an attempt to determine values of variables at run time and predictions of assignments. It determines if their inputs are constants or of similar values. An expression, e.g. a module variable access, an expensive operation, may be constant across the module of the function scope and then there needs to be none, or no repeated module variable look-up. Consider e.g. the module attribute ``__name__`` which likely is only ever read, so its value could be predicted to a constant string known at compile time. This can then be used as input to the constant folding. .. code-block:: python if __name__ == "__main__": # Your test code might be here use_something_not_use_by_program() .. admonition:: Status From modules attributes, only ``__name__`` are currently actually optimized. Also possible would be at least ``__doc__``. In the future, this may improve as SSA is expanded to module variables. Built-in Name Lookups --------------------- Also built-in exception name references are optimized if they are used as module level read only variables: .. code-block:: python try: something() except ValueError: # The ValueError is a slow global name lookup normally. pass .. admonition:: Status This works for all built-in names. When an assignment is done to such a name, or it's even local, then of course it is not done. Builtin Call Prediction ----------------------- For builtin calls like ``type``, ``len``, or ``range`` it is often possible to predict the result at compile time, esp. for constant inputs the resulting value often can be precomputed by Nuitka. It can simply determine the result or the raised exception and replace the builtin call with it allowing for more constant folding or code path folding. .. code-block:: python type("string") # predictable result, builtin type str. len([1, 2]) # predictable result range(3, 9, 2) # predictable result range(3, 9, 0) # predictable exception, range raises due to 0. .. admonition:: Status The builtin call prediction is considered implemented. We can simply during compile time emulate the call and use its result or raised exception. But we may not cover all the built-ins there are yet. Sometimes the result of a built-in should not be predicted when the result is big. A ``range()`` call e.g. may give too big values to include the result in the binary. Then it is not done. .. code-block:: python range( 100000 ) # We do not want this one to be expanded .. admonition:: Status This is considered mostly implemented. Please file bugs for built-ins that are pre-computed, but should not be computed by Nuitka at compile time with specific values. Conditional Statement Prediction -------------------------------- For conditional statements, some branches may not ever be taken, because of the conditions being possible to predict. In these cases, the branch not taken and the condition check is removed. This can typically predict code like this: .. code-block:: python if __name__ == "__main__": # Your test code might be here use_something_not_use_by_program() or .. code-block:: python if False: # Your deactivated code might be here It will also benefit from constant propagations, or enable them because once some branches have been removed, other things may become more predictable, so this can trigger other optimization to become possible. Every branch removed makes optimization more likely. With some code branches removed, access patterns may be more friendly. Imagine e.g. that a function is only called in a removed branch. It may be possible to remove it entirely, and that may have other consequences too. .. admonition:: Status This is considered implemented, but for the maximum benefit, more constants need to be determined at compile time. Exception Propagation --------------------- For exceptions that are determined at compile time, there is an expression that will simply do raise the exception. These can be propagated upwards, collecting potentially "side effects", i.e. parts of expressions that were executed before it occurred, and still have to be executed. Consider the following code: .. code-block:: python print side_effect_having() + (1 / 0) print something_else() The ``(1 / 0)`` can be predicted to raise a ``ZeroDivisionError`` exception, which will be propagated through the ``+`` operation. That part is just Constant Propagation as normal. The call to ``side_effect_having`` will have to be retained though, but the ``print`` statement does and can be turned into an explicit raise. The statement sequence can then be aborted and as such the ``something_else`` call needs no code generation or consideration anymore. To that end, Nuitka works with a special node that raises an exception and has so called "side_effects" children, yet can be used in generated code as an expression. .. admonition:: Status The propagation of exceptions is mostly implemented, but needs handling in every kind of operations, and not all of them might do it already. As work progresses or examples arise, the coverage will be extended. Feel free to generate bug reports with non-working examples. Exception Scope Reduction ------------------------- Consider the following code: .. code-block:: python try: b = 8 print range(3, b, 0) print "Will not be executed" except ValueError, e: print e The ``try`` block is bigger than it needs to be. The statement ``b = 8`` cannot cause a ``ValueError`` to be raised. As such it can be moved to outside the try without any risk. .. code-block:: python b = 8 try: print range(3, b, 0) print "Will not be executed" except ValueError as e: print e .. admonition:: Status This is considered done. For every kind of operation, we trace if it may raise an exception. We do however *not* track properly yet, what can do a ``ValueError`` and what cannot. Exception Block Inlining ------------------------ With the exception propagation it is then possible to transform this code: .. code-block:: python try: b = 8 print range(3, b, 0) print "Will not be executed" except ValueError, e: print e .. code-block:: python try: raise ValueError, "range() step argument must not be zero" except ValueError, e: print e Which then can be reduced by avoiding the raise and catch of the exception, making it: .. code-block:: python e = ValueError( "range() step argument must not be zero" ) print e .. admonition:: Status This is not implemented yet. Empty Branch Removal -------------------- For loops and conditional statements that contain only code without effect, it should be possible to remove the whole construct: .. code-block:: python for i in range(1000): pass The loop could be removed, at maximum it should be considered an assignment of variable ``i`` to ``999`` and no more. .. admonition:: Status This is not implemented yet, as it requires us to track iterators, and their side effects, as well as loop values, and exit conditions. Too much yet, but we will get there. Another example: .. code-block:: python if side_effect_free: pass The condition check should be removed in this case, as its evaluation is not needed. It may be difficult to predict that ``side_effect_free`` has no side effects, but many times this might be possible. .. admonition:: Status This is considered implemented. The conditional statement nature is removed if both branches are empty, only the condition is evaluated, and checked for truth (in cases that could raise an exception). Unpacking Prediction -------------------- When the length of the right hand side of an assignment to a sequence can be predicted, the unpacking can be replaced with multiple assignments. .. code-block:: python a, b, c = 1, side_effect_free(), 3 .. code-block:: python a = 1 b = side_effect_free() c = 3 This is of course only really safe if the left hand side cannot raise an exception while building the assignment targets. We do this now, but only for constants, because we currently have no ability to predict if an expression can raise an exception or not. .. admonition:: Status Not implemented yet. Will need us to see through the unpacking of what is an iteration over a tuple, we created ourselves. We are not there yet, but we will get there. Built-in Type Inference ----------------------- When a construct like ``in xrange()`` or ``in range()`` is used, it is possible to know what the iteration does and represent that, so that iterator users can use that instead. I consider that: .. code-block:: python for i in xrange(1000): something(i) could translate ``xrange(1000)`` into an object of a special class that does the integer looping more efficiently. In case ``i`` is only assigned from there, this could be a nice case for a dedicated class. .. admonition:: Status Future work, not even started. Quicker Function Calls ---------------------- Functions are structured so that their parameter parsing and ``tp_call`` interface is separate from the actual function code. This way the call can be optimized away. One problem is that the evaluation order can differ. .. code-block:: python def f(a, b, c): return a, b, c f(c = get1(), b = get2(), a = get3()) This will have to evaluate first ``get1()``, then ``get2()`` and only then ``get3()`` and then make the function call with these values. Therefore it will be necessary to have a staging of the parameters before making the actual call, to avoid an re-ordering of the calls to ``get1()``, ``get2()``, and ``get3()``. .. admonition:: Status Not even started. A re-formulation that avoids the dictionary to call the function, and instead uses temporary variables appears to be relatively straight forward once we do that kind of parameter analysis. Lowering of iterated Container Types ------------------------------------ In some cases, accesses to ``list`` constants can become ``tuple`` constants instead. Consider that: .. code-block:: python for x in [a, b, c]: something(x) Can be optimized into this: .. code-block:: python for x in (a, b, c): something(x) This allows for simpler, faster code to be generated, and less checks needed, because e.g. the ``tuple`` is clearly immutable, whereas the ``list`` needs a check to assert that. This is also possible for sets. .. admonition:: Status Implemented, even works for non-constants. Needs other optimization to become generally useful, and will itself help other optimization to become possible. This allows us to e.g. only treat iteration over tuples, and not care about sets. In theory something similar is also possible for ``dict``. For the later it will be non-trivial though to maintain the order of execution without temporary values introduced. The same thing is done for pure constants of these types, they change to ``tuple`` values when iterated. Credits ======= Contributors to Nuitka ---------------------- Thanks go to these individuals for their much valued contributions to Nuitka. Contributors have the license to use Nuitka for their own code even if Closed Source. The order is sorted by time. - Li Xuan Ji: Contributed patches for general portability issue and enhancements to the environment variable settings. - Nicolas Dumazet: Found and fixed reference counting issues, ``import`` packages work, improved some of the English and generally made good code contributions all over the place, solved code generation TODOs, did tree building cleanups, core stuff. - Khalid Abu Bakr: Submitted patches for his work to support MinGW and Windows, debugged the issues, and helped me to get cross compile with MinGW from Linux to Windows. This was quite a difficult stuff. - Liu Zhenhai: Submitted patches for Windows support, making the inline Scons copy actually work on Windows as well. Also reported import related bugs, and generally helped me make the Windows port more usable through his testing and information. - Christopher Tott: Submitted patches for Windows, and general as well as structural cleanups. - Pete Hunt: Submitted patches for MacOS X support. - "ownssh": Submitted patches for built-ins module guarding, and made massive efforts to make high quality bug reports. Also the initial "standalone" mode implementation was created by him. - Juan Carlos Paco: Submitted cleanup patches, creator of the `Nuitka GUI `__, creator of the `Ninja IDE plugin `__ for Nuitka. - "dr. Equivalent": Submitted the Nuitka Logo. - Johan Holmberg: Submitted patch for Python3 support on MacOS X. - Umbra: Submitted patches to make the Windows port more usable, adding user provided application icons, as well as MSVC support for large constants and console applications. - David Cortesi: Submitted patches and test cases to make MacOS port more usable, specifically for the Python3 standalone support of Qt. Projects used by Nuitka ----------------------- * The `CPython project `__ Thanks for giving us CPython, which is the base of Nuitka. We are nothing without it. * The `GCC project `__ Thanks for not only the best compiler suite, but also thanks for supporting C++11 which helped to get Nuitka off the ground. Your compiler was the first usable for Nuitka and with little effort. * The `Scons project `__ Thanks for tackling the difficult points and providing a Python environment to make the build results. This is such a perfect fit to Nuitka and a dependency that will likely remain. * The `valgrind project `__ Luckily we can use Valgrind to determine if something is an actual improvement without the noise. And it's also helpful to determine what's actually happening when comparing. * The `NeuroDebian project `__ Thanks for hosting the build infrastructure that the Debian and sponsor Yaroslav Halchenko uses to provide packages for all Ubuntu versions. * The `openSUSE Buildservice `__ Thanks for hosting this excellent service that allows us to provide RPMs for a large variety of platforms and make them available immediately nearly at release time. * The `MinGW64 project `__ Thanks for porting the gcc to Windows. This allowed portability of Nuitka with relatively little effort. * The `Buildbot project `__ Thanks for creating an easy to deploy and use continuous integration framework that also runs on Windows and is written and configured in Python code. This allows to run the Nuitka tests long before release time. Updates for this Manual ======================= This document is written in REST. That is an ASCII format which is readable as ASCII, but used to generate PDF or HTML documents. You will find the current source under: http://nuitka.net/gitweb/?p=Nuitka.git;a=blob_plain;f=README.rst And the current PDF under: http://nuitka.net/doc/README.pdf Nuitka-0.5.18.1/Changelog.rst0000644000175000017500000122700712651072046016066 0ustar hayenhayen00000000000000Nuitka Release 0.5.18 ===================== This release mainly has a scalability focus. While there are few compatibility improvements, the larger goal has been to make Nuitka compilation and the final C compilation faster. Bug Fixes --------- - Compatibility: The nested arguments functions can now be called using their keyword arguments. .. code-block:: python def someFunction(a,(b,c)): return a, b, c someFunction(a = 1, **{".1" : (2,3)}) - Compatibility: Generators with Python3.4 or higher now also have a ``__del__`` attribute, and therefore properly participate in finalization. This should improve their interactions with garbage collection reference cycles, although no issues had been observed so far. - Windows: Was outputting command line arguments debug information at program start. `Issue#284 `__. Fixed in 0.5.17.1 already. Optimization ------------ - Code generated for parameter parsing is now a *lot* less verbose. Python level loops and conditionals to generate code for each variable has been replaced with C level generic code. This will speed up the backend compilation by a lot. - Function calls with constant arguments were speed up specifically, as their call is now fully prepared, and yet using less code. Variable arguments are also faster, and all defaulted arguments are also much faster. Method calls are not affected by these improvements though. - Nested argument functions now have a quick call entry point as well, making them faster to call too. - The ``slice`` built-in, and internal creation of slices (e.g. in re-formulations of Python3 slices as subscripts) cannot raise. `Issue#262 `__. - Standalone: Avoid inclusion of bytecode of ``unittest.test``, ``sqlite3.test``, ``distutils.test``, and ``ensurepip``. These are not needed, but simply bloat the amount of bytecode used on e.g. MacOS. `Issue#272 `__. - Speed up compilation with Nuitka itself by avoid to copying and constructing variable lists as much as possible using an always accurate variable registry. Cleanups -------- - Nested argument functions of Python2 are now re-formulated into a wrapping function that directly calls the actual function body with the unpacking of nested arguments done in nodes explicitly. This allows for better optimization and checks of these steps and potential in-lining of these functions too. - Unified slice object creation and built-in ``slice`` nodes, these were two distinct nodes before. - The code generation for all statement kinds is now done via dispatching from a dictionary instead of long ``elif`` chains. - Named nodes more often consistently, e.g. all loop related nodes start with ``Loop`` now, making them easier to group. - Parameter specifications got simplified to work without variables where it is possible. Organizational -------------- - Nuitka is now available on the social code platforms gitlab as well. Summary ------- Long standing weaknesses have been addressed in this release, also quite a few structural cleanups have been performed, e.g. strengthening the role of the variable registry to always be accurate, is groundlaying to further improvement of optimization. However, this release cycle was mostly dedicated to performance of the actual compilation, and more accurate information was needed to e.g. not search for information that should be instant. Upcoming releases will focus on usability issues and further optimization, it was nice however to see speedups of created code even from these scalability improvements. Nuitka Release 0.5.17 ===================== This release is a major feature release, as it adds full support for Python3.5 and its coroutines. In addition, in order to properly support coroutines, the generator implementation got enhanced. On top of that, there is the usual range of corrections. Bug Fixes --------- - Windows: Command line arguments that are unicode strings were not properly working. - Compatibility: Fix, only the code object attached to exceptions contained all variable names, but not the one of the function object. - Python3: Support for virtualenv on Windows was using non-portable code and therefore failing. `Issue#266 `__. - The tree displayed with ``--display-tree`` duplicated all functions and did not resolve source lines for functions. It also displayed unused functions, which is not helpful. - Generators with parameters leaked C level memory for each instance of them leading to memory bloat for long running programs that use a lot of generators. Fixed in 0.5.16.1 already. - Don't drop positional arguments when called with ``--run``, also make it an error if they are present without that option. New Features ------------ - Added full support for Python3.5, coroutines work now too. Optimization ------------ - Optimized frame access of generators to not use both a local frame variable and the frame object stored in the generator object itself. This gave about 1% speed up to setting them up. - Avoid having multiple code objects for functions that can raise and have local variables. Previously one code object would be used to create the function (with parameter variable names only) and when raising an exception, another one would be used (with all local variable names). Creating them both at start-up was wasteful and also needed two tuples to be created, thus more constants setup code. - The entry point for generators is now shared code instead of being generated for each one over and over. This should make things more cache local and also results in less generated C code. - When creating frame codes, avoid working with strings, but use proper emission for less memory churn during code generation. Organizational -------------- - Updated the key for the Debian/Ubuntu repositories to remain valid for 2 more years. - Added support for Fedora 23. - MinGW32 is no more supported, use MinGW64 in the 32 bits variant, which has less issues. Cleanups -------- - Detecting function type ahead of times, allows to handle generators different from normal functions immediately. - Massive removal of code duplication between normal functions and generator functions. The later are now normal functions creating generator objects, which makes them much more lightweight. - The ``return`` statement in generators is now immediately set to the proper node as opposed to doing this in variable closure phase only. We can now use the ahead knowledge of the function type. - The ``nonlocal`` statement is now immediately checked for syntax errors as opposed to doing that only in variable closure phase. - The name of contraction making functions is no longer skewed to empty, but the real thing instead. The code name is solved differently now. - The ``local_locals`` mode for function node was removed, it was always true ever since Python2 list contractions stop using pseudo functions. - The outline nodes allowed to provide a body when creating them, although creating that body required using the outline node already to create temporary variables. Removed that argument. - Removed PyLint false positive annotations no more needed for PyLint 1.5 and solved some TODOs. - Code objects are now mostly created from specs (not yet complete) which are attached and shared between statement frames and function creations nodes, in order to have less guess work to do. Tests ----- - Added the CPython3.5 test suite. - Updated generated doctests to fix typos and use common code in all CPython test suites. Summary ------- This release continues to address technical debt. Adding support for Python3.5 was the major driving force, while at the same time removing obstacles to the changes that were needed for coroutine support. With Python3.5 sorted out, it will be time to focus on general optimization again, but there is more technical debt related to classes, so the cleanup has to continue. Nuitka Release 0.5.16 ===================== This is a maintenance release, largely intended to put out improved support for new platforms and minor corrections. It should improve the speed for standalone mode, and compilation in general for some use cases, but this is mostly to clean up open ends. Bug Fixes --------- - Fix, the ``len`` built-in could give false values for dictionary and set creations with the same element. .. code-block:: python # This was falsely optimized to 2 even if "a is b and a == b" was true. len({a, b}) - Python: Fix, the ``gi_running`` attribute of generators is no longer an ``int``, but ``bool`` instead. - Python3: Fix, the ``int`` built-in with two arguments, value and base, raised ``UnicodeDecodeError`` instead of ``ValueError`` for illegal bytes given as value. - Python3: Using ``tokenize.open`` to read source code, instead of reading manually and decoding from ``tokenize.detect_encoding``, this handles corner cases more compatible. - Fix, the PyLint warnings plug-in could crash in some cases, make sure it's more robust. - Windows: Fix, the combination of AnaConda Python, MinGW 64 bits and mere acceleration was not working. `Issue#254 `__. - Standalone: Preserve not only namespace packages created by ``.pth`` files, but also make the imports done by them. This makes it more compatible with uses of it in Fedora 22. - Standalone: The extension modules could be duplicated, turned this into an error and cache finding them during compile time and during early import resolution to avoid duplication. - Standalone: Handle "not found" from ``ldd`` output, on some systems not all the libraries wanted are accessible for every library. - Python3.5: Fixed support for namespace packages, these were not yet working for that version yet. - Python3.5: Fixes lack of support for unpacking in normal ``tuple``, ``list``, and ``set`` creations. .. code-block:: python [*a] # this has become legal in 3.5 and now works too. Now also gives compatible ``SyntaxError`` for earlier versions. Python2 was good already. - Python3.5: Fix, need to reduce compiled functions to ``__qualname__`` value, rather than just ``__name__`` or else pickling methods doesn't work. - Python3.5: Fix, added ``gi_yieldfrom`` attribute to generator objects. - Windows: Fixed harmless warnings for Visual Studio 2015 in ``--debug`` mode. Optimization ------------ - Re-formulate ``exec`` and ``eval`` to default to ``globals()`` as the default for the locals dictionary in modules. - The ``try`` node was making a description of nodes moved to the outside when shrinking its scope, which was using a lot of time, just to not be output, now these can be postponed. - Refactored how freezing of bytecode works. Uncompiled modules are now explicit nodes too, and in the registry. We only have one or the other of it, avoiding to compile both. Tests ----- - When ``strace`` or ``dtruss`` are not found, given proper error message, so people know what to do. - The doc tests extracted and then generated for CPython3 test suites were not printing the expressions of the doc test, leading to largely decreased test coverage here. - The CPython 3.4 test suite is now also using common runner code, and avoids ignoring all Nuitka warnings, instead more white listing was added. - Started to run CPython 3.5 test suite almost completely, but coroutines are blocking some parts of that, so these tests that use this feature are currently skipped. - Removed more CPython tests that access the network and are generally useless to testing Nuitka. - When comparing outputs, normalize typical temporary file names used on posix systems. - Coverage tests have made some progress, and some changes were made due to its results. - Added test to cover too complex code module of ``idna`` module. - Added Python3.5 only test for unpacking variants. Cleanups -------- - Prepare plug-in interface to allow suppression of import warnings to access the node doing it, making the import node is accessible. - Have dedicated class function body object, which is a specialization of the function body node base class. This allowed removing class specific code from that class. - The use of "win_target" as a scons parameter was useless. Make more consistent use of it as a flag indicator in the scons file. - Compiled types were mixing uses of ``compiled_`` prefixes, something with a space, sometimes with an underscore. Organizational -------------- - Improved support for Python3.5 missing compatibility with new language features. - Updated the Developer Manual with changes that SSA is now a fact. - Added Python3.5 Windows MSI downloads. - Added repository for Ubuntu Wily (15.10) for download. Removed Ubuntu Utopic package download, no longer supported by Ubuntu. - Added repository with RPM packages for Fedora 22. Summary ------- So this release is mostly to lower the technical debt incurred that holds it back from supporting making more interesting changes. Upcoming releases may have continue that trend for some time. This release is mostly about catching up with Python3.5, to make sure we did not miss anything important. The new function body variants will make it easier to implement coroutines, and help with optimization and compatibility problems that remain for Python3 classes. Ultimately it will be nice to require a lot less checks for when function in-line is going to be acceptable. Also code generation will need a continued push to use the new structure in preparation for making type specific code generation a reality. Nuitka Release 0.5.15 ===================== This release enables SSA based optimization, the huge leap, not so much in terms of actual performance increase, but for now making the things possible that will allow it. This has been in the making literally for years. Over and over, there was just "one more thing" needed. But now it's there. The release includes much stuff, and there is a perspective on the open tasks in the summary, but first out to the many details. Bug Fixes --------- - Standalone: Added implicit import for ``reportlab`` package configuration dynamic import. Fixed in 0.5.14.1 already. - Standalone: Fix, compilation of the ``ctypes`` module could happen for some import patterns, and then prevented the distribution to contain all necessary libraries. Now it is made certain to not include compiled and frozen form both. `Issue#241 `__. Fixed in 0.5.14.1 already. - Fix, compilation for conditional statements where the boolean check on the condition cannot raise, could fail compilation. `Issue#240 `__. Fixed in 0.5.14.2 already. - Fix, the ``__import__`` built-in was making static optimization assuming compile time constants to be strings, which in the error case they are not, which was crashing the compiler. `Issue#240 `__. .. code-block:: python __import__(("some.module",)) # tuples don't work This error became only apparent, because now in some cases, Nuitka forward propagates values. - Windows: Fix, when installing Python2 only for the user, the detection of it via registry failed as it was only searching system key. This was `a github pull request `__. Fixed in 0.5.14.3 already. - Some modules have extremely complex expressions requiring too deep recursion to work on all platforms. These modules are now included entirely as bytecode fallback. `Issue#240 `__. - The standard library may contain broken code due to installation mistakes. We have to ignore their ``SyntaxError``. `Issue#244 `__. - Fix, pickling compiled methods was failing with the wrong kind of error, because they should not implement ``__reduce__``, but only ``__deepcopy__``. `Issue#219 `__. - Fix, when running under ``wine``, the check for scons binary was fooled by existance of ``/usr/bin/scons``. `Issue#251 `__. New Features ------------ - Added experimental support for Python3.5, coroutines don't work yet, but it works perfectly as a 3.4 replacement. - Added experimental Nuitka plug-in framework, and use it for the packaging of Qt plugins in standalone mode. The API is not yet stable nor polished. - New option ``--debugger`` that makes ``--run`` execute directly in ``gdb`` and gives a stack trace on crash. - New option ``--profile`` executes compiled binary and outputs measured performance with ``vmprof``. This is work in progress and not functional yet. - Started work on ``--graph`` to render the SSA state into diagrams. This is work in progress and not functional yet. - Plug-in framework added. Not yet ready for users. Working ``PyQt4`` and ``PyQt5`` plug-in support. Experimental Windows ``multiprocessing`` support. Experimental PyLint warnings disable support. More to come. - Added support for AnaConda accelerated mode on MacOS by modifying the rpath to the Python DLL. - Added experimental support for ``multiprocessing`` on Windows, which needs money patching of the module to support compiled methods. Optimization ------------ - The SSA analysis is now enabled by default, eliminating variables that are not shared, and can be forward propagated. This is currently limited mostly to compile time constants, but things won't remain that way. - Code generation for many constructs now takes into account if a specific operation can raise or not. If e.g. an attribute look-up is known to not raise, then that is now decided by the node the looked is done to, and then more often can determine this, or even directly the value. - Calls to C-API that we know cannot raise, no longer check, but merely assert the result. - For attribute look-up and other operations that might be known to not raise, we now only assert that it succeeds. - Built-in loop-ups cannot fail, merely assert that. - Creation of built-in exceptions never raises, merely assert that too. - More Python operation slots now have their own computations and some of these gained overloads for more compile time constant optimization. - When taking an iterator cannot raise, this is now detected more often. - The ``try``/``finally`` construct is now represented by duplicating the final block into all kinds of handlers (``break``, ``continue``, ``return``, or ``except``) and optimized separately. This allows for SSA to trace values more correctly. - The ``hash`` built-in now has dedicated node and code generation too. This is mostly intended to represent the side effects of dictionary look-up, but gives more compact and faster code too. - Type ``type`` built-in cannot raise and has no side effect. - Speed improvement for in-place float operations for ``+=`` and ``*=``, as these will be common cases. Tests ----- - Made the construct based testing executable with Python3. - Removed warnings using the new PyLint warnings plug-in for the reflected test. Nuitka now uses the PyLint annotations to not warn. Also do not go into PyQt for reflected test, not needed. Many Python3 improvements for cases where there are differences to report. - The optimization tests no longer use 2to3 anymore, made the tests portable to all versions. - Checked more in-place operations for speed. Organizational -------------- - Many improvements to the coverage taking. We can hope to see public data from this, some improvements were triggered from this already, but full runs of the test suite with coverage data collection are yet to be done. Summary ------- The release includes many important new directions. Coverage analysis will be important to remain certain of test coverage of Nuitka itself. This is mostly done, but needs more work to complete. Then the graphing surely will help us to debug and understand code examples. So instead of tracing, and reading stuff, we should visualize things, to more clearly see, how things evolve under optimization iteration, and where exactly one thing goes wrong. This will be improved as it proves necessary to do just that. So far, this has been rare. Expect this to become end user capable with time. If only to allow you to understand why Nuitka won't optimize code of yours, and what change of Nuitka it will need to improve. The comparative performance benchmarking is clearly the most important thing to have for users. It deserves to be the top priority. Thanks to the PyPy tool ``vmprof``, we may already be there on the data taking side, but the presenting and correlation part, is still open and a fair bit of work. It will be most important to empower users to make competent performance bug reports, now that Nuitka enters the phase, where these things matter. As this is a lot of ground to cover. More than ever. We can make this compiler, but only if you help, it will arrive in your life time. Nuitka Release 0.5.14 ===================== This release is an intermediate step towards value propagation, which is not considered ready for stable release yet. The major point is the elimination of the ``try``/``finally`` expressions, as they are problems to SSA. The ``try``/``finally`` statement change is delayed. There are also a lot of bug fixes, and enhancements to code generation, as well as major cleanups of code base. Bug Fixes --------- - Python3: Added support assignments trailing star assignment. .. code-block:: python *a, b = 1, 2 This raised ``ValueError`` before. - Python3: Properly detect illegal double star assignments. .. code-block:: python *a, *b = c - Python3: Properly detect the syntax error to star assign from non-tuple/list. .. code-block:: python *a = 1 - Python3.4: Fixed a crash of the binary when copying dictionaries with split tables received as star arguments. - Python3: Fixed reference loss, when using ``raise a from b`` where ``b`` was an exception instance. Fixed in 0.5.13.8 already. - Windows: Fix, the flag ``--disable-windows-console`` was not properly handled for MinGW32 run time resulting in a crash. - Python2.7.10: Was not recognizing this as a 2.7.x variant and therefore not applying minor version compatibility levels properly. - Fix, when choosing to have frozen source references, code objects were not use the same value as ``__file__`` did for its filename. - Fix, when re-executing itself to drop the ``site`` module, make sure we find the same file again, and not according to the ``PYTHONPATH`` changes coming from it. `Issue#223 `__. Fixed in 0.5.13.4 already. - Enhanced code generation for ``del variable`` statements, where it's clear that the value must be assigned. - When pressing CTRL-C, the stack traces from both Nuitka and Scons were given, we now avoid the one from Scons. - Fix, the dump from ``--xml`` no longer contains functions that have become unused during analysis. - Standalone: Creating or running programs from inside unicode paths was not working on Windows. `Issue#231 `__ `Issue#229 `__ and. Fixed in 0.5.13.7 already. - Namespace package support was not yet complete, importing the parent of a package was still failing. `Issue#230 `__. Fixed in 0.5.13.7 already. - Python2.6: Compatibility for exception check messages enhanced with newest minor releases. - Compatibility: The ``NameError`` in classes needs to say ``global name`` and not just ``name`` too. - Python3: Fixed creation of XML representation, now done without ``lxml`` as it doesn't support needed features on that version. Fixed in 0.5.13.5 already. - Python2: Fix, when creating code for the largest negative constant to still fit into ``int``, that was only working in the main module. `Issue#228 `__. Fixed in 0.5.13.5 already. - Compatibility: The ``print`` statement raised an assertion on unicode objects that could not be encoded with ``ascii`` codec. New Features ------------ - Added support for Windows 10. - Followed changes for Python 3.5 beta 2. Still only usable as a Python 3.4 replacement, no new features. - Using a self compiled Python running from the source tree is now supported. - Added support for ``AnaConda`` Python distribution. As it doesn't install the Python DLL, we copy it along for acceleration mode. - Added support for Visual Studio 2015. `Issue#222 `__. Fixed in 0.5.13.3 already. - Added support for self compiled Python versions running from build tree, this is intended to help debug things on Windows. Optimization ------------ - Function in-lining is now present in the code, but still disabled, because it needs more changes in other areas, before we can generally do it. - Trivial outlines, result of re-formulations or function in-lining, are now in-lined, in case they just return an expression. - The re-formulation for ``or`` and ``and`` has been giving up, eliminating the use of a ``try``/``finally`` expression, at the cost of dedicated boolean nodes and code generation for these. This saves around 8% of compile time memory for Nuitka, and allows for faster and more complete optimization, and gets rid of a complicated structure for analysis. - When a frame is used in an exception, its locals are detached. This was done more often than necessary and even for frames that are not necessary our own ones. This will speed up some exception cases. - When the default arguments, or the keyword default arguments (Python3) or the annotations (Python3) were raising an exception, the function definition is now replaced with the exception, saving a code generation. This happens frequently with Python2/Python3 compatible code guarded by version checks. - The SSA analysis for loops now properly traces "break" statement situations and merges the post-loop situation from all of them. This significantly allows for and improves optimization of code following the loop. - The SSA analysis of ``try``/``finally`` statements has been greatly enhanced. The handler for ``finally`` is now optimized for exception raise and no exception raise individually, as well as for ``break``, ``continue`` and ``return`` in the tried code. The SSA analysis for after the statement is now the result of merging these different cases, should they not abort. - The code generation for `del` statements is now taking advantage should there be definite knowledge of previous value. This speed them up slightly. - The SSA analysis of `del` statements now properly decided if the statement can raise or not, allowing for more optimization. - For list contractions, the re-formulation was enhanced using the new outline construct instead of a pseudo function, leading to better analysis and code generation. - Comparison chains are now re-formulated into outlines too, allowing for better analysis of them. - Exceptions raised in function creations, e.g. in default values, are now propagated, eliminating the function's code. This happens most often with Python2/Python3 in branches. On the other hand, function creations that cannot are also annotated now. - Closure variables that become unreferenced outside of the function become normal variables leading to better tracing and code generation for them. - Function creations cannot raise except their defaults, keyword defaults or annotations do. - Built-in references can now be converted to strings at compile time, e.g. when printed. Organizational -------------- - Removed gitorious mirror of the git repository, they shut down. - Make it more clear in the documentation that Python2 is needed at compile time to create Python3 executables. Cleanups -------- - Moved more parts of code generation to their own modules, and used registry for code generation for more expression kinds. - Unified ``try``/``except`` and ``try``/``finally`` into a single construct that handles both through ``try``/``except``/``break``/``continue``/``return`` semantics. Finally is now solved via duplicating the handler into cases necessary. No longer are nodes annotated with information if they need to publish the exception or not, this is now all done with the dedicated nodes. - The ``try``/``finally`` expressions have been replaced with outline function bodies, that instead of side effect statements, are more like functions with return values, allowing for easier analysis and dedicated code generation of much lower complexity. - No more "tolerant" flag for release nodes, we now decide this fully based on SSA information. - Added helper for assertions that code flow does not reach certain positions, e.g. a function must return or raise, aborting statements do not continue and so on. - To keep cloning of code parts as simple as possible, the limited use of ``makeCloneAt`` has been changed to a new ``makeClone`` which produces identical copies, which is what we always do. And a generic cloning based on "details" has been added, requiring to make constructor arguments and details complete and consistent. - The re-formulation code helpers have been improved to be more convenient at creating nodes. - The old ``nuitka.codegen`` module ``Generator`` was still used for many things. These now all got moved to appropriate code generation modules, and their users got updated, also moving some code generator functions in the process. - The module ``nuitka.codegen.CodeTemplates`` got replaces with direct uses of the proper topic module from ``nuitka.codegen.templates``, with some more added, and their names harmonized to be more easily recognizable. - Added more assertions to the generated code, to aid bug finding. - The autoformat now sorts pylint markups for increased consistency. - Releases no longer have a ``tolerant`` flag, this was not needed anymore as we use SSA. - Handle CTRL-C in scons code preventing per job messages that are not helpful and avoid tracebacks from scons, also remove more unused tools like ``rpm`` from out in-line copy. Tests ----- - Added the CPython3.4 test suite. - The CPython3.2, CPython3.3, and CPython3.4 test suite now run with Python2 giving the same errors. Previously there were a few specific errors, some with line numbers, some with different ``SyntaxError`` be raised, due to different order of checks. This increases the coverage of the exception raising tests somewhat. - Also the CPython3.x test suites now all pass with debug Python, as does the CPython 2.6 test suite with 2.6 now. - Added tests to cover all forms of unpacking assignments supported in Python3, to be sure there are no other errors unknown to us. - Started to document the reference count tests, and to make it more robust against SSA optimization. This will take some time and is work in progress. - Made the compile library test robust against modules that raise a syntax error, checking that Nuitka does the same. - Refined more tests to be directly execuable with Python3, this is an ongoing effort. Summary ------- This release is clearly major. It represents a huge step forward for Nuitka as it improves nearly every aspect of code generation and analysis. Removing the ``try``/``finally`` expression nodes proved to be necessary in order to even have the correct SSA in their cases. Very important optimization was blocked by it. Going forward, the ``try``/``finally`` statements will be removed and dead variable elimination will happen, which then will give function inlining. This is expected to happen in one of the next releases. This release is a consolidation of 8 hotfix releases, and many refactorings needed towards the next big step, which might also break things, and for that reason is going to get its own release cycle. Nuitka Release 0.5.13 ===================== This release contains the first use of SSA for value propagation and massive amounts of bug fixes and optimization. Some of the bugs that were delivered as hotfixes, were only revealed when doing the value propagation as they still could apply to real code. Bug Fixes --------- - Fix, relative imports in packages were not working with absolute imports enabled via future flags. Fixed in 0.5.12.1 already. - Loops were not properly degrading knowledge from inside the loop at loop exit, and therefore this could have lead missing checks and releases in code generation for cases, for ``del`` statements in the loop body. Fixed in 0.5.12.1 already. - The ``or`` and ``and`` re-formulation could trigger false assertions, due to early releases for compatibility. Fixed in 0.5.12.1 already. - Fix, optimizion of calls of constant objects (always an exception), crashed the compiler. This corrects `Issue#202 `__. Fixed in 0.5.12.2 already. - Standalone: Added support for ``site.py`` installations with a leading ``def`` or ``class`` statement, which is defeating our attempt to patch ``__file__`` for it. This corrects `Issue#189 `__. - Compatibility: In full compatibility mode, the tracebacks of ``or`` and ``and`` expressions are now as wrong as they are in CPython. Does not apply to ``--improved`` mode. - Standalone: Added missing dependency on ``QtGui`` by ``QtWidgets`` for PyQt5. - MacOS: Improved parsing of ``otool`` output to avoid duplicate entries, which can also be entirely wrong in the case of Qt plugins at least. - Avoid relative paths for main program with file reference mode ``original``, as it otherwise changes as the file moves. - MinGW: The created modules depended on MinGW to be in ``PATH`` for their usage. This is no longer necessary, as we now link these libraries statically for modules too. - Windows: For modules, the option ``--run`` to immediately load the modules had been broken for a while. - Standalone: Ignore Windows DLLs that were attempted to be loaded, but then failed to load. This happens e.g. when both PySide and PyQt are installed, and could cause the dreaded conflicting DLLs message. The DLL loaded in error is now ignored, which avoids this. - MinGW: The resource file used might be empty, in which case it doesn't get created, avoiding an error due to that. - MinGW: Modules can now be created again. The run time relative code uses an API that is WinXP only, and MinGW failed to find it without guidance. Optimization ------------ - Make direct calls out of called function creations. Initially this applies to lambda functions only, but it's expected to become common place in coming releases. This is now 20x faster than CPython. .. code-block:: python # Nuitka avoids creating a function object, parsing function arguments: (lambda x:x)(something) - Propagate assignments from non-mutable constants forward based on SSA information. This is the first step of using SSA for real compile time optimization. - Specialized the creation of call nodes at creation, avoiding to have all kinds be the most flexible form (keyword and plain arguments), but instead only what kind of call they really are. This saves lots of memory, and makes the tree faster to visit. - Added support for optimizing the ``slice`` built-in with compile time constant arguments to constants. The re-formulation for slices in Python3 uses these a lot. And the lack of this optimization prevented a bunch of optimization in this area. For Python2 the built-in is optimized too, but not as important probably. - Added support for optimizing ``isinstance`` calls with compile time constant arguments. This avoids static exception raises in the ``exec`` re-formulation which tests for ``file`` type, and then optimization couldn't tell that a ``str`` is not a ``file`` instance. Now it can. - Lower in-place operations on immutable types to normal operations. This will allow to compile time compute these more accurately. - The re-formulation of loops puts the loop condition as a conditional statement with break. The ``not`` that needs to apply was only added in later optimization, leading to unnecessary compile time efforts. - Removed per variable trace visit from optimization, removing useless code and compile time overhead. We are going to optimize things by making decision in assignment and reference nodes based on forward looking statements using the last trace collection. New Features ------------ - Added experimental support for Python 3.5, which seems to be passing the test suites just fine. The new ``@`` matrix multiplicator operators are not yet supported though. - Added support for patching source on the fly. This is used to work around a (now fixed) issue with ``numexpr.cpuinfo`` making type checks with the ``is`` operation, about the only thing we cannot detect. Organizational -------------- - Added repository for Ubuntu Vivid (15.04) for download. Removed Ubuntu Saucy and Ubuntu Raring package downloads, these are no longer supported by Ubuntu. - Added repository for Debian Stretch, after Jessie release. - Make it more clear in the documentation that in order to compile Python3, a Python2 is needed to execute Scons, but that the end result is a Python3 binary. - The PyLint checker tool now can operate on directories given on the command line, and whitelists an error that is Windows only. Cleanups -------- - Split up standalone code further, moving ``depends.exe`` handling to a separate module. - Reduced code complexity of scons interface. - Cleaned up where trace collection is being done. It was partially still done inside the collection itself instead in the owner. - In case of conflicting DLLs for standalone mode, these are now output with nicer formatting, that makes it easy to recognize what is going on. - Moved code to fetch ``depends.exe`` to dedicated module, so it's not as much in the way of standalone code. Tests ----- - Made ``BuiltinsTest`` directly executable with Python3. - Added construct test to demonstrate the speed up of direct lambda calls. - The deletion of ``@test`` for the CPython test suite is more robust now, esp. on Windows, the symbolic links are now handled. - Added test to cover ``or`` usage with in-place assignment. - Cover local relative ``import from .`` with ``absolute_import`` future flag enabled. - Again, more basic tests are now directly executable with Python3. Summary ------- This release is major due to amount of ground covered. The reduction in memory usage of Nuitka itself (the C++ compiler will still use much memory) is very massive and an important aspect of scalability too. Then the SSA changes are truly the first sign of major improvements to come. In their current form, without eliminating dead assignments, the full advantage is not taken yet, but the next releases will do this, and that's a major milestone to Nuitka. The other optimization mostly stem from looking at things closer, and trying to work towards function in-lining, for which we are making a lot of progress now. Nuitka Release 0.5.12 ===================== This release contains massive amounts of corrections for long standing issues in the import recursion mechanism, as well as for standalone issues now visible after the ``__file__`` and ``__path__`` values have changed to become runtime dependent values. Bug Fixes --------- - Fix, the ``__path__`` attribute for packages was still the original filename's directory, even in file reference mode was ``runtime``. - The use of ``runtime`` as default file reference mode for executables, even if not in standalone mode, was making acceleration harder than necessary. Changed to ``original`` for that case. Fixed in 0.5.11.1 already. - The constant value for the smallest ``int`` that is not yet a ``long`` is created using ``1`` due to C compiler limitations, but ``1`` was not yet initialized properly, if this was a global constant, i.e. used in multiple modules. Fixed in 0.5.11.2 already. - Standalone: Recent fixes around ``__path__`` revealed issues with PyWin32, where modules from ``win32com.shell`` were not properly recursed to. Fixed in 0.5.11.2 already. - The importing of modules with the same name as a built-in module inside a package falsely assumed these were the built-ins which need not exist, and then didn't recurse into them. This affected standalone mode the most, as the module was then missing entirely. This corrects `Issue#178 `__. .. code-block:: python # Inside "x.y" module: import x.y.exceptions - Similarily, the importing of modules with the same name as standard library modules could go wrong. This corrects `Issue#184 `__. .. code-block:: python # Inside "x.y" module: import x.y.types - Importing modules on Windows and MacOS was not properly checking the checking the case, making it associate wrong modules from files with mismatching case. This corrects `Issue#188 `__. - Standalone: Importing with ``from __future__ import absolute_import`` would prefer relative imports still. This corrects `Issue#187 `__. - Python3: Code generation for ``try``/``return expr``/``finally`` could loose exceptions when ``expr`` raised an exception, leading to a ``RuntimeError`` for ``NULL`` return value. The real exception was lost. - Lambda expressions that were directly called with star arguments caused the compiler to crash. .. code-block:: python (lambda *args:args)(*args) # was crashing Nuitka New Optimization ---------------- - Focusing on compile time memory usage, cyclic dependencies of trace merges that prevented them from being released, even when replaced were removed. - More memory efficient updating of global SSA traces, reducing memory usage during optimization by ca. 50%. - Code paths that cannot and therefore must not happen are now more clearly indicated to the backend compiler, allowing for slightly better code to be generated by it, as it can tell that certain code flows need not be merged. New Features ------------ - Standalone: On systems, where ``.pth`` files inject Python packages at launch, these are now detected, and taking into account. Previously Nuitka did not recognize them, due to lack of ``__init__.py`` files. These are mostly pip installations of e.g. ``zope.interface``. - Added option ``--explain-imports`` to debug the import resolution code of Nuitka. - Added options ``--show-memory`` to display the amount of memory used in total and how it's spread across the different node types during compilation. - The option ``--trace-execution`` now also covers early program initialisation before any Python code runs, to ease finding bugs in this domain as well. Organizational -------------- - Changed default for file reference mode to ``original`` unless standalone or module mode are used. For mere acceleration, breaking the reading of data files from ``__file__`` is useless. - Added check that the in-line copy of scons is not run with Python3, which is not supported. Nuitka works fine with Python3, but a Python2 is required to execute scons. - Discover more kinds of Python2 installations on Linux/MacOS installations. - Added instructions for MacOS to the download page. Cleanups -------- - Moved ``oset`` and ``odict`` modules which provide ordered sets and dictionaries into a new package ``nuitka.container`` to clean up the top level scope. - Moved ``SyntaxErrors`` to ``nuitka.tree`` package, where it is used to format error messages. - Moved ``nuitka.Utils`` package to ``nuitka.utils.Utils`` creating a whole package for utils, so as to better structure them for their purpose. Summary ------- This release is a major maintenance release. Support for namespace modules injected by ``*.pth`` is a major step for new compatibility. The import logic improvements expand the ability of standalone mode widely. Many more use cases will now work out of the box, and less errors will be found on case insensitive systems. There is aside of memory issues, no new optimization though as many of these improvements could not be delivered as hotfixes (too invasive code changes), and should be out to the users as a stable release. Real optimization changes have been postponed to be next release. Nuitka Release 0.5.11 ===================== The last release represented a significant change and introduced a few regressions, which got addressed with hot fix releases. But it also had a focus on cleaning up open optimization issues that were postponed in the last release. New Features ------------ - The filenames of source files as found in the ``__file__`` attribute are now made relative for all modes, not just standalone mode. This makes it possible to put data files along side compiled modules in a deployment. This solves `Issue#170 `__. Bug Fixes --------- - Local functions that reference themselves were not released. They now are. .. code-block:: python def someFunction(): def f(): f() # referencing 'f' in 'f' caused the garbage collection to fail. Recent changes to code generation attached closure variable values to the function object, so now they can be properly visited. This corrects `Issue#45 `__. Fixed in 0.5.10.1 already. - Python2.6: The complex constants with real or imaginary parts ``-0.0`` were collapsed with constants of value ``0.0``. This became more evident after we started to optimize the ``complex`` built-in. Fixed in 0.5.10.1 already. .. code-block:: python complex(0.0, 0.0) complex(-0.0, -0.0) # Could be confused with the above. - Complex call helpers could leak references to their arguments. This was a regression. Fixed in 0.5.10.1 already. - Parameter variables offered as closure variables were not properly released, only the cell object was, but not the value. This was a regression. Fixed in 0.5.10.1 already. - Compatibility: The exception type given when accessing local variable values not initialized in a closure taking function, needs to be ``NameError`` and ``UnboundLocalError`` for accesses in the providing function. Fixed in 0.5.10.1 already. - Fix support for "venv" on systems, where the system Python uses symbolic links too. This is the case on at least on Mageia Linux. Fixed in 0.5.10.2 already. - Python3.4: On systems where ``long`` and ``Py_ssize_t`` are different (e.g. Win64) iterators could be corrupted if used by uncompiled Python code. Fixed in 0.5.10.2 already. - Fix, generator objects didn't release weak references to them properly. Fixed in 0.5.10.2 already. - Compatiblity: The ``__closure__`` attributes of functions was so far not supported, and rarely missing. Recent changes made it easy to expose, so now it was added. This corrects `Issue#45 `__. - MacOS: A linker warning about deprecated linker option ``-s`` was solved by removing the option. - Compatibility: Nuitka was enforcing that the ``__doc__`` attribute to be a string object, and gave a misleading error message. This check must not be done though, ``__doc__`` can be any type in Python. This corrects `Issue#177 `__. New Optimization ---------------- - Variables that need not be shared, because the uses in closure taking functions were eliminated, no longer use cell objects. - The ``try``/``except`` and ``try``/``finally`` statements now both have actual merging for SSA, allowing for better optimization of code behind it. .. code-block:: python def f(): try: a = something() except: return 2 # Since the above exception handling cannot continue the code flow, # we do not have to invalidate the trace of "a", and e.g. do not have # to generate code to check if it's assigned. return a Since ``try``/``finally`` is used in almost all re-formulations of complex Python constructs this is improving SSA application widely. The uses of ``try``/``except`` in user code will no longer degrade optimization and code generation efficiency as much as they did. - The ``try``/``except`` statement now reduces the scope of tried block if possible. When no statement raised, already the handling was removed, but leading and trailing statements that cannot raise, were not considered. .. code-block:: python def f(): try: b = 1 a = something() c = 1 except: return 2 This is now optimized to. .. code-block:: python def f(): b = 1 try: a = something() except: return 2 c = 1 The impact may on execution speed may be marginal, but it is definitely going to improve the branch merging to be added later. Note that ``c`` can only be optimized, because the exception handler is aborting, otherwise it would change behaviour. - The creation of code objects for standalone mode and now all code objects was creating a distinct filename object for every function in a module, despite them being same content. This was wasteful for module loading. Now it's done only once. Also, when having multiple modules, the code to build the run time filename used for code objects, was calling import logic, and doing lookups to find ``os.path.join`` again and again. These are now cached, speeding up the use of many modules as well. Cleanups -------- - Nuitka used to have "variable usage profiles" and still used them to decide if a global variable is written to, in which case, it stays away from doing optimization of it to built-in lookups, and later calls. The have been replaced by "global variable traces", which collect the traces to a variable across all modules and functions. While this is now only a replacement, and getting rid of old code, and basing on SSA, later it will also allow to become more correct and more optimized. - The standalone now queries its hidden dependencies from a plugin framework, which will become an interface to Nuitka internals in the future. Testing ------- - The use of deep hashing of constants allows us to check if constants become mutated during the run-time of a program. This allows to discover corruption should we encounter it. - The tests of CPython are now also run with Python in debug mode, but only on Linux, enhancing reference leak coverage. - The CPython test parts which had been disabled due to reference cycles involving compiled functions, or usage of ``__closure__`` attribute, were reactivated. Organizational -------------- - Since Google Code has shutdown, it has been removed from the Nuitka git mirrors. Summary ------- This release brings exciting new optimization with the focus on the ``try`` constructs, now being done more optimal. It is also a maintenance release, bringing out compatibility improvements, and important bug fixes, and important usability features for the deployment of modules and packages, that further expand the use cases of Nuitka. The git flow had to be applied this time to get out fixes for regression bug fixes, that the big change of the last release brought, so this is also to consolidate these and the other corrections into a full release before making more invasive changes. The cleanups are leading the way to expanded SSA applied to global variable and shared variable values as well. Already the built-in detect is now based on global SSA information, which was an important step ahead. Nuitka Release 0.5.10 ===================== This release has a focus on code generation optimization. Doing major changes away from "C++-ish" code to "C-ish" code, many constructs are now faster or got looked at and optimized. Bug Fixes --------- - Compatibility: The variable name in locals for the iterator provided to the generator expression should be ``.0``, now it is. - Generators could leak frames until program exit, these are now properly freed immediately. New Optimization ---------------- - Faster exception save and restore functions that might be in-lined by the backend C compiler. - Faster error checks for many operations, where these errors are expected, e.g. instance attribute lookups. - Do not create traceback and locals dictionary for frame when ``StopIteration`` or ``GeneratorExit`` are raised. These tracebacks were wasted, as they were immediately released afterwards. - Closure variables to functions and parameters of generator functions are now attached to the function and generator objects. - The creation of functions with closure taking was accelerated. - The creation and destruction of generator objects was accelerated. - The re-formulation for in-place assignments got simplified and got faster doing so. - In-place operations of ``str`` were always copying the string, even if was not necessary. This corrects `Issue#124 `__. .. code-block:: python a += b # Was not re-using the storage of "a" in case of strings - Python2: Additions of ``int`` for Python2 are now even faster. - Access to local variable values got slightly accelerated at the expense of closure variables. - Added support for optimizing the ``complex`` built-in. - Removing unused temporary and local variables as a result of optimization, these previously still allocated storage. Cleanup ------- - The use of C++ classes for variable objects was removed. Closure variables are now attached as ``PyCellObject`` to the function objects owning them. - The use of C++ context classes for closure taking and generator parameters has been replaced with attaching values directly to functions and generator objects. - The indentation of code template instantiations spanning multiple was not in all cases proper. We were using emission objects that handle it new lines in code and mere ``list`` objects, that don't handle them in mixed forms. Now only the emission objects are used. - Some templates with C++ helper functions that had no variables got changed to be properly formatted templates. - The internal API for handling of exceptions is now more consistent and used more efficiently. - The printing helpers got cleaned up and moved to static code, removing any need for forward declaration. - The use of ``INCREASE_REFCOUNT_X`` was removed, it got replaced with proper ``Py_XINCREF`` usages. The function was once required before "C-ish" lifted the need to do everything in one function call. - The use of ``INCREASE_REFCOUNT`` got reduced. See above for why that is any good. The idea is that ``Py_INCREF`` must be good enough, and that we want to avoid the C function it was, even if in-lined. - The ``assertObject`` function that checks if an object is not ``NULL`` and has positive reference count, i.e. is sane, got turned into a preprocessor macro. - Deep hashes of constant values created in ``--debug`` mode, which cover also mutable values, and attempt to depend on actual content. These are checked at program exit for corruption. This may help uncover bugs. Organizational -------------- - Speedcenter has been enhanced with better graphing and has more benchmarks now. More work will be needed to make it useful. - Updates to the Developer Manual, reflecting the current near finished state of "C-ish" code generation. Tests ----- - New reference count tests to cover generator expressions and their usage got added. - Many new construct based tests got added, these will be used for performance graphing, and serve as micro benchmarks now. - Again, more basic tests are directly executable with Python3. Summary ------- This is the next evolution of "C-ish" coming to pass. The use of C++ has for all practical purposes vanished. It will remain an ongoing activity to clear that up and become real C. The C++ classes were a huge road block to many things, that now will become simpler. One example of these were in-place operations, which now can be dealt with easily. Also, lots of polishing and tweaking was done while adding construct benchmarks that were made to check the impact of these changes. Here, generators probably stand out the most, as some of the missed optimization got revealed and then addressed. Their speed increases will be visible to some programs that depend a lot on generators. This release is clearly major in that the most important issues got addressed, future releases will provide more tuning and completeness, but structurally the "C-ish" migration has succeeded, and now we can reap the benefits in the coming releases. More work will be needed for all in-place operations to be accelerated. More work will be needed to complete this, but it's good that this is coming to an end, so we can focus on SSA based optimization for the major gains to be had. Nuitka Release 0.5.9 ==================== This release is mostly a maintenance release, bringing out minor compatibility improvements, and some standalone improvements. Also new options to control the recursion into modules are added. Bug Fixes --------- - Compatibility: Checks for iterators were using ``PyIter_Check`` which is buggy when running outside of Python core, because it's comparing pointers we don't see. Replaced with ``HAS_ITERNEXT`` helper which compares against the pointer as extracting for a real non-iterator object. .. code-block:: python class Iterable: def __init__(self): self.consumed = 2 def __iter__(self): return Iterable() iter(Iterable()) # This is suppose to raise, but didn't with Nuitka - Python3: Errors when creating class dictionaries raised by the ``__prepare__`` dictionary (e.g. ``enum`` classes with wrong identifiers) were not immediately raised, but only by the ``type`` call. This was not observable, but might have caused issues potentially. - Standalone MacOS: Shared libraries and extension modules didn't have their DLL load paths updated, but only the main binary. This is not sufficient for more complex programs. - Standalone Linux: Shared libraries copied into the ``.dist`` folder were read-only and executing ``chrpath`` could potentially then fail. This has not been observed, but is a conclusion of MacOS fix. - Standalone: When freezing standard library, the path of Nuitka and the current directory remained in the search path, which could lead to looking at the wrong files. Organizational -------------- - The ``getattr`` built-in is now optimized for compile time constants if possible, even in the presence of a ``default`` argument. This is more a cleanup than actually useful yet. - The calling of ``PyCFunction`` from normal Python extension modules got accelerated, especially for the no or single argument cases where Nuitka now avoids building the tuple. New Features ------------ - Added the option ``--recurse-pattern`` to include modules per filename, which for Python3 is the only way to not have them in a package automatically. - Added the option ``--generate-c++-only`` to only generate the C++ source code without starting the compiler. Mostly used for debugging and testing coverage. In the later case we do not want the C++ compiler to create any binary, but only to measure what would have been used. Organizational -------------- - Renamed the debug option ``--c++-only`` to ``--recompile-c++-only`` to make its purpose more clear and there now is ``--generate-c++-only`` too. Tests ----- - Added support for taking coverage of Nuitka in a test run on a given input file. - Added support for taking coverage for all Nuitka test runners, migrating them all to common code for searching. - Added uniform way of reporting skipped tests, not generally used yet. Summary ------- This release marks progress towards having coverage testing. Recent releases had made it clear that not all code of Nuitka is actually used at least once in our release tests. We aim at identifying these. Another direction was to catch cases, where Nuitka leaks exceptions or is subject to leaked exceptions, which revealed previously unnoticed errors. Important changes have been delayed, e.g. the closure variables will not yet use C++ objects to share storage, but proper ``PyCellObject`` for improved compatibility, and to approach a more "C-ish" status. These is unfinished code that does this. And the forward propagation of values is not enabled yet again either. So this is an interim step to get the bug fixes and improvements accumulated out. Expect more actual changes in the next releases. Nuitka Release 0.5.8 ==================== This release has mainly a focus on cleanups and compatibility improvements. It also advances standalone support, and a few optimization improvements, but it mostly is a maintenance release, attacking long standing issues. Bug Fixes --------- - Compatibility Windows MacOS: Fix importing on case insensitive systems. It was not always working properly, if there was both a package ``Something`` and ``something``, by merit of having files ``Something/__init__.py`` and ``something.py``. - Standalone: The search path was preferring system directories and therefore could have conflicting DLLs. `Issue#144 `__. - Fix, the optimization of ``getattr`` with predictable result was crashing the compilation. This was a regression, fixed in 0.5.7.1 already. - Compatibility: The name mangling inside classes also needs to be applied to global variables. - Fix, proving ``clang++`` for ``CXX`` was mistakingly thinking of it as a ``g++`` and making version checks on it. - Python3: Declaring ``__class__`` global is now a ``SyntaxError`` before Python3.4. - Standalone Python3: Making use of module state in extension modules was not working properly. New Features ------------ - The filenames of source files as found in the ``__file__`` attribute are now made relative in standalone mode. This should make it more apparent if things outside of the distribution folder are used, at the cost of tracebacks. Expect the default ability to copy the source code along in an upcoming release. - Added experimental standalone mode support for PyQt5. At least headless mode should be working, plug-ins (needed for anything graphical) are not yet copied and will need more work. Cleanup ------- - No longer using ``imp.find_module`` anymore. To solve the casing issues we needed to make our own module finding implementation finally. - The name mangling was handled during code generation only. Moved to tree building instead. - More code generation cleanups. The compatible line numbers are now attached during tree building and therefore better preserved, as well as that code no longer polluting code generation as much. Organizational -------------- - No more packages for openSUSE 12.1/12.2/12.3 and Fedora 17/18/19 as requested by the openSUSE Build Service. - Added RPM packages for Fedora 21 and CentOS 7 on openSUSE Build Service. Tests ----- - Lots of test refinements for the CPython test suites to be run continuously in Buildbot for both Windows and Linux. Summary ------- This release brings about two major changes, each with the risk to break things. One is that we finally started to have our own import logic, which has the risk to cause breakage, but apparently currently rather improved compatibility. The case issues were not fixable with standard library code. The second one is that the ``__file__`` attributes for standalone mode is now no longer pointing to the original install and therefore will expose missing stuff sooner. This will have to be followed up with code to scan for missing "data" files later on. For SSA based optimization, there are cleanups in here, esp. the one removing the name mangling, allowing to remove special code for class variables. This makes the SSA tree more reliable. Hope is that the big step (forward propagation through variables) can be made in one of the next releases. Nuitka Release 0.5.7 ==================== This release is brings a newly supported platform, bug fixes, and again lots of cleanups. Bug Fixes --------- - Fix, creation of dictionary and set literals with non-hashable indexes did not raise an exception. .. code-block:: python {[]: None} # This is now a TypeError New Optimization ---------------- - Calls to the ``dict`` built-in with only keyword arguments are now optimized to mere dictionary creations. This is new for the case of non-constant arguments only of course. .. code-block:: python dict(a = b, c = d) # equivalent to {"a" : b, "c" : d} - Slice ``del`` with indexable arguments are now using optimized code that avoids Python objects too. This was already done for slice look-ups. - Added support for ``bytearray`` built-in. Organizational -------------- - Added support for OpenBSD with fiber implementation from library, as it has no context support. Cleanups -------- - Moved slicing solutions for Python3 to the re-formulation stage. So far the slice nodes were used, but only at code generation time, there was made a distinction between Python2 and Python3 for them. Now these nodes are purely Python2 and slice objects are used universally for Python3. Tests ----- - The test runners now have common code to scan for the first file to compile, an implementation of the ``search`` mode. This will allow to introduce the ability to search for pattern matches, etc. - More tests are directly executable with Python3. - Added ``recurse_none`` mode to test comparison, making using extra options for that purpose unnecessary. Summary ------- This solves long standing issues with slicing and subscript not being properly distinguished in the Nuitka code. It also contains major bug fixes that really problematic. Due to the involved nature of these fixes they are made in this new release. Nuitka Release 0.5.6 ==================== This release brings bug fixes, important new optimization, newly supported platforms, and important compatibility improvements. Progress on all fronts. Bug Fixes --------- - Closure taking of global variables in member functions of classes that had a class variable of the same name was binding to the class variable as opposed to the module variable. - Overwriting compiled function's ``__doc__`` attribute more than once could corrupt the old value, leading to crashes. `Issue#156 `__. Fixed in 0.5.5.2 already. - Compatibility Python2: The ``exec`` statement ``execfile`` were changing ``locals()`` was given as an argument. .. code-block:: python def function(): a = 1 exec code in locals() # Cannot change local "a". exec code in None # Can change local "a" exec code Previously Nuitka treated all 3 variants the same. - Compatibility: Empty branches with a condition were reduced to only the condition, but they need in fact to also check the truth value: .. code-block:: python if condition: pass # must be treated as bool(condition) # and not (bug) condition - Detection of Windows virtualenv was not working properly. Fixed in 0.5.5.2 already. - Large enough constants structures are now unstreamed via ``marshal`` module, avoiding large codes being generated with no point. Fixed in 0.5.5.2 already. - Windows: Pressing CTRL-C gave two stack traces, one from the re-execution of Nuitka which was rather pointless. Fixed in 0.5.5.1 already. - Windows: Searching for virtualenv environments didn't terminate in all cases. Fixed in 0.5.5.1 already. - During installation from PyPI with Python3 versions, there were errors given for the Python2 only scons files. `Issue#153 `__. Fixed in 0.5.5.3 already. - Fix, the arguments of ``yield from`` expressions could be leaked. - Fix, closure taking of a class variable could have in a sub class where the module variable was meant. .. code-block:: python var = 1 class C: var = 2 class D: def f(): # was C.var, now correctly addressed top level var return var - Fix, setting ``CXX`` environment variable because the installed gcc has too low version, wasn't affecting the version check at all. - Fix, on Debian/Ubuntu with ``hardening-wrapper`` installed the version check was always failing, because these report a shortened version number to Scons. New Optimization ---------------- - Local variables that must be assigned also have no side effects, making use of SSA. This allows for a host of optimization to be applied to them as well, often yielding simpler access/assign code, and discovering in more cases that frames are not necessary. - Micro optimization to ``dict`` built-in for simpler code generation. Organizational -------------- - Added support for ARM "hard float" architecture. - Added package for Ubuntu 14.10 for download. - Added package for openSUSE 13.2 for download. - Donations were used to buy a Cubox-i4 Pro. It got Debian Jessie installed on it, and will be used to run an even larger amount of tests. - Made it more clear in the user documentation that the ``.exe`` suffix is used for all platforms, and why. - Generally updated information in user manual and developer manual about the optimization status. - Using Nikola 7.1 with external filters instead of our own, outdated branch for the web site. Cleanups -------- - PyLint clean for the first time ever. We now have a Buildbot driven test that this stays that way. - Massive indentation cleanup of keyword argument calls. We have a rule to align the keywords, but as this was done manually, it could easily get out of touch. Now with a "autoformat" tool based on RedBaron, it's correct. Also, spacing around arguments is now automatically corrected. More to come. - For ``exec`` statements, the coping back to local variables is now an explicit node in the tree, leader to cleaner code generation, as it now uses normal variable assignment code generation. - The ``MaybeLocalVariables`` became explicit about which variable they might be, and contribute to its SSA trace as well, which was incomplete before. - Removed some cases of code duplication that were marked as TODO items. This often resulted in cleanups. - Do not use ``replaceWith`` on child nodes, that potentially were re-used during their computation. Summary ------- The release is mainly the result of consolidation work. While the previous release contained many important enhancements, this is another important step towards full SSA, closing one loop whole (class variables and ``exec`` functions), as well as applying it to local variables, largely extending its use. The amount of cleanups is tremendous, in huge part due to infrastructure problems that prevented release repeatedly. This reduces the technological debt very much. More importantly, it would appear that now eliminating local and temporary variables that are not necessary is only a small step away. But as usual, while this may be easy to implement now, it will uncover more bugs in existing code, that we need to address before we continue. Nuitka Release 0.5.5 ==================== This release is finally making full use of SSA analysis knowledge for code generation, leading to many enhancements over previous releases. It also adds support for Python3.4, which has been longer in the making, due to many rather subtle issues. In fact, even more work will be needed to fully solve remaining minor issues, but these should affect no real code. And then there is much improved support for using standalone mode together with virtualenv. This combination was not previously supported, but should work now. New Features ------------ - Added support for Python3.4 This means support for ``clear`` method of frames to close generators, dynamic ``__qualname__``, affected by ``global`` statements, tuples as ``yield from`` arguments, improved error messages, additional checks, and many more detail changes. New Optimization ---------------- - Using SSA knowledge, local variable assignments now no longer need to check if they need to release previous values, they know definitely for the most cases. .. code-block:: python def f(): a = 1 # This used to check if old value of "a" needs a release ... - Using SSA knowledge, local variable references now no longer need to check for raising exceptions, let alone produce exceptions for cases, where that cannot be. .. code-block:: python def f(): a = 1 return a # This used to check if "a" is assigned - Using SSA knowledge, local variable references now are known if they can raise the ``UnboundLocalError`` exception or not. This allows to eliminate frame usages for many cases. Including the above example. - Using less memory for keeping variable information. - Also using less memory for constant nodes. Bug Fixes --------- - The standalone freezing code was reading Python source as UTF-8 and not using the code that handles the Python encoding properly. On some platforms there are files in standard library that are not encoded like that. - The fiber implementation for Linux amd64 was not working with glibc from RHEL 5. Fixed to use now multiple ``int`` to pass pointers as necessary. Also use ``uintptr_t`` instead of ``intprt_t`` to transport pointers, which may be more optimal. - Line numbers for exceptions were corrupted by ``with`` statements due to setting line numbers even for statements marked as internal. - Partial support for ``win32com`` by adding support for its hidden ``__path__`` change. - Python3: Finally figured out proper chaining of exceptions, given proper context messages for exception raised during the handling of exceptions. - Corrected C++ memory leak for each closure variable taken, each time a function object was created. - Python3: Raising exceptions with tracebacks already attached, wasn't using always them, but producing new ones instead. - Some constants could cause errors, as they cannot be handled with the ``marshal`` module as expected, e.g. ``(int,)``. - Standalone: Make sure to propagate ``sys.path`` to the Python instance used to check for standard library import dependencies. This is important for virtualenv environments, which need ``site.py`` to set the path, which is not executed in that mode. - Windows: Added support for different path layout there, so using virtualenv should work there too. - The code object flag "optimized" (fast locals as opposed to locals dictionary) for functions was set wrongly to value for the parent, but for frames inside it, one with the correct value. This lead to more code objects than necessary and false ``co_flags`` values attached to the function. - Options passed to ``nuitka-python`` could get lost. .. code-block:: sh nuitka-python program.py argument1 argument2 ... The above is supposed to compile program.py, execute it immediately and pass the arguments to it. But when Nuitka decides to restart itself, it would forget these options. It does so to e.g. disable hash randomization as it would affect code generation. - Raising tuples exception as exceptions was not compatible (Python2) or reference leaking (Python3). Tests ----- - Running ``2to3`` is now avoided for tests that are already running on both Python2 and Python3. - Made XML based optimization tests work with Python3 too. Previously these were only working on Python2. - Added support for ignoring messages that come from linking against self compiled Pythons. - Added test case for threaded generators that tortures the fiber layer a bit and exposed issues on RHEL 5. - Made reference count test of compiled functions generic. No more code duplication, and automatic detection of shared stuff. Also a more clear interface for disabling test cases. - Added Python2 specific reference counting tests, so the other cases can be executed with Python3 directly, making debugging them less tedious. Cleanups -------- - Really important removal of "variable references". They didn't solve any problem anymore, but their complexity was not helpful either. This allowed to make SSA usable finally, and removed a lot of code. - Removed special code generation for parameter variables, and their dedicated classes, no more needed, as every variable access code is now optimized like this. - Stop using C++ class methods at all. Now only the destructor of local variables is actually supposed to do anything, and their are no methods anymore. The unused ``var_name`` got removed, ``setVariableValue`` is now done manually. - Moved assertions for the fiber layer to a common place in the header, so they are executed on all platforms in debug mode. - As usual, also a bunch of cleanups for PyLint were applied. - The ``locals`` built-in code now uses code generation for accessing local variable values instead having its own stuff. Organizational -------------- - The Python version 3.4 is now officially supported. There are a few problems open, that will be addressed in future releases, none of which will affect normal people though. - Major cleanup of Nuitka options. - Windows specific stuff is now in a dedicated option group. This includes options for icon, disabling console, etc. - There is now a dedicated group for controlling backend compiler choices and options. - Also pickup ``g++44`` automatically, which makes using Nuitka on CentOS5 more automatic. Summary ------- This release represents a very important step ahead. Using SSA for real stuff will allow us to build the trust necessary to take the next steps. Using the SSA information, we could start implementing more optimizations. Nuitka Release 0.5.4 ==================== This release is aiming at preparatory changes to enable optimization based on SSA analysis, introducing a variable registry, so that variables no longer trace their references to themselves. Otherwise, MinGW64 support has been added, and lots of bug fixes were made to improve the compatibility. New Optimization ---------------- - Using new variable registry, now properly detecting actual need for sharing variables. Optimization may discover that it is unnecessary to share a variable, and then it no longer is. This also allows ``--debug`` without it reporting unused variable warnings on Python3. - Scons startup has been accelerated, removing scans for unused tools, and avoiding making more than one gcc version check. Bug Fixes --------- - Compatibility: In case of unknown encodings, Nuitka was not giving the name of the problematic encoding in the error message. Fixed in 0.5.3.3 already. - Submodules with the same name as built-in modules were wrongly shadowed. Fixed in 0.5.3.2 already. - Python3: Added implementations of ``is_package`` to the meta path based loader. - Python3.4: Added ``find_spec`` implementation to the meta path based loader for increased compatibility. - Python3: Corrections for ``--debug`` to work with Python3 and MSVC compiler more often. - Fixed crash with ``--show-scons`` when no compiler was found. Fixed in 0.5.3.5 already. - Standalone: Need to blacklist ``lib2to3`` from standard library as well. Fixed in 0.5.3.4 already. - Python3: Adapted to changes in ``SyntaxError`` on newer Python releases, there is now a ``msg`` that can override ``reason``. - Standalone Windows: Preserve ``sys.executable`` as it might be used to fork binaries. - Windows: The caching of Scons was not arch specific, and files could be used again, even if changing the arch from ```x86`` to ``x86_64`` or back. - Windows: On 32 bit Python it can happen that with large number of generators running concurrently (>1500), one cannot be started anymore. Raising an ``MemoryError`` now. Organizational -------------- - Added support for MinGW64. Currently needs to be run with ``PATH`` environment properly set up. - Updated internal version of Scons to 2.3.2, which breaks support for VS 2008, but adds support for VS 2013 and VS 2012. The VS 2013 is now the recommended compiler. - Added RPM package and repository for RHEL 7. - The output of ``--show-scons`` now includes the used compiler, including the MSVC version. - Added option ``--msvc`` to select the MSVC compiler version to use, which overrides automatic selection of the latest. - Added option ``-python-flag=no_warnings`` to disable user and deprecation warnings at run time. - Repository for Ubuntu Raring was removed, no more supported by Ubuntu. Cleanups -------- - Made technical and logical sharing decisions separate functions and implement them in a dedicated variable registry. - The Scons file has seen a major cleanup. Summary ------- This release is mostly a maintenance release. The Scons integrations has been heavily visited, as has been Python3 and esp. Python3.4 compatibility, and results from the now possible debug test runs. Standalone should be even more practical now, and MinGW64 is an option for those cases, where MSVC is too slow. Nuitka Release 0.5.3 ==================== This release is mostly a follow up, resolving points that have become possible to resolve after completing the C-ish evolution of Nuitka. So this is more of a service release. New Features ------------ - Improved mode ``--improved`` now sets error lines more properly than CPython does in many cases. - The ``-python-flag=-S`` mode now preserves ``PYTHONPATH`` and therefore became usable with virtualenv. New Optimization ---------------- - Line numbers of frames no longer get set unless an exception occurs, speeding up the normal path of execution. - For standalone mode, using ``--python-flag-S`` is now always possible and yields less module usage, resulting in smaller binaries and faster compilation. Bug Fixes --------- - Corrected an issue for frames being optimized away where in fact they are still necessary. `Issue#140 `__. Fixed in 0.5.2.1 already. - Fixed handling of exception tests as side effects. These could be remainders of optimization, but didn't have code generation. Fixed in 0.5.2.1 already. - Previously Nuitka only ever used the statement line as the line number for all the expression, even if it spawned multiple lines. Usually nothing important, and often even more correct, but sometimes not. Now the line number is most often the same as CPython in full compatibility mode, or better, see above. `Issue#9 `__. - Python3.4: Standalone mode for Windows is working now. - Standalone: Undo changes to ``PYTHONPATH`` or ``PYTHONHOME`` allowing potentially forked CPython programs to run properly. - Standalone: Fixed import error when using PyQt and Python3. New Tests --------- - For our testing approach, the improved line number handling means we can undo lots of changes that are no more necessary. - The compile library test has been extended to cover a third potential location where modules may live, covering the ``matplotlib`` module as a result. Cleanups -------- - In Python2, the list contractions used to be re-formulated to be function calls that have no frame stack entry of their own right. This required some special handling, in e.g. closure taking, and determining variable sharing across functions. This now got cleaned up to be properly in-lined in a ``try``/``finally`` expression. - The line number handling got simplified by pushing it into error exits only, removing the need to micro manage a line number stack which got removed. - Use ``intptr_t`` over ``unsigned long`` to store fiber code pointers, increasing portability. Organizational -------------- - Providing own Debian/Ubuntu repositories for all relevant distributions. - Windows MSI files for Python 3.4 were added. - Hosting of the web site was moved to metal server with more RAM and performance. Summary ------- This release brings about structural simplification that is both a follow-up to C-ish, as well as results from a failed attempt to remove static "variable references" and be fully SSA based. It incorporates changes aimed at making this next step in Nuitka evolution smaller. Nuitka Release 0.5.2 ==================== This is a major release, with huge changes to code generation that improve performance in a significant way. It is a the result of a long development period, and therefore contains a huge jump ahead. New Features ------------ - Added experimental support for Python 3.4, which is still work in progress. - Added support for virtualenv on MacOS. - Added support for virtualenv on Windows. - Added support for MacOS X standalone mode. - The code generation uses no header files anymore, therefore adding a module doesn't invalidate all compiled object files from caches anymore. - Constants code creation is now distributed, and constants referenced in a module are declared locally. This means that changing a module doesn't affect the validity of other modules object files from caches anymore. New Optimization ---------------- - C-ish code generation uses less C++ classes and generates more C-like code. Explicit temporary objects are now used for statement temporary variables. - The constants creation code is no more in a single file, but distributed across all modules, with only shared values created in a single file. This means improved scalability. There are remaining bad modules, but more often, standalone mode is now fast. - Exception handling no longer uses C++ exception, therefore has become much faster. - Loops that only break are eliminated. - Dead code after loops that do not break is now removed. - The ``try``/``finally`` and ``try``/``except`` constructs are now eliminated, where that is possible. - The ``try``/``finally`` part of the re-formulation for ``print`` statements is now only done when printing to a file, avoiding useless node tree bloat. - Tuples and lists are now generated with faster code. - Locals and global variables are now access with more direct code. - Added support for the anonymous ``code`` type built-in. - Added support for ``compile`` built-in. - Generators that statically return immediately, e.g. due to optimization results, are no longer using frame objects. - The complex call helpers use no pseudo frames anymore. Previous code generation required to have them, but with C-ish code generation that is no more necessary, speeding up those kind of calls. - Modules with only code that cannot raise, need not have a frame created for them. This avoids useless code size bloat because of them. Previously the frame stack entry was mandatory. Bug Fixes --------- - Windows: The resource files were cached by Scons and re-used, even if the input changed. The could lead to corrupted incremental builds. `Issue#129 `__. Fixed in 0.5.1.1 already. - Windows: For functions with too many local variables, the MSVC failed with an error "C1026: parser stack overflow, program too complex". The rewritten code generation doesn't burden the compiler as much. `Issue#127 `__. - Compatibility: The timing deletion of nested call arguments was different from C++. This shortcoming has been addressed in the rewritten code generation. `Issue#62 `__. - Compatibility: The ``__future__`` flags and ``CO_FREECELL`` were not present in frame flags. These were then not always properly inherited to ``eval`` and ``exec`` in all cases. - Compatibility: Compiled frames for Python3 had ``f_restricted`` attribute, which is Python2 only. Removed it. - Compatibility: The ``SyntaxError`` of having a ``continue`` in a finally clause is now properly raised. - Python2: The ``exec`` statement with no locals argument provided, was preventing list contractions to take closure variables. - Python2: Having the ASCII encoding declared in a module wasn't working. - Standalone: Included the ``idna`` encoding as well. `Issue#135 `__. - Standalone: For virtualenv, the file ``orig-prefix.txt`` needs to be present, now it's copied into the "dist" directory as well. `Issue#126 `__. Fixed in 0.5.1.1 already. - Windows: Handle cases, where Python and user program are installed on different volumes. - Compatibility: Can now finally use ``execfile`` as an expression. `Issue#5 `__ is finally fixed after all this time thanks to C-ish code generation. - Compatibility: The order or call arguments deletion is now finally compatible. `Issue#62 `__ also is finally fixed. This too is thanks to C-ish code generation. - Compatibility: Code object flags are now more compatible for Python3. - Standalone: Removing "rpath" settings of shared libraries and extension modules included. This makes standalone binaries more robust on Fedora 20. - Python2: Wasn't falsely rejecting ``unicode`` strings as values for ``int`` and ``long`` variants with base argument provided. - Windows: For Python3.2 and 64 bits, global variable accesses could give false ``NameError`` exceptions. Fixed in 0.5.1.6 already. - Compatibility: Many ``exec`` and ``eval`` details have become more correctly, the argument handling is more compatible, and e.g. future flags are now passed along properly. - Compatibility: Using ``open`` with no arguments is now giving the same error. Organizational -------------- - Replying to email from the `issue tracker `__ works now. - Added option name alias ``--xml`` for ``--dump-xml``. - Added option name alias ``--python-dbg`` for ``--python-debug``, which actually might make it a bit more clear that it is about using the CPython debug run time. - Remove option ``--dump-tree``, it had been broken for a long time and unused in favor of XML dumps. - New digital art folder with 3D version of Nuitka logo. Thanks to Juan Carlos for creating it. - Using "README.rst" instead of "README.txt" to make it look better on web pages. - More complete whitelisting of missing imports in standard library. These should give no warnings anymore. - Updated the Nuitka GUI to the latest version, with enhanced features. - The builds of releases and update of the `downloads page `__ is now driven by Buildbot. Page will be automatically updated as updated binaries arrive. Cleanups -------- - Temporary keeper variables and the nodes to handle them are now unified with normal temporary variables, greatly simplifying variable handling on that level. - Less code is coming from templates, more is actually derived from the node tree instead. - Releasing the references to temporary variables is now always explicit in the node tree. - The publishing and preservation of exceptions in frames was turned into explicit nodes. - Exception handling is now done with a single handle that checks with branches on the exception. This eliminates exception handler nodes. - The ``dir`` built-in with no arguments is now re-formulated to ``locals`` or ``globals`` with their ``.keys()`` attribute taken. - Dramatic amounts of cleanups to code generation specialties, that got done right for the new C-ish code generation. New Tests --------- - Warnings from MSVC are now error exits for ``--debug`` mode too, expanding the coverage of these tests. - The outputs with ``python-dbg`` can now also be compared, allowing to expand test coverage for reference counts. - Many of the basic tests are now executable with Python3 directly. This allows for easier debug. - The library compilation test is now also executed with Python3. Summary ------- This release would deserve more than a minor number increase. The C-ish code generation, is a huge body of work. In many ways, it lays ground to taking benefit of SSA results, that previously would not have been possible. In other ways, it's incomplete in not yet taking full advantage yet. The release contains so many improvements, that are not yet fully realized, but as a compiler, it also reflects a stable and improved state. The important changes are about making SSA even more viable. Many of the problematic cases, e.g. exception handlers, have been stream lined. A whole class of variables, temporary keepers, has been eliminated. This is big news in this domain. For the standalone users, there are lots of refinements. There is esp. a lot of work to create code that doesn't show scalability issues. While some remain, the most important problems have been dealt with. Others are still in the pipeline. More work will be needed to take full advantage. This has been explained in a `separate post `__ in greater detail. Nuitka Release 0.5.1 ==================== This release brings corrections and major improvements to how standalone mode performs. Much of it was contributed via patches and bug reports. Bug Fixes --------- - There was a crash when using ``next`` on a non-iterable. Fixed in 0.5.0.1 already. - Module names with special characters not allowed in C identifiers were not fully supported. `Issue#118 `__. Fixed in 0.5.0.1 already. - Name mangling for classes with leading underscores was not removing them from resulting attribute names. This broke at ``__slots__`` with private attributes for such classes. `Issue#119 `__. Fixed in 0.5.0.1 already. - Standalone on Windows might need "cp430" encoding. `Issue#120 `__. Fixed in 0.5.0.2 already. - Standalone mode didn't work with ``lxml.etree`` due to lack of hard coded dependencies. When a shared library imports things, Nuitka cannot detect it easily. - Wasn't working on MacOS 64 bits due to using Linux 64 bits specific code. `Issue#123 `__. Fixed in 0.5.0.2 already. - On MinGW the constants blob was not properly linked on some installations, this is now done differently (see below). New Features ------------ - Memory usages are now traced with ``--show-progress`` allowing us to trace where things go wrong. New Optimization ---------------- - Standalone mode now includes standard library as bytecode by default. This is workaround scalability issues with many constants from many modules. Future releases are going to undo it. - On Windows the constants blob is now stored as a resource, avoiding compilation via C code for MSVC as well. MinGW was changed to use the same code. New Tests --------- - Expanded test coverage for "standalone mode" demonstrating usage of "hex" encoding, PySide, and PyGtk packages. Summary ------- This release is mostly an interim maintenance release for standalone. Major changes that provide optimization beyond that, termed "C-ish code generation" are delayed for future releases. This release makes standalone practical which is an important point. Instead of hour long compilation, even for small programs, we are down to less than a minute. The solution of the scalability issues with many constants from many modules will be top priority going forward. Since they are about how even single use constants are created all in one place, this will be easy, but as large changes are happening in "C-ish code generation", we are waiting for these to complete. Nuitka Release 0.5.0 ==================== This release breaks interface compatibility, therefore the major version number change. Also "standalone mode" has seen significant improvements on both Windows, and Linux. Should work much better now. But consider that this part of Nuitka is still in its infancy. As it is not the top priority of mine for Nuitka, which primarily is intended as an super compatible accelerator of Python, it will continue to evolve nearby. There is also many new optimization based on structural improvements in the direction of actual SSA. Bug Fixes --------- - The "standalone mode" was not working on all Redhat, Fedora, and openSUSE platforms and gave warnings with older compilers. Fixed in 0.4.7.1 already. - The "standalone mode" was not including all useful encodings. `Issue#116 `__. Fixed in 0.4.7.2 already. - The "standalone mode" was defaulting to ``--python-flag=-S`` which disables the parsing of "site" module. That unfortunately made it necessary to reach some modules without modifying ``PYTHONPATH`` which conflicts with the "out-of-the-box" experience. - The "standalone mode" is now handling packages properly and generally working on Windows as well. - The syntax error of having an all catching except clause and then a more specific one wasn't causing a ``SyntaxError`` with Nuitka. .. code-block:: python try: something() except: somehandling(): except TypeError: notallowed() - A corruption bug was identified, when re-raising exceptions, the top entry of the traceback was modified after usage. Depending on ``malloc`` this was potentially causing an endless loop when using it for output. New Features ------------ - Windows: The "standalone" mode now properly detects used DLLs using `Dependency Walker `__ which it offers to download and extra for you. It is used as a replacement to ``ldd`` on Linux when building the binary, and as a replacement of ``strace`` on Linux when running the tests to check that nothing is loaded from the outside. New Optimization ---------------- - When iterating over ``list``, ``set``, this is now automatically lowered to ``tuples`` avoiding the mutable container types. So the following code is now equivalent: .. code-block:: python for x in [ a, b, c ]: ... # same as for x in (a, b, c): ... For constants, this is even more effective, because for mutable constants, no more is it necessary to make a copy. - Python2: The iteration of large ``range`` is now automatically lowered to ``xrange`` which is faster to loop over, and more memory efficient. - Added support for the ``xrange`` built-in. - The statement only expression optimization got generalized and now is capable of removing useless parts of operations, not only the whole thing when it has not side effects. .. code-block:: python [a,b] # same as a b This works for all container types. Another example is ``type`` built-in operation with single argument. When the result is not used, it need not be called. .. code-block:: python type(a) # same as a And another example ``is`` and ``is not`` have no effect of their own as well, therefore: .. code-block:: python a is b # same as a b - Added proper handling of conditional expression branches in SSA based optimization. So far these branches were ignored, which only acceptable for temporary variables as created by tree building, but not other variable types. This is preparatory for introducing SSA for local variables. Organizational -------------- - The option ``--exe`` is now ignored and creating an executable is the default behavior of ``nuitka``, a new option ``--module`` allows to produce extension modules. - The binary ``nuitka-python`` was removed, and is replaced by ``nuitka-run`` with now only implies ``--execute`` on top of what ``nuitka`` is. - Using dedicated `Buildbot `__ for continuous integration testing and release creation as well. - The `Downloads `__ now offers MSI files for Win64 as well. - Discontinued the support for cross compilation to Win32. That was too limited and the design choice is to have a running CPython instance of matching architecture at Nuitka compile time. New Tests --------- - Expanded test coverage for "standalone mode" demonstrating usage of "hex" encoding, and PySide package. Summary ------- The "executable by default" interface change improves on the already high ease of use. The new optimization do not give all that much in terms of numbers, but are all signs of structural improvements, and it is steadily approaching the point, where the really interesting stuff will happen. The progress for standalone mode is of course significant. It is still not quite there yet, but it is making quick progress now. This will attract a lot of attention hopefully. As for optimization, the focus for it has shifted to making exception handlers work optimal by default (publish the exception to sys.exc_info() and create traceback only when necessary) and be based on standard branches. Removing special handling of exception handlers, will be the next big step. This release includes some correctness fixes stemming from that work already. Nuitka Release 0.4.7 ==================== This release includes important new features, lots of polishing cleanups, and some important performance improvements as well. Bug Fixes --------- - The RPM packages didn't build due to missing in-line copy of Scons. Fixed in 0.4.6.1 already. - The recursion into modules and unfreezing them was not working for packages and modules anymore. Fixed in 0.4.6.2 already. - The Windows installer was not including Scons. Fixed in 0.4.6.3 already. - Windows: The immediate execution as performed by ``nuitka --execute`` was not preserving the exit code. `Issue#26 `__. - Python3.3: Packages without ``__init.py__`` were not properly embedding the name-space package as well. - Python3: Fix, modules and packages didn't add themselves to ``sys.modules`` which they should, happened only for programs. - Python3.3: Packages should set ``__package`` to their own name, not the one of their parents. - Python3.3: The ``__qualname__`` of nested classes was corrected. - For modules that recursed to other modules, an infinite loop could be triggered when comparing types with rich comparisons. `Issue#115 `__. New Features ------------ - The "standalone" mode allows to compile standalone binaries for programs and run them without Python installation. The DLLs loaded by extension modules on Windows need to be added manually, on Linux these are determined automatically already. To achieve running without Python installation, Nuitka learned to freeze bytecode as an alternative to compiling modules, as some modules need to be present when the CPython library is initialized. - New option ``--python-flag`` allows to specify flags to the compiler that the "python" binary normally would. So far ``-S`` and ``-v`` are supported, with sane aliases ``no_site`` and ``trace_imports``. The recommended use of ``--python-flag=-S`` is to avoid dependency creep in standalone mode compilations, because the ``site`` module often imports many useless things that often don't apply to target systems. New Optimization ---------------- - Faster frame stack handling for functions without ``try``/``except`` (or ``try``/``finally`` in Python3). This gives a speed boost to "PyStone" of ca. 2.5% overall. - Python2: Faster attribute getting and setting, handling special cases at compile time. This gives a minor speed boost to "PyStone" of ca. 0.5% overall. - Python2: Much quicker calls of ``__getattr__`` and ``__setattr__`` as this is now using the quicker call method avoiding temporary tuples. - Don't treat variables usages used in functions called directly by their owner as shared. This leads to more efficient code generation for contractions and class bodies. - Create ``unicode`` constants directly from their UTF-8 string representation for Python2 as well instead of un-streaming. So far this was only done for Python3. Affects only program start-up. - Directly create ``int`` and ``long`` constants outside of ``2**31`` and ``2**32-1``, but only limited according to actual platform values. Affects only program start-up. - When creating ``set`` values, no longer use a temporary ``tuple`` value, but use a properly generated helper functions instead. This makes creating sets much faster. - Directly create ``set`` constants instead of un-streaming them. Affects only program start-up. - For correct line numbers in traceback, the current frame line number must be updated during execution. This was done more often than necessary, e.g. loops set the line number before loop entry, and at first statement. - Module variables are now accessed even faster, the gain for "PyStone" is only 0.1% and mostly the result of leaner code. Organizational -------------- - The "standalone mode" code (formerly known as "portable mode" has been redone and activated. This is a feature that a lot of people expect from a compiler naturally. And although the overall goal for Nuitka is of course acceleration, this kind of packaging is one of the areas where CPython needs improvement. - Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04 and 11.10, no more supported. - Added package for openSUSE 13.1 for download. - Nuitka is now part of Arch and can be installed with ``pacman -S nuitka``. - Using dedicated `Buildbot `__ for continuous integration testing. Not yet public. - Windows: In order to speed up repeated compilation on a platform without ``ccache``, added Scons level caching in the build directory. - Disabled hash randomization for inside Nuitka (but not in ultimately created binaries) for a more stable output, because dictionary constants will not change around. This makes the build results possible to cache for ``ccache`` and Scons as well. Tests ----- - The ``programs`` tests cases now fail if module or directory recursion is not working, being executed in another directory. - Added test runner for packages, with initial test case for package with recursion and sub-packages. - Made some test cases more strict by reducing ``PYTHONPATH`` provision. - Detect use of extra flags in tests that don't get consumed avoiding ineffective flags. - Use ``--execute`` on Windows as well, the issue that prevented it has been solved after all. Cleanups -------- - The generated code uses ``const_``, ``var_``, ``par_`` prefixes in the generated code and centralized the decision about these into single place. - Module variables no longer use C++ classes for their access, but instead accessor functions, leading to much less code generated per module variable and removing the need to trace their usage during code generation. - The test runners now share common code in a dedicated module, previously they replicated it all, but that turned out to be too tedious. - Massive general cleanups, many of which came from new contributor Juan Carlos Paco. - Moved standalone and freezer related codes to dedicated package ``nuitka.freezer`` to not pollute the ``nuitka`` package name space. - The code generation use variable identifiers and their accesses was cleaned up. - Removed several not-so-special case identifier classes because they now behave more identical and all work the same way, so a parameters can be used to distinguish them. - Moved main program, function object, set related code generation to dedicated modules. Summary ------- This release marks major technological progress with the introduction of the much sought standalone mode and performance improvements from improved code generation. The major break through for SSA optimization was not yet achieved, but this is again making progress in the direction of it. Harmonizing variables of different kinds was an important step ahead. Also very nice is the packaging progress, Nuitka was accepted into Arch after being in Debian Testing for a while already. Hope is to see more of this kind of integration in the future. Nuitka Release 0.4.6 ==================== This release includes progress on all fronts. The primary focus was to advance SSA optimization over older optimization code that was already in place. In this domain, there are mostly cleanups. Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds an installed MSVC compiler automatically, properly handles architecture of Python and Windows. This improves usability a lot. Then this is also very much about bug fixes. There have been several hot fixes for the last release, but a complicated and major issue forced a new release, and many other small issues. And then there is performance. As can be seen in the `performance graph `__, this release is the fastest so far. This came mainly from examining the need for comparison slots for compiled types. And last, but not least, this also expands the base of supported platforms, adding Gentoo, and self compiled Python to the mix. Bug Fixes --------- - Support Nuitka being installed to a path that contains spaces and handle main programs with spaces in their paths. `Issue#106 `__. Fixed in 0.4.5.1 already. - Support Python being installed to a path that contains spaces. `Issue#106 `__. Fixed in 0.4.5.2 already. - Windows: User provided constants larger than 65k didn't work with MSVC. `Issue#108 `__. Fixed in 0.4.5.3 already. - Windows: The option ``--windows-disable-console`` was not effective with MSVC. `Issue#107 `__. Fixed in 0.4.5.3 already. - Windows: For some users, Scons was detecting their MSVC installation properly already from registry, but it didn't honor the target architecture. `Issue#99 `__. Fixed in 0.4.5.3 already. - When creating Python modules, these were marked as executable ("x" bit), which they are of course not. Fixed in 0.4.5.3 already. - Python3.3: On architectures where ``Py_ssize_t`` is not the same as ``long`` this could lead to errors. Fixed in 0.4.5.3 already. - Code that was using nested mutable constants and changed the nested ones was not executing correctly. `Issue#112 `__. - Python2: Due to list contractions being re-formulated as functions, ``del`` was rejected for the variables assigned in the contraction. `Issue#111 `__. .. code-block:: python [ expr(x) for x in iterable() ] del x # Should work, was gave an unjustified SyntaxError. New Features ------------ - Compiled types when used in Python comparison now work. Code like this will work: .. code-block:: python def f(): pass assert type(f) == types.FunctionType This of course also works for ``in`` operator, and is another step ahead in compatibility, and surprising too. And best of all, this works even if the checking code is not compiled with Nuitka. - Windows: Detecting MSVC installation from registry, if no compiler is already present in PATH. - Windows: Now options ``--mingw`` to force compilation with MinGW. New Optimization ---------------- - Rich comparisons (``==``, ``<``, and the like) are now faster than ever before due to a full implementation of its own in Nuitka that eliminates a bit of the overhead. In the future, we will aim at giving it type hints to make it even faster. This gives a minor speed boost to PyStone of ca. 0.7% overall. - Integer comparisons are now treated preferably, as they are in CPython, which gives 1.3% speed boost to CPython. - The SSA based analysis is now used to provide variable scopes for temporary variables as well as reference count needs. Cleanups -------- - Replaced "value friend" based optimization code with SSA based optimization, which allowed to remove complicated and old code that was still used mainly in optimization of ``or`` and ``and`` expressions. - Delayed declaration of temp variables and their reference type is now performed based on information from SSA, which may given more accurate results. Not using "variable usage" profiles for this anymore. - The Scons interface and related code got a massive overhaul, making it more consistent and better documented. Also updated the internal copy to 2.3.0 for the platforms that use it, mostly Windows. - Stop using ``os.system`` and ``subprocess.call(..., shell = True)`` as it is not really portable at all, use ``subprocess.call(..., shell = False)`` instead. - As usual lots of cleanups related to line length issues and PyLint. Organizational -------------- - Added support for Gentoo Linux. - Added support for self compiled Python versions with and without debug enabled. `Issue#110 `__ - Added use of Nuitka fonts for headers in manuals. - Does not install in-line copy of Scons only on systems where it is not going to be used, that is mostly non-Windows, and Linux where it is not already present. This makes for cleaner RPM packages. Summary ------- While the SSA stuff is not yet bearing performance fruits, it starts to carry weight. Taking over the temporary variable handling now also means we can apply the same stuff to local variables later. To make up for the delay in SSA driven performance improvements, there is more traditional code acceleration for rich comparisons, making it significant, and the bug fixes make Nuitka more compatible than ever. So give this a roll, it's worth it. And feel free to `join the mailing list `_ or `make a donation `__ to support Nuitka. Nuitka Release 0.4.5 ==================== This release incorporates very many bug fixes, most of which were already part of hot fixes, usability improvements, documentation improvements, new logo, simpler Python3 on Windows, warnings for recursion options, and so on. So it's mostly a consolidation release. Bug Fixes --------- - When targeting Python 3.x, Nuitka was using "python" to run Scons to run it under Python 2.x, which is not good enough on systems, where that is already Python3. Improved to only do the guessing where necessary (i.e. when using the in-line copy of Scons) and then to prefer "python2". `Issue#95 `__. Fixed in 0.4.4.1 already. - When using Nuitka created binaries inside a "virtualenv", created programs would instantly crash. The attempt to load and patch ``inspect`` module was not making sure that ``site`` module was already imported, but inside the "virtualenv", it cannot be found unless. `Issue#96 `__. Fixed in 0.4.4.1 already. - The option ``--recurse-directory`` to include plugin directories was broken. `Issue#97 `__. Fixed in 0.4.4.2 already. - Python3: Files with "BOM" marker causes the compiler to crash. `Issue#98 `__. Fixed in 0.4.4.2 already. - Windows: The generated code for ``try``/``return``/``finally`` was working with gcc (and therefore MinGW), but not with MSVC, causing crashes. `Issue#102 `__. Fixed in 0.4.4.2 already. - The option ``--recurse-all`` did not recurse to package ``__init__.py`` files in case ``from x.y import z`` syntax was used. `Issue#100 `__. Fixed in 0.4.4.2 already. - Python3 on MacOS: Corrected link time error. Fixed in 0.4.4.2 already. - Python3.3 on Windows: Fixed crash with too many arguments to a kwonly argument using function. Fixed in 0.4.4.2 already. - Python3.3 on Windows: Using "yield from" resulted in a link time error. Fixed in 0.4.4.2 already. - Windows: Added back XML manifest, found a case where it is needed to prevent clashes with binary modules. - Windows: Generators only worked in the main Python threads. Some unusual threading modules therefore failed. - Using ``sys.prefix`` to find the Python installation instead of hard coded paths. `Issue#103 `__. New Features ------------ - Windows: Python3 finds Python2 installation to run Scons automatically now. Nuitka itself runs under Python3 just fine, but in order to build the generated C++ code into binaries, it uses Scons which still needs Python2. Nuitka will now find the Python2 installation searching Windows registry instead of requiring hard coded paths. - Windows: Python2 and Python3 find their headers now even if Python is not installed to specific paths. The installation path now is passed on to Scons which then uses it. - Better error checking for ``--recurse-to`` and ``--recurse-not-to`` arguments, tell the user not to use directory paths. - Added a warning for ``--recurse-to`` arguments that end up having no effect to the final result. Cleanups -------- - Import mechanism got cleaned up, stopped using "PyImport_ExtendInittab". It does not handle packages, and the ``sys.meta_path`` based importer is now well proven. - Moved some of the constraint collection code mess into proper places. It still remains a mess. Organizational -------------- - Added ``LICENSE.txt`` file with Apache License 2.0 text to make it more immediately obvious which license Nuitka is under. - Added section about Nuitka license to the "`User Manual `__". - Added `Nuitka Logo `__ to the distribution. - Use Nuitka Logo as the bitmap in the Windows installer. - Use Nuitka Logo in the documentation ("`User Manual `__" and "`Developer Manual `__"). - Enhanced documentation to number page numbers starting after table of contents, removed header/footer from cover pages. Summary ------- This release is mostly the result of improvements made based on the surge of users after Europython 2013. Some people went to extents and reported their experience very detailed, and so I could aim at making e.g. their misconceptions about how recursion options work, more obvious through warnings and errors. This release is not addressing performance improvements. The next release will be able to focus on that. I am taking my claim of full compatibility very serious, so any time it's broken, it's the highest priority to restore it. Nuitka Release 0.4.4 ==================== This release marks the point, where Nuitka for the first time supports all major current Python versions and all major features. It adds Python 3.3 support and it adds support for threading. And then there is a massive amount of fixes that improve compatibility even further. Aside of that, there is major performance work. One side is the optimization of call performance (to CPython non-compiled functions) and to compiled functions, both. This gave a serious improvement to performance. Then of course, we are making other, long term performance progress, as in "--experimental" mode, the SSA code starts to optimize unused code away. That code is not yet ready for prime time yet, but the trace structure will hold. New Features ------------ - Python3.3 support. The test suite of CPython3.3 passes now too. The ``yield from`` is now supported, but the improved argument parsing error messages are not implemented yet. - Tracing user provided constants, now Nuitka warns about too large constants produced during optimization. - Line numbers of expressions are now updates as evaluation progresses. This almost corrects. Finally improves `Issue#9 `__. Now only expression parts that cannot raise, do not update, which can still cause difference, but much less often, and then definitely useless. - Experimental support for threads. Threading appears to work just fine in the most cases. It's not as optimal as I wanted it to be, but that's going to change with time. New Optimization ---------------- - Previous corrections for ``==``, ``!=``, and ``<=``, caused a performance regression for these operations in case of handling identical objects. For built-in objects of sane types (not ``float``), these operations are now accelerated again. The overreaching acceleration of ``>=`` was still there (bug, see below) and has been adapted too. - Calling non-compiled Python functions from compiled functions was slower than in CPython. It is now just as fast. - Calling compiled functions without keyword arguments has been accelerated with a dedicated entry point that may call the implementation directly and avoid parameter parsing almost entirely. - Making calls to compiled and non-compiled Python functions no longer requires to build a temporary tuple and therefore is much faster. - Parameter parsing code is now more compact, and re-uses error raises, or creates them on the fly, instead of hard coding it. Saves binary size and should be more cache friendly. Bug Fixes --------- - Corrected false optimization of ``a >= a`` on C++ level. When it's not done during Nuitka compile time optimization, the rich comparison helper still contained short cuts for ``>=``. This is now the same for all the comparison operators. - Calling a function with default values, not providing it, and not providing a value for a value without default, was not properly detecting the error, and instead causing a run time crash. .. code-block:: python def f(a, b = 2): pass f(b = 2) This now properly raises the ``TypeError`` exception. - Constants created with ``+`` could become larger than the normally enforced limits. Not as likely to become huge, but still potentially an issue. - The ``vars`` built-in, when used on something without ``__dict__`` attribute, was giving ``AttributeError`` instead of ``TypeError``. - When re-cursing to modules at compile time, script directory and current directory were used last, while at run time, it was the other way around, which caused overloaded standard library modules to not be embedded. Corrects `Issue#94 `__. Thanks for the patch to James Michael DuPont. - Super without arguments was not raising the correct ``RuntimeError`` exception in functions that cannot be methods, but ``UnboundLocalError`` instead. .. code-block:: python def f(): super() # Error, cannot refer to first argument of f - Generators no longer use ``raise StopIteration`` for return statements, because that one is not properly handled in ``try``/``except`` clauses, where it's not supposed to trigger, while ``try``/``finally`` should be honored. - Exception error message when throwing non-exceptions into generators was not compatible. - The use of ``return`` with value in generators is a ``SyntaxError`` before Python3.3, but that was not raised. - Variable names of the "__var" style need to be mangled. This was only done for classes, but not for functions contained in classes, there they are now mangled too. - Python3: Exceptions raised with causes were not properly chaining. - Python3: Specifying the file encoding corrupted line numbers, making them all of by one. Cleanups -------- - For containers (``tuple``, ``list``, ``set``, ``dict``) defined on the source code level, Nuitka immediately created constant references from them. For function calls, class creations, slice objects, this code is now re-used, and its dictionaries and tuples, may now become constants immediately, reducing noise in optimization steps. - The parameter parsing code got cleaned up. There were a lot of relics from previously explored paths. And error raises were part of the templates, but now are external code. - Global variable management moved to module objects and out of "Variables" module. - Make sure, nodes in the tree are not shared by accident. This helped to find a case of duplicate use in the complex call helpers functions. Code generation will now notice this kind of duplication in debug mode. - The complex call helper functions were manually taking variable closure, which made these functions inconsistent to other functions, e.g. no variable version was allocated to assignments. Removing the manual setting of variables allowed a huge reduction of code volume, as it became more generic code. - Converting user provided constants to create containers into constants immediately, to avoid noise from doing this in optimization. - The ``site`` module is now imported explicitly in the ``__main__`` module, so it can be handled by the recursion code as well. This will help portable mode. - Many line length 80 changes, improved comments. New Tests --------- - The CPython3.3 test suite was added, and run with both Python3.2 and Python3.3, finding new bugs. - The ``doctest`` to code generation didn't successfully handle all tests, most notably, "test_generators.py" was giving a ``SyntaxError`` and therefore not actually active. Correcting that improved the coverage of generator testing. Organizational -------------- - The portable code is still delayed. Support for Python3.3 was a higher priority, but the intention is to get it into shape for Europython still. Added notes about it being disabled it in the "`User Manual `__" documentation. Summary ------- This release is in preparation for Europython 2013. Wanted to get this much out, as it changes the status slides quite a bit, and all of that was mostly done in my Cyprus holiday a while ago. The portable code has not seen progress. The idea here is to get this into a development version later. Nuitka Release 0.4.3 ==================== This release expands the reach of Nuitka substantially, as new platforms and compilers are now supported. A lot of polish has been applied. Under the hood there is the continued and in-progress effort to implement SSA form in Nuitka. New Features ------------ - Support for new compiler: Microsoft Visual C++. You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows. - Support for NetBSD. Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic "fibers" implementation. - Support for Python3 under Windows too. Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to ``C:\Python26`` or ``C:\Python27`` for Nuitka to be able to find it. - Enhanced Python 3.3 compatibility. The support the newest version of Python has been extended, improving compatibility for many minor corner cases. - Added warning when a user compiles a module and executes it immediately when that references ``__name__``. Because very likely the intention was to create an executable. And esp. if there is code like this: .. code-block:: python if __name__ == "__main__": main() In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command .. code-block:: sh nuitka --execute module is behavioral more like python -c "import module" and that was a trap for new users. - All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don't have to implement for specific architectures anymore. Bug Fixes --------- - Dictionary creation was not fully compatible. As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn't observe this before, and it's likely the new dictionary implementation that finds it. Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing. - Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was. - Using a ``__future__`` import inside a function was giving an assertion, instead of the proper syntax error. - Python3: Do not set the attributes ``sys.exc_type``, ``sys.exc_value``, ``sys.exc_traceback``. - Python3: Annotations of function worked only as long as their definition was not referring to local variables. New Optimization ---------------- - Calls with no positional arguments are now using the faster call methods. The generated C++ code was using the ``()`` constant at call site, when doing calls that use no positional arguments, which is of course useless. - For Windows now uses OS "Fibers" for Nuitka "Fibers". Using threads for fibers was causing only overhead and with this API, MSVC had less issues too. Organizational -------------- - Accepting `Donations `__ via Paypal, please support funding travels, website, etc. - The "`User Manual `__" has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated. - The Changelog has been improved for consistency, wordings, and styles. - Nuitka is now available on the social code platforms as well * `Bitbucket `__ * `Github `__ * `Gitorious `__ * `Google Code `__ - Removed "clean-up.sh", which is practically useless, as tests now clean up after themselves reasonably, and with ``git clean -dfx`` working better. - Removed "create-environment.sh" script, which was only setting the ``PATH`` variable, which is not necessary. - Added ``check-with-pylint --emacs`` option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint. - Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated. Cleanups -------- - Removed useless manifest and resource file creation under Windows. Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it. - PyLint massive cleanups and annotations bringing down the number of warnings by a lot. - Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes. - Do not track needed tuple, list, and dict creation code variants in context, but e.g. in ``nuitka.codegen.TupleCodes`` module instead. - Introduced an "internal" module to host the complex call helper functions, instead of just adding it to any module that first uses it. New Tests --------- - Added basic tests for order evaluation, where there currently were None. - Added support for "2to3" execution under Windows too, so we can run tests for Python3 installations too. Summary ------- The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now. Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization. Nuitka Release 0.4.2 ==================== This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the `performance diagrams `__ got expanded. New Features ------------ - Support for FreeBSD. Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some "Linuxisms" in the build process. - New option for warning about compile time detected exception raises. Nuitka can now warn about exceptions that will be raised at run time. - Basic Python3.3 support. The test suite of CPython3.2 passes and fails in a compatible way. New feature ``yield from`` is not yet supported, and the improved argument parsing error messages are not implemented yet. Bug Fixes --------- - Nuitka already supported compilation of "main directories", i.e. directories with a "__main__.py" file inside. The resulting binary name was "__main__.exe" though, but now it is "directory.exe" .. code-block:: sh # ls directory __main__.py # nuitka --exe directory # ls directory directory.exe This makes this usage more obvious, and fixes the older issue `Issue#49 `__ for this feature. - Evaluation order of binary operators was not enforced. Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems. Corrects `Issue#61 `__. - Providing an ``# coding: no-exist`` was crashing under Python2, and ignored under Python3, now it does the compatible thing for both. - Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are. .. code-block:: python global a # Not in a function, but on module level. Pointless but legal! a = 1 Effectively these statements can be ignored. Corrects part of `Issue#65 `__. - Future imports are only legal when they are at the start of the file. This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error. Corrects part of `Issue#65 `__. - Raising exceptions from context was leaking references. .. code-block:: python raise ValueError() from None Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments. Corrects `Issue#76 `__. - Importing the module that became ``__main__`` through the module name, didn't recurse to it. This also gives a warning. PyBench does it, and then stumbles over the non-found "pybench" module. Of course, programmers should use ``sys.modules[ "__main__" ]`` to access main module code. Not only because the duplicated modules don't share data. Corrects `Issue#68 `__. - Compiled method ``repr`` leaked references when printed. When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects ``__del__`` to run at some point. Corrects `Issue#81 `__. - The ``super`` built-in leaked references to given object. This was added, because Python3 needs it. It supplies the arguments to ``super`` automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing ``__del__`` to run. Corrects `Issue#81 `__. - The ``raise`` statement didn't enforce type of third argument. This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it's None (which is also legal), no reference to None is leaked. - Python3 built-in exceptions were strings instead of exceptions. A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn't help their usage of course, fixed. Corrects `Issue#82 `__. - The ``-nan`` and ``nan`` both exist and make a difference. A older story continued. There is a sign to ``nan``, which can be copied away and should be present. This is now also supported by Nuitka. Corrects `Issue#75 `__. - Wrong optimization of ``a == a``, ``a != a``, ``a <= a`` on C++ level. While it's not done during Nuitka optimization, the rich comparison helpers still contained short cuts for ``==``, ``!=``, and ``<=``. - The ``sys.executable`` for ``nuitka-python --python-version 3.2`` was still ``python``. When determining the value for ``sys.executable`` the CPython library code looks at the name ``exec`` had received. It was ``python`` in all cases, but now it depends on the running version, so it propagates. - Keyword only functions with default values were loosing references to defaults. .. code-block:: python def f(*, a = X()) pass f() f() # Can crash, X() should already be released. This is now corrected. Of course, a Python3 only issue. - Pressing CTRL-C didn't generate ``KeyboardInterrupt`` in compiled code. Nuitka never executes "pending calls". It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this. - For ``with`` statements with ``return``, ``break``, or ``continue`` to leave their body, the ``__exit__`` was not called. .. code-block:: python with a: # This called a.__enter__(). return 2 # This didn't call a.__exit__(None, None, None). This is of course quite huge, and unfortunately wasn't covered by any test suite so far. Turns out, the re-formulation of ``with`` statements, was wrongly using ``try/except/else``, but these ignore the problematic statements. Only ``try/finally`` does. The enhanced re-formulation now does the correct thing. Corrects `Issue#59 `__. - Starting with Python3, absolute imports are now the default. This was already present for Python3.3, and it turns out that all of Python3 does it. New Optimization ---------------- - Constants are now much less often created with ``pickle`` module, but created directly. This esp. applies for nested constants, now more values become ``is`` identical instead of only ``==`` identical, which indicates a reduced memory usage. .. code-block:: python a = ("something_special",) b = "something_special" assert a[0] is b # Now true This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the "==" operator will be able to shortcut dramatically in cases of identical objects. Constants now created without ``pickle`` usage, cover ``float``, ``list``, and ``dict``, which is enough for PyStone to not use it at all, which has been added support for as well. - Continue statements might be optimized away. A terminal ``continue`` in a loop, was not optimized away: .. code-block:: python while 1: something continue # Now optimized away The trailing ``continue`` has no effect and can therefore be removed. .. code-block:: python while 1: something - Loops with only break statements are optimized away. .. code-block:: python while 1: break A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate "if ... : break" checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed. New Tests --------- - Added tests for the found issues. - Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered. - Running the "CPython3.2" test suite with Python3.3 based Nuitka works and found a few minor issues. Organizational -------------- - The `Downloads `__ page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and "ownssh" for contributing an RPM spec file. The page got improved with logos for the distributions. - Added "ownssh" as contributor. - Revamped the "`User Manual `__" in terms of layout, structure, and content. Summary ------- This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around. Nuitka Release 0.4.1 ==================== This release is the first follow-up with a focus on optimization. The major highlight is progress towards SSA form in the node tree. Also a lot of cleanups have been performed, for both the tree building, which is now considered mostly finished, and will be only reviewed. And for the optimization part there have been large amounts of changes. New Features ------------ - Python 3.3 experimental support * Now compiles many basic tests. Ported the dictionary quick access and update code to a more generic and useful interface. * Added support for ``__qualname__`` to classes and functions. * Small compatibility changes. Some exceptions changed, absolute imports are now default, etc. * For comparison tests, the hash randomization is disabled. - Python 3.2 support has been expanded. The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code. Bug fixes --------- - Default values were not "is" identical. .. code-block:: python def defaultKeepsIdentity(arg = "str_value"): print arg is "str_value" defaultKeepsIdentity() This now prints "True" as it does with CPython. The solution is actually a general code optimization, see below. `Issue#55 `__ - Usage of ``unicode`` built-in with more than one argument could corrupt the encoding argument string. An implementation error of the ``unicode`` was releasing references to arguments converted to default encoding, which could corrupt it. - Assigning Python3 function annotations could cause a segmentation fault. New Optimization ---------------- - Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization. .. code-block:: python # Now optimizing: raise TypeError, 1/0 # into (minus normalization): raise ZeroDivisionError, "integer division or modulo by zero" # Now optimizing: (1/0).something # into (minus normalization): raise ZeroDivisionError, "integer division or modulo by zero" # Now optimizing: function(a, 1/0).something # into (minus normalization), notice the side effects of first checking # function and a as names to be defined, these may be removed only if # they can be demonstrated to have no effect. function a raise ZeroDivisionError, "integer division or modulo by zero" There is more examples, where the raise propagation is new, but you get the idea. - Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things. .. code-block:: python # This was already optimized, as it's a compile time constant. a if ("a",) else b a if True else b # These are now optimized, as their truth value is known. a if (c,) else b a if not (c,) else b This is simply taking advantage of infrastructure that now exists. Each node kind can overload "getTruthValue" and benefit from it. Help would be welcome to review which ones can be added. - Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused. - Code generation for constants now shares element values used in tuples. The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up. Cleanups -------- - Optimization was initially designed around visitors that each did one thing, and did it well. It turns out though, that this approach is unnecessary, and constraint collection, allows for the most consistent results. All remaining optimization has been merged into constraint collection. - The names of modules containing node classes were harmonized to always be plural. In the beginning, this was used to convey the information that only a single node kind would be contained, but that has long changed, and is unimportant information. - The class names of nodes were stripped from the "CPython" prefix. Originally the intent was to express strict correlation to CPython, but with increasing amounts of re-formulations, this was not used at all, and it's also not important enough to dominate the class name. - The re-formulations performed in tree building have moved out of the "Building" module, into names "ReformulationClasses" e.g., so they are easier to locate and review. Helpers for node building are now in a separate module, and generally it's much easier to find the content of interest now. - Added new re-formulation of ``print`` statements. The conversion to strings is now made explicit in the node tree. New Tests --------- - Added test to cover default value identity. Organizational -------------- - The upload of `Nuitka to PyPI `__ has been repaired and now properly displays project information again. Summary ------- The quicker release is mostly a consolidation effort, without actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization. Nuitka Release 0.4.0 ==================== This release brings massive progress on all fronts. The big highlight is of course: Full Python3.2 support. With this release, the test suite of CPython3.2 is considered passing when compiled with Nuitka. Then lots of work on optimization and infrastructure. The major goal of this release was to get in shape for actual optimization. This is also why for the first time, it is tested that some things are indeed compile time optimized to spot regressions easier. And we are having performance diagrams, `even if weak ones `__: New Features ------------ - Python3.2 is now fully supported. - Fully correct ``metaclass =`` semantics now correctly supported. It had been working somewhat previously, but now all the corner cases are covered too. - Keyword only parameters. - Annotations of functions return value and their arguments. - Exception causes, chaining, automatic deletion of exception handlers ``as`` values. - Added support for starred assigns. - Unicode variable names are also supported, although it's of course ugly, to find a way to translate these to C++ ones. Bug fixes --------- - Checking compiled code with ``instance(some_function, types.FunctionType)`` as "zope.interfaces" does, was causing compatibility problems. Now this kind of check passes for compiled functions too. `Issue#53 `__ - The frame of modules had an empty locals dictionary, which is not compatible to CPython which puts the globals dictionary there too. Also discussed in `Issue#53 `__ - For nested exceptions and interactions with generator objects, the exceptions in "sys.exc_info()" were not always fully compatible. They now are. - The ``range`` builtin was not raising exceptions if given arguments appeared to not have side effects, but were still illegal, e.g. ``range([], 1, -1)`` was optimized away if the value was not used. - Don't crash on imported modules with syntax errors. Instead, the attempted recursion is simply not done. - Doing a ``del`` on ``__defaults`` and ``__module__`` of compiled functions was crashing. This was noticed by a Python3 test for ``__kwdefaults__`` that exposed this compiled functions weakness. - Wasn't detecting duplicate arguments, if one of them was not a plain arguments. Star arguments could collide with normal ones. - The ``__doc__`` of classes is now only set, where it was in fact specified. Otherwise it only polluted the name space of ``locals()``. - When ``return`` from the tried statements of a ``try/finally`` block, was overridden, by the final block, a reference was leaked. Example code: .. code-block:: python try: return 1 finally: return 2 - Raising exception instances with value, was leaking references, and not raising the ``TypeError`` error it is supposed to do. - When raising with multiple arguments, the evaluation order of them was not enforced, it now is. This fixes a reference leak when raising exceptions, where building the exception was raising an exception. New Optimization ---------------- - Optimizing attribute access to compile time constants for the first time. The old registry had no actual user yet. - Optimizing subscript and slices for all compile time constants beyond constant values, made easy by using inheritance. - Built-in references now convert to strings directly, e.g. when used in a print statement. Needed for the testing approach "compiled file contains only prints with constant value". - Optimizing calls to constant nodes directly into exceptions. - Optimizing built-in ``bool`` for arguments with known truth value. This would be creations of tuples, lists, and dictionaries. - Optimizing ``a is b`` and ``a is not b`` based on aliasing interface, which at this time effectively is limited to telling that ``a is a`` is true and ``a is not a`` is false, but this will expand. - Added support for optimizing ``hasattr``, ``getattr``, and ``setattr`` built-ins as well. The ``hasattr`` was needed for the ``class`` re-formulation of Python3 anyway. - Optimizing ``getattr`` with string argument and no default to simple attribute access. - Added support for optimizing ``isinstance`` built-in. - Was handling "BreakException" and "ContinueException" in all loops that used ``break`` or ``continue`` instead of only where necessary. - When catching "ReturnValueException", was raising an exception where a normal return was sufficient. Raising them now only where needed, which also means, function need not catch them ever. Cleanups -------- - The handling of classes for Python2 and Python3 have been re-formulated in Python more completely. * The calling of the determined "metaclass" is now in the node tree, so this call may possible to in-line in the future. This eliminated some static C++ code. * Passing of values into dictionary creation function is no longer using hard coded special parameters, but temporary variables can now have closure references, making this normal and visible to the optimization. * Class dictionary creation functions are therefore no longer as special as they used to be. * There is no class creation node anymore, it's merely a call to ``type`` or the metaclass detected. - Re-formulated complex calls through helper functions that process the star list and dict arguments and do merges, checks, etc. * Moves much C++ code into the node tree visibility. * Will allow optimization to eliminate checks and to compile time merge, once in-line functions and loop unrolling are supported. - Added "return None" to function bodies without a an aborting statement at the end, and removed the hard coded fallback from function templates. Makes it explicit in the node tree and available for optimization. - Merged C++ classes for frame exception keeper with frame guards. * The exception is now saved in the compiled frame object, making it potentially more compatible to start with. * Aligned module and function frame guard usage, now using the same class. * There is now a clear difference in the frame guard classes. One is for generators and one is for functions, allowing to implement their different exception behavior there. - The optimization registries for calls, subscripts, slices, and attributes have been replaced with attaching them to nodes. * The ensuing circular dependency has been resolved by more local imports for created nodes. * The package "nuitka.transform.optimization.registries" is no more. * New per node methods "computeNodeCall", "computeNodeSubscript", etc. dispatch the optimization process to the nodes directly. - Use the standard frame guard code generation for modules too. * Added a variant "once", that avoids caching of frames entirely. - The variable closure taking has been cleaned up. * Stages are now properly numbered. * Python3 only stage is not executed for Python2 anymore. * Added comments explaining things a bit better. * Now an early step done directly after building a tree. - The special code generation used for unpacking from iterators and catching "StopIteration" was cleaned up. * Now uses template, Generator functions, and proper identifiers. - The ``return`` statements in generators are now re-formulated into ``raise StopIteration`` for generators, because that's what they really are. Allowed to remove special handling of ``return`` nodes in generators. - The specialty of CPython2.6 yielding non-None values of lambda generators, was so far implemented in code generation. This was moved to tree building as a re-formulation, making it subject to normal optimization. - Mangling of attribute names in functions contained in classes, has been moved into the early tree building. So far it was done during code generation, making it invisible to the optimization stages. - Removed tags attribute from node classes. This was once intended to make up for non-inheritance of similar node kinds, but since we have function references, the structure got so clean, it's no more needed. - Introduced new package ``nuitka.tree``, where the building of node trees, and operations on them live, as well as recursion and variable closure. - Removed ``nuitka.transform`` and move its former children ``nuitka.optimization`` and ``nuitka.finalization`` one level up. The deeply nested structure turned out to have no advantage. - Checks for Python version was sometimes "> 300", where of course ">= 300" is the only thing that makes sense. - Split out helper code for exception raising from the handling of exception objects. New Tests --------- - The complete CPython3.2 test suite was adapted (no ``__code__``, no ``__closure__``, etc.) and is now passing, but only without "--debug", because otherwise some of the generated C++ triggers (harmless) warnings. - Added new test suite designed to prove that expressions that are known to be compile time constant are indeed so. This works using the XML output done with "--dump-xml" and then searching it to only have print statements with constant values. - Added new basic CPython3.2 test "Functions32" and "ParameterErrors32" to cover keyword only parameter handling. - Added tests to cover generator object and exception interactions. - Added tests to cover ``try/finally`` and ``return`` in one or both branches correctly handling the references. - Added tests to cover evaluation order of arguments when raising exceptions. Organizational -------------- - Changed my email from GMX over to Gmail, the old one will still continue to work. Updated the copyright notices accordingly. - Uploaded `Nuitka to PyPI `__ as well. Summary ------- This release marks a milestone. The support of Python3 is here. The re-formulation of complex calls, and the code generation improvements are quite huge. More re-formulation could be done for argument parsing, but generally this is now mostly complete. The 0.3.x series had a lot releases. Many of which brought progress with re-formulations that aimed at making optimization easier or possible. Sometimes small things like making "return None" explicit. Sometimes bigger things, like making class creations normal functions, or getting rid of ``or`` and ``and``. All of this was important ground work, to make sure, that optimization doesn't deal with complex stuff. So, the 0.4.x series begins with this. The focus from now on can be almost purely optimization. This release contains already some of it, with frames being optimized away, with the assignment keepers from the ``or`` and ``and`` re-formulation being optimized away. This will be about achieving goals from the "ctypes" plan as discussed in the developer manual. Also the performance page will be expanded with more benchmarks and diagrams as I go forward. I have finally given up on "codespeed", and do my own diagrams. Nuitka Release 0.3.25 ===================== This release brings about changes on all fronts, bug fixes, new features. Also very importantly Nuitka no longer uses C++11 for its code, but mere C++03. There is new re-formulation work, and re-factoring of functions. But the most important part is this: Mercurial unit tests are working. Nearly. With the usual disclaimer of me being wrong, all remaining errors are errors of the test, or minor things. Hope is that these unit tests can be added as release tests to Nuitka. And once that is done, the next big Python application can come. Bug fixes --------- - Local variables were released when an exception was raised that escaped the local function. They should only be released, after another exception was raised somewhere. `Issue#39 `__. - Identifiers of nested tuples and lists could collide. .. code-block:: python a = ((1, 2), 3) b = ((1,), 2, 3) Both tuples had the same name previously, not the end of the tuple is marked too. Fixed in 0.3.24.1 already. - The ``__name__`` when used read-only in modules in packages was optimized to a string value that didn't contain the package name. - Exceptions set when entering compiled functions were unset at function exit. New Features ------------ - Compiled frames support. Before, Nuitka was creating frames with the standard CPython C/API functions, and tried its best to cache them. This involved some difficulties, but as it turns out, it is actually possible to instead provide a compatible type of our own, that we have full control over. This will become the base of enhanced compatibility. Keeping references to local variables attached to exception tracebacks is something we may be able to solve now. - Enhanced Python3 support, added support for ``nonlocal`` declarations and many small corrections for it. - Writable ``__defaults__`` attribute for compiled functions, actually changes the default value used at call time. Not supported is changing the amount of default parameters. Cleanups -------- - Keep the functions along with the module and added "FunctionRef" node kind to point to them. - Reformulated ``or`` and ``and`` operators with the conditional expression construct which makes the "short-circuit" branch. - Access ``self`` in methods from the compiled function object instead of pointer to context object, making it possible to access the function object. - Removed "OverflowCheck" module and its usage, avoids one useless scan per function to determine the need for "locals dictionary". - Make "compileTree" of "MainControl" module to only do what the name says and moved the rest out, making the top level control clearer. - Don't export module entry points when building executable and not modules. These exports cause MinGW and MSVC compilers to create export libraries. New Optimization ---------------- - More efficient code for conditional expressions in conditions: .. code-block:: python if a if b else c See above, this code is now the typical pattern for each ``or`` and ``and``, so this was much needed now. Organizational -------------- - The remaining uses of C++11 have been removed. Code generated with Nuitka and complementary C++ code now compile with standard C++03 compilers. This lowers the Nuitka requirements and enables at least g++ 4.4 to work with Nuitka. - The usages of the GNU extension operation ``a ?: b`` have replaced with standard C++ constructs. This is needed to support MSVC which doesn't have this. - Added examples for the typical use cases to the "`User Manual `__". - The "compare_with_cpython" script has gained an option to immediately remove the Nuitka outputs (build directory and binary) if successful. Also the temporary files are now put under "/var/tmp" if available. - Debian package improvements, registering with "doc-base" the "`User Manual `__" so it is easier to discover. Also suggest "mingw32" package which provides the cross compiler to Windows. - Partial support for MSVC (Visual Studio 2008 to be exact, the version that works with CPython2.6 and CPython2.7). All basic tests that do not use generators are working now, but those will currently cause crashes. - Renamed the ``--g++-only`` option to ``--c++-only``. The old name is no longer correct after clang and MSVC have gained support, and it could be misunderstood to influence compiler selection, rather than causing the C++ source code to not be updated, so manual changes will the used. This solves `Issue#47 `__. - Catch exceptions for ``continue``, ``break``, and ``return`` only where needed for ``try``/``finally`` and loop constructs. New Tests --------- - Added CPython3.2 test suite as "tests/CPython32" from 3.2.3 and run it with CPython2.7 to check that Nuitka gives compatible error messages. It is not expected to pass yet on Python3.2, but work will be done towards this goal. - Make CPython2.7 test suite runner also execute the generated "doctest" modules. - Enabled tests for default parameters and their reference counts. Summary ------- This release marks an important point. The compiled frames are exciting new technology, that will allow even better integration with CPython, while improving speed. Lowering the requirements to C++03 means, we will become usable on Android and with MSVC, which will make adoption of Nuitka on Windows easier for many. Structurally the outstanding part is the function as references cleanup. This was a blocker for value propagation, because now functions references can be copied, whereas previously this was duplicating the whole function body, which didn't work, and wasn't acceptable. Now, work can resume in this domain. Also very exciting when it comes to optimization is the remove of special code for ``or`` and ``and`` operators, as these are now only mere conditional expressions. Again, this will make value propagation easier with two special cases less. And then of course, with Mercurial unit tests running compiled with Nuitka, an important milestone has been hit. For a while now, the focus will be on completing Python3 support, XML based optimization regression tests, benchmarks, and other open ends. Once that is done, and more certainty about Mercurial tests support, I may call it a 0.4 and start with local type inference for actual speed gains. Nuitka Release 0.3.24 ===================== This release contains progress on many fronts, except performance. The extended coverage from running the CPython 2.7 and CPython 3.2 (partially) test suites shows in a couple of bug fixes and general improvements in compatibility. Then there is a promised new feature that allows to compile whole packages. Also there is more Python3 compatibility, the CPython 3.2 test suite now succeeds up to "test_builtin.py", where it finds that ``str`` doesn't support the new parameters it has gained, future releases will improve on this. And then of course, more re-formulation work, in this case, class definitions are now mere simple functions. This and later function references, is the important and only progress towards type inference. Bug fixes --------- - The compiled method type can now be used with ``copy`` module. That means, instances with methods can now be copied too. `Issue#40 `__. Fixed in 0.3.23.1 already. - The ``assert`` statement as of Python2.7 creates the ``AssertionError`` object from a given value immediately, instead of delayed as it was with Python2.6. This makes a difference for the form with 2 arguments, and if the value is a tuple. `Issue#41 `__. Fixed in 0.3.23.1 already. - Sets written like this didn't work unless they were predicted at compile time: .. code-block:: python { value } This apparently rarely used Python2.7 syntax didn't have code generation yet and crashed the compiler. `Issue#42 `__. Fixed in 0.3.23.1 already. - For Python2, the default encoding for source files is ``ascii``, and it is now enforced by Nuitka as well, with the same ``SyntaxError``. - Corner cases of ``exec`` statements with nested functions now give proper ``SyntaxError`` exceptions under Python2. - The ``exec`` statement with a tuple of length 1 as argument, now also gives a ``TypeError`` exception under Python2. - For Python2, the ``del`` of a closure variable is a ``SyntaxError``. New Features ------------ - Added support creating compiled packages. If you give Nuitka a directory with an "__init__.py" file, it will compile that package into a ".so" file. Adding the package contents with ``--recurse-dir`` allows to compile complete packages now. Later there will be a cleaner interface likely, where the later is automatic. - Added support for providing directories as main programs. It's OK if they contain a "__main__.py" file, then it's used instead, otherwise give compatible error message. - Added support for optimizing the ``super`` built-in. It was already working correctly, but not optimized on CPython2. But for CPython3, the variant without any arguments required dedicated code. - Added support for optimizing the ``unicode`` built-in under Python2. It was already working, but will become the basis for the ``str`` built-in of Python3 in future releases. - For Python3, lots of compatibility work has been done. The Unicode issues appear to be ironed out now. The ``del`` of closure variables is allowed and supported now. Built-ins like ``ord`` and ``chr`` work more correctly and attributes are now interned strings, so that monkey patching classes works. Organizational -------------- - Migrated "bin/benchmark.sh" to Python as "misc/run-valgrind.py" and made it a bit more portable that way. Prefers "/var/tmp" if it exists and creates temporary files in a secure manner. Triggered by the Debian "insecure temp file" bug. - Migrated "bin/make-dependency-graph.sh" to Python as "misc/make-dependency-graph.py" and made a more portable and powerful that way. The filtering is done a more robust way. Also it creates temporary files in a secure manner, also triggered by the Debian "insecure temp file" bug. And it creates SVG files and no longer PostScript as the first one is more easily rendered these days. - Removed the "misc/gist" git sub-module, which was previously used by "misc/make-doc.py" to generate HTML from "`User Manual `__" and "`Developer Manual `__". These are now done with Nikola, which is much better at it and it integrates with the web site. - Lots of formatting improvements to the change log, and manuals: * Marking identifiers with better suited ReStructured Text markup. * Added links to the bug tracker all Issues. * Unified wordings, quotation, across the documents. Cleanups -------- - The creation of the class dictionaries is now done with normal function bodies, that only needed to learn how to throw an exception when directly called, instead of returning ``NULL``. Also the assignment of ``__module__`` and ``__doc__`` in these has become visible in the node tree, allowing their proper optimization. These re-formulation changes allowed to remove all sorts of special treatment of ``class`` code in the code generation phase, making things a lot simpler. - There was still a declaration of ``PRINT_ITEMS`` and uses of it, but no definition of it. - Code generation for "main" module and "other" modules are now merged, and no longer special. - The use of raw strings was found unnecessary and potentially still buggy and has been removed. The dependence on C++11 is getting less and less. New Tests --------- - Updated CPython2.6 test suite "tests/CPython26" to 2.6.8, adding tests for recent bug fixes in CPython. No changes to Nuitka were needed in order to pass, which is always good news. - Added CPython2.7 test suite as "tests/CPython27" from 2.7.3, making it public for the first time. Previously a private copy of some age, with many no longer needed changes had been used by me. Now it is up to par with what was done before for "tests/CPython26", so this pending action is finally done. - Added test to cover Python2 syntax error of having a function with closure variables nested inside a function that is an overflow function. - Added test "BuiltinSuper" to cover ``super`` usage details. - Added test to cover ``del`` on nested scope as syntax error. - Added test to cover ``exec`` with a tuple argument of length 1. - Added test to cover ``barry_as_FLUFL`` future import to work. - Removed "Unicode" from known error cases for CPython3.2, it's now working. Summary ------- This release brought forward the most important remaining re-formulation changes needed for Nuitka. Removing class bodies, makes optimization yet again simpler. Still, making function references, so they can be copied, is missing for value propagation to progress. Generally, as usual, a focus has been laid on correctness. This is also the first time, I am release with a known bug though: That is `Issue#39 `__ which I believe now, may be the root cause of the mercurial tests not yet passing. The solution will be involved and take a bit of time. It will be about "compiled frames" and be a (invasive) solution. It likely will make Nuitka faster too. But this release includes lots of tiny improvements, for Python3 and also for Python2. So I wanted to get this out now. As usual, please check it out, and let me know how you fare. Nuitka Release 0.3.23 ===================== This release is the one that completes the Nuitka "sun rise phase". All of Nuitka is now released under `Apache License 2.0 `__ which is a very liberal license, and compatible with basically all Free Software licenses there are. It's only asking to allow integration, of what you send back, and patent grants for the code. In the first phase of Nuitka development, I wanted to keep control over Nuitka, so it wouldn't repeat mistakes of other projects. This is no longer a concern for me, it's not going to happen anymore. I would like to thank Debian Legal team, for originally bringing to my attention, that this license will be better suited, than any copyright assignment could be. Bug fixes --------- - The compiled functions could not be used with ``multiprocessing`` or ``copy.copy``. `Issue#19 `__. Fixed in 0.3.22.1 already. - In-place operations for slices with not both bounds specified crashed the compiler. `Issue#36 `__. Fixed in 0.3.22.1 already. - Cyclic imports could trigger an endless loop, because module import expressions became the parent of the imported module object. `Issue#37 `__. Fixed in 0.3.22.2 already. - Modules named ``proc`` or ``func`` could not be compiled to modules or embedded due to a collision with identifiers of CPython2.7 includes. `Issue#38 `__. Fixed in 0.3.22.2 already. New Features ------------ - The fix for `Issue#19 `__ also makes pickling of compiled functions available. As it is the case for non-compiled functions in CPython, no code objects are stored, only names of module level variables. Organizational -------------- - Using the Apache License 2.0 for all of Nuitka now. - Speedcenter has been re-activated, but is not yet having a lot of benchmarks yet, subject to change. .. admonition:: Update We have given up on speedcenter meanwhile, and generate static pages with graphs instead. New Tests --------- - Changed the "CPython26" tests to no longer disable the parts that relied on copying of functions to work, as `Issue#19 `__ is now supported. - Extended in-place assignment tests to cover error cases of `Issue#36 `__. - Extended compile library test to also try and compile the path where ``numpy`` lives. This is apparently another path, where Debian installs some modules, and compiling this would have revealed `Issue#36 `__ sooner. Summary ------- The release contains bug fixes, and the huge step of changing `the license `__. It is made in preparation to `PyCON EU `__. Nuitka Release 0.3.22 ===================== This release is a continuation of the trend of previous releases, and added more re-formulations of Python that lower the burden on code generation and optimization. It also improves Python3 support substantially. In fact this is the first release to not only run itself under Python3, but for Nuitka to *compile itself* with Nuitka under Python3, which previously only worked for Python2. For the common language subset, it's quite fine now. Bug fixes --------- - List contractions produced extra entries on the call stack, after they became functions, these are no more existent. That was made possible my making frame stack entries an optional element in the node tree, left out for list contractions. - Calling a compiled function in an exception handler cleared the exception on return, it no longer does that. - Reference counter handling with generator ``throw`` method is now correct. - A module "builtins" conflicted with the handling of the Python ``builtins`` module. Those now use different identifiers. New Features ------------ - New ``metaclass`` syntax for the ``class`` statement works, and the old ``__metaclass__`` attribute is properly ignored. .. code-block:: python # Metaclass syntax in Python3, illegal in Python2 class X(metaclass = Y): pass .. code-block:: python # Metaclass syntax in Python2, no effect in Python3 class X: __metaclass__ = Y .. note:: The way to make a use of a metaclass in a portable way, is to create a based class that has it and then inherit from it. Sad, isn' it. Surely, the support for ``__metaclass__`` could still live. .. code-block:: python # For Python2/3 compatible source, we create a base class that has the # metaclass used and doesn't require making a choice. CPythonNodeMetaClassBase = NodeCheckMetaClass("CPythonNodeMetaClassBase", (object,), {}) - The ``--dump-xml`` option works with Nuitka running under Python3. This was not previously supported. - Python3 now also has compatible parameter errors and compatible exception error messages. - Python3 has changed scope rules for list contractions (assignments don't affect outside values) and this is now respected as well. - Python3 has gained support for recursive programs and stand alone extension modules, these are now both possible as well. New Optimization ---------------- - Avoid frame stack entries for functions that cannot raise exceptions, i.e. where they would not be used. This avoids overhead for the very simple functions. And example of this can be seen here: .. code-block:: python def simple(): return 7 - Optimize ``len`` built-in for non-constant, but known length values. An example can be seen here: .. code-block:: python # The range isn't constructed at compile time, but we still know its # length. len(range(10000000)) # The string isn't constructed at compile time, but we still know its # length. len("*" * 1000) # The tuple isn't constructed, instead it's known length is used, and # side effects are maintained. len((a(), b())) This new optimization applies to all kinds of container creations and the ``range`` built-in initially. - Optimize conditions for non-constant, but known truth values. At this time, known truth values of non-constants means ``range`` built-in calls with know size and container creations. An example can be seen here: .. code-block:: python if (a,): print "In Branch" It's clear, that the tuple will be true, we just need to maintain the side effect, which we do. - Optimize ``or`` and ``and`` operators for known truth values. See above for what has known truth values currently. This will be most useful to predict conditions that need not be evaluated at all due to short circuit nature, and to avoid checking against constant values. Previously this could not be optimized, but now it can: .. code-block:: python # The access and call to "something()" cannot possibly happen 0 and something() # Can be replaced with "something()", as "1" is true. If it had a side effect, it # would be maintained. 1 and something() # The access and call to "something()" cannot possibly happen, the value is already # decided, it's "1". 1 or something() # Can be replaced with "something()", as "0" is false. If it had a side effect, it # would be maintained. 0 or something() - Optimize print arguments to become strings. The arguments to ``print`` statements are now converted to strings at compile time if possible. .. code-block:: python print 1 becomes: .. code-block:: python print "1" - Combine print arguments to single ones. When multiple strings are printed, these are now combined. .. code-block:: python print "1+1=", 1+1 becomes: .. code-block:: python print "1+1= 2" Organizational -------------- - Enhanced Python3 support, enabling support for most basic tests. - Check files with PyLint in deterministic (alphabetical) order. Cleanups -------- - Frame stack entries are now part of the node tree instead of part of the template for every function, generator, class or module. - The ``try``/``except``/``else`` has been re-formulated to use an indicator variable visible in the node tree, that tells if a handler has been executed or not. - Side effects are now a dedicated node, used in several optimization to maintain the effect of an expression with known value. New Tests --------- - Expanded and adapted basic tests to work for Python3 as well. - Added reference count tests for generator functions ``throw``, ``send``, and ``close`` methods. - Cover calling a function with ``try``/``except`` in an exception handler twice. No test was previously doing that. Summary ------- This release offers enhanced compatibility with Python3, as well as the solution to many structural problems. Calculating lengths of large non-constant values at compile time, is technically a break through, as is avoiding lengthy calculations. The frame guards as nodes is a huge improvement, making that costly operational possible to be optimized away. There still is more work ahead, before value propagation will be safe enough to enable, but we are seeing the glimpse of it already. Not for long, and looking at numbers will make sense. Nuitka Release 0.3.21 ===================== This releases contains some really major enhancements, all heading towards enabling value propagation inside Nuitka. Assignments of all forms are now all simple and explicit, and as a result, now it will be easy to start tracking them. Contractions have become functions internally, with statements use temporary variables, complex unpacking statement were reduced to more simple ones, etc. Also there are the usual few small bug fixes, and a bunch of organizational improvements, that make the release complete. Bug fixes --------- - The built-in ``next`` could causes a program crash when iterating past the end of an iterator. `Issue#34 `__. Fixed in 0.3.20.1 already. - The ``set`` constants could cause a compiler error, as that type was not considered in the "mutable" check yet. Fixed in 0.3.20.2 already. - Performance regression. Optimize expression for exception types caught as well again, this was lost in last release. - Functions that contain ``exec``, are supposed to have a writable locals. But when removing that ``exec`` statement as part of optimization, this property of the function could get lost. - The so called "overflow functions" are once again correctly handled. These once were left behind in some refactoring and had not been repaired until now. An overflow function is a nested function with an ``exec`` or a star import. - The syntax error for ``return`` outside of a function, was not given, instead the code returned at run time. Fixed to raise a ``SyntaxError`` at compile time. New Optimization ---------------- - Avoid ``tuple`` objects to be created when catching multiple exception types, instead call exception match check function multiple times. - Removal of dead code following ``break``, ``continue``, ``return``, and ``raise``. Code that follows these statements, or conditional statements, where all branches end with it. .. note:: These may not actually occur often in actual code, but future optimization may produce them more frequently, and their removal may in turn make other possible optimization. - Detect module variables as "read only" after all writes have been detected to not be executed as removed. Previously the "read only indicator" was determined only once and then stayed the same. - Expanded conditional statement optimization to detect cases, where condition is a compile time constant, not just a constant value. - Optimize away assignments from a variable to the same variable, they have no effect. The potential side effect of accessing the variable is left intact though, so exceptions will be raised still. .. note:: An exception is where ``len = len`` actually does have an impact, because that variable becomes assignable. The "compile itself" test of Nuitka found that to happen with ``long`` from the ``nuitka.__past__`` module. - Created Python3 variant of quick ``unicode`` string access, there was no such thing in the CPython C/API, but we make the distinction in the source code, so it makes sense to have it. - Created an optimized implementation for the built-in ``iter`` with 2 parameters as well. This allows for slightly more efficient code to be created with regards to reference handling, rather than using the CPython C/API. - For all types of variable assigned in the generated code, there are now methods that accept already taken references or not, and the code generator picks the optimal variant. This avoids the drop of references, that e.g. the local variable will insist to take. - Don't use a "context" object for generator functions (and generator expressions) that don't need one. And even if it does to store e.g. the given parameter values, avoid to have a "common context" if there is no closure taken. This avoids useless ``malloc`` calls and speeds up repeated generator object creation. Organizational -------------- - Changed the Scons build file database to reside in the build directory as opposed to the current directory, not polluting it anymore. Thanks for the patch go to Michael H Kent, very much appreciated. - The ``--experimental`` option is no longer available outside of checkouts of git, and even there not on stable branches (``master``, ``hotfix/...``). It only pollutes ``--help`` output as stable releases have no experimental code options, not even development version will make a difference. - The binary "bin/Nuitka.py" has been removed from the git repository. It was deprecated a while ago, not part of the distribution and served no good use, as it was a symbolic link only anyway. - The ``--python-version`` option is applied at Nuitka start time to re-launch Nuitka with the given Python version, to make sure that the Python run time used for computations and link time Python versions are the same. The allowed values are now checked (2.6, 2.7 and 3.2) and the user gets a nice error with wrong values. - Added ``--keep-pythonpath`` alias for ``--execute-with-pythonpath`` option, probably easier to remember. - Support ``--debug`` with clang, so it can also be used to check the generated code for all warnings, and perform assertions. Didn't report anything new. - The contents environment variable ``CXX`` determines the default C++ compiler when set, so that checking with ``CXX=g++-4.7 nuitka-python ...`` has become supported. - The ``check-with-pylint`` script now has a real command line option to control the display of ``TODO`` items. Cleanups -------- - Changed complex assignments, i.e. assignments with multiple targets to such using a temporary variable and multiple simple assignments instead. .. code-block:: python a = b = c .. code-block:: python _tmp = c b = _tmp a = _tmp In CPython, when one assignment raises an exception, the whole thing is aborted, so the complexity of having multiple targets is no more needed, now that we have temporary variables in a block. All that was really needed, was to evaluate the complete source expression only once, but that made code generation contain ugly loops that are no more needed. - Changed unpacking assignments to use temporary variables. Code like this: .. code-block:: python a, b = c Is handled more like this: .. code-block:: python _tmp_iter = iter(c) _tmp1 = next(_tmp_iter) _tmp2 = next(_tmp_iter) if not finished(_tmp_iter): raise ValueError("too many values to unpack") a = _tmp1 b = _tmp2 In reality, not really ``next`` is used, as it wouldn't raise the correct exception for unpacking, and the ``finished`` check is more condensed into it. Generally this cleanup allowed that the ``AssignTargetTuple`` and associated code generation was removed, and in the future value propagation may optimize these ``next`` and ``iter`` calls away where possible. At this time, this is not done yet. - Exception handlers assign caught exception value through assignment statement. Previously the code generated for assigning from the caught exception was not considered part of the handler. It now is the first statement of an exception handler or not present, this way it may be optimized as well. - Exception handlers now explicitly catch more than one type. Catching multiple types worked by merits of the created tuple object working with the Python C/API function called, but that was not explicit at all. Now every handler has a tuple of exceptions it catches, which may only be one, or if None, it's all. - Contractions are now functions as well. Contractions (list, dict, and set) are now re-formulated as function bodies that contain for loops and conditional statements. This allowed to remove a lot of special code that dealt with them and will make these easier to understand for optimization and value propagation. - Global is handled during tree building. Previously the global statement was its own node, which got removed during the optimization phase in a dedicated early optimization that applied its effect, and then removed the node. It was determined, that there is no reason to not immediately apply the effect of the global variable and take closure variables and add them to the provider of that ``global`` statement, allowing to remove the node class. - Read only module variable detection integrated to constraint collection. The detection of read only module variables was so far done as a separate step, which is no more necessary as the constraint collection tracks the usages of module variables anyway, so this separate and slow step could be removed. New Tests --------- - Added test to cover order of calls for complex assignments that unpack, to see that they make a fresh iterator for each part of a complex assignment. - Added test that unpacks in an exception catch. It worked, due to the generic handling of assignment targets by Nuitka, and I didn't even know it can be done, example: .. code-block:: python try: raise ValueError(1,2) except ValueError as (a,b): print "Unpacking caught exception and unpacked", a, b Will assign ``a=1`` and ``b=2``. - Added test to cover return statements on module level and class level, they both must give syntax errors. - Cover exceptions from accessing unassigned global names. - Added syntax test to show that star imports do not allow other names to be imported at the same time as well. - Python3 is now also running the compile itself test successfully. Summary ------- The progress made towards value propagation and type inference is *very* significant, and makes those appears as if they are achievable. Nuitka Release 0.3.20 ===================== This time there are a few bug fixes and some really major cleanups, lots of new optimization and preparations for more. And then there is a new compiler clang and a new platform supported. MacOS X appears to work mostly, thanks for the patches from Pete Hunt. Bug fixes --------- - The use of a local variable name as an expression was not covered and lead to a compiler crash. Totally amazing, but true, nothing in the test suite of CPython covered this. `Issue#30 `__. Fixed in release 0.3.19.1 already. - The use of a closure variable name as an expression was not covered as well. And in this case corrupted the reference count. `Issue#31 `__. Fixed in release 0.3.19.1 already. - The ``from x import *`` attempted to respect ``__all__`` but failed to do so. `Issue#32 `__. Fixed in release 0.3.19.2 already. - The ``from x import *`` didn't give a ``SyntaxError`` when used on Python3. Fixed in release 0.3.19.2 already. - The syntax error messages for "global for function argument name" and "duplicate function argument name" are now identical as well. - Parameter values of generator function could cause compilation errors when used in the closure of list contractions. Fixed. New Features ------------ - Added support for disabling the console for Windows binaries. Thanks for the patch go to Michael H Kent. - Enhanced Python3 support for syntax errors, these are now also compatible. - Support for MacOS X was added. - Support for using the clang compiler was added, it can be enforced via ``--clang`` option. Currently this option is mainly intended to allow testing the "MacOS X" support as good as possible under Linux. New Optimization ---------------- - Enhanced all optimization that previously worked on "constants" to work on "compile time constants" instead. A "compile time constant" can currently also be any form of a built-in name or exception reference. It is intended to expand this in the future. - Added support for built-ins ``bin``, ``oct``, and ``hex``, which also can be computed at compile time, if their arguments are compile time constant. - Added support for the ``iter`` built-in in both forms, one and two arguments. These cannot be computed at compile time, but now will execute faster. - Added support for the ``next`` built-in, also in its both forms, one and two arguments. These also cannot be computed at compile time, but now will execute faster as well. - Added support for the ``open`` built-in in all its form. We intend for future releases to be able to track file opens for including them into the executable if data files. - Optimize the ``__debug__`` built-in constant as well. It cannot be assigned, yet code can determine a mode of operation from it, and apparently some code does. When compiling the mode is decided. - Optimize the ``Ellipsis`` built-in constant as well. It falls in the same category as ``True``, ``False``, ``None``, i.e. names of built-in constants that a singletons. - Added support for anonymous built-in references, i.e. built-ins which have names that are not normally accessible. An example is ``type(None)`` which is not accessible from anywhere. Other examples of such names are ``compiled_method_or_function``. Having these as represented internally, and flagged as "compile time constants", allows the compiler to make more compile time optimization and to generate more efficient C++ code for it that won't e.g. call the ``type`` built-in with ``None`` as an argument. - All built-in names used in the program are now converted to "built-in name references" in a first step. Unsupported built-ins like e.g. ``zip``, for which Nuitka has no own code or understanding yet, remained as "module variables", which made access to them slow, and difficult to recognize. - Added optimization for module attributes ``__file__``, ``__doc__`` and ``__package__`` if they are read only. It's the same as ``__name__``. - Added optimization for slices and subscripts of "compile time constant" values. These will play a more important role, once value propagation makes them more frequent. Organizational -------------- - Created a "change log" from the previous release announcements. It's as ReStructured Text and converted to PDF for the release as well, but I chose not to include that in Debian, because it's so easy to generate the PDF on that yourself. - The posting of release announcements is now prepared by a script that converts the ReStructured Text to HTML and adds it to Wordpress as a draft posting or updates it, until it's release time. Simple, sweet and elegant. Cleanups -------- - Split out the ``nuitka.nodes.Nodes`` module into many topic nodes, so that there are now ``nuitka.nodes.BoolNodes`` or ``nuitka.nodes.LoopNodes`` to host nodes of similar kinds, so that it is now cleaner. - Split ``del`` statements into their own node kind, and use much simpler node structures for them. The following blocks are absolutely the same: .. code-block:: python del a, b.c, d .. code-block:: python del a del b.c del d So that's now represented in the node tree. And even more complex looking cases, like this one, also the same: .. code-block:: python del a, (b.c, d) This one gives a different parse tree, but the same bytecode. And so Nuitka need no longer concern itself with this at all, and can remove the tuple from the parse tree immediately. That makes them easy to handle. As you may have noted already, it also means, there is no way to enforce that two things are deleted or none at all. - Turned the function and class builder statements into mere assignment statements, where defaults and base classes are handled by wrapping expressions. Previously they are also kind of assignment statements too, which is not needed. Now they were reduced to only handle the ``bases`` for classes and the ``defaults`` for functions and make optional. - Refactored the decorator handling to the tree building stage, presenting them as function calls on "function body expression" or class body expression". This allowed to remove the special code for decorators from code generation and C++ templates, making decorations easy subjects for future optimization, as they practically are now just function calls. .. code-block:: python @some_classdecorator class C: @staticmethod def f(): pass It's just a different form of writing things. Nothing requires the implementation of decorators, it's just functions calls with function bodies before the assignment. The following is only similar: .. code-block:: python class C: def f(): pass f = staticmethod(f) C = some_classdecorator(C) It's only similar, because the assignment to an intermediate value of ``C`` and ``f`` is not done, and if an exception was raised by the decoration, that name could persist. For Nuitka, the function and class body, before having a name, are an expression, and so can of course be passed to decorators already. - The in-place assignments statements are now handled using temporary variable blocks Adding support for scoped temporary variables and references to them, it was possible to re-formulate in-place assignments expressions as normal look-ups, in-place operation call and then assignment statement. This allowed to remove static templates and will yield even better generated code in the future. - The for loop used to have has a "source" expression as child, and the iterator over it was only taken at the code generation level, so that step was therefore invisible to optimization. Moved it to tree building stage instead, where optimization can work on it then. - Tree building now generally allows statement sequences to be ``None`` everywhere, and pass statements are immediately eliminated from them immediately. Empty statement sequences are now forbidden to exist. - Moved the optimization for ``__name__`` to compute node of variable references, where it doesn't need anything complex to replace with the constant value if it's only read. - Added new bases classes and mix-in classes dedicated to expressions, giving a place for some defaults. - Made the built-in code more reusable. New Tests --------- - Added some more diagnostic tests about complex assignment and ``del`` statements. - Added syntax test for star import on function level, that must fail on Python3. - Added syntax test for duplicate argument name. - Added syntax test for global on a function argument name. Summary ------- The decorator and building changes, the assignment changes, and the node cleanups are all very important progress for the type inference work, because they remove special casing the that previously would have been required. Lambdas and functions now really are the same thing right after tree building. The in-place assignments are now merely done using standard assignment code, the built functions and classes are now assigned to names in assignment statements, much *more* consistency there. Yet, even more work will be needed in the same direction. There may e.g. be work required to cover ``with`` statements as well. And assignments will become no more complex than unpacking from a temporary variable. For this release, there is only minimal progress on the Python3 front, despite the syntax support, which is only miniscule progress. The remaining tasks appear all more or less difficult work that I don't want to touch now. There are still remaining steps, but we can foresee that a release may be done that finally actually does type inference and becomes the effective Python compiler this project is all about. Nuitka Release 0.3.19 ===================== This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things in this are justifying a new release. Bug fixes --------- - The man pages of ``nuitka`` and ``nuitka-python`` had no special layout for the option groups and broken whitespace for ``--recurse-to`` option. Also ``--g++-only`` was only partially bold. Released as 0.3.18.1 hot fix already. - The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hot fix already. - Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of ``/`` instead of using a ``joinpath`` call. Released as 0.3.18.3 already. - A call to the range built-in with no arguments would crash the compiler, see `Issue#29 `__. Released as 0.3.18.4 already. - Compatibility Fix: When rich comparison operators returned false value other ``False``, for comparison chains, these would not be used, but ``False`` instead, see . - The support for ``__import__`` didn't cover keyword arguments, these were simply ignored. See `Issue#28 `__. Fixed, but no warning is given yet. New Features ------------ - A new option has been added, one can now specify ``--recurse-directory`` and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements. - Added support for the ``exec`` built-in of Python3, this enables us to run one more basic test, ``GlobalStatement.py`` with Python3. The test ``ExecEval.py`` nearly works now. New Optimization ---------------- - The no arguments ``range()`` call now optimized into the static CPython exception it raises. - Parts of comparison chains with constant arguments are now optimized away. Cleanups -------- - Simplified the ``CPythonExpressionComparison`` node, it now always has only 2 operands. If there are more, the so called "comparison chain", it's done via ``and`` with assignments to temporary variables, which are expressed by a new node type ``CPythonExpressionTempVariableRef``. This allowed to remove ``expression_temps`` from C++ code templates and generation, reducing the overall complexity. - When executing a module (``--execute`` but not ``--exe``), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment. - The calls to the variadic ``MAKE_TUPLE`` were replaced with calls the ``MAKE_TUPLExx`` (where ``xx`` is the number of arguments), that are generated on a as-needed basis. This gives more readable code, because no ``EVAL_ORDERED_xx`` is needed at call site anymore. - Many node classes have moved to new modules in ``nuitka.nodes`` and grouped by theme. That makes them more accessible. - The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the ``sys.abiflags`` and works with Python3. - The replacing of ``.py`` in filenames was made more robust. No longer is ``str.replace`` used, but instead proper means to assure that having ``.py`` as other parts of the filenames won't be a trouble. - Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements. - As always, some PyLint work, and some minor ``TODO`` were solved. Organizational -------------- - Added more information to the "`Developer Manual `__", e.g. documenting the tree changes for ``assert`` to become a conditional statement with a raise statement, etc. - The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise. - Added support to specify the binary under test with a ``NUITKA`` environment, so the test framework can run with installed version of Nuitka too. - Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for ``os.execl`` not propagating exit codes under Windows. See `Issue#26 `__ for more information. - For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the ``PATH`` or even installed to execute the binary. New Tests --------- - The ``basic``, ``programs``, ``syntax``, and ``reflected`` were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations. - Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains. - The ``Referencing.py`` test was made portable to Python3. - Cover no arguments ``range()`` exception as well. - Added test to demonstrate that ``--recurse-directory`` actually works. This is using an ``__import__`` that cannot be predicted at run time (yet). - The created source package is now tested on pbuilder chroots to be pass installation and the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn't break without notice. Summary ------- This releases contains many changes. The "temporary variable ref" and "assignment expression" work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. in-place assignments can be reduced to assignments to temporary variables and conditional statements. While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with "value propagation" working. These are the real focus. The old comparison chain handling was a big wart. Working, but no way understood by any form of analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimization to see through them. Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of in-place assignments. With an "assignment expression" to a "temporary variable ref", these become the same as user code using such a variable. There will be more of these to find. So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the ``__import__`` related one, were not suitable for hot fix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already, also worth to get out. Nuitka Release 0.3.18 ===================== This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via ``apt-get install nuitka``. The release cycle was too short to have much focus. It merely includes fixes, which were available as hot fixes, and some additional optimization and node tree cleanups, as well as source cleanups. But not much else. Bug fixes --------- - Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. `Issue#16 `__. Released as 0.3.17a hot fix already. - Nuitka was considering directories to contain packages that had no "__init__.py" which could lead to errors when it couldn't find the package later in the compilation process. Released as 0.3.17a hot fix already. - When providing ``locals()`` to ``exec`` statements, this was not making the ``locals()`` writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn't consider the actual value ``locals()``. Released as 0.3.17b hot fix already. - Compatibility Fix: When no defaults are given, CPython uses ``None`` for ``func.func_defaults``, but Nuitka had been using ``None``. New Optimization ---------------- - If the condition of assert statements can be predicted, these are now optimized in a static raise or removed. - For built-in name references, there is now dedicated code to look them up, that doesn't check the module level at all. Currently these are used in only a few cases though. - Cleaner code is generated for the simple case of ``print`` statements. This is not only faster code, it's also more readable. Cleanups -------- - Removed the ``CPythonStatementAssert`` node. It's not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with ``AssertionError`` and the assertion argument. This allowed to remove code and complexity from the subsequent steps of Nuitka, and enabled existing optimization to work on assert statements as well. - Moved built-in exception names and built-in names to a new module ``nuitka.Builtins`` instead of having in other places. This was previously a bit spread-out and misplaced. - Added cumulative ``tags`` to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks. - New node for built-in name loopups, which allowed to remove tricks played with adding module variable lookups for ``staticmethod`` when adding them for ``__new__`` or module variable lookups for ``str`` when predicting the result of ``type('a')``, which was unlikely to cause a problem, but an important ``TODO`` item still. Organizational -------------- - The `"Download" <../pages/download.html>`__ page is now finally updated for releases automatically. This closes `Issue#7 `__ completely. Up to this release, I had to manually edit that page, but now mastered the art of upload via XMLRCP and a Python script, so that don't loose as much time with editing, checking it, etc. - The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages. - Made sure the test runners worth with bare ``python2.6`` as well. New Tests --------- - Added some tests intended for type inference development. Summary ------- This releases contains not as much changes as others, mostly because it's the intended base for a Debian upload. The ``exec`` fix was detected by continued work on the branch ``feature/minimize_CPython26_tests_diff`` branch, but that work is now complete. It is being made pretty (many git rebase iterations) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made. But of course, the real excitement is the "type inference" work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient. So when I learn something. e.g. ``assert`` is not special, I apply it to the ``develop`` branch immediately, to keep the differences as small as possible, and to immediately benefit from such improvements. Nuitka Release 0.3.17 ===================== This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, lots of very important organisational work, and yet again improved compatibility and cleanups. Also huge is the advance in making ``--deep`` go away and making the recursion of Nuitka controllable, which means a lot for scalability of projects that use a lot of packages that use other packages, because now you can choose which ones to embed and which ones one. The release cycle had a focus on improving the quality of the test scripts, the packaging, and generally to prepare the work on "type inference" in a new feature branch. I have also continued to work towards CPython3.2 compatibility, and this version, while not there, supports Python3 with a large subset of the basic tests programs running fine (of course via ``2to3`` conversion) without trouble. There is still work to do, exceptions don't seem to work fully yet, parameter parsing seems to have changed, etc. but it seems that CPython3.2 is going to work one day. And there has been a lot of effort, to address the Debian packaging to be cleaner and more complete, addressing issues that prevented it from entering the Debian repository. Bug fixes --------- - Fixed the handling of modules and packages of the same name, but with different casing. Problem showed under Windows only. Released as 0.3.16a hot fix already. - Fixed an error where the command line length of Windows was exceeded when many modules were embedded, Christopher Tott provided a fix for it. Released as 0.3.16a hot fix already. - Fix, avoid to introduce new variables for where built-in exception references are sufficient. Released as 0.3.16b hot fix already. - Fix, add the missing ``staticmethod`` decorator to ``__new__`` methods before resolving the scopes of variables, this avoids the use of that variable before it was assigned a scope. Released as 0.3.16b hot fix already. New Features ------------ - Enhanced compatibility again, provide enough ``co_varnames`` in the code objects, so that slicing them up to ``code_object.co_argcount`` will work. They are needed by ``inspect`` module and might be used by some decorators as well. - New options to control the recursion: ``--recurse-none`` (do not warn about not-done recursions) ``--recurse-all`` (recurse to all otherwise warned modules) ``--recurse-to`` (confirm to recurse to those modules) ``--recurse-not-to`` (confirm to not recurse to those modules) New Optimization ---------------- - The optimization of constant conditional expressions was not done yet. Added this missing constant propagation case. - Eliminate near empty statement sequences (only contain a pass statement) in more places, giving a cleaner node structure for many constructs. - Use the pickle "protocol 2" on CPython2 except for ``unicode`` strings where it does not work well. It gives a more compressed and binary representation, that is generally more efficient to un-stream as well. Also use the cPickle protocol, the use of ``pickle`` was not really necessary anymore. Organizational -------------- - Added a "`Developer Manual `__" to the release. It's incomplete, but it details some of the existing stuff, coding rules, plans for "type inference", etc. - Improved the ``--help`` output to use ``metavar`` where applicable. This makes it more readable for some options. - Instead of error message, give help output when no module or program file name was given. This makes Nuitka help out more convenient. - Consistently use ``#!/usr/bin/env python`` for all scripts, this was previously only done for some of them. - Ported the PyLint check script to Python as well, enhancing it on the way to check the exit code, and to only output changes things, as well as making the output of warnings for ``TODO`` items optional. - All scripts used for testing, PyLint checking, etc. now work with Python3 as well. Most useful on Arch Linux, where it's also already the default for ``Python``. - The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together. - Make the tests run without any dependence on ``PATH`` to contain the executables of Nuitka. This makes it easier to use. - Add license texts to 3rd party file that were missing them, apply ``licensecheck`` results to cleanup Nuitka. Also removed own copyright statement from in-line copy of Scons, it had been added by accident only. - Release the tests that I own as well as the Debian packaging I created under "Apache License 2.0" which is very liberal, meaning every project will be able to use it. - Don't require copyright assignment for contributions anymore, instead only "Apache License 2.0", the future Nuitka license, so that the code won't be a problem when changing the license of all of Nuitka to that license. - Give contributors listed in the "`User Manual `__" an exception to the GPL terms until Nuitka is licensed under "Apache License 2.0" as well. - Added an ``--experimental`` option which can be used to control experimental features, like the one currently being added on ``feature/ctypes_annotation``, where "type inference" is currently only activated when that option is given. For this stable release, it does nothing. - Check the static C++ files of Nuitka with ``cppcheck`` as well. Didn't find anything. - Arch Linux packages have been contributed, these are linked for download, but the stable package may lag behind a bit. Cleanups -------- - Changed ``not`` boolean operation to become a normal operator. Changed ``and`` and ``or`` boolean operators to a new base class, and making their interface more similar to that of operations. - Added cumulative ``tags`` to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks. - Enhanced the "visitor" interface to provide more kinds of callbacks, enhanced the way "each scope" visiting is achieved by generalizing is as "child has not tag 'closure_taker'" and that for every "node that has tag 'closure_taker'". - Moved ``SyntaxHighlighting`` module to ``nuitka.gui`` package where it belongs. - More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that were white listed. - Consistently use ``nuitka`` in test scripts, as there isn't a ``Nuitka.py`` on all platforms. The later is scheduled for removal. - Some more PyLint cleanups. New Tests --------- - Make sure the basic tests pass with CPython or else fail the test. This is to prevent false positives, where a test passes, but only because it fails in CPython early on and then does so with Nuitka too. For the syntax tests we make sure they fail. - The basic tests can now be run with ``PYTHON=python3.2`` and use ``2to3`` conversion in that case. Also the currently not passing tests are not run, so the passing tests continue to do so, with this run from the release test script ``check-release``. - Include the syntax tests in release tests as well. - Changed many existing tests so that they can run under CPython3 too. Of course this is via ``2to3`` conversion. - Don't fail if the CPython test suites are not there. Currently they remain largely unpublished, and as such are mostly only available to me (exception, ``feature/minimize_CPython26_tests_diff`` branch references the CPython2.6 tests repository, but that remains work in progress). - For the compile itself test: Make the presence of the Scons in-line copy optional, the Debian package doesn't contain it. - Also make it more portable, so it runs under Windows too, and allow to choose the Python version to test. Check this test with both CPython2.6 and CPython2.7 not only the default Python. - Before releasing, test that the created Debian package builds fine in a minimal Debian ``unstable`` chroot, and passes all the tests included in the package (``basics``, ``syntax``, ``programs``, ``reflected``). Also many other Debian packaging improvements. Summary ------- The "git flow" was used again in this release cycle and proved to be useful not only for hot fix, but also for creating the branch ``feature/ctypes_annotation`` and rebasing it often while things are still flowing. The few hot fixes didn't require a new release, but the many organizational improvements and the new features did warrant the new release, because of e.g. the much better test handling in this release and the improved recursion control. The work on Python3 support has slowed down a bit. I mostly only added some bits for compatibility, but generally it has slowed down. I wanted to make sure it doesn't regress by accident, so running with CPython3.2 is now part of the normal release tests. What's still missing is more "hg" completeness. Only the ``co_varnames`` work for ``inspect`` was going in that direction, and this has slowed down. It was more important to make Nuitka's recursion more accessible with the new options, so that was done first. And of course, the real excitement is the "type inference" work. It will give a huge boost to Nuitka, and I am happy that it seems to go well. With this in place, new benchmarks may make sense. I am working on getting it off the ground, so other people can work on it too. My idea of ``ctypes`` native calls may become true sooner than expected. To support that, I would like to add more tools to make sure we discover changes earlier on, checking the XML representations of tests to discover improvements and regressions more clearly. Nuitka Release 0.3.16 ===================== This time there are many bug fixes, some important scalability work, and again improved compatibility and cleanups. The release cycle had a focus on fixing the bug reports I received. I have also continued to look at CPython3 compatibility, and this is the first version to support Python3 somewhat, at least some of the basic tests programs run (of course via ``2to3`` conversion) without trouble. I don't know when, but it seems that it's going to work one day. Also there has an effort to make the Debian packaging cleaner, addressing all kinds of small issues that prevented it from entering the Debian repository. It's still not there, but it's making progress. Bug fixes --------- - Fixed a packaging problem for Linux and x64 platform, the new ``swapFiber.S`` file for the fiber management was not included. Released as 0.3.15a hot fix already. - Fixed an error where optimization was performed on removed unreachable code, which lead to an error. Released as 0.3.15b hot fix already. - Fixed an issue with ``__import__`` and recursion not happening in any case, because when it did, it failed due to not being ported to new internal APIs. Released as 0.3.15c hot fix already. - Fixed ``eval()`` and ``locals()`` to be supported in generator expressions and contractions too. Released as 0.3.15d hot fix already. - Fixed the Windows batch files ``nuitka.bat`` and ``nuitka-python.bat`` to not output the ``rem`` statements with the copyright header. Released as 0.3.15d hot fix already. - Fixed re-raise with ``raise``, but without a current exception set. Released as 0.3.15e hot fix already. - Fixed ``vars()`` call on the module level, needs to be treated as ``globals()``. Released as 0.3.15e hot fix already. - Fix handling of broken new lines in source files. Read the source code in "universal line ending mode". Released as 0.3.15f hot fix already. - Fixed handling of constant module attribute ``__name__`` being replaced. Don't replace local variables of the same name too. Released as 0.3.15g hot fix already. - Fixed assigning to ``True``, ``False`` or ``None``. There was this old ``TODO``, and some code has compatibility craft that does it. Released as 0.3.15g hot fix already. - Fix constant dictionaries not always being recognized as shared. Released as 0.3.15g hot fix already. - Fix generator function objects to not require a return frame to exist. In finalize cleanup it may not. - Fixed non-execution of cleanup codes that e.g. flush ``sys.stdout``, by adding ``Py_Finalize()``. - Fix ``throw()`` method of generator expression objects to not check arguments properly. - Fix missing fallback to subscript operations for slicing with non-indexable objects. - Fix, in-place subscript operations could fail to apply the update, if the intermediate object was e.g. a list and the handle just not changed by the operation, but e.g. the length did. - Fix, the future spec was not properly preserving the future division flag. New Optimization ---------------- - The optimization scales now much better, because per-module optimization only require the module to be reconsidered, but not all modules all the time. With many modules recursed into, this makes a huge difference in compilation time. - The creation of dictionaries from constants is now also optimized. New Features ------------ - As a new feature functions now have the ``func_defaults`` and ``__defaults__`` attribute. It works only well for non-nested parameters and is not yet fully integrated into the parameter parsing. This improves the compatibility somewhat already though. - The names ``True``, ``False`` and ``None`` are now converted to constants only when they are read-only module variables. - The ``PYTHONPATH`` variable is now cleared when immediately executing a compiled binary unless ``--execute-with-pythonpath`` is given, in which case it is preserved. This allows to make sure that a binary is in fact containing everything required. Organizational -------------- - The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together. - The in-line copy of Scons is not checked with PyLint anymore. We of course don't care. - Program tests are no longer executed in the program directory, so failed module inclusions become immediately obvious. - The basic tests can now be run with ``PYTHON=python3.2`` and use ``2to3`` conversion in that case. Cleanups -------- - Moved ``tags`` to a separate module, make optimization emit only documented tags, checked against the list of allowed ones. - The Debian package has seen lots of improvements, to make it "lintian clean", even in pedantic mode. The homepage of Nuitka is listed, a watch file can check for new releases, the git repository and the gitweb are referenced, etc. - Use ``os.path.join`` in more of the test code to achieve more Windows portability for them. - Some more PyLint cleanups. New Tests --------- - There is now a ``Crasher`` test, for tests that crashed Nuitka previously. - Added a program test where the imported module does a ``sys.exit()`` and make sure it really doesn't continue after the ``SystemExit`` exception that creates. - Cover the type of ``__builtins__`` in the main program and in imported modules in tests too. It's funny and differs between module and dict in CPython2. - Cover a final ``print`` statement without newline in the test. Must still receive a newline, which only happens when ``Py_Finalize()`` is called. - Added test with functions that makes a ``raise`` without an exception set. - Cover the calling of ``vars()`` on module level too. - Cover the use of eval in contractions and generator expressions too. - Cover ``func_defaults`` and ``__default__`` attributes for a function too. - Added test function with two ``raise`` in an exception handler, so that one becomes dead code and removed without the crash. Summary ------- The "git flow" was really great in this release cycle. There were many hot fix releases being made, so that the bugs could be addressed immediately without requiring the overhead of a full release. I believe that this makes Nuitka clearly one of the best supported projects. This quick turn-around also encourages people to report more bugs, which is only good. And the structure is there to hold it. Of course, the many bug fixes meant that there is not as much new development, but that is not the priority, correctness is. The work on Python3 is a bit strange. I don't need Python3 at all. I also believe it is that evil project to remove cruft from the Python core and make developers of all relevant Python software, add compatibility cruft to their software instead. Yet, I can't really stop to work on it. It has that appeal of small fixups here and there, and then something else works too. Python3 work is like when I was first struggling with Nuitka to pass the CPython2 unit tests for a first time. It's fun. And then it finds real actual bugs that apply to CPython2 too. Not doing ``Py_Finalize`` (but having to), the slice operations shortcomings, the bug of subscript in-place, and so on. There is likely more things hidden, and the earlier Python3 is supported, the more benefit from increased test covered. What's missing is more "hg" completeness. I think only the ``raise`` without exception set and the ``func_defaults`` issue were going into its direction, but it won't be enough yet. Nuitka Release 0.3.15 ===================== This is to inform you about the new stable release of Nuitka. This time again many organizational improvements, some bug fixes, much improved compatibility and cleanups. This release cycle had a focus on packaging Nuitka for easier consumption, i.e. automatic packaging, making automatic uploads, improvement documentation, and generally cleaning things up, so that Nuitka becomes more compatible and ultimately capable to run the "hg" test suite. It's not there yet, but this is a huge jump for usability of Nuitka and its compatibility, again. Then lots of changes that make Nuitka approach Python3 support, the generated C++ for at least one large example is compiling with this new release. It won't link, but there will be later releases. And there is a lot of cleanup going on, geared towards compatibility with line numbers in the frame object. Bug fixes --------- - The main module was using ``__main__`` in tracebacks, but it must be ````. Released as 0.3.14a hot fix already. - Workaround for "execfile cannot be used as an expression". It wasn't possible to use ``execfile`` in an expression, only as a statement. But then there is crazy enough code in e.g. mercurial that uses it in a lambda function, which made the issue more prominent. The fix now allows it to be an expression, except on the class level, which wasn't seen yet. - The in-line copy of Scons was not complete enough to work for "Windows" or with ``--windows-target`` for cross compile. Fixed. - Cached frames didn't release the "back" frame, therefore holding variables of these longer than CPython does, which could cause ordering problems. Fixed for increased compatibility. - Handle "yield outside of function" syntax error in compiled source correctly. This one was giving a Nuitka backtrace, now it gives a ``SyntaxError`` as it needs to. - Made syntax/indentation error output absolutely identical to CPython. - Using the frame objects ``f_lineno`` may fix endless amounts bugs related to traceback line numbers. New Features ------------ - Guesses the location of the MinGW compiler under Windows to default install location, so it need not be added to ``PATH`` environment variable. Removes the need to modify ``PATH`` environment just for Nuitka to find it. - Added support for "lambda generators". You don't want to know what it is. Lets just say, it was the last absurd language feature out there, plus that didn't work. It now works perfect. Organizational -------------- - You can now download a Windows installer and a Debian package that works on Debian Testing, current Ubuntu and Mint Linux. - New release scripts give us the ability to have hot fix releases as download packages immediately. That means the "git flow" makes even more beneficial to the users. - Including the generated "README.pdf" in the distribution archives, so it can be read instead of "README.txt". The text file is fairly readable, due to the use of ReStructured Text, but the PDF is even nicer to read, due to e.g. syntax highlighting of the examples. - Renamed the main binaries to ``nuitka`` and ``nuitka-python``, so that there is no dependency on case sensitive file systems. - For Windows there are batch files ``nuitka.bat`` and ``nuitka-python.bat`` to make Nuitka directly executable without finding the ``Python.exe``, which the batch files can tell from their own location. - There are now man pages of ``nuitka`` and ``nuitka-python`` with examples for the most common use cases. They are of course included in the Debian package. - Don't strip the binary when executing it to analyse compiled binary with ``valgrind``. It will give better information that way, without changing the code. New Optimization ---------------- - Implemented ``swapcontext`` alike (``swapFiber``) for x64 to achieve 8 times speedup for Generators. It doesn't do useless syscalls to preserve signal masks. Now Nuitka is faster at frame switching than CPython on x64, which is already good by design. Cleanups -------- - Using the frame objects to store current line of execution avoids the need to store it away in helper code at all. It ought to also help a lot with threading support, and makes Nuitka even more compatible, because now line numbers will be correct even outside tracebacks, but for mere stack frame dumps. - Moved the ``for_return`` detection from code generation to tree building where it belongs. Yield statements used as return statements need slightly different code for Python2.6 difference. That solved an old ``TODO``. - Much Python3 portability work. Sometimes even improving existing code, the Python compiler code had picked up a few points, where the latest Nuitka didn't work with Python3 anymore, when put to actual compile. The test covered only syntax, but e.g. meta classes need different code in CPython3, and that's now supported. Also helper code was made portable in more places, but not yet fully. This will need more work. - Cleaned up uses of debug defines, so they are now more consistent and in one place. - Some more PyLint cleanups. New Tests --------- - The tests are now executed by Python scripts and cover ``stderr`` output too. Before we only checked ``stdout``. This unveiled a bunch of issues Nuitka had, but went unnoticed so far, and triggered e.g. the frame line number improvements. - Separate syntax tests. - The scripts to run the tests now are all in pure Python. This means, no more MinGW shell is needed to execute the tests. Summary ------- The Debian package, Windows installer, etc. are now automatically updated and uploaded. From here on, there can be such packages for the hot fix releases too. The exception tracebacks are now correct by design, and better covered. The generator performance work showed that the approach taken by Nuitka is in fact fast. It was fast on ARM already, but it's nice to see that it's now also fast on x64. Programs using generators will be affected a lot by this. Overall, this release brings Nuitka closer to usability. Better binary names, man pages, improved documentation, issue tracker, etc. all there now. I am in fact now looking for a sponsor for the Debian package to upload it into Debian directly. .. admonition:: Update The upload to Debian happened for 0.3.18 and was done by Yaroslav Halchenko. What's missing is more "hg" completeness. The frame release issue helped it, but ``inspect.getargs()`` doesn't work yet, and is a topic for a future release. Won't be easy, as ``func_defaults`` will be an invasive change too. Nuitka Release 0.3.14 ===================== This is to inform you about the new stable release of Nuitka. This time it contains mostly organisational improvements, some bug fixes, improved compatibility and cleanups. It is again the result of working towards compilation of a real program (Mercurial). This time, I have added support for proper handling of compiled types by the ``inspect`` module. Bug fixes --------- - Fix for "Missing checks in parameter parsing with star list, star dict and positional arguments". There was whole in the checks for argument counts, now the correct error is given. Fixed in 0.3.13a already. - The simple slice operations with 2 values, not extended with 3 values, were not applying the correct order for evaluation. Fixed in 0.3.13a already. - The simple slice operations couldn't handle ``None`` as the value for lower or upper index. Fixed in 0.3.11a already. - The in-place simple slice operations evaluated the slice index expressions twice, which could cause problems if they had side effects. Fixed in 0.3.11a already. New Features ------------ - Run time patching the ``inspect`` module so it accepts compiled functions, compiled methods, and compiled generator objects. The ``test_inspect`` test of CPython is nearly working unchanged with this. - The generator functions didn't have ``CO_GENERATOR`` set in their code object, setting it made compatible with CPython in this regard too. The inspect module will therefore return correct value for ``inspect.isgeneratorfunction()`` too. New Optimization ---------------- - Slice indexes that are ``None`` are now constant propagated as well. - Slightly more efficient code generation for dual star arg functions, removing useless checks. Cleanups -------- - Moved the Scons, static C++ files, and assembler files to new package ``nuitka.build`` where also now ``SconsInterface`` module lives. - Moved the Qt dialog files to ``nuitka.gui`` - Moved the "unfreezer" code to its own static C++ file. - Some PyLint cleanups. New Tests --------- - New test ``Recursion`` to cover recursive functions. - New test ``Inspection`` to cover the patching of ``inspect`` module. - Cover ``execfile`` on the class level as well in ``ExecEval`` test. - Cover evaluation order of simple slices in ``OrderCheck`` too. Organizational -------------- - There is a new issue tracker available under http://bugs.nuitka.net Please register and report issues you encounter with Nuitka. I have put all the known issues there and started to use it recently. It's Roundup based like http://bugs.python.org is, so people will find it familiar. - The ``setup.py`` is now apparently functional. The source releases for download are made it with, and it appears the binary distributions work too. We may now build a windows installer. It's currently in testing, we will make it available when finished. Summary ------- The new source organisation makes packaging Nuitka really easy now. From here, we can likely provide "binary" package of Nuitka soon. A windows installer will be nice. The patching of ``inspect`` works wonders for compatibility for those programs that insist on checking types, instead of doing duck typing. The function call problem, was an issue found by the Mercurial test suite. For the "hg.exe" to pass all of its test suite, more work may be needed, this is the overall goal I am currently striving for. Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization. Nuitka Release 0.3.13 ===================== This release is mostly the result of working towards compilation of a real programs (Mercurial) and to merge and finalize the frame stack work. Now Nuitka has a correct frame stack at all times, and supports ``func_code`` and ``gi_code`` objects, something previously thought to be impossible. Actually now it's only the "bytecode" objects that won't be there. And not attributes of ``func_code`` are meaningful yet, but in theory can be supported. Due to the use of the "git flow" for Nuitka, most of the bugs listed here were already fixed in on the stable release before this release. This time there were 5 such hot fix releases, sometimes fixing multiple bugs. Bug fixes --------- - In case of syntax errors in the main program, an exception stack was giving that included Nuitka code. Changed to make the same output as CPython does. Fixed in 0.3.12a already. - The star import (``from x import *``) didn't work for submodules. Providing ``*`` as the import list to the respective code allowed to drop the complex lookups we were doing before, and to simply trust CPython C/API to do it correctly. Fixed in 0.3.12 already. - The absolute import is *not* the default of CPython 2.7 it seems. A local ``posix`` package shadows the standard library one. Fixed in 0.3.12 already. - In ``--deep`` mode, a module may contain a syntax error. This is e.g. true of "PyQt" with ``port_v3`` included. These files contain Python3 syntax and fail to be imported in Python2, but that is not to be considered an error. These modules are now skipped with a warning. Fixed in 0.3.12b already. - The code to import modules wasn't using the ``__import__`` built-in, which prevented ``__import__`` overriding code to work. Changed import to use the built-in. Fixed in 0.3.12c already. - The code generated for the ``__import__`` built-in with constant values was doing relative imports only. It needs to attempt relative and absolut imports. Fixed in 0.3.12c already. - The code of packages in "__init__.py" believed it was outside of the package, giving problems for package local imports. Fixed in 0.3.12d already. - It appears that "Scons", which Nuitka uses internally and transparent to you, to execute the compilation and linking tasks, was sometimes not building the binaries or shared libraries, due to a false caching. As a workaround, these are now erased before doing the build. Fixed in 0.3.12d already. - The use of ``in`` and ``not in`` in comparison chains (e.g. ``a < b < c`` is one), wasn't supported yet. The use of these in comparison chains ``a in b in c`` is very strange. Only in the ``test_grammar.py`` it was ever used I believe. Anyway, it's supported now, solving this ``TODO`` and reducing the difference. Fixed in 0.3.12e already. - The order of evaluation for ``in`` and ``not in`` operators wasn't enforced in a portable way. Now it is correct on "ARM" too. Fixed in 0.3.12e already. New Optimization ---------------- - The built-ins ``GeneratorExit`` and ``StopIteration`` are optimized to their Python C/API names where possible as well. Cleanups -------- - The ``__file__`` attribute of modules was the relative filename, but for absolute filenames these become a horrible mess at least on Linux. - Added assertion helpers for sane frame and code objects and use them. - Make use of ``assertObject`` in more places. - Instead of using ``os.path.sep`` all over, added a helper ``Utils.joinpath`` that hides this and using ``os.path.join``. This gives more readable code. - Added traces to the "unfreezer" guarded by a define. Helpful in analyzing import problems. - Some PyLint cleanups removing dead code, unused variables, useless pass statement, etc. New Tests --------- - New tests to cover ``SyntaxError`` and ``IndentationError`` from ``--deep`` imports and in main program. - New test to cover evaluation order of ``in`` and ``not in`` comparisons. - New test to cover package local imports made by the "__init__.py" of the package. Organizational -------------- - Drop "compile_itself.sh" in favor of the new "compile_itself.py", because the later is more portable. - The logging output is now nicer, and for failed recursions, outputs the line that is having the problem. Summary ------- The frame stack work and the ``func_code`` are big for compatibility. The ``func_code`` was also needed for "hg" to work. For Mercurial to pass all of its test suite, more work will be needed, esp. the ``inspect`` module needs to be run-time patched to accept compiled functions and generators too. Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization. Nuitka Release 0.3.12 ===================== This is to inform you about the new release of Nuitka many bug fixes, and substantial improvements especially in the organizational area. There is a new "`User Manual `__" (`PDF `__), with much improved content, a ``sys.meta_path`` based import mechanism for ``--deep`` mode, git flow goodness. This release is generally also the result of working towards compilation of a real programs (Mercurial) and to get things work more nicely on Windows by default. Thanks go to Liu Zhenhai for helping me with this goal. Due to the use of the "git flow", most of the bugs listed here were already fixed in on the stable release before this release. And there were many of these. Bug fixes --------- - The order of evaluation for base classes and class dictionaries was not enforced. Apparently nothing in the CPython test suite did that, I only noticed during debugging that Nuitka gave a different error than CPython did, for a class that had an undefined base class, because both class body and base classes were giving an error. Fixed in 0.3.11a already. - Method objects didn't hold a reference to the used class. The effect was only noticed when ``--python-debug`` was used, i.e. the debug version of Python linked, because then the garbage collector makes searches. Fixed in 0.3.11b already. - Set ``sys.executable`` on Linux as well. On Debian it is otherwise ``/usr/bin/python`` which might be a different version of Python entirely. Fixed in 0.3.11c already. - Embedded modules inside a package could hide package variables of the same name. Learned during PyCON DE about this corner case. Fixed in 0.3.11d already. - Packages could be duplicated internally. This had no effect on generated code other than appearing twice in the list if frozen modules. Fixed in 0.3.11d already. - When embedding modules from outside current directory, the look-up failed. The embedding only ever worked for the compile itself and programs test cases, because they are all in the current directory then. Fixed in 0.3.11e already. - The check for ARM target broke Windows support in the Scons file. Fixed in 0.3.11f already. - The star import from external modules failed with an error in ``--deep`` mode. Fixed in 0.3.11g already. - Modules with a parent package could cause a problem under some circumstances. Fixed in 0.3.11h already. - One call variant, with both list and dict star arguments and keyword arguments, but no positional parameters, didn't have the required C++ helper function implemented. Fixed in 0.3.11h already. - The detection of the CPU core count was broken on my hexacore at least. Gave 36 instead of 6, which is a problem for large programs. Fixed in 0.3.11h already. - The in-line copy of Scons didn't really work on Windows, which was sad, because we added it to simplify installation on Windows precisely because of this. - Cleaning up the build directory from old sources and object files wasn't portable to Windows and therefore wasn't effective there. - From imports where part of the imported were found modules and parts were not, didn't work. Solved by the feature branch ``meta_path_import`` that was merged for this release. - Newer MinGW gave warnings about the default visibility not being possible to apply to class members. Fixed by not setting this default visibility anymore on Windows. - The ``sys.executable`` gave warnings on Windows because of backslashes in the path. Using a raw string to prevent such problems. - The standard library path was hard coded. Changed to run time detection. Cleanups -------- - Version checks on Python runtime now use a new define ``PYTHON_VERSION`` that makes it easier. I don't like ``PY_VERSION_HEX``, because it is so unreadable. Makes some of the checks a lot more safe. - The ``sys.meta_path`` based import from the ``meta_path_import`` feature branch allowed the cleanup the way importing is done. It's a lot less code now. - Removed some unused code. We will aim at making Nuitka the tool to detect dead code really. - Moved ``nuitka.Nodes`` to ``nuitka.nodes.Nodes``, that is what the package is intended for, the split will come later. New Tests --------- - New tests for import variants that previously didn't work: Mixed imports. Imports from a package one level up. Modules hidden by a package variable, etc. - Added test of function call variant that had no test previously. Only found it when compiling "hg". Amazing how nothing in my tests, CPython tests, etc. used it. - Added test to cover the partial success of import statements. - Added test to cover evaluation order of class definitions. Organizational -------------- - Migrated the "README.txt" from org-mode to ReStructured Text, which allows for a more readable document, and to generate a nice "`User Manual `__" in PDF form. - The amount of information in "README.txt" was increased, with many more subjects are now covered, e.g. "git flow" and how to join Nuitka development. It's also impressive to see what code blocks and syntax highlighting can do for readability. - The Nuitka git repository has seen multiple hot fixes. These allowed to publish bug fixes immediately after they were made, and avoided the need for a new release just to get these out. This really saves me a lot of time too, because I can postpone releasing the new version until it makes sense because of other things. - Then there was a feature branch ``meta_path_import`` that lived until being merged to ``develop`` to improve the import code, which is now released on ``master`` as stable. Getting that feature right took a while. - And there is the feature branch ``minimize_CPython26_tests_diff`` which has some success already in documenting the required changes to the "CPython26" test suite and in reducing the amount of differences, while doing it. We have a frame stack working there, albeit in too ugly code form. - The release archives are now built using ``setuptools``. You can now also download a zip file, which is probably more Windows friendly. The intention is to work on that to make ``setup.py`` produce a Nuitka install that won't rely on any environment variables at all. Right now ``setup.py`` won't even allow any other options than ``sdist`` to be given. - Ported "compile_itself.sh" to "compile_itself.py", i.e. ported it to Python. This way, we can execute it easily on Windows too, where it currently still fails. Replacing ``diff``, ``rm -rf``, etc. is a challenge, but it reduces the dependency on MSYS tools on Windows. - The compilation of standard library is disabled by default, but ``site`` or ``dist`` packages are now embedded. To include even standard library, there is a ``--really-deep`` option that has to be given in addition to ``--deep``, which forces this. Summary ------- Again, huge progress. The improved import mechanism is very beautiful. It appears that little is missing to compile real world programs like "hg" with Nuitka. The next release cycle will focus on that and continue to improve the Windows support which appears to have some issues. Nuitka Release 0.3.11 ===================== This is to inform you about the new release of Nuitka with some bug fixes and portability work. This release is generally cleaning up things, and makes Nuitka portable to ARM Linux. I used to host the Nuitka homepage on that machine, but now that it's no longer so, I can run heavy compile jobs on it. To my surprise, it found many portability problems. So I chose to fix that first, the result being that Nuitka now works on ARM Linux too. Bug fixes --------- - The order of slice expressions was not correct on x86 as well, and I found that with new tests only. So the porting to ARM revealed a bug category, I previously didn't consider. - The use of ``linux2`` in the Scons file is potentially incompatible with Linux 3.0, although it seems that at least on Debian the ``sys.platform`` was changed back to ``linux2``. Anyway, it's probably best to allow just anything that starts with ``linux`` these days. - The ``print`` statement worked like a ``print`` function, i.e. it first evaluated all printed expressions, and did the output only then. That is incompatible in case of exceptions, where partial outputs need to be done, and so that got fixed. New Optimization ---------------- - Function calls now each have a dedicated helper function, avoiding in some cases unnecessary work. We will may build further on this and in-line ``PyObject_Call`` differently for the special cases. Cleanups -------- - Moved many C++ helper declarations and in-line implementations to dedicated header files for better organisation. - Some dependencies were removed and consolidated to make the dependency graph sane. - Multiple decorators were in reverse order in the node tree. The code generation reversed it back, so no bug, yet that was a distorted tree. Finding this came from the ARM work, because the "reversal" was in fact just the argument evaluation order of C++ under x86/x64, but on ARM that broke. Correcting it highlighted this issue. - The deletion of slices, was not using ``Py_ssize`` for indexes, disallowing some kinds of optimization, so that was harmonized. - The function call code generation got a general overhaul. It is now more consistent, has more helpers available, and creates more readable code. - PyLint is again happier than ever. New Tests --------- - There is a new basic test ``OrderChecks`` that covers the order of expression evaluation. These problems were otherwise very hard to detect, and in some cases not previously covered at all. - Executing Nuitka with Python3 (it won't produce correct Python3 C/API code) is now part of the release tests, so non-portable code of Nuitka gets caught. Organizational -------------- - Support for ARM Linux. I will make a separate posting on the challenges of this. Suffice to say now, that C++ leaves way too much things unspecified. - The Nuitka git repository now uses "git flow". The new git policy will be detailed in another `separate posting `__. - There is an unstable ``develop`` branch in which the development occurs. For this release ca. 40 commits were done to this branch, before merging it. I am also doing more fine grained commits now. - Unlike previously, there is ``master`` branch for the stable release. - There is a script "make-dependency-graph.sh" (Update: meanwhile it was renamed to "make-dependency-graph.py") to produce a dependency graphs of Nuitka. I detected a couple of strange things through this. - The Python3 ``__pycache__`` directories get removed too by the cleanup script. Numbers ------- We only have "PyStone" now, and on a new machine, so the numbers cannot be compared to previous releases: python 2.6:: Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second Nuitka 0.3.11 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second So this a speedup factor of 258%, last time on another machine it was 240%. Yet it only proves that the generated and compiled are more efficient than bytecode, but Nuitka doesn't yet do the relevant optimization. Only once it does, the factor will be significantly higher. Summary ------- Overall, there is quite some progress. Nuitka is a lot cleaner now, which will help us later only. I wanted to get this out, mostly because of the bug fixes, and of course just in case somebody attempts to use it on ARM. Nuitka Release 0.3.10 ===================== This new release is major milestone 2 work, enhancing practically all areas of Nuitka. The focus was roundup and breaking new grounds with structural optimization enhancements. Bug fixes --------- - Exceptions now correctly stack. When you catch an exception, there always was the exception set, but calling a new function, and it catching the exception, the values of ``sys.exc_info()`` didn't get reset after the function returned. This was a small difference (of which there are nearly none left now) but one that might effect existing code, which affects code that calls functions in exception handling to check something about it. So it's good this is resolved now too. Also because it is difficult to understand, and now it's just like CPython behaves, which means that we don't have to document anything at all about it. - Using ``exec`` in generator functions got fixed up. I realized that this wouldn't work while working on other things. It's obscure yes, but it ought to work. - Lambda generator functions can now be nested and in generator functions. There were some problems here with the allocation of closure variables that got resolved. - List contractions could not be returned by lambda functions. Also a closure issue. - When using a mapping for globals to ``exec`` or ``eval`` that had a side effect on lookup, it was evident that the lookup was made twice. Correcting this also improves the performance for the normal case. New Optimization ---------------- - Statically raised as well as predicted exceptions are propagated upwards, leading to code and block removal where possible, while maintaining the side effects. This is brand new and doesn't do everything possible yet. Most notable, the matching of raised exception to handlers is not yet performed. - Built-in exception name references and creation of instances of them are now optimized as well, which leads to faster exception raising/catching for these cases. - More kinds of calls to built-ins are handled, positional parameters are checked and more built-ins are covered. Notable is that now checks are performed if you didn't potentially overload e.g. the ``len`` with your own version in the module. Locally it was always detected already. So it's now also safe. - All operations and comparisons are now simulated if possible and replaced with their result. - In the case of predictable true or false conditions, not taken branches are removed. - Empty branches are now removed from most constructs, leading to sometimes cleaner code generated. Cleanups -------- - Removed the lambda body node and replaced it with function body. This is a great win for the split into body and builder. Regular functions and lambda functions now only differ in how the created body is used. - Large cleanup of the operation/comparison code. There is now only use of a simulator function, which exists for every operator and comparison. This one is then used in a prediction call, shared with the built-in predictions. - Added a ``Tracing`` module to avoid future imports of ``print_function``, which annoyed me many times by causing syntax failures for when I quickly added a print statement, not noting it must have the braces. - PyLint is happier than ever. New Tests --------- - Enhanced ``OverflowFunctions`` test to cover even deeper nesting of overflow functions taking closure from each level. While it's not yet working, this makes clearer what will be needed. Even if this code is obscure, I would like to be that correct here. - Made ``Operators`` test to cover the `` operator as well. - Added to ``ListContractions`` the case where a contraction is returned by a lambda function, but still needs to leak its loop variable. - Enhanced ``GeneratorExpressions`` test to cover lambda generators, which is really crazy code: .. code-block:: python def y(): yield((yield 1),(yield 2)) - Added to ``ExecEval`` a case where the ``exec`` is inside a generator, to cover that too. - Activated the testing of ``sys.exc_info()`` in ``ExceptionRaising`` test. This was previously commented out, and now I added stuff to illustrate all of the behavior of CPython there. - Enhanced ``ComparisonChains`` test to demonstrate that the order of evaluations is done right and that side effects are maintained. - Added ``BuiltinOverload`` test to show that overloaded built-ins are actually called and not the optimized version. So code like this has to print 2 lines: .. code-block:: python from __builtin__ import len as _len def len(x): print x return _len(x) print len(range(9)) Organizational -------------- - Changed "README.txt" to no longer say that "Scons" is a requirement. Now that it's included (patched up to work with ``ctypes`` on Windows), we don't have to say that anymore. - Documented the status of optimization and added some more ideas. - There is now an option to dump the node tree after optimization as XML. Not currently use, but is for regression testing, to identify where new optimization and changes have an impact. This make it more feasible to be sure that Nuitka is only becoming better. - Executable with Python3 again, although it won't do anything, the necessary code changes were done. Summary ------- It's nice to see, that I some long standing issues were resolved, and that structural optimization has become almost a reality. The difficult parts of exception propagation are all in place, now it's only details. With that we can eliminate and predict even more of the stupid code of "pybench" at compile time, achieving more infinite speedups. Nuitka Release 0.3.9 ==================== This is about the new release of Nuitka which some bug fixes and offers a good speed improvement. This new release is major milestone 2 work, enhancing practically all areas of Nuitka. The main focus was on faster function calls, faster class attributes (not instance), faster unpacking, and more built-ins detected and more thoroughly optimizing them. Bug fixes --------- - Exceptions raised inside with statements had references to the exception and traceback leaked. - On Windows the binaries ``sys.executable`` pointed to the binary itself instead of the Python interpreter. Changed, because some code uses ``sys.executable`` to know how to start Python scripts. - There is a bug (fixed in their repository) related to C++ raw strings and C++ "trigraphs" that affects Nuitka, added a workaround that makes Nuitka not emit "trigraphs" at all. - The check for mutable constants was erroneous for tuples, which could lead to assuming a tuple with only mutable elements to be not mutable, which is of course wrong. New Optimization ---------------- This time there are so many new optimization, it makes sense to group them by the subject. Exceptions ~~~~~~~~~~ - The code to add a traceback is now our own, which made it possible to use frames that do not contain line numbers and a code object capable of lookups. - Raising exceptions or adding to tracebacks has been made way faster by reusing a cached frame objects for the task. - The class used for saving exceptions temporarily (e.g. used in ``try``/``finally`` code, or with statement) has been improved so it doesn't make a copy of the exception with a C++ ``new`` call, but it simply stores the exception properties itself and creates the exception object only on demand, which is more efficient. - When catching exceptions, the addition of tracebacks is now done without exporting and re-importing the exception to Python, but directly on the exception objects traceback, this avoids a useless round trip. Function Calls ~~~~~~~~~~~~~~ - Uses of PyObject_Call provide ``NULL`` as the dictionary, instead of an empty dictionary, which is slightly faster for function calls. - There are now dedicated variants for complex function calls with ``*`` and ``**`` arguments in all forms. These can take advantage of easier cases. For example, a merge with star arguments is only needed if there actually were any of these. - The check for non-string values in the ``**`` arguments can now be completely short-cut for the case of a dictionary that has never had a string added. There is now code that detects this case and skips the check, eliminating it as a performance concern. Parameter Parsing ~~~~~~~~~~~~~~~~~ - Reversed the order in which parameters are checked. Now the keyword dictionary is iterated first and only then the positional arguments after that is done. This iteration is not only much faster (avoiding repeated lookups for each possible parameter), it also can be more correct, in case the keyword argument is derived from a dictionary and its keys mutate it when being compared. - Comparing parameter names is now done with a fast path, in which the pointer values are compare first. This can avoid a call to the comparison at all, which has become very likely due to the interning of parameter name strings, see below. - Added a dedicated call to check for parameter equality with rich equality comparison, which doesn't raise an exception. - Unpacking of tuples is now using dedicated variants of the normal unpacking code instead of rolling out everything themselves. Attribute Access ~~~~~~~~~~~~~~~~ - The class type (in executables, not yet for extension modules) is changed to a faster variant of our own making that doesn't consider the restricted mode a possibility. This avoids very expensive calls, and makes accessing class attributes in compiled code and in non-compiled code faster. - Access to attributes (but not of instances) got in-lined and therefore much faster. Due to other optimization, a specific step to intern the string used for attribute access is not necessary with Nuitka at all anymore. This made access to attributes about 50% faster which is big of course. Constants ~~~~~~~~~ - The bug for mutable tuples also caused non-mutable tuples to be considered as mutable, which lead to less efficient code. - The constant creation with the g++ bug worked around, can now use raw strings to create string constants, without resorting to un-pickling them as a work around. This allows us to use ``PyString_FromStringAndSize`` to create strings again, which is obviously faster, and had not been done, because of the confusion caused by the g++ bug. - For string constants that are usable as attributes (i.e. match the identifier regular expression), these are now interned, directly after creation. With this, the check for identical value of pointers for parameters has a bigger chance to succeed, and this saves some memory too. - For empty containers (set, dict, list, tuple) the constants created are now are not unstreamed, but created with the dedicated API calls, saving a bit of code and being less ugly. - For mutable empty constant access (set, dict, list) the values are no longer made by copying the constant, but instead with the API functions to create new ones. This makes code like ``a = []`` a tiny bit faster. - For slice indices the code generation now takes advantage of creating a C++ ``Py_ssize_t`` from constant value if possible. Before it was converting the integer constant at run time, which was of course wasteful even if not (very) slow. Iteration ~~~~~~~~~ - The creation of iterators got our own code. This avoids a function call and is otherwise only a small gain for anything but sequence iterators. These may be much faster to create now, as it avoids another call and repeated checks. - The next on iterator got our own code too, which has simpler code flow, because it avoids the double check in case of NULL returned. - The unpack check got simlar code to the next iterator, it also has simpler code flow now and avoids double checks. Built-ins ~~~~~~~~~ - Added support for the ``list``, ``tuple``, ``dict``, ``str``, ``float`` and ``bool`` built-ins along with optimizing their use with constant parameter. - Added support for the ``int`` and ``long`` built-ins, based on a new "call spec" object, that detects parameter errors at compile time and raises appropriate exceptions as required, plus it deals with keyword arguments just as well. So, to Nuitka it doesn't matter now it you write ``int(value) ``or ``int(x = value)`` anymore. The ``base`` parameter of these built-ins is also supported. The use of this call spec mechanism will the expanded, currently it is not applied to the built-ins that take only one parameter. This is a work in progress as is the whole built-ins business as not all the built-ins are covered yet. Cleanups ~~~~~~~~ - In 0.3.8 per module global classes were introduced, but the ``IMPORT_MODULE`` kept using the old universal class, this got resolved and the old class is now fully gone. - Using ``assertObject`` in more cases, and in more places at all, catches errors earlier on. - Moved the addition to tracebacks into the ``_PythonException`` class, where it works directly on the contained traceback. This is cleaner as it no longer requires to export exceptions to Python, just to add a traceback entry. - Some ``PyLint`` cleanups were done, reducing the number of reports a bit, but there is still a lot to do. - Added a ``DefaultValueIdentifier`` class that encapsulates the access to default values in the parameter parsing more cleanly. - The module ``CodeTemplatesListContractions`` was renamed to ``CodeTemplatesContractions`` to reflect the fact that it deals with all kinds of contractions (also set and dict contractions), not just list contractions. - Moved the with related template to its own module ``CodeTemplatesWith``, so its easier to find. - The options handling for g++ based compilers was cleaned up, so that g++ 4.6 and MinGW are better supported now. - Documented more aspects of the Scons build file. - Some more generated code white space fixes. - Moved some helpers to dedicated files. There is now ``calling.hpp`` for function calls, an ``importing.cpp`` for import related stuff. - Moved the manifest generation to the scons file, which now produces ready to use executables. New Tests --------- - Added a improved version of "pybench" that can cope with the "0 ms" execution time that Nuitka has for some if its sub-tests. - Reference counting test for with statement was added. - Micro benchmarks to demonstrate try finally performance when an exception travels through it. - Micro benchmark for with statement that eats up exceptions raised inside the block. - Micro benchmarks for the read and write access to class attributes. - Enhanced ``Printing`` test to cover the trigraphs constant bug case. Output is required to make the error detectable. - Enhanced ``Constants`` test to cover repeated mutation of mutable tuple constants, this covers the bug mentioned. Organizational -------------- - Added a credits section to the "README.txt" where I give credit to the people who contributed to Nuitka, and the projects it is using. I will make it a separate posting to cite these. - Documented the requirements on the compiler more clearly, document the fact that we require scons and which version of Python (2.6 or 2.7). - The is now a codespeed implementation up and running with historical data for up to Nuitka 0.3.8 runs of "PyStone" and with pybench. It will be updated for 0.3.9 once I have the infrastructure in place to do that automatically. - The cleanup script now also removes .so files. - The handling of options for g++ got improved, so it's the same for g++ and MinGW compilers, plus adequate errors messages are given, if the compiler version is too low. - There is now a ``--unstriped`` option that just keeps the debug information in the file, but doesn't keep the assertions. This will be helpful when looking at generated assembler code from Nuitka to not have the distortions that ``--debug`` causes (reduced optimization level, assertions, etc.) and instead a clear view. Nuitka Release 0.3.8 ==================== This is to inform you about the new release of Nuitka with some real news and a slight performance increase. The significant news is added "Windows Support". You can now hope to run Nuitka on Windows too and have it produce working executables against either the standard Python distribution or a MinGW compiled Python. There are still some small things to iron out, and clearly documentation needs to be created, and esp. the DLL hell problem of ``msvcr90.dll`` vs. ``msvcrt.dll``, is not yet fully resolved, but appears to be not as harmful, at least not on native Windows. I am thanking Khalid Abu Bakr for making this possible. I was surprised to see this happen. I clearly didn't make it easy. He found a good way around ``ucontext``, identifier clashes, and a very tricky symbol problems where the CPython library under Windows exports less than under Linux. Thanks a whole lot. Currently the Windows support is considered experimental and works with MinGW 4.5 or higher only. Otherwise there have been the usual round of performance improvements and more cleanups. This release is otherwise milestone 2 work only, which will have to continue for some time more. Bug fixes --------- - Lambda generators were not fully compatible, their simple form could yield an extra value. The behavior for Python 2.6 and 2.7 is also different and Nuitka now mimics both correctly, depending on the used Python version - The given parameter count cited in the error message in case of too many parameters, didn't include the given keyword parameters in the error message. - There was an ``assert False`` right after warning about not found modules in the ``--deep`` mode, which was of course unnecessary. New Optimization ---------------- - When unpacking variables in assignments, the temporary variables are now held in a new temporary class that is designed for the task specifically. This avoids the taking of a reference just because the ``PyObjectTemporary`` destructor insisted on releasing one. The new class ``PyObjectTempHolder`` hands the existing reference over and releases only in case of exceptions. - When unpacking variable in for loops, the value from the iterator may be directly assigned, if it's to a variable. In general this would be possible for every assignment target that cannot raise, but the infrastructure cannot tell yet, which these would be. This will improve with more milestone 3 work. - Branches with only ``pass`` inside are removed, ``pass`` statements are removed before the code generation stage. This makes it easier to achieve and decide empty branches. - There is now a global variable class per module. It appears that it is indeed faster to roll out a class per module accessing the ``module *`` rather than having one class and use a ``module **``, which is quite disappointing from the C++ compiler. - Also ``MAKE_LIST`` and ``MAKE_TUPLE`` have gained special cases for the 0 arguments case. Even when the size of the variadic template parameters should be known to the compiler, it seems, it wasn't eliminating the branch, so this was a speedup measured with valgrind. - Empty tried branches are now replaced when possible with ``try``/``except`` statements, ``try``/``finally`` is simplified in this case. This gives a cleaner tree structure and less verbose C++ code which the compiler threw away, but was strange to have in the first place. - In conditions the ``or`` and ``and`` were evaluated with Python objects instead of with C++ bool, which was unnecessary overhead. - List contractions got more clever in how they assign from the iterator value. It now uses a ``PyObjectTemporary`` if it's assigned to multiple values, a ``PyObjectTempHolder`` if it's only assigned once, to something that could raise, or a ``PyObject *`` if an exception cannot be raised. This avoids temporary references completely for the common case. Cleanups -------- - The ``if``, ``for``, and ``while`` statements had always empty ``else`` nodes which were then also in the generated C++ code as empty branches. No harm to performance, but this got cleaned up. - Some more generated code white space fixes. New Tests --------- - The CPython 2.7 test suite now also has the ``doctests`` extracted to static tests, which improves test coverage for Nuitka again. This was previously only done for CPython 2.6 test suite, but the test suites are different enough to make this useful, e.g. to discover newly changed behavior like with the lambda generators. - Added Shed Skin 0.7.1 examples as benchmarks, so we can start to compare Nuitka performance in these tests. These will be the focus of numbers for the 0.4.x release series. - Added a micro benchmark to check unpacking behavior. Some of these are needed to prove that a change is an actual improvement, when its effect can go under in noise of in-line vs. no in-line behavior of the C++ compiler. - Added "pybench" benchmark which reveals that Nuitka is for some things much faster, but there are still fields to work on. This version needed changes to stand the speed of Nuitka. These will be subject of a later posting. Organizational -------------- - There is now a "tests/benchmarks/micro" directory to contain tiny benchmarks that just look at a single aspect, but have no other meaning, e.g. the "PyStone" extracts fall into this category. - There is now a ``--windows-target`` option that attempts a cross-platform build on Linux to Windows executable. This is using "MingGW-cross-env" cross compilation tool chain. It's not yet working fully correctly due to the DLL hell problem with the C runtime. I hope to get this right in subsequent releases. - The ``--execute`` option uses wine to execute the binary if it's a cross-compile for windows. - Native windows build is recognized and handled with MinGW 4.5, the VC++ is not supported yet due to missing C++0x support. - The basic test suite ran with Windows so far only and some adaptations were necessary. Windows new lines are now ignored in difference check, and addresses under Windows are upper case, small things. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.8 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.27 This machine benchmarks at 185185 pystones/second This is a 140% speed increase of 0.3.8 compared to CPython, up from 132% compared to the previous release. Nuitka Release 0.3.7 ==================== This is about the new release with focus on performance and cleanups. It indicates significant progress with the milestone this release series really is about as it adds a ``compiled_method`` type. So far functions, generator function, generator expressions were compiled objects, but in the context of classes, functions were wrapped in CPython ``instancemethod`` objects. The new ``compiled_method`` is specifically designed for wrapping ``compiled_function`` and therefore more efficient at it. Bug fixes --------- - When using ``Python`` or ``Nuitka.py`` to execute some script, the exit code in case of "file not found" was not the same as CPython. It should be 2, not 1. - The exit code of the created programs (``--deep`` mode) in case of an uncaught exception was 0, now it an error exit with value 1, like CPython does it. - Exception tracebacks created inside ``with`` statements could contain duplicate lines, this was corrected. New Optimization ---------------- - Global variable assignments now also use ``assign0`` where no reference exists. The assignment code for module variables is actually faster if it needs not drop the reference, but clearly the code shouldn't bother to take it on the outside just for that. This variant existed, but wasn't used as much so far. - The instance method objects are now Nuitka's own compiled type too. This should make things slightly faster by itself. - Our new compiled method objects support dedicated method parsing code, where ``self`` is passed directly, allowing to make calls taking a fast path in parameter parsing. This avoids allocating/freeing a ``tuple`` object per method call, while reduced 3% ticks in "PyStone" benchmark, so that's significant. - Solved a ``TODO`` of ``BUILTIN_RANGE`` to change it to pre-allocating the list in the final size as we normally do everywhere else. This was a tick reduction of 0.4% in "PyStone" benchmark, but the measurement method normalizes on loop speed, so it's not visible in the numbers output. - Parameter variables cannot possibly be uninitialized at creation and most often they are never subject to a ``del`` statement. Adding dedicated C++ variable classes gave a big speedup, around 3% of "PyStone" benchmark ticks. - Some abstract object operations were re-implemented, which allows to avoid function calls e.g. in the ``ITERATOR_NEXT`` case, this gave a few percent on "PyStone" as well. Cleanups -------- - New package ``nuitka.codegen`` to contain all code generation related stuff, moved ``nuitka.templates`` to ``nuitka.codegen.templates`` as part of that. - Inside the ``nuitka.codegen`` package the ``MainControl`` module now longer reaches into ``Generator`` for simple things, but goes through ``CodeGeneration`` for everything now. - The ``Generator`` module uses almost no tree nodes anymore, but instead gets information passed in function calls. This allows for a cleanup of the interface towards ``CodeGeneration``. Gives a cleaner view on the C++ code generation, and generally furthers the goal of other than C++ language backends. - More "PyLint" work, many of the reported warnings have been addressed, but it's not yet happy. - Defaults for ``yield`` and ``return`` are ``None`` and these values are now already added (as constants) during tree building so that no such special cases need to be dealt with in ``CodeGeneration`` and future analysis steps. - Parameter parsing code has been unified even further, now the whole entry point is generated by one of the function in the new ``nuitka.codegen.ParameterParsing`` module. - Split variable, exception, built-in helper classes into separate header files. New Tests --------- - The exit codes of CPython execution and Nuitka compiled programs are now compared as well. - Errors messages of methods are now covered by the ``ParameterErrors`` test as well. Organizational -------------- - A new script "benchmark.sh" (now called "run-valgrind.py") script now starts "kcachegrind" to display the valgrind result directly. One can now use it to execute a test and inspect valgrind information right away, then improve it. Very useful to discover methods for improvements, test them, then refine some more. - The "check-release.sh" script needs to unset ``NUITKA_EXTRA_OPTIONS`` or else the reflection test will trip over the changed output paths. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.7 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.28 This machine benchmarks at 178571 pystones/second This is a 132% speed of 0.3.7 compared to CPython, up from 109% compare to the previous release. This is a another small increase, that can be fully attributed to milestone 2 measures, i.e. not analysis, but purely more efficient C++ code generation and the new compiled method type. One can now safely assume that it is at least twice as fast, but I will try and get the PyPy or Shedskin test suite to run as benchmarks to prove it. No milestone 3 work in this release. I believe it's best to finish with milestone 2 first, because these are quite universal gains that we should have covered. Nuitka Release 0.3.6 ==================== The major point this for this release is cleanup work, and generally bug fixes, esp. in the field of importing. This release cleans up many small open ends of Nuitka, closing quite a bunch of consistency ``TODO`` items, and then aims at cleaner structures internally, so optimization analysis shall become "easy". It is a correctness and framework release, not a performance improvement at all. Bug fixes --------- - Imports were not respecting the ``level`` yet. Code like this was not working, now it is: .. code-block:: python from .. import something - Absolute and relative imports were e.g. both tried all the time, now if you specify absolute or relative imports, it will be attempted in the same way than CPython does. This can make a difference with compatibility. - Functions with a "locals dict" (using ``locals`` built-in or ``exec`` statement) were not 100% compatible in the way the locals dictionary was updated, this got fixed. It seems that directly updating a dict is not what CPython does at all, instead it only pushes things to the dictionary, when it believes it has to. Nuitka now does the same thing, making it faster and more compatible at the same time with these kind of corner cases. - Nested packages didn't work, they do now. Nuitka itself is now successfully using nested packages (e.g. ``nuitka.transform.optimizations``) New Features ------------ - The ``--lto`` option becomes usable. It's not measurably faster immediately, and it requires g++ 4.6 to be available, but then it at least creates smaller binaries and may provide more optimization in the future. New Optimization ---------------- - Exceptions raised by pre-computed built-ins, unpacking, etc. are now transformed to raising the exception statically. Cleanups -------- - There is now a ``getVariableForClosure`` that a variable provider can use. Before that it guessed from ``getVariableForReference`` or ``getVariableForAssignment`` what might be the intention. This makes some corner cases easier. - Classes, functions and lambdas now also have separate builder and body nodes, which enabled to make getSameScopeNodes() really simple. Either something has children which are all in a new scope or it has them in the same scope. - Twisted workarounds like ``TransitiveProvider`` are no longer needed, because class builder and class body were separated. - New packages ``nuitka.transform.optimizations`` and ``nuitka.transform.finalizations``, where the first was ``nuitka.optimizations`` before. There is also code in ``nuitka.transform`` that was previously in a dedicated module. This allowed to move a lot of displaced code. - ``TreeBuilding`` now has fast paths for all 3 forms, things that need a "provider", "node", and "source_ref"; things that need "node" and "source_ref"; things that need nothing at all, e.g. pass. - Variables now avoid building duplicated instances, but instead share one. Better for analysis of them. New Tests --------- - The Python 2.7 test suite is no longer run with Python 2.6 as it will just crash with the same exception all the time, there is no ``importlib`` in 2.6, but every test is using that through test_support. - Nested packages are now covered with tests too. - Imports of upper level packages are covered now too. Organizational -------------- - Updated the "README.txt" with the current plan on optimization. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.6 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.31 This machine benchmarks at 161290 pystones/second This is 109% for 0.3.6, but no change from the previous release. No surprise, because no new effective new optimization means have been implemented. Stay tuned for future release for actual progress. Nuitka Release 0.3.5 ==================== This new release of Nuitka is an overall improvement on many fronts, there is no real focus this time, likely due to the long time it was in the making. The major points are more optimization work, largely enhanced import handling and another improvement on the performance side. But there are also many bug fixes, more test coverage, usability and compatibility. Something esp. noteworthy to me and valued is that many important changes were performed or at least triggered by Nicolas Dumazet, who contributed a lot of high quality commits as you can see from the gitweb history. He appears to try and compile Mercurial and Nuitka, and this resulted in important contributions. Bug fixes --------- - Nicolas found a reference counting bug with nested parameter calls. Where a function had parameters of the form ``a, (b,c)`` it could crash. This got fixed and covered with a reference count test. - Another reference count problem when accessing the locals dictionary was corrected. - Values ``0.0`` and ``-0.0`` were treated as the same. They are not though, they have a different sign that should not get lost. - Nested contractions didn't work correctly, when the contraction was to iterate over another contraction which needs a closure. The problem was addressing by splitting the building of a contraction from the body of the contraction, so that these are now 2 nodes, making it easy for the closure handling to get things right. - Global statements in function with local ``exec()`` would still use the value from the locals dictionary. Nuitka is now compatible to CPython with this too. - Nicolas fixed problems with modules of the same name inside different packages. We now use the full name including parent package names for code generation and look-ups. - The ``__module__`` attribute of classes was only set after the class was created. Now it is already available in the class body. - The ``__doc__`` attribute of classes was not set at all. Now it is. - The relative import inside nested packages now works correctly. With Nicolas moving all of Nuitka to a package, the compile itself exposed many weaknesses. - A local re-raise of an exception didn't have the original line attached but the re-raise statement line. New Features ------------ - Modules and packages have been unified. Packages can now also have code in "__init__.py" and then it will be executed when the package is imported. - Nicolas added the ability to create deep output directory structures without having to create them beforehand. This makes ``--output-dir=some/deep/path`` usable. - Parallel build by Scons was added as an option and enabled by default, which enhances scalability for ``--deep`` compilations a lot. - Nicolas enhanced the CPU count detection used for the parallel build. Turned out that ``multithreading.cpu_count()`` doesn't give us the number of available cores, so he contributed code to determine that. - Support for upcoming g++ 4.6 has been added. The use of the new option ``--lto`` has been been prepared, but right now it appears that the C++ compiler will need more fixes, before we can this feature with Nuitka. - The ``--display-tree`` feature got an overhaul and now displays the node tree along with the source code. It puts the cursor on the line of the node you selected. Unfortunately I cannot get it to work two-way yet. I will ask for help with this in a separate posting as we can really use a "python-qt" expert it seems. - Added meaningful error messages in the "file not found" case. Previously I just didn't care, but we sort of approach end user usability with this. New Optimization ---------------- - Added optimization for the built-in ``range()`` which otherwise requires a module and ``builtin`` module lookup, then parameter parsing. Now this is much faster with Nuitka and small ranges (less than 256 values) are converted to constants directly, avoiding run time overhead entirely. - Code for re-raise statements now use a simple re-throw of the exception where possible, and only do the hard work where the re-throw is not inside an exception handler. - Constant folding of operations and comparisons is now performed if the operands are constants. - Values of some built-ins are pre-computed if the operands are constants. - The value of module attribute ``__name__`` is replaced by a constant unless it is assigned to. This is the first sign of upcoming constant propagation, even if only a weak one. - Conditional statement and/or their branches are eliminated where constant conditions allow it. Cleanups -------- - Nicolas moved the Nuitka source code to its own ``nuitka`` package. That is going to make packaging it a lot easier and allows cleaner code. - Nicolas introduced a fast path in the tree building which often delegates (or should do that) to a function. This reduced a lot of the dispatching code and highlights more clearly where such is missing right now. - Together we worked on the line length issues of Nuitka. We agreed on a style and very long lines will vanish from Nuitka with time. Thanks for pushing me there. - Nicolas also did provide many style fixes and general improvements, e.g. using ``PyObjectTemporary`` in more places in the C++ code, or not using ``str.find`` where ``x in y`` is a better choice. - The node structure got cleaned up towards the direction that assigments always have an assignment as a child. A function definition, or a class definition, are effectively assignments, and in order to not have to treat this as special cases everywhere, they need to have assignment targets as child nodes. Without such changes, optimization will have to take too many things into account. This is not yet completed. - Nicolas merged some node tree building functions that previously handled deletion and assigning differently, giving us better code reuse. - The constants code generation was moved to a ``__constants.cpp`` where it doesn't make __main__.cpp so much harder to read anymore. - The module declarations have been moved to their own header files. - Nicolas cleaned up the scripts used to test Nuitka big time, removing repetitive code and improving the logic. Very much appreciated. - Nicolas also documented a things in the Nuitka source code or got me to document things that looked strange, but have reasons behind it. - Nicolas solved the ``TODO`` related to built-in module accesses. These will now be way faster than before. - Nicolas also solved the ``TODO`` related to the performance of "locals dict" variable accesses. - Generator.py no longer contains classes. The Contexts objects are supposed to contain the state, and as such the generator objects never made much sense. - Also with the help of Scons community, I figured out how to avoid having object files inside the ``src`` directory of Nuitka. That should also help packaging, now all build products go to the .build directory as they should. - The vertical white space of the generated C++ got a few cleanups, trailing/leading new line is more consistent now, and there were some assertions added that it doesn't happen. New Tests --------- - The CPython 2.6 tests are now also run by CPython 2.7 and the other way around and need to report the same test failure reports, which found a couple of issues. - Now the test suite is run with and without ``--debug`` mode. - Basic tests got extended to cover more topics and catch more issues. - Program tests got extended to cover code in packages. - Added more exec scope tests. Currently inlining of exec statements is disabled though, because it requires entirely different rules to be done right, it has been pushed back to the next release. Organizational -------------- - The ``g++-nuitka`` script is no more. With the help of the Scons community, this is now performed inside the scons and only once instead of each time for every C++ file. - When using ``--debug``, the generated C++ is compiled with ``-Wall`` and ``-Werror`` so that some form of bugs in the generated C++ code will be detected immediately. This found a few issues already. - There is a new git merge policy in place. Basically it says, that if you submit me a pull request, that I will deal with it before publishing anything new, so you can rely on the current git to provide you a good base to work on. I am doing more frequent pre-releases already and I would like to merge from your git. - The "README.txt" was updated to reflect current optimization status and plans. There is still a lot to do before constant propagation can work, but this explains things a bit better now. I hope to expand this more and more with time. - There is now a "misc/clean-up.sh" script that prints the commands to erase all the temporary files sticking around in the source tree. That is for you if you like me, have other directories inside, ignored, that you don't want to delete. - Then there is now a script that prints all source filenames, so you can more easily open them all in your editor. - And very important, there is now a "check-release.sh" script that performs all the tests I think should be done before making a release. - Pylint got more happy with the current Nuitka source. In some places, I added comments where rules should be granted exceptions. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.5 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.31 This machine benchmarks at 161290 pystones/second This is 109% for 0.3.5, up from 91% before. Overall this release is primarily an improvement in the domain of compatibility and contains important bug and feature fixes to the users. The optimization framework only makes a first showing of with the framework to organize them. There is still work to do to migrate optimization previously present It will take more time before we will see effect from these. I believe that even more cleanups of ``TreeBuilding``, ``Nodes`` and ``CodeGeneration`` will be required, before everything is in place for the big jump in performance numbers. But still, passing 100% feels good. Time to rejoice. Nuitka Release 0.3.4 ==================== This new release of Nuitka has a focus on re-organizing the Nuitka generated source code and a modest improvement on the performance side. For a long time now, Nuitka has generated a single C++ file and asked the C++ compiler to translate it to an executable or shared library for CPython to load. This was done even when embedding many modules into one (the "deep" compilation mode, option ``--deep``). This was simple to do and in theory ought to allow the compiler to do the most optimization. But for large programs, the resulting source code could have exponential compile time behavior in the C++ compiler. At least for the GNU g++ this was the case, others probably as well. This is of course at the end a scalability issue of Nuitka, which now has been addressed. So the major advancement of this release is to make the ``--deep`` option useful. But also there have been a performance improvements, which end up giving us another boost for the "PyStone" benchmark. Bug fixes --------- - Imports of modules local to packages now work correctly, closing the small compatibility gap that was there. - Modules with a "-" in their name are allowed in CPython through dynamic imports. This lead to wrong C++ code created. (Thanks to Li Xuan Ji for reporting and submitting a patch to fix it.) - There were warnings about wrong format used for ``Ssize_t`` type of CPython. (Again, thanks to Li Xuan Ji for reporting and submitting the patch to fix it.) - When a wrong exception type is raised, the traceback should still be the one of the original one. - Set and dict contractions (Python 2.7 features) declared local variables for global variables used. This went unnoticed, because list contractions don't generate code for local variables at all, as they cannot have such. - Using the ``type()`` built-in to create a new class could attribute it to the wrong module, this is now corrected. New Features ------------ - Uses Scons to execute the actual C++ build, giving some immediate improvements. - Now caches build results and Scons will only rebuild as needed. - The direct use of ``__import__()`` with a constant module name as parameter is also followed in "deep" mode. With time, non-constants may still become predictable, right now it must be a real CPython constant string. New Optimization ---------------- - Added optimization for the built-ins ``ord()`` and ``chr()``, these require a module and built-in module lookup, then parameter parsing. Now these are really quick with Nuitka. - Added optimization for the ``type()`` built-in with one parameter. As above, using from builtin module can be very slow. Now it is instantaneous. - Added optimization for the ``type()`` built-in with three parameters. It's rarely used, but providing our own variant, allowed to fix the bug mentioned above. Cleanups -------- - Using scons is a big cleanup for the way how C++ compiler related options are applied. It also makes it easier to re-build without Nuitka, e.g. if you were using Nuitka in your packages, you can easily build in the same way than Nuitka does. - Static helpers source code has been moved to ".hpp" and ".cpp" files, instead of being in ".py" files. This makes C++ compiler messages more readable and allows us to use C++ mode in Emacs etc., making it easier to write things. - Generated code for each module ends up in a separate file per module or package. - Constants etc. go to their own file (although not named sensible yet, likely going to change too) - Module variables are now created by the ``CPythonModule`` node only and are unique, this is to make optimization of these feasible. This is a pre-step to module variable optimization. New Tests --------- - Added "ExtremeClosure" from my Python quiz, it was not covered by existing tests. - Added test case for program that imports a module with a dash in its name. - Added test case for main program that starts with a dash. - Extended the built-in tests to cover ``type()`` as well. Organizational -------------- - There is now a new environment variable ``NUITKA_SCONS`` which should point to the directory with the ``SingleExe.scons`` file for Nuitka. The scons file could be named better, because it is actually one and the same who builds extension modules and executables. - There is now a new environment variable ``NUITKA_CPP`` which should point to the directory with the C++ helper code of Nuitka. - The script "create-environment.sh" can now be sourced (if you are in the top level directory of Nuitka) or be used with eval. In either case it also reports what it does. .. admonition:: Update The script has become obsolete now, as the environment variables are no longer necessary. - To cleanup the many "Program.build" directories, there is now a "clean-up.sh" script for your use. Can be handy, but if you use git, you may prefer its clean command. .. admonition:: Update The script has become obsolete now, as Nuitka test executions now by default delete the build results. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.4:: Pystone(1.1) time for 50000 passes = 0.34 This machine benchmarks at 147059 pystones/second This is 91% for 0.3.4, up from 80% before. Nuitka Release 0.3.3 ==================== This release of Nuitka continues the focus on performance. It also cleans up a few open topics. One is "doctests", these are now extracted from the CPython 2.6 test suite more completely. The other is that the CPython 2.7 test suite is now passed completely. There is some more work ahead though, to extract all of the "doctests" and to do that for both versions of the tests. This means an even higher level of compatibility has been achieved, then there is performance improvements, and ever cleaner structure. Bug fixes --------- Generators ~~~~~~~~~~ - Generator functions tracked references to the common and the instance context independently, now the common context is not released before the instance contexts are. - Generator functions didn't check the arguments to ``throw()`` the way they are in CPython, now they are. - Generator functions didn't trace exceptions to "stderr" if they occurred while closing unfinished ones in "del". - Generator functions used the slightly different wordings for some error messages. Function Calls ~~~~~~~~~~~~~~ - Extended call syntax with ``**`` allows that to use a mapping, and it is now checked if it really is a mapping and if the contents has string keys. - Similarly, extended call syntax with ``*`` allows a sequence, it is now checked if it really is a sequence. - Error message for duplicate keyword arguments or too little arguments now describe the duplicate parameter and the callable the same way CPython does. - Now checks to the keyword argument list first before considering the parameter counts. This is slower in the error case, but more compatible with CPython. Classes ~~~~~~~ - The "locals()" built-in when used in the class scope (not in a method) now is correctly writable and writes to it change the resulting class. - Name mangling for private identifiers was not always done entirely correct. Others ~~~~~~ - Exceptions didn't always have the correct stack reported. - The pickling of some tuples showed that "cPickle" can have non-reproducible results, using "pickle" to stream constants now New Optimization ---------------- - Access to instance attributes has become faster by writing specific code for the case. This is done in JIT way, attempting at run time to optimize attribute access for instances. - Assignments now often consider what's cheaper for the other side, instead of taking a reference to a global variable, just to have to release it. - The function call code built argument tuples and dictionaries as constants, now that is true for every tuple usage. Cleanups -------- - The static helper classes, and the prelude code needed have been moved to separate C++ files and are now accessed "#include". This makes the code inside C++ files as opposed to a Python string and therefore easier to read and or change. New Features ------------ - The generator functions and generator expressions have the attribute "gi_running" now. These indicate if they are currently running. New Tests --------- - The script to extract the "doctests" from the CPython test suite has been rewritten entirely and works with more doctests now. Running these tests created increased the test coverage a lot. - The Python 2.7 test suite has been added. Organizational -------------- - One can now run multiple "compare_with_cpython" instances in parallel, which enables background test runs. - There is now a new environment variable "NUITKA_INCLUDE" which needs to point to the directory Nuitka's C++ includes live in. Of course the "create-environment.sh" script generates that for you easily. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.3:: Pystone(1.1) time for 50000 passes = 0.36 This machine benchmarks at 138889 pystones/second This is 80% for 0.3.3, up from 66% before. Nuitka Release 0.3.2 ==================== This release of Nuitka continues the focus on performance. But this release also revisits the topic of feature parity. Before, feature parity had been reached "only" with Python 2.6. This is of course a big thing, but you know there is always more, e.g. Python 2.7. With the addition of set contractions and dict contractions in this very release, Nuitka is approaching Python support for 2.7, and then there are some bug fixes. Bug fixes --------- - Calling a function with ``**`` and using a non-dict for it was leading to wrong behavior. Now a mapping is good enough as input for the ``**`` parameter and it's checked. - Deeply nested packages "package.subpackage.module" were not found and gave a warning from Nuitka, with the consequence that they were not embedded in the executable. They now are. - Some error messages for wrong parameters didn't match literally. For example "function got multiple..." as opposed to "function() got multiple..." and alike. - Files that ended in line with a "#" but without a new line gave an error from "ast.parse". As a workaround, a new line is added to the end of the file if it's "missing". - More correct exception locations for complex code lines. I noted that the current line indication should not only be restored when the call at hand failed, but in any case. Otherwise sometimes the exception stack would not be correct. It now is - more often. Right now, this has no systematic test. - Re-raised exceptions didn't appear on the stack if caught inside the same function, these are now correct. - For ``exec`` the globals argument needs to have "__builtins__" added, but the check was performed with the mapping interface. That is not how CPython does it, and so e.g. the mapping could use a default value for "__builtins__" which could lead to incorrect behavior. Clearly a corner case, but one that works fully compatible now. New Optimization ---------------- - The local and shared local variable C++ classes have a flag "free_value" to indicate if an "PY_DECREF" needs to be done when releasing the object. But still the code used "Py_XDECREF" (which allows for "NULL" values to be ignored.) when the releasing of the object was done. Now the inconsistency of using "NULL" as "object" value with "free_value" set to true was removed. - Tuple constants were copied before using them without a point. They are immutable anyway. Cleanups -------- - Improved more of the indentation of the generated C++ which was not very good for contractions so far. Now it is. Also assignments should be better now. - The generation of code for contractions was made more general and templates split into multiple parts. This enabled reuse of the code for list contractions in dictionary and set contractions. - The with statement has its own template now and got cleaned up regarding indentation. New Tests --------- - There is now a script to extract the "doctests" from the CPython test suite and it generates Python source code from them. This can be compiled with Nuitka and output compared to CPython. Without this, the doctest parts of the CPython test suite is mostly useless. Solving this improved test coverage, leading to many small fixes. I will dedicate a later posting to the tool, maybe it is useful in other contexts as well. - Reference count tests have been expanded to cover assignment to multiple assignment targets, and to attributes. - The deep program test case, now also have a module in a sub-package to cover this case as well. Organizational -------------- - The `gitweb interface `__ might be considered an alternative to downloading the source if you want to provide a pointer, or want to take a quick glance at the source code. You can already download with git, follow the link below to the page explaining it. - The "README.txt" has documented more of the differences and I consequently updated the Differences page. There is now a distinction between generally missing functionality and things that don't work in ``--deep`` mode, where Nuitka is supposed to create one executable. I will make it a priority to remove the (minor) issues of ``--deep`` mode in the next release, as this is only relatively little work, and not a good difference to have. We want these to be empty, right? But for the time being, I document the known differences there. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.2:: Pystone(1.1) time for 50000 passes = 0.39 This machine benchmarks at 128205 pystones/second This is 66% for 0.3.2, slightly up from the 58% of 0.3.1 before. The optimization done were somewhat fruitful, but as you can see, they were also more cleanups, not the big things. Nuitka Release 0.3.1 ==================== This release of Nuitka continues the focus on performance and contains only cleanups and optimization. Most go into the direction of more readable code, some aim at making the basic things faster, with good results as to performance as you can see below. New Optimization ---------------- - Constants in conditions of conditional expressions (``a if cond else d``), ``if``/``elif`` or ``while`` are now evaluated to ``true`` or ``false`` directly. Before there would be temporary python object created from it which was then checked if it had a truth value. All of that is obviously overhead only. And it hurts the typically ``while 1:`` infinite loop case badly. - Do not generate code to catch ``BreakException`` or ``ContinueException`` unless a ``break`` or ``continue`` statement being in a ``try: finally:`` block inside that loop actually require this. Even while uncaught exceptions are cheap, it is still an improvement worthwhile and it clearly improves the readability for the normal case. - The compiler more aggressively prepares tuples, lists and dicts from the source code as constants if their contents is "immutable" instead of building at run time. An example of a "mutable" tuple would be ``({},)`` which is not safe to share, and therefore will still be built at run time. For dictionaries and lists, copies will be made, under the assumption that copying a dictionary will always be faster, than making it from scratch. - The parameter parsing code was dynamically building the tuple of argument names to check if an argument name was allowed by checking the equivalent of ``name in argument_names``. This was of course wasteful and now a pre-built constant is used for this, so it should be much faster to call functions with keyword arguments. - There are new templates files and also actual templates now for the ``while`` and ``for`` loop code generation. And I started work on having a template for assignments. Cleanups -------- - Do not generate code for the else of ``while`` and ``for`` loops if there is no such branch. This uncluttered the generated code somewhat. - The indentation of the generated C++ was not very good and whitespace was often trailing, or e.g. a real tab was used instead of "\t". Some things didn't play well together here. Now much of the generated C++ code is much more readable and white space cleaner. For optimization to be done, the humans need to be able to read the generated code too. Mind you, the aim is not to produce usable C++, but on the other hand, it must be possible to understand it. - To the same end of readability, the empty ``else {}`` branches are avoided for ``if``, ``while`` and ``for`` loops. While the C++ compiler can be expected to remove these, they seriously cluttered up things. - The constant management code in ``Context`` was largely simplified. Now the code is using the ``Constant`` class to find its way around the problem that dicts, sets, etc. are not hashable, or that ``complex`` is not being ordered; this was necessary to allow deeply nested constants, but it is also a simpler code now. - The C++ code generated for functions now has two entry points, one for Python calls (arguments as a list and dictionary for parsing) and one where this has happened successfully. In the future this should allow for faster function calls avoiding the building of argument tuples and dictionaries all-together. - For every function there was a "traceback adder" which was only used in the C++ exception handling before exit to CPython to add to the traceback object. This was now in-lined, as it won't be shared ever. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.1:: Pystone(1.1) time for 50000 passes = 0.41 This machine benchmarks at 121951 pystones/second This is 58% for 0.3.1, up from the 25% before. So it's getting somewhere. As always you will find its latest version here. Nuitka Release 0.3.0 ==================== This release 0.3.0 is the first release to focus on performance. In the 0.2.x series Nuitka achieved feature parity with CPython 2.6 and that was very important, but now it is time to make it really useful. Optimization has been one of the main points, although I was also a bit forward looking to Python 2.7 language constructs. This release is the first where I really started to measure things and removed the most important bottlenecks. New Features ------------ - Added option to control ``--debug``. With this option the C++ debug information is present in the file, otherwise it is not. This will give much smaller ".so" and ".exe" files than before. - Added option ``--no-optimization`` to disable all optimization. It enables C++ asserts and compiles with less aggressive C++ compiler optimization, so it can be used for debugging purposes. - Support for Python 2.7 set literals has been added. Performance Enhancements ------------------------ - Fast global variables: Reads of global variables were fast already. This was due to a trick that is now also used to check them and to do a much quicker update if they are already set. - Fast ``break``/``continue`` statements: To make sure these statements execute the finally handlers if inside a try, these used C++ exceptions that were caught by ``try``/``finally`` in ``while`` or ``for`` loops. This was very slow and had very bad performance. Now it is checked if this is at all necessary and then it's only done for the rare case where a ``break``/``continue`` really is inside the tried block. Otherwise it is now translated to a C++ ``break``/``continue`` which the C++ compiler handles more efficiently. - Added ``unlikely()`` compiler hints to all errors handling cases to allow the C++ compiler to generate more efficient branch code. - The for loop code was using an exception handler to make sure the iterated value was released, using ``PyObjectTemporary`` for that instead now, which should lead to better generated code. - Using constant dictionaries and copy from them instead of building them at run time even when contents was constant. New Tests --------- - Merged some bits from the CPython 2.7 test suite that do not harm 2.6, but generally it's a lot due to some ``unittest`` module interface changes. - Added CPython 2.7 tests ``test_dictcomps.py`` and ``test_dictviews.py`` which both pass when using Python 2.7. - Added another benchmark extract from "PyStone" which uses a while loop with break. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.0:: Pystone(1.1) time for 50000 passes = 0.52 This machine benchmarks at 96153.8 pystones/second That's a 25% speedup now and a good start clearly. It's not yet in the range of where i want it to be, but there is always room for more. And the ``break``/``continue`` exception was an important performance regression fix. Nuitka Release 0.2.4 ==================== This release 0.2.4 is likely the last 0.2.x release, as it's the one that achieved feature parity with CPython 2.6, which was the whole point of the release series, so time to celebrate. I have stayed away (mostly) from any optimization, so as to not be premature. From now on speed optimization is going to be the focus though. Because right now, frankly, there is not much of a point to use Nuitka yet, with only a minor run time speed gain in trade for a long compile time. But hopefully we can change that quickly now. New Features ------------ - The use of exec in a local function now adds local variables to scope it is in. - The same applies to ``from module_name import *`` which is now compiled correctly and adds variables to the local variables. Bug Fixes --------- - Raises ``UnboundLocalError`` when deleting a local variable with ``del`` twice. - Raises ``NameError`` when deleting a global variable with ``del`` twice. - Read of to uninitialized closure variables gave ``NameError``, but ``UnboundLocalError`` is correct and raised now. Cleanups -------- - There is now a dedicated pass over the node tree right before code generation starts, so that some analysis can be done as late as that. Currently this is used for determining which functions should have a dictionary of locals. - Checking the exported symbols list, fixed all the cases where a ``static`` was missing. This reduces the "module.so" sizes. - With gcc the "visibility=hidden" is used to avoid exporting the helper classes. Also reduces the "module.so" sizes, because classes cannot be made static otherwise. New Tests --------- - Added "DoubleDeletions" to cover behaviour of ``del``. It seems that this is not part of the CPython test suite. - The "OverflowFunctions" (those with dynamic local variables) now has an interesting test, exec on a local scope, effectively adding a local variable while a closure variable is still accessible, and a module variable too. This is also not in the CPython test suite. - Restored the parts of the CPython test suite that did local star imports or exec to provide new variables. Previously these have been removed. - Also "test_with.py" which covers PEP 343 has been reactivated, the with statement works as expected. Nuitka Release 0.2.3 ==================== This new release is marking a closing in on feature parity to CPython 2.6 which is an important mile stone. Once this is reached, a "Nuitka 0.3.x" series will strive for performance. Bug Fixes --------- - Generator functions no longer leak references when started, but not finished. - Yield can in fact be used as an expression and returns values that the generator user ``send()`` to it. Reduced Differences / New Features ---------------------------------- - Generator functions already worked quite fine, but now they have the ``throw()``, ``send()`` and ``close()`` methods. - Yield is now an expression as is ought to be, it returns values put in by ``send()`` on the generator user. - Support for extended slices: .. code-block:: python x = d[:42, ..., :24:, 24, 100] d[:42, ..., :24:, 24, 100] = "Strange" del d[:42, ..., :24:, 24, 100] Tests Work ---------- - The "test_contextlib" is now working perfectly due to the generator functions having a correct ``throw()``. Added that test back, so context managers are now fully covered. - Added a basic test for "overflow functions" has been added, these are the ones which have an unknown number of locals due to the use of language constructs ``exec`` or ``from bla import *`` on the function level. This one currently only highlights the failure to support it. - Reverted removals of extended slice syntax from some parts of the CPython test suite. Cleanups -------- - The compiled generator types are using the new C++0x type safe enums feature. - Resolved a circular dependency between ``TreeBuilding`` and ``TreeTransforming`` modules. Nuitka Release 0.2.2 ==================== This is some significant progress, a lot of important things were addressed. Bug Fixes --------- - Scope analysis is now done during the tree building instead of sometimes during code generation, this fixed a few issues that didn't show up in tests previously. - Reference leaks of generator expressions that were not fishing, but then deleted are not more. - Inlining of exec is more correct now. - More accurate exception lines when iterator creation executes compiled code, e.g. in a for loop - The list of base classes of a class was evaluated in the context of the class, now it is done in the context of the containing scope. - The first iterated of a generator expression was evaluated in its own context, now it is done in the context of the containing scope. Reduced Differences ------------------- - With the enhanced scope analysis, ``UnboundLocalError`` is now correctly supported. - Generator expressions (but not yet functions) have a ``throw()``, ``send()`` and ``close()`` method. - Exec can now write to local function namespace even if ``None`` is provided at run time. - Relative imports inside packages are now correctly resolved at compile time when using ``--deep``. Cleanups -------- - The compiled function type got further enhanced and cleaned up. - The compiled generator expression function type lead to a massive cleanup of the code for generator expressions. - Cleaned up namespaces, was still using old names, or "Py*" which is reserved to core CPython. - Overhaul of the code responsible for ``eval`` and ``exec``, it has been split, and it pushed the detection defaults to the C++ compiler which means, we can do it at run time or compile time, depending on circumstances. - Made ``PyTemporaryObject`` safer to use, disabling copy constructor it should be also a relief to the C++ compiler if it doesn't have to eliminate all its uses. - The way delayed work is handled in ``TreeBuilding`` step has been changed to use closured functions, should be more readable. - Some more code templates have been created, making the code generation more readable in some parts. More to come. New Features ------------ - As I start to consider announcing Nuitka, I moved the version logic so that the version can now be queried with ``--version``. New Optimization ---------------- - Name lookups for ``None``, ``True`` and ``False`` and now always detected as constants, eliminating many useless module variable lookups. New Tests --------- - More complete test of generator expressions. - Added test program for packages with relative imports inside the package. - The built-in ``dir()`` in a function was not having fully deterministic output list, now it does. Summary ------- Overall, the amount of differences between CPython and Nuitka is heading towards zero. Also most of the improvements done in this release were very straightforward cleanups and not much work was required, mostly things are about cleanups and then it becomes easily right. The new type for the compiled generator expressions was simple to create, esp. as I could check what CPython does in its source code. For optimization purposes, I decided that generator expressions and generator functions will be separate compiled types, as most of their behavior will not be shared. I believe optimizing generator expressions to run well is an important enough goal to warrant that they have their own implementation. Now that this is done, I will repeat it with generator functions. Generator functions already work quite fine, but like generator expressions did before this release, they can leak references if not finished , and they don't have the ``throw()`` method, which seems very important to the correct operation of ``contextlib``. So I will introduce a decicated type for these too, possibly in the next release. Nuitka Release 0.2.1 ==================== The march goes on, this is another minor release with a bunch of substantial improvements: Bug Fixes --------- - Packages now also can be embedded with the ``--deep`` option too, before they could not be imported from the executable. - In-lined exec with their own future statements leaked these to the surrounding code. Reduced Differences ------------------- - The future print function import is now supported too. Cleanups -------- - Independence of the compiled function type. When I started it was merely ``PyCFunction`` and then a copy of it patched at run time, using increasingly less code from CPython. Now it's nothing at all anymore. - This lead to major cleanup of run time compiled function creation code, no more ``methoddefs``, ``PyCObject`` holding context, etc. - PyLint was used to find the more important style issues and potential bugs, also helping to identify some dead code. Summary ------- The major difference now is the lack of a throw method for generator functions. I will try to address that in a 0.2.2 release if possible. The plan is that the 0.2.x series will complete these tasks, and 0.3 could aim at some basic optimization finally. Nuitka Release 0.2 ================== Good day, this is a major step ahead, improvements everywhere. Bug fixes --------- - Migrated the Python parser from the deprecated and problematic ``compiler`` module to the ``ast`` module which fixes the ``d[a,] = b`` parser problem. A pity it was not available at the time I started, but the migration was relatively painless now. - I found and fixed wrong encoding of binary data into C++ literals. Now Nuitka uses C++0x raw strings, and these problems are gone. - The decoding of constants was done with the ``marshal`` module, but that appears to not deeply care enough about unicode encoding it seems. Using ``cPickle`` now, which seems less efficient, but is more correct. - Another difference is gone: The ``continue`` and ``break`` inside loops do no longer prevent the execution of finally blocks inside the loop. Organizational -------------- - I now maintain the "README.txt" in org-mode, and intend to use it as the issue tracker, but I am still a beginner at that. .. admonition:: Update Turned out I never master it, and used ReStructured Text instead. - There is a public git repository for you to track Nuitka releases. Make your changes and then ``git pull --rebase``. If you encounter conflicts in things you consider useful, please submit the patches and a pull offer. When you make your clones of Nuitka public, use ``nuitka-unofficial`` or not the name ``Nuitka`` at all. - There is a now a `mailing list `__ available too. Reduced Differences ------------------- - Did you know you could write ``lambda : (yield something)`` and it gives you a lambda that creates a generator that produces that one value? Well, now Nuitka has support for lambda generator functions. - The ``from __future__ import division`` statement works as expected now, leading to some newly passing CPython tests. - Same for ``from __future__ import unicode_literals`` statement, these work as expected now, removing many differences in the CPython tests that use this already. New Features ------------ - The ``Python`` binary provided and ``Nuitka.py`` are now capable of accepting parameters for the program executed, in order to make it even more of a drop-in replacement to ``python``. - Inlining of ``exec`` statements with constant expressions. These are now compiled at compile time, not at run time anymore. I observed that an increasing number of CPython tests use exec to do things in isolation or to avoid warnings, and many more these tests will now be more effective. I intend to do the same with eval expressions too, probably in a minor release. Summary ------- So give it a whirl. I consider it to be substantially better than before, and the list of differences to CPython is getting small enough, plus there is already a fair bit of polish to it. Just watch out that it needs gcc-4.5 or higher now. Nuitka Release 0.1.1 ==================== I just have just updated Nuitka to version 0.1.1 which is a bug fix release to 0.1, which corrects many of the small things: - Updated the CPython test suite to 2.6.6rc and minimized much of existing differences in the course. - Compiles standalone executable that includes modules (with --deep option), but packages are not yet included successfully. - Reference leaks with exceptions are no more. - sys.exc_info() works now mostly as expected (it's not a stack of exceptions). - More readable generated code, better organisation of C++ template code. - Restored debug option ``--g++-only``. The biggest thing probably is the progress with exception tracebacks objects in exception handlers, which were not there before (always ``None``). Having these in place will make it much more compatible. Also with manually raised exceptions and assertions, tracebacks will now be more correct to the line. On a bad news, I discovered that the ``compiler`` module that I use to create the AST from Python source code, is not only deprecated, but also broken. I created the `CPython bug `__ about it, basically it cannot distinguish some code of the form ``d[1,] = None`` from ``d[1] = None``. This will require a migration of the ``ast`` module, which should not be too challenging, but will take some time. I am aiming at it for a 0.2 release. Generating wrong code (Nuitka sees ``d[1] = None`` in both cases) is a show blocker and needs a solution. So, yeah. It's better, it's there, but still experimental. You will find its latest version here. Please try it out and let me know what you think in the comments section. Nuitka Release 0.1 (Releasing Nuitka to the World) ================================================== Obviously this is very exciting step for me. I am releasing Nuitka today. Finally. For a long time I knew I would, but actually doing it, is a different beast. Reaching my goals for release turned out to be less far away than I hope, so instead of end of August, I can already release it now. Currently it's not more than 4% faster than CPython. No surprise there, if all you did, is removing the bytecode interpretation so far. It's not impressive at all. It's not even a reason to use it. But it's also only a start. Clearly, once I get into optimizing the code generation of Nuitka, it will only get better, and then probably in sometimes dramatic steps. But I see this as a long term goal. I want to have infrastructure in the code place, before doing lots of possible optimization that just make Nuitka unmaintainable. And I will want to have a look at what others did so far in the domain of type inference and how to apply that for my project. I look forward to the reactions about getting this far. The supported language volume is amazing, and I have a set of nice tricks used. For example the way generator functions are done is a clever hack. Where to go from here? Well, I guess, I am going to judge it by the feedback I receive. I personally see "constant propagation" as a laudable first low hanging fruit, that could be solved. Consider this readable code on the module level: .. code-block:: python meters_per_nautical_mile = 1852 def convertMetersToNauticalMiles(meters): return meters / meters_per_nautical_mile def convertNauticalMilesToMeters(miles): return miles * meters_per_nautical_mile Now imagine you are using this very frequently in code. Quickly you determine that the following will be much faster: .. code-block:: python def convertMetersToNauticalMiles(meters): return meters / 1852 def convertNauticalMilesToMeters(miles): return miles * 1852 Still good? Well, probably next step you are going to in-line the function calls entirely. For optimization, you are making your code less readable. I do not all appreciate that. My first goal is there to make the more readable code perform as well or better as the less readable variant. Nuitka-0.5.18.1/tests/0000755000175000017500000000000012651072347014602 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/packages/0000755000175000017500000000000012651072347016360 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/packages/sub_package/0000755000175000017500000000000012651072347020624 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/packages/sub_package/kitty/0000755000175000017500000000000012651072347021770 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/packages/sub_package/kitty/smallkitty.py0000644000175000017500000000161612651071040024530 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # size = "small" Nuitka-0.5.18.1/tests/packages/sub_package/kitty/bigkitty.py0000644000175000017500000000161412651071040024157 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # size = "big" Nuitka-0.5.18.1/tests/packages/sub_package/kitty/speak/0000755000175000017500000000000012651072347023073 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/packages/sub_package/kitty/speak/miau.py0000644000175000017500000000163512651071040024372 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def speak(): print "miau" Nuitka-0.5.18.1/tests/packages/sub_package/kitty/speak/__init__.py0000644000175000017500000000200012651071040025161 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: print("__loader__ present:", __loader__ is not None) except NameError: print("No __loader__ found, OK for Python2") Nuitka-0.5.18.1/tests/packages/sub_package/kitty/speak/purr.py0000644000175000017500000000163712651071040024431 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def speak(): print "mrrruu" Nuitka-0.5.18.1/tests/packages/sub_package/kitty/speak/hello.py0000644000175000017500000000203412651071040024534 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def speak(): print( "hello kitty" ) # Test Issue#115, in recursing modules, this was misbehaving. import types assert type(speak) == types.FunctionType Nuitka-0.5.18.1/tests/packages/sub_package/kitty/__init__.py0000644000175000017500000000204712651071040024071 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("__name__ is ", __name__) print("__package__ is ", __package__) import sys print("From sys.modules", sys.modules["kitty"]) from kitty.speak.hello import speak Nuitka-0.5.18.1/tests/packages/run_all.py0000755000175000017500000000560612651071040020365 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, createSearchMode, compareWithCPython, withExtendedExtraOptions, getTempDir ) python_version = setup() search_mode = createSearchMode() for filename in sorted(os.listdir('.')): if not os.path.isdir(filename) or filename.endswith(".build"): continue extra_flags = [ "expect_success", "remove_output", "module_mode", "two_step_execution" ] # The use of "__main__" in the test package gives a warning. if filename == "sub_package": extra_flags.append("ignore_warnings") active = search_mode.consider( dirname = None, filename = filename ) if active: my_print("Consider output of recursively compiled program:", filename) for filename_main in os.listdir(filename): if not os.path.isdir(os.path.join(filename,filename_main)): continue if filename_main not in ("..", '.'): break else: sys.exit( """\ Error, no package in dir '%s' found, incomplete test case.""" % filename ) extensions = [ "--recurse-to=%s" % os.path.basename(filename_main) ] if not "--output-dir" in os.environ.get("NUITKA_EXTRA_OPTIONS", ""): extensions.append("--output-dir=%s" % getTempDir()) with withExtendedExtraOptions(*extensions): compareWithCPython( dirname = filename, filename = filename_main, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.18.1/tests/standalone/0000755000175000017500000000000012651072347016732 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/standalone/TkInterUsing.py0000644000175000017500000000161612651071040021663 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import tkinter Nuitka-0.5.18.1/tests/standalone/IdnaUsing.py0000644000175000017500000000176412651071040021162 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import idna.core import sys print(idna.core, "idna.idnadata" in sys.modules) Nuitka-0.5.18.1/tests/standalone/run_all.py0000755000175000017500000003660612651071040020743 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys, shutil # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, hasModule, compareWithCPython, decideFilenameVersionSkip, getRuntimeTraceOfLoadedFiles, createSearchMode ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() search_mode.mayFailFor( # Do not expect PySide to work yet, because it has that bug still # where it won't call compiled functions as slots. "PySideUsing.py" ) for filename in sorted(os.listdir('.')): if not filename.endswith(".py"): continue if not decideFilenameVersionSkip(filename): continue path = os.path.relpath(filename) active = search_mode.consider(dirname = None, filename = filename) if not active: my_print("Skipping", filename) continue extra_flags = [ "expect_success", "standalone", "remove_output" ] if filename == "PySideUsing.py": # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): my_print("Skipping", filename, "not relevant.") continue if not hasModule("PySide.QtCore"): my_print( "Skipping", filename, "PySide not installed for", python_version, "but test needs it." ) continue # For the warnings. extra_flags.append("ignore_stderr") if "PyQt4" in filename: # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): my_print("Skipping", filename, "not relevant.") continue if not hasModule("PyQt4.QtGui"): my_print( "Skipping", filename, "PyQt4 not installed for", python_version, "but test needs it." ) continue # For the plug-in information. extra_flags.append("ignore_infos") if "Idna" in filename: if not hasModule("idna.core"): my_print( "Skipping", filename, "idna not installed for", python_version, "but test needs it." ) continue # For the warnings of Python2. if python_version.startswith("2"): extra_flags.append("ignore_stderr") if "PyQt5" in filename: # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): my_print("Skipping", filename, "not relevant.") continue if not hasModule("PyQt5.QtGui"): my_print( "Skipping", filename, "PyQt5 not installed for", python_version, "but test needs it." ) continue # For the plug-in information. extra_flags.append("ignore_infos") # TODO: Temporary only if os.name == "nt" and "PyQt" in filename: continue if "PySide" in filename or "PyQt" in filename: extra_flags.append("plugin_enable:qt-plugins") if filename == "CtypesUsing.py": extra_flags.append("plugin_disable:pylint-warnings") if filename == "GtkUsing.py": # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): my_print("Skipping", filename, "not relevant.") continue if not hasModule("pygtk"): my_print( "Skipping", filename, "pygtk not installed for", python_version, "but test needs it." ) continue # For the warnings. extra_flags.append("ignore_stderr") if filename.startswith("Win"): if os.name != "nt": my_print("Skipping", filename, "windows only.") continue if filename == "Win32ComUsing.py": if not hasModule("win32com"): my_print( "Skipping", filename, "win32com not installed for", python_version, "but test needs it." ) continue if filename == "LxmlUsing.py": if not hasModule("lxml.etree"): my_print( "Skipping", filename, "lxml.etree not installed for", python_version, "but test needs it." ) continue if filename == "TkInterUsing.py": if not hasModule("tkinter"): my_print( "Skipping", filename, "tkinter not installed for", python_version, "but test needs it." ) continue if filename not in ("PySideUsing.py", "PyQt4Using.py", "PyQt5Using.py", "PyQt4Plugins.py", "PyQt5Plugins.py", "GtkUsing.py", "LxmlUsing.py", "Win32ComUsing.py", "IdnaUsing.py"): extra_flags += [ "no_site" ] my_print("Consider output of recursively compiled program:", filename) # First compare so we know the program behaves identical. compareWithCPython( dirname = None, filename = filename, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) # Second use "strace" on the result. loaded_filenames = getRuntimeTraceOfLoadedFiles( path = os.path.join( filename[:-3] + ".dist", filename[:-3] + ".exe" ) ) current_dir = os.path.normpath(os.getcwd()) current_dir = os.path.normcase(current_dir) illegal_access = False for loaded_filename in loaded_filenames: loaded_filename = os.path.normpath(loaded_filename) loaded_filename = os.path.normcase(loaded_filename) if loaded_filename.startswith(current_dir): continue if loaded_filename.startswith(os.path.abspath(current_dir)): continue if loaded_filename.startswith("/etc/"): continue if loaded_filename.startswith("/proc/") or loaded_filename == "/proc": continue if loaded_filename.startswith("/dev/"): continue if loaded_filename.startswith("/tmp/"): continue if loaded_filename.startswith("/usr/lib/locale/"): continue if loaded_filename.startswith("/usr/share/locale/"): continue if loaded_filename.startswith("/usr/share/X11/locale/"): continue if loaded_filename.startswith("/lib/libc.") or \ loaded_filename.startswith("/lib64/libc."): continue if loaded_filename.startswith("/lib/libdl.") or \ loaded_filename.startswith("/lib64/libdl."): continue if loaded_filename.startswith("/lib/libm.") or \ loaded_filename.startswith("/lib64/libm."): continue if loaded_filename.startswith("/lib/libz.") or \ loaded_filename.startswith("/lib64/libz."): continue if loaded_filename.startswith("/lib/libutil.") or \ loaded_filename.startswith("/lib64/libutil."): continue if loaded_filename.startswith("/lib/libpthread.") or \ loaded_filename.startswith("/lib64/libpthread."): continue if loaded_filename.startswith("/lib/libgcc_s.") or \ loaded_filename.startswith("/lib64/libgcc_s."): continue # Loaded by C library potentially for DNS lookups. if os.path.basename(loaded_filename).startswith("libnss_") or \ os.path.basename(loaded_filename).startswith("libnsl"): continue # Loaded by dtruss on MacOS X. if loaded_filename.startswith("/usr/lib/dtrace/"): continue # Loaded by cowbuilder and pbuilder on Debian if os.path.basename(loaded_filename) == ".ilist": continue if "cowdancer" in loaded_filename: continue if "eatmydata" in loaded_filename: continue # Loading from home directories is OK too. if loaded_filename.startswith("/home/") or \ loaded_filename.startswith("/data/") or \ loaded_filename.startswith("/root/") or \ loaded_filename in ("/home", "/data", "/root"): continue if os.path.basename(loaded_filename) == "gconv-modules.cache": continue # TODO: Unclear, loading gconv from filesystem of installed system # may be OK or not. I think it should be. if "/gconv/" in loaded_filename: continue if os.path.basename(loaded_filename).startswith("libicu"): continue # Loading from caches is OK. if loaded_filename.startswith("/var/cache/"): continue # PySide accesses its directory. if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/PySide": continue # GTK accesses package directories only. if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/gtk-2.0/gtk": continue if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/glib": continue if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/gtk-2.0/gio": continue if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/gobject": continue if loaded_filename == "/usr/bin/python3.2mu": continue # Accessing SE-Linux is OK. if loaded_filename in ("/sys/fs/selinux", "/selinux"): continue # The access to .pth files has no effect. if loaded_filename.endswith(".pth"): continue # Looking at site-package dir alone is alone. if loaded_filename.endswith("site-packages"): continue loaded_basename = os.path.basename(loaded_filename).upper() # Windows baseline DLLs if loaded_basename in ( "SHELL32.DLL","USER32.DLL","KERNEL32.DLL", "NTDLL.DLL", "NETUTILS.DLL", "LOGONCLI.DLL", "GDI32.DLL", "RPCRT4.DLL", "ADVAPI32.DLL", "SSPICLI.DLL", "SECUR32.DLL", "KERNELBASE.DLL", "WINBRAND.DLL", "DSROLE.DLL", "DNSAPI.DLL", "SAMCLI.DLL", "WKSCLI.DLL", "SAMLIB.DLL", "WLDAP32.DLL", "NTDSAPI.DLL", "CRYPTBASE.DLL", "W32TOPL", "WS2_32.DLL", "SPPC.DLL", "MSSIGN32.DLL", "CERTCLI.DLL", "WEBSERVICES.DLL", "AUTHZ.DLL", "CERTENROLL.DLL", "VAULTCLI.DLL", "REGAPI.DLL", "BROWCLI.DLL", "WINNSI.DLL", "DHCPCSVC6.DLL", "PCWUM.DLL", "CLBCATQ.DLL", "IMAGEHLP.DLL", "MSASN1.DLL", "DBGHELP.DLL", "DEVOBJ.DLL", "DRVSTORE.DLL", "CABINET.DLL", "SCECLI.DLL", "SPINF.DLL", "SPFILEQ.DLL", "GPAPI.DLL", "NETJOIN.DLL", "W32TOPL.DLL", "NETBIOS.DLL", "DXGI.DLL", "DWRITE.DLL", "D3D11.DLL", "WLANAPI.DLL", "WLANUTIL.DLL", "ONEX.DLL", "EAPPPRXY.DLL", "MFPLAT.DLL", "AVRT.DLL", "ELSCORE.DLL", "INETCOMM.DLL", "MSOERT2.DLL", "IEUI.DLL", "MSCTF.DLL", "MSFEEDS.DLL", "UIAUTOMATIONCORE.DLL", "PSAPI.DLL", "EFSADU.DLL", "MFC42U.DLL", "ODBC32.DLL", "OLEDLG.DLL", "NETAPI32.DLL", "LINKINFO.DLL", "DUI70.DLL", "ADVPACK.DLL", "NTSHRUI.DLL", "WINSPOOL.DRV", "EFSUTIL.DLL", "WINSCARD.DLL", "SHDOCVW.DLL", "IEFRAME.DLL", "D2D1.DLL", "GDIPLUS.DLL", "OCCACHE.DLL", "IEADVPACK.DLL", "MLANG.DLL", "MSI.DLL", "MSHTML.DLL", "COMDLG32.DLL", "PRINTUI.DLL", "PUIAPI.DLL", "ACLUI.DLL", "WTSAPI32.DLL", "FMS.DLL", "DFSCLI.DLL", "HLINK.DLL", "MSRATING.DLL", "PRNTVPT.DLL", "IMGUTIL.DLL", "MSLS31.DLL", "VERSION.DLL", "NORMALIZ.DLL", "IERTUTIL.DLL", "WININET.DLL", "WINTRUST.DLL", "XMLLITE.DLL", "APPHELP.DLL", "PROPSYS.DLL", "RSTRTMGR.DLL", "NCRYPT.DLL", "BCRYPT.DLL", "MMDEVAPI.DLL", "MSILTCFG.DLL", "DEVMGR.DLL", "DEVRTL.DLL", "NEWDEV.DLL", "VPNIKEAPI.DLL", "WINHTTP.DLL", "WEBIO.DLL", "NSI.DLL", "DHCPCSVC.DLL", "CRYPTUI.DLL", "ESENT.DLL", "DAVHLPR.DLL", "CSCAPI.DLL", "ATL.DLL", "OLEAUT32.DLL", "SRVCLI.DLL", "RASDLG.DLL", "MPRAPI.DLL", "RTUTILS.DLL", "RASMAN.DLL", "MPRMSG.DLL", "SLC.DLL", "CRYPTSP.DLL", "RASAPI32.DLL", "TAPI32.DLL", "EAPPCFG.DLL", "NDFAPI.DLL", "WDI.DLL", "COMCTL32.DLL", "UXTHEME.DLL", "IMM32.DLL", "OLEACC.DLL", "WINMM.DLL", "WINDOWSCODECS.DLL", "DWMAPI.DLL", "DUSER.DLL", "PROFAPI.DLL", "URLMON.DLL", "SHLWAPI.DLL", "LPK.DLL", "USP10.DLL", "CFGMGR32.DLL", "MSIMG32.DLL", "POWRPROF.DLL", "SETUPAPI.DLL", "WINSTA.DLL", "CRYPT32.DLL", "IPHLPAPI.DLL", "MPR.DLL", "CREDUI.DLL", "NETPLWIZ.DLL", "OLE32.DLL", "ACTIVEDS.DLL", "ADSLDPC.DLL", "USERENV.DLL", "APPREPAPI.DLL", "BCP47LANGS.DLL", "BCRYPTPRIMITIVES.DLL", "CERTCA.DLL", "CHARTV.DLL", "COMBASE.DLL", "COML2.DLL", "DCOMP.DLL", "DPAPI.DLL", "DSPARSE.DLL", "FECLIENT.DLL", "FIREWALLAPI.DLL", "FLTLIB.DLL", "MRMCORER.DLL", "NTASN1.DLL", "SECHOST.DLL", "SETTINGSYNCPOLICY.DLL", "SHCORE.DLL", "TBS.DLL", "TWINAPI.APPCORE.DLL", "TWINAPI.DLL", "VIRTDISK.DLL", "WEBSOCKET.DLL", "WEVTAPI.DLL", "WINMMBASE.DLL", "WMICLNT.DLL"): continue # Win API can be assumed. if loaded_basename.startswith("API-MS-WIN"): continue # MSVC run time DLLs, seem to sometimes come from system. TODO: # clarify if that means we did it wrong. if loaded_basename in ("MSVCRT.DLL", "MSVCR90.DLL"): continue my_print("Should not access '%s'." % loaded_filename) illegal_access = True if illegal_access: sys.exit(1) shutil.rmtree(filename[:-3] + ".dist") search_mode.finish() Nuitka-0.5.18.1/tests/standalone/LxmlUsing.py0000644000175000017500000000177312651071040021223 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import lxml.etree tree = lxml.etree.fromstring("value") assert tree.tag == "root" assert tree.text == "value" Nuitka-0.5.18.1/tests/standalone/PyQt4Plugins.py0000644000175000017500000000171212651071040021615 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from PyQt4 import QtGui print(QtGui.QImageReader.supportedImageFormats()) Nuitka-0.5.18.1/tests/standalone/PyQt5Using.py0000644000175000017500000000436412651071040021270 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function # This test is playing with configuration settings and checking that works. from PyQt5.QtCore import QSettings from PyQt5.QtCore import QCoreApplication import sys app = QCoreApplication([]) app.setOrganizationName("BOGUS_NAME") app.setOrganizationDomain("bogosity.com") app.setApplicationName("BOGUS") settings = QSettings() byte_string = b'\xde\xad\xbe\xef' settings.clear() settings.setValue("bogus_byte_string",byte_string) settings.sync() return_string = settings.value("bogus_byte_string",b'\x00\x00\x00\x00') if sys.version_info >= (3,): assert return_string == byte_string, (repr(return_string), "!=", byte_string) print("OK.") # This test is using signals and will only work if PySide properly accepts # compiled functions as callables. from PyQt5.QtCore import pyqtSlot, pyqtSignal, QObject, QMetaObject class Communicate(QObject): speak = pyqtSignal(int) def __init__(self,name = "",parent = None): QObject.__init__(self,parent) self.setObjectName(name) class Speaker(QObject): @pyqtSlot(int) def on_communicator_speak(self, stuff): print(stuff) speaker = Speaker() someone = Communicate(name = "communicator",parent = speaker) QMetaObject.connectSlotsByName(speaker) print("The answer is:",end = "") # emit 'speak' signal someone.speak.emit(42) print("Slot should have made output by now.") Nuitka-0.5.18.1/tests/standalone/ShlibUsing.py0000644000175000017500000000177112651071040021346 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: import pyexpat print(pyexpat.__doc__) except ImportError: print("Skipped, no pyexpat module installed.") Nuitka-0.5.18.1/tests/standalone/PythonExecuting.py0000644000175000017500000000225612651071040022433 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Execute CPython from a standalone binary. The standalone environment should not harm CPython execution by not having set modified PYTHONPATH or PYTHONHOME. """ import subprocess, sys subprocess.call( [ sys.executable, "-c", "print('From forked CPython', 27)" ] ) Nuitka-0.5.18.1/tests/standalone/Issue116_2.py0000644000175000017500000000163512651071040021037 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("4B4159".decode("hex")) Nuitka-0.5.18.1/tests/standalone/PyQt4Using.py0000644000175000017500000000316112651071040021261 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This test is using signals and will only work if PySide properly accepts # compiled functions as callables. from __future__ import print_function from PyQt4.QtCore import pyqtSlot, pyqtSignal, QObject, QMetaObject class Communicate(QObject): speak = pyqtSignal(int) def __init__(self,name = "",parent = None): QObject.__init__(self,parent) self.setObjectName(name) class Speaker(QObject): @pyqtSlot(int) def on_communicator_speak(self, stuff): print(stuff) speaker = Speaker() someone = Communicate(name = "communicator",parent = speaker) QMetaObject.connectSlotsByName(speaker) print("The answer is:",end = "") # emit 'speak' signal someone.speak.emit(42) print("Slot should have made output by now.") Nuitka-0.5.18.1/tests/standalone/GtkUsing.py0000644000175000017500000000175712651071040021036 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import warnings warnings.filterwarnings("ignore", "") import sys import gtk import pygtk pygtk.require("2.0") Nuitka-0.5.18.1/tests/standalone/Win32ComUsing.py0000644000175000017500000000171412651071040021643 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from win32com import storagecon from win32com.shell import shell, shellcon Nuitka-0.5.18.1/tests/standalone/CtypesUsing.py0000644000175000017500000000464612651071040021560 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, ctypes if os.name == "nt": # adapted from http://code.activestate.com/recipes/578513 from ctypes import wintypes # Lets allow this to match Windows API it reflects, # pylint: disable=C0103 class PROCESS_MEMORY_COUNTERS_EX(ctypes.Structure): _fields_ = [ ("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD), ("PeakWorkingSetSize", ctypes.c_size_t), ("WorkingSetSize", ctypes.c_size_t), ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), ("QuotaPagedPoolUsage", ctypes.c_size_t), ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), ("QuotaNonPagedPoolUsage", ctypes.c_size_t), ("PagefileUsage", ctypes.c_size_t), ("PeakPagefileUsage", ctypes.c_size_t), ("PrivateUsage", ctypes.c_size_t), ] GetProcessMemoryInfo = ctypes.windll.psapi.GetProcessMemoryInfo GetProcessMemoryInfo.argtypes = [ wintypes.HANDLE, ctypes.POINTER(PROCESS_MEMORY_COUNTERS_EX), wintypes.DWORD, ] GetProcessMemoryInfo.restype = wintypes.BOOL counters = PROCESS_MEMORY_COUNTERS_EX() rv = GetProcessMemoryInfo( ctypes.windll.kernel32.GetCurrentProcess(), ctypes.byref(counters), ctypes.sizeof(counters) ) if not rv: raise ctypes.WinError() print("OK.") else: # TODO: How to get this to work. if False: libc = ctypes.CDLL("libc") printf = libc.printf printf("Hello, %s\n", "World!") print("OK.") Nuitka-0.5.18.1/tests/standalone/PySideUsing.py0000644000175000017500000000314212651071040021474 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This test is using signals and will only work if PySide properly accepts # compiled functions as callables. from __future__ import print_function from PySide.QtCore import Slot, Signal, QObject, QMetaObject class Communicate(QObject): speak = Signal(int) def __init__(self,name = "",parent = None): QObject.__init__(self,parent) self.setObjectName(name) class Speaker(QObject): @Slot(int) def on_communicator_speak(self, stuff): print(stuff) speaker = Speaker() someone = Communicate(name = "communicator",parent = speaker) QMetaObject.connectSlotsByName(speaker) print("The answer is:",end = "") # emit 'speak' signal someone.speak.emit(42) print("Slot should have made output by now.") Nuitka-0.5.18.1/tests/syntax/0000755000175000017500000000000012651072347016130 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/syntax/Importing32.py0000644000175000017500000000174712651071040020615 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def starImportFailure(): from doctest import * try: sys print( "but it does not" ) except NameError: print( "and it does" ) print("Star import needs to respect __all__", starImportFailure()) Nuitka-0.5.18.1/tests/syntax/SyntaxError.py0000644000175000017500000000154512651071040020774 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Test the syntax error case: def nested(): claxss ProxyBase(metaclass=ProxyType): pass Nuitka-0.5.18.1/tests/syntax/TryFinallyContinue.py0000644000175000017500000000155312651071040022275 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): for i in range(10): try: undefined finally: continue Nuitka-0.5.18.1/tests/syntax/TryExceptAllNotLast.py0000644000175000017500000000152712651071040022361 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): try: raise A except: print "caught" except A: print "hit" Nuitka-0.5.18.1/tests/syntax/YieldFromInModule.py0000644000175000017500000000150212651071040022014 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # x = (i for i in (yield from range(7)) if (yield from range(2))) Nuitka-0.5.18.1/tests/syntax/ContinueWithoutLoop.py0000644000175000017500000000152212651071040022471 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def test_from_format(self): if sys.version_info >= (3, 3): continue Nuitka-0.5.18.1/tests/syntax/GeneratorReturn_2.py0000644000175000017500000000155712651071040022046 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def g(): for a in range(3): yield a return 7 print("Yielder with return value", list(g())) Nuitka-0.5.18.1/tests/syntax/IndentationError.py0000644000175000017500000000143712651071040021762 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def someFunc(): a b Nuitka-0.5.18.1/tests/syntax/UnpackTwoStars32.py0000644000175000017500000000145112651071040021565 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # foo, *bar, baz, *a, b = name.split('/') Nuitka-0.5.18.1/tests/syntax/UnpackNoTuple.py0000644000175000017500000000141012651071040021213 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # *a = 1 Nuitka-0.5.18.1/tests/syntax/FutureUnknown.py0000644000175000017500000000144512651071040021325 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import not_existing Nuitka-0.5.18.1/tests/syntax/NonlocalForParameter32.py0000644000175000017500000000143312651071040022712 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(a): nonlocal a Nuitka-0.5.18.1/tests/syntax/run_all.py0000755000175000017500000000352612651071040020134 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, compareWithCPython, createSearchMode, decideFilenameVersionSkip ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() for filename in sorted(os.listdir('.')): if not filename.endswith(".py"): continue if not decideFilenameVersionSkip(filename): continue active = search_mode.consider( dirname = None, filename = filename ) if active: extra_flags = ["expect_failure", "remove_output"] compareWithCPython( dirname = None, filename = filename, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.18.1/tests/syntax/NonAsciiWithoutEncoding_2.py0000644000175000017500000000156312651071040023453 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # No encoding declared, but non-ASCII in that string. s = "'\0\"\n\r\t abcd\x85é\U00012fff\uD800\U0001D121xxx." Nuitka-0.5.18.1/tests/syntax/ClassReturn.py0000644000175000017500000000142712651071040020740 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class X: return 3 Nuitka-0.5.18.1/tests/syntax/YieldInAsync35.py0000644000175000017500000000144412651071040021175 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # async def g(): yield from None Nuitka-0.5.18.1/tests/syntax/ExecWithNesting_2.py0000644000175000017500000000212712651071040021762 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # In Python2, it is not allowed to take closure variables, in a function that # has an "exec". For Python3, the problem doesn't exist, as there is no exec # statement anymore. def someFunctionWithUnqualifiedExecAndCallback(): exec "def f(): pass" def callback(): return nested someFunctionWithUnqualifiedExecAndCallback() Nuitka-0.5.18.1/tests/syntax/MisplacedFutureImport.py0000644000175000017500000000151212651071040022755 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): from __future__ import print_function print(locals()) Nuitka-0.5.18.1/tests/syntax/BreakWithoutLoop.py0000644000175000017500000000154212651071040021733 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # marker1 def test_from_format(self): if sys.version_info >= (3, 3): break # marker2 Nuitka-0.5.18.1/tests/syntax/NonlocalNotFound32.py0000644000175000017500000000154712651071040022065 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def testNonlocal(): x = 0 y = 0 def f(): nonlocal z f() testNonlocal() Nuitka-0.5.18.1/tests/syntax/ModuleReturn.py0000644000175000017500000000144112651071040021114 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # lalala = 3 return lelele = 7 Nuitka-0.5.18.1/tests/syntax/UnknownEncoding.py0000644000175000017500000000142712651071040021601 0ustar hayenhayen00000000000000# coding: no-existing # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/syntax/ClosureDel_2.py0000644000175000017500000000171212651071040020752 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def deletingClosure(): a = 1 def closureTaker(): return a del a try: x = closureTaker() except Exception as e: print "Occurred %r" % e deletingClosure() Nuitka-0.5.18.1/tests/syntax/DuplicateArgument.py0000644000175000017500000000142712651071040022110 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(a, a): pass Nuitka-0.5.18.1/tests/syntax/GlobalForParameter.py0000644000175000017500000000143112651071040022176 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(a): global a Nuitka-0.5.18.1/tests/syntax/StarImportExtra.py0000644000175000017500000000157712651071040021611 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: from sys import not_there, * except Exception as e: print( "Star import with extra stuff not present gave", e ) Nuitka-0.5.18.1/tests/syntax/YieldInModule.py0000644000175000017500000000144612651071040021177 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # x = (i for i in (yield) if (yield)) Nuitka-0.5.18.1/tests/syntax/FutureBraces.py0000644000175000017500000000147412651071040021067 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import braces print( "Look ma, braces." ) Nuitka-0.5.18.1/tests/syntax/LateFutureImport.py0000644000175000017500000000160412651071040021743 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Not allowed to do future imports that are not the first statements of the # module. a = 1 from __future__ import print_function Nuitka-0.5.18.1/tests/reflected/0000755000175000017500000000000012651072347016537 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/reflected/compile_itself.py0000755000175000017500000002476512651071040022115 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys, shutil, time, difflib, subprocess # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, getTempDir ) python_version = setup() # TODO: This ought to no longer be necessary, removing it could highlight bugs. # No random hashing, it makes comparing outputs futile. if "PYTHONHASHSEED" not in os.environ: os.environ["PYTHONHASHSEED"] = '0' nuitka_main_path = os.path.join("..", "..", "bin", "nuitka") tmp_dir = getTempDir() # TODO: Could detect this more automatic. PACKAGE_LIST = ( "nuitka", "nuitka/nodes", "nuitka/tree", "nuitka/importing", "nuitka/build", "nuitka/freezer", "nuitka/gui", "nuitka/codegen", "nuitka/codegen/templates", "nuitka/optimizations", "nuitka/finalizations", "nuitka/plugins", "nuitka/plugins/standard", "nuitka/plugins/user", "nuitka/containers", "nuitka/utils", ) def readSource(filename): if python_version < "3": return open(filename, "rb").read() else: return open(filename, "rb").read().decode("latin1") def diffRecursive(dir1, dir2): done = set() for filename in os.listdir(dir1): path1 = os.path.join(dir1, filename) path2 = os.path.join(dir2, filename) done.add(path1) # Skip these binary files of course. if filename.endswith(".o") or \ filename.endswith(".os") or \ filename.endswith(".obj"): continue # Skip scons build database if filename == ".sconsign.dblite": continue if not os.path.exists(path2): sys.exit("Only in %s: %s" % (dir1, filename)) if os.path.isdir(path1): diffRecursive(path1, path2) elif os.path.isfile(path1): fromdate = time.ctime(os.stat(path1).st_mtime) todate = time.ctime(os.stat(path2).st_mtime) diff = difflib.unified_diff( a = readSource(path1).splitlines(), b = readSource(path2).splitlines(), fromfile = path1, tofile = path2, fromfiledate = fromdate, tofiledate = todate, n = 3 ) result = list(diff) if result: for line in result: my_print(line) sys.exit(1) else: assert False, path1 for filename in os.listdir(dir2): path1 = os.path.join(dir1, filename) path2 = os.path.join(dir2, filename) if path1 in done: continue if not os.path.exists(path1): sys.exit("Only in %s: %s" % (dir2, filename)) def executePASS1(): my_print("PASS 1: Compiling from compiler running from .py files.") base_dir = os.path.join("..", "..") for package in PACKAGE_LIST: package = package.replace('/', os.path.sep) source_dir = os.path.join(base_dir, package) target_dir = package if os.path.exists(target_dir): shutil.rmtree(target_dir) os.mkdir(target_dir) for filename in os.listdir(target_dir): if filename.endswith(".so"): path = os.path.join(target_dir, filename) os.unlink(path) for filename in sorted(os.listdir(source_dir)): if not filename.endswith(".py"): continue if filename.startswith(".#"): continue path = os.path.join(source_dir, filename) if filename != "__init__.py": my_print("Compiling", path) command = [ os.environ["PYTHON"], nuitka_main_path, "--module", "--recurse-none", "--plugin-enable=pylint-warnings", "--output-dir=%s" % target_dir, path ] command += os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() result = subprocess.call( command ) if result != 0: sys.exit(result) else: shutil.copyfile(path, os.path.join(target_dir, filename)) my_print("Compiling", nuitka_main_path) shutil.copyfile(nuitka_main_path, "nuitka.py") command = [ os.environ["PYTHON"], nuitka_main_path, "--recurse-none", "--output-dir=.", "--python-flag=-S", "nuitka.py" ] command += os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() result = subprocess.call( command ) if result != 0: sys.exit(result) scons_inline_copy_path = os.path.join( base_dir, "nuitka", "build", "inline_copy" ) if os.path.exists(scons_inline_copy_path): shutil.copytree( scons_inline_copy_path, os.path.join("nuitka", "build", "inline_copy") ) shutil.copy( os.path.join(base_dir, "nuitka", "build", "SingleExe.scons"), os.path.join("nuitka", "build", "SingleExe.scons") ) shutil.copytree( os.path.join(base_dir, "nuitka", "build", "static_src"), os.path.join("nuitka", "build", "static_src") ) shutil.copytree( os.path.join(base_dir, "nuitka", "build", "include"), os.path.join("nuitka", "build", "include") ) def compileAndCompareWith(nuitka): base_dir = os.path.join("..", "..") for package in PACKAGE_LIST: package = package.replace('/', os.path.sep) source_dir = os.path.join(base_dir, package) for filename in sorted(os.listdir(source_dir)): if not filename.endswith(".py"): continue if filename.startswith(".#"): continue path = os.path.join(source_dir, filename) if filename != "__init__.py": my_print("Compiling", path) target = filename.replace(".py", ".build") target_dir = os.path.join(tmp_dir, target) if os.path.exists(target_dir): shutil.rmtree(target_dir) command = [ nuitka, "--module", "--recurse-none", "--plugin-enable=pylint-warnings", "--output-dir=%s"% tmp_dir, path ] command += os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() result = subprocess.call( command ) if result != 0: sys.exit(result) diffRecursive(os.path.join(package, target), target_dir) shutil.rmtree(target_dir) if os.name == "nt": target_filename = filename.replace(".py", ".pyd") else: target_filename = filename.replace(".py", ".so") os.unlink(os.path.join(tmp_dir, target_filename)) def executePASS2(): my_print( "PASS 2: Compiling from compiler running from .exe and many .so files." ) # Windows will load the compiled modules (pyd) only from PYTHONPATH, so we # have to add it. if os.name == "nt": os.environ["PYTHONPATH"] = ':'.join(PACKAGE_LIST) compileAndCompareWith(os.path.join('.', "nuitka.exe")) # Undo the damage from above. if os.name == "nt": del os.environ["PYTHONPATH"] my_print("OK.") def executePASS3(): my_print( "PASS 3: Compiling from compiler running from .py files to single .exe." ) exe_path = os.path.join(tmp_dir, "nuitka.exe") if os.path.exists(exe_path): os.unlink(exe_path) build_path = os.path.join(tmp_dir, "nuitka.build") if os.path.exists(build_path): shutil.rmtree(build_path) path = os.path.join("..", "..", "bin", "nuitka") my_print("Compiling", path) command = [ os.environ["PYTHON"], nuitka_main_path, path, "--output-dir=%s" % tmp_dir, "--python-flag=-S", "--recurse-all" ] result = subprocess.call( command ) if result != 0: sys.exit(result) shutil.rmtree(build_path) my_print("OK.") def executePASS4(): my_print("PASS 4: Compiling the compiler running from single exe.") exe_path = os.path.join(tmp_dir, "nuitka.exe") compileAndCompareWith(exe_path) my_print("OK.") def executePASS5(): my_print( "PASS 5: Compiling the compiler 'nuitka' package to single '.so' file." ) path = os.path.join("..", "..", "nuitka") command = [ os.environ["PYTHON"], nuitka_main_path, "--plugin-enable=pylint-warnings", "--output-dir=%s" % tmp_dir, "--recurse-all", "--recurse-not-to=PyQt4", "--recurse-not-to=nuitka.build.inline_copy", "--recurse-dir=%s" % path, "--module", path ] result = subprocess.call( command ) if result != 0: sys.exit(result) os.unlink(os.path.join(tmp_dir, "nuitka.so")) shutil.rmtree(os.path.join(tmp_dir, "nuitka.build")) cross_compilation = False executePASS1() if cross_compilation: my_print("PASS 2: Skipped for cross-compilation case.") else: executePASS2() executePASS3() if cross_compilation: my_print("PASS 4: Skipped for cross-compilation case.") else: executePASS4() shutil.rmtree("nuitka") executePASS5() os.unlink(os.path.join(tmp_dir, "nuitka.exe")) os.rmdir(tmp_dir) Nuitka-0.5.18.1/tests/test_common.py0000644000175000017500000005646412651071040017507 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Common test infrastructure functions. To be used by test runners. """ from __future__ import print_function import os, sys, subprocess, tempfile, atexit, shutil, re, ast, doctest # Make sure we flush after every print, the "-u" option does more than that # and this is easy enough. def my_print(*args, **kwargs): print(*args, **kwargs) sys.stdout.flush() def check_output(*popenargs, **kwargs): if "stdout" in kwargs: raise ValueError("stdout argument not allowed, it will be overridden.") process = subprocess.Popen( stdout = subprocess.PIPE, *popenargs, **kwargs ) output, _unused_err = process.communicate() retcode = process.poll() if retcode: cmd = kwargs.get("args") if cmd is None: cmd = popenargs[0] raise subprocess.CalledProcessError(retcode, cmd, output = output) return output def check_result(*popenargs, **kwargs): if "stdout" in kwargs: raise ValueError("stdout argument not allowed, it will be overridden.") process = subprocess.Popen( stdout = subprocess.PIPE, *popenargs, **kwargs ) _unused_output, _unused_err = process.communicate() retcode = process.poll() if retcode: return False else: return True def setup(needs_io_encoding = False, silent = False): # Go its own directory, to have it easy with path knowledge. os.chdir( os.path.dirname( os.path.abspath(sys.modules[ "__main__" ].__file__) ) ) if "PYTHON" not in os.environ: os.environ["PYTHON"] = sys.executable # Allow providing 33, 27, and expand that to python2.7 if len(os.environ["PYTHON"]) == 2 and \ os.environ["PYTHON"].isdigit() and \ os.name != "nt": os.environ["PYTHON"] = "python%s.%s" % ( os.environ["PYTHON"][0], os.environ["PYTHON"][1] ) if needs_io_encoding and "PYTHONIOENCODING" not in os.environ: os.environ["PYTHONIOENCODING"] = "utf-8" version_output = check_output( ( os.environ["PYTHON"], "-c", """\ import sys, os;\ print(".".join(str(s) for s in list(sys.version_info)[:3]));\ print(("x86_64" if "AMD64" in sys.version else "x86") if os.name == "nt" else os.uname()[4]);\ """, ), stderr = subprocess.STDOUT ) global python_version, python_arch python_version = version_output.split(b"\n")[0].strip() python_arch = version_output.split(b"\n")[1].strip() if sys.version.startswith('3'): python_arch = python_arch.decode() python_version = python_version.decode() os.environ["PYTHON_VERSION"] = python_version if not silent: my_print("Using concrete python", python_version, "on", python_arch) assert type(python_version) is str, repr(python_version) assert type(python_arch) is str, repr(python_arch) if "COVERAGE_FILE" not in os.environ: os.environ["COVERAGE_FILE"] = os.path.join( os.path.dirname(__file__), "..", ".coverage" ) return python_version tmp_dir = None def getTempDir(): # Create a temporary directory to work in, automatically remove it in case # it is empty in the end. global tmp_dir if tmp_dir is None: tmp_dir = tempfile.mkdtemp( prefix = os.path.basename( os.path.dirname( os.path.abspath(sys.modules[ "__main__" ].__file__) ) ) + '-', dir = tempfile.gettempdir() if not os.path.exists("/var/tmp") else "/var/tmp" ) def removeTempDir(): try: shutil.rmtree(tmp_dir) except OSError: pass atexit.register(removeTempDir) return tmp_dir def convertUsing2to3(path, force = False): command = [ os.environ["PYTHON"], "-m", "py_compile", path ] if not force and "xrange" not in open(path).read(): with open(os.devnull, 'w') as stderr: if check_result(command, stderr = stderr): return path, False filename = os.path.basename(path) new_path = os.path.join(getTempDir(), filename) shutil.copy(path, new_path) # On Windows, we cannot rely on 2to3 to be in the path. if os.name == "nt": command = [ sys.executable, os.path.join( os.path.dirname(sys.executable), "Tools/Scripts/2to3.py" ) ] else: command = [ "2to3" ] command += [ "-w", "-n", "--no-diffs", new_path ] with open(os.devnull, 'w') as devnull: check_output( command, stderr = devnull ) return new_path, True def decideFilenameVersionSkip(filename): assert type(filename) is str assert type(python_version) is str # Skip runner scripts by default. if filename.startswith("run_"): return False # Skip tests that require Python 2.7 at least. if filename.endswith("27.py") and python_version.startswith("2.6"): return False if filename.endswith("_2.py") and python_version.startswith('3'): return False # Skip tests that require Python 3.2 at least. if filename.endswith("32.py") and python_version < "3.2": return False # Skip tests that require Python 3.3 at least. if filename.endswith("33.py") and python_version < "3.3": return False # Skip tests that require Python 3.4 at least. if filename.endswith("34.py") and python_version < "3.4": return False # Skip tests that require Python 3.5 at least. if filename.endswith("35.py") and python_version < "3.5": return False return True def compareWithCPython(dirname, filename, extra_flags, search_mode, needs_2to3): if dirname is None: path = filename else: path = os.path.join(dirname, filename) # Apply 2to3 conversion if necessary. if needs_2to3: path, converted = convertUsing2to3(path) else: converted = False command = [ sys.executable, os.path.join("..", "..", "bin", "compare_with_cpython"), path, "silent" ] if extra_flags is not None: command += extra_flags command += search_mode.getExtraFlags(dirname, filename) try: result = subprocess.call( command ) except KeyboardInterrupt: result = 2 # Cleanup, some tests apparently forget that. try: if os.path.isdir("@test"): shutil.rmtree("@test") elif os.path.isfile("@test"): os.unlink("@test") except OSError: # This seems to work for broken "lnk" files. if os.name == "nt": os.system("rmdir /S /Q @test") if os.path.exists("@test"): raise if result != 0 and \ result != 2 and \ search_mode.abortOnFinding(dirname, filename): my_print("Error exit!", result) sys.exit(result) if converted: os.unlink(path) if result == 2: sys.stderr.write("Interrupted, with CTRL-C\n") sys.exit(2) def hasDebugPython(): global python_version # On Debian systems, these work. debug_python = os.path.join("/usr/bin/", os.environ["PYTHON"] + "-dbg") if os.path.exists(debug_python): return True # For other Python, if it's the one also executing the runner, which is # very probably the case, we check that. We don't check the provided # binary here, this could be done as well. if sys.executable == os.environ["PYTHON"] and \ hasattr(sys, "gettotalrefcount"): return True # Otherwise no. return False def getArchitecture(): if os.name == "nt": if "AMD64" in sys.version: return "x86_64" else: return "x86" else: return os.uname()[4] def getDependsExePath(): if "APPDATA" not in os.environ: sys.exit("Error, standalone mode cannot find 'APPDATA' environment.") nuitka_app_dir = os.path.join(os.environ["APPDATA"],"nuitka") depends_dir = os.path.join( nuitka_app_dir, python_arch, ) depends_exe = os.path.join( depends_dir, "depends.exe" ) assert os.path.exists(depends_exe), depends_exe return depends_exe def isExecutableCommand(command): path = os.environ["PATH"] suffixes = (".exe",) if os.name == "nt" else ("",) for part in path.split(os.pathsep): if not part: continue for suffix in suffixes: if os.path.isfile(os.path.join(part, command + suffix)): return True return False def getRuntimeTraceOfLoadedFiles(path, trace_error = True): """ Returns the files loaded when executing a binary. """ result = [] if os.name == "posix": if sys.platform == "darwin": if not isExecutableCommand("dtruss"): sys.exit( """\ Error, needs 'dtruss' on your system to scan used libraries.""" ) args = ( "sudo", "dtruss", "-f", "-t", "open", path ) else: if not isExecutableCommand("strace"): sys.exit( """\ Error, needs 'strace' on your system to scan used libraries.""" ) args = ( "strace", "-e", "file", "-s4096", # Some paths are truncated otherwise. path ) process = subprocess.Popen( args = args, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) _stdout_strace, stderr_strace = process.communicate() open(path+".strace","wb").write(stderr_strace) for line in stderr_strace.split(b"\n"): if process.returncode != 0 and trace_error: my_print(line) if not line: continue # Don't consider files not found. The "site" module checks lots # of things. if b"ENOENT" in line: continue if line.startswith(b"stat(") and b"S_IFDIR" in line: continue # Allow stats on the python binary, and stuff pointing to the # standard library, just not uses of it. It will search there # for stuff. if line.startswith(b"lstat(") or \ line.startswith(b"stat(") or \ line.startswith(b"readlink("): filename = line[line.find(b"(")+2:line.find(b", ")-1] if filename in (b"/usr", b"/usr/bin"): continue if filename == b"/usr/bin/python" + python_version[:3].encode("utf8"): continue if filename in (b"/usr/bin/python", b"/usr/bin/python2", b"/usr/bin/python3"): continue result.extend( os.path.abspath(match) for match in re.findall(b'"(.*?)(?:\\\\0)?"', line) ) if sys.version.startswith('3'): result = [s.decode("utf-8") for s in result] elif os.name == "nt": subprocess.call( ( getDependsExePath(), "-c", "-ot%s" % path + ".depends", "-f1", "-pa1", "-ps1", "-pp0", "-pl1", path ) ) inside = False for line in open(path + ".depends"): if "| Module Dependency Tree |" in line: inside = True continue if not inside: continue if "| Module List |" in line: break if ']' not in line: continue # Skip missing DLLs, apparently not needed anyway. if '?' in line[:line.find(']')]: continue dll_filename = line[line.find(']')+2:-1] assert os.path.isfile(dll_filename), dll_filename # The executable itself is of course exempted. if os.path.normcase(dll_filename) == \ os.path.normcase(os.path.abspath(path)): continue dll_filename = os.path.normcase(dll_filename) result.append(dll_filename) os.unlink(path + ".depends") result = list(sorted(set(result))) return result def hasModule(module_name): result = subprocess.call( ( os.environ["PYTHON"], "-c" "import %s" % module_name ), stdout = open(os.devnull,'w'), stderr = subprocess.STDOUT ) return result == 0 m1 = {} m2 = {} def snapObjRefCntMap(before): import gc if before: global m1 m = m1 else: global m2 m = m2 for x in gc.get_objects(): if x is m1: continue if x is m2: continue m[ str(x) ] = sys.getrefcount(x) def checkReferenceCount(checked_function, max_rounds = 10): assert sys.exc_info() == (None, None, None), sys.exc_info() print(checked_function.__name__ + ": ", end = "") sys.stdout.flush() ref_count1 = 17 ref_count2 = 17 explain = False import gc assert max_rounds > 0 for count in range(max_rounds): gc.collect() ref_count1 = sys.gettotalrefcount() # @UndefinedVariable if explain and count == max_rounds - 1: snapObjRefCntMap(True) checked_function() # Not allowed, but happens when bugs occur. assert sys.exc_info() == (None, None, None), sys.exc_info() gc.collect() if explain and count == max_rounds - 1: snapObjRefCntMap(False) ref_count2 = sys.gettotalrefcount() # @UndefinedVariable if ref_count1 == ref_count2: result = True print("PASSED") break # print count, ref_count1, ref_count2 else: result = False print("FAILED", ref_count1, ref_count2, "leaked", ref_count2 - ref_count1) if explain: assert m1 assert m2 for key in m1.keys(): if key not in m2: print('*' * 80) print(key) elif m1[key] != m2[key]: print('*' * 80) print(key) else: pass # print m1[key] assert sys.exc_info() == (None, None, None), sys.exc_info() gc.collect() sys.stdout.flush() return result def createSearchMode(): search_mode = len(sys.argv) > 1 and sys.argv[1] == "search" start_at = sys.argv[2] if len(sys.argv) > 2 else None coverage_mode = len(sys.argv) > 1 and sys.argv[1] == "coverage" class SearchModeBase: def __init__(self): self.may_fail = [] def consider(self, dirname, filename): return True def finish(self): pass def abortOnFinding(self, dirname, filename): for candidate in self.may_fail: if self._match(dirname, filename, candidate): return False return True def getExtraFlags(self, dirname, filename): return [] def mayFailFor(self, *names): self.may_fail += names def _match(self, dirname, filename, candidate): parts = [dirname, filename] while None in parts: parts.remove(None) assert parts path = os.path.join(*parts) return candidate in ( dirname, filename, filename.replace(".py", ""), path, path.replace(".py", "") ) def isCoverage(self): return False if coverage_mode: class SearchModeCoverage(SearchModeBase): def getExtraFlags(self, dirname, filename): return ["coverage"] def isCoverage(self): return True return SearchModeCoverage() elif search_mode and start_at: start_at = start_at.replace('/', os.path.sep) class SearchModeByPattern(SearchModeBase): def __init__(self): SearchModeBase.__init__(self) self.active = False def consider(self, dirname, filename): if self.active: return True self.active = self._match(dirname, filename, start_at) return self.active def finish(self): if not self.active: sys.exit("Error, became never active.") return SearchModeByPattern() else: class SearchModeImmediate(SearchModeBase): def abortOnFinding(self, dirname, filename): return search_mode and \ SearchModeBase.abortOnFinding(self, dirname, filename) return SearchModeImmediate() def reportSkip(reason, dirname, filename): my_print("Skipped, %s (%s)." % (os.path.join(dirname, filename), reason)) def executeReferenceChecked(prefix, names, tests_skipped, tests_stderr): import gc gc.disable() extract_number = lambda name: int(name.replace(prefix, "")) # Find the function names. matching_names = tuple( name for name in names if name.startswith(prefix) and name[-1].isdigit() ) old_stderr = sys.stderr # Everything passed result = True for name in sorted(matching_names, key = extract_number): number = extract_number(name) # print(tests_skipped) if number in tests_skipped: my_print(name + ": SKIPPED (%s)" % tests_skipped[number]) continue # Avoid unraisable output. try: if number in tests_stderr: sys.stderr = open(os.devnull, "wb") except Exception: # Windows if not checkReferenceCount(names[name]): result = False else: if not checkReferenceCount(names[name]): result = False if number in tests_stderr: new_stderr = sys.stderr sys.stderr = old_stderr new_stderr.close() gc.enable() return result from contextlib import contextmanager @contextmanager def withPythonPathChange(python_path): if type(python_path) in (tuple, list): python_path = os.pathsep.join(python_path) if python_path: if "PYTHONPATH" in os.environ: old_path = os.environ["PYTHONPATH"] os.environ["PYTHONPATH"] += os.pathsep + python_path else: old_path = None os.environ["PYTHONPATH"] = python_path yield if python_path: if old_path is None: del os.environ["PYTHONPATH"] else: os.environ["PYTHONPATH"] = old_path @contextmanager def withExtendedExtraOptions(*args): assert args old_value = os.environ.get("NUITKA_EXTRA_OPTIONS", None) value = old_value for arg in args: if value is None: value = arg else: value += ' ' + arg os.environ[ "NUITKA_EXTRA_OPTIONS" ] = value yield if old_value is None: del os.environ[ "NUITKA_EXTRA_OPTIONS" ] else: os.environ[ "NUITKA_EXTRA_OPTIONS" ] = old_value def indentedCode(codes, count): """ Indent code, used for generating test codes. """ return '\n'.join( ' ' * count + line if line else "" for line in codes ) def convertToPython(doctests, line_filter = None): """ Convert give doctest string to static Python code. """ code = doctest.script_from_examples(doctests) if code.endswith('\n'): code += "#\n" else: assert False output = [] inside = False def getPrintPrefixed(evaluated): try: node = ast.parse(evaluated.lstrip(), "eval") except SyntaxError: return evaluated if node.body[0].__class__.__name__ == "Expr": count = 0 while evaluated.startswith(' ' * count): count += 1 if sys.version_info < (3,): modified = (count-1) * ' ' + "print " + evaluated return (count-1) * ' ' + ("print 'Line %d'" % line_number) + "\n" + modified else: modified = (count-1) * ' ' + "print(" + evaluated + "\n)\n" return (count-1) * ' ' + ("print('Line %d'" % line_number) + ")\n" + modified else: return evaluated def getTried(evaluated): if sys.version_info < (3,): return """ try: %(evaluated)s except Exception as __e: print "Occurred", type(__e), __e """ % { "evaluated" : indentedCode(getPrintPrefixed(evaluated).split('\n'), 4) } else: return """ try: %(evaluated)s except Exception as __e: print("Occurred", type(__e), __e) """ % { "evaluated" : indentedCode(getPrintPrefixed(evaluated).split('\n'), 4) } def isOpener(evaluated): evaluated = evaluated.lstrip() if evaluated == "": return False if evaluated.split()[0] in ("def", "class", "for", "while", "try:", "except", "except:", "finally:", "else:"): return True else: return False for line_number, line in enumerate(code.split('\n')): # print "->", inside, line if line_filter is not None and line_filter(line): continue if inside and len(line) > 0 and line[0].isalnum() and not isOpener(line): output.append(getTried('\n'.join(chunk))) chunk = [] inside = False if inside and not (line.startswith('#') and line.find("SyntaxError:") != -1): chunk.append(line) elif line.startswith('#'): if line.find("SyntaxError:") != -1: # print "Syntax error detected" if inside: # print "Dropping chunk", chunk chunk = [] inside = False else: del output[-1] elif isOpener(line): inside = True chunk = [line] elif line.strip() == "": output.append(line) else: output.append(getTried(line)) return '\n'.join(output).rstrip() + '\n' Nuitka-0.5.18.1/tests/basics/0000755000175000017500000000000012651072347016046 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/basics/Crashers.py0000644000175000017500000000362412651071040020164 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import random def getrandom(): return random.random() def optimizerCrashIssue13(): try: print("Something with side effects that might raise.") except Exception as x: print("Caught it") raise print("Should not reach this") raise x # Just so it won't be optimized away entirely, the run time has no issue. optimizerCrashIssue13() def codegeneratorCrashIssue15(): f = float("nan") g = getrandom() # Prevent optimization of "nan"-constant return f+g # Just so it won't be optimized away entirely. codegeneratorCrashIssue15() def codegeneratorCrashIssue30(): f = getrandom() # Prevent optimization f # Will be optimized way in later versions of Nuitka. # TODO: May already be the case. def runtimeCrashIssue(): # Some C functions don't set an exception when called. from itertools import count import sys try: # This will set an error and return a value for at least Python3.2 count(1, sys.maxsize+5) except TypeError: # For Python2.6, the two arguments variant didn't exist yet. assert sys.version_info < (2,7) runtimeCrashIssue() Nuitka-0.5.18.1/tests/basics/HelloWorld.py0000644000175000017500000000240412651071040020460 0ustar hayenhayen00000000000000# -*- coding: utf-8 -*- # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print "Hello World from module main code" def printHelloWorld(): print "Hello World from function main code" print printHelloWorld printHelloWorld() def printHelloWorld2(arg): print arg print printHelloWorld2 printHelloWorld2("Hello World from function positional argument") printHelloWorld2(arg = "Hello World from function keyword argument") def printHelloWorld3(arg = "Hello World from function default argument"): print arg print printHelloWorld3 printHelloWorld3() Nuitka-0.5.18.1/tests/basics/OrderChecks27.py0000644000175000017500000000311212651071040020747 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def raiseOrderCheck(): print("Checking order of raises:") def exception_type(): print("exception_type", end = "") return ValueError def exception_value(): print("exception_value", end = "") return 1 def exception_tb(): print("exception_value", end = "") return None print("3 args", end = "") try: raise exception_type(), exception_value(), exception_tb() except Exception as e: print("caught", repr(e)) print("2 args", end = "") try: raise exception_type(), exception_value() except Exception as e: print("caught", repr(e)) print("1 args", end = "") try: raise exception_type() except Exception as e: print("caught", repr(e)) raiseOrderCheck() Nuitka-0.5.18.1/tests/basics/run_xml.py0000755000175000017500000000626112651071040020101 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import os, sys, subprocess, tempfile, shutil # Go its own directory, to have it easy with path knowledge. nuitka1 = sys.argv[1] nuitka2 = sys.argv[2] search_mode = len(sys.argv) > 3 and sys.argv[3] == "search" start_at = sys.argv[4] if len(sys.argv) > 4 else None if start_at: active = False else: active = True def check_output(*popenargs, **kwargs): from subprocess import Popen, PIPE, CalledProcessError if "stdout" in kwargs: raise ValueError("stdout argument not allowed, it will be overridden.") process = Popen(stdout = PIPE, *popenargs, **kwargs) output, unused_err = process.communicate() retcode = process.poll() if retcode: cmd = kwargs.get("args") if cmd is None: cmd = popenargs[0] raise CalledProcessError(retcode, cmd, output = output) return output my_dir = os.path.dirname(os.path.abspath(__file__)) for filename in sorted(os.listdir(my_dir)): if not filename.endswith(".py") or filename.startswith("run_"): continue path = os.path.relpath(os.path.join(my_dir, filename)) if not active and start_at in (filename, path): active = True if active: # TODO: Reactivate Python3 support here. if False: new_path = os.path.join(tempfile.gettempdir(), filename) shutil.copy(path, new_path) path = new_path # On Windows, we cannot rely on 2to3 to be in the path. if os.name == "nt": command = sys.executable + ' ' + os.path.join(os.path.dirname(sys.executable), "Tools/Scripts/2to3.py") else: command = "2to3" result = subprocess.call( command + " -w -n --no-diffs " + path, stderr = open(os.devnull, 'w'), shell = True ) command = "%s %s '%s' '%s' %s" % ( sys.executable, os.path.join(my_dir, "..", "..", "bin", "compare_with_xml"), nuitka1, nuitka2, path, ) result = subprocess.call( command, shell = True ) if result == 2: sys.stderr.write("Interrupted, with CTRL-C\n") sys.exit(2) if result != 0 and search_mode: print("Error exit!", result) sys.exit(result) else: print("Skipping", filename) Nuitka-0.5.18.1/tests/basics/Varargs.py0000644000175000017500000000407012651071040020013 0ustar hayenhayen00000000000000# -*- coding: utf-8 -*- # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def plain_list_dict_args_function(plain, *arg_list, **arg_dict): print "plain", plain, "arg_list", arg_list, "arg_dict", arg_dict def plain_list_args_function(plain, *arg_list): print plain, arg_list def plain_dict_args_function(plain, **arg_dict): print plain, arg_dict print "Function with plain arg and varargs dict:" plain_dict_args_function(1, a = 2, b = 3, c = 4) plain_dict_args_function(1) print "Function with plain arg and varargs list:" plain_list_args_function(1, 2, 3, 4) plain_list_args_function(1) print "Function with plain arg, varargs list and varargs dict:" plain_list_dict_args_function(1, 2, z = 3) plain_list_dict_args_function(1, 2, 3) plain_list_dict_args_function(1, a = 2, b = 3, c = 4) def list_dict_args_function(*arg_list, **arg_dict): print arg_list, arg_dict def list_args_function(*arg_list): print arg_list def dict_args_function(**arg_dict): print arg_dict print "Function with plain arg and varargs dict:" dict_args_function(a = 2, b = 3, c = 4) dict_args_function() print "Function with plain arg and varargs list:" list_args_function(2, 3, 4) list_args_function() print "Function with plain arg, varargs list and varargs dict:" list_dict_args_function(2, z = 3) list_dict_args_function(2, 3) list_dict_args_function(a = 2, b = 3, c = 4) Nuitka-0.5.18.1/tests/basics/Decorators.py0000644000175000017500000000312312651071040020511 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def decorator1(f): print("Executing decorator 1") def deco_f(): return f() + 2 return deco_f def decorator2(f): print("Executing decorator 2") def deco_f(): return f() * 2 return deco_f # Result of function now depends on correct order of applying the decorators @decorator1 @decorator2 def function1(): return 3 print(function1()) def deco_returner1(): print("Executing decorator returner D1") return decorator1 def deco_returner2(): print("Executing decorator returner D2") return decorator2 @deco_returner1() @deco_returner2() def function2(): return 3 print(function2()) # Same as function2, but without decorator syntax. def function3(): return 3 function3 = deco_returner1()(deco_returner2()(function3)) print(function3()) Nuitka-0.5.18.1/tests/basics/TryReturnFinally.py0000644000175000017500000000542312651071040021706 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # In this test we show that return in try/finally executes the finally part # just fine. from __future__ import print_function import sys def eight(): return 8 def nine(): return 9 def raisy1(): return 1 / 0 def raisy2(): return 1() def raisy3(arg): raise TypeError(arg) def returnInTried(for_call): try: print("returnInTried with exception info in tried block:", sys.exc_info()) return for_call() finally: print("returnInTried with exception info in final block:", sys.exc_info()) def returnInFinally(for_call): try: print("returnInFinally with exception info in tried block:", sys.exc_info()) finally: print("returnInFinally with exception info in final block:", sys.exc_info()) return for_call() print("Standard try finally with return in tried block:") print("result", returnInTried(eight)) print('*' * 80) print("Standard try finally with return in final block:") print("result", returnInFinally(nine)) print('*' * 80) print("Exception raising try finally with return in tried block:") try: print("result", returnInTried(raisy1)) except Exception as e: print("Gave exception", repr(e)) print('*' * 80) print("Exception raising try finally with return in final block:") try: print("result", returnInFinally(raisy2)) except Exception as e: print("Gave exception", repr(e)) print('*' * 80) try: raisy3("unreal 1") except Exception as outer_e: print("Exception raising try finally with return in tried block:") try: print("result", returnInTried(raisy1)) except Exception as e: print("Gave exception", repr(e)) print("Handler exception remains:", repr(outer_e)) print('*' * 80) try: raisy3("unreal 2") except Exception as outer_e: print("Exception raising try finally with return in final block:") try: print("result", returnInFinally(raisy2)) except Exception as e: print("Gave exception", repr(e)) print("Handler exception remains:", repr(outer_e)) print('*' * 80) Nuitka-0.5.18.1/tests/basics/BuiltinsTest.py0000644000175000017500000002245512651071040021046 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def someFunctionWritingLocals(): x = 1 r = locals() # This is without effect on r. It doesn't mention y at all y = 2 # This adds z to the locals, but only that. r[ 'z' ] = 3 del x try: z except Exception as e: print("Accessing z writing to locals gives Exception", e) return r, y def someFunctionWritingLocalsContainingExec(): x = 1 r = locals() # This is without effect on r. It doesn't mention y at all y = 2 # This adds z to the locals, but only that. r[ 'z' ] = 3 try: z except Exception as e: print("Accessing z writing to locals in exec function gives Exception", e) return r, y # Note: This exec is dead code, and still changes the behaviour of # CPython, because it detects exec during parse already. exec("") print("Testing locals():") print(someFunctionWritingLocals()) print(someFunctionWritingLocalsContainingExec()) def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" import pprint return pprint.pformat(d) print("Vars on module level", displayDict(vars())) module_locals = locals() # Patch away "__file__" path in a hard to detect way. This will make sure, # repeated calls to locals really get the same dictionary. import os module_locals["__file__"] = os.path.basename(module_locals[ "__file__" ]) del module_locals print("Use of locals on the module level", displayDict(locals())) def someFunctionUsingGlobals(): g = globals() g["hallo"] = "du" global hallo print("hallo", hallo) print("Testing globals():") someFunctionUsingGlobals() print("Testing dir():") print("Module dir", end = ' ') def someFunctionUsingDir(): x = someFunctionUsingGlobals() print("Function dir", dir()) return x someFunctionUsingDir() print("Making a new type, with type() and 3 args:", end = ' ') new_class = type("Name", (object,), {}) print(new_class, new_class()) print("None has type", type(None)) print("Constant ranges", range(2), range(1, 6), range(3, 0, -1), range(3, 8, 2), range(5, -5, -3)) print("Border cases", range(0), range(-1), range(-1, 1)) print("Corner case large negative value", range(-2**100)) print("Corner case with large start/end values in small range", range(2**100,2**100+2)) try: print("Range with 0 step gives:", end = ' ') print(range(3, 8, 0)) except ValueError as e: print(repr(e)) try: print("Range with float:", end = ' ') print(range(1.0)) except TypeError as e: print("Gives exception:", repr(e)) try: print("Empty range call", end = ' ') print(range()) except TypeError as e: print("Gives exception:", e) print("List from iterable", list("abc"), list()) print("List from sequence", list(sequence = (0, 1, 2))) print("Tuple from iterable", tuple("cda"), tuple()) print("Tuple from sequence", tuple(sequence = (0, 1, 2))) print("Dictionary from iterable and keywords", dict(("ab", (1, 2)), f = 1, g = 1)) print("More constant dictionaries", {"two": 2, "one": 1}, {}, dict()) g = {"two": 2, "one": 1} print("Variable dictionary", dict(g)) print("Found during optimization", dict(dict({"le": 2, "la": 1}), fu = 3), dict(named = dict({"le": 2, "la": 1}))) print("Floats from constants", float("3.0"), float(x = 9.0), float()) print("Found during optimization", float(float("3.2")), float(x = float(11.0))) print("Complex from constants", complex("3.0j"), complex(real = 9.0), complex(imag = 9.0), complex(1,2), complex()) print("Found during optimization", complex(float("3.2")), complex(real = float(11.0)), complex(imag = float(11.0))) print("Strs from constants", str("3.3"), str(object = 9.1), str()) print("Found during optimization", str(float("3.3")), str(object = float(12.0))) print("Bools from constants", bool("3.3"), bool(x = 9.1), bool(0), bool()) print("Found during optimization", bool(float("3.3")), bool(x = float(0.0))) print("Ints from constants", int('3'), int(x = '9'), int('f', 16), int(x = 'e', base = 16), int("0101", base = 2), int(0), int()) print("Found ints during optimization", int(int('3')), int(x = int(0.0))) try: print("Longs from constants", long('3'), long(x = '9'), long('f', 16), long(x = 'e', base = 16), long("0101", base = 2), long(0), long()) print("Found longs during optimization", long(long('3')), long(x = long(0.0))) except NameError: print("Python3 has no long type.") pass try: print("Int with only base", int(base = 2), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") try: print("Int with large base", int(2, 37), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") try: print("Long with only base", int(base = 2), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") print("Oct from constants", oct(467), oct(0)) print("Found during optimization", oct(int('3'))) print("Hex from constants", hex(467), hex(0)) print("Found during optimization", hex(int('3'))) print("Bin from constants", bin(467), bin(0)) print("Found during optimization", bin(int('3'))) try: int(1,2,3) except Exception as e: print("Too many args gave", repr(e)) try: int(y = 1) except Exception as e: print("Wrong arg", repr(e)) f = 3 print("Unoptimized call of int", int('0' * f, base = 16)) d = { 'x' : "12", "base" : 8 } print("Dict star argument call of int", int(**d)) base = 16 try: value = unicode("20") except NameError: print("Python3: Has unicode by default.") value = "20" print("Unoptimized calls of int with unicode args", int(value, base), int(value)) base = 37 try: print("Int with large base", int(2, base), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") try: print(chr()) except Exception as e: print("Disallowed without args", repr(e)) try: print(ord()) except Exception as e: print("Disallowed without args", repr(e)) try: print(ord(s = 1)) except Exception as e: print("Disallowed keyword args", repr(e)) try: print(ord(1, 2)) except Exception as e: print("Too many plain args", repr(e)) try: print(ord(1, s = 2)) except Exception as e: print("Too many args, some keywords", repr(e)) try: print(str('1', offer = 2)) except Exception as e: print("Too many args, some keywords", repr(e)) # TODO: This is calls, not really builtins. a = 2 print("Can optimize the star list argness away", int(*(a,)), end = ' ') print("Can optimize the empty star list arg away", int(*tuple()), end = ' ') print("Can optimize the empty star dict arg away", int(**dict())) print("Dict building with keyword arguments", dict(), dict(a = f)) print( "Dictionary entirely from constant args", displayDict( dict(q = "Guido", w = "van", e = "Rossum", r = "invented", t = "Python", y = "") ) ) a = 5 print("Instance check recognises", isinstance(a, int)) try: print("Instance check with too many arguments", isinstance(a, long, int)) except Exception as e: print("Too many args", repr(e)) try: print("Instance check with too many arguments", isinstance(a)) except Exception as e: print("Too few args", repr(e)) def usingIterToCheckIterable(a): try: iter(a) except TypeError: print("not iterable") else: print("ok") usingIterToCheckIterable(1) print("Nested constant, dict inside a list, referencing a built-in compile time constant", end = ' ') print([dict(type = int)]) print("nan and -nan sign checks:") from math import copysign print(copysign(1.0, float("nan"))) print(copysign(1.0, float("-nan"))) print("Using != to detect nan floats:") a = float("nan") if a != a: print("is nan") else: print("isn't nan") class CustomStr(str): pass class CustomBytes(bytes): pass class CustomByteArray(bytearray): pass values = [ b'100', b'', bytearray(b'100'), CustomStr("100"), CustomBytes(b'100'), CustomByteArray(b'100') ] for x in values: try: print("int", repr(x), int(x), int(x,2)) except (TypeError, ValueError) as e: print("caught", repr(e)) try: print("long", repr(x), long(x), long(x,2)) except (TypeError, ValueError) as e: print("caught", repr(e)) except NameError: print("Python3 has no long") z = range(5) try: next(z) except TypeError as e: print("caught", repr(e)) try: open() except TypeError as e: print("Open without arguments gives", repr(e)) print("Type of id values:", type(id(id))) Nuitka-0.5.18.1/tests/basics/Classes.py0000644000175000017500000001043012651071040020000 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class SimpleClass: " The class documentation." # Leading space on purpose. # TODO: Doesn't work with Python3, because we don't yet make our own dict # visible. print locals() print "Class locals, while building", str(locals()).replace("'__locals__': {...}, ", "") class_var = 1 def __init__(self, init_parameter): self.x = init_parameter def normal_method(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 return self.arg1, self.arg2, self.x, self @staticmethod def static_method(): return "something" print "Simple class:", SimpleClass print "Lives in", SimpleClass.__module__ print "Documentation", SimpleClass.__doc__ print "Instantiate simple class:", SimpleClass(14) print "Call simple class normal method:", SimpleClass(11).normal_method(1, 2) print "Call simple class static method:", SimpleClass(11).static_method() class MetaClass(type): def __init__(cls, name, bases, dictionary): print "MetaClass is called." cls.addedin = 5 print MetaClass class ComplexClass: __metaclass__ = MetaClass print ComplexClass, dir(ComplexClass) print ComplexClass, hasattr(ComplexClass, "addedin") and ComplexClass.addedin def function(): x = 1 class DynamicClass: y = x return DynamicClass x = 2 print function(), function().y def strangeClassBehaviour(): class StrangeClass(object): count = 0 def __new__(cls): print "__new__" cls.count += 1 return object.__new__(cls) def __del__(self): print "__del__" cls = self.__class__ cls.count -= 1 assert cls.count >= 0 x = StrangeClass() return x.count print "Strange class with __new__ and __del__ overloads", strangeClassBehaviour() class ClosureLocalizer: function = function # Using what looks like a method as a decorator. def deco(f): # @NoSelf f.decorated = True return f @deco def x(self): pass print "Class with a name from module level renamed to local", ClosureLocalizer.function print "Class method decorated", ClosureLocalizer().x.decorated print "Class with decorator and meta class:" def classdecorator(cls): print "cls decorator", cls.addedin return cls @classdecorator class MyClass: __metaclass__ = MetaClass print "Class that updates its locals:", class DictUpdating: a = 1 locals().update({ 'b' : 2 }) for f in range(6): locals()[ "test_%s" % f ] = f print "Changed values", DictUpdating.b, DictUpdating.test_4 def functionThatOffersClosureToPassThroughClass(x): class Foo: global x x = 1 def __call__(self, y): return x + y return Foo() print functionThatOffersClosureToPassThroughClass(6)(2), print x class NameCollisionClosure: def x(self): return x print NameCollisionClosure, NameCollisionClosure().x() class ClassesWithNestedClass: class NestedClass(object): def getDict(self): return { 'a':2 } print ClassesWithNestedClass, ClassesWithNestedClass().NestedClass, ClassesWithNestedClass().NestedClass().getDict() secondary = "global closure wins" class ClassWithModuleVariableCollisionMain(): secondary = None def __init__(self): self.secondary = self.Child() self.value = self.secondary.attr class Child(): def __init__(self): self.attr = secondary print ClassWithModuleVariableCollisionMain, ClassWithModuleVariableCollisionMain().value Nuitka-0.5.18.1/tests/basics/YieldFrom33.py0000644000175000017500000000626112651071040020452 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def g(): for a in range(3): yield a return 7 def h(): yield 4 yield 5 def f(): print("Yielded from returner", (yield from g())) print("Yielded from non-return value", (yield from h())) print( "Result", list( f() ) ) print( "Yielder with return value", list(g()) ) # This will raise when looking up any attribute. class Broken: def __iter__(self): return self def __next__(self): return 1 def __getattr__(self, attr): 1/0 def test_broken_getattr_handling(): def g(): yield from Broken() print( "Next with send: ", end = "" ) try: gi = g() next(gi) gi.send(1) except Exception as e: print( "Caught", repr(e) ) print( "Next with throw: ", end = "" ) try: gi = g() next(gi) gi.throw(AttributeError) except Exception as e: print( "Caught", repr(e) ) print( "Next with close: ", end = "" ) try: gi = g() next(gi) gi.close() print( "All good" ) except Exception as e: print( "Caught", repr(e) ) test_broken_getattr_handling() def test_throw_catched_subgenerator_handling(): def g1(): try: print("Starting g1") yield "g1 ham" yield from g2() yield "g1 eggs" finally: print("Finishing g1") def g2(): try: print("Starting g2") yield "g2 spam" yield "g2 more spam" except LunchError: print("Caught LunchError in g2") yield "g2 lunch saved" yield "g2 yet more spam" class LunchError(Exception): pass g = g1() for i in range(2): x = next(g) print("Yielded %s" % (x,)) e = LunchError("tomato ejected") print( "Throw returned", g.throw(e) ) print( "Sub thrown" ) for x in g: print("Yielded %s" % (x,)) test_throw_catched_subgenerator_handling() def give_cpython_generator(): # TODO: This relies on eval not being inlined, which will become untrue. return eval( "( x for x in range(3) )" ) def gen_compiled(): yield from give_cpython_generator() yield from range(7) print( list( gen_compiled() ) ) Nuitka-0.5.18.1/tests/basics/InplaceOperations.py0000644000175000017500000000241412651071040022025 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function x = 1 x += 2 print("Plain in-place:", x) z = [1, 2, 3] z[1] += 5 print("List in-place:", z[1]) h = { 'a' : 3 } h['a'] += 2 print("Dictionary in-place:", h['a']) class B: a = 1 B.a += 2 print("Class attribute in-place:", B.a) h = [1, 2, 3, 4] h[1:2] += (2,3) print("List 'sclice' in-place [x:y]", h) h[:1] += (9,9) print("List 'sclice' in-place [:y]", h) h[2:] += (6,6) print("List 'sclice' in-place [y:]", h) h[:] += (5,5,5) print("List 'sclice' in-place [:]", h) Nuitka-0.5.18.1/tests/basics/ExceptionRaising33.py0000644000175000017500000000266012651071040022032 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Testing exception changes between generator switches:") def yieldExceptionInteraction(): def yield_raise(): try: raise KeyError("caught") except KeyError: yield from sys.exc_info() yield from sys.exc_info() yield from sys.exc_info() g = yield_raise() print("Initial yield from catch in generator", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("Second yield from the catch reentered", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("After leaving the catch generator yielded", next(g)) yieldExceptionInteraction() Nuitka-0.5.18.1/tests/basics/Looping.py0000644000175000017500000000607012651071040020017 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def cond(): return False def loopingFunction(a = 1*2): c = [] f = [c, a] for a in range(6 or 8): for b in range(8): if a == b: c.append((a,b,True)) elif a < b: c.append((b,a,False)) else: c.append((a,b,False)) if a != b: z = 1 else: z = 0 if z == 0: continue if z == 1 and b == 6: break if a == b: z = 0 print c print f f = 1 while f < (10 or 8): m = 1 f += 1 print "m=", m x = [u for u in range(8)] print x x = [(u,v) for (u,v) in zip(range(8),reversed(range(8))) ] print x x = [(u if u%2==0 else 0) for u in range(10)] print x x = [(u if u%2==0 else 0) for u in (a if cond() else range(9))] print x y = [ [ 3+ (l if l else -1) for l in [m,m+1] ] for m in [f for f in range(2)] ] print "f=", f print "y=", y if x: l = "YES" else: l = "NO" if x: l = "yes" else: if True: l = "no" print "Triple and chain" if m and l and f: print "OK" print "Triple or chain" if m or l or f: print "Okey" print "Nested if not chain" if not m: if not l: print "ok" print "Braced if not chain with 'or'" if not (m or l): print "oki" print "Braced if not chain with 'and'" if not (m and l): print "oki" d=1 print "Nested if chain with outer else" if a: if b or c: if d: print "inside nest" else: print "outer else" print x while False: pass else: print "Executed else branch for False condition while loop" while True: break else: print "Executed else branch for True condition while loop" for x in range(7): pass else: print "Executed else branch for no break for loop" for x in range(7): break else: print "Executed else branch despite break in for loop" x = iter(range(5)) while next(x): pass else: print "Executed else branch of while loop without break" loopingFunction() Nuitka-0.5.18.1/tests/basics/Branching.py0000644000175000017500000000770412651071040020310 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Some random branching to cover most common cases. """ from __future__ import print_function def branchingFunction(a, b, c): print("branchingFunction:", a, b, c) print("a or b", a or b) print("a and b", a and b) print("not a", not a) print("not b", not b) print("Simple branch with both branches") if a: l = "YES" else: l = "NO" print(a, "->", l) print("Simple not branch with both branches") if not a: l = "YES" else: l = "NO" print(not a, "->", l) print("Simple branch with a nested branch in else path:") if a: m = "yes" else: if True: m = "no" print(a, "->", m) print("Triple 'and' chain:") v = "NO" if a and b and c: v = "YES" print(a, b, c, "->", v) print("Triple or chain:") k = "NO" if a or b or c: k = "YES" print(a, b, c, "->", k) print("Nested 'if not' chain:") p = "NO" if not a: if not b: p = "YES" print("not a, not b", not a, not b, "->", p) print("or condition in braces:") q = "NO" if (a or b): q = "YES" print("(a or b) ->", q) print("Braced if not with two 'or'") if not (a or b or c): q = "YES" else: q = "NO" print("not (a or b or c)", q) print("Braced if not with one 'or'") q = "NO" if not (b or b): q = "YES" print("not (b or b)", q) print("Expression a or b", a or b) print("Expression not(a or b)", not(a or b)) print("Expression a and (b+5)", a and (b+5)) print("Expression (b if b else 2)", (b if b else 2)) print("Expression (a and (b if b else 2))", (a and (b if b else 2))) print("Braced if not chain with 'and' and conditional expression:") if not (a and (b if b else 2)): print("oki") print("Nested if chain with outer else:") d=1 if a: if b or c: if d: print("inside nest") else: print("outer else") print("Complex conditional expression:") v = (3 if a-1 else 0) or \ (b or (c*2 if c else 6) if b-1 else a and b and c) print(v) if True: print("Predictable branch taken") branchingFunction(1,0,3) x = 3 def optimizationVictim(): if x: pass else: pass if x: pass pass optimizationVictim() def dontOptimizeSideEffects(): print("Lets see, if conditional expression in known true values are correctly handled:") def returnTrue(): print("function 'returnTrue' was called as expected") return True def returnFalse(): print("function 'returnFalse' should not have beeen called") return False if (returnTrue() or returnFalse(),): print("Taken branch as expected.") else: print("Bad branch taken.") dontOptimizeSideEffects() def dontOptimizeTruthCheck(): class A: def __nonzero__(self): raise ValueError __bool__ = __nonzero__ a = A() if a: pass try: print("Check that branch conditions are not optimized way: ", end = "") dontOptimizeTruthCheck() print("FAIL.") except ValueError: print("OK.") Nuitka-0.5.18.1/tests/basics/Printing_2.py0000644000175000017500000000263312651071040020424 0ustar hayenhayen00000000000000# -*- coding: utf-8 -*- # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # All of these should be identical with correct software behavior. print "Output with newline." print "Output", "with", "newline." print "Output trailing spaces ", "with ", "newline." print "Output ", print "with ", print "newline." print "Output\twith tab" print "Output\t", print "with tab" # These ones gave errors with previous literal bugs: print "changed 2" print "foo%sbar%sfred%sbob?????" a = "partial print" # b doesn't exist try: print a, undefined_global # @UndefinedVariable except Exception, e: print "then occurred", repr(e) print "No newline at the end", x = 1 print """ New line is no soft space, is it """, x Nuitka-0.5.18.1/tests/basics/Referencing.py0000644000175000017500000003554212651071040020645 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( executeReferenceChecked, # @UnresolvedImport my_print, # @UnresolvedImport ) if not hasattr(sys, "gettotalrefcount"): my_print("Warning, using non-debug Python makes this test ineffective.") sys.gettotalrefcount = lambda : 0 x = 17 # Just a function return a constant. Functions don't become any smaller. Let's # get that right. def simpleFunction1(): return 1 # Do a bit of math with a local variable, assigning to its value and then doing # an overwrite of that, trying that math again. This should cover local access # a bit. def simpleFunction2(): y = 3 * x # @UnusedVariable y = 3 return x*2*y # A local function is being returned. This covers creation of local functions # and their release. No closure variables involved yet. def simpleFunction3(): def contained(): return x return contained # Again, local function being return, but this time with local variable taken # as a closure. We use value from defaulted argument, so it cannot be replaced. def simpleFunction4(a = 1): y = a def contained(): return y return contained # Default argument and building a list as local variables. Also return them, # so they are not optimized away. def simpleFunction5(a = 2): c = 1 f = [a, a + c] return c, f def simpleFunction6(): for _b in range(6): pass for _c in (1, 2, 3, 4, 5, 6): pass def simpleFunction7(b = 1): for _b in range(6): pass def simpleFunction8(): c = [] c.append(x) def simpleFunction9(a = 1*2): if a == a: pass u = None def simpleFunction10(a = 1*2): x = [u for u in range(8)] def simpleFunction11(): f = 1 while f < 8: f += 1 v = None def simpleFunction12(): a = [(u,v) for (u,v) in zip(range(8),range(8))] return a def cond(): return 1 def simpleFunction13(a = 1*2): pass def simpleFunction14p(x): try: simpleFunction14p(1,1) except TypeError as _e: pass try: simpleFunction14p(1,1) except TypeError: pass def simpleFunction14(): simpleFunction14p(3) def simpleFunction15p(x): try: try: x += 1 finally: try: x *= 1 finally: _z = 1 except: pass def simpleFunction15(): simpleFunction15p([1]) def simpleFunction16(): class EmptyClass: pass return EmptyClass def simpleFunction17(): class EmptyObjectClass: pass return EmptyObjectClass() def simpleFunction18(): closured = 1 class NonEmptyClass: def __init__(self, a, b): self.a = a self.b = b inside = closured return NonEmptyClass(133, 135) def simpleFunction19(): lam = lambda l : l+1 return lam(9), lam def simpleFunction20(): try: a = [] a[1] except IndexError as _e: pass def simpleFunction21(): class EmptyBaseClass: def base(self): return 3 class EmptyObjectClass(EmptyBaseClass): pass result = EmptyObjectClass() c = result.base() return result, c def simpleFunction22(): return True is False and False is not None def simpleFunction23(): not 2 def simpleFunction24p(x): pass def simpleFunction24(): simpleFunction24p(x = 3) def simpleFunction25(): class X: f = 1 def inplace_adder(b): X.f += b return inplace_adder(6**8) def simpleFunction26(): class X: f = [5] def inplace_adder(b): X.f += b return inplace_adder([1, 2]) def simpleFunction27(): a = { 'g': 8 } def inplace_adder(b): a[ 'g' ] += b return inplace_adder(3) def simpleFunction28(): a = { 'g': [8], 'h': 2 } def inplace_adder(b): a[ 'g' ] += b return inplace_adder([3, 5]) def simpleFunction29(): return '3' in '7' def simpleFunction30(): def generatorFunction(): yield 1 yield 2 yield 3 def simpleFunction31(): def generatorFunction(): yield 1 yield 2 yield 3 a = [] for y in generatorFunction(): a.append(y) for z in generatorFunction(): a.append(z) def simpleFunction32(): def generatorFunction(): yield 1 gen = generatorFunction() next(gen) def simpleFunction33(): def generatorFunction(): a = 1 yield a a = [] for y in generatorFunction(): a.append(y) def simpleFunction34(): try: raise ValueError except: pass def simpleFunction35(): try: raise ValueError(1,2,3) except: pass def simpleFunction36(): try: raise (TypeError, (3,x,x,x)) except TypeError: pass def simpleFunction37(): l = [1, 2, 3] try: _a, _b = l except ValueError: pass def simpleFunction38(): class Base: pass class Parent(Base): pass def simpleFunction39(): class Parent(object): pass def simpleFunction40(): def myGenerator(): yield 1 myGenerator() def simpleFunction41(): a = b = 2 return a, b def simpleFunction42(): a = b = 2 * x return a, b def simpleFunction43(): class D: pass a = D() a.b = 1 def simpleFunction47(): def reraisy(): def raisingFunction(): raise ValueError(3) def reraiser(): raise try: raisingFunction() except: reraiser() try: reraisy() except: pass def simpleFunction48(): class BlockExceptions: def __enter__(self): pass def __exit__(self, exc, val, tb): return True with BlockExceptions(): raise ValueError() template = "lala %s lala" def simpleFunction49(): c = 3 d = 4 a = x, y = b, e = (c,d) return a, y, b, e b = range(10) def simpleFunction50(): def getF(): def f(): for i in b: yield i return f f = getF() for x in range(2): _r = list(f()) def simpleFunction51(): g = ( x for x in range(9) ) try: g.throw(ValueError, 9) except ValueError as _e: pass def simpleFunction52(): g = ( x for x in range(9) ) try: g.throw(ValueError(9)) except ValueError as _e: pass def simpleFunction53(): g = ( x for x in range(9) ) try: g.send(9) except TypeError as _e: pass def simpleFunction54(): g = ( x for x in range(9) ) next(g) try: g.send(9) except TypeError as _e: pass def simpleFunction55(): g = ( x for x in range(9) ) try: g.close() except ValueError as _e: pass def simpleFunction56(): def f(): f() try: f() except RuntimeError: pass def simpleFunction57(): x = 1 y = 2 def f(a = x, b = y): return a, b f() f(2) f(3,4) def simpleFunction58(): a = 3 b = 5 try: a = a * 2 return a finally: a / b def simpleFunction59(): a = 3 b = 5 try: a = a * 2 return a finally: return a / b class X: def __del__(self): # Super used to reference leak. x = super() raise ValueError(1) def simpleFunction63(): def superUser(): X() try: superUser() except Exception: pass def simpleFunction64(): x = 2 y = 3 # @UnusedVariable z = eval("x * y") return z def simpleFunction65(): import array a = array.array('b', b"") assert a == eval(repr(a), {"array": array.array}) d = { 'x' : 2, 'y' : 3 } z = eval(repr(d), d) return z def simpleFunction66(): import types return type(simpleFunction65) == types.FunctionType def simpleFunction67(): length = 100000 pattern = "1234567890\00\01\02\03\04\05\06" q, r = divmod(length, len(pattern)) teststring = pattern * q + pattern[:r] return teststring def simpleFunction68(): from random import randrange x = randrange(18) def simpleFunction69(): pools = [tuple() ] g = ((len(pool) == 0,) for pool in pools) next(g) def simpleFunction70(): def gen(): try: undefined_yyy # @UndefinedVariable except Exception: pass yield sys.exc_info() try: undefined_xxx # @UndefinedVariable except Exception: return list(gen()) def simpleFunction71(): try: undefined_global # @UndefinedVariable except Exception: try: try: raise finally: undefined_global # @UndefinedVariable except Exception: pass def simpleFunction72(): try: for _i in range(10): try: undefined_global # @UndefinedVariable finally: break except Exception: pass def simpleFunction73(): for _i in range(10): try: undefined_global # @UndefinedVariable finally: return 7 def simpleFunction74(): import os # @Reimport return os def simpleFunction75(): def raising_gen(): try: raise TypeError except TypeError: yield g = raising_gen() next(g) try: g.throw(RuntimeError()) except RuntimeError: pass def simpleFunction76(): class MyException(Exception): def __init__(self, obj): self.obj = obj class MyObj: pass def inner_raising_func(): raise MyException(MyObj()) try: inner_raising_func() except MyException: try: try: raise finally: raise except MyException: pass def simpleFunction77(): class weirdstr(str): def __getitem__(self, index): return weirdstr(str.__getitem__(self, index)) (weirdstr("1234")) # filter(lambda x: x>="33", weirdstr("1234")) def simpleFunction78(): a = "x = 2" exec(a) def simpleFunction79(): "some doc" simpleFunction79.__doc__ = simpleFunction79.__doc__.replace("doc", "dok") simpleFunction79.__doc__ += " and more" + simpleFunction79.__name__ def simpleFunction80(): "some doc" del simpleFunction80.__doc__ def simpleFunction81(): def f(): yield 1 j j = 1 x = list(f()) def simpleFunction82(): def f(): yield 1 j j = 1 x = f.__doc__ def simpleFunction83(): x = list(range(7)) x[2] = 5 j = 3 x += [h*2 for h in range(j)] def simpleFunction84(): x = tuple(range(7)) j = 3 x += tuple([h*2 for h in range(j)]) def simpleFunction85(): x = list(range(7)) x[2] = 3 x *= 2 def simpleFunction86(): x = "something" x += "" def simpleFunction87(): x = 7 x += 2000 def simpleFunction88(): class C: def __iadd__(self, other): return self x = C() x += C() def simpleFunction89(): x = [1,2] x += [3,4] def anyArgs(*args, **kw): pass def simpleFunction90(): some_tuple = ( simpleFunction89, simpleFunction89, simpleFunction89, ) anyArgs(*some_tuple) def simpleFunction91(): some_dict = { 'a' : simpleFunction90, } anyArgs(**some_dict) def simpleFunction92(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(*some_tuple, **some_dict) def simpleFunction93(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(some_tuple, *some_tuple, **some_dict) def simpleFunction94(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(*some_tuple, b = some_dict, **some_dict) def simpleFunction95(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(some_tuple, *some_tuple, b = some_dict, **some_dict) def simpleFunction96(): some_tuple = ( simpleFunction89, ) anyArgs(some_tuple, *some_tuple) # Complex call with dictionary and key arguments only. def simpleFunction97(): some_dict = { 'a' : simpleFunction90, 'd' : simpleFunction91 } anyArgs(b = some_dict, c = 1, **some_dict) def simpleFunction98(): some_tuple = ( simpleFunction89, ) anyArgs(*some_tuple, b = some_tuple) def simpleFunction99(): some_dict = { 'a' : simpleFunction90, } anyArgs(some_dict, **some_dict) def simpleFunction100(): def h(f): def g(): return f return g def f(): pass h(f) def simpleFunction101(): def orMaking(a, b): x = "axa" x += a or b orMaking('x', "") #################################### class SomeClassWithAttributeAccess(object): READING = 1 def use(self): return self.READING def simpleFunction102(): SomeClassWithAttributeAccess().use() SomeClassWithAttributeAccess().use() #################################### def getInt(): return 3 def simpleFunction103(): try: raise getInt() except TypeError: pass #################################### class ClassWithGeneratorMethod: def generator_method(self): yield self def simpleFunction104(): return list(ClassWithGeneratorMethod().generator_method()) #################################### # These need stderr to be wrapped. tests_stderr = (63,) # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.18.1/tests/basics/PrintFuture.py0000644000175000017500000000154012651071040020674 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("hallo welt", end = ',') print("this is the end") Nuitka-0.5.18.1/tests/basics/GlobalStatement.py0000644000175000017500000000720712651071040021500 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys x = 2 def someFunction1(): x = 3 return x def someFunction2(): global x x = 4 return x def someFunction3(): return x def someNestedGlobalUser1(): z = 1 # Nested function that uses a global z doesn't affect the local variable z at all. def setZ(): global z z = 3 setZ() return z def someNestedGlobalUser2(): z = 1 # Nested function that uses a global z doesn't affect the local variable z at # all. This doesn't change if it's done inside an exec block. exec(""" def setZ(): global z z = 3 setZ() """) return z def someNestedGlobalUser3a(): # Nested function that uses a exec variable scope z and a global z, changes z to be # the global one only. We verify that by looking at locals. This means that the global # statement inside the function of exec changes the effect of the z. exec(""" z = 1 def setZ(): global z z = 3 setZ() """) return z, locals().keys() == ["setZ"] def someNestedGlobalUser3b(): # Nested function that uses a exec variable scope z and a global z, changes # z to be the global one only. We verify that by looking at locals. exec(""" z = 1 """) if sys.version_info[0] < 3: return z, locals().keys() == ['z'] else: return locals().keys() == [] def someNestedGlobalUser4(): z = 1 # This one proves that the local variable z is entirely ignored, and that the global z # has the value 2 inside setZ(). exec(""" z = 2 def setZ(): global z z = 3*z setZ() """) return z def someNestedGlobalUser5(): z = 1 # Without a global statement, z affects the local variable z. exec(""" z = 3 """) return z def someNestedGlobalUser6(): # Without a global statement, a local variable z is created. exec(""" z = 7 """) return z print("Function that shadows a global variable with a local variable") print(someFunction1()) print("Function that accesses and changes a global variable declared with a global statement") print(someFunction2()) print("Function that uses a global variable") print(someFunction3()) print("Functions that uses a global variable in a nested function in various ways:") print(someNestedGlobalUser1, someNestedGlobalUser1()) del z print(someNestedGlobalUser2, someNestedGlobalUser2()) del z print(someNestedGlobalUser3a, someNestedGlobalUser3a()) del z print(someNestedGlobalUser3b, someNestedGlobalUser3b()) print(someNestedGlobalUser4, (someNestedGlobalUser4(), z)) del z print(someNestedGlobalUser5, someNestedGlobalUser5()) z = 9 print(someNestedGlobalUser6, (someNestedGlobalUser6(), z)) x = 7 def f(): x = 1 def g(): global x def i(): def h(): return x return h() return i() return g() print(f()) global global_already global_already = 1 Nuitka-0.5.18.1/tests/basics/OrderChecks.py0000644000175000017500000003420612651072046020615 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def separator(): print('*' * 80) def dictOrderCheck(): def key1(): print("key1 called") return 1 def key2(): print("key2 called") return 2 def value1(): print("value1 called") return 11 def value2(): print("value2 called") return 22 print("Checking order of calls in dictionary creation from callables:") print( { key1() : value1(), key2() : value2() } ) def listOrderCheck(): def value1(): print("value1 called") return 11 def value2(): print("value2 called") return 22 print([value1(), value2() ]) def sliceOrderCheck(): print("Slices:") d = list(range(10)) def lvalue(): print("lvalue", end = ' ') return d def rvalue(): print("rvalue", end = ' ') return range(2) def rvalue4(): print("rvalue", end = ' ') return range(4) def low(): print("low", end = ' ') return 0 def high(): print("high", end = ' ') return 4 def step(): print("step", end = ' ') return 2 print("Complex slice lookup:", end = ' ') print(lvalue()[ low() : high() : step() ]) print("Complex slice assignment:", end = ' ') lvalue()[ low() : high() : step() ] = rvalue() print(d) print("Complex slice del:", end = ' ') del lvalue()[ low() : high() : step() ] print(d) print("Complex inplace slice operation", end = ' ') # TODO: This gives an error in CPython, but not in Nuitka. # lvalue()[ low() : high() : step() ] += rvalue() print(d) d = list(range(10)) print("Simple slice lookup", end = ' ') print(lvalue()[ low() : high() ]) print("Simple slice assignment", end = ' ') lvalue()[ 3 + low() : 3 + high() ] = rvalue() print(d) print("Simple slice del", end = ' ') del lvalue()[ 3 + low() : 3 + high() ] print(d) print("Simple inplace slice operation", end = ' ') lvalue()[ low() : high() ] += rvalue4() print(d) def subscriptOrderCheck(): print("Subscripts:") d={} def lvalue(): print("lvalue", end = ' ') return d def rvalue(): print("rvalue", end = ' ') return 2 def subscript(): print("subscript", end = ' ') return 1 print("Assigning subscript:") lvalue()[ subscript() ] = rvalue() print(d) print("Lookup subscript:") print(lvalue()[ subscript() ]) print("Deleting subscript:") del lvalue()[ subscript() ] print(d) def attributeOrderCheck(): def lvalue(): print("lvalue", end = ' ') return lvalue def rvalue(): print("rvalue", end = ' ') return 2 print("Attribute assignment order:") lvalue().xxx = rvalue() print("Assigned was indeed:", lvalue.xxx) print("Checking attribute assignment to unassigned value from unassigned:") try: undefined_global_zzz.xxx = undefined_global_yyy # @UndefinedVariable except Exception as e: print("Expected exception caught:", repr(e)) else: assert False def compareOrderCheck(): def lvalue(): print("lvalue", end = ' ') return 1 def rvalue(): print("rvalue", end = ' ') return 2 print("Comparisons:") print("==", lvalue() == rvalue()) print("<=", lvalue() <= rvalue()) print(">=", lvalue() >= rvalue()) print("!=", lvalue() != rvalue()) print('>', lvalue() > rvalue()) print('<', lvalue() < rvalue()) print("Comparison used in bool context:") print("==", "yes" if lvalue() == rvalue() else "no") print("<=", "yes" if lvalue() <= rvalue() else "no") print(">=", "yes" if lvalue() >= rvalue() else "no") print("!=", "yes" if lvalue() != rvalue() else "no") print('>', "yes" if lvalue() > rvalue() else "no") print('<', "yes" if lvalue() < rvalue() else "no") def operatorOrderCheck(): def left(): print("left operand", end = ' ') return 1 def middle(): print("middle operand", end = ' ') return 3 def right(): print("right operand", end = ' ') return 2 print("Operations:") print('+', left() + middle() + right()) print('-', left() - middle() - right()) print('*', left() * middle() * right()) print('/', left() / middle() / right()) print('%', left() % middle() % right()) print("**", left() ** middle() ** right()) def generatorOrderCheck(): print("Generators:") def default1(): print("default1", end = ' ') return 1 def default2(): print("default2", end = ' ') return 2 def default3(): print("default3", end = ' ') return 3 def value(x): print("value", x, end = ' ') return x def generator(a = default1(), b = default2(), c = default3()): print("generator entry") yield value(a) yield value(b) yield value(c) print("generator exit") result = list(generator()) print("Result", result) def classOrderCheck(): print("Checking order of class constructions:") class B1: pass class B2: pass def base1(): print("base1", end = ' ') return B1 def base2(): print("base2", end = ' ') return B2 def deco1(cls): print("deco1", end = ' ') return cls def deco2(cls): print("deco2") return B2 @deco2 @deco1 class X(base1(), base2()): print("class body", end = ' ') print def inOrderCheck(): print("Checking order of in operator:") def container(): print("container", end = ' ') return [3] def searched(): print("searched", end = ' ') return 3 print("in:", searched() in container()) print("not in:", searched() not in container()) def unpackOrderCheck(): print("Unpacking values:") class Iterable: def __init__(self): self.consumed = 2 def __iter__(self): return Iterable() def __del__(self): print("Deleted with", self.consumed) def next(self): print("Next with", self.consumed) if self.consumed: self.consumed -=1 else: raise StopIteration return self.consumed __next__ = next iterable = Iterable() try: x, y = a, b = iterable except Exception as e: print("Caught", repr(e)) return a, b, x, y def superOrderCheck(): print("Built-in super:") try: super(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught super 2", repr(e)) def isinstanceOrderCheck(): print("Built-in isinstance:") try: isinstance(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught isinstance 2", repr(e)) def rangeOrderCheck(): print("Built-in range:") try: range(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught range 3", repr(e)) try: range(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught range 2", repr(e)) def importOrderCheck(): print("Built-in __import__:") def name(): print("name", end = ' ') def globals(): # @ReservedAssignment print("globals", end = ' ') def locals(): # @ReservedAssignment print("locals", end = ' ') def fromlist(): print("fromlist", end = ' ') def level(): print("level") try: __import__(name(), globals(), locals(), fromlist(), level()) except Exception as e: print("Expected exception caught __import__ 5", repr(e)) def hasattrOrderCheck(): print("Built-in hasattr:") try: hasattr(zzz, yyy) # @UndefinedVariable except Exception as e: print("Expected exception caught hasattr", repr(e)) def getattrOrderCheck(): print("Built-in getattr:") try: getattr(zzz, yyy) # @UndefinedVariable except Exception as e: print("Expected exception caught getattr 2", repr(e)) try: getattr(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught getattr 3", repr(e)) def default(): print("default used") print("Default side effects:", end = ' ') print(getattr(1, "real", default())) def typeOrderCheck(): print("Built-in type:") try: type(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught type 3", repr(e)) def iterOrderCheck(): print("Built-in iter:") try: iter(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught iter 2", repr(e)) def openOrderCheck(): print("Built-in open:") try: open(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught open 3", repr(e)) def unicodeOrderCheck(): print("Built-in unicode:") try: unicode(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught unicode", repr(e)) def longOrderCheck(): print("Built-in long:") try: long(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught long 2", repr(e)) def intOrderCheck(): print("Built-in int:") try: int(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught int", repr(e)) def nextOrderCheck(): print("Built-in next:") try: next(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught next 2", repr(e)) def callOrderCheck(): print("Checking nested call arguments:") class A: def __del__(self): print("Doing del inner object") def check(obj): print("Outer function") def p(obj): print("Inner function") check(p(A())) def boolOrderCheck(): print("Checking order of or/and arguments:") class A(int): def __init__(self, value): self.value = value def __del__(self): print("Doing del of %s" % self) def __bool__(self): print("Test of %s" % self) return self.value != 0 __nonzero__ = __bool__ def __str__(self): return "<%s %r>" % (self.__class__.__name__, self.value) class B(A): pass class C(A): pass print("Two arguments, A or B:") for a in range(2): for b in range(2): print("Case %d or %d" % (a,b)) r = A(a) or B(b) print(r) del r # TODO: The order of deletion does not exactly match, which we accept for # now. if True: print("Three arguments, A or B or C:") for a in range(2): for b in range(2): for c in range(2): print("Case %d or %d or %d" % (a,b,c)) r = A(a) or B(b) or C(c) print(r) del r print("Two arguments, A and B:") for a in range(2): for b in range(2): print("Case %d and %d" % (a,b)) r = A(a) and B(b) print(r) del r # See above if True: print("Three arguments, A and B and C:") for a in range(2): for b in range(2): for c in range(2): print("Case %d and %d and %d" % (a,b,c)) r = A(a) and B(b) and C(c) print(r) del r def comparisonChainOrderCheck(): print("Checking order of comparison chains:") class A(int): def __init__(self, value): self.value = value def __del__(self): print("Doing del of %s" % self) def __le__(self, other): print("Test of %s <= %s" % (self, other)) return self.value <= other.value def __str__(self): return "<%s %r>" % (self.__class__.__name__, self.value) class B(A): pass class C(A): pass # See above, not really doing it right. if False: print("Three arguments, A <= B <= C:") for a in range(3): for b in range(3): for c in range(3): print("Case %d <= %d <= %d" % (a,b,c)) r = A(a) <= B(b) <= C(c) print(r) del r dictOrderCheck() separator() listOrderCheck() separator() subscriptOrderCheck() separator() attributeOrderCheck() separator() operatorOrderCheck() separator() compareOrderCheck() separator() sliceOrderCheck() separator() generatorOrderCheck() separator() classOrderCheck() separator() inOrderCheck() separator() unpackOrderCheck() separator() superOrderCheck() separator() isinstanceOrderCheck() separator() rangeOrderCheck() separator() importOrderCheck() separator() hasattrOrderCheck() separator() getattrOrderCheck() separator() typeOrderCheck() separator() iterOrderCheck() separator() openOrderCheck() separator() unicodeOrderCheck() separator() nextOrderCheck() separator() longOrderCheck() separator() intOrderCheck() separator() callOrderCheck() separator() boolOrderCheck() separator() comparisonChainOrderCheck() Nuitka-0.5.18.1/tests/basics/ListContractions.py0000644000175000017500000000511512651071040021711 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def displayDict(d): result = '{' for key, value in sorted(d.items()): result += "%s: %s" % (key, value) result += '}' print("List contraction on the module level:") x = [(u if u%2==0 else 0) for u in range(10)] print(x) print("List contraction on the function level:") def someFunction(): x = [(u if u%2==0 else 0) for u in range(10)] print(x) someFunction() print("List contractions with no, 1 one 2 conditions:") def otherFunction(): print([ x for x in range(8) ]) print([ x for x in range(8) if x % 2 == 1 ]) print([ x for x in range(8) if x % 2 == 1 if x > 4 ]) otherFunction() print("Complex list contractions with more than one for:") def complexContractions(): print([ (x,y) for x in range(3) for y in range(5) ]) seq = range(3) res = [(i, j, k) for i in iter(seq) for j in iter(seq) for k in iter(seq)] print(res) complexContractions() print("Contraction for 2 for statements and one final if referring to first for:") def trickyContraction(): class Range: def __init__(self, value): self.value = value def __iter__(self): print("Giving range iter to", self.value) return iter(range(self.value)) def Cond(y): print("Checking against", y) return y == 1 r = [ (x,z,y) for x in Range(3) for z in Range(2) for y in Range(4) if Cond(y) ] print("result is", r) trickyContraction() def lambdaWithcontraction(x): l = lambda x : [ z for z in range(x) ] r = l(x) print("Lambda contraction locals:", displayDict(locals())) lambdaWithcontraction(3) print("Contraction that gets a 'del' on the iterator variable:", end = ' ') def allowedDelOnIteratorVariable(z): x = 2 del x return [ x*z for x in range(z) ] print(allowedDelOnIteratorVariable(3)) Nuitka-0.5.18.1/tests/basics/Inspection.py0000644000175000017500000000757712651071040020540 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import inspect, types, sys, pprint def compiledFunction(a, b): pass assert inspect.isfunction(compiledFunction) is True assert isinstance(compiledFunction, types.FunctionType) assert isinstance(compiledFunction, (int, types.FunctionType)) print("Compiled spec:", inspect.getargspec(compiledFunction)) print("Compiled args:", inspect.formatargspec(*inspect.getargspec(compiledFunction))) # Even this works. assert type(compiledFunction) == types.FunctionType class CompiledClass: def __init__(self): pass def compiledMethod(self): pass assert inspect.isfunction(CompiledClass) is False assert isinstance(CompiledClass, types.FunctionType) is False assert inspect.ismethod(compiledFunction) is False assert inspect.ismethod(CompiledClass) is False assert inspect.ismethod(CompiledClass.compiledMethod) == ( sys.version_info < (3,) ) assert inspect.ismethod(CompiledClass().compiledMethod) is True assert bool(type(CompiledClass.compiledMethod) == types.MethodType) == ( sys.version_info < (3,) ) print("Compiled method:", inspect.getargspec(CompiledClass().compiledMethod)) print("Compiled class:", inspect.formatargspec(*inspect.getargspec(CompiledClass().compiledMethod))) def compiledGenerator(): yield 1 assert inspect.isfunction(compiledGenerator) is True assert inspect.isgeneratorfunction(compiledGenerator) is True assert isinstance(compiledGenerator(), types.GeneratorType) is True assert type(compiledGenerator()) == types.GeneratorType assert isinstance(compiledGenerator, types.GeneratorType) is False assert inspect.ismethod(compiledGenerator()) is False assert inspect.isfunction(compiledGenerator()) is False assert inspect.isgenerator(compiledFunction) is False assert inspect.isgenerator(compiledGenerator) is False assert inspect.isgenerator(compiledGenerator()) is True def someFunction(a): assert inspect.isframe(sys._getframe()) # print("Running frame getframeinfo()", inspect.getframeinfo(sys._getframe())) # TODO: The locals of the frame are not updated. # print("Running frame arg values", inspect.getargvalues(sys._getframe())) someFunction(2) class C: print("Class locals", str(sys._getframe().f_locals).replace(", '__locals__': {...}", "").replace("'__qualname__': 'C', ", "")) print("Class flags", sys._getframe().f_code.co_flags) def f(): print("Func locals", sys._getframe().f_locals) print("Func flags", sys._getframe().f_code.co_flags) f() def g(): yield("Generator object locals", sys._getframe().f_locals) yield("Generator object flags", sys._getframe().f_code.co_flags) for line in g(): print(*line) print("Generator function flags", g.__code__.co_flags) def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" return pprint.pformat(d) print("Module frame locals", displayDict(sys._getframe().f_locals)) print("Module flags", sys._getframe().f_code.co_flags) print("Module code name", sys._getframe().f_code.co_name) print("Module frame dir", dir(sys._getframe())) Nuitka-0.5.18.1/tests/basics/Recursion.py0000644000175000017500000000155512651071040020364 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # count = 0 def recurse(): global count count += 1 if count < 50: recurse() recurse() Nuitka-0.5.18.1/tests/basics/ParameterErrors32.py0000644000175000017500000000361312651071040021672 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def kwfunc(a, *, k): pass print( "Call function with mixed arguments with too wrong keyword argument." ) try: kwfunc( k = 3, b = 5 ) except TypeError as e: print( repr(e) ) print( "Call function with mixed arguments with too little positional arguments." ) try: kwfunc( k = 3 ) except TypeError as e: print( repr(e) ) print( "Call function with mixed arguments with too little positional arguments." ) try: kwfunc( 3 ) except TypeError as e: print( repr(e) ) print( "Call function with mixed arguments with too many positional arguments." ) try: kwfunc( 1,2,k=3 ) except TypeError as e: print( repr(e) ) def kwfuncdefaulted(a, b = None, *, c = None): pass print( "Call function with mixed arguments and defaults but too many positional arguments." ) try: kwfuncdefaulted(1, 2, 3) except TypeError as e: print( repr(e) ) def kwfunc2(a, *, k, l, m): pass print( "Call function with mixed arguments with too little positional and keyword-only arguments." ) try: kwfunc2( 1, l = 2 ) except TypeError as e: print( repr(e) ) try: kwfunc2( 1 ) except TypeError as e: print( repr(e) ) Nuitka-0.5.18.1/tests/basics/MainPrograms.py0000644000175000017500000000266212651071040021012 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("Module name is", __name__) class SomeClass: pass print("Class inside main module names its module as", repr(SomeClass.__module__)) if __name__ == "__main__": print("Executed as __main__:") import sys, os # The sys.argv[0] might contain ".exe", ".py" or no suffix at all. # Remove it, so the "diff" output is more acceptable. args = sys.argv[:] args[0] = os.path.basename(args[0]).replace(".exe", ".py").replace(".py", "") print("Arguments were (stripped argv[0] suffix):", repr(args)) # Output the flags, so we can test if we are compatible with these too. print("The sys.flags are:", sys.flags) Nuitka-0.5.18.1/tests/basics/Asserts.py0000644000175000017500000000322312651071040020031 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def testAssert1(): assert False return 1 def testAssert2(): assert True return 1 def testAssert3(): assert False, "argument" return 1 try: print("Function that will assert.") testAssert1() print("No exception.") except Exception as e: print("Raised", type(e), e) try: print("Function that will not assert.") testAssert2() print("No exception.") except Exception as e: print("Raised", type(e), e) try: print("Function that will assert with argument.") testAssert3() print("No exception.") except Exception as e: print("Raised", type(e), e) try: print("Assertion with tuple argument.", end = "") assert False, (3,) except AssertionError as e: print(str(e)) try: print("Assertion with plain argument.", end = "") assert False, 3 except AssertionError as e: print(str(e)) Nuitka-0.5.18.1/tests/basics/DefaultParameters.py0000644000175000017500000000416412651071040022022 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function module_level = 1 def defaultValueTest1(no_default, some_default_constant = 1): return some_default_constant def defaultValueTest2(no_default, some_default_computed = module_level*2): local_var = no_default return local_var, some_default_computed def defaultValueTest3(no_default, func_defaulted = defaultValueTest1(module_level)): return [ func_defaulted for _i in range(8) ] def defaultValueTest4(no_default, funced_defaulted = lambda x: x**2): c = 1 d = 1 return ( i+c+d for i in range(8) ) def defaultValueTest5(no_default, tuple_defaulted = (1,2,3)): return tuple_defaulted def defaultValueTest6(no_default, list_defaulted = [1,2,3]): list_defaulted.append(5) return list_defaulted print(defaultValueTest1("ignored")) # The change of the default variable doesn't influence the default # parameter of defaultValueTest2, that means it's also calculated # at the time the function is defined. module_level = 7 print(defaultValueTest2("also ignored")) print(defaultValueTest3("nono not again")) print(list(defaultValueTest4("unused"))) print(defaultValueTest5("unused")) print(defaultValueTest6("unused"), end = "") print(defaultValueTest6("unused")) print(defaultValueTest6.__defaults__) defaultValueTest6.func_defaults = ([1,2,3],) print(defaultValueTest6.__defaults__) print(defaultValueTest6(1)) Nuitka-0.5.18.1/tests/basics/TryExceptContinue.py0000644000175000017500000000425712651071040022051 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def tryWhileExceptContinueTest(): print("Check if continue is executed in a except handler using for loop:") global undefined x = 0 while x < 10: x += 1 try: if x % 2 == 1: undefined except: print(x, end = ' ') continue print('-', end = ' ') print() def tryForExceptContinueTest(): print("Check if continue is executed in a except handler using for loop:") for x in range(10): try: if x % 2 == 1: undefined except: print(x, end = ' ') continue print('-', end = ' ') print() def tryWhileExceptBreakTest(): print("Check if break is executed in a except handler using while loop:") x = 0 while x < 10: x += 1 try: if x == 5: undefined except: print(x, end = ' ') break print('-', end = ' ') print() def tryForExceptBreakTest(): print("Check if break is executed in a except handler using for loop:") for x in range(10): try: if x == 5: undefined except: print(x, end = ' ') break print('-', end = ' ') print() tryWhileExceptContinueTest() tryWhileExceptBreakTest() tryForExceptContinueTest() tryForExceptBreakTest() Nuitka-0.5.18.1/tests/basics/FunctionObjects.py0000644000175000017500000000310712651071040021505 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def func(arg1, arg2, arg3, **star): """ Some documentation. """ pass print("Starting out: func, __name__:", func, func.__name__) print("Changing its name:") func.__name__ = "renamed" print("With new name: func, __name__:", func, func.__name__) print("Documentation initially:", func.__doc__) print("Changing its doc:") func.__doc__ = "changed doc" + chr(0) + " with 0 character" print("Documentation updated:", repr(func.__doc__)) print("Setting its dict") func.my_value = "attached value" print("Reading its dict", func.my_value) print("__code__ dir") print(dir(func.__code__)) def func2(arg1, arg2 = "default_arg2", arg3 = "default_arg3"): x = arg2 + arg3 return x print("func __defaults__", func2.__defaults__) print("function varnames", func2.__code__.co_varnames) Nuitka-0.5.18.1/tests/basics/Unpacking35.py0000644000175000017500000000350412651071040020476 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def tupleUnpacking(): return (*a,b,*c) def listUnpacking(): return [*a,b,*c] def setUnpacking(): return {*a,b,*c} def dictUnpacking(): return {"a" : 1, **d} a = range(3) b = 5 c = range(8,10) d = {"a" : 2} print("Tuple unpacked", tupleUnpacking()) print("List unpacked", listUnpacking()) print("Set unpacked", setUnpacking()) print("Dict unpacked", dictUnpacking()) non_iterable = 2.0 def tupleUnpackingError(): try: return (*a,*non_iterable,*c) except Exception as e: return e def listUnpackingError(): try: return [*a,*non_iterable,*c] except Exception as e: return e def setUnpackingError(): try: return {*a,*non_iterable,*c} except Exception as e: return e def dictUnpackingError(): try: return {"a" : 1, **non_iterable} except Exception as e: return e print("Tuple unpacked error:", tupleUnpackingError()) print("List unpacked error:", listUnpackingError()) print("Set unpacked error:", setUnpackingError()) print("Dict unpacked error:", dictUnpackingError()) Nuitka-0.5.18.1/tests/basics/run_all.py0000755000175000017500000001004312651071040020042 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, decideFilenameVersionSkip, compareWithCPython, hasDebugPython, withPythonPathChange, createSearchMode ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() if python_version >= "3.4": # These tests don't work with 3.4 yet, and the list is considered the major # TODO for 3.4 support. search_mode.mayFailFor( # Prepared dictionaries of "enum.Enums" are not used early enough "Classes34.py", ) # Create large constants test on the fly, if it's not there, not going to # add it to release archives for no good reason. if not os.path.exists("BigConstants.py"): with open("BigConstants.py", 'w') as output: output.write( "# Automatically generated test, not part of releases or git.\n\n" ) output.write( "print('%s')\n" % ("1234" * 17000) ) # Now run all the tests in this directory. for filename in sorted(os.listdir('.')): if not filename.endswith(".py"): continue if not decideFilenameVersionSkip(filename): continue extra_flags = [ # No error exits normally, unless we break tests, and that we would # like to know. "expect_success", # Keep no temporary files. "remove_output", # Include imported files, mostly "test_common" module. "recurse_all", # Use the original __file__ value, at least one case warns about things # with filename included. "original_file" ] # This test should be run with the debug Python, and makes outputs to # standard error that might be ignored. if filename.startswith("Referencing"): extra_flags.append("python_debug") # This tests warns about __import__() used. if filename == "OrderChecks.py": extra_flags.append("ignore_warnings") # This tests warns about an package relative import despite # being in no package. if filename == "Importing.py": extra_flags.append("ignore_warnings") # TODO: Nuitka does not give output for ignored exception in dtor, this is # not fully compatible and potentially an error. if filename == "YieldFrom33.py": extra_flags.append("ignore_stderr") active = search_mode.consider( dirname = None, filename = filename ) if active: if filename.startswith("Referencing") and not hasDebugPython(): my_print("Skipped (no debug Python)") continue needs_2to3 = python_version.startswith('3') and \ not filename.endswith("32.py") and \ not filename.endswith("33.py") with withPythonPathChange(".."): compareWithCPython( dirname = None, filename = filename, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = needs_2to3 ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.18.1/tests/basics/Operators.py0000644000175000017500000000371112651071040020365 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # a = 3 b = 7 c = [7, 8] d = 15 print '+', a + b print '-', a - b print '*', a * b print '/', a / b print "//", a // b print '%', b % a print "& (2)", a & b print "| (2)", a | b print "& (3)", a & b & d print "| (3)", a | b | d print "^ (2)", a ^ b print "^ (3)", a ^ b ^ d print "**", a ** b print "<<", a << b print ">>", b >> a print "in", b in c print "not in", b not in c print '<', a < b print '>', a > b print "==", a == b print "<=", a <= b print ">=", a >= b print "!=", a != b print "is", a is b print "is not", a is not b print '~', ~ b print '-', - b print '+', + b l = {('a', 'c') : "a,c", 'b' : 2, 'c' : 3, 'd' : 4 } l[ 'l', ] = '6' print "Extended slicing:" print "Should be a,c:", l[ 'a', 'c'] print "Short form of extended slicing:" d = {} # d[1] = 1 d[1,] = 2 d[1,2] = 3 d[1,2,3] = 4 L = list(d) L.sort() print L s = "Some information" ss = s[-1] print "Constant subscript of string", ss print "Repr" print `L`, `ss` print `0L` print repr(L), repr(ss) print repr(3L) print "Slicing on a list:" l = [1, 3, 5, 7, 11, 13, 17] print l[None:None] n = None print l[n:n] print l[3:n] print l[n:3] value = None try: x = value[1] except Exception as e: print "Indexing None gives", repr(e) Nuitka-0.5.18.1/tests/basics/ExceptionRaising32.py0000644000175000017500000000167712651071040022040 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def raisy(): raise ValueError() from None try: print("Raising exception in a function 'from None':") raisy() except (ValueError,TypeError) as e: print("Caught as", repr(e)) Nuitka-0.5.18.1/tests/basics/Slots.py0000644000175000017500000000311412651071040017510 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class W1(object): def __init__(self): self.__hidden = 5 class W2(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 class _W1(object): def __init__(self): self.__hidden = 5 class _W2(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 class a_W1(object): def __init__(self): self.__hidden = 5 class a_W2(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 class W1_(object): def __init__(self): self.__hidden = 5 class W2_(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 for w in (W1, W2, _W1, _W2, a_W1, a_W2, W1_, W2_): try: print(w) print(dir(w)) a = w() except AttributeError: print( "bug in %s" % w ) Nuitka-0.5.18.1/tests/basics/ParameterErrors.py0000644000175000017500000001274412651071040021532 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def functionNoParameters(): pass print("Call a function with no parameters with a plain argument:") try: functionNoParameters(1) except TypeError as e: print(repr(e)) print("Call a function with no parameters with a keyword argument:") try: functionNoParameters(z = 1) except TypeError as e: print(repr(e)) def functionOneParameter(a): print(a) print("Call a function with one parameter with two plain arguments:") try: functionOneParameter(1, 1) except TypeError as e: print(repr(e)) print("Call a function with one parameter too many, and duplicate arguments:") try: functionOneParameter(6, a = 4, *(1, 2, 3)) except TypeError as e: print(repr(e)) print("Call a function with two parameters with three plain arguments:") def functionTwoParameters(a, b): print(a, b) try: functionTwoParameters(1, 2, 3) except TypeError as e: print(repr(e)) print("Call a function with two parameters with one plain argument:") try: functionTwoParameters(1) except TypeError as e: print(repr(e)) print("Call a function with two parameters with three plain arguments:") try: functionTwoParameters(1, 2, 3) except TypeError as e: print(repr(e)) print("Call a function with two parameters with one keyword argument:") try: functionTwoParameters(a = 1) except TypeError as e: print(repr(e)) print("Call a function with two parameters with three keyword arguments:") try: functionTwoParameters(a = 1, b = 2, c = 3) except TypeError as e: print(repr(e)) class MethodContainer: def methodNoParameters(self): pass def methodOneParameter(self, a): print(a) def methodTwoParameters(self, a, b): print(a, b) obj = MethodContainer() print("Call a method with no parameters with a plain argument:") try: obj.methodNoParameters(1) except TypeError as e: print(repr(e)) print("Call a method with no parameters with a keyword argument:") try: obj.methodNoParameters(z = 1) except TypeError as e: print(repr(e)) print("Call a method with one parameter with two plain arguments:") try: obj.methodOneParameter(1, 1) except TypeError as e: print(repr(e)) print("Call a method with two parameters with three plain arguments:") try: obj.methodTwoParameters(1, 2, 3) except TypeError as e: print(repr(e)) print("Call a method with two parameters with one plain argument:") try: obj.methodTwoParameters(1) except TypeError as e: print(repr(e)) print("Call a method with two parameters with one keyword argument:") try: obj.methodTwoParameters(a = 1) except TypeError as e: print(repr(e)) print("Call a method with two parameters with three keyword arguments:") try: obj.methodTwoParameters(a = 1, b = 2, c = 3) except TypeError as e: print(repr(e)) def functionPosBothStarArgs(a, b, c, *l, **d): print(a, b, c, l, d) l = [2] d = { "other" : 7 } print("Call a function with both star arguments and too little arguments:") try: functionPosBothStarArgs(1, *l, **d) except TypeError as e: print(repr(e)) print("Call a function with defaults with too little arguments:") def functionWithDefaults(a, b, c, d = 3): print(a, b, c, d) try: functionWithDefaults(1) except TypeError as e: print(repr(e)) print("Call a function with defaults with too many arguments:") try: functionWithDefaults(1) except TypeError as e: print(repr(e)) print("Complex call with invalid star list and star arguments:") try: a = 1 b = 2.0 functionWithDefaults(1,c = 3,*a,**b) except TypeError as e: print(repr(e)) try: a = 1 b = 2.0 functionWithDefaults(1,*a,**b) except TypeError as e: print(repr(e)) try: a = 1 b = 2.0 functionWithDefaults(c = 1, *a,**b) except TypeError as e: print(repr(e)) try: a = 1 b = 2.0 functionWithDefaults(*a,**b) except TypeError as e: print(repr(e)) try: a = 1 functionWithDefaults(*a) except TypeError as e: print(repr(e)) try: a = 1 MethodContainer(*a) except TypeError as e: print(repr(e)) try: a = 1 MethodContainer()(*a) except TypeError as e: print(repr(e)) try: a = 1 MethodContainer.methodTwoParameters(*a) except TypeError as e: print(repr(e)) try: a = 1 None(*a) except TypeError as e: print(repr(e)) try: a = 1 None(**a) except TypeError as e: print(repr(e)) print("Call object with name as both keyword and in star dict argument:") try: a = {'a' : 3} None(a = 2, **a) except TypeError as e: print(repr(e)) print("Call function with only defaulted value given as keyword argument:") def functionwithTwoArgsOneDefaulted(a, b = 5): pass try: functionwithTwoArgsOneDefaulted(b = 12) except TypeError as e: print(repr(e)) Nuitka-0.5.18.1/tests/basics/BuiltinOverload.py0000644000175000017500000000206012651071040021505 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function try: from __builtin__ import len as _len except ImportError: from builtins import len as _len def len(x): # @ReservedAssignment print("Private built-in called with argument", repr(x)) return _len(x) print("Calling built-in len", len(range(9))) Nuitka-0.5.18.1/tests/basics/Lamdas.py0000644000175000017500000000323212651071040017606 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def lamdaContainer(x): f = lambda c : c g = lambda c : c if x else c*c # h = lambda c: 'a' <= c <= 'z' y = f(x) z = g(4) print("Lambda with conditional expression gives", z) if 'a' <= x <= y <= 'z': print("Four") if 'a' <= x <= 'z': print("Yes") if 'a' <= x > 'z': print("Yes1") if 'a' <= ('1' if x else '2') > 'z': print("Yes2") if 'a' <= ('1' if x else '2') > 'z' > undefined_global: # @UndefinedVariable print("Yes3") z = lambda huhu = y : huhu print("Lambda defaulted gives", z()) lamdaContainer('b') def lambdaGenerator(): x = lambda : (yield 3) gen = x() print("Lambda generator gives", next(gen)) lambdaGenerator() def lambdaDirectCall(): args = range(7) x = (lambda *args:args)(*args) print("Lambda direct call gave", x) lambdaDirectCall() Nuitka-0.5.18.1/tests/basics/ModuleAttributes.py0000644000175000017500000000323612651071040021705 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Some module documentation. With newline and stuff.""" from __future__ import print_function import os, sys print("doc:", __doc__) print("filename:", os.path.basename(__file__)) print("builtins:", __builtins__) print("debug", __debug__) print("debug in builtins", __builtins__.__debug__) print("__initializing__", end = ' ') try: print(__initializing__) # @UndefinedVariable except NameError: print("not found") def checkFromFunction(): frame = sys._getframe(1) def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" return repr(d) print("Globals", displayDict(frame.f_globals)) print("Locals", displayDict(frame.f_locals)) print("Is identical", frame.f_locals is frame.f_globals) checkFromFunction() Nuitka-0.5.18.1/tests/basics/ThreadedGenerators.py0000644000175000017500000000220612651071040022157 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # From Issue#146, this has crashed in the past. import threading def some_generator(): yield 1 def run(): for i in range(10000): for j in some_generator(): pass def main(): workers = [threading.Thread(target = run) for i in range(5)] for t in workers: t.start() for t in workers: t.join() if __name__ == "__main__": main() Nuitka-0.5.18.1/tests/basics/Importing.py0000644000175000017500000000460112651071040020356 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def localImporter1(): import os return os def localImporter1a(): import os as my_os_name return my_os_name def localImporter2(): from os import path return path def localImporter2a(): from os import path as renamed return renamed print("Direct module import", localImporter1()) print("Direct module import using rename", localImporter1a()) print("From module import", localImporter2()) print("From module import using rename", localImporter2a()) from os import * print("Star import gave us", path) import os.path as myname print("As import gave", myname) def localImportFailure(): try: from os import path, lala, listdir except Exception as e: print("gives", type(e), repr(e)) try: print(listdir) except UnboundLocalError: print("and listdir was not imported", end = ' ') print("but path was", path) print("From import that fails in the middle", end = ' ') localImportFailure() def nonPackageImportFailure(): try: # Not allowed without being a package, should raise ValueError from . import whatever except Exception as e: print(type(e), repr(e)) print("Package import fails in non-package:", end = ' ') nonPackageImportFailure() def importBuiltinTupleFailure(): try: value = "something", # Not allowed to not be constant string, optimization might be fooled # though. __import__(value) except Exception as e: print(type(e), repr(e)) print("The __import__ built-in optimization can handle tuples:", end = ' ') importBuiltinTupleFailure() Nuitka-0.5.18.1/tests/basics/Unicode.py0000644000175000017500000000210312651071040017767 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(u"gfcrk") print(repr(u"g\xfcrk")) print(r"""\x00""") print("\ttest\n") print(""" something with new lines""") print(u"favicon.ico (32\xd732)") # TODO: Python3 has a problem here, hard to find, disabled for now. if False: encoding = "utf-16-be" print("[\uDC80]".encode(encoding)) print("[\\udc80]") Nuitka-0.5.18.1/tests/basics/DoubleDeletions.py0000644000175000017500000000210012651071040021457 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function a = 3 del a try: del a except NameError as e: print("Raised expected exception:", repr(e)) def someFunction(b, c): b = 1 del b try: del b except UnboundLocalError as e: print("Raised expected exception:", repr(e)) someFunction(3, 4) Nuitka-0.5.18.1/tests/basics/TryExceptFrames.py0000644000175000017500000000240712651071040021475 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys class X: def __del__(self): print "X.__del__ occurred" def raising(doit): _x = X() if doit: 1 / 0 # Call it without an exception raising(False) def catcher(): try: raising(True) except ZeroDivisionError: print "Catching" print "Top traceback code is '%s'." % sys.exc_info()[2].tb_frame.f_code.co_name print "Previous frame locals (module) are", sys.exc_info()[2].tb_next.tb_frame.f_locals pass catcher() print "Good bye." Nuitka-0.5.18.1/tests/basics/Functions_2.py0000644000175000017500000000443512651071040020604 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def local_function(a,z = 9): b = `a*a+1` c = (a,b,a**32,a+a) d = long('0') # @UnusedVariable e = int("77") # @UnusedVariable d = long(b) e = long(1+1) return a, b, c, d, e, z print("Call function with many variables calculated and returned", local_function(1,z = 5)) print("Function with nested args:") def nested_args_function((a,b), c): return a, b, c print(nested_args_function((1, 2), 3)) try: nested_args_function((1, 2, 3), 3) except ValueError, e: print("Calling nested with too long tuple gave:", e) try: nested_args_function((1,), 3) except ValueError, e: print("Calling nested with too short tuple gave:", e) def deeply_nested_function(((a,), b, c, (d, (e,f)))): return a, b, c, d, e, f print("Deeply nested function", deeply_nested_function(((1,), 2, 3, (4, (5, 6))))) print("Function with nested args that have defaults:") def default_giver(): class R: def __iter__(self): print("Giving iter") return iter(range(2)) return R() def nested_args_function_with_defaults((a,b) = default_giver(), c = 5): return a, b, c print("Calling it.") print(nested_args_function_with_defaults()) def comp_args1((a, b)): return a,b def comp_args2((a, b) = (3, 4)): return a, b def comp_args3(a, (b, c)): return a, b, c def comp_args4(a = 2, (b, c) = (3, 4)): return a, b, c print("Complex args functions", comp_args1((2, 1)), comp_args2(), comp_args2((7,9)), comp_args3(7, (8,9)), comp_args4()) Nuitka-0.5.18.1/tests/basics/Classes34.py0000644000175000017500000000244212651071040020153 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from enum import Enum print("Enum class with duplicate enumeration values:") try: class Color(Enum): red = 1 green = 2 blue = 3 red = 4 print("not allowed to get here") except Exception as e: print("Occurred", e) print("Class variable that conflicts with closure variable:") def testClassNamespaceOverridesClosure(): # See #17853. x = 42 class X: locals()['x'] = 43 y = x print("should be 43:", X.y) testClassNamespaceOverridesClosure() Nuitka-0.5.18.1/tests/basics/Constants27.py0000644000175000017500000000143212651071040020532 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # x = ( { 1, } ) print x Nuitka-0.5.18.1/tests/basics/MinimalClass.py0000644000175000017500000000154012651071040020761 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Very minimal class example, to be used for debugging. a = 1 class B: b = a print(B.b) Nuitka-0.5.18.1/tests/basics/Assignments32.py0000644000175000017500000001332412651071040021050 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # a, b = 1, 2 # simple sequence assignment print(a, b) a, b = ['green', 'blue'] # list assignment print(a, b) a, b = 'XY' # string assignment print(a, b) a, b = range(1,5,2) # any iterable will do print(a, b) (a,b), c = "XY", "Z" # a = 'X', b = 'Y', c = 'Z' print(a, b, c) try: (a,b), c = "XYZ" # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) try: (a,b), c = "XY" # ERROR -- need more than 1 value to unpack except Exception as e: print(repr(e)) print(a, b, c) (a,b), c, = [1,2],'this' # a = '1', b = '2', c = 'this' print(a, b, c) try: (a,b), (c,) = [1,2],'this' # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) # extended sequence unpacking a, *b = 1,2,3,4,5 # a = 1, b = [2,3,4,5] print(a, b) *a, b = 1,2,3,4,5 # a = [1,2,3,4], b = 5 print(a, b) a, *b, c = 1,2,3,4,5 # a = 1, b = [2,3,4], c = 5 print(a, b) a, *b = 'X' # a = 'X', b = [] print(a, b) *a, b = 'X' # a = [], b = 'X' print(a, b) a, *b, c = "XY" # a = 'X', b = [], c = 'Y' print(a, b) a, *b, c = "X...Y" # a = 'X', b = ['.','.','.'], c = 'Y' print(a, b, c) a, b, *c = 1,2,3 # a = 1, b = 2, c = [3] print(a, b, c) a, b, c, *d = 1,2,3 # a = 1, b = 2, c = 3, d = [] print(a, b, c, d) (a,b), c = [1,2],'this' # a = '1', b = '2', c = 'this' print(a, b, c) (a,b), *c = [1,2],'this' # a = '1', b = '2', c = ['this'] print(a, b, c) (a,b), c, *d = [1,2],'this' # a = '1', b = '2', c = 'this', d = [] print(a, b, c, d) (a,b), *c, d = [1,2],'this' # a = '1', b = '2', c = [], d = 'this' print(a, b, c, d) (a,b), (c, *d) = [1,2],'this' # a = '1', b = '2', c = 't', d = ['h', 'i', 's'] print(a, b, c, d) *a, = (1,2) # a = [1,2] try: *a, = 1 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a) *a, = [1] # a = [1] print(a) *a, = (1,) # a = [1] print(a) try: *a, = (1) # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a) *a, b = [1] # a = [], b = 1 print(a, b) *a, b = (1,) # a = [], b = 1 print(a, b) try: (a,b),c = 1,2,3 # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) try: (a,b), *c = 1,2,3 # ERROR - 'int' object is not iterable except Exception as e: print(repr(e)) print(a, b, c) (a,b), *c = 'XY', 2, 3 # a = 'X', b = 'Y', c = [2,3] print(a, b, c) # extended sequence unpacking -- NESTED try: (a,b),c = 1,2,3 # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) *(a,b), c = 1,2,3 # a = 1, b = 2, c = 3 print(a, b, c) *(a,b), = 1,2 # a = 1, b = 2 print(a, b) *(a,b), = 'XY' # a = 'X', b = 'Y' print(a, b) try: *(a, b), = 'this' # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b) *(a, *b), = 'this' # a = 't', b = ['h', 'i', 's'] print(a, b) *(a, *b), c = 'this' # a = 't', b = ['h', 'i'], c = 's' print(a, b, c) *(a,*b), = 1,2,3,3,4,5,6,7 # a = 1, b = [2, 3, 3, 4, 5, 6, 7] print(a, b) try: *(a,*b), (*c,) = 1,2,3,3,4,5,6,7 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a, b, c) *(a,*b), c = 1,2,3,3,4,5,6,7 # a = 1, b = [2, 3, 3, 4, 5, 6], c = 7 print(a, b, c) *(a,*b), (*c,) = 1,2,3,4,5,'XY' # a = 1, b = [2, 3, 4, 5], c = ['X', 'Y'] print(a, b, c) *(a,*b), c, d = 1,2,3,3,4,5,6,7 # a = 1, b = [2, 3, 3, 4, 5], c = 6, d = 7 try: *(a,*b), (c, d) = 1,2,3,3,4,5,6,7 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a, b, c, d) try: *(a,*b), (*c, d) = 1,2,3,3,4,5,6,7 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a, b, c, d) try: *(a,b), c = 'XY', 3 # ERROR -- need more than 1 value to unpack except Exception as e: print(repr(e)) print(a, b) *(*a,b), c = 'XY', 3 # a = [], b = 'XY', c = 3 print(a, b, c) (a,b), c = 'XY', 3 # a = 'X', b = 'Y', c = 3 print(a, b, c) *(a,b), c = 'XY', 3, 4 # a = 'XY', b = 3, c = 4 print(a, b, c) *(*a,b), c = 'XY', 3, 4 # a = ['XY'], b = 3, c = 4 print(a, b, c) try: (a,b), c = 'XY', 3, 4 # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) Nuitka-0.5.18.1/tests/basics/TryExceptFinally.py0000644000175000017500000000436012651071040021656 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # "Some doc" from __future__ import print_function def one(): return 1 def tryScope1(x): try: try: x += one() finally: print("Finally is executed") try: _z = one() finally: print("Deep Nested finally is executed") except: print("Exception occurred") else: print("No exception occurred") tryScope1(1) print('*' * 20) tryScope1([1]) def tryScope2(x, someExceptionClass): try: x += 1 except someExceptionClass as e: print("Exception class from argument occurred:", someExceptionClass, repr(e)) else: print("No exception occurred") def tryScope3(x): if x: try: x += 1 except TypeError: print("TypeError occurred") else: print("Not taken") print('*' * 20) tryScope2(1, TypeError) tryScope2([1], TypeError) print('*' * 20) tryScope3(1) tryScope3([1]) tryScope3([]) print('*' * 20) def tryScope4(x): try: x += 1 except: print("exception occurred") else: print("no exception occurred") finally: print("finally obeyed") tryScope4(1) tryScope4([1]) def tryScope5(): import sys print("Exception info is initially", sys.exc_info()) try: try: undefined_global += 1 # @UndefinedVariable finally: print("Exception info in 'finally' clause is", sys.exc_info()) except: pass tryScope5() Nuitka-0.5.18.1/tests/basics/Assignments.py0000644000175000017500000001277612651071040020715 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys def someFunction(): a = 2 print("Simple assignment to variable:", a) b = c = 3 print("Assignment to 2 variables", b, c) z = [1, 2, 3] z[2] = z[1] = 5 print("Assignment to list subscripts:", z) d, e = 1, 2 print("Assignment to variable tuple:", d, e) [f, g] = 7, 9 print("Assignment to variable list:", f, g) j = [h, i] = (7, 9) print("Complex Assignment from variabe list:", j, type(j), h, i) a, (b,c) = 1, (2,3) print("Assigment to nested tuples:", a, b, c) v = [1, 2, 3, 4] v[2:3] = (8,9) print("Assignment to list slice", v) def varargsFunction(*args): f1, f2, f3, f4 = args print("Assignment from list", f1, f2, f3, f4) def otherFunction(): class Iterable: def __iter__(self): return iter(range(3)) a, b, c = Iterable() print("Assignments from iterable", a ,b ,c) print("Assignments from too small iterable",end = ' ') try: f, g = 1, except Exception as e: print("gave", type(e), repr(e)) try: print(f) except UnboundLocalError: print("Variable f is untouched") try: print(g) except UnboundLocalError: print("Variable g is untouched") print("Assignments from too large iterable", end = ' ') try: d, j = 1, 2, 3 except Exception as e: print("gave", type(e), repr(e)) try: print(d) except UnboundLocalError: print("Variable d is untouched") try: print(j) except UnboundLocalError: print("Variable j is untouched") class BasicIterClass: def __init__(self, n): self.n = n self.i = 0 def __next__(self): res = self.i if res >= self.n: raise StopIteration self.i = res + 1 return res if sys.version_info[0] < 3: def next(self): return self.__next__() class IteratingSequenceClass: def __init__(self, n): self.n = n def __iter__(self): return BasicIterClass(self.n) try: a, b, c = IteratingSequenceClass(2) except ValueError: print("Exception from iterating over too short class", sys.exc_info()) def anotherFunction(): d = {} print("Assignment to dictionary with comma subscript:", end = "") # d[ "f" ] = 3 d['a', 'b'] = 6 d['c', 'b'] = 9 print(sorted(d.items())) def swapVariables(): print("Strange swap form:") a = 1 b = 2 a, b, a = b, a, b print(a, b) def interuptedUnpack(): a = 1 b = 2 print("Assignment from a too short tuple to multiple targets:", end = ' ') try: s = a, c, d = s except ValueError as e: print("gives ValueError", repr(e)) try: print(c) except UnboundLocalError as e: print("and then nothing is assigned:", repr(e)) else: del d del a, b z = [] try: a, z.unknown, b = 1, 2, 3 except AttributeError: print("Interrupted unpack, leaves value assigned", a) def multiTargetInterrupt(): a = 1 b = 2 print("Multiple, overlapping targets", end = "") d = c, d = a, b # @UnusedVariable print(d, c, end = "") del c del d c, d = d = a, b print(d, c) print("Error during multiple assignments", end = "") del c del d e = 9 z = [] try: c, d = e, z.a = a, b except AttributeError: print("having attribute error", c, d, e) del c del d e = 9 print("Error during multiple assignments", end = "") try: c, d = z.a, e = a, b except AttributeError: print("having attribute error", c, d, e) def optimizeableTargets(): a = [1, 2] a[int(1)] = 3 print("Optimizable slice operation, results in", a) def complexDel(): a = b = c = d = 1 del a, b, (c, d) try: print(c) except UnboundLocalError as e: print("yes, del worked", repr(e)) def sliceDel(): # Python3 ranges are not lists. a = list(range(6)) del a[2:4] print("Del slice operation, results in", a) def globalErrors(): global unassigned_1, unassigned_2 try: unassigned_1 = unassigned_1 except NameError as e: print("Accessing unassigned global gives", repr(e)) try: del unassigned_2 except NameError as e: print("Del on unassigned global gives", repr(e)) someFunction() varargsFunction(1,2,3,4) otherFunction() anotherFunction() swapVariables() interuptedUnpack() multiTargetInterrupt() optimizeableTargets() complexDel() sliceDel() globalErrors() Nuitka-0.5.18.1/tests/basics/OverflowFunctions_2.py0000644000175000017500000000272512651071040022330 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def starImporterFunction(): from sys import * # @UnusedWildImport print "Version", version.split()[0].split('.')[:-1] starImporterFunction() def deepExec(): for_closure = 3 def deeper(): for_closure_as_well = 4 def execFunction(): code = "f=2" # Can fool it to nest exec code in None, None print "Locals now", locals() print "Closure one level up was taken", for_closure_as_well print "Closure two levels up was taken", for_closure print "Globals still work", starImporterFunction print "Added local from code", f # @UndefinedVariable execFunction() deeper() deepExec() Nuitka-0.5.18.1/tests/basics/TryYieldFinally.py0000644000175000017500000000442512651071040021476 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def tryContinueFinallyTest(): for x in range(10): try: if x % 2 == 1: continue finally: yield x yield '-' def tryBreakFinallyTest(): for x in range(10): try: if x == 5: break finally: yield x yield '-' def tryFinallyAfterYield(): try: yield 3 finally: print("Executing finally") def tryReturnFinallyYield(): try: return finally: yield 1 def tryReturnExceptYield(): try: return except StopIteration: print("Caught StopIteration") yield 2 except: yield 1 else: print("No exception") def tryStopIterationExceptYield(): try: raise StopIteration except StopIteration: print("Caught StopIteration") yield 2 except: yield 1 else: print("No exception") print("Check if finally is executed in a continue using for loop:") print(tuple(tryContinueFinallyTest())) print("Check if finally is executed in a break using for loop:") print(tuple(tryBreakFinallyTest())) print("Check what try yield finally something does:") print(tuple(tryFinallyAfterYield())) print("Check if yield is executed in finally after return:") print(tuple(tryReturnFinallyYield())) print("Check if yield is executed in except after return:") print(tuple(tryReturnExceptYield())) print("Check if yield is executed in except after StopIteration:") print(tuple(tryReturnExceptYield())) Nuitka-0.5.18.1/tests/basics/Functions32.py0000644000175000017500000000656312651071040020534 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def displayDict(d): result = '{' for key, value in sorted(d.items()): result += "%s: %s" % (key, value) result += '}' def kwonlysimple(*, a): return a print("Keyword only function case: ", kwonlysimple(a = 3)) def kwonlysimpledefaulted(*, a = 5): return a print("Keyword only function, using default value: ", kwonlysimpledefaulted()) def default1(): print("Called", default1) return 1 def default2(): print("Called", default2) return 2 def default3(): print("Called", default3) return 3 def default4(): print("Called", default4) return 4 def annotation1(): print("Called", annotation1) return "a1" def annotation2(): print("Called", annotation2) return "a2" def annotation3(): print("Called", annotation3) return "a3" def annotation4(): print("Called", annotation4) return "a4" def annotation5(): print("Called", annotation5) return "a5" def annotation6(): print("Called", annotation6) return "a6" def annotation7(): print("Called", annotation7) return "a7" def annotation8(): print("Called", annotation8) return "a8" def annotation9(): print("Called", annotation9) return "a9" print("Defining function with annotations, and defaults as functions for everything:") def kwonlyfunc(x: annotation1(), y: annotation2() = default1(), z: annotation3() = default2(), *, a: annotation4(), b: annotation5() = default3(), c: annotation6() = default4(), d: annotation7(), **kw: annotation8()) -> annotation9(): print(x, y, z, a, b, c, d) print("__kwdefaults__", displayDict(kwonlyfunc.__kwdefaults__)) print("Keyword only function called:") kwonlyfunc( 7, a = 8, d = 12 ) print("OK.") print("Annotations come out as", sorted( kwonlyfunc.__annotations__ )) kwonlyfunc.__annotations__ = {} print("After updating to None it is", kwonlyfunc.__annotations__) kwonlyfunc.__annotations__ = { "k" : 9 } print("After updating to None it is", kwonlyfunc.__annotations__) def kwonlystarfunc(*, a, b, **d): return a, b, d print("kwonlystarfunc", kwonlystarfunc(a = 8, b = 12, k = 9, j = 7)) def deeplyNestedNonLocalWrite(): x = 0 y = 0 def f(): def g(): nonlocal x x = 3 return x return g() return f(), x print("Deeply nested non local writing function", deeplyNestedNonLocalWrite()) def deletingClosureVariable(): try: x = 1 def g(): nonlocal x del x g() g() except Exception as e: return repr(e) print("Using deleted non-local variable:", deletingClosureVariable()) Nuitka-0.5.18.1/tests/basics/Future32.py0000644000175000017500000000160012651071040020021 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import barry_as_FLUFL print( eval("1 <> 2") ) print( eval('"a"<>"b"') ) print( eval("range(7) <> range(7)") ) Nuitka-0.5.18.1/tests/basics/ExceptionRaising.py0000644000175000017500000003426212651071040021667 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys print("Raising an exception type in a function:") def raiseExceptionClass(): raise ValueError try: raiseExceptionClass() except Exception as e: print("Caught exception type", e, repr(e), type(e)) print("Inside handler, sys.exc_info is this", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raising an exception instance in a function:") def raiseExceptionInstance(): raise ValueError("hallo") try: raiseExceptionInstance() except Exception as f: print("Caught exception instance", f, repr(f), type(f)) print("Inside handler, sys.exc_info is this", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raising an exception, then catch it to re-raise it:") def raiseExceptionAndReraise(arg): try: return arg / arg except: raise try: raiseExceptionAndReraise(0) except: print("Catched reraised", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Access an undefined global variable in a function:") def raiseNonGlobalError(): return undefined_value # @UndefinedVariable try: raiseNonGlobalError() except: print("NameError caught", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raise a new style class as an exception, should be rejected:") def raiseIllegalError(): class X(object): pass raise X() try: raiseIllegalError() except TypeError as E: print("New style class exception correctly rejected:", E) except: print(sys.exc_info()) assert False, "Error, new style class exception was not rejected" print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raise an old-style class, version dependent outcome:") class ClassicClassException: pass def raiseCustomError(): raise ClassicClassException() try: try: raiseCustomError() except ClassicClassException: print("Caught classic class exception") except: print("Default catch", sys.exc_info()) assert False, "Error, old style class exception was not caught" except TypeError as e: print("Python3 hates to even try and catch classic classes", e) else: print("Classic exception catching was considered fine.") print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Check lazy exception creation:") def checkExceptionConversion(): try: raise Exception("some string") except Exception as err: print("Catched raised object", err, type(err)) try: raise Exception, "some string" except Exception as err: print("Catched raised type, value pair", err, type(err)) checkExceptionConversion() print('*' * 20) print("Check exc_info scope:") def checkExcInfoScope(): try: raise ValueError except: assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None if sys.version_info[0] < 3: print("Exc_info remains visible after exception handler for Python2") assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None else: print("Exc_info is clear after exception handler for Python3") assert sys.exc_info()[0] is None assert sys.exc_info()[1] is None assert sys.exc_info()[2] is None def subFunction(): print("Entering with exception info", sys.exc_info()) assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None try: print("Trying") except: pass print("After trying something and didn't have an exception, info is", sys.exc_info()) print("Call a function inside the exception handler and check there too.") try: raise KeyError except: assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None subFunction() print("Call it twice and see.") try: raise "me" except: assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None subFunction() subFunction() if sys.version_info[0] < 3: sys.exc_clear() checkExcInfoScope() print('*' * 20) # Check that the sys.exc_info is cleared again, after being set inside the # function checkExcInfoScope, it should now be clear again. assert sys.exc_info()[0] is None, sys.exc_info()[0] assert sys.exc_info()[1] is None assert sys.exc_info()[2] is None print("Check catching subclasses") def checkDerivedCatch(): class A(BaseException): pass class B(A): def __init__(self): pass a = A() b = B() try: raise A, b except B, v: print("Caught B", v) except A, v: print("Didn't catch as B, but as A, Python3 does that", v) else: print("Not caught A class, not allowed to happen.") try: raise B, a except TypeError, e: print("TypeError with pair form for class not taking args:", e) checkDerivedCatch() print('*' * 20) def checkNonCatch1(): print("Testing if the else branch is executed in the optimizable case:") try: 0 except TypeError: print("Should not catch") else: print("Executed else branch correctly") checkNonCatch1() print('*' * 20) def checkNonCatch2(): try: print("Testing if the else branch is executed in the non-optimizable case:") except TypeError: print("Should not catch") else: print("Executed else branch correctly") checkNonCatch2() print('*' * 20) print("Checking raise that with exception arguments that raise error themselves.") def checkRaisingRaise(): def geterror(): return 1/0 try: geterror() except Exception as e: print("Had exception", e) try: raise TypeError, geterror() except Exception as e: print("Had exception", e) try: raise TypeError, 7, geterror() except Exception as e: print("Had exception", e) checkRaisingRaise() print('*' * 20) print("Checking a re-raise that isn't one:") def checkMisRaise(): raise try: checkMisRaise() except Exception as e: print("Without existing exception, re-raise gives:", e) print('*' * 20) print("Raising an exception in an exception handler gives:") def nestedExceptions(a, b): try: a / b except ZeroDivisionError: a / b try: nestedExceptions(1, 0) except Exception as e: print("Nested exception gives", e) print('*' * 20) print("Checking unpacking from an exception as a sequence:") def unpackingCatcher(): try: raise ValueError(1,2) except ValueError as (a,b): print("Unpacking caught exception and unpacked", a, b) unpackingCatcher() print("Afterwards, exception info is", sys.exc_info()) print('*' * 20) print("Testing exception that escapes __del__ and therefore cannot be raised") def devide(a,b): return a / b def unraisableExceptionInDel(): class C: def __del__(self): c = devide(1,0) print(c) def f(): C() f() unraisableExceptionInDel() print('*' * 20) print("Testing exception changes between generator switches:") def yieldExceptionInteraction(): def yield_raise(): print("Yield finds at generator entry", sys.exc_info()[0]) try: raise KeyError("caught") except KeyError: yield sys.exc_info()[0] yield sys.exc_info()[0] yield sys.exc_info()[0] g = yield_raise() print("Initial yield from catch in generator", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("Second yield from the catch reentered", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("After leaving the catch generator yielded", next(g)) yieldExceptionInteraction() print('*' * 20) print("Testing exception change between generator switches while handling an own exception") def yieldExceptionInteraction2(): def yield_raise(): print("Yield finds at generator entry", sys.exc_info()[0]) try: raise ValueError("caught") except ValueError: yield sys.exc_info()[0] yield sys.exc_info()[0] yield sys.exc_info()[0] try: undefined_global # @UndefinedVariable except Exception: print("Checking from outside of generator with", sys.exc_info()[0]) g = yield_raise() v = next(g) print("Initial yield from catch in generator:", v) print("Checking from outside the generator:", sys.exc_info()[0]) print("Second yield from the catch reentered:", next(g)) print("Checking from outside the generation again:", sys.exc_info()[0]) print("After leaving the catch generator yielded:", next(g)) print("After exiting the trying branch:", sys.exc_info()[0]) yieldExceptionInteraction2() print("After function exit, no exception", sys.exc_info()) print('*' * 20) print("Check what happens if a function attempts to clear the exception in a handler") def clearingException(): def clearit(): try: if sys.version_info[0] < 3: sys.exc_clear() except KeyError: pass try: raise KeyError except: print("Before clearing, it's", sys.exc_info()) clearit() print("After clearing, it's", sys.exc_info()) clearingException() print('*' * 20) print("Check that multiple exceptions can be caught in a handler through a variable:") def multiCatchViaTupleVariable(): some_exceptions = (KeyError, ValueError) try: raise KeyError except some_exceptions: print("Yes, indeed.") multiCatchViaTupleVariable() def raiseValueWithValue(): try: raise ValueError(1,2,3), (ValueError(1,2,3)) except Exception as e: print("Gives", e) print("Check exception given when value is raised with value", raiseValueWithValue()) # Make sure the "repr" of exceptions is fine a = IOError print("IOError is represented correctly:", repr(a)) def raising(): raise ValueError def not_raising(): pass def raiseWithFinallyNotCorruptingLineNumber(): try: try: raising() finally: not_raising() except ValueError: print("Traceback is in tried block line", sys.exc_info()[2].tb_lineno) raiseWithFinallyNotCorruptingLineNumber() def wideCatchMustPublishException(): print("At entry, no exception", sys.exc_info()) try: undefined_global # @UndefinedVariable except: print("Inside handler:", sys.exc_info()) pass print("Outside handler:", sys.exc_info()) print("Check that a unqualified catch properly preserves exception") wideCatchMustPublishException() print("Check if a nested exception handler does overwrite re-raised") def checkReraiseAfterNestedTryExcept(): def reraise(): try: raise TypeError("outer") except Exception: try: raise KeyError("nested") except KeyError: print("Current exception inside nested handler", sys.exc_info()) pass print("Current exception after nested handler exited", sys.exc_info()) # Which one does this pick raise try: reraise() except Exception as e: print("Catched", repr(e)) checkReraiseAfterNestedTryExcept() def checkReraiseByFunction(): def reraise(): raise try: try: raise TypeError("outer") except Exception: reraise() except Exception as e: import traceback print("Exception traceback of re-raise:") print('-' * 40) traceback.print_exc() print('-' * 40) print("OK.") # TODO: Enable this, once the actual traceback of a function # re-raise isn't wrong (contains itself) anymore. if False: checkReraiseByFunction() def checkNoRaiseExceptionDictBuilding(arg): a = { () : arg } b = { None : arg } c = { Ellipsis : arg } d = { 1.0j : arg } e = { 1.0 : arg } f = { long(0) : arg } g = { 0 : arg } h = { type : arg } return a, b, c, d, e, f, g, h checkNoRaiseExceptionDictBuilding(1) def checkRaiseExceptionDictBuildingRange(arg): try: i = { range(10) : arg } except Exception as e: print("Raised", repr(e)) else: print("No exception, OK for Python2") return i print("Check if range raises:") checkRaiseExceptionDictBuildingRange(2) def checkRaiseExceptionDictBuildingTuple(arg): try: i = { (2, []) : arg } except Exception as e: print("Raised", repr(e)) else: return i print("Check if mutable tuple raises:") checkRaiseExceptionDictBuildingTuple(3) def checkRaiseExceptionDictBuildingList(arg): try: i = { [2, ()] : arg } except Exception as e: print("Raised", repr(e)) else: return i print("Check if list raises:") checkRaiseExceptionDictBuildingList(4) Nuitka-0.5.18.1/tests/basics/WithStatements.py0000644000175000017500000001140612651071040021372 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys x = 0 # This is used to trace the exact interaction with the context manager to # uncover and decide ordering and correctness of calls. class MyContextManager(object): def __getattribute__(self, attribute_name): print("Asking context manager attribute", attribute_name) return object.__getattribute__(self, attribute_name) def __enter__(self): global x x += 1 print("Entered context manager with counter value", x) return x def __exit__(self, exc_type, exc_value, traceback): print("Context manager exit sees", exc_type, exc_value, traceback) print("Published to context manager exit is", sys.exc_info()) return False print("Use context manager and raise no exception in the body:") with MyContextManager() as x: print("x has become", x) print("Use context manager and raise an exception in the body:") try: with MyContextManager() as x: print("x has become", x) raise Exception("Lalala") print(x) except Exception as e: print("Caught raised exception", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) # Python3 ranges are not lists l = list(range(3)) print("Use context manager and assign to subscription target:") with MyContextManager() as l[0]: print("Complex assignment target works", l[0]) try: with MyContextManager(): sys.exit(9) except BaseException as e: print("Caught base exception", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("Use context manager and fail to assign to attribute:") try: with MyContextManager() as l.wontwork: sys.exit(9) except BaseException as e: print("Caught base exception", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("Use context manager to do nothing inside:") with MyContextManager() as x: pass if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) # Use context manager and fail to assign. def returnFromContextBlock(): # Use context manager to do nothing. with MyContextManager() as x: return 7 if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("Use context manager to return value:") r = returnFromContextBlock() print("Return value", r) class NonContextManager1: def __enter__(self): return self class NonContextManager2: def __exit__(self): return self print("Use incomplete context managers:") try: with NonContextManager1() as x: print(x) except Exception as e: print("Caught for context manager without __exit__", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) try: with NonContextManager2() as x: print(x) except Exception as e: print("Caught for context manager without __enter__", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) class NotAtAllContextManager: pass try: with NotAtAllContextManager() as x: print(x) except Exception as e: print("Caught for context manager without any special methods", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) class MeanContextManager: def __enter__(self): raise ValueError("Nah, I won't play") def __exit__(self): print("Called exit, yes") print("Use mean context manager:") try: with MeanContextManager() as x: print(x) except Exception as e: print("Caught from mean manager", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) class CatchingContextManager(object): def __enter__(self): pass def __exit__(self, exc_type, exc_value, traceback): return True print("Suppressing exception from context manager body:") with CatchingContextManager(): raise ZeroDivisionError if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("OK") Nuitka-0.5.18.1/tests/basics/Referencing33.py0000644000175000017500000000514612651071040021010 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( executeReferenceChecked, my_print, ) if not hasattr(sys, "gettotalrefcount"): my_print("Warning, using non-debug Python makes this test ineffective.") sys.gettotalrefcount = lambda : 0 def simpleFunction1(): def g(): for a in range(20): yield a def h(): yield 4 yield 5 yield 6 def f(): yield from g() yield from h() x = list( f() ) def simpleFunction2(): def g(): for a in range(20): yield a def h(): yield 4 yield 5 yield 6 raise TypeError def f(): yield from g() yield from h() try: x = list( f() ) except TypeError: pass # Broken iterator class. class Broken: def __iter__(self): return self def __next__(self): return 1 def __getattr__(self, attr): 1/0 def simpleFunction3(): def g(): yield from Broken() try: gi = g() next(gi) except Exception: pass def simpleFunction4(): def g(): yield from Broken() try: gi = g() next(gi) gi.throw(AttributeError) except Exception: pass def simpleFunction5(): def g(): yield from (2,3) return list( g() ) # These need stderr to be wrapped. tests_stderr = (3, 4) # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.18.1/tests/basics/TrickAssignments32.py0000644000175000017500000001340312651071040022043 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def someFunctionThatReturnsDeletedValueViaAttributeLookup(): class C: def __getattr__(self, attr_name): nonlocal a del a c = C() a = 1 c.something return a try: someFunctionThatReturnsDeletedValueViaAttributeLookup() except UnboundLocalError: print("OK, object attribute look-up correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaAttributeSetting(): class C: def __setattr__(self, attr_name, value): nonlocal a del a c = C() a = 1 c.something = 1 return a try: someFunctionThatReturnsDeletedValueViaAttributeSetting() except UnboundLocalError: print("OK, object attribute setting correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaAttributeDel(): class C: def __delattr__(self, attr_name): nonlocal a del a return True c = C() a = 1 del c.something return a try: someFunctionThatReturnsDeletedValueViaAttributeDel() except UnboundLocalError: print("OK, object attribute del correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaItemLookup(): class C: def __getitem__(self, attr_name): nonlocal a del a c = C() a = 1 c[2] return a try: someFunctionThatReturnsDeletedValueViaItemLookup() except UnboundLocalError: print("OK, object subscript look-up correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaItemSetting(): class C: def __setitem__(self, attr_name, value): nonlocal a del a c = C() a = 1 c[2] = 3 return a try: someFunctionThatReturnsDeletedValueViaItemSetting() except UnboundLocalError: print("OK, object subscript setting correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaItemDel(): class C: def __delitem__(self, attr_name): nonlocal a del a c = C() a = 1 del c[2] return a try: someFunctionThatReturnsDeletedValueViaItemDel() except UnboundLocalError: print("OK, object subscript del correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaCall(): class C: def __call__(self): nonlocal a del a c = C() a = 1 c() return a try: someFunctionThatReturnsDeletedValueViaCall() except UnboundLocalError: print("OK, object call correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaAdd(): class C: def __add__(self, other): nonlocal a del a c = C() a = 1 c += 1 return a try: someFunctionThatReturnsDeletedValueViaAdd() except UnboundLocalError: print("OK, object add correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaSub(): class C: def __add__(self, other): nonlocal a del a c = C() a = 1 c += 1 return a try: someFunctionThatReturnsDeletedValueViaSub() except UnboundLocalError: print("OK, object sub correctly deleted an item.") else: print("Ouch.!") # TODO: There is a whole lot more operations to cover. def someFunctionThatReturnsDeletedValueViaNot(): class C: def __bool__(self): nonlocal a del a return False c = C() a = 1 not c return a try: someFunctionThatReturnsDeletedValueViaNot() except UnboundLocalError: print("OK, object not correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaRepr(): class C: def __repr__(self): nonlocal a del a return "" c = C() a = 1 repr(c) return a try: someFunctionThatReturnsDeletedValueViaRepr() except UnboundLocalError: print("OK, object repr correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaStr(): class C: def __str__(self): nonlocal a del a return "" c = C() a = 1 str(c) return a try: someFunctionThatReturnsDeletedValueViaStr() except UnboundLocalError: print("OK, object str correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaCompare(): class C: def __lt__(self, other): nonlocal a del a return "" c = C() a = 1 c < None return a try: someFunctionThatReturnsDeletedValueViaCompare() except UnboundLocalError: print("OK, object compare correctly deleted an item.") else: print("Ouch.!") # TODO: The "del" operation may surrect a variable value by "__del__". # TODO: There must be way more than these. Nuitka-0.5.18.1/tests/basics/ExtremeClosure.py0000644000175000017500000000256512651071040021363 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function a = 1 b = 1 def someFunction(): a = a class someClass(): b = b someClass() try: someFunction() except UnboundLocalError as e: print("Expected unbound local error occurred:", repr(e)) try: class anotherClass(): b = undefined_global # @UndefinedVariable except NameError as e: print("Expected name error occurred:", repr(e)) # TODO: This is not passing yet. if False: try: class yetanotherClass(): b = 1 del b print(b) except NameError as e: print("Expected name error occurred:", repr(e)) Nuitka-0.5.18.1/tests/basics/Constants.py0000644000175000017500000001071212651071040020362 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Playing around with constants only. """ from __future__ import print_function try: long except NameError: long = int # @ReservedAssignment def displayDict(d): result = '{' for key, value in sorted(d.items()): result += "%s: %s" % (key, value) result += '}' print("A bunch of constants and their representation:") for value in (0, 3, -4, 17, "hey", (0,), 0.0, -0.0): print(value, ':', repr(value)) print("Comparing constants, optimizable:") print(1 == 0) print("Representation of long constants:") a = long(0) print(repr(long(0)), repr(a) == "0L") print("Identity of empty dictionary constants:") print({} is {}) a = ({}, []) a[0][1] = 2 a[1].append(3) print("Mutable list and dict inside an immutable tuple:") print(a) print("Empty list and dict are hopefully unchanged:") print(({}, [])) def argChanger(a): a[0][1] = 2 a[1].append(3) return a print("Mutable list and dict inside an immutable tuple as arguments:") print(argChanger(({}, []))) print("Empty list and dict are hopefully still unchanged:") print(({}, [])) print("Set constants:") print(set(["foo"])) def mutableConstantChanger(): a = ([1, 2], [3]) print("Start out with value:") print(a) a[1].append(5) print("Changed to value:") print(a) d = { 'l': [], 'm' : [] } print("Start out with value:") print(d) d['l'].append(7) print("Changed to value:") print(d) spec = dict(qual = [], storage = set(), type = [], function = set(), q = 1) spec["type"].insert(0, 2) spec["storage"].add(3) print("Dictionary created from dict built-in.") print(sorted(spec)) mutableConstantChanger() print("Redo constant changes, to catch corruptions:") mutableConstantChanger() def defaultKeepsIdentity(arg = "str_value"): print("Default constant values are still shared if immutable:") print(arg is "str_value") defaultKeepsIdentity() # Dictionary creation from call arguments. def dd(**d): return d def f(): def one(): print("one") def two(): print("two") a = dd(qual = one(), storage = two(), type = [], function = []) print("f mutable", displayDict(a)) a = dd(qual = 1, storage = 2, type = 3, function = 4) print("f immutable", displayDict(a)) # TODO: This exposes a bug in how the star dict argument should populate the # dictionary first instead of last, and the called arguments might have to # come from pairs so hashing does not reorder. # x = { "p" : 7 } # a = dd(qual=[], storage=[], type=[], function=[],**x) # print "f ext mutable", a # x = { "p" : 8 } # a = dd(qual=1, storage=2, type=3, function=4,**x) # print "f ext immutable", a f() # Dictionary creation one after another x={} x["function"] = [] x["type"] = [] x["storage"] = [] x["qual"] = [] print("Manual built dictionary:", x) x={} x["function"] = 1 x["type"] = 2 x["storage"] = 3 x["qual"] = 4 print("Manual built dictionary:", x) # Constants in the code must be created differently. d = { "qual" : [], "storage" : [], "type2" : [], "function" : [] } print("Mutable values dictionary constant:", displayDict(d)) d = { "qual" : 1, "storage" : 2, "type2" : 3, "function" : 4 } print("Immutable values dictionary constant:", displayDict(d)) # Constants that might be difficult min_signed_int = int(-(2**(8*8-1)-1)-1) print("Small int:", min_signed_int, type(min_signed_int)) min_signed_int = int(-(2**(8*4-1)-1)-1) print("Small int", min_signed_int, type(min_signed_int)) # Constants that might be difficult min_signed_long = long(-(2**(8*8-1)-1)-1) print("Small long", min_signed_long, type(min_signed_long)) min_signed_long = long(-(2**(8*4-1)-1)-1) print("Small long", min_signed_long, type(min_signed_long)) Nuitka-0.5.18.1/tests/basics/Referencing35.py0000644000175000017500000000641312651071040021010 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( executeReferenceChecked, my_print, ) if not hasattr(sys, "gettotalrefcount"): my_print("Warning, using non-debug Python makes this test ineffective.") sys.gettotalrefcount = lambda : 0 def run_async(coro): values = [] result = None while True: try: values.append(coro.send(None)) except StopIteration as ex: result = ex.args[0] if ex.args else None break return values, result def raisy(): raise TypeError def simpleFunction1(): async def foo(): return run_async(foo()) #################################### class AsyncIteratorWrapper: def __init__(self, obj): self._it = iter(obj) async def __aiter__(self): return self async def __anext__(self): try: value = next(self._it) except StopIteration: raise StopAsyncIteration return value def simpleFunction3(): async def f(): result = [] async for letter in AsyncIteratorWrapper("abcdefg"): result.append(letter) return result run_async(f()) #################################### def simpleFunction2(): async def foo(): return 7 run_async(foo()) def simpleFunction4(): async def foo(): raise StopIteration try: run_async(foo()) except RuntimeError: pass #################################### class ClassWithAsyncMethod: async def async_method(self): return self def simpleFunction5(): run_async(ClassWithAsyncMethod().async_method()) #################################### class BadAsyncIter: def __init__(self): self.weight = 1 async def __aiter__(self): return self def __anext__(self): return () def simpleFunction7(): async def foo(): async for i in BadAsyncIter(): print('never going to happen') try: run_async(foo()) except TypeError: pass # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.18.1/tests/basics/Classes32.py0000644000175000017500000000565412651071040020161 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Call order of Python3 metaclasses:") def a(): x = 1 class A: print("Class body a.A is evaluating closure x", x) print("Called", a) return A def b(): class B: pass print("Called", b) return B from collections import OrderedDict def displayable(dictionary): return sorted(dictionary.items()) def m(): class M(type): def __new__(cls, class_name, bases, attrs, **over): print("Metaclass M.__new__ cls", cls, "name", class_name, "bases", bases, "dict", displayable(attrs), "extra class defs", displayable(over)) return type.__new__(cls, class_name, bases, attrs) def __init__(self, name, bases, attrs, **over): print("Metaclass M.__init__", name, bases, displayable(attrs), displayable(over)) super().__init__(name, bases, attrs) def __prepare__(metacls, bases, **over): # @NoSelf print("Metaclass M.__prepare__", metacls, bases, displayable(over)) return OrderedDict() print("Called", m) return M def d(): print("Called", d) return 1 def e(): print("Called", e) return 2 class C1(a(), b(), other = d(), metaclass = m(), yet_other = e()): import sys # TODO: Enable this. # print("C1 locals type is", type(sys._getframe().f_locals)) print("OK, class created", C1) print("Attribute C1.__dict__ has type", type(C1.__dict__)) print("Function local classes can be made global and get proper __qualname__:") def someFunctionWithLocalClassesMadeGlobal(): # Affects __qualname__ only in Python3.4 or higher, not in Python3.2 global C class C: pass class D: pass try: print("Nested class qualname is", D.__qualname__) except AttributeError: # Python3.2 pass try: print("Local class made global qualname is", C.__qualname__) except AttributeError: pass someFunctionWithLocalClassesMadeGlobal() print("Function in a class with private name") class someClassWithPrivateArgumentNames: def f(self, *, __kw:1): pass print(someClassWithPrivateArgumentNames.f.__annotations__) print("OK.") Nuitka-0.5.18.1/tests/basics/Functions.py0000644000175000017500000002726412651071040020370 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function var_on_module_level = 1 def closureTest1(some_arg): x = 3 def enclosed(f = "default_value"): return x return enclosed print("Call closured function returning function:", closureTest1(some_arg = "ignored")()) def closureTest2(some_arg): def enclosed(f = "default_value"): return x x = 4 return enclosed print("Call closured function returning function:", closureTest2(some_arg = "ignored")()) def defaultValueTest1(no_default, some_default_constant = 1): return some_default_constant print("Call function with 2 parameters, one defaulted, and check that the default value is used:", defaultValueTest1("ignored")) def defaultValueTest1a(no_default, some_default_constant_1 = 1, some_default_constant_2 = 2): return some_default_constant_2 - some_default_constant_1 print("Call function with 3 parameters, 2 defaulted, check they are used correctly:", defaultValueTest1a("ignored")) def defaultValueTest2(no_default, some_default_variable = var_on_module_level*2): return some_default_variable print("Call function with 2 parameters, 1 defaulted with an expression, check its result", defaultValueTest2("ignored")) var_on_module_level = 2 print("Call function with 2 parameters, 1 defaulted with an expression, values have changed since, check its result", defaultValueTest2("ignored")) def contractionTest(): j = 2 return [ j + i for i in range(8) ] print("Call function that returns a list contraction:", contractionTest()) def defaultValueTest3a(no_default, funced_defaulted = defaultValueTest2(var_on_module_level)): return [ i + funced_defaulted for i in range(8) ] print("Call function that has a default value coming from a function call:", defaultValueTest3a("ignored")) def defaultValueTest3b(no_default, funced_defaulted = defaultValueTest2(var_on_module_level)): local_var = [ funced_defaulted + i for i in range(8) ] return local_var print("Call function that returns a list contraction result via a local variable:", defaultValueTest3b("ignored")) def defaultValueTest3c(no_default, funced_defaulted = defaultValueTest2(var_on_module_level)): local_var = [ [ j+funced_defaulted+1 for j in range(i) ] for i in range(8) ] return local_var print("Call function that returns a nested list contraction with input from default parameter", defaultValueTest3c("ignored")) def defaultValueTest4(no_default, funced_defaulted = lambda x: x**2): return funced_defaulted(4) print("Call function that returns value calculated by a lambda function as default parameter", defaultValueTest4("ignored")) def defaultValueTest4a(no_default, funced_defaulted = lambda x: x**2): c = 1 d = funced_defaulted(1) r = ( i+j+c+d for i, j in zip(range(8), range(9)) ) l = [] for x in r: l.append(x) return l print("Call function that has a lambda calculated default parameter and a generator expression", defaultValueTest4a("ignored")) def defaultValueTest4b(no_default, funced_defaulted = lambda x: x**3): d = funced_defaulted(1) # Nested generators l = [] for x in ( (d+j for j in range(4)) for i in range(8) ): for y in x: l.append(y) return l print("Call function that has a lambda calculated default parameter and a nested generator expression", defaultValueTest4b("ignored")) def defaultValueTest5(no_default, tuple_defaulted = (1,2,3)): return tuple_defaulted print("Call function with default value that is a tuple", defaultValueTest5("ignored")) def defaultValueTest6(no_default, list_defaulted = [1,2,3]): return list_defaulted print("Call function with default value that is a list", defaultValueTest6("ignored")) def lookup(unused, something): something.very.namelookup.chaining() something.very.namelookup.chaining() x = len("hey") def in_test(a): # if 7 in a: # print "hey" 8 in a # @NoEffect 9 not in a # @NoEffect def printing(): print("Hallo") print("du") print("da") def my_deco(function): def new_function(c, d): return function(d, c) return new_function @my_deco def decoriert(a,b): def subby(a): return 2 + a return 1+subby(b) print("Function with decoration", decoriert(3, 9)) #def var_test(a): # b = len(a) # c = len(a) def user(): global a return a a = "oh common" some_constant_tuple = (2,5,7) some_semiconstant_tuple = (2,5,a) f = a * 2 print(defaultValueTest1("ignored")) # The change of the default variable doesn't influence the default # parameter of defaultValueTest2, that means it's also calculated # at the time the function is defined. module_level = 7 print(defaultValueTest2("also ignored")) def starArgTest(a, b, c): return a, b, c print("Function called with star arg from tuple:") star_list_arg = (11, 44, 77) print(starArgTest(*star_list_arg)) print("Function called with star arg from list:") star_list_arg = [7, 8, 9] print(starArgTest(*star_list_arg)) star_dict_arg = { 'a' : 9, 'b' : 3, 'c': 8 } print("Function called with star arg from dict") print(starArgTest(**star_dict_arg)) lambda_func = lambda a, b : a < b lambda_args = (8, 9) print("Lambda function called with star args from tuple:") print(lambda_func(*lambda_args)) print("Lambda function called with star args from list:") lambda_args = [8, 7] print(lambda_func(*lambda_args)) print("Generator function without context:") def generator_without_context_function(): gen = ( x for x in range(9) ) return tuple(gen) print(generator_without_context_function()) print("Generator function with 2 iterateds:") def generator_with_2_fors(): return tuple( (x, y) for x in range(2) for y in range(3) ) print(generator_with_2_fors()) def someYielder(): yield 1 yield 2 def someYieldFunctionUser(): print("someYielder", someYielder()) result = [] for a in someYielder(): result.append(a) return result print("Function that uses some yielding function coroutine:") print(someYieldFunctionUser()) def someLoopYielder(): for i in (0, 1, 2): yield i def someLoopYieldFunctionUser(): result = [] for a in someLoopYielder(): result.append(a) return result print("Function that uses some yielding function coroutine that loops:") print(someLoopYieldFunctionUser()) def someGeneratorClosureUser(): def someGenerator(): def userOfGeneratorLocalVar(): return x+1 x = 2 yield userOfGeneratorLocalVar() yield 6 gen = someGenerator() return [next(gen), next(gen) ] print("Function generator that uses a local function accessing its local variables to yield:") print(someGeneratorClosureUser()) def someClosureUsingGeneratorUser(): offered = 7 def someGenerator(): yield offered return next(someGenerator()) print("Function generator that yield from its closure:") print(someClosureUsingGeneratorUser()) print("Function call with both star args and named args:") def someFunction(a, b, c, d): print(a, b, c, d) someFunction(a = 1, b = 2, **{ 'c' : 3, 'd' : 4 }) print("Order of evaluation of function and args:") def getFunction(): print("getFunction", end = "") def x(y, u, a, k): return y, u, k, a return x def getPlainArg1(): print("getPlainArg1", end = "") return 9 def getPlainArg2(): print("getPlainArg2", end = "") return 13 def getKeywordArg1(): print("getKeywordArg1", end = "") return 'a' def getKeywordArg2(): print("getKeywordArg2", end = "") return 'b' getFunction()(getPlainArg1(), getPlainArg2(), k = getKeywordArg1(), a = getKeywordArg2()) print() def getListStarArg(): print("getListStarArg", end = "") return [1] def getDictStarArg(): print("getDictStarArg", end = "") return { 'k' : 9 } print("Same with star args:") getFunction()(getPlainArg1(), a = getKeywordArg1(), *getListStarArg(), **getDictStarArg()) print() print("Dictionary creation order:") d = { getKeywordArg1() : getPlainArg1(), getKeywordArg2() : getPlainArg2() } print() print("Throwing an exception to a generator function:") def someGeneratorFunction(): try: yield 1 yield 2 except: yield 3 yield 4 gen1 = someGeneratorFunction() print("Fresh Generator Function throwing gives", end = "") try: print(gen1.throw(ValueError)), except ValueError: print("exception indeed") gen2 = someGeneratorFunction() print("Used Generator Function throwing gives", end = "") next(gen2) print(gen2.throw(ValueError), "indeed") gen3 = someGeneratorFunction() print("Fresh Generator Function close gives", end = "") print(gen3.close()) gen4 = someGeneratorFunction() print("Used Generator Function that mis-catches close gives", end = "") next(gen4) try: print(gen4.close(), end = "") except RuntimeError: print("runtime exception indeed") gen5 = someGeneratorFunction() print("Used Generator Function close gives", end = "") next(gen5) next(gen5) next(gen5) print(gen5.close(), end = "") def receivingGenerator(): while True: a = yield 4 yield a print("Generator function that receives", end = "") gen6 = receivingGenerator() print(next(gen6), end = "") print(gen6.send(5), end = "") print(gen6.send(6), end = "") print(gen6.send(7), end = "") print(gen6.send(8)) print("Generator function whose generator is copied", end = "") def generatorFunction(): yield 1 yield 2 gen7 = generatorFunction() next(gen7) gen8 = iter(gen7) print(next(gen8)) def doubleStarArgs(*a, **d): return a, d try: from UserDict import UserDict except ImportError: print("Using Python3, making own non-dict dict:") class UserDict(dict): pass print("Function that has keyword argument matching the list star arg name", end = "") print(doubleStarArgs(1, **UserDict(a = 2))) def generatorFunctionUnusedArg(a): yield 1 generatorFunctionUnusedArg(3) def closureHavingGenerator(arg): def gen(x = 1): yield arg return gen() print("Function generator that has a closure and default argument", end = "") print(list(closureHavingGenerator(3))) def functionWithDualStarArgsAndKeywordsOnly(a1, a2, a3, a4, b): return a1, a2, a3, a4, b l = [1, 2, 3] d = {'b': 8} print("Dual star args, but not positional call", functionWithDualStarArgsAndKeywordsOnly(a4 = 1, *l, **d)) def posDoubleStarArgsFunction(a, b, c, *l, **d): return a, b, c, l, d l = [2] d = { "other" : 7, 'c' : 3 } print("Dual star args consuming function", posDoubleStarArgsFunction(1, *l, **d)) import inspect, sys for value in sorted(dir()): main_value = getattr(sys.modules[ "__main__" ], value) if inspect.isfunction(main_value): print(main_value, main_value.__code__.co_varnames[:main_value.__code__.co_argcount]) # inspect.getargs( main_value.func_code ) # TODO: Make this work as well, currently disabled, because of nested arguments not # being compatible yet. # print main_value, main_value.func_code.co_varnames, inspect.getargspec( main_value ) pass Nuitka-0.5.18.1/tests/basics/TryContinueFinally.py0000644000175000017500000000406112651071040022210 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def tryWhileContinueFinallyTest(): print("Check if finally is executed in a continue using for loop:") x = 0 while x < 10: x += 1 try: if x % 2 == 1: continue finally: print(x, end = ' ') print('-', end = ' ') print() def tryForContinueFinallyTest(): print("Check if finally is executed in a continue using for loop:") for x in range(10): try: if x % 2 == 1: continue finally: print(x, end = ' ') print('-', end = ' ') print() def tryWhileBreakFinallyTest(): print("Check if finally is executed in a break using while loop:") x = 0 while x < 10: x += 1 try: if x == 5: break finally: print(x, end = ' ') print('-', end = ' ') print() def tryForBreakFinallyTest(): print("Check if finally is executed in a break using for loop:") for x in range(10): try: if x == 5: break finally: print(x, end = ' ') print('-', end = ' ') print() tryWhileContinueFinallyTest() tryWhileBreakFinallyTest() tryForContinueFinallyTest() tryForBreakFinallyTest() Nuitka-0.5.18.1/tests/basics/BuiltinSuper.py0000644000175000017500000000622712651071040021041 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function # Python2 will fallback to this variable, which Python3 will ignore. __class__ = "Using module level __class__ variable, would be wrong for Python3" # @ReservedAssignment class ClassWithUnderClassClosure: def g(self): def h(): print("Variable __class__ in ClassWithUnderClassClosure is", __class__) h() try: print("ClassWithUnderClassClosure: Super in ClassWithUnderClassClosure is", super()) except Exception as e: print("ClassWithUnderClassClosure: Occurred during super call", repr(e)) print("Class with a method that has a local function accessing __class__:") ClassWithUnderClassClosure().g() class ClassWithoutUnderClassClosure: def g(self): __class__ = "Providing __class__ ourselves, then it must be used" # @ReservedAssignment print(__class__) try: print("ClassWithoutUnderClassClosure: Super", super()) except Exception as e: print("ClassWithoutUnderClassClosure: Occurred during super call", repr(e)) ClassWithoutUnderClassClosure().g() # For Python2 only. __class__ = "Global __class__" # @ReservedAssignment def deco(C): print("Decorating", repr(C)) class D(C): pass return D @deco class X: __class__ = "some string" def f1(self): print("f1", locals()) try: print("f1", __class__) except Exception as e: print("Accessing __class__ in f1 gave", repr(e)) def f2(self): print("f2", locals()) def f4(self): print("f4", self) self = X() print("f4", self) try: print("f4", super()) print("f4", super().__self__) except TypeError: import sys assert sys.version < (3,) f5 = lambda x: __class__ def f6(self_by_another_name): # @NoSelf try: print("f6", super()) except TypeError: import sys assert sys.version < (3,) def f7(self): try: yield super() except TypeError: import sys assert sys.version < (3,) print("Early pre-class calls begin") print("Set in class __class__", __class__) # f1(1) f2(2) print("Early pre-class calls end") del __class__ x = X() x.f1() x.f2() x.f4() print("f5", x.f5()) x.f6() print("f7", list(x.f7())) Nuitka-0.5.18.1/tests/basics/Empty.py0000644000175000017500000000140212651071040017500 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/basics/Referencing32.py0000644000175000017500000000550312651071040021004 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( executeReferenceChecked, my_print, ) if not hasattr(sys, "gettotalrefcount"): my_print("Warning, using non-debug Python makes this test ineffective.") sys.gettotalrefcount = lambda : 0 def simpleFunction1(): def abc(*, exc=IOError): pass for _ in range(100): abc() def simpleFunction2(): def abc(*, exc=IOError): raise ValueError from None try: abc() except (ValueError, TypeError): pass def simpleFunction3(): try: class A(Exception): pass class B(Exception): pass try: raise A('foo') except A as e1: raise B(str(e1)) from e1 except Exception: pass def simpleFunction4(): a = 1 def nonlocal_writer(): nonlocal a for a in range(10): pass nonlocal_writer() assert a == 9, a def simpleFunction5(): x = 2 def local_func(a: int, b: x*x): pass local_func(x, x) def simpleFunction6(): # Make sure exception state is cleaned up as soon as the except # block is left. class MyException(Exception): def __init__(self, obj): self.obj = obj class MyObj: pass def inner_raising_func(): local_ref = obj raise MyException(obj) # "except" block raising another exception obj = MyObj() try: try: inner_raising_func() except: raise KeyError except KeyError as e: pass # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.18.1/tests/basics/ComparisonChains.py0000644000175000017500000000762312651071040021655 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def simple_comparisons(x, y): if 'a' <= x <= y <= 'z': print("One") if 'a' <= x <= 'z': print("Two") if 'a' <= x > 'z': print("Three") print("Simple comparisons:") simple_comparisons('c', 'd') def side_effect(): print("") return 7 def side_effect_comparisons(): print("Should have side effect:") print(1 < side_effect() < 9) print("Should not have side effect due to short circuit:") print(3 < 2 < side_effect() < 9) print("Check for expected side effects only:") side_effect_comparisons() def function_torture_is(): a = (1, 2, 3) for x in a: for y in a: for z in a: print(x, y, z, ':', x is y is z, x is not y is not z) function_torture_is() print("Check if lambda can have expression chains:", end = "") def function_lambda_with_chain(): a = (1, 2, 3) x = lambda x : x[0] < x[1] < x[2] print("lambda result is", x(a)) function_lambda_with_chain() print("Check if generators can have expression chains:", end = "") def generator_function_with_chain(): x = (1, 2, 3) yield x[0] < x[1] < x[2] print(list(generator_function_with_chain())) print("Check if list contractions can have expression chains:", end = "") def contraction_with_chain(): return [ x[0] < x[1] < x[2] for x in [(1, 2, 3) ] ] print(contraction_with_chain()) print("Check if generator expressions can have expression chains:", end = "") def genexpr_with_chain(): return ( x[0] < x[1] < x[2] for x in [(1, 2, 3) ] ) print(list(genexpr_with_chain())) print("Check if class bodies can have expression chains:", end = "") class class_with_chain: x = (1, 2, 3) print(x[0] < x[1] < x[2]) x = (1, 2, 3) print(x[0] < x[1] < x[2]) class CustomOps(int): def __lt__(self, other): print("enter <", self, other) return True def __gt__(self, other): print("enter >", self, other) return False print("Custom ops, to enforce chain eval order and short circuit:", end = "") print(CustomOps(7) < CustomOps(8) > CustomOps(6)) print("Custom ops, doing short circuit:", end = "") print(CustomOps(8) > CustomOps(7) < CustomOps(6)) def inOperatorChain(): print("In operator chains:") print(3 in [3,4] in [[3,4]]) print(3 in [3,4] not in [[3,4]]) if 3 in [3,4] in [[3,4]]: print("Yes") else: print("No") if 3 in [3,4] not in [[3,4]]: print("Yes") else: print("No") inOperatorChain() # Make sure the values are called and order is correct: class A(object): def __init__(self, name, value): self.name = name self.value = value def __repr__(self): return "" % (self.name, self.value) def __lt__(self, other): print("less than called for:", self, other, self.value, other.value, self.value < other.value) if self.value < other.value: print("good") return 7 else: print("bad") return 0 a = A('a',1) b = A('b',2) c = A('c',0) print(a < b < c) print('*' * 80) a = A('a',2) b = A('b',1) c = A('c',0) print(a < b < c) Nuitka-0.5.18.1/tests/basics/Referencing_2.py0000644000175000017500000000472112651071040021061 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( executeReferenceChecked, checkReferenceCount, my_print, ) if not hasattr(sys, "gettotalrefcount"): my_print("Warning, using non-debug Python makes this test ineffective.") sys.gettotalrefcount = lambda : 0 x = 17 # Python2 only syntax things are here. def simpleFunction1(): try: raise TypeError, (3,x,x,x) except TypeError: pass def simpleFunction2(): try: raise ValueError(1,2,3), ValueError(1,2,3) except Exception: pass def simpleFunction3(): try: raise ValueError, 2, None except Exception: pass def simpleFunction4(): try: raise ValueError, 2, 3 except Exception: pass def simpleFunction5(): def nested_args_function((a,b), c): return a, b, c nested_args_function((1, 2), 3) def simpleFunction6(): def nested_args_function((a,b), c): return a, b, c try: nested_args_function((1,), 3) except ValueError: pass def simpleFunction7(): def nested_args_function((a,b), c): return a, b, c try: nested_args_function((1, 2, 3), 3) except ValueError: pass # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.18.1/tests/basics/GeneratorExpressions.py0000644000175000017500000001255712651071040022610 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import inspect print("Generator expression that demonstrates the timing:") def iteratorCreationTiming(): def getIterable(x): print("Getting iterable", x) return Iterable(x) class Iterable: def __init__(self, x): self.x = x self.values = list(range(x)) self.count = 0 def __iter__(self): print("Giving iterator now", self.x) return self def __next__(self): print("Next of", self.x, "is", self.count) if len(self.values) > self.count: self.count += 1 return self.values[ self.count - 1 ] else: print("Raising StopIteration for", self.x) raise StopIteration # Python2/3 compatibility. next = __next__ def __del__(self): print("Deleting", self.x) gen = ( (y,z) for y in getIterable(3) for z in getIterable(2) ) print("Using generator", gen) next(gen) res = tuple(gen) print(res) print('*' * 20) try: next(gen) except StopIteration: print("Usage past end gave StopIteration exception as expected.") try: print("Generator state then is", inspect.getgeneratorstate(gen)) # @UndefinedVariable except AttributeError: pass print("Its frame is now", gen.gi_frame) print("Early aborting generator:") gen2 = ( (y,z) for y in getIterable(3) for z in getIterable(2) ) del gen2 iteratorCreationTiming() print("Generator expressions that demonstrate the use of conditions:") print(tuple( x for x in range(8) if x % 2 == 1 )) print(tuple( x for x in range(8) if x % 2 == 1 for z in range(8) if z == x )) print(tuple( x for (x,y) in zip(list(range(2)),list(range(4))))) print("Directory of generator expressions:") for_dir = ( x for x in [1] ) gen_dir = dir(for_dir) print(sorted( g for g in gen_dir )) def genexprSend(): x = ( x for x in range(9) ) print("Sending too early:") try: x.send(3) except TypeError as e: print("Gave expected TypeError with text:", e) z = next(x) y = x.send(3) print("Send return value", y) print("And then next gave", next(x)) print("Throwing an exception to it.") try: x.throw(2) assert False except TypeError as e: print("Gave expected TypeError:", e) print("Throwing an exception to it.") try: x.throw(ValueError(2)) except ValueError as e: print("Gave expected ValueError:", e) try: next(x) print("Next worked even after thrown error") except StopIteration as e: print("Gave expected stop iteration after throwing exception in it:", e) print("Throwing another exception from it.") try: x.throw(ValueError(5)) except ValueError as e: print("Gave expected ValueError with text:", e) print("Generator expressions have send too:") genexprSend() def genexprClose(): x = ( x for x in range(9) ) print("Immediate close:") x.close() print("Closed once") x.close() print("Closed again without any trouble") genexprClose() def genexprThrown(): def checked(z): if z == 3: raise ValueError return z x = ( checked(x) for x in range(9) ) try: for count, value in enumerate(x): print(count, value) except ValueError: print(count+1, ValueError) try: next(x) print("Allowed to do next() after raised exception from the generator expression") except StopIteration: print("Exception in generator, disallowed next() afterwards.") genexprThrown() def nestedExpressions(): a = [x for x in range(10)] b = (x for x in (y for y in a)) print("nested generator expression", list(b)) nestedExpressions() def lambdaGenerators(): a = 1 x = lambda : (yield a) print("Simple lambda generator", x, x(), list(x())) y = lambda : ((yield 1),(yield 2)) print("Complex lambda generator", y, y(), list(y())) lambdaGenerators() def functionGenerators(): # Like lambdaGenerators, to show how functions behave differently if at all. a = 1 def x(): yield a print("Simple function generator", x, x(), list(x())) def y(): yield((yield 1),(yield 2)) print("Complex function generator", y, y(), list(y())) functionGenerators() def strangeLambdaGeneratorExpression(): x = ((yield) for i in (1,2) if (yield)) print("Strange lambda generator expression") print(list(x)) strangeLambdaGeneratorExpression() Nuitka-0.5.18.1/tests/basics/LateClosureAssignment.py0000644000175000017500000000521512651071040022663 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def closureTest1(): # Assign, but the value is not supposed to be used by the function, instead the later # update is effective. d = 1 def subby(): return d d = 22222*2222 return subby() def closureTest2(): # Using a closure variable that is not initialized at the time it is closured should # work as well. def subby(): return d d = 2222*2222 return subby() def closureTest3(): def subby(): return undefined_global # @UndefinedVariable try: return subby() except NameError: return 88 d = 1 def scopeTest4(): try: return d d = 1 except UnboundLocalError as e: return repr(e) print("Test closure where value is overwritten:", closureTest1()) print("Test closure where value is assigned only late:", closureTest2()) print("Test function where closured value is never assigned:", closureTest3()) print("Scope test where UnboundLocalError is expected:", scopeTest4()) def function(): pass class ClosureLocalizerClass: print("Function before assigned in a class:", function) function = 1 print("Function after it was assigned in class:", function) ClosureLocalizerClass() def ClosureLocalizerFunction(): try: function = function print("Function didn't give unbound local error") except UnboundLocalError as e: print("Function gave unbound local error when accessing function before assignment:", repr(e)) ClosureLocalizerFunction() class X: def __init__(self, x): self.x = x def changingClosure(): print("Changing a closure taken value after it was taken.") a = 1 def closureTaker(): return X(a) x = closureTaker() a=2 print("Closure value first time:", x.x) x = closureTaker() print("Closure value second time:", x.x) changingClosure() Nuitka-0.5.18.1/tests/basics/ExecEval.py0000644000175000017500000001477212651071040020114 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import tempfile, sys, os print "eval 3+3=", eval("3+3") print "eval 4+4=", eval(" 4+4") def functionEval1(): return eval(" 5+5") print "eval in a function with nothing provided", functionEval1() def functionEval2(): a = [2] g = {} r = eval("1+3", g) return r, g.keys(), a print "eval in a function with globals provided", functionEval2() def functionEval3(): result = [] for x in eval("(1,2)"): result.append(x) return result print "eval in a for loop as iterator giver", functionEval3() print "exec on a global level", exec( "d=2+2" ) print "2+2=",d # @UndefinedVariable def functionExec1(): a = 1 code = "a=2" exec( code ) return a def functionExec2(): a = 1 code = "a=2" exec code in globals(), locals() return a print "exec in function without and with locals() provided:", functionExec1(), functionExec2() tmp_filename = tempfile.gettempdir() + "/execfile.py" f = open(tmp_filename, 'w') f.write("e=7\nf=8\n") f.close() execfile(tmp_filename) print "execfile with defaults f,g=", e, f # @UndefinedVariable global_vars = { 'e' : '0', 'f' : 0 } local_vars = dict(global_vars) execfile(tmp_filename, global_vars) print "execfile with globals dict:", global_vars.keys() execfile(tmp_filename, global_vars, local_vars) print "execfile with globals and locals dict:", local_vars def functionExecfile(): e = 0 f = 0 global_vars = { 'e' : '0', 'f' : 0 } local_vars = dict(global_vars) print "execfile with globals and locals dict in a function:", x = execfile(tmp_filename, global_vars, local_vars) print x, print global_vars.keys(), local_vars, e, f functionExecfile() class classExecfile: e = 0 f = 0 print "execfile in a class:", # TODO: Won't work yet, Issue#5 # print execfile( tmp_filename ), execfile(tmp_filename) print "execfile changed local values:", e, f f = 7 def functionExecNonesTuple(): f = 0 exec("f=1", None, None) print "Exec with None as optimizable tuple args did update locals:", f def functionExecNonesSyntax(): f = 0 exec "f=2" in None, None print "Exec with None as optimizable normal args did update locals:", f functionExecNonesTuple() functionExecNonesSyntax() print "Global value is untouched", f def functionEvalNones2(): f = 11 code = 'f' g = None l = None f1 = eval (code, l, g) print "Eval with None arguments from variables did access locals:", f1 functionEvalNones2() def functionExecNonesTuple2(): f = 0 code = "f=1" g = None l = None exec(code, l, g) print "Exec with None as tuple args from variable did update locals:", f def functionExecNonesSyntax2(): f = 0 code = "f=2" g = None l = None exec code in l, g print "Exec with None as normal args did update locals:", f functionExecNonesTuple2() functionExecNonesSyntax2() print "Exec with a future division definition and one without:" exec """ from __future__ import division from __future__ import print_function print( "3/2 is with future division", 3/2 ) """ exec """ from __future__ import print_function print( "3/2 is without future division", 3/2 ) """ x = 1 y = 1 def functionGlobalsExecShadow(): global x print "Global x outside is", x y = 0 print "Local y is initially", y print "Locals initially", locals() exec """ from __future__ import print_function x = 2 print( "Exec local x is", x ) """ print "Function global x referenced as local x in exec is", x exec """ from __future__ import print_function print( "Re-exec local x", x ) """ print "Locals after exec assigning to local x", locals() exec """ from __future__ import print_function global x x = 3 print( "Exec global x is inside exec", x ) """ print "Global x referenced as global x in exec is", x exec """ from __future__ import print_function def change_y(): global y y = 4 print( "Exec function global y is", y ) y = 7 change_y() # TODO: The below will not work print( "Exec local y is", y ) """ # print "Local y is afterwards", y def print_global_y(): global y # TODO: The below will not work print "Global y outside", y print_global_y() print "Outside y", y functionGlobalsExecShadow() def functionWithClosureProvidedByExec(): code = "ValueError = TypeError" exec code in None, None def func(): print "Closure from exec not used", ValueError func() functionWithClosureProvidedByExec() x = 2 def functionWithExecAffectingClosure(): x = 4 code = "d=3;x=5" space = locals() exec code in space def closureMaker(): return x return d, closureMaker() # @UndefinedVariable print "Closure in a function with exec to not none", functionWithExecAffectingClosure() def generatorFunctionWithExec(): yield 1 code = "y = 2" exec code yield y print "Exec in a generator function", tuple(generatorFunctionWithExec()) def evalInContractions(): r1 = list( eval(str(s)) for s in range(3) ) r2 = [ eval(str(s)) for s in range(4) ] return r1, r2 print "Eval in a list contraction or generator expression", evalInContractions() def execDefinesFunctionToLocalsExplicity(): exec """\ def makeAddPair(a, b): def addPair(c, d): return (a + c, b + d) return addPair """ in locals() if sys.version_info < (3,): assert makeAddPair # @UndefinedVariable return "yes" print "Exec adds functions declares in explicit locals() given.", execDefinesFunctionToLocalsExplicity() os.unlink(tmp_filename) def execWithShortTuple(): try: exec("print hey",) except Exception as e: return "gives exception: " + repr(e) print "Exec with too short tuple argument:", execWithShortTuple() Nuitka-0.5.18.1/tests/optimizations/0000755000175000017500000000000012651072347017513 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/optimizations/Attributes.py0000644000175000017500000000204012651071040022174 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print((1).imag) print(int.__name__) print((1).__class__) print(getattr(1, "real")) print(getattr(1, "real", None)) print(hasattr(1, "real")) print((0.0).real) Nuitka-0.5.18.1/tests/optimizations/Operations.py0000644000175000017500000000201112651072046022176 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(not bool) print(not {}) print(not 7) # TODO: Needs some SSA now. # print(bool or len) # print(False or dict) print(type(Ellipsis)) Nuitka-0.5.18.1/tests/optimizations/Conditions.py0000644000175000017500000000163112651071040022164 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(1 if [1,2] else 2) Nuitka-0.5.18.1/tests/optimizations/Subscripts.py0000644000175000017500000000175012651071040022216 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( (1,2,3)[1] ) print( (1,2,3)[1:] ) print( (1,2,3)[:2] ) print( (1,2,3)[:] ) print( (1,2,3)[1:2] ) Nuitka-0.5.18.1/tests/optimizations/run_all.py0000755000175000017500000001422112651071040021511 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys, subprocess # The test runner needs "lxml" itself. try: import lxml.etree except ImportError: print("Warning, no 'lxml' module installed, cannot do XML based tests.") sys.exit(0) # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, convertUsing2to3, createSearchMode, hasModule, check_output ) python_version = setup() # The Python version used by Nuitka needs "lxml" too. if not hasModule("lxml.etree"): print("Warning, no 'lxml' module installed, cannot run XML based tests.") sys.exit(0) search_mode = createSearchMode() def getKind(node): result = node.attrib[ "kind" ] result = result.replace("Statements", "") result = result.replace("Statement", "") result = result.replace("Expression", "") return result def getRole(node, role): for child in node: if child.tag == "role" and child.attrib["name"] == role: return child else: return None def getSourceRef(node): return "%s:%s" % ( filename, node.attrib["line"] ) def checkSequence(statements): for statement in statements: kind = getKind(statement) # Printing is fine. if kind == "PrintValue": print_arg, = getRole(statement, "value") if getKind(print_arg) not in ("ConstantRef", "ModuleFileAttributeRef"): sys.exit( "%s: Error, print of non-constant %s." % ( getSourceRef(statement), getKind(print_arg) ) ) continue if kind == "PrintNewline": continue # Printing in Python3 is a function call whose return value is ignored. if kind == "Only": only_expression = getRole(statement, "expression")[0] if getKind(only_expression) == "CallNoKeywords": called_expression = getRole(only_expression, "called")[0] if getKind(called_expression) == "BuiltinRef": if called_expression.attrib["builtin_name"] == "print": continue if kind == "Frame": checkSequence(getRole(statement, "statements")) continue if kind == "AssignmentVariable": assign_source, = getRole(statement, "source") source_kind = getKind(assign_source) if source_kind not in("ConstantRef", "ImportModuleHard", "ModuleFileAttributeRef"): sys.exit("Error, assignment from of non-constant %s." % source_kind) continue print(lxml.etree.tostring(statement, pretty_print = True)) sys.exit("Error, non-print statement of unknown kind '%s'." % kind) for filename in sorted(os.listdir('.')): if not filename.endswith(".py") or filename.startswith("run_"): continue active = search_mode.consider( dirname = None, filename = filename ) extra_flags = ["expect_success"] if active: # Apply 2to3 conversion if necessary. if python_version.startswith('3'): filename, changed = convertUsing2to3(filename) else: changed = False my_print("Consider", filename, end = ' ') command = [ os.environ["PYTHON"], os.path.abspath(os.path.join("..", "..", "bin", "nuitka")), "--dump-xml", "--module", filename ] if search_mode.isCoverage(): # To avoid re-execution, which is not acceptable to coverage. if "PYTHONHASHSEED" not in os.environ: os.environ["PYTHONHASHSEED"] = '0' # Coverage modules hates Nuitka to re-execute, and so we must avoid # that. python_path = subprocess.check_output( [ os.environ["PYTHON"], "-c" "import sys, os; print(os.pathsep.join(sys.path))" ] ) if sys.version_info >= (3,): python_path = python_path.decode("utf8") os.environ["PYTHONPATH"] = python_path.strip() command.insert(2, "--must-not-re-execute") command = command[0:1] + [ "-S", "-m", "coverage", "run", "--rcfile", os.devnull, "-a", ] + command[1:] result = check_output( command ) # Parse the result into XML and check it. root = lxml.etree.fromstring(result) module_body = root[0] module_statements_sequence = module_body[0] assert len(module_statements_sequence) == 1 module_statements = next(iter(module_statements_sequence)) checkSequence(module_statements) if changed: os.unlink(filename) my_print("OK.") else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.18.1/tests/optimizations/Calls.py0000644000175000017500000000173112651071040021112 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # TODO: This is only a placeholder, currently no calls will be optimized print(range(4)) Nuitka-0.5.18.1/tests/optimizations/Len.py0000644000175000017500000000231412651071040020570 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print(len(range(266))) print(len(range(266,9999))) print(len(range(266,9999,3))) print(len(range(266,9999,-3))) print(len(range(22266,9999,-3))) print(len(range(22266,9998,-3))) print(len(range(22266,9997,-3))) print(len(range(22266,9996,-3))) print(len(range(0,3,3))) print(len(iter((1, 2)))) Nuitka-0.5.18.1/tests/programs/0000755000175000017500000000000012651072347016434 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_module_collision/0000755000175000017500000000000012651072347023447 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_module_collision/Something/0000755000175000017500000000000012651072347025404 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_module_collision/Something/__init__.py0000644000175000017500000000145212651071040027504 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Importing Something/__init__.py") Nuitka-0.5.18.1/tests/programs/package_module_collision/something.py0000644000175000017500000000144112651071040026003 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Importing something.py") Nuitka-0.5.18.1/tests/programs/package_module_collision/PackageAndModuleNamedSameMain.py0000644000175000017500000000160512651071040031514 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import Something import something print("This would collide on Windows generated source names:") print(Something) print(something) Nuitka-0.5.18.1/tests/programs/plugin_import/0000755000175000017500000000000012651072347021324 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/plugin_import/PluginImportMain.py0000644000175000017500000000153612651071040025126 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # name = "some_module" module = getattr(__import__("some_package", fromlist = [name]), name) Nuitka-0.5.18.1/tests/programs/plugin_import/some_package/0000755000175000017500000000000012651072347023742 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/plugin_import/some_package/some_module.py0000644000175000017500000000141712651071040026614 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "hi" ) Nuitka-0.5.18.1/tests/programs/plugin_import/some_package/__init__.py0000644000175000017500000000140312651071040026036 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Nuitka-0.5.18.1/tests/programs/main_raises2/0000755000175000017500000000000012651072347021010 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/main_raises2/ErrorInFunctionMain.py0000644000175000017500000000207412651071040025245 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Just plain exception from the function level, supposed to report the correct file and line def generator_function(): import ErrorRaising x = ( lambda : ErrorRaising.raiseException() for z in range(3) ) next(x)() def normal_function(): y = generator_function() y() normal_function() Nuitka-0.5.18.1/tests/programs/main_raises2/ErrorRaising.py0000644000175000017500000000145212651071040023757 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def raiseException(): return 1 / 0 Nuitka-0.5.18.1/tests/programs/case_imports1/0000755000175000017500000000000012651072347021205 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports1/CasedImportingMain.py0000644000175000017500000000145112651071040025262 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import Some_Module import some_package Nuitka-0.5.18.1/tests/programs/case_imports1/path1/0000755000175000017500000000000012651072347022222 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports1/path1/Some_Package/0000755000175000017500000000000012651072347024540 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports1/path1/Some_Package/__init__.py0000644000175000017500000000143712651071040026643 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package") Nuitka-0.5.18.1/tests/programs/case_imports1/path1/Some_Module.py0000644000175000017500000000143612651071040024775 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module") Nuitka-0.5.18.1/tests/programs/case_imports1/path2/0000755000175000017500000000000012651072347022223 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports1/path2/some_package/0000755000175000017500000000000012651072347024641 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports1/path2/some_package/__init__.py0000644000175000017500000000143712651071040026744 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is some_package") Nuitka-0.5.18.1/tests/programs/case_imports1/path2/some_module.py0000644000175000017500000000143612651071040025076 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is some_module") Nuitka-0.5.18.1/tests/programs/absolute_import/0000755000175000017500000000000012651072347021644 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/absolute_import/AbsoluteImportMain.py0000644000175000017500000000150112651071040025756 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import foobar if __name__ == "__main__": foobar.Foobar() Nuitka-0.5.18.1/tests/programs/absolute_import/foobar/0000755000175000017500000000000012651072347023114 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/absolute_import/foobar/util.py0000644000175000017500000000143612651071040024434 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def foo(): return "hi" Nuitka-0.5.18.1/tests/programs/absolute_import/foobar/foobar.py0000644000175000017500000000166012651071040024726 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import absolute_import, print_function from foobar import util from . import local class Foobar(object): def __init__(self): print(util.foo()) Nuitka-0.5.18.1/tests/programs/absolute_import/foobar/local.py0000644000175000017500000000144712651071040024553 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is for relative import.") Nuitka-0.5.18.1/tests/programs/absolute_import/foobar/__init__.py0000644000175000017500000000143612651071040025216 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from .foobar import Foobar Nuitka-0.5.18.1/tests/programs/reimport_main_dynamic/0000755000175000017500000000000012651072347023005 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/reimport_main_dynamic/ImportItselfDynamicMain.py0000644000175000017500000000162112651071040030077 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Here I am before import", __name__) c = "ImportItselfDynamicMain" __import__(c) print("Here I am after import", __name__) Nuitka-0.5.18.1/tests/programs/dash_import/0000755000175000017500000000000012651072347020745 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/dash_import/DashImportMain.py0000644000175000017500000000157112651071040024167 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # __import__("dash-module") b = "dash-module" __import__(b) __import__("plus+module") c = "plus+module" __import__(c) Nuitka-0.5.18.1/tests/programs/dash_import/dash-module.py0000644000175000017500000000146212651071040023511 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Module with dash imported as", __name__) Nuitka-0.5.18.1/tests/programs/dash_import/plus+module.py0000644000175000017500000000146212651071040023553 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Module with plus imported as", __name__) Nuitka-0.5.18.1/tests/programs/with space/0000755000175000017500000000000012651072347020463 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/with space/Space Main.py0000755000175000017500000000147412651071040022733 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Hello from main program with space in its paths" ) Nuitka-0.5.18.1/tests/programs/package_contains_main/0000755000175000017500000000000012651072347022731 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_contains_main/local.py0000644000175000017500000000144212651071040024363 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print ("Imported local package") Nuitka-0.5.18.1/tests/programs/package_contains_main/PackageContainsMain.py0000644000175000017500000000142612651071040027132 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from . import local Nuitka-0.5.18.1/tests/programs/package_contains_main/__init__.py0000644000175000017500000000140212651071040025024 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/dash_main/0000755000175000017500000000000012651072347020357 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/dash_main/Dash-Main.py0000644000175000017500000000145112651071040022460 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is running from",__file__) Nuitka-0.5.18.1/tests/programs/run_all.py0000755000175000017500000001125312651071040020434 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys # Find common code relative in file system. Not using packages for test stuff. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), ".." ) ) ) from test_common import ( my_print, setup, createSearchMode, compareWithCPython, withPythonPathChange ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() extra_options = os.environ.get("NUITKA_EXTRA_OPTIONS","") for filename in sorted(os.listdir('.')): if not os.path.isdir(filename) or \ filename.endswith(".build") or \ filename.endswith(".dist"): continue filename = os.path.relpath(filename) # For these, we expect that they will fail. expected_errors = [ "module_exits", "main_raises", "main_raises2", "package_contains_main" ] # Allowed after Python3, packages need no more "__init__.py" if python_version < "3.3": expected_errors.append("package_missing_init") if filename not in expected_errors: extra_flags = ["expect_success"] else: extra_flags = ["expect_failure"] if filename in ("reimport_main_static", "package_missing_init", "dash_import", "package_contains_main", "case_imports3", "import_variants", "package_init_import"): extra_flags.append("ignore_warnings") extra_flags.append("remove_output") extra_flags.append("recurse_all") # Use the original __file__ value, at least one case warns about things # with filename included. extra_flags.append("original_file") # Cannot include the files with syntax errors, these would then become # ImportError, but that's not the test. In all other cases, use two # step execution, which will not add the program original source to # PYTHONPATH. if filename != "syntax_errors": extra_flags.append("two_step_execution") else: extra_flags.append("binary_python_path") if filename == "plugin_import": os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options + \ " --recurse-directory=%s/some_package" % ( os.path.abspath(filename) ) elif filename == "reimport_main_dynamic": if python_version < '3': os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options + \ " --recurse-directory=%s" % ( os.path.abspath(filename) ) else: os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options + \ " --recurse-pattern=%s/*.py" % ( os.path.abspath(filename) ) extra_flags.append("ignore_warnings") else: os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options active = search_mode.consider( dirname = None, filename = filename ) if active: my_print("Consider output of recursively compiled program:", filename) for filename_main in os.listdir(filename): if filename_main.endswith("Main.py"): break if filename_main.endswith("Main"): break else: sys.exit( """\ Error, no file ends with 'Main.py' or 'Main' in %s, incomplete test case.""" % ( filename ) ) extra_python_path = [ os.path.abspath(os.path.join(filename,entry)) for entry in os.listdir(filename) if entry.startswith("path") ] with withPythonPathChange(extra_python_path): compareWithCPython( dirname = filename, filename = filename_main, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.18.1/tests/programs/Issue16/0000755000175000017500000000000012651072347017673 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/Issue16/err/0000755000175000017500000000000012651072347020463 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/Issue16/err/CrasherModule16.py0000644000175000017500000000160012651071040023723 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # $Id$ class BaseOptions(object): _save_attr = () if __debug__: if __name__ == "__main__": o = BaseOptions() Nuitka-0.5.18.1/tests/programs/Issue16/err/__init__.py0000644000175000017500000000140212651071040022556 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/Issue16/Issue16Main.py0000644000175000017500000000144112651071040022276 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from err import CrasherModule16 Nuitka-0.5.18.1/tests/programs/module_exits/0000755000175000017500000000000012651072347021135 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/module_exits/ErrorExitingModule.py0000644000175000017500000000155312651071040025267 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print( type(__builtins__) ) sys.exit("Module doing sys.exit") print( "This won't happen!" ) Nuitka-0.5.18.1/tests/programs/module_exits/Main.py0000644000175000017500000000153212651071040022361 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( type(__builtins__) ) import ErrorExitingModule print( "Should not get here!" ) Nuitka-0.5.18.1/tests/programs/reimport_main_static/0000755000175000017500000000000012651072347022650 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/reimport_main_static/ImportItselfStaticMain.py0000644000175000017500000000160212651071040027604 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Here I am before import", __name__) import ImportItselfStaticMain print("Here I am after import", __name__) Nuitka-0.5.18.1/tests/programs/package_program/0000755000175000017500000000000012651072347021556 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_program/PackageAsMain/0000755000175000017500000000000012651072347024202 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_program/PackageAsMain/__main__.py0000644000175000017500000000150712651071040026264 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Hello world!", __name__, sys.modules[ __name__ ]) Nuitka-0.5.18.1/tests/programs/relative_import/0000755000175000017500000000000012651072347021641 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/relative_import/dircache.py0000644000175000017500000000140212651071040023737 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/relative_import/RelativeImportMain.py0000644000175000017500000000151412651071040025754 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import absolute_import import dircache print( dircache ) Nuitka-0.5.18.1/tests/programs/package_missing_init/0000755000175000017500000000000012651072347022603 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_missing_init/PackageMissingInitMain.py0000644000175000017500000000144112651071040027460 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package.some_module Nuitka-0.5.18.1/tests/programs/package_missing_init/some_package/0000755000175000017500000000000012651072347025221 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_missing_init/some_package/some_module.py0000644000175000017500000000166512651071040030100 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "This must be Python3.3 or higher, which no longer needs __init__.py to accept a package." ) import sys print("The parent path is", sys.modules[ "some_package"].__path__) Nuitka-0.5.18.1/tests/programs/stdlib_overload/0000755000175000017500000000000012651072347021610 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/stdlib_overload/some_package/0000755000175000017500000000000012651072347024226 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/stdlib_overload/some_package/star_importing.py0000644000175000017500000000166412651071040027635 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "some_package.star_importing, doing the star import" ) print("Before", sorted(dir())) from .pyexpat import * lala = 1 print("After", sorted(dir())) print( "Finished" ) Nuitka-0.5.18.1/tests/programs/stdlib_overload/some_package/pyexpat.py0000644000175000017500000000145412651071040026263 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # defined_in_pyexpat_subpackage = "see me" Nuitka-0.5.18.1/tests/programs/stdlib_overload/some_package/__init__.py0000644000175000017500000000140212651071040026321 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/stdlib_overload/some_package/normal_importing.py0000644000175000017500000000153112651071040030145 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import pyexpat print( "Imported pyexpat, should use our one." ) print( dir(pyexpat) ) Nuitka-0.5.18.1/tests/programs/stdlib_overload/StdlibOverloadMain.py0000644000175000017500000000222412651071040025671 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Main importing nearby package") import pyexpat try: print(pyexpat.defined_in_pyexpat ) except AttributeError: print("Must be Python3, where absolute imports are default.") print( "Main importing from package doing star import" ) from some_package import star_importing print( "Main importing from package doing normal import" ) from some_package import normal_importing print( "Done." ) Nuitka-0.5.18.1/tests/programs/stdlib_overload/pyexpat.py0000644000175000017500000000144112651071040023641 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # defined_in_pyexpat = "see me" Nuitka-0.5.18.1/tests/programs/syntax_errors/0000755000175000017500000000000012651072347021356 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/syntax_errors/SyntaxErroring.py0000644000175000017500000000144012651071040024712 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class x(metaclass=y): pass Nuitka-0.5.18.1/tests/programs/syntax_errors/Main.py0000644000175000017500000000210212651071040022574 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import os try: from SyntaxErroring import x except Exception as e: print("Importing with syntax error gave:", type(e), e) try: from IndentationErroring import x except Exception as e: print("Importing with indentation error gave:", type(e), e) print("Finished.") Nuitka-0.5.18.1/tests/programs/syntax_errors/IndentationErroring.py0000644000175000017500000000142712651071040025705 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): x y Nuitka-0.5.18.1/tests/programs/case_imports3/0000755000175000017500000000000012651072347021207 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports3/CasedImportingMain.py0000644000175000017500000000205712651071040025267 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: import some_module except ImportError: print("Cannot import wrongly cased module.") else: print("OK, imported wrongly cased module.") try: import some_package except ImportError: print("Cannot import wrongly cased package.") else: print("OK, imported wrongly cased package.") Nuitka-0.5.18.1/tests/programs/case_imports3/path1/0000755000175000017500000000000012651072347022224 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports3/path1/Some_Package/0000755000175000017500000000000012651072347024542 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports3/path1/Some_Package/__init__.py0000644000175000017500000000145212651071040026642 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package from path1") Nuitka-0.5.18.1/tests/programs/case_imports3/path1/Some_Module.py0000644000175000017500000000145112651071040024774 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module from path1") Nuitka-0.5.18.1/tests/programs/case_imports3/path2/0000755000175000017500000000000012651072347022225 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports3/path2/Some_Package/0000755000175000017500000000000012651072347024543 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports3/path2/Some_Package/__init__.py0000644000175000017500000000145212651071040026643 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package from path2") Nuitka-0.5.18.1/tests/programs/case_imports3/path2/Some_Module.py0000644000175000017500000000145112651071040024775 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module from path2") Nuitka-0.5.18.1/tests/programs/case_imports2/0000755000175000017500000000000012651072347021206 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports2/CasedImportingMain.py0000644000175000017500000000145112651071040025263 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import Some_Module import some_package Nuitka-0.5.18.1/tests/programs/case_imports2/path1/0000755000175000017500000000000012651072347022223 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports2/path1/some_package/0000755000175000017500000000000012651072347024641 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports2/path1/some_package/__init__.py0000644000175000017500000000140212651071040026734 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/case_imports2/path1/some_module.py0000644000175000017500000000140212651071040025067 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/case_imports2/path2/0000755000175000017500000000000012651072347022224 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports2/path2/Some_Package/0000755000175000017500000000000012651072347024542 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/case_imports2/path2/Some_Package/__init__.py0000644000175000017500000000143712651071040026645 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package") Nuitka-0.5.18.1/tests/programs/case_imports2/path2/Some_Module.py0000644000175000017500000000143612651071040024777 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module") Nuitka-0.5.18.1/tests/programs/main_raises/0000755000175000017500000000000012651072347020726 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/main_raises/ErrorMain.py0000644000175000017500000000161712651071040023170 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Just plain exception from the module level, supposed to report the correct file and line import ErrorRaising ErrorRaising.raiseException() Nuitka-0.5.18.1/tests/programs/main_raises/ErrorRaising.py0000644000175000017500000000145212651071040023675 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def raiseException(): return 1 / 0 Nuitka-0.5.18.1/tests/programs/package_overload/0000755000175000017500000000000012651072347021722 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_overload/foo/0000755000175000017500000000000012651072347022505 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_overload/foo/bar2.py0000644000175000017500000000140212651071040023667 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/package_overload/foo/bar.py0000644000175000017500000000140212651071040023605 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/package_overload/foo/__init__.py0000644000175000017500000000143712651071040024610 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # bar = 42 not_overloaded = 45 Nuitka-0.5.18.1/tests/programs/package_overload/Main.py0000644000175000017500000000156512651071040023154 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from foo import bar, not_overloaded print(bar, not_overloaded) from foo import bar2, not_overloaded print( bar2 ) Nuitka-0.5.18.1/tests/programs/unicode_bom/0000755000175000017500000000000012651072347020717 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/unicode_bom/unicode_bom.py0000644000175000017500000000152112651071040023540 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # import unicodedata print( "This is from file with BOM unicode marker" ) Nuitka-0.5.18.1/tests/programs/unicode_bom/UnicodeBomMain.py0000644000175000017500000000150612651071040024111 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Importing unicode BOM file:") import unicode_bom print("OK.") Nuitka-0.5.18.1/tests/programs/package_init_import/0000755000175000017500000000000012651072347022444 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_init_import/some_package/0000755000175000017500000000000012651072347025062 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_init_import/some_package/PackageLocal.py0000644000175000017500000000144312651071040027731 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Imported PackageLocal" ) Nuitka-0.5.18.1/tests/programs/package_init_import/some_package/__init__.py0000644000175000017500000000162012651071040027157 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: import PackageLocal except ImportError: print( "This must be Python3, doing local import then." ) from . import PackageLocal Nuitka-0.5.18.1/tests/programs/package_init_import/Main.py0000644000175000017500000000147212651071040023673 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package print( some_package.PackageLocal ) Nuitka-0.5.18.1/tests/programs/import_variants/0000755000175000017500000000000012651072347021655 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/import_variants/ImportVariationsMain.py0000644000175000017500000000171212651071040026334 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** Main: Importing" ) import some_package.Child2 print( "*** Main: Imported" ) print("*** Main: Some package", some_package) print("*** Main: Imported package child", some_package.Child2) Nuitka-0.5.18.1/tests/programs/import_variants/some_package/0000755000175000017500000000000012651072347024273 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/import_variants/some_package/Child2.py0000644000175000017500000000205012651071040025734 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("*** Child2: Begin", __name__) try: import Child1 except ImportError: print( "This must be Python3, doing local import then." ) from . import Child1 print("*** Child2: Child2 is in", __package__) print("*** Child2: Imported nearby child", Child1) print( "*** Child2: End" ) Nuitka-0.5.18.1/tests/programs/import_variants/some_package/Child3.py0000644000175000017500000000147312651071040025745 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** Child3: Begin" ) print( "*** Child3: End" ) Nuitka-0.5.18.1/tests/programs/import_variants/some_package/Child1.py0000644000175000017500000000161712651071040025743 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** Child1: Begin" ) from . import Child3 as localname print("*** Child1: Imported Child3", localname) print( "*** Child1: End" ) Nuitka-0.5.18.1/tests/programs/import_variants/some_package/__init__.py0000644000175000017500000000160612651071040026374 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** some_package: Coming from '%s'" % __file__.replace(".pyc", ".py") ) print( "*** some_package: Path is '%s'" % __path__ ) Nuitka-0.5.18.1/tests/programs/deep/0000755000175000017500000000000012651072347017351 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/deep/DeepProgramMain.py0000644000175000017500000000154112651071040022723 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package.DeepChild import some_package.deep_package.DeepDeepChild print( "Done." ) Nuitka-0.5.18.1/tests/programs/deep/some_package/0000755000175000017500000000000012651072347021767 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/deep/some_package/DeepChild.py0000644000175000017500000000154612651071040024155 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Importing child." ) class A: pass print("Class defined here, has these vars", vars(A)) Nuitka-0.5.18.1/tests/programs/deep/some_package/DeepBrother.py0000644000175000017500000000165412651071040024537 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is deep brother module talking.", __name__) def someBrotherFunction(): pass print("The __module__ of function here is", someBrotherFunction.__module__) Nuitka-0.5.18.1/tests/programs/deep/some_package/__init__.py0000644000175000017500000000140212651071040024062 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/deep/some_package/deep_package/0000755000175000017500000000000012651072347024357 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/deep/some_package/deep_package/DeepDeepChild.py0000644000175000017500000000151012651071040027332 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "This is DeepDeepChild talking." ) from .. import DeepBrother Nuitka-0.5.18.1/tests/programs/deep/some_package/deep_package/__init__.py0000644000175000017500000000140212651071040026452 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/tests/programs/package_code/0000755000175000017500000000000012651072347021021 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_code/some_package/0000755000175000017500000000000012651072347023437 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/tests/programs/package_code/some_package/SomeModule.py0000644000175000017500000000145412651071040026053 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Thanks for importing SomeModule" ) Nuitka-0.5.18.1/tests/programs/package_code/some_package/__init__.py0000644000175000017500000000145112651071040025536 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Thanks for importing", __name__) Nuitka-0.5.18.1/tests/programs/package_code/PackageInitCodeMain.py0000644000175000017500000000144012651071040025136 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package.SomeModule Nuitka-0.5.18.1/tests/run-tests0000755000175000017500000004425412651071040016472 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Main front-end to the tests of Nuitka. Has many options, read --help output. """ from __future__ import print_function import os, sys, tempfile, subprocess if "nocheck" in os.environ.get("DEB_BUILD_OPTIONS", "").split(): print("Skipped all tests as per DEB_BUILD_OPTIONS environment.") sys.exit(0) from optparse import OptionParser parser = OptionParser() parser.add_option( "--skip-basic-tests", action = "store_false", dest = "basic_tests", default = True, help = """\ The basic tests, execute these to check if Nuitka is healthy. Default is %default.""" ) parser.add_option( "--skip-syntax-tests", action = "store_false", dest = "syntax_tests", default = True, help = """\ The syntax tests, execute these to check if Nuitka handles Syntax errors fine. Default is %default.""" ) parser.add_option( "--skip-program-tests", action = "store_false", dest = "program_tests", default = True, help = """\ The programs tests, execute these to check if Nuitka handles programs, e.g. import recursions, etc. fine. Default is %default.""" ) parser.add_option( "--skip-package-tests", action = "store_false", dest = "package_tests", default = True, help = """\ The packages tests, execute these to check if Nuitka handles packages, e.g. import recursions, etc. fine. Default is %default.""" ) parser.add_option( "--skip-optimizations-tests", action = "store_false", dest = "optimization_tests", default = True, help = """\ The optimization tests, execute these to check if Nuitka does optimize certain constructs fully away. Default is %default.""" ) parser.add_option( "--skip-standalone-tests", action = "store_false", dest = "standalone_tests", default = os.name != "posix" or os.uname()[0] != "NetBSD", help = """\ The standalone tests, execute these to check if Nuitka standalone mode, e.g. not referring to outside, important 3rd library packages like PyQt fine. Default is %default.""" ) parser.add_option( "--skip-reflection-test", action = "store_false", dest = "reflection_test", default = True, help = """\ The reflection test compiles Nuitka with Nuitka, and then Nuitka with the compile Nuitka and compares the outputs. Default is %default.""" ) parser.add_option( "--skip-cpython26-tests", action = "store_false", dest = "cpython26", default = True, help = """\ The standard CPython2.6 test suite. Execute this for all corner cases to be covered. With Python 2.7 this covers exception behavior quite well. Default is %default.""" ) parser.add_option( "--skip-cpython27-tests", action = "store_false", dest = "cpython27", default = True, help = """\ The standard CPython2.7 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython32-tests", action = "store_false", dest = "cpython32", default = True, help = """\ The standard CPython3.2 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython33-tests", action = "store_false", dest = "cpython33", default = True, help = """\ The standard CPython3.3 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython34-tests", action = "store_false", dest = "cpython34", default = True, help = """\ The standard CPython3.4 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is %default.""" ) parser.add_option( "--no-python2.6", action = "store_true", dest = "no26", default = False, help = """\ Do not use Python2.6 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python2.7", action = "store_true", dest = "no27", default = False, help = """\ Do not use Python2.7 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.2", action = "store_true", dest = "no32", default = False, help = """\ Do not use Python3.2 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.3", action = "store_true", dest = "no33", default = False, help = """\ Do not use Python3.3 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.4", action = "store_true", dest = "no34", default = False, help = """\ Do not use Python3.4 even if available on the system. Default is %default.""" ) parser.add_option( "--coverage", action = "store_true", dest = "coverage", default = False, help = """\ Make a coverage analysis, that does not really check. Default is %default.""" ) options, positional_args = parser.parse_args() if positional_args: parser.print_help() sys.exit("\nError, no positional argument allowed.") # Go its own directory, to have it easy with path knowledge. os.chdir(os.path.dirname(os.path.abspath(__file__))) os.chdir("..") if options.coverage and os.path.exists(".coverage"): os.unlink(".coverage") # Add the local bin directory to search path start. os.environ["PATH"] = \ os.path.join( os.getcwd(), "bin" ) + \ os.pathsep + \ os.environ["PATH"] def checkExecutableCommand(command): """ Check if a command is executable. """ # Many cases, pylint: disable=R0911,R0912 # Do respect given options to disable specific Python versions if command == "python2.6" and options.no26: return False if command == "python2.7" and options.no27: return False if command == "python3.2" and options.no32: return False if command == "python3.3" and options.no33: return False if command == "python3.4" and options.no34: return False # Shortcuts for python versions, also needed for Windows as it won't have # the version number in the Python binaries at all. if command == "python2.6" and sys.version_info[0:2] == (2,6): return True if command == "python2.7" and sys.version_info[0:2] == (2,7): return True if command == "python3.2" and sys.version_info[0:2] == (3,2): return True if command == "python3.3" and sys.version_info[0:2] == (3,3): return True if command == "python3.4" and sys.version_info[0:2] == (3,4): return True path = os.environ[ "PATH" ] suffixes = (".exe",) if os.name == "nt" else ("",) for part in path.split(os.pathsep): if not part: continue for suffix in suffixes: if os.path.exists(os.path.join(part, command + suffix)): return True return False def setExtraFlags(where, name, flags): if where is not None: tmp_dir = tempfile.gettempdir() # Try to avoid RAM disk /tmp and use the disk one instead. if tmp_dir == "/tmp" and os.path.exists("/var/tmp"): tmp_dir = "/var/tmp" where = os.path.join(tmp_dir, name, where) if not os.path.exists(where): os.makedirs(where) os.environ[ "NUITKA_EXTRA_OPTIONS" ] = flags + " --output-dir=" + where else: os.environ[ "NUITKA_EXTRA_OPTIONS" ] = flags def executeSubTest(command, hide_output = False): if options.coverage and "search" in command: command = command.replace("search", "coverage") parts = command.split() parts[0] = parts[0].replace('/', os.path.sep) # On Windows, we need to specify Python ourselves. if os.name == "nt": parts.insert(0, sys.executable) print("Run '%s' in '%s'." % (' '.join(parts), os.getcwd())) sys.stdout.flush() if hide_output: result = subprocess.call( parts, stdout = open(os.devnull, 'w') ) else: result = subprocess.call( parts ) if result != 0: sys.exit(result) def execute_tests(where, use_python, flags): # Many cases, pylint: disable=R0912,R0915 print( "Executing test case called %s with CPython %s and extra flags '%s'." % ( where, use_python, flags ) ) if os.name == "nt": if use_python == "python2.6": if sys.version.startswith("2.6"): os.environ[ "PYTHON" ] = sys.executable else: os.environ[ "PYTHON" ] = r"C:\Python26\python.exe" elif use_python == "python2.7": if sys.version.startswith("2.7"): os.environ[ "PYTHON" ] = sys.executable else: os.environ[ "PYTHON" ] = r"C:\Python27\python.exe" elif use_python == "python3.2": if sys.version.startswith("3.2"): os.environ[ "PYTHON" ] = sys.executable else: os.environ[ "PYTHON" ] = r"C:\Python32\python.exe" elif use_python == "python3.3": if sys.version.startswith("3.3"): os.environ[ "PYTHON" ] = sys.executable else: os.environ[ "PYTHON" ] = r"C:\Python33\python.exe" elif use_python == "python3.4": if sys.version.startswith("3.4"): os.environ[ "PYTHON" ] = sys.executable else: os.environ[ "PYTHON" ] = r"C:\Python34\python.exe" else: assert False, use_python else: os.environ[ "PYTHON" ] = use_python if options.basic_tests: print("Running the basic tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "basics", flags) executeSubTest("./tests/basics/run_all.py search") if options.syntax_tests: print("Running the syntax tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "syntax", flags) executeSubTest("./tests/syntax/run_all.py search") if options.program_tests: print("Running the program tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "programs", flags) executeSubTest("./tests/programs/run_all.py search") if options.package_tests: print("Running the package tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "packages", flags) executeSubTest("./tests/packages/run_all.py search") # At least one Debian Jessie, these versions won't have lxml installed, so # don't run them there. Also these won't be very version dependent in their # results. if use_python != "python2.6" and use_python != "python3.2": if options.optimization_tests: print("Running the optimizations tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "optimizations", flags) executeSubTest("./tests/optimizations/run_all.py search") if options.standalone_tests and not options.coverage: print("Running the standalone tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(None, "standalone", flags) executeSubTest("./tests/standalone/run_all.py search") if options.reflection_test and not options.coverage: print("Running the reflection test with options '%s' with %s:" % (flags, use_python)) setExtraFlags(None, "reflected", flags) executeSubTest("./tests/reflected/compile_itself.py search") if not use_python.startswith("python3"): if os.path.exists("./tests/CPython26/run_all.py"): if options.cpython26: print("Running the CPython 2.6 tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "26tests", flags) executeSubTest("./tests/CPython26/run_all.py search") else: print("The CPython2.6 tests are not present, not run.") # Running the Python 2.7 test suite with CPython 2.6 gives little # insight, because "importlib" will not be there and that's it. if use_python != "python2.6": if os.path.exists("./tests/CPython27/run_all.py"): if options.cpython27: print("Running the CPython 2.7 tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "27tests", flags) executeSubTest("./tests/CPython27/run_all.py search") else: print("The CPython2.7 tests are not present, not run.") if "--debug" not in flags: # Not running the Python 3.2 test suite with CPython2.6, as that's about # the same as CPython2.7 and won't have any new insights. if use_python != "python2.6" and \ use_python != "python2.7" or not options.coverage: if os.path.exists("./tests/CPython32/run_all.py"): if options.cpython32: setExtraFlags(where, "32tests", flags) executeSubTest("./tests/CPython32/run_all.py search") else: print("The CPython3.2 tests are not present, not run.") # Running the Python 3.3 test suite only with CPython3.x. if not use_python.startswith("python2"): if os.path.exists("./tests/CPython33/run_all.py"): if options.cpython33: setExtraFlags(where, "33tests", flags) executeSubTest("./tests/CPython33/run_all.py search") else: print("The CPython3.3 tests are not present, not run.") # Running the Python 3.3 test suite only with CPython3.x. if not use_python.startswith("python2"): if os.path.exists("./tests/CPython34/run_all.py"): if options.cpython33: setExtraFlags(where, "34tests", flags) executeSubTest("./tests/CPython34/run_all.py search") else: print("The CPython3.4 tests are not present, not run.") if "NUITKA_EXTRA_OPTIONS" in os.environ: del os.environ[ "NUITKA_EXTRA_OPTIONS" ] assert checkExecutableCommand("python2.6") or \ checkExecutableCommand("python2.7") or \ checkExecutableCommand("python3.2") or \ checkExecutableCommand("python3.3") or \ checkExecutableCommand("python3.4") # Just the quick syntax test, full tests are run later. if checkExecutableCommand("python3.2"): executeSubTest( "bin/nuitka --python-version=3.2 --version", hide_output = True ) if checkExecutableCommand("python2.6"): execute_tests("python2.6-debug", "python2.6", "--debug") else: print("Cannot execute tests with Python 2.6, disabled or not installed.") if checkExecutableCommand("python2.7"): execute_tests("python2.7-debug", "python2.7", "--debug") else: print("Cannot execute tests with Python 2.7, disabled or not installed.") # Temporary measure, the nodebug tests pass, debug doesn't for the CPython3.2 # test suite without unused closure variable removal. if False and checkExecutableCommand("python3.2"): execute_tests("python3.2-debug", "python3.2", "--debug") else: print("Cannot execute tests with Python 3.2, disabled or not installed.") if checkExecutableCommand("python2.6"): execute_tests("python2.6-nodebug", "python2.6", "") else: print("Cannot execute tests with Python 2.6, disabled or not installed.") if checkExecutableCommand("python2.7"): execute_tests("python2.7-nodebug", "python2.7", "") else: print("Cannot execute tests with Python 2.7, disabled or not installed.") if checkExecutableCommand("python3.2"): execute_tests("python3.2-nodebug", "python3.2", "") else: print("Cannot execute tests with Python 3.2, disabled or not installed.") if checkExecutableCommand("python3.3"): execute_tests("python3.2-nodebug", "python3.3", "") else: print("Cannot execute tests with Python 3.3, disabled or not installed.") if checkExecutableCommand("python3.4"): execute_tests("python3.2-nodebug", "python3.4", "") else: print("Cannot execute tests with Python 3.4, disabled or not installed.") if options.coverage: def copyToGlobalCoverageData(source, target): coverage_dir = os.environ.get("COVERAGE_DIR", None) if coverage_dir is None: return assert 0 == subprocess.call( ( "C:\\MinGW\\msys\\1.0\\bin\\scp.exe" if os.name == "nt" else "scp", source, os.path.join( coverage_dir, target ) ) ) if os.name == "nt": suffix = "win" else: import platform suffix = platform.uname()[0] + '.' + platform.uname()[4] with open("data.coverage", "w") as data_file: source_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__))) data_file.write("NUITKA_SOURCE_DIR='%s'\n" % source_dir) nuitka_id = subprocess.check_output( "cd '%s'; git rev-parse HEAD" % source_dir, shell = True ) nuitka_id = nuitka_id.strip() if sys.version_info > (3,): nuitka_id = nuitka_id.decode() data_file.write("NUITKA_COMMIT='%s'\n" % nuitka_id) copyToGlobalCoverageData("data.coverage", "data.coverage." + suffix) copyToGlobalCoverageData(".coverage", ".coverage." + suffix) print("OK.") Nuitka-0.5.18.1/nuitka/0000755000175000017500000000000012651072347014733 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/nodes/0000755000175000017500000000000012651072347016043 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/nodes/ImportNodes.py0000644000175000017500000003512612651071040020654 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes related to importing modules or names. Normally imports are mostly relatively static, but Nuitka also attempts to cover the uses of "__import__" built-in and other import techniques, that allow dynamic values. If other optimizations make it possible to predict these, the compiler can go deeper that what it normally could. The import expression node can recurse. An "__import__" built-in may be converted to it, once the module name becomes a compile time constant. """ from logging import warning from nuitka.importing.Importing import ( findModule, getModuleNameAndKindFromFilename ) from nuitka.importing.Recursion import decideRecursion, recurseTo from nuitka.importing.Whitelisting import getModuleWhiteList from nuitka.utils import Utils from .ConstantRefNodes import ExpressionConstantRef from .NodeBases import ( ExpressionChildrenHavingBase, ExpressionMixin, NodeBase, StatementChildrenHavingBase ) class ExpressionImportModule(NodeBase, ExpressionMixin): kind = "EXPRESSION_IMPORT_MODULE" # Set of modules, that we failed to import, and gave warning to the user # about it. _warned_about = set() def __init__(self, module_name, import_list, level, source_ref): assert type(module_name) is str, type(module_name) NodeBase.__init__( self, source_ref = source_ref ) self.module_name = module_name self.import_list = import_list self.level = level self.module = None self.attempted_recurse = False self.found_modules = () def getDetails(self): return { "module_name" : self.module_name, "level" : self.level, "import_list" : self.import_list } def getModuleName(self): return self.module_name def getImportList(self): return self.import_list def getLevel(self): if self.level == 0: if self.source_ref.getFutureSpec().isAbsoluteImport(): return 0 else: return -1 else: return self.level def getModule(self): return self.module def setModule(self, module): # Modules have no parent. assert module.parent is None self.module = module def _consider(self, constraint_collection, module_filename, module_package): assert module_package is None or \ (type(module_package) is str and module_package != "") module_filename = Utils.normpath(module_filename) module_name, module_kind = getModuleNameAndKindFromFilename(module_filename) if module_kind is not None: decision, reason = decideRecursion( module_filename = module_filename, module_name = module_name, module_package = module_package, module_kind = module_kind ) if decision: module_relpath = Utils.relpath(module_filename) imported_module, added_flag = recurseTo( module_package = module_package, module_filename = module_filename, module_relpath = module_relpath, module_kind = module_kind, reason = reason ) if added_flag: constraint_collection.signalChange( "new_code", imported_module.getSourceReference(), "Recursed to module." ) return imported_module elif decision is None and module_kind == "py": if module_package is None: module_fullpath = module_name else: module_fullpath = module_package + '.' + module_name if module_filename not in self._warned_about and \ module_fullpath not in getModuleWhiteList(): self._warned_about.add(module_filename) warning( """\ Not recursing to '%(full_path)s' (%(filename)s), please specify \ --recurse-none (do not warn), \ --recurse-all (recurse to all), \ --recurse-not-to=%(full_path)s (ignore it), \ --recurse-to=%(full_path)s (recurse to it) to change.""" % { "full_path" : module_fullpath, "filename" : module_filename } ) def _attemptRecursion(self, constraint_collection): assert self.getModule() is None parent_module = self.getParentModule() if parent_module.isCompiledPythonPackage(): parent_package = parent_module.getFullName() else: parent_package = self.getParentModule().getPackage() module_package, module_filename, _finding = findModule( importing = self, module_name = self.getModuleName(), parent_package = parent_package, level = self.getLevel(), warn = True ) if module_filename is not None: imported_module = self._consider( constraint_collection = constraint_collection, module_filename = module_filename, module_package = module_package ) if imported_module is not None: self.setModule(imported_module) self.found_modules = [] import_list = self.getImportList() if import_list and imported_module.isCompiledPythonPackage(): for import_item in import_list: if import_item == '*': continue module_package, module_filename, _finding = findModule( importing = self, module_name = import_item, parent_package = imported_module.getFullName(), level = -1, warn = False ) if module_filename is not None: sub_imported_module = self._consider( constraint_collection = constraint_collection, module_filename = module_filename, module_package = module_package ) if sub_imported_module is not None: self.found_modules.append(sub_imported_module) def computeExpression(self, constraint_collection): # Attempt to recurse if not already done. if not self.attempted_recurse: self._attemptRecursion( constraint_collection = constraint_collection ) self.attempted_recurse = True if self.getModule() is not None: from nuitka.ModuleRegistry import addUsedModule addUsedModule(self.getModule()) for found_module in self.found_modules: addUsedModule(found_module) # When a module is recursed to and included, we know it won't raise, # right? But even if you import, that successful import may still raise # and we don't know how to check yet. constraint_collection.onExceptionRaiseExit( BaseException ) return self, None, None class ExpressionImportModuleHard(NodeBase, ExpressionMixin): """ Hard code import, e.g. of "sys" module as done in Python mechanics. """ kind = "EXPRESSION_IMPORT_MODULE_HARD" def __init__(self, module_name, import_name, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.module_name = module_name self.import_name = import_name def getDetails(self): return { "module_name" : self.module_name, "import_name" : self.import_name } def getModuleName(self): return self.module_name def getImportName(self): return self.import_name def computeExpression(self, constraint_collection): # TODO: May return a module reference of some sort in the future with # embedded modules. return self, None, None def mayHaveSideEffects(self): if self.module_name == "sys" and self.import_name == "stdout": return False elif self.module_name == "__future__": return False else: return True def mayRaiseException(self, exception_type): return self.mayHaveSideEffects() class ExpressionBuiltinImport(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_IMPORT" named_children = ( "import_name", "globals", "locals", "fromlist", "level" ) def __init__(self, name, import_globals, import_locals, fromlist, level, source_ref): if fromlist is None: fromlist = ExpressionConstantRef( constant = (), source_ref = source_ref ) if level is None: level = 0 if source_ref.getFutureSpec().isAbsoluteImport() else -1 level = ExpressionConstantRef( constant = level, source_ref = source_ref ) ExpressionChildrenHavingBase.__init__( self, values = { "import_name" : name, "globals" : import_globals, "locals" : import_locals, "fromlist" : fromlist, "level" : level }, source_ref = source_ref ) getImportName = ExpressionChildrenHavingBase.childGetter("import_name") getFromList = ExpressionChildrenHavingBase.childGetter("fromlist") getGlobals = ExpressionChildrenHavingBase.childGetter("globals") getLocals = ExpressionChildrenHavingBase.childGetter("locals") getLevel = ExpressionChildrenHavingBase.childGetter("level") def computeExpression(self, constraint_collection): module_name = self.getImportName() fromlist = self.getFromList() level = self.getLevel() # TODO: In fact, if the module is not a package, we don't have to insist # on the "fromlist" that much, but normally it's not used for anything # but packages, so it will be rare. if module_name.isExpressionConstantRef() and \ fromlist.isExpressionConstantRef() and \ level.isExpressionConstantRef(): if module_name.isStringConstant(): new_node = ExpressionImportModule( module_name = module_name.getConstant(), import_list = fromlist.getConstant(), level = level.getConstant(), source_ref = self.getSourceReference() ) # Importing may raise an exception obviously. constraint_collection.onExceptionRaiseExit(BaseException) return ( new_node, "new_import", "Replaced __import__ call with module import expression." ) else: # Non-strings is going to raise an error. new_node, change_tags, message = constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : __import__(module_name.getConstant()), description = "Replaced __import__ call with non-string module name argument." ) # Must fail, must not go on when it doesn't. assert change_tags == "new_raise" return new_node, change_tags, message # Importing may raise an exception obviously. constraint_collection.onExceptionRaiseExit(BaseException) # TODO: May return a module or module variable reference of some sort in # the future with embedded modules. return self, None, None class StatementImportStar(StatementChildrenHavingBase): kind = "STATEMENT_IMPORT_STAR" named_children = ("module",) def __init__(self, module_import, source_ref): StatementChildrenHavingBase.__init__( self, values = { "module" : module_import }, source_ref = source_ref ) getModule = StatementChildrenHavingBase.childGetter("module") def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getModule()) # Need to invalidate everything, and everything could be assigned to # something else now. constraint_collection.removeAllKnowledge() return self, None, None class ExpressionImportName(ExpressionChildrenHavingBase): kind = "EXPRESSION_IMPORT_NAME" named_children = ( "module", ) def __init__(self, module, import_name, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "module" : module }, source_ref = source_ref ) self.import_name = import_name assert module is not None def getImportName(self): return self.import_name def getDetails(self): return { "import_name" : self.getImportName() } def getDetail(self): return "import %s from %s" % ( self.getImportName(), self.getModule().getModuleName() ) getModule = ExpressionChildrenHavingBase.childGetter("module") def computeExpression(self, constraint_collection): # TODO: May return a module or module variable reference of some sort in # the future with embedded modules. return self, None, None Nuitka-0.5.18.1/nuitka/nodes/OperatorNodes.py0000644000175000017500000002532112651071040021171 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for unary and binary operations. No short-circuit involved, boolean 'not' is an unary operation like '-' is, no real difference. """ import math from nuitka import PythonOperators from .NodeBases import ExpressionChildrenHavingBase class ExpressionOperationBase(ExpressionChildrenHavingBase): inplace_suspect = False def __init__(self, operator, simulator, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = values, source_ref = source_ref ) self.operator = operator self.simulator = simulator def markAsInplaceSuspect(self): self.inplace_suspect = True def unmarkAsInplaceSuspect(self): self.inplace_suspect = False def isInplaceSuspect(self): return self.inplace_suspect def getDetail(self): return self.operator def getDetails(self): return { "operator" : self.operator } def getOperator(self): return self.operator def getSimulator(self): return self.simulator def isKnownToBeIterable(self, count): # TODO: Could be true, if the arguments said so return None class ExpressionOperationBinary(ExpressionOperationBase): kind = "EXPRESSION_OPERATION_BINARY" named_children = ("left", "right") nice_children = tuple(child_name + " operand" for child_name in named_children) def __init__(self, operator, left, right, source_ref): assert left.isExpression() and right.isExpression, (left, right) ExpressionOperationBase.__init__( self, operator = operator, simulator = PythonOperators.binary_operator_functions[ operator ], values = { "left" : left, "right" : right }, source_ref = source_ref ) def computeExpression(self, constraint_collection): # This is using many returns based on many conditions, # pylint: disable=R0912 # TODO: May go down to MemoryError for compile time constant overflow # ones. constraint_collection.onExceptionRaiseExit(BaseException) operator = self.getOperator() operands = self.getOperands() left, right = operands if left.isCompileTimeConstant() and right.isCompileTimeConstant(): left_value = left.getCompileTimeConstant() right_value = right.getCompileTimeConstant() if operator == "Mult" and right.isNumberConstant(): iter_length = left.getIterationLength() if iter_length is not None: if iter_length * right_value > 256: return self, None, None if left.isNumberConstant(): if left.isIndexConstant() and right.isIndexConstant(): # Estimate with logarithm, if the result of number # calculations is computable with acceptable effort, # otherwise, we will have to do it at runtime. if left_value != 0 and right_value != 0: if math.log10(abs(left_value)) + math.log10(abs(right_value)) > 20: return self, None, None elif operator == "Mult" and left.isNumberConstant(): iter_length = right.getIterationLength() if iter_length is not None: if iter_length * left_value > 256: return self, None, None elif operator == "Add" and \ left.isKnownToBeIterable(None) and \ right.isKnownToBeIterable(None): iter_length = left.getIterationLength() + \ right.getIterationLength() if iter_length > 256: return self, None, None return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( left_value, right_value ), description = "Operator '%s' with constant arguments." % operator ) else: # The value of these nodes escaped and could change its contents. constraint_collection.removeKnowledge(left) constraint_collection.removeKnowledge(right) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) return self, None, None def getOperands(self): return (self.getLeft(), self.getRight()) getLeft = ExpressionChildrenHavingBase.childGetter("left") getRight = ExpressionChildrenHavingBase.childGetter("right") class ExpressionOperationUnary(ExpressionOperationBase): kind = "EXPRESSION_OPERATION_UNARY" named_children = ("operand",) def __init__(self, operator, operand, source_ref): assert operand.isExpression(), operand ExpressionOperationBase.__init__( self, operator = operator, simulator = PythonOperators.unary_operator_functions[ operator ], values = { "operand" : operand }, source_ref = source_ref ) def computeExpression(self, constraint_collection): operator = self.getOperator() operand = self.getOperand() if operand.isCompileTimeConstant(): operand_value = operand.getCompileTimeConstant() return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( operand_value, ), description = "Operator '%s' with constant argument." % operator ) else: # TODO: May go down to MemoryError for compile time constant overflow # ones. constraint_collection.onExceptionRaiseExit(BaseException) # The value of that node escapes and could change its contents. constraint_collection.removeKnowledge(operand) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) return self, None, None getOperand = ExpressionChildrenHavingBase.childGetter("operand") def getOperands(self): return (self.getOperand(),) @staticmethod def isExpressionOperationUnary(): return True class ExpressionOperationNOT(ExpressionOperationUnary): kind = "EXPRESSION_OPERATION_NOT" def __init__(self, operand, source_ref): ExpressionOperationUnary.__init__( self, operator = "Not", operand = operand, source_ref = source_ref ) def getDetails(self): return {} def computeExpression(self, constraint_collection): return self.getOperand().computeExpressionOperationNot( not_node = self, constraint_collection = constraint_collection ) def mayRaiseException(self, exception_type): return self.getOperand().mayRaiseExceptionBool(exception_type) def mayRaiseExceptionBool(self, exception_type): return self.getOperand().mayRaiseExceptionBool(exception_type) def getTruthValue(self): result = self.getOperand().getTruthValue() # Need to invert the truth value of operand of course here. return None if result is None else not result def mayHaveSideEffects(self): operand = self.getOperand() if operand.mayHaveSideEffects(): return True return operand.mayHaveSideEffectsBool() def mayHaveSideEffectsBool(self): return self.getOperand().mayHaveSideEffectsBool() def extractSideEffects(self): operand = self.getOperand() # TODO: Find the common ground of these, and make it an expression # method. if operand.isExpressionMakeSequence(): return operand.extractSideEffects() if operand.isExpressionMakeDict(): return operand.extractSideEffects() return (self,) class ExpressionOperationBinaryInplace(ExpressionOperationBinary): kind = "EXPRESSION_OPERATION_BINARY_INPLACE" def __init__(self, operator, left, right, source_ref): ExpressionOperationBinary.__init__( self, operator = operator, left = left, right = right, source_ref = source_ref ) @staticmethod def isExpressionOperationBinary(): return True def computeExpression(self, constraint_collection): # In-place operation requires extra care to avoid corruption of # values. left = self.getLeft() right = self.getRight() if left.isCompileTimeConstant(): # Then we made a mistake currently. assert not left.isMutable(), self source_ref = self.getSourceReference() result = ExpressionOperationBinary( left = left, right = right, operator = self.getOperator()[1:], source_ref = source_ref ) constraint_collection.signalChange( tags = "new_expression", source_ref = source_ref, message = """\ Lowered in-place binary operation of compile time constant to binary operation.""" ) return result.computeExpression(constraint_collection) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) # The value of these nodes escaped and could change its contents. constraint_collection.removeKnowledge(left) constraint_collection.removeKnowledge(right) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) return self, None, None Nuitka-0.5.18.1/nuitka/nodes/ConditionalNodes.py0000644000175000017500000005552412651071040021651 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Conditional nodes. These is the conditional expression '(a if b else c)' and the conditional statement, 'if a: ... else: ...' and there is no 'elif', because that is expressed via nesting of conditional statements. """ from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinBool from nuitka.optimizations.TraceCollections import ConstraintCollectionBranch from .Checkers import checkStatementsSequenceOrNone from .NodeBases import ExpressionChildrenHavingBase, StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, wrapExpressionWithNodeSideEffects, wrapStatementWithSideEffects ) class ExpressionConditional(ExpressionChildrenHavingBase): kind = "EXPRESSION_CONDITIONAL" named_children = ( "condition", "expression_yes", "expression_no" ) def __init__(self, condition, expression_yes, expression_no, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "condition" : condition, "expression_yes" : expression_yes, "expression_no" : expression_no }, source_ref = source_ref ) def getBranches(self): return ( self.getExpressionYes(), self.getExpressionNo() ) getExpressionYes = ExpressionChildrenHavingBase.childGetter( "expression_yes" ) getExpressionNo = ExpressionChildrenHavingBase.childGetter( "expression_no" ) getCondition = ExpressionChildrenHavingBase.childGetter( "condition" ) def computeExpressionRaw(self, constraint_collection): # Query the truth value after the expression is evaluated, once it is # evaluated in onExpression, it is known. constraint_collection.onExpression( expression = self.getCondition() ) condition = self.getCondition() condition_may_raise = condition.mayRaiseException(BaseException) if condition_may_raise: constraint_collection.onExceptionRaiseExit( BaseException ) # No need to look any further, if the condition raises, the branches do # not matter at all. if condition.willRaiseException(BaseException): return condition, "new_raise", """\ Conditional expression already raises implicitly in condition, removing \ branches.""" if not condition_may_raise and condition.mayRaiseExceptionBool(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) # Decide this based on truth value of condition. truth_value = condition.getTruthValue() # TODO: We now know that condition evaluates to true for the yes branch # and to not true for no branch, the branch should know that. yes_branch = self.getExpressionYes() # Continue to execute for yes branch unless we know it's not going to be # relevant. if truth_value is not False: branch_yes_collection = ConstraintCollectionBranch( parent = constraint_collection, name = "conditional expression yes branch" ) branch_yes_collection.computeBranch( branch = yes_branch ) # May have just gone away, so fetch it again. yes_branch = self.getExpressionYes() # If it's aborting, it doesn't contribute to merging. if yes_branch.willRaiseException(BaseException): branch_yes_collection = None else: branch_yes_collection = None no_branch = self.getExpressionNo() # Continue to execute for yes branch. if truth_value is not True: branch_no_collection = ConstraintCollectionBranch( parent = constraint_collection, name = "conditional expression no branch" ) branch_no_collection.computeBranch( branch = no_branch ) # May have just gone away, so fetch it again. no_branch = self.getExpressionNo() # If it's aborting, it doesn't contribute to merging. if no_branch.willRaiseException(BaseException): branch_no_collection = None else: branch_no_collection = None # Merge into parent execution. constraint_collection.mergeBranches( branch_yes_collection, branch_no_collection ) if truth_value is True: return ( wrapExpressionWithNodeSideEffects( new_node = self.getExpressionYes(), old_node = condition ), "new_expression", "Conditional expression predicted to yes case" ) elif truth_value is False: return ( wrapExpressionWithNodeSideEffects( new_node = self.getExpressionNo(), old_node = condition ), "new_expression", "Conditional expression predicted to no case" ) else: return self, None, None def mayHaveSideEffectsBool(self): if self.getCondition().mayHaveSideEffectsBool(): return True if self.getExpressionYes().mayHaveSideEffectsBool(): return True if self.getExpressionNo().mayHaveSideEffectsBool(): return True return False def mayRaiseException(self, exception_type): condition = self.getCondition() if condition.mayRaiseException(exception_type): return True if condition.mayRaiseExceptionBool(exception_type): return True yes_branch = self.getExpressionYes() # Handle branches that became empty behind our back if yes_branch is not None and \ yes_branch.mayRaiseException(exception_type): return True no_branch = self.getExpressionNo() # Handle branches that became empty behind our back if no_branch is not None and \ no_branch.mayRaiseException(exception_type): return True return False def mayRaiseExceptionBool(self, exception_type): if self.getCondition().mayRaiseExceptionBool(exception_type): return True if self.getExpressionYes().mayRaiseExceptionBool(exception_type): return True if self.getExpressionNo().mayRaiseExceptionBool(exception_type): return True return False class ExpressionConditionalBoolBase(ExpressionChildrenHavingBase): named_children = ( "left", "right" ) def __init__(self, left, right, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "left" : left, "right" : right, }, source_ref = source_ref ) getLeft = ExpressionChildrenHavingBase.childGetter( "left" ) getRight = ExpressionChildrenHavingBase.childGetter( "right" ) def computeExpressionRaw(self, constraint_collection): # Query the truth value after the expression is evaluated, once it is # evaluated in onExpression, it is known. constraint_collection.onExpression( expression = self.getLeft() ) left = self.getLeft() left_may_raise = left.mayRaiseException(BaseException) if left_may_raise: constraint_collection.onExceptionRaiseExit( BaseException ) # No need to look any further, if the condition raises, the branches do # not matter at all. if left.willRaiseException(BaseException): return left, "new_raise", """\ Conditional %s statements already raises implicitly in condition, removing \ branches.""" % self.conditional_kind if not left_may_raise and left.mayRaiseExceptionBool(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) # Decide this based on truth value of condition. truth_value = left.getTruthValue() truth_value_use_left = self.conditional_kind == "or" truth_value_use_right = not truth_value_use_left right = self.getRight() # Continue to execute for yes branch unless we know it's not going to be # relevant. if truth_value is not truth_value_use_left: # TODO: We now know that left evaluates and we should tell the # branch that. branch_yes_collection = ConstraintCollectionBranch( parent = constraint_collection, name = "boolean %s right branch" % self.conditional_kind ) branch_yes_collection.computeBranch( branch = right ) # May have just gone away, so fetch it again. right = self.getRight() # If it's aborting, it doesn't contribute to merging. if right.willRaiseException(BaseException): branch_yes_collection = None else: branch_yes_collection = None if branch_yes_collection: # Merge into parent execution. constraint_collection.mergeBranches( branch_yes_collection, None ) if truth_value is truth_value_use_left: return ( left, "new_expression", "Conditional '%s' expression predicted to left value." % self.conditional_kind ) elif truth_value is truth_value_use_right: return ( wrapExpressionWithNodeSideEffects( new_node = right, old_node = left ), "new_expression", "Conditional '%s' expression predicted right value." % self.conditional_kind ) else: return self, None, None def mayRaiseException(self, exception_type): left = self.getLeft() if left.mayRaiseException(exception_type): return True if left.mayRaiseExceptionBool(exception_type): return True right = self.getRight() if right.mayRaiseException(exception_type): return True return False def mayRaiseExceptionBool(self, exception_type): if self.getLeft().mayRaiseExceptionBool(exception_type): return True if self.getRight().mayRaiseExceptionBool(exception_type): return True return False def mayHaveSideEffectsBool(self): if self.getLeft().mayHaveSideEffectsBool(): return True if self.getRight().mayHaveSideEffectsBool(): return True return False class ExpressionConditionalOR(ExpressionConditionalBoolBase): kind = "EXPRESSION_CONDITIONAL_OR" def __init__(self, left, right, source_ref): ExpressionConditionalBoolBase.__init__( self, left = left, right = right, source_ref = source_ref ) self.conditional_kind = "or" class ExpressionConditionalAND(ExpressionConditionalBoolBase): kind = "EXPRESSION_CONDITIONAL_AND" def __init__(self, left, right, source_ref): ExpressionConditionalBoolBase.__init__( self, left = left, right = right, source_ref = source_ref ) self.conditional_kind = "and" class StatementConditional(StatementChildrenHavingBase): kind = "STATEMENT_CONDITIONAL" named_children = ( "condition", "yes_branch", "no_branch" ) checkers = { "yes_branch" : checkStatementsSequenceOrNone, "no_branch" : checkStatementsSequenceOrNone, } def __init__(self, condition, yes_branch, no_branch, source_ref): StatementChildrenHavingBase.__init__( self, values = { "condition" : condition, "yes_branch" : yes_branch, "no_branch" : no_branch }, source_ref = source_ref ) getCondition = StatementChildrenHavingBase.childGetter("condition") getBranchYes = StatementChildrenHavingBase.childGetter("yes_branch") setBranchYes = StatementChildrenHavingBase.childSetter("yes_branch") getBranchNo = StatementChildrenHavingBase.childGetter("no_branch") setBranchNo = StatementChildrenHavingBase.childSetter("no_branch") def isStatementAborting(self): yes_branch = self.getBranchYes() if yes_branch is not None: if yes_branch.isStatementAborting(): no_branch = self.getBranchNo() if no_branch is not None: return no_branch.isStatementAborting() else: return False else: return False else: return False def mayRaiseException(self, exception_type): condition = self.getCondition() if condition.mayRaiseException(exception_type): return True if condition.mayRaiseExceptionBool(exception_type): return True yes_branch = self.getBranchYes() # Handle branches that became empty behind our back if yes_branch is not None and \ yes_branch.mayRaiseException(exception_type): return True no_branch = self.getBranchNo() # Handle branches that became empty behind our back if no_branch is not None and \ no_branch.mayRaiseException(exception_type): return True return False def needsFrame(self): condition = self.getCondition() if condition.mayRaiseException(BaseException): return True if condition.mayRaiseExceptionBool(BaseException): return True yes_branch = self.getBranchYes() # Handle branches that became empty behind our back if yes_branch is not None and \ yes_branch.needsFrame(): return True no_branch = self.getBranchNo() # Handle branches that became empty behind our back if no_branch is not None and \ no_branch.needsFrame(): return True return False def computeStatement(self, constraint_collection): # This is rather complex stuff, pylint: disable=R0912,R0915 constraint_collection.onExpression( expression = self.getCondition() ) condition = self.getCondition() condition_may_raise = condition.mayRaiseException(BaseException) if condition_may_raise: constraint_collection.onExceptionRaiseExit( BaseException ) # No need to look any further, if the condition raises, the branches do # not matter at all. if condition.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = condition, node = self ) return result, "new_raise", """\ Conditional statements already raises implicitly in condition, removing \ branches.""" if not condition_may_raise and condition.mayRaiseExceptionBool(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) # Query the truth value after the expression is evaluated, once it is # evaluated in onExpression, it is known. truth_value = condition.getTruthValue() # TODO: We now know that condition evaluates to true for the yes branch # and to not true for no branch, the branch collection should know that. yes_branch = self.getBranchYes() no_branch = self.getBranchNo() # Handle branches that became empty behind our back. if yes_branch is not None: if not yes_branch.getStatements(): yes_branch = None if no_branch is not None: if not no_branch.getStatements(): no_branch = None # Consider to not remove branches that we know won't be taken. if yes_branch is not None and truth_value is False: constraint_collection.signalChange( tags = "new_statements", source_ref = yes_branch.source_ref, message = "Removed conditional branch not taken due to false condition value." ) self.setBranchYes(None) yes_branch = None if no_branch is not None and truth_value is True: constraint_collection.signalChange( tags = "new_statements", source_ref = no_branch.source_ref, message = "Removed 'else' branch not taken due to true condition value." ) self.setBranchNo(None) no_branch = None # Continue to execute for yes branch unless we know it's not going to be # relevant. if yes_branch is not None: branch_yes_collection = ConstraintCollectionBranch( parent = constraint_collection, name = "conditional yes branch", ) branch_yes_collection.computeBranch( branch = yes_branch ) # May have just gone away, so fetch it again. yes_branch = self.getBranchYes() # If it's aborting, it doesn't contribute to merging. if yes_branch is None or yes_branch.isStatementAborting(): branch_yes_collection = None else: branch_yes_collection = None # Continue to execute for yes branch. if no_branch is not None: branch_no_collection = ConstraintCollectionBranch( parent = constraint_collection, name = "conditional no branch" ) branch_no_collection.computeBranch( branch = no_branch ) # May have just gone away, so fetch it again. no_branch = self.getBranchNo() # If it's aborting, it doesn't contribute to merging. if no_branch is None or no_branch.isStatementAborting(): branch_no_collection = None else: branch_no_collection = None # Merge into parent execution. constraint_collection.mergeBranches( branch_yes_collection, branch_no_collection ) # Both branches may have become empty, which case, the statement needs # not remain. if yes_branch is None and no_branch is None: # Need to keep the boolean check. if truth_value is None: condition = ExpressionBuiltinBool( value = condition, source_ref = condition.getSourceReference() ) if condition.mayHaveSideEffects(): # With both branches eliminated, the condition remains as a side # effect. result = makeStatementExpressionOnlyReplacementNode( expression = condition, node = self ) return result, "new_statements", """\ Both branches have no effect, reduced to evaluate condition.""" else: return None, "new_statements", """\ Removed conditional statement without effect.""" # Note: Checking the condition late, so that the surviving branch got # processed already. Returning without doing that, will corrupt the SSA # results. TODO: Could pretend the other branch didn't exist to save # complexity the merging of processing. if truth_value is not None: if truth_value is True: choice = "true" new_statement = self.getBranchYes() else: choice = "false" new_statement = self.getBranchNo() new_statement = wrapStatementWithSideEffects( new_node = new_statement, old_node = condition, allow_none = True # surviving branch may empty ) return new_statement, "new_statements", """\ Condition for branch was predicted to be always %s.""" % choice # If there is no "yes" branch, remove that. Maybe a bad idea though. if yes_branch is None: # Would be eliminated already, if there wasn't any "no" branch # either. assert no_branch is not None from .OperatorNodes import ExpressionOperationNOT new_statement = StatementConditional( condition = ExpressionOperationNOT( operand = condition, source_ref = condition.getSourceReference() ), yes_branch = no_branch, no_branch = None, source_ref = self.getSourceReference() ) return new_statement, "new_statements", """\ Empty 'yes' branch for conditional statement treated with inverted condition check.""" return self, None, None def mayReturn(self): yes_branch = self.getBranchYes() if yes_branch is not None and yes_branch.mayReturn(): return True no_branch = self.getBranchNo() if no_branch is not None and no_branch.mayReturn(): return True return False def mayBreak(self): yes_branch = self.getBranchYes() if yes_branch is not None and yes_branch.mayBreak(): return True no_branch = self.getBranchNo() if no_branch is not None and no_branch.mayBreak(): return True return False def mayContinue(self): yes_branch = self.getBranchYes() if yes_branch is not None and yes_branch.mayContinue(): return True no_branch = self.getBranchNo() if no_branch is not None and no_branch.mayContinue(): return True return False Nuitka-0.5.18.1/nuitka/nodes/BuiltinFormatNodes.py0000644000175000017500000000372412651071040022160 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Format nodes bin/oct/hex. These will most often be used for outputs, and the hope is, the type prediction or the result prediction will help to be smarter, but generally these should not be that much about performance critical. """ from nuitka.optimizations import BuiltinOptimization from .NodeBases import ExpressionBuiltinSingleArgBase class ExpressionBuiltinBin(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_BIN" builtin_spec = BuiltinOptimization.builtin_bin_spec class ExpressionBuiltinOct(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_OCT" builtin_spec = BuiltinOptimization.builtin_oct_spec class ExpressionBuiltinHex(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_HEX" builtin_spec = BuiltinOptimization.builtin_hex_spec class ExpressionBuiltinId(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ID" builtin_spec = BuiltinOptimization.builtin_id_spec def computeExpression(self, constraint_collection): # Note: Quite impossible to predict the pointer value or anything, but # we know the result will be a long. return self, None, None def getIntValue(self): return self Nuitka-0.5.18.1/nuitka/nodes/YieldNodes.py0000644000175000017500000000777212651071040020456 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Yield node. The yield node returns to the caller of the generator and therefore may execute absolutely arbitrary code, from the point of view of this code. It then returns something, which may often be 'None', but doesn't have to be. Often it will be used as a statement, which should also be reflected in a dedicated node. """ from .NodeBases import ExpressionChildrenHavingBase class ExpressionYield(ExpressionChildrenHavingBase): """ Yielding an expression. Typical code: yield expression Can only happen in a generator. Kind of explicitly suspends and resumes the execution. The user may inject any kind of exception or give any return value. The value of "None" is the most common though, esp. if it's not used. """ kind = "EXPRESSION_YIELD" named_children = ("expression",) def __init__(self, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.exception_preserving = False def markAsExceptionPreserving(self): self.exception_preserving = True def isExceptionPreserving(self): return self.exception_preserving getExpression = ExpressionChildrenHavingBase.childGetter("expression") def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getExpression()) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) # Nothing possible really here. return self, None, None class ExpressionYieldFrom(ExpressionChildrenHavingBase): """ Yielding from an expression. Typical code: yield from expression (Python3) Can only happen in a generator and only in Python3. Similar to yield, but implies a loop and exception propagation to the yield from generator if such. Kind of explicitly suspends and resumes the execution. The user may inject any kind of exception or give any return value. Having a return value is what makes Python3 generators special, and with yield from, that value is the expression result. """ kind = "EXPRESSION_YIELD_FROM" named_children = ("expression",) def __init__(self, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.exception_preserving = False def markAsExceptionPreserving(self): self.exception_preserving = True def isExceptionPreserving(self): return self.exception_preserving getExpression = ExpressionChildrenHavingBase.childGetter("expression") def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getExpression()) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) # Nothing possible really here. return self, None, None Nuitka-0.5.18.1/nuitka/nodes/BuiltinRefNodes.py0000644000175000017500000001704612651071040021446 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tree nodes for built-in references. There is 2 major types of built-in references. One is the values from built-ins, the other is built-in exceptions. They work differently and mean different things, but they have similar origin, that is, access to variables only ever read. """ from nuitka.Builtins import ( builtin_anon_names, builtin_exception_names, builtin_exception_values, builtin_names ) from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ConstantRefNodes import ExpressionConstantRef from .ExceptionNodes import ExpressionBuiltinMakeException from .NodeBases import CompileTimeConstantExpressionMixin, NodeBase class ExpressionBuiltinRefBase(CompileTimeConstantExpressionMixin, NodeBase): def __init__(self, builtin_name, source_ref): NodeBase.__init__(self, source_ref = source_ref) CompileTimeConstantExpressionMixin.__init__(self) self.builtin_name = builtin_name def getDetails(self): return { "builtin_name" : self.builtin_name } def getBuiltinName(self): return self.builtin_name def isKnownToBeHashable(self): return True def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False def getStrValue(self): return ExpressionConstantRef( constant = str(self.getCompileTimeConstant()), user_provided = True, source_ref = self.getSourceReference(), ) class ExpressionBuiltinRef(ExpressionBuiltinRefBase): kind = "EXPRESSION_BUILTIN_REF" def __init__(self, builtin_name, source_ref): assert builtin_name in builtin_names, builtin_name ExpressionBuiltinRefBase.__init__( self, builtin_name = builtin_name, source_ref = source_ref ) def isCompileTimeConstant(self): return True def getCompileTimeConstant(self): return __builtins__[ self.builtin_name ] def computeExpression(self, constraint_collection): quick_names = { "None" : None, "True" : True, "False" : False, "__debug__" : __debug__, "Ellipsis" : Ellipsis, } if self.builtin_name in quick_names: new_node = ExpressionConstantRef( constant = quick_names[self.builtin_name], source_ref = self.getSourceReference() ) return new_node, "new_constant", """\ Built-in constant '%s' resolved.""" % self.builtin_name return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): from nuitka.optimizations.OptimizeBuiltinCalls import computeBuiltinCall # Anything may happen. On next pass, if replaced, we might be better # but not now. constraint_collection.onExceptionRaiseExit(BaseException) new_node, tags, message = computeBuiltinCall( call_node = call_node, called = self ) if self.builtin_name in ("dir", "eval", "exec", "execfile", "locals", "vars"): # Just inform the collection that all has escaped. constraint_collection.onLocalsUsage() return new_node, tags, message def getStringValue(self): return repr(self.getCompileTimeConstant()) def isKnownToBeIterable(self, count): # TODO: Why yes, some may be, could be told here. return None class ExpressionBuiltinOriginalRef(ExpressionBuiltinRef): kind = "EXPRESSION_BUILTIN_ORIGINAL_REF" def isCompileTimeConstant(self): # TODO: Actually the base class should not be constant and this # one should be. return False def computeExpression(self, constraint_collection): # Needs whole program analysis, we don't really know much about it. return self, None, None class ExpressionBuiltinAnonymousRef(ExpressionBuiltinRefBase): kind = "EXPRESSION_BUILTIN_ANONYMOUS_REF" def __init__(self, builtin_name, source_ref): assert builtin_name in builtin_anon_names, builtin_name ExpressionBuiltinRefBase.__init__( self, builtin_name = builtin_name, source_ref = source_ref ) def isCompileTimeConstant(self): return True def getCompileTimeConstant(self): return builtin_anon_names[ self.builtin_name ] def computeExpression(self, constraint_collection): return self, None, None def getStringValue(self): return repr(self.getCompileTimeConstant()) class ExpressionBuiltinExceptionRef(ExpressionBuiltinRefBase): kind = "EXPRESSION_BUILTIN_EXCEPTION_REF" def __init__(self, exception_name, source_ref): assert exception_name in builtin_exception_names ExpressionBuiltinRefBase.__init__( self, builtin_name = exception_name, source_ref = source_ref ) def getDetails(self): return { "exception_name" : self.builtin_name } getExceptionName = ExpressionBuiltinRefBase.getBuiltinName def isCompileTimeConstant(self): return True def mayRaiseException(self, exception_type): return False def getCompileTimeConstant(self): return builtin_exception_values[self.builtin_name] def computeExpression(self, constraint_collection): # Not much that can be done here. return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): exception_name = self.getExceptionName() # TODO: Keyword only arguments of it, are not properly handled yet by # the built-in call code. if exception_name == "ImportError" and python_version >= 330: if call_kw is not None and \ (not call_kw.isExpressionConstantRef() or call_kw.getConstant() != {}): return call_node, None, None def createBuiltinMakeException(args, source_ref): return ExpressionBuiltinMakeException( exception_name = exception_name, args = args, source_ref = source_ref ) new_node = BuiltinOptimization.extractBuiltinArgs( node = call_node, builtin_class = createBuiltinMakeException, builtin_spec = BuiltinOptimization.makeBuiltinParameterSpec( exception_name = exception_name ) ) # TODO: Don't allow this to happen. if new_node is None: return call_node, None, None return new_node, "new_expression", "Detected built-in exception making." Nuitka-0.5.18.1/nuitka/nodes/ExceptionNodes.py0000644000175000017500000002311012651071040021326 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes related to raising and making exceptions. """ from .NodeBases import ( ExpressionChildrenHavingBase, ExpressionMixin, NodeBase, StatementChildrenHavingBase ) from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions class StatementRaiseException(StatementChildrenHavingBase): kind = "STATEMENT_RAISE_EXCEPTION" named_children = ( "exception_type", "exception_value", "exception_trace", "exception_cause" ) def __init__(self, exception_type, exception_value, exception_trace, exception_cause, source_ref): if exception_type is None: assert exception_value is None if exception_value is None: assert exception_trace is None StatementChildrenHavingBase.__init__( self, values = { "exception_type" : exception_type, "exception_value" : exception_value, "exception_trace" : exception_trace, "exception_cause" : exception_cause }, source_ref = source_ref ) self.reraise_finally = False getExceptionType = StatementChildrenHavingBase.childGetter( "exception_type" ) getExceptionValue = StatementChildrenHavingBase.childGetter( "exception_value" ) getExceptionTrace = StatementChildrenHavingBase.childGetter( "exception_trace" ) getExceptionCause = StatementChildrenHavingBase.childGetter( "exception_cause" ) def isStatementReraiseException(self): return self.getExceptionType() is None def isStatementAborting(self): return True @staticmethod def isImplicit(): return False def computeStatement(self, constraint_collection): constraint_collection.onExpression( expression = self.getExceptionType(), allow_none = True ) exception_type = self.getExceptionType() # TODO: Limit by type. constraint_collection.onExceptionRaiseExit(BaseException) if exception_type is not None and \ exception_type.willRaiseException(BaseException): from .NodeMakingHelpers import makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = exception_type, node = self ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception type.""" constraint_collection.onExpression( expression = self.getExceptionValue(), allow_none = True ) exception_value = self.getExceptionValue() if exception_value is not None and exception_value.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( exception_type, exception_value ) ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception value.""" constraint_collection.onExpression( expression = self.getExceptionTrace(), allow_none = True ) exception_trace = self.getExceptionTrace() if exception_trace is not None and \ exception_trace.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( exception_type, exception_value, exception_trace ) ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception traceback.""" constraint_collection.onExpression( expression = self.getExceptionCause(), allow_none = True ) exception_cause = self.getExceptionCause() if exception_cause is not None and \ exception_cause.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( exception_type, exception_cause, ) ) return result, "new_raise", """ Explicit raise already raises implicitly building exception cause.""" return self, None, None def needsFrame(self): return not self.isStatementReraiseException() class StatementRaiseExceptionImplicit(StatementRaiseException): kind = "STATEMENT_RAISE_EXCEPTION_IMPLICIT" @staticmethod def isStatementRaiseException(): return True @staticmethod def isImplicit(): return True class ExpressionRaiseException(ExpressionChildrenHavingBase): """ This node type is only produced via optimization. CPython only knows exception raising as a statement, but often the raising of exceptions can be predicted to occur as part of an expression, which it replaces then. """ kind = "EXPRESSION_RAISE_EXCEPTION" named_children = ( "exception_type", "exception_value" ) def __init__(self, exception_type, exception_value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "exception_type" : exception_type, "exception_value" : exception_value }, source_ref = source_ref ) def willRaiseException(self, exception_type): # One thing is clear, it will raise. TODO: Match exception_type more # closely if it is predictable. return exception_type is BaseException getExceptionType = ExpressionChildrenHavingBase.childGetter( "exception_type" ) getExceptionValue = ExpressionChildrenHavingBase.childGetter( "exception_value" ) def computeExpression(self, constraint_collection): return self, None, None def computeExpressionDrop(self, statement, constraint_collection): result = StatementRaiseExceptionImplicit( exception_type = self.getExceptionType(), exception_value = self.getExceptionValue(), exception_trace = None, exception_cause = None, source_ref = self.getSourceReference() ) return result, "new_raise", """\ Propagated implicit raise expression to raise statement.""" class ExpressionBuiltinMakeException(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_MAKE_EXCEPTION" named_children = ( "args", ) def __init__(self, exception_name, args, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "args" : tuple(args), }, source_ref = source_ref ) self.exception_name = exception_name def getDetails(self): return { "exception_name" : self.exception_name } def getExceptionName(self): return self.exception_name getArgs = ExpressionChildrenHavingBase.childGetter("args") def computeExpression(self, constraint_collection): return self, None, None def mayRaiseException(self, exception_type): for arg in self.getArgs(): if arg.mayRaiseException(exception_type): return True return False class ExpressionCaughtExceptionTypeRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_CAUGHT_EXCEPTION_TYPE_REF" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Might be predictable based on the exception handler this is in. return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False class ExpressionCaughtExceptionValueRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_CAUGHT_EXCEPTION_VALUE_REF" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Might be predictable based on the exception handler this is in. return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False class ExpressionCaughtExceptionTracebackRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_CAUGHT_EXCEPTION_TRACEBACK_REF" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeExpression(self, constraint_collection): return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False Nuitka-0.5.18.1/nuitka/nodes/FutureSpecs.py0000644000175000017500000000705612651071040020662 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Specification record for future flags. A source reference also implies a specific set of future flags in use by the parser at that location. Can be different inside a module due to e.g. the in-lining of "exec" statements with their own future imports, or in-lining of code from other modules. """ from nuitka.PythonVersions import python_version from nuitka.utils.InstanceCounters import counted_del, counted_init # These defaults have changed with Python versions. _future_division_default = python_version >= 300 _future_absolute_import_default = python_version >= 300 _future_generator_stop_default = python_version >= 360 class FutureSpec: @counted_init def __init__(self): self.future_division = _future_division_default self.unicode_literals = False self.absolute_import = _future_absolute_import_default self.future_print = False self.barry_bdfl = False self.generator_stop = _future_generator_stop_default __del__ = counted_del() def clone(self): result = FutureSpec() result.future_division = self.future_division result.unicode_literals = self.unicode_literals result.absolute_import = self.absolute_import result.future_print = self.future_print result.barry_bdfl = self.barry_bdfl return result def isFutureDivision(self): return self.future_division def enableFutureDivision(self): self.future_division = True def enableFuturePrint(self): self.future_print = True def enableUnicodeLiterals(self): self.unicode_literals = True def enableAbsoluteImport(self): self.absolute_import = True def enableBarry(self): self.barry_bdfl = True def enableGeneratorStop(self): self.generator_stop = True def isAbsoluteImport(self): return self.absolute_import def isGeneratorStop(self): return self.generator_stop def asFlags(self): """ Create a list of C identifiers to represent the flag values. This is for use in code generation only. """ result = [] if self.future_division and python_version < 300: result.append("CO_FUTURE_DIVISION") if self.unicode_literals: result.append("CO_FUTURE_UNICODE_LITERALS") if self.absolute_import and python_version < 300: result.append("CO_FUTURE_ABSOLUTE_IMPORT") if self.future_print and python_version < 300: result.append("CO_FUTURE_PRINT_FUNCTION") if self.barry_bdfl and python_version >= 300: result.append("CO_FUTURE_BARRY_AS_BDFL") if self.generator_stop and python_version >= 350: result.append("CO_FUTURE_GENERATOR_STOP") return tuple(result) Nuitka-0.5.18.1/nuitka/nodes/TryNodes.py0000644000175000017500000004230712651071040020157 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for try/except/finally handling. This is the unified low level solution to trying a block, and executing code when it returns, break, continues, or raises an exception. See Developer Manual for how this maps to try/finally and try/except as in Python. """ from nuitka.optimizations.TraceCollections import ConstraintCollectionBranch from .Checkers import checkStatementsSequence, checkStatementsSequenceOrNone from .NodeBases import StatementChildrenHavingBase from .StatementNodes import StatementsSequence class StatementTry(StatementChildrenHavingBase): kind = "STATEMENT_TRY" named_children = ( "tried", "except_handler", "break_handler", "continue_handler", "return_handler" ) checkers = { "tried" : checkStatementsSequence, "except_handler" : checkStatementsSequenceOrNone, "break_handler" : checkStatementsSequenceOrNone, "continue_handler" : checkStatementsSequenceOrNone, "return_handler" : checkStatementsSequenceOrNone } def __init__(self, tried, except_handler, break_handler, continue_handler, return_handler, source_ref): StatementChildrenHavingBase.__init__( self, values = { "tried" : tried, "except_handler" : except_handler, "break_handler" : break_handler, "continue_handler" : continue_handler, "return_handler" : return_handler }, source_ref = source_ref ) getBlockTry = StatementChildrenHavingBase.childGetter( "tried" ) setBlockTry = StatementChildrenHavingBase.childSetter( "tried" ) getBlockExceptHandler = StatementChildrenHavingBase.childGetter( "except_handler" ) setBlockExceptHandler = StatementChildrenHavingBase.childSetter( "except_handler" ) getBlockBreakHandler = StatementChildrenHavingBase.childGetter( "break_handler" ) setBlockBreakHandler = StatementChildrenHavingBase.childSetter( "break_handler" ) getBlockContinueHandler = StatementChildrenHavingBase.childGetter( "continue_handler" ) setBlockContinueHandler = StatementChildrenHavingBase.childSetter( "continue_handler" ) getBlockReturnHandler = StatementChildrenHavingBase.childGetter( "return_handler" ) setBlockReturnHandler = StatementChildrenHavingBase.childSetter( "return_handler" ) def computeStatement(self, constraint_collection): # This node has many children to handle, pylint: disable=R0912,R0914 tried = self.getBlockTry() except_handler = self.getBlockExceptHandler() break_handler = self.getBlockBreakHandler() continue_handler = self.getBlockContinueHandler() return_handler = self.getBlockReturnHandler() # The tried block must be considered as a branch, if it is not empty # already. collection_start = ConstraintCollectionBranch( parent = constraint_collection, name = "try start" ) abort_context = constraint_collection.makeAbortStackContext( catch_breaks = break_handler is not None, catch_continues = continue_handler is not None, catch_returns = return_handler is not None, catch_exceptions = True, ) with abort_context: # As a branch point for the many types of handlers. result = tried.computeStatementsSequence( constraint_collection = constraint_collection ) # We might be done entirely already. if result is None: return None, "new_statements", "Removed now empty try statement." # Might be changed. if result is not tried: self.setBlockTry(result) tried = result break_collections = constraint_collection.getLoopBreakCollections() continue_collections = constraint_collection.getLoopContinueCollections() return_collections = constraint_collection.getFunctionReturnCollections() exception_collections = constraint_collection.getExceptionRaiseCollections() tried_may_raise = tried.mayRaiseException(BaseException) # Exception handling is useless if no exception is to be raised. # TODO: signal the change. if not tried_may_raise: if except_handler is not None: self.setBlockExceptHandler(None) except_handler = None # If tried may raise, even empty exception handler has a meaning to # ignore that exception. if tried_may_raise: collection_exception_handling = ConstraintCollectionBranch( parent = collection_start, name = "except handler" ) if not exception_collections: for statement in tried.getStatements(): if statement.mayRaiseException(BaseException): assert False, statement.asXmlText() assert False collection_exception_handling.mergeMultipleBranches(exception_collections) if except_handler is not None: result = except_handler.computeStatementsSequence( constraint_collection = collection_exception_handling ) # Might be changed. if result is not except_handler: self.setBlockExceptHandler(result) except_handler = result if break_handler is not None: if not tried.mayBreak(): self.setBlockBreakHandler(None) break_handler = None if break_handler is not None: collection_break = ConstraintCollectionBranch( parent = collection_start, name = "break handler" ) collection_break.mergeMultipleBranches(break_collections) result = break_handler.computeStatementsSequence( constraint_collection = collection_break ) # Might be changed. if result is not break_handler: self.setBlockBreakHandler(result) break_handler = result if continue_handler is not None: if not tried.mayContinue(): self.setBlockContinueHandler(None) continue_handler = None if continue_handler is not None: collection_continue = ConstraintCollectionBranch( parent = collection_start, name = "continue handler" ) collection_continue.mergeMultipleBranches(continue_collections) result = continue_handler.computeStatementsSequence( constraint_collection = collection_continue ) # Might be changed. if result is not continue_handler: self.setBlockContinueHandler(result) continue_handler = result if return_handler is not None: if not tried.mayReturn(): self.setBlockReturnHandler(None) return_handler = None if return_handler is not None: collection_return = ConstraintCollectionBranch( parent = collection_start, name = "return handler" ) collection_return.mergeMultipleBranches(return_collections) result = return_handler.computeStatementsSequence( constraint_collection = collection_return ) # Might be changed. if result is not return_handler: self.setBlockReturnHandler(result) return_handler = result if return_handler is not None: if return_handler.getStatements()[0].isStatementReturn() and \ return_handler.getStatements()[0].getExpression().isExpressionReturnedValueRef(): self.setBlockReturnHandler(None) return_handler = None # Merge exception handler only if it is used. Empty means it is not # aborting, as it swallows the exception. if tried_may_raise and ( except_handler is None or \ not except_handler.isStatementAborting() ): constraint_collection.mergeBranches( collection_yes = collection_exception_handling, collection_no = None ) # An empty exception handler means we have to swallow exception. if not tried_may_raise and \ break_handler is None and \ continue_handler is None and \ return_handler is None: return tried, "new_statements", "Removed all try handlers." tried_statements = tried.getStatements() pre_statements = [] while tried_statements: tried_statement = tried_statements[0] if tried_statement.mayRaiseException(BaseException): break if break_handler is not None and \ tried_statement.mayBreak(): break if continue_handler is not None and \ tried_statement.mayContinue(): break if return_handler is not None and \ tried_statement.mayReturn(): break pre_statements.append(tried_statement) tried_statements = list(tried_statements) del tried_statements[0] post_statements = [] if except_handler is not None and except_handler.isStatementAborting(): while tried_statements: tried_statement = tried_statements[-1] if tried_statement.mayRaiseException(BaseException): break if break_handler is not None and \ tried_statement.mayBreak(): break if continue_handler is not None and \ tried_statement.mayContinue(): break if return_handler is not None and \ tried_statement.mayReturn(): break post_statements.insert(0, tried_statement) tried_statements = list(tried_statements) del tried_statements[-1] if pre_statements or post_statements: assert tried_statements # Should be dealt with already tried.setStatements(tried_statements) result = StatementsSequence( statements = pre_statements + [self] + post_statements, source_ref = self.getSourceReference() ) def explain(): # TODO: We probably don't want to say this for re-formulation ones. result = "Reduced scope of tried block." if pre_statements: result += " Leading statements at %s." % ( ','.join( x.getSourceReference().getAsString() + '/' + str(x) for x in pre_statements ) ) if post_statements: result += " Trailing statements at %s." % ( ','.join( x.getSourceReference().getAsString() + '/' + str(x) for x in post_statements ) ) return result return ( result, "new_statements", explain ) return self, None, None def mayReturn(self): # TODO: If we optimized return handler away, this would be not needed # or even non-optimal. if self.getBlockTry().mayReturn(): return True except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.mayReturn(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.mayReturn(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.mayReturn(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.mayReturn(): return True return False def mayBreak(self): # TODO: If we optimized return handler away, this would be not needed # or even non-optimal. if self.getBlockTry().mayBreak(): return True except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.mayBreak(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.mayBreak(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.mayBreak(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.mayBreak(): return True return False def mayContinue(self): # TODO: If we optimized return handler away, this would be not needed # or even non-optimal. if self.getBlockTry().mayContinue(): return True except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.mayContinue(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.mayContinue(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.mayContinue(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.mayContinue(): return True return False def isStatementAborting(self): except_handler = self.getBlockExceptHandler() if except_handler is None or not except_handler.isStatementAborting(): return False break_handler = self.getBlockBreakHandler() if break_handler is not None and not break_handler.isStatementAborting(): return False continue_handler = self.getBlockContinueHandler() if continue_handler is not None and not continue_handler.isStatementAborting(): return False return_handler = self.getBlockReturnHandler() if return_handler is not None and not return_handler.isStatementAborting(): return False return self.getBlockTry().isStatementAborting() def mayRaiseException(self, exception_type): tried = self.getBlockTry() if tried.mayRaiseException(exception_type): except_handler = self.getBlockExceptHandler() if except_handler is not None and \ except_handler.mayRaiseException(exception_type): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and \ break_handler.mayRaiseException(exception_type): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and \ continue_handler.mayRaiseException(exception_type): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and \ return_handler.mayRaiseException(exception_type): return True return False def needsFrame(self): except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.needsFrame(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.needsFrame(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.needsFrame(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.needsFrame(): return True return self.getBlockTry().needsFrame() Nuitka-0.5.18.1/nuitka/nodes/CodeObjectSpecs.py0000644000175000017500000000611412651071040021403 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code object specifications. For code objects that will be attached to module, function, and generator objects, as well as tracebacks. They might be shared. """ from nuitka.PythonVersions import python_version class CodeObjectSpec: def __init__(self, code_name, code_kind, arg_names, kw_only_count, has_starlist, has_stardict): assert type(has_starlist) is bool assert type(has_stardict) is bool self.code_name = code_name self.code_kind = code_kind self.arg_names = tuple(arg_names) for arg_name in arg_names: assert type(arg_name) is str self.kw_only_count = kw_only_count self.has_starlist = has_starlist self.has_stardict = has_stardict self.local_names = () def __repr__(self): return """\ """ % self.getDetails() def getDetails(self): result = { "code_name" : self.code_name, "code_kind" : self.code_kind, "arg_names" : self.arg_names, "local_names" : self.local_names, "kw_only_count" : self.kw_only_count, "has_starlist" : self.has_starlist, "has_stardict" : self.has_stardict, } if python_version >= 300: result["kw_only_count"] = self.kw_only_count return result def getKind(self): return self.code_kind def updateLocalNames(self, local_names): """ Add detected local variables after closure has been decided. """ self.local_names = tuple( local_name for local_name in local_names if local_name not in self.arg_names ) def getVarNames(self): return self.arg_names + self.local_names def getArgumentCount(self): return len(self.arg_names) - \ (1 if self.has_stardict else 0) - \ (1 if self.has_starlist else 0) - \ self.kw_only_count def getKwOnlyParameterCount(self): return self.kw_only_count def getCodeObjectName(self): return self.code_name def hasStarListArg(self): return self.has_starlist def hasStarDictArg(self): return self.has_stardict Nuitka-0.5.18.1/nuitka/nodes/BuiltinIteratorNodes.py0000644000175000017500000002712512651071040022522 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in iterator nodes. These play a role in for loops, and in unpacking. They can something be predicted to succeed or fail, in which case, code can become less complex. The length of things is an important optimization issue for these to be good. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.optimizations import BuiltinOptimization from .NodeBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase, StatementChildrenHavingBase ) class ExpressionBuiltinLen(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_LEN" builtin_spec = BuiltinOptimization.builtin_len_spec def getIntegerValue(self): return self.getValue().getIterationLength() def computeExpression(self, constraint_collection): from .NodeMakingHelpers import ( makeConstantReplacementNode, wrapExpressionWithNodeSideEffects ) new_node, change_tags, change_desc = ExpressionBuiltinSingleArgBase.\ computeExpression( self, constraint_collection = constraint_collection ) if new_node is self: arg_length = self.getIntegerValue() if arg_length is not None: change_tags = "new_constant" change_desc = "Predicted len argument" new_node = wrapExpressionWithNodeSideEffects( new_node = makeConstantReplacementNode(arg_length, self), old_node = self.getValue() ) if new_node.isExpressionSideEffects(): change_desc += " maintaining side effects" return new_node, change_tags, change_desc class ExpressionBuiltinIter1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ITER1" def computeExpression(self, constraint_collection): value = self.getValue() # Iterator of an iterator can be removed. if value.isIteratorMaking(): return value, "new_builtin", "Eliminated useless iterator creation." return value.computeExpressionIter1( iter_node = self, constraint_collection = constraint_collection ) def isIteratorMaking(self): return True def isKnownToBeIterable(self, count): if count is None: return True # TODO: Should ask value if it is. return None def getIterationLength(self): return self.getValue().getIterationLength() def extractSideEffects(self): # Iterator making is the side effect itself. if self.getValue().isCompileTimeConstant(): return () else: return (self,) def mayHaveSideEffects(self): if self.getValue().isCompileTimeConstant(): return self.getValue().isKnownToBeIterable(None) return True def mayRaiseException(self, exception_type): value = self.getValue() if value.mayRaiseException(exception_type): return True if value.isKnownToBeIterable(None): return False return True def isKnownToBeIterableAtMin(self, count): assert type(count) is int iter_length = self.getValue().getIterationMinLength() return iter_length is not None and iter_length < count def isKnownToBeIterableAtMax(self, count): assert type(count) is int iter_length = self.getValue().getIterationMaxLength() return iter_length is not None and count <= iter_length def onRelease(self, constraint_collection): # print "onRelease", self pass class ExpressionBuiltinNext1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_NEXT1" def __init__(self, value, source_ref): ExpressionBuiltinSingleArgBase.__init__( self, value = value, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Predict iteration result if possible via SSA variable trace of # the iterator state. # Assume exception is possible. TODO: We might query the next from the # source with a computeExpressionNext slot, but we delay that. constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None class ExpressionSpecialUnpack(ExpressionBuiltinNext1): kind = "EXPRESSION_SPECIAL_UNPACK" def __init__(self, value, count, expected, source_ref): ExpressionBuiltinNext1.__init__( self, value = value, source_ref = source_ref ) self.count = count self.expected = expected def getDetails(self): result = ExpressionBuiltinNext1.getDetails(self) result["count"] = self.getCount() result["expected"] = self.getExpected() return result def getCount(self): return self.count def getExpected(self): return self.expected class StatementSpecialUnpackCheck(StatementChildrenHavingBase): kind = "STATEMENT_SPECIAL_UNPACK_CHECK" named_children = ( "iterator", ) def __init__(self, iterator, count, source_ref): StatementChildrenHavingBase.__init__( self, values = { "iterator" : iterator }, source_ref = source_ref ) self.count = count def getDetails(self): return { "count" : self.getCount(), } def getCount(self): return self.count getIterator = StatementChildrenHavingBase.childGetter("iterator") def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getIterator()) iterator = self.getIterator() if iterator.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if iterator.willRaiseException(BaseException): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = iterator, node = self ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception type.""" # Remove the check if it can be decided at compile time. if iterator.isKnownToBeIterableAtMax(0): return None, "new_statements", """\ Determined iteration end check to be always true.""" constraint_collection.onExceptionRaiseExit( BaseException ) return self, None, None class ExpressionBuiltinIter2(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_ITER2" named_children = ( "callable", "sentinel", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, callable_arg, sentinel, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "callable" : callable_arg, "sentinel" : sentinel, }, source_ref = source_ref ) getCallable = ExpressionChildrenHavingBase.childGetter("callable") getSentinel = ExpressionChildrenHavingBase.childGetter("sentinel") def computeExpression(self, constraint_collection): # TODO: The "callable" should be investigated here, maybe it is not # really callable, or raises an exception. return self, None, None def isIteratorMaking(self): return True class ExpressionBuiltinNext2(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_NEXT2" named_children = ( "iterator", "default" ) def __init__(self, iterator, default, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "iterator" : iterator, "default" : default, }, source_ref = source_ref ) getIterator = ExpressionChildrenHavingBase.childGetter("iterator") getDefault = ExpressionChildrenHavingBase.childGetter("default") def computeExpression(self, constraint_collection): # TODO: The "iterator" should be investigated here, if it is iterable, # or if the default is raising. return self, None, None class ExpressionAsyncIter(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_ASYNC_ITER" def computeExpression(self, constraint_collection): value = self.getValue() return value.computeExpressionAsyncIter( iter_node = self, constraint_collection = constraint_collection ) # TODO: Questionable optimization based on this. Better to this in the slot # iteration. def isIteratorMaking(self): return True def isKnownToBeIterable(self, count): if count is None: return True # TODO: Should ask value if it is. return None def getIterationLength(self): return self.getValue().getIterationLength() def extractSideEffects(self): # Iterator making is the side effect itself. if self.getValue().isCompileTimeConstant(): return () else: return (self,) def mayHaveSideEffects(self): if self.getValue().isCompileTimeConstant(): return self.getValue().isKnownToBeIterable(None) return True def mayRaiseException(self, exception_type): value = self.getValue() if value.mayRaiseException(exception_type): return True if value.isKnownToBeIterable(None): return False return True def isKnownToBeIterableAtMin(self, count): assert type(count) is int iter_length = self.getValue().getIterationMinLength() return iter_length is not None and iter_length < count def isKnownToBeIterableAtMax(self, count): assert type(count) is int iter_length = self.getValue().getIterationMaxLength() return iter_length is not None and count <= iter_length def onRelease(self, constraint_collection): # print "onRelease", self pass class ExpressionAsyncNext(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_ASYNC_NEXT" def __init__(self, value, source_ref): ExpressionBuiltinSingleArgBase.__init__( self, value = value, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Predict iteration result if possible via SSA variable trace of # the iterator state. # Assume exception is possible. TODO: We might query the next from the # source with a computeExpressionAsyncNext slot, but we delay that. constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None Nuitka-0.5.18.1/nuitka/nodes/SubscriptNodes.py0000644000175000017500000001477212651072046021373 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Subscript node. Subscripts are important when working with lists and dictionaries. Tracking them can allow to achieve more compact code, or predict results at compile time. There will be a method "computeExpressionSubscript" to aid predicting them. """ from .NodeBases import ExpressionChildrenHavingBase, StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, makeStatementOnlyNodesFromExpressions ) class StatementAssignmentSubscript(StatementChildrenHavingBase): kind = "STATEMENT_ASSIGNMENT_SUBSCRIPT" named_children = ( "source", "expression", "subscript" ) def __init__(self, expression, subscript, source, source_ref): StatementChildrenHavingBase.__init__( self, values = { "source" : source, "expression" : expression, "subscript" : subscript }, source_ref = source_ref ) getSubscribed = StatementChildrenHavingBase.childGetter("expression") getSubscript = StatementChildrenHavingBase.childGetter("subscript") getAssignSource = StatementChildrenHavingBase.childGetter("source") def computeStatement(self, constraint_collection): constraint_collection.onExpression( expression = self.getAssignSource() ) source = self.getAssignSource() # No assignment will occur, if the assignment source raises, so strip it # away. if source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_raise", """\ Subscript assignment raises exception in assigned value, removed assignment.""" constraint_collection.onExpression(self.getSubscribed()) subscribed = self.getSubscribed() if subscribed.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, subscribed ) ) return result, "new_raise", """\ Subscript assignment raises exception in subscribed, removed assignment.""" constraint_collection.onExpression( self.getSubscript() ) subscript = self.getSubscript() if subscript.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, subscribed, subscript ) ) return result, "new_raise", """ Subscript assignment raises exception in subscript value, removed \ assignment.""" return subscribed.computeExpressionSetSubscript( set_node = self, subscript = subscript, value_node = source, constraint_collection = constraint_collection ) class StatementDelSubscript(StatementChildrenHavingBase): kind = "STATEMENT_DEL_SUBSCRIPT" named_children = ( "expression", "subscript" ) def __init__(self, expression, subscript, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression, "subscript" : subscript }, source_ref = source_ref ) getSubscribed = StatementChildrenHavingBase.childGetter("expression") getSubscript = StatementChildrenHavingBase.childGetter("subscript") def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getSubscribed()) subscribed = self.getSubscribed() if subscribed.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = subscribed, node = self ) return result, "new_raise", """\ Subscript 'del' raises exception in subscribed value, removed del.""" constraint_collection.onExpression(self.getSubscript()) subscript = self.getSubscript() if subscript.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( subscribed, subscript ) ) return result, "new_raise", """\ Subscript 'del' raises exception in subscript value, removed del.""" return subscribed.computeExpressionDelSubscript( set_node = self, subscript = subscript, constraint_collection = constraint_collection ) class ExpressionSubscriptLookup(ExpressionChildrenHavingBase): kind = "EXPRESSION_SUBSCRIPT_LOOKUP" named_children = ( "subscribed", "subscript" ) def __init__(self, subscribed, subscript, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "subscribed" : subscribed, "subscript" : subscript }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("subscribed") getSubscript = ExpressionChildrenHavingBase.childGetter("subscript") def computeExpression(self, constraint_collection): return self.getLookupSource().computeExpressionSubscript( lookup_node = self, subscript = self.getSubscript(), constraint_collection = constraint_collection ) def isKnownToBeIterable(self, count): return None Nuitka-0.5.18.1/nuitka/nodes/ReturnNodes.py0000644000175000017500000000574412651071040020664 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Return node This one exits functions. The only other exit is the default exit of functions with 'None' value, if no return is done. """ from .NodeBases import ExpressionMixin, NodeBase, StatementChildrenHavingBase class StatementReturn(StatementChildrenHavingBase): kind = "STATEMENT_RETURN" named_children = ("expression",) nice_children = ("return value",) def __init__(self, expression, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) getExpression = StatementChildrenHavingBase.childGetter( "expression" ) def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return self.getExpression().mayRaiseException(exception_type) def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getExpression()) expression = self.getExpression() if expression.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit(BaseException) if expression.willRaiseException(BaseException): from .NodeMakingHelpers import makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = expression, node = self ) return result, "new_raise", """\ Return statement raises in returned expression, removed return.""" constraint_collection.onFunctionReturn() return self, None, None class ExpressionReturnedValueRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_RETURNED_VALUE_REF" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Might be predictable based on the exception handler this is in. return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False def mayRaiseException(self, exception_type): return False Nuitka-0.5.18.1/nuitka/nodes/NodeMakingHelpers.py0000644000175000017500000002370612651072046021760 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ These are just helpers to create nodes, often to replace existing nodes These are for use in optimizations and computations, and therefore cover mostly exceptions and constants. Often cyclic dependencies kicks in, which is why this module is mostly only imported locally. Note: It's intended to be reversed, this module will make the local imports instead, as these local imports look ugly everywhere else, making it more difficult to use. """ from logging import warning from nuitka.Builtins import builtin_names from nuitka.Constants import isConstant from nuitka.Options import isDebug, shallWarnImplicitRaises def makeConstantReplacementNode(constant, node): from .ConstantRefNodes import ExpressionConstantRef return ExpressionConstantRef( constant = constant, source_ref = node.getSourceReference() ) def makeRaiseExceptionReplacementExpression(expression, exception_type, exception_value): from .ExceptionNodes import ExpressionRaiseException from .BuiltinRefNodes import ExpressionBuiltinExceptionRef source_ref = expression.getSourceReference() assert type(exception_type) is str if shallWarnImplicitRaises(): warning( "%s: Will always raise exception '%s(%s)'", expression.getSourceReference().getAsString(), exception_type, exception_value ) result = ExpressionRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = exception_type, source_ref = source_ref ), exception_value = makeConstantReplacementNode( constant = exception_value, node = expression ), source_ref = source_ref ) return result def makeRaiseExceptionReplacementExpressionFromInstance(expression, exception): assert isinstance(exception, Exception) args = exception.args if type(args) is tuple and len(args) == 1: value = args[0] else: assert type(args) is tuple value = args return makeRaiseExceptionReplacementExpression( expression = expression, exception_type = exception.__class__.__name__, exception_value = value ) def makeCompileTimeConstantReplacementNode(value, node): # This needs to match code in isCompileTimeConstantValue if isConstant(value): return makeConstantReplacementNode( constant = value, node = node ) elif type(value) is type: if value.__name__ in builtin_names: from .BuiltinRefNodes import ExpressionBuiltinRef return ExpressionBuiltinRef( builtin_name = value.__name__, source_ref = node.getSourceReference() ) else: return node else: return node def getComputationResult(node, computation, description): """ With a computation function, execute it and return constant result or exception node. """ # Try and turn raised exceptions into static raises. pylint: disable=W0703 try: result = computation() except Exception as e: new_node = makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = e ) change_tags = "new_raise" change_desc = description + " Predicted to raise an exception." else: new_node = makeCompileTimeConstantReplacementNode( value = result, node = node ) if isDebug(): assert new_node is not node, (node, result) if new_node is not node: change_tags = "new_constant" change_desc = description + " Predicted constant result." else: change_tags = None change_desc = None return new_node, change_tags, change_desc def makeStatementExpressionOnlyReplacementNode(expression, node): from .StatementNodes import StatementExpressionOnly return StatementExpressionOnly( expression = expression, source_ref = node.getSourceReference() ) def mergeStatements(statements, allow_none = False): """ Helper function that merges nested statement sequences. """ merged_statements = [] for statement in statements: if statement is None and allow_none: pass elif type(statement) in (tuple, list): merged_statements += mergeStatements(statement, allow_none) elif statement.isStatement() or statement.isStatementsFrame(): merged_statements.append(statement) elif statement.isStatementsSequence(): merged_statements.extend(mergeStatements(statement.getStatements())) else: assert False, statement return merged_statements def makeStatementsSequenceReplacementNode(statements, node): from .StatementNodes import StatementsSequence return StatementsSequence( statements = mergeStatements(statements), source_ref = node.getSourceReference() ) def convertNoneConstantToNone(node): if node is None: return None elif node.isExpressionConstantRef() and node.getConstant() is None: return None else: return node def wrapExpressionWithSideEffects(side_effects, old_node, new_node): assert new_node.isExpression() from .SideEffectNodes import ExpressionSideEffects if side_effects: new_node = ExpressionSideEffects( expression = new_node, side_effects = side_effects, source_ref = old_node.getSourceReference() ) return new_node def wrapExpressionWithNodeSideEffects(new_node, old_node): return wrapExpressionWithSideEffects( side_effects = old_node.extractSideEffects(), old_node = old_node, new_node = new_node ) def wrapStatementWithSideEffects(new_node, old_node, allow_none = False): assert new_node is not None or allow_none side_effects = old_node.extractSideEffects() if side_effects: from .StatementNodes import StatementExpressionOnly side_effects = tuple( StatementExpressionOnly( expression = side_effect, source_ref = side_effect.getSourceReference() ) for side_effect in side_effects ) if new_node is not None: new_node = makeStatementsSequenceReplacementNode( statements = side_effects + (new_node,), node = old_node ) else: new_node = makeStatementsSequenceReplacementNode( statements = side_effects, node = old_node ) return new_node def makeStatementOnlyNodesFromExpressions(expressions): from .StatementNodes import StatementExpressionOnly, StatementsSequence statements = tuple( StatementExpressionOnly( expression = expression, source_ref = expression.getSourceReference() ) for expression in expressions ) if not statements: return None elif len(statements) == 1: return statements[0] else: return StatementsSequence( statements = statements, source_ref = statements[0].getSourceReference() ) def makeComparisonNode(left, right, comparator, source_ref): from .ComparisonNodes import ( ExpressionComparison, ExpressionComparisonIs, ExpressionComparisonIsNOT ) if comparator == "Is": result = ExpressionComparisonIs( left = left, right = right, source_ref = source_ref ) elif comparator == "IsNot": result = ExpressionComparisonIsNOT( left = left, right = right, source_ref = source_ref ) else: result = ExpressionComparison( left = left, right = right, comparator = comparator, source_ref = source_ref ) result.setCompatibleSourceReference( source_ref = right.getCompatibleSourceReference() ) return result def makeVariableRefNode(variable, source_ref): if variable.isTempVariable(): from .VariableRefNodes import ExpressionTempVariableRef return ExpressionTempVariableRef( variable = variable, source_ref = source_ref ) else: from .VariableRefNodes import ExpressionVariableRef return ExpressionVariableRef( variable_name = variable.getName(), variable = variable, source_ref = source_ref ) def makeVariableTargetRefNode(variable, source_ref): if variable.isTempVariable(): from .AssignNodes import ExpressionTargetTempVariableRef return ExpressionTargetTempVariableRef( variable = variable, source_ref = source_ref ) else: from .AssignNodes import ExpressionTargetVariableRef return ExpressionTargetVariableRef( variable_name = variable.getName(), variable = variable, source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/nodes/BuiltinDecodingNodes.py0000644000175000017500000000372512651071040022445 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Builtin ord/chr nodes These are good for optimizations, as they give a very well known result. In the case of 'chr', it's one of 256 strings, and in case of 'ord' it's one of 256 numbers, so these can answer quite a few questions at compile time. """ from nuitka.optimizations import BuiltinOptimization from .NodeBases import ( ExpressionBuiltinNoArgBase, ExpressionBuiltinSingleArgBase ) class ExpressionBuiltinOrd0(ExpressionBuiltinNoArgBase): kind = "EXPRESSION_BUILTIN_ORD0" def __init__(self, source_ref): ExpressionBuiltinNoArgBase.__init__( self, builtin_function = ord, source_ref = source_ref ) class ExpressionBuiltinOrd(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ORD" builtin_spec = BuiltinOptimization.builtin_ord_spec def isKnownToBeIterable(self, count): return False class ExpressionBuiltinChr(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_CHR" builtin_spec = BuiltinOptimization.builtin_chr_spec def isKnownToBeIterable(self, count): if self.mayRaiseException(BaseException): return None return count is None or count == 1 Nuitka-0.5.18.1/nuitka/nodes/AssignNodes.py0000644000175000017500000005327712651072046020644 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Assignment related nodes. The most simple assignment statement ``a = b`` is what we have here. All others are either re-formulated using temporary variables, e.g. ``a, b = c`` or are attribute, slice, subscript assignments. The deletion is a separate node unlike in CPython where assigning to ``NULL`` is internally what deletion is. But deleting is something entirely different to us during code generation, which is why we keep them separate. Tracing assignments in SSA form is the core of optimization for which we use the traces. """ from nuitka import Options, VariableRegistry from .NodeBases import NodeBase, StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, makeStatementsSequenceReplacementNode ) from .VariableRefNodes import ExpressionTempVariableRef, ExpressionVariableRef class StatementAssignmentVariable(StatementChildrenHavingBase): """ Assignment to a variable from an expression. All assignment forms that are not to attributes, slices, subscripts use this. The source might be a complex expression. The target can be any kind of variable, temporary, local, global, etc. Assigning a variable is something we trace in a new version, this is hidden behind target variable reference, which has this version once it can be determined. """ kind = "STATEMENT_ASSIGNMENT_VARIABLE" named_children = ( "source", "variable_ref" ) inplace_suspect = None def __init__(self, variable_ref, source, source_ref): assert variable_ref is not None, source_ref assert source is not None, source_ref assert variable_ref.isTargetVariableRef() StatementChildrenHavingBase.__init__( self, values = { "source" : source, "variable_ref" : variable_ref }, source_ref = source_ref ) self.variable_trace = None def getDetail(self): variable_ref = self.getTargetVariableRef() variable = variable_ref.getVariable() if variable is not None: return "to variable %s" % variable else: return "to variable %s" % self.getTargetVariableRef() getTargetVariableRef = StatementChildrenHavingBase.childGetter( "variable_ref" ) getAssignSource = StatementChildrenHavingBase.childGetter( "source" ) setAssignSource = StatementChildrenHavingBase.childSetter( "source" ) def markAsInplaceSuspect(self): self.inplace_suspect = True def isInplaceSuspect(self): return self.inplace_suspect def mayRaiseException(self, exception_type): return self.getAssignSource().mayRaiseException(exception_type) def computeStatement(self, constraint_collection): # This is very complex stuff, pylint: disable=R0912 # TODO: Way too ugly to have global trace kinds just here, and needs to # be abstracted somehow. But for now we let it live here: pylint: disable=R0915 # Let assignment source may re-compute first. constraint_collection.onExpression(self.getAssignSource()) source = self.getAssignSource() # No assignment will occur, if the assignment source raises, so give up # on this, and return it as the only side effect. if source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_raise", """\ Assignment raises exception in assigned value, removed assignment.""" variable_ref = self.getTargetVariableRef() variable = variable_ref.getVariable() # Not allowed anymore at this point. assert variable is not None # Assigning from and to the same variable, can be optimized away # immediately, there is no point in doing it. Exceptions are of course # module variables that collide with built-in names. # TODO: The variable type checks ought to become unnecessary, as they # are to be a feature of the trace. Assigning from known assigned is # supposed to be possible to eliminate. If we get that wrong, we are # doing it wrong. if not variable.isModuleVariable() and \ source.isExpressionVariableRef() and \ source.getVariable() == variable: # A variable access that has a side effect, must be preserved, # so it can e.g. raise an exception, otherwise we can be fully # removed. if source.mayHaveSideEffects(): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_statements", """\ Reduced assignment of variable '%s' from itself to mere access of it.""" % variable.getName() else: return None, "new_statements", """\ Removed assignment of variable '%s' from itself which is known to be defined.""" % variable.getName() # If the assignment source has side effects, we can simply evaluate them # beforehand, we have already visited and evaluated them before. if source.isExpressionSideEffects(): statements = [ makeStatementExpressionOnlyReplacementNode( side_effect, self ) for side_effect in source.getSideEffects() ] statements.append(self) parent = self.parent result = makeStatementsSequenceReplacementNode( statements = statements, node = self, ) result.parent = parent # Need to update it. self.setAssignSource(source.getExpression()) source = self.getAssignSource() constraint_collection.signalChange( tags = "new_statements", message = """\ Side effects of assignments promoted to statements.""", source_ref = self.getSourceReference() ) # Set-up the trace to the trace collection, so future references will # find this assignment. self.variable_trace = constraint_collection.onVariableSet( assign_node = self ) global_trace = VariableRegistry.getGlobalVariableTrace(variable) if global_trace is not None: # and Options.isExperimental(): last_trace = global_trace.getMatchingAssignTrace(self) if last_trace is not None: if variable.isLocalVariable() or variable.isTempVariable(): if source.isCompileTimeConstant(): # Can safely forward propagate only non-mutable constants. if not source.isMutable(): provider = self.getParentVariableProvider() if variable.isTempVariable() or \ (not provider.isExpressionClassBody() and \ not provider.isUnoptimized() ): if last_trace.hasDefiniteUsages(): self.variable_trace.setReplacementNode( lambda usage : source.makeClone() ) propagated = True else: propagated = False if not last_trace.hasPotentialUsages() and not last_trace.hasNameUsages(): # This limitation may fall later. if not variable.isSharedLogically(): if not last_trace.getPrevious().isUninitTrace(): # TODO: We could well decide, if that's even necessary. result = StatementDelVariable( variable_ref = self.getTargetVariableRef(), tolerant = True, source_ref = self.getSourceReference() ) else: result = None return ( result, "new_statements", "Dropped %s assignment statement to '%s'." % ( "propagated" if propagated else "dead", self.getTargetVariableRef().getVariableName() ) ) else: # Something might be possible still. pass elif Options.isExperimental() and \ source.isExpressionFunctionCreation() and \ not source.getFunctionRef().getFunctionBody().isExpressionGeneratorBody() and \ not source.getFunctionRef().getFunctionBody().isExpressionClassBody() and \ not source.getDefaults() and \ not source.getKwDefaults() and \ not source.getAnnotations(): # TODO: These are very mutable, right? provider = self.getParentVariableProvider() if variable.isTempVariable() or \ (not provider.isUnoptimized() and \ not provider.isExpressionClassBody()): # This limitation may fall later. if not variable.isSharedLogically(): if last_trace.getDefiniteUsages() <= 1 and \ not last_trace.hasPotentialUsages() and \ not last_trace.hasNameUsages(): if last_trace.getDefiniteUsages() == 1: self.variable_trace.setReplacementNode( lambda usage : source.makeClone() ) propagated = True else: propagated = False if not last_trace.getPrevious().isUninitTrace(): # TODO: We could well decide, if that's even necessary. result = StatementDelVariable( variable_ref = self.getTargetVariableRef(), tolerant = True, source_ref = self.getSourceReference() ) else: result = None return ( result, "new_statements", "Dropped %s assignment statement to '%s'." % ( "propagated" if propagated else "dead", self.getTargetVariableRef().getVariableName() ) ) else: # More cases thinkable. pass return self, None, None def needsReleasePreviousValue(self): previous = self.variable_trace.getPrevious() if previous.mustNotHaveValue(): return False elif previous.mustHaveValue(): return True else: return None class StatementDelVariable(StatementChildrenHavingBase): """ Deleting a variable. All del forms that are not to attributes, slices, subscripts use this. The target can be any kind of variable, temporary, local, global, etc. Deleting a variable is something we trace in a new version, this is hidden behind target variable reference, which has this version once it can be determined. Tolerance means that the value might be unset. That can happen with re-formulation of ours, and Python3 exception variables. """ kind = "STATEMENT_DEL_VARIABLE" named_children = ( "variable_ref", ) def __init__(self, variable_ref, tolerant, source_ref): assert variable_ref is not None assert variable_ref.isTargetVariableRef() assert tolerant is True or tolerant is False StatementChildrenHavingBase.__init__( self, values = { "variable_ref" : variable_ref }, source_ref = source_ref ) self.variable_trace = None self.previous_trace = None self.tolerant = tolerant def getDetail(self): variable_ref = self.getTargetVariableRef() variable = variable_ref.getVariable() if variable is not None: return "to variable %s" % variable else: return "to variable %s" % self.getTargetVariableRef() def getDetails(self): return { "tolerant" : self.tolerant } # TODO: Value propagation needs to make a difference based on this. def isTolerant(self): return self.tolerant getTargetVariableRef = StatementChildrenHavingBase.childGetter( "variable_ref" ) def computeStatement(self, constraint_collection): variable_ref = self.getTargetVariableRef() variable = variable_ref.getVariable() self.previous_trace = constraint_collection.getVariableCurrentTrace(variable) # First eliminate us entirely if we can. if self.tolerant and self.previous_trace.isUninitTrace(): return ( None, "new_statements", "Removed tolerant 'del' statement of '%s' without effect." % ( variable.getName(), ) ) # The "del" is a potential use of a value. TODO: This could be made more # beautiful indication, as it's not any kind of usage. self.previous_trace.addPotentialUsage() # If not tolerant, we may exception exit now during the __del__ if not self.tolerant and not self.previous_trace.mustHaveValue(): constraint_collection.onExceptionRaiseExit(BaseException) # Record the deletion, needs to start a new version then. constraint_collection.onVariableDel( variable_ref = variable_ref ) constraint_collection.onVariableContentEscapes(variable) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Need to fetch the potentially invalidated variable. A "del" on a # or shared value, may easily assign the global variable in "__del__". self.variable_trace = constraint_collection.getVariableCurrentTrace(variable) return self, None, None def mayHaveSideEffects(self): return True def mayRaiseException(self, exception_type): if self.tolerant: return False else: if self.variable_trace is not None: variable = self.getTargetVariableRef().getVariable() # TODO: This condition must become unnecessary, but enhancing # SSA to notice potential escapes. if not variable.isModuleVariable() and \ not variable.isMaybeLocalVariable() and not \ variable.isSharedTechnically(): # Temporary variables deletions won't raise, just because we don't # create them that way. We can avoid going through SSA in these # cases. if variable.isTempVariable(): return False # If SSA knows, that's fine. if self.previous_trace is not None and \ self.previous_trace.mustHaveValue(): return False return True class StatementReleaseVariable(NodeBase): """ Releasing a variable. Just release the value, which of course is not to be used afterwards. Typical code: Function exit. """ kind = "STATEMENT_RELEASE_VARIABLE" def __init__(self, variable, source_ref): assert variable is not None, source_ref NodeBase.__init__( self, source_ref = source_ref ) self.variable = variable self.variable_trace = None def getDetail(self): return "of variable %s" % self.variable def getDetails(self): return { "variable" : self.variable } def getDetailsForDisplay(self): if self.variable.getOwner() is not self.getParentVariableProvider(): return { "variable" : self.variable.getName(), "owner" : self.variable.getOwner().getCodeName() } else: return { "variable" : self.variable.getName(), } def getVariable(self): return self.variable def setVariable(self, variable): self.variable = variable def computeStatement(self, constraint_collection): self.variable_trace = constraint_collection.onVariableRelease( variable = self.variable ) if self.variable_trace.isUninitTrace(): return ( None, "new_statements", "Uninitialized variable '%s' is not released." % ( self.variable.getName() ) ) constraint_collection.onVariableContentEscapes(self.variable) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # TODO: We might be able to remove ourselves based on the trace # we belong to. return self, None, None def mayHaveSideEffects(self): # May execute __del__ code, it would be sweet to be able to predict # that another reference will still be active for a value though. return True def mayRaiseException(self, exception_type): # By default, __del__ is not allowed to raise an exception. return False class ExpressionTargetVariableRef(ExpressionVariableRef): kind = "EXPRESSION_TARGET_VARIABLE_REF" # TODO: Remove default and correct argument order later. def __init__(self, variable_name, source_ref, variable = None): ExpressionVariableRef.__init__(self, variable_name, source_ref) self.variable_version = None # TODO: Remove setVariable, once not needed anymore and in-line to # here. if variable is not None: self.setVariable(variable) assert variable.getName() == variable_name def getDetailsForDisplay(self): if self.variable is None: return { "name" : self.variable_name } else: return { "name" : self.variable_name, "variable" : self.variable, "version" : self.variable_version } def computeExpression(self, constraint_collection): assert False, self.parent @staticmethod def isTargetVariableRef(): return True def getVariableVersion(self): assert self.variable_version is not None, self return self.variable_version def setVariable(self, variable): ExpressionVariableRef.setVariable(self, variable) self.variable_version = variable.allocateTargetNumber() assert self.variable_version is not None class ExpressionTargetTempVariableRef(ExpressionTempVariableRef): kind = "EXPRESSION_TARGET_TEMP_VARIABLE_REF" def __init__(self, variable, source_ref): ExpressionTempVariableRef.__init__(self, variable, source_ref) self.variable_version = variable.allocateTargetNumber() def computeExpression(self, constraint_collection): assert False, self.parent @staticmethod def isTargetVariableRef(): return True def getVariableVersion(self): return self.variable_version # Python3 only, it updates temporary variables that are closure variables. def setVariable(self, variable): ExpressionTempVariableRef.setVariable(self, variable) self.variable_version = self.variable.allocateTargetNumber() Nuitka-0.5.18.1/nuitka/nodes/SliceNodes.py0000644000175000017500000002475712651071040020451 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Slice nodes. Slices are important when working with lists. Tracking them can allow to achieve more compact code, or predict results at compile time. There will be a method "computeExpressionSlice" to aid predicting them. """ from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ConstantRefNodes import ExpressionConstantRef from .NodeBases import ( ChildrenHavingMixin, ExpressionChildrenHavingBase, ExpressionSpecBasedComputationMixin, NodeBase, StatementChildrenHavingBase ) from .NodeMakingHelpers import ( convertNoneConstantToNone, makeStatementExpressionOnlyReplacementNode, makeStatementOnlyNodesFromExpressions ) class StatementAssignmentSlice(StatementChildrenHavingBase): kind = "STATEMENT_ASSIGNMENT_SLICE" named_children = ( "source", "expression", "lower", "upper" ) def __init__(self, expression, lower, upper, source, source_ref): assert python_version < 300 StatementChildrenHavingBase.__init__( self, values = { "source" : source, "expression" : expression, "lower" : lower, "upper" : upper }, source_ref = source_ref ) getLookupSource = StatementChildrenHavingBase.childGetter("expression") getLower = StatementChildrenHavingBase.childGetter("lower") getUpper = StatementChildrenHavingBase.childGetter("upper") getAssignSource = StatementChildrenHavingBase.childGetter("source") def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getAssignSource()) source = self.getAssignSource() # No assignment will occur, if the assignment source raises, so strip it # away. if source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_raise", """\ Slice assignment raises exception in assigned value, removed assignment.""" constraint_collection.onExpression(self.getLookupSource()) lookup_source = self.getLookupSource() if lookup_source.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, lookup_source ) ) return result, "new_raise", """\ Slice assignment raises exception in sliced value, removed assignment.""" constraint_collection.onExpression(self.getLower(), allow_none = True) lower = self.getLower() if lower is not None and lower.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, lookup_source, lower ) ) return result, "new_raise", """\ Slice assignment raises exception in lower slice boundary value, removed \ assignment.""" constraint_collection.onExpression(self.getUpper(), allow_none = True) upper = self.getUpper() if upper is not None and upper.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, lookup_source, lower, upper ) ) return result, "new_raise", """\ Slice assignment raises exception in upper slice boundary value, removed \ assignment.""" return lookup_source.computeExpressionSetSlice( set_node = self, lower = lower, upper = upper, value_node = source, constraint_collection = constraint_collection ) class StatementDelSlice(StatementChildrenHavingBase): kind = "STATEMENT_DEL_SLICE" named_children = ( "expression", "lower", "upper" ) def __init__(self, expression, lower, upper, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression, "lower" : lower, "upper" : upper }, source_ref = source_ref ) getLookupSource = StatementChildrenHavingBase.childGetter("expression") getLower = StatementChildrenHavingBase.childGetter("lower") getUpper = StatementChildrenHavingBase.childGetter("upper") def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getLookupSource()) lookup_source = self.getLookupSource() if lookup_source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = lookup_source, node = self ) return result, "new_raise", """\ Slice del raises exception in sliced value, removed del""" constraint_collection.onExpression(self.getLower(), allow_none = True) lower = self.getLower() if lower is not None and lower.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( lookup_source, lower ) ) return result, "new_raise", """ Slice del raises exception in lower slice boundary value, removed del""" constraint_collection.onExpression(self.getUpper(), allow_none = True) upper = self.getUpper() if upper is not None and upper.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( lookup_source, lower, upper ) ) return result, "new_raise", """ Slice del raises exception in upper slice boundary value, removed del""" return lookup_source.computeExpressionDelSlice( set_node = self, lower = lower, upper = upper, constraint_collection = constraint_collection ) class ExpressionSliceLookup(ExpressionChildrenHavingBase): kind = "EXPRESSION_SLICE_LOOKUP" named_children = ( "expression", "lower", "upper" ) checkers = { "upper" : convertNoneConstantToNone, "lower" : convertNoneConstantToNone } def __init__(self, expression, lower, upper, source_ref): assert python_version < 300 ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression, "upper" : upper, "lower" : lower }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("expression") getLower = ExpressionChildrenHavingBase.childGetter("lower") setLower = ExpressionChildrenHavingBase.childSetter("lower") getUpper = ExpressionChildrenHavingBase.childGetter("upper") setUpper = ExpressionChildrenHavingBase.childSetter("upper") def computeExpression(self, constraint_collection): lookup_source = self.getLookupSource() return lookup_source.computeExpressionSlice( lookup_node = self, lower = self.getLower(), upper = self.getUpper(), constraint_collection = constraint_collection ) def isKnownToBeIterable(self, count): # TODO: Should ask SliceRegistry return None class ExpressionBuiltinSlice(ChildrenHavingMixin, NodeBase, ExpressionSpecBasedComputationMixin): kind = "EXPRESSION_BUILTIN_SLICE" named_children = ( "start", "stop", "step" ) builtin_spec = BuiltinOptimization.builtin_slice_spec def __init__(self, start, stop, step, source_ref): NodeBase.__init__( self, source_ref = source_ref ) if start is None: start = ExpressionConstantRef( constant = None, source_ref = source_ref ) if stop is None: stop = ExpressionConstantRef( constant = None, source_ref = source_ref ) if step is None: step = ExpressionConstantRef( constant = None, source_ref = source_ref ) ChildrenHavingMixin.__init__( self, values = { "start" : start, "stop" : stop, "step" : step } ) def computeExpression(self, constraint_collection): start = self.getStart() stop = self.getStop() step = self.getStep() args = ( start, stop, step ) return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = args ) def mayRaiseException(self, exception_type): return self.getStart().mayRaiseException(exception_type) or \ self.getStop().mayRaiseException(exception_type) or \ self.getStep().mayRaiseException(exception_type) getStart = ChildrenHavingMixin.childGetter("start") getStop = ChildrenHavingMixin.childGetter("stop") getStep = ChildrenHavingMixin.childGetter("step") Nuitka-0.5.18.1/nuitka/nodes/ParameterSpecs.py0000644000175000017500000003347112651071040021330 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module maintains the parameter specification classes. These are used for function, lambdas, generators. They are also a factory for the respective variable objects. One of the difficulty of Python and its parameter parsing is that they are allowed to be nested like this: (a,b), c Much like in assignments, which are very similar to parameters, except that parameters may also be assigned from a dictionary, they are no less flexible. """ from nuitka import Variables from nuitka.PythonVersions import python_version class TooManyArguments(Exception): def __init__(self, real_exception): Exception.__init__(self) self.real_exception = real_exception def getRealException(self): return self.real_exception class ParameterSpec: # These got many attributes, in part duplicating name and instance of # variables, pylint: disable=R0902 def __init__(self, name, normal_args, kw_only_args, list_star_arg, dict_star_arg, default_count): assert None not in normal_args self.owner = None self.name = name self.normal_args = tuple(normal_args) self.normal_variables = None assert list_star_arg is None or type(list_star_arg) is str, \ list_star_arg assert dict_star_arg is None or type(dict_star_arg) is str, \ dict_star_arg self.list_star_arg = list_star_arg self.dict_star_arg = dict_star_arg self.list_star_variable = None self.dict_star_variable = None self.default_count = default_count self.kw_only_args = tuple(kw_only_args) self.kw_only_variables = None def makeClone(self): return ParameterSpec( name = self.name, normal_args = self.normal_args, kw_only_args = self.kw_only_args, list_star_arg = self.list_star_arg, dict_star_arg = self.dict_star_arg, default_count = self.default_count ) def checkValid(self): arg_names = self.getParameterNames() # Check for duplicate arguments, could happen. for arg_name in arg_names: if arg_names.count(arg_name) != 1: return "duplicate argument '%s' in function definition" % arg_name return None def __repr__(self): parts = [ str(normal_arg) for normal_arg in self.normal_args ] if self.list_star_arg is not None: parts.append("*%s" % self.list_star_arg) if self.dict_star_variable is not None: parts.append("**%s" % self.dict_star_variable) if parts: return "" % ','.join(parts) else: return "" def getArgumentCount(self): return len(self.normal_args) def setOwner(self, owner): if self.owner is not None: return self.owner = owner self.normal_variables = [] for normal_arg in self.normal_args: if type(normal_arg) is str: normal_variable = Variables.ParameterVariable( owner = self.owner, parameter_name = normal_arg ) else: assert False, normal_arg self.normal_variables.append(normal_variable) if self.list_star_arg: self.list_star_variable = Variables.ParameterVariable( owner = owner, parameter_name = self.list_star_arg ) else: self.list_star_variable = None if self.dict_star_arg: self.dict_star_variable = Variables.ParameterVariable( owner = owner, parameter_name = self.dict_star_arg ) else: self.dict_star_variable = None self.kw_only_variables = [ Variables.ParameterVariable( owner = self.owner, parameter_name = kw_only_arg ) for kw_only_arg in self.kw_only_args ] def getDefaultCount(self): return self.default_count def hasDefaultParameters(self): return self.getDefaultCount() > 0 def getTopLevelVariables(self): return self.normal_variables + self.kw_only_variables def getAllVariables(self): result = list(self.normal_variables) result += self.kw_only_variables if self.list_star_variable is not None: result.append(self.list_star_variable) if self.dict_star_variable is not None: result.append(self.dict_star_variable) return result def getParameterNames(self): result = list(self.normal_args) result += self.kw_only_args if self.list_star_arg is not None: result.append(self.list_star_arg) if self.dict_star_arg is not None: result.append(self.dict_star_arg) return result def getStarListArgumentName(self): return self.list_star_arg def getListStarArgVariable(self): return self.list_star_variable def getStarDictArgumentName(self): return self.dict_star_arg def getDictStarArgVariable(self): return self.dict_star_variable def getKwOnlyVariables(self): return self.kw_only_variables def allowsKeywords(self): # Abstract method, pylint: disable=R0201 return True def getKeywordRefusalText(self): return "%s() takes no keyword arguments" % self.name def getArgumentNames(self): return self.normal_args def getKwOnlyParameterNames(self): return self.kw_only_args def getKwOnlyParameterCount(self): return len(self.kw_only_args) # Note: Based loosely on "inspect.getcallargs" with corrections. def matchCall(func_name, args, star_list_arg, star_dict_arg, num_defaults, positional, pairs, improved = False): # This is of incredible code complexity, but there really is no other way to # express this with less statements, branches, or variables. # pylint: disable=R0912,R0914,R0915 assert type(positional) is tuple, positional assert type(pairs) in (tuple, list), pairs # Make a copy, we are going to modify it. pairs = list(pairs) result = {} assigned_tuple_params = [] def assign(arg, value): if type(arg) is str: # Normal case: result[ arg ] = value else: # Tuple argument case: assigned_tuple_params.append(arg) value = iter(value.getIterationValues()) for i, subarg in enumerate(arg): try: subvalue = next(value) except StopIteration: raise TooManyArguments( ValueError( "need more than %d %s to unpack" % ( i, "values" if i > 1 else "value" ) ) ) # Recurse into tuple argument values, could be more tuples. assign(subarg, subvalue) # Check that not too many values we provided. try: next(value) except StopIteration: pass else: raise TooManyArguments( ValueError("too many values to unpack") ) def isAssigned(arg): if type(arg) is str: return arg in result return arg in assigned_tuple_params num_pos = len(positional) num_total = num_pos + len(pairs) num_args = len(args) for arg, value in zip(args, positional): assign(arg, value) # Python3 does this check earlier. if python_version >= 300 and not star_dict_arg: for pair in pairs: if pair[0] not in args: message = "'%s' is an invalid keyword argument for this function" % pair[0] raise TooManyArguments( TypeError(message) ) if star_list_arg: if num_pos > num_args: assign(star_list_arg, positional[ -(num_pos-num_args) : ]) else: assign(star_list_arg, ()) elif 0 < num_args < num_total: if num_defaults == 0: if num_args == 1: raise TooManyArguments( TypeError( "%s() takes exactly one argument (%d given)" % ( func_name, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s expected %d arguments, got %d" % ( func_name, num_args, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s() takes at most %d %s (%d given)" % ( func_name, num_args, "argument" if num_args == 1 else "arguments", num_total ) ) ) elif num_args == 0 and num_total: if star_dict_arg: if num_pos: # Could use num_pos, but Python also uses num_total. raise TooManyArguments( TypeError( "%s() takes exactly 0 arguments (%d given)" % ( func_name, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s() takes no arguments (%d given)" % ( func_name, num_total ) ) ) named_argument_names = [ pair[0] for pair in pairs ] for arg in args: if type(arg) is str and arg in named_argument_names: if isAssigned(arg): raise TooManyArguments( TypeError( "%s() got multiple values for keyword argument '%s'" % ( func_name, arg ) ) ) else: new_pairs = [] for pair in pairs: if arg == pair[0]: assign(arg, pair[1]) else: new_pairs.append(pair) assert len(new_pairs) == len(pairs) - 1 pairs = new_pairs # Fill in any missing values with the None to indicate "default". if num_defaults > 0: for arg in args[ -num_defaults : ]: if not isAssigned(arg): assign(arg, None) if star_dict_arg: assign(star_dict_arg, pairs) elif pairs: unexpected = next(iter(dict(pairs))) if improved: message = "%s() got an unexpected keyword argument '%s'" % ( func_name, unexpected ) else: message = "'%s' is an invalid keyword argument for this function" % unexpected raise TooManyArguments( TypeError(message) ) unassigned = num_args - len( [ arg for arg in args if isAssigned(arg) ] ) if unassigned: num_required = num_args - num_defaults if num_required > 0 or improved: if num_defaults == 0 and num_args != 1: raise TooManyArguments( TypeError( "%s expected %d arguments, got %d" % ( func_name, num_args, num_total ) ) ) if num_required == 1: arg_desc = "1 argument" if python_version < 350 else "one argument" else: arg_desc = "%d arguments" % num_required raise TooManyArguments( TypeError( "%s() takes %s %s (%d given)" % ( func_name, "at least" if num_defaults > 0 else "exactly", arg_desc, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s expected %s%s, got %d" % ( func_name, ( "at least " if python_version < 300 else "" ) if num_defaults > 0 else "exactly ", "%d arguments" % num_required, num_total ) ) ) return result Nuitka-0.5.18.1/nuitka/nodes/ContainerOperationNodes.py0000644000175000017500000002274412651072046023216 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Operations on Containers. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from .NodeBases import ExpressionChildrenHavingBase, StatementChildrenHavingBase class ExpressionListOperationAppend(ExpressionChildrenHavingBase): kind = "EXPRESSION_LIST_OPERATION_APPEND" named_children = ( "list", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, list_arg, value, source_ref): assert list_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "list" : list_arg, "value" : value }, source_ref = source_ref ) getList = ExpressionChildrenHavingBase.childGetter("list") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getList()) return self, None, None class ExpressionListOperationExtend(ExpressionChildrenHavingBase): kind = "EXPRESSION_LIST_OPERATION_EXTEND" named_children = ( "list", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, list_arg, value, source_ref): assert list_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "list" : list_arg, "value" : value }, source_ref = source_ref ) getList = ExpressionChildrenHavingBase.childGetter("list") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getList()) return self, None, None class ExpressionListOperationPop(ExpressionChildrenHavingBase): kind = "EXPRESSION_LIST_OPERATION_POP" named_children = ( "list", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, list_arg, source_ref): assert list_arg is not None ExpressionChildrenHavingBase.__init__( self, values = { "list" : list_arg, }, source_ref = source_ref ) getList = ExpressionChildrenHavingBase.childGetter("list") def computeExpression(self, constraint_collection): # We might be able to tell that element, or know that it cannot exist # and raise an exception instead. constraint_collection.removeKnowledge(self.getList()) return self, None, None class ExpressionSetOperationAdd(ExpressionChildrenHavingBase): kind = "EXPRESSION_SET_OPERATION_ADD" named_children = ( "set", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, set_arg, value, source_ref): assert set_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "set" : set_arg, "value" : value }, source_ref = source_ref ) getSet = ExpressionChildrenHavingBase.childGetter( "set" ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getSet()) return self, None, None class ExpressionSetOperationUpdate(ExpressionChildrenHavingBase): kind = "EXPRESSION_SET_OPERATION_UPDATE" named_children = ( "set", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, set_arg, value, source_ref): assert set_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "set" : set_arg, "value" : value }, source_ref = source_ref ) getSet = ExpressionChildrenHavingBase.childGetter( "set" ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getSet()) return self, None, None class ExpressionDictOperationSet(ExpressionChildrenHavingBase): kind = "EXPRESSION_DICT_OPERATION_SET" named_children = ( "dict", "key", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, key, value, source_ref): assert dict_arg is not None assert key is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, "value" : value }, source_ref = source_ref ) getDict = ExpressionChildrenHavingBase.childGetter("dict") getKey = ExpressionChildrenHavingBase.childGetter("key") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getDict()) return self, None, None class StatementDictOperationRemove(StatementChildrenHavingBase): kind = "STATEMENT_DICT_OPERATION_REMOVE" named_children = ( "dict", "key" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, key, source_ref): assert dict_arg is not None assert key is not None StatementChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, }, source_ref = source_ref ) getDict = StatementChildrenHavingBase.childGetter("dict") getKey = StatementChildrenHavingBase.childGetter("key") def computeStatement(self, constraint_collection): constraint_collection.onExpression(self.getDict()) dicte = self.getDict() if dicte.willRaiseException(BaseException): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = dicte, node = self ) return result, "new_raise", """\ Dictionary remove already raises implicitly accessing dictionary.""" constraint_collection.onExpression(self.getKey()) key = self.getKey() if key.willRaiseException(BaseException): from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions result = makeStatementOnlyNodesFromExpressions( expressions = ( dicte, key ) ) return result, "new_raise", """ Dictionary remove already raises implicitly building key.""" # TODO: Be less lossy about it. constraint_collection.removeKnowledge(dicte) return self, None, None class ExpressionDictOperationGet(ExpressionChildrenHavingBase): kind = "EXPRESSION_DICT_OPERATION_GET" named_children = ( "dict", "key" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, key, source_ref): assert dict_arg is not None assert key is not None ExpressionChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, }, source_ref = source_ref ) getDict = ExpressionChildrenHavingBase.childGetter("dict") getKey = ExpressionChildrenHavingBase.childGetter("key") def computeExpression(self, constraint_collection): return self, None, None class ExpressionDictOperationUpdate(ExpressionChildrenHavingBase): kind = "EXPRESSION_DICT_OPERATION_UPDATE" named_children = ( "dict", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, value, source_ref): assert dict_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "value" : value }, source_ref = source_ref ) getSet = ExpressionChildrenHavingBase.childGetter( "dict" ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, constraint_collection): constraint_collection.removeKnowledge(self.getSet()) return self, None, None Nuitka-0.5.18.1/nuitka/nodes/BuiltinDictNodes.py0000644000175000017500000001262612651072046021623 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for the calls to the 'dict' built-in. """ from nuitka.nodes.DictionaryNodes import ( ExpressionKeyValuePair, ExpressionMakeDict ) from nuitka.nodes.NodeMakingHelpers import wrapExpressionWithNodeSideEffects from nuitka.optimizations.BuiltinOptimization import builtin_dict_spec from .BuiltinIteratorNodes import ExpressionBuiltinIter1 from .ConstantRefNodes import ExpressionConstantRef from .NodeBases import ExpressionChildrenHavingBase class ExpressionBuiltinDict(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_DICT" named_children = ( "pos_arg", "pairs" ) def __init__(self, pos_arg, pairs, source_ref): assert type(pos_arg) not in (tuple, list), source_ref assert type(pairs) in (tuple, list), source_ref ExpressionChildrenHavingBase.__init__( self, values = { "pos_arg" : pos_arg, "pairs" : tuple( ExpressionKeyValuePair( ExpressionConstantRef(key, source_ref), value, value.getSourceReference() ) for key, value in pairs ) }, source_ref = source_ref ) getPositionalArgument = ExpressionChildrenHavingBase.childGetter("pos_arg") getNamedArgumentPairs = ExpressionChildrenHavingBase.childGetter("pairs") def hasOnlyConstantArguments(self): pos_arg = self.getPositionalArgument() if pos_arg is not None and not pos_arg.isCompileTimeConstant(): return False for arg_pair in self.getNamedArgumentPairs(): if not arg_pair.getKey().isCompileTimeConstant(): return False if not arg_pair.getValue().isCompileTimeConstant(): return False return True def computeExpression(self, constraint_collection): pos_arg = self.getPositionalArgument() pairs = self.getNamedArgumentPairs() if pos_arg is None: new_node = ExpressionMakeDict( pairs = self.getNamedArgumentPairs(), source_ref = self.source_ref ) # This cannot raise anymore than its arguments, as the keys will # be known as hashable, due to being Python parameters before. return ( new_node, "new_expression", "Replace 'dict' built-in call dictionary creation from arguments." ) pos_iteration_length = pos_arg.getIterationLength() if pos_iteration_length == 0: new_node = ExpressionMakeDict( pairs = self.getNamedArgumentPairs(), source_ref = self.source_ref ) # Maintain potential side effects from the positional arguments. new_node = wrapExpressionWithNodeSideEffects( old_node = ExpressionBuiltinIter1( value = pos_arg, source_ref = self.source_ref ), new_node = new_node ) # Just in case, the iteration may do that. if pos_arg.mayRaiseExceptionIter(BaseException): constraint_collection.onExceptionRaiseExit(BaseException) return ( new_node, "new_expression", "Replace 'dict' built-in call dictionary creation from arguments." ) if pos_iteration_length is not None and \ pos_iteration_length + len(pairs) < 256 and \ self.hasOnlyConstantArguments(): if pos_arg is not None: pos_args = ( pos_arg, ) else: pos_args = None return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : builtin_dict_spec.simulateCall( (pos_args, self.getNamedArgumentPairs()) ), description = "Replace 'dict' call with constant arguments." ) else: constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): pos_arg = self.getPositionalArgument() # TODO: Determining if it's sufficient is not easy but possible. if pos_arg is not None: return True for arg_pair in self.getNamedArgumentPairs(): if arg_pair.mayRaiseException(exception_type): return True return False Nuitka-0.5.18.1/nuitka/nodes/FrameNodes.py0000644000175000017500000001776712651071040020447 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Frame nodes. The frame attaches name and other frame properties to a scope, where it is optional. For use in tracebacks, their created frame objects, potentially cached are essential. Otherwise, they are similar to statement sequences, so they inherit from them. """ from nuitka.PythonVersions import python_version from .StatementNodes import StatementsSequence def checkFrameStatements(value): """ Check that frames statements list value proper. Must not be None, must not contain None, and of course only statements sequences, or statements, may be empty. """ assert value is not None assert None not in value for statement in value: assert statement.isStatement() or statement.isStatementsFrame(), \ statement return tuple(value) class StatementsFrame(StatementsSequence): kind = "STATEMENTS_FRAME" checkers = { "statements" : checkFrameStatements } def __init__(self, statements, guard_mode, code_object, source_ref): StatementsSequence.__init__( self, statements = statements, source_ref = source_ref ) # TODO: Why not have multiple classes for this. self.guard_mode = guard_mode self.code_object = code_object self.needs_frame_exception_preserve = False def getDetails(self): result = { "guard_mode" : self.guard_mode, "code_object" : self.code_object } result.update(StatementsSequence.getDetails(self)) return result def getDetailsForDisplay(self): result = self.getDetails() result.update(self.code_object.getDetails()) return result def getGuardMode(self): return self.guard_mode def needsExceptionFramePreservation(self): if python_version < 300: preserving = ("full", "once") else: preserving = ("full", "once", "generator") return self.guard_mode in preserving def getVarNames(self): return self.code_object.getVarNames() def updateLocalNames(self): """ For use during variable closure phase. """ provider = self.getParentVariableProvider() if provider.isExpressionFunctionBody(): self.code_object.updateLocalNames( [ variable.getName() for variable in provider.getLocalVariables() ] ) def markAsFrameExceptionPreserving(self): self.needs_frame_exception_preserve = True def needsFrameExceptionPreserving(self): return self.needs_frame_exception_preserve def getCodeObject(self): return self.code_object def getCodeObjectHandle(self, context): provider = self.getParentVariableProvider() is_optimized = not provider.isCompiledPythonModule() and \ not provider.isExpressionClassBody() and \ not provider.hasLocalsDict() new_locals = not provider.isCompiledPythonModule() and \ (python_version < 340 or ( not provider.isExpressionClassBody() and \ not provider.hasLocalsDict())) # TODO: Why do this accessing a node, do this outside. return context.getCodeObjectHandle( code_object = self.code_object, filename = self.getParentModule().getRunTimeFilename(), line_number = 1 if provider.isCompiledPythonModule() else self.source_ref.getLineNumber(), is_optimized = is_optimized, new_locals = new_locals, has_closure = provider.isExpressionFunctionBody() and \ provider.getClosureVariables() != () and \ not provider.isExpressionClassBody(), future_flags = provider.getSourceReference().getFutureSpec().\ asFlags() ) def computeStatementsSequence(self, constraint_collection): # The extraction of parts of the frame that can be moved before or after # the frame scope, takes it toll to complexity, pylint: disable=R0912 new_statements = [] statements = self.getStatements() for count, statement in enumerate(statements): # May be frames embedded. if statement.isStatementsFrame(): new_statement = statement.computeStatementsSequence( constraint_collection = constraint_collection ) else: new_statement = constraint_collection.onStatement( statement = statement ) if new_statement is not None: if new_statement.isStatementsSequence() and \ not new_statement.isStatementsFrame(): new_statements.extend(new_statement.getStatements()) else: new_statements.append(new_statement) if statement is not statements[-1] and \ new_statement.isStatementAborting(): constraint_collection.signalChange( "new_statements", statements[count+1].getSourceReference(), "Removed dead statements." ) break if not new_statements: return None # If our statements changed just now, they are not immediately usable, # so do this in two steps. Next time we can reduce the frame scope just # as well. if statements != tuple(new_statements): self.setStatements(new_statements) return self # Determine statements inside the frame, that need not be in a frame, # because they wouldn't raise an exception. outside_pre = [] while new_statements and \ not new_statements[0].needsFrame(): outside_pre.append(new_statements[0]) del new_statements[0] outside_post = [] while new_statements and \ not new_statements[-1].needsFrame(): outside_post.insert(0, new_statements[-1]) del new_statements[-1] if outside_pre or outside_post: from .NodeMakingHelpers import makeStatementsSequenceReplacementNode if new_statements: self.setStatements(new_statements) return makeStatementsSequenceReplacementNode( statements = outside_pre + [self] + \ outside_post, node = self ) else: constraint_collection.signalChange( "new_statements", self.getSourceReference(), "Removed useless frame object of '%s'." % self.code_object.getCodeObjectName() ) return makeStatementsSequenceReplacementNode( statements = outside_pre + outside_post, node = self ) else: if statements != new_statements: self.setStatements(new_statements) return self Nuitka-0.5.18.1/nuitka/nodes/TypeNodes.py0000644000175000017500000001237312651071040020322 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ The type1 node. This one just determines types. It's great for optimization. We may be able to predict its value, but knowing it. In that case, we have a built-in name reference for that type to convert to, or when checking the result of it, we will then know it's limited after the fact. """ from nuitka.Builtins import builtin_names from .BuiltinRefNodes import ExpressionBuiltinAnonymousRef, ExpressionBuiltinRef from .NodeBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase ) class ExpressionBuiltinType1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_TYPE1" def computeExpression(self, constraint_collection): value = self.getValue() if value.isCompileTimeConstant(): value = value.getCompileTimeConstant() type_name = value.__class__.__name__ if type_name in builtin_names: new_node = ExpressionBuiltinRef( builtin_name = type_name, source_ref = self.getSourceReference() ) else: new_node = ExpressionBuiltinAnonymousRef( builtin_name = type_name, source_ref = self.getSourceReference() ) return ( new_node, "new_builtin", "Replaced predictable type lookup with builtin type '%s'." % ( type_name ) ) return self, None, None def computeExpressionDrop(self, statement, constraint_collection): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = self.getValue(), node = statement ) return result, "new_statements", """\ Removed type taking for unused result.""" def mayRaiseException(self, exception_type): return self.getValue().mayRaiseException(exception_type) def mayHaveSideEffects(self): return self.getValue().mayHaveSideEffects() class ExpressionBuiltinSuper(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_SUPER" named_children = ( "type", "object" ) def __init__(self, super_type, super_object, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "type" : super_type, "object" : super_object }, source_ref = source_ref ) getType = ExpressionChildrenHavingBase.childGetter("type") getObject = ExpressionChildrenHavingBase.childGetter("object") def computeExpression(self, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) # TODO: Quite some cases should be possible to predict. return self, None, None class ExpressionBuiltinIsinstance(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_ISINSTANCE" named_children = ( "instance", "classes" ) def __init__(self, instance, classes, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "instance" : instance, "classes" : classes }, source_ref = source_ref ) getInstance = ExpressionChildrenHavingBase.childGetter("instance") getCls = ExpressionChildrenHavingBase.childGetter("classes") def computeExpression(self, constraint_collection): # TODO: Quite some cases should be possible to predict. instance = self.getInstance() # TODO: Should be possible to query run time type instead, but we don't # have that method yet. Later this will be essential. if not instance.isCompileTimeConstant(): constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None cls = self.getCls() if not cls.isCompileTimeConstant(): constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None # So if both are compile time constant, we are able to compute it. return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : isinstance( instance.getCompileTimeConstant(), cls.getCompileTimeConstant() ), description = "Built-in call to 'isinstance' computed." ) Nuitka-0.5.18.1/nuitka/nodes/Checkers.py0000644000175000017500000000221512651071040020131 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node children checkers. The role of checkers is to make sure that node children have specific value types only. """ def checkStatementsSequenceOrNone(value): assert value is None or value.kind == "STATEMENTS_SEQUENCE", value return value def checkStatementsSequence(value): assert value is not None and value.kind == "STATEMENTS_SEQUENCE", value return value Nuitka-0.5.18.1/nuitka/nodes/ExecEvalNodes.py0000644000175000017500000002541212651071040021073 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes concern with exec and eval builtins. These are the dynamic codes, and as such rather difficult. We would like to eliminate or limit their impact as much as possible, but it's difficult to do. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.PythonVersions import python_version from .NodeBases import ExpressionChildrenHavingBase, StatementChildrenHavingBase from .NodeMakingHelpers import ( convertNoneConstantToNone, makeStatementOnlyNodesFromExpressions ) class ExpressionBuiltinEval(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_EVAL" named_children = ( "source", "globals", "locals" ) def __init__(self, source_code, globals_arg, locals_arg, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source_code, "globals" : globals_arg, "locals" : locals_arg, }, source_ref = source_ref ) getSourceCode = ExpressionChildrenHavingBase.childGetter("source") getGlobals = ExpressionChildrenHavingBase.childGetter("globals") getLocals = ExpressionChildrenHavingBase.childGetter("locals") def computeExpression(self, constraint_collection): # TODO: Attempt for constant values to do it. return self, None, None # Note: Python3 only so far. if python_version >= 300: class ExpressionBuiltinExec(ExpressionBuiltinEval): kind = "EXPRESSION_BUILTIN_EXEC" def __init__(self, source_code, globals_arg, locals_arg, source_ref): ExpressionBuiltinEval.__init__( self, source_code = source_code, globals_arg = globals_arg, locals_arg = locals_arg, source_ref = source_ref ) def needsLocalsDict(self): return True def computeExpression(self, constraint_collection): # TODO: Attempt for constant values to do it. return self, None, None def computeExpressionDrop(self, statement, constraint_collection): if self.getParentVariableProvider().isEarlyClosure(): result = StatementExec( source_code = self.getSourceCode(), globals_arg = self.getGlobals(), locals_arg = self.getLocals(), source_ref = self.getSourceReference() ) return result, "new_statements", """\ Replaced built-in exec call to exec statement in early closure context.""" else: return statement, None, None # Note: Python2 only if python_version < 300: class ExpressionBuiltinExecfile(ExpressionBuiltinEval): kind = "EXPRESSION_BUILTIN_EXECFILE" named_children = ("source", "globals", "locals") def __init__(self, source_code, globals_arg, locals_arg, source_ref): ExpressionBuiltinEval.__init__( self, source_code = source_code, globals_arg = globals_arg, locals_arg = locals_arg, source_ref = source_ref ) def needsLocalsDict(self): return True def computeExpressionDrop(self, statement, constraint_collection): # In this case, the copy-back must be done and will only be done # correctly by the code for exec statements. provider = self.getParentVariableProvider() if provider.isExpressionClassBody(): result = StatementExec( source_code = self.getSourceCode(), globals_arg = self.getGlobals(), locals_arg = self.getLocals(), source_ref = self.getSourceReference() ) return result, "new_statements", """\ Changed 'execfile' with unused result to 'exec' on class level.""" else: return statement, None, None class StatementExec(StatementChildrenHavingBase): kind = "STATEMENT_EXEC" named_children = ( "source", "globals", "locals" ) def __init__(self, source_code, globals_arg, locals_arg, source_ref): StatementChildrenHavingBase.__init__( self, values = { "globals" : globals_arg, "locals" : locals_arg, "source" : source_code }, source_ref = source_ref, ) def setChild(self, name, value): if name in ("globals", "locals"): value = convertNoneConstantToNone(value) return StatementChildrenHavingBase.setChild(self, name, value) getSourceCode = StatementChildrenHavingBase.childGetter("source") getGlobals = StatementChildrenHavingBase.childGetter("globals") getLocals = StatementChildrenHavingBase.childGetter("locals") def needsLocalsDict(self): return self.getLocals().mayBeNone() def computeStatement(self, constraint_collection): constraint_collection.onExpression( expression = self.getSourceCode() ) source_code = self.getSourceCode() if source_code.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if source_code.willRaiseException(BaseException): result = source_code return ( result, "new_raise", """\ Exec statement raises implicitly when determining source code argument.""" ) constraint_collection.onExpression( expression = self.getGlobals(), allow_none = True ) globals_arg = self.getGlobals() if globals_arg is not None and globals_arg.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if globals_arg is not None and \ globals_arg.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source_code, globals_arg ) ) return ( result, "new_raise", """\ Exec statement raises implicitly when determining globals argument.""" ) constraint_collection.onExpression( expression = self.getLocals(), allow_none = True ) locals_arg = self.getLocals() if locals_arg is not None and locals_arg.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if locals_arg is not None and \ locals_arg.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source_code, globals_arg, locals_arg ) ) return ( result, "new_raise", """\ Exec statement raises implicitly when determining locals argument.""" ) constraint_collection.onExceptionRaiseExit( BaseException ) str_value = self.getSourceCode().getStrValue() if False and str_value is not None: # TODO: Don't forget to consider side effects of source code, # locals_arg, and globals_arg. # TODO: This needs to be re-done. exec_body = None return ( exec_body, "new_statements", "In-lined constant exec statement." ) return self, None, None class StatementLocalsDictSync(StatementChildrenHavingBase): kind = "STATEMENT_LOCALS_DICT_SYNC" named_children = ( "locals", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, locals_arg, source_ref): StatementChildrenHavingBase.__init__( self, values = { "locals" : locals_arg, }, source_ref = source_ref, ) def computeStatement(self, constraint_collection): if self.getParentVariableProvider().isCompiledPythonModule(): return None, "new_statements", "Removed sync back to locals without locals." constraint_collection.removeAllKnowledge() return self, None, None getLocals = ExpressionChildrenHavingBase.childGetter("locals") def mayRaiseException(self, exception_type): return False class ExpressionBuiltinCompile(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_COMPILE" named_children = ( "source", "filename", "mode", "flags", "dont_inherit", "optimize" ) def __init__(self, source_code, filename, mode, flags, dont_inherit, optimize, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source_code, "filename" : filename, "mode" : mode, "flags" : flags, "dont_inherit" : dont_inherit, "optimize" : optimize }, source_ref = source_ref ) getSourceCode = ExpressionChildrenHavingBase.childGetter("source") getFilename = ExpressionChildrenHavingBase.childGetter("filename") getMode = ExpressionChildrenHavingBase.childGetter("mode") getFlags = ExpressionChildrenHavingBase.childGetter("flags") getDontInherit = ExpressionChildrenHavingBase.childGetter("dont_inherit") getOptimize = ExpressionChildrenHavingBase.childGetter("optimize") def computeExpression(self, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) # TODO: Attempt for constant values to do it. return self, None, None Nuitka-0.5.18.1/nuitka/nodes/PrintNodes.py0000644000175000017500000001444112651071040020473 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Print nodes. Right now there is only the print statement, but in principle, there should also be the print function here. These perform output, which can be combined if possible, and could be detected to fail, which would be perfect. Predicting the behavior of 'print' is not trivial at all, due to many special cases. """ from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, makeStatementsSequenceReplacementNode, wrapStatementWithSideEffects ) class StatementPrintValue(StatementChildrenHavingBase): kind = "STATEMENT_PRINT_VALUE" named_children = ( "dest", "value" ) def __init__(self, dest, value, source_ref): StatementChildrenHavingBase.__init__( self, values = { "value" : value, "dest" : dest }, source_ref = source_ref ) assert value is not None getDestination = StatementChildrenHavingBase.childGetter( "dest" ) getValue = StatementChildrenHavingBase.childGetter( "value" ) setValue = StatementChildrenHavingBase.childSetter( "value" ) def computeStatement(self, constraint_collection): constraint_collection.onExpression( expression = self.getDestination(), allow_none = True ) dest = self.getDestination() if dest is not None and dest.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if dest is not None and dest.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = dest, node = self ) return result, "new_raise", """\ Exception raise in 'print' statement destination converted to explicit raise.""" constraint_collection.onExpression( expression = self.getValue() ) value = self.getValue() if value.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if value.willRaiseException(BaseException): if dest is not None: result = wrapStatementWithSideEffects( new_node = makeStatementExpressionOnlyReplacementNode( expression = value, node = self ), old_node = dest ) else: result = makeStatementExpressionOnlyReplacementNode( expression = value, node = self ) return result, "new_raise", """\ Exception raise in 'print' statement arguments converted to explicit raise.""" constraint_collection.onExceptionRaiseExit( BaseException ) if dest is None: if value.isExpressionSideEffects(): self.setValue(value.getExpression()) statements = [ makeStatementExpressionOnlyReplacementNode( side_effect, self ) for side_effect in value.getSideEffects() ] statements.append(self) result = makeStatementsSequenceReplacementNode( statements = statements, node = self, ) return result, "new_statements", """\ Side effects printed item promoted to statements.""" if value.isCompileTimeConstant(): if not (value.isExpressionConstantRef() and \ value.isUnicodeConstant()): new_value = value.getStrValue() assert new_value is not None, value if value is not new_value: self.setValue(new_value) return self, None, None def mayRaiseException(self, exception_type): return True class StatementPrintNewline(StatementChildrenHavingBase): kind = "STATEMENT_PRINT_NEWLINE" named_children = ( "dest", ) def __init__(self, dest, source_ref): StatementChildrenHavingBase.__init__( self, values = { "dest" : dest }, source_ref = source_ref ) getDestination = StatementChildrenHavingBase.childGetter( "dest" ) def computeStatement(self, constraint_collection): # TODO: Reactivate below optimizations for prints. constraint_collection.onExpression( expression = self.getDestination(), allow_none = True ) dest = self.getDestination() if dest is not None and dest.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit( BaseException ) if dest is not None and dest.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = dest, node = self ) return result, "new_raise", """\ Exception raise in 'print' statement destination converted to explicit raise.""" constraint_collection.onExceptionRaiseExit( BaseException ) return self, None, None def mayRaiseException(self, exception_type): return True Nuitka-0.5.18.1/nuitka/nodes/BuiltinVarsNodes.py0000644000175000017500000000317512651071040021643 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Builtin vars node. Not used much, esp. not in the form with arguments. Maybe used in some meta programming, and hopefully can be predicted, because at run time, it is hard to support. """ from .NodeBases import ExpressionChildrenHavingBase class ExpressionBuiltinVars(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_VARS" named_children = ( "source", ) def __init__(self, source, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source, }, source_ref = source_ref ) getSource = ExpressionChildrenHavingBase.childGetter("source") def computeExpression(self, constraint_collection): # TODO: Should be possible to predict this. return self, None, None def mayBeNone(self): return None Nuitka-0.5.18.1/nuitka/nodes/BuiltinOpenNodes.py0000644000175000017500000000402112651071040021620 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'open' builtin. This is a rather two sided beast, as it may be read or write. And we would like to be able to track it, so we can include files into the executable, or write more efficiently. """ from .NodeBases import ExpressionChildrenHavingBase class ExpressionBuiltinOpen(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_OPEN" named_children = ( "filename", "mode", "buffering" ) def __init__(self, filename, mode, buffering, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "filename" : filename, "mode" : mode, "buffering" : buffering }, source_ref = source_ref ) getFilename = ExpressionChildrenHavingBase.childGetter("filename") getMode = ExpressionChildrenHavingBase.childGetter("mode") getBuffering = ExpressionChildrenHavingBase.childGetter("buffering") def computeExpression(self, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) # Note: Quite impossible to predict without further assumptions, but we could look # at the arguments at least. return self, None, None Nuitka-0.5.18.1/nuitka/nodes/ComparisonNodes.py0000644000175000017500000002013012651072046021510 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for comparisons. """ from nuitka import PythonOperators from .NodeBases import ExpressionChildrenHavingBase from .NodeMakingHelpers import ( makeComparisonNode, makeConstantReplacementNode, wrapExpressionWithSideEffects ) class ExpressionComparison(ExpressionChildrenHavingBase): kind = "EXPRESSION_COMPARISON" named_children = ( "left", "right" ) def __init__(self, left, right, comparator, source_ref): assert left.isExpression() assert right.isExpression() assert comparator in PythonOperators.all_comparison_functions, comparator ExpressionChildrenHavingBase.__init__( self, values = { "left" : left, "right" : right }, source_ref = source_ref ) self.comparator = comparator if comparator in ("Is", "IsNot"): assert self.__class__ is not ExpressionComparison def getOperands(self): return ( self.getLeft(), self.getRight() ) getLeft = ExpressionChildrenHavingBase.childGetter("left") getRight = ExpressionChildrenHavingBase.childGetter("right") def getComparator(self): return self.comparator def getDetails(self): return { "comparator" : self.comparator } def getSimulator(self): return PythonOperators.all_comparison_functions[self.comparator] def computeExpression(self, constraint_collection): left, right = self.getOperands() if left.isCompileTimeConstant() and right.isCompileTimeConstant(): left_value = left.getCompileTimeConstant() right_value = right.getCompileTimeConstant() return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( left_value, right_value ), description = "Comparison with constant arguments." ) # The value of these nodes escaped and could change its contents. constraint_collection.removeKnowledge(left) constraint_collection.removeKnowledge(right) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None def computeExpressionOperationNot(self, not_node, constraint_collection): if self.comparator in PythonOperators.comparison_inversions: left, right = self.getOperands() result = makeComparisonNode( left = left, right = right, comparator = PythonOperators.comparison_inversions[ self.comparator ], source_ref = self.source_ref ) return result, "new_expression", """\ Replaced negated comparison with inverse comparison.""" return not_node, None, None class ExpressionComparisonIsIsNotBase(ExpressionComparison): def __init__(self, left, right, comparator, source_ref): ExpressionComparison.__init__( self, left = left, right = right, comparator = comparator, source_ref = source_ref ) assert comparator in ("Is", "IsNot") self.match_value = comparator == "Is" def getDetailsForDisplay(self): return ExpressionComparison.getDetails(self) def getDetails(self): return {} def isExpressionComparison(self): # Virtual method, pylint: disable=R0201 return True def mayRaiseException(self, exception_type): return self.getLeft().mayRaiseException(exception_type) or \ self.getRight().mayRaiseException(exception_type) def mayRaiseExceptionBool(self, exception_type): return False def computeExpression(self, constraint_collection): left, right = self.getOperands() if constraint_collection.mustAlias(left, right): result = makeConstantReplacementNode( constant = self.match_value, node = self ) if left.mayHaveSideEffects() or right.mayHaveSideEffects(): result = wrapExpressionWithSideEffects( side_effects = self.extractSideEffects(), old_node = self, new_node = result ) return result, "new_constant", """\ Determined values to alias and therefore result of %s comparison.""" % ( self.comparator ) if constraint_collection.mustNotAlias(left, right): result = makeConstantReplacementNode( constant = not self.match_value, node = self ) if left.mayHaveSideEffects() or right.mayHaveSideEffects(): result = wrapExpressionWithSideEffects( side_effects = self.extractSideEffects(), old_node = self, new_node = result ) return result, "new_constant", """\ Determined values to not alias and therefore result of %s comparison.""" % ( self.comparator ) return ExpressionComparison.computeExpression( self, constraint_collection = constraint_collection ) def extractSideEffects(self): left, right = self.getOperands() return left.extractSideEffects() + right.extractSideEffects() def computeExpressionDrop(self, statement, constraint_collection): from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions result = makeStatementOnlyNodesFromExpressions( expressions = self.getOperands() ) return result, "new_statements", """\ Removed %s comparison for unused result.""" % self.comparator class ExpressionComparisonIs(ExpressionComparisonIsIsNotBase): kind = "EXPRESSION_COMPARISON_IS" def __init__(self, left, right, source_ref): ExpressionComparisonIsIsNotBase.__init__( self, left = left, right = right, comparator = "Is", source_ref = source_ref ) class ExpressionComparisonIsNOT(ExpressionComparisonIsIsNotBase): kind = "EXPRESSION_COMPARISON_IS_NOT" def __init__(self, left, right, source_ref): ExpressionComparisonIsIsNotBase.__init__( self, left = left, right = right, comparator = "IsNot", source_ref = source_ref ) class ExpressionComparisonExceptionMatch(ExpressionComparison): kind = "EXPRESSION_COMPARISON_EXCEPTION_MATCH" def __init__(self, left, right, source_ref): ExpressionComparison.__init__( self, left = left, right = right, comparator = "exception_match", source_ref = source_ref ) def getDetails(self): return {} def isExpressionComparison(self): # Virtual method, pylint: disable=R0201 return True def getSimulator(self): assert False return PythonOperators.all_comparison_functions[self.comparator] Nuitka-0.5.18.1/nuitka/nodes/BuiltinTypeNodes.py0000644000175000017500000002275712651071040021660 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in type nodes tuple/list/float/int etc. These are all very simple and have predictable properties, because we know their type and that should allow some important optimizations. """ from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .NodeBases import ( ChildrenHavingMixin, ExpressionBuiltinSingleArgBase, ExpressionSpecBasedComputationMixin, NodeBase ) from .NodeMakingHelpers import ( makeConstantReplacementNode, wrapExpressionWithNodeSideEffects ) class ExpressionBuiltinTypeBase(ExpressionBuiltinSingleArgBase): pass class ExpressionBuiltinTuple(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_TUPLE" builtin_spec = BuiltinOptimization.builtin_tuple_spec class ExpressionBuiltinList(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_LIST" builtin_spec = BuiltinOptimization.builtin_list_spec class ExpressionBuiltinSet(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_SET" builtin_spec = BuiltinOptimization.builtin_set_spec class ExpressionBuiltinFloat(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_FLOAT" builtin_spec = BuiltinOptimization.builtin_float_spec class ExpressionBuiltinBool(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_BOOL" builtin_spec = BuiltinOptimization.builtin_bool_spec def computeExpression(self, constraint_collection): value = self.getValue() if value is not None: truth_value = self.getValue().getTruthValue() if truth_value is not None: result = wrapExpressionWithNodeSideEffects( new_node = makeConstantReplacementNode( constant = truth_value, node = self, ), old_node = self.getValue() ) return ( result, "new_constant", "Predicted truth value of built-in bool argument" ) return ExpressionBuiltinTypeBase.computeExpression(self, constraint_collection) class ExpressionBuiltinIntLongBase(ChildrenHavingMixin, NodeBase, ExpressionSpecBasedComputationMixin): named_children = ("value", "base") # Note: Version specific, may be allowed or not. try: int(base = 2) except TypeError: base_only_value = False else: base_only_value = True # To be overloaded by child classes with int/long. builtin = int def __init__(self, value, base, source_ref): NodeBase.__init__(self, source_ref = source_ref) if value is None and self.base_only_value: value = makeConstantReplacementNode( constant = '0', node = self ) ChildrenHavingMixin.__init__( self, values = { "value" : value, "base" : base } ) getValue = ChildrenHavingMixin.childGetter("value") getBase = ChildrenHavingMixin.childGetter("base") def computeExpression(self, constraint_collection): value = self.getValue() base = self.getBase() given_values = [] if value is None: if base is not None: if not self.base_only_value: return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin(base = 2), description = """\ %s built-in call with only base argument""" % self.builtin.__name__ ) given_values = () elif base is None: given_values = (value,) else: given_values = (value, base) return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = given_values ) class ExpressionBuiltinInt(ExpressionBuiltinIntLongBase): kind = "EXPRESSION_BUILTIN_INT" builtin_spec = BuiltinOptimization.builtin_int_spec builtin = int class ExpressionBuiltinUnicodeBase(ChildrenHavingMixin, NodeBase, ExpressionSpecBasedComputationMixin): named_children = ( "value", "encoding", "errors" ) def __init__(self, value, encoding, errors, source_ref): NodeBase.__init__( self, source_ref = source_ref ) ChildrenHavingMixin.__init__( self, values = { "value" : value, "encoding" : encoding, "errors" : errors } ) getValue = ChildrenHavingMixin.childGetter("value") getEncoding = ChildrenHavingMixin.childGetter("encoding") getErrors = ChildrenHavingMixin.childGetter("errors") def computeExpression(self, constraint_collection): args = [ self.getValue(), self.getEncoding(), self.getErrors() ] while args and args[-1] is None: del args[-1] for arg in args: # The value of that node escapes and could change its contents. constraint_collection.removeKnowledge(arg) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = tuple(args) ) if python_version < 300: class ExpressionBuiltinStr(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_STR" builtin_spec = BuiltinOptimization.builtin_str_spec def computeExpression(self, constraint_collection): new_node, change_tags, change_desc = ExpressionBuiltinTypeBase.computeExpression( self, constraint_collection ) if new_node is self: str_value = self.getValue().getStrValue() if str_value is not None: new_node = wrapExpressionWithNodeSideEffects( new_node = str_value, old_node = self.getValue() ) change_tags = "new_expression" change_desc = "Predicted 'str' built-in result" return new_node, change_tags, change_desc class ExpressionBuiltinLong(ExpressionBuiltinIntLongBase): kind = "EXPRESSION_BUILTIN_LONG" builtin_spec = BuiltinOptimization.builtin_long_spec builtin = long class ExpressionBuiltinUnicode(ExpressionBuiltinUnicodeBase): kind = "EXPRESSION_BUILTIN_UNICODE" builtin_spec = BuiltinOptimization.builtin_unicode_spec else: class ExpressionBuiltinStr(ExpressionBuiltinUnicodeBase): kind = "EXPRESSION_BUILTIN_STR" builtin_spec = BuiltinOptimization.builtin_str_spec class ExpressionBuiltinBytearray(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_BYTEARRAY" builtin_spec = BuiltinOptimization.builtin_bytearray_spec def __init__(self, value, source_ref): if value is None: value = ExpressionConstantRef( constant = b"", source_ref = source_ref ) ExpressionBuiltinTypeBase.__init__( self, value = value, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Quite impossible as this has a variable result, but we could # look at the arguments at least. return self, None, None class ExpressionBuiltinComplex(ChildrenHavingMixin, NodeBase, ExpressionSpecBasedComputationMixin): kind = "EXPRESSION_BUILTIN_COMPLEX" named_children = ( "real", "imag", ) builtin_spec = BuiltinOptimization.builtin_complex_spec def __init__(self, real, imag, source_ref): NodeBase.__init__( self, source_ref = source_ref ) ChildrenHavingMixin.__init__( self, values = { "real" : real, "imag" : imag, } ) def computeExpression(self, constraint_collection): start = self.getReal() stop = self.getImag() args = ( start, stop, ) return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = args ) getReal = ChildrenHavingMixin.childGetter("real") getImag = ChildrenHavingMixin.childGetter("imag") Nuitka-0.5.18.1/nuitka/nodes/DictionaryNodes.py0000644000175000017500000001640012651072046021510 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes that build dictionaries. The "pair" is a sub-structure of the dictionary, representing a key/value pair that is the child of the dictionary creation. """ from nuitka import Constants from nuitka.PythonVersions import python_version from .NodeBases import ( ExpressionChildrenHavingBase, SideEffectsFromChildrenMixin ) from .NodeMakingHelpers import ( makeConstantReplacementNode, makeStatementOnlyNodesFromExpressions ) class ExpressionKeyValuePair(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_KEY_VALUE_PAIR" if python_version < 350: named_children = ( "key", "value" ) else: named_children = ( "key", "value" ) def __init__(self, key, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "key" : key, "value" : value }, source_ref = source_ref ) getKey = ExpressionChildrenHavingBase.childGetter("key") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, constraint_collection): key = self.getKey() hashable = key.isKnownToBeHashable() # If not known to be hashable, that can raise an exception. if not hashable: constraint_collection.onExceptionRaiseExit( TypeError ) if hashable is False: # TODO: If it's not hashable, we should turn it into a raise, it's # just difficult to predict the exception value precisely, as it # could be e.g. (2, []), and should then complain about the list. pass return self, None, None def mayRaiseException(self, exception_type): key = self.getKey() return key.mayRaiseException(exception_type) or \ key.isKnownToBeHashable() is not True or \ self.getValue().mayRaiseException(exception_type) class ExpressionMakeDict(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_DICT" named_children = ( "pairs", ) def __init__(self, pairs, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "pairs" : tuple(pairs), }, source_ref = source_ref ) getPairs = ExpressionChildrenHavingBase.childGetter("pairs") def computeExpression(self, constraint_collection): pairs = self.getPairs() for pair in pairs: key = pair.getKey() # TODO: Mutable key should cause an exception raise to be produced. if not key.isExpressionConstantRef() or not key.isKnownToBeHashable(): return self, None, None value = pair.getValue() if not value.isExpressionConstantRef(): return self, None, None constant_value = Constants.createConstantDict( keys = [ pair.getKey().getConstant() for pair in pairs ], values = [ pair.getValue().getConstant() for pair in pairs ] ) new_node = makeConstantReplacementNode( constant = constant_value, node = self ) return new_node, "new_constant", """\ Created dictionary found to be constant.""" def mayRaiseException(self, exception_type): for pair in self.getPairs(): if pair.mayRaiseException(exception_type): return True return False def mayHaveSideEffectsBool(self): return False def isKnownToBeIterable(self, count): return count is None or count == len(self.getPairs()) def getIterationLength(self): pair_count = len(self.getPairs()) # Hashing may consume elements. if pair_count >= 2: return None else: return pair_count def getIterationMinLength(self): pair_count = len(self.getPairs()) if pair_count == 0: return 0 else: return 1 def getIterationMaxLength(self): return len(self.getPairs()) def canPredictIterationValues(self): # Dictionaries are fully predictable, pylint: disable=R0201 # TODO: For some things, that may not be true, when key collisions # happen for example. We will have to check that then. return True def getIterationValue(self, count): return self.getPairs()[count].getKey() def getTruthValue(self): return self.getIterationLength() > 0 def mayBeNone(self): return False def isMapping(self): # Dictionaries are always mappings, but this is a virtual method, # pylint: disable=R0201 return True def isMappingWithConstantStringKeys(self): for pair in self.getPairs(): key = pair.getKey() if not key.isExpressionConstantRef() or not key.isStringConstant(): return False return True def getMappingStringKeyPairs(self): return [ ( pair.getKey().getConstant(), pair.getValue() ) for pair in self.getPairs() ] def getMappingPairs(self): return self.getPairs() # TODO: Missing computeExpressionIter1 here. For now it would require us to # add lots of temporary variables for keys, which then becomes the tuple, # but for as long as we don't have efficient forward propagation of these, # we won't do that. Otherwise we loose execution order of values with them # remaining as side effects. We could limit ourselves to cases where # isMappingWithConstantStringKeys is true, or keys had no side effects, but # that feels wasted effort as we are going to have full propagation. def computeExpressionDrop(self, statement, constraint_collection): expressions = [] for pair in self.getPairs(): expressions.extend(pair.extractSideEffects()) result = makeStatementOnlyNodesFromExpressions( expressions = expressions ) return result, "new_statements", """\ Removed sequence creation for unused sequence.""" def computeExpressionIter1(self, iter_node, constraint_collection): return iter_node, None, None Nuitka-0.5.18.1/nuitka/nodes/NodeBases.py0000644000175000017500000015063412651072046020265 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node base classes. These classes provide the generic base classes available for nodes. """ from nuitka import Options, Tracing, TreeXML, Variables from nuitka.__past__ import iterItems from nuitka.Constants import isCompileTimeConstantValue from nuitka.containers.odict import OrderedDict from nuitka.PythonVersions import python_version from nuitka.utils.InstanceCounters import counted_del, counted_init from nuitka.VariableRegistry import addVariableUsage, removeVariableUsage from .NodeMakingHelpers import ( getComputationResult, wrapExpressionWithSideEffects ) class NodeCheckMetaClass(type): kinds = set() def __new__(cls, name, bases, dictionary): # This is in conflict with either PyDev or Pylint, pylint: disable=C0204 assert len(bases) == len(set(bases)) return type.__new__(cls, name, bases, dictionary) def __init__(cls, name, bases, dictionary): if not name.endswith("Base"): assert ("kind" in dictionary), name kind = dictionary["kind"] assert type(kind) is str, name assert kind not in NodeCheckMetaClass.kinds, name NodeCheckMetaClass.kinds.add(kind) def convert(value): if value in ("AND", "OR", "NOT"): return value else: return value.title() kind_to_name_part = "".join( [convert(x) for x in kind.split('_')] ) assert name.endswith(kind_to_name_part), \ (name, kind_to_name_part) # Automatically add checker methods for everything to the common # base class checker_method = "is" + kind_to_name_part def checkKind(self): return self.kind == kind if not hasattr(NodeBase, checker_method): setattr(NodeBase, checker_method, checkKind) type.__init__(cls, name, bases, dictionary) # For every node type, there is a test, and then some more members, # For Python2/3 compatible source, we create a base class that has the metaclass # used and doesn't require making a choice. NodeMetaClassBase = NodeCheckMetaClass("NodeMetaClassBase", (object,), {}) class NodeBase(NodeMetaClassBase): kind = None @counted_init def __init__(self, source_ref): # The base class has no __init__ worth calling. # Check source reference to meet basic standards, so we note errors # when they occur. assert source_ref is not None assert source_ref.line is not None self.parent = None self.source_ref = source_ref __del__ = counted_del() def __repr__(self): # This is to avoid crashes, because of bugs in detail. # pylint: disable=W0703 try: detail = self.getDetail() except Exception as e: detail = "detail raises exception %s" % e if not detail: return "" % self.getDescription() else: return "" % (self.getDescription(), detail) def getDescription(self): """ Description of the node, intended for use in __repr__ and graphical display. """ return "%s at %s" % (self.kind, self.source_ref.getAsString()) def getDetails(self): """ Details of the node, intended for re-creation. We are not using the pickle mechanisms, but this is basically part of what the constructor call needs. Real children will also be added. """ # Virtual method, pylint: disable=R0201 return {} def getDetailsForDisplay(self): """ Details of the node, intended for use in __repr__ and dumps. This is also used for XML. """ return self.getDetails() def getDetail(self): """ Details of the node, intended for use in __repr__ and graphical display. """ return str(self.getDetails())[1:-1] def makeClone(self): try: # Using star dictionary arguments here for generic use. result = self.__class__( source_ref = self.source_ref, **self.getDetails() ) except TypeError: print("Problem cloning", self.__class__) raise effective_source_ref = self.getCompatibleSourceReference() if effective_source_ref is not self.source_ref: result.setCompatibleSourceReference(effective_source_ref) return result def makeCloneAt(self, source_ref): result = self.makeClone() result.source_ref = source_ref return result def getParent(self): """ Parent of the node. Every node except modules have to have a parent. """ if self.parent is None and not self.isCompiledPythonModule(): # print self.getVisitableNodesNamed() assert False, (self, self.source_ref) return self.parent def getChildName(self): """ Return the role in the current parent, subject to changes. """ parent = self.getParent() for key, value in parent.child_values.items(): if self is value: return key if type(value) is tuple: if self in value: return key, value.index(self) # TODO: Not checking tuples yet return None def getChildNameNice(self): child_name = self.getChildName() if hasattr(self.parent, "nice_children"): return self.parent.nice_children[self.parent.named_children.index(child_name)] else: return child_name def getParentFunction(self): """ Return the parent that is a function. """ parent = self.getParent() while parent is not None and not parent.isExpressionFunctionBodyBase(): parent = parent.getParent() return parent def getParentModule(self): """ Return the parent that is module. """ parent = self while not parent.isCompiledPythonModule(): if hasattr(parent, "provider"): # After we checked, we can use it, will be much faster route # to take. parent = parent.provider else: parent = parent.getParent() return parent def isParentVariableProvider(self): # Check if it's a closure giver, in which cases it can provide variables, return isinstance(self, ClosureGiverNodeBase) def getParentVariableProvider(self): parent = self.getParent() while not parent.isParentVariableProvider(): parent = parent.getParent() return parent def getParentReturnConsumer(self): parent = self.getParent() while not parent.isParentVariableProvider() and \ not parent.isExpressionOutlineBody(): parent = parent.getParent() return parent def getParentStatementsFrame(self): current = self.getParent() while True: if current.isStatementsFrame(): return current if current.isParentVariableProvider(): return None if current.isExpressionOutlineBody(): return None current = current.getParent() def getSourceReference(self): return self.source_ref def setCompatibleSourceReference(self, source_ref): """ Bug compatible line numbers information. As CPython outputs the last bit of bytecode executed, and not the line of the operation. For example calls, output the line of the last argument, as opposed to the line of the operation start. For tests, we wants to be compatible. In improved more, we are not being fully compatible, and just drop it altogether. """ # Getting the same source reference can be dealt with quickly, so do # this first. if self.source_ref is not source_ref and \ Options.isFullCompat() and \ self.source_ref != source_ref: # An attribute outside of "__init__", so we save one memory for the # most cases. Very few cases involve splitting across lines. # pylint: disable=W0201 self.effective_source_ref = source_ref def getCompatibleSourceReference(self): """ Bug compatible line numbers information. See above. """ return getattr(self, "effective_source_ref", self.source_ref) def asXml(self): line = self.getSourceReference().getLineNumber() result = TreeXML.Element( "node", kind = self.__class__.__name__, line = "%s" % line ) compat_line = self.getCompatibleSourceReference().getLineNumber() if compat_line != line: result.attrib["compat_line"] = str(compat_line) for key, value in iterItems(self.getDetailsForDisplay()): value = str(value) if value.startswith('<') and value.endswith('>'): value = value[1:-1] result.set(key, str(value)) for name, children in self.getVisitableNodesNamed(): if type(children) not in (list, tuple): children = (children,) role = TreeXML.Element( "role", name = name ) result.append(role) for child in children: if child is not None: role.append( child.asXml() ) return result def asXmlText(self): xml = self.asXml() return TreeXML.toString(xml) def dump(self, level = 0): Tracing.printIndented(level, self) Tracing.printSeparator(level) for visitable in self.getVisitableNodes(): visitable.dump(level + 1) Tracing.printSeparator(level) @staticmethod def isCompiledPythonModule(): # For overload by module nodes return False def isExpression(self): return self.kind.startswith("EXPRESSION_") def isStatement(self): return self.kind.startswith("STATEMENT_") def isExpressionBuiltin(self): return self.kind.startswith("EXPRESSION_BUILTIN_") def isExpressionSideEffects(self): # Virtual method, pylint: disable=R0201 # We need to provide this, as these node kinds are only imported if # necessary, but we test against them. return False def isStatementReraiseException(self): # Virtual method, pylint: disable=R0201 return False def isExpressionMakeSequence(self): # Virtual method, pylint: disable=R0201 return False def isIteratorMaking(self): # Virtual method, pylint: disable=R0201 return False def isNumberConstant(self): # Virtual method, pylint: disable=R0201 return False def isExpressionCall(self): # Virtual method, pylint: disable=R0201 return False def visit(self, context, visitor): visitor(self) for visitable in self.getVisitableNodes(): visitable.visit(context, visitor) def getVisitableNodes(self): # Virtual method, pylint: disable=R0201 return () def getVisitableNodesNamed(self): # Virtual method, pylint: disable=R0201 return () def replaceWith(self, new_node): self.parent.replaceChild( old_node = self, new_node = new_node ) def getName(self): # Virtual method, pylint: disable=R0201 return None def mayHaveSideEffects(self): """ Unless we are told otherwise, everything may have a side effect. """ # Virtual method, pylint: disable=R0201 return True def isOrderRelevant(self): return self.mayHaveSideEffects() def mayHaveSideEffectsBool(self): """ Unless we are told otherwise, everything may have a side effect. """ # Virtual method, pylint: disable=R0201 return True def extractSideEffects(self): """ Unless defined otherwise, the expression is the side effect. """ return (self,) def mayRaiseException(self, exception_type): """ Unless we are told otherwise, everything may raise everything. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionBool(self, exception_type): """ Unless we are told otherwise, everything may raise being checked. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionIter(self, exception_type): """ Unless we are told otherwise, everything may raise being iterated. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionAttributeLookup(self, exception_type, attribute_name): """ Unless we are told otherwise, everything may raise for attribute access. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionAttributeLookupSpecial(self, exception_type, attribute_name): """ Unless we are told otherwise, everything may raise for attribute access. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionAttributeLookupObject(self, exception_type, attribute): """ Unless we are told otherwise, everything may raise for attribute access. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionAttributeCheck(self, exception_type, attribute_name): """ Unless we are told otherwise, everything may raise for attribute check. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayRaiseExceptionAttributeCheckObject(self, exception_type, attribute): """ Unless we are told otherwise, everything may raise for attribute check. """ # Virtual method, pylint: disable=R0201,W0613 return True def mayReturn(self): return "_RETURN" in self.kind def mayBreak(self): # For overload, pylint: disable=R0201 return False def mayContinue(self): # For overload, pylint: disable=R0201 return False def needsFrame(self): """ Unless we are tolder otherwise, this depends on exception raise. """ return self.mayRaiseException(BaseException) def willRaiseException(self, exception_type): """ Unless we are told otherwise, nothing may raise anything. """ # Virtual method, pylint: disable=R0201,W0613 return False def isIndexable(self): """ Unless we are told otherwise, it's not indexable. """ # Virtual method, pylint: disable=R0201 return False def isStatementAborting(self): """ Is the node aborting, control flow doesn't continue after this node. """ assert self.isStatement(), self.kind return False def needsLocalsDict(self): """ Node requires a locals dictionary by provider. """ # Virtual method, pylint: disable=R0201 return False def getIntegerValue(self): """ Node as integer value, if possible.""" # Virtual method, pylint: disable=R0201 return None def getIntValue(self): """ Value that "int" or "PyNumber_Int" (sp) would give, if known. Otherwise it is "None" to indicate unknown. Users must not forget to take side effects into account, when replacing a node with its string value. """ # Virtual method, pylint: disable=R0201 return None class CodeNodeBase(NodeBase): def __init__(self, name, code_prefix, source_ref): assert name is not None NodeBase.__init__( self, source_ref = source_ref ) self.name = name self.code_prefix = code_prefix # The code name is determined on demand only. self.code_name = None # The "UID" values of children kinds are kept here. self.uids = {} def getName(self): return self.name def getFullName(self): result = self.getName() current = self while True: current = current.getParent() if current is None: break name = current.getName() if name is not None: result = "%s__%s" % (name, result) assert '<' not in result, result return result def getCodeName(self): if self.code_name is None: provider = self.getParentVariableProvider() parent_name = provider.getCodeName() uid = "_%d" % provider.getChildUID(self) assert isinstance(self, CodeNodeBase) if self.name: name = uid + '_' + self.name.strip("<>") else: name = uid self.code_name = "%s%s_of_%s" % (self.code_prefix, name, parent_name) return self.code_name def getChildUID(self, node): if node.kind not in self.uids: self.uids[node.kind] = 0 self.uids[node.kind] += 1 return self.uids[node.kind] class ChildrenHavingMixin: named_children = () checkers = {} def __init__(self, values): assert type(self.named_children) is tuple and len(self.named_children) # Check for completeness of given values, everything should be there # but of course, might be put to None. assert set(values.keys()) == set(self.named_children) self.child_values = dict(values) for key, value in self.child_values.items(): if key in self.checkers: value = self.child_values[key] = self.checkers[key](value) assert type(value) is not list, key if type(value) is tuple: assert None not in value, key for val in value: val.parent = self elif value is not None: value.parent = self elif value is None: pass else: assert False, type(value) def setChild(self, name, value): """ Set a child value. Do not overload, provider self.checkers instead. """ # Only accept legal child names assert name in self.child_values, name # Lists as inputs are OK, but turn them into tuples. if type(value) is list: value = tuple(value) if name in self.checkers: value = self.checkers[name](value) # Re-parent value to us. if type(value) is tuple: for val in value: val.parent = self elif value is not None: value.parent = self # Determine old value, and inform it about loosing its parent. old_value = self.child_values[name] assert old_value is not value, value self.child_values[name] = value def getChild(self, name): # Only accept legal child names assert name in self.child_values, name return self.child_values[name] def hasChild(self, name): return name in self.child_values @staticmethod def childGetter(name): def getter(self): return self.getChild(name) return getter @staticmethod def childSetter(name): def setter(self, value): self.setChild(name, value) return setter def getVisitableNodes(self): result = [] for name in self.named_children: value = self.child_values[ name ] if value is None: pass elif type(value) is tuple: result += list(value) elif isinstance(value, NodeBase): result.append(value) else: raise AssertionError( self, "has illegal child", name, value, value.__class__ ) return tuple(result) def getVisitableNodesNamed(self): result = [] for name in self.named_children: value = self.child_values[ name ] result.append((name, value)) return result def replaceChild(self, old_node, new_node): if new_node is not None and not isinstance(new_node, NodeBase): raise AssertionError( "Cannot replace with", new_node, "old", old_node, "in", self ) # Find the replaced node, as an added difficulty, what might be # happening, is that the old node is an element of a tuple, in which we # may also remove that element, by setting it to None. for key, value in self.child_values.items(): if value is None: pass elif type(value) is tuple: if old_node in value: if new_node is not None: self.setChild( key, tuple( (val if val is not old_node else new_node) for val in value ) ) else: self.setChild( key, tuple( val for val in value if val is not old_node ) ) return key elif isinstance(value, NodeBase): if old_node is value: self.setChild(key, new_node) return key else: assert False, (key, value, value.__class__) raise AssertionError( "Didn't find child", old_node, "in", self ) def makeClone(self): values = {} for key, value in self.child_values.items(): assert type(value) is not list, key if value is None: values[key] = None elif type(value) is tuple: values[key] = tuple( v.makeClone() for v in value ) else: values[key] = value.makeClone() values.update( self.getDetails() ) try: # Using star dictionary arguments here for generic use, # pylint: disable=E1123 result = self.__class__( source_ref = self.source_ref, **values ) except TypeError: print("Problem cloning", self.__class__) raise effective_source_ref = self.getCompatibleSourceReference() if effective_source_ref is not self.source_ref: result.setCompatibleSourceReference(effective_source_ref) return result class ClosureGiverNodeBase(CodeNodeBase): """ Mix-in for nodes that provide variables for closure takers. """ def __init__(self, name, code_prefix, source_ref): CodeNodeBase.__init__( self, name = name, code_prefix = code_prefix, source_ref = source_ref ) self.providing = OrderedDict() self.temp_variables = OrderedDict() self.temp_scopes = OrderedDict() self.preserver_id = 0 def hasProvidedVariable(self, variable_name): return variable_name in self.providing def getProvidedVariable(self, variable_name): if variable_name not in self.providing: self.providing[variable_name] = self.createProvidedVariable( variable_name = variable_name ) return self.providing[variable_name] def createProvidedVariable(self, variable_name): # Virtual method, pylint: disable=R0201 assert type(variable_name) is str return None def registerProvidedVariables(self, *variables): for variable in variables: self.registerProvidedVariable(variable) def registerProvidedVariable(self, variable): assert variable is not None self.providing[variable.getName()] = variable def getProvidedVariables(self): return self.providing.values() def allocateTempScope(self, name, allow_closure = False): self.temp_scopes[name] = self.temp_scopes.get(name, 0) + 1 # TODO: Instead of using overly long code name, could just visit parents # and make sure to allocate the scope at the top. if allow_closure: return "%s_%s_%d" % ( self.getCodeName(), name, self.temp_scopes[name] ) else: return "%s_%d" % ( name, self.temp_scopes[name] ) def allocateTempVariable(self, temp_scope, name): if temp_scope is not None: full_name = "%s__%s" % ( temp_scope, name ) else: assert name != "result" full_name = name # No duplicates please. assert full_name not in self.temp_variables, full_name result = Variables.TempVariable( owner = self, variable_name = full_name ) self.temp_variables[full_name] = result addVariableUsage(result, self) return result def getTempVariable(self, temp_scope, name): if temp_scope is not None: full_name = "%s__%s" % (temp_scope, name) else: full_name = name return self.temp_variables[full_name] def getTempVariables(self): return tuple(self.temp_variables.values()) def removeTempVariable(self, variable): del self.temp_variables[variable.getName()] removeVariableUsage(variable, self) def allocatePreserverId(self): if python_version >= 300: self.preserver_id += 1 return self.preserver_id class ClosureTakerMixin: """ Mixin for nodes that accept variables from closure givers. """ def __init__(self, provider, early_closure): assert provider.isParentVariableProvider(), provider self.provider = provider self.early_closure = early_closure self.taken = set() self.temp_variables = set() def getParentVariableProvider(self): return self.provider def getClosureVariable(self, variable_name): result = self.provider.getVariableForClosure( variable_name = variable_name ) assert result is not None, variable_name # There is no maybe with closures. It means, it is closure variable in # this case. if result.isMaybeLocalVariable(): result = result.getMaybeVariable() if not result.isModuleVariable(): self.addClosureVariable(result) return result def addClosureVariable(self, variable): self.taken.add(variable) return variable def getClosureVariables(self): return tuple( sorted( [ take for take in self.taken if not take.isModuleVariable() ], key = lambda x : x.getName() ) ) def hasTakenVariable(self, variable_name): for variable in self.taken: if variable.getName() == variable_name: return True return False def getTakenVariable(self, variable_name): for variable in self.taken: if variable.getName() == variable_name: return variable return None def isEarlyClosure(self): """ Early closure taking means immediate binding of references. Normally it's good to lookup name references immediately, but not for functions. In case of a function body it is not allowed to do that, because a later assignment needs to be queried first. Nodes need to indicate via this if they would like to resolve references at the same time as assignments. """ return self.early_closure class ExpressionMixin: def isCompileTimeConstant(self): """ Has a value that we can use at compile time. Yes or no. If it has such a value, simulations can be applied at compile time and e.g. operations or conditions, or even calls may be executed against it. """ # Virtual method, pylint: disable=R0201 return False def getCompileTimeConstant(self): assert self.isCompileTimeConstant(), self assert False def getTruthValue(self): """ Return known truth value. The "None" value indicates unknown. """ if self.isCompileTimeConstant(): return bool(self.getCompileTimeConstant()) else: return None def mayBeNone(self): """ Could this evaluate to be "None". Yes or no. Defaults to pessimistic yes.""" # For overload, pylint: disable=R0201 return True def isKnownToBeIterable(self, count): """ Can be iterated at all (count is None) or exactly count times. Yes or no. If it can be iterated a known number of times, it may be asked to unpack itself. """ # Virtual method, pylint: disable=R0201,W0613 return False def isKnownToBeIterableAtMin(self, count): # Virtual method, pylint: disable=R0201,W0613 return False def isKnownToBeIterableAtMax(self, count): # Virtual method, pylint: disable=R0201,W0613 return False def getIterationLength(self): """ Value that "len" or "PyObject_Size" would give, if known. Otherwise it is "None" to indicate unknown. """ # Virtual method, pylint: disable=R0201 return None def getIterationMinLength(self): """ Value that "len" or "PyObject_Size" would give at minimum, if known. Otherwise it is "None" to indicate unknown. """ return self.getIterationLength() def getIterationMaxLength(self): """ Value that "len" or "PyObject_Size" would give at maximum, if known. Otherwise it is "None" to indicate unknown. """ return self.getIterationLength() def getStringValue(self): """ Node as string value, if possible.""" # Virtual method, pylint: disable=R0201 return None def getStrValue(self): """ Value that "str" or "PyObject_Str" would give, if known. Otherwise it is "None" to indicate unknown. Users must not forget to take side effects into account, when replacing a node with its string value. """ string_value = self.getStringValue() if string_value is not None: from .NodeMakingHelpers import makeConstantReplacementNode return makeConstantReplacementNode( node = self, constant = string_value ) return None def getTypeValue(self): """ Type of the node. """ from .TypeNodes import ExpressionBuiltinType1 return ExpressionBuiltinType1( value = self.makeClone(), source_ref = self.getSourceReference() ) def isKnownToBeHashable(self): """ Is the value hashable, i.e. suitable for dictionary/set keying.""" # Virtual method, pylint: disable=R0201 # Unknown by default. return None def onRelease(self, constraint_collection): # print "onRelease", self pass def computeExpressionRaw(self, constraint_collection): """ Compute an expression. Default behavior is to just visit the child expressions first, and then the node "computeExpression". For a few cases this needs to be overloaded, e.g. conditional expressions. """ # First apply the sub-expressions, as they are evaluated before. sub_expressions = self.getVisitableNodes() for count, sub_expression in enumerate(sub_expressions): assert sub_expression.isExpression(), (self, sub_expression) expression = constraint_collection.onExpression( expression = sub_expression ) if expression.willRaiseException(BaseException): wrapped_expression = wrapExpressionWithSideEffects( side_effects = sub_expressions[:count], old_node = sub_expression, new_node = expression ) return ( wrapped_expression, "new_raise", "For '%s' the expression '%s' will raise." % ( self.getChildNameNice(), expression.getChildNameNice() ) ) # Then ask ourselves to work on it. return self.computeExpression( constraint_collection = constraint_collection ) def isKnownToHaveAttribute(self, attribute_name): # Virtual method, pylint: disable=R0201,W0613 return None def computeExpressionAttribute(self, lookup_node, attribute_name, constraint_collection): # By default, an attribute lookup may change everything about the lookup # source. constraint_collection.removeKnowledge(self) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) if not self.isKnownToHaveAttribute(attribute_name): constraint_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionAttributeSpecial(self, lookup_node, attribute_name, constraint_collection): # By default, an attribute lookup may change everything about the lookup # source. Virtual method, pylint: disable=W0613 constraint_collection.removeKnowledge(lookup_node) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSetAttribute(self, set_node, attribute_name, value_node, constraint_collection): # By default, an attribute lookup may change everything about the lookup # source. Virtual method, pylint: disable=W0613 constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(value_node) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) # Better mechanics? return set_node, None, None def computeExpressionDelAttribute(self, set_node, attribute_name, constraint_collection): # By default, an attribute lookup may change everything about the lookup # source. Virtual method, pylint: disable=W0613 constraint_collection.removeKnowledge(self) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) # Better mechanics? return set_node, None, None def computeExpressionSubscript(self, lookup_node, subscript, constraint_collection): # By default, an subscript may change everything about the lookup # source. constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(subscript) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) constraint_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSetSubscript(self, set_node, subscript, value_node, constraint_collection): # By default, an subscript may change everything about the lookup # source. constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(subscript) constraint_collection.removeKnowledge(value_node) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionDelSubscript(self, set_node, subscript, constraint_collection): # By default, an subscript may change everything about the lookup # source. constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(subscript) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionSlice(self, lookup_node, lower, upper, constraint_collection): # By default, a slicing may change everything about the lookup source. constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(lower) constraint_collection.removeKnowledge(upper) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSetSlice(self, set_node, lower, upper, value_node, constraint_collection): # By default, an subscript may change everything about the lookup # source. constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(lower) constraint_collection.removeKnowledge(upper) constraint_collection.removeKnowledge(value_node) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionDelSlice(self, set_node, lower, upper, constraint_collection): # By default, an subscript may change everything about the lookup # source. constraint_collection.removeKnowledge(self) constraint_collection.removeKnowledge(lower) constraint_collection.removeKnowledge(upper) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): # The called and the arguments escape for good. self.onContentEscapes(constraint_collection) if call_args is not None: call_args.onContentEscapes(constraint_collection) if call_kw is not None: call_kw.onContentEscapes(constraint_collection) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return call_node, None, None def computeExpressionIter1(self, iter_node, constraint_collection): self.onContentEscapes(constraint_collection) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return iter_node, None, None def computeExpressionAsyncIter(self, iter_node, constraint_collection): self.onContentEscapes(constraint_collection) # Any code could be run, note that. constraint_collection.onControlFlowEscape(self) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return iter_node, None, None def computeExpressionOperationNot(self, not_node, constraint_collection): # Virtual method, pylint: disable=R0201 # The value of that node escapes and could change its contents. constraint_collection.removeKnowledge(not_node) # Any code could be run, note that. constraint_collection.onControlFlowEscape(not_node) # Any exception may be raised. constraint_collection.onExceptionRaiseExit(BaseException) return not_node, None, None def computeExpressionDrop(self, statement, constraint_collection): if not self.mayHaveSideEffects(): return None, "new_statements", "Removed statement without effect." return statement, None, None def onContentEscapes(self, constraint_collection): pass class CompileTimeConstantExpressionMixin(ExpressionMixin): # TODO: Do this for all computations, do this in the base class of all # nodes. computed_attribute = None def __init__(self): pass def isCompileTimeConstant(self): """ Has a value that we can use at compile time. Yes or no. If it has such a value, simulations can be applied at compile time and e.g. operations or conditions, or even calls may be executed against it. """ return True def isMutable(self): # Virtual method, pylint: disable=R0201 return False def mayHaveSideEffects(self): # Virtual method, pylint: disable=R0201 return False def mayHaveSideEffectsBool(self): # Virtual method, pylint: disable=R0201 return False def mayRaiseException(self, exception_type): # Virtual method, pylint: disable=R0201,W0613 return False def mayRaiseExceptionBool(self, exception_type): # Virtual method, pylint: disable=R0201,W0613 return False def mayRaiseExceptionAttributeLookup(self, exception_type, attribute_name): # Virtual method, pylint: disable=W0613 # We remember it from our computation. return not self.computed_attribute def mayRaiseExceptionAttributeLookupSpecial(self, exception_type, attribute_name): # Virtual method, pylint: disable=W0613 # We remember it from our computation. return not self.computed_attribute def mayRaiseExceptionAttributeCheck(self, exception_type): # Virtual method, pylint: disable=R0201,W0613 # Checking attributes of compile time constants never raises. return False def mayBeNone(self): return self.getCompileTimeConstant() is None def computeExpressionOperationNot(self, not_node, constraint_collection): return constraint_collection.getCompileTimeComputationResult( node = not_node, computation = lambda : not self.getCompileTimeConstant(), description = """\ Compile time constant negation truth value pre-computed.""" ) def isKnownToHaveAttribute(self, attribute_name): if self.computed_attribute is None: self.computed_attribute = hasattr(self.getCompileTimeConstant(), attribute_name) return self.computed_attribute def computeExpressionAttribute(self, lookup_node, attribute_name, constraint_collection): value = self.getCompileTimeConstant() if self.computed_attribute is None: self.computed_attribute = hasattr(value, attribute_name) # If it raises, or the attribute itself is a compile time constant, # then do execute it. if not self.computed_attribute or \ isCompileTimeConstantValue(getattr(value, attribute_name)): return constraint_collection.getCompileTimeComputationResult( node = lookup_node, computation = lambda : getattr(value, attribute_name), description = "Attribute lookup to '%s' pre-computed." % ( attribute_name ) ) return lookup_node, None, None def computeExpressionSubscript(self, lookup_node, subscript, constraint_collection): if subscript.isCompileTimeConstant(): return constraint_collection.getCompileTimeComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ subscript.getCompileTimeConstant() ], description = "Subscript of constant with constant value." ) # TODO: Look-up of subscript to index may happen. constraint_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSlice(self, lookup_node, lower, upper, constraint_collection): # TODO: Could be happy with predictable index values and not require # constants. if lower is not None: if upper is not None: if lower.isCompileTimeConstant() and upper.isCompileTimeConstant(): return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ lower.getCompileTimeConstant() : upper.getCompileTimeConstant() ], description = """\ Slicing of constant with constant indexes.""" ) else: if lower.isCompileTimeConstant(): return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ lower.getCompileTimeConstant() : ], description = """\ Slicing of constant with constant lower index only.""" ) else: if upper is not None: if upper.isCompileTimeConstant(): return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ : upper.getCompileTimeConstant() ], description = """\ Slicing of constant with constant upper index only.""" ) else: return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ : ], description = "Slicing of constant with no indexes." ) return lookup_node, None, None class ExpressionSpecBasedComputationMixin(ExpressionMixin): builtin_spec = None def computeBuiltinSpec(self, constraint_collection, given_values): assert self.builtin_spec is not None, self for value in given_values: if value is not None and not value.isCompileTimeConstant(): constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None if not self.builtin_spec.isCompileTimeComputable(given_values): constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_spec.simulateCall(given_values), description = "Built-in call to '%s' pre-computed." % ( self.builtin_spec.getName() ) ) class ExpressionChildrenHavingBase(ChildrenHavingMixin, NodeBase, ExpressionMixin): def __init__(self, values, source_ref): NodeBase.__init__( self, source_ref = source_ref ) ChildrenHavingMixin.__init__( self, values = values ) class StatementChildrenHavingBase(ChildrenHavingMixin, NodeBase): def __init__(self, values, source_ref): NodeBase.__init__(self, source_ref = source_ref) ChildrenHavingMixin.__init__( self, values = values ) def computeStatementSubExpressions(self, constraint_collection, expressions): """ Compute a statement. Default behavior is to just visit the child expressions first, and then the node "computeStatement". For a few cases this needs to be overloaded. """ for count, expression in enumerate(expressions): assert expression.isExpression(), (self, expression) expression = constraint_collection.onExpression( expression = expression ) if expression.willRaiseException(BaseException): wrapped_expression = wrapExpressionWithSideEffects( side_effects = expressions[:count], old_node = expression, new_node = expression ) return ( wrapped_expression, "new_raise", "For '%s' the expression '%s' will raise." % ( self.getChildNameNice(), expression.getChildNameNice() ) ) return self, None, None class ExpressionBuiltinNoArgBase(NodeBase, ExpressionMixin): def __init__(self, builtin_function, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.builtin_function = builtin_function def computeExpression(self, constraint_collection): # The lambda is there for make sure that no argument parsing will reach # the built-in function at all, pylint: disable=W0108 return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_function(), description = "No argument form of '%s' built-in" % self.builtin_function.__name__ ) class ExpressionBuiltinSingleArgBase(ExpressionChildrenHavingBase, ExpressionSpecBasedComputationMixin): named_children = ( "value", ) def __init__(self, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "value" : value, }, source_ref = source_ref ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, constraint_collection): value = self.getValue() if value is None: return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = () ) else: return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = (value,) ) class SideEffectsFromChildrenMixin: def mayHaveSideEffects(self): for child in self.getVisitableNodes(): if child.mayHaveSideEffects(): return True return False def extractSideEffects(self): # No side effects at all but from the children. result = [] for child in self.getVisitableNodes(): result.extend( child.extractSideEffects() ) return tuple(result) Nuitka-0.5.18.1/nuitka/nodes/ClassNodes.py0000644000175000017500000001234012651071040020440 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for classes and their creations. The classes are are at the core of the language and have their complexities. """ from nuitka.PythonVersions import python_version from .Checkers import checkStatementsSequenceOrNone from .FunctionNodes import ExpressionFunctionBodyBase from .IndicatorMixins import MarkLocalsDictIndicator from .NodeBases import ChildrenHavingMixin, ExpressionChildrenHavingBase class ExpressionClassBody(ExpressionFunctionBodyBase, MarkLocalsDictIndicator): kind = "EXPRESSION_CLASS_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } def __init__(self, provider, name, doc, flags, source_ref): while provider.isExpressionOutlineBody(): provider = provider.getParentVariableProvider() ExpressionFunctionBodyBase.__init__( self, provider = provider, name = name, is_class = True, code_prefix = "class", flags = flags, source_ref = source_ref ) MarkLocalsDictIndicator.__init__(self) self.doc = doc assert self.isEarlyClosure() def getDetails(self): return { "name" : self.getFunctionName(), "ref_name" : self.getCodeName(), "provider" : self.provider.getCodeName(), "doc" : self.doc } def getDetail(self): return "named %s" % self.getFunctionName() getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") def getDoc(self): return self.doc def getVariableForClosure(self, variable_name): # print( "getVariableForClosure", self, variable_name ) # The class bodies provide no closure, except under CPython3.x, there # they provide "__class__" but nothing else. if variable_name == "__class__": if python_version < 300: return self.provider.getVariableForClosure( "__class__" ) else: return ExpressionFunctionBodyBase.getVariableForClosure( self, variable_name = "__class__" ) else: return self.provider.getVariableForClosure( variable_name ) def markAsDirectlyCalled(self): pass def markAsExecContaining(self): pass def markAsUnqualifiedExecContaining(self, source_ref): pass def mayHaveSideEffects(self): # The function definition has no side effects, calculating the defaults # would be, but that is done outside of this. return False def mayRaiseException(self, exception_type): return self.getBody().mayRaiseException(exception_type) class ExpressionSelectMetaclass(ExpressionChildrenHavingBase): kind = "EXPRESSION_SELECT_METACLASS" named_children = ( "metaclass", "bases" ) def __init__(self, metaclass, bases, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "metaclass" : metaclass, "bases" : bases }, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Meta class selection is very computable, and should be done. return self, None, None getMetaclass = ExpressionChildrenHavingBase.childGetter("metaclass") getBases = ExpressionChildrenHavingBase.childGetter("bases") class ExpressionBuiltinType3(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_TYPE3" named_children = ("type_name", "bases", "dict") def __init__(self, type_name, bases, type_dict, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "type_name" : type_name, "bases" : bases, "dict" : type_dict }, source_ref = source_ref ) getTypeName = ExpressionChildrenHavingBase.childGetter("type_name") getBases = ExpressionChildrenHavingBase.childGetter("bases") getDict = ExpressionChildrenHavingBase.childGetter("dict") def computeExpression(self, constraint_collection): # TODO: Should be compile time computable if bases and dict are. return self, None, None Nuitka-0.5.18.1/nuitka/nodes/SideEffectNodes.py0000644000175000017500000000730512651071040021401 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node that models side effects. Sometimes, the effect of an expression needs to be had, but the value itself does not matter at all. """ from .NodeBases import ExpressionChildrenHavingBase from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions def checkSideEffects(value): real_value = [] for child in value: if child.isExpressionSideEffects(): real_value.extend(child.getSideEffects()) real_value.append(child.getExpression()) else: assert child.isExpression() real_value.append(child) return tuple(real_value) class ExpressionSideEffects(ExpressionChildrenHavingBase): kind = "EXPRESSION_SIDE_EFFECTS" named_children = ( "side_effects", "expression" ) checkers = { "side_effects" : checkSideEffects } def __init__(self, side_effects, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "side_effects" : tuple(side_effects), "expression" : expression }, source_ref = source_ref ) getSideEffects = ExpressionChildrenHavingBase.childGetter("side_effects") setSideEffects = ExpressionChildrenHavingBase.childSetter("side_effects") getExpression = ExpressionChildrenHavingBase.childGetter("expression") def isExpressionSideEffects(self): return True def computeExpression(self, constraint_collection): side_effects = self.getSideEffects() new_side_effects = [] for side_effect in side_effects: if side_effect.mayHaveSideEffects(): new_side_effects.append(side_effect) expression = self.getExpression() if expression.isExpressionSideEffects(): new_side_effects.extend( expression.getSideEffects() ) expression.setSideEffects(new_side_effects) return expression, "new_expression", "Remove nested side effects." if new_side_effects != side_effects: self.setSideEffects(new_side_effects) if not new_side_effects: return expression, "new_expression", "Removed empty side effects." return self, None, None def willRaiseException(self, exception_type): for child in self.getVisitableNodes(): if child.willRaiseException(exception_type): return True return False def getTruthValue(self): return self.getExpression().getTruthValue() def computeExpressionDrop(self, statement, constraint_collection): # Side effects can become statements. expressions = self.getSideEffects() + (self.getExpression(),) result = makeStatementOnlyNodesFromExpressions( expressions = expressions ) return result, "new_statements", """\ Turned side effects of expression only statement into statements.""" Nuitka-0.5.18.1/nuitka/nodes/CallNodes.py0000644000175000017500000001503512651071040020252 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Call node Function calls and generally calling expressions are the same thing. This is very important, because it allows to predict most things, and avoid expensive operations like parameter parsing at run time. There will be a method "computeExpressionCall" to aid predicting them in other nodes. """ from .NodeBases import ExpressionChildrenHavingBase class ExpressionCall(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL" named_children = ( "called", "args", "kw" ) def __init__(self, called, args, kw, source_ref): assert called.isExpression() assert args.isExpression() assert kw.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, "args" : args, "kw" : kw, }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") getCallArgs = ExpressionChildrenHavingBase.childGetter("args") getCallKw = ExpressionChildrenHavingBase.childGetter("kw") def isExpressionCall(self): return True def computeExpression(self, constraint_collection): called = self.getCalled() return called.computeExpressionCall( call_node = self, call_args = self.getCallArgs(), call_kw = self.getCallKw(), constraint_collection = constraint_collection ) def extractPreCallSideEffects(self): args = self.getCallArgs() kw = self.getCallKw() return args.extractSideEffects() + kw.extractSideEffects() class ExpressionCallNoKeywords(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL_NO_KEYWORDS" named_children = ( "called", "args" ) def __init__(self, called, args, source_ref): assert called.isExpression() assert args.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, "args" : args }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") getCallArgs = ExpressionChildrenHavingBase.childGetter("args") def computeExpression(self, constraint_collection): return self.getCalled().computeExpressionCall( call_node = self, call_args = self.getCallArgs(), call_kw = None, constraint_collection = constraint_collection ) @staticmethod def getCallKw(): return None def isExpressionCall(self): return True def extractPreCallSideEffects(self): args = self.getCallArgs() return args.extractSideEffects() class ExpressionCallKeywordsOnly(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL_KEYWORDS_ONLY" named_children = ( "called", "kw" ) def __init__(self, called, kw, source_ref): assert called.isExpression() assert kw.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, "kw" : kw }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") getCallKw = ExpressionChildrenHavingBase.childGetter("kw") def computeExpression(self, constraint_collection): called = self.getCalled() return called.computeExpressionCall( call_node = self, call_args = None, call_kw = self.getCallKw(), constraint_collection = constraint_collection ) @staticmethod def getCallArgs(): return None def isExpressionCall(self): return True def extractPreCallSideEffects(self): kw = self.getCallKw() return kw.extractSideEffects() class ExpressionCallEmpty(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL_EMPTY" named_children = ( "called", ) def __init__(self, called, source_ref): assert called.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") def computeExpression(self, constraint_collection): called = self.getCalled() return called.computeExpressionCall( call_node = self, call_args = None, call_kw = None, constraint_collection = constraint_collection ) @staticmethod def getCallKw(): return None @staticmethod def getCallArgs(): return None def isExpressionCall(self): return True @staticmethod def extractPreCallSideEffects(): return () def makeExpressionCall(called, args, kw, source_ref): """ Make the most simple call node possible. By avoiding the more complex classes, we can achieve that there is less work to do for analysis. """ has_kw = not kw.isExpressionConstantRef() or kw.getConstant() != {} has_args = not args.isExpressionConstantRef() or args.getConstant() != () if has_kw: if has_args: return ExpressionCall(called, args, kw, source_ref) else: return ExpressionCallKeywordsOnly(called, kw, source_ref) else: if has_args: return ExpressionCallNoKeywords(called, args, source_ref) else: return ExpressionCallEmpty(called, source_ref) Nuitka-0.5.18.1/nuitka/nodes/FunctionNodes.py0000644000175000017500000006523712651072046021204 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for functions and their creations. Lambdas are functions too. The functions are at the core of the language and have their complexities. Creating a CPython function object is an optional thing. Some things might only be used to be called directly, while knowing exactly what it is. So the "ExpressionFunctionCreation" might be used to provide that kind of CPython reference, and may escape. Coroutines and generators live in their dedicated module and share base classes. """ from nuitka import Options, VariableRegistry, Variables from nuitka.optimizations.FunctionInlining import convertFunctionCallToOutline from nuitka.PythonVersions import python_version from nuitka.tree.Extractions import updateVariableUsage from .Checkers import checkStatementsSequenceOrNone from .IndicatorMixins import ( MarkLocalsDictIndicator, MarkUnoptimizedFunctionIndicator ) from .NodeBases import ( ChildrenHavingMixin, ClosureGiverNodeBase, ClosureTakerMixin, CompileTimeConstantExpressionMixin, ExpressionChildrenHavingBase, ExpressionMixin, NodeBase, SideEffectsFromChildrenMixin ) from .NodeMakingHelpers import ( makeConstantReplacementNode, makeRaiseExceptionReplacementExpressionFromInstance, wrapExpressionWithSideEffects ) from .ParameterSpecs import TooManyArguments, matchCall class ExpressionFunctionBodyBase(ClosureTakerMixin, ChildrenHavingMixin, ClosureGiverNodeBase, ExpressionMixin): def __init__(self, provider, name, code_prefix, is_class, flags, source_ref): ClosureTakerMixin.__init__( self, provider = provider, early_closure = is_class ) ClosureGiverNodeBase.__init__( self, name = name, code_prefix = code_prefix, source_ref = source_ref ) ChildrenHavingMixin.__init__( self, values = { "body" : None # delayed } ) self.flags = flags # Hack: This allows some APIs to work although this is not yet # officially a child yet. Important during building. self.parent = provider # Python3.4: Might be overridden by global statement on the class name. # TODO: Make this class only code. if python_version >= 340: self.qualname_provider = provider # Non-local declarations. self.non_local_declarations = [] # Register ourselves immediately with the module. provider.getParentModule().addFunction(self) self.constraint_collection = None @staticmethod def isExpressionFunctionBodyBase(): return True def getContainingClassDictCreation(self): current = self while not current.isCompiledPythonModule(): if current.isExpressionClassBody(): return current current = current.getParentVariableProvider() return None def hasFlag(self, flag): return flag in self.flags def getLocalsMode(self): if python_version >= 300: return "updated" elif self.isEarlyClosure() or self.isUnoptimized(): return "updated" else: return "copy" def hasVariableName(self, variable_name): return variable_name in self.providing or variable_name in self.temp_variables def getVariables(self): return self.providing.values() def getLocalVariables(self): return [ variable for variable in self.providing.values() if variable.isLocalVariable() ] def getUserLocalVariables(self): return tuple( variable for variable in self.providing.values() if variable.isLocalVariable() and not variable.isParameterVariable() if variable.getOwner() is self ) def removeClosureVariable(self, variable): assert variable in self.providing.values(), (self.providing, variable) del self.providing[variable.getName()] assert not variable.isParameterVariable() or \ variable.getOwner() is not self self.taken.remove(variable) VariableRegistry.removeVariableUsage(variable, self) def demoteClosureVariable(self, variable): assert variable.isLocalVariable() self.taken.remove(variable) assert variable.getOwner() is not self new_variable = Variables.LocalVariable( owner = self, variable_name = variable.getName() ) self.providing[variable.getName()] = new_variable updateVariableUsage( provider = self, old_variable = variable, new_variable = new_variable ) VariableRegistry.removeVariableUsage(variable, self) VariableRegistry.addVariableUsage(new_variable, self) def removeUserVariable(self, variable): assert variable in self.providing.values(), (self.providing, variable) del self.providing[variable.getName()] assert not variable.isParameterVariable() or \ variable.getOwner() is not self VariableRegistry.removeVariableUsage(variable, self) def getVariableForAssignment(self, variable_name): # print("ASS func", self, variable_name) if self.hasTakenVariable(variable_name): result = self.getTakenVariable(variable_name) else: result = self.getProvidedVariable(variable_name) return result def getVariableForReference(self, variable_name): # print( "REF func", self, variable_name ) if self.hasProvidedVariable(variable_name): result = self.getProvidedVariable(variable_name) else: result = self.getClosureVariable( variable_name = variable_name ) # Remember that we need that closure variable for something, so # we don't create it again all the time. if not result.isModuleVariable(): self.registerProvidedVariable(result) # For "exec" containing/star import containing, we get a # closure variable already, but if it is a module variable, # only then make it a maybe local variable. if not self.isExpressionClassBody() and self.isUnoptimized() and result.isModuleVariable(): result = Variables.MaybeLocalVariable( owner = self, maybe_variable = result ) self.registerProvidedVariable(result) return result def getVariableForClosure(self, variable_name): # print( "getVariableForClosure", self, variable_name ) if self.hasProvidedVariable(variable_name): return self.getProvidedVariable(variable_name) else: return self.provider.getVariableForClosure(variable_name) def createProvidedVariable(self, variable_name): # print("createProvidedVariable", self, variable_name) return Variables.LocalVariable( owner = self, variable_name = variable_name ) def addNonlocalsDeclaration(self, names, source_ref): self.non_local_declarations.append( (names, source_ref) ) def getNonlocalDeclarations(self): return self.non_local_declarations def getFunctionName(self): return self.name def getFunctionQualname(self): """ Function __qualname__ new in CPython3.3 Should contain some kind of full name descriptions for the closure to recognize and will be used for outputs. """ function_name = self.getFunctionName() if python_version < 340: provider = self.getParentVariableProvider() else: provider = self.qualname_provider if provider.isCompiledPythonModule(): return function_name elif provider.isExpressionClassBody(): return provider.getFunctionQualname() + '.' + function_name else: return provider.getFunctionQualname() + ".." + function_name def computeExpression(self, constraint_collection): assert False # Function body is quite irreplaceable. return self, None, None def mayRaiseException(self, exception_type): body = self.getBody() if body is None: return False else: return self.getBody().mayRaiseException(exception_type) class ExpressionFunctionBody(ExpressionFunctionBodyBase, MarkLocalsDictIndicator, MarkUnoptimizedFunctionIndicator): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, leading to many methods, pylint: disable=R0901 kind = "EXPRESSION_FUNCTION_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } if python_version >= 340: qualname_setup = None def __init__(self, provider, name, doc, parameters, flags, source_ref): while provider.isExpressionOutlineBody(): provider = provider.getParentVariableProvider() if name == "": assert python_version >= 300 ExpressionFunctionBodyBase.__init__( self, provider = provider, name = name, code_prefix = "function", is_class = False, flags = flags, source_ref = source_ref ) MarkLocalsDictIndicator.__init__(self) MarkUnoptimizedFunctionIndicator.__init__(self) self.doc = doc # Indicator if the return value exception might be required. self.return_exception = False # Indicator if the function needs to be created as a function object. self.needs_creation = False # Indicator if the function is called directly. self.needs_direct = False # Indicator if the function is used outside of where it's defined. self.cross_module_use = False self.parameters = parameters self.parameters.setOwner(self) self.registerProvidedVariables( *self.parameters.getAllVariables() ) def getDetails(self): return { "name" : self.getFunctionName(), "ref_name" : self.getCodeName(), "parameters" : self.getParameters(), "provider" : self.provider.getCodeName(), "doc" : self.doc } def getDetail(self): return "named %s with %s" % (self.getFunctionName(), self.parameters) def getParent(self): assert False def getDoc(self): return self.doc def getParameters(self): return self.parameters getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") def needsCreation(self): return self.needs_creation def markAsNeedsCreation(self): self.needs_creation = True def needsDirectCall(self): return self.needs_direct def markAsDirectlyCalled(self): self.needs_direct = True def isCrossModuleUsed(self): return self.cross_module_use def markAsCrossModuleUsed(self): self.cross_module_use = True def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): # TODO: Until we have something to re-order the arguments, we need to # skip this. For the immediate need, we avoid this complexity, as a # re-ordering will be needed. assert False, self def isCompileTimeConstant(self): # TODO: It's actually pretty much compile time accessible maybe. return None def mayHaveSideEffects(self): # The function definition has no side effects, calculating the defaults # would be, but that is done outside of this. return False def mayRaiseException(self, exception_type): return self.getBody().mayRaiseException(exception_type) def markAsExceptionReturnValue(self): self.return_exception = True def needsExceptionReturnValue(self): return self.return_exception def convertNoneConstantOrEmptyDictToNone(node): if node is None: return None elif node.isExpressionConstantRef() and node.getConstant() is None: return None elif node.isExpressionConstantRef() and node.getConstant() == {}: return None else: return node # TODO: Function direct call node ought to be here too. class ExpressionFunctionCreation(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_FUNCTION_CREATION" # Note: The order of evaluation for these is a bit unexpected, but # true. Keyword defaults go first, then normal defaults, and annotations of # all kinds go last. # A bug of CPython3.x not fixed before version 3.4, see bugs.python.org/issue16967 kw_defaults_before_defaults = python_version < 340 if kw_defaults_before_defaults: named_children = ( "kw_defaults", "defaults", "annotations", "function_ref" ) else: named_children = ( "defaults", "kw_defaults", "annotations", "function_ref" ) checkers = { "kw_defaults" : convertNoneConstantOrEmptyDictToNone, } def __init__(self, function_ref, code_object, defaults, kw_defaults, annotations, source_ref): assert kw_defaults is None or kw_defaults.isExpression() assert annotations is None or annotations.isExpression() assert function_ref.isExpressionFunctionRef() ExpressionChildrenHavingBase.__init__( self, values = { "function_ref" : function_ref, "defaults" : tuple(defaults), "kw_defaults" : kw_defaults, "annotations" : annotations }, source_ref = source_ref ) self.code_object = code_object def getName(self): return self.getFunctionRef().getName() def getCodeObject(self): return self.code_object def getDetails(self): return { "code_object" : self.code_object } def computeExpression(self, constraint_collection): defaults = self.getDefaults() side_effects = [] for default in defaults: if default.willRaiseException(BaseException): result = wrapExpressionWithSideEffects( side_effects = side_effects, old_node = self, new_node = default ) return result, "new_raise", "Default value contains raise." kw_defaults = self.getKwDefaults() if kw_defaults is not None: if kw_defaults.willRaiseException(BaseException): result = wrapExpressionWithSideEffects( side_effects = side_effects, old_node = self, new_node = kw_defaults ) return result, "new_raise", "Keyword default values contain raise." side_effects.append(kw_defaults) annotations = self.getAnnotations() if annotations is not None and annotations.willRaiseException(BaseException): result = wrapExpressionWithSideEffects( side_effects = side_effects, old_node = self, new_node = annotations ) return result, "new_raise", "Annotation values contain raise." # TODO: Function body may know something too. return self, None, None getFunctionRef = ExpressionChildrenHavingBase.childGetter("function_ref") getDefaults = ExpressionChildrenHavingBase.childGetter("defaults") getKwDefaults = ExpressionChildrenHavingBase.childGetter("kw_defaults") getAnnotations = ExpressionChildrenHavingBase.childGetter("annotations") def mayRaiseException(self, exception_type): for default in self.getDefaults(): if default.mayRaiseException(exception_type): return True kw_defaults = self.getKwDefaults() if kw_defaults is not None and kw_defaults.mayRaiseException(exception_type): return True annotations = self.getAnnotations() if annotations is not None and annotations.mayRaiseException(exception_type): return True return False def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) # TODO: Until we have something to re-order the keyword arguments, we # need to skip this. For the immediate need, we avoid this complexity, # as a re-ordering will be needed. if call_kw is not None and \ (not call_kw.isExpressionConstantRef() or call_kw.getConstant() != {}): return call_node, None, None if call_kw is not None: return call_node, None, None if call_args is None: args_tuple = () elif call_args.isExpressionConstantRef() or \ call_args.isExpressionMakeTuple(): args_tuple = call_args.getIterationValues() else: # TODO: Can this even happen, i.e. does the above check make # sense. assert False, call_args return call_node, None, None function_body = self.getFunctionRef().getFunctionBody() # TODO: Actually the above disables it entirely, as it is at least # the empty dictionary node in any case. We will need some enhanced # interfaces for "matchCall" to work on. call_spec = function_body.getParameters() try: args_dict = matchCall( func_name = self.getName(), args = call_spec.getArgumentNames(), star_list_arg = call_spec.getStarListArgumentName(), star_dict_arg = call_spec.getStarDictArgumentName(), num_defaults = call_spec.getDefaultCount(), positional = args_tuple, pairs = () ) values = [ args_dict[name] for name in call_spec.getParameterNames() ] result = ExpressionFunctionCall( function = self, values = values, source_ref = call_node.getSourceReference() ) return ( result, "new_statements", # TODO: More appropriate tag maybe. """\ Replaced call to created function body '%s' with direct \ function call.""" % self.getName() ) except TooManyArguments as e: result = wrapExpressionWithSideEffects( new_node = makeRaiseExceptionReplacementExpressionFromInstance( expression = call_node, exception = e.getRealException() ), old_node = call_node, side_effects = call_node.extractPreCallSideEffects() ) return ( result, "new_raise", # TODO: More appropriate tag maybe. """Replaced call to created function body '%s' to argument \ error""" % self.getName() ) def getCallCost(self, values): # TODO: Ought to use values. If they are all constant, how about we # assume no cost, pylint: disable=W0613 if not Options.isExperimental(): return None function_body = self.getFunctionRef().getFunctionBody() if function_body.isExpressionGeneratorObjectBody(): # TODO: That's not even allowed, is it? assert False return None if function_body.isExpressionClassBody(): return None # TODO: Lying for the demo, this is too limiting, but needs frames to # be allowed twice in a context. if function_body.mayRaiseException(BaseException): return 60 return 20 def createOutlineFromCall(self, provider, values): return convertFunctionCallToOutline( provider = provider, function_ref = self.getFunctionRef(), values = values ) class ExpressionFunctionRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_FUNCTION_REF" def __init__(self, function_body, source_ref): assert function_body.isExpressionFunctionBody() or \ function_body.isExpressionClassBody() or \ function_body.isExpressionGeneratorObjectBody() or \ function_body.isExpressionCoroutineObjectBody() NodeBase.__init__( self, source_ref = source_ref ) self.function_body = function_body def getName(self): return self.function_body.getName() def getDetails(self): return { "function_body" : self.function_body } def getDetailsForDisplay(self): return { "function" : self.function_body.getCodeName() } def getFunctionBody(self): return self.function_body def computeExpressionRaw(self, constraint_collection): function_body = self.getFunctionBody() owning_module = function_body.getParentModule() # Make sure the owning module is added to the used set. This is most # important for helper functions, or modules, which otherwise have # become unused. from nuitka.ModuleRegistry import addUsedModule addUsedModule(owning_module) owning_module.addUsedFunction(function_body) from nuitka.optimizations.TraceCollections import \ ConstraintCollectionFunction # TODO: Doesn't this mean, we can do this multiple times by doing it # in the reference. We should do it in the body, and there we should # limit us to only doing it once per module run, e.g. being based on # presence in used functions of the module already. old_collection = function_body.constraint_collection function_body.constraint_collection = ConstraintCollectionFunction( parent = constraint_collection, function_body = function_body ) statements_sequence = function_body.getBody() if statements_sequence is not None and \ not statements_sequence.getStatements(): function_body.setStatements(None) statements_sequence = None if statements_sequence is not None: result = statements_sequence.computeStatementsSequence( constraint_collection = function_body.constraint_collection ) if result is not statements_sequence: function_body.setBody(result) function_body.constraint_collection.updateFromCollection(old_collection) # TODO: Function collection may now know something. return self, None, None def mayHaveSideEffects(self): # Using a function has no side effects. return False class ExpressionFunctionCall(ExpressionChildrenHavingBase): """ Shared function call. This is for calling created function bodies with multiple users. Not clear if such a thing should exist. But what this will do is to have respect for the fact that there are multiple such calls. """ kind = "EXPRESSION_FUNCTION_CALL" named_children = ( "function", "values" ) def __init__(self, function, values, source_ref): assert function.isExpressionFunctionCreation() ExpressionChildrenHavingBase.__init__( self, values = { "function" : function, "values" : tuple(values), }, source_ref = source_ref ) def computeExpression(self, constraint_collection): function = self.getFunction() values = self.getArgumentValues() # TODO: This needs some design. cost = function.getCallCost(values) if function.getFunctionRef().getFunctionBody().mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit(BaseException) if cost is not None and cost < 50: result = function.createOutlineFromCall( provider = self.getParentVariableProvider(), values = values ) return result, "new_statements", "Function call in-lined." return self, None, None def mayRaiseException(self, exception_type): function = self.getFunction() if function.getFunctionRef().getFunctionBody().mayRaiseException(exception_type): return True values = self.getArgumentValues() for value in values: if value.mayRaiseException(exception_type): return True return False getFunction = ExpressionChildrenHavingBase.childGetter("function") getArgumentValues = ExpressionChildrenHavingBase.childGetter("values") # Needed for Python3.3 and higher class ExpressionFunctionQualnameRef(CompileTimeConstantExpressionMixin, NodeBase): kind = "EXPRESSION_FUNCTION_QUALNAME_REF" def __init__(self, function_body, source_ref): NodeBase.__init__(self, source_ref = source_ref) CompileTimeConstantExpressionMixin.__init__(self) self.function_body = function_body def computeExpression(self, constraint_collection): result = makeConstantReplacementNode( node = self, constant = self.function_body.getFunctionQualname() ) return result, "new_constant", "Delayed __qualname__ resolution." Nuitka-0.5.18.1/nuitka/nodes/StatementNodes.py0000644000175000017500000002473712651071040021354 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for statements. """ from .NodeBases import NodeBase, StatementChildrenHavingBase def checkStatements(value): """ Check that statements list value property. Must not be None, must not contain None, and of course only statements, may be empty. """ assert value is not None assert None not in value for statement in value: assert statement.isStatement() or statement.isStatementsFrame(), \ statement.asXmlText() return tuple(value) class StatementsSequence(StatementChildrenHavingBase): kind = "STATEMENTS_SEQUENCE" named_children = ( "statements", ) checkers = { "statements" : checkStatements } def __init__(self, statements, source_ref): StatementChildrenHavingBase.__init__( self, values = { "statements" : statements }, source_ref = source_ref ) getStatements = StatementChildrenHavingBase.childGetter("statements") setStatements = StatementChildrenHavingBase.childSetter("statements") def getDetailsForDisplay(self): if self.getStatements(): return { "statement_count" : len(self.getStatements()) } else: return { "statement_count" : 0 } # Overloading name based automatic check, so that derived ones know it too. def isStatementsSequence(self): # Virtual method, pylint: disable=R0201 return True def trimStatements(self, statement): assert statement.parent is self old_statements = list(self.getStatements()) assert statement in old_statements, \ (statement, self) new_statements = old_statements[ : old_statements.index(statement)+1 ] self.setChild("statements", new_statements) def removeStatement(self, statement): assert statement.parent is self statements = list(self.getStatements()) statements.remove(statement) self.setChild("statements", statements) def mergeStatementsSequence(self, statement_sequence): assert statement_sequence.parent is self old_statements = list(self.getStatements()) assert statement_sequence in old_statements, \ (statement_sequence, self) merge_index = old_statements.index(statement_sequence) new_statements = tuple(old_statements[ : merge_index ]) + \ statement_sequence.getStatements() + \ tuple(old_statements[ merge_index+1 : ]) self.setChild("statements", new_statements) def mayHaveSideEffects(self): # Statement sequences have a side effect if one of the statements does. for statement in self.getStatements(): if statement.mayHaveSideEffects(): return True return False def mayRaiseException(self, exception_type): for statement in self.getStatements(): if statement.mayRaiseException(exception_type): return True return False def needsFrame(self): for statement in self.getStatements(): if statement.needsFrame(): return True return False def mayReturn(self): for statement in self.getStatements(): if statement.mayReturn(): return True return False def mayBreak(self): for statement in self.getStatements(): if statement.mayBreak(): return True return False def mayContinue(self): for statement in self.getStatements(): if statement.mayContinue(): return True return False def mayRaiseExceptionOrAbort(self, exception_type): return self.mayRaiseException(exception_type) or \ self.mayReturn() or \ self.mayBreak() or \ self.mayContinue() def isStatementAborting(self): return self.getStatements()[-1].isStatementAborting() def computeStatement(self, constraint_collection): # Don't want to be called like this. assert False, self def computeStatementsSequence(self, constraint_collection): new_statements = [] statements = self.getStatements() assert statements, self for count, statement in enumerate(statements): # May be frames embedded. if statement.isStatementsFrame(): new_statement = statement.computeStatementsSequence( constraint_collection ) else: new_statement = constraint_collection.onStatement( statement = statement ) if new_statement is not None: if new_statement.isStatementsSequence() and \ not new_statement.isStatementsFrame(): new_statements.extend( new_statement.getStatements() ) else: new_statements.append( new_statement ) if statement is not statements[-1] and \ new_statement.isStatementAborting(): constraint_collection.signalChange( "new_statements", statements[count+1].getSourceReference(), "Removed dead statements." ) break if statements != new_statements: if new_statements: self.setStatements(new_statements) return self else: return None else: return self class StatementExpressionOnly(StatementChildrenHavingBase): kind = "STATEMENT_EXPRESSION_ONLY" named_children = ( "expression", ) def __init__(self, expression, source_ref): assert expression.isExpression() StatementChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) def getDetail(self): return "expression %s" % self.getExpression() def mayHaveSideEffects(self): return self.getExpression().mayHaveSideEffects() def mayRaiseException(self, exception_type): return self.getExpression().mayRaiseException(exception_type) getExpression = StatementChildrenHavingBase.childGetter( "expression" ) def computeStatement(self, constraint_collection): constraint_collection.onExpression( expression = self.getExpression() ) expression = self.getExpression() if expression.mayRaiseException(BaseException): constraint_collection.onExceptionRaiseExit(BaseException) result, change_tags, change_desc = expression.computeExpressionDrop( statement = self, constraint_collection = constraint_collection ) if result is not self: return result, change_tags, change_desc return self, None, None class StatementGeneratorEntry(NodeBase): kind = "STATEMENT_GENERATOR_ENTRY" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def mayRaiseException(self, exception_type): # Anything might be thrown into a generator, representing that is the # whole point of this statement. return True def computeStatement(self, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) # Nothing we can about it. return self, None, None class StatementPreserveFrameException(NodeBase): kind = "STATEMENT_PRESERVE_FRAME_EXCEPTION" def __init__(self, preserver_id, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.preserver_id = preserver_id def getDetails(self): return { "preserver_id" : self.preserver_id } def getPreserverId(self): return self.preserver_id def computeStatement(self, constraint_collection): # For Python2 generators, it's not necessary to preserve, the frame # decides it. TODO: This check makes only sense once. if self.getParentStatementsFrame().needsExceptionFramePreservation(): return self, None, None else: return ( None, "new_statements", "Removed frame preservation for generators." ) def mayRaiseException(self, exception_type): return False def needsFrame(self): return True class StatementRestoreFrameException(NodeBase): kind = "STATEMENT_RESTORE_FRAME_EXCEPTION" def __init__(self, preserver_id, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.preserver_id = preserver_id def getDetails(self): return { "preserver_id" : self.preserver_id } def getPreserverId(self): return self.preserver_id def computeStatement(self, constraint_collection): return self, None, None def mayRaiseException(self, exception_type): return False class StatementPublishException(NodeBase): kind = "STATEMENT_PUBLISH_EXCEPTION" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeStatement(self, constraint_collection): # TODO: Determine the need for it. return self, None, None def mayRaiseException(self, exception_type): return False Nuitka-0.5.18.1/nuitka/nodes/GlobalsLocalsNodes.py0000644000175000017500000001065112651071040022117 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Globals/locals/dir1 nodes These nodes give access to variables, highly problematic, because using them, the code may change or access anything about them, so nothing can be trusted anymore, if we start to not know where their value goes. The "dir()" call without arguments is reformulated to locals or globals calls. """ from nuitka.PythonVersions import python_version from .NodeBases import ( ExpressionBuiltinSingleArgBase, ExpressionMixin, NodeBase, StatementChildrenHavingBase ) class ExpressionBuiltinGlobals(NodeBase, ExpressionMixin): kind = "EXPRESSION_BUILTIN_GLOBALS" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeExpression(self, constraint_collection): return self, None, None def mayHaveSideEffects(self): return False def mayRaiseException(self, exception_type): return False def mayBeNone(self): return None class ExpressionBuiltinLocals(NodeBase, ExpressionMixin): kind = "EXPRESSION_BUILTIN_LOCALS" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeExpression(self, constraint_collection): # Just inform the collection that all escaped. constraint_collection.onLocalsUsage() return self, None, None def needsLocalsDict(self): return self.getParentVariableProvider().isEarlyClosure() and \ not self.getParent().isStatementReturn() def mayHaveSideEffects(self): if python_version < 300: return False provider = self.getParentVariableProvider() if provider.isCompiledPythonModule(): return False # TODO: That's not true. return self.getParentVariableProvider().isExpressionClassBody() def mayRaiseException(self, exception_type): return self.mayHaveSideEffects() def mayBeNone(self): return None class StatementSetLocals(StatementChildrenHavingBase): kind = "STATEMENT_SET_LOCALS" named_children = ( "new_locals", ) def __init__(self, new_locals, source_ref): StatementChildrenHavingBase.__init__( self, values = { "new_locals" : new_locals, }, source_ref = source_ref ) def needsLocalsDict(self): return True def mayRaiseException(self, exception_type): return self.getNewLocals().mayRaiseException(exception_type) getNewLocals = StatementChildrenHavingBase.childGetter("new_locals") def computeStatement(self, constraint_collection): # Make sure that we don't even assume "unset" of things not set yet for # anything. constraint_collection.removeAllKnowledge() constraint_collection.onExpression(self.getNewLocals()) new_locals = self.getNewLocals() if new_locals.willRaiseException(BaseException): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = new_locals, node = self ) return result, "new_raise", """\ Setting locals already raises implicitly building new locals.""" return self, None, None class ExpressionBuiltinDir1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_DIR1" def computeExpression(self, constraint_collection): # TODO: Quite some cases should be possible to predict. return self, None, None def mayBeNone(self): return None Nuitka-0.5.18.1/nuitka/nodes/VariableRefNodes.py0000644000175000017500000002246012651072046021570 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for variable references. These represent all variable references in the node tree. Can be in assignments and its expressions, changing the meaning of course dramatically. """ from nuitka import Builtins, VariableRegistry, Variables from .ConstantRefNodes import ExpressionConstantRef from .NodeBases import ExpressionMixin, NodeBase def _isReadOnlyUnterdeterminedModuleVariable(variable): return variable.isModuleVariable() and \ variable.getReadOnlyIndicator() is None def _isReadOnlyModuleVariable(variable): return ( variable.isModuleVariable() and \ variable.getReadOnlyIndicator() is True ) or variable.isMaybeLocalVariable() class ExpressionVariableRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_VARIABLE_REF" def __init__(self, variable_name, source_ref, variable = None): NodeBase.__init__( self, source_ref = source_ref ) self.variable_name = variable_name self.variable = variable if variable is not None: assert variable.getName() == variable_name self.variable_trace = None self.global_trace = None def getDetails(self): if self.variable is None: return { "variable_name" : self.variable_name } else: return { "variable_name" : self.variable_name, "variable" : self.variable } def getDetail(self): if self.variable is None: return self.variable_name else: return repr(self.variable) @staticmethod def isTargetVariableRef(): return False def getVariableName(self): return self.variable_name def getVariable(self): return self.variable def setVariable(self, variable): assert isinstance(variable, Variables.Variable), repr(variable) self.variable = variable def computeExpression(self, constraint_collection): variable = self.variable assert variable is not None self.variable_trace = constraint_collection.getVariableCurrentTrace( variable = variable ) replacement = self.variable_trace.getReplacementNode(self) if replacement is not None: constraint_collection.signalChange( "new_expression", self.source_ref, "Value propagated for '%s' from '%s'." % ( self.variable.getName(), replacement.getSourceReference().getAsString() ) ) # Need to compute the replacement still. return replacement.computeExpression(constraint_collection) if not self.variable_trace.mustHaveValue(): # TODO: This could be way more specific surely. constraint_collection.onExceptionRaiseExit( BaseException ) self.global_trace = VariableRegistry.getGlobalVariableTrace(variable) # TODO: Maybe local variables are factored into this strangely. if self.global_trace is None and variable.isModuleVariable(): constraint_collection.assumeUnclearLocals() elif (variable.isModuleVariable() and not self.global_trace.hasDefiniteWrites() ) or \ variable.isMaybeLocalVariable(): if self.variable_name in Builtins.builtin_exception_names: from .BuiltinRefNodes import ExpressionBuiltinExceptionRef new_node = ExpressionBuiltinExceptionRef( exception_name = self.variable_name, source_ref = self.getSourceReference() ) change_tags = "new_builtin_ref" change_desc = """\ Module variable '%s' found to be built-in exception reference.""" % ( self.variable_name ) elif self.variable_name in Builtins.builtin_names and \ self.variable_name != "pow": from .BuiltinRefNodes import ExpressionBuiltinRef new_node = ExpressionBuiltinRef( builtin_name = self.variable_name, source_ref = self.getSourceReference() ) change_tags = "new_builtin_ref" change_desc = """\ Module variable '%s' found to be built-in reference.""" % ( self.variable_name ) elif self.variable_name == "__name__": new_node = ExpressionConstantRef( constant = variable.getOwner().getParentModule().\ getFullName(), source_ref = self.getSourceReference() ) change_tags = "new_constant" change_desc = """\ Replaced read-only module attribute '__name__' with constant value.""" elif self.variable_name == "__package__": new_node = ExpressionConstantRef( constant = variable.getOwner().getPackage(), source_ref = self.getSourceReference() ) change_tags = "new_constant" change_desc = """\ Replaced read-only module attribute '__package__' with constant value.""" else: self.variable_trace.addUsage() # Probably should give a warning once about it. new_node = self change_tags = None change_desc = None return new_node, change_tags, change_desc self.variable_trace.addUsage() return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) constraint_collection.onControlFlowEscape(self) if self.global_trace is None and \ self.variable_name in ("dir", "eval", "exec", "execfile", "locals", "vars") and \ self.variable.isModuleVariable(): # Just inform the collection that all escaped. constraint_collection.onLocalsUsage() return call_node, None, None def onContentEscapes(self, constraint_collection): constraint_collection.onVariableContentEscapes(self.variable) def isKnownToBeIterable(self, count): return None def mayHaveSideEffects(self): return not self.variable_trace.mustHaveValue() def mayRaiseException(self, exception_type): variable_trace = self.variable_trace return variable_trace is None or not self.variable_trace.mustHaveValue() class ExpressionTempVariableRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_TEMP_VARIABLE_REF" def __init__(self, variable, source_ref): assert variable.isTempVariable() NodeBase.__init__(self, source_ref = source_ref) self.variable = variable self.variable_trace = None def getDetailsForDisplay(self): return { "name" : self.variable.getName() } def getDetails(self): return { "variable" : self.variable } def getDetail(self): return self.variable.getName() def getVariableName(self): return self.variable.getName() def getVariable(self): return self.variable @staticmethod def isTargetVariableRef(): return False def computeExpression(self, constraint_collection): self.variable_trace = constraint_collection.getVariableCurrentTrace( variable = self.variable ) replacement = self.variable_trace.getReplacementNode(self) if replacement is not None: return replacement, "new_expression", "Value propagated for temp '%s'." % self.variable.getName() self.variable_trace.addUsage() # Nothing to do here. return self, None, None def onContentEscapes(self, constraint_collection): constraint_collection.onVariableContentEscapes(self.variable) def mayHaveSideEffects(self): # Can't happen return False def mayRaiseException(self, exception_type): # Can't happen return False def isKnownToBeIterableAtMin(self, count): # TODO: See through the variable current trace. return None def isKnownToBeIterableAtMax(self, count): # TODO: See through the variable current trace. return None # Python3 only, it updates temporary variables that are closure variables. def setVariable(self, variable): self.variable = variable Nuitka-0.5.18.1/nuitka/nodes/ConstantRefNodes.py0000644000175000017500000002170412651072046021634 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for constant expressions. Can be all common built-in types. """ from logging import warning from nuitka.__past__ import iterItems, unicode # pylint: disable=W0622 from nuitka.Constants import ( getConstantIterationLength, isConstant, isHashable, isIndexConstant, isIterableConstant, isMutable, isNumberConstant ) from nuitka.Options import isDebug from .NodeBases import CompileTimeConstantExpressionMixin, NodeBase from .NodeMakingHelpers import ( makeRaiseExceptionReplacementExpression, wrapExpressionWithSideEffects ) class ExpressionConstantRef(CompileTimeConstantExpressionMixin, NodeBase): kind = "EXPRESSION_CONSTANT_REF" user_provided = False def __init__(self, constant, source_ref, user_provided = False): NodeBase.__init__(self, source_ref = source_ref) CompileTimeConstantExpressionMixin.__init__(self) assert isConstant(constant), repr(constant) self.constant = constant # Memory saving method, have the attribute only where necessary. if user_provided: self.user_provided = user_provided if not user_provided and isDebug(): try: size = len(constant) if type(constant) in (str, unicode): max_size = 1000 else: max_size = 256 if size > max_size: warning( "Too large constant (%s %d) encountered at %s.", type(constant), size, source_ref.getAsString() ) except TypeError: pass def __repr__(self): return "" % ( self.kind, self.constant, self.source_ref.getAsString(), self.user_provided ) def getDetails(self): return { "constant" : self.constant, "user_provided" : self.user_provided } def getDetailsForDisplay(self): return { "constant" : repr(self.constant), "user_provided" : self.user_provided } def getDetail(self): return repr(self.constant) def computeExpression(self, constraint_collection): # Cannot compute any further, this is already the best. return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, constraint_collection): # The arguments don't matter. All constant values cannot be called, and # we just need to make and error out of that. new_node = wrapExpressionWithSideEffects( new_node = makeRaiseExceptionReplacementExpression( expression = self, exception_type = "TypeError", exception_value = "'%s' object is not callable" % type(self.constant).__name__ ), old_node = call_node, side_effects = call_node.extractPreCallSideEffects() ) constraint_collection.onExceptionRaiseExit(TypeError) return new_node, "new_raise", "Predicted call of constant value to exception raise." def getCompileTimeConstant(self): return self.constant getConstant = getCompileTimeConstant def isMutable(self): return isMutable(self.constant) def isKnownToBeHashable(self): return isHashable(self.constant) def isNumberConstant(self): return isNumberConstant(self.constant) def isIndexConstant(self): return isIndexConstant(self.constant) def isIndexable(self): return self.constant is None or self.isNumberConstant() def isKnownToBeIterable(self, count): if isIterableConstant(self.constant): return count is None or \ getConstantIterationLength(self.constant) == count else: return False def isKnownToBeIterableAtMin(self, count): length = self.getIterationLength() return length is not None and length >= count def canPredictIterationValues(self): return self.isKnownToBeIterable(None) def getIterationValue(self, count): assert count < len(self.constant) return ExpressionConstantRef( constant = self.constant[count], source_ref = self.source_ref ) def getIterationValues(self): source_ref = self.getSourceReference() return tuple( ExpressionConstantRef( constant = value, source_ref = source_ref, user_provided = self.user_provided ) for value in self.constant ) def isMapping(self): return type(self.constant) is dict def isMappingWithConstantStringKeys(self): assert self.isMapping() for key in self.constant: if type(key) not in (str, unicode): return False return True def getMappingPairs(self): assert self.isMapping() pairs = [] source_ref = self.getSourceReference() for key, value in iterItems(self.constant): pairs.append( ExpressionConstantRef( constant = key, source_ref = source_ref ), ExpressionConstantRef( constant = value, source_ref = source_ref ) ) return pairs def getMappingStringKeyPairs(self): assert self.isMapping() pairs = [] source_ref = self.getSourceReference() for key, value in iterItems(self.constant): pairs.append( ( key, ExpressionConstantRef( constant = value, source_ref = source_ref ) ) ) return pairs def isBoolConstant(self): return type(self.constant) is bool def mayHaveSideEffects(self): # Constants have no side effects return False def extractSideEffects(self): # Constants have no side effects return () def getIntegerValue(self): if self.isNumberConstant(): return int(self.constant) else: return None def getStringValue(self): if self.isStringConstant(): return self.constant else: return None def getIterationLength(self): if isIterableConstant(self.constant): return getConstantIterationLength(self.constant) else: return None def isIterableConstant(self): return isIterableConstant(self.constant) def isUnicodeConstant(self): return type(self.constant) is unicode def isStringConstant(self): return type(self.constant) is str def getStrValue(self): if type(self.constant) is str: # Nothing to do. return self else: try: return ExpressionConstantRef( constant = str(self.constant), user_provided = self.user_provided, source_ref = self.getSourceReference(), ) except UnicodeEncodeError: # Unicode constants may not be possible to encode. return None def computeExpressionIter1(self, iter_node, constraint_collection): if type(self.constant) in (list, set, frozenset, dict): result = ExpressionConstantRef( constant = tuple(self.constant), user_provided = self.user_provided, source_ref = self.getSourceReference() ) self.replaceWith(result) return ( iter_node, "new_constant", """\ Iteration over constant %s changed to tuple.""" % type(self.constant).__name__ ) if not isIterableConstant(self.constant): # Any exception may be raised. constraint_collection.onExceptionRaiseExit(TypeError) return iter_node, None, None Nuitka-0.5.18.1/nuitka/nodes/GeneratorNodes.py0000644000175000017500000001070712651071040021326 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for generator objects and their creations. Generators are turned into normal functions that create generator objects, whose implementation lives here. The creation itself also lives here. """ from nuitka.PythonVersions import python_version from .Checkers import checkStatementsSequenceOrNone from .FunctionNodes import ExpressionFunctionBodyBase from .IndicatorMixins import ( MarkLocalsDictIndicator, MarkUnoptimizedFunctionIndicator ) from .NodeBases import ChildrenHavingMixin, ExpressionChildrenHavingBase from .ReturnNodes import StatementReturn class ExpressionMakeGeneratorObject(ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_GENERATOR_OBJECT" named_children = ( "generator_ref", ) getGeneratorRef = ExpressionChildrenHavingBase.childGetter("generator_ref") def __init__(self, generator_ref, code_object, source_ref): assert generator_ref.getFunctionBody().isExpressionGeneratorObjectBody() ExpressionChildrenHavingBase.__init__( self, values = { "generator_ref" : generator_ref, }, source_ref = source_ref ) self.code_object = code_object def getDetails(self): return { "code_object" : self.code_object } def getCodeObject(self): return self.code_object def computeExpression(self, constraint_collection): # TODO: Generator body may know something too. return self, None, None def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False class ExpressionGeneratorObjectBody(ExpressionFunctionBodyBase, MarkLocalsDictIndicator, MarkUnoptimizedFunctionIndicator): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, pylint: disable=R0901 kind = "EXPRESSION_GENERATOR_OBJECT_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } if python_version >= 340: qualname_setup = None def __init__(self, provider, name, flags, source_ref): ExpressionFunctionBodyBase.__init__( self, provider = provider, name = name, is_class = False, code_prefix = "genexpr" if name == "" else "genobj", flags = flags, source_ref = source_ref ) MarkLocalsDictIndicator.__init__(self) MarkUnoptimizedFunctionIndicator.__init__(self) self.needs_generator_return_exit = False def getFunctionName(self): return self.name def getFunctionQualname(self): return self.getParentVariableProvider().getFunctionQualname() def markAsNeedsGeneratorReturnHandling(self, value): self.needs_generator_return_exit = max( self.needs_generator_return_exit, value ) def needsGeneratorReturnHandling(self): return self.needs_generator_return_exit == 2 def needsGeneratorReturnExit(self): return bool(self.needs_generator_return_exit) @staticmethod def needsCreation(): return False getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") class StatementGeneratorReturn(StatementReturn): kind = "STATEMENT_GENERATOR_RETURN" def __init__(self, expression, source_ref): StatementReturn.__init__( self, expression = expression, source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/nodes/IndicatorMixins.py0000644000175000017500000000430512651071040021510 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module for node class mixins that indicate runtime determined node facts. These come into play after finalization only. All of the these attributes (and we could use properties instead) are determined once or from a default and then used like this. """ class MarkLocalsDictIndicator: def __init__(self): self.needs_locals_dict = False def hasLocalsDict(self): return self.needs_locals_dict def markAsLocalsDict(self): self.needs_locals_dict = True class MarkUnoptimizedFunctionIndicator: """ Mixin for indication that a function contains an exec or star import. These do not access global variables directly, but check a locals dictionary first, because they do. """ def __init__(self): self.unoptimized_locals = False self.unqualified_exec = False self.exec_source_ref = None def markAsExecContaining(self): self.unoptimized_locals = True def markAsUnqualifiedExecContaining(self, source_ref): self.unqualified_exec = True # Let the first one win. if self.exec_source_ref is None: self.exec_source_ref = source_ref markAsStarImportContaining = markAsExecContaining def isUnoptimized(self): return self.unoptimized_locals def isUnqualifiedExec(self): return self.unoptimized_locals and self.unqualified_exec def getExecSourceRef(self): return self.exec_source_ref Nuitka-0.5.18.1/nuitka/nodes/__init__.py0000644000175000017500000000150112651071040020136 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/nodes/BuiltinRangeNodes.py0000644000175000017500000003006712651071040021764 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'range' builtin. This is a rather complex beast as it has many cases, is difficult to know if it's sizable enough to compute, and there are complex cases, where the bad result of it can be predicted still, and these are interesting for warnings. """ import math from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .NodeBases import ExpressionBuiltinNoArgBase, ExpressionChildrenHavingBase from .NodeMakingHelpers import makeConstantReplacementNode class ExpressionBuiltinRange0(ExpressionBuiltinNoArgBase): kind = "EXPRESSION_BUILTIN_RANGE0" def __init__(self, source_ref): ExpressionBuiltinNoArgBase.__init__( self, builtin_function = range, source_ref = source_ref ) def mayHaveSideEffects(self): return False def mayBeNone(self): return False class ExpressionBuiltinRangeBase(ExpressionChildrenHavingBase): """ Base class for range nodes with 1/2/3 arguments. """ builtin_spec = BuiltinOptimization.builtin_range_spec def __init__(self, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = values, source_ref = source_ref ) def getTruthValue(self): length = self.getIterationLength() if length is None: return None else: return length > 0 def mayHaveSideEffects(self): for child in self.getVisitableNodes(): if child.mayHaveSideEffects(): return True if child.getIntegerValue() is None: return True if python_version >= 270 and \ child.isExpressionConstantRef() and \ type(child.getConstant()) is float: return True return False def mayRaiseException(self, exception_type): for child in self.getVisitableNodes(): if child.mayRaiseException(exception_type): return True # TODO: Should take exception_type value into account here. if child.getIntegerValue() is None: return True if python_version >= 270 and \ child.isExpressionConstantRef() and \ type(child.getConstant()) is float: return True step = self.getStep() # false alarm, pylint: disable=E1128 # A step of 0 will raise. if step is not None and step.getIntegerValue() == 0: return True return False def computeBuiltinSpec(self, constraint_collection, given_values): assert self.builtin_spec is not None, self if not self.builtin_spec.isCompileTimeComputable(given_values): constraint_collection.onExceptionRaiseExit(BaseException) # TODO: Raise exception known step 0. return self, None, None return constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_spec.simulateCall( given_values ), description = "Built-in call to '%s' computed." % ( self.builtin_spec.getName() ) ) def computeExpressionIter1(self, iter_node, constraint_collection): # TODO: Support Python3 range objects too. if python_version >= 300: return iter_node, None, None iteration_length = self.getIterationLength() if iteration_length is not None and iteration_length > 256: result = ExpressionBuiltinXrange( low = self.getLow(), high = self.getHigh(), step = self.getStep(), source_ref = self.getSourceReference() ) self.replaceWith(result) return ( iter_node, "new_expression", "Replaced 'range' with 'xrange' built-in call." ) # No exception will be raised on ranges. return iter_node, None, None @staticmethod def getLow(): return None @staticmethod def getHigh(): return None @staticmethod def getStep(): return None def mayBeNone(self): return False class ExpressionBuiltinRange1(ExpressionBuiltinRangeBase): kind = "EXPRESSION_BUILTIN_RANGE1" named_children = ( "low", ) def __init__(self, low, source_ref): assert low is not None ExpressionBuiltinRangeBase.__init__( self, values = { "low" : low, }, source_ref = source_ref ) getLow = ExpressionChildrenHavingBase.childGetter("low") def computeExpression(self, constraint_collection): # TODO: Support Python3 range objects too. if python_version >= 300: return self, None, None low = self.getLow() return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = ( low, ) ) def getIterationLength(self): low = self.getLow().getIntegerValue() if low is None: return None return max(0, low) def canPredictIterationValues(self): return self.getIterationLength() is not None def getIterationValue(self, element_index): length = self.getIterationLength() if length is None: return None if element_index > length: return None # TODO: Make sure to cast element_index to what CPython will give, for # now a downcast will do. return makeConstantReplacementNode( constant = int(element_index), node = self ) def isKnownToBeIterable(self, count): return count is None or count == self.getIterationLength() class ExpressionBuiltinRange2(ExpressionBuiltinRangeBase): kind = "EXPRESSION_BUILTIN_RANGE2" named_children = ("low", "high") def __init__(self, low, high, source_ref): ExpressionBuiltinRangeBase.__init__( self, values = { "low" : low, "high" : high }, source_ref = source_ref ) getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") builtin_spec = BuiltinOptimization.builtin_range_spec def computeExpression(self, constraint_collection): if python_version >= 300: return self, None, None low = self.getLow() high = self.getHigh() return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = ( low, high ) ) def getIterationLength(self): low = self.getLow() high = self.getHigh() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None return max(0, high - low) def canPredictIterationValues(self): return self.getIterationLength() is not None def getIterationValue(self, element_index): low = self.getLow() high = self.getHigh() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None result = low + element_index if result >= high: return None else: return makeConstantReplacementNode( constant = result, node = self ) def isKnownToBeIterable(self, count): return count is None or count == self.getIterationLength() class ExpressionBuiltinRange3(ExpressionBuiltinRangeBase): kind = "EXPRESSION_BUILTIN_RANGE3" named_children = ( "low", "high", "step" ) def __init__(self, low, high, step, source_ref): ExpressionBuiltinRangeBase.__init__( self, values = { "low" : low, "high" : high, "step" : step }, source_ref = source_ref ) getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") getStep = ExpressionChildrenHavingBase.childGetter("step") builtin_spec = BuiltinOptimization.builtin_range_spec def computeExpression(self, constraint_collection): if python_version >= 300: return self, None, None low = self.getLow() high = self.getHigh() step = self.getStep() return self.computeBuiltinSpec( constraint_collection = constraint_collection, given_values = ( low, high, step ) ) def getIterationLength(self): low = self.getLow() high = self.getHigh() step = self.getStep() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None step = step.getIntegerValue() if step is None: return None # Give up on this, will raise ValueError. if step == 0: return None if low < high: if step < 0: estimate = 0 else: estimate = math.ceil(float(high - low) / step) else: if step > 0: estimate = 0 else: estimate = math.ceil(float(high - low) / step) estimate = round(estimate) assert estimate >= 0 return int(estimate) def canPredictIterationValues(self): return self.getIterationLength() is not None def getIterationValue(self, element_index): low = self.getLow().getIntegerValue() if low is None: return None high = self.getHigh().getIntegerValue() if high is None: return None step = self.getStep().getIntegerValue() result = low + step * element_index if result >= high: return None else: return makeConstantReplacementNode( constant = result, node = self ) def isKnownToBeIterable(self, count): return count is None or count == self.getIterationLength() class ExpressionBuiltinXrange(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_XRANGE" named_children = ("low", "high", "step") def __init__(self, low, high, step, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "low" : low, "high" : high, "step" : step }, source_ref = source_ref ) def computeExpression(self, constraint_collection): return self, None, None getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") getStep = ExpressionChildrenHavingBase.childGetter("step") Nuitka-0.5.18.1/nuitka/nodes/ContainerMakingNodes.py0000644000175000017500000001530412651071040022447 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes that build containers. """ from .NodeBases import ( ExpressionChildrenHavingBase, SideEffectsFromChildrenMixin ) from .NodeMakingHelpers import ( getComputationResult, makeStatementOnlyNodesFromExpressions, wrapExpressionWithSideEffects ) class ExpressionMakeSequenceBase(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): named_children = ( "elements", ) def __init__(self, sequence_kind, elements, source_ref): assert sequence_kind in ("TUPLE", "LIST", "SET"), sequence_kind for element in elements: assert element.isExpression(), element self.sequence_kind = sequence_kind.lower() ExpressionChildrenHavingBase.__init__( self, values = { "elements" : tuple(elements), }, source_ref = source_ref ) def isExpressionMakeSequence(self): return True def getSequenceKind(self): return self.sequence_kind getElements = ExpressionChildrenHavingBase.childGetter("elements") def getSimulator(self): # Abstract method, pylint: disable=R0201 return None def computeExpression(self, constraint_collection): elements = self.getElements() for count, element in enumerate(elements): if element.willRaiseException(BaseException): result = wrapExpressionWithSideEffects( side_effects = elements[:count], new_node = element, old_node = self ) return result, "new_raise", "Sequence creation raises exception" # TODO: CompileTimeConstant should be good enough. for element in elements: if not element.isExpressionConstantRef(): return self, None, None simulator = self.getSimulator() assert simulator is not None # The simulator is in fact callable if not None, pylint: disable=E1102 return getComputationResult( node = self, computation = lambda : simulator( element.getConstant() for element in self.getElements() ), description = "%s with constant arguments." % simulator.__name__.title() ) def mayHaveSideEffectsBool(self): return False def isKnownToBeIterable(self, count): return count is None or count == len(self.getElements()) def getIterationValue(self, count): return self.getElements()[count] @staticmethod def canPredictIterationValues(): return True def getIterationValues(self): return self.getElements() def getTruthValue(self): return self.getIterationLength() > 0 def mayRaiseException(self, exception_type): for element in self.getElements(): if element.mayRaiseException(exception_type): return True return False def mayBeNone(self): return False def computeExpressionDrop(self, statement, constraint_collection): result = makeStatementOnlyNodesFromExpressions( expressions = self.getElements() ) return result, "new_statements", """\ Removed sequence creation for unused sequence.""" class ExpressionMakeTuple(ExpressionMakeSequenceBase): kind = "EXPRESSION_MAKE_TUPLE" def __init__(self, elements, source_ref): ExpressionMakeSequenceBase.__init__( self, sequence_kind = "TUPLE", elements = elements, source_ref = source_ref ) def getSimulator(self): return tuple def getIterationLength(self): return len(self.getElements()) class ExpressionMakeList(ExpressionMakeSequenceBase): kind = "EXPRESSION_MAKE_LIST" def __init__(self, elements, source_ref): ExpressionMakeSequenceBase.__init__( self, sequence_kind = "LIST", elements = elements, source_ref = source_ref ) def getSimulator(self): return list def getIterationLength(self): return len(self.getElements()) def computeExpressionIter1(self, iter_node, constraint_collection): result = ExpressionMakeTuple( elements = self.getElements(), source_ref = self.source_ref ) self.replaceWith(result) return iter_node, "new_expression", """\ Iteration over list reduced to tuple.""" class ExpressionMakeSet(ExpressionMakeSequenceBase): kind = "EXPRESSION_MAKE_SET" def __init__(self, elements, source_ref): ExpressionMakeSequenceBase.__init__( self, sequence_kind = "SET", elements = elements, source_ref = source_ref ) def getSimulator(self): return set def getIterationLength(self): element_count = len(self.getElements()) # Hashing may consume elements. if element_count >= 2: return None else: return element_count def getIterationMinLength(self): element_count = len(self.getElements()) if element_count == 0: return 0 else: return 1 def getIterationMaxLength(self): return len(self.getElements()) def mayRaiseException(self, exception_type): for element in self.getElements(): if not element.isKnownToBeHashable(): return True if element.mayRaiseException(exception_type): return True return False def computeExpressionIter1(self, iter_node, constraint_collection): result = ExpressionMakeTuple( elements = self.getElements(), source_ref = self.source_ref ) self.replaceWith(result) return iter_node, "new_expression", """\ Iteration over set reduced to tuple.""" Nuitka-0.5.18.1/nuitka/nodes/BuiltinHashNodes.py0000644000175000017500000000366112651072046021622 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'hash' builtin. This is a specific thing, which must be calculated at run time, but we can predict things about its type, and the fact that it won't raise an exception for some types, so it is still useful. Also calls to it can be accelerated slightly. """ from .NodeBases import ExpressionChildrenHavingBase class ExpressionBuiltinHash(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_HASH" named_children = ( "value", ) def __init__(self, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "value" : value, }, source_ref = source_ref ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, constraint_collection): value = self.getValue() # TODO: Have a computation slot for hashing. if not value.isKnownToBeHashable(): constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): return not self.getValue().isKnownToBeHashable() Nuitka-0.5.18.1/nuitka/nodes/LoopNodes.py0000644000175000017500000002163012651071040020306 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Loop nodes. There are for and loop nodes, but both are reduced to loops with break/continue statements for it. These re-formulations require that optimization of loops has to be very general, yet the node type for loop, becomes very simple. """ from nuitka.optimizations.TraceCollections import ConstraintCollectionBranch from nuitka.tree.Extractions import getVariablesWritten from .Checkers import checkStatementsSequenceOrNone from .NodeBases import NodeBase, StatementChildrenHavingBase class StatementLoop(StatementChildrenHavingBase): kind = "STATEMENT_LOOP" named_children = ( "body", ) checkers = { "body" : checkStatementsSequenceOrNone } def __init__(self, body, source_ref): StatementChildrenHavingBase.__init__( self, values = { "body" : body }, source_ref = source_ref ) self.loop_variables = None getLoopBody = StatementChildrenHavingBase.childGetter("body") setLoopBody = StatementChildrenHavingBase.childSetter("body") def mayReturn(self): loop_body = self.getLoopBody() if loop_body is not None and loop_body.mayReturn(): return True return False def mayBreak(self): # The loop itself may never break another loop. return False def mayContinue(self): # The loop itself may never continue another loop. return False def isStatementAborting(self): loop_body = self.getLoopBody() if loop_body is None: return True else: return not loop_body.mayBreak() def mayRaiseException(self, exception_type): # Loops can only raise, if their body does, but they also issue the # async exceptions, so we must make them do it all the time. return True # loop_body = self.getLoopBody() # return loop_body is not None and \ # self.getLoopBody().mayRaiseException(exception_type) def computeLoopBody(self, constraint_collection): abort_context = constraint_collection.makeAbortStackContext( catch_breaks = True, catch_continues = True, catch_returns = False, catch_exceptions = False, ) with abort_context: loop_body = self.getLoopBody() if loop_body is not None: # Look ahead. what will be written and degrade about that if we # are in the first iteration, later we will have more precise # knowledge. if self.loop_variables is None: self.loop_variables = getVariablesWritten( loop_body ) loop_entry_traces = set() # Mark all variables as loop wrap around that are written in # the loop and hit a 'continue'. for variable in self.loop_variables: loop_entry_traces.add( constraint_collection.markActiveVariableAsLoopMerge( variable = variable ) ) result = loop_body.computeStatementsSequence( constraint_collection = constraint_collection ) # Might be changed. if result is not loop_body: self.setLoopBody(result) loop_body = result if loop_body is not None: # Emulate terminal continue if not aborting. if not loop_body.isStatementAborting(): constraint_collection.onLoopContinue() continue_collections = constraint_collection.getLoopContinueCollections() self.loop_variables = set() for loop_entry_trace in loop_entry_traces: variable = loop_entry_trace.getVariable() loop_end_traces = set() for continue_collection in continue_collections: loop_end_trace = continue_collection.getVariableCurrentTrace(variable) if loop_end_trace is not loop_entry_trace: loop_end_traces.add(loop_end_trace) if loop_end_traces: loop_entry_trace.addLoopContinueTraces(loop_end_traces) self.loop_variables.add(variable) # If we break, the outer collections becomes a merge of all those breaks # or just the one, if there is only one. break_collections = constraint_collection.getLoopBreakCollections() return loop_body, break_collections def computeStatement(self, constraint_collection): outer_constraint_collection = constraint_collection constraint_collection = ConstraintCollectionBranch( parent = constraint_collection, name = "loop" ) loop_body, break_collections = self.computeLoopBody(constraint_collection) # Consider trailing "continue" statements, these have no effect, so we # can remove them. if loop_body is not None: assert loop_body.isStatementsSequence() statements = loop_body.getStatements() assert statements # Cannot be empty # If the last statement is a "continue" statement, it can simply # be discarded. last_statement = statements[-1] if last_statement.isStatementLoopContinue(): if len(statements) == 1: self.setLoopBody(None) loop_body = None else: last_statement.replaceWith(None) constraint_collection.signalChange( "new_statements", last_statement.getSourceReference(), """\ Removed useless terminal 'continue' as last statement of loop.""" ) if break_collections: outer_constraint_collection.mergeMultipleBranches(break_collections) # Consider leading "break" statements, they should be the only, and # should lead to removing the whole loop statement. Trailing "break" # statements could also be handled, but that would need to consider if # there are other "break" statements too. Numbering loop exits is # nothing we have yet. if loop_body is not None: assert loop_body.isStatementsSequence() statements = loop_body.getStatements() assert statements # Cannot be empty if len(statements) == 1 and statements[-1].isStatementLoopBreak(): return None, "new_statements", """\ Removed useless loop with immediate 'break' statement.""" # Also consider the threading intermission. TODO: We ought to make it # explicit, so we can see it potentially disrupting and changing the # global variables. It may also raise. outer_constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None class StatementLoopContinue(NodeBase): kind = "STATEMENT_LOOP_CONTINUE" def __init__(self, source_ref): NodeBase.__init__(self, source_ref = source_ref) def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return False def mayContinue(self): return True def computeStatement(self, constraint_collection): # This statement being aborting, will already tell everything. constraint_collection.onLoopContinue() return self, None, None class StatementLoopBreak(NodeBase): kind = "STATEMENT_LOOP_BREAK" def __init__(self, source_ref): NodeBase.__init__(self, source_ref = source_ref) def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return False def mayBreak(self): return True def computeStatement(self, constraint_collection): # This statement being aborting, will already tell everything. constraint_collection.onLoopBreak() return self, None, None Nuitka-0.5.18.1/nuitka/nodes/CoroutineNodes.py0000644000175000017500000001170712651071040021350 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for coroutine objects and their creations. Coroutines are turned into normal functions that create generator objects, whose implementation lives here. The creation itself also lives here. """ from nuitka.PythonVersions import python_version from .Checkers import checkStatementsSequenceOrNone from .FunctionNodes import ExpressionFunctionBodyBase from .IndicatorMixins import ( MarkLocalsDictIndicator, MarkUnoptimizedFunctionIndicator ) from .NodeBases import ChildrenHavingMixin, ExpressionChildrenHavingBase class ExpressionMakeCoroutineObject(ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_COROUTINE_OBJECT" named_children = ( "coroutine_ref", ) getCoroutineRef = ExpressionChildrenHavingBase.childGetter("coroutine_ref") def __init__(self, coroutine_ref, code_object, source_ref): assert coroutine_ref.getFunctionBody().isExpressionCoroutineObjectBody() ExpressionChildrenHavingBase.__init__( self, values = { "coroutine_ref" : coroutine_ref, }, source_ref = source_ref ) self.code_object = code_object def getDetails(self): return { "code_object" : self.code_object } def getCodeObject(self): return self.code_object def getDetailsForDisplay(self): return { "coroutine" : self.getCoroutineRef().getFunctionBody().getCodeName() } def computeExpression(self, constraint_collection): # TODO: Coroutine body may know something too. return self, None, None def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False class ExpressionCoroutineObjectBody(ExpressionFunctionBodyBase, MarkLocalsDictIndicator, MarkUnoptimizedFunctionIndicator): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, pylint: disable=R0901 kind = "EXPRESSION_COROUTINE_OBJECT_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } if python_version >= 340: qualname_setup = None def __init__(self, provider, name, flags, source_ref): while provider.isExpressionOutlineBody(): provider = provider.getParentVariableProvider() ExpressionFunctionBodyBase.__init__( self, provider = provider, name = name, code_prefix = "coroutine", is_class = False, flags = flags, source_ref = source_ref ) MarkLocalsDictIndicator.__init__(self) MarkUnoptimizedFunctionIndicator.__init__(self) self.needs_generator_return_exit = False def getFunctionName(self): return self.name def getFunctionQualname(self): return self.getParentVariableProvider().getFunctionQualname() def markAsNeedsGeneratorReturnHandling(self, value): self.needs_generator_return_exit = max( self.needs_generator_return_exit, value ) def needsGeneratorReturnHandling(self): return self.needs_generator_return_exit == 2 def needsGeneratorReturnExit(self): return bool(self.needs_generator_return_exit) @staticmethod def needsCreation(): return False getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") class ExpressionAsyncWait(ExpressionChildrenHavingBase): kind = "EXPRESSION_ASYNC_WAIT" named_children = ("expression",) def __init__(self, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) def computeExpression(self, constraint_collection): # TODO: Might be predictable based awaitable analysis or for constants. return self, None, None getValue = ChildrenHavingMixin.childGetter("expression") Nuitka-0.5.18.1/nuitka/nodes/OutlineNodes.py0000644000175000017500000001126512651071040021017 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Outline nodes. We use them for re-formulations and for in-lining of code. They are expressions that get their value from return statements in their code body. They do not own anything by themselves. It's just a way of having try/finally for the expressions, or multiple returns, without running in a too different context. """ from .NodeBases import ChildrenHavingMixin, ExpressionChildrenHavingBase class ExpressionOutlineBody(ExpressionChildrenHavingBase): """ Outlined code. This is for a call to a piece of code to be executed in a specific context. It contains an exclusively owned function body, that has no other references, and can be considered part of the calling context. It must return a value, to use as expression value. """ kind = "EXPRESSION_OUTLINE_BODY" named_children = ( "body", ) def __init__(self, provider, name, source_ref, body = None): assert name != "" ExpressionChildrenHavingBase.__init__( self, values = { "body" : body }, source_ref = source_ref ) self.provider = provider self.name = name self.temp_scope = None # Hack: This allows some APIs to work although this is not yet # officially a child yet. Important during building. self.parent = provider def getDetails(self): return { "provider" : self.provider, "name" : self.name } getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") def getOutlineTempScope(self): # We use our own name as a temp_scope, cached from the parent, if the # scope is None. if self.temp_scope is None: self.temp_scope = self.provider.allocateTempScope(self.name) return self.temp_scope def allocateTempVariable(self, temp_scope, name): if temp_scope is None: temp_scope = self.getOutlineTempScope() return self.provider.allocateTempVariable( temp_scope = temp_scope, name = name ) def allocateTempScope(self, name): # Let's scope the temporary scopes by the outline they come from. return self.provider.allocateTempScope( name = self.name + '$' + name ) def getContainingClassDictCreation(self): return self.getParentVariableProvider().getContainingClassDictCreation() def computeExpressionRaw(self, constraint_collection): owning_module = self.getParentModule() # Make sure the owning module is added to the used set. This is most # important for helper functions, or modules, which otherwise have # become unused. from nuitka.ModuleRegistry import addUsedModule addUsedModule(owning_module) abort_context = constraint_collection.makeAbortStackContext( catch_breaks = False, catch_continues = False, catch_returns = True, catch_exceptions = False ) with abort_context: body = self.getBody() result = body.computeStatementsSequence( constraint_collection = constraint_collection ) if result is not body: self.setBody(result) body = result return_collections = constraint_collection.getFunctionReturnCollections() constraint_collection.mergeMultipleBranches(return_collections) if body.getStatements()[0].isStatementReturn(): return ( body.getStatements()[0].getExpression(), "new_expression", "Outline is now simple expression, use directly." ) # TODO: Function outline may become too trivial to outline and return # collections may tell us something. return self, None, None Nuitka-0.5.18.1/nuitka/nodes/AttributeNodes.py0000644000175000017500000003274212651072046021355 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Attribute nodes Knowing attributes of an object is very important, esp. when it comes to 'self' and objects and classes. There will be a methods "computeExpression*Attribute" to aid predicting them, with many variants for setting, deleting, and accessing. Also there is some complication in the form of special lookups, that won't go through the normal path, but just check slots. Due to ``getattr`` and ``setattr`` built-ins, there is also a different in the computations for objects and for compile time known strings. This reflects what CPython also does with "tp_getattr" and "tp_getattro". These nodes are therefore mostly delegating the work to expressions they work on, and let them decide and do the heavy lifting of optimization and annotation is happening in the nodes that implement these compute slots. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from .NodeBases import ExpressionChildrenHavingBase, StatementChildrenHavingBase from .NodeMakingHelpers import wrapExpressionWithNodeSideEffects class StatementAssignmentAttribute(StatementChildrenHavingBase): """ Assignment to an attribute. Typically from code like: source.attribute_name = expression Both source and expression may be complex expressions, the source is evaluated first. Assigning to an attribute has its on slot on the source, which gets to decide if it knows it will work or not, and what value it will be. """ kind = "STATEMENT_ASSIGNMENT_ATTRIBUTE" named_children = ( "source", "expression" ) def __init__(self, expression, attribute_name, source, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression, "source" : source, }, source_ref = source_ref ) self.attribute_name = attribute_name def getDetails(self): return { "attribute_name" : self.attribute_name } def getDetail(self): return "to attribute %s" % self.attribute_name def getAttributeName(self): return self.attribute_name def setAttributeName(self, attribute_name): self.attribute_name = attribute_name getLookupSource = StatementChildrenHavingBase.childGetter("expression") getAssignSource = StatementChildrenHavingBase.childGetter("source") def computeStatement(self, constraint_collection): lookup_source = self.getLookupSource() source = self.getAssignSource() result, change_tags, change_desc = self.computeStatementSubExpressions( constraint_collection = constraint_collection, expressions = (lookup_source, source), ) if result is not self: return result, change_tags, change_desc return lookup_source.computeExpressionSetAttribute( set_node = self, attribute_name = self.attribute_name, value_node = source, constraint_collection = constraint_collection ) class StatementDelAttribute(StatementChildrenHavingBase): """ Deletion of an attribute. Typically from code like: del source.attribute_name The source may be complex expression. Deleting an attribute has its on slot on the source, which gets to decide if it knows it will work or not, and what value it will be. """ kind = "STATEMENT_DEL_ATTRIBUTE" named_children = ( "expression", ) def __init__(self, expression, attribute_name, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.attribute_name = attribute_name def getDetails(self): return { "attribute_name" : self.attribute_name } def getDetail(self): return "to attribute %s" % self.attribute_name def getAttributeName(self): return self.attribute_name def setAttributeName(self, attribute_name): self.attribute_name = attribute_name getLookupSource = StatementChildrenHavingBase.childGetter("expression") def computeStatement(self, constraint_collection): lookup_source = self.getLookupSource() result, change_tags, change_desc = self.computeStatementSubExpressions( constraint_collection = constraint_collection, expressions = (lookup_source,) ) if result is not self: return result, change_tags, change_desc return lookup_source.computeExpressionDelAttribute( set_node = self, attribute_name = self.attribute_name, constraint_collection = constraint_collection ) class ExpressionAttributeLookup(ExpressionChildrenHavingBase): """ Looking up an attribute of an object. Typically code like: source.attribute_name """ kind = "EXPRESSION_ATTRIBUTE_LOOKUP" named_children = ( "source", ) def __init__(self, source, attribute_name, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source }, source_ref = source_ref ) self.attribute_name = attribute_name def getAttributeName(self): return self.attribute_name def setAttributeName(self, attribute_name): self.attribute_name = attribute_name def getDetails(self): return { "attribute_name" : self.getAttributeName() } def getDetail(self): return "attribute %s from %s" % ( self.getAttributeName(), self.getLookupSource() ) getLookupSource = ExpressionChildrenHavingBase.childGetter( "source" ) def computeExpression(self, constraint_collection): return self.getLookupSource().computeExpressionAttribute( lookup_node = self, attribute_name = self.getAttributeName(), constraint_collection = constraint_collection ) def mayRaiseException(self, exception_type): return self.getLookupSource().mayRaiseExceptionAttributeLookup( exception_type = exception_type, attribute_name = self.getAttributeName() ) def isKnownToBeIterable(self, count): # TODO: Could be known. We would need for computeExpressionAttribute to # either return a new node, or a decision maker. return None class ExpressionAttributeLookupSpecial(ExpressionAttributeLookup): """ Special lookup up an attribute of an object. Typically from code like this: with source: pass These directly go to slots, and are performed for with statements of Python2.7 or higher. """ kind = "EXPRESSION_ATTRIBUTE_LOOKUP_SPECIAL" def computeExpression(self, constraint_collection): return self.getLookupSource().computeExpressionAttributeSpecial( lookup_node = self, attribute_name = self.getAttributeName(), constraint_collection = constraint_collection ) class ExpressionBuiltinGetattr(ExpressionChildrenHavingBase): """ Built-in "getattr". Typical code like this: getattr(source, attribute, default) The default is optional, but computed before the lookup is done. """ kind = "EXPRESSION_BUILTIN_GETATTR" named_children = ("source", "attribute", "default") @calledWithBuiltinArgumentNamesDecorator def __init__(self, object_arg, name, default, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : object_arg, "attribute" : name, "default" : default }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("source") getAttribute = ExpressionChildrenHavingBase.childGetter("attribute") getDefault = ExpressionChildrenHavingBase.childGetter("default") def computeExpression(self, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) default = self.getDefault() if default is None or not default.mayHaveSideEffects(): attribute = self.getAttribute() attribute_name = attribute.getStringValue() if attribute_name is not None: source = self.getLookupSource() if source.isKnownToHaveAttribute(attribute_name): # If source has side effects, they must be evaluated, before # the lookup, meaning, a temporary variable should be assigned. # For now, we give up in this case. side_effects = source.extractSideEffects() if not side_effects: result = ExpressionAttributeLookup( source = source, attribute_name = attribute_name, source_ref = self.source_ref ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = attribute ) return ( result, "new_expression", """Replaced call to built-in 'getattr' with constant \ attribute '%s' to mere attribute lookup""" % attribute_name ) return self, None, None class ExpressionBuiltinSetattr(ExpressionChildrenHavingBase): """ Built-in "setattr". Typical code like this: setattr(source, attribute, value) """ kind = "EXPRESSION_BUILTIN_SETATTR" named_children = ("source", "attribute", "value") @calledWithBuiltinArgumentNamesDecorator def __init__(self, object_arg, name, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : object_arg, "attribute" : name, "value" : value }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("source") getAttribute = ExpressionChildrenHavingBase.childGetter("attribute") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, constraint_collection): constraint_collection.onExceptionRaiseExit(BaseException) # Note: Might be possible to predict or downgrade to mere attribute set. return self, None, None class ExpressionBuiltinHasattr(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_HASATTR" named_children = ("source", "attribute") @calledWithBuiltinArgumentNamesDecorator def __init__(self, object_arg, name, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : object_arg, "attribute" : name, }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("source") getAttribute = ExpressionChildrenHavingBase.childGetter("attribute") def computeExpression(self, constraint_collection): # We do at least for compile time constants optimization here, but more # could be done, were we to know shapes. source = self.getLookupSource() if source.isCompileTimeConstant(): attribute = self.getAttribute() attribute_name = attribute.getStringValue() if attribute_name is not None: # If source or attribute have side effects, they must be # evaluated, before the lookup. result, tags, change_desc = constraint_collection.getCompileTimeComputationResult( node = self, computation = lambda : hasattr( source.getCompileTimeConstant(), attribute_name ), description = "Call to 'hasattr' pre-computed." ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = attribute ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = source ) return result, tags, change_desc constraint_collection.onExceptionRaiseExit(BaseException) return self, None, None Nuitka-0.5.18.1/nuitka/nodes/ModuleNodes.py0000644000175000017500000004647312651072046020645 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module/Package nodes The top of the tree. Packages are also modules. Modules are what hold a program together and cross-module optimizations are the most difficult to tackle. """ import re from nuitka import Options, Variables from nuitka.containers.oset import OrderedSet from nuitka.importing.Importing import ( findModule, getModuleNameAndKindFromFilename ) from nuitka.importing.Recursion import decideRecursion, recurseTo from nuitka.optimizations.TraceCollections import ConstraintCollectionModule from nuitka.PythonVersions import python_version from nuitka.SourceCodeReferences import SourceCodeReference, fromFilename from nuitka.utils import Utils from .Checkers import checkStatementsSequenceOrNone from .CodeObjectSpecs import CodeObjectSpec from .ConstantRefNodes import ExpressionConstantRef from .FutureSpecs import FutureSpec from .NodeBases import ( ChildrenHavingMixin, ClosureGiverNodeBase, ExpressionMixin, NodeBase ) class PythonModuleMixin: def __init__(self, name, package_name): assert type(name) is str, type(name) assert '.' not in name, name assert package_name is None or \ (type(package_name) is str and package_name != "") self.name = name self.package_name = package_name self.package = None def getName(self): return self.name def getPackage(self): return self.package_name def getFullName(self): if self.package_name: return self.package_name + '.' + self.getName() else: return self.getName() @staticmethod def isMainModule(): return False @staticmethod def isInternalModule(): return False def attemptRecursion(self): # Make sure the package is recursed to. # Return the list of newly added modules. result = [] if self.package_name is not None and self.package is None: package_package, package_filename, _finding = findModule( importing = self, module_name = self.package_name, parent_package = None, level = 1, warn = python_version < 330 ) # TODO: Temporary, if we can't find the package for Python3.3 that # is semi-OK, maybe. if python_version >= 330 and not package_filename: return [] assert package_filename is not None, self.package_name _package_name, package_kind = getModuleNameAndKindFromFilename(package_filename) # assert _package_name == self.package_name, (package_filename, _package_name, self.package_name) decision, _reason = decideRecursion( module_filename = package_filename, module_name = self.package_name, module_package = package_package, module_kind = package_kind ) if decision is not None: imported_module, is_added = recurseTo( module_package = package_package, module_filename = package_filename, module_relpath = Utils.relpath(package_filename), module_kind = "py", reason = "Containing package of recursed module '%s'." % self.getFullName(), ) self.package = imported_module if is_added: result.append(imported_module) if self.package: from nuitka.ModuleRegistry import addUsedModule addUsedModule(self.package) # print "Recursed to package", self.package_name result.extend(self.package.attemptRecursion()) return result def getCodeName(self): # Abstract method, pylint: disable=R0201 return None def getCompileTimeFilename(self): return Utils.abspath(self.getSourceReference().getFilename()) def getRunTimeFilename(self): reference_mode = Options.getFileReferenceMode() if reference_mode == "original": return self.getCompileTimeFilename() elif reference_mode == "frozen": return "" % self.getFullName() else: filename = self.getCompileTimeFilename() full_name = self.getFullName() result = Utils.basename(filename) current = filename levels = full_name.count('.') if self.isCompiledPythonPackage(): levels += 1 for _i in range(levels): current = Utils.dirname(current) result = Utils.joinpath(Utils.basename(current), result) return result class CompiledPythonModule(PythonModuleMixin, ChildrenHavingMixin, ClosureGiverNodeBase): """ Compiled Python Module """ kind = "COMPILED_PYTHON_MODULE" named_children = ( "body", ) checkers = { "body": checkStatementsSequenceOrNone } def __init__(self, name, package_name, source_ref): ClosureGiverNodeBase.__init__( self, name = name, code_prefix = "module", source_ref = source_ref ) PythonModuleMixin.__init__( self, name = name, package_name = package_name ) ChildrenHavingMixin.__init__( self, values = { "body" : None # delayed }, ) self.variables = {} # The list functions contained in that module. self.functions = OrderedSet() self.active_functions = OrderedSet() self.cross_used_functions = OrderedSet() # SSA trace based information about the module. self.constraint_collection = None self.code_object = CodeObjectSpec( code_name = "" if self.isMainModule() else self.getName(), code_kind = "Module", arg_names = (), kw_only_count = 0, has_stardict = False, has_starlist = False, ) def getCodeObject(self): return self.code_object def getDetails(self): return { "filename" : self.source_ref.getFilename(), "package" : self.package_name, "name" : self.name } def asXml(self): result = super(CompiledPythonModule, self).asXml() for function_body in self.active_functions: result.append(function_body.asXml()) return result def asGraph(self, computation_counter): from graphviz import Digraph # @UnresolvedImport pylint: disable=F0401,I0021 graph = Digraph("cluster_%d" % computation_counter, comment = "Graph for %s" % self.getName()) graph.body.append("style=filled") graph.body.append("color=lightgrey") graph.body.append("label=Iteration_%d" % computation_counter) def makeTraceNodeName(variable_trace): return "%d/ %s %s %s" % ( computation_counter, variable_trace.getVariable(), variable_trace.getVersion(), variable_trace.__class__.__name__ ) for function_body in self.active_functions: constraint_collection = function_body.constraint_collection for (_variable, _version), variable_trace in constraint_collection.getVariableTracesAll().items(): node = makeTraceNodeName(variable_trace) previous = variable_trace.getPrevious() if variable_trace.hasDefiniteUsages(): graph.attr("node", style = "filled", color = "blue") elif variable_trace.hasPotentialUsages(): graph.attr("node", style = "filled", color = "yellow") else: graph.attr("node", style = "filled", color = "red") graph.node(node) if type(previous) is tuple: for previous in previous: graph.edge(makeTraceNodeName(previous), node) elif previous is not None: graph.edge(makeTraceNodeName(previous), node) return graph getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") @staticmethod def isCompiledPythonModule(): return True def getParent(self): assert False def getParentVariableProvider(self): return None def hasVariableName(self, variable_name): return variable_name in self.variables or variable_name in self.temp_variables def getVariables(self): return self.variables.values() def getFilename(self): return self.source_ref.getFilename() def getVariableForAssignment(self, variable_name): return self.getProvidedVariable(variable_name) def getVariableForReference(self, variable_name): return self.getProvidedVariable(variable_name) def getVariableForClosure(self, variable_name): return self.getProvidedVariable( variable_name = variable_name ) def createProvidedVariable(self, variable_name): assert variable_name not in self.variables result = Variables.ModuleVariable( module = self, variable_name = variable_name ) self.variables[variable_name] = result return result @staticmethod def getContainingClassDictCreation(): return None def isEarlyClosure(self): # Modules should immediately closure variables on use. # pylint: disable=R0201 return True def getCodeName(self): def r(match): c = match.group() if c == '.': return '$' else: return "$$%d$" % ord(c) return "".join( re.sub("[^a-zA-Z0-9_]", r ,c) for c in self.getFullName() ) def addFunction(self, function_body): assert function_body not in self.functions self.functions.add(function_body) def getFunctions(self): return self.functions def startTraversal(self): self.active_functions = OrderedSet() def addUsedFunction(self, function_body): assert function_body in self.functions assert function_body.isExpressionFunctionBody() or \ function_body.isExpressionClassBody() or \ function_body.isExpressionGeneratorObjectBody() or \ function_body.isExpressionCoroutineObjectBody() if function_body not in self.active_functions: self.active_functions.add(function_body) def getUsedFunctions(self): return self.active_functions def getUnusedFunctions(self): for function in self.functions: if function not in self.active_functions: yield function def addCrossUsedFunction(self, function_body): if function_body not in self.cross_used_functions: self.cross_used_functions.add(function_body) def getCrossUsedFunctions(self): return self.cross_used_functions def getOutputFilename(self): main_filename = self.getFilename() if main_filename.endswith(".py"): result = main_filename[:-3] else: result = main_filename # There are some characters that somehow are passed to shell, by # Scons or unknown, so lets avoid them for now. return result.replace(')',"").replace('(',"") # TODO: Can't really use locals for modules, this should probably be made # sure to not be used. @staticmethod def getLocalsMode(): return "copy" def computeModule(self): old_collection = self.constraint_collection self.constraint_collection = ConstraintCollectionModule() module_body = self.getBody() if module_body is not None: result = module_body.computeStatementsSequence( constraint_collection = self.constraint_collection ) if result is not module_body: self.setBody(result) new_modules = self.attemptRecursion() for new_module in new_modules: self.constraint_collection.signalChange( source_ref = new_module.getSourceReference(), tags = "new_code", message = "Recursed to module package." ) self.constraint_collection.updateFromCollection(old_collection) def getTraceCollections(self): yield self.constraint_collection for function in self.getUsedFunctions(): yield function.constraint_collection def hasUnclearLocals(self): for collection in self.getTraceCollections(): if collection.hasUnclearLocals(): return True return False class CompiledPythonPackage(CompiledPythonModule): kind = "COMPILED_PYTHON_PACKAGE" def __init__(self, name, package_name, source_ref): assert name CompiledPythonModule.__init__( self, name = name, package_name = package_name, source_ref = source_ref ) def getOutputFilename(self): return Utils.dirname(self.getFilename()) def makeUncompiledPythonModule(module_name, filename, bytecode, is_package, user_provided): parts = module_name.rsplit('.', 1) name = parts[-1] package_name = parts[0] if len(parts) == 2 else None source_ref = fromFilename(filename) if is_package: return UncompiledPythonPackage( name = name, package_name = package_name, bytecode = bytecode, filename = filename, user_provided = user_provided, source_ref = source_ref ) else: return UncompiledPythonModule( name = name, package_name = package_name, bytecode = bytecode, filename = filename, user_provided = user_provided, source_ref = source_ref ) class UncompiledPythonModule(PythonModuleMixin, NodeBase): """ Compiled Python Module """ kind = "UNCOMPILED_PYTHON_MODULE" def __init__(self, name, package_name, bytecode, filename, user_provided, source_ref): NodeBase.__init__( self, source_ref = source_ref ) PythonModuleMixin.__init__( self, name = name, package_name = package_name ) self.bytecode = bytecode self.filename = filename self.user_provided = user_provided def isUserProvided(self): return self.user_provided def getByteCode(self): return self.bytecode @staticmethod def isPackage(): return False class UncompiledPythonPackage(UncompiledPythonModule): kind = "UNCOMPILED_PYTHON_PACKAGE" @staticmethod def isPackage(): return True class SingleCreationMixin: created = set() def __init__(self): assert self.__class__ not in self.created self.created.add(self.__class__) class PythonMainModule(CompiledPythonModule, SingleCreationMixin): kind = "PYTHON_MAIN_MODULE" def __init__(self, main_added, source_ref): CompiledPythonModule.__init__( self, name = "__main__", package_name = None, source_ref = source_ref ) SingleCreationMixin.__init__(self) self.main_added = main_added @staticmethod def isMainModule(): return True def getOutputFilename(self): if self.main_added: return Utils.dirname(self.getFilename()) else: return CompiledPythonModule.getOutputFilename(self) class PythonInternalModule(CompiledPythonModule, SingleCreationMixin): kind = "PYTHON_INTERNAL_MODULE" def __init__(self): CompiledPythonModule.__init__( self, name = "__internal__", package_name = None, source_ref = SourceCodeReference.fromFilenameAndLine( filename = "internal", line = 0, future_spec = FutureSpec() ) ) SingleCreationMixin.__init__(self) @staticmethod def isInternalModule(): return True def getOutputFilename(self): return "__internal" class PythonShlibModule(PythonModuleMixin, NodeBase): kind = "PYTHON_SHLIB_MODULE" avoid_duplicates = set() def __init__(self, name, package_name, source_ref): NodeBase.__init__( self, source_ref = source_ref ) PythonModuleMixin.__init__( self, name = name, package_name = package_name ) # That would be a mistake we just made. assert Utils.basename(source_ref.getFilename()) != "" # That is too likely a bug. assert name != "__main__" # Duplicates should be avoided by us caching elsewhere before creating # the object. assert self.getFullName() not in self.avoid_duplicates, self.getFullName() self.avoid_duplicates.add(self.getFullName()) def getDetails(self): return { "name" : self.name, "package_name" : self.package_name } def getFilename(self): return self.getSourceReference().getFilename() def startTraversal(self): pass class ExpressionModuleFileAttributeRef(NodeBase, ExpressionMixin): kind = "EXPRESSION_MODULE_FILE_ATTRIBUTE_REF" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def mayRaiseException(self, exception_type): return False def computeExpression(self, constraint_collection): # There is not a whole lot to do here, the path will change at run # time if Options.getFileReferenceMode() != "runtime": result = ExpressionConstantRef( constant = self.getRunTimeFilename(), source_ref = self.getSourceReference() ) return result, "new_expression", "Using original '__file__' value." return self, None, None def getCompileTimeFilename(self): return self.getParentModule().getCompileTimeFilename() def getRunTimeFilename(self): return self.getParentModule().getRunTimeFilename() Nuitka-0.5.18.1/nuitka/ModuleRegistry.py0000644000175000017500000001001112651071040020241 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This to keep track of used modules. There is a set of root modules, which are user specified, and must be processed. As they go, they add more modules to active modules list and move done modules out of it. That process can be restarted and modules will be fetched back from the existing set of modules. """ from nuitka.containers.oset import OrderedSet from nuitka.PythonVersions import python_version from nuitka.utils import Utils # One or more root modules, i.e. entry points that must be there. root_modules = OrderedSet() # To be traversed modules active_modules = OrderedSet() # Already traversed modules done_modules = set() # Uncompiled modules uncompiled_modules = set() def addRootModule(module): root_modules.add(module) def getRootModules(): return root_modules def hasRootModule(module_name): for module in root_modules: if module.getFullName() == module_name: return True return False def addUncompiledModule(module): uncompiled_modules.add(module) def getUncompiledModules(): return sorted( uncompiled_modules, key = lambda module : module.getFullName() ) def _normalizeModuleFilename(filename): if python_version >= 300: filename = filename.replace("__pycache__", "") suffix = ".cpython-%d.pyc" % (python_version // 10) if filename.endswith(suffix): filename = filename[:-len(suffix)] + ".py" else: if filename.endswith(".pyc"): filename = filename[:-3] + ".py" if Utils.basename(filename) == "__init__.py": filename = Utils.dirname(filename) return filename def getUncompiledModule(module_name, module_filename): for uncompiled_module in uncompiled_modules: if module_name == uncompiled_module.getFullName(): if Utils.areSamePaths( _normalizeModuleFilename(module_filename), _normalizeModuleFilename(uncompiled_module.filename) ): return uncompiled_module return None def removeUncompiledModule(module): uncompiled_modules.remove(module) def startTraversal(): # Using global here, as this is really a singleton, in the form of a module, # pylint: disable=W0603 global active_modules, done_modules active_modules = OrderedSet(root_modules) done_modules = set() for active_module in active_modules: active_module.startTraversal() def addUsedModule(module): if module not in done_modules and module not in active_modules: active_modules.add(module) module.startTraversal() def nextModule(): if active_modules: result = active_modules.pop() done_modules.add(result) return result else: return None def remainingCount(): return len(active_modules) def getDoneModules(): return sorted( done_modules, key = lambda module : module.getFullName() ) def getDoneUserModules(): return sorted( [ module for module in done_modules if not module.isInternalModule() if not module.isMainModule() ], key = lambda module : module.getFullName() ) def removeDoneModule(module): done_modules.remove(module) Nuitka-0.5.18.1/nuitka/utils/0000755000175000017500000000000012651072347016073 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/utils/InstanceCounters.py0000644000175000017500000000452712651071040021731 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Instance counter primitives We don't use a meta class as it's unnecessary complex, and portable meta classes have their difficulties, and want to count classes, who already have a meta class. This is going to expanded with time. """ from nuitka.Options import isShowMemory from nuitka.Tracing import printIndented, printLine counted_inits = {} counted_dels = {} def counted_init(init): if isShowMemory(): def wrapped_init(self, *args, **kw): name = self.__class__.__name__ assert type(name) is str if name not in counted_inits: counted_inits[name] = 0 counted_inits[name] += 1 init(self, *args, **kw) return wrapped_init else: return init empty_del = lambda x : 0 def counted_del(del_func = empty_del): if isShowMemory(): def wrapped_del(self): # This cannot be necessary, because in program finalization, the # global variables were assign to None. if counted_dels is None: return name = self.__class__.__name__ assert type(name) is str if name not in counted_dels: counted_dels[name] = 0 counted_dels[name] += 1 if del_func is not empty_del: del_func(self) return wrapped_del else: return empty_del def printStats(): printLine("Init/del calls:") for name, count in sorted(counted_inits.items()): dels = counted_dels.get(name, 0) printIndented(1, name, count, dels, count - dels) Nuitka-0.5.18.1/nuitka/utils/Utils.py0000644000175000017500000002023112651071040017530 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Utility module. Here the small things for file/dir names, Python version, CPU counting, memory usage, etc. that fit nowhere else and don't deserve their own names. """ import os import subprocess import sys from nuitka.PythonVersions import python_version def getOS(): if os.name == "nt": return "Windows" elif os.name == "posix": return os.uname()[0] # @UndefinedVariable else: assert False, os.name def getArchitecture(): if getOS() == "Windows": if "AMD64" in sys.version: return "x86_64" else: return "x86" else: return os.uname()[4] # @UndefinedVariable def relpath(path): try: return os.path.relpath(path) except ValueError: # On Windows, paths on different devices prevent it to work. Use that # full path then. if getOS() == "Windows": return os.path.abspath(path) raise def abspath(path): return os.path.abspath(path) def isAbsolutePath(path): return os.path.isabs(path) def joinpath(*parts): return os.path.join(*parts) def splitpath(path): return tuple( element for element in os.path.split(path) if element ) def basename(path): return os.path.basename(path) def dirname(path): return os.path.dirname(path) def normpath(path): return os.path.normpath(path) def normcase(path): return os.path.normcase(path) def getExtension(path): return os.path.splitext(path)[1] def isFile(path): return os.path.isfile(path) def isDir(path): return os.path.isdir(path) def isLink(path): return os.path.islink(path) def areSamePaths(path1, path2): path1 = normcase(abspath(normpath(path1))) path2 = normcase(abspath(normpath(path2))) return path1 == path2 def readLink(path): return os.readlink(path) # @UndefinedVariable def listDir(path): """ Give a sorted path, base filename pairs of a directory.""" return sorted( [ ( joinpath(path, filename), filename ) for filename in os.listdir(path) ] ) def getFileList(path): for root, _dirnames, filenames in os.walk(path): for filename in filenames: yield joinpath(root, filename) def deleteFile(path, must_exist): if must_exist or isFile(path): os.unlink(path) def makePath(path): os.makedirs(path) def getCoreCount(): cpu_count = 0 # Try to sum up the CPU cores, if the kernel shows them. try: # Try to get the number of logical processors with open("/proc/cpuinfo") as cpuinfo_file: cpu_count = cpuinfo_file.read().count("processor\t:") except IOError: pass if not cpu_count: import multiprocessing cpu_count = multiprocessing.cpu_count() return cpu_count def callExec(args): """ Do exec in a portable way preserving exit code. On Windows, unfortunately there is no real exec, so we have to spawn a new process instead. """ # On Windows os.execl does not work properly if getOS() != "Windows": # The star arguments is the API of execl os.execl(*args) else: args = list(args) del args[1] try: sys.exit( subprocess.call(args) ) except KeyboardInterrupt: # There was a more relevant stack trace already, so abort this # right here, pylint: disable=W0212 os._exit(2) def encodeNonAscii(var_name): """ Encode variable name that is potentially not ASCII to ASCII only. For Python3, unicode identifiers can be used, but these are not possible in C++03, so we need to replace them. """ if python_version < 300: return var_name else: # Using a escaping here, because that makes it safe in terms of not # to occur in the encoding escape sequence for unicode use. var_name = var_name.replace("$$", "$_$") var_name = var_name.encode("ascii", "xmlcharrefreplace") var_name = var_name.decode("ascii") return var_name.replace("&#", "$$").replace(';', "") def isExecutableCommand(command): path = os.environ["PATH"] suffixes = (".exe",) if getOS() == "Windows" else ("",) for part in path.split(os.pathsep): if not part: continue for suffix in suffixes: if isFile(joinpath(part, command + suffix)): return True return False def getOwnProcessMemoryUsage(): """ Memory usage of own process in bytes. """ if getOS() == "Windows": # adapted from http://code.activestate.com/recipes/578513 import ctypes from ctypes import wintypes # Lets allow this to match Windows API it reflects, # pylint: disable=C0103 class PROCESS_MEMORY_COUNTERS_EX(ctypes.Structure): _fields_ = [ ("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD), ("PeakWorkingSetSize", ctypes.c_size_t), ("WorkingSetSize", ctypes.c_size_t), ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), ("QuotaPagedPoolUsage", ctypes.c_size_t), ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), ("QuotaNonPagedPoolUsage", ctypes.c_size_t), ("PagefileUsage", ctypes.c_size_t), ("PeakPagefileUsage", ctypes.c_size_t), ("PrivateUsage", ctypes.c_size_t), ] GetProcessMemoryInfo = ctypes.windll.psapi.GetProcessMemoryInfo GetProcessMemoryInfo.argtypes = [ wintypes.HANDLE, ctypes.POINTER(PROCESS_MEMORY_COUNTERS_EX), wintypes.DWORD, ] GetProcessMemoryInfo.restype = wintypes.BOOL counters = PROCESS_MEMORY_COUNTERS_EX() rv = GetProcessMemoryInfo( ctypes.windll.kernel32.GetCurrentProcess(), # @UndefinedVariable ctypes.byref(counters), ctypes.sizeof(counters) ) if not rv: raise ctypes.WinError() return counters.PrivateUsage else: # Posix only code, pylint: disable=F0401,I0021 import resource # @UnresolvedImport # The value is from "getrusage", which has OS dependent scaling, at least # MacOS and Linux different if getOS() == "Darwin": factor = 1 else: factor = 1024 return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * factor def getHumanReadableProcessMemoryUsage(value = None): if value is None: value = getOwnProcessMemoryUsage() if abs(value) < 1024*1014: return "%.2f KB (%d bytes)" % ( value / 1024.0, value ) elif abs(value) < 1024*1014*1024: return "%.2f MB (%d bytes)" % ( value / (1024*1024.0), value ) elif abs(value) < 1024*1014*1024*1024: return "%.2f GB (%d bytes)" % ( value / (1024*1024*1024.0), value ) else: return "%d bytes" % value class MemoryWatch: def __init__(self): self.start = getOwnProcessMemoryUsage() self.stop = None def finish(self): self.stop = getOwnProcessMemoryUsage() def asStr(self): return getHumanReadableProcessMemoryUsage(self.stop - self.start) Nuitka-0.5.18.1/nuitka/utils/__init__.py0000644000175000017500000000150112651071040020166 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/importing/0000755000175000017500000000000012651072347016743 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/importing/StandardLibrary.py0000644000175000017500000001053312651071040022371 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Access to standard library distinction. For code to be in the standard library means that it's not written by the user for sure. We treat code differently based on that information, by e.g. including as byte code. To determine if a module from the standard library, we can abuse the attribute "__file__" of the "os" module like it's done in "isStandardLibraryPath" of this module. """ import os from nuitka.utils import Utils def getStandardLibraryPaths(): """ Get the standard library paths. """ # Using the function object to cache its result, avoiding global variable # usage. if not hasattr(getStandardLibraryPaths, "result"): os_filename = os.__file__ if os_filename.endswith(".pyc"): os_filename = os_filename[:-1] os_path = Utils.normcase(Utils.dirname(os_filename)) stdlib_paths = set([os_path]) # Happens for virtualenv situation, some modules will come from the link # this points to. if Utils.isLink(os_filename): os_filename = Utils.readLink(os_filename) stdlib_paths.add(Utils.normcase(Utils.dirname(os_filename))) # Another possibility is "orig-prefix.txt" file near the os.py, which # points to the original install. orig_prefix_filename = Utils.joinpath(os_path, "orig-prefix.txt") if Utils.isFile(orig_prefix_filename): # Scan upwards, until we find a "bin" folder, with "activate" to # locate the structural path to be added. We do not know for sure # if there is a sub-directory under "lib" to use or not. So we try # to detect it. search = os_path lib_part = "" while os.path.splitdrive(search)[1] not in (os.path.sep, ""): if Utils.isFile(Utils.joinpath(search,"bin/activate")) or \ Utils.isFile(Utils.joinpath(search,"scripts/activate")): break lib_part = Utils.joinpath(Utils.basename(search), lib_part) search = Utils.dirname(search) assert search and lib_part stdlib_paths.add( Utils.normcase( Utils.joinpath( open(orig_prefix_filename).read(), lib_part, ) ) ) # And yet another possibility, for MacOS Homebrew created virtualenv # at least is a link ".Python", which points to the original install. python_link_filename = Utils.joinpath(os_path, "..", ".Python") if Utils.isLink(python_link_filename): stdlib_paths.add( Utils.normcase( Utils.joinpath( Utils.readLink(python_link_filename), "lib" ) ) ) getStandardLibraryPaths.result = [ Utils.normcase(stdlib_path) for stdlib_path in stdlib_paths ] return getStandardLibraryPaths.result def isStandardLibraryPath(path): """ Check if a path is in the standard library. """ path = Utils.normcase(path) # In virtualenv, the "site.py" lives in a place that suggests it is not in # standard library, although it is. if os.path.basename(path) == "site.py": return True # These never are in standard library paths. if "dist-packages" in path or "site-packages" in path: return False for candidate in getStandardLibraryPaths(): if path.startswith(candidate): return True return False Nuitka-0.5.18.1/nuitka/importing/Whitelisting.py0000644000175000017500000002042312651071040021755 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Whitelist modules that are not found, but probably that's acceptable. """ import sys from logging import warning def getModuleWhiteList(): return ( "mac", "nt", "os2", "posix", "_emx_link", "riscos", "ce", "riscospath", "riscosenviron", "Carbon.File", "org.python.core", "_sha", "_sha256", "array", "_sha512", "_md5", "_subprocess", "msvcrt", "cPickle", "marshal", "imp", "sys", "itertools", "cStringIO", "time", "zlib", "thread", "math", "errno", "operator", "signal", "gc", "exceptions", "win32process", "unicodedata", "__builtin__", "fcntl", "_socket", "_ssl", "pwd", "spwd", "_random", "grp", "_io", "_string", "select", "__main__", "_winreg", "_warnings", "_sre", "_functools", "_hashlib", "_collections", "_locale", "_codecs", "_weakref", "_struct", "_dummy_threading", "binascii", "datetime", "_ast", "xxsubtype", "_bytesio", "cmath", "_fileio", "aetypes", "aepack", "MacOS", "cd", "cl", "gdbm", "gl", "GL", "aetools", "_bisect", "_heapq", "_symtable", "syslog", "_datetime", "_elementtree", "_pickle", "_posixsubprocess", "_thread", "atexit", "pyexpat", "_imp", "_sha1", "faulthandler", "_osx_support", "sysconfig", "copyreg", "ipaddress", "reprlib", "win32event", "win32file", # Python-Qt4 does these if missing python3 parts: "PyQt4.uic.port_v3.string_io", "PyQt4.uic.port_v3.load_plugin", "PyQt4.uic.port_v3.ascii_upper", "PyQt4.uic.port_v3.proxy_base", "PyQt4.uic.port_v3.as_string", # CPython3 does these: "builtins", "UserDict", "os.path", "StringIO", # "test_array", "_testcapi", # test_applesingle.py "applesingle", # test_buffer.py "_testbuffer", # test_bsddb.py "bsddb.test", # test_collections.py "collections.abc", # test_compile.py "__package__.module", "__mangled_mod", # test_ctypes "ctypes.test", # test_dbm.py "dbm.dumb", # test_dbm_ndbm.py "dbm.ndbm", # test_distutils.py "distutils.tests", "distutils.mwerkscompiler", # test_docxmlrpc.py "xmlrpc.server", # test_emails.py "email.test.test_email", "email.test.test_email_renamed", "email.test.test_email_codecs", # test_email_codecs.py "email.test", # test_enum.py "enum", # test_file.py "_pyio", # test_frozen.py "__hello__", "__phello__", "__phello__.spam", "__phello__.foo", # test_fork1.py "fake test module", # test_html.py "html", "html.entities", # test_http_cookiejar.py "urllib.request", "http", # test_imp.py "importlib.test.import_", "pep3147.foo", "pep3147", # test_import.py "RAnDoM", "infinite_reload", "test_trailing_slash", "nonexistent_xyzzy", "_parent_foo.bar", "_parent_foo", "test_unc_path", # test_importhooks.py "hooktestmodule", "hooktestpackage", "hooktestpackage.sub", "reloadmodule", "hooktestpackage.sub.subber", "hooktestpackage.oldabs", "hooktestpackage.newrel", "hooktestpackage.sub.subber.subest", "hooktestpackage.futrel", "sub", "hooktestpackage.newabs", # test_imporlib.py" "importlib.test.__main__", "importlib", # test_inspect.py "inspect_fodder3", "test.test_import", # test_imageop.py "imgfile", # test_json.py "json.tests", # test_lib2to3.py "lib2to3.tests", # test_logging.py "win32evtlog", "win32evtlogutil", "pywintypes", # test_lzma.py "lzma", # test_macostools.py "macostools", # test_namespace_pkgs.py "foo.one", "foo.two", "parent.child.one", "parent.child.two", "parent.child.three", "bar.two", "a_test", # test_new.py "Spam", # test_ossaudiodev.py "ossaudiodev", # test_pathlib.py "pathlib", # test_platform.py "gestalt", # test_pkg.py "t1", "t2", "t2.sub", "t2.sub.subsub", "t3.sub.subsub", "t5", "t6", "t7", "t7.sub", "t7.sub.subsub", "t8", # test_pkgutil.py "foo", "foo.bar", "foo.baz", "zipimport", "pkg", "pkg.subpkg", "pkg.subpkg.c", "pkg.subpkg.d", # test_urllib.py "urllib.parse", # test_urllib_response.py "urllib.response", # test_repr.py """areallylongpackageandmodulenametotestreprtruncation.\ areallylongpackageandmodulenametotestreprtruncation""", # test_robotparser.py "urllib.error", "urllib.robotparser", # test_runpy.py "test.script_helper", # test_selectors.py "selectors", # test_statistics.py "statistics", # test_shelve.py "test.test_dbm", # test_strftime.py "java", # test_strop.py "strop", # test_sqlite3.py "sqlite3.test", # test_sundry.py "distutils.emxccompiler", "os2emxpath", # test_tcl.py "tkinter", # test_tk.py "runtktests", "tkinter.test", "tkinter.test.support", # test_tools.py "analyze_dxp", "test_unparse", "importlib.machinery", # test_traceback.py "test_bug737473", # test_tracemalloc "tracemalloc", # test_typing.py "mock", "typing.io", "typing.re", # test_unittest.py "unittest.test", # test_xml_etree.py "xml.parsers.expat.errors", # test_zipimport_support.py "test_zipped_doctest", "zip_pkg", # test/test_zipimport_support.py "test.test_cmd_line_script", # Python3: modules that no longer exist "commands", "dummy_thread", "_dummy_thread", "httplib", "Queue", "sets", # Python2: modules that don't yet exit "http.client", "queue", "winreg", # Very old modules with older names "simplejson", "sets", # Standalone mode "site" import flexibilities "sitecustomize", "usercustomize", "apport_python_hook", "_frozen_importlib", # Standard library stuff that is optional "comtypes.server.inprocserver", "_tkinter", "_scproxy", "EasyDialogs", "SOCKS", "rourl2path", "_winapi", "win32api", "win32con", "_gestalt", "java.lang", "vms_lib", "ic", "readline", "termios", "_sysconfigdata", "al", "AL", "sunaudiodev", "SUNAUDIODEV", "Audio_mac", "nis", "test.test_MimeWriter", "dos", "win32pipe", "Carbon", "Carbon.Files", "sgi", "ctypes.macholib.dyld", "bsddb3", "_pybsddb", "_xmlrpclib", "netbios", "win32wnet", "email.Parser", "elementree.cElementTree", "elementree.ElementTree", "_gbdm", "resource", "crypt", "bz2", "dbm", "mmap", "Mailman", # Nuitka tests "test_common", # Mercurial test "statprof", "email.Generator", "email.Utils", # setuptools does a lot of speculative stuff "wincertstore", "setuptools_svn", # reportlab does use this if present only and warns about itself. "pyfribidi2", "macfs", ) def isWhiteListedNotExistingModule(module_name): result = module_name in getModuleWhiteList() if not result and module_name in sys.builtin_module_names: warning("""\ Your CPython version has a built-in module '%s', that is not whitelisted please report this to http://bugs.nuitka.net.""", module_name) return result Nuitka-0.5.18.1/nuitka/importing/Recursion.py0000644000175000017500000003043612651071040021261 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Recursion into other modules. """ import glob import sys from logging import debug, warning import marshal from nuitka import ModuleRegistry, Options from nuitka.importing import ImportCache, Importing, StandardLibrary from nuitka.plugins.PluginBase import Plugins from nuitka.PythonVersions import python_version from nuitka.tree.SourceReading import readSourceCodeFromFilename from nuitka.utils import Utils def recurseTo(module_package, module_filename, module_relpath, module_kind, reason): from nuitka.tree import Building from nuitka.nodes.ModuleNodes import makeUncompiledPythonModule if not ImportCache.isImportedModuleByPath(module_relpath): module, source_ref, source_filename = Building.decideModuleTree( filename = module_filename, package = module_package, is_top = False, is_main = False, is_shlib = module_kind == "shlib" ) # Check if the module name is known. In order to avoid duplicates, # learn the new filename, and continue build if its not. if not ImportCache.isImportedModuleByName(module.getFullName()): debug( "Recurse to import '%s' from %s. (%s)", module.getFullName(), module_relpath, reason ) if module_kind == "py" and source_filename is not None: try: Building.createModuleTree( module = module, source_ref = source_ref, source_code = readSourceCodeFromFilename( module_name = module.getFullName(), source_filename = source_filename ), is_main = False ) except (SyntaxError, IndentationError) as e: if module_filename not in Importing.warned_about: Importing.warned_about.add(module_filename) warning( """\ Cannot recurse to import module '%s' (%s) because of '%s'""", module_relpath, module_filename, e.__class__.__name__ ) return None, False except Building.CodeTooComplexCode: if module_filename not in Importing.warned_about: Importing.warned_about.add(module_filename) warning( """\ Cannot recurse to import module '%s' (%s) because code is too complex.""", module_relpath, module_filename, ) if Options.isStandaloneMode(): module = makeUncompiledPythonModule( module_name = module.getFullName(), filename = module_filename, bytecode = marshal.dumps( compile( readSourceCodeFromFilename(module.getFullName(), module_filename), module_filename, "exec" ) ), is_package = module.isCompiledPythonPackage(), user_provided = True, ) ModuleRegistry.addUncompiledModule(module) return None, False ImportCache.addImportedModule(module) is_added = True else: module = ImportCache.getImportedModuleByName( module.getFullName() ) is_added = False assert not module_relpath.endswith("/__init__.py"), module return module, is_added else: return ImportCache.getImportedModuleByPath(module_relpath), False def decideRecursion(module_filename, module_name, module_package, module_kind): # Many branches, which make decisions immediately, by returning # pylint: disable=R0911,R0912 Plugins.onModuleEncounter( module_filename, module_name, module_package, module_kind ) if module_kind == "shlib": if Options.isStandaloneMode(): return True, "Shared library for inclusion." else: return False, "Shared library cannot be inspected." if module_package is None: full_name = module_name else: full_name = module_package + '.' + module_name no_case_modules = Options.getShallFollowInNoCase() for no_case_module in no_case_modules: if full_name == no_case_module: return ( False, "Module listed explicitly to not recurse to." ) if full_name.startswith(no_case_module + '.'): return ( False, "Module in package listed explicitly to not recurse to." ) any_case_modules = Options.getShallFollowModules() for any_case_module in any_case_modules: if full_name == any_case_module: return ( True, "Module listed explicitly to recurse to." ) if full_name.startswith(any_case_module + '.'): return ( True, "Module in package listed explicitly to recurse to." ) if Options.shallFollowNoImports(): return ( False, "Requested to not recurse at all." ) if StandardLibrary.isStandardLibraryPath(module_filename): return ( Options.shallFollowStandardLibrary(), "Requested to %srecurse to standard library." % ( "" if Options.shallFollowStandardLibrary() else "not " ) ) if Options.shallFollowAllImports(): return ( True, "Requested to recurse to all non-standard library modules." ) # Means, we were not given instructions how to handle things. return ( None, "Default behavior, not recursing without request." ) def considerFilename(module_filename): module_filename = Utils.normpath(module_filename) if Utils.isDir(module_filename): module_filename = Utils.abspath(module_filename) module_name = Utils.basename(module_filename) module_relpath = Utils.relpath(module_filename) return module_filename, module_relpath, module_name elif module_filename.endswith(".py"): module_name = Utils.basename(module_filename)[:-3] module_relpath = Utils.relpath(module_filename) return module_filename, module_relpath, module_name else: return None def isSameModulePath(path1, path2): if Utils.basename(path1) == "__init__.py": path1 = Utils.dirname(path1) if Utils.basename(path2) == "__init__.py": path2 = Utils.dirname(path2) return Utils.abspath(path1) == Utils.abspath(path2) def _checkPluginPath(plugin_filename, module_package): # Many branches, for the decision is very complex, pylint: disable=R0912 debug( "Checking detail plug-in path '%s' '%s':", plugin_filename, module_package ) plugin_info = considerFilename( module_filename = plugin_filename ) if plugin_info is not None: module, is_added = recurseTo( module_filename = plugin_info[0], module_relpath = plugin_info[1], module_package = module_package, module_kind = "py", reason = "Lives in plug-in directory." ) if module: if not is_added: warning( "Recursed to %s '%s' at '%s' twice.", "package" if module.isCompiledPythonPackage() else "module", module.getName(), plugin_info[0] ) if not isSameModulePath(module.getFilename(), plugin_info[0]): warning( "Duplicate ignored '%s'.", plugin_info[1] ) return debug( "Recursed to %s %s %s", module.getName(), module.getPackage(), module ) if module.isCompiledPythonPackage(): package_filename = module.getFilename() if Utils.isDir(package_filename): # Must be a namespace package. assert python_version >= 330 package_dir = package_filename # Only include it, if it contains actual modules, which will # recurse to this one and find it again. useful = False else: package_dir = Utils.dirname(package_filename) # Real packages will always be included. useful = True debug( "Package directory %s", package_dir ) for sub_path, sub_filename in Utils.listDir(package_dir): if sub_filename in ("__init__.py", "__pycache__"): continue assert sub_path != plugin_filename if Importing.isPackageDir(sub_path) or \ sub_path.endswith(".py"): _checkPluginPath(sub_path, module.getFullName()) else: # Modules should always be included. useful = True if useful: ModuleRegistry.addRootModule(module) ImportCache.addImportedModule(module) else: warning("Failed to include module from '%s'.", plugin_info[0]) def checkPluginPath(plugin_filename, module_package): debug( "Checking top level plug-in path %s %s", plugin_filename, module_package ) plugin_info = considerFilename( module_filename = plugin_filename ) if plugin_info is not None: # File or package makes a difference, handle that if Utils.isFile(plugin_info[0]) or \ Importing.isPackageDir(plugin_info[0]): _checkPluginPath(plugin_filename, module_package) elif Utils.isDir(plugin_info[0]): for sub_path, sub_filename in Utils.listDir(plugin_info[0]): assert sub_filename != "__init__.py" if Importing.isPackageDir(sub_path) or \ sub_path.endswith(".py"): _checkPluginPath(sub_path, None) else: warning("Failed to include module from '%s'.", plugin_info[0]) else: warning("Failed to recurse to directory '%s'.", plugin_filename) def checkPluginFilenamePattern(pattern): debug( "Checking plug-in pattern '%s':", pattern, ) if Utils.isDir(pattern): sys.exit("Error, pattern cannot be a directory name.") found = False for filename in glob.iglob(pattern): if filename.endswith(".pyc"): continue if not Utils.isFile(filename): continue found = True _checkPluginPath(filename, None) if not found: warning("Didn't match any files against pattern '%s'." % pattern) Nuitka-0.5.18.1/nuitka/importing/Importing.py0000644000175000017500000004034012651071040021253 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Locating modules and package source on disk. The actual import of a module would already execute code that changes things. Consider a module that does "os.system()", it would be executed. People often connect to databases, and these kind of things, at import time. Not a good style, but it's being done. Therefore CPython exhibits the interfaces in an "imp" module or more modern "importlib" in standard library, which one can use those to know ahead of time, to tell what filename an import would load from. For us unfortunately there is nothing in CPython that gives the fully compatible functionality we need for packages and search paths exactly like CPython really does, so we implement here a multiple step search process that is compatible. """ from __future__ import print_function import imp import os import sys from logging import warning from nuitka import Options from nuitka.containers import oset from nuitka.plugins.PluginBase import Plugins from nuitka.PythonVersions import python_version from nuitka.utils import Utils from .PreloadedPackages import getPreloadedPackagePath, isPreloadedPackagePath from .Whitelisting import isWhiteListedNotExistingModule _debug_module_finding = Options.shallExplainImports() warned_about = set() # Directory where the main script lives. Should attempt to import from there. main_path = None def setMainScriptDirectory(main_dir): """ Initialize the main script directory. We use this as part of the search path for modules. """ # We need to set this from the outside, pylint: disable=W0603 global main_path main_path = main_dir def isPackageDir(dirname): """ Decide if a directory is a package. Before Python3.3 it's required to have a "__init__.py" file, but then it became impossible to decide, and for extra fun, there is also the extra packages provided via "*.pth" file tricks by "site.py" loading. """ return Utils.isDir(dirname) and \ ( python_version >= 330 or Utils.isFile(Utils.joinpath(dirname, "__init__.py")) or isPreloadedPackagePath(dirname) ) def getPackageNameFromFullName(full_name): if '.' in full_name: return full_name[:full_name.rfind('.')] else: return None def getExtensionModuleSuffixes(): for suffix, _mode, kind in imp.get_suffixes(): if kind == imp.C_EXTENSION: yield suffix def getModuleNameAndKindFromFilename(module_filename): if Utils.isDir(module_filename): module_name = Utils.basename(module_filename) module_kind = "py" elif module_filename.endswith(".py"): module_name = Utils.basename(module_filename)[:-3] module_kind = "py" else: for suffix, _mode, kind in imp.get_suffixes(): if kind != imp.C_EXTENSION: continue if module_filename.endswith(suffix): module_name = Utils.basename(module_filename)[:-len(suffix)] module_kind = "shlib" break else: module_kind = None module_name = None return module_name, module_kind def warnAbout(importing, module_name, parent_package, level): # This probably should not be dealt with here. if module_name == "": return if not isWhiteListedNotExistingModule(module_name): key = module_name, parent_package, level if key not in warned_about: warned_about.add(key) if parent_package is None: full_name = module_name else: full_name = module_name if Plugins.suppressUnknownImportWarning(importing, full_name): return if level == 0: level_desc = "as absolute import" elif level == -1: level_desc = "as relative or absolute import" elif level == 1: level_desc = "%d package level up" % level else: level_desc = "%d package levels up" % level if parent_package is not None: warning( "%s: Cannot find '%s' in package '%s' %s.", importing.getSourceReference().getAsString(), module_name, parent_package, level_desc ) else: warning( "%s: Cannot find '%s' %s.", importing.getSourceReference().getAsString(), module_name, level_desc ) def normalizePackageName(module_name): # The "os.path" is strangely hacked into the "os" module, dispatching per # platform, we either cannot look into it, or we require that we resolve it # here correctly. if module_name == "os.path": module_name = Utils.basename(os.path.__name__) return module_name def findModule(importing, module_name, parent_package, level, warn): """ Find a module with given package name as parent. The package name can be None of course. Level is the same as with "__import__" built-in. Warnings are optional. Returns a triple of package name the module is in, module name and filename of it, which can be a directory. """ # We have many branches here, because there are a lot of cases to try. # pylint: disable=R0912 if _debug_module_finding: print( "findModule: Enter to search %r in package %r level %s." % ( module_name, parent_package, level ) ) # Do not allow star imports to get here. We just won't find modules with # that name, but it would be wasteful. assert module_name != '*' tried_names = [] if level > 1: # TODO: Should give a warning and return not found if the levels # exceed the package name. if parent_package is not None: parent_package = '.'.join(parent_package.split('.')[:-level+1]) if parent_package == "": parent_package = None else: return None, None, "not-found" # Try relative imports first if we have a parent package. if level != 0 and parent_package is not None: full_name = normalizePackageName(parent_package + '.' + module_name) if full_name.endswith('.'): full_name = full_name[:-1] tried_names.append(full_name) try: module_filename = _findModule( module_name = full_name, ) except ImportError: # For relative import, that is OK, we will still try absolute. pass else: if _debug_module_finding: print( "findModule: Relative imported module '%s' as '%s' in filename '%s':" % ( module_name, full_name, module_filename ) ) return getPackageNameFromFullName(full_name), module_filename, "relative" if level <= 1 and module_name != "": module_name = normalizePackageName(module_name) tried_names.append(module_name) package_name = getPackageNameFromFullName(module_name) # Built-in module names must not be searched any further. if module_name in sys.builtin_module_names: if _debug_module_finding: print( "findModule: Absolute imported module '%s' in as built-in':" % ( module_name, ) ) return package_name, None, "built-in" try: module_filename = _findModule( module_name = module_name, ) except ImportError: # For relative import, that is OK, we will still try absolute. pass else: if _debug_module_finding: print( "findModule: Found absolute imported module '%s' in filename '%s':" % ( module_name, module_filename ) ) return package_name, module_filename, "absolute" if warn: warnAbout( importing = importing, module_name = module_name, parent_package = parent_package, level = level, ) return None, None, "not-found" # Some platforms are case insensitive. case_sensitive = not sys.platform.startswith(("win", "cygwin", "darwin")) def _findModuleInPath2(module_name, search_path): """ This is out own module finding low level implementation. Just the full module name and search path are given. This is then tasked to raise "ImportError" or return a path if it finds it, or None, if it is a built-in. """ # We have many branches here, because there are a lot of cases to try. # pylint: disable=R0912 # We may have to decide between package and module, therefore build # a list of candidates. candidates = oset.OrderedSet() considered = set() for entry in search_path: # Don't try again, just with an entry of different casing or complete # duplicate. if Utils.normcase(entry) in considered: continue considered.add(Utils.normcase(entry)) package_directory = os.path.join(entry, module_name) # First, check for a package with an init file, that would be the # first choice. if Utils.isDir(package_directory): for suffix in (".py", ".pyc"): package_file_name = "__init__" + suffix file_path = os.path.join(package_directory, package_file_name) if Utils.isFile(file_path): candidates.add( (entry, 1, package_directory) ) break else: if python_version >= 330: candidates.add( (entry, 2, package_directory) ) # Then, check out suffixes of all kinds. for suffix, _mode, _type in imp.get_suffixes(): file_path = Utils.joinpath(entry, module_name + suffix) if Utils.isFile(file_path): candidates.add( (entry, 1, file_path) ) break if _debug_module_finding: print("Candidates", candidates) if candidates: # Ignore lower priority matches from package directories without # "__init__.py" file. min_prio = min(candidate[1] for candidate in candidates) candidates = [ candidate for candidate in candidates if candidate[1] == min_prio ] # On case sensitive systems, no resolution needed. if case_sensitive: return candidates[0][2] else: for candidate in candidates: dir_listing = os.listdir(candidate[0]) for filename in dir_listing: if Utils.joinpath(candidate[0], filename) == candidate[2]: return candidate[2] # Only exact case matches matter, all candidates were ignored, # lets just fall through to raising the import error. # Nothing found. raise ImportError def getPackageSearchPath(package_name): assert main_path is not None if package_name is None: return [os.getcwd(), main_path] + sys.path elif '.' in package_name: parent_package_name, package_name = package_name.rsplit('.', 1) result = [] for element in getPackageSearchPath(parent_package_name): package_dir = Utils.joinpath( element, package_name ) if isPackageDir(package_dir): result.append(package_dir) return result else: preloaded_path = getPreloadedPackagePath(package_name) if preloaded_path is not None: return preloaded_path def getPackageDirCandidates(element): yield Utils.joinpath(element, package_name), False # Hack for PyWin32. TODO: Move this __path__ extensions to # plug-in decisions. if package_name == "win32com": yield Utils.joinpath(element, "win32comext"), True result = [] for element in getPackageSearchPath(None): for package_dir, force_package in getPackageDirCandidates(element): if isPackageDir(package_dir) or force_package: result.append(package_dir) return result def _findModuleInPath(module_name, package_name): if _debug_module_finding: print("_findModuleInPath: Enter", module_name, "in", package_name) # The "site" module must be located based on PYTHONPATH before it was # executed, while we normally search in PYTHONPATH after it was executed, # and on some systems, that fails. if package_name is None and module_name == "site": candidate = os.environ.get("NUITKA_SITE_FILENAME", "") if candidate: return candidate # Free pass for built-in modules, the need not exist. if package_name is None and imp.is_builtin(module_name): return None, module_name search_path = getPackageSearchPath(package_name) if _debug_module_finding: print("_findModuleInPath: Using search path", search_path) try: module_filename = _findModuleInPath2( module_name = module_name, search_path = search_path ) except SyntaxError: # Warn user, as this is kind of unusual. warning( "%s: Module cannot be imported due to syntax errors.", module_name if package_name is None else package_name + '.' + module_name, ) return None if _debug_module_finding: print("_findModuleInPath: _findModuleInPath2 gave", module_filename) return module_filename module_search_cache = {} def _findModule(module_name): if _debug_module_finding: print( "_findModule: Enter to search '%s'." % ( module_name, ) ) assert not module_name.endswith('.'), module_name key = module_name if key in module_search_cache: result = module_search_cache[key] if _debug_module_finding: print("_findModule: Cached result (see previous call).") if result is ImportError: raise ImportError else: return result try: module_search_cache[key] = _findModule2(module_name) except ImportError: new_module_name = Plugins.considerFailedImportReferrals(module_name) if new_module_name is None: module_search_cache[key] = ImportError raise else: module_search_cache[key] = _findModule(new_module_name) return module_search_cache[key] def _findModule2(module_name): # Need a real module name. assert module_name != "" if '.' in module_name: package_part = module_name[ : module_name.rfind('.') ] module_name = module_name[ module_name.rfind('.') + 1 : ] else: package_part = None preloaded_path = getPreloadedPackagePath(module_name) if preloaded_path is not None: return preloaded_path[0] return _findModuleInPath( module_name = module_name, package_name = package_part ) Nuitka-0.5.18.1/nuitka/importing/PreloadedPackages.py0000644000175000017500000000733612651071040022651 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module abstracts what site.py is normally doing in .pth files. This tries to extract "namespaces" packages that were manually created and point to package directories, which need no "__init__.py" to count as a package. Nuitka will pretend for those that there be one, but without content. """ import sys from logging import warning from nuitka.utils import Utils def getLoadedPackages(): """ Extract packages with no __file__, i.e. they got added manually. They are frequently created with "*.pth" files that then check for the "__init__.py" to exist, and when it doesn't, then they create during the loading of "site.py" an package with "__path__" set. """ for module_name, module in sys.modules.items(): if not hasattr(module, "__path__"): continue if hasattr(module, "__file__"): continue yield module_name, module def detectPreLoadedPackagePaths(): result = {} for package_name, module in getLoadedPackages(): result[package_name] = list(module.__path__) return result preloaded_packages = None def getPreloadedPackagePaths(): # We need to set this from the outside, pylint: disable=W0603 global preloaded_packages if preloaded_packages is None: preloaded_packages = detectPreLoadedPackagePaths() return preloaded_packages def setPreloadedPackagePaths(value): # We need to set this from the outside, pylint: disable=W0603 global preloaded_packages preloaded_packages = value def getPreloadedPackagePath(package_name): return getPreloadedPackagePaths().get(package_name, None) def isPreloadedPackagePath(path): path = Utils.normcase(path) for paths in getPreloadedPackagePaths().values(): for element in paths: if Utils.normcase(element) == path: return True return False def detectPthImportedPackages(): if not hasattr(sys.modules["site"], "getsitepackages"): return () pth_imports = set() for prefix in sys.modules["site"].getsitepackages(): if not Utils.isDir(prefix): continue for path, filename in Utils.listDir(prefix): if filename.endswith(".pth"): try: for line in open(path, "rU"): if line.startswith("import "): if ';' in line: line = line[:line.find(';')] for part in line[7:].split(','): pth_imports.add(part.strip()) except OSError: warning("Python installation problem, cannot read file '%s'.") return tuple(sorted(pth_imports)) pth_imported_packages = () def setPthImportedPackages(value): # We need to set this from the outside, pylint: disable=W0603 global pth_imported_packages pth_imported_packages = value def getPthImportedPackages(): return pth_imported_packages Nuitka-0.5.18.1/nuitka/importing/__init__.py0000644000175000017500000000150112651071040021036 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/importing/ImportCache.py0000644000175000017500000000436112651071040021504 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Import cache. This is not about caching the search of modules in the file system, but about maintaining a cache of module trees built. It can happen that modules become unused, and then dropped from active modules, and then later active again, via another import, and in this case, we should not start anew, but reuse what we already found out about it. """ from nuitka.plugins.PluginBase import Plugins from nuitka.utils import Utils imported_modules = {} imported_by_name = {} def addImportedModule(imported_module): key = ( Utils.relpath(imported_module.getFilename()), imported_module.getFullName() ) if key in imported_modules: assert imported_module is imported_modules[key], key else: Plugins.onModuleDiscovered(imported_module) imported_modules[key] = imported_module imported_by_name[imported_module.getFullName()] = imported_module # We don't expect that to happen. assert not imported_module.isMainModule() def isImportedModuleByPath(module_relpath): for key in imported_modules: if key[0] == module_relpath: return True return False def isImportedModuleByName(full_name): return full_name in imported_by_name def getImportedModuleByName(full_name): return imported_by_name[full_name] def getImportedModuleByPath(module_relpath): for key in imported_modules: if key[0] == module_relpath: return imported_modules[key] raise KeyError(module_relpath) Nuitka-0.5.18.1/nuitka/SourceCodeReferences.py0000644000175000017500000000721512651071040021334 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Source code reference record. All the information to lookup line and file of a code location, together with the future flags in use there. """ from nuitka.nodes.FutureSpecs import FutureSpec from nuitka.utils.InstanceCounters import counted_del, counted_init class SourceCodeReference(object): # TODO: Measure the access speed impact of slots. The memory savings is # not worth it (only a few percent). __slots__ = ["filename", "line", "future_spec", "internal"] @classmethod def fromFilenameAndLine(cls, filename, line, future_spec): result = cls() result.filename = filename result.line = line result.future_spec = future_spec return result __del__ = counted_del() @counted_init def __init__(self): self.line = None self.filename = None self.future_spec = None self.internal = False def __repr__(self): return "<%s to %s:%s>" % (self.__class__.__name__, self.filename, self.line) def __cmp__(self, other): if other is None: return -1 assert isinstance(other, SourceCodeReference), other result = cmp(self.filename, other.filename) if result == 0: result = cmp(self.line, other.line) if result == 0: result = cmp(self.internal, other.internal) return result def _clone(self, line): """ Make a copy it itself. """ result = SourceCodeReference.fromFilenameAndLine( filename = self.filename, line = line, future_spec = self.future_spec ) result.internal = self.internal return result def atInternal(self): """ Make a copy it itself but mark as internal code. Avoids useless copies, by returning an internal object again if it is already internal. """ if not self.internal: result = self._clone(self.line) result.internal = True return result else: return self def atLineNumber(self, line): """ Make a reference to the same file, but different line. Avoids useless copies, by returning same object if the line is the same. """ assert type(line) is int, line if self.line != line: return self._clone(line) else: return self def getLineNumber(self): return self.line def getFilename(self): return self.filename def getFutureSpec(self): return self.future_spec def getAsString(self): return "%s:%s" % (self.filename, self.line) def isInternal(self): return self.internal def fromFilename(filename): return SourceCodeReference.fromFilenameAndLine( filename = filename, line = 1, future_spec = FutureSpec(), ) Nuitka-0.5.18.1/nuitka/Tracing.py0000644000175000017500000000271412651071040016665 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Outputs to the user. Printing with intends or plain, mostly a compensation for the print strangeness. We want to avoid "from __future__ import print_function" in every file out there, which makes adding another debug print rather tedious. This should cover all calls/uses of "print" we have to do, and the make it easy to simply to "print for_debug" without much hassle (braces). """ from __future__ import print_function import sys def printIndented(level, *what): print(" " * level, *what) def printSeparator(level = 0): print(" " * level, '*' * 10) def printLine(*what): print(*what) def printError(message): print( message, file = sys.stderr ) Nuitka-0.5.18.1/nuitka/codegen/0000755000175000017500000000000012651072347016337 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/codegen/Indentation.py0000644000175000017500000000260212651071040021152 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Indentation of code. Language independent, the amount of the spaces is not configurable, as it needs to be the same as in templates. """ def _indentedCode(codes, count): return '\n'.join( ' ' * count + line if (line and not line.startswith('#')) else line for line in codes ) def indented(codes, level = 1, vert_block = False): if type(codes) is str: codes = codes.split('\n') if vert_block and codes != [""]: codes.insert(0, "") codes.append("") return _indentedCode(codes, level * 4) def getCommentCode(comment, emit): emit("// " + comment) Nuitka-0.5.18.1/nuitka/codegen/OperationCodes.py0000644000175000017500000001311212651071040021612 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Codes for operations. There are unary and binary operations. Many of them have specializations and of course types could play into it. Then there is also the added difficulty of in-place assignments, which have other operation variants. """ from . import OperatorCodes from .ErrorCodes import getErrorExitBoolCode, getErrorExitCode, getReleaseCode from .Helpers import generateChildExpressionsCode def generateOperationBinaryCode(to_name, expression, emit, context): left_arg_name, right_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) inplace = expression.isInplaceSuspect() assert not inplace or not expression.getLeft().isCompileTimeConstant(), \ expression getOperationCode( to_name = to_name, operator = expression.getOperator(), arg_names = (left_arg_name, right_arg_name), in_place = inplace, emit = emit, context = context ) def generateOperationUnaryCode(to_name, expression, emit, context): arg_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) inplace = expression.isInplaceSuspect() assert not inplace or not expression.getOperand().isCompileTimeConstant(), \ expression getOperationCode( to_name = to_name, operator = expression.getOperator(), arg_names = (arg_name,), in_place = inplace, emit = emit, context = context ) def getOperationCode(to_name, operator, arg_names, in_place, emit, context): # This needs to have one case per operation of Python, and there are many # of these, # pylint: disable=R0912 prefix_args = () ref_count = 1 if operator == "Pow": helper = "POWER_OPERATION" elif operator == "IPow" and in_place: helper = "POWER_OPERATION_INPLACE" elif operator == "IPow": helper = "POWER_OPERATION2" elif operator == "Add": helper = "BINARY_OPERATION_ADD" elif operator == "IAdd" and in_place: helper = "BINARY_OPERATION_ADD_INPLACE" elif operator == "IMult" and in_place: helper = "BINARY_OPERATION_MUL_INPLACE" elif operator == "Sub": helper = "BINARY_OPERATION_SUB" elif operator == "Div": helper = "BINARY_OPERATION_DIV" elif operator == "Mult": helper = "BINARY_OPERATION_MUL" elif operator == "Mod": helper = "BINARY_OPERATION_REMAINDER" elif len(arg_names) == 2: helper = "BINARY_OPERATION" prefix_args = ( OperatorCodes.binary_operator_codes[ operator ], ) elif len(arg_names) == 1: impl_helper, ref_count = OperatorCodes.unary_operator_codes[ operator ] helper = "UNARY_OPERATION" prefix_args = ( impl_helper, ) else: assert False, operator # We must assume to write to a variable is "in_place" is active, not e.g. # a constant reference. That was asserted before calling us. if in_place: res_name = context.getBoolResName() # We may have not specialized this one yet, so lets use generic in-place # code, or the helper specified. if helper == "BINARY_OPERATION": emit( "%s = BINARY_OPERATION_INPLACE( %s, &%s, %s );" % ( res_name, OperatorCodes.binary_operator_codes[ operator ], arg_names[0], arg_names[1], ) ) else: emit( "%s = %s( &%s, %s );" % ( res_name, helper, arg_names[0], arg_names[1], ) ) ref_count = 0 emit( "%s = %s;" % ( to_name, arg_names[0] ) ) for arg_name in arg_names: getReleaseCode( arg_name, emit, context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) if ref_count: context.addCleanupTempName(to_name) else: emit( "%s = %s( %s );" % ( to_name, helper, ", ".join(prefix_args + arg_names) ) ) for arg_name in arg_names: getReleaseCode( arg_name, emit, context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) if ref_count: context.addCleanupTempName(to_name) Nuitka-0.5.18.1/nuitka/codegen/PrintCodes.py0000644000175000017500000000657012651071040020760 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Print related codes. This is broken down to to level on printing one individual item, and a new line potentially. The heavy lifting for 'softspace', etc. is happening in the C helper functions. """ from .ErrorCodes import getErrorExitBoolCode, getReleaseCode, getReleaseCodes from .Helpers import generateExpressionCode def generatePrintValueCode(statement, emit, context): destination = statement.getDestination() value = statement.getValue() if destination is not None: dest_name = context.allocateTempName("print_dest", unique = True) generateExpressionCode( expression = destination, to_name = dest_name, emit = emit, context = context ) else: dest_name = None value_name = context.allocateTempName("print_value", unique = True) generateExpressionCode( expression = value, to_name = value_name, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) if dest_name is not None: print_code = "PRINT_ITEM_TO( %s, %s ) == false" % ( dest_name, value_name ) else: print_code = "PRINT_ITEM( %s ) == false" % ( value_name, ) getErrorExitBoolCode( condition = print_code, emit = emit, context = context ) getReleaseCodes( release_names = (dest_name, value_name), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generatePrintNewlineCode(statement, emit, context): destination = statement.getDestination() if destination is not None: dest_name = context.allocateTempName("print_dest", unique = True) generateExpressionCode( expression = destination, to_name = dest_name, emit = emit, context = context ) else: dest_name = None old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) if dest_name is not None: print_code = "PRINT_NEW_LINE_TO( %s ) == false" % ( dest_name, ) else: print_code = "PRINT_NEW_LINE() == false" getErrorExitBoolCode( condition = print_code, emit = emit, context = context ) getReleaseCode( release_name = dest_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) Nuitka-0.5.18.1/nuitka/codegen/BlobCodes.py0000644000175000017500000000303512651071040020533 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Blob codes for storing binary data semi-efficiently. This module offers means to store and encode binary blobs in C++ semi efficiently. The "StreamData" class is used in two places, for constants and for freezing of bytecode. """ class StreamData: def __init__(self): self.stream_data = bytes() def getStreamDataCode(self, value, fixed_size = False): offset = self.stream_data.find(value) if offset == -1: offset = len(self.stream_data) self.stream_data += value if fixed_size: return "&constant_bin[ %d ]" % offset else: return "&constant_bin[ %d ], %d" % ( offset, len(value) ) def getBytes(self): return self.stream_data Nuitka-0.5.18.1/nuitka/codegen/LoopCodes.py0000644000175000017500000000633612651071040020575 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Loop codes. Code generation for loops, breaking them, or continuing them. In Nuitka, there are no for-loops or while-loops at this point. They have been re-formulated in a simpler loop without a condition, and statements there-in that break under certain conditions. See Developer Manual for how the CPython loops are mapped to these nodes. """ from .ErrorCodes import getErrorExitBoolCode from .ExceptionCodes import getExceptionUnpublishedReleaseCode from .LabelCodes import getGotoCode, getLabelCode def generateLoopBreakCode(statement, emit, context): # Functions used for generation all accept statement, but this one does # not use it. pylint: disable=W0613 getExceptionUnpublishedReleaseCode(emit, context) break_target = context.getLoopBreakTarget() getGotoCode(break_target, emit) def generateLoopContinueCode(statement, emit, context): # Functions used for generation all accept statement, but this one does # not use it. pylint: disable=W0613 getExceptionUnpublishedReleaseCode(emit, context) continue_target = context.getLoopContinueTarget() getGotoCode(continue_target, emit) def generateLoopCode(statement, emit, context): loop_start_label = context.allocateLabel("loop_start") if not statement.isStatementAborting(): loop_end_label = context.allocateLabel("loop_end") else: loop_end_label = None getLabelCode(loop_start_label, emit) old_loop_break = context.setLoopBreakTarget(loop_end_label) old_loop_continue = context.setLoopContinueTarget(loop_start_label) # TODO: Should come from registry instead. from .CodeGeneration import generateStatementSequenceCode generateStatementSequenceCode( statement_sequence = statement.getLoopBody(), allow_none = True, emit = emit, context = context ) context.setLoopBreakTarget(old_loop_break) context.setLoopContinueTarget(old_loop_continue) # Note: We are using the wrong line here, but it's an exception, it's unclear what line it would be anyway. old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) getErrorExitBoolCode( condition = "CONSIDER_THREADING() == false", emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) getGotoCode(loop_start_label, emit) if loop_end_label is not None: getLabelCode(loop_end_label, emit) Nuitka-0.5.18.1/nuitka/codegen/LoaderCodes.py0000644000175000017500000000550612651071040021070 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with module loaders. This is for generating the look-up table for the modules included in a binary or distribution folder. """ from .Indentation import indented from .templates.CodeTemplatesLoader import ( template_metapath_loader_body, template_metapath_loader_compiled_module_entry, template_metapath_loader_compiled_package_entry, template_metapath_loader_shlib_module_entry ) def getModuleMetapathLoaderEntryCode(module_name, module_identifier, is_shlib, is_package): if is_shlib: assert module_name != "__main__" assert not is_package return template_metapath_loader_shlib_module_entry % { "module_name" : module_name } elif is_package: return template_metapath_loader_compiled_package_entry % { "module_name" : module_name, "module_identifier" : module_identifier, } else: return template_metapath_loader_compiled_module_entry % { "module_name" : module_name, "module_identifier" : module_identifier, } def getMetapathLoaderBodyCode(other_modules): metapath_loader_inittab = [] metapath_module_decls = [] for other_module in other_modules: metapath_loader_inittab.append( getModuleMetapathLoaderEntryCode( module_name = other_module.getFullName(), module_identifier = other_module.getCodeName(), is_shlib = other_module.isPythonShlibModule(), is_package = other_module.isCompiledPythonPackage() ) ) if not other_module.isPythonShlibModule(): metapath_module_decls.append( "MOD_INIT_DECL( %s );" % other_module.getCodeName() ) return template_metapath_loader_body % { "use_loader" : 1 if other_modules else 0, "metapath_module_decls" : indented(metapath_module_decls, 0), "metapath_loader_inittab" : indented(metapath_loader_inittab) } Nuitka-0.5.18.1/nuitka/codegen/FrameCodes.py0000644000175000017500000004030212651071040020705 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Frame codes This is about frame stacks and their management. There are different kinds of frames for different uses. """ from nuitka.PythonVersions import python_version from . import Contexts, Emission from .ExceptionCodes import getTracebackMakingIdentifier from .GlobalsLocalsCodes import getLoadLocalsCode from .Indentation import indented from .ModuleCodes import getModuleAccessCode from .templates.CodeTemplatesFrames import ( template_frame_guard_cache_decl, template_frame_guard_coroutine, template_frame_guard_frame_decl, template_frame_guard_full_block, template_frame_guard_full_exception_handler, template_frame_guard_full_return_handler, template_frame_guard_generator, template_frame_guard_generator_exception_handler, template_frame_guard_generator_return_handler, template_frame_guard_once ) def generateStatementsFrameCode(statement_sequence, emit, context): # This is a wrapper that provides also handling of frames, which got a # lot of variants and details, therefore lots of branches. # pylint: disable=R0912,R0915 context = Contexts.PythonStatementCContext(context) provider = statement_sequence.getParentVariableProvider() guard_mode = statement_sequence.getGuardMode() parent_exception_exit = context.getExceptionEscape() # Allow stacking of frame handles. old_frame_handle = context.getFrameHandle() if guard_mode != "pass_through": if provider.isExpressionGeneratorObjectBody(): context.setFrameHandle("generator->m_frame") elif provider.isExpressionCoroutineObjectBody(): context.setFrameHandle("coroutine->m_frame") elif provider.isCompiledPythonModule(): context.setFrameHandle("frame_module") else: context.setFrameHandle("frame_function") context.setExceptionEscape( context.allocateLabel("frame_exception_exit") ) else: context.setFrameHandle("PyThreadState_GET()->frame") needs_preserve = statement_sequence.needsFrameExceptionPreserving() if statement_sequence.mayReturn() and guard_mode != "pass_through": parent_return_exit = context.getReturnTarget() context.setReturnTarget( context.allocateLabel("frame_return_exit") ) else: parent_return_exit = None # Now generate the statements code into a local buffer, to we can wrap # the frame stuff around it. local_emit = Emission.SourceCodeCollector() from .CodeGeneration import _generateStatementSequenceCode _generateStatementSequenceCode( statement_sequence = statement_sequence, emit = local_emit, context = context ) if statement_sequence.mayRaiseException(BaseException) or \ guard_mode == "generator": frame_exception_exit = context.getExceptionEscape() else: frame_exception_exit = None if parent_return_exit is not None: frame_return_exit = context.getReturnTarget() else: frame_return_exit = None if guard_mode == "generator": assert provider.isExpressionGeneratorObjectBody() or provider.isExpressionCoroutineObjectBody() # TODO: This case should care about "needs_preserve", as for # Python3 it is actually not a stub of empty code. getFrameGuardLightCode( code_identifier = statement_sequence.getCodeObjectHandle( context = context ), codes = local_emit.codes, parent_exception_exit = parent_exception_exit, frame_exception_exit = frame_exception_exit, parent_return_exit = parent_return_exit, frame_return_exit = frame_return_exit, provider = provider, emit = emit, context = context ) elif guard_mode == "pass_through": # This case does not care about "needs_preserve", as for that kind # of frame, it is an empty code stub anyway. local_emit.emitTo(emit) elif guard_mode == "full": assert provider.isExpressionFunctionBody() or \ provider.isExpressionClassBody() getFrameGuardHeavyCode( frame_identifier = context.getFrameHandle(), code_identifier = statement_sequence.getCodeObjectHandle( context ), parent_exception_exit = parent_exception_exit, parent_return_exit = parent_return_exit, frame_exception_exit = frame_exception_exit, frame_return_exit = frame_return_exit, codes = local_emit.codes, needs_preserve = needs_preserve, provider = provider, emit = emit, context = context ) elif guard_mode == "once": getFrameGuardOnceCode( frame_identifier = context.getFrameHandle(), code_identifier = statement_sequence.getCodeObjectHandle( context = context ), parent_exception_exit = parent_exception_exit, parent_return_exit = parent_return_exit, frame_exception_exit = frame_exception_exit, frame_return_exit = frame_return_exit, codes = local_emit.codes, needs_preserve = needs_preserve, provider = provider, emit = emit, context = context ) else: assert False, guard_mode context.setExceptionEscape(parent_exception_exit) if frame_return_exit is not None: context.setReturnTarget(parent_return_exit) context.setFrameHandle(old_frame_handle) # Complain if any temporary was not dealt with yet. assert not context.getCleanupTempnames(), \ context.getCleanupTempnames() def getFrameGuardHeavyCode(frame_identifier, code_identifier, codes, needs_preserve, parent_exception_exit, parent_return_exit, frame_exception_exit, frame_return_exit, provider, emit, context): # We really need this many parameters here. pylint: disable=R0913 no_exception_exit = context.allocateLabel("frame_no_exception") context.addFrameDeclaration( template_frame_guard_cache_decl % { "frame_identifier" : frame_identifier, } ) context.addFrameDeclaration( template_frame_guard_frame_decl % { "frame_identifier" : frame_identifier, } ) emit( template_frame_guard_full_block % { "frame_identifier" : frame_identifier, "code_identifier" : code_identifier, "codes" : indented(codes, 0), "module_identifier" : getModuleAccessCode(context = context), "no_exception_exit" : no_exception_exit, "needs_preserve" : 1 if needs_preserve else 0, } ) if frame_return_exit is not None: emit( template_frame_guard_full_return_handler % { "frame_identifier" : frame_identifier, "return_exit" : parent_return_exit, "frame_return_exit" : frame_return_exit, "needs_preserve" : 1 if needs_preserve else 0, } ) if frame_exception_exit is not None: frame_locals_name, locals_code = getFrameLocalsUpdateCode( provider = provider, context = context ) emit( template_frame_guard_full_exception_handler % { "frame_identifier" : frame_identifier, "frame_locals_name" : frame_locals_name, "store_frame_locals" : indented( locals_code, 2, vert_block = True ), "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = "exception_lineno" ), "parent_exception_exit" : parent_exception_exit, "frame_exception_exit" : frame_exception_exit, "needs_preserve" : 1 if needs_preserve else 0, } ) emit("%s:;\n" % no_exception_exit) def getFrameGuardOnceCode(frame_identifier, code_identifier, codes, parent_exception_exit, parent_return_exit, frame_exception_exit, frame_return_exit, needs_preserve, provider, emit, context): # We really need this many parameters here. pylint: disable=R0913 # Used for modules only currently, but that ought to change. assert parent_return_exit is None and frame_return_exit is None if not provider.isCompiledPythonModule(): _frame_locals_name, locals_code = getFrameLocalsUpdateCode( provider = provider, context = context ) # TODO: Not using locals, which is only OK for modules assert False, locals_code context.addFrameDeclaration( template_frame_guard_frame_decl % { "frame_identifier" : frame_identifier, } ) emit( template_frame_guard_once % { "frame_identifier" : frame_identifier, "code_identifier" : code_identifier, "codes" : indented(codes, 0), "module_identifier" : getModuleAccessCode(context = context), "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = "exception_lineno" ), "parent_exception_exit" : parent_exception_exit, "frame_exception_exit" : frame_exception_exit, "no_exception_exit" : context.allocateLabel( "frame_no_exception" ), "needs_preserve" : 1 if needs_preserve else 0 } ) def getFrameGuardLightCode(code_identifier, codes, parent_exception_exit, parent_return_exit, frame_exception_exit, frame_return_exit, provider, emit, context): context.markAsNeedsExceptionVariables() assert frame_exception_exit is not None if context.getOwner().isExpressionGeneratorObjectBody(): kind = "generator" template = template_frame_guard_generator else: kind = "coroutine" template = template_frame_guard_coroutine context.addFrameDeclaration( template_frame_guard_cache_decl % { "frame_identifier" : "frame_" + kind, } ) no_exception_exit = context.allocateLabel("frame_no_exception") emit( template % { "frame_cache_identifier" : "cache_frame_" + kind, "code_identifier" : code_identifier, "codes" : indented(codes, 0), "module_identifier" : getModuleAccessCode(context = context), "no_exception_exit" : no_exception_exit, } ) if frame_return_exit is not None: emit( template_frame_guard_generator_return_handler % { "frame_identifier" : "%s->m_frame" % kind, "return_exit" : parent_return_exit, "frame_return_exit" : frame_return_exit, } ) frame_locals_name, locals_code = getFrameLocalsUpdateCode( provider = provider, context = context ) if kind == "generator": template = template_frame_guard_generator_exception_handler else: template = template_frame_guard_generator_exception_handler # TODO: Don't create locals for StopIteration or GeneratorExit, that is just # wasteful. emit( template % { "frame_identifier" : "%s->m_frame" % kind, "frame_locals_name" : frame_locals_name, "store_frame_locals" : indented( locals_code, 2, vert_block = True ), "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = "exception_lineno" ), "frame_exception_exit" : frame_exception_exit, "parent_exception_exit" : parent_exception_exit, "no_exception_exit" : no_exception_exit, } ) def getFrameLocalsUpdateCode(provider, context): locals_codes = Emission.SourceCodeCollector() context.setCurrentSourceCodeReference( provider.getSourceReference() ) frame_locals_name = context.allocateTempName( "frame_locals", unique = True ) getLoadLocalsCode( to_name = frame_locals_name, provider = provider, mode = "updated", emit = locals_codes.emit, context = context ) if context.needsCleanup(frame_locals_name): context.removeCleanupTempName(frame_locals_name) return frame_locals_name, locals_codes.codes def generateFramePreserveExceptionCode(statement, emit, context): emit("// Preserve existing published exception.") if python_version < 300: emit( "PRESERVE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: preserver_id = statement.getPreserverId() if preserver_id == 0 and python_version < 300: emit( "PRESERVE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: context.addExceptionPreserverVariables(preserver_id) emit( """\ exception_preserved_type_%(preserver_id)d = PyThreadState_GET()->exc_type; Py_XINCREF( exception_preserved_type_%(preserver_id)d ); exception_preserved_value_%(preserver_id)d = PyThreadState_GET()->exc_value; Py_XINCREF( exception_preserved_value_%(preserver_id)d ); exception_preserved_tb_%(preserver_id)d = (PyTracebackObject *)PyThreadState_GET()->exc_traceback; Py_XINCREF( exception_preserved_tb_%(preserver_id)d ); """ % { "preserver_id" : preserver_id, } ) def generateFrameRestoreExceptionCode(statement, emit, context): emit("// Restore previous exception.") if python_version < 300: emit( "RESTORE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: preserver_id = statement.getPreserverId() if preserver_id == 0 and python_version < 300: emit( "RESTORE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: # pylint: disable=C0301 emit( """\ SET_CURRENT_EXCEPTION( exception_preserved_type_%(preserver_id)d, exception_preserved_value_%(preserver_id)d, exception_preserved_tb_%(preserver_id)d );""" % { "preserver_id" : preserver_id, } ) Nuitka-0.5.18.1/nuitka/codegen/CallCodes.py0000644000175000017500000003000212651072046020531 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for calls. The different kinds of calls get dedicated code. Most notable, calls with only positional arguments, are attempted through helpers that might be able to execute them without creating the argument dictionary at all. """ from .ConstantCodes import getConstantAccess from .ErrorCodes import getErrorExitCode, getReleaseCode, getReleaseCodes from .Helpers import generateChildExpressionCode from .LineNumberCodes import emitLineNumberUpdateCode from .templates.CodeTemplatesCalls import ( template_call_function_with_args_decl, template_call_function_with_args_impl ) from .templates.CodeTemplatesModules import ( template_header_guard, template_helper_impl_decl ) def generateCallCode(to_name, expression, emit, context): # There is a whole lot of different cases, for each of which, we create # optimized code, constant, with and without positional or keyword arguments # each, so there is lots of branches here. pylint: disable=R0912 called_name = generateChildExpressionCode( expression = expression.getCalled(), emit = emit, context = context ) call_args = expression.getCallArgs() call_kw = expression.getCallKw() if call_kw is None or \ (call_kw.isExpressionConstantRef() and call_kw.getConstant() == {}): if call_args is None or call_args.isExpressionConstantRef(): context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) if call_args is not None: call_args_value = call_args.getConstant() else: call_args_value = () assert type(call_args_value) is tuple if call_args is not None and call_args.isMutable(): call_arg_names = [] for call_arg_element in call_args_value: call_arg_name = context.allocateTempName("call_arg_element") getConstantAccess( to_name = call_arg_name, constant = call_arg_element, emit = emit, context = context, ) call_arg_names.append(call_arg_name) getCallCodePosArgsQuick( to_name = to_name, called_name = called_name, arg_names = call_arg_names, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) elif call_args_value: getCallCodeFromTuple( to_name = to_name, called_name = called_name, arg_tuple = context.getConstantCode( constant = call_args_value ), arg_size = len(call_args_value), needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: getCallCodeNoArgs( to_name = to_name, called_name = called_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) elif call_args.isExpressionMakeTuple(): call_arg_names = [] for call_arg_element in call_args.getElements(): call_arg_name = generateChildExpressionCode( child_name = call_args.getChildName() + "_element", expression = call_arg_element, emit = emit, context = context, ) call_arg_names.append(call_arg_name) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getCallCodePosArgsQuick( to_name = to_name, called_name = called_name, arg_names = call_arg_names, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: args_name = generateChildExpressionCode( expression = call_args, emit = emit, context = context ) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getCallCodePosArgs( to_name = to_name, called_name = called_name, args_name = args_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: if call_args is None or \ (call_args.isExpressionConstantRef() and \ call_args.getConstant() == ()): call_kw_name = generateChildExpressionCode( expression = call_kw, emit = emit, context = context ) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getCallCodeKeywordArgs( to_name = to_name, called_name = called_name, call_kw_name = call_kw_name, emit = emit, context = context ) else: call_args_name = generateChildExpressionCode( expression = call_args, emit = emit, context = context ) call_kw_name = generateChildExpressionCode( expression = call_kw, emit = emit, context = context ) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getCallCodePosKeywordArgs( to_name = to_name, called_name = called_name, call_args_name = call_args_name, call_kw_name = call_kw_name, emit = emit, context = context ) def getCallCodeNoArgs(to_name, called_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION_NO_ARGS( %s );" % ( to_name, called_name ) ) getReleaseCode( release_name = called_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) # Outside helper code relies on some quick call to be present. quick_calls_used = set([1, 2, 3]) def getCallCodePosArgsQuick(to_name, called_name, arg_names, needs_check, emit, context): arg_size = len(arg_names) quick_calls_used.add(arg_size) # For 0 arguments, NOARGS is supposed to be used. assert arg_size > 0 emitLineNumberUpdateCode(emit, context) emit( """\ { PyObject *call_args[] = { %s }; %s = CALL_FUNCTION_WITH_ARGS%d( %s, call_args ); } """ % ( ", ".join(arg_names), to_name, arg_size, called_name, ) ) getReleaseCodes( release_names = [called_name] + arg_names, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodeFromTuple(to_name, called_name, arg_tuple, arg_size, needs_check, emit, context): quick_calls_used.add(arg_size) # For 0 arguments, NOARGS is supposed to be used. assert arg_size > 0 emitLineNumberUpdateCode(emit, context) emit( """\ %s = CALL_FUNCTION_WITH_ARGS%d( %s, &PyTuple_GET_ITEM( %s, 0 ) ); """ % ( to_name, arg_size, called_name, arg_tuple, ) ) getReleaseCode( release_name = called_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodePosArgs(to_name, called_name, args_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION_WITH_POSARGS( %s, %s );" % ( to_name, called_name, args_name ) ) getReleaseCodes( release_names = (called_name, args_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodeKeywordArgs(to_name, called_name, call_kw_name, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION_WITH_KEYARGS( %s, %s );" % ( to_name, called_name, call_kw_name ) ) getReleaseCodes( release_names = (called_name, call_kw_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodePosKeywordArgs(to_name, called_name, call_args_name, call_kw_name, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION( %s, %s, %s );" % ( to_name, called_name, call_args_name, call_kw_name ) ) getReleaseCodes( release_names = (called_name, call_args_name, call_kw_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallsDecls(): result = [] for quick_call_used in sorted(quick_calls_used): result.append( template_call_function_with_args_decl % { "args_count" : quick_call_used } ) return template_header_guard % { "header_guard_name" : "__NUITKA_CALLS_H__", "header_body" : '\n'.join(result) } def getCallsCode(): result = [] result.append( template_helper_impl_decl % {} ) for quick_call_used in sorted(quick_calls_used): result.append( template_call_function_with_args_impl % { "args_count" : quick_call_used } ) return '\n'.join(result) Nuitka-0.5.18.1/nuitka/codegen/ExceptionCodes.py0000644000175000017500000001553012651071040021616 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Exception handling. """ from nuitka.PythonVersions import python_version from .Helpers import generateExpressionCode from .templates. \ CodeTemplatesExceptions import template_publish_exception_to_handler def getExceptionIdentifier(exception_type): assert "PyExc" not in exception_type, exception_type if exception_type == "NotImplemented": return "Py_NotImplemented" return "PyExc_%s" % exception_type def generateExceptionRefCode(to_name, expression, emit, context): exception_type = expression.getExceptionName() emit( "%s = %s;" % ( to_name, getExceptionIdentifier(exception_type), ) ) # Lets have context in the API for consistency with everything else. assert context def getTracebackMakingIdentifier(context, lineno_name): frame_handle = context.getFrameHandle() assert frame_handle is not None return "MAKE_TRACEBACK( %s, %s )" % ( frame_handle, lineno_name ) def generateExceptionCaughtTypeCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=W0613 keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is None: emit( "%s = PyThreadState_GET()->exc_type;" % ( to_name, ) ) else: emit( "%s = %s;" % ( to_name, keeper_variables[0] ) ) def generateExceptionCaughtValueCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=W0613 keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[1] is None: emit( "%s = PyThreadState_GET()->exc_value;" % ( to_name, ) ) else: if python_version >= 270: emit( "%s = %s;" % ( to_name, keeper_variables[1] ) ) else: emit( "%s = %s ? %s : Py_None;" % ( to_name, keeper_variables[1], keeper_variables[1] ) ) def generateExceptionCaughtTracebackCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=W0613 keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[2] is None: emit( "%s = PyThreadState_GET()->exc_traceback;" % ( to_name, ) ) else: emit( """\ if ( %(keeper_tb)s != NULL ) { %(to_name)s = (PyObject *)%(keeper_tb)s; Py_INCREF( %(to_name)s ); } else { %(to_name)s = (PyObject *)%(tb_making)s; } """ % { "to_name" : to_name, "keeper_tb" : keeper_variables[2], "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = keeper_variables[3] ) } ) context.addCleanupTempName(to_name) def getExceptionUnpublishedReleaseCode(emit, context): keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is not None: emit("Py_DECREF( %s );" % keeper_variables[0]) emit("Py_XDECREF( %s );" % keeper_variables[1]) emit("Py_XDECREF( %s );" % keeper_variables[2]) def generateExceptionPublishCode(statement, emit, context): # This statement has no attributes really, pylint: disable=W0613 # TODO: Should this be necessary, something else would have required # them already, or it's wrong. context.markAsNeedsExceptionVariables() # Current variables cannot be used anymore now. keeper_type, keeper_value, keeper_tb, keeper_lineno = context.setExceptionKeeperVariables( (None, None, None, None) ) emit( template_publish_exception_to_handler % { "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = keeper_lineno ), "keeper_tb" : keeper_tb, "keeper_lineno" : keeper_lineno, "frame_identifier" : context.getFrameHandle() } ) emit( "NORMALIZE_EXCEPTION( &%s, &%s, &%s );" % ( keeper_type, keeper_value, keeper_tb ) ) if python_version >= 300: emit( "PyException_SetTraceback( %s, (PyObject *)%s );" % ( keeper_value, keeper_tb ) ) emit( "PUBLISH_EXCEPTION( &%s, &%s, &%s );" % ( keeper_type, keeper_value, keeper_tb ) ) def generateBuiltinMakeExceptionCode(to_name, expression, emit, context): from .CallCodes import getCallCodeNoArgs, getCallCodePosArgsQuick exception_arg_names = [] for exception_arg in expression.getArgs(): exception_arg_name = context.allocateTempName("make_exception_arg") generateExpressionCode( to_name = exception_arg_name, expression = exception_arg, emit = emit, context = context ) exception_arg_names.append(exception_arg_name) exception_type = expression.getExceptionName() if exception_arg_names: getCallCodePosArgsQuick( to_name = to_name, called_name = getExceptionIdentifier(exception_type), arg_names = exception_arg_names, needs_check = False, emit = emit, context = context ) else: getCallCodeNoArgs( to_name = to_name, called_name = getExceptionIdentifier(exception_type), needs_check = False, emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/Pickling.py0000644000175000017500000000456212651071040020445 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module to hide the complexity of using pickle. It should be simple, but it is not yet. Not all the pickle modules are well behaved. """ import pickletools from logging import warning from nuitka import Constants from nuitka.__past__ import unicode # pylint: disable=W0622 from nuitka.PythonVersions import python_version # Work around for CPython 3.x removal of "cpickle". try: import cPickle as cpickle except ImportError: import pickle as cpickle if python_version >= 300: # Python3: The protocol 3 adds support for bytes type. pickle_protocol = 3 else: pickle_protocol = 2 def getStreamedConstant(constant_value): # Note: The marshal module cannot persist all unicode strings and # therefore cannot be used. Instead we use pickle. try: saved = cpickle.dumps( constant_value, protocol = 0 if type(constant_value) is unicode else pickle_protocol ) except TypeError: warning("Problem with persisting constant '%r'." % constant_value) raise saved = pickletools.optimize(saved) # Check that the constant is restored correctly. try: restored = cpickle.loads( saved ) except: warning("Problem with persisting constant '%r'." % constant_value) raise if not Constants.compareConstants(restored, constant_value): raise AssertionError( "Streaming of constant changed value", constant_value, "!=", restored, "types:", type(constant_value), type(restored) ) return saved Nuitka-0.5.18.1/nuitka/codegen/ClassCodes.py0000644000175000017500000000736112651071040020730 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Codes for classes. Most the class specific stuff is solved in re-formulation. Only the selection of the metaclass remains as specific. """ from nuitka.PythonVersions import python_version from .ErrorCodes import getErrorExitCode, getReleaseCode, getReleaseCodes from .Helpers import generateChildExpressionsCode from .PythonAPICodes import generateCAPIObjectCode0 def _getMetaclassVariableCode(context): assert python_version < 300 return "GET_STRING_DICT_VALUE( moduledict_%s, (Nuitka_StringObject *)%s )" % ( context.getModuleCodeName(), context.getConstantCode( constant = "__metaclass__" ) ) def generateSelectMetaclassCode(to_name, expression, emit, context): metaclass_name, bases_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) if python_version < 300: assert metaclass_name is None args = [ bases_name, _getMetaclassVariableCode(context = context) ] else: args = [ metaclass_name, bases_name ] emit( "%s = SELECT_METACLASS( %s );" % ( to_name, ", ".join(args) ) ) # Can only fail with Python3. if python_version >= 300: getErrorExitCode( check_name = to_name, emit = emit, context = context ) getReleaseCodes( release_names = args, emit = emit, context = context ) else: getReleaseCode( release_name = bases_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinSuperCode(to_name, expression, emit, context): type_name, object_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = BUILTIN_SUPER( %s, %s );" % ( to_name, type_name if type_name is not None else "NULL", object_name if object_name is not None else "NULL" ) ) getReleaseCodes( release_names = (type_name, object_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinIsinstanceCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "BUILTIN_ISINSTANCE", arg_desc = ( ("isinstance_inst", expression.getInstance()), ("isinstance_cls", expression.getCls()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/ListCodes.py0000644000175000017500000001150412651072046020577 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for lists. Right now only the creation is done here. But more should be added later on. """ from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .Helpers import generateChildExpressionsCode, generateExpressionCode from .PythonAPICodes import generateCAPIObjectCode def generateListCreationCode(to_name, expression, emit, context): elements = expression.getElements() emit( "%s = PyList_New( %d );" % ( to_name, len(elements) ) ) context.addCleanupTempName(to_name) element_name = context.allocateTempName("list_element") for count, element in enumerate(elements): generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) if not context.needsCleanup(element_name): emit("Py_INCREF( %s );" % element_name) else: context.removeCleanupTempName(element_name) emit( "PyList_SET_ITEM( %s, %d, %s );" % ( to_name, count, element_name ) ) def generateListOperationAppendCode(to_name, expression, emit, context): res_name = context.getIntResName() list_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PyList_Check( %s ) );" % list_arg_name) emit( "%s = PyList_Append( %s, %s );" % ( res_name, list_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (list_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) # Only assign if necessary. if context.isUsed(to_name): emit( "%s = Py_None;" % to_name ) else: context.forgetTempName(to_name) def generateListOperationExtendCode(to_name, expression, emit, context): list_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PyList_Check( %s ) );" % list_arg_name) emit( "%s = _PyList_Extend( (PyListObject *)%s, %s );" % ( to_name, list_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (list_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateListOperationPopCode(to_name, expression, emit, context): list_arg_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # TODO: Have a dedicated helper instead, this could be more efficient. emit("assert( PyList_Check( %s ) );" % list_arg_name) emit( '%s = PyObject_CallMethod( %s, (char *)"pop", NULL );' % ( to_name, list_arg_name ) ) getReleaseCode( release_name = list_arg_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinListCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PySequence_List", arg_desc = ( ("list_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/AttributeCodes.py0000644000175000017500000002647512651072046021644 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Attribute related codes. Attribute lookup, setting. """ from nuitka import Options from .ConstantCodes import getConstantCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .Helpers import generateChildExpressionsCode, generateExpressionCode from .LabelCodes import getBranchingCode from .PythonAPICodes import generateCAPIObjectCode, generateCAPIObjectCode0 def generateAssignmentAttributeCode(statement, emit, context): lookup_source = statement.getLookupSource() attribute_name = statement.getAttributeName() value = statement.getAssignSource() value_name = context.allocateTempName("assattr_name") generateExpressionCode( to_name = value_name, expression = value, emit = emit, context = context ) target_name = context.allocateTempName("assattr_target") generateExpressionCode( to_name = target_name, expression = lookup_source, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) if attribute_name == "__dict__": getAttributeAssignmentDictSlotCode( target_name = target_name, value_name = value_name, emit = emit, context = context ) elif attribute_name == "__class__": getAttributeAssignmentClassSlotCode( target_name = target_name, value_name = value_name, emit = emit, context = context ) else: getAttributeAssignmentCode( target_name = target_name, value_name = value_name, attribute_name = getConstantCode( context = context, constant = attribute_name ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateDelAttributeCode(statement, emit, context): target_name = context.allocateTempName("attrdel_target") generateExpressionCode( to_name = target_name, expression = statement.getLookupSource(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( statement.getLookupSource().getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getAttributeDelCode( target_name = target_name, attribute_name = getConstantCode( context = context, constant = statement.getAttributeName() ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateAttributeLookupCode(to_name, expression, emit, context): source_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) attribute_name = expression.getAttributeName() getAttributeLookupCode( to_name = to_name, source_name = source_name, attribute_name = attribute_name, needs_check = expression.getLookupSource().mayRaiseExceptionAttributeLookup( exception_type = BaseException, attribute_name = attribute_name ), emit = emit, context = context ) def getAttributeLookupCode(to_name, source_name, attribute_name, needs_check, emit, context): if attribute_name == "__dict__": emit( "%s = LOOKUP_ATTRIBUTE_DICT_SLOT( %s );" % ( to_name, source_name ) ) elif attribute_name == "__class__": emit( "%s = LOOKUP_ATTRIBUTE_CLASS_SLOT( %s );" % ( to_name, source_name ) ) else: emit( "%s = LOOKUP_ATTRIBUTE( %s, %s );" % ( to_name, source_name, getConstantCode( context = context, constant = attribute_name ) ) ) getReleaseCode( release_name = source_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getAttributeCheckBoolCode(source_name, attr_name, needs_check, emit, context): res_name = context.getIntResName() emit( "%s = PyObject_HasAttr( %s, %s );" % ( res_name, source_name, attr_name ) ) getReleaseCodes( release_names = (source_name, attr_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, needs_check = needs_check, emit = emit, context = context ) getBranchingCode("%s == 1" % res_name, emit, context) def getAttributeAssignmentCode(target_name, attribute_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_ATTRIBUTE( %s, %s, %s );" % ( res_name, target_name, attribute_name, value_name ) ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) getReleaseCodes( release_names = (value_name, target_name, attribute_name), emit = emit, context = context ) def getAttributeAssignmentDictSlotCode(target_name, value_name, emit, context): """ Code for special case target.__dict__ = value """ res_name = context.getBoolResName() emit( "%s = SET_ATTRIBUTE_DICT_SLOT( %s, %s );" % ( res_name, target_name, value_name ) ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) getReleaseCodes( release_names = (value_name, target_name), emit = emit, context = context ) def getAttributeAssignmentClassSlotCode(target_name, value_name, emit, context): """ Get code for special case target.__class__ = value """ res_name = context.getBoolResName() emit( "%s = SET_ATTRIBUTE_CLASS_SLOT( %s, %s );" % ( res_name, target_name, value_name ) ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) getReleaseCodes( release_names = (value_name, target_name), emit = emit, context = context ) def getAttributeDelCode(target_name, attribute_name, emit, context): res_name = context.getIntResName() emit( "%s = PyObject_DelAttr( %s, %s );" % ( res_name, target_name, attribute_name ) ) getReleaseCodes( release_names = (target_name, attribute_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) def generateAttributeLookupSpecialCode(to_name, expression, emit, context): source_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) attribute_name = expression.getAttributeName() getAttributeLookupSpecialCode( to_name = to_name, source_name = source_name, attr_name = getConstantCode( context = context, constant = attribute_name ), needs_check = expression.getLookupSource().mayRaiseExceptionAttributeLookupSpecial( exception_type = BaseException, attribute_name = attribute_name ), emit = emit, context = context ) def getAttributeLookupSpecialCode(to_name, source_name, attr_name, needs_check, emit, context): emit( "%s = LOOKUP_SPECIAL( %s, %s );" % ( to_name, source_name, attr_name, ) ) getReleaseCodes( release_names = (source_name, attr_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) def generateBuiltinHasattrCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "BUILTIN_HASATTR", arg_desc = ( ("hasattr_value", expression.getLookupSource()), ("hasattr_attr", expression.getAttribute()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinGetattrCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_GETATTR", arg_desc = ( ("getattr_target", expression.getLookupSource()), ("getattr_attr", expression.getAttribute()), ("getattr_default", expression.getDefault()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinSetattrCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "BUILTIN_SETATTR", arg_desc = ( ("setattr_target", expression.getLookupSource()), ("setattr_attr", expression.getAttribute()), ("setattr_value", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, ) Nuitka-0.5.18.1/nuitka/codegen/templates/0000755000175000017500000000000012651072347020335 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesMain.py0000644000175000017500000002237512651071040024243 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for main program. """ template_main_program = """\ // The main program for C++. It needs to prepare the interpreter and then // calls the initialization code of the "__main__" module. #include "structseq.h" #if %(python_no_warnings)d extern PyObject *const_str_plain_ignore; #endif #ifdef _NUITKA_WINMAIN_ENTRY_POINT int __stdcall WinMain( HINSTANCE hInstance, HINSTANCE hPrevInstance, char* lpCmdLine, int nCmdShow ) { #if defined(__MINGW32__) && !defined(_W64) /* MINGW32 */ int argc = _argc; char** argv = _argv; #else /* MSVC, MINGW64 */ int argc = __argc; char** argv = __argv; #endif #else int main( int argc, char **argv ) { #endif #ifdef _NUITKA_TRACE puts("main(): Entered."); #endif #ifdef __FreeBSD__ // 754 requires that FP exceptions run in "no stop" mode by default, and // until C vendors implement C99's ways to control FP exceptions, Python // requires non-stop mode. Alas, some platforms enable FP exceptions by // default. Here we disable them. fp_except_t m; m = fpgetmask(); fpsetmask( m & ~FP_X_OFL ); #endif #ifdef _NUITKA_STANDALONE #ifdef _NUITKA_TRACE puts("main(): Prepare standalone environment."); #endif prepareStandaloneEnvironment(); #endif // Initialize CPython library environment. Py_DebugFlag = %(python_sysflag_debug)d; #if %(python_sysflag_py3k_warning)d Py_Py3kWarningFlag = %(python_sysflag_py3k_warning)d; #endif #if %(python_sysflag_division_warning)d Py_DivisionWarningFlag = #if %(python_sysflag_py3k_warning)d Py_Py3kWarningFlag || #endif %(python_sysflag_division_warning)d; #endif Py_InspectFlag = %(python_sysflag_inspect)d; Py_InteractiveFlag = %(python_sysflag_interactive)d; Py_OptimizeFlag = %(python_sysflag_optimize)d; Py_DontWriteBytecodeFlag = %(python_sysflag_dont_write_bytecode)d; Py_NoUserSiteDirectory = %(python_sysflag_no_user_site)d; Py_IgnoreEnvironmentFlag = %(python_sysflag_ignore_environment)d; #if %(python_sysflag_tabcheck)d Py_TabcheckFlag = %(python_sysflag_tabcheck)d; #endif Py_VerboseFlag = %(python_sysflag_verbose)d; #if %(python_sysflag_unicode)d Py_UnicodeFlag = %(python_sysflag_unicode)d; #endif Py_BytesWarningFlag = %(python_sysflag_bytes_warning)d; #if %(python_sysflag_hash_randomization)d Py_HashRandomizationFlag = %(python_sysflag_hash_randomization)d; #endif // We want to import the site module, but only after we finished our own // setup. The site module import will be the first thing, the main module // does. Py_NoSiteFlag = 1; // Initial command line handling only. #ifdef _NUITKA_TRACE puts("main(): Calling convert/setCommandLineParameters."); #endif #if PYTHON_VERSION >= 300 wchar_t **argv_unicode = convertCommandLineParameters( argc, argv ); #endif #if PYTHON_VERSION < 300 bool is_multiprocess_forking = setCommandLineParameters( argc, argv, true ); #else bool is_multiprocess_forking = setCommandLineParameters( argc, argv_unicode, true ); #endif // Initialize the embedded CPython interpreter. #ifdef _NUITKA_TRACE puts("main(): Calling Py_Initialize."); #endif Py_Initialize(); #ifdef _NUITKA_TRACE puts("main(): Returned from Py_Initialize."); #endif // Lie about it, believe it or not, there are "site" files, that check // against later imports, see below. Py_NoSiteFlag = %(python_sysflag_no_site)s; // Set the command line parameters for run time usage. #ifdef _NUITKA_TRACE puts("main(): Calling setCommandLineParameters."); #endif #if PYTHON_VERSION < 300 setCommandLineParameters( argc, argv, false ); #else setCommandLineParameters( argc, argv_unicode, false ); #endif #ifdef _NUITKA_STANDALONE #ifdef _NUITKA_TRACE puts("main(): Restore standalone environment."); #endif restoreStandaloneEnvironment(); #endif // Initialize the built-in module tricks used. #ifdef _NUITKA_TRACE puts("main(): Calling _initBuiltinModule()."); #endif _initBuiltinModule(); #ifdef _NUITKA_TRACE puts("main(): Returned from _initBuiltinModule."); #endif // Initialize the constant values used. #ifdef _NUITKA_TRACE puts("main(): Calling createGlobalConstants()."); #endif createGlobalConstants(); #ifdef _NUITKA_TRACE puts("main(): Calling _initBuiltinOriginalValues()."); #endif _initBuiltinOriginalValues(); // Revert the wrong "sys.flags" value, it's used by "site" on at least // Debian for Python 3.3, more uses may exist. #if %(python_sysflag_no_site)s == 0 #if PYTHON_VERSION >= 330 PyStructSequence_SetItem( PySys_GetObject( "flags" ), 6, const_int_0 ); #elif PYTHON_VERSION >= 320 PyStructSequence_SetItem( PySys_GetObject( "flags" ), 7, const_int_0 ); #elif PYTHON_VERSION >= 260 PyStructSequence_SET_ITEM( PySys_GetObject( (char *)"flags" ), 9, const_int_0 ); #endif #endif // Initialize the compiled types of Nuitka. PyType_Ready( &Nuitka_Generator_Type ); PyType_Ready( &Nuitka_Function_Type ); PyType_Ready( &Nuitka_Method_Type ); PyType_Ready( &Nuitka_Frame_Type ); #if PYTHON_VERSION >= 350 PyType_Ready( &Nuitka_Coroutine_Type ); PyType_Ready( &Nuitka_CoroutineWrapper_Type ); #endif #if PYTHON_VERSION < 300 _initSlotCompare(); #endif #if PYTHON_VERSION >= 270 _initSlotIternext(); #endif enhancePythonTypes(); // Set the sys.executable path to the original Python executable on Linux // or to python.exe on Windows. PySys_SetObject( (char *)"executable", %(sys_executable)s ); patchBuiltinModule(); patchTypeComparison(); // Allow to override the ticker value, to remove checks for threads in // CPython core from impact on benchmarks. char const *ticker_value = getenv( "NUITKA_TICKER" ); if ( ticker_value != NULL ) { _Py_Ticker = atoi( ticker_value ); assert ( _Py_Ticker >= 20 ); } #ifdef _NUITKA_STANDALONE setEarlyFrozenModulesFileAttribute(); #endif // Disable Python warnings if requested to. #if %(python_no_warnings)d // Should be same as: // warnings.simplefilter("ignore", UserWarning) // warnings.simplefilter("ignore", DeprecationWarning) // There is no C-API to control warnings. We don't care if it actually // works, i.e. return code of "simplefilter" function is not checked. { PyObject *warnings = PyImport_ImportModule( "warnings" ); if ( warnings != NULL ) { PyObject *simplefilter = PyObject_GetAttrString( warnings, "simplefilter" ); if ( simplefilter != NULL ) { PyObject *result1 = PyObject_CallFunctionObjArgs( simplefilter, const_str_plain_ignore, PyExc_UserWarning, NULL ); assert( result1 ); Py_XDECREF( result1 ); PyObject *result2 = PyObject_CallFunctionObjArgs( simplefilter, const_str_plain_ignore, PyExc_DeprecationWarning, NULL ); assert( result2 ); Py_XDECREF( result2 ); } } } #endif // Enable meta path based loader. setupMetaPathBasedLoader(); #if _NUITKA_PROFILE startProfiling(); #endif // Execute the main module. In case of multiprocessing making a fork on // Windows, we should execute something else instead. #if _NUITKA_MODULE_COUNT > 1 if (unlikely( is_multiprocess_forking )) { #ifdef _NUITKA_TRACE puts("main(): Calling __parents_main__."); #endif PyObject *result = IMPORT_COMPILED_MODULE( PyUnicode_FromString("__parents_main__"), "__parents_main__" ); if ( result == NULL ) { PyErr_PrintEx( 0 ); Py_Exit( 1 ); } } else #endif { assert( !is_multiprocess_forking ); #ifdef _NUITKA_TRACE puts("main(): Calling __main__."); #endif // Execute the "__main__" module init function. MOD_INIT_NAME( __main__ )(); } #if _NUITKA_PROFILE stopProfiling(); #endif #ifndef __NUITKA_NO_ASSERT__ checkGlobalConstants(); checkModuleConstants___main__(); #endif if ( ERROR_OCCURRED() ) { // Cleanup code may need a frame, so put one back. PyThreadState_GET()->frame = MAKE_MODULE_FRAME( %(code_identifier)s, module___main__ ); PyErr_PrintEx( 0 ); Py_Exit( 1 ); } else { Py_Exit( 0 ); } // The above branches both do "Py_Exit()" calls which are not supposed to // return. NUITKA_CANNOT_GET_HERE( main ); } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesExceptions.py0000644000175000017500000000372712651071040025500 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for handling exceptions. """ template_publish_exception_to_handler = """\ if ( %(keeper_tb)s == NULL ) { %(keeper_tb)s = %(tb_making)s; } else if ( %(keeper_lineno)s != -1 ) { %(keeper_tb)s = ADD_TRACEBACK( %(keeper_tb)s, %(frame_identifier)s, %(keeper_lineno)s ); } """ template_error_catch_quick_exception = """\ if ( %(condition)s ) { if ( !ERROR_OCCURRED() ) { exception_type = %(quick_exception)s; Py_INCREF( exception_type ); exception_value = NULL; exception_tb = NULL; } else { FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); } %(release_temps)s %(line_number_code)s goto %(exception_exit)s; }""" template_error_catch_exception = """\ if ( %(condition)s ) { assert( ERROR_OCCURRED() ); FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(release_temps)s %(line_number_code)s goto %(exception_exit)s; }""" template_error_format_string_exception = """\ if ( %(condition)s ) { %(release_temps)s %(set_exception)s %(line_number_code)s goto %(exception_exit)s; } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesFunction.py0000644000175000017500000000746312651071040025145 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Normal function (no generator, not yielding) related templates. """ template_function_make_declaration = """\ static PyObject *MAKE_FUNCTION_%(function_identifier)s( %(function_creation_arg_spec)s ); """ template_function_direct_declaration = """\ %(file_scope)s PyObject *impl_%(function_identifier)s( %(direct_call_arg_spec)s ); """ template_function_closure_making = """\ PyCellObject **closure = (PyCellObject **)malloc(%(closure_count)d * sizeof(PyCellObject *)); %(closure_copy)s """ template_make_function_with_context_template = """ static PyObject *MAKE_FUNCTION_%(function_identifier)s( %(function_creation_args)s ) { // Copy the parameter default values and closure values over. %(closure_making)s PyObject *result = Nuitka_Function_New( %(function_impl_identifier)s, %(function_name_obj)s, #if PYTHON_VERSION >= 330 %(function_qualname_obj)s, #endif %(code_identifier)s, %(defaults)s, #if PYTHON_VERSION >= 300 %(kw_defaults)s, %(annotations)s, #endif %(module_identifier)s, %(function_doc)s, closure, %(closure_count)d ); return result; } """ template_make_function_without_context_template = """ static PyObject *MAKE_FUNCTION_%(function_identifier)s( %(function_creation_args)s ) { PyObject *result = Nuitka_Function_New( %(function_impl_identifier)s, %(function_name_obj)s, #if PYTHON_VERSION >= 330 %(function_qualname_obj)s, #endif %(code_identifier)s, %(defaults)s, #if PYTHON_VERSION >= 300 %(kw_defaults)s, %(annotations)s, #endif %(module_identifier)s, %(function_doc)s ); return result; } """ template_function_body = """\ static PyObject *impl_%(function_identifier)s( %(parameter_objects_decl)s ) { // Preserve error status for checks #ifndef __NUITKA_NO_ASSERT__ NUITKA_MAY_BE_UNUSED bool had_error = ERROR_OCCURRED(); #endif // Local variable declarations. %(function_locals)s // Actual function code. %(function_body)s %(function_exit)s } """ template_function_exception_exit = """\ function_exception_exit: %(function_cleanup)s\ assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return NULL; """ template_function_return_exit = """\ function_return_exit: %(function_cleanup)s CHECK_OBJECT( tmp_return_value ); assert( had_error || !ERROR_OCCURRED() ); return tmp_return_value; """ function_direct_body_template = """\ %(file_scope)s PyObject *impl_%(function_identifier)s( %(direct_call_arg_spec)s ) { #ifndef __NUITKA_NO_ASSERT__ NUITKA_MAY_BE_UNUSED bool had_error = ERROR_OCCURRED(); assert(!had_error); // Do not enter inlined functions with error set. #endif // Local variable declarations. %(function_locals)s // Actual function code. %(function_body)s %(function_exit)s } """ function_dict_setup = """\ // Locals dictionary setup. PyObject *locals_dict = PyDict_New(); """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesModules.py0000644000175000017500000002006412651071040024760 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Main module code templates This for the main program in case of executables, the module templates and stuff related to importing, and of course the generated code license. """ template_global_copyright = """\ // Generated code for Python source for module '%(name)s' // created by Nuitka version %(version)s // This code is in part copyright %(year)s Kay Hayen. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. """ template_module_body_template = """ #include "nuitka/prelude.hpp" #include "__helpers.hpp" // The _module_%(module_identifier)s is a Python object pointer of module type. // Note: For full compatibility with CPython, every module variable access // needs to go through it except for cases where the module cannot possibly // have changed in the mean time. PyObject *module_%(module_identifier)s; PyDictObject *moduledict_%(module_identifier)s; // The module constants used %(constant_decl_codes)s static bool constants_created = false; static void createModuleConstants( void ) { %(constant_init_codes)s constants_created = true; } #ifndef __NUITKA_NO_ASSERT__ void checkModuleConstants_%(module_identifier)s( void ) { // The module may not have been used at all. if (constants_created == false) return; %(constant_check_codes)s } #endif // The module code objects. %(module_code_objects_decl)s static void createModuleCodeObjects(void) { %(module_code_objects_init)s } // The module function declarations. %(module_functions_decl)s // The module function definitions. %(module_functions_code)s #if PYTHON_VERSION >= 300 static struct PyModuleDef mdef_%(module_identifier)s = { PyModuleDef_HEAD_INIT, "%(module_name)s", /* m_name */ NULL, /* m_doc */ -1, /* m_size */ NULL, /* m_methods */ NULL, /* m_reload */ NULL, /* m_traverse */ NULL, /* m_clear */ NULL, /* m_free */ }; #endif // The exported interface to CPython. On import of the module, this function // gets called. It has to have an exact function name, in cases it's a shared // library export. This is hidden behind the MOD_INIT_DECL. MOD_INIT_DECL( %(module_identifier)s ) { #if defined(_NUITKA_EXE) || PYTHON_VERSION >= 300 static bool _init_done = false; // Modules might be imported repeatedly, which is to be ignored. if ( _init_done ) { return MOD_RETURN_VALUE( module_%(module_identifier)s ); } else { _init_done = true; } #endif #ifdef _NUITKA_MODULE // In case of a stand alone extension module, need to call initialization // the init here because that's the first and only time we are going to get // called here. // Initialize the constant values used. _initBuiltinModule(); createGlobalConstants(); // Initialize the compiled types of Nuitka. PyType_Ready( &Nuitka_Generator_Type ); PyType_Ready( &Nuitka_Function_Type ); PyType_Ready( &Nuitka_Method_Type ); PyType_Ready( &Nuitka_Frame_Type ); #if PYTHON_VERSION >= 350 PyType_Ready( &Nuitka_Coroutine_Type ); PyType_Ready( &Nuitka_CoroutineWrapper_Type ); #endif #if PYTHON_VERSION < 300 _initSlotCompare(); #endif #if PYTHON_VERSION >= 270 _initSlotIternext(); #endif patchBuiltinModule(); patchTypeComparison(); #endif #if _NUITKA_MODULE // Enable meta path based loader if not already done. setupMetaPathBasedLoader(); #endif createModuleConstants(); createModuleCodeObjects(); // puts( "in init%(module_identifier)s" ); // Create the module object first. There are no methods initially, all are // added dynamically in actual code only. Also no "__doc__" is initially // set at this time, as it could not contain NUL characters this way, they // are instead set in early module code. No "self" for modules, we have no // use for it. #if PYTHON_VERSION < 300 module_%(module_identifier)s = Py_InitModule4( "%(module_name)s", // Module Name NULL, // No methods initially, all are added // dynamically in actual module code only. NULL, // No __doc__ is initially set, as it could // not contain NUL this way, added early in // actual code. NULL, // No self for modules, we don't use it. PYTHON_API_VERSION ); #else module_%(module_identifier)s = PyModule_Create( &mdef_%(module_identifier)s ); #endif moduledict_%(module_identifier)s = (PyDictObject *)((PyModuleObject *)module_%(module_identifier)s)->md_dict; CHECK_OBJECT( module_%(module_identifier)s ); // Seems to work for Python2.7 out of the box, but for Python3, the module // doesn't automatically enter "sys.modules", so do it manually. #if PYTHON_VERSION >= 300 { int r = PyObject_SetItem( PySys_GetObject( (char *)"modules" ), %(module_name_obj)s, module_%(module_identifier)s ); assert( r != -1 ); } #endif // For deep importing of a module we need to have "__builtins__", so we set // it ourselves in the same way than CPython does. Note: This must be done // before the frame object is allocated, or else it may fail. PyObject *module_dict = PyModule_GetDict( module_%(module_identifier)s ); if ( PyDict_GetItem( module_dict, const_str_plain___builtins__ ) == NULL ) { PyObject *value = (PyObject *)builtin_module; // Check if main module, not a dict then. #if !defined(_NUITKA_EXE) || !%(is_main_module)s value = PyModule_GetDict( value ); #endif #ifndef __NUITKA_NO_ASSERT__ int res = #endif PyDict_SetItem( module_dict, const_str_plain___builtins__, value ); assert( res == 0 ); } #if PYTHON_VERSION >= 330 #if _MODULE_LOADER PyDict_SetItem( module_dict, const_str_plain___loader__, metapath_based_loader ); #else PyDict_SetItem( module_dict, const_str_plain___loader__, Py_None ); #endif #endif // Temp variables if any %(temps_decl)s // Module code. %(module_code)s return MOD_RETURN_VALUE( module_%(module_identifier)s ); %(module_exit)s """ template_module_exception_exit = """\ module_exception_exit: RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return MOD_RETURN_VALUE( NULL ); }""" template_module_noexception_exit = """\ }""" template_helper_impl_decl = """\ // This file contains helper functions that are automatically created from // templates. #include "nuitka/prelude.hpp" extern PyObject *callPythonFunction( PyObject *func, PyObject **args, int count ); """ template_header_guard = """\ #ifndef %(header_guard_name)s #define %(header_guard_name)s %(header_body)s #endif """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesVariables.py0000644000175000017500000001323712651071040025264 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the variable handling. """ template_write_local_unclear_ref0 = """\ { PyObject *old = %(identifier)s; %(identifier)s = %(tmp_name)s; Py_XDECREF( old ); } """ template_write_local_unclear_ref1 = """\ { PyObject *old = %(identifier)s; %(identifier)s = %(tmp_name)s; Py_INCREF( %(identifier)s ); Py_XDECREF( old ); } """ template_write_local_empty_ref0 = """\ assert( %(identifier)s == NULL ); %(identifier)s = %(tmp_name)s; """ template_write_local_empty_ref1 = """\ assert( %(identifier)s == NULL ); Py_INCREF( %(tmp_name)s ); %(identifier)s = %(tmp_name)s; """ template_write_local_clear_ref0 = """\ { PyObject *old = %(identifier)s; assert( old != NULL ); %(identifier)s = %(tmp_name)s; Py_DECREF( old ); } """ template_write_local_inplace = """\ %(identifier)s = %(tmp_name)s; """ template_write_shared_inplace = """\ PyCell_SET( %(identifier)s, %(tmp_name)s ); """ template_write_local_clear_ref1 = """\ { PyObject *old = %(identifier)s; assert( old != NULL ); %(identifier)s = %(tmp_name)s; Py_INCREF( %(identifier)s ); Py_DECREF( old ); } """ template_write_shared_unclear_ref0 = """\ { PyObject *old = PyCell_GET( %(identifier)s ); PyCell_SET( %(identifier)s, %(tmp_name)s ); Py_XDECREF( old ); } """ template_write_shared_unclear_ref1 = """\ { PyObject *old = PyCell_GET( %(identifier)s ); PyCell_SET( %(identifier)s, %(tmp_name)s ); Py_INCREF( %(tmp_name)s ); Py_XDECREF( old ); } """ template_write_shared_clear_ref0 = """\ assert( PyCell_GET( %(identifier)s ) == NULL ); PyCell_SET( %(identifier)s, %(tmp_name)s ); """ template_write_shared_clear_ref1 = """\ assert( PyCell_GET( %(identifier)s ) == NULL ); Py_INCREF( %(tmp_name)s ); PyCell_SET( %(identifier)s, %(tmp_name)s ); """ template_read_local = """\ %(tmp_name)s = %(identifier)s; """ template_del_local_tolerant = """\ Py_XDECREF( %(identifier)s ); %(identifier)s = NULL; """ template_del_shared_tolerant = """\ if ( %(identifier)s ) { Py_XDECREF( PyCell_GET( %(identifier)s )); PyCell_SET( %(identifier)s, NULL ); } """ template_del_local_intolerant = """\ %(result)s = %(identifier)s != NULL; if ( %(result)s == true ) { Py_DECREF( %(identifier)s ); %(identifier)s = NULL; } """ # TODO: Storage will not be NULL. What is this used for. template_del_shared_intolerant = """\ %(result)s = %(identifier)s != NULL && PyCell_GET( %(identifier)s ) != NULL; if ( %(result)s == true ) { Py_DECREF( PyCell_GET( %(identifier)s ) ); PyCell_SET( %(identifier)s, NULL ); } """ template_del_local_known = """\ CHECK_OBJECT( %(identifier)s ); Py_DECREF( %(identifier)s ); %(identifier)s = NULL; """ template_del_shared_known = """\ Py_DECREF( PyCell_GET( %(identifier)s ) ); PyCell_SET( %(identifier)s, NULL ); """ template_release_unclear = """\ Py_XDECREF( %(identifier)s ); %(identifier)s = NULL; """ template_release_clear = """\ CHECK_OBJECT( (PyObject *)%(identifier)s ); Py_DECREF( %(identifier)s ); %(identifier)s = NULL; """ # TODO: Storage will not be NULL. template_read_shared_unclear = """\ if ( %(identifier)s == NULL ) { %(tmp_name)s = NULL; } else { %(tmp_name)s = PyCell_GET( %(identifier)s ); } """ template_read_shared_known = """\ %(tmp_name)s = PyCell_GET( %(identifier)s ); """ # For module variable values, need to lookup in module dictionary or in # built-in dictionary. template_read_mvar_unclear = """\ %(tmp_name)s = GET_STRING_DICT_VALUE( moduledict_%(module_identifier)s, (Nuitka_StringObject *)%(var_name)s ); if (unlikely( %(tmp_name)s == NULL )) { %(tmp_name)s = GET_STRING_DICT_VALUE( dict_builtin, (Nuitka_StringObject *)%(var_name)s ); } """ template_read_maybe_local_unclear = """\ %(tmp_name)s = PyDict_GetItem( %(locals_dict)s, %(var_name)s ); if ( %(tmp_name)s == NULL ) { %(fallback)s } """ template_del_global_unclear = """\ %(res_name)s = PyDict_DelItem( (PyObject *)moduledict_%(module_identifier)s, %(var_name)s ); if ( %(res_name)s == -1 ) CLEAR_ERROR_OCCURRED(); """ template_update_locals_dict_value = """\ DICT_SYNC_FROM_VARIABLE( %(dict_name)s, %(var_name)s, %(access_code)s ); """ template_set_locals_dict_value = """\ if ( %(access_code)s ) { int res = PyDict_SetItem( %(dict_name)s, %(var_name)s, %(access_code)s ); assert( res == 0 ); } """ template_update_locals_mapping_value = """\ %(tmp_name)s = MAPPING_SYNC_FROM_VARIABLE( %(mapping_name)s, %(var_name)s, %(access_code)s ); """ template_set_locals_mapping_value = """\ if %(check_code)s { %(tmp_name)s = SET_SUBSCRIPT( %(mapping_name)s, %(var_name)s, %(access_code)s ); } """ # TODO: Unused now. template_assign_from_frame_locals = """\ if ( %(frame_identifier)s->f_locals == NULL ) { %(frame_identifier)s->f_locals = PyDict_New(); } %(to_name)s = %(frame_identifier)s->f_locals; """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesIterators.py0000644000175000017500000000447712651071040025336 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the iterator handling. """ template_iterator_check = """\ // Check if iterator has left-over elements. CHECK_OBJECT( %(iterator_name)s ); assert( HAS_ITERNEXT( %(iterator_name)s ) ); %(attempt_name)s = (*Py_TYPE( %(iterator_name)s )->tp_iternext)( %(iterator_name)s ); if (likely( %(attempt_name)s == NULL )) { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL ) { if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration )) { CLEAR_ERROR_OCCURRED(); } else { FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(release_temps_1)s goto %(exception_exit)s; } } } else { Py_DECREF( %(attempt_name)s ); // TODO: Could avoid PyErr_Format. #if PYTHON_VERSION < 300 PyErr_Format( PyExc_ValueError, "too many values to unpack" ); #else PyErr_Format( PyExc_ValueError, "too many values to unpack (expected %(count)d)" ); #endif FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(release_temps_2)s goto %(exception_exit)s; }""" template_loop_break_next = """\ if ( %(to_name)s == NULL ) { if ( CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED() ) { %(break_indicator_code)s goto %(break_target)s; } else { %(release_temps)s FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(line_number_code)s goto %(exception_target)s; } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesCalls.py0000644000175000017500000002733212651072046024422 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for calling functions with positional args only very quickly. """ template_call_function_with_args_decl = """\ extern PyObject *CALL_FUNCTION_WITH_ARGS%(args_count)d( PyObject *called, PyObject **args );""" template_call_function_with_args_impl = """\ PyObject *CALL_FUNCTION_WITH_ARGS%(args_count)d( PyObject *called, PyObject **args ) { CHECK_OBJECT( called ); // Check if arguments are valid objects in debug mode. #ifndef __NUITKA_NO_ASSERT__ for( size_t i = 0; i < %(args_count)d; i++ ) { CHECK_OBJECT( args[ i ] ); } #endif if ( Nuitka_Function_Check( called ) ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } Nuitka_FunctionObject *function = (Nuitka_FunctionObject *)called; PyObject *result; if ( function->m_args_simple && %(args_count)d == function->m_args_positional_count ) { for( Py_ssize_t i = 0; i < %(args_count)d; i++ ) { Py_INCREF( args[ i ] ); } result = function->m_c_code( function, args ); } else if ( function->m_args_simple && %(args_count)d + function->m_defaults_given == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_positional_count ); #else PyObject *python_pars[ function->m_args_positional_count ]; #endif memcpy( python_pars, args, %(args_count)d * sizeof(PyObject *) ); memcpy( python_pars + %(args_count)d, &PyTuple_GET_ITEM( function->m_defaults, 0 ), function->m_defaults_given * sizeof(PyObject *) ); for( Py_ssize_t i = 0; i < function->m_args_positional_count; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsPos( function, python_pars, args, %(args_count)d )) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } else if ( Nuitka_Method_Check( called ) ) { Nuitka_MethodObject *method = (Nuitka_MethodObject *)called; // Unbound method without arguments, let the error path be slow. if ( method->m_object != NULL ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } Nuitka_FunctionObject *function = method->m_function; PyObject *result; if ( function->m_args_simple && %(args_count)d + 1 == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_positional_count ); #else PyObject *python_pars[ function->m_args_positional_count ]; #endif python_pars[ 0 ] = method->m_object; Py_INCREF( method->m_object ); for( Py_ssize_t i = 0; i < %(args_count)d; i++ ) { python_pars[ i + 1 ] = args[ i ]; Py_INCREF( args[ i ] ); } result = function->m_c_code( function, python_pars ); } else if ( function->m_args_simple && %(args_count)d + 1 + function->m_defaults_given == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_positional_count ); #else PyObject *python_pars[ function->m_args_positional_count ]; #endif python_pars[ 0 ] = method->m_object; Py_INCREF( method->m_object ); memcpy( python_pars+1, args, %(args_count)d * sizeof(PyObject *) ); memcpy( python_pars+1 + %(args_count)d, &PyTuple_GET_ITEM( function->m_defaults, 0 ), function->m_defaults_given * sizeof(PyObject *) ); for( Py_ssize_t i = 1; i < function->m_args_overall_count; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsMethodPos( function, python_pars, method->m_object, args, %(args_count)d ) ) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } } else if ( PyCFunction_Check( called ) ) { // Try to be fast about wrapping the arguments. int flags = PyCFunction_GET_FLAGS( called ); if ( flags & METH_NOARGS ) { #if %(args_count)d == 0 PyCFunction method = PyCFunction_GET_FUNCTION( called ); PyObject *self = PyCFunction_GET_SELF( called ); // Recursion guard is not strictly necessary, as we already have // one on our way to here. #ifdef _NUITKA_FULL_COMPAT if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } #endif PyObject *result = (*method)( self, NULL ); #ifdef _NUITKA_FULL_COMPAT Py_LeaveRecursiveCall(); #endif if ( result != NULL ) { // Some buggy C functions do set an error, but do not indicate it // and Nuitka inner workings can get upset/confused from it. DROP_ERROR_OCCURRED(); return result; } else { // Other buggy C functions do this, return NULL, but with // no error set, not allowed. if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in PyObject_Call" ); } return NULL; } #else PyErr_Format( PyExc_TypeError, "%%s() takes no arguments (%(args_count)d given)", ((PyCFunctionObject *)called)->m_ml->ml_name ); return NULL; #endif } else if ( flags & METH_O ) { #if %(args_count)d == 1 PyCFunction method = PyCFunction_GET_FUNCTION( called ); PyObject *self = PyCFunction_GET_SELF( called ); // Recursion guard is not strictly necessary, as we already have // one on our way to here. #ifdef _NUITKA_FULL_COMPAT if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } #endif PyObject *result = (*method)( self, args[0] ); #ifdef _NUITKA_FULL_COMPAT Py_LeaveRecursiveCall(); #endif if ( result != NULL ) { // Some buggy C functions do set an error, but do not indicate it // and Nuitka inner workings can get upset/confused from it. DROP_ERROR_OCCURRED(); return result; } else { // Other buggy C functions do this, return NULL, but with // no error set, not allowed. if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in PyObject_Call" ); } return NULL; } #else PyErr_Format(PyExc_TypeError, "%%s() takes exactly one argument (%(args_count)d given)", ((PyCFunctionObject *)called)->m_ml->ml_name ); return NULL; #endif } else { PyCFunction method = PyCFunction_GET_FUNCTION( called ); PyObject *self = PyCFunction_GET_SELF( called ); PyObject *pos_args = MAKE_TUPLE( args, %(args_count)d ); PyObject *result; assert( flags && METH_VARARGS ); // Recursion guard is not strictly necessary, as we already have // one on our way to here. #ifdef _NUITKA_FULL_COMPAT if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } #endif if ( flags && METH_KEYWORDS ) { result = (*(PyCFunctionWithKeywords)method)( self, pos_args, NULL ); } else { result = (*method)( self, pos_args ); } #ifdef _NUITKA_FULL_COMPAT Py_LeaveRecursiveCall(); #endif if ( result != NULL ) { // Some buggy C functions do set an error, but do not indicate it // and Nuitka inner workings can get upset/confused from it. DROP_ERROR_OCCURRED(); Py_DECREF( pos_args ); return result; } else { // Other buggy C functions do this, return NULL, but with // no error set, not allowed. if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in PyObject_Call" ); } Py_DECREF( pos_args ); return NULL; } } } else if ( PyFunction_Check( called ) ) { return callPythonFunction( called, args, %(args_count)d ); } PyObject *pos_args = MAKE_TUPLE( args, %(args_count)d ); PyObject *result = CALL_FUNCTION( called, pos_args, NULL ); Py_DECREF( pos_args ); return result; } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesFreezer.py0000644000175000017500000000357312651071040024760 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for code to load frozen bytecode. """ template_frozen_modules = """\ // This provides the frozen (compiled bytecode) files that are included if // any. #include // Blob from which modules are unstreamed. #if defined(_WIN32) && defined(_NUITKA_EXE) extern const unsigned char* constant_bin; #else extern "C" const unsigned char constant_bin[]; #endif #define stream_data constant_bin // These modules should be loaded as bytecode. They may e.g. have to be loadable // during "Py_Initialize" already, or for irrelevance, they are only included // in this un-optimized form. These are not compiled by Nuitka, and therefore // are not accelerated at all, merely bundled with the binary or module, so // that CPython library can start out finding them. void copyFrozenModulesTo( void* destination ) { _frozen frozen_modules[] = { %(frozen_modules)s { NULL, NULL, 0 } }; memcpy( destination, frozen_modules, ( _NUITKA_FROZEN + 1 ) * sizeof( struct _frozen ) ); } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesConstants.py0000644000175000017500000000540712651071040025330 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the constants handling. """ template_constants_reading = """ #include "nuitka/prelude.hpp" // Sentinel PyObject to be used for all our call iterator endings. It will // become a PyCObject pointing to NULL. It's address is unique, and that's // enough for us to use it as sentinel value. PyObject *_sentinel_value = NULL; %(constant_declarations)s #if defined(_WIN32) && defined(_NUITKA_EXE) #include const unsigned char* constant_bin; struct __initResourceConstants { __initResourceConstants() { constant_bin = (const unsigned char*)LockResource( LoadResource( NULL, FindResource(NULL, MAKEINTRESOURCE(3), RT_RCDATA) ) ); } } __initResourceConstants_static_initializer; #else extern "C" const unsigned char constant_bin[]; #endif static void _createGlobalConstants( void ) { NUITKA_MAY_BE_UNUSED PyObject *exception_type, *exception_value; NUITKA_MAY_BE_UNUSED PyTracebackObject *exception_tb; #ifdef _MSC_VER // Prevent unused warnings in case of simple programs, the attribute // NUITKA_MAY_BE_UNUSED doesn't work for MSVC. (void *)exception_type; (void *)exception_value; (void *)exception_tb; #endif %(constant_inits)s } // In debug mode we can check that the constants were not tampered with in any // given moment. We typically do it at program exit, but we can add extra calls // for sanity. #ifndef __NUITKA_NO_ASSERT__ void checkGlobalConstants( void ) { %(constant_checks)s } #endif void createGlobalConstants( void ) { if ( _sentinel_value == NULL ) { #if PYTHON_VERSION < 300 _sentinel_value = PyCObject_FromVoidPtr( NULL, NULL ); #else // The NULL value is not allowed for a capsule, so use something else. _sentinel_value = PyCapsule_New( (void *)27, "sentinel", NULL ); #endif assert( _sentinel_value ); _createGlobalConstants(); } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesCoroutines.py0000644000175000017500000000537412651071040025511 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Coroutines function (await/async) related templates. """ template_coroutine_object_decl_template = """\ static void %(function_identifier)s( Nuitka_CoroutineObject *coroutine ); """ template_coroutine_object_body_template = """ static void %(function_identifier)s( Nuitka_CoroutineObject *coroutine ) { CHECK_OBJECT( (PyObject *)coroutine ); assert( Nuitka_Coroutine_Check( (PyObject *)coroutine ) ); // Local variable initialization %(function_var_inits)s // Actual function code. %(function_body)s %(coroutine_exit)s } """ template_coroutine_exception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); function_exception_exit: assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); coroutine->m_yielded = NULL; return; """ template_coroutine_noexception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); coroutine->m_yielded = NULL; return; """ template_coroutine_return_exit = """\ function_return_exit:; coroutine->m_returned = tmp_return_value; return; """ template_make_coroutine_without_context_template = """ %(to_name)s = Nuitka_Coroutine_New( %(coroutine_identifier)s, self->m_name, self->m_qualname, %(code_identifier)s, NULL, 0 ); """ template_make_coroutine_with_context_template = """ { %(closure_making)s %(to_name)s = Nuitka_Coroutine_New( %(coroutine_identifier)s, self->m_name, self->m_qualname, %(code_identifier)s, closure, %(closure_count)d ); } """ template_coroutine_await = """ { PyObject *awaitable = _PyCoro_GetAwaitableIter( %(value)s ); if (likely( awaitable != NULL )) { %(to_name) = COROUTINE_AWAIT( awaitable ); } else { %(to_name) = NULL; } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/__init__.py0000644000175000017500000000150112651071040022430 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesFrames.py0000644000175000017500000002274112651071040024571 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code templates for frames of all kinds. """ template_frame_guard_cache_decl = """\ static PyFrameObject *cache_%(frame_identifier)s = NULL; """ template_frame_guard_frame_decl = """\ PyFrameObject *%(frame_identifier)s; """ # Frame in a function template_frame_guard_full_block = """\ MAKE_OR_REUSE_FRAME( cache_%(frame_identifier)s, %(code_identifier)s, %(module_identifier)s ); %(frame_identifier)s = cache_%(frame_identifier)s; // Push the new frame as the currently active one. pushFrameStack( %(frame_identifier)s ); // Mark the frame object as in use, ref count 1 will be up for reuse. Py_INCREF( %(frame_identifier)s ); assert( Py_REFCNT( %(frame_identifier)s ) == 2 ); // Frame stack #if PYTHON_VERSION >= 340 %(frame_identifier)s->f_executing += 1; #endif // Framed code: %(codes)s #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif // Put the previous frame back on top. popFrameStack(); #if PYTHON_VERSION >= 340 %(frame_identifier)s->f_executing -= 1; #endif Py_DECREF( %(frame_identifier)s ); goto %(no_exception_exit)s; """ template_frame_guard_full_return_handler = """\ %(frame_return_exit)s:; #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif popFrameStack(); #if PYTHON_VERSION >= 340 %(frame_identifier)s->f_executing -= 1; #endif Py_DECREF( %(frame_identifier)s ); goto %(return_exit)s; """ template_frame_guard_full_exception_handler = """\ %(frame_exception_exit)s:; #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif { bool needs_detach = false; if ( exception_tb == NULL ) { exception_tb = %(tb_making)s; needs_detach = true; } else if ( exception_lineno != -1 ) { PyTracebackObject *traceback_new = MAKE_TRACEBACK( %(frame_identifier)s, exception_lineno ); traceback_new->tb_next = exception_tb; exception_tb = traceback_new; needs_detach = true; } if (needs_detach) { %(store_frame_locals)s detachFrame( exception_tb, %(frame_locals_name)s ); } } popFrameStack(); #if PYTHON_VERSION >= 340 %(frame_identifier)s->f_executing -= 1; #endif Py_DECREF( %(frame_identifier)s ); // Return the error. goto %(parent_exception_exit)s; """ # Frame for a module. TODO: Use it for functions called only once. # TODO: The once guard need not take a reference count in its frame class. template_frame_guard_once = """\ // Frame without reuse. %(frame_identifier)s = MAKE_MODULE_FRAME( %(code_identifier)s, %(module_identifier)s ); // Push the new frame as the currently active one, and we should be exclusively // owning it. pushFrameStack( %(frame_identifier)s ); assert( Py_REFCNT( %(frame_identifier)s ) == 1 ); #if PYTHON_VERSION >= 340 %(frame_identifier)s->f_executing += 1; #endif // Framed code: %(codes)s // Restore frame exception if necessary. #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif popFrameStack(); assertFrameObject( %(frame_identifier)s ); Py_DECREF( %(frame_identifier)s ); goto %(no_exception_exit)s; %(frame_exception_exit)s:; #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif if ( exception_tb == NULL ) { exception_tb = %(tb_making)s; } else if ( exception_tb->tb_frame != %(frame_identifier)s ) { PyTracebackObject *traceback_new = MAKE_TRACEBACK( %(frame_identifier)s, exception_lineno ); traceback_new->tb_next = exception_tb; exception_tb = traceback_new; } // Put the previous frame back on top. popFrameStack(); #if PYTHON_VERSION >= 340 %(frame_identifier)s->f_executing -= 1; #endif Py_DECREF( %(frame_identifier)s ); // Return the error. goto %(parent_exception_exit)s; %(no_exception_exit)s:;""" template_generator_initial_throw = """\ // Throwing into not started generators is possible. As they don't stand any // chance to deal with them, we might as well create traceback on the // outside, if ( generator->m_exception_type ) { generator->m_yielded = NULL; exception_type = generator->m_exception_type; generator->m_exception_type = NULL; exception_value = generator->m_exception_value; generator->m_exception_value = NULL; exception_tb = generator->m_exception_tb;; generator->m_exception_tb = NULL; if (exception_tb == NULL) { %(set_error_line_number)s goto %(frame_exception_exit)s; } else { goto function_exception_exit; } } """ # Frame in a generator template_frame_guard_generator = """\ MAKE_OR_REUSE_FRAME( %(frame_cache_identifier)s, %(code_identifier)s, %(module_identifier)s ); generator->m_frame = %(frame_cache_identifier)s; Py_INCREF( generator->m_frame ); #if PYTHON_VERSION >= 340 generator->m_frame->f_gen = (PyObject *)generator; #endif Py_CLEAR( generator->m_frame->f_back ); generator->m_frame->f_back = PyThreadState_GET()->frame; Py_INCREF( generator->m_frame->f_back ); PyThreadState_GET()->frame = generator->m_frame; Py_INCREF( generator->m_frame ); #if PYTHON_VERSION >= 340 generator->m_frame->f_executing += 1; #endif #if PYTHON_VERSION >= 300 // Accept currently existing exception as the one to publish again when we // yield or yield from. PyThreadState *thread_state = PyThreadState_GET(); generator->m_frame->f_exc_type = thread_state->exc_type; if ( generator->m_frame->f_exc_type == Py_None ) generator->m_frame->f_exc_type = NULL; Py_XINCREF( generator->m_frame->f_exc_type ); generator->m_frame->f_exc_value = thread_state->exc_value; Py_XINCREF( generator->m_frame->f_exc_value ); generator->m_frame->f_exc_traceback = thread_state->exc_traceback; Py_XINCREF( generator->m_frame->f_exc_traceback ); #endif // Framed code: %(codes)s #if PYTHON_VERSION >= 340 generator->m_frame->f_executing -= 1; #endif #if PYTHON_VERSION >= 300 Py_CLEAR( generator->m_frame->f_exc_type ); Py_CLEAR( generator->m_frame->f_exc_value ); Py_CLEAR( generator->m_frame->f_exc_traceback ); #endif Py_DECREF( generator->m_frame ); goto %(no_exception_exit)s; """ # Frame in a coroutine template_frame_guard_coroutine = """\ MAKE_OR_REUSE_FRAME( %(frame_cache_identifier)s, %(code_identifier)s, %(module_identifier)s ); coroutine->m_frame = %(frame_cache_identifier)s; Py_INCREF( coroutine->m_frame ); coroutine->m_frame->f_gen = (PyObject *)coroutine; Py_CLEAR( coroutine->m_frame->f_back ); coroutine->m_frame->f_back = PyThreadState_GET()->frame; Py_INCREF( coroutine->m_frame->f_back ); PyThreadState_GET()->frame = coroutine->m_frame; Py_INCREF( coroutine->m_frame ); coroutine->m_frame->f_executing += 1; // Accept currently existing exception as the one to publish again when we // yield or yield from. PyThreadState *thread_state = PyThreadState_GET(); coroutine->m_frame->f_exc_type = thread_state->exc_type; if ( coroutine->m_frame->f_exc_type == Py_None ) coroutine->m_frame->f_exc_type = NULL; Py_XINCREF( coroutine->m_frame->f_exc_type ); coroutine->m_frame->f_exc_value = thread_state->exc_value; Py_XINCREF( coroutine->m_frame->f_exc_value ); coroutine->m_frame->f_exc_traceback = thread_state->exc_traceback; Py_XINCREF( coroutine->m_frame->f_exc_traceback ); // Framed code: %(codes)s coroutine->m_frame->f_executing -= 1; Py_CLEAR( coroutine->m_frame->f_exc_type ); Py_CLEAR( coroutine->m_frame->f_exc_value ); Py_CLEAR( coroutine->m_frame->f_exc_traceback ); Py_DECREF( coroutine->m_frame ); goto %(no_exception_exit)s; """ # TODO: This cannot happen, can it? template_frame_guard_generator_return_handler = """\ %(frame_return_exit)s:; #if PYTHON_VERSION >= 300 Py_CLEAR( %(frame_identifier)s->f_exc_type ); Py_CLEAR( %(frame_identifier)s->f_exc_value ); Py_CLEAR( %(frame_identifier)s->f_exc_traceback ); #endif Py_DECREF( %(frame_identifier)s ); goto %(return_exit)s; """ template_frame_guard_generator_exception_handler = """\ %(frame_exception_exit)s:; // If it's not an exit exception, consider and create a traceback for it. if ( !EXCEPTION_MATCH_GENERATOR( exception_type ) ) { int needs_detach = false; if ( exception_tb == NULL ) { exception_tb = %(tb_making)s; needs_detach = true; } else if ( exception_tb->tb_frame != %(frame_identifier)s ) { PyTracebackObject *traceback_new = MAKE_TRACEBACK( %(frame_identifier)s, exception_lineno ); traceback_new->tb_next = exception_tb; exception_tb = traceback_new; needs_detach = true; } if (needs_detach) { %(store_frame_locals)s detachFrame( exception_tb, %(frame_locals_name)s ); } } #if PYTHON_VERSION >= 300 Py_CLEAR( %(frame_identifier)s->f_exc_type ); Py_CLEAR( %(frame_identifier)s->f_exc_value ); Py_CLEAR( %(frame_identifier)s->f_exc_traceback ); #endif Py_DECREF( %(frame_identifier)s ); // Return the error. goto %(parent_exception_exit)s; %(no_exception_exit)s:; """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesLoader.py0000644000175000017500000000375512651071040024566 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the loading of embedded modules. """ template_metapath_loader_compiled_module_entry = """\ { (char *)"%(module_name)s", MOD_INIT_NAME( %(module_identifier)s ), NUITKA_COMPILED_MODULE },""" template_metapath_loader_compiled_package_entry = """\ { (char *)"%(module_name)s", MOD_INIT_NAME( %(module_identifier)s ), NUITKA_COMPILED_PACKAGE },""" template_metapath_loader_shlib_module_entry = """\ { (char *)"%(module_name)s", NULL, NUITKA_SHLIB_MODULE },""" template_metapath_loader_body = """\ // Code to register embedded modules if any. #if %(use_loader)d == 1 #include "nuitka/unfreezing.hpp" // Table for lookup to find "frozen" modules or DLLs, i.e. the ones included in // or along this binary. %(metapath_module_decls)s static struct Nuitka_MetaPathBasedLoaderEntry meta_path_loader_entries[] = { %(metapath_loader_inittab)s { NULL, NULL, 0 } }; void setupMetaPathBasedLoader( void ) { static bool init_done = false; if ( init_done == false ) { registerMetaPathBasedUnfreezer( meta_path_loader_entries ); init_done = true; } } #else void setupMetaPathBasedLoader( void ) { } #endif """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/CodeTemplatesGeneratorFunction.py0000644000175000017500000000604612651071040027010 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Generator function (with yield) related templates. """ template_genfunc_yielder_decl_template = """\ static void %(function_identifier)s_context( Nuitka_GeneratorObject *generator ); """ template_genfunc_yielder_body_template = """ static void %(function_identifier)s_context( Nuitka_GeneratorObject *generator ) { CHECK_OBJECT( (PyObject *)generator ); assert( Nuitka_Generator_Check( (PyObject *)generator ) ); // Local variable initialization %(function_var_inits)s // Actual function code. %(function_body)s %(generator_exit)s } """ template_generator_exception_exit = """\ RESTORE_ERROR_OCCURRED( PyExc_StopIteration, NULL, NULL ); Py_INCREF( PyExc_StopIteration ); generator->m_yielded = NULL; return; function_exception_exit: assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); generator->m_yielded = NULL; return; """ template_generator_noexception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); generator->m_yielded = NULL; return; """ template_generator_return_exit = """\ // The above won't return, but we need to make it clear to the compiler // as well, or else it will complain and/or generate inferior code. assert(false); return; function_return_exit: #if PYTHON_VERSION < 330 RESTORE_ERROR_OCCURRED( PyExc_StopIteration, NULL, NULL ); #else RESTORE_ERROR_OCCURRED( PyExc_StopIteration, tmp_return_value, NULL ); #endif Py_INCREF( PyExc_StopIteration ); generator->m_yielded = NULL; return; """ template_generator_making_without_context = """\ %(to_name)s = Nuitka_Generator_New( %(generator_identifier)s_context, %(generator_name_obj)s, #if PYTHON_VERSION >= 350 %(generator_qualname_obj)s, #endif %(code_identifier)s, NULL, 0 ); """ template_generator_making_with_context = """\ { %(closure_making)s %(to_name)s = Nuitka_Generator_New( %(generator_identifier)s_context, %(generator_name_obj)s, #if PYTHON_VERSION >= 350 %(generator_qualname_obj)s, #endif %(code_identifier)s, closure, %(closure_count)d ); } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.18.1/nuitka/codegen/templates/TemplateDebugWrapper.py0000644000175000017500000000466712651071040024774 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nuitka templates can have more checks that the normal '%' operation. This wraps strings with a class derived from "str" that does more checks. """ from nuitka.__past__ import iterItems from nuitka.Options import isDebug def enableDebug(globals_dict): templates = dict(globals_dict) class TemplateWrapper: """ Wrapper around templates. To better trace and control template usage. """ def __init__(self, name, value): self.name = name self.value = value def __str__(self): return self.value def __mod__(self, other): assert type(other) is dict, self.name for key in other.keys(): if "%%(%s)" % key not in self.value: from logging import warning warning( "Extra value '%s' provided to template '%s'.", key, self.name ) try: return self.value % other except KeyError as e: raise KeyError(self.name, *e.args) def split(self, sep): return self.value.split(sep) for template_name, template_value in iterItems(templates): # Ignore internal attribute like "__name__" that the module will also # have of course. if template_name.startswith('_'): continue if type(template_value) is str: globals_dict[template_name] = TemplateWrapper( template_name, template_value ) def checkDebug(globals_dict): if isDebug(): enableDebug(globals_dict) Nuitka-0.5.18.1/nuitka/codegen/ConditionalCodes.py0000644000175000017500000003243712651071040022130 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Conditional statements related codes. Branches, conditions, truth checks. """ from nuitka import Options from .AttributeCodes import getAttributeCheckBoolCode from .ComparisonCodes import ( getBuiltinIsinstanceBoolCode, getComparisonExpressionBoolCode ) from .Emission import SourceCodeCollector from .ErrorCodes import getErrorExitBoolCode, getReleaseCode from .Helpers import generateExpressionCode from .LabelCodes import getBranchingCode, getGotoCode, getLabelCode def generateConditionCode(condition, emit, context): # The complexity is needed to avoid unnecessary complex generated C++ # pylint: disable=R0914,R0915 if condition.isExpressionConstantRef(): # TODO: Must not happen, optimization catches this. assert False value = condition.getConstant() if value: getGotoCode(context.getTrueBranchTarget(), emit) else: getGotoCode(context.getFalseBranchTarget(), emit) elif condition.isExpressionComparison(): left_name = context.allocateTempName("compare_left") generateExpressionCode( to_name = left_name, expression = condition.getLeft(), emit = emit, context = context ) right_name = context.allocateTempName("compare_right") generateExpressionCode( to_name = right_name, expression = condition.getRight(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(condition.getSourceReference()) getComparisonExpressionBoolCode( comparator = condition.getComparator(), left_name = left_name, right_name = right_name, needs_check = condition.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif condition.isExpressionOperationNOT(): # Lets just switch the targets temporarily to get at "NOT" without # any effort really. true_target = context.getTrueBranchTarget() false_target = context.getFalseBranchTarget() context.setTrueBranchTarget(false_target) context.setFalseBranchTarget(true_target) generateConditionCode( condition = condition.getOperand(), emit = emit, context = context ) context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) elif condition.isExpressionConditional(): expression_yes = condition.getExpressionYes() expression_no = condition.getExpressionNo() condition = condition.getCondition() old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() select_true = context.allocateLabel("select_true") select_false = context.allocateLabel("select_false") # TODO: Could be avoided in some cases. select_end = context.allocateLabel("select_end") context.setTrueBranchTarget(select_true) context.setFalseBranchTarget(select_false) generateConditionCode( condition = condition, emit = emit, context = context, ) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) getLabelCode(select_true,emit) generateConditionCode( condition = expression_yes, emit = emit, context = context, ) getGotoCode(select_end, emit) getLabelCode(select_false,emit) generateConditionCode( condition = expression_no, emit = emit, context = context, ) getLabelCode(select_end,emit) elif condition.isExpressionBuiltinHasattr(): source_name = context.allocateTempName("hasattr_source") attr_name = context.allocateTempName("hasattr_attr") generateExpressionCode( to_name = source_name, expression = condition.getLookupSource(), emit = emit, context = context ) generateExpressionCode( to_name = attr_name, expression = condition.getAttribute(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( condition.getAttribute().getSourceReference() if Options.isFullCompat() else condition.getSourceReference() ) getAttributeCheckBoolCode( source_name = source_name, attr_name = attr_name, needs_check = condition.getLookupSource().mayRaiseExceptionAttributeCheckObject( exception_type = BaseException, attribute = condition.getAttribute() ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif condition.isExpressionBuiltinIsinstance(): inst_name = context.allocateTempName("isinstance_inst") cls_name = context.allocateTempName("isinstance_cls") generateExpressionCode( to_name = inst_name, expression = condition.getInstance(), emit = emit, context = context ) generateExpressionCode( to_name = cls_name, expression = condition.getCls(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(condition.getSourceReference()) getBuiltinIsinstanceBoolCode( inst_name = inst_name, cls_name = cls_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: condition_name = context.allocateTempName("cond_value") truth_name = context.allocateTempName("cond_truth", "int") generateExpressionCode( to_name = condition_name, expression = condition, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(condition.getSourceReference()) getConditionCheckTrueCode( to_name = truth_name, value_name = condition_name, needs_check = condition.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) getReleaseCode( release_name = condition_name, emit = emit, context = context ) getBranchingCode( condition = "%s == 1" % truth_name, emit = emit, context = context ) def getConditionCheckTrueCode(to_name, value_name, needs_check, emit, context): emit( "%s = CHECK_IF_TRUE( %s );" % ( to_name, value_name ) ) getErrorExitBoolCode( condition = "%s == -1" % to_name, needs_check = needs_check, emit = emit, context = context ) def generateConditionalAndOrCode(to_name, expression, emit, context): # This is a complex beast, handling both "or" and "and" expressions, # and it needs to micro manage details. # pylint: disable=R0914 if expression.isExpressionConditionalOR(): prefix = "or_" else: prefix = "and_" true_target = context.allocateLabel(prefix + "left") false_target = context.allocateLabel(prefix + "right") end_target = context.allocateLabel(prefix + "end") old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() truth_name = context.allocateTempName(prefix + "left_truth", "int") left_name = context.allocateTempName(prefix + "left_value") right_name = context.allocateTempName(prefix + "right_value") left_value = expression.getLeft() generateExpressionCode( to_name = left_name, expression = left_value, emit = emit, context = context ) # We need to treat this mostly manually here. We remember to release # this, and we better do this manually later. needs_ref1 = context.needsCleanup(left_name) getConditionCheckTrueCode( to_name = truth_name, value_name = left_name, needs_check = left_value.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) if expression.isExpressionConditionalOR(): context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) else: context.setTrueBranchTarget(false_target) context.setFalseBranchTarget(true_target) getBranchingCode( condition = "%s == 1" % truth_name, emit = emit, context = context ) getLabelCode(false_target,emit) # So it's not the left value, then lets release that one right away, it # is not needed, but we remember if it should be added above. getReleaseCode( release_name = left_name, emit = emit, context = context ) # Evaluate the "right" value then. generateExpressionCode( to_name = right_name, expression = expression.getRight(), emit = emit, context = context ) # Again, remember the reference count to manage it manually. needs_ref2 = context.needsCleanup(right_name) if needs_ref2: context.removeCleanupTempName(right_name) if not needs_ref2 and needs_ref1: emit("Py_INCREF( %s );" % right_name) emit( "%s = %s;" % ( to_name, right_name ) ) getGotoCode(end_target, emit) getLabelCode(true_target, emit) if not needs_ref1 and needs_ref2: emit("Py_INCREF( %s );" % left_name) emit( "%s = %s;" % ( to_name, left_name ) ) getLabelCode(end_target, emit) if needs_ref1 or needs_ref2: context.addCleanupTempName(to_name) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) def generateConditionalCode(to_name, expression, emit, context): true_target = context.allocateLabel("condexpr_true") false_target = context.allocateLabel("condexpr_false") end_target = context.allocateLabel("condexpr_end") old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) generateConditionCode( condition = expression.getCondition(), emit = emit, context = context ) getLabelCode(true_target,emit) generateExpressionCode( to_name = to_name, expression = expression.getExpressionYes(), emit = emit, context = context ) needs_ref1 = context.needsCleanup(to_name) # Must not clean this up in other expression. if needs_ref1: context.removeCleanupTempName(to_name) real_emit = emit emit = SourceCodeCollector() generateExpressionCode( to_name = to_name, expression = expression.getExpressionNo(), emit = emit, context = context ) needs_ref2 = context.needsCleanup(to_name) # TODO: Need to buffer generated code, so we can emit extra reference if # not same. if needs_ref1 and not needs_ref2: getGotoCode(end_target, real_emit) getLabelCode(false_target, real_emit) for line in emit.codes: real_emit(line) emit = real_emit emit("Py_INCREF( %s );" % to_name) context.addCleanupTempName(to_name) elif not needs_ref1 and needs_ref2: real_emit("Py_INCREF( %s );" % to_name) getGotoCode(end_target, real_emit) getLabelCode(false_target, real_emit) for line in emit.codes: real_emit(line) emit = real_emit else: getGotoCode(end_target, real_emit) getLabelCode(false_target, real_emit) for line in emit.codes: real_emit(line) emit = real_emit getLabelCode(end_target,emit) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) Nuitka-0.5.18.1/nuitka/codegen/RaisingCodes.py0000644000175000017500000002601312651071040021252 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for implicit and explicit exception raises. Exceptions from other operations are consider ErrorCodes domain. """ from nuitka.Options import isDebug from .Helpers import generateChildExpressionsCode, generateExpressionCode from .LabelCodes import getGotoCode from .LineNumberCodes import emitErrorLineNumberUpdateCode from .PythonAPICodes import getReferenceExportCode2 def generateRaiseCode(statement, emit, context): exception_type = statement.getExceptionType() exception_value = statement.getExceptionValue() exception_tb = statement.getExceptionTrace() exception_cause = statement.getExceptionCause() context.markAsNeedsExceptionVariables() # Exception cause is only possible with simple raise form. if exception_cause is not None: assert exception_type is not None assert exception_value is None assert exception_tb is None raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) raise_cause_name = context.allocateTempName("raise_cause") generateExpressionCode( to_name = raise_cause_name, expression = exception_cause, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(exception_cause.getSourceReference()) getRaiseExceptionWithCauseCode( raise_type_name = raise_type_name, raise_cause_name = raise_cause_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif exception_type is None: assert exception_cause is None assert exception_value is None assert exception_tb is None getReRaiseExceptionCode( emit = emit, context = context ) elif exception_value is None and exception_tb is None: raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value = exception_type.getCompatibleSourceReference() ) getRaiseExceptionWithTypeCode( raise_type_name = raise_type_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif exception_tb is None: raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) raise_value_name = context.allocateTempName("raise_value") generateExpressionCode( to_name = raise_value_name, expression = exception_value, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(exception_value.getSourceReference()) context.setCurrentSourceCodeReference( statement.getCompatibleSourceReference() ) getRaiseExceptionWithValueCode( raise_type_name = raise_type_name, raise_value_name = raise_value_name, implicit = statement.isImplicit(), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) raise_value_name = context.allocateTempName("raise_value") generateExpressionCode( to_name = raise_value_name, expression = exception_value, emit = emit, context = context ) raise_tb_name = context.allocateTempName("raise_tb") generateExpressionCode( to_name = raise_tb_name, expression = exception_tb, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(exception_tb.getSourceReference()) getRaiseExceptionWithTracebackCode( raise_type_name = raise_type_name, raise_value_name = raise_value_name, raise_tb_name = raise_tb_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateRaiseExpressionCode(to_name, expression, emit, context): arg_names = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # Missed optimization opportunity, please report it, this should not # normally happen. We are supposed to propagate this upwards. if isDebug(): parent = expression.parent assert parent.isExpressionSideEffects() or \ parent.isExpressionConditional(), \ (expression, expression.parent) # That's how we indicate exception to the surrounding world. emit("%s = NULL;" % to_name) getRaiseExceptionWithValueCode( raise_type_name = arg_names[0], raise_value_name = arg_names[1], implicit = True, emit = emit, context = context ) def getReRaiseExceptionCode(emit, context): keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is None: emit( """\ RERAISE_EXCEPTION( &exception_type, &exception_value, &exception_tb );""" ) frame_handle = context.getFrameHandle() if frame_handle: emit( """\ if (exception_tb && exception_tb->tb_frame == %(frame_identifier)s) \ %(frame_identifier)s->f_lineno = exception_tb->tb_lineno;""" % { "frame_identifier" : context.getFrameHandle() } ) else: keeper_type, keeper_value, keeper_tb, keeper_lineno = context.getExceptionKeeperVariables() emit( """\ // Re-raise. exception_type = %(keeper_type)s; exception_value = %(keeper_value)s; exception_tb = %(keeper_tb)s; exception_lineno = %(keeper_lineno)s; """ % { "keeper_type" : keeper_type, "keeper_value" : keeper_value, "keeper_tb" : keeper_tb, "keeper_lineno" : keeper_lineno } ) getGotoCode(context.getExceptionEscape(), emit) def getRaiseExceptionWithCauseCode(raise_type_name, raise_cause_name, emit, context): context.markAsNeedsExceptionVariables() emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode2(raise_type_name, emit, context) emit("exception_value = NULL;") getReferenceExportCode2(raise_cause_name, emit, context) emitErrorLineNumberUpdateCode(emit, context) emit( """\ RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, \ %s );""" % raise_cause_name ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) if context.needsCleanup(raise_cause_name): context.removeCleanupTempName(raise_cause_name) def getRaiseExceptionWithTypeCode(raise_type_name, emit, context): context.markAsNeedsExceptionVariables() emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode2(raise_type_name, emit, context) emitErrorLineNumberUpdateCode(emit, context) emit( "RAISE_EXCEPTION_WITH_TYPE( &exception_type, &exception_value, &exception_tb );" ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) def getRaiseExceptionWithValueCode(raise_type_name, raise_value_name, implicit, emit, context): emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode2(raise_type_name, emit, context) emit( "exception_value = %s;" % raise_value_name ) getReferenceExportCode2(raise_value_name, emit, context) emitErrorLineNumberUpdateCode(emit, context) emit( "RAISE_EXCEPTION_%s( &exception_type, &exception_value, &exception_tb );" % ( ("IMPLICIT" if implicit else "WITH_VALUE") ) ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) if context.needsCleanup(raise_value_name): context.removeCleanupTempName(raise_value_name) def getRaiseExceptionWithTracebackCode(raise_type_name, raise_value_name, raise_tb_name, emit, context): emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode2(raise_type_name, emit, context) emit( "exception_value = %s;" % raise_value_name ) getReferenceExportCode2(raise_value_name, emit, context) emit( "exception_tb = (PyTracebackObject *)%s;" % raise_tb_name ) getReferenceExportCode2(raise_tb_name, emit, context) if False: # TODO: May be wrong, pylint: disable=W0125 emitErrorLineNumberUpdateCode(emit, context) emit( "RAISE_EXCEPTION_WITH_TRACEBACK( &exception_type, &exception_value, &exception_tb);" ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) if context.needsCleanup(raise_value_name): context.removeCleanupTempName(raise_value_name) if context.needsCleanup(raise_tb_name): context.removeCleanupTempName(raise_tb_name) Nuitka-0.5.18.1/nuitka/codegen/DictCodes.py0000644000175000017500000002226512651072046020555 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for dictionaries. """ from nuitka import Options from nuitka.PythonVersions import python_version from .ErrorCodes import getErrorExitBoolCode, getErrorExitCode, getReleaseCodes from .Helpers import generateChildExpressionsCode, generateExpressionCode def generateBuiltinDictCode(to_name, expression, emit, context): if expression.getPositionalArgument(): seq_name = context.allocateTempName("dict_seq") generateExpressionCode( to_name = seq_name, expression = expression.getPositionalArgument(), emit = emit, context = context, allow_none = True ) else: seq_name = None if expression.getNamedArgumentPairs(): # If there is no sequence to mix in, then directly generate # into to_name. if seq_name is None: getDictionaryCreationCode( to_name = to_name, pairs = expression.getNamedArgumentPairs(), emit = emit, context = context ) dict_name = None else: dict_name = context.allocateTempName("dict_arg") getDictionaryCreationCode( to_name = dict_name, pairs = expression.getNamedArgumentPairs(), emit = emit, context = context ) else: dict_name = None if seq_name is not None: emit( "%s = TO_DICT( %s, %s );" % ( to_name, seq_name, "NULL" if dict_name is None else dict_name ) ) getReleaseCodes( release_names = (seq_name, dict_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateDictionaryCreationCode(to_name, expression, emit, context): getDictionaryCreationCode( to_name = to_name, pairs = expression.getPairs(), emit = emit, context = context ) def getDictionaryCreationCode(to_name, pairs, emit, context): emit( "%s = _PyDict_NewPresized( %d );" % ( to_name, len(pairs) ) ) context.addCleanupTempName(to_name) def generateValueCode(dict_value_name, pair): generateExpressionCode( to_name = dict_value_name, expression = pair.getValue(), emit = emit, context = context ) def generateKeyCode(dict_key_name, pair): generateExpressionCode( to_name = dict_key_name, expression = pair.getKey(), emit = emit, context = context ) # Strange as it is, CPython evaluates the key/value pairs strictly in order, # but for each pair, the value first. for pair in pairs: dict_key_name = context.allocateTempName("dict_key") dict_value_name = context.allocateTempName("dict_value") if python_version < 350: generateValueCode(dict_value_name, pair) generateKeyCode(dict_key_name, pair) else: generateKeyCode(dict_key_name, pair) generateValueCode(dict_value_name, pair) if pair.getKey().isKnownToBeHashable(): emit( "PyDict_SetItem( %s, %s, %s );" % ( to_name, dict_key_name, dict_value_name ) ) else: res_name = context.getIntResName() emit( "%s = PyDict_SetItem( %s, %s, %s );" % ( res_name, to_name, dict_key_name, dict_value_name ) ) getErrorExitBoolCode( condition = "%s != 0" % res_name, emit = emit, context = context ) if context.needsCleanup(dict_value_name): emit("Py_DECREF( %s );" % dict_value_name) context.removeCleanupTempName(dict_value_name) if context.needsCleanup(dict_key_name): emit("Py_DECREF( %s );" % dict_key_name) context.removeCleanupTempName(dict_key_name) def getDictOperationRemoveCode(dict_name, key_name, emit, context): res_name = context.getBoolResName() emit( "%s = DICT_REMOVE_ITEM( %s, %s );" % ( res_name, dict_name, key_name ) ) getReleaseCodes( release_names = (dict_name, key_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def generateDictOperationUpdateCode(to_name, expression, emit, context): res_name = context.getIntResName() dict_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PyDict_Check( %s ) );" % dict_arg_name) emit( "%s = PyDict_Update( %s, %s );" % ( res_name, dict_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (dict_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) # Only assign if necessary. if context.isUsed(to_name): emit( "%s = Py_None;" % to_name ) else: context.forgetTempName(to_name) def generateDictOperationRemoveCode(statement, emit, context): if statement.isStatementDictOperationRemove(): dict_name = context.allocateTempName("remove_dict", unique = True) key_name = context.allocateTempName("remove_key", unique = True) generateExpressionCode( to_name = dict_name, expression = statement.getDict(), emit = emit, context = context ) generateExpressionCode( to_name = key_name, expression = statement.getKey(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( statement.getKey().getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getDictOperationRemoveCode( dict_name = dict_name, key_name = key_name, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(old_source_ref) def generateDictOperationGetCode(to_name, expression, emit, context): dict_name, key_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = DICT_GET_ITEM( %s, %s );" % ( to_name, dict_name, key_name ) ) getReleaseCodes( release_names = (dict_name, key_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateDictOperationSetCode(to_name, expression, emit, context): dict_name, key_name, value_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) res_name = context.getIntResName() emit( "%s = PyDict_SetItem( %s, %s, %s );" % ( res_name, dict_name, key_name, value_name ) ) getReleaseCodes( release_names = (dict_name, key_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s != 0" % res_name, emit = emit, context = context ) # Only assign if necessary. if context.isUsed(to_name): emit( "%s = Py_None;" % to_name ) else: context.forgetTempName(to_name) Nuitka-0.5.18.1/nuitka/codegen/GeneratorCodes.py0000644000175000017500000001751112651071040021607 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled function objects. """ from nuitka.PythonVersions import python_version from .ConstantCodes import getConstantCode from .ErrorCodes import ( getErrorVariableDeclarations, getExceptionKeeperVariableNames, getExceptionPreserverVariableNames ) from .Indentation import indented from .LineNumberCodes import getErrorLineNumberUpdateCode from .templates.CodeTemplatesFrames import template_generator_initial_throw from .templates.CodeTemplatesFunction import ( function_dict_setup, template_function_closure_making ) from .templates.CodeTemplatesGeneratorFunction import ( template_generator_exception_exit, template_generator_making_with_context, template_generator_making_without_context, template_generator_noexception_exit, template_generator_return_exit, template_genfunc_yielder_body_template, template_genfunc_yielder_decl_template ) from .VariableCodes import getLocalVariableInitCode, getVariableCode def getGeneratorObjectDeclCode(function_identifier): return template_genfunc_yielder_decl_template % { "function_identifier" : function_identifier, } def getGeneratorObjectCode(context, function_identifier, user_variables, temp_variables, function_codes, needs_exception_exit, needs_generator_return): function_locals = [] for user_variable in user_variables + temp_variables: function_locals.append( getLocalVariableInitCode( variable = user_variable, ) ) if context.hasLocalsDict(): function_locals += function_dict_setup.split('\n') if context.needsExceptionVariables(): function_locals.extend(getErrorVariableDeclarations()) for keeper_index in range(1, context.getKeeperVariableCount()+1): function_locals.extend(getExceptionKeeperVariableNames(keeper_index)) for preserver_id in context.getExceptionPreserverCounts(): function_locals.extend(getExceptionPreserverVariableNames(preserver_id)) function_locals += [ "%s%s%s;" % ( tmp_type, ' ' if not tmp_type.endswith('*') else "", tmp_name ) for tmp_name, tmp_type in context.getTempNameInfos() ] function_locals += context.getFrameDeclarations() # TODO: Could avoid this unless try/except or try/finally with returns # occur. if context.hasTempName("generator_return"): function_locals.append("tmp_generator_return = false;") if context.hasTempName("return_value"): function_locals.append("tmp_return_value = NULL;") for tmp_name, tmp_type in context.getTempNameInfos(): if tmp_name.startswith("tmp_outline_return_value_"): function_locals.append("%s = NULL;" % tmp_name) if needs_exception_exit: generator_exit = template_generator_exception_exit % {} else: generator_exit = template_generator_noexception_exit % {} if needs_generator_return: generator_exit += template_generator_return_exit % {} return template_genfunc_yielder_body_template % { "function_identifier" : function_identifier, "function_body" : indented(function_codes), "function_var_inits" : indented(function_locals), "generator_exit" : generator_exit } def generateMakeGeneratorObjectCode(to_name, expression, emit, context): generator_object_body = expression.getGeneratorRef().getFunctionBody() closure_variables = generator_object_body.getClosureVariables() if python_version < 350 or context.isForDirectCall(): generator_name_obj = getConstantCode( constant = generator_object_body.getFunctionName(), context = context ) else: generator_name_obj = "self->m_name" if python_version < 350: generator_qualname_obj = "NULL" elif not context.isForDirectCall(): generator_qualname_obj = "self->m_qualname" else: generator_qualname_obj = getConstantCode( constant = generator_object_body.getFunctionQualname(), context = context ) code_identifier = context.getCodeObjectHandle( code_object = expression.getCodeObject(), filename = generator_object_body.getParentModule().getRunTimeFilename(), line_number = generator_object_body.getSourceReference().getLineNumber(), is_optimized = True, new_locals = not generator_object_body.needsLocalsDict(), has_closure = len(closure_variables) > 0, future_flags = generator_object_body.getSourceReference().getFutureSpec().asFlags() ) if closure_variables: closure_copy = [] for count, variable in enumerate(closure_variables): variable_code = getVariableCode( context = context, variable = variable ) # Generators might not use them, but they still need to be put there. # TODO: But they don't have to be cells. if not variable.isSharedTechnically(): closure_copy.append( "closure[%d] = PyCell_NEW0( %s );" % ( count, variable_code ) ) else: closure_copy.append( "closure[%d] = %s;" % ( count, variable_code ) ) closure_copy.append( "Py_INCREF( closure[%d] );" % count ) closure_making = template_function_closure_making % { "closure_copy" : indented(closure_copy), "closure_count" : len(closure_variables) } emit( template_generator_making_with_context % { "closure_making" : closure_making, "to_name" : to_name, "generator_identifier" : generator_object_body.getCodeName(), "generator_name_obj" : generator_name_obj, "generator_qualname_obj" : generator_qualname_obj, "code_identifier" : code_identifier, "closure_count" : len(closure_variables) } ) else: emit( template_generator_making_without_context % { "to_name" : to_name, "generator_identifier" : generator_object_body.getCodeName(), "generator_name_obj" : generator_name_obj, "generator_qualname_obj" : generator_qualname_obj, "code_identifier" : code_identifier } ) context.addCleanupTempName(to_name) def generateGeneratorEntryCode(statement, emit, context): context.setCurrentSourceCodeReference(statement.getSourceReference()) emit( template_generator_initial_throw % { "frame_exception_exit" : context.getExceptionEscape(), "set_error_line_number" : getErrorLineNumberUpdateCode(context) } ) Nuitka-0.5.18.1/nuitka/codegen/IteratorCodes.py0000644000175000017500000001613412651071040021452 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Iteration related codes. Next variants and unpacking with related checks. """ from nuitka.PythonVersions import python_version from .ErrorCodes import ( getErrorExitCode, getErrorExitReleaseCode, getReleaseCode ) from .Helpers import generateChildExpressionsCode, generateExpressionCode from .Indentation import indented from .LineNumberCodes import getLineNumberUpdateCode from .PythonAPICodes import generateCAPIObjectCode from .templates.CodeTemplatesIterators import ( template_iterator_check, template_loop_break_next ) def generateBuiltinNext1Code(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = %s;" % ( to_name, "ITERATOR_NEXT( %s )" % value_name, ) ) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, quick_exception = "StopIteration", emit = emit, context = context ) context.addCleanupTempName(to_name) def getBuiltinLoopBreakNextCode(to_name, value, emit, context): emit( "%s = %s;" % ( to_name, "ITERATOR_NEXT( %s )" % value, ) ) getReleaseCode( release_name = value, emit = emit, context = context ) break_target = context.getLoopBreakTarget() if type(break_target) is tuple: break_indicator_code = "%s = true;" % break_target[1] break_target = break_target[0] else: break_indicator_code = "" emit( template_loop_break_next % { "to_name" : to_name, "break_indicator_code" : break_indicator_code, "break_target" : break_target, "release_temps" : indented( getErrorExitReleaseCode(context), 2 ), "line_number_code" : indented( getLineNumberUpdateCode(context), 2 ), "exception_target" : context.getExceptionEscape() } ) context.addCleanupTempName(to_name) def getUnpackNextCode(to_name, value, expected, count, emit, context): if python_version < 350: emit( "%s = UNPACK_NEXT( %s, %s );" % ( to_name, value, count - 1 ) ) else: emit( "%s = UNPACK_NEXT( %s, %s, %s );" % ( to_name, value, count - 1, expected ) ) getErrorExitCode( check_name = to_name, quick_exception = "StopIteration", emit = emit, context = context ) getReleaseCode( release_name = value, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateSpecialUnpackCode(to_name, expression, emit, context): value_name = context.allocateTempName("unpack") generateExpressionCode( to_name = value_name, expression = expression.getValue(), emit = emit, context = context ) getUnpackNextCode( to_name = to_name, value = value_name, count = expression.getCount(), expected = expression.getExpected(), emit = emit, context = context ) def generateUnpackCheckCode(statement, emit, context): iterator_name = context.allocateTempName("iterator_name") generateExpressionCode( to_name = iterator_name, expression = statement.getIterator(), emit = emit, context = context ) # These variable cannot collide, as it's used very locally. attempt_name = context.allocateTempName("iterator_attempt", unique = True) release_code = getErrorExitReleaseCode(context) emit( template_iterator_check % { "iterator_name" : iterator_name, "attempt_name" : attempt_name, "count" : statement.getCount(), "exception_exit" : context.getExceptionEscape(), "release_temps_1" : indented(release_code, 2), "release_temps_2" : indented(release_code), } ) getReleaseCode( release_name = iterator_name, emit = emit, context = context ) def generateBuiltinNext2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_NEXT2", arg_desc = ( ("next_arg", expression.getIterator()), ("next_default", expression.getDefault()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinIter1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "MAKE_ITERATOR", arg_desc = ( ("iter_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinIter2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_ITER2", arg_desc = ( ("iter_callable", expression.getCallable()), ("iter_sentinel", expression.getSentinel()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinLenCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_LEN", arg_desc = ( ("len_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/BuiltinCodes.py0000644000175000017500000002146612651071040021273 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in codes This is code generation for built-in references, and some built-ins like range, bin, etc. """ from nuitka import Builtins from .ConstantCodes import getConstantCode from .ErrorCodes import getAssertionCode, getErrorExitCode, getReleaseCodes from .Helpers import generateChildExpressionsCode from .PythonAPICodes import generateCAPIObjectCode, generateCAPIObjectCode0 def generateBuiltinRefCode(to_name, expression, emit, context): builtin_name = expression.getBuiltinName() emit( "%s = LOOKUP_BUILTIN( %s );" % ( to_name, getConstantCode( constant = builtin_name, context = context ) ) ) getAssertionCode( check = "%s != NULL" % to_name, emit = emit ) # Gives no reference def generateBuiltinAnonymousRefCode(to_name, expression, emit, context): # Functions used for generation all accept context, but this one does # not use it. pylint: disable=W0613 builtin_name = expression.getBuiltinName() emit( "%s = (PyObject *)%s;" % ( to_name, Builtins.builtin_anon_codes[builtin_name] ) ) def generateBuiltinType1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_TYPE1", arg_desc = ( ("type_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinType3Code(to_name, expression, emit, context): type_name, bases_name, dict_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = BUILTIN_TYPE3( %s, %s, %s, %s );" % ( to_name, getConstantCode( constant = context.getModuleName(), context = context ), type_name, bases_name, dict_name ), ) getReleaseCodes( release_names = (type_name, bases_name, dict_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinOpenCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_OPEN", arg_desc = ( ("open_filename", expression.getFilename()), ("open_mode", expression.getMode()), ("open_buffering", expression.getBuffering()), ), may_raise = expression.mayRaiseException(BaseException), none_null = True, source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinRange1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_RANGE", arg_desc = ( ("range_arg", expression.getLow()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinRange2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_RANGE2", arg_desc = ( ("range2_low", expression.getLow()), ("range2_high", expression.getHigh()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinRange3Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_RANGE3", arg_desc = ( ("range3_low", expression.getLow()), ("range3_high", expression.getHigh()), ("range3_step", expression.getStep()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinXrangeCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_XRANGE", arg_desc = ( ("xrange_low", expression.getLow()), ("xrange_high", expression.getHigh()), ("xrange_step", expression.getStep()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, none_null = True, ) def generateBuiltinFloatCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "TO_FLOAT", arg_desc = ( ("float_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinComplexCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "TO_COMPLEX", arg_desc = ( ("real_arg", expression.getReal()), ("imag_arg", expression.getImag()) ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinBoolCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "TO_BOOL", arg_desc = ( ("bool_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinBinCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_BIN", arg_desc = ( ("bin_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinOctCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_OCT", arg_desc = ( ("oct_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinHexCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_HEX", arg_desc = ( ("hex_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinBytearrayCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_BYTEARRAY", arg_desc = ( ("bytearray_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/LineNumberCodes.py0000644000175000017500000000454012651071040021717 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Generate code that updates the source code line. """ def getCurrentLineNumberCode(context): frame_handle = context.getFrameHandle() if frame_handle is None: return "" else: source_ref = context.getCurrentSourceCodeReference() if source_ref.isInternal(): return "" else: return str(source_ref.getLineNumber()) def getLineNumberUpdateCode(context): lineno_value = getCurrentLineNumberCode(context) if lineno_value: frame_handle = context.getFrameHandle() return "%s->f_lineno = %s;" % ( frame_handle, lineno_value ) else: return "" def getErrorLineNumberUpdateCode(context): lineno_value = getCurrentLineNumberCode(context) if lineno_value: return "exception_lineno = %s;" % ( lineno_value ) else: return "" def emitErrorLineNumberUpdateCode(emit, context): update_code = getErrorLineNumberUpdateCode(context) if update_code: emit(update_code) def emitLineNumberUpdateCode(emit, context): code = getLineNumberUpdateCode(context) if code: emit(code) def getSetLineNumberCodeRaw(to_name, emit, context): assert context.getFrameHandle() is not None emit( "%s->f_lineno = %s;" % ( context.getFrameHandle(), to_name ) ) def getLineNumberCode(to_name, emit, context): assert context.getFrameHandle() is not None emit( "%s = %s->f_lineno;" % ( to_name, context.getFrameHandle() ) ) Nuitka-0.5.18.1/nuitka/codegen/StringCodes.py0000644000175000017500000000713512651071040021130 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ String related codes, str and unicode. """ from nuitka.PythonVersions import python_version from .PythonAPICodes import generateCAPIObjectCode def generateBuiltinUnicodeCode(to_name, expression, emit, context): encoding = expression.getEncoding() errors = expression.getErrors() if encoding is None and errors is None: generateCAPIObjectCode( to_name = to_name, capi = "PyObject_Unicode", arg_desc = ( ( "str_arg" if python_version < 300 \ else "unicode_arg", expression.getValue() ), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) else: generateCAPIObjectCode( to_name = to_name, capi = "TO_UNICODE3", arg_desc = ( ("unicode_arg", expression.getValue()), ("unicode_encoding", encoding), ("unicode_errors", errors), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context, ) def generateBuiltinStrCode(to_name, expression, emit, context): if python_version < 300: generateCAPIObjectCode( to_name = to_name, capi = "PyObject_Str", arg_desc = ( ("str_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) else: return generateBuiltinUnicodeCode(to_name, expression, emit, context) def generateBuiltinChrCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_CHR", arg_desc = ( ("chr_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinOrdCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_ORD", arg_desc = ( ("ord_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/Emission.py0000644000175000017500000000246712651071040020475 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Emission of source code. Code generation is driven via "emit", which is to receive lines of code and this is to collect them, providing the emit implementation. Sometimes nested use of these will occur. """ class SourceCodeCollector: def __init__(self): self.codes = [] def __call__(self, code): self.emit(code) def emit(self,code): for line in code.split('\n'): self.codes.append(line) def emitTo(self, emit): for code in self.codes: emit(code) self.codes = None Nuitka-0.5.18.1/nuitka/codegen/LabelCodes.py0000644000175000017500000000423212651071040020674 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ C labels, small helpers. Much things are handled with "goto" statements in the generated code, error exits, finally blocks, etc. this provides just the means to emit a label or the goto statement itself. """ from .CppStrings import encodeString def getGotoCode(label, emit): assert label is not None emit( "goto %s;" % label ) def getLabelCode(label, emit): assert label is not None emit( "%s:;" % label ) def getBranchingCode(condition, emit, context): true_target = context.getTrueBranchTarget() false_target = context.getFalseBranchTarget() if true_target is not None and false_target is None: emit( "if ( %s ) goto %s;" % ( condition, true_target ) ) elif true_target is None and false_target is not None: emit( "if (!( %s )) goto %s;" % ( condition, false_target ) ) else: assert true_target is not None and false_target is not None emit( """\ if ( %s ) { goto %s; } else { goto %s; }""" % ( condition, true_target, false_target ) ) def getStatementTrace(source_desc, statement_repr): return 'puts( "Execute: " %s );' % ( encodeString(source_desc + b" " + statement_repr), ) Nuitka-0.5.18.1/nuitka/codegen/Namify.py0000644000175000017500000001544512651071040020132 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Namify constants. This determines the identifier names of constants in the generated code. We try to have readable names where possible, and resort to hash codes only when it is really necessary. """ import hashlib import math import re from logging import warning from nuitka.__past__ import long, unicode # pylint: disable=W0622 class ExceptionCannotNamify(Exception): pass def namifyConstant(constant): # Many branches, statements and every case has a return, this is a huge case # statement, that encodes the naming policy of constants, with often complex # conditions, pylint: disable=R0911,R0912,R0915 if type(constant) is int: if constant == 0: result = "int_0" elif constant > 0: result = "int_pos_%d" % constant else: result = "int_neg_%d" % abs(constant) if len(result) > 32: result = _digest(result) return result elif type(constant) is long: if constant == 0: result = "long_0" elif constant > 0: result = "long_pos_%d" % constant else: result = "long_neg_%d" % abs(constant) if len(result) > 32: result = _digest(result) return result elif constant is None: return "none" elif constant is True: return "true" elif constant is False: return "false" elif constant is Ellipsis: return "ellipsis" elif type(constant) is str: return "str_" + _namifyString(constant) elif type(constant) is bytes: return "bytes_" + _namifyString(constant) elif type(constant) is unicode: if _isAscii(constant): return "unicode_" + _namifyString(str(constant)) else: # Others are better digested to not cause compiler trouble return "unicode_digest_" + _digest(repr(constant)) elif type(constant) is float: if math.isnan(constant): return "float_%s_nan" % ( "minus" if math.copysign(1, constant) < 0 else "plus" ) return "float_%s" % repr(constant).replace('.', '_').\ replace('-', "_minus_").replace('+', "") elif type(constant) is complex: value = "%s__%s" % (constant.real, constant.imag) value = value.replace('+', 'p').replace('-', 'm').\ replace('.', '_') if value.startswith('(') and value.endswith(')'): value = value[1:-1] return "complex_%s" % value elif type(constant) is dict: if constant == {}: return "dict_empty" else: return "dict_" + _digest(repr(constant)) elif type(constant) is set: if constant == set(): return "set_empty" else: return "set_" + _digest(repr(constant)) elif type(constant) is frozenset: if constant == frozenset(): return "frozenset_empty" else: return "frozenset_" + _digest(repr(constant)) elif type(constant) is tuple: if constant == (): return "tuple_empty" else: try: result = '_'.join( namifyConstant(value) for value in constant ) if len(result) > 60: result = _digest(repr(constant)) return "tuple_" + result + "_tuple" except ExceptionCannotNamify: warning("Couldn't namify '%r'" % (constant,)) return "tuple_" + _digest(repr(constant)) elif type(constant) is list: if constant == []: return "list_empty" else: try: result = '_'.join( namifyConstant(value) for value in constant ) if len(result) > 60: result = _digest(repr(constant)) return "list_" + result + "_list" except ExceptionCannotNamify: warning("Couldn't namify '%r'" % value) return "list_" + _digest(repr(constant)) elif type(constant) is range: # Python3 type only. return "range_%s" % ( str(constant)[6:-1].replace(' ', "").replace(',', '_') ) elif type(constant) is slice: return "slice_%s_%s_%s" % ( namifyConstant(constant.start), namifyConstant(constant.stop), namifyConstant(constant.step) ) elif type(constant) is type: return "type_%s" % constant.__name__ raise ExceptionCannotNamify("%r" % constant) _re_str_needs_no_digest = re.compile(r"^([a-z]|[A-Z]|[0-9]|_){1,40}$", re.S) def _namifyString(string): # Many branches case has a return, encodes the naming policy of strings # constants, with often complex decisions to make, pylint: disable=R0911 if string in ("", b""): return "empty" elif string == ' ': return "space" elif string == '.': return "dot" elif string == '\n': return "newline" elif type(string) is str and \ _re_str_needs_no_digest.match(string) and \ '\n' not in string: # Some strings can be left intact for source code readability. return "plain_" + string elif len(string) == 1: return "chr_%d" % ord(string) elif len(string) > 2 and string[0] == '<' and string[-1] == '>' and \ _re_str_needs_no_digest.match(string[1:-1]) and \ '\n' not in string: return "angle_" + string[1:-1] else: # Others are better digested to not cause compiler trouble return "digest_" + _digest(repr(string)) def _isAscii(string): try: _unused = str(string) return True except UnicodeEncodeError: return False def _digest(value): if str is not unicode: return hashlib.md5(value).hexdigest() else: if type(value) is bytes: return hashlib.md5(value).hexdigest() else: return hashlib.md5(value.encode("utf-8")).hexdigest() Nuitka-0.5.18.1/nuitka/codegen/ModuleCodes.py0000644000175000017500000001234212651071040021103 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled module objects. """ from nuitka import Options from .CodeObjectCodes import getCodeObjectsDeclCode, getCodeObjectsInitCode from .ConstantCodes import ( allocateNestedConstants, getConstantCode, getConstantInitCodes ) from .ErrorCodes import ( getErrorVariableDeclarations, getExceptionKeeperVariableNames, getExceptionPreserverVariableNames ) from .Indentation import indented from .templates.CodeTemplatesModules import ( template_global_copyright, template_module_body_template, template_module_exception_exit, template_module_noexception_exit ) from .VariableCodes import getLocalVariableInitCode def getModuleAccessCode(context): return "module_%s" % context.getModuleCodeName() def getModuleValues(context, module_name, module_identifier, codes, function_decl_codes, function_body_codes, temp_variables, is_main_module, is_internal_module): # For the module code, lots of arguments and attributes come together. # pylint: disable=R0914 # Temporary variable initializations local_var_inits = [ getLocalVariableInitCode( variable = variable ) for variable in temp_variables ] if context.needsExceptionVariables(): local_var_inits.extend(getErrorVariableDeclarations()) for keeper_index in range(1, context.getKeeperVariableCount()+1): local_var_inits.extend(getExceptionKeeperVariableNames(keeper_index)) for preserver_id in context.getExceptionPreserverCounts(): local_var_inits.extend(getExceptionPreserverVariableNames(preserver_id)) local_var_inits += [ "%s%s%s;" % ( tmp_type, ' ' if not tmp_type.endswith('*') else "", tmp_name ) for tmp_name, tmp_type in context.getTempNameInfos() ] for tmp_name, tmp_type in context.getTempNameInfos(): if tmp_name.startswith("tmp_outline_return_value_"): local_var_inits.append("%s = NULL;" % tmp_name) local_var_inits += context.getFrameDeclarations() if context.needsExceptionVariables(): module_exit = template_module_exception_exit else: module_exit = template_module_noexception_exit module_body_template_values = { "module_name" : module_name, "module_name_obj" : getConstantCode( context = context, constant = module_name ), "is_main_module" : 1 if is_main_module else 0, "module_identifier" : module_identifier, "module_functions_decl" : function_decl_codes, "module_functions_code" : function_body_codes, "temps_decl" : indented(local_var_inits), "module_code" : indented(codes), "module_exit" : module_exit, "module_code_objects_decl" : indented( getCodeObjectsDeclCode(context), 0 ), "module_code_objects_init" : indented( getCodeObjectsInitCode(context), 1 ) } allocateNestedConstants(context) # Force internal module to not need constants init, by making all its # constants be shared. if is_internal_module: for constant in context.getConstants(): context.global_context.countConstantUse(constant) return module_body_template_values def getModuleCode(module_context, template_values): header = template_global_copyright % { "name" : module_context.getName(), "version" : Options.getVersion(), "year" : Options.getYear() } decls, inits, checks = getConstantInitCodes(module_context) if module_context.needsModuleFilenameObject(): decls.append("static PyObject *module_filename_obj;") template_values["constant_decl_codes"] = indented( decls, 0 ) template_values["constant_init_codes"] = indented( inits, 1 ) template_values["constant_check_codes"] = indented( checks, 1 ) return header + template_module_body_template % template_values def generateModuleFileAttributeCode(to_name, expression, emit, context): # The expression doesn't really matter, but it is part of the API for # the expression registry, pylint: disable=W0613 emit( "%s = module_filename_obj;" % ( to_name, ) ) context.markAsNeedsModuleFilenameObject() Nuitka-0.5.18.1/nuitka/codegen/VariableCodes.py0000644000175000017500000005367312651071040021417 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Low level variable code generation. """ from nuitka import Options, Variables from nuitka.PythonVersions import python_version from .ConstantCodes import getConstantCode from .Emission import SourceCodeCollector from .ErrorCodes import ( getAssertionCode, getErrorFormatExitBoolCode, getErrorFormatExitCode ) from .Helpers import generateExpressionCode from .Indentation import indented from .templates.CodeTemplatesVariables import ( template_del_global_unclear, template_del_local_intolerant, template_del_local_known, template_del_local_tolerant, template_del_shared_intolerant, template_del_shared_known, template_del_shared_tolerant, template_read_local, template_read_maybe_local_unclear, template_read_mvar_unclear, template_read_shared_known, template_read_shared_unclear, template_release_clear, template_release_unclear, template_write_local_clear_ref0, template_write_local_clear_ref1, template_write_local_empty_ref0, template_write_local_empty_ref1, template_write_local_inplace, template_write_local_unclear_ref0, template_write_local_unclear_ref1, template_write_shared_clear_ref0, template_write_shared_clear_ref1, template_write_shared_inplace, template_write_shared_unclear_ref0, template_write_shared_unclear_ref1 ) def generateAssignmentVariableCode(statement, emit, context): variable_ref = statement.getTargetVariableRef() value = statement.getAssignSource() tmp_name = context.allocateTempName("assign_source") generateExpressionCode( expression = value, to_name = tmp_name, emit = emit, context = context ) getVariableAssignmentCode( tmp_name = tmp_name, variable = variable_ref.getVariable(), needs_release = statement.needsReleasePreviousValue(), in_place = statement.inplace_suspect, emit = emit, context = context ) # Ownership of that reference must have been transfered. assert not context.needsCleanup(tmp_name) def generateDelVariableCode(statement, emit, context): old_source_ref = context.setCurrentSourceCodeReference( statement.getSourceReference() ) getVariableDelCode( variable = statement.getTargetVariableRef().getVariable(), tolerant = statement.isTolerant(), needs_check = statement.isTolerant() or \ statement.mayRaiseException(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateVariableReleaseCode(statement, emit, context): variable = statement.getVariable() if variable.isSharedTechnically(): # TODO: We might start to not allocate the cell object, then a check # would be due. But currently we always allocate it. needs_check = False else: needs_check = not statement.variable_trace.mustHaveValue() getVariableReleaseCode( variable = statement.getVariable(), needs_check = needs_check, emit = emit, context = context ) def generateVariableReferenceCode(to_name, expression, emit, context): getVariableAccessCode( to_name = to_name, variable = expression.getVariable(), needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) def getVariableCodeName(in_context, variable): if in_context: # Closure case: return "closure_" + variable.getCodeName() elif variable.isParameterVariable(): return "par_" + variable.getCodeName() elif variable.isTempVariable(): return "tmp_" + variable.getCodeName() else: return "var_" + variable.getCodeName() def _getLocalVariableCode(context, variable): # Now must be local or temporary variable, we return in each case- # pylint: disable=R0911 user = context.getOwner() owner = variable.getOwner() if owner is user: result = getVariableCodeName( in_context = False, variable = variable ) return result, variable.isSharedTechnically() elif context.isForDirectCall(): if user.isExpressionGeneratorObjectBody(): closure_index = user.getClosureVariables().index(variable) return "generator->m_closure[%d]" % closure_index, True elif user.isExpressionCoroutineObjectBody(): closure_index = user.getClosureVariables().index(variable) return "coroutine->m_closure[%d]" % closure_index, True else: result = getVariableCodeName( in_context = True, variable = variable ) return result, variable.isSharedTechnically() else: closure_index = user.getClosureVariables().index(variable) if user.isExpressionGeneratorObjectBody(): return "generator->m_closure[%d]" % closure_index, True elif user.isExpressionCoroutineObjectBody(): return "coroutine->m_closure[%d]" % closure_index, True else: return "self->m_closure[%d]" % closure_index, True def getVariableCode(context, variable): # Modules are simple. if variable.isModuleVariable(): return getVariableCodeName( in_context = False, variable = variable ) result, _is_cell = _getLocalVariableCode(context, variable) return result def getLocalVariableObjectAccessCode(context, variable): assert variable.isLocalVariable() code, is_cell = _getLocalVariableCode(context, variable) if is_cell: return code + "->ob_ref" else: return code def getLocalVariableInitCode(variable, init_from = None): assert not variable.isModuleVariable() if variable.isSharedTechnically(): type_name = "PyCellObject *" else: type_name = "PyObject *" code_name = getVariableCodeName( in_context = False, variable = variable ) if variable.isSharedTechnically(): # TODO: Single out "init_from" only user, so it becomes sure that we # get a reference transferred here in these cases. if init_from is not None: init_value = "PyCell_NEW1( %s )" % init_from else: init_value = "PyCell_EMPTY()" else: if init_from is None: init_from = "NULL" init_value = "%s" % init_from return "%s%s = %s;" % ( type_name, code_name, init_value ) def getVariableAssignmentCode(context, emit, variable, tmp_name, needs_release, in_place): # Many different cases, as this must be, pylint: disable=R0912,R0915 assert isinstance(variable, Variables.Variable), variable # For transfer of ownership. if context.needsCleanup(tmp_name): ref_count = 1 else: ref_count = 0 if variable.isModuleVariable(): emit( "UPDATE_STRING_DICT%s( moduledict_%s, (Nuitka_StringObject *)%s, %s );" % ( ref_count, context.getModuleCodeName(), getConstantCode( constant = variable.getName(), context = context ), tmp_name ) ) if ref_count: context.removeCleanupTempName(tmp_name) elif variable.isLocalVariable(): if in_place: # Releasing is not an issue here, local variable reference never # gave a reference, and the in-place code deals with possible # replacement/release. if variable.isSharedTechnically(): template = template_write_shared_inplace else: template = template_write_local_inplace elif variable.isSharedTechnically(): if ref_count: if needs_release is False: template = template_write_shared_clear_ref0 else: template = template_write_shared_unclear_ref0 else: if needs_release is False: template = template_write_shared_clear_ref1 else: template = template_write_shared_unclear_ref1 else: if ref_count: if needs_release is False: template = template_write_local_empty_ref0 elif needs_release is True: template = template_write_local_clear_ref0 else: template = template_write_local_unclear_ref0 else: if needs_release is False: template = template_write_local_empty_ref1 elif needs_release is True: template = template_write_local_clear_ref1 else: template = template_write_local_unclear_ref1 emit( template % { "identifier" : getVariableCode(context, variable), "tmp_name" : tmp_name } ) if ref_count: context.removeCleanupTempName(tmp_name) elif variable.isTempVariable(): if variable.isSharedTechnically(): if ref_count: template = template_write_shared_unclear_ref0 else: template = template_write_shared_unclear_ref1 else: if ref_count: if needs_release is False: template = template_write_local_empty_ref0 elif needs_release is True: template = template_write_local_clear_ref0 else: template = template_write_local_unclear_ref0 else: if needs_release is False: template = template_write_local_empty_ref1 elif needs_release is True: template = template_write_local_clear_ref1 else: template = template_write_local_unclear_ref1 emit( template % { "identifier" : getVariableCode(context, variable), "tmp_name" : tmp_name } ) if ref_count: context.removeCleanupTempName(tmp_name) else: assert False, variable def getVariableAccessCode(to_name, variable, needs_check, emit, context): # Many different cases, as this must be, pylint: disable=R0912 assert isinstance(variable, Variables.Variable), variable if variable.isModuleVariable(): emit( template_read_mvar_unclear % { "module_identifier" : context.getModuleCodeName(), "tmp_name" : to_name, "var_name" : getConstantCode( context = context, constant = variable.getName() ) } ) if needs_check: if python_version < 340 and \ not context.isCompiledPythonModule() and \ not context.getOwner().isExpressionClassBody(): error_message = "global name '%s' is not defined" else: error_message = "name '%s' is not defined" getErrorFormatExitCode( check_name = to_name, exception = "PyExc_NameError", args = ( error_message, variable.getName() ), emit = emit, context = context ) elif Options.isDebug(): emit("CHECK_OBJECT( %s );" % to_name) return elif variable.isMaybeLocalVariable(): fallback_emit = SourceCodeCollector() getVariableAccessCode( to_name = to_name, variable = variable.getMaybeVariable(), needs_check = True, emit = fallback_emit, context = context ) emit( template_read_maybe_local_unclear % { "locals_dict" : "locals_dict", "fallback" : indented(fallback_emit.codes), "tmp_name" : to_name, "var_name" : getConstantCode( context = context, constant = variable.getName() ) } ) return elif variable.isLocalVariable(): if variable.isSharedTechnically(): if not needs_check: template = template_read_shared_unclear else: template = template_read_shared_known emit( template % { "tmp_name" : to_name, "identifier" : getVariableCode(context, variable) } ) else: template = template_read_local emit( template % { "tmp_name" : to_name, "identifier" : getLocalVariableObjectAccessCode(context, variable) } ) if needs_check: if variable.getOwner() is not context.getOwner(): getErrorFormatExitCode( check_name = to_name, exception = "PyExc_NameError", args = ( """\ free variable '%s' referenced before assignment in enclosing scope""", variable.getName() ), emit = emit, context = context ) else: getErrorFormatExitCode( check_name = to_name, exception = "PyExc_UnboundLocalError", args = ( """\ local variable '%s' referenced before assignment""", variable.getName() ), emit = emit, context = context ) elif Options.isDebug(): emit("CHECK_OBJECT( %s );" % to_name) return elif variable.isTempVariable(): if variable.isSharedTechnically(): template = template_read_shared_unclear emit( template % { "tmp_name" : to_name, "identifier" : getVariableCode(context, variable) } ) if needs_check: getErrorFormatExitCode( check_name = to_name, exception = "PyExc_NameError", args = ( """\ free variable '%s' referenced before assignment in enclosing scope""", variable.getName() ), emit = emit, context = context ) elif Options.isDebug(): emit("CHECK_OBJECT( %s );" % to_name) return else: template = template_read_local emit( template % { "tmp_name" : to_name, "identifier" : getVariableCode(context, variable) } ) if needs_check: getErrorFormatExitCode( check_name = to_name, exception = "PyExc_UnboundLocalError", args = ( """\ local variable '%s' referenced before assignment""", variable.getName() ), emit = emit, context = context ) elif Options.isDebug(): emit("CHECK_OBJECT( %s );" % to_name) return assert False, variable def getVariableDelCode(variable, tolerant, needs_check, emit, context): # Many different cases, as this must be, pylint: disable=R0912 assert isinstance(variable, Variables.Variable), variable if variable.isModuleVariable(): check = not tolerant res_name = context.getIntResName() emit( template_del_global_unclear % { "module_identifier" : context.getModuleCodeName(), "res_name" : res_name, "var_name" : getConstantCode( context = context, constant = variable.getName() ) } ) # TODO: Apply needs_check for module variables too. if check: getErrorFormatExitBoolCode( condition = "%s == -1" % res_name, exception = "PyExc_NameError", args = ( "%sname '%s' is not defined" % ( "global " if not context.isCompiledPythonModule() else "", variable.getName() ), ), emit = emit, context = context ) elif variable.isLocalVariable(): if not needs_check: if variable.isSharedTechnically(): template = template_del_shared_known else: template = template_del_local_known emit( template % { "identifier" : getVariableCode( variable = variable, context = context ) } ) elif tolerant: if variable.isSharedTechnically(): template = template_del_shared_tolerant else: template = template_del_local_tolerant emit( template % { "identifier" : getVariableCode( variable = variable, context = context ) } ) else: res_name = context.getBoolResName() if variable.isSharedTechnically(): template = template_del_shared_intolerant else: template = template_del_local_intolerant emit( template % { "identifier" : getVariableCode( variable = variable, context = context ), "result" : res_name } ) if variable.getOwner() is context.getOwner(): getErrorFormatExitBoolCode( condition = "%s == false" % res_name, exception = "PyExc_UnboundLocalError", args = ("""\ local variable '%s' referenced before assignment""" % ( variable.getName() ), ), emit = emit, context = context ) else: getErrorFormatExitBoolCode( condition = "%s == false" % res_name, exception = "PyExc_NameError", args = ("""\ free variable '%s' referenced before assignment in enclosing scope""" % ( variable.getName() ), ), emit = emit, context = context ) elif variable.isTempVariable(): if tolerant: # Temp variables use similar classes, can use same templates. if variable.isSharedTechnically(): template = template_del_shared_tolerant else: template = template_del_local_tolerant emit( template % { "identifier" : getVariableCode( variable = variable, context = context ) } ) else: res_name = context.getBoolResName() if variable.isSharedTechnically(): template = template_del_shared_intolerant else: template = template_del_local_intolerant emit( template % { "identifier" : getVariableCode( variable = variable, context = context ), "result" : res_name } ) getAssertionCode( check = "%s != false" % res_name, emit = emit ) else: assert False, variable def getVariableReleaseCode(variable, needs_check, emit, context): assert isinstance(variable, Variables.Variable), variable assert not variable.isModuleVariable() # TODO: We could know, if we could loop, and only set the # variable to NULL then, using a different template. if needs_check: template = template_release_unclear else: template = template_release_clear emit( template % { "identifier" : getVariableCode( variable = variable, context = context ) } ) Nuitka-0.5.18.1/nuitka/codegen/ImportCodes.py0000644000175000017500000002046012651071040021130 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Import related codes. That is import as expression, and star import. """ from .ConstantCodes import getConstantCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .GlobalsLocalsCodes import getLoadGlobalsCode, getLoadLocalsCode from .Helpers import generateExpressionCode, generateExpressionsCode from .LineNumberCodes import emitLineNumberUpdateCode from .ModuleCodes import getModuleAccessCode def generateBuiltinImportCode(to_name, expression, emit, context): # We know that 5 expressions are created, pylint: disable=W0632 module_name, globals_name, locals_name, import_list_name, level_name = \ generateExpressionsCode( expressions = ( expression.getImportName(), expression.getGlobals(), expression.getLocals(), expression.getFromList(), expression.getLevel() ), names = ( "import_modulename", "import_globals", "import_locals", "import_fromlist", "import_level" ), emit = emit, context = context ) if expression.getGlobals() is None: globals_name = context.allocateTempName("import_globals") getLoadGlobalsCode( to_name = globals_name, emit = emit, context = context ) if expression.getLocals() is None: provider = expression.getParentVariableProvider() if provider.isCompiledPythonModule(): locals_name = globals_name else: locals_name = context.allocateTempName("import_locals") getLoadLocalsCode( to_name = locals_name, provider = provider, mode = provider.getLocalsMode(), emit = emit, context = context ) getBuiltinImportCode( to_name = to_name, module_name = module_name, globals_name = globals_name, locals_name = locals_name, import_list_name = import_list_name, level_name = level_name, emit = emit, context = context ) def getBuiltinImportCode(to_name, module_name, globals_name, locals_name, import_list_name, level_name, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = IMPORT_MODULE( %s, %s, %s, %s, %s );" % ( to_name, module_name, globals_name, locals_name, import_list_name, level_name ) ) getReleaseCodes( release_names = ( module_name, globals_name, locals_name, import_list_name, level_name ), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateImportModuleHardCode(to_name, expression, emit, context): getImportModuleHardCode( to_name = to_name, module_name = expression.getModuleName(), import_name = expression.getImportName(), needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) def getImportModuleHardCode(to_name, module_name, import_name, needs_check, emit, context): if module_name == "sys": emit( """%s = PySys_GetObject( (char *)"%s" );""" % ( to_name, import_name ) ) elif module_name in ("os", "__future__"): emit( """%s = PyObject_GetAttrString(PyImport_ImportModule("%s"), "%s");""" % ( to_name, module_name, import_name ) ) else: assert False, module_name if needs_check: getErrorExitCode( check_name = to_name, emit = emit, context = context ) def generateImportModuleCode(to_name, expression, emit, context): provider = expression.getParentVariableProvider() globals_name = context.allocateTempName("import_globals") getLoadGlobalsCode( to_name = globals_name, emit = emit, context = context ) if provider.isCompiledPythonModule(): locals_name = globals_name else: locals_name = context.allocateTempName("import_locals") getLoadLocalsCode( to_name = locals_name, provider = expression.getParentVariableProvider(), mode = "updated", emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(expression.getSourceReference()) getBuiltinImportCode( to_name = to_name, module_name = getConstantCode( constant = expression.getModuleName(), context = context ), globals_name = globals_name, locals_name = locals_name, import_list_name = getConstantCode( constant = expression.getImportList(), context = context ), level_name = getConstantCode( constant = expression.getLevel(), context = context ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateImportStarCode(statement, emit, context): module_name = context.allocateTempName("star_imported") generateImportModuleCode( to_name = module_name, expression = statement.getModule(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) res_name = context.getBoolResName() if not context.hasLocalsDict(): emit( "%s = IMPORT_MODULE_STAR( %s, true, %s );" % ( res_name, getModuleAccessCode( context = context ), module_name ) ) else: emit( "%s = IMPORT_MODULE_STAR( locals_dict, false, %s );" % ( res_name, module_name ) ) getReleaseCode( release_name = module_name, emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateImportNameCode(to_name, expression, emit, context): from_arg_name = context.allocateTempName("import_name_from") generateExpressionCode( to_name = from_arg_name, expression = expression.getModule(), emit = emit, context = context ) emit( "%s = IMPORT_NAME( %s, %s );" % ( to_name, from_arg_name, getConstantCode( constant = expression.getImportName(), context = context ) ) ) getReleaseCode( release_name = from_arg_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.18.1/nuitka/codegen/FunctionCodes.py0000644000175000017500000006102612651072046021455 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled function objects. """ from nuitka.PythonVersions import python_version from .ConstantCodes import getConstantCode from .CoroutineCodes import getCoroutineObjectDeclCode from .Emission import SourceCodeCollector from .ErrorCodes import ( getErrorExitCode, getErrorVariableDeclarations, getExceptionKeeperVariableNames, getExceptionPreserverVariableNames, getMustNotGetHereCode, getReleaseCode ) from .GeneratorCodes import getGeneratorObjectDeclCode from .Helpers import generateExpressionCode from .Indentation import indented from .LabelCodes import getLabelCode from .ModuleCodes import getModuleAccessCode from .PythonAPICodes import getReferenceExportCode from .templates.CodeTemplatesFunction import ( function_dict_setup, function_direct_body_template, template_function_body, template_function_closure_making, template_function_direct_declaration, template_function_exception_exit, template_function_make_declaration, template_function_return_exit, template_make_function_with_context_template, template_make_function_without_context_template ) from .TupleCodes import getTupleCreationCode from .VariableCodes import ( getLocalVariableInitCode, getVariableCode, getVariableCodeName ) def getClosureVariableProvisionCode(context, closure_variables): result = [] for variable in closure_variables: result.append( getVariableCode( context = context, variable = variable ) ) return result def _getFunctionCreationArgs(defaults_name, kw_defaults_name, annotations_name, closure_variables): result = [] if defaults_name is not None: result.append("PyObject *defaults") if kw_defaults_name is not None: result.append("PyObject *kw_defaults") if annotations_name is not None: result.append("PyObject *annotations") for closure_variable in closure_variables: result.append( "PyCellObject *%s" % ( getVariableCodeName( variable = closure_variable, in_context = True ) ) ) return result def getFunctionMakerDecl(function_identifier, defaults_name, kw_defaults_name, annotations_name, closure_variables): function_creation_arg_spec = _getFunctionCreationArgs( defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = closure_variables ) return template_function_make_declaration % { "function_identifier" : function_identifier, "function_creation_arg_spec" : ", ".join( function_creation_arg_spec ) } def getFunctionEntryPointIdentifier(function_identifier): return "impl_" + function_identifier def getFunctionMakerCode(function_name, function_qualname, function_identifier, code_identifier, closure_variables, defaults_name, kw_defaults_name, annotations_name, function_doc, context): # We really need this many parameters here and functions have many details, # that we express as variables, pylint: disable=R0914 function_creation_args = _getFunctionCreationArgs( defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = closure_variables ) if python_version < 330 or function_qualname == function_name: function_qualname_obj = "NULL" else: function_qualname_obj = getConstantCode( constant = function_qualname, context = context ) if closure_variables: closure_copy = [] for count, closure_variable in enumerate(closure_variables): closure_copy.append( "closure[%d] = %s;" % ( count, getVariableCodeName( True, closure_variable ) ) ) closure_copy.append( "Py_INCREF( closure[%d] );" %count ) closure_making = template_function_closure_making % { "closure_copy" : indented(closure_copy), "closure_count" : len(closure_variables) } result = template_make_function_with_context_template % { "function_name_obj" : getConstantCode( constant = function_name, context = context ), "function_qualname_obj" : function_qualname_obj, "function_identifier" : function_identifier, "function_impl_identifier" : getFunctionEntryPointIdentifier( function_identifier = function_identifier, ), "function_creation_args" : ", ".join( function_creation_args ), "code_identifier" : code_identifier, "closure_making" : closure_making, "function_doc" : getConstantCode( constant = function_doc, context = context ), "defaults" : "defaults" if defaults_name else "NULL", "kw_defaults" : "kw_defaults" if kw_defaults_name else "NULL", "annotations" : "annotations" if annotations_name else context.getConstantCode({}), "closure_count" : len(closure_variables), "module_identifier" : getModuleAccessCode( context = context ), } else: result = template_make_function_without_context_template % { "function_name_obj" : getConstantCode( constant = function_name, context = context ), "function_qualname_obj" : function_qualname_obj, "function_identifier" : function_identifier, "function_impl_identifier" : getFunctionEntryPointIdentifier( function_identifier = function_identifier, ), "function_creation_args" : ", ".join( function_creation_args ), "code_identifier" : code_identifier, "function_doc" : getConstantCode( constant = function_doc, context = context ), "defaults" : "defaults" if defaults_name else "NULL", "kw_defaults" : "kw_defaults" if kw_defaults_name else "NULL", "annotations" : "annotations" if annotations_name else context.getConstantCode({}), "module_identifier" : getModuleAccessCode( context = context ), } return result def generateFunctionCreationCode(to_name, expression, emit, context): # This is about creating functions, which is detail ridden stuff, # pylint: disable=R0914 function_body = expression.getFunctionRef().getFunctionBody() code_object = expression.getCodeObject() defaults = expression.getDefaults() kw_defaults = expression.getKwDefaults() annotations = expression.getAnnotations() defaults_first = not expression.kw_defaults_before_defaults assert function_body.needsCreation(), function_body def handleKwDefaults(): if kw_defaults: kw_defaults_name = context.allocateTempName("kw_defaults") assert not kw_defaults.isExpressionConstantRef() or \ kw_defaults.getConstant() != {}, kw_defaults.getConstant() generateExpressionCode( to_name = kw_defaults_name, expression = kw_defaults, emit = emit, context = context ) else: kw_defaults_name = None return kw_defaults_name def handleDefaults(): if defaults: defaults_name = context.allocateTempName("defaults") getTupleCreationCode( to_name = defaults_name, elements = defaults, emit = emit, context = context ) else: defaults_name = None return defaults_name if defaults_first: defaults_name = handleDefaults() kw_defaults_name = handleKwDefaults() else: kw_defaults_name = handleKwDefaults() defaults_name = handleDefaults() if annotations: annotations_name = context.allocateTempName("annotations") generateExpressionCode( to_name = annotations_name, expression = annotations, emit = emit, context = context, ) else: annotations_name = None function_identifier = function_body.getCodeName() # Creation code needs to be done only once. if not context.hasHelperCode(function_identifier): code_identifier = context.getCodeObjectHandle( code_object = code_object, filename = function_body.getParentModule().getRunTimeFilename(), line_number = function_body.getSourceReference().getLineNumber(), is_optimized = not function_body.needsLocalsDict(), new_locals = True, has_closure = function_body.getClosureVariables() != (), future_flags = function_body.getSourceReference().getFutureSpec().asFlags() ) maker_code = getFunctionMakerCode( function_name = function_body.getFunctionName(), function_qualname = function_body.getFunctionQualname(), function_identifier = function_identifier, code_identifier = code_identifier, closure_variables = function_body.getClosureVariables(), defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, function_doc = function_body.getDoc(), context = context ) context.addHelperCode(function_identifier, maker_code) function_decl = getFunctionMakerDecl( function_identifier = function_body.getCodeName(), defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = function_body.getClosureVariables() ) context.addDeclaration(function_identifier, function_decl) getFunctionCreationCode( to_name = to_name, function_identifier = function_body.getCodeName(), defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = function_body.getClosureVariables(), emit = emit, context = context ) getReleaseCode( release_name = annotations_name, emit = emit, context = context ) def getFunctionCreationCode(to_name, function_identifier, defaults_name, kw_defaults_name, annotations_name, closure_variables, emit, context): args = [] if defaults_name is not None: args.append(getReferenceExportCode(defaults_name, context)) if kw_defaults_name is not None: args.append(kw_defaults_name) if annotations_name is not None: args.append(annotations_name) args += getClosureVariableProvisionCode( context = context, closure_variables = closure_variables ) emit( "%s = MAKE_FUNCTION_%s( %s );" % ( to_name, function_identifier, ", ".join(args) ) ) if context.needsCleanup(defaults_name): context.removeCleanupTempName(defaults_name) if context.needsCleanup(kw_defaults_name): context.removeCleanupTempName(kw_defaults_name) # No error checks, this supposedly, cannot fail. context.addCleanupTempName(to_name) def getDirectFunctionCallCode(to_name, function_identifier, arg_names, closure_variables, needs_check, emit, context): function_identifier = getFunctionEntryPointIdentifier( function_identifier = function_identifier ) suffix_args = getClosureVariableProvisionCode( context = context, closure_variables = closure_variables ) # TODO: We ought to not assume references for direct calls, or make a # profile if an argument needs a reference at all. Most functions don't # bother to release a called argument by "del" or assignment to it. We # could well know that ahead of time. for arg_name in arg_names: if context.needsCleanup(arg_name): context.removeCleanupTempName(arg_name) else: emit("Py_INCREF( %s );" % arg_name) if arg_names: emit( """ { PyObject *dir_call_args[] = {%s}; %s = %s( dir_call_args%s%s ); }""" % ( ", ".join( arg_names ), to_name, function_identifier, ", " if suffix_args else "", ", ".join(suffix_args) ) ) else: emit( "%s = %s( NULL%s%s );" % ( to_name, function_identifier, ", " if suffix_args else "", ", ".join(suffix_args) ) ) # Arguments are owned to the called in direct function call. for arg_name in arg_names: if context.needsCleanup(arg_name): context.removeCleanupTempName(arg_name) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) def getFunctionDirectClosureArgs(closure_variables): result = [] for closure_variable in closure_variables: if closure_variable.isSharedTechnically(): result.append( "PyCellObject *%s" % ( getVariableCodeName( in_context = True, variable = closure_variable ) ) ) else: # TODO: The reference is only needed for Python3, could make it # version dependent. result.append( "PyObject *&%s" % ( getVariableCodeName( in_context = True, variable = closure_variable ) ) ) return result def getFunctionDirectDecl(function_identifier, closure_variables, file_scope): parameter_objects_decl = [ "PyObject **python_pars" ] parameter_objects_decl += getFunctionDirectClosureArgs(closure_variables) result = template_function_direct_declaration % { "file_scope" : file_scope, "function_identifier" : function_identifier, "direct_call_arg_spec" : ", ".join(parameter_objects_decl), } return result def getFunctionCode(context, function_identifier, parameters, closure_variables, user_variables, temp_variables, function_codes, function_doc, file_scope, needs_exception_exit): # Functions have many details, that we express as variables, with many # branches to decide, pylint: disable=R0912,R0914 function_locals = [] if context.hasLocalsDict(): function_locals += function_dict_setup.split('\n') function_cleanup = "Py_DECREF( locals_dict );\n" else: function_cleanup = "" if parameters is not None: for count, variable in enumerate(parameters.getAllVariables()): function_locals.append( getLocalVariableInitCode( variable = variable, init_from = "python_pars[ %d ]" % count ) ) # User local variable initializations function_locals += [ getLocalVariableInitCode( variable = variable, ) for variable in user_variables + tuple( variable for variable in temp_variables ) ] if context.needsExceptionVariables(): function_locals.extend(getErrorVariableDeclarations()) for keeper_index in range(1, context.getKeeperVariableCount()+1): function_locals.extend(getExceptionKeeperVariableNames(keeper_index)) for preserver_id in context.getExceptionPreserverCounts(): function_locals.extend(getExceptionPreserverVariableNames(preserver_id)) function_locals += [ "%s%s%s;" % ( tmp_type, ' ' if not tmp_type.endswith('*') else "", tmp_name ) for tmp_name, tmp_type in context.getTempNameInfos() ] function_locals += context.getFrameDeclarations() # TODO: Could avoid this unless try/except or try/finally with returns # occur. if context.hasTempName("return_value"): function_locals.append("tmp_return_value = NULL;") for tmp_name, tmp_type in context.getTempNameInfos(): if tmp_name.startswith("tmp_outline_return_value_"): function_locals.append("%s = NULL;" % tmp_name) function_doc = getConstantCode( context = context, constant = function_doc ) result = "" emit = SourceCodeCollector() getMustNotGetHereCode( reason = "Return statement must have exited already.", context = context, emit = emit ) function_exit = indented(emit.codes) + "\n\n" del emit if needs_exception_exit: function_exit += template_function_exception_exit % { "function_cleanup" : function_cleanup, } if context.hasTempName("return_value"): function_exit += indented( template_function_return_exit % { "function_cleanup" : indented(function_cleanup), } ) if context.isForCreatedFunction(): parameter_objects_decl = ["Nuitka_FunctionObject const *self"] else: parameter_objects_decl = [] parameter_objects_decl += [ "PyObject **python_pars" ] if context.isForDirectCall(): parameter_objects_decl += getFunctionDirectClosureArgs(closure_variables) result += function_direct_body_template % { "file_scope" : file_scope, "function_identifier" : function_identifier, "direct_call_arg_spec" : ", ".join( parameter_objects_decl ), "function_locals" : indented(function_locals), "function_body" : indented(function_codes), "function_exit" : function_exit } else: result += template_function_body % { "function_identifier" : function_identifier, "parameter_objects_decl" : ", ".join(parameter_objects_decl), "function_locals" : indented(function_locals), "function_body" : indented(function_codes), "function_exit" : function_exit } return result def getExportScopeCode(cross_module): if cross_module: return "NUITKA_CROSS_MODULE" else: return "NUITKA_LOCAL_MODULE" def generateFunctionDeclCode(function_body): if function_body.isExpressionGeneratorObjectBody(): return getGeneratorObjectDeclCode( function_identifier = function_body.getCodeName(), ) elif function_body.isExpressionCoroutineObjectBody(): return getCoroutineObjectDeclCode( function_identifier = function_body.getCodeName(), ) elif function_body.isExpressionClassBody(): return getFunctionDirectDecl( function_identifier = function_body.getCodeName(), closure_variables = function_body.getClosureVariables(), file_scope = getExportScopeCode( cross_module = False ) ) elif function_body.needsDirectCall(): return getFunctionDirectDecl( function_identifier = function_body.getCodeName(), closure_variables = function_body.getClosureVariables(), file_scope = getExportScopeCode( cross_module = function_body.isCrossModuleUsed() ) ) else: return None def generateFunctionCallCode(to_name, expression, emit, context): assert expression.getFunction().isExpressionFunctionCreation() function_body = expression.getFunction().getFunctionRef().getFunctionBody() function_identifier = function_body.getCodeName() argument_values = expression.getArgumentValues() arg_names = [] for count, arg_value in enumerate(argument_values): arg_name = context.allocateTempName("dircall_arg%d" % (count+1)) generateExpressionCode( to_name = arg_name, expression = arg_value, emit = emit, context = context ) arg_names.append(arg_name) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getDirectFunctionCallCode( to_name = to_name, function_identifier = function_identifier, arg_names = arg_names, closure_variables = function_body.getClosureVariables(), needs_check = expression.getFunction().getFunctionRef().\ getFunctionBody().mayRaiseException(BaseException), emit = emit, context = context ) def generateFunctionOutlineCode(to_name, expression, emit, context): assert expression.isExpressionOutlineBody() # Need to set return target, to assign to_name from. old_return_release_mode = context.getReturnReleaseMode() return_target = context.allocateLabel("outline_result") old_return_target = context.setReturnTarget(return_target) return_value_name = context.allocateTempName("outline_return_value") old_return_value_name = context.setReturnValueName(return_value_name) from .CodeGeneration import generateStatementSequenceCode generateStatementSequenceCode( statement_sequence = expression.getBody(), emit = emit, context = context, allow_none = False ) getMustNotGetHereCode( reason = "Return statement must have exited already.", context = context, emit = emit ) getLabelCode(return_target, emit) emit( "%s = %s;" % ( to_name, return_value_name ) ) context.addCleanupTempName(to_name) # Restore previous "return" handling. context.setReturnTarget(old_return_target) context.setReturnReleaseMode(old_return_release_mode) context.setReturnValueName(old_return_value_name) Nuitka-0.5.18.1/nuitka/codegen/SetCodes.py0000644000175000017500000001103212651072046020413 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for sets. Right now only the creation, and set add code is done here. But more should be added later on. """ from .ErrorCodes import getErrorExitBoolCode, getReleaseCodes from .Helpers import generateChildExpressionsCode, generateExpressionCode from .PythonAPICodes import generateCAPIObjectCode def generateSetCreationCode(to_name, expression, emit, context): emit( "%s = PySet_New( NULL );" % ( to_name, ) ) context.addCleanupTempName(to_name) element_name = context.allocateTempName("set_element") for element in expression.getElements(): generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) if element.isKnownToBeHashable(): emit( "PySet_Add( %s, %s );" % ( to_name, element_name ) ) else: res_name = context.getIntResName() emit( "%s = PySet_Add( %s, %s );" % ( res_name, to_name, element_name ) ) getErrorExitBoolCode( condition = "%s != 0" % res_name, emit = emit, context = context ) if context.needsCleanup(element_name): emit("Py_DECREF( %s );" % element_name) context.removeCleanupTempName(element_name) def generateSetOperationAddCode(to_name, expression, emit, context): res_name = context.getIntResName() set_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PySet_Check( %s ) );" % set_arg_name) emit( "%s = PySet_Add( %s, %s );" % ( res_name, set_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (set_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) # Only assign if necessary. if context.isUsed(to_name): emit( "%s = Py_None;" % to_name ) else: context.forgetTempName(to_name) def generateSetOperationUpdateCode(to_name, expression, emit, context): res_name = context.getIntResName() set_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PySet_Check( %s ) );" % set_arg_name) emit( "%s = _PySet_Update( %s, %s );" % ( res_name, set_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (set_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) # Only assign if necessary. if context.isUsed(to_name): emit( "%s = Py_None;" % to_name ) else: context.forgetTempName(to_name) def generateBuiltinSetCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PySet_New", arg_desc = ( ("set_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/MainCodes.py0000644000175000017500000000717212651071040020547 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Main program code generation. This is for the actual entry point code, which is mostly fed by a template, but also customized through a lot of runtime configuration values. Examples of these are sys.executable, and sys.flags, but of course also the frame object data (filename, etc). """ import os import sys from nuitka import Options from .ConstantCodes import getModuleConstantCode from .templates.CodeTemplatesMain import template_main_program def generateMainCode(main_module, codes, context): python_flags = Options.getPythonFlags() if context.mayRaiseException(): code_identifier = context.getCodeObjectHandle( code_object = main_module.getCodeObject(), filename = main_module.getRunTimeFilename(), line_number = 1, is_optimized = False, new_locals = False, has_closure = False, future_flags = main_module.getSourceReference().getFutureSpec().\ asFlags() ) else: code_identifier = "NULL" main_code = template_main_program % { "sys_executable" : getModuleConstantCode( constant = sys.executable, ), "python_sysflag_debug" : sys.flags.debug, "python_sysflag_py3k_warning" : ( sys.flags.py3k_warning if hasattr(sys.flags, "py3k_warning") else 0 ), "python_sysflag_division_warning" : ( sys.flags.division_warning if hasattr(sys.flags, "division_warning") else 0 ), #"python_sysflag_division_new" : sys.flags.division_new, #not supported "python_sysflag_inspect" : sys.flags.inspect, "python_sysflag_interactive" : sys.flags.interactive, "python_sysflag_optimize" : sys.flags.optimize, "python_sysflag_dont_write_bytecode" : sys.flags.dont_write_bytecode, "python_sysflag_no_site" : os.environ.get( "NUITKA_SITE_FLAG", 1 if "no_site" in Options.getPythonFlags() else 0 ), "python_sysflag_no_user_site" : sys.flags.no_user_site, "python_sysflag_ignore_environment" : sys.flags.ignore_environment, "python_sysflag_tabcheck" : ( sys.flags.tabcheck if hasattr(sys.flags, "tabcheck") else 0 ), "python_sysflag_verbose" : 1 if "trace_imports" in python_flags else 0, "python_sysflag_unicode" : ( sys.flags.unicode if hasattr(sys.flags, "unicode") else 0 ), "python_sysflag_bytes_warning" : sys.flags.bytes_warning, "python_sysflag_hash_randomization" : ( sys.flags.hash_randomization if (hasattr(sys.flags, "hash_randomization") and \ "no_randomization" not in python_flags) else 0 ), "python_no_warnings" : 1 if "no_warnings" in python_flags else 0, "code_identifier" : code_identifier } return codes + main_code Nuitka-0.5.18.1/nuitka/codegen/ReturnCodes.py0000644000175000017500000000656412651071040021146 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Return codes This handles code generation for return statements of normal functions and of generator functions. Also the value currently being returned, and intercepted by a try statement is accessible this way. """ from nuitka.PythonVersions import python_version from .ExceptionCodes import getExceptionUnpublishedReleaseCode from .Helpers import generateExpressionCode from .LabelCodes import getGotoCode def generateReturnCode(statement, emit, context): getExceptionUnpublishedReleaseCode(emit, context) return_value = statement.getExpression() if not return_value.isExpressionReturnedValueRef(): return_value_name = context.getReturnValueName() if context.getReturnReleaseMode(): emit("Py_DECREF( %s );" % return_value_name) generateExpressionCode( to_name = return_value_name, expression = return_value, emit = emit, context = context ) if context.needsCleanup(return_value_name): context.removeCleanupTempName(return_value_name) else: emit( "Py_INCREF( %s );" % return_value_name ) getGotoCode( label = context.getReturnTarget(), emit = emit ) def generateReturnedValueRefCode(to_name, expression, emit, context): # We don't need the expression, pylint: disable=W0613 return_value_name = context.getReturnValueName() emit( "%s = %s;" % ( to_name, return_value_name ) ) def generateGeneratorReturnCode(statement, emit, context): if python_version >= 330: return_value_name = context.getGeneratorReturnValueName() expression = statement.getExpression() if context.getReturnReleaseMode(): emit("Py_DECREF( %s );" % return_value_name) generateExpressionCode( to_name = return_value_name, expression = expression, emit = emit, context = context ) if context.needsCleanup(return_value_name): context.removeCleanupTempName(return_value_name) else: emit( "Py_INCREF( %s );" % return_value_name ) elif statement.getParentVariableProvider().needsGeneratorReturnHandling(): return_value_name = context.getGeneratorReturnValueName() generator_return_name = context.allocateTempName( "generator_return", "bool", unique = True ) emit("%s = true;" % generator_return_name) getGotoCode(context.getReturnTarget(), emit) Nuitka-0.5.18.1/nuitka/codegen/ComparisonCodes.py0000644000175000017500000002151012651072046021774 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Comparison related codes. Rich comparisons, "in", and "not in", also "is", and "is not", and the "isinstance" check as used in conditions, as well as exception matching. """ from . import OperatorCodes from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .Helpers import generateExpressionCode from .LabelCodes import getBranchingCode def generateComparisonExpressionCode(to_name, expression, emit, context): left_name = context.allocateTempName("compexpr_left") right_name = context.allocateTempName("compexpr_right") generateExpressionCode( to_name = left_name, expression = expression.getLeft(), emit = emit, context = context ) generateExpressionCode( to_name = right_name, expression = expression.getRight(), emit = emit, context = context ) getComparisonExpressionCode( to_name = to_name, comparator = expression.getComparator(), left_name = left_name, right_name = right_name, needs_check = expression.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) def getComparisonExpressionCode(to_name, comparator, left_name, right_name, needs_check, emit, context): if comparator in OperatorCodes.normal_comparison_codes: helper = OperatorCodes.normal_comparison_codes[ comparator ] assert helper.startswith("SEQUENCE_CONTAINS") emit( "%s = %s( %s, %s );" % ( to_name, helper, left_name, right_name ) ) getReleaseCode( release_name = left_name, emit = emit, context = context ) getReleaseCode( release_name = right_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) elif comparator in OperatorCodes.rich_comparison_codes: helper = "RICH_COMPARE_%s" % ( OperatorCodes.rich_comparison_codes[ comparator ] ) if not context.mayRecurse() and comparator == "Eq": helper += "_NORECURSE" emit( "%s = %s( %s, %s );" % ( to_name, helper, left_name, right_name ) ) getReleaseCodes( release_names = (left_name, right_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) elif comparator == "Is": emit( "%s = BOOL_FROM( %s == %s );" % ( to_name, left_name, right_name ) ) getReleaseCodes( release_names = (left_name, right_name), emit = emit, context = context ) elif comparator == "IsNot": emit( "%s = BOOL_FROM( %s != %s );" % ( to_name, left_name, right_name ) ) getReleaseCodes( release_names = (left_name, right_name), emit = emit, context = context ) elif comparator == "exception_match": operator_res_name = context.allocateTempName( "cmp_exception_match", "int" ) emit( "%s = EXCEPTION_MATCH_BOOL( %s, %s );" % ( operator_res_name, left_name, right_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, needs_check = needs_check, emit = emit, context = context ) emit( "%s = BOOL_FROM( %s != 0 );" % ( to_name, operator_res_name ) ) else: assert False, comparator def getComparisonExpressionBoolCode(comparator, left_name, right_name, needs_check, emit, context): if comparator in OperatorCodes.normal_comparison_codes: operator_res_name = context.allocateTempName("cmp_" + comparator, "int") emit( "%s = PySequence_Contains( %s, %s );" % ( operator_res_name, right_name, # sequence goes first. left_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, emit = emit, needs_check = needs_check, context = context ) condition = "%s == %d" % ( operator_res_name, 1 if comparator == "In" else 0 ) elif comparator in OperatorCodes.rich_comparison_codes: operator_res_name = context.allocateTempName("cmp_" + comparator, "int") helper = OperatorCodes.rich_comparison_codes[comparator] if not context.mayRecurse() and comparator == "Eq": helper += "_NORECURSE" emit( "%s = RICH_COMPARE_BOOL_%s( %s, %s );" % ( operator_res_name, helper, left_name, right_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, needs_check = needs_check, emit = emit, context = context ) condition = "%s == 1" % ( operator_res_name, ) elif comparator == "Is": operator_res_name = context.allocateTempName("is", "bool") emit( "%s = ( %s == %s );" % ( operator_res_name, left_name, right_name ) ) condition = operator_res_name elif comparator == "IsNot": operator_res_name = context.allocateTempName("isnot", "bool") emit( "%s = ( %s != %s );" % ( operator_res_name, left_name, right_name ) ) condition = operator_res_name elif comparator == "exception_match": operator_res_name = context.allocateTempName("exc_match_" + comparator, "int") emit( "%s = EXCEPTION_MATCH_BOOL( %s, %s );" % ( operator_res_name, left_name, right_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, needs_check = needs_check, emit = emit, context = context ) condition = "%s == 1" % ( operator_res_name ) else: assert False, comparator getReleaseCode( release_name = left_name, emit = emit, context = context ) getReleaseCode( release_name = right_name, emit = emit, context = context ) getBranchingCode(condition, emit, context) def getBuiltinIsinstanceBoolCode(inst_name, cls_name, emit, context): res_name = context.getIntResName() emit( "%s = Nuitka_IsInstance( %s, %s );" % ( res_name, inst_name, cls_name ) ) getReleaseCodes( release_names = (inst_name, cls_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) getBranchingCode( condition = "%s == 1" % res_name, emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/PythonAPICodes.py0000644000175000017500000001036112651071040021470 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for standard CPython/API calls. This is generic stuff. """ from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .Helpers import generateExpressionCode def generateCAPIObjectCodeCommon(to_name, capi, arg_desc, may_raise, ref_count, source_ref, emit, context, none_null = False): arg_names = [] for arg_name, arg_expression in arg_desc: if arg_expression is None and none_null: arg_names.append("NULL") else: arg_name = context.allocateTempName(arg_name) generateExpressionCode( to_name = arg_name, expression = arg_expression, emit = emit, context = context ) arg_names.append(arg_name) context.setCurrentSourceCodeReference(source_ref) getCAPIObjectCode( to_name = to_name, capi = capi, arg_names = arg_names, may_raise = may_raise, ref_count = ref_count, emit = emit, context = context ) def generateCAPIObjectCode(to_name, capi, arg_desc, may_raise, source_ref, emit, context, none_null = False): generateCAPIObjectCodeCommon( to_name = to_name, capi = capi, arg_desc = arg_desc, may_raise = may_raise, ref_count = 1, source_ref = source_ref, emit = emit, context = context, none_null = none_null ) def generateCAPIObjectCode0(to_name, capi, arg_desc, may_raise, source_ref, emit, context, none_null = False): generateCAPIObjectCodeCommon( to_name = to_name, capi = capi, arg_desc = arg_desc, may_raise = may_raise, ref_count = 0, source_ref = source_ref, emit = emit, context = context, none_null = none_null ) def getCAPIObjectCode(to_name, capi, arg_names, may_raise, ref_count, emit, context): emit( "%s = %s( %s );" % ( to_name, capi, ", ".join(arg_names) ) ) getReleaseCodes( release_names = ( arg_name for arg_name in arg_names if arg_name != "NULL" ), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = may_raise, emit = emit, context = context ) if ref_count: context.addCleanupTempName(to_name) def getCAPIIntCode(res_name, capi, args, emit, context): emit( "%s = %s( %s );" % ( res_name, capi, ", ".join(args) ) ) # TODO: Order, potentially for arg in args: getReleaseCode( release_name = arg, emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) def getReferenceExportCode(base_name, context): if context.needsCleanup(base_name): return base_name else: return "INCREASE_REFCOUNT( %s )" % base_name def getReferenceExportCode2(base_name, emit, context): if not context.needsCleanup(base_name): emit("Py_INCREF( %s );" % base_name) Nuitka-0.5.18.1/nuitka/codegen/ExpressionCodes.py0000644000175000017500000000425312651071040022017 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Expression codes, side effects, or statements that are an unused expression. When you write "f()", i.e. you don't use the return value, that is an expression only statement. """ from .ErrorCodes import getReleaseCode from .Helpers import generateExpressionCode def generateExpressionOnlyCode(statement, emit, context): return getStatementOnlyCode( value = statement.getExpression(), emit = emit, context = context ) def getStatementOnlyCode(value, emit, context): tmp_name = context.allocateTempName( base_name = "unused", type_code = "NUITKA_MAY_BE_UNUSED PyObject *", unique = True ) # An error of the expression is dealt inside of this, not necessary here. generateExpressionCode( expression = value, to_name = tmp_name, emit = emit, context = context ) getReleaseCode( release_name = tmp_name, emit = emit, context = context ) def generateSideEffectsCode(to_name, expression, emit, context): for side_effect in expression.getSideEffects(): getStatementOnlyCode( value = side_effect, emit = emit, context = context ) generateExpressionCode( to_name = to_name, expression = expression.getExpression(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/GlobalsLocalsCodes.py0000644000175000017500000002201312651071040022373 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for locals and globals handling. This also includes writing back to locals for exec statements. """ from nuitka.PythonVersions import python_version from .ConstantCodes import getConstantCode from .ErrorCodes import getErrorExitBoolCode from .Helpers import generateExpressionCode from .ModuleCodes import getModuleAccessCode from .PythonAPICodes import generateCAPIObjectCode from .templates.CodeTemplatesVariables import ( template_set_locals_dict_value, template_set_locals_mapping_value, template_update_locals_dict_value, template_update_locals_mapping_value ) from .VariableCodes import ( getLocalVariableObjectAccessCode, getVariableAssignmentCode ) def generateBuiltinLocalsCode(to_name, expression, emit, context): provider = expression.getParentVariableProvider() getLoadLocalsCode( to_name = to_name, provider = provider, mode = provider.getLocalsMode(), emit = emit, context = context ) def generateBuiltinGlobalsCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=W0613 getLoadGlobalsCode( to_name = to_name, emit = emit, context = context ) def getLoadGlobalsCode(to_name, emit, context): assert type(to_name) is str emit( "%(to_name)s = ((PyModuleObject *)%(module_identifier)s)->md_dict;" % { "to_name" : to_name, "module_identifier" : getModuleAccessCode(context) }, ) def _getLocalVariableList(provider): if provider.isExpressionFunctionBody(): include_closure = not provider.isUnoptimized() elif provider.isExpressionClassBody(): include_closure = False else: include_closure = True return [ variable for variable in provider.getVariables() if not variable.isModuleVariable() if not variable.isMaybeLocalVariable() if (include_closure or variable.getOwner() is provider) ] def _getVariableDictUpdateCode(target_name, variable, initial, is_dict, emit, context): # TODO: Variable could known to be set here, get a hand at that # information. access_code = getLocalVariableObjectAccessCode( variable = variable, context = context ) if is_dict: if initial: template = template_set_locals_dict_value else: template = template_update_locals_dict_value emit( template % { "dict_name" : target_name, "var_name" : getConstantCode( constant = variable.getName(), context = context ), "access_code" : access_code, } ) else: if initial: template = template_set_locals_mapping_value else: template = template_update_locals_mapping_value res_name = context.getBoolResName() emit( template % { "mapping_name" : target_name, "var_name" : getConstantCode( constant = variable.getName(), context = context ), "access_code" : access_code, "tmp_name" : res_name } ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getLoadLocalsCode(to_name, provider, mode, emit, context): if provider.isCompiledPythonModule(): # Optimization will have made this "globals". assert False, provider elif not context.hasLocalsDict(): local_list = _getLocalVariableList( provider = provider, ) # TODO: Use DictCodes ? emit( "%s = PyDict_New();" % ( to_name, ) ) context.addCleanupTempName(to_name) for local_var in local_list: _getVariableDictUpdateCode( target_name = to_name, variable = local_var, is_dict = True, initial = True, emit = emit, context = context ) else: if mode == "copy": emit( "%s = PyDict_Copy( locals_dict );" % ( to_name, ) ) context.addCleanupTempName(to_name) elif mode == "updated": local_list = _getLocalVariableList( provider = provider ) emit( """\ %s = locals_dict; Py_INCREF( locals_dict );""" % ( to_name ) ) for local_var in local_list: _getVariableDictUpdateCode( target_name = to_name, variable = local_var, is_dict = python_version < 300 or \ not context.getFunction().isExpressionClassBody(), initial = False, emit = emit, context = context ) context.addCleanupTempName(to_name) else: assert False def generateSetLocalsCode(statement, emit, context): new_locals_name = context.allocateTempName("set_locals", unique = True) generateExpressionCode( to_name = new_locals_name, expression = statement.getNewLocals(), emit = emit, context = context ) ref_count = context.needsCleanup(new_locals_name) emit( """\ Py_DECREF(locals_dict); locals_dict = %s;""" % ( new_locals_name ) ) if not ref_count: emit( "Py_INCREF(locals_dict);" ) if ref_count: context.removeCleanupTempName(new_locals_name) def getStoreLocalsCode(locals_name, provider, emit, context): assert not provider.isCompiledPythonModule() for variable in provider.getVariables(): if not variable.isModuleVariable() and \ not variable.isMaybeLocalVariable(): key_name = getConstantCode( context = context, constant = variable.getName() ) value_name = context.allocateTempName("locals_value", unique = True) # This should really be a template. emit( "%s = PyObject_GetItem( %s, %s );" % ( value_name, locals_name, key_name, ) ) getErrorExitBoolCode( condition = """\ %s == NULL && !EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_KeyError )""" % value_name, emit = emit, context = context ) emit("CLEAR_ERROR_OCCURRED();") emit("if ( %s != NULL )" % value_name) emit('{') context.addCleanupTempName(value_name) getVariableAssignmentCode( variable = variable, tmp_name = value_name, needs_release = None, # TODO: Could be known maybe. in_place = False, emit = emit, context = context ) emit('}') def generateBuiltinDir1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PyObject_Dir", arg_desc = ( ("dir_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinVarsCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "LOOKUP_VARS", arg_desc = ( ("vars_arg", expression.getSource()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/OperatorCodes.py0000644000175000017500000000565712651071040021464 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Operator code tables These are mostly used to look up the Python C/API from operations or a wrapper used. """ from nuitka.PythonVersions import python_version binary_operator_codes = { # Those commented out in this section have fully specialized variants already. # "Add" : "PyNumber_Add", # "Sub" : "PyNumber_Subtract", # "Div" : "PyNumber_Divide", # "Mult" : "PyNumber_Multiply", # "Mod" : "PyNumber_Remainder", # These have their own variants only to make sure the generic code is in-lined # but the CPython code is not in-lined. # "Pow" : "PyNumber_Power", # "IPow" : "PyNumber_InPlacePower", # The others are generic code and would be faster if they had a specialized variant too. "FloorDiv" : "PyNumber_FloorDivide", "TrueDiv" : "PyNumber_TrueDivide", "LShift" : "PyNumber_Lshift", "RShift" : "PyNumber_Rshift", "BitAnd" : "PyNumber_And", "BitOr" : "PyNumber_Or", "BitXor" : "PyNumber_Xor", "IAdd" : "PyNumber_InPlaceAdd", "ISub" : "PyNumber_InPlaceSubtract", "IMult" : "PyNumber_InPlaceMultiply", "IDiv" : "PyNumber_InPlaceDivide", "IFloorDiv" : "PyNumber_InPlaceFloorDivide", "ITrueDiv" : "PyNumber_InPlaceTrueDivide", "IMod" : "PyNumber_InPlaceRemainder", "ILShift" : "PyNumber_InPlaceLshift", "IRShift" : "PyNumber_InPlaceRshift", "IBitAnd" : "PyNumber_InPlaceAnd", "IBitOr" : "PyNumber_InPlaceOr", "IBitXor" : "PyNumber_InPlaceXor", } # Python 3.5 only operator if python_version >= 350: binary_operator_codes["MatMult"] = "PyNumber_MatrixMultiply" binary_operator_codes["IMatMult"] = "PyNumber_InPlaceMatrixMultiply" unary_operator_codes = { "UAdd" : ("PyNumber_Positive", 1), "USub" : ("PyNumber_Negative", 1), "Invert" : ("PyNumber_Invert", 1), "Repr" : ("PyObject_Repr", 1), "Not" : ("UNARY_NOT", 0) } rich_comparison_codes = { "Lt" : "LT", "LtE" : "LE", "Eq" : "EQ", "NotEq" : "NE", "Gt" : "GT", "GtE" : "GE" } normal_comparison_codes = { "In" : "SEQUENCE_CONTAINS", "NotIn" : "SEQUENCE_CONTAINS_NOT" } Nuitka-0.5.18.1/nuitka/codegen/IntegerCodes.py0000644000175000017500000000702112651071040021251 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Integer related codes, long and int. """ from nuitka.PythonVersions import python_version from .ErrorCodes import getErrorExitCode, getReleaseCodes from .Helpers import generateChildExpressionsCode from .PythonAPICodes import generateCAPIObjectCode def generateBuiltinLongCode(to_name, expression, emit, context): assert python_version < 300 value = expression.getValue() base = expression.getBase() assert value is not None if base is None: generateCAPIObjectCode( to_name = to_name, capi = "PyNumber_Long", arg_desc = ( ("long_arg", value), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) else: value_name, base_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = TO_LONG2( %s, %s );" % ( to_name, value_name, base_name ) ) getReleaseCodes( release_names = (value_name, base_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinIntCode(to_name, expression, emit, context): value = expression.getValue() base = expression.getBase() assert value is not None if base is None: generateCAPIObjectCode( to_name = to_name, capi = "PyNumber_Int", arg_desc = ( ("int_arg", value), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, ) else: value_name, base_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = TO_INT2( %s, %s );" % ( to_name, value_name, base_name ) ) getReleaseCodes( release_names = (value_name, base_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.18.1/nuitka/codegen/BranchCodes.py0000644000175000017500000000427212651071040021056 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Branch related codes. """ from .ConditionalCodes import generateConditionCode from .LabelCodes import getGotoCode, getLabelCode def generateBranchCode(statement, emit, context): from .CodeGeneration import generateStatementSequenceCode true_target = context.allocateLabel("branch_yes") false_target = context.allocateLabel("branch_no") end_target = context.allocateLabel("branch_end") old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) generateConditionCode( condition = statement.getCondition(), emit = emit, context = context ) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) getLabelCode(true_target, emit) generateStatementSequenceCode( statement_sequence = statement.getBranchYes(), emit = emit, context = context ) if statement.getBranchNo() is not None: getGotoCode(end_target, emit) getLabelCode(false_target, emit) generateStatementSequenceCode( statement_sequence = statement.getBranchNo(), emit = emit, context = context ) getLabelCode(end_target, emit) else: getLabelCode(false_target, emit) Nuitka-0.5.18.1/nuitka/codegen/TryCodes.py0000644000175000017500000002560312651071040020440 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Try statement and related code generation. For Nuitka, all try/except and try/finally are dealt with this, where the finally block gets duplicated into handlers. So this is a common low level structure used, where exception handling and everything is made explicit. """ from nuitka import Options from .ErrorCodes import getMustNotGetHereCode from .ExceptionCodes import getExceptionUnpublishedReleaseCode from .Helpers import generateExpressionCode from .IteratorCodes import getBuiltinLoopBreakNextCode from .LabelCodes import getGotoCode, getLabelCode from .VariableCodes import getVariableAssignmentCode def generateTryCode(statement, emit, context): # The try construct is the most complex for code generation. We may need to # react on break, continue, return, raise in the handlers. For exception # and return handlers, we need to be able to re-raise or re-return. # So this is full of detail stuff, pylint: disable=R0912,R0914,R0915 if generateTryNextExceptStopIterationCode(statement, emit, context): return # TODO: This should come from Helpers module. from .CodeGeneration import generateStatementSequenceCode # Get the statement sequences involved. All except the tried block can be # None. For the tried block it would be a missed optimization. Also not all # the handlers must be None, then it's also a missed optimization. tried_block = statement.getBlockTry() except_handler = statement.getBlockExceptHandler() continue_handler = statement.getBlockContinueHandler() break_handler = statement.getBlockBreakHandler() return_handler = statement.getBlockReturnHandler() tried_block_may_raise = tried_block.mayRaiseException(BaseException) assert tried_block_may_raise or \ continue_handler is not None or \ break_handler is not None or \ return_handler is not None, statement.asXmlText() # The tried statements might raise, for which we define an escape. tried_handler_escape = context.allocateLabel("try_except_handler") if tried_block_may_raise: old_exception_escape = context.setExceptionEscape(tried_handler_escape) # The tried statements might continue, for which we define an escape. continue_handler_escape = context.allocateLabel("try_continue_handler") if continue_handler is not None: old_continue_target = context.setLoopContinueTarget(continue_handler_escape) # The tried statements might break, for which we define an escape. break_handler_escape = context.allocateLabel("try_break_handler") if break_handler is not None: old_break_target = context.setLoopBreakTarget(break_handler_escape) # The tried statements might return, for which we define an escape. return_handler_escape = context.allocateLabel("try_return_handler") if return_handler is not None: old_return_target = context.setReturnTarget(return_handler_escape) # Now the tried block can be generated, cannot be "None" or else the # optimization failed. emit("// Tried code:") generateStatementSequenceCode( statement_sequence = tried_block, emit = emit, allow_none = False, context = context ) # Restore the old escape targets as preserved above, during the handlers, # the parent handlers should be back in effect. if tried_block_may_raise: context.setExceptionEscape(old_exception_escape) if continue_handler: context.setLoopContinueTarget(old_continue_target) if break_handler: context.setLoopBreakTarget(old_break_target) if return_handler: context.setReturnTarget(old_return_target) post_label = None if not tried_block.isStatementAborting(): if post_label is None: post_label = context.allocateLabel("try_end") getGotoCode(post_label, emit) else: getMustNotGetHereCode( reason = "tried codes exits in all cases", context = context, emit = emit ) if return_handler is not None: assert tried_block.mayReturn() emit("// Return handler code:") getLabelCode(return_handler_escape, emit) # During the return value, the value being returned is in a variable, # and therefore needs to be released before being updated. old_return_value_release = context.setReturnReleaseMode(True) generateStatementSequenceCode( statement_sequence = return_handler, emit = emit, allow_none = False, context = context ) context.setReturnReleaseMode(old_return_value_release) assert return_handler.isStatementAborting() if tried_block_may_raise: emit("// Exception handler code:") getLabelCode(tried_handler_escape, emit) # Need to preserve exception state. keeper_type, keeper_value, keeper_tb, keeper_lineno = \ context.allocateExceptionKeeperVariables() old_keepers = context.setExceptionKeeperVariables( (keeper_type, keeper_value, keeper_tb, keeper_lineno) ) assert keeper_type is not None # TODO: That normalization and chaining is only necessary if the # exception is published. emit( """\ %(keeper_type)s = exception_type; %(keeper_value)s = exception_value; %(keeper_tb)s = exception_tb; %(keeper_lineno)s = exception_lineno; exception_type = NULL; exception_value = NULL; exception_tb = NULL; exception_lineno = -1; """ % { "keeper_type" : keeper_type, "keeper_value" : keeper_value, "keeper_tb" : keeper_tb, "keeper_lineno" : keeper_lineno } ) generateStatementSequenceCode( statement_sequence = except_handler, emit = emit, allow_none = True, context = context ) if except_handler is None or not except_handler.isStatementAborting(): getExceptionUnpublishedReleaseCode(emit, context) if post_label is None: post_label = context.allocateLabel("try_end") getGotoCode(post_label, emit) getMustNotGetHereCode( reason = "exception handler codes exits in all cases", context = context, emit = emit ) context.setExceptionKeeperVariables(old_keepers) else: assert except_handler is None if break_handler is not None: assert tried_block.mayBreak() emit("// try break handler code:") getLabelCode(break_handler_escape, emit) generateStatementSequenceCode( statement_sequence = break_handler, emit = emit, allow_none = False, context = context ) assert break_handler.isStatementAborting() if continue_handler is not None: assert tried_block.mayContinue() emit("// try continue handler code:") getLabelCode(continue_handler_escape, emit) generateStatementSequenceCode( statement_sequence = continue_handler, emit = emit, allow_none = False, context = context ) assert continue_handler.isStatementAborting() emit("// End of try:") if post_label is not None: getLabelCode(post_label, emit) def generateTryNextExceptStopIterationCode(statement, emit, context): # This has many branches which mean this optimized code generation is not # applicable, we return each time. pylint: disable=R0911,R0912 except_handler = statement.getBlockExceptHandler() if except_handler is None: return False if statement.getBlockBreakHandler() is not None: return False if statement.getBlockContinueHandler() is not None: return False if statement.getBlockReturnHandler() is not None: return False tried_statements = statement.getBlockTry().getStatements() if len(tried_statements) != 1: return False handling_statements = except_handler.getStatements() if len(handling_statements) != 1: return False tried_statement = tried_statements[0] if not tried_statement.isStatementAssignmentVariable(): return False assign_source = tried_statement.getAssignSource() if not assign_source.isExpressionBuiltinNext1(): return False handling_statement = handling_statements[0] if not handling_statement.isStatementConditional(): return False yes_statements = handling_statement.getBranchYes().getStatements() no_statements = handling_statement.getBranchNo().getStatements() if len(yes_statements) != 1: return False if not yes_statements[0].isStatementLoopBreak(): return False if len(no_statements) != 1: return False if not no_statements[0].isStatementReraiseException() or \ not no_statements[0].isStatementReraiseException(): return False tmp_name = context.allocateTempName("next_source") generateExpressionCode( expression = assign_source.getValue(), to_name = tmp_name, emit = emit, context = context ) tmp_name2 = context.allocateTempName("assign_source") old_source_ref = context.setCurrentSourceCodeReference( assign_source.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getBuiltinLoopBreakNextCode( to_name = tmp_name2, value = tmp_name, emit = emit, context = context ) getVariableAssignmentCode( tmp_name = tmp_name2, variable = tried_statement.getTargetVariableRef().getVariable(), needs_release = None, in_place = False, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) if context.needsCleanup(tmp_name2): context.removeCleanupTempName(tmp_name2) return True Nuitka-0.5.18.1/nuitka/codegen/__init__.py0000644000175000017500000000150112651071040020432 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/codegen/Helpers.py0000644000175000017500000001300712651071040020301 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Helpers for code generation. This dispatch building of expressions and statements, as well as providing typical support functions to building parts. """ from nuitka.Tracing import printError expression_dispatch_dict = {} def setExpressionDispatchDict(dispatch_dict): # Using global here, as this is really a singleton, in the form of a module, # and this is to break the cyclic dependency it has, pylint: disable=W0603 # Please call us only once. global expression_dispatch_dict assert not expression_dispatch_dict expression_dispatch_dict = dispatch_dict def generateExpressionCode(to_name, expression, emit, context, allow_none = False): try: _generateExpressionCode( to_name = to_name, expression = expression, emit = emit, context = context, allow_none = allow_none ) except Exception: printError( "Problem with %r at %s" % ( expression, "" if expression is None else expression.getSourceReference().getAsString() ) ) raise def _generateExpressionCode(to_name, expression, emit, context, allow_none = False): # This is a dispatching function for every expression. if expression is None and allow_none: return None # Make sure we don't generate code twice for any node, this uncovers bugs # where nodes are shared in the tree, which is not allowed. assert not hasattr(expression, "code_generated"), expression expression.code_generated = True old_source_ref = context.setCurrentSourceCodeReference(expression.getSourceReference()) if not expression.isExpression(): printError("No expression %r" % expression) expression.dump() assert False, expression expression_dispatch_dict[expression.kind]( to_name = to_name, expression = expression, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateExpressionsCode(names, expressions, emit, context): assert len(names) == len(expressions) result = [] for name, expression in zip(names, expressions): if expression is not None: to_name = context.allocateTempName(name) generateExpressionCode( to_name = to_name, expression = expression, emit = emit, context = context ) else: to_name = None result.append(to_name) return result def generateChildExpressionsCode(expression, emit, context): value_names = [] for child_name in expression.named_children: child_value = expression.getChild(child_name) # Allocate anyway, so names are aligned. value_name = context.allocateTempName(child_name + "_name") if child_value is not None: generateExpressionCode( to_name = value_name, expression = child_value, emit = emit, context = context ) value_names.append(value_name) else: context.forgetTempName(value_name) value_names.append(None) return value_names def generateChildExpressionCode(expression, emit, context, child_name = None): assert expression is not None if child_name is None: child_name = expression.getChildName() # Allocate anyway, so names are aligned. value_name = context.allocateTempName(child_name + "_name") generateExpressionCode( to_name = value_name, expression = expression, emit = emit, context = context ) return value_name statement_dispatch_dict = {} def setStatementDispatchDict(dispatch_dict): # Using global here, as this is really a singleton, in the form of a module, # and this is to break the cyclic dependency it has, pylint: disable=W0603 # Please call us only once. global statement_dispatch_dict assert not statement_dispatch_dict statement_dispatch_dict = dispatch_dict def generateStatementCode(statement, emit, context): try: statement_dispatch_dict[statement.kind]( statement = statement, emit = emit, context = context ) # Complain if any temporary was not dealt with yet. assert not context.getCleanupTempnames(), \ context.getCleanupTempnames() except Exception: printError( "Problem with %r at %s" % ( statement, statement.getSourceReference().getAsString() ) ) raise Nuitka-0.5.18.1/nuitka/codegen/TupleCodes.py0000644000175000017500000000577312651071040020761 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tuple codes """ from .ConstantCodes import getConstantAccess from .Helpers import generateExpressionCode from .PythonAPICodes import generateCAPIObjectCode def _areConstants(expressions): for expression in expressions: if not expression.isExpressionConstantRef(): return False if expression.isMutable(): return False return True def generateTupleCreationCode(to_name, expression, emit, context): return getTupleCreationCode( to_name = to_name, elements = expression.getElements(), emit = emit, context = context ) def getTupleCreationCode(to_name, elements, emit, context): if _areConstants(elements): getConstantAccess( to_name = to_name, constant = tuple( element.getConstant() for element in elements ), emit = emit, context = context ) else: emit( "%s = PyTuple_New( %d );" % ( to_name, len(elements) ) ) context.addCleanupTempName(to_name) element_name = context.allocateTempName("tuple_element") for count, element in enumerate(elements): generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) if not context.needsCleanup(element_name): emit("Py_INCREF( %s );" % element_name) else: context.removeCleanupTempName(element_name) emit( "PyTuple_SET_ITEM( %s, %d, %s );" % ( to_name, count, element_name ) ) def generateBuiltinTupleCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PySequence_Tuple", arg_desc = ( ("tuple_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/IdCodes.py0000644000175000017500000000331312651071040020210 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Codes for id and hash """ from .PythonAPICodes import generateCAPIObjectCode def generateBuiltinIdCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PyLong_FromVoidPtr", arg_desc = ( ("id_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinHashCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_HASH", arg_desc = ( ("hash_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/YieldCodes.py0000644000175000017500000000547612651071040020736 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Yield related codes. The normal "yield", and the Python 3.3 or higher "yield from" variant. """ from .ErrorCodes import getErrorExitCode, getReleaseCode from .Helpers import generateChildExpressionsCode def generateYieldCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # In handlers, we must preserve/restore the exception. preserve_exception = expression.isExceptionPreserving() emit( "%s = %s( generator, %s );" % ( to_name, "YIELD" if not preserve_exception else "YIELD_IN_HANDLER", value_name if context.needsCleanup(value_name) else "INCREASE_REFCOUNT( %s )" % value_name ) ) if context.needsCleanup(value_name): context.removeCleanupTempName(value_name) getErrorExitCode( check_name = to_name, emit = emit, context = context ) # Comes as only borrowed. # context.addCleanupTempName(to_name) def generateYieldFromCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # In handlers, we must preserve/restore the exception. preserve_exception = expression.isExceptionPreserving() emit( "%s = %s( generator, %s );" % ( to_name, "YIELD_FROM" if not preserve_exception else "YIELD_FROM_IN_HANDLER", value_name if context.needsCleanup(value_name) else "INCREASE_REFCOUNT( %s )" % value_name ) ) if not context.needsCleanup(value_name): context.addCleanupTempName(value_name) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.18.1/nuitka/codegen/Contexts.py0000644000175000017500000006201312651071040020507 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation contexts. """ import hashlib from nuitka import Options from nuitka.__past__ import iterItems from nuitka.Constants import constant_builtin_types from nuitka.PythonVersions import python_version from .Namify import namifyConstant # Many methods won't use self, but it's the interface. pylint: disable=R0201 class TempMixin: # Lots of details, everything gets to store bits here, to indicate # code generation states, and there are many, pylint: disable=R0902 def __init__(self): self.tmp_names = {} self.tmp_types = {} self.forgotten_names = set() self.labels = {} # For exception handling self.needs_exception_variables = False self.exception_escape = None self.loop_continue = None self.loop_break = None # For branches self.true_target = None self.false_target = None self.keeper_variable_count = 0 self.exception_keepers = (None, None, None, None) self.preserver_variable_counts = set() def formatTempName(self, base_name, number): if number is None: return "tmp_{name}".format( name = base_name ) else: return "tmp_{name}_{number:d}".format( name = base_name, number = number ) def allocateTempName(self, base_name, type_name = "PyObject *", unique = False): if unique: number = None else: number = self.tmp_names.get(base_name, 0) number += 1 self.tmp_names[base_name] = number if base_name not in self.tmp_types: self.tmp_types[base_name] = type_name else: assert self.tmp_types[base_name] == type_name, \ (self.tmp_types[base_name], type_name) return self.formatTempName( base_name = base_name, number = number ) def hasTempName(self, base_name): return base_name in self.tmp_names def forgetTempName(self, tmp_name): self.forgotten_names.add(tmp_name) def getTempNameInfos(self): result = [] for base_name, count in sorted(iterItems(self.tmp_names)): if count is not None: for number in range(1,count+1): tmp_name = self.formatTempName( base_name = base_name, number = number ) if tmp_name not in self.forgotten_names: result.append( ( tmp_name, self.tmp_types[base_name] ) ) else: tmp_name = self.formatTempName( base_name = base_name, number = None ) if tmp_name not in self.forgotten_names: result.append( ( tmp_name, self.tmp_types[base_name] ) ) return result def getExceptionEscape(self): return self.exception_escape def setExceptionEscape(self, label): result = self.exception_escape self.exception_escape = label return result def getLoopBreakTarget(self): return self.loop_break def setLoopBreakTarget(self, label): result = self.loop_break self.loop_break = label return result def getLoopContinueTarget(self): return self.loop_continue def setLoopContinueTarget(self, label): result = self.loop_continue self.loop_continue = label return result def allocateLabel(self, label): result = self.labels.get(label, 0) result += 1 self.labels[label] = result return "{name}_{number:d}".format( name = label, number = result ) def needsExceptionVariables(self): return self.needs_exception_variables def markAsNeedsExceptionVariables(self): self.needs_exception_variables = True def allocateExceptionKeeperVariables(self): self.keeper_variable_count += 1 return ( "exception_keeper_type_%d" % self.keeper_variable_count, "exception_keeper_value_%d" % self.keeper_variable_count, "exception_keeper_tb_%d" % self.keeper_variable_count, "exception_keeper_lineno_%s" % self.keeper_variable_count ) def getKeeperVariableCount(self): return self.keeper_variable_count def getExceptionKeeperVariables(self): return self.exception_keepers def setExceptionKeeperVariables(self, keeper_vars): result = self.exception_keepers self.exception_keepers = tuple(keeper_vars) return result def getExceptionPreserverCounts(self): return self.preserver_variable_counts def addExceptionPreserverVariables(self, count): assert count != 0 self.preserver_variable_counts.add(count) def getTrueBranchTarget(self): return self.true_target def getFalseBranchTarget(self): return self.false_target def setTrueBranchTarget(self, label): self.true_target = label def setFalseBranchTarget(self, label): self.false_target = label class CodeObjectsMixin: def __init__(self): # Code objects needed made unique by a key. self.code_objects = {} def getCodeObjects(self): return sorted(iterItems(self.code_objects)) def getCodeObjectHandle(self, code_object, filename, line_number, is_optimized, new_locals, has_closure, future_flags): key = ( filename, code_object.getCodeObjectName(), line_number, code_object.getVarNames(), code_object.getArgumentCount(), code_object.getKwOnlyParameterCount(), code_object.getKind(), is_optimized, new_locals, code_object.hasStarListArg(), code_object.hasStarDictArg(), has_closure, future_flags ) if key not in self.code_objects: self.code_objects[key] = "codeobj_%s" % self._calcHash(key) return self.code_objects[key] if python_version < 300: def _calcHash(self, key): hash_value = hashlib.md5( "%s%s%d%s%d%d%s%s%s%s%s%s%s" % key ) return hash_value.hexdigest() else: def _calcHash(self, key): hash_value = hashlib.md5( ("%s%s%d%s%d%d%s%s%s%s%s%s%s" % key).encode("utf-8") ) return hash_value.hexdigest() class PythonContextBase: def __init__(self): self.temp_counts = {} self.source_ref = None def isCompiledPythonModule(self): return False def allocateTempNumber(self, tmp_scope): result = self.temp_counts.get(tmp_scope, 0) + 1 self.temp_counts[ tmp_scope ] = result return result class PythonChildContextBase(PythonContextBase): def __init__(self, parent): PythonContextBase.__init__(self) self.parent = parent def getConstantCode(self, constant): return self.parent.getConstantCode(constant) def getModuleCodeName(self): return self.parent.getModuleCodeName() def getModuleName(self): return self.parent.getModuleName() def addHelperCode(self, key, code): return self.parent.addHelperCode(key, code) def hasHelperCode(self, key): return self.parent.hasHelperCode(key) def addDeclaration(self, key, code): self.parent.addDeclaration(key, code) def _getConstantDefaultPopulation(): # Note: Can't work with set here, because we need to put in some values that # cannot be hashed. result = [ # Basic values that the helper code uses all the times. (), {}, "", True, False, 0, 1, # For Python3 empty bytes, no effect for Python2, same as "", used for # code objects. b"", # Python mechanics, used in various helpers. "__module__", "__class__", "__name__", "__metaclass__", "__dict__", "__doc__", "__file__", "__enter__", "__exit__", "__builtins__", "__all__", "__cmp__", "__iter__", # Patched module name. "inspect", # Names of built-ins used in helper code. "compile", "range", "open", "__import__" ] if python_version >= 300: # For Python3 modules result.append( "__cached__" ) # For Python3 print result += ( "print", "end", "file" ) if python_version >= 330: # Modules have that attribute starting with 3.3 result.append( "__loader__" ) if python_version >= 340: result.append( # YIELD_FROM uses this starting 3.4, with 3.3 other code is used. "send" ) if python_version >= 330: result += ( # YIELD_FROM uses this "throw", "close", ) if python_version < 300: # For patching Python2 internal class type result += ( "__getattr__", "__setattr__", "__delattr__", ) # For patching Python2 "sys" attributes for current exception result += ( "exc_type", "exc_value", "exc_traceback" ) # The xrange built-in is Python2 only. if python_version < 300: result.append( "xrange" ) # Executables only if not Options.shallMakeModule(): result.append( "__main__" ) # The "site" module is referenced in inspect patching. result.append( "site" ) # Built-in original values if not Options.shallMakeModule(): result += [ "type", "len", "range", "repr", "int", "iter", ] if python_version < 300: result.append( "long", ) # Disabling warnings at startup if "no_warnings" in Options.getPythonFlags(): result.append( "ignore" ) if python_version >= 350: # Patching the types module. result.append( "types" ) return result class PythonGlobalContext: def __init__(self): self.constants = {} self.constant_use_count = {} for constant in _getConstantDefaultPopulation(): code = self.getConstantCode(constant) # Force them to be global. self.countConstantUse(code) self.countConstantUse(code) self.needs_exception_variables = False def getConstantCode(self, constant): # Use in user code, or for constants building code itself if constant is None: key = "Py_None" elif constant is True: key = "Py_True" elif constant is False: key = "Py_False" elif constant is Ellipsis: key = "Py_Ellipsis" elif constant in constant_builtin_types: type_name = constant.__name__ if constant is int and python_version >= 300: type_name = "long" if constant is str and python_version < 300: type_name = "string" if constant is str and python_version > 300: type_name = "unicode" key = "(PyObject *)&Py%s_Type" % type_name.title() else: key = "const_" + namifyConstant(constant) if key not in self.constants: self.constants[key] = constant return key def countConstantUse(self, constant): if constant not in self.constant_use_count: self.constant_use_count[constant] = 0 self.constant_use_count[constant] += 1 def getConstantUseCount(self, constant): return self.constant_use_count[constant] def getConstants(self): return self.constants class FrameDeclarationsMixin: def __init__(self): self.frame_declarations = [] def addFrameDeclaration(self, frame_decl): self.frame_declarations.append(frame_decl) def getFrameDeclarations(self): return self.frame_declarations class PythonModuleContext(PythonContextBase, TempMixin, CodeObjectsMixin, FrameDeclarationsMixin): # Plenty of attributes, because it's storing so many different things. # pylint: disable=R0902 def __init__(self, module, module_name, code_name, filename, global_context): PythonContextBase.__init__(self) TempMixin.__init__(self) CodeObjectsMixin.__init__(self) FrameDeclarationsMixin.__init__(self) self.module = module self.name = module_name self.code_name = code_name self.filename = filename self.global_context = global_context self.declaration_codes = {} self.helper_codes = {} self.constants = set() self.return_release_mode = False self.frame_handle = None self.return_exit = True self.return_name = None self.needs_module_filename_object = False def __repr__(self): return "" % self.filename def getOwner(self): return self.module def isCompiledPythonModule(self): return True def hasLocalsDict(self): return False def getFrameHandle(self): return self.frame_handle def setFrameHandle(self, frame_handle): self.frame_handle = frame_handle def getName(self): return self.name def getFilename(self): return self.filename def mayRaiseException(self): body = self.module.getBody() return body is not None and body.mayRaiseException(BaseException) getModuleName = getName def getModuleCodeName(self): return self.code_name # There cannot be local variable in modules no need to consider the name. # pylint: disable=W0613 def hasClosureVariable(self, var_name): return False # pylint: enable=W0613 def setFrameGuardMode(self, guard_mode): assert guard_mode == "once" def addHelperCode(self, key, code): assert key not in self.helper_codes, key self.helper_codes[ key ] = code def hasHelperCode(self, key): return key in self.helper_codes def getHelperCodes(self): return self.helper_codes def addDeclaration(self, key, code): assert key not in self.declaration_codes self.declaration_codes[ key ] = code def getDeclarations(self): return self.declaration_codes def getReturnValueName(self): return self.return_name def setReturnValueName(self, value): result = self.return_name self.return_name = value return result def setReturnReleaseMode(self, value): result = self.return_release_mode self.return_release_mode = value return result def getReturnReleaseMode(self): return self.return_release_mode def getReturnTarget(self): return self.return_exit def setReturnTarget(self, label): result = self.return_exit self.return_exit = label return result def mayRecurse(self): return False def getConstantCode(self, constant): result = self.global_context.getConstantCode(constant) if result not in self.constants: self.constants.add(result) self.global_context.countConstantUse(result) return result def getConstants(self): return self.constants def markAsNeedsModuleFilenameObject(self): self.needs_module_filename_object = True def needsModuleFilenameObject(self): return self.needs_module_filename_object class PythonFunctionContext(PythonChildContextBase, TempMixin, FrameDeclarationsMixin): def __init__(self, parent, function): PythonChildContextBase.__init__( self, parent = parent ) TempMixin.__init__(self) FrameDeclarationsMixin.__init__(self) self.function = function self.return_exit = None self.setExceptionEscape("function_exception_exit") self.setReturnTarget("function_return_exit") self.return_release_mode = False self.return_name = None self.frame_handle = None def __repr__(self): return "" % ( "function" if not self.function.isExpressionClassBody() else "class", self.function.getName() ) def getFunction(self): return self.function def getOwner(self): return self.function def hasLocalsDict(self): return self.function.hasLocalsDict() def hasClosureVariable(self, var_name): return var_name in self.function.getClosureVariableNames() def getFrameHandle(self): return self.frame_handle def setFrameHandle(self, frame_handle): self.frame_handle = frame_handle def getReturnValueName(self): if self.return_name is None: self.return_name = self.allocateTempName("return_value", unique = True) return self.return_name def setReturnValueName(self, value): result = self.return_name self.return_name = value return result def getReturnTarget(self): return self.return_exit def setReturnTarget(self, label): result = self.return_exit self.return_exit = label return result def setReturnReleaseMode(self, value): result = self.return_release_mode self.return_release_mode = value return result def getReturnReleaseMode(self): return self.return_release_mode def mayRecurse(self): # TODO: Determine this at compile time for enhanced optimizations. return True def getCodeObjectHandle(self, **kw): return self.parent.getCodeObjectHandle(**kw) class PythonFunctionDirectContext(PythonFunctionContext): def isForDirectCall(self): return True def isForCrossModuleUsage(self): return self.function.isCrossModuleUsed() def isForCreatedFunction(self): return False class PythonGeneratorObjectContext(PythonFunctionContext): def isForDirectCall(self): return False def isForCrossModuleUsage(self): return self.function.isCrossModuleUsed() def isForCreatedFunction(self): return False class PythonCoroutineObjectContext(PythonGeneratorObjectContext): pass class PythonFunctionCreatedContext(PythonFunctionContext): def isForDirectCall(self): return False def isForCreatedFunction(self): return True class PythonStatementCContext(PythonChildContextBase): def __init__(self, parent): PythonChildContextBase.__init__( self, parent = parent ) self.cleanup_names = [] self.current_source_ref = None self.last_source_ref = None def getOwner(self): return self.parent.getOwner() def isCompiledPythonModule(self): return self.parent.isCompiledPythonModule() def getFunction(self): return self.parent.getFunction() def hasLocalsDict(self): return self.parent.hasLocalsDict() def isForDirectCall(self): return self.parent.isForDirectCall() def allocateTempName(self, base_name, type_code = "PyObject *", unique = False): return self.parent.allocateTempName(base_name, type_code, unique) def getIntResName(self): return self.allocateTempName("res", "int", unique = True) def getBoolResName(self): return self.allocateTempName("result", "bool", unique = True) def getReturnValueName(self): return self.parent.getReturnValueName() def setReturnValueName(self, value): return self.parent.setReturnValueName(value) def getGeneratorReturnValueName(self): if python_version >= 330: return self.allocateTempName( "return_value", "PyObject *", unique = True ) else: return self.allocateTempName( "generator_return", "bool", unique = True ) def getExceptionEscape(self): return self.parent.getExceptionEscape() def setExceptionEscape(self, label): return self.parent.setExceptionEscape(label) def getLoopBreakTarget(self): return self.parent.getLoopBreakTarget() def setLoopBreakTarget(self, label): return self.parent.setLoopBreakTarget(label) def getLoopContinueTarget(self): return self.parent.getLoopContinueTarget() def setLoopContinueTarget(self, label): return self.parent.setLoopContinueTarget(label) def getTrueBranchTarget(self): return self.parent.getTrueBranchTarget() def setTrueBranchTarget(self, label): return self.parent.setTrueBranchTarget(label) def getFalseBranchTarget(self): return self.parent.getFalseBranchTarget() def setFalseBranchTarget(self, label): return self.parent.setFalseBranchTarget(label) def getReturnTarget(self): return self.parent.getReturnTarget() def setReturnTarget(self, label): return self.parent.setReturnTarget(label) def getReturnReleaseMode(self): return self.parent.getReturnReleaseMode() def setReturnReleaseMode(self, value): return self.parent.setReturnReleaseMode(value) def allocateLabel(self, label): return self.parent.allocateLabel(label) def addCleanupTempName(self, tmp_name): assert tmp_name not in self.cleanup_names, tmp_name self.cleanup_names.append(tmp_name) def removeCleanupTempName(self, tmp_name): assert tmp_name in self.cleanup_names, tmp_name self.cleanup_names.remove(tmp_name) def needsCleanup(self, tmp_name): return tmp_name in self.cleanup_names def isUsed(self, tmp_name): if tmp_name.startswith("tmp_unused_"): return False else: return True def forgetTempName(self, tmp_name): self.parent.forgetTempName(tmp_name) def getCleanupTempnames(self): return self.cleanup_names def getFrameHandle(self): return self.parent.getFrameHandle() def setFrameHandle(self, frame_handle): return self.parent.setFrameHandle(frame_handle) def allocateExceptionKeeperVariables(self): return self.parent.allocateExceptionKeeperVariables() def getExceptionKeeperVariables(self): return self.parent.getExceptionKeeperVariables() def setExceptionKeeperVariables(self, keeper_vars): return self.parent.setExceptionKeeperVariables(keeper_vars) def addExceptionPreserverVariables(self, count): self.parent.addExceptionPreserverVariables(count) def needsExceptionVariables(self): return self.parent.needsExceptionVariables() def markAsNeedsExceptionVariables(self): self.parent.markAsNeedsExceptionVariables() def addFrameDeclaration(self, frame_decl): self.parent.addFrameDeclaration(frame_decl) def mayRecurse(self): return self.parent.mayRecurse() def getCodeObjectHandle(self, **kw): return self.parent.getCodeObjectHandle(**kw) def getCurrentSourceCodeReference(self): return self.current_source_ref def setCurrentSourceCodeReference(self, value): result = self.current_source_ref self.current_source_ref = value if value is not None: self.last_source_ref = result return result def getLastSourceCodeReference(self): result = self.last_source_ref # self.last_source_ref = None return result def markAsNeedsModuleFilenameObject(self): self.parent.markAsNeedsModuleFilenameObject() Nuitka-0.5.18.1/nuitka/codegen/IndexCodes.py0000644000175000017500000000317612651071040020732 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for index values. This is not for actual subscripts (see SubscriptCodes), but to convert generic values to indexes. Also the maximum and minimum index values are abstracted here. """ from .ErrorCodes import getErrorExitBoolCode def getMaxIndexCode(to_name, emit): emit( "%s = PY_SSIZE_T_MAX;" % to_name ) def getMinIndexCode(to_name, emit): emit( "%s = 0;" % to_name ) def getIndexCode(to_name, value_name, emit, context): emit( "%s = CONVERT_TO_INDEX( %s );" % ( to_name, value_name, ) ) getErrorExitBoolCode( condition = "%s == -1 && ERROR_OCCURRED()" % to_name, emit = emit, context = context ) def getIndexValueCode(to_name, value, emit): emit( "%s = %d;" % ( to_name, value ) ) Nuitka-0.5.18.1/nuitka/codegen/ErrorCodes.py0000644000175000017500000002150512651071040020750 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Error codes These are the helper functions that will emit the error exit codes. They can abstractly check conditions or values directly. The release of statement temporaries from context is automatic. Also formatting errors is done here, avoiding PyErr_Format as much as possible. And releasing of values, as this is what the error case commonly does. """ from nuitka import Options from nuitka.PythonVersions import python_version from .ExceptionCodes import getExceptionIdentifier from .Indentation import getCommentCode, indented from .LineNumberCodes import getErrorLineNumberUpdateCode from .templates.CodeTemplatesExceptions import ( template_error_catch_exception, template_error_catch_quick_exception, template_error_format_string_exception ) def getErrorExitReleaseCode(context): temp_release = '\n'.join( "Py_DECREF( %s );" % tmp_name for tmp_name in context.getCleanupTempnames() ) keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is not None: temp_release += "\nPy_DECREF( %s );" % keeper_variables[0] temp_release += "\nPy_XDECREF( %s );" % keeper_variables[1] temp_release += "\nPy_XDECREF( %s );" % keeper_variables[2] return temp_release def getErrorExitBoolCode(condition, emit, context, needs_check = True, quick_exception = None): assert not condition.endswith(';') if not needs_check: getAssertionCode("!(%s)" % condition, emit) return context.markAsNeedsExceptionVariables() if quick_exception: emit( indented( template_error_catch_quick_exception % { "condition" : condition, "exception_exit" : context.getExceptionEscape(), "quick_exception" : getExceptionIdentifier(quick_exception), "release_temps" : indented( getErrorExitReleaseCode(context) ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context) ) }, 0 ) ) else: emit( indented( template_error_catch_exception % { "condition" : condition, "exception_exit" : context.getExceptionEscape(), "release_temps" : indented( getErrorExitReleaseCode(context) ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context) ) }, 0 ) ) def getErrorExitCode(check_name, emit, context, quick_exception = None, needs_check = True): if needs_check: getErrorExitBoolCode( condition = "%s == NULL" % check_name, quick_exception = quick_exception, emit = emit, context = context ) else: getAssertionCode("%s != NULL" % check_name, emit) def getErrorFormatExitBoolCode(condition, exception, args, emit, context): assert not condition.endswith(';') context.markAsNeedsExceptionVariables() if len(args) == 1 and type(args[0]) is str: from .ConstantCodes import getModuleConstantCode set_exception = [ "exception_type = %s;" % exception, "Py_INCREF( exception_type );", "exception_value = %s;" % getModuleConstantCode( constant = args[0], ), "exception_tb = NULL;" ] else: set_exception = [ "exception_type = %s;" % exception, "Py_INCREF( exception_type );", "exception_value = Py%s_FromFormat( %s );" % ( "String" if python_version < 300 else "Unicode", ", ".join( '"%s"' % arg for arg in args ) ), "exception_tb = NULL;" ] if python_version >= 300: keeper_vars = context.getExceptionKeeperVariables() if keeper_vars[0] is not None: set_exception.append( "ADD_EXCEPTION_CONTEXT( &%s, &%s );" % ( keeper_vars[0], keeper_vars[1] ) ) else: set_exception.append( "NORMALIZE_EXCEPTION( &exception_type, &exception_value, &exception_tb );" ) set_exception.append( "CHAIN_EXCEPTION( exception_value );" ) emit( template_error_format_string_exception % { "condition" : condition, "exception_exit" : context.getExceptionEscape(), "set_exception" : indented(set_exception), "release_temps" : indented( getErrorExitReleaseCode(context) ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context) ) } ) def getErrorVariableDeclarations(): return ( "PyObject *exception_type = NULL, *exception_value = NULL;", "PyTracebackObject *exception_tb = NULL;", "NUITKA_MAY_BE_UNUSED int exception_lineno = -1;" ) def getExceptionKeeperVariableNames(keeper_index): # For finally handlers of Python3, which have conditions on assign and # use. debug = Options.isDebug() and python_version >= 300 if debug: keeper_obj_init = " = NULL" else: keeper_obj_init = "" return ( "PyObject *exception_keeper_type_%d%s;" % ( keeper_index, keeper_obj_init ), "PyObject *exception_keeper_value_%d%s;" % ( keeper_index, keeper_obj_init ), "PyTracebackObject *exception_keeper_tb_%d%s;" % ( keeper_index, keeper_obj_init ), "NUITKA_MAY_BE_UNUSED int exception_keeper_lineno_%d%s;" % ( keeper_index, " = -1" if debug else "" ) ) def getExceptionPreserverVariableNames(preserver_id): # For finally handlers of Python3, which have conditions on assign and # use. debug = Options.isDebug() and python_version >= 300 if debug: preserver_obj_init = " = NULL" else: preserver_obj_init = "" return ( "PyObject *exception_preserved_type_%d%s;" % ( preserver_id, preserver_obj_init ), "PyObject *exception_preserved_value_%d%s;" % ( preserver_id, preserver_obj_init ), "PyTracebackObject *exception_preserved_tb_%d%s;" % ( preserver_id, preserver_obj_init ), ) def getErrorFormatExitCode(check_name, exception, args, emit, context): getErrorFormatExitBoolCode( condition = "%s == NULL" % check_name, exception = exception, args = args, emit = emit, context = context ) def getReleaseCode(release_name, emit, context): assert release_name is None or len(release_name) > 2 if context.needsCleanup(release_name): emit("Py_DECREF( %s );" % release_name) context.removeCleanupTempName(release_name) def getReleaseCodes(release_names, emit, context): for release_name in release_names: getReleaseCode( release_name = release_name, emit = emit, context = context ) def getMustNotGetHereCode(reason, context, emit): getCommentCode(reason, emit) provider = context.getOwner() emit( "NUITKA_CANNOT_GET_HERE( %(function_identifier)s );" % { "function_identifier" : provider.getCodeName() } ) if provider.isExpressionGeneratorObjectBody(): emit("return;") elif provider.isExpressionCoroutineObjectBody(): emit("return;") elif provider.isCompiledPythonModule(): emit("return MOD_RETURN_VALUE( NULL );") else: emit("return NULL;") def getAssertionCode(check, emit): emit("assert( %s );" % check) Nuitka-0.5.18.1/nuitka/codegen/EvalCodes.py0000644000175000017500000002513212651071040020546 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Eval/exec/execfile/compile built-in related codes. """ from nuitka import Options from nuitka.PythonVersions import python_version from .ConstantCodes import getConstantCode from .ErrorCodes import getErrorExitCode, getReleaseCode, getReleaseCodes from .GlobalsLocalsCodes import getStoreLocalsCode from .Helpers import generateExpressionCode def generateBuiltinCompileCode(to_name, expression, emit, context): source_name = context.allocateTempName("compile_source") filename_name = context.allocateTempName("compile_filename") mode_name = context.allocateTempName("compile_mode") generateExpressionCode( to_name = source_name, expression = expression.getSourceCode(), emit = emit, context = context ) generateExpressionCode( to_name = filename_name, expression = expression.getFilename(), emit = emit, context = context ) generateExpressionCode( to_name = mode_name, expression = expression.getMode(), emit = emit, context = context ) if expression.getFlags() is not None: flags_name = context.allocateTempName("compile_flags") generateExpressionCode( to_name = flags_name, expression = expression.getFlags(), emit = emit, context = context ) else: flags_name = "NULL" if expression.getDontInherit() is not None: dont_inherit_name = context.allocateTempName("compile_dont_inherit") generateExpressionCode( to_name = dont_inherit_name, expression = expression.getDontInherit(), emit = emit, context = context ) else: dont_inherit_name = "NULL" if expression.getOptimize() is not None: optimize_name = context.allocateTempName("compile_dont_inherit") generateExpressionCode( to_name = optimize_name, expression = expression.getOptimize(), emit = emit, context = context ) else: optimize_name = "NULL" context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getBuiltinCompileCode( to_name = to_name, source_name = source_name, filename_name = filename_name, mode_name = mode_name, flags_name = flags_name, dont_inherit_name = dont_inherit_name, optimize_name = optimize_name, emit = emit, context = context ) def getBuiltinCompileCode(to_name, source_name, filename_name, mode_name, flags_name, dont_inherit_name, optimize_name, emit, context): if python_version < 300: args = ( source_name, filename_name, mode_name, flags_name, dont_inherit_name ) else: args = ( source_name, filename_name, mode_name, flags_name, dont_inherit_name, optimize_name ) emit( "%s = COMPILE_CODE( %s );" % ( to_name, ", ".join(args) ) ) getReleaseCodes( release_names = (source_name, filename_name, mode_name, flags_name, dont_inherit_name, optimize_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getBuiltinEvalCode(to_name, source_name, filename_name, globals_name, locals_name, mode_name, emit, context): compiled_name = context.allocateTempName("eval_compiled") getBuiltinCompileCode( to_name = compiled_name, source_name = source_name, filename_name = filename_name, mode_name = mode_name, flags_name = "NULL", dont_inherit_name = "NULL", optimize_name = "NULL", emit = emit, context = context ) emit( "%s = EVAL_CODE( %s, %s, %s );" % ( to_name, compiled_name, globals_name, locals_name ) ) getReleaseCodes( release_names = (compiled_name, globals_name, locals_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateExecCode(statement, emit, context): source_arg = statement.getSourceCode() globals_arg = statement.getGlobals() locals_arg = statement.getLocals() source_name = context.allocateTempName("eval_source") globals_name = context.allocateTempName("eval_globals") locals_name = context.allocateTempName("eval_locals") generateExpressionCode( to_name = source_name, expression = source_arg, emit = emit, context = context ) generateExpressionCode( to_name = globals_name, expression = globals_arg, emit = emit, context = context ) generateExpressionCode( to_name = locals_name, expression = locals_arg, emit = emit, context = context ) source_ref = statement.getSourceReference() # Filename with origin in improved mode. if Options.isFullCompat(): filename_name = getConstantCode( constant = "", context = context ) else: filename_name = getConstantCode( constant = "" % source_ref.getAsString(), context = context ) old_source_ref = context.setCurrentSourceCodeReference( locals_arg.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) compiled_name = context.allocateTempName("exec_compiled") getBuiltinCompileCode( to_name = compiled_name, source_name = source_name, filename_name = filename_name, mode_name = getConstantCode( constant = "exec", context = context ), flags_name = "NULL", dont_inherit_name = "NULL", optimize_name = "NULL", emit = emit, context = context ) to_name = context.allocateTempName("exec_result") emit( "%s = EVAL_CODE( %s, %s, %s );" % ( to_name, compiled_name, globals_name, locals_name ) ) getReleaseCodes( release_names = (compiled_name, globals_name, locals_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) # Immediately release the exec result. context.addCleanupTempName(to_name) getReleaseCode( release_name = to_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def _generateEvalCode(to_name, node, emit, context): source_name = context.allocateTempName("eval_source") globals_name = context.allocateTempName("eval_globals") locals_name = context.allocateTempName("eval_locals") generateExpressionCode( to_name = source_name, expression = node.getSourceCode(), emit = emit, context = context ) generateExpressionCode( to_name = globals_name, expression = node.getGlobals(), emit = emit, context = context ) generateExpressionCode( to_name = locals_name, expression = node.getLocals(), emit = emit, context = context ) if node.isExpressionBuiltinEval() or \ (python_version >= 300 and node.isExpressionBuiltinExec()): filename = "" else: filename = "" getBuiltinEvalCode( to_name = to_name, source_name = source_name, globals_name = globals_name, locals_name = locals_name, filename_name = getConstantCode( constant = filename, context = context ), mode_name = getConstantCode( constant = "eval" if node.isExpressionBuiltinEval() else "exec", context = context ), emit = emit, context = context ) def generateEvalCode(to_name, expression, emit, context): return _generateEvalCode( to_name = to_name, node = expression, emit = emit, context = context ) def generateExecfileCode(to_name, expression, emit, context): assert python_version < 300 return _generateEvalCode( to_name = to_name, node = expression, emit = emit, context = context ) def generateLocalsDictSyncCode(statement, emit, context): locals_arg = statement.getLocals() locals_name = context.allocateTempName("eval_locals") generateExpressionCode( to_name = locals_name, expression = locals_arg, emit = emit, context = context ) provider = statement.getParentVariableProvider() old_source_ref = context.setCurrentSourceCodeReference( statement.getSourceReference() ) getStoreLocalsCode( locals_name = locals_name, provider = provider, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) Nuitka-0.5.18.1/nuitka/codegen/CodeGeneration.py0000644000175000017500000006423512651072046021605 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ The code generation. No language specifics at all are supposed to be present here. Instead it is using primitives from the given generator to build code sequences (list of strings). As such this is the place that knows how to take a condition and two code branches and make a code block out of it. But it doesn't contain any target language syntax. """ from nuitka import Options from nuitka.__past__ import iterItems from nuitka.PythonVersions import python_version from . import Contexts, Emission, Helpers from .AttributeCodes import ( generateAssignmentAttributeCode, generateAttributeLookupCode, generateAttributeLookupSpecialCode, generateBuiltinGetattrCode, generateBuiltinHasattrCode, generateBuiltinSetattrCode, generateDelAttributeCode ) from .BranchCodes import generateBranchCode from .BuiltinCodes import ( generateBuiltinAnonymousRefCode, generateBuiltinBinCode, generateBuiltinBoolCode, generateBuiltinBytearrayCode, generateBuiltinComplexCode, generateBuiltinFloatCode, generateBuiltinHexCode, generateBuiltinOctCode, generateBuiltinOpenCode, generateBuiltinRange1Code, generateBuiltinRange2Code, generateBuiltinRange3Code, generateBuiltinRefCode, generateBuiltinType1Code, generateBuiltinType3Code, generateBuiltinXrangeCode ) from .CallCodes import generateCallCode, getCallsCode, getCallsDecls from .ClassCodes import ( generateBuiltinIsinstanceCode, generateBuiltinSuperCode, generateSelectMetaclassCode ) from .ComparisonCodes import generateComparisonExpressionCode from .ConditionalCodes import ( generateConditionalAndOrCode, generateConditionalCode ) from .ConstantCodes import generateConstantReferenceCode from .CoroutineCodes import ( generateAsyncIterCode, generateAsyncNextCode, generateAsyncWaitCode, generateMakeCoroutineObjectCode, getCoroutineObjectCode ) from .DictCodes import ( generateBuiltinDictCode, generateDictionaryCreationCode, generateDictOperationGetCode, generateDictOperationRemoveCode, generateDictOperationSetCode, generateDictOperationUpdateCode ) from .EvalCodes import ( generateBuiltinCompileCode, generateEvalCode, generateExecCode, generateExecfileCode, generateLocalsDictSyncCode ) from .ExceptionCodes import ( generateBuiltinMakeExceptionCode, generateExceptionCaughtTracebackCode, generateExceptionCaughtTypeCode, generateExceptionCaughtValueCode, generateExceptionPublishCode, generateExceptionRefCode ) from .ExpressionCodes import generateExpressionOnlyCode, generateSideEffectsCode from .FrameCodes import ( generateFramePreserveExceptionCode, generateFrameRestoreExceptionCode, generateStatementsFrameCode ) from .FunctionCodes import ( generateFunctionCallCode, generateFunctionCreationCode, generateFunctionDeclCode, generateFunctionOutlineCode, getExportScopeCode, getFunctionCode, getFunctionDirectDecl ) from .GeneratorCodes import ( generateGeneratorEntryCode, generateMakeGeneratorObjectCode, getGeneratorObjectCode ) from .GlobalsLocalsCodes import ( generateBuiltinDir1Code, generateBuiltinGlobalsCode, generateBuiltinLocalsCode, generateBuiltinVarsCode, generateSetLocalsCode ) from .Helpers import generateStatementCode from .IdCodes import generateBuiltinHashCode, generateBuiltinIdCode from .ImportCodes import ( generateBuiltinImportCode, generateImportModuleCode, generateImportModuleHardCode, generateImportNameCode, generateImportStarCode ) from .IntegerCodes import generateBuiltinIntCode, generateBuiltinLongCode from .IteratorCodes import ( generateBuiltinIter1Code, generateBuiltinIter2Code, generateBuiltinLenCode, generateBuiltinNext1Code, generateBuiltinNext2Code, generateSpecialUnpackCode, generateUnpackCheckCode ) from .LabelCodes import getStatementTrace from .ListCodes import ( generateBuiltinListCode, generateListCreationCode, generateListOperationAppendCode, generateListOperationExtendCode, generateListOperationPopCode ) from .LoaderCodes import getMetapathLoaderBodyCode from .LoopCodes import ( generateLoopBreakCode, generateLoopCode, generateLoopContinueCode ) from .ModuleCodes import ( generateModuleFileAttributeCode, getModuleCode, getModuleValues ) from .OperationCodes import ( generateOperationBinaryCode, generateOperationUnaryCode ) from .PrintCodes import generatePrintNewlineCode, generatePrintValueCode from .RaisingCodes import generateRaiseCode from .ReturnCodes import ( generateGeneratorReturnCode, generateReturnCode, generateReturnedValueRefCode ) from .SetCodes import ( generateBuiltinSetCode, generateSetCreationCode, generateSetOperationAddCode, generateSetOperationUpdateCode ) from .SliceCodes import ( generateAssignmentSliceCode, generateBuiltinSliceCode, generateDelSliceCode, generateSliceLookupCode ) from .StringCodes import ( generateBuiltinChrCode, generateBuiltinOrdCode, generateBuiltinStrCode, generateBuiltinUnicodeCode ) from .SubscriptCodes import ( generateAssignmentSubscriptCode, generateDelSubscriptCode, generateSubscriptLookupCode ) from .TryCodes import generateTryCode from .TupleCodes import generateBuiltinTupleCode, generateTupleCreationCode from .VariableCodes import ( generateAssignmentVariableCode, generateDelVariableCode, generateVariableReferenceCode, generateVariableReleaseCode ) from .YieldCodes import generateYieldCode, generateYieldFromCode _generated_functions = {} def generateFunctionBodyCode(function_body, context): function_identifier = function_body.getCodeName() if function_identifier in _generated_functions: return _generated_functions[function_identifier] # TODO: Generate both codes, and base direct/etc. decisions on context. if function_body.isExpressionGeneratorObjectBody(): function_context = Contexts.PythonGeneratorObjectContext( parent = context, function = function_body ) elif function_body.isExpressionCoroutineObjectBody(): function_context = Contexts.PythonCoroutineObjectContext( parent = context, function = function_body ) elif function_body.isExpressionClassBody(): function_context = Contexts.PythonFunctionDirectContext( parent = context, function = function_body ) elif function_body.needsCreation(): function_context = Contexts.PythonFunctionCreatedContext( parent = context, function = function_body ) else: function_context = Contexts.PythonFunctionDirectContext( parent = context, function = function_body ) function_codes = Emission.SourceCodeCollector() generateStatementSequenceCode( statement_sequence = function_body.getBody(), allow_none = True, emit = function_codes, context = function_context ) needs_exception_exit = function_body.mayRaiseException(BaseException) if function_body.isExpressionGeneratorObjectBody(): function_code = getGeneratorObjectCode( context = function_context, function_identifier = function_identifier, user_variables = function_body.getUserLocalVariables(), temp_variables = function_body.getTempVariables(), function_codes = function_codes.codes, needs_exception_exit = needs_exception_exit, needs_generator_return = function_body.needsGeneratorReturnExit() ) elif function_body.isExpressionCoroutineObjectBody(): function_code = getCoroutineObjectCode( context = function_context, function_identifier = function_identifier, user_variables = function_body.getUserLocalVariables(), temp_variables = function_body.getTempVariables(), function_codes = function_codes.codes, needs_exception_exit = needs_exception_exit, needs_generator_return = function_body.needsGeneratorReturnExit() ) elif function_body.isExpressionClassBody(): function_code = getFunctionCode( context = function_context, function_identifier = function_identifier, parameters = None, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), temp_variables = function_body.getTempVariables(), function_codes = function_codes.codes, function_doc = function_body.getDoc(), needs_exception_exit = needs_exception_exit, file_scope = getExportScopeCode( cross_module = False ) ) else: parameters = function_body.getParameters() function_code = getFunctionCode( context = function_context, function_identifier = function_identifier, parameters = parameters, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), temp_variables = function_body.getTempVariables(), function_codes = function_codes.codes, function_doc = function_body.getDoc(), needs_exception_exit = needs_exception_exit, file_scope = getExportScopeCode( cross_module = function_body.isCrossModuleUsed() ) ) return function_code def _generateStatementSequenceCode(statement_sequence, emit, context): if statement_sequence is None: return for statement in statement_sequence.getStatements(): if Options.shallTraceExecution(): source_ref = statement.getSourceReference() statement_repr = repr(statement) source_repr = source_ref.getAsString() if python_version >= 300: statement_repr = statement_repr.encode("utf8") source_repr = source_repr.encode("utf8") emit( getStatementTrace( source_repr, statement_repr ) ) # Might contain frame statement sequences as children. if statement.isStatementsFrame(): generateStatementsFrameCode( statement_sequence = statement, emit = emit, context = context ) else: generateStatementCode( statement = statement, emit = emit, context = context ) def generateStatementSequenceCode(statement_sequence, emit, context, allow_none = False): if allow_none and statement_sequence is None: return None assert statement_sequence.kind == "STATEMENTS_SEQUENCE", statement_sequence statement_context = Contexts.PythonStatementCContext(context) _generateStatementSequenceCode( statement_sequence = statement_sequence, emit = emit, context = statement_context ) # Complain if any temporary was not dealt with yet. assert not statement_context.getCleanupTempnames(), \ statement_context.getCleanupTempnames() def prepareModuleCode(global_context, module, module_name): # As this not only creates all modules, but also functions, it deals # also with its functions. assert module.isCompiledPythonModule(), module context = Contexts.PythonModuleContext( module = module, module_name = module_name, code_name = module.getCodeName(), filename = module.getFilename(), global_context = global_context ) context.setExceptionEscape("module_exception_exit") statement_sequence = module.getBody() codes = Emission.SourceCodeCollector() generateStatementSequenceCode( statement_sequence = statement_sequence, emit = codes, allow_none = True, context = context, ) function_decl_codes = [] function_body_codes = [] for function_body in module.getUsedFunctions(): function_code = generateFunctionBodyCode( function_body = function_body, context = context ) assert type(function_code) is str function_body_codes.append(function_code) function_decl = generateFunctionDeclCode( function_body = function_body ) if function_decl is not None: function_decl_codes.append(function_decl) for function_body in module.getCrossUsedFunctions(): assert function_body.isCrossModuleUsed() function_decl = getFunctionDirectDecl( function_identifier = function_body.getCodeName(), closure_variables = function_body.getClosureVariables(), file_scope = getExportScopeCode( cross_module = function_body.isCrossModuleUsed() ) ) function_decl_codes.append(function_decl) for _identifier, code in sorted(iterItems(context.getHelperCodes())): function_body_codes.append(code) for _identifier, code in sorted(iterItems(context.getDeclarations())): function_decl_codes.append(code) function_body_codes = "\n\n".join(function_body_codes) function_decl_codes = "\n\n".join(function_decl_codes) template_values = getModuleValues( module_name = module_name, module_identifier = module.getCodeName(), codes = codes.codes, function_decl_codes = function_decl_codes, function_body_codes = function_body_codes, temp_variables = module.getTempVariables(), is_main_module = module.isMainModule(), is_internal_module = module.isInternalModule(), context = context ) if python_version >= 330: context.getConstantCode("__loader__") return template_values, context def generateModuleCode(module_context, template_values): return getModuleCode( module_context = module_context, template_values = template_values ) def generateHelpersCode(other_modules): calls_decl_code = getCallsDecls() loader_code = getMetapathLoaderBodyCode(other_modules) calls_body_code = getCallsCode() return calls_decl_code, calls_body_code + loader_code def makeGlobalContext(): return Contexts.PythonGlobalContext() Helpers.setExpressionDispatchDict( { "EXPRESSION_ATTRIBUTE_LOOKUP" : generateAttributeLookupCode, "EXPRESSION_ATTRIBUTE_LOOKUP_SPECIAL" : generateAttributeLookupSpecialCode, "EXPRESSION_BUILTIN_SLICE" : generateBuiltinSliceCode, "EXPRESSION_BUILTIN_HASH" : generateBuiltinHashCode, "EXPRESSION_BUILTIN_ID" : generateBuiltinIdCode, "EXPRESSION_BUILTIN_COMPILE" : generateBuiltinCompileCode, "EXPRESSION_BUILTIN_EXECFILE" : generateExecfileCode, "EXPRESSION_BUILTIN_EVAL" : generateEvalCode, "EXPRESSION_BUILTIN_EXEC" : generateEvalCode, "EXPRESSION_BUILTIN_ITER1" : generateBuiltinIter1Code, "EXPRESSION_BUILTIN_ITER2" : generateBuiltinIter2Code, "EXPRESSION_BUILTIN_NEXT1" : generateBuiltinNext1Code, "EXPRESSION_BUILTIN_NEXT2" : generateBuiltinNext2Code, "EXPRESSION_BUILTIN_TYPE1" : generateBuiltinType1Code, "EXPRESSION_BUILTIN_TYPE3" : generateBuiltinType3Code, "EXPRESSION_BUILTIN_IMPORT" : generateBuiltinImportCode, "EXPRESSION_BUILTIN_BOOL" : generateBuiltinBoolCode, "EXPRESSION_BUILTIN_BYTEARRAY" : generateBuiltinBytearrayCode, "EXPRESSION_BUILTIN_INT" : generateBuiltinIntCode, "EXPRESSION_BUILTIN_LONG" : generateBuiltinLongCode, "EXPRESSION_BUILTIN_FLOAT" : generateBuiltinFloatCode, "EXPRESSION_BUILTIN_COMPLEX" : generateBuiltinComplexCode, "EXPRESSION_BUILTIN_LEN" : generateBuiltinLenCode, "EXPRESSION_BUILTIN_STR" : generateBuiltinStrCode, "EXPRESSION_BUILTIN_UNICODE" : generateBuiltinUnicodeCode, "EXPRESSION_BUILTIN_CHR" : generateBuiltinChrCode, "EXPRESSION_BUILTIN_ORD" : generateBuiltinOrdCode, "EXPRESSION_BUILTIN_BIN" : generateBuiltinBinCode, "EXPRESSION_BUILTIN_OCT" : generateBuiltinOctCode, "EXPRESSION_BUILTIN_HEX" : generateBuiltinHexCode, "EXPRESSION_BUILTIN_TUPLE" : generateBuiltinTupleCode, "EXPRESSION_BUILTIN_LIST" : generateBuiltinListCode, "EXPRESSION_BUILTIN_SET" : generateBuiltinSetCode, "EXPRESSION_BUILTIN_DICT" : generateBuiltinDictCode, "EXPRESSION_BUILTIN_LOCALS" : generateBuiltinLocalsCode, "EXPRESSION_BUILTIN_GLOBALS" : generateBuiltinGlobalsCode, "EXPRESSION_BUILTIN_SUPER" : generateBuiltinSuperCode, "EXPRESSION_BUILTIN_ISINSTANCE" : generateBuiltinIsinstanceCode, "EXPRESSION_BUILTIN_DIR1" : generateBuiltinDir1Code, "EXPRESSION_BUILTIN_VARS" : generateBuiltinVarsCode, "EXPRESSION_BUILTIN_HASATTR" : generateBuiltinHasattrCode, "EXPRESSION_BUILTIN_GETATTR" : generateBuiltinGetattrCode, "EXPRESSION_BUILTIN_SETATTR" : generateBuiltinSetattrCode, "EXPRESSION_BUILTIN_OPEN" : generateBuiltinOpenCode, "EXPRESSION_BUILTIN_RANGE1" : generateBuiltinRange1Code, "EXPRESSION_BUILTIN_RANGE2" : generateBuiltinRange2Code, "EXPRESSION_BUILTIN_RANGE3" : generateBuiltinRange3Code, "EXPRESSION_BUILTIN_XRANGE" : generateBuiltinXrangeCode, "EXPRESSION_BUILTIN_MAKE_EXCEPTION" : generateBuiltinMakeExceptionCode, "EXPRESSION_BUILTIN_REF" : generateBuiltinRefCode, "EXPRESSION_BUILTIN_EXCEPTION_REF" : generateExceptionRefCode, "EXPRESSION_BUILTIN_ANONYMOUS_REF" : generateBuiltinAnonymousRefCode, "EXPRESSION_CAUGHT_EXCEPTION_TYPE_REF" : generateExceptionCaughtTypeCode, "EXPRESSION_CAUGHT_EXCEPTION_VALUE_REF" : generateExceptionCaughtValueCode, "EXPRESSION_CAUGHT_EXCEPTION_TRACEBACK_REF" : generateExceptionCaughtTracebackCode, "EXPRESSION_CALL_EMPTY" : generateCallCode, "EXPRESSION_CALL_KEYWORDS_ONLY" : generateCallCode, "EXPRESSION_CALL_NO_KEYWORDS" : generateCallCode, "EXPRESSION_CALL" : generateCallCode, "EXPRESSION_CONSTANT_REF" : generateConstantReferenceCode, "EXPRESSION_CONDITIONAL" : generateConditionalCode, "EXPRESSION_CONDITIONAL_OR" : generateConditionalAndOrCode, "EXPRESSION_CONDITIONAL_AND" : generateConditionalAndOrCode, "EXPRESSION_COMPARISON" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_IS" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_IS_NOT" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_EXCEPTION_MATCH" : generateComparisonExpressionCode, "EXPRESSION_DICT_OPERATION_UPDATE" : generateDictOperationUpdateCode, "EXPRESSION_DICT_OPERATION_GET" : generateDictOperationGetCode, "EXPRESSION_DICT_OPERATION_SET" : generateDictOperationSetCode, "EXPRESSION_FUNCTION_CREATION" : generateFunctionCreationCode, "EXPRESSION_FUNCTION_CALL" : generateFunctionCallCode, "EXPRESSION_IMPORT_MODULE" : generateImportModuleCode, "EXPRESSION_IMPORT_MODULE_HARD" : generateImportModuleHardCode, "EXPRESSION_IMPORT_NAME" : generateImportNameCode, "EXPRESSION_LIST_OPERATION_APPEND" : generateListOperationAppendCode, "EXPRESSION_LIST_OPERATION_EXTEND" : generateListOperationExtendCode, "EXPRESSION_LIST_OPERATION_POP" : generateListOperationPopCode, "EXPRESSION_MODULE_FILE_ATTRIBUTE_REF" : generateModuleFileAttributeCode, "EXPRESSION_MAKE_GENERATOR_OBJECT" : generateMakeGeneratorObjectCode, "EXPRESSION_MAKE_COROUTINE_OBJECT" : generateMakeCoroutineObjectCode, "EXPRESSION_MAKE_SET" : generateSetCreationCode, "EXPRESSION_MAKE_TUPLE" : generateTupleCreationCode, "EXPRESSION_MAKE_LIST" : generateListCreationCode, "EXPRESSION_MAKE_DICT" : generateDictionaryCreationCode, "EXPRESSION_OPERATION_BINARY" : generateOperationBinaryCode, "EXPRESSION_OPERATION_BINARY_INPLACE" : generateOperationBinaryCode, "EXPRESSION_OPERATION_UNARY" : generateOperationUnaryCode, "EXPRESSION_OPERATION_NOT" : generateOperationUnaryCode, "EXPRESSION_OUTLINE_BODY" : generateFunctionOutlineCode, "EXPRESSION_RETURNED_VALUE_REF" : generateReturnedValueRefCode, "EXPRESSION_SUBSCRIPT_LOOKUP" : generateSubscriptLookupCode, "EXPRESSION_SLICE_LOOKUP" : generateSliceLookupCode, "EXPRESSION_SET_OPERATION_ADD" : generateSetOperationAddCode, "EXPRESSION_SET_OPERATION_UPDATE" : generateSetOperationUpdateCode, "EXPRESSION_SIDE_EFFECTS" : generateSideEffectsCode, "EXPRESSION_SPECIAL_UNPACK" : generateSpecialUnpackCode, "EXPRESSION_TEMP_VARIABLE_REF" : generateVariableReferenceCode, "EXPRESSION_VARIABLE_REF" : generateVariableReferenceCode, "EXPRESSION_YIELD" : generateYieldCode, "EXPRESSION_YIELD_FROM" : generateYieldFromCode, "EXPRESSION_SELECT_METACLASS" : generateSelectMetaclassCode, "EXPRESSION_ASYNC_WAIT" : generateAsyncWaitCode, "EXPRESSION_ASYNC_ITER" : generateAsyncIterCode, "EXPRESSION_ASYNC_NEXT" : generateAsyncNextCode, } ) Helpers.setStatementDispatchDict( { "STATEMENT_ASSIGNMENT_VARIABLE" : generateAssignmentVariableCode, "STATEMENT_ASSIGNMENT_ATTRIBUTE" : generateAssignmentAttributeCode, "STATEMENT_ASSIGNMENT_SUBSCRIPT" : generateAssignmentSubscriptCode, "STATEMENT_ASSIGNMENT_SLICE" : generateAssignmentSliceCode, "STATEMENT_DEL_VARIABLE" : generateDelVariableCode, "STATEMENT_DEL_ATTRIBUTE" : generateDelAttributeCode, "STATEMENT_DEL_SUBSCRIPT" : generateDelSubscriptCode, "STATEMENT_DEL_SLICE" : generateDelSliceCode, "STATEMENT_DICT_OPERATION_REMOVE" : generateDictOperationRemoveCode, "STATEMENT_RELEASE_VARIABLE" : generateVariableReleaseCode, "STATEMENT_EXPRESSION_ONLY" : generateExpressionOnlyCode, "STATEMENT_RETURN" : generateReturnCode, "STATEMENT_GENERATOR_RETURN" : generateGeneratorReturnCode, "STATEMENT_CONDITIONAL" : generateBranchCode, "STATEMENT_TRY" : generateTryCode, "STATEMENT_PRINT_VALUE" : generatePrintValueCode, "STATEMENT_PRINT_NEWLINE" : generatePrintNewlineCode, "STATEMENT_IMPORT_STAR" : generateImportStarCode, "STATEMENT_LOOP" : generateLoopCode, "STATEMENT_LOOP_BREAK" : generateLoopBreakCode, "STATEMENT_LOOP_CONTINUE" : generateLoopContinueCode, "STATEMENT_RAISE_EXCEPTION" : generateRaiseCode, "STATEMENT_RAISE_EXCEPTION_IMPLICIT" : generateRaiseCode, "STATEMENT_SPECIAL_UNPACK_CHECK" : generateUnpackCheckCode, "STATEMENT_EXEC" : generateExecCode, "STATEMENT_LOCALS_DICT_SYNC" : generateLocalsDictSyncCode, "STATEMENT_SET_LOCALS" : generateSetLocalsCode, "STATEMENT_GENERATOR_ENTRY" : generateGeneratorEntryCode, "STATEMENT_PRESERVE_FRAME_EXCEPTION" : generateFramePreserveExceptionCode, "STATEMENT_RESTORE_FRAME_EXCEPTION" : generateFrameRestoreExceptionCode, "STATEMENT_PUBLISH_EXCEPTION" : generateExceptionPublishCode } ) Nuitka-0.5.18.1/nuitka/codegen/SubscriptCodes.py0000644000175000017500000001670612651071040021644 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Subscript related code generation. There is special handling for integer indexes, which can be dealt with much faster than general subscript lookups. """ from nuitka import Options from nuitka.Constants import isIndexConstant from .ErrorCodes import getErrorExitBoolCode, getErrorExitCode, getReleaseCodes from .Helpers import ( generateChildExpressionsCode, generateExpressionCode, generateExpressionsCode ) def generateAssignmentSubscriptCode(statement, emit, context): subscribed = statement.getSubscribed() subscript = statement.getSubscript() value = statement.getAssignSource() integer_subscript = False if subscript.isExpressionConstantRef(): constant = subscript.getConstant() if isIndexConstant(constant): constant_value = int(constant) if abs(constant_value) < 2**31: integer_subscript = True value_name = context.allocateTempName("ass_subvalue") generateExpressionCode( to_name = value_name, expression = value, emit = emit, context = context ) subscribed_name = context.allocateTempName("ass_subscribed") generateExpressionCode( to_name = subscribed_name, expression = subscribed, emit = emit, context = context ) subscript_name = context.allocateTempName("ass_subscript") generateExpressionCode( to_name = subscript_name, expression = subscript, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) if integer_subscript: getIntegerSubscriptAssignmentCode( subscribed_name = subscribed_name, subscript_name = subscript_name, subscript_value = constant_value, value_name = value_name, emit = emit, context = context ) else: getSubscriptAssignmentCode( target_name = subscribed_name, subscript_name = subscript_name, value_name = value_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateDelSubscriptCode(statement, emit, context): subscribed = statement.getSubscribed() subscript = statement.getSubscript() target_name, subscript_name = generateExpressionsCode( expressions = (subscribed, subscript), names = ("delsubscr_target", "delsubscr_subscript"), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( subscript.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSubscriptDelCode( target_name = target_name, subscript_name = subscript_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateSubscriptLookupCode(to_name, expression, emit, context): subscribed_name, subscript_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) return getSubscriptLookupCode( to_name = to_name, subscribed_name = subscribed_name, subscript_name = subscript_name, emit = emit, context = context ) def getIntegerSubscriptLookupCode(to_name, target_name, subscript_name, subscript_value, emit, context): emit( "%s = LOOKUP_SUBSCRIPT_CONST( %s, %s, %s );" % ( to_name, target_name, subscript_name, subscript_value ) ) getReleaseCodes( release_names = (target_name, subscript_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getSubscriptLookupCode(to_name, subscript_name, subscribed_name, emit, context): emit( "%s = LOOKUP_SUBSCRIPT( %s, %s );" % ( to_name, subscribed_name, subscript_name, ) ) getReleaseCodes( release_names = (subscribed_name, subscript_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getIntegerSubscriptAssignmentCode(subscribed_name, subscript_name, subscript_value, value_name, emit, context): assert abs(subscript_value) < 2**31 res_name = context.allocateTempName("ass_subscript_res", "int") emit( "%s = SET_SUBSCRIPT_CONST( %s, %s, %s, %s );" % ( res_name, subscribed_name, subscript_name, subscript_value, value_name, ) ) getReleaseCodes( release_names = (subscribed_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSubscriptAssignmentCode(target_name, subscript_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_SUBSCRIPT( %s, %s, %s );" % ( res_name, target_name, subscript_name, value_name, ) ) getReleaseCodes( release_names = (target_name, subscript_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSubscriptDelCode(target_name, subscript_name, emit, context): res_name = context.getBoolResName() emit( "%s = DEL_SUBSCRIPT( %s, %s );" % ( res_name, target_name, subscript_name, ) ) getReleaseCodes( release_names = (target_name, subscript_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/SliceCodes.py0000644000175000017500000003405312651071040020720 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for slicing. This is about slice lookups, assignments, and deletions. There is also a special case, for using index values instead of objects. The slice objects are also created here, and can be used for indexing. """ from nuitka import Options from nuitka.Constants import isNumberConstant from nuitka.PythonVersions import python_version from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .Helpers import ( generateChildExpressionsCode, generateExpressionCode, generateExpressionsCode ) from .IndexCodes import ( getIndexCode, getIndexValueCode, getMaxIndexCode, getMinIndexCode ) def generateSliceRangeIdentifier(lower, upper, scope, emit, context): lower_name = context.allocateTempName( scope + "slicedel_index_lower", "Py_ssize_t" ) upper_name = context.allocateTempName( scope + "_index_upper", "Py_ssize_t" ) def isSmallNumberConstant(node): value = node.getConstant() if isNumberConstant(value): return abs(int(value)) < 2**63-1 else: return False if lower is None: getMinIndexCode( to_name = lower_name, emit = emit ) elif lower.isExpressionConstantRef() and isSmallNumberConstant(lower): getIndexValueCode( to_name = lower_name, value = int(lower.getConstant()), emit = emit ) else: value_name = context.allocateTempName(scope + "_lower_index_value") generateExpressionCode( to_name = value_name, expression = lower, emit = emit, context = context ) getIndexCode( to_name = lower_name, value_name = value_name, emit = emit, context = context ) if upper is None: getMaxIndexCode( to_name = upper_name, emit = emit ) elif upper.isExpressionConstantRef() and isSmallNumberConstant(upper): getIndexValueCode( to_name = upper_name, value = int(upper.getConstant()), emit = emit ) else: value_name = context.allocateTempName(scope + "_upper_index_value") generateExpressionCode( to_name = value_name, expression = upper, emit = emit, context = context ) getIndexCode( to_name = upper_name, value_name = value_name, emit = emit, context = context ) return lower_name, upper_name def _decideSlicing(lower, upper): return (lower is None or lower.isIndexable()) and \ (upper is None or upper.isIndexable()) def generateSliceLookupCode(to_name, expression, emit, context): assert python_version < 300 lower = expression.getLower() upper = expression.getUpper() if _decideSlicing(lower, upper): lower_name, upper_name = generateSliceRangeIdentifier( lower = lower, upper = upper, scope = "slice", emit = emit, context = context ) source_name = context.allocateTempName("slice_source") generateExpressionCode( to_name = source_name, expression = expression.getLookupSource(), emit = emit, context = context ) getSliceLookupIndexesCode( to_name = to_name, source_name = source_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) else: source_name, lower_name, upper_name = generateExpressionsCode( names = ("slice_source", "slice_lower", "slice_upper"), expressions = ( expression.getLookupSource(), expression.getLower(), expression.getUpper() ), emit = emit, context = context ) getSliceLookupCode( to_name = to_name, source_name = source_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) def generateAssignmentSliceCode(statement, emit, context): assert python_version < 300 lookup_source = statement.getLookupSource() lower = statement.getLower() upper = statement.getUpper() value = statement.getAssignSource() value_name = context.allocateTempName("sliceass_value") generateExpressionCode( to_name = value_name, expression = value, emit = emit, context = context ) target_name = context.allocateTempName("sliceass_target") generateExpressionCode( to_name = target_name, expression = lookup_source, emit = emit, context = context ) if _decideSlicing(lower, upper): lower_name, upper_name = generateSliceRangeIdentifier( lower = lower, upper = upper, scope = "sliceass", emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceAssignmentIndexesCode( target_name = target_name, lower_name = lower_name, upper_name = upper_name, value_name = value_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: lower_name, upper_name = generateExpressionsCode( names = ( "sliceass_lower", "sliceass_upper" ), expressions = ( lower, upper ), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceAssignmentCode( target_name = target_name, upper_name = upper_name, lower_name = lower_name, value_name = value_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateDelSliceCode(statement, emit, context): assert python_version < 300 target = statement.getLookupSource() lower = statement.getLower() upper = statement.getUpper() target_name = context.allocateTempName("slicedel_target") generateExpressionCode( to_name = target_name, expression = target, emit = emit, context = context ) if _decideSlicing(lower, upper): lower_name, upper_name = generateSliceRangeIdentifier( lower = lower, upper = upper, scope = "slicedel", emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( (upper or lower or statement).getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceDelIndexesCode( target_name = target_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: lower_name, upper_name = generateExpressionsCode( names = ( "slicedel_lower", "slicedel_upper" ), expressions = ( lower, upper ), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( (upper or lower or target).getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceDelCode( target_name = target_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateBuiltinSliceCode(to_name, expression, emit, context): lower_name, upper_name, step_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = MAKE_SLICEOBJ3( %s, %s, %s );" % ( to_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None", step_name if step_name is not None else "Py_None", ) ) getReleaseCodes( release_names = (lower_name, upper_name, step_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = False, # Note: Cannot fail emit = emit, context = context ) context.addCleanupTempName(to_name) def getSliceLookupCode(to_name, source_name, lower_name, upper_name, emit, context): emit( "%s = LOOKUP_SLICE( %s, %s, %s );" % ( to_name, source_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None" ) ) getReleaseCodes( release_names = (source_name, lower_name, upper_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getSliceLookupIndexesCode(to_name, lower_name, upper_name, source_name, emit, context): emit( "%s = LOOKUP_INDEX_SLICE( %s, %s, %s );" % ( to_name, source_name, lower_name, upper_name, ) ) getReleaseCode( release_name = source_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getSliceAssignmentIndexesCode(target_name, lower_name, upper_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_INDEX_SLICE( %s, %s, %s, %s );" % ( res_name, target_name, lower_name, upper_name, value_name ) ) getReleaseCodes( release_names = (value_name, target_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSliceAssignmentCode(target_name, lower_name, upper_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_SLICE( %s, %s, %s, %s );" % ( res_name, target_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None", value_name ) ) getReleaseCodes( release_names = (target_name, lower_name, upper_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSliceDelIndexesCode(target_name, lower_name, upper_name, emit, context): res_name = context.getBoolResName() emit( "%s = DEL_INDEX_SLICE( %s, %s, %s );" % ( res_name, target_name, lower_name, upper_name ) ) getReleaseCode( release_name = target_name, emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSliceDelCode(target_name, lower_name, upper_name, emit, context): res_name = context.getBoolResName() emit( "%s = DEL_SLICE( %s, %s, %s );" % ( res_name, target_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None" ) ) getReleaseCodes( release_names = (target_name, lower_name, upper_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) Nuitka-0.5.18.1/nuitka/codegen/ConstantCodes.py0000644000175000017500000007756412651071040021470 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Low level constant code generation. This deals with constants, there creation, there access, and some checks about them. Even mutable constants should not change during the course of the program. There are shared constants, which are created for multiple modules to use, you can think of them as globals. And there are module local constants, which are for a single module only. """ import ctypes import re import struct from logging import warning import marshal from nuitka import Options from nuitka.__past__ import iterItems, long, unicode # pylint: disable=W0622 from nuitka.codegen import Emission from nuitka.Constants import ( constant_builtin_types, getConstantWeight, isMutable ) from .BlobCodes import StreamData from .Emission import SourceCodeCollector from .Indentation import indented from .Pickling import getStreamedConstant from .templates.CodeTemplatesConstants import template_constants_reading def generateConstantReferenceCode(to_name, expression, emit, context): """ Assign the constant behind the expression to to_name.""" getConstantAccess( to_name = to_name, constant = expression.getConstant(), emit = emit, context = context ) # One global stream of constant information. In the future it might make # sense to have per module ones, for better locality of indexes within it, # but we don't do this yet. stream_data = StreamData() # TODO: This is deprecated, and should be removed. def getConstantCode(context, constant): return context.getConstantCode(constant) def getConstantCodeName(context, constant): return context.getConstantCode(constant) # TODO: The determination of this should already happen in Building or in a # helper not during code generation. _match_attribute_names = re.compile(r"[a-zA-Z_][a-zA-Z0-9_]*$") def _isAttributeName(value): # TODO: The exception is to make sure we intern the ".0" argument name # used for generator expressions, iterator value. return _match_attribute_names.match(value) or value == ".0" # Indicator to standalone mode code, if we need pickling module early on, which # we try to avoid, but can happen with things we cannot create directly. _needs_pickle = False def needsPickleInit(): return _needs_pickle def _getUnstreamCode2(constant_value): saved = getStreamedConstant( constant_value = constant_value ) assert type(saved) is bytes # We need to remember having to use pickle, pylint: disable=W0603 global _needs_pickle _needs_pickle = True return stream_data.getStreamDataCode(saved) def _getUnstreamCode(constant_value, constant_identifier): """ Get code to assign given constant value to an identifier from a stream. This uses pickle, and usage should be minimized. """ return "%s = UNSTREAM_CONSTANT( %s );" % ( constant_identifier, _getUnstreamCode2(constant_value) ) sizeof_long = ctypes.sizeof(ctypes.c_long) max_unsigned_long = 2**(sizeof_long*8)-1 # The gcc gives a warning for -2**sizeof_long*8-1, which is still an "int", but # seems to not work (without warning) as literal, so avoid it. min_signed_long = -(2**(sizeof_long*8-1)-1) done = set() def _getConstantInitValueCode(constant_value, constant_type): """ Return code, if possible, to create a constant. It's only used for module local constants, like error messages, and provides no caching of the values. When it returns "None", it is in error. """ # This function is a case driven by returns, pylint: disable=R0911 if constant_type is unicode: try: encoded = constant_value.encode("utf-8") if str is not unicode: return "UNSTREAM_UNICODE( %s )" % ( stream_data.getStreamDataCode(encoded) ) else: return "UNSTREAM_STRING( %s, %d, %d )" % ( stream_data.getStreamDataCode(encoded, fixed_size = True), len(constant_value), 1 if _isAttributeName(constant_value) else 0 ) except UnicodeEncodeError: # TODO: try and use "surrogateescape" for this return None elif constant_type is str: # Python3: Strings that can be encoded as UTF-8 are done more or less # directly. When they cannot be expressed as UTF-8, that is rare not we # can indeed use pickling. assert str is not unicode if len(constant_value) == 1: return "UNSTREAM_CHAR( %d, %d )" % ( ord(constant_value[0]), 1 if _isAttributeName(constant_value) else 0 ) else: return "UNSTREAM_STRING( %s, %d )" % ( stream_data.getStreamDataCode(constant_value), 1 if _isAttributeName(constant_value) else 0 ) elif constant_type is bytes: assert str is unicode return "UNSTREAM_BYTES( %s )" % ( stream_data.getStreamDataCode(constant_value) ) else: return None def decideMarshal(constant_value): """ Decide of a constant can be created using "marshal" module methods. This is not the case for everything. A prominent exception is types, they are constants, but the "marshal" module refuses to work with them. """ constant_type = type(constant_value) if constant_type is type: # Types cannot be marshaled, there is no choice about it. return False elif constant_type is dict: # Look at all the keys an values, if one of it cannot be marshaled, # or should not, that is it. for key, value in iterItems(constant_value): if not decideMarshal(key): return False if not decideMarshal(value): return False elif constant_type in (tuple, list, set, frozenset): for element_value in constant_value: if not decideMarshal(element_value): return False return True def isMarshalConstant(constant_value): """ Decide if we want to use marshal to create a constant. The reason we do this, is because creating dictionaries with 700 elements creates a lot of C code, while gaining usually no performance at all. The MSVC compiler is especially notorious about hanging like forever with this active, due to its optimizer not scaling. Therefore we use a constant "weight" (how expensive it is), and apply that to decide. If marshal is not possible, or constant "weight" is too large, we don't do it. Also, for some constants, marshal can fail, and return other values. Check that too. In that case, we have to create it. """ if not decideMarshal(constant_value): return False if getConstantWeight(constant_value) < 20: return False marshal_value = marshal.dumps(constant_value) restored = marshal.loads(marshal_value) # TODO: Potentially warn about these. return constant_value == restored def attemptToMarshal(constant_identifier, constant_value, emit): """ Try and marshal a value, if so decided. Indicate with return value. See above for why marshal is only used in problematic cases. """ if not isMarshalConstant(constant_value): return False marshal_value = marshal.dumps(constant_value) restored = marshal.loads(marshal_value) # TODO: The check in isMarshalConstant is currently preventing this from # happening. if constant_value != restored: warning("Problem with marshal of constant %r", constant_value) return False emit( "%s = PyMarshal_ReadObjectFromString( (char *)%s );" % ( constant_identifier, stream_data.getStreamDataCode(marshal_value) ) ) return True def _addConstantInitCode(context, emit, check, constant_type, constant_value, constant_identifier, module_level): """ Emit code for a specific constant to be prepared during init. This may be module or global init. Code makes sure that nested constants belong into the same scope. """ # This is just a wrapper to make sure that hash values become initialized # for every constant too. if constant_value in constant_builtin_types: return if constant_value is None: return if constant_value is False: return if constant_value is True: return if constant_value is Ellipsis: return # Do not repeat ourselves. if constant_identifier in done: return if Options.shallTraceExecution(): emit("""puts("Creating constant: %s");""" % constant_identifier) # Then it's a real named constant not yet created. __addConstantInitCode(context, emit, check, constant_type, constant_value, constant_identifier, module_level) if Options.isDebug(): emit( """\ hash_%(constant_identifier)s = DEEP_HASH( %(constant_identifier)s );""" % { "constant_identifier" : constant_identifier } ) check( """\ CHECK_OBJECT( %(constant_identifier)s ); assert( hash_%(constant_identifier)s == DEEP_HASH( %(constant_identifier)s ) );""" % { "constant_identifier" : constant_identifier } ) if Options.isExperimental(): check( """\ if ( hash_%(constant_identifier)s == -1 ) puts("Note: Weak hash for: %(constant_identifier)s.");""" % { "constant_identifier" : constant_identifier } ) def __addConstantInitCode(context, emit, check, constant_type, constant_value, constant_identifier, module_level): """ Emit code for a specific constant to be prepared during init. This may be module or global init. Code makes sure that nested constants belong into the same scope. """ # This has many cases, that all return, and do a lot. # pylint: disable=R0911,R0912,R0914,R0915 # For the module level, we only mean to create constants that are used only # inside of it. For the global level, it must must be single use. if module_level: if context.global_context.getConstantUseCount(constant_identifier) != 1: return else: if context.getConstantUseCount(constant_identifier) == 1: return # Adding it to "done". We cannot have recursive constants, so this is OK # to be done now. done.add(constant_identifier) # Use shortest code for ints and longs. if constant_type is long: # See above, same for long values. Note: These are of course not # existent with Python3 which would have covered it before. if constant_value >= 0 and constant_value <= max_unsigned_long: emit ( "%s = PyLong_FromUnsignedLong( %sul );" % ( constant_identifier, constant_value ) ) return elif constant_value < 0 and constant_value >= min_signed_long: emit ( "%s = PyLong_FromLong( %sl );" % ( constant_identifier, constant_value ) ) return elif constant_value == min_signed_long-1: # There are compilers out there, that give warnings for the literal # MININT when used. We work around that warning here. emit( """\ %s = PyLong_FromLong( %sl ); // To be corrected with -1 in-place next lines. CHECK_OBJECT( const_int_pos_1 ); %s = PyNumber_InPlaceSubtract( %s, PyLong_FromLong( 1 ) );""" % ( constant_identifier, min_signed_long, constant_identifier, constant_identifier ) ) return else: # Note, other longs cannot be handled like that yet. We might create # code that does it better in the future, abusing e.g. internal # representation of "long" integer values. pass elif constant_type is int: if constant_value >= min_signed_long: emit( "%s = PyInt_FromLong( %sl );" % ( constant_identifier, constant_value ) ) return else: # There are compilers out there, that give warnings for the literal # MININT when used. We work around that warning here. assert constant_value == min_signed_long-1 emit( """\ %s = PyInt_FromLong( %sl ); // To be corrected in next line. %s = PyNumber_InPlaceSubtract( %s, PyInt_FromLong( 1 ) );""" % ( constant_identifier, min_signed_long, constant_identifier, constant_identifier ) ) return if constant_type is unicode: # Attempting to marshal is OK, but esp. Python2 cannot do it for all # "unicode" values. if attemptToMarshal(constant_identifier, constant_value, emit): return try: encoded = constant_value.encode("utf-8") if str is not unicode: emit( "%s = UNSTREAM_UNICODE( %s );" % ( constant_identifier, stream_data.getStreamDataCode(encoded) ) ) else: emit( "%s = UNSTREAM_STRING( %s, %d );" % ( constant_identifier, stream_data.getStreamDataCode(encoded), 1 if _isAttributeName(constant_value) else 0 ) ) return except UnicodeEncodeError: # So fall back to below code, which will unstream it then. pass elif constant_type is str: # Python3: Strings that can be encoded as UTF-8 are done more or less # directly. When they cannot be expressed as UTF-8, that is rare not we # can indeed use pickling. assert str is not unicode if len(constant_value) == 1: emit( "%s = UNSTREAM_CHAR( %d, %d );" % ( constant_identifier, ord(constant_value[0]), 1 if _isAttributeName(constant_value) else 0 ) ) else: emit( "%s = UNSTREAM_STRING( %s, %d );" % ( constant_identifier, stream_data.getStreamDataCode(constant_value), 1 if _isAttributeName(constant_value) else 0 ) ) return elif constant_type is bytes: # Python3 only, for Python2, bytes do not happen. assert str is unicode emit( "%s = UNSTREAM_BYTES( %s );" % ( constant_identifier, stream_data.getStreamDataCode(constant_value) ) ) return if constant_type is float: emit( "%s = UNSTREAM_FLOAT( %s );" % ( constant_identifier, stream_data.getStreamDataCode( value = struct.pack("= 32 and cv <= 127: if octal and c in b'0123456789': result += '" "' result += chr(cv) octal = False else: result += r'\%o' % cv octal = True result = result.replace('" "\\', '\\') return '"%s"' % result def encodeString(value): """ Encode a string, so that it gives a C++ string literal. """ # Not all compilers don't allow arbitrary large C++ strings. result = _encodeString(value[:16000 ]) value = value[16000:] while len(value) > 0: result += ' ' result += _encodeString(value[:16000 ]) value = value[16000:] return result Nuitka-0.5.18.1/nuitka/codegen/CoroutineCodes.py0000644000175000017500000002145612651071040021633 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled coroutine objects. """ from .ErrorCodes import ( getErrorExitCode, getErrorVariableDeclarations, getExceptionKeeperVariableNames, getExceptionPreserverVariableNames, getReleaseCode ) from .Helpers import generateChildExpressionsCode from .Indentation import indented from .templates.CodeTemplatesCoroutines import ( template_coroutine_exception_exit, template_coroutine_noexception_exit, template_coroutine_object_body_template, template_coroutine_object_decl_template, template_coroutine_return_exit, template_make_coroutine_with_context_template, template_make_coroutine_without_context_template ) from .templates.CodeTemplatesFunction import ( function_dict_setup, template_function_closure_making ) from .VariableCodes import getLocalVariableInitCode, getVariableCode def getCoroutineObjectDeclCode(function_identifier): return template_coroutine_object_decl_template % { "function_identifier" : function_identifier, } def getCoroutineObjectCode(context, function_identifier, user_variables, temp_variables, function_codes, needs_exception_exit, needs_generator_return): function_locals = [] for user_variable in user_variables + temp_variables: function_locals.append( getLocalVariableInitCode( variable = user_variable, ) ) if context.hasLocalsDict(): function_locals += function_dict_setup.split('\n') if context.needsExceptionVariables(): function_locals.extend(getErrorVariableDeclarations()) for keeper_index in range(1, context.getKeeperVariableCount()+1): function_locals.extend(getExceptionKeeperVariableNames(keeper_index)) for preserver_id in context.getExceptionPreserverCounts(): function_locals.extend(getExceptionPreserverVariableNames(preserver_id)) function_locals += [ "%s%s%s;" % ( tmp_type, ' ' if not tmp_type.endswith('*') else "", tmp_name ) for tmp_name, tmp_type in context.getTempNameInfos() ] function_locals += context.getFrameDeclarations() # TODO: Could avoid this unless try/except or try/finally with returns # occur. if context.hasTempName("generator_return"): function_locals.append("tmp_generator_return = false;") if context.hasTempName("return_value"): function_locals.append("tmp_return_value = NULL;") for tmp_name, tmp_type in context.getTempNameInfos(): if tmp_name.startswith("tmp_outline_return_value_"): function_locals.append("%s = NULL;" % tmp_name) if needs_exception_exit: generator_exit = template_coroutine_exception_exit % { "function_identifier" : function_identifier, } else: generator_exit = template_coroutine_noexception_exit % { "function_identifier" : function_identifier, } if needs_generator_return: generator_exit += template_coroutine_return_exit % {} return template_coroutine_object_body_template % { "function_identifier" : function_identifier, "function_body" : indented(function_codes), "function_var_inits" : indented(function_locals), "coroutine_exit" : generator_exit } def generateMakeCoroutineObjectCode(to_name, expression, emit, context): coroutine_object_body = expression.getCoroutineRef().getFunctionBody() closure_variables = coroutine_object_body.getClosureVariables() code_identifier = context.getCodeObjectHandle( code_object = expression.getCodeObject(), filename = coroutine_object_body.getParentModule().getRunTimeFilename(), line_number = coroutine_object_body.getSourceReference().getLineNumber(), is_optimized = True, new_locals = not coroutine_object_body.needsLocalsDict(), has_closure = len(closure_variables) > 0, future_flags = coroutine_object_body.getSourceReference().getFutureSpec().asFlags() ) if closure_variables: # TODO: Copy duplication with generator codes, ought to be shared. closure_copy = [] for count, variable in enumerate(closure_variables): variable_code = getVariableCode( context = context, variable = variable ) # Generators might not use them, but they still need to be put there. # TODO: But they don't have to be cells. if not variable.isSharedTechnically(): closure_copy.append( "closure[%d] = PyCell_NEW0( %s );" % ( count, variable_code ) ) else: closure_copy.append( "closure[%d] = %s;" % ( count, variable_code ) ) closure_copy.append( "Py_INCREF( closure[%d] );" % count ) closure_making = template_function_closure_making % { "closure_copy" : indented(closure_copy), "closure_count" : len(closure_variables) } emit( template_make_coroutine_with_context_template % { "closure_making" : closure_making, "coroutine_identifier" : coroutine_object_body.getCodeName(), "to_name" : to_name, "code_identifier" : code_identifier, "closure_count" : len(closure_variables) } ) else: emit( template_make_coroutine_without_context_template % { "coroutine_identifier" : coroutine_object_body.getCodeName(), "to_name" : to_name, "code_identifier" : code_identifier, } ) context.addCleanupTempName(to_name) def generateAsyncWaitCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = %s( coroutine, %s );" % ( to_name, "AWAIT_COROUTINE", value_name if context.needsCleanup(value_name) else "INCREASE_REFCOUNT( %s )" % value_name ) ) if not context.needsCleanup(value_name): context.addCleanupTempName(value_name) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateAsyncIterCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = MAKE_ASYNC_ITERATOR( coroutine, %s );" % ( to_name, value_name ) ) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateAsyncNextCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = ASYNC_ITERATOR_NEXT( coroutine, %s );" % ( to_name, value_name, ) ) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, quick_exception = "StopAsyncIteration", emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.18.1/nuitka/Options.py0000644000175000017500000005652112651072223016742 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Options module """ version_string = """\ Nuitka V0.5.18.1 Copyright (C) 2016 Kay Hayen.""" import logging import sys from optparse import SUPPRESS_HELP, OptionGroup, OptionParser from nuitka.PythonVersions import ( getSupportedPythonVersions, getSupportedPythonVersionStr, python_version_str ) from nuitka.utils import Utils # Indicator if we were called as "nuitka-run" in which case we assume some # other defaults and work a bit different with parameters. is_nuitka_run = Utils.basename(sys.argv[0]).lower().startswith("nuitka-run") def getVersion(): return version_string.split()[1][1:] def getYear(): return int(version_string.split()[4]) if not is_nuitka_run: usage = "usage: %prog [--module] [--execute] [options] main_module.py" else: usage = "usage: %prog [options] main_module.py" parser = OptionParser( usage = usage, version = getVersion() ) # This option is obsolete, and module should be used. parser.add_option( "--exe", action = "store_true", dest = "obsolete_executable", default = False, help = SUPPRESS_HELP ) parser.add_option( "--module", action = "store_false", dest = "executable", default = True, help = """\ Create an extension module executable instead of a program. Defaults to off.""" ) parser.add_option( "--standalone", "--portable", action = "store_true", dest = "is_standalone", default = False, help = """\ Enable standalone mode in build. This allows you to transfer the created binary to other machines without it relying on an existing Python installation. It implies these options: "--recurse-all --recurse-stdlib". You may also want to use "--python-flag=no_site" to avoid the "site.py" module. Defaults to off.""", ) parser.add_option( "--nofreeze-stdlib", action = "store_false", dest = "freeze_stdlib", default = True, help = """\ In standalone mode by default all modules of standard library will be frozen as bytecode. This compiles them all and as a result compilation time will increase very much. """, ) parser.add_option( "--python-version", action = "store", dest = "python_version", choices = getSupportedPythonVersions(), default = None, help = """\ Major version of Python to be used, one of %s. Defaults to what you run Nuitka with (currently %s)""" % ( getSupportedPythonVersionStr(), python_version_str ) ) parser.add_option( "--python-debug", "--python-dbg", action = "store_true", dest = "python_debug", default = None, help = """\ Use debug version or not. Default uses what you are using to run Nuitka, most likely a non-debug version.""" ) parser.add_option( "--python-flag", action = "append", dest = "python_flags", default = [], help = """\ Python flags to use. Default uses what you are using to run Nuitka, this enforces a specific mode. These are options that also exist to standard Python executable. Currently supported: "-S" (alias nosite), "static_hashes" (not use Randomization), "no_warnings" (do not give Python runtime warnings). Default empty.""" ) parser.add_option( "--warn-implicit-exceptions", action = "store_true", dest = "warn_implicit_exceptions", default = False, help = """\ Given warnings for implicit exceptions detected at compile time.""", ) recurse_group = OptionGroup( parser, "Control the recursion into imported modules" ) recurse_group.add_option( "--recurse-stdlib", action = "store_true", dest = "recurse_stdlib", default = False, help = """\ Also descend into imported modules from standard library. Defaults to off.""" ) recurse_group.add_option( "--recurse-none", action = "store_true", dest = "recurse_none", default = False, help = """\ When --recurse-none is used, do not descend into any imported modules at all, overrides all other recursion options. Defaults to off.""" ) recurse_group.add_option( "--recurse-all", "--recurse-on", action = "store_true", dest = "recurse_all", default = False, help = """\ When --recurse-all is used, attempt to descend into all imported modules. Defaults to off.""" ) recurse_group.add_option( "--recurse-to", action = "append", dest = "recurse_modules", metavar = "MODULE/PACKAGE", default = [], help = """\ Recurse to that module, or if a package, to the whole package. Can be given multiple times. Default empty.""" ) recurse_group.add_option( "--recurse-not-to", action = "append", dest = "recurse_not_modules", metavar = "MODULE/PACKAGE", default = [], help = """\ Do not recurse to that module, or if a package, to the whole package in any case, overrides all other options. Can be given multiple times. Default empty.""" ) recurse_group.add_option( "--recurse-plugins", "--recurse-directory", action = "append", dest = "recurse_extra", metavar = "MODULE/PACKAGE", default = [], help = """\ Recurse into that directory, no matter if it's used by the given main program in a visible form. Overrides all other recursion options. Can be given multiple times. Default empty.""" ) recurse_group.add_option( "--recurse-files", "--recurse-pattern", action = "append", dest = "recurse_extra_files", metavar = "PATTERN", default = [], help = """\ Recurse into files matching the PATTERN. Overrides all recursion other options. Can be given multiple times. Default empty.""" ) parser.add_option_group(recurse_group) execute_group = OptionGroup( parser, "Immediate execution after compilation" ) execute_group.add_option( "--run", "--execute", action = "store_true", dest = "immediate_execution", default = is_nuitka_run, help = """\ Execute immediately the created binary (or import the compiled module). Defaults to %s.""" % ("on" if is_nuitka_run else "off") ) execute_group.add_option( "--debugger", "--gdb", action = "store_true", dest = "debugger", default = False, help = """\ Execute inside "gdb" to automatically get a stack trace. Defaults to off.""" ) execute_group.add_option( "--execute-with-pythonpath", "--keep-pythonpath", action = "store_true", dest = "keep_pythonpath", default = False, help = """\ When immediately executing the created binary (--execute), don't reset PYTHONPATH. When all modules are successfully included, you ought to not need PYTHONPATH anymore.""" ) parser.add_option_group(execute_group) dump_group = OptionGroup( parser, "Dump options for internal tree" ) dump_group.add_option( "--dump-xml", "--xml", action = "store_true", dest = "dump_xml", default = False, help = "Dump the final result of optimization as XML, then exit." ) dump_group.add_option( "--display-tree", action = "store_true", dest = "display_tree", default = False, help = """\ Display the final result of optimization in a GUI, then exit.""" ) parser.add_option_group(dump_group) codegen_group = OptionGroup( parser, "Code generation choices" ) codegen_group.add_option( "--improved", "--enhanced", action = "store_true", dest = "improved", default = False, help = """\ Allow minor deviations from CPython behavior, e.g. better tracebacks, which are not really incompatible, but different.""", ) codegen_group.add_option( "--code-gen-no-statement-lines", action = "store_false", dest = "statement_lines", default = True, help = SUPPRESS_HELP, # help = """\ # Statements shall have their line numbers set. Disable this for less precise # exceptions and slightly faster code. Not recommended. Defaults to off.""" ) codegen_group.add_option( "--file-reference-choice", action = "store", dest = "file_reference_mode", choices = ("original", "runtime", "frozen"), default = None, help = """\ Select what value "__file__" is going to be. With "runtime" (default for standalone binary mode and module mode), the created binaries and modules, use the location of themselves to deduct the value of "__file__". Included packages pretend to be in directories below that location. This allows you to include data files in deployments. If you merely seek acceleration, it's better for you to use the "original" value, where the source files location will be used. With "frozen" a notation "" is used. For compatibility reasons, the "__file__" value will always have ".py" suffix independent of what it really is.""" ) codegen_group.add_option( "--no-optimization", action = "store_true", dest = "no_optimize", default = False, help = SUPPRESS_HELP # """Disable all unnecessary optimizations on Python level. Defaults to off.""" ) parser.add_option_group(codegen_group) outputdir_group = OptionGroup( parser, "Output directory choices" ) outputdir_group.add_option( "--output-dir", action = "store", dest = "output_dir", metavar = "DIRECTORY", default = "", help = """\ Specify where intermediate and final output files should be put. DIRECTORY will be populated with C++ files, object files, etc. Defaults to current directory. """ ) outputdir_group.add_option( "--remove-output", action = "store_true", dest = "remove_build", default = False, help = """\ Removes the build directory after producing the module or exe file. Defaults to off.""" ) parser.add_option_group(outputdir_group) windows_group = OptionGroup( parser, "Windows specific output control" ) debug_group = OptionGroup( parser, "Debug features" ) debug_group.add_option( "--debug", action = "store_true", dest = "debug", default = False, help = """\ Executing all self checks possible to find errors in Nuitka, do not use for production. Defaults to off.""" ) debug_group.add_option( "--unstripped", "--no-strip", "--unstriped", action = "store_true", dest = "unstripped", default = False, help = """\ Keep debug info in the resulting object file for better debugger interaction. Defaults to off.""" ) debug_group.add_option( "--profile", action = "store_true", dest = "profile", default = False, help = """\ Enable vmprof based profiling of time spent. Defaults to off.""" ) debug_group.add_option( "--graph", action = "store_true", dest = "graph", default = False, help = """\ Create graph of optimization process. Defaults to off.""" ) debug_group.add_option( "--trace-execution", action = "store_true", dest = "trace_execution", default = False, help = """\ Traced execution output, output the line of code before executing it. Defaults to off.""" ) debug_group.add_option( "--recompile-c++-only", action = "store_true", dest = "recompile_cpp_only", default = False, help = """\ Take existing files and compile them again.Allows compiling edited C++ files with the C++ compiler for quick debugging changes to the generated source. Defaults to off. Depends on compiling Python source to determine which files it should look at.""" ) debug_group.add_option( "--generate-c++-only", action = "store_true", dest = "generate_cpp_only", default = False, help = """\ Generate only C++ source code, and do not compile it to binary or module. This is for debugging and code coverage analysis that doesn't waste CPU. Defaults to off.""" ) debug_group.add_option( "--experimental", action = "store_true", dest = "experimental", default = False, help = """\ Use features declared as 'experimental'. May have no effect if no experimental features are present in the code. Defaults to off.""" ) debug_group.add_option( "--explain-imports", action = "store_true", dest = "explain_imports", default = False, help = SUPPRESS_HELP ) # This is for testing framework, "coverage.py" hates to loose the process. And # we can use it to make sure it's not done unknowingly. parser.add_option( "--must-not-re-execute", action = "store_false", dest = "allow_reexecute", default = True, help = SUPPRESS_HELP ) parser.add_option_group(debug_group) cpp_compiler_group = OptionGroup( parser, "Backend C++ compiler choice" ) cpp_compiler_group.add_option( "--clang", action = "store_true", dest = "clang", default = False, help = """\ Enforce the use of clang (needs clang 3.2 or higher). Defaults to off.""" ) cpp_compiler_group.add_option( "--mingw", action = "store_true", dest = "mingw", default = False, help = """\ Enforce the use of MinGW on Windows. Defaults to off.""" ) cpp_compiler_group.add_option( "--msvc", action = "store", dest = "msvc", default = None, help = """\ Enforce the use of specific MSVC version on Windows. Allowed values are e.g. 9.0, 9.0exp, specify an illegal value for a list of installed compilers. Defaults to the most recent version.""" ) cpp_compiler_group.add_option( "-j", "--jobs", action = "store", dest = "jobs", metavar = 'N', default = Utils.getCoreCount(), help = """\ Specify the allowed number of parallel C++ compiler jobs. Defaults to the system CPU count.""", ) cpp_compiler_group.add_option( "--lto", action = "store_true", dest = "lto", default = False, help = """\ Use link time optimizations if available and usable (g++ 4.6 and higher). Defaults to off.""" ) parser.add_option_group(cpp_compiler_group) tracing_group = OptionGroup( parser, "Tracing features" ) tracing_group.add_option( "--show-scons", action = "store_true", dest = "show_scons", default = False, help = """\ Operate Scons in non-quiet mode, showing the executed commands. Defaults to off.""" ) tracing_group.add_option( "--show-progress", action = "store_true", dest = "show_progress", default = False, help = """Provide progress information and statistics. Defaults to off.""" ) tracing_group.add_option( "--show-memory", action = "store_true", dest = "show_memory", default = False, help = """Provide memory information and statistics. Defaults to off.""" ) tracing_group.add_option( "--show-modules", action = "store_true", dest = "show_inclusion", default = False, help = """Provide a final summary on included modules. Defaults to off.""" ) tracing_group.add_option( "--verbose", action = "store_true", dest = "verbose", default = False, help = """\ Output details of actions taken, esp. in optimizations. Can become a lot. Defaults to off.""" ) parser.add_option_group(tracing_group) windows_group.add_option( "--windows-disable-console", action = "store_true", dest = "win_disable_console", default = False, help = """\ When compiling for Windows, disable the console window. Defaults to off.""" ) windows_group.add_option( "--windows-icon", "--icon", action = "store", dest = "icon_path", metavar = "ICON_PATH", default = None, help = "Add executable icon (Windows only).", ) parser.add_option_group(windows_group) plugin_group = OptionGroup( parser, "Plugin control" ) plugin_group.add_option( "--plugin-enable", "--enable-plugin", action = "append", dest = "plugins_enabled", default = [], help = """\ Enabled plugins. Must be plug-in names. Use --plugin-list to query the full list and exit. Default empty.""" ) plugin_group.add_option( "--plugin-disable", "--disable-plugin", action = "append", dest = "plugins_disabled", default = [], help = """\ Disabled plugins. Must be plug-in names. Use --plugin-list to query the full list and exit. Default empty.""" ) plugin_group.add_option( "--plugin-no-detection", action = "store_false", dest = "detect_missing_plugins", default = True, help = """\ Plugins can detect if they might be used, and the you can disable the warning via --plugin-disable=plugin-that-warned, or you can use this option to disable the mechanism entirely, which also speeds up compilation slightly of course as this detection code is run in vain once you are certain of which plug-ins to use. Defaults to off.""" ) parser.add_option_group(plugin_group) # First, isolate the first non-option arguments. if is_nuitka_run: count = 0 for count, arg in enumerate(sys.argv): if count == 0: continue if arg[0] != '-': break # Treat "--" as a terminator. if arg == "--": count += 1 break if count > 0: extra_args = sys.argv[count+1:] sys.argv = sys.argv[0:count+1] else: extra_args = [] options, positional_args = parser.parse_args() if not positional_args: parser.print_help() sys.exit(""" Error, need positional argument with python module or main program.""") if not options.immediate_execution and len(positional_args) > 1: parser.print_help() sys.exit(""" Error, need only one positional argument unless --run is specified.""") if options.verbose: logging.getLogger().setLevel(logging.DEBUG) else: logging.getLogger().setLevel(logging.INFO) # Standalone mode implies an executable, not importing "site" module, which is # only for this machine, recursing to all modules, and even including the # standard library. if options.is_standalone: options.executable = True options.recurse_all = True def shallTraceExecution(): return options.trace_execution def shallExecuteImmediately(): return options.immediate_execution def shallRunInDebugger(): return options.debugger def shallDumpBuiltTreeXML(): return options.dump_xml def shallDisplayBuiltTree(): return options.display_tree def shallOnlyExecCppCall(): return options.recompile_cpp_only def shallNotDoExecCppCall(): return options.generate_cpp_only def shallHaveStatementLines(): return options.statement_lines def getFileReferenceMode(): if options.file_reference_mode is None: value = ("runtime" if shallMakeModule() or isStandaloneMode() else "original") else: value = options.file_reference_mode return value def shallMakeModule(): return not options.executable def shallFollowStandardLibrary(): return options.recurse_stdlib def shallFollowNoImports(): return options.recurse_none def shallFollowAllImports(): return options.recurse_all def getShallFollowModules(): return sum([ x.split(',') for x in options.recurse_modules ], []) def isAllowedToReexecute(): return options.allow_reexecute for any_case_module in getShallFollowModules(): if any_case_module.startswith('.'): bad = True else: for char in "/\\:": if char in any_case_module: bad = True break else: bad = False if bad: sys.exit(""" Error, '--recurse-to' takes only module names, not directory path '%s'.""" % \ any_case_module) def getShallFollowInNoCase(): return sum([ x.split(',') for x in options.recurse_not_modules ], []) for no_case_module in getShallFollowInNoCase(): if no_case_module.startswith('.'): bad = True else: for char in "/\\:": if char in no_case_module: bad = True break else: bad = False if bad: sys.exit(""" Error, '--recurse-not-to' takes only module names, not directory path '%s'.""" % \ no_case_module) def getShallFollowExtra(): return sum([ x.split(',') for x in options.recurse_extra ], []) def getShallFollowExtraFilePatterns(): return sum([ x.split(',') for x in options.recurse_extra_files ], []) def shallWarnImplicitRaises(): return options.warn_implicit_exceptions def isDebug(): return options.debug or options.debugger def isPythonDebug(): return options.python_debug or sys.flags.debug def isOptimize(): return not options.no_optimize def isUnstripped(): return options.unstripped or options.profile def isProfile(): return options.profile def shouldCreateGraph(): return options.graph def getOutputPath(path): if options.output_dir: return Utils.normpath(Utils.joinpath(options.output_dir, path)) else: return path def getOutputDir(): return options.output_dir if options.output_dir else '.' def getPositionalArgs(): return tuple(positional_args) def getMainArgs(): return tuple(extra_args) def shallOptimizeStringExec(): return False def shallClearPythonPathEnvironment(): return not options.keep_pythonpath def isShowScons(): return options.show_scons def getJobLimit(): return int(options.jobs) def isLto(): return options.lto def isClang(): return options.clang def isMingw(): return options.mingw def getMsvcVersion(): return options.msvc def shallDisableConsoleWindow(): return options.win_disable_console def isFullCompat(): return not options.improved def isShowProgress(): return options.show_progress def isShowMemory(): return options.show_memory def isShowInclusion(): return options.show_inclusion def isRemoveBuildDir(): return options.remove_build and not options.generate_cpp_only def getIntendedPythonVersion(): return options.python_version def isExperimental(): return hasattr(options, "experimental") and options.experimental def shallExplainImports(): return options.explain_imports def isStandaloneMode(): return options.is_standalone def getIconPath(): return options.icon_path def getPythonFlags(): result = [] for part in options.python_flags: if part in ("-S", "nosite", "no_site"): result.append("no_site") elif part in ("static_hashes", "norandomization", "no_randomization"): result.append("no_randomization") elif part in ("-v", "trace_imports", "trace_import"): result.append("trace_imports") elif part in ("no_warnings", "nowarnings"): result.append("no_warnings") else: logging.warning("Unsupported flag '%s'.", part) return result def shallFreezeAllStdlib(): return options.freeze_stdlib def getPluginsEnabled(): return options.plugins_enabled def getPluginsDisabled(): return options.plugins_disabled def shallDetectMissingPlugins(): return options.detect_missing_plugins def getPluginOptions(plugin_name): # TODO: This should come from command line, pylint: disable=W0613 return {} Nuitka-0.5.18.1/nuitka/profiler/0000755000175000017500000000000012651072347016555 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/profiler/__init__.py0000644000175000017500000000141512651071040020654 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/nuitka/profiler/__main__.py0000644000175000017500000000467512651071040020650 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Profiling for Nuitka and CPython. This provides the capability of comparing performance results of Nuitka and CPython relatively to one another. """ from __future__ import print_function import runpy import sys import tempfile import vmprof # pylint: disable=F0401,I0021 def _namelen(e): if e.startswith("py:"): return len(e.split(':')[1]) else: return len(e) def show(stats): p = stats.top_profile() if not p: print("no stats") return p.sort(key = lambda x: x[1], reverse = True) top = p[0][1] max_len = max([_namelen(e[0]) for e in p]) print(" vmprof output:") print(" %: name:" + ' ' * (max_len - 3) + "location:") for k, v in p: v = "%.1f%%" % (float(v) / top * 100) if v == "0.0%": v = "<0.1%" if k.startswith("py:"): _, func_name, lineno, filename = k.split(':', 3) lineno = int(lineno) print(" %s %s %s:%d" % (v.ljust(7), func_name.ljust(max_len + 1), filename, lineno)) else: print(" %s %s" % (v.ljust(7), k.ljust(max_len + 1))) def main(): with tempfile.NamedTemporaryFile() as prof_file: vmprof.enable(prof_file.fileno(), 0.001) try: program = sys.argv[1] del sys.argv[1] # sys.argv = [args.program] + args.args runpy.run_path(program, run_name = "__main__") except BaseException as e: if not isinstance(e, (KeyboardInterrupt, SystemExit)): raise vmprof.disable() stats = vmprof.read_profile( prof_file.name, virtual_only = True ) show(stats) main() Nuitka-0.5.18.1/nuitka/Builtins.py0000644000175000017500000001461212651071040017067 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-ins module. Information about built-ins of the running Python. """ import functools import sys from types import BuiltinFunctionType, FunctionType, GeneratorType from nuitka.__past__ import iterItems from nuitka.PythonVersions import python_version def _getBuiltinExceptionNames(): def isExceptionName(builtin_name): if builtin_name.endswith("Error") or \ builtin_name.endswith("Exception"): return True elif builtin_name in ("StopIteration", "GeneratorExit", "SystemExit", "NotImplemented", "KeyboardInterrupt", "StopAsyncIteration"): return True else: return False # Hide Python3 changes for built-in exception names try: import exceptions names = [ str(name) for name in dir(exceptions) if isExceptionName(name) ] values = {} for key in names: values[key] = getattr(exceptions, key) for key in dir(sys.modules["__builtin__"]): name = str(key) if isExceptionName(name): names.append(key) values[name] = getattr(sys.modules["__builtin__"], key) except ImportError: exceptions = {} for key, value in sys.modules["builtins"].__dict__.items(): if isExceptionName(key): exceptions[key] = value names = [ key for key, value in exceptions.items() ] values = {} for key, value in exceptions.items(): values[key] = value return names, values builtin_exception_names, builtin_exception_values = _getBuiltinExceptionNames() # Just to make sure it's covering these cases correctly. assert "ValueError" in builtin_exception_names assert "StopIteration" in builtin_exception_names assert "GeneratorExit" in builtin_exception_names assert "AssertionError" in builtin_exception_names assert "BaseException" in builtin_exception_names assert "Exception" in builtin_exception_names assert "NotImplemented" in builtin_exception_names assert "StopAsyncIteration" in builtin_exception_names or python_version < 350 def _getBuiltinNames(): names = [ str(x) for x in __builtins__.keys() ] for builtin_exception_name in builtin_exception_names: if builtin_exception_name in names: names.remove(builtin_exception_name) names.remove("__doc__") names.remove("__name__") names.remove("__package__") warnings = [] for builtin_name in names: if builtin_name.endswith("Warning"): warnings.append(builtin_name) for builtin_name in warnings: names.remove(builtin_name) return names, warnings builtin_names, builtin_warnings = _getBuiltinNames() assert "__import__" in builtin_names assert "int" in builtin_names assert "__doc__" not in builtin_names assert "sys" not in builtin_names builtin_all_names = builtin_names + builtin_exception_names + builtin_warnings def getBuiltinTypeNames(): result = [] for builtin_name in builtin_names: if isinstance(__builtins__[builtin_name],type): result.append(builtin_name) return tuple(sorted(result)) builtin_type_names = getBuiltinTypeNames() def _getAnonBuiltins(): with open(sys.executable) as any_file: anon_names = { # Strangely not Python3 types module "NoneType" : type(None), "ellipsis" : type(Ellipsis), # see above "NotImplementedType" : type(NotImplemented), "function" : FunctionType, "builtin_function_or_method" : BuiltinFunctionType, # Can't really have it any better way. "compiled_function" : BuiltinFunctionType, "generator" : GeneratorType, "compiled_generator" : GeneratorType, # see above "code" : type(_getAnonBuiltins.__code__), "file" : type(any_file) } anon_codes = { "NoneType" : "Py_TYPE( Py_None )", "ellipsis" : "&PyEllipsis_Type", "NotImplementedType" : "Py_TYPE( Py_NotImplemented )", "function" : "&PyFunction_Type", "builtin_function_or_method" : "&PyCFunction_Type", "compiled_function" : "&Nuitka_Function_Type", "compiled_generator" : "&Nuitka_Generator_Type", "code" : "&PyCode_Type", "file" : "&PyFile_Type" } if python_version < 300: from types import ClassType, InstanceType, MethodType anon_names["classobj"] = ClassType anon_codes["classobj"] = "&PyClass_Type" anon_names["instance"] = InstanceType anon_codes["instance"] = "&PyInstance_Type" anon_names["instancemethod"] = MethodType anon_codes["instancemethod"] = "&PyMethod_Type" return anon_names, anon_codes builtin_anon_names, builtin_anon_codes = _getAnonBuiltins() def calledWithBuiltinArgumentNamesDecorator(f): """ Allow a function to be called with an "_arg" if a built-in name. This avoids using built-in names in Nuitka source, while enforcing a policy how to make them pretty. """ @functools.wraps(f) def wrapper(*args, **kw): new_kw = {} for key, value in iterItems(kw): if key in builtin_all_names: key = key + "_arg" new_kw[key] = value return f(*args, **new_kw) return wrapper Nuitka-0.5.18.1/nuitka/PythonVersions.py0000644000175000017500000000570212651071040020310 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Python version specifics. This abstracts the Python version decisions. This makes decisions based on the numbers, and attempts to give them meaningful names. Where possible it should attempt to make run time detections. """ import os import re import sys def getSupportedPythonVersions(): return ("2.6", "2.7", "3.2", "3.3", "3.4", "3.5") def getSupportedPythonVersionStr(): supported_python_versions = getSupportedPythonVersions() supported_python_versions_str = repr(supported_python_versions)[1:-1] supported_python_versions_str = re.sub( r"(.*),(.*)$", r"\1, or\2", supported_python_versions_str ) return supported_python_versions_str def _getPythonVersion(): big, major, minor = sys.version_info[0:3] return big * 100 + major * 10 + minor python_version = _getPythonVersion() python_version_full_str = '.'.join(str(s) for s in sys.version_info[0:3]) python_version_str = '.'.join(str(s) for s in sys.version_info[0:2]) def isAtLeastSubVersion(version): if version < 280 and \ python_version >= 280 and \ python_version < 300: return True if version // 10 != python_version // 10: return False return python_version >= version def doShowUnknownEncodingName(): # Python 3.3.3 or higher does it, 3.4 always did. if python_version >= 333: return True # Python2.7 after 2.7.6 does it. if isAtLeastSubVersion(276): return True # Debian back ports do it. if "2.7.5+" in sys.version or "3.3.2+" in sys.version: return True return False def getErrorMessageExecWithNestedFunction(): """ Error message of the concrete Python in case an exec occurs in a function that takes a closure variable. """ assert python_version < 300 # Need to use "exec" to detect the syntax error, pylint: disable=W0122 try: exec(""" def f(): exec "" def nested(): return closure""") except SyntaxError as e: return e.message.replace("'f'", "'%s'") def isUninstalledPython(): return "Anaconda" in sys.version or \ "WinPython" in sys.version or \ (os.name == "nt" and python_version >= 350) Nuitka-0.5.18.1/nuitka/MainControl.py0000644000175000017500000006132012651071040017521 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This is the main actions of Nuitka. This can do all the steps to translate one module to a target language using the Python C/API, to compile it to either an executable or an extension module. """ import os import shutil import subprocess import sys from logging import info, warning from nuitka.importing import Importing, Recursion from nuitka.plugins.PluginBase import Plugins from nuitka.PythonVersions import isUninstalledPython, python_version from nuitka.tree import SyntaxErrors from nuitka.utils import InstanceCounters, Utils from . import ModuleRegistry, Options, Tracing, TreeXML from .build import SconsInterface from .codegen import CodeGeneration, ConstantCodes, MainCodes from .finalizations import Finalization from .freezer.BytecodeModuleFreezer import generateBytecodeFrozenCode from .freezer.Standalone import ( copyUsedDLLs, detectEarlyImports, detectLateImports ) from .optimizations import Optimization from .tree import Building def createNodeTree(filename): """ Create a node tree. Turn that source code into a node tree structure. If recursion into imported modules is available, more trees will be available during optimization, or immediately through recursed directory paths. """ # First, build the raw node tree from the source code. main_module = Building.buildModuleTree( filename = filename, package = None, is_top = True, is_main = not Options.shallMakeModule() ) ModuleRegistry.addRootModule(main_module) # First remove old object files and old generated files, old binary or # module, and standalone mode program directory if any, they can only do # harm. source_dir = getSourceDirectoryPath(main_module) if not Options.shallOnlyExecCppCall(): cleanSourceDirectory(source_dir) if Options.isStandaloneMode(): standalone_dir = getStandaloneDirectoryPath(main_module) shutil.rmtree(standalone_dir, ignore_errors = True) Utils.makePath(standalone_dir) Utils.deleteFile( path = getResultFullpath(main_module), must_exist = False ) # Second, do it for the directories given. for plugin_filename in Options.getShallFollowExtra(): Recursion.checkPluginPath( plugin_filename = plugin_filename, module_package = None ) for pattern in Options.getShallFollowExtraFilePatterns(): Recursion.checkPluginFilenamePattern( pattern = pattern ) # Then optimize the tree and potentially recursed modules. Optimization.optimize() return main_module def dumpTreeXML(tree): xml_root = tree.asXml() TreeXML.dump(xml_root) def displayTree(tree): # pragma: no cover # Import only locally so the Qt4 dependency doesn't normally come into play # when it's not strictly needed. from .gui import TreeDisplay TreeDisplay.displayTreeInspector(tree) def getTreeFilenameWithSuffix(tree, suffix): return tree.getOutputFilename() + suffix def getSourceDirectoryPath(main_module): assert main_module.isCompiledPythonModule() return Options.getOutputPath( path = Utils.basename( getTreeFilenameWithSuffix(main_module, ".build") ) ) def getStandaloneDirectoryPath(main_module): return Options.getOutputPath( path = Utils.basename( getTreeFilenameWithSuffix(main_module, ".dist") ) ) def getResultBasepath(main_module): assert main_module.isCompiledPythonModule() if Options.isStandaloneMode(): return Utils.joinpath( getStandaloneDirectoryPath(main_module), Utils.basename( getTreeFilenameWithSuffix(main_module, "") ) ) else: return Options.getOutputPath( path = Utils.basename( getTreeFilenameWithSuffix(main_module, "") ) ) def getResultFullpath(main_module): result = getResultBasepath(main_module) if Options.shallMakeModule(): if Utils.getOS() == "Windows": result += ".pyd" else: result += ".so" else: result += ".exe" return result def cleanSourceDirectory(source_dir): if Utils.isDir(source_dir): for path, _filename in Utils.listDir(source_dir): if Utils.getExtension(path) in (".cpp", ".hpp", ".c", ".o", ".os", ".obj", ".bin", ".res", ".rc", ".manifest"): Utils.deleteFile(path, True) else: Utils.makePath(source_dir) static_source_dir = Utils.joinpath( source_dir, "static" ) if Utils.isDir(static_source_dir): for path, _filename in Utils.listDir(static_source_dir): if Utils.getExtension(path) in (".o", ".os", ".obj"): Utils.deleteFile(path, True) win32_source_dir = Utils.joinpath( static_source_dir, "win32_ucontext_src" ) if Utils.getOS() == "Windows": Utils.deleteFile( Utils.joinpath(win32_source_dir, "fibers_win32.obj"), False ) def pickSourceFilenames(source_dir, modules): collision_filenames = set() seen_filenames = set() # Our output. module_filenames = {} def getFilenames(module): base_filename = Utils.joinpath( source_dir, "module." + module.getFullName() if not module.isInternalModule() else module.getFullName() ) # Note: Could detect if the file system is cases sensitive in source_dir # or not, but that's probably not worth the effort. False positives do # no harm at all. collision_filename = Utils.normcase(base_filename) return base_filename, collision_filename # First pass, check for collisions. for module in modules: if module.isPythonShlibModule(): continue base_filename = base_filename, collision_filename = getFilenames(module) if collision_filename in seen_filenames: collision_filenames.add(collision_filename) seen_filenames.add(collision_filename) # Count up for colliding filenames. collision_counts = {} # Second pass, this time sorted, so we get deterministic results. We will # apply an @1/@2 to disambiguate the filenames. for module in sorted(modules, key = lambda x : x.getFullName()): if module.isPythonShlibModule(): continue base_filename = base_filename, collision_filename = getFilenames(module) if collision_filename in collision_filenames: collision_counts[ collision_filename ] = \ collision_counts.get(collision_filename, 0) + 1 hash_suffix = "@%d" % collision_counts[ collision_filename ] else: hash_suffix = "" base_filename += hash_suffix module_filenames[module] = base_filename + ".cpp" return module_filenames standalone_entry_points = [] def makeSourceDirectory(main_module): """ Get the full list of modules imported, create code for all of them. """ # We deal with a lot of details here, but rather one by one, and split makes # no sense, pylint: disable=R0912,R0914 assert main_module.isCompiledPythonModule() # The global context used to generate code. global_context = CodeGeneration.makeGlobalContext() assert main_module in ModuleRegistry.getDoneModules() # We might have chosen to include it as bytecode, and only compiled it for # fun, and to find its imports. In this case, now we just can drop it. Or # a module may shadow a frozen module, but be a different one, then we can # drop the frozen one. # TODO: This really should be done when the compiled module comes into # existence. for module in ModuleRegistry.getDoneUserModules(): if module.isCompiledPythonModule(): uncompiled_module = ModuleRegistry.getUncompiledModule( module_name = module.getFullName(), module_filename = module.getCompileTimeFilename() ) if uncompiled_module is not None: # We now need to decide which one to keep, compiled or uncompiled # module. Some uncompiled modules may have been asked by the user # or technically required. By default, frozen code if it exists # is preferred, as it will be from standalone mode adding it. if uncompiled_module.isUserProvided(): ModuleRegistry.removeDoneModule(module) else: ModuleRegistry.removeUncompiledModule(uncompiled_module) # Lets check if the recurse-to modules are actually present, and warn the # user if one of those was not found. for any_case_module in Options.getShallFollowModules(): for module in ModuleRegistry.getDoneUserModules(): if module.getFullName() == any_case_module: break else: warning( "Didn't recurse to '%s', apparently not used." % \ any_case_module ) # Prepare code generation, i.e. execute finalization for it. for module in ModuleRegistry.getDoneModules(): if module.isCompiledPythonModule(): Finalization.prepareCodeGeneration(module) # Pick filenames. source_dir = getSourceDirectoryPath(main_module) module_filenames = pickSourceFilenames( source_dir = source_dir, modules = ModuleRegistry.getDoneModules() ) # First pass, generate code and use constants doing so, but prepare the # final code generation only, because constants code will be added at the # end only. prepared_modules = {} for module in ModuleRegistry.getDoneModules(): if module.isCompiledPythonModule(): cpp_filename = module_filenames[module] prepared_modules[cpp_filename] = CodeGeneration.prepareModuleCode( global_context = global_context, module = module, module_name = module.getFullName(), ) # Main code constants need to be allocated already too. if module is main_module and not Options.shallMakeModule(): prepared_modules[cpp_filename][1].getConstantCode(0) # Second pass, generate the actual module code into the files. for module in ModuleRegistry.getDoneModules(): if module.isCompiledPythonModule(): cpp_filename = module_filenames[module] template_values, module_context = prepared_modules[cpp_filename] source_code = CodeGeneration.generateModuleCode( module_context = module_context, template_values = template_values ) # The main of an executable module gets a bit different code. if module is main_module and not Options.shallMakeModule(): source_code = MainCodes.generateMainCode( main_module = main_module, context = module_context, codes = source_code ) writeSourceCode( filename = cpp_filename, source_code = source_code ) if Options.isShowInclusion(): info("Included compiled module '%s'." % module.getFullName()) elif module.isPythonShlibModule(): target_filename = Utils.joinpath( getStandaloneDirectoryPath(main_module), *module.getFullName().split('.') ) if Utils.getOS() == "Windows": target_filename += ".pyd" else: target_filename += ".so" target_dir = Utils.dirname(target_filename) if not Utils.isDir(target_dir): Utils.makePath(target_dir) shutil.copy( module.getFilename(), target_filename ) standalone_entry_points.append( (target_filename, module.getPackage()) ) else: assert False, module writeSourceCode( filename = Utils.joinpath(source_dir, "__constants.cpp"), source_code = ConstantCodes.getConstantsDefinitionCode( context = global_context ) ) helper_decl_code, helper_impl_code = CodeGeneration.generateHelpersCode( ModuleRegistry.getDoneUserModules() ) writeSourceCode( filename = Utils.joinpath(source_dir, "__helpers.hpp"), source_code = helper_decl_code ) writeSourceCode( filename = Utils.joinpath(source_dir, "__helpers.cpp"), source_code = helper_impl_code ) def runScons(main_module, quiet): # Scons gets transported many details, that we express as variables, and # have checks for them, leading to many branches, pylint: disable=R0912 python_version_str = "%d.%d" % (sys.version_info[0], sys.version_info[1]) if hasattr(sys, "abiflags"): if Options.isPythonDebug() or \ hasattr(sys, "getobjects"): if sys.abiflags.startswith('d'): python_version_str += sys.abiflags else: python_version_str += 'd' + sys.abiflags else: python_version_str += sys.abiflags def asBoolStr(value): return "true" if value else "false" options = { "name" : Utils.basename( getTreeFilenameWithSuffix(main_module, "") ), "result_name" : getResultBasepath(main_module), "source_dir" : getSourceDirectoryPath(main_module), "debug_mode" : asBoolStr(Options.isDebug()), "python_debug" : asBoolStr(Options.isPythonDebug()), "unstripped_mode" : asBoolStr(Options.isUnstripped()), "module_mode" : asBoolStr(Options.shallMakeModule()), "optimize_mode" : asBoolStr(Options.isOptimize()), "full_compat" : asBoolStr(Options.isFullCompat()), "experimental" : asBoolStr(Options.isExperimental()), "trace_mode" : asBoolStr(Options.shallTraceExecution()), "python_version" : python_version_str, "target_arch" : Utils.getArchitecture(), "python_prefix" : sys.prefix, "nuitka_src" : SconsInterface.getSconsDataPath(), "module_count" : "%d" % ( len(ModuleRegistry.getDoneUserModules()) + 1 ) } # Ask Scons to cache on Windows, except where the directory is thrown # away. On non-Windows you can should use ccache instead. if not Options.isRemoveBuildDir() and Utils.getOS() == "Windows": options["cache_mode"] = "true" if Options.isLto(): options["lto_mode"] = "true" if Options.shallDisableConsoleWindow(): options["win_disable_console"] = "true" if Options.isStandaloneMode(): options["standalone_mode"] = "true" if not Options.isStandaloneMode() and \ not Options.shallMakeModule() and \ isUninstalledPython(): options["uninstalled_python"] = "true" if ModuleRegistry.getUncompiledModules(): options["frozen_modules"] = str( len(ModuleRegistry.getUncompiledModules()) ) if Options.isShowScons(): options["show_scons"] = "true" if Options.isMingw(): options["mingw_mode"] = "true" if Options.getMsvcVersion(): msvc_version = Options.getMsvcVersion() msvc_version = msvc_version.replace("exp", "Exp") if '.' not in msvc_version: msvc_version += ".0" options["msvc_version"] = msvc_version if Options.isClang(): options["clang_mode"] = "true" if Options.getIconPath(): options["icon_path"] = Options.getIconPath() if Options.isProfile(): options["profile_mode"] = "true" return SconsInterface.runScons(options, quiet), options def writeSourceCode(filename, source_code): # Prevent accidental overwriting. When this happens the collision detection # or something else has failed. assert not Utils.isFile(filename), filename if python_version >= 300: with open(filename, "wb") as output_file: output_file.write(source_code.encode("latin1")) else: with open(filename, 'w') as output_file: output_file.write(source_code) def writeBinaryData(filename, binary_data): # Prevent accidental overwriting. When this happens the collision detection # or something else has failed. assert not Utils.isFile(filename), filename assert type(binary_data) is bytes with open(filename, "wb") as output_file: output_file.write(binary_data) def callExec(args, clean_path, add_path): old_python_path = os.environ.get("PYTHONPATH", None) if clean_path and old_python_path is not None: os.environ["PYTHONPATH"] = "" if add_path: if "PYTHONPATH" in os.environ: os.environ["PYTHONPATH"] += ':' + Options.getOutputDir() else: os.environ["PYTHONPATH"] = Options.getOutputDir() # We better flush these, "os.execl" won't do it anymore. sys.stdout.flush() sys.stderr.flush() # Add the main arguments, previous separated. args += Options.getPositionalArgs()[1:] + Options.getMainArgs() Utils.callExec(args) def executeMain(binary_filename, clean_path): args = (binary_filename, binary_filename) if Options.shallRunInDebugger(): args = ("/usr/bin/gdb", "gdb", "-ex=run", "-ex=where", "--args", binary_filename) callExec( clean_path = clean_path, add_path = False, args = args ) def executeModule(tree, clean_path): python_command = "__import__('%s')" % tree.getName() args = ( sys.executable, "python", "-c", python_command, ) callExec( clean_path = clean_path, add_path = True, args = args ) def compileTree(main_module): source_dir = getSourceDirectoryPath(main_module) if not Options.shallOnlyExecCppCall(): # Now build the target language code for the whole tree. makeSourceDirectory( main_module = main_module ) if Options.isStandaloneMode(): for module in detectLateImports(): ModuleRegistry.addUncompiledModule(module) frozen_code = generateBytecodeFrozenCode() if frozen_code is not None: writeSourceCode( filename = Utils.joinpath( source_dir, "__frozen.cpp" ), source_code = frozen_code ) writeBinaryData( filename = Utils.joinpath(source_dir, "__constants.bin"), binary_data = ConstantCodes.stream_data.getBytes() ) else: source_dir = getSourceDirectoryPath(main_module) if not Utils.isFile(Utils.joinpath(source_dir, "__helpers.hpp")): sys.exit("Error, no previous build directory exists.") if Options.isShowProgress() or Options.isShowMemory(): Tracing.printLine( "Total memory usage before running scons: {memory}:".format( memory = Utils.getHumanReadableProcessMemoryUsage() ) ) if Options.isShowMemory(): InstanceCounters.printStats() if Options.shallNotDoExecCppCall(): return True, {} # Run the Scons to build things. result, options = runScons( main_module = main_module, quiet = not Options.isShowScons() ) return result, options data_files = [] def main(): """ Main program flow of Nuitka At this point, options will be parsed already, Nuitka will be executing in the desired version of Python with desired flags, and we just get to execute the task assigned. We might be asked to only re-compile generated C++, dump only an XML representation of the internal node tree after optimization, etc. """ # Main has to fullfil many options, leading to many branches and statements # to deal with them. pylint: disable=R0912 positional_args = Options.getPositionalArgs() assert len(positional_args) > 0 filename = Options.getPositionalArgs()[0] # Inform the importing layer about the main script directory, so it can use # it when attempting to follow imports. Importing.setMainScriptDirectory( main_dir = Utils.dirname(Utils.abspath(filename)) ) # Detect to be frozen modules if any, so we can consider to not recurse # to them. if Options.isStandaloneMode(): for module in detectEarlyImports(): ModuleRegistry.addUncompiledModule(module) if module.getName() == "site": origin_prefix_filename = Utils.joinpath( Utils.dirname(module.getCompileTimeFilename()), "orig-prefix.txt" ) if Utils.isFile(origin_prefix_filename): data_files.append( (filename, "orig-prefix.txt") ) # Turn that source code into a node tree structure. try: main_module = createNodeTree( filename = filename ) except (SyntaxError, IndentationError) as e: # Syntax or indentation errors, output them to the user and abort. sys.exit( SyntaxErrors.formatOutput(e) ) if Options.shallDumpBuiltTreeXML(): for module in ModuleRegistry.getDoneModules(): dumpTreeXML(module) elif Options.shallDisplayBuiltTree(): displayTree(main_module) else: result, options = compileTree( main_module = main_module ) # Exit if compilation failed. if not result: sys.exit(1) if Options.shallNotDoExecCppCall(): sys.exit(0) # Remove the source directory (now build directory too) if asked to. if Options.isRemoveBuildDir(): shutil.rmtree( getSourceDirectoryPath(main_module) ) if Options.isStandaloneMode(): binary_filename = options["result_name"] + ".exe" standalone_entry_points.insert( 0, (binary_filename, None) ) dist_dir = getStandaloneDirectoryPath(main_module) for module in ModuleRegistry.getDoneUserModules(): standalone_entry_points.extend( Plugins.considerExtraDlls(dist_dir, module) ) if Utils.getOS() == "NetBSD": warning("Standalone mode on NetBSD is not functional, due to $ORIGIN linkage not being supported.") copyUsedDLLs( dist_dir = dist_dir, standalone_entry_points = standalone_entry_points ) for source_filename, target_filename in data_files: shutil.copy2( source_filename, Utils.joinpath( getStandaloneDirectoryPath(main_module), target_filename ) ) # Modules should not be executable, but Scons creates them like it, fix # it up here. if Utils.getOS() != "Windows" and Options.shallMakeModule(): subprocess.call( ( "chmod", "-x", getResultFullpath(main_module) ) ) # Execute the module immediately if option was given. if Options.shallExecuteImmediately(): if Options.shallMakeModule(): executeModule( tree = main_module, clean_path = Options.shallClearPythonPathEnvironment() ) else: executeMain( binary_filename = getResultFullpath(main_module), clean_path = Options.shallClearPythonPathEnvironment() ) Nuitka-0.5.18.1/nuitka/build/0000755000175000017500000000000012651072347016032 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/SingleExe.scons0000644000175000017500000013055712651071040020764 0ustar hayenhayen00000000000000# -*- python -*- # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # The Nuitka scons file. If you have Scons or platform knowledge, please be # especially invited and contribute improvements. # # This file is used to build an executable or shared library. Nuitka needs no # build process for itself, although it can be compiled using the same method. import os import platform import re import shutil import signal import subprocess import sys ARGUMENTS = ARGUMENTS # @UndefinedVariable # The directory containing the C++ files generated by Nuitka to be built using # scons. They are referred to as sources from here on. The ARGUMENTS is what # Scons provides to us. source_name = ARGUMENTS["name"] source_dir = ARGUMENTS["source_dir"] # The directory containing Nuitka provided C++ files to be built and where it # should be used. nuitka_src = ARGUMENTS["nuitka_src"] static_src = os.path.join(source_dir, "static") # We tell Scons that it is a variant to be built, so object files don't end up # inside that directory which need not be writable. VariantDir(static_src, os.path.join(nuitka_src, "static_src"), 0) # @UndefinedVariable # The name of executable or extension module that we produce. result_basepath = ARGUMENTS["result_name"] def getBoolOption(option_name, default): """ Small helper for boolean mode flags.""" value = ARGUMENTS.get(option_name, "True" if default else "False") return value.lower() in ("yes", "true", '1') # The directory to use for cache directory. cache_mode = getBoolOption("cache_mode", False) # Module mode: Create a Python extension module, create an executable otherwise. module_mode = getBoolOption("module_mode", False) # Debug mode: Less optimizations, debug information in the resulting binary. debug_mode = getBoolOption("debug_mode", False) # Profiling mode: Outputs vmprof based information from program run. profile_mode = getBoolOption("profile_mode", False) # Python version to target. python_version = ARGUMENTS["python_version"] # Python debug mode: reference count checking, assertions in CPython core. python_debug = getBoolOption("python_debug", False) # Optimization mode: Optimize as much as currently possible. optimize_mode = getBoolOption("optimize_mode", True) # Full compatibility, even where it's stupid, i.e. do not provide information, # even if available, in order to assert maximum compatibility. Intended to # control level of compatibility to absurd. full_compat_mode = getBoolOption("full_compat", False) # Experimental mode. Do things that are not yet safe to do. experimental_mode = getBoolOption("experimental", False) # Tracing mode. Output program progress. trace_mode = getBoolOption("trace_mode", False) # LTO mode: Use link time optimizations of g++ compiler if available and known # good with the compiler in question. The 4.5 one didn't have good enough # support, the compiled result would not run correctly. lto_mode = getBoolOption("lto_mode", False) # Windows target mode: Compile for Windows. Used to be an option, but we # no longer cross compile this way. win_target = os.name == "nt" # Windows subsystem mode: Disable console for windows builds. win_disable_console = getBoolOption("win_disable_console", False) # Windows might be running a Python whose DLL we have to use. uninstalled_python = getBoolOption("uninstalled_python", False) # Unstriped mode: Do not remove debug symbols. unstripped_mode = getBoolOption("unstripped_mode", False) # Clang compiler mode, default on MacOS X and FreeBSD, optional on Linux. clang_mode = getBoolOption("clang_mode", False) if sys.platform == "darwin" or "freebsd" in sys.platform: clang_mode = True # MinGW compiler mode, optional and interesting to Windows only. mingw_mode = getBoolOption("mingw_mode", False) # Shared library and compiled modules count, determines the need for the # compiled module loader. module_count = int(ARGUMENTS["module_count"]) # Frozen modules count, determines the need for the bytecode frozen # modules loader. frozen_modules = int(ARGUMENTS.get("frozen_modules", 0)) # Standalone mode standalone_mode = getBoolOption("standalone_mode", False) # Show scons mode, output information about Scons operation show_scons_mode = getBoolOption("show_scons", False) # Home of Python to be compiled against, used to find include files and # libraries to link against. python_prefix = ARGUMENTS["python_prefix"] # Target arch, uses for compiler choice and quick linking of constants binary # data. target_arch = ARGUMENTS["target_arch"] # Icon for executable (windows-only) icon_path = ARGUMENTS.get("icon_path", None) # Forced MSVC version (windows-only) msvc_version = ARGUMENTS.get("msvc_version", None) # Amount of jobs to use. job_count = GetOption("num_jobs") # @UndefinedVariable # Self-compiled Windows has different paths somewhat. win_from_source_mode = win_target and \ os.path.exists(os.path.join(python_prefix, "PCBuild")) if "CXX" in os.environ: cxx_dirname = os.path.dirname(os.environ["CXX"]) if os.path.isdir(cxx_dirname): os.environ["PATH"] = os.pathsep.join([ cxx_dirname ] + os.environ["PATH"].split(os.pathsep)) def getExecutablePath(filename, initial): """ Find an execute in either normal PATH, or Scons detected PATH. """ # Variable substitution from environment is needed, because this can contain # "$CC" which should be looked up too. while filename.startswith('$'): filename = env[filename[1:]] # Append ".exe" suffix on Windows if not already present. if win_target and not filename.lower().endswith(".exe"): filename += ".exe" # Either look at the initial "PATH" as given from the outside or look at the # current environment. if initial: search_path = os.environ["PATH"] else: search_path = env._dict["ENV"]["PATH"] # Now check in each path element, much like the shell will. path_elements = search_path.split(os.pathsep) for path_element in path_elements: full = os.path.join(path_element, filename) if os.path.exists(full): return full else: return None def createEnvironment(compiler_tools): args = {} # If we are on Windows, and MinGW is not enforced, lets see if we can # find "cl.exe", and if we do, disable automatic scan. if win_target and \ not mingw_mode and \ (getExecutablePath("cl", initial = True) is not None or \ getExecutablePath("gcc", initial = True) is not None): args["MSVC_USE_SCRIPT"] = False return Environment( # @UndefinedVariable # We want the outside environment to be passed through. ENV = os.environ, # Extra tools configuration for scons. tools = compiler_tools, # The shared libraries should not be named "lib...", because CPython # requires the filename "module_name.so" to load it. SHLIBPREFIX = "", # Under windows, specify the target architecture is needed for Scons # to pick up MSVC. TARGET_ARCH = target_arch, MSVC_VERSION = msvc_version, **args ) if mingw_mode: # Force usage of MinGW. compiler_tools = ["mingw"] else: # Everything else should use default. compiler_tools = ["default"] # Create Scons environment, the main control tool. Don't include "mingw" on # Windows immediately, we will default to MSVC if available. env = createEnvironment( compiler_tools = compiler_tools ) # On Windows, in case MSVC was not found and not previously forced, retry with # it and use that instead then as a fallback. Using both tools in one call # seems to not work in terms of fallback when both exist. if win_target and \ compiler_tools == ["default"] and \ getExecutablePath(env["CXX"], initial = False) is None: env = createEnvironment( compiler_tools = ["mingw"] ) if show_scons_mode: print "Initial CXX:", env.get("CXX", None) print "Initial CXXVERSION:", env.get("CXXVERSION", None) if clang_mode: # If requested by the user, use the clang compiler, overriding what was # said in environment. env["CXX"] = "clang" env["CXXVERSION"] = None elif "CXX" in os.environ: # If the environment variable CXX is set, use that. env["CXX"] = os.environ["CXX"] env["CXXVERSION"] = None # To work around Windows not supporting command lines of greater than 10K by # default: def setupSpawn(env): def spawn(sh, escape, cmd, args, env): # The "del" appears to not work, but is used with large amounts of # files to link. So, lets do this ourselves, plus it avoids a process # spawn. if cmd == "del": assert len(args) == 2 os.unlink(args[1]) return 0 # For quoted arguments that end in a backslash, things don't work well # this is a workaround for it. def removeTrailingSlashQuote(arg): if arg.endswith(r'\"'): return arg[:-1] + '\\"' else: return arg newargs = ' '.join( removeTrailingSlashQuote(arg) for arg in args[1:] ) cmdline = cmd + ' ' + newargs startupinfo = subprocess.STARTUPINFO() # Remove useless quoted include directories to "windres", which does not # handle them properly in its command line parsing, while they are not # used at all. if cmd == "windres": cmdline = re.sub('--include-dir ".*?"', "", cmdline) # CPython2.6 compatibility try: from subprocess import STARTF_USESHOWWINDOW except ImportError: from _subprocess import STARTF_USESHOWWINDOW startupinfo.dwFlags |= STARTF_USESHOWWINDOW proc = subprocess.Popen( cmdline, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, startupinfo = startupinfo, shell = False, env = env ) data, err = proc.communicate() rv = proc.wait() if cmd == "cl": data = data[data.find("\r\n") + 2 :] source_basenames = [ os.path.basename(source_file) for source_file in source_files ] def check(line): return line in ("", "Generating Code...") or \ line in source_basenames data = "\r\n".join( line for line in data.split("\r\n") if not check(line) ) elif cmd == "rc": data = data[data.find("reserved.\r") + 13:] data = '\n'.join( line for line in data.split('\n') if not "identifier truncated to" ) elif cmd == "link" and module_mode: data = "\r\n".join( line for line in data.split("\r\n") if " Creating library" not in line # On localized compilers, the message to ignore is not as clear. if not (module_mode and ".exp" in line) ) if data.rstrip(): if not show_scons_mode: print cmdline print data, if err: print err, return rv env["SPAWN"] = spawn def getGccVersion(): # Update CXXVERSION in env, after we changed it. import SCons # @UnresolvedImport pipe = SCons.Action._subproc( env, [env["CXX"], "--version"], stdin = "devnull", stderr = "devnull", stdout = subprocess.PIPE ) line = pipe.stdout.readline() match = re.search(r'[0-9]+(\.[0-9]+){2}', line) if match: return match.group(0) else: return None def getClangVersion(): import SCons # @UnresolvedImport pipe = SCons.Action._subproc( env, [env["CXX"], "--version"], stdin = "devnull", stderr = "devnull", stdout = subprocess.PIPE ) line = pipe.stdout.readline() match = re.search(r'LLVM ([0-9]+(\.[0-9]+){1})', line) if match: return match.group(1) match = re.search(r'([0-9]+(\.[0-9]+){1})', line) if match: return match.group(1) else: return None # Some versions of Scons have "CXX" not set or None, make it explicit. if env.get("CXX", None) is None: env["CXX"] = None orig_cxx = env["CXX"] orig_cxx_version = env.get("CXXVERSION", None) # Remove "g++" as the compiler, in case it's not really existing or not high # enough version. if "g++" in (env["CXX"] or "") and "clang" not in (env["CXX"] or ""): gpp_version = getGccVersion() if gpp_version is None or int(gpp_version.replace('.', "")) < 440: env["CXX"] = None # Detect compiler to be used from supported ones, if there is no usable g++ # link. if not win_target and env["CXX"] is None: # Try out the Debian names, as no "gcc" default compiler package needs to # be installed. for candidate in ("g++-5.1", "g++-4.9", "g++-4.8", "g++-4.7", "g++-4.6", "g++-4.5", "g++-4.4", "g++44", "clang", "eg++"): for system_path in ("/usr/bin", "/usr/local/bin"): if os.path.exists(os.path.join(system_path, candidate)): env["CXX"] = candidate env["CC"] = candidate.replace("++", "cc") if "clang" not in candidate: # Update CXXVERSION in env, after we changed it. env["CXXVERSION"] = getGccVersion() break if env["CXX"] is not None: break if env["CXX"] is None: sys.exit("""\ The g++ compiler '%s' (version %s) doesn't have the sufficient \ version (>= 4.4).""" % (orig_cxx, orig_cxx_version)) if env["CXX"] is None or \ getExecutablePath(env["CXX"], initial = False) is None: if win_target: sys.exit("""\ Error, cannot locate suitable C++ compiler. You have the following options: a) If a suitable Visual Studio version is installed, it will not be located, automatically unless you install pywin32 for the Python installation below "%s". b) To make it find Visual Studio execute from Start Menu the 'Visual Studio Command Prompt' or "vcvarsall.bat". That will add Visual Studio to the PATH. And it will be detected. c) Install MinGW to 'C:\\MinGW' then it is automatically picked up. But be sure to pick the proper variant (32/64 bits, your Python arch is '%s'). If you have it installed, using the matching Python will help. """ % (sys.exec_prefix, target_arch)) else: sys.exit("Error, cannot locate suitable C++ compiler.") gcc_mode = "g++" in env["CXX"] or "clang" in env["CXX"] msvc_mode = win_target and not gcc_mode mingw_mode = win_target and gcc_mode if msvc_mode: def getMsvcVersionString(): import SCons.Tool.MSCommon.vc # @UnresolvedImport return SCons.Tool.MSCommon.vc.get_default_version(env) def getMsvcVersion(): value = getMsvcVersionString() value = value.replace("exp","") return float(value) if show_scons_mode: print "Scons compiler: Using", print getExecutablePath(env["CXX"], initial = False), if win_target and msvc_mode: print "(MSVC %s)" % getMsvcVersionString() print if win_target: setupSpawn(env) env["BUILD_DIR"] = source_dir # Store the file signatures database with the rest of the source files sconsign_dir = os.path.abspath(os.path.join(source_dir, ".sconsign")) if not os.path.exists(sconsign_dir): os.makedirs(sconsign_dir) env.SConsignFile(sconsign_dir) # Support for clang. if "clang" in env["CXX"]: env.Append( CCFLAGS = ["-w"] ) env.Append( CPPDEFINES = ["_XOPEN_SOURCE"] ) env.Append( LINKFLAGS = ["-lstdc++" ] ) # Don't export anything by default, this should create smaller executables. env.Append( CCFLAGS = [ "-fvisibility=hidden", "-fvisibility-inlines-hidden" ] ) if debug_mode and not clang_mode: env.Append( CCFLAGS = ["-Wunused-but-set-variable"] ) clang_version = getClangVersion() clang_version = int(clang_version.replace('.', "") + '0') if gcc_mode: if not win_target: env.Append( CCFLAGS = ["-fvisibility=hidden"] ) env.Append( CXXFLAGS = ["-fvisibility-inlines-hidden"] ) # Support details for real g++, not clang++. if "g++" in env["CXX"] and "clang" not in env["CXX"]: # Don't export anything by default, this should create smaller executables. if env.get("CXXVERSION", None) is None or \ env.get("CXXVERSION").count('.') < 2: env["CXXVERSION"] = getGccVersion() # Version dependent options. gpp_version = int(env["CXXVERSION"].replace('.', "")) # Enforce the minimum version, selecting a potentially existing g++-4.5 # binary if it's not high enough. This is esp. useful under Debian which # allows all compiler to exist next to each other and where g++ might not be # good enough, but g++-4.5 would be. if gpp_version < 440: sys.exit( """\ The g++ compiler %s (version %s) doesn't have the sufficient \ version (>= 4.4).""" % (env["CXX"], env["CXXVERSION"]) ) # Older g++ complains about aliasing with Py_True and Py_False, but we don't # care. if gpp_version < 450: env.Append( CCFLAGS = ["-fno-strict-aliasing"] ) # For LTO mode, the version requirement is even higher, so try that too. if lto_mode and gpp_version < 460: sys.exit("""\ The g++ compiler %s (version %s) doesn't have the sufficient \ version for lto mode (>= 4.6).""" % (env[ "CXX" ], env[ "CXXVERSION" ])) # For g++ 4.6 there are some new interesting functions. if gpp_version >= 460: env.Append( CCFLAGS = ["-fpartial-inlining"] ) if debug_mode: env.Append( CCFLAGS = ["-Wunused-but-set-variable"] ) # Use link time optimizations so that gcc can help with optimization across # files, but unfortunately at this time it seriously slows down the compiled # code. This may be because it needs -O3 option to be effective. if gpp_version >= 460 and lto_mode: env.Append(CCFLAGS = ["-flto"]) env.Append(LINKFLAGS = ["-flto=%d" % job_count]) # env.Append( LINKFLAGS = [ "-Wsuggest-attribute=noreturn" ] ) # env.Append( LINKFLAGS = [ "-Wsuggest-attribute=pure" ] ) # env.Append( LINKFLAGS = [ "-Wsuggest-attribute=const" ] ) # env.Append( CCFLAGS = [ "-Wnoexcept" ] ) if debug_mode: env.Append(LINKFLAGS = ["-O2"]) if optimize_mode: env.Append( LINKFLAGS = [ "-O3", "-fpartial-inlining", "-freorder-functions", ] ) # Give a warning if LTO mode was specified, but won't be used. if lto_mode and gpp_version < 460: print >> sys.stderr, "Warning, LTO mode specified, but not available." # The var-tracking does not scale, disable it. Should we really need it, we # can enable it. TODO: Does this cause a performance loss? env.Append(CCFLAGS = ["-fno-var-tracking"]) if msvc_mode: env.Append(CCFLAGS = ["/EHsc", "/J", "/Gd"]) env.Append(LINKFLAGS = ["/INCREMENTAL:NO"]) # Stack size 4MB or 8MB, we need more than the default 1MB. if target_arch == "x86_64": env.Append(CCFLAGS = ["/F8388608"]) else: env.Append(CCFLAGS = ["/F4194304"]) # The 32 bits MinGW does not default for API level properly, so help it. if mingw_mode: # Windows XP env.Append( CPPDEFINES = ["_WIN32_WINNT=0x0501"] ) if debug_mode: if gcc_mode: # Allow g++/clang to point out all kinds of inconsistency to us by # raising an error. env.Append( CCFLAGS = [ "-Wall", "-Werror", # Unfortunately Py_INCREF(Py_False) triggers aliasing warnings, # which are unfounded, so disable them. "-Wno-error=strict-aliasing", "-Wno-strict-aliasing", # At least for self-compiled Python3.2, and MinGW this happens # and has little use anyway. "-Wno-error=format", "-Wno-format" ] ) elif msvc_mode: # Disable warnings that system headers already show. env.Append( CCFLAGS = [ "/W4", "/wd4505", "/wd4127", "/wd4100", "/wd4702", "/wd4189", "/wd4211", "/WX" ] ) # Disable warnings, that CPython headers already show. if python_version >= "3.4": env.Append( CCFLAGS = [ "/wd4512", "/wd4510", "/wd4610" ] ) # As for sequence points, we are abusing it, so we have to allow it. if "g++" in env["CXX"]: env.Append( CCFLAGS = ["-Wno-sequence-point"] ) if full_compat_mode: env.Append( CPPDEFINES = ["_NUITKA_FULL_COMPAT"] ) if experimental_mode: env.Append( CPPDEFINES = ["_NUITKA_EXPERIMENTAL"] ) if profile_mode: env.Append( CPPDEFINES = ["_NUITKA_PROFILE"] ) if trace_mode: env.Append( CPPDEFINES = ["_NUITKA_TRACE"] ) if standalone_mode: env.Append( CPPDEFINES = ["_NUITKA_STANDALONE"] ) if "linux" in sys.platform: env.Append( LIBS = ["dl"] ) if win_target: # For MinGW and cross compilation, we need to tell the subsystem # to target as well as to automatically import everything used. if gcc_mode: env.Append( LINKFLAGS = ["-Wl,--enable-auto-import"] ) if win_disable_console: env.Append( LINKFLAGS = ["-Wl,--subsystem,windows"] ) if win_disable_console: env.Append( CPPDEFINES = ["_NUITKA_WINMAIN_ENTRY_POINT"] ) # This is used for "PathRemoveFileSpec" used in Windows code. env.Append( LIBS = ["Shlwapi"] ) # This is used for "GetCommandLineW" used in Windows code for Python3. if python_version >= "3": env.Append( LIBS = ["Shell32"] ) if python_debug: env.Append( CPPDEFINES = ["Py_DEBUG"] ) def detectHostMultiarch(): import commands for line in commands.getoutput("dpkg-architecture").split('\n'): if line.startswith("DEB_HOST_MULTIARCH="): return line.split('=', 1)[1] else: return None if gcc_mode and "linux" in sys.platform: if python_version.startswith("3.3"): host_multiarch = detectHostMultiarch() if host_multiarch is not None: env.Append( CCFLAGS = [ "-I" + os.path.join( "/usr/include/", host_multiarch, "python" + python_version ) ] ) # For Nuitka, it generally is OK to break out of the virtualenv, and use the # original install. Mind you, this is not about executing anything, this is # about building, and finding the headers to compile against that Python, we # do not care about any site packages, and so on. if not win_target and os.path.islink(os.path.join(python_prefix, ".Python")): # Some virtualenv, at least on MacOS, have such handy links. python_prefix = os.path.normpath( os.path.join( os.readlink(os.path.join(python_prefix, ".Python")), # @UndefinedVariable ".." ) ) elif not win_target and python_version >= "3.3" and \ os.path.exists(os.path.join(python_prefix, "bin/activate")): # For "venv" of Python3.3, it seems necessary to find include files # and libraries on the outside. python_binary = os.path.join(python_prefix, "bin", "python") python_binary = os.path.realpath(python_binary) python_prefix = os.path.normpath( os.path.join( python_binary, "..", ".." ) ) elif os.path.exists(os.path.join(python_prefix, "Lib/orig-prefix.txt")): python_prefix = open( os.path.join(python_prefix, "Lib/orig-prefix.txt") ).read() # Trailing spaces in the python prefix, please not. assert python_prefix == python_prefix.strip() if win_target: # On Windows, the installation layout is relatively fixed. python_header_path = os.path.join(python_prefix, "include") else: # The python header path is a combination of python version and debug # indication, make sure the headers are found by adding it to the C++ # include path. python_header_path = os.path.join( python_prefix, "include", "python" + python_version ) if not os.path.exists(os.path.join(python_header_path, "Python.h")): # Not, for --python-debug other headers are used than for normal # compilation. sys.exit( """\ Error, no 'Python.h' %s headers can be found at '%s', dependency \ not satisfied!""" % ( "debug" if python_debug else "development", python_header_path ) ) env.Append(CPPPATH = [python_header_path]) # To support self built Python on Windows, need to also add the "PC" directory, # that a normal install won't have. if win_from_source_mode: env.Append(CPPPATH = [os.path.join(python_prefix, "PC")]) def getWindowsPythonDLLPath(): dll_filename = "python%s%s.dll" % ( python_version.replace('.', ""), "_d" if python_debug else "" ) if uninstalled_python: return os.path.join( python_prefix, dll_filename ) elif target_arch == "x86_64": return os.path.join( os.environ["WINDIR"], "SysWOW64", dll_filename ) else: return os.path.join( os.environ["WINDIR"], dll_filename ) if win_target: # MinGW for 64 bits needs this due to CPython being compiled with MSVC. if win_from_source_mode: win_lib_path = os.path.join(python_prefix, "PCBuild") else: win_lib_path = os.path.join(python_prefix, "libs") if win_target and target_arch == "x86_64" and gcc_mode: new_win_lib_path = os.path.join(source_dir, "libs") if not os.path.exists(new_win_lib_path): os.makedirs(new_win_lib_path) command = "gendef - %s" % getWindowsPythonDLLPath() defs_filename = os.path.join( new_win_lib_path, "python%s.defs" % python_version.replace('.', "") ) defs_file = open( defs_filename, 'w' ) startupinfo = subprocess.STARTUPINFO() # CPython2.6 compatibility try: from subprocess import STARTF_USESHOWWINDOW except ImportError: from _subprocess import STARTF_USESHOWWINDOW startupinfo.dwFlags |= STARTF_USESHOWWINDOW try: proc = subprocess.Popen( command, stdin = subprocess.PIPE, stdout = defs_file, stderr = subprocess.PIPE, startupinfo = startupinfo, shell = False ) except WindowsError: sys.exit("Error, cannot find gendef.exe which is required though.") data, err = proc.communicate() rv = proc.wait() assert rv == 0 defs_file.close() defs_file = open( defs_filename, 'r' ) defs = defs_file.read() defs_file.close() defs_file = open( defs_filename, 'w' ) defs_file.write(defs.replace("Py_InitModule4", "Py_InitModule4_64")) defs_file.close() shutil.copy( getWindowsPythonDLLPath(), new_win_lib_path ) lib_filename = os.path.join( new_win_lib_path, "libpython%s.a" % python_version.replace('.', "") ) dll_tool = os.path.join( os.path.dirname( getExecutablePath(env["CXX"], initial = False) ), "dlltool" ) result = subprocess.call( ( dll_tool, "-d", defs_filename, "-l", lib_filename ), stdout = subprocess.PIPE ) if result != 0: sys.exit("Error, call to 'dlltool.exe' failed.") win_lib_path = new_win_lib_path env.Append(LIBPATH = [win_lib_path]) if python_debug: env.Append(LIBS = ["python" + python_version.replace('.', "") + "_d"]) else: env.Append(LIBS = ["python" + python_version.replace('.', "")]) else: # Debian and Ubuntu distinguish the system libraries like this. if python_debug and \ python_prefix == "/usr" and \ not python_version.startswith('3') and \ platform.dist()[0].lower() in ("debian", "ubuntu"): env.Append(LIBS = ["python" + python_version + "_d"]) else: env.Append(LIBS = ["python" + python_version]) if python_prefix != "/usr" and "linux" in sys.platform: env.Append( LIBS = ["dl", "pthread", "util", 'm'] ) if gcc_mode and "clang" not in env["CXX"]: env.Append( LINKFLAGS = ["-export-dynamic"] ) # Add the python library path to the library path python_lib_path = os.path.join(python_prefix, "lib") env.Append(LIBPATH = [python_lib_path]) # For NetBSD the rpath is required, on FreeBSD it's warned as unused. if "netbsd" in sys.platform: env.Append( LINKFLAGS = ["-rpath=" + python_lib_path] ) # The static include files reside in Nuitka installation, which may be where # the "nuitka.build" package lives. nuitka_include = os.path.join( nuitka_src, "include" ) # We have include files in the build directory and the static include directory # that is located inside Nuitka installation. env.Append( CPPPATH = [ source_dir, nuitka_include ] ) if debug_mode or unstripped_mode: # Use debug format, so we get good tracebacks from it. if gcc_mode: env.Append( CCFLAGS = ["-g"] ) env.Append( ASFLAGS = ["-g"] ) if "g++" in env["CXX"]: env.Append( CCFLAGS = ["-feliminate-unused-debug-types"] ) elif msvc_mode: env.Append( CCFLAGS = ["/Z7"] ) # Higher MSVC versions need this for parallel compilation if job_count > 1 and getMsvcVersion() >= 11: env.Append( CCFLAGS = ["/FS"] ) env.Append( LINKFLAGS = ["/DEBUG"] ) else: if gcc_mode: if sys.platform == "darwin": env.Append( LINKFLAGS = ["-Wno-deprecated-declarations"] ) else: env.Append( LINKFLAGS = ["-s"] ) # When debugging, optimize less than when optimizing, when not remove # assertions. if debug_mode: if not optimize_mode: if gcc_mode or msvc_mode: env.Append( CCFLAGS = ["-O2"] ) else: env.Append( CPPDEFINES = ["__NUITKA_NO_ASSERT__"] ) if optimize_mode: if gcc_mode: env.Append( CCFLAGS = ["-O3"] ) # Check inlining of calls, except in debug mode, where it will all be # wrong due to additional code. # if not debug_mode: # env.Append( CCFLAGS = [ "-Winline" ] ) elif msvc_mode: env.Append( CCFLAGS = ["/Ox"] ) # MinGW for 64 bits needs this due to CPython bugs. if win_target and target_arch == "x86_64" and gcc_mode: env.Append(CPPDEFINES = ["MS_WIN64"]) # Set load libpython from binary directory default if standalone_mode and gcc_mode and sys.platform != "darwin": env.Append( LINKFLAGS = ["-Wl,-R,'$$ORIGIN'"]) def getLinkerArch(): if "linux" in sys.platform: if target_arch == "x86_64": return "elf64-x86-64" elif target_arch == "armv5tel": # Debian arch "armel" return "elf32-littlearm" elif target_arch == "armv7l": # Debian arch "armhf" return "elf32-littlearm" else: # TODO: Detect from objdump mayhaps. return None else: # TODO: Missing for MacOS, FreeBSD, other Linux return None constants_bin_filename = os.path.join(source_dir,"__constants.bin") if win_target and not module_mode: # On Windows constants are accesses as a resource, except in shared # libraries, where that option is not available. constants_generated_filename = None elif gcc_mode and getLinkerArch() is not None: env.Append( LINKFLAGS = [ "-Wl,-b", "-Wl,binary", "-Wl,%s" % constants_bin_filename, "-Wl,-b", "-Wl,%s" % getLinkerArch(), "-Wl,-defsym", "-Wl,%sconstant_bin=_binary_%s___constants_bin_start" % ( '_' if mingw_mode else "", "".join(re.sub("[^a-zA-Z0-9_]",'_',c) for c in source_dir) ) ] ) constants_generated_filename = None else: constants_generated_filename = os.path.join( source_dir, "__constants_data.c" ) with open(constants_generated_filename, 'w') as output: output.write("const unsigned char constant_bin[] =\n{\n") for count, stream_byte in enumerate( open(constants_bin_filename, "rb").read() ): if count % 16 == 0: if count > 0: output.write('\n') output.write(" ") output.write(" 0x%02x," % ord(stream_byte)) output.write("\n};\n"); env.Append( CPPDEFINES = [ "_NUITKA_FROZEN=%d" % frozen_modules, "_NUITKA_MODULE_COUNT=%d" % module_count ] ) # Tell compiler to create a shared library or program. if module_mode: if "g++" in env["CXX"]: env.Append( CCFLAGS = ["-shared"] ) elif "clang" in env["CXX"]: pass elif msvc_mode: env.Append( CCFLAGS = ["/LD"] ) else: assert False else: if msvc_mode: env.Append( CCFLAGS = ["/MT"] ) if module_mode: env.Append( CPPDEFINES = ["_NUITKA_MODULE"] ) else: env.Append( CPPDEFINES = ["_NUITKA_EXE"] ) def discoverSourceFiles(): result = [] def getStatic(sub_path): return os.path.join(static_src, sub_path.replace('/', os.path.sep)) result.append(getStatic("CompiledFunctionType.cpp")) result.append(getStatic("CompiledMethodType.cpp")) result.append(getStatic("CompiledGeneratorType.cpp")) if python_version >= "3.5": result.append(getStatic("CompiledCoroutineType.cpp")) result.append(getStatic("CompiledFrameType.cpp")) result.append(getStatic("CompiledCodeHelpers.cpp")) result.append(getStatic("InspectPatcher.cpp")) if win_target: result.append(getStatic("win32_ucontext_src/fibers_win32.cpp")) elif "openbsd" in sys.platform: result.append(getStatic("libcoro_ucontext_src/fibers_coro.cpp")) result.append(getStatic("libcoro_ucontext_src/coro.c")) env.Append( CPPDEFINES = ["CORO_SJLJ"] ) elif target_arch == "x86_64" and "linux" in sys.platform: result.append(getStatic("x64_ucontext_src/fibers_x64.cpp")) result.append(getStatic("x64_ucontext_src/swapfiber.S")) elif target_arch == "armv5tel": result.append(getStatic("arm_ucontext_src/fibers_arm.cpp")) result.append(getStatic("arm_ucontext_src/ucontext.cpp")) result.append(getStatic("arm_ucontext_src/getcontext.asm")) else: # Variant based on getcontext/setcontext/swapcontext/makecontext result.append(getStatic("gen_ucontext_src/fibers_gen.cpp")) for filename in os.listdir(source_dir): if filename.endswith(".cpp"): result.append(os.path.join(source_dir, filename)) # If more than one module is included, we need the unfreezer. if module_count > 1: result.append(getStatic("MetaPathBasedLoader.cpp")) if constants_generated_filename is not None: result.append(constants_generated_filename) return result source_targets = [] # Prepare the use of a custom specs file for windows targets. We change the used # specs for linking to avoid the use of the wrong (for CPython) run time # library. if win_target: rc_content = [] # Which files to depend on. rc_file_dependencies = [] if not module_mode: rc_content.append( '3 RCDATA "%s"' % constants_bin_filename.replace('\\', '/') ) rc_file_dependencies.append(constants_bin_filename) if python_version < "3.3": manifest_filename = os.path.join( source_dir, "resources.manifest" ) if win_from_source_mode: # Also consider a Python built from source. python_exe = os.path.join( python_prefix, "PCBuild", "python_d.exe" if python_debug else "python.exe" ) else: # Find the "python.exe" in a normal install, this is assuming we # broke out of the virtualenv already. python_exe = os.path.join( python_prefix, "python.exe" ) assert os.path.exists(python_exe), python_exe if msvc_mode: try: result = subprocess.call( ( env["MT"], "-inputresource:%s;#1" % python_exe, "-out:%s" % manifest_filename ), stdout = subprocess.PIPE ) except OSError: result = -1 if result != 0: sys.exit( """\ Error, call to 'mt.exe' (%s) to extract manifest failed (error code %d.""" % ( env["MT"], result ) ) else: try: result = subprocess.call( ( "windres.exe", "--input", python_exe, "--output", manifest_filename, "--output-format", "rc" ), stdout = subprocess.PIPE ) except WindowsError: sys.exit("Error, cannot find 'windres.exe' but it is required.") if result != 0: sys.exit("Error, call to 'windres.exe' to extract manifest failed.") # Need to reduce to what we use, and remove the quoting of the # actual manifest. We created a full "rc" file, but that we do # not have use for. manifest_file = open(manifest_filename, 'r') inside = False contents = [] for line in manifest_file: if "RT_MANIFEST" in line: inside = True continue if not inside: continue line = line.replace(r"\r\n", "\r\n") line = line.replace('""', '"') contents.append( line[line.find('"')+1:line.rfind('"')] ) manifest_file = open(manifest_filename, 'w') manifest_file.write("".join(contents[1:-1])) manifest_file.close() rc_content.append( "1 RT_MANIFEST resources.manifest" ) rc_file_dependencies.append(manifest_filename) if icon_path: rc_content.append( '2 ICON MOVEABLE PURE LOADONCALL DISCARDABLE "%s"' % ( icon_path.replace('\\', '/') ) ) rc_file_dependencies.append(icon_path) if rc_content: rc_filename = os.path.join(source_dir, "resources.rc") rc_file = open(rc_filename, 'w') rc_content.insert( 0, '#include "winuser.h"' ) rc_file.write('\n'.join(rc_content)) rc_file.close() res_target = env.RES(rc_filename) for rc_file_dependency in rc_file_dependencies: Depends(res_target, rc_file_dependency) # @UndefinedVariable source_targets.append( res_target ) source_files = discoverSourceFiles() if module_mode: # For Python modules, the standard shared library extension is not what # gets used. if win_target: module_suffix = ".pyd" else: module_suffix = ".so" env["SHLIBSUFFIX"] = module_suffix target = env.SharedLibrary( result_basepath, source_files + source_targets ) else: target = env.Program( result_basepath + ".exe", source_files + source_targets ) # Avoid dependency on MinGW libraries. if win_target and gcc_mode: env.Append( LINKFLAGS = [ "-static-libgcc", "-static-libstdc++" ] ) # On some architectures, makecontext cannot pass pointers reliably. if target_arch == "x86_64" and "linux" in sys.platform: env.Append(CPPDEFINES = ["_NUITKA_MAKECONTEXT_INTS"]) # Avoid IO for compilation as much as possible, this should make the # compilation more memory hungry, but also faster. if gcc_mode: env.Append(CCFLAGS = "-pipe") if "CPPFLAGS" in os.environ: env.Append(CCFLAGS = os.environ["CPPFLAGS"].split()) if "CCFLAGS" in os.environ: env.Append(CCFLAGS = os.environ["CCFLAGS"].split()) if "CXXFLAGS" in os.environ: env.Append(CCFLAGS = os.environ["CXXFLAGS"].split()) if "LDFLAGS" in os.environ: env.Append(LINKFLAGS = os.environ["LDFLAGS"].split()) # Remove the target file to avoid cases where it falsely doesn't get rebuild # and then lingers from previous builds, if os.path.exists(target[0].abspath): os.unlink(target[0].abspath) if show_scons_mode: print "scons: Told to run compilation on %d CPUs." % job_count # Cached, when done, by the fastest possible algorithm and right inside the # build directory. Makes no sense of course, if that is removed later on by # Nuitka. if cache_mode: CacheDir(os.path.join(source_dir, "cache-" + target_arch)) # @UndefinedVariable Decider("MD5-timestamp") # @UndefinedVariable # Before we go, also lets turn KeyboardInterrupt into a mere error exit. def signalHandler(signal, frame): sys.exit(2) signal.signal(signal.SIGINT, signalHandler) Default(target) # @UndefinedVariable # Copy the Python DLL to the target directory, so the executable can be run. if uninstalled_python: if win_target: shutil.copy( getWindowsPythonDLLPath(), os.path.dirname(result_basepath) or "." ) elif sys.platform == "darwin": python_dll_filename = "libpython" + python_version + ".dylib" result = subprocess.call( ( "install_name_tool", "--change", python_dll_filename, os.path.join(python_lib_path, python_dll_filename), result_basepath + ".exe" ), stdout = subprocess.PIPE ) if result != 0: sys.exit("Error, call to 'install_name_tool' to fix Python library path failed.") Nuitka-0.5.18.1/nuitka/build/include/0000755000175000017500000000000012651072347017455 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/include/nuitka/0000755000175000017500000000000012651072347020750 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/include/nuitka/builtins.hpp0000644000175000017500000000476512651072046023322 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_BUILTINS_H__ #define __NUITKA_BUILTINS_H__ #include "__helpers.hpp" extern PyModuleObject *builtin_module; extern PyDictObject *dict_builtin; #include "nuitka/calling.hpp" NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_BUILTIN( PyObject *name ) { CHECK_OBJECT( (PyObject *)dict_builtin ); CHECK_OBJECT( name ); assert( Nuitka_String_Check( name ) ); PyObject *result = GET_STRING_DICT_VALUE( dict_builtin, (Nuitka_StringObject *)name ); CHECK_OBJECT( result ); return result; } class PythonBuiltin { public: explicit PythonBuiltin( PyObject **name ) { this->name = (Nuitka_StringObject **)name; this->value = NULL; } PyObject *asObject0() { if ( this->value == NULL ) { this->value = LOOKUP_BUILTIN( (PyObject *)*this->name ); } CHECK_OBJECT( this->value ); return this->value; } void update( PyObject *new_value ) { CHECK_OBJECT( new_value ); this->value = new_value; } private: PythonBuiltin( PythonBuiltin const & ) { assert( false ); } Nuitka_StringObject **name; PyObject *value; }; extern void _initBuiltinModule(); #ifdef _NUITKA_EXE // Original builtin values, currently only used for assertions. extern PyObject *_python_original_builtin_value_type; extern PyObject *_python_original_builtin_value_len; extern PyObject *_python_original_builtin_value_range; extern PyObject *_python_original_builtin_value_repr; extern PyObject *_python_original_builtin_value_int; extern PyObject *_python_original_builtin_value_iter; extern PyObject *_python_original_builtin_value_long; extern void _initBuiltinOriginalValues(); #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/importing.hpp0000644000175000017500000000222712651071040023461 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_IMPORTING_H__ #define __NUITKA_IMPORTING_H__ extern PyObject *IMPORT_MODULE( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items, PyObject *level ); extern bool IMPORT_MODULE_STAR( PyObject *target, bool is_module, PyObject *module ); extern PyObject *IMPORT_COMPILED_MODULE( PyObject *module_name, char const *name ); #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/compiled_coroutine.hpp0000644000175000017500000000542112651071040025333 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_COROUTINE_H__ #define __NUITKA_COMPILED_COROUTINE_H__ // Compiled coroutine type. // Another cornerstone of the integration into CPython. Try to behave as well as // normal coroutine objects do or even better. #if PYTHON_VERSION >= 350 // The Nuitka_GeneratorObject is the storage associated with a compiled // generator object instance of which there can be many for each code. typedef struct { PyObject_HEAD PyObject *m_name; PyObject *m_qualname; PyObject *m_yieldfrom; Fiber m_yielder_context; Fiber m_caller_context; // Weak references are supported for generator objects in CPython. PyObject *m_weakrefs; int m_running; void *m_code; PyObject *m_yielded; PyObject *m_returned; PyObject *m_exception_type, *m_exception_value; PyTracebackObject *m_exception_tb; PyFrameObject *m_frame; PyCodeObject *m_code_object; // Closure variables given, if any, we reference cells here. PyCellObject **m_closure; Py_ssize_t m_closure_given; // Was it ever used, is it still running, or already finished. Generator_Status m_status; } Nuitka_CoroutineObject; extern PyTypeObject Nuitka_Coroutine_Type; typedef void (*coroutine_code)( Nuitka_CoroutineObject * ); extern PyObject *Nuitka_Coroutine_New( coroutine_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyCellObject **closure, Py_ssize_t closure_given ); static inline bool Nuitka_Coroutine_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Coroutine_Type; } typedef struct { PyObject_HEAD Nuitka_CoroutineObject *m_coroutine; } Nuitka_CoroutineWrapperObject; extern PyTypeObject Nuitka_CoroutineWrapper_Type; extern PyObject *AWAIT_COROUTINE( Nuitka_CoroutineObject *coroutine, PyObject *awaitable ); extern PyObject *MAKE_ASYNC_ITERATOR( Nuitka_CoroutineObject *coroutine, PyObject *value ); extern PyObject *ASYNC_ITERATOR_NEXT( Nuitka_CoroutineObject *coroutine, PyObject *value ); #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/compiled_method.hpp0000644000175000017500000000331312651071040024602 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_METHOD_H__ #define __NUITKA_COMPILED_METHOD_H__ // Compiled function and compile generator types may be referenced. #include "compiled_function.hpp" #include "compiled_generator.hpp" // The backbone of the integration into CPython. Try to behave as well as normal // method objects, or even better. // The Nuitka_MethodObject is the storage associated with a compiled method // instance of which there can be many for each code. typedef struct { PyObject_HEAD Nuitka_FunctionObject *m_function; PyObject *m_weakrefs; PyObject *m_object; PyObject *m_class; } Nuitka_MethodObject; extern PyTypeObject Nuitka_Method_Type; // Make a method out of a function. extern PyObject *Nuitka_Method_New( Nuitka_FunctionObject *function, PyObject *object, PyObject *klass ); static inline bool Nuitka_Method_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Method_Type; } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/fibers.hpp0000644000175000017500000000433112651071040022721 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_FIBERS_H__ #define __NUITKA_FIBERS_H__ #if defined( _WIN32 ) #include #elif defined( __OpenBSD__ ) extern "C" { #include "coro.h" } #else #include #endif typedef struct _Fiber { #if defined( _WIN32 ) LPVOID fiber; #elif defined( __OpenBSD__ ) struct coro_context coro_ctx; void *sptr; #else ucontext_t f_context; void *start_stack; #endif } Fiber; extern "C" void _initFiber( Fiber *to ); extern "C" void _swapFiber( Fiber *to, Fiber *from ); extern "C" int _prepareFiber( Fiber *to, void *code, uintptr_t arg ); extern "C" void _releaseFiber( Fiber *to ); // Have centralized assertions as wrappers in debug mode, or directly access // the fiber implementions of a given platform. #ifdef __NUITKA_NO_ASSERT__ #define initFiber _initFiber #define swapFiber _swapFiber #define prepareFiber _prepareFiber #define releaseFiber _releaseFiber #else static inline void initFiber( Fiber *to ) { assert( to ); _initFiber( to ); } static inline void swapFiber( Fiber *to, Fiber *from ) { assert( to != NULL ); assert( from != NULL ); _swapFiber( to, from ); } static inline int prepareFiber( Fiber *to, void *code, uintptr_t arg ) { assert( to != NULL ); assert( code != NULL ); CHECK_OBJECT( (PyObject *)arg ); return _prepareFiber( to, code, arg ); } static inline void releaseFiber( Fiber *to ) { assert( to != NULL ); _releaseFiber( to ); } #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/prelude.hpp0000644000175000017500000001464012651072046023122 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_PRELUDE_H__ #define __NUITKA_PRELUDE_H__ /* Disable assert statements if asked to not assert. */ #ifdef __NUITKA_NO_ASSERT__ #define NDEBUG #endif /* Include the CPython version numbers, and define our own take of what version * numbers should be. */ #include "patchlevel.h" #define PYTHON_VERSION (PY_MAJOR_VERSION*100+PY_MINOR_VERSION*10+PY_MICRO_VERSION) /* This is needed or else we can't create modules name "proc" or "func". For */ /* Python3, the name collision can't happen, so we can limit it to Python2. */ #if PYTHON_VERSION < 300 #define initproc python_initproc #define initfunc python_initfunc #define initstate system_initstate #endif /* Include the Python C-API header files. */ #include "Python.h" #include "methodobject.h" #include "frameobject.h" #include "pydebug.h" #include "marshal.h" /* See above. */ #if PYTHON_VERSION < 300 #undef initproc #undef initfunc #undef initstate #endif /* Include the C header files most often used. */ #include /* Using "_alloca" extension due to MSVC restrictions. */ #ifdef _MSC_VER #include #endif /* An idea I first saw used with Cython, hint the compiler about branches * that are more or less likely to be taken. And hint the compiler about * things that we assume to be normally true. If other compilers can do * similar, I would be grateful for instructions. */ #ifdef __GNUC__ #define likely(x) __builtin_expect(!!(x), 1) #define unlikely(x) __builtin_expect(!!(x), 0) #else #define likely(x) (x) #define unlikely(x) (x) #endif /* A way to not give warnings about things that are declared, but might not * be used like in-line helper functions in headers or static per module variables from headers. */ #ifdef __GNUC__ #define NUITKA_MAY_BE_UNUSED __attribute__((__unused__)) #else #define NUITKA_MAY_BE_UNUSED #endif /* A way to indicate that a specific function won't return, so the C compiler * can create better code. */ #ifdef __GNUC__ #define NUITKA_NO_RETURN __attribute__((__noreturn__)) #elif defined(_MSC_VER) #define NUITKA_NO_RETURN __declspec(noreturn) #else #define NUITKA_NO_RETURN #endif /* This is used to indicate code control flows we know cannot happen. */ #define NUITKA_CANNOT_GET_HERE(NAME) assert(false && #NAME); abort(); #ifdef __GNUC__ #define NUITKA_FORCE_INLINE __attribute__((always_inline)) #else #define NUITKA_FORCE_INLINE #endif /* Python3 removed PyInt instead of renaming PyLong, and PyObject_Str instead * of renaming PyObject_Unicode. Define this to be easily portable. */ #if PYTHON_VERSION >= 300 #define PyInt_FromString PyLong_FromString #define PyInt_FromLong PyLong_FromLong #define PyInt_AsLong PyLong_AsLong #define PyInt_FromSsize_t PyLong_FromSsize_t #define PyNumber_Int PyNumber_Long #define PyObject_Unicode PyObject_Str #endif /* String handling that uses the proper version of strings for Python3 or not, * which makes it easier to write portable code. */ #if PYTHON_VERSION < 300 #define Nuitka_String_AsString PyString_AsString #define Nuitka_String_AsString_Unchecked PyString_AS_STRING #define Nuitka_String_Check PyString_Check #define Nuitka_StringObject PyStringObject #define Nuitka_StringIntern PyString_InternInPlace #else #define Nuitka_String_AsString _PyUnicode_AsString /* Note: There seems to be no variant that does it without checks, so we rolled * our own. */ #define Nuitka_String_AsString_Unchecked _PyUnicode_AS_STRING #define Nuitka_String_Check PyUnicode_Check #define Nuitka_StringObject PyUnicodeObject #define Nuitka_StringIntern PyUnicode_InternInPlace #endif /* With the idea to reduce the amount of exported symbols in the DLLs, make it * clear that the module "init" function should of course be exported, but not * for executable, where we call it ourselves from the main code. */ #if defined( _NUITKA_EXE ) #if PYTHON_VERSION < 300 #define NUITKA_MODULE_INIT_FUNCTION void #else #define NUITKA_MODULE_INIT_FUNCTION PyObject * #endif #elif defined( __GNUC__ ) #if PYTHON_VERSION < 300 #define NUITKA_MODULE_INIT_FUNCTION PyMODINIT_FUNC __attribute__(( visibility( "default" ))) #else #define NUITKA_MODULE_INIT_FUNCTION extern "C" __attribute__(( visibility( "default" ))) PyObject * #endif #else #define NUITKA_MODULE_INIT_FUNCTION PyMODINIT_FUNC #endif /* The name of the entry point for DLLs changed between CPython versions, and * this is here to hide that. */ #if PYTHON_VERSION < 300 #define MOD_INIT_NAME( name ) init##name #define MOD_INIT_DECL( name ) NUITKA_MODULE_INIT_FUNCTION init##name( void ) #define MOD_RETURN_VALUE( value ) #else #define MOD_INIT_NAME( name ) PyInit_##name #define MOD_INIT_DECL( name ) NUITKA_MODULE_INIT_FUNCTION PyInit_##name( void ) #define MOD_RETURN_VALUE( value ) value #endif #if PYTHON_VERSION < 300 typedef long Py_hash_t; #endif /* These two express if a directly called function should be exported (C level) * or if it can be local to the file. */ #define NUITKA_CROSS_MODULE #define NUITKA_LOCAL_MODULE static /* Due to ABI issues, it seems that on Windows the symbols used by * "_PyObject_GC_TRACK" are not exported and we need to use a function that does * it instead. */ #if defined( _WIN32 ) #define Nuitka_GC_Track PyObject_GC_Track #define Nuitka_GC_UnTrack PyObject_GC_UnTrack #else #define Nuitka_GC_Track _PyObject_GC_TRACK #define Nuitka_GC_UnTrack _PyObject_GC_UNTRACK #endif #include "nuitka/helpers.hpp" #include "nuitka/compiled_function.hpp" /* Sentinel PyObject to be used for all our call iterator endings. */ extern PyObject *_sentinel_value; #include "nuitka/compiled_generator.hpp" #include "nuitka/compiled_method.hpp" #include "nuitka/compiled_frame.hpp" #if PYTHON_VERSION >= 350 #include "nuitka/compiled_coroutine.hpp" #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/compiled_function.hpp0000644000175000017500000001156112651072046025162 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_FUNCTION_H__ #define __NUITKA_COMPILED_FUNCTION_H__ #include "Python.h" #include "frameobject.h" // Compiled function type. // The backbone of the integration into CPython. Try to behave as well as normal // functions and built-in functions, or even better. struct Nuitka_FunctionObject; // The actual function code with arguments as an array. typedef PyObject *(*function_impl_code)( Nuitka_FunctionObject const *, PyObject ** ); // The Nuitka_FunctionObject is the storage associated with a compiled function // instance of which there can be many for each code. struct Nuitka_FunctionObject { PyObject_HEAD PyObject *m_name; PyObject *m_module; PyObject *m_doc; PyCodeObject *m_code_object; Py_ssize_t m_args_overall_count; Py_ssize_t m_args_positional_count; Py_ssize_t m_args_keywords_count; bool m_args_simple; Py_ssize_t m_args_star_list_index; Py_ssize_t m_args_star_dict_index; // Same as code_object->co_varnames PyObject **m_varnames; function_impl_code m_c_code; PyObject *m_dict; PyObject *m_weakrefs; // List of defaults, for use in __defaults__ and parameter parsing. PyObject *m_defaults; Py_ssize_t m_defaults_given; // Closure taken objects, for use in __closure__ and for accessing it. PyCellObject **m_closure; Py_ssize_t m_closure_given; #if PYTHON_VERSION >= 300 // List of keyword only defaults, for use in __kwdefaults__ and parameter // parsing. PyObject *m_kwdefaults; // Annotations to the function arguments and return value. PyObject *m_annotations; #endif #if PYTHON_VERSION >= 330 PyObject *m_qualname; #endif long m_counter; }; extern PyTypeObject Nuitka_Function_Type; // Make a function without context. #if PYTHON_VERSION < 300 extern PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc ); #elif PYTHON_VERSION < 330 extern PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc ); #else extern PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc ); #endif // Make a function with context. #if PYTHON_VERSION < 300 extern PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ); #elif PYTHON_VERSION < 330 extern PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ); #else extern PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ); #endif static inline bool Nuitka_Function_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Function_Type; } static inline PyObject *Nuitka_Function_GetName( PyObject *object ) { return ((Nuitka_FunctionObject *)object)->m_name; } extern bool parseArgumentsPos( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ); extern bool parseArgumentsMethodPos( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *object, PyObject **args, Py_ssize_t args_size ); extern PyObject *Nuitka_CallFunctionPosArgsKwArgs( Nuitka_FunctionObject const *function, PyObject **args, Py_ssize_t args_size, PyObject *kw ); extern PyObject *Nuitka_CallMethodFunctionPosArgsKwArgs( Nuitka_FunctionObject const *function, PyObject *object, PyObject **args, Py_ssize_t args_size, PyObject *kw ); #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/0000755000175000017500000000000012651072347022227 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/cells.hpp0000644000175000017500000000341112651071040024026 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_CELLS_H__ #define __NUITKA_CELLS_H__ NUITKA_MAY_BE_UNUSED static PyCellObject *PyCell_NEW0( PyObject *value ) { CHECK_OBJECT( value ); PyCellObject *result; result = (PyCellObject *)PyObject_GC_New( PyCellObject, &PyCell_Type ); assert( result != NULL ); result->ob_ref = value; Py_INCREF( value ); Nuitka_GC_Track( result ); return result; } NUITKA_MAY_BE_UNUSED static PyCellObject *PyCell_NEW1( PyObject *value ) { CHECK_OBJECT( value ); PyCellObject *result; result = (PyCellObject *)PyObject_GC_New( PyCellObject, &PyCell_Type ); assert( result != NULL ); result->ob_ref = value; Nuitka_GC_Track( result ); return result; } NUITKA_MAY_BE_UNUSED static PyCellObject *PyCell_EMPTY( void ) { PyCellObject *result; result = (PyCellObject *)PyObject_GC_New( PyCellObject, &PyCell_Type ); assert( result != NULL ); result->ob_ref = NULL; Nuitka_GC_Track( result ); return result; } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/iterators.hpp0000644000175000017500000001647112651071040024752 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_ITERATORS_H__ #define __NUITKA_HELPER_ITERATORS_H__ #if PYTHON_VERSION >= 270 // Initialize value for "tp_iternext" to compare with, needed by HAS_ITERNEXT // which emulates PyCheck_Iter but bug free. extern iternextfunc default_iternext; extern void _initSlotIternext( void ); #endif // This is like "PyIter_Check" but without bugs due to shared library pointers. NUITKA_MAY_BE_UNUSED static inline bool HAS_ITERNEXT( PyObject *value ) { #if PYTHON_VERSION < 300 if ( !PyType_HasFeature( Py_TYPE( value ), Py_TPFLAGS_HAVE_ITER ) ) { return false; } #endif iternextfunc tp_iternext = Py_TYPE( value )->tp_iternext; if ( tp_iternext == NULL ) { return false; } #if PYTHON_VERSION >= 270 return tp_iternext != default_iternext; #else return true; #endif } // Stolen from CPython implementation, so we can access it. typedef struct { PyObject_HEAD #if PYTHON_VERSION < 340 long it_index; #else Py_ssize_t it_index; #endif PyObject *it_seq; } seqiterobject; NUITKA_MAY_BE_UNUSED static PyObject *MAKE_ITERATOR( PyObject *iterated ) { #if PYTHON_VERSION < 300 getiterfunc tp_iter = NULL; if ( PyType_HasFeature( Py_TYPE( iterated ), Py_TPFLAGS_HAVE_ITER )) { tp_iter = Py_TYPE( iterated )->tp_iter; } #else getiterfunc tp_iter = Py_TYPE( iterated )->tp_iter; #endif if ( tp_iter ) { PyObject *result = (*tp_iter)( iterated ); if (unlikely( result == NULL )) { return NULL; } else if (unlikely( !HAS_ITERNEXT( result )) ) { PyErr_Format( PyExc_TypeError, "iter() returned non-iterator of type '%s'", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } else { return result; } } else if ( PySequence_Check( iterated ) ) { seqiterobject *result = PyObject_GC_New( seqiterobject, &PySeqIter_Type ); assert( result ); result->it_index = 0; result->it_seq = INCREASE_REFCOUNT( iterated ); Nuitka_GC_Track( result ); return (PyObject *)result; } else { PyErr_Format( PyExc_TypeError, "'%s' object is not iterable", Py_TYPE( iterated )->tp_name ); return NULL; } } NUITKA_MAY_BE_UNUSED static PyObject *ITERATOR_NEXT( PyObject *iterator ) { CHECK_OBJECT( iterator ); iternextfunc iternext = Py_TYPE( iterator )->tp_iternext; if (unlikely( iternext == NULL )) { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 330 "%s object is not an iterator", #else "'%s' object is not an iterator", #endif Py_TYPE( iterator )->tp_name ); return NULL; } PyObject *result = (*iternext)( iterator ); #if PYTHON_VERSION < 330 if ( result == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error ) { if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { CLEAR_ERROR_OCCURRED(); } } } #endif return result; } NUITKA_MAY_BE_UNUSED static PyObject *BUILTIN_NEXT1( PyObject *iterator ) { CHECK_OBJECT( iterator ); iternextfunc iternext = Py_TYPE( iterator )->tp_iternext; if (unlikely( iternext == NULL )) { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 330 "%s object is not an iterator", #else "'%s' object is not an iterator", #endif Py_TYPE( iterator )->tp_name ); return NULL; } PyObject *result = (*iternext)( iterator ); if (unlikely( result == NULL )) { // The iteration can return NULL with no error, which means // StopIteration. if ( !ERROR_OCCURRED() ) { PyErr_SetNone( PyExc_StopIteration ); } return NULL; } else { CHECK_OBJECT( result ); } return result; } NUITKA_MAY_BE_UNUSED static PyObject *BUILTIN_NEXT2( PyObject *iterator, PyObject *default_value ) { CHECK_OBJECT( iterator ); CHECK_OBJECT( default_value ); PyObject *result = (*Py_TYPE( iterator )->tp_iternext)( iterator ); if (unlikely( result == NULL )) { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL ) { if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration )) { DROP_ERROR_OCCURRED(); return INCREASE_REFCOUNT( default_value ); } else { return NULL; } } else { return INCREASE_REFCOUNT( default_value ); } } else { CHECK_OBJECT( result ); } return result; } #if PYTHON_VERSION < 350 NUITKA_MAY_BE_UNUSED static PyObject *UNPACK_NEXT( PyObject *iterator, int seq_size_so_far ) #else NUITKA_MAY_BE_UNUSED static PyObject *UNPACK_NEXT( PyObject *iterator, int seq_size_so_far, int expected ) #endif { CHECK_OBJECT( iterator ); assert( HAS_ITERNEXT( iterator ) ); PyObject *result = (*Py_TYPE( iterator )->tp_iternext)( iterator ); if (unlikely( result == NULL )) { #if PYTHON_VERSION < 300 if (unlikely( !ERROR_OCCURRED() )) #else if (unlikely( !ERROR_OCCURRED() || EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) )) #endif { #if PYTHON_VERSION < 350 if ( seq_size_so_far == 1 ) { PyErr_Format( PyExc_ValueError, "need more than 1 value to unpack" ); } else { PyErr_Format( PyExc_ValueError, "need more than %d values to unpack", seq_size_so_far ); } #else PyErr_Format( PyExc_ValueError, "not enough values to unpack (expected %d, got %d)", expected, seq_size_so_far ); #endif } return NULL; } CHECK_OBJECT( result ); return result; } NUITKA_MAY_BE_UNUSED static bool UNPACK_ITERATOR_CHECK( PyObject *iterator ) { CHECK_OBJECT( iterator ); assert( HAS_ITERNEXT( iterator ) ); PyObject *attempt = (*Py_TYPE( iterator )->tp_iternext)( iterator ); if (likely( attempt == NULL )) { return CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED(); } else { Py_DECREF( attempt ); PyErr_Format( PyExc_ValueError, "too many values to unpack" ); return false; } } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/richcomparisons.hpp0000644000175000017500000002173512651071040026140 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_RICHCOMPARISONS_H__ #define __NUITKA_HELPER_RICHCOMPARISONS_H__ static inline bool IS_SANE_TYPE( PyTypeObject *type ) { return #if PYTHON_VERSION < 300 type == &PyString_Type || type == &PyInt_Type || #endif type == &PyLong_Type || type == &PyList_Type || type == &PyTuple_Type; } extern PyObject *MY_RICHCOMPARE( PyObject *v, PyObject *w, int op ); extern PyObject *MY_RICHCOMPARE_NORECURSE( PyObject *v, PyObject *w, int op ); NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_LT( PyObject *operand1, PyObject *operand2 ) { PyObject *result = MY_RICHCOMPARE( operand1, operand2, Py_LT ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_LE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return INCREASE_REFCOUNT( Py_True ); } PyObject *result = MY_RICHCOMPARE( operand1, operand2, Py_LE ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_EQ( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return INCREASE_REFCOUNT( Py_True ); } return MY_RICHCOMPARE( operand1, operand2, Py_EQ ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_EQ_NORECURSE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return INCREASE_REFCOUNT( Py_True ); } return MY_RICHCOMPARE_NORECURSE( operand1, operand2, Py_EQ ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_NE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return INCREASE_REFCOUNT( Py_False ); } return MY_RICHCOMPARE( operand1, operand2, Py_NE ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_GT( PyObject *operand1, PyObject *operand2 ) { return MY_RICHCOMPARE( operand1, operand2, Py_GT ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_GE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return INCREASE_REFCOUNT( Py_True ); } return MY_RICHCOMPARE( operand1, operand2, Py_GE ); } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_LT( PyObject *operand1, PyObject *operand2 ) { PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_LT ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_LE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_LE ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_EQ( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_EQ ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_EQ_NORECURSE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE_NORECURSE( operand1, operand2, Py_EQ ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_NE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 0; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_NE ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_GT( PyObject *operand1, PyObject *operand2 ) { PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_GT ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_GE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_GE ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/subscripts.hpp0000644000175000017500000002453512651071040025137 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_SUBSCRIPTS_H__ #define __NUITKA_HELPER_SUBSCRIPTS_H__ extern PyObject *BUILTIN_CHR( unsigned char c ); NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SUBSCRIPT_CONST( PyObject *source, PyObject *const_subscript, Py_ssize_t int_subscript ) { CHECK_OBJECT( source ); CHECK_OBJECT( const_subscript ); PyTypeObject *type = Py_TYPE( source ); PyMappingMethods *mapping_methods = type->tp_as_mapping; PyObject *result; if ( mapping_methods && mapping_methods->mp_subscript ) { if ( PyList_CheckExact( source ) ) { Py_ssize_t list_size = PyList_GET_SIZE( source ); if ( int_subscript < 0 ) { if ( -int_subscript > list_size ) { PyErr_Format( PyExc_IndexError, "list index out of range" ); return NULL; } int_subscript += list_size; } else { if ( int_subscript >= list_size ) { PyErr_Format( PyExc_IndexError, "list index out of range" ); return NULL; } } return INCREASE_REFCOUNT( ((PyListObject *)source)->ob_item[ int_subscript ] ); } #if PYTHON_VERSION < 300 else if ( PyString_CheckExact( source ) ) { Py_ssize_t string_size = PyString_GET_SIZE( source ); if ( int_subscript < 0 ) { if ( -int_subscript > string_size ) { PyErr_Format( PyExc_IndexError, "string index out of range" ); return NULL; } int_subscript += string_size; } else { if ( int_subscript >= string_size ) { PyErr_Format( PyExc_IndexError, "string index out of range" ); return NULL; } } unsigned char c = ((PyStringObject *)source)->ob_sval[ int_subscript ]; return BUILTIN_CHR( c ); } #else else if ( PyUnicode_CheckExact( source ) ) { if ( int_subscript < 0 ) { #if PYTHON_VERSION < 330 int_subscript += PyUnicode_GET_SIZE( source ); #else int_subscript += PyUnicode_GET_LENGTH( source ); #endif } result = type->tp_as_sequence->sq_item( source, int_subscript ); } #endif else { result = mapping_methods->mp_subscript( source, const_subscript ); } } else if ( type->tp_as_sequence ) { result = PySequence_GetItem( source, int_subscript ); } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 270 "'%s' object is unsubscriptable", #elif PYTHON_VERSION >= 300 || PYTHON_VERSION <= 272 "'%s' object is not subscriptable", #else "'%s' object has no attribute '__getitem__'", #endif Py_TYPE( source )->tp_name ); return NULL; } if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SUBSCRIPT( PyObject *source, PyObject *subscript ) { CHECK_OBJECT( source ); CHECK_OBJECT( subscript ); PyTypeObject *type = Py_TYPE( source ); PyMappingMethods *mapping = type->tp_as_mapping; PyObject *result; if ( mapping != NULL && mapping->mp_subscript != NULL ) { result = mapping->mp_subscript( source, subscript ); } else if ( type->tp_as_sequence != NULL ) { if ( PyIndex_Check( subscript ) ) { Py_ssize_t index = PyNumber_AsSsize_t( subscript, NULL ); if ( index == -1 && ERROR_OCCURRED() ) { return NULL; } result = PySequence_GetItem( source, index ); } else if ( type->tp_as_sequence->sq_item ) { PyErr_Format( PyExc_TypeError, "sequence index must be integer, not '%s'", Py_TYPE( subscript )->tp_name ); return NULL; } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 270 "'%s' object is unsubscriptable", #elif PYTHON_VERSION >= 300 || PYTHON_VERSION <= 272 "'%s' object is not subscriptable", #else "'%s' object has no attribute '__getitem__'", #endif Py_TYPE( source )->tp_name ); return NULL; } } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 270 "'%s' object is unsubscriptable", #elif PYTHON_VERSION >= 300 || PYTHON_VERSION <= 272 "'%s' object is not subscriptable", #else "'%s' object has no attribute '__getitem__'", #endif Py_TYPE( source )->tp_name ); return NULL; } if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool SET_SUBSCRIPT_CONST( PyObject *target, PyObject *subscript, Py_ssize_t int_subscript, PyObject *value ) { CHECK_OBJECT( value ); CHECK_OBJECT( target ); CHECK_OBJECT( subscript ); PyMappingMethods *mapping_methods = Py_TYPE( target )->tp_as_mapping; if ( mapping_methods != NULL && mapping_methods->mp_ass_subscript ) { if ( PyList_CheckExact( target ) ) { Py_ssize_t list_size = PyList_GET_SIZE( target ); if ( int_subscript < 0 ) { if ( -int_subscript > list_size ) { PyErr_Format( PyExc_IndexError, "list assignment index out of range" ); return false; } int_subscript += list_size; } PyListObject *target_list = (PyListObject *)target; PyObject *old_value = target_list->ob_item[ int_subscript ]; target_list->ob_item[ int_subscript ] = INCREASE_REFCOUNT( value ); Py_DECREF( old_value ); return true; } else { int res = mapping_methods->mp_ass_subscript( target, subscript, value ); if (unlikely( res == -1 )) { return false; } return true; } } else if ( Py_TYPE( target )->tp_as_sequence ) { if ( PyIndex_Check( subscript ) ) { Py_ssize_t key_value = PyNumber_AsSsize_t( subscript, PyExc_IndexError ); if ( key_value == -1 ) { if ( ERROR_OCCURRED() ) { return false; } } return SEQUENCE_SETITEM( target, key_value, value ); } else if ( Py_TYPE( target )->tp_as_sequence->sq_ass_item ) { PyErr_Format( PyExc_TypeError, "sequence index must be integer, not '%s'", Py_TYPE( subscript )->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } } NUITKA_MAY_BE_UNUSED static bool SET_SUBSCRIPT( PyObject *target, PyObject *subscript, PyObject *value ) { CHECK_OBJECT( value ); CHECK_OBJECT( target ); CHECK_OBJECT( subscript ); PyMappingMethods *mapping_methods = Py_TYPE( target )->tp_as_mapping; if ( mapping_methods != NULL && mapping_methods->mp_ass_subscript ) { int res = mapping_methods->mp_ass_subscript( target, subscript, value ); if (unlikely( res == -1 )) { return false; } } else if ( Py_TYPE( target )->tp_as_sequence ) { if ( PyIndex_Check( subscript ) ) { Py_ssize_t key_value = PyNumber_AsSsize_t( subscript, PyExc_IndexError ); if ( key_value == -1 ) { if ( ERROR_OCCURRED() ) { return false; } } return SEQUENCE_SETITEM( target, key_value, value ); } else if ( Py_TYPE( target )->tp_as_sequence->sq_ass_item ) { PyErr_Format( PyExc_TypeError, "sequence index must be integer, not '%s'", Py_TYPE( subscript )->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } return true; } NUITKA_MAY_BE_UNUSED static bool DEL_SUBSCRIPT( PyObject *target, PyObject *subscript ) { CHECK_OBJECT( target ); CHECK_OBJECT( subscript ); int status = PyObject_DelItem( target, subscript ); if (unlikely( status == -1 )) { return false; } return true; } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/rangeobjects.hpp0000644000175000017500000000261512651071040025377 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_RANGEOBJECTS_H__ #define __NUITKA_RANGEOBJECTS_H__ // Python3 range objects #if PYTHON_VERSION >= 300 typedef struct { PyObject_HEAD PyObject *start; PyObject *stop; PyObject *step; PyObject *length; } _rangeobject; NUITKA_MAY_BE_UNUSED static PyObject *PyRange_Start( PyObject *range ) { return ((_rangeobject *)range)->start; } NUITKA_MAY_BE_UNUSED static PyObject *PyRange_Stop( PyObject *range ) { return ((_rangeobject *)range)->stop; } NUITKA_MAY_BE_UNUSED static PyObject *PyRange_Step( PyObject *range ) { return ((_rangeobject *)range)->step; } #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/boolean.hpp0000644000175000017500000000523012651071040024344 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_BOOLEAN_H__ #define __NUITKA_HELPER_BOOLEAN_H__ #if PYTHON_VERSION >= 300 #define nb_nonzero nb_bool #endif NUITKA_MAY_BE_UNUSED static int CHECK_IF_TRUE( PyObject *object ) { CHECK_OBJECT( object ); if ( object == Py_True ) { return 1; } else if ( object == Py_False || object == Py_None ) { return 0; } else { Py_ssize_t result; if ( Py_TYPE( object )->tp_as_number != NULL && Py_TYPE( object )->tp_as_number->nb_nonzero != NULL ) { result = (*Py_TYPE( object )->tp_as_number->nb_nonzero)( object ); } else if ( Py_TYPE( object )->tp_as_mapping != NULL && Py_TYPE( object )->tp_as_mapping->mp_length != NULL ) { result = (*Py_TYPE( object )->tp_as_mapping->mp_length)( object ); } else if ( Py_TYPE( object )->tp_as_sequence != NULL && Py_TYPE( object )->tp_as_sequence->sq_length != NULL ) { result = (*Py_TYPE( object )->tp_as_sequence->sq_length)( object ); } else { return 1; } if ( result > 0 ) { return 1; } else if ( result == 0 ) { return 0; } else { return -1; } } } NUITKA_MAY_BE_UNUSED static int CHECK_IF_FALSE( PyObject *object ) { int result = CHECK_IF_TRUE( object ); if ( result == 0 ) return 1; if ( result == 1 ) return 0; return -1; } NUITKA_MAY_BE_UNUSED static PyObject *BOOL_FROM( bool value ) { CHECK_OBJECT( Py_True ); CHECK_OBJECT( Py_False ); return value ? Py_True : Py_False; } NUITKA_MAY_BE_UNUSED static PyObject *UNARY_NOT( PyObject *object ) { int res = CHECK_IF_TRUE( object ); if ( res == 0 ) return Py_True; if ( res == 1 ) return Py_False; return NULL; } #undef nb_nonzero #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/slices.hpp0000644000175000017500000001726012651072046024224 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_SLICES_H__ #define __NUITKA_HELPER_SLICES_H__ static inline bool IS_INDEXABLE( PyObject *value ) { return value == Py_None || #if PYTHON_VERSION < 300 PyInt_Check( value ) || #endif PyLong_Check( value ) || PyIndex_Check( value ); } #if PYTHON_VERSION < 300 // Note: It appears that Python3 has no index slicing operations anymore, but // uses slice objects all the time. That's fine and make sure we adhere to it by // guarding the presence of the helpers. NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SLICE( PyObject *source, PyObject *lower, PyObject *upper ) { CHECK_OBJECT( source ); CHECK_OBJECT( lower ); CHECK_OBJECT( upper ); PySequenceMethods *tp_as_sequence = Py_TYPE( source )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_slice && IS_INDEXABLE( lower ) && IS_INDEXABLE( upper ) ) { Py_ssize_t ilow = 0; if ( lower != Py_None ) { ilow = CONVERT_TO_INDEX( lower ); if ( ilow == -1 && ERROR_OCCURRED() ) { return NULL; } } Py_ssize_t ihigh = PY_SSIZE_T_MAX; if ( upper != Py_None ) { ihigh = CONVERT_TO_INDEX( upper ); if ( ihigh == -1 && ERROR_OCCURRED() ) { return NULL; } } PyObject *result = PySequence_GetSlice( source, ilow, ihigh ); if (unlikely( result == NULL )) { return NULL; } return result; } else { PyObject *slice = PySlice_New( lower, upper, NULL ); if (unlikely( slice == NULL )) { return NULL; } PyObject *result = PyObject_GetItem( source, slice ); Py_DECREF( slice ); if (unlikely( result == NULL )) { return NULL; } return result; } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_INDEX_SLICE( PyObject *source, Py_ssize_t lower, Py_ssize_t upper ) { CHECK_OBJECT( source ); PyObject *result = PySequence_GetSlice( source, lower, upper ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool SET_SLICE( PyObject *target, PyObject *lower, PyObject *upper, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( lower ); CHECK_OBJECT( upper ); CHECK_OBJECT( value ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice && IS_INDEXABLE( lower ) && IS_INDEXABLE( upper ) ) { Py_ssize_t lower_int = 0; if ( lower != Py_None ) { lower_int = CONVERT_TO_INDEX( lower ); if ( lower_int == -1 && ERROR_OCCURRED() ) { return false; } } Py_ssize_t upper_int = PY_SSIZE_T_MAX; if ( upper != Py_None ) { upper_int = CONVERT_TO_INDEX( upper ); if ( upper_int == -1 && ERROR_OCCURRED() ) { return false; } } int status = PySequence_SetSlice( target, lower_int, upper_int, value ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = PySlice_New( lower, upper, NULL ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_SetItem( target, slice, value ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool SET_INDEX_SLICE( PyObject *target, Py_ssize_t lower, Py_ssize_t upper, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( value ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice ) { int status = PySequence_SetSlice( target, lower, upper, value ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = _PySlice_FromIndices( lower, upper ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_SetItem( target, slice, value ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool DEL_SLICE( PyObject *target, PyObject *lower, PyObject *upper ) { CHECK_OBJECT( target ); CHECK_OBJECT( lower ); CHECK_OBJECT( upper ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice && IS_INDEXABLE( lower ) && IS_INDEXABLE( upper ) ) { Py_ssize_t lower_int = 0; if ( lower != Py_None ) { lower_int = CONVERT_TO_INDEX( lower ); if ( lower_int == -1 && ERROR_OCCURRED() ) { return false; } } Py_ssize_t upper_int = PY_SSIZE_T_MAX; if ( upper != Py_None ) { upper_int = CONVERT_TO_INDEX( upper ); if ( upper_int == -1 && ERROR_OCCURRED() ) { return false; } } int status = PySequence_DelSlice( target, lower_int, upper_int ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = PySlice_New( lower, upper, NULL ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_DelItem( target, slice ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool DEL_INDEX_SLICE( PyObject *target, Py_ssize_t lower, Py_ssize_t upper ) { CHECK_OBJECT( target ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice ) { int status = PySequence_DelSlice( target, lower, upper ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = _PySlice_FromIndices( lower, upper ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_DelItem( target, slice ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } #endif // Note: Cannot fail NUITKA_MAY_BE_UNUSED static PyObject *MAKE_SLICEOBJ3( PyObject *start, PyObject *stop, PyObject *step ) { CHECK_OBJECT( start ); CHECK_OBJECT( stop ); CHECK_OBJECT( step ); return PySlice_New( start, stop, step ); } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/dictionaries.hpp0000644000175000017500000002137312651072046025417 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_DICTIONARIES_H__ #define __NUITKA_DICTIONARIES_H__ static inline Py_ssize_t DICT_SIZE( PyObject *dict ) { CHECK_OBJECT( dict ); return ((PyDictObject *)dict)->ma_used; } static inline PyDictObject *MODULE_DICT( PyModuleObject *module ) { PyDictObject *dict = (PyDictObject *)(module->md_dict); return dict; } static inline PyDictObject *MODULE_DICT( PyObject *module ) { return MODULE_DICT( (PyModuleObject *)module ); } #if PYTHON_VERSION < 330 // Quick dictionary lookup for a string value. typedef PyDictEntry *Nuitka_DictEntryHandle; static PyDictEntry *GET_STRING_DICT_ENTRY( PyDictObject *dict, Nuitka_StringObject *key ) { assert( PyDict_CheckExact( dict ) ); assert( Nuitka_String_Check( key ) ); #if PYTHON_VERSION < 300 Py_hash_t hash = key->ob_shash; #else Py_hash_t hash = key->hash; #endif // Only improvement would be to identify how to ensure that the hash is // computed already. Calling hash early on could do that potentially. if ( hash == -1 ) { #if PYTHON_VERSION < 300 hash = PyString_Type.tp_hash( (PyObject *)key ); key->ob_shash = hash; #else hash = PyUnicode_Type.tp_hash( (PyObject *)key ); key->hash = hash; #endif } PyDictEntry *entry = dict->ma_lookup( dict, (PyObject *)key, hash ); // The "entry" cannot be NULL, it can only be empty for a string dict // lookup, but at least assert it. assert( entry != NULL ); return entry; } NUITKA_MAY_BE_UNUSED static PyObject *GET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle ) { return handle->me_value; } NUITKA_MAY_BE_UNUSED static void SET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle, PyObject *value ) { handle->me_value = value; } static PyObject *GET_STRING_DICT_VALUE( PyDictObject *dict, Nuitka_StringObject *key ) { return GET_STRING_DICT_ENTRY( dict, key )->me_value; } #else // Quick dictionary lookup for a string value. struct PyDictKeyEntry { /* Cached hash code of me_key. */ Py_hash_t me_hash; PyObject *me_key; PyObject *me_value; /* This field is only meaningful for combined tables */ }; typedef PyDictKeyEntry *(*dict_lookup_func)(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject ***value_addr); // Stolen from CPython3.3 dictobject.c struct _dictkeysobject { Py_ssize_t dk_refcnt; Py_ssize_t dk_size; dict_lookup_func dk_lookup; Py_ssize_t dk_usable; PyDictKeyEntry dk_entries[1]; }; typedef PyObject **Nuitka_DictEntryHandle; static Nuitka_DictEntryHandle GET_STRING_DICT_ENTRY( PyDictObject *dict, Nuitka_StringObject *key ) { assert( PyDict_CheckExact( dict ) ); assert( Nuitka_String_Check( key ) ); Py_hash_t hash = key->_base._base.hash; // Only improvement would be to identify how to ensure that the hash is computed // already. Calling hash early on could do that potentially. if ( hash == -1 ) { hash = PyUnicode_Type.tp_hash( (PyObject *)key ); key->_base._base.hash = hash; } PyObject **value_addr; PyDictKeyEntry *entry = dict->ma_keys->dk_lookup( dict, (PyObject *)key, hash, &value_addr ); // The "entry" cannot be NULL, it can only be empty for a string dict lookup, but at // least assert it. assert( entry != NULL ); return value_addr; } NUITKA_MAY_BE_UNUSED static PyObject *GET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle ) { return *handle; } NUITKA_MAY_BE_UNUSED static void SET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle, PyObject *value ) { *handle = value; } NUITKA_MAY_BE_UNUSED static PyObject *GET_STRING_DICT_VALUE( PyDictObject *dict, Nuitka_StringObject *key ) { Nuitka_DictEntryHandle handle = GET_STRING_DICT_ENTRY( dict, key ); return GET_DICT_ENTRY_VALUE( handle ); } #endif NUITKA_MAY_BE_UNUSED static bool DICT_SET_ITEM( PyObject *dict, PyObject *key, PyObject *value ) { CHECK_OBJECT( dict ); CHECK_OBJECT( key ); CHECK_OBJECT( value ); int status = PyDict_SetItem( dict, key, value ); if (unlikely( status != 0 )) { return false; } return true; } NUITKA_MAY_BE_UNUSED static inline bool DICT_SET_ITEM( PyDictObject *dict, PyObject *key, PyObject *value ) { return DICT_SET_ITEM( (PyObject *)dict, key, value ); } NUITKA_MAY_BE_UNUSED static bool DICT_REMOVE_ITEM( PyObject *dict, PyObject *key ) { int status = PyDict_DelItem( dict, key ); if (unlikely( status == -1 )) { return false; } return true; } NUITKA_MAY_BE_UNUSED static PyObject *DICT_GET_ITEM( PyObject *dict, PyObject *key ) { CHECK_OBJECT( dict ); assert( PyDict_Check( dict ) ); CHECK_OBJECT( key ); PyObject *result = PyDict_GetItem( dict, key ); if ( result == NULL ) { if (unlikely( PyErr_Occurred() )) { return NULL; } PyErr_SetObject( PyExc_KeyError, key ); return NULL; } else { return INCREASE_REFCOUNT( result ); } } // Convert to dictionary, helper for builtin dict mainly. NUITKA_MAY_BE_UNUSED static PyObject *TO_DICT( PyObject *seq_obj, PyObject *dict_obj ) { PyObject *result = PyDict_New(); if ( seq_obj != NULL ) { int res; if ( PyObject_HasAttrString( seq_obj, "keys" ) ) { res = PyDict_Merge( result, seq_obj, 1 ); } else { res = PyDict_MergeFromSeq2( result, seq_obj, 1 ); } if ( res == -1 ) { return NULL; } } if ( dict_obj != NULL ) { int res = PyDict_Merge( result, dict_obj, 1 ); if ( res == -1 ) { return NULL; } } return result; } NUITKA_MAY_BE_UNUSED static void UPDATE_STRING_DICT0( PyDictObject *dict, Nuitka_StringObject *key, PyObject *value ) { Nuitka_DictEntryHandle entry = GET_STRING_DICT_ENTRY( dict, key ); PyObject *old = GET_DICT_ENTRY_VALUE( entry ); // Values are more likely (more often) set than not set, in that case // speculatively try the quickest access method. if (likely( old != NULL )) { Py_INCREF( value ); SET_DICT_ENTRY_VALUE( entry, value ); CHECK_OBJECT( old ); Py_DECREF( old ); } else { DICT_SET_ITEM( dict, (PyObject *)key, value ); } } NUITKA_MAY_BE_UNUSED static void UPDATE_STRING_DICT1( PyDictObject *dict, Nuitka_StringObject *key, PyObject *value ) { Nuitka_DictEntryHandle entry = GET_STRING_DICT_ENTRY( dict, key ); PyObject *old = GET_DICT_ENTRY_VALUE( entry ); // Values are more likely (more often) set than not set, in that case // speculatively try the quickest access method. if (likely( old != NULL )) { SET_DICT_ENTRY_VALUE( entry, value ); Py_DECREF( old ); } else { DICT_SET_ITEM( dict, (PyObject *)key, value ); Py_DECREF( value ); } } // TODO: Have mapping.hpp NUITKA_MAY_BE_UNUSED static void DICT_SYNC_FROM_VARIABLE( PyObject *dict, PyObject *key, PyObject *value ) { if ( value ) { assert( PyDict_CheckExact( dict ) ); UPDATE_STRING_DICT0( (PyDictObject *)dict, (Nuitka_StringObject *)key, value ); } else { int res = PyDict_DelItem( dict, key ); if ( res != 0 ) { CLEAR_ERROR_OCCURRED(); } } } // TODO: Have mapping.hpp NUITKA_MAY_BE_UNUSED static bool MAPPING_SYNC_FROM_VARIABLE( PyObject *mapping, PyObject *key, PyObject *value ) { if ( value ) { int res = PyObject_SetItem( mapping, key, value ); return res == 0; } else { PyObject *test_value = PyObject_GetItem( mapping, key ); if ( test_value ) { Py_DECREF( test_value ); int res = PyObject_DelItem( mapping, key ); return res == 0; } else { PyErr_Clear(); return true; } } } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/printing.hpp0000644000175000017500000000414112651071040024557 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_PRINTING_H__ #define __NUITKA_PRINTING_H__ // Helper functions for print. Need to play nice with Python softspace // behavior. extern bool PRINT_NEW_LINE( void ); extern bool PRINT_ITEM( PyObject *object ); extern bool PRINT_STRING( char const *str ); extern bool PRINT_ITEM_TO( PyObject *file, PyObject *object ); extern bool PRINT_NEW_LINE_TO( PyObject *file ); extern PyObject *GET_STDOUT(); extern PyObject *GET_STDERR(); // ----------------------------------------------------------------------- // Helper functions to debug the run time operation of the compiled binary // manually or in debug modes. // Print the reference count of the object. extern void PRINT_REFCOUNT( PyObject *object ); // Print the full traceback stack. // TODO: Could be ported, the "printf" stuff would need to be split. On Python3 // the normal C print output gets lost. #if PYTHON_VERSION < 300 extern void PRINT_TRACEBACK( PyTracebackObject *traceback ); #endif // Print the exception state, including NULL values. extern void PRINT_EXCEPTION( PyObject *exception_type, PyObject *exception_value, PyObject *exception_tb ); // Print the representation of the object, or "" if it's not set. extern bool PRINT_REPR( PyObject *object ); // Print the word , as an alternative to pointers. extern bool PRINT_NULL( void ); #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/indexes.hpp0000644000175000017500000000256212651072046024400 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_INDEXES_H__ #define __NUITKA_HELPER_INDEXES_H__ #include "nuitka/helper/raising.hpp" NUITKA_MAY_BE_UNUSED static Py_ssize_t CONVERT_TO_INDEX( PyObject *value ) { CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if ( PyInt_Check( value ) ) { return PyInt_AS_LONG( value ); } else #endif if ( PyIndex_Check( value ) ) { return PyNumber_AsSsize_t( value, NULL ); } else { PyErr_Format( PyExc_TypeError, "slice indices must be integers or None or have an __index__ method" ); return -1; } } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/operations.hpp0000644000175000017500000006531712651071040025124 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_OPERATIONS_H__ #define __NUITKA_OPERATIONS_H__ #if PYTHON_VERSION < 300 #define NEW_STYLE_NUMBER( o ) PyType_HasFeature( Py_TYPE( o ), Py_TPFLAGS_CHECKTYPES ) #else #define NEW_STYLE_NUMBER( o ) ( true ) #endif typedef PyObject *(unary_api)( PyObject * ); NUITKA_MAY_BE_UNUSED static PyObject *UNARY_OPERATION( unary_api api, PyObject *operand ) { CHECK_OBJECT( operand ); PyObject *result = api( operand ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } typedef PyObject *(binary_api)( PyObject *, PyObject * ); NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION( binary_api api, PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); PyObject *result = api( operand1, operand2 ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool BINARY_OPERATION_INPLACE( binary_api api, PyObject **operand1, PyObject *operand2 ) { assert( operand1 ); CHECK_OBJECT( *operand1 ); CHECK_OBJECT( operand2 ); // TODO: There is not really much point in these things. PyObject *result = BINARY_OPERATION( api, *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } // We got an object handed, that we have to release. Py_DECREF( *operand1 ); // That's our return value then. As we use a dedicated variable, it's // OK that way. *operand1 = result; return true; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_ADD( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); #if PYTHON_VERSION < 300 // Something similar for Python3 should exist too. if ( PyInt_CheckExact( operand1 ) && PyInt_CheckExact( operand2 ) ) { long a, b, i; a = PyInt_AS_LONG( operand1 ); b = PyInt_AS_LONG( operand2 ); i = a + b; // Detect overflow, in which case, a "long" object would have to be // created, which we won't handle here. if (likely(!( (i^a) < 0 && (i^b) < 0 ) )) { return PyInt_FromLong( i ); } } #endif binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_add; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_add; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_add; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif // Special case for "+", also works as sequence concat. PySequenceMethods *seq_methods = Py_TYPE( operand1 )->tp_as_sequence; if ( seq_methods && seq_methods->sq_concat ) { PyObject *result = (*seq_methods->sq_concat)( operand1, operand2 ); if (unlikely( result == NULL )) { return NULL; } return result; } PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for +: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } #if PYTHON_VERSION < 300 #include #define PyStringObject_SIZE ( offsetof( PyStringObject, ob_sval ) + 1 ) NUITKA_MAY_BE_UNUSED static bool STRING_RESIZE( PyObject **value, Py_ssize_t newsize ) { PyStringObject *sv; _Py_DEC_REFTOTAL; _Py_ForgetReference( *value ); *value = (PyObject *)PyObject_REALLOC( (char *)*value, PyStringObject_SIZE + newsize ); if (unlikely( *value == NULL )) { PyErr_NoMemory(); return false; } _Py_NewReference( *value ); sv = (PyStringObject *)*value; Py_SIZE( sv ) = newsize; sv->ob_sval[newsize] = '\0'; sv->ob_shash = -1; return true; } NUITKA_MAY_BE_UNUSED static bool STRING_ADD_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { assert( PyString_CheckExact( *operand1 ) ); assert( !PyString_CHECK_INTERNED( *operand1 ) ); assert( PyString_CheckExact( operand2 ) ); Py_ssize_t operand1_size = PyString_GET_SIZE( *operand1 ); Py_ssize_t operand2_size = PyString_GET_SIZE( operand2 ); Py_ssize_t new_size = operand1_size + operand2_size; if (unlikely( new_size < 0 )) { PyErr_Format( PyExc_OverflowError, "strings are too large to concat" ); return false; } if (unlikely( STRING_RESIZE( operand1, new_size ) == false )) { return false; } memcpy( PyString_AS_STRING( *operand1 ) + operand1_size, PyString_AS_STRING( operand2 ), operand2_size ); return true; } #else NUITKA_MAY_BE_UNUSED static bool UNICODE_ADD_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { #if PYTHON_VERSION < 330 Py_ssize_t operand1_size = PyUnicode_GET_SIZE( *operand1 ); Py_ssize_t operand2_size = PyUnicode_GET_SIZE( operand2 ); Py_ssize_t new_size = operand1_size + operand2_size; if (unlikely( new_size < 0 )) { PyErr_Format( PyExc_OverflowError, "strings are too large to concat" ); return false; } if (unlikely( PyUnicode_Resize( operand1, new_size ) != 0 )) { return false; } /* copy 'w' into the newly allocated area of 'v' */ memcpy( PyUnicode_AS_UNICODE( *operand1 ) + operand1_size, PyUnicode_AS_UNICODE( operand2 ), operand2_size * sizeof( Py_UNICODE ) ); return true; #else PyUnicode_Append( operand1, operand2 ); return !ERROR_OCCURRED(); #endif } #endif static bool FLOAT_ADD_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { assert( PyFloat_CheckExact( *operand1 ) ); assert( PyFloat_CheckExact( operand2 ) ); PyFPE_START_PROTECT("add", return false); PyFloat_AS_DOUBLE( *operand1 ) += PyFloat_AS_DOUBLE( operand2 ); PyFPE_END_PROTECT( *operand1 ); return true; } static bool FLOAT_MUL_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { assert( PyFloat_CheckExact( *operand1 ) ); assert( PyFloat_CheckExact( operand2 ) ); PyFPE_START_PROTECT("mul", return false); PyFloat_AS_DOUBLE( *operand1 ) *= PyFloat_AS_DOUBLE( operand2 ); PyFPE_END_PROTECT( *operand1 ); return true; } NUITKA_MAY_BE_UNUSED static bool BINARY_OPERATION_ADD_INPLACE( PyObject **operand1, PyObject *operand2 ) { assert( operand1 ); CHECK_OBJECT( *operand1 ); CHECK_OBJECT( operand2 ); #if PYTHON_VERSION < 300 // Something similar for Python3 should exist too. if ( PyInt_CheckExact( *operand1 ) && PyInt_CheckExact( operand2 ) ) { long a, b, i; a = PyInt_AS_LONG( *operand1 ); b = PyInt_AS_LONG( operand2 ); i = a + b; // Detect overflow, in which case, a "long" object would have to be // created, which we won't handle here. TODO: Add an else for that // case. if (likely(!( (i^a) < 0 && (i^b) < 0 ) )) { PyObject *result = PyInt_FromLong( i ); Py_DECREF( *operand1 ); *operand1 = result; return true; } } #endif #if PYTHON_VERSION < 300 if ( Py_REFCNT( *operand1 ) == 1 ) { // We more or less own the operand, so we might re-use its storage and // execute stuff in-place. if ( PyString_CheckExact( *operand1 ) && !PyString_CHECK_INTERNED( *operand1 ) && PyString_CheckExact( operand2 ) ) { return STRING_ADD_INCREMENTAL( operand1, operand2 ); } else if ( PyFloat_CheckExact( *operand1 ) && PyFloat_CheckExact( operand2 ) ) { return FLOAT_ADD_INCREMENTAL( operand1, operand2 ); } } // Strings are to be treated differently. if ( PyString_CheckExact( *operand1 ) && PyString_CheckExact( operand2 ) ) { PyString_Concat( operand1, operand2 ); return !ERROR_OCCURRED(); } #else if ( Py_REFCNT( *operand1 ) == 1 ) { // We more or less own the operand, so we might re-use its storage and // execute stuff in-place. if ( PyUnicode_CheckExact( *operand1 ) && !PyUnicode_CHECK_INTERNED( *operand1 ) && PyUnicode_CheckExact( operand2 ) ) { return UNICODE_ADD_INCREMENTAL( operand1, operand2 ); } else if ( PyFloat_CheckExact( *operand1 ) && PyFloat_CheckExact( operand2 ) ) { return FLOAT_ADD_INCREMENTAL( operand1, operand2 ); } } // Strings are to be treated differently. if ( PyUnicode_CheckExact( *operand1 ) && PyUnicode_CheckExact( operand2 ) ) { PyObject *result = PyUnicode_Concat( *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } Py_DECREF( *operand1 ); *operand1 = result; return true; } #endif PyObject *result = PyNumber_InPlaceAdd( *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } // We got an object handed, that we have to release. Py_DECREF( *operand1 ); // That's our return value then. As we use a dedicated variable, it's // OK that way. *operand1 = result; return true; } NUITKA_MAY_BE_UNUSED static bool BINARY_OPERATION_MUL_INPLACE( PyObject **operand1, PyObject *operand2 ) { assert( operand1 ); CHECK_OBJECT( *operand1 ); CHECK_OBJECT( operand2 ); if ( Py_REFCNT( *operand1 ) == 1 ) { if ( PyFloat_CheckExact( *operand1 ) && PyFloat_CheckExact( operand2 ) ) { return FLOAT_MUL_INCREMENTAL( operand1, operand2 ); } } PyObject *result = PyNumber_InPlaceMultiply( *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } // We got an object handed, that we have to release. Py_DECREF( *operand1 ); // That's our return value then. As we use a dedicated variable, it's // OK that way. *operand1 = result; return true; } static PyObject *SEQUENCE_REPEAT( ssizeargfunc repeatfunc, PyObject *seq, PyObject *n ) { if (unlikely( !PyIndex_Check( n ) )) { PyErr_Format( PyExc_TypeError, "can't multiply sequence by non-int of type '%s'", Py_TYPE( n )->tp_name ); return NULL; } PyObject *index_value = PyNumber_Index( n ); if (unlikely( index_value == NULL )) { return NULL; } /* We're done if PyInt_AsSsize_t() returns without error. */ #if PYTHON_VERSION < 300 Py_ssize_t count = PyInt_AsSsize_t( index_value ); #else Py_ssize_t count = PyLong_AsSsize_t( index_value ); #endif Py_DECREF( index_value ); if (unlikely( count == -1 )) // Note: -1 is an unlikely repetition count { PyObject *exception = GET_ERROR_OCCURRED(); if (unlikely( exception )) { if ( !EXCEPTION_MATCH_BOOL_SINGLE( exception, PyExc_OverflowError ) ) { return NULL; } PyErr_Format( PyExc_OverflowError, "cannot fit '%s' into an index-sized integer", Py_TYPE( n )->tp_name ); return NULL; } } PyObject *result = (*repeatfunc)( seq, count ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_MUL( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_multiply; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_multiply; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_multiply; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif // Special case for "+", also works as sequence concat. PySequenceMethods *seq_methods1 = Py_TYPE( operand1 )->tp_as_sequence; PySequenceMethods *seq_methods2 = Py_TYPE( operand2 )->tp_as_sequence; if ( seq_methods1 != NULL && seq_methods1->sq_repeat ) { return SEQUENCE_REPEAT( seq_methods1->sq_repeat, operand1, operand2 ); } if ( seq_methods2 != NULL && seq_methods2->sq_repeat ) { return SEQUENCE_REPEAT( seq_methods2->sq_repeat, operand2, operand1 ); } PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for *: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_SUB( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_subtract; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_subtract; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_subtract; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for -: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } #if PYTHON_VERSION < 300 NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_DIV( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_divide; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_divide; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_divide; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for /: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } #endif NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_REMAINDER( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_remainder; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_remainder; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_remainder; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for %%: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } NUITKA_MAY_BE_UNUSED static PyObject *POWER_OPERATION( PyObject *operand1, PyObject *operand2 ) { PyObject *result = PyNumber_Power( operand1, operand2, Py_None ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *POWER_OPERATION2( PyObject *operand1, PyObject *operand2 ) { PyObject *result = PyNumber_InPlacePower( operand1, operand2, Py_None ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool POWER_OPERATION_INPLACE( PyObject **operand1, PyObject *operand2 ) { PyObject *result = PyNumber_InPlacePower( *operand1, operand2, Py_None ); if (unlikely( result == NULL )) { return false; } if ( result != *operand1 ) { Py_DECREF( *operand1 ); *operand1 = result; } return true; } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/attributes.hpp0000644000175000017500000003560012651072046025126 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_ATTRIBUTES_H__ #define __NUITKA_HELPER_ATTRIBUTES_H__ #if PYTHON_VERSION < 300 NUITKA_MAY_BE_UNUSED static PyObject *FIND_ATTRIBUTE_IN_CLASS( PyClassObject *klass, PyObject *attr_name ) { CHECK_OBJECT( klass ); CHECK_OBJECT( attr_name ); assert( PyClass_Check( klass )); assert( PyString_Check( attr_name ) ); PyObject *result = GET_STRING_DICT_VALUE( (PyDictObject *)klass->cl_dict, (PyStringObject *)attr_name ); if ( result == NULL ) { Py_ssize_t base_count = PyTuple_Size( klass->cl_bases ); for ( Py_ssize_t i = 0; i < base_count; i++ ) { result = FIND_ATTRIBUTE_IN_CLASS( (PyClassObject *)PyTuple_GetItem( klass->cl_bases, i ), attr_name ); if ( result != NULL ) { break; } } } return result; } #endif #if PYTHON_VERSION < 300 extern PyObject *CALL_FUNCTION_WITH_ARGS2( PyObject *called, PyObject **args ); static PyObject *LOOKUP_INSTANCE( PyObject *source, PyObject *attr_name ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); assert( PyInstance_Check( source ) ); assert( PyString_Check( attr_name ) ); PyInstanceObject *source_instance = (PyInstanceObject *)source; // The special cases have their own variant on the code generation level // as we are called with constants only. assert( attr_name != const_str_plain___dict__ ); assert( attr_name != const_str_plain___class__ ); // Try the instance dict first. PyObject *result = GET_STRING_DICT_VALUE( (PyDictObject *)source_instance->in_dict, (PyStringObject *)attr_name ); if ( result ) { return INCREASE_REFCOUNT( result ); } // Next see if a class has it result = FIND_ATTRIBUTE_IN_CLASS( source_instance->in_class, attr_name ); if ( result != NULL ) { descrgetfunc func = Py_TYPE( result )->tp_descr_get; if ( func ) { result = func( result, source, (PyObject *)source_instance->in_class ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else { return INCREASE_REFCOUNT( result ); } } else { if (unlikely( !CHECK_AND_CLEAR_ATTRIBUTE_ERROR_OCCURRED() )) { return NULL; } } // Finally allow a __getattr__ to handle it or else it's an error. if (unlikely( source_instance->in_class->cl_getattr == NULL )) { PyErr_Format( PyExc_AttributeError, "%s instance has no attribute '%s'", PyString_AS_STRING( source_instance->in_class->cl_name ), PyString_AS_STRING( attr_name ) ); return NULL; } else { PyObject *args[] = { source, attr_name }; return CALL_FUNCTION_WITH_ARGS2( source_instance->in_class->cl_getattr, args ); } } #endif NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_ATTRIBUTE( PyObject *source, PyObject *attr_name ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); #if PYTHON_VERSION < 300 if ( PyInstance_Check( source ) ) { PyObject *result = LOOKUP_INSTANCE( source, attr_name ); return result; } else #endif { PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro != NULL ) { PyObject *result = (*type->tp_getattro)( source, attr_name ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *result = (*type->tp_getattr)( source, Nuitka_String_AsString_Unchecked( attr_name ) ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%s'", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return NULL; } } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_ATTRIBUTE_DICT_SLOT( PyObject *source ) { CHECK_OBJECT( source ); #if PYTHON_VERSION < 300 if (likely( PyInstance_Check( source ) )) { PyInstanceObject *source_instance = (PyInstanceObject *)source; return INCREASE_REFCOUNT( source_instance->in_dict ); } else #endif { PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro != NULL ) { PyObject *result = (*type->tp_getattro)( source, const_str_plain___dict__ ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *result = (*type->tp_getattr)( source, (char *)"__dict__" ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '__dict__'", type->tp_name ); return NULL; } } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_ATTRIBUTE_CLASS_SLOT( PyObject *source ) { CHECK_OBJECT( source ); #if PYTHON_VERSION < 300 if (likely( PyInstance_Check( source ) )) { PyInstanceObject *source_instance = (PyInstanceObject *)source; return INCREASE_REFCOUNT( (PyObject *)source_instance->in_class ); } else #endif { PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro != NULL ) { PyObject *result = (*type->tp_getattro)( source, const_str_plain___class__ ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *result = (*type->tp_getattr)( source, (char *)"__class__" ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '__class__'", type->tp_name ); return NULL; } } } NUITKA_MAY_BE_UNUSED static PyObject *BUILTIN_HASATTR( PyObject *source, PyObject *attr_name ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); int res = PyObject_HasAttr( source, attr_name ); if (unlikely( res == -1 )) { return NULL; } return BOOL_FROM(res == 1); } #if PYTHON_VERSION < 300 extern PyObject *CALL_FUNCTION_WITH_ARGS3( PyObject *called, PyObject **args ); static bool SET_INSTANCE( PyObject *target, PyObject *attr_name, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( attr_name ); CHECK_OBJECT( value ); assert( PyInstance_Check( target ) ); assert( PyString_Check( attr_name ) ); PyInstanceObject *target_instance = (PyInstanceObject *)target; // The special cases should get their own SET_ATTRIBUTE_xxxx_SLOT variants // on the code generation level as SET_ATTRIBUTE is called with constants // only. assert( attr_name != const_str_plain___dict__ ); assert( attr_name != const_str_plain___class__ ); if ( target_instance->in_class->cl_setattr != NULL ) { PyObject *args[] = { target, attr_name, value }; PyObject *result = CALL_FUNCTION_WITH_ARGS3( target_instance->in_class->cl_setattr, args ); if (unlikely( result == NULL )) { return false; } Py_DECREF( result ); return true; } else { int status = PyDict_SetItem( target_instance->in_dict, attr_name, value ); if (unlikely( status != 0 )) { return false; } return true; } } #endif NUITKA_MAY_BE_UNUSED static bool SET_ATTRIBUTE( PyObject *target, PyObject *attr_name, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( attr_name ); CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if ( PyInstance_Check( target ) ) { return SET_INSTANCE( target, attr_name, value ); } else #endif { PyTypeObject *type = Py_TYPE( target ); if ( type->tp_setattro != NULL ) { int status = (*type->tp_setattro)( target, attr_name, value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_setattr != NULL ) { int status = (*type->tp_setattr)( target, Nuitka_String_AsString_Unchecked( attr_name ), value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_getattr == NULL && type->tp_getattro == NULL ) { PyErr_Format( PyExc_TypeError, "'%s' object has no attributes (assign to %s)", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object has only read-only attributes (assign to %s)", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool SET_ATTRIBUTE_DICT_SLOT( PyObject *target, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if ( likely( PyInstance_Check( target ) )) { PyInstanceObject *target_instance = (PyInstanceObject *)target; if (unlikely( !PyDict_Check( value ) )) { PyErr_SetString( PyExc_TypeError, "__dict__ must be set to a dictionary" ); return false; } PyObject *old = target_instance->in_dict; target_instance->in_dict = INCREASE_REFCOUNT( value ); Py_DECREF( old ); } else #endif { PyTypeObject *type = Py_TYPE( target ); if ( type->tp_setattro != NULL ) { int status = (*type->tp_setattro)( target, const_str_plain___dict__, value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_setattr != NULL ) { int status = (*type->tp_setattr)( target, (char *)"__dict__", value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_getattr == NULL && type->tp_getattro == NULL ) { PyErr_Format( PyExc_TypeError, "'%s' object has no attributes (assign to __dict__)", type->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object has only read-only attributes (assign to __dict__)", type->tp_name ); return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool SET_ATTRIBUTE_CLASS_SLOT( PyObject *target, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if (likely( PyInstance_Check( target ) )) { PyInstanceObject *target_instance = (PyInstanceObject *)target; if (unlikely( !PyClass_Check( value ) )) { PyErr_SetString( PyExc_TypeError, "__class__ must be set to a class" ); return false; } PyObject *old = (PyObject *)( target_instance->in_class ); target_instance->in_class = (PyClassObject *)INCREASE_REFCOUNT( value ); Py_DECREF( old ); } else #endif { PyTypeObject *type = Py_TYPE( target ); if ( type->tp_setattro != NULL ) { int status = (*type->tp_setattro)( target, const_str_plain___class__, value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_setattr != NULL ) { int status = (*type->tp_setattr)( target, (char *)"__class__", value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_getattr == NULL && type->tp_getattro == NULL ) { PyErr_Format( PyExc_TypeError, "'%s' object has no attributes (assign to __class__)", type->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object has only read-only attributes (assign to __class__)", type->tp_name ); return false; } } return true; } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SPECIAL( PyObject *source, PyObject *attr_name ) { #if PYTHON_VERSION < 300 if ( PyInstance_Check( source ) ) { return LOOKUP_INSTANCE( source, attr_name ); } #endif // TODO: There is heavy optimization in CPython to avoid it. Potentially // that's worth it to imitate that. PyObject *result = _PyType_Lookup( Py_TYPE( source ), attr_name ); if (likely( result )) { descrgetfunc func = Py_TYPE( result )->tp_descr_get; if ( func == NULL ) { return INCREASE_REFCOUNT( result ); } else { PyObject *func_result = func( result, source, (PyObject *)( Py_TYPE( source ) ) ); if (unlikely( func_result == NULL )) { return NULL; } CHECK_OBJECT( func_result ); return func_result; } } PyErr_SetObject( PyExc_AttributeError, attr_name ); return NULL; } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/raising.hpp0000644000175000017500000003661712651071040024376 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_RAISING_H__ #define __NUITKA_HELPER_RAISING_H__ #if PYTHON_VERSION < 266 #define WRONG_EXCEPTION_TYPE_ERROR_MESSAGE "exceptions must be classes or instances, not %s" #elif PYTHON_VERSION < 300 #define WRONG_EXCEPTION_TYPE_ERROR_MESSAGE "exceptions must be old-style classes or derived from BaseException, not %s" #else #define WRONG_EXCEPTION_TYPE_ERROR_MESSAGE "exceptions must derive from BaseException" #endif #if PYTHON_VERSION >= 300 static void CHAIN_EXCEPTION( PyObject *exception_value ) { // Implicit chain of exception already existing. PyThreadState *thread_state = PyThreadState_GET(); // Normalize existing exception first. TODO: Will normally be done already. NORMALIZE_EXCEPTION( &thread_state->exc_type, &thread_state->exc_value, (PyTracebackObject **)&thread_state->exc_traceback ); PyObject *old_exc_value = thread_state->exc_value; if ( old_exc_value != NULL && old_exc_value != Py_None && old_exc_value != exception_value ) { PyObject *current = old_exc_value; while( true ) { PyObject *context = PyException_GetContext( current ); if (!context) break; CHECK_OBJECT( context ); Py_DECREF( context ); CHECK_OBJECT( context ); if ( context == exception_value ) { PyException_SetContext( current, NULL ); break; } current = context; } CHECK_OBJECT( old_exc_value ); Py_INCREF( old_exc_value ); PyException_SetContext( exception_value, old_exc_value ); CHECK_OBJECT( thread_state->exc_traceback ); PyException_SetTraceback( old_exc_value, thread_state->exc_traceback ); } } #endif NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_WITH_TYPE( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { *exception_value = NULL; *exception_tb = NULL; #if PYTHON_VERSION < 300 // Next, repeatedly, replace a tuple exception with its first item while( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 ) { PyObject *tmp = *exception_type; *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); Py_INCREF( *exception_type ); Py_DECREF( tmp ); } #endif if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 270 if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)*exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); Py_DECREF( *exception_type ); Py_DECREF( *exception_value ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } #endif #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); // TODO: Ever true? if ( *exception_tb ) { PyTracebackObject *prev = (PyTracebackObject *)PyException_GetTraceback( *exception_value ); if ( prev != NULL ) { assert( (*exception_tb)->tb_next == NULL ); (*exception_tb)->tb_next = prev; } PyException_SetTraceback( *exception_value, (PyObject *)(*exception_tb ? *exception_tb : (PyTracebackObject *)Py_None ) ); } *exception_tb = (PyTracebackObject *)PyException_GetTraceback( *exception_value ); #endif return; } else { Py_DECREF( *exception_type ); PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( *exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } #if PYTHON_VERSION >= 300 NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_WITH_CAUSE( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb, PyObject *exception_cause ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( exception_cause ); *exception_tb = NULL; #if PYTHON_VERSION >= 330 // None is not a cause. if ( exception_cause == Py_None ) { Py_DECREF( exception_cause ); exception_cause = NULL; } else #endif if ( PyExceptionClass_Check( exception_cause ) ) { PyObject *old_exception_cause = exception_cause; exception_cause = PyObject_CallObject( exception_cause, NULL ); Py_DECREF( old_exception_cause ); if (unlikely( exception_cause == NULL )) { Py_DECREF( *exception_type ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } #if PYTHON_VERSION >= 330 if (unlikely( exception_cause != NULL && !PyExceptionInstance_Check( exception_cause ) )) #else if (unlikely( !PyExceptionInstance_Check( exception_cause ) )) #endif { Py_DECREF( *exception_type ); Py_XDECREF( *exception_tb ); Py_XDECREF( exception_cause ); #ifdef _NUITKA_FULL_COMPAT PyErr_Format( PyExc_TypeError, "exception causes must derive from BaseException" ); #else PyErr_Format( PyExc_TypeError, "exception causes must derive from BaseException (%s does not)", Py_TYPE( exception_cause )->tp_name ); #endif FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_DECREF( *exception_tb ); Py_XDECREF( exception_cause ); PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } PyException_SetCause( *exception_value, exception_cause ); #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); PyException_SetCause( *exception_value, exception_cause ); #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); #endif return; } else { Py_DECREF( *exception_type ); Py_XDECREF( exception_cause ); PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } #endif NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_WITH_VALUE( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( *exception_value ); *exception_tb = NULL; // Non-empty tuple exceptions are the first element. while (unlikely( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 )) { *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); } if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 270 if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)*exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { if (unlikely( *exception_value != NULL && *exception_value != Py_None )) { PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } // The type is rather a value, so we are overriding it here. *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_IMPLICIT( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( *exception_value ); *exception_tb = NULL; // Non-empty tuple exceptions are the first element. while (unlikely( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 )) { *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); } if ( PyExceptionClass_Check( *exception_type ) ) { #if PYTHON_VERSION >= 340 NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); CHAIN_EXCEPTION( *exception_value ); #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { #if PYTHON_VERSION >= 340 CHAIN_EXCEPTION( *exception_value ); #endif // The type is rather a value, so we are overriding it here. *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 340 CHAIN_EXCEPTION( *exception_value ); #endif return; } } NUITKA_MAY_BE_UNUSED static inline void RAISE_EXCEPTION_WITH_TRACEBACK( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( *exception_value ); if ( *exception_tb == (PyTracebackObject *)Py_None ) { Py_DECREF( *exception_tb ); *exception_tb = NULL; } // Non-empty tuple exceptions are the first element. while (unlikely( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 )) { *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); } if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 270 if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)*exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { if (unlikely( *exception_value != NULL && *exception_value != Py_None )) { PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } // The type is rather a value, so we are overriding it here. *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } NUITKA_MAY_BE_UNUSED static void RERAISE_EXCEPTION( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { PyThreadState *tstate = PyThreadState_GET(); assert( tstate ); *exception_type = tstate->exc_type != NULL ? tstate->exc_type : Py_None; Py_INCREF( *exception_type ); *exception_value = tstate->exc_value; Py_XINCREF( *exception_value ); *exception_tb = (PyTracebackObject *)tstate->exc_traceback; Py_XINCREF( *exception_tb ); CHECK_OBJECT( *exception_type ); if ( *exception_type == Py_None ) { #if PYTHON_VERSION >= 300 Py_DECREF( *exception_type ); *exception_type = INCREASE_REFCOUNT( PyExc_RuntimeError ); *exception_value = PyUnicode_FromString( "No active exception to reraise" ); *exception_tb = NULL; #else PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( *exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); #endif } } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helper/sequences.hpp0000644000175000017500000000565412651071040024732 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_SEQUENCES_H__ #define __NUITKA_HELPER_SEQUENCES_H__ NUITKA_MAY_BE_UNUSED static PyObject *SEQUENCE_CONTAINS( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return NULL; } return BOOL_FROM( result == 1 ); } NUITKA_MAY_BE_UNUSED static PyObject *SEQUENCE_CONTAINS_NOT( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return NULL; } return BOOL_FROM( result == 0 ); } NUITKA_MAY_BE_UNUSED static bool SEQUENCE_CONTAINS_BOOL( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return NULL; } return result == 1; } NUITKA_MAY_BE_UNUSED static bool SEQUENCE_CONTAINS_NOT_BOOL( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return NULL; } return result == 0; } NUITKA_MAY_BE_UNUSED static bool SEQUENCE_SETITEM( PyObject *sequence, Py_ssize_t index, PyObject *value ) { CHECK_OBJECT( sequence ); CHECK_OBJECT( value ); PySequenceMethods *sequence_methods = Py_TYPE( sequence )->tp_as_sequence; if ( sequence_methods != NULL && sequence_methods->sq_ass_item ) { if ( index < 0 ) { if ( sequence_methods->sq_length ) { Py_ssize_t length = (*sequence_methods->sq_length)( sequence ); if ( length < 0 ) { return false; } index += length; } } int res = sequence_methods->sq_ass_item( sequence, index, value ); if (unlikely( res == -1 )) { return false; } return true; } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( sequence )->tp_name ); return false; } } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/compiled_frame.hpp0000644000175000017500000000330412651071040024414 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_FRAME_H__ #define __NUITKA_COMPILED_FRAME_H__ // Create a frame object for the given code object, frame or module. extern PyFrameObject *MAKE_MODULE_FRAME( PyCodeObject *code, PyObject *module ); extern PyFrameObject *MAKE_FUNCTION_FRAME( PyCodeObject *code, PyObject *module ); // Create a code object for the given filename and function name #if PYTHON_VERSION < 300 extern PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int flags ); #else extern PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int kw_only_count, int flags ); #endif extern PyTypeObject Nuitka_Frame_Type; static inline bool Nuitka_Frame_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Frame_Type; } extern PyFrameObject *duplicateFrame( PyFrameObject *old_frame ); #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/unfreezing.hpp0000644000175000017500000000333412651071040023625 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_UNFREEZING_H__ #define __NUITKA_UNFREEZING_H__ // This define guards these definitions from being used without the unfreezing // mode actually being active at all. #define NUITKA_COMPILED_MODULE 0 #define NUITKA_SHLIB_MODULE 1 #define NUITKA_COMPILED_PACKAGE 2 struct Nuitka_MetaPathBasedLoaderEntry { // Full module name, including package char *name; // Entry function if compiled module, otherwise NULL. #if PYTHON_VERSION < 300 void (*python_initfunc)( void ); #else PyObject * (*python_initfunc)( void ); #endif // Flags: NUITKA_COMPILED_MODULE or NUITKA_SHLIB_MODULE int flags; }; // For embedded modules, to be unpacked. Used by main program/package only extern void registerMetaPathBasedUnfreezer( struct Nuitka_MetaPathBasedLoaderEntry *loader_entries ); // For use as the "__loader__" attribute of compiled modules. #if PYTHON_VERSION >= 330 extern PyObject *metapath_based_loader; #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/calling.hpp0000644000175000017500000000541512651071040023064 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_CALLING_H__ #define __NUITKA_CALLING_H__ #include "__helpers.hpp" extern PyObject *const_tuple_empty; NUITKA_MAY_BE_UNUSED static PyObject *CALL_FUNCTION( PyObject *function_object, PyObject *positional_args, PyObject *named_args ) { assert( !ERROR_OCCURRED() ); CHECK_OBJECT( function_object ); CHECK_OBJECT( positional_args ); assert( named_args == NULL || Py_REFCNT( named_args ) > 0 ); ternaryfunc call_slot = Py_TYPE( function_object )->tp_call; if (unlikely( call_slot == NULL )) { PyErr_Format( PyExc_TypeError, "'%s' object is not callable", function_object->ob_type->tp_name ); return NULL; } if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object") )) { return NULL; } PyObject *result = (*call_slot)( function_object, positional_args, named_args ); Py_LeaveRecursiveCall(); if ( result == NULL ) { if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in CALL_FUNCTION" ); } return NULL; } else { // Some buggy C functions do this, and Nuitka inner workings can get // upset from it. DROP_ERROR_OCCURRED(); return result; } } // Function call variant with no arguments provided at all. extern PyObject *CALL_FUNCTION_NO_ARGS( PyObject *called ); // Function call variants with positional arguments tuple. NUITKA_MAY_BE_UNUSED static PyObject *CALL_FUNCTION_WITH_POSARGS( PyObject *function_object, PyObject *positional_args ) { return CALL_FUNCTION( function_object, positional_args, NULL ); } NUITKA_MAY_BE_UNUSED static PyObject *CALL_FUNCTION_WITH_KEYARGS( PyObject *function_object, PyObject *named_args ) { return CALL_FUNCTION( function_object, const_tuple_empty, named_args ); } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/frame_stack.hpp0000644000175000017500000001276712651071040023742 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_FRAME_STACK_H__ #define __NUITKA_FRAME_STACK_H__ #define MAKE_OR_REUSE_FRAME( cache_identifier, code_identifier, module_identifier ) \ if ( isFrameUnusable( cache_identifier ) ) \ { \ Py_XDECREF( cache_identifier ); \ cache_identifier = MAKE_FUNCTION_FRAME( code_identifier, module_identifier ); \ } \ inline static void assertCodeObject( PyCodeObject *code_object ) { CHECK_OBJECT( (PyObject *)code_object ); } inline static void assertFrameObject( PyFrameObject *frame_object ) { CHECK_OBJECT( (PyObject *)frame_object ); assertCodeObject( frame_object->f_code ); } NUITKA_MAY_BE_UNUSED static PyFrameObject *INCREASE_REFCOUNT( PyFrameObject *frame_object ) { assertFrameObject( frame_object ); Py_INCREF( frame_object ); return frame_object; } NUITKA_MAY_BE_UNUSED static bool isFrameUnusable( PyFrameObject *frame_object ) { if ( frame_object ) CHECK_OBJECT( (PyObject *)frame_object ); bool result = // Never used. frame_object == NULL || // Still in use Py_REFCNT( frame_object ) > 1 || #if PYTHON_VERSION < 340 // Last used by another thread (TODO: Could just set it when re-using) frame_object->f_tstate != PyThreadState_GET() || #endif // Was detached from (TODO: When detaching, can't we just have another // frame guard instead) frame_object->f_back != NULL; #if _DEBUG_REFRAME if (result && frame_object) { PRINT_STRING("NOT REUSING FRAME:"); PRINT_ITEM((PyObject *)frame_object); PRINT_REFCOUNT( (PyObject *)frame_object ); if ( frame_object->f_back ) PRINT_ITEM( (PyObject *)frame_object->f_back ); PRINT_NEW_LINE(); } #endif return result; } NUITKA_MAY_BE_UNUSED inline static void popFrameStack( void ) { PyThreadState *tstate = PyThreadState_GET(); PyFrameObject *old = tstate->frame; #if _DEBUG_FRAME printf( "Taking off frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)old ) ), PyString_AsString( PyObject_Repr( (PyObject *)old->f_code ) ) ); #endif tstate->frame = old->f_back; old->f_back = NULL; // We might be very top level, e.g. in a thread, and therefore do not insist // on value. Py_XDECREF( tstate->frame ); #if _DEBUG_FRAME if (tstate->frame) { printf( "Now at top frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)tstate->frame ) ), PyString_AsString( PyObject_Repr( (PyObject *)tstate->frame->f_code ) ) ); } else { printf( "Now at top no frame\n"); } #endif } NUITKA_MAY_BE_UNUSED inline static void pushFrameStack( PyFrameObject *frame_object ) { assertFrameObject( frame_object ); PyThreadState *tstate = PyThreadState_GET(); // Look at current frame. PyFrameObject *old = tstate->frame; #if _DEBUG_FRAME if ( old ) { assertCodeObject( old->f_code ); printf( "Upstacking to frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)old ) ), PyString_AsString( PyObject_Repr( (PyObject *)old->f_code ) ) ); } #endif // No recursion allowed of course, assert against it. assert( old != frame_object ); // Push the new frame as the currently active one. tstate->frame = frame_object; // We don't allow touching cached frame objects where this is not true. assert( frame_object->f_back == NULL ); if ( old != NULL ) { assertFrameObject( old ); frame_object->f_back = old; Py_INCREF( frame_object->f_back ); } #if _DEBUG_FRAME printf( "Now at top frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)tstate->frame ) ), PyString_AsString( PyObject_Repr( (PyObject *)tstate->frame->f_code ) ) ); #endif } #if _DEBUG_REFRAME static inline void dumpFrameStack( void ) { PyFrameObject *current = PyThreadState_GET()->frame; int total = 0; while( current ) { total++; current = current->f_back; } current = PyThreadState_GET()->frame; puts( ">--------->" ); while( current ) { printf( "Frame stack %d: %s %s\n", total--, PyString_AsString( PyObject_Str( (PyObject *)current ) ), PyString_AsString( PyObject_Str( (PyObject *)current->f_code ) ) ); current = current->f_back; } puts( ">---------<" ); } #endif // Make a replacement for the traceback frame, that we again own it exclusively // enough so that the line numbers are detached. extern void detachFrame( PyTracebackObject *traceback, PyObject *locals ); #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/compiled_generator.hpp0000644000175000017500000001540512651071040025315 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_GENERATOR_H__ #define __NUITKA_COMPILED_GENERATOR_H__ #include "Python.h" #include "methodobject.h" #include "frameobject.h" // Compiled generator function type. // Another cornerstone of the integration into CPython. Try to behave as well as // normal generator objects do or even better. #include "fibers.hpp" // Status of the generator object. enum Generator_Status { status_Unused, // Not used so far status_Running, // Running, used but didn't stop yet status_Finished // Stopped, no more values to come }; // The Nuitka_GeneratorObject is the storage associated with a compiled // generator object instance of which there can be many for each code. typedef struct { PyObject_HEAD PyObject *m_name; #if PYTHON_VERSION >= 350 PyObject *m_qualname; PyObject *m_yieldfrom; #endif Fiber m_yielder_context; Fiber m_caller_context; // Weak references are supported for generator objects in CPython. PyObject *m_weakrefs; int m_running; void *m_code; PyObject *m_yielded; PyObject *m_exception_type, *m_exception_value; PyTracebackObject *m_exception_tb; PyFrameObject *m_frame; PyCodeObject *m_code_object; // Closure variables given, if any, we reference cells here. PyCellObject **m_closure; Py_ssize_t m_closure_given; // Was it ever used, is it still running, or already finished. Generator_Status m_status; } Nuitka_GeneratorObject; extern PyTypeObject Nuitka_Generator_Type; typedef void (*generator_code)( Nuitka_GeneratorObject * ); #if PYTHON_VERSION < 350 extern PyObject *Nuitka_Generator_New( generator_code code, PyObject *name, PyCodeObject *code_object, PyCellObject **closure, Py_ssize_t closure_given ); #else extern PyObject *Nuitka_Generator_New( generator_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyCellObject **closure, Py_ssize_t closure_given ); #endif static inline bool Nuitka_Generator_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Generator_Type; } static inline PyObject *Nuitka_Generator_GetName( PyObject *object ) { return ((Nuitka_GeneratorObject *)object)->m_name; } static inline PyObject *YIELD( Nuitka_GeneratorObject *generator, PyObject *value ) { CHECK_OBJECT( value ); generator->m_yielded = value; #if PYTHON_VERSION >= 340 generator->m_frame->f_executing -= 1; #endif // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); #if PYTHON_VERSION >= 340 generator->m_frame->f_executing += 1; #endif // Check for thrown exception. if (unlikely( generator->m_exception_type )) { RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } CHECK_OBJECT( generator->m_yielded ); return generator->m_yielded; } #if PYTHON_VERSION >= 300 static inline PyObject *YIELD_IN_HANDLER( Nuitka_GeneratorObject *generator, PyObject *value ) { CHECK_OBJECT( value ); generator->m_yielded = value; // When yielding from an exception handler in Python3, the exception // preserved to the frame is restore, while the current one is put there. PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD exit:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; #if PYTHON_VERSION >= 340 generator->m_frame->f_executing -= 1; #endif // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); #if PYTHON_VERSION >= 340 generator->m_frame->f_executing += 1; #endif // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD return:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; // Check for thrown exception. if (unlikely( generator->m_exception_type )) { RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } return generator->m_yielded; } #endif #if PYTHON_VERSION >= 330 extern PyObject *YIELD_FROM( Nuitka_GeneratorObject *generator, PyObject *target ); extern PyObject *YIELD_FROM_IN_HANDLER( Nuitka_GeneratorObject *generator, PyObject *target ); #endif #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/helpers.hpp0000644000175000017500000005164412651072046023131 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPERS_H__ #define __NUITKA_HELPERS_H__ #define _DEBUG_FRAME 0 #define _DEBUG_REFRAME 0 #define _DEBUG_EXCEPTIONS 0 #define _DEBUG_COROUTINE 0 extern PyObject *const_tuple_empty; extern PyObject *const_str_plain___dict__; extern PyObject *const_str_plain___class__; extern PyObject *const_str_plain___enter__; extern PyObject *const_str_plain___exit__; // From CPython, to allow us quick access to the dictionary of an module, the // structure is normally private, but we need it for quick access to the module // dictionary. typedef struct { PyObject_HEAD PyObject *md_dict; } PyModuleObject; // Most fundamental, because we use it for debugging in everything else. #include "nuitka/helper/printing.hpp" static PyObject *INCREASE_REFCOUNT( PyObject *object ); // Helper to check that an object is valid and has positive reference count. #define CHECK_OBJECT( value ) ( assert( value != NULL ), assert( Py_REFCNT( value ) > 0 ) ); #include "nuitka/exceptions.hpp" // Helper functions for reference count handling in the fly. NUITKA_MAY_BE_UNUSED static PyObject *INCREASE_REFCOUNT( PyObject *object ) { CHECK_OBJECT( object ); Py_INCREF( object ); return object; } NUITKA_MAY_BE_UNUSED static PyObject *DECREASE_REFCOUNT( PyObject *object ) { CHECK_OBJECT( object ); Py_DECREF( object ); return object; } // For checking values if they changed or not. #ifndef __NUITKA_NO_ASSERT__ extern Py_hash_t DEEP_HASH( PyObject *value ); #endif // For profiling of Nuitka compiled binaries #if _NUITKA_PROFILE extern void startProfiling( void ); extern void stopProfiling( void ); #endif #include "nuitka/helper/boolean.hpp" #include "nuitka/helper/dictionaries.hpp" #include "nuitka/helper/rangeobjects.hpp" #if PYTHON_VERSION >= 300 static char *_PyUnicode_AS_STRING( PyObject *unicode ) { #if PYTHON_VERSION < 330 PyObject *bytes = _PyUnicode_AsDefaultEncodedString( unicode, NULL ); if (unlikely( bytes == NULL )) { return NULL; } return PyBytes_AS_STRING( bytes ); #else return PyUnicode_AsUTF8( unicode ); #endif } #endif #include "nuitka/helper/raising.hpp" #include "helper/operations.hpp" #include "nuitka/helper/richcomparisons.hpp" #include "nuitka/helper/sequences.hpp" static inline bool Nuitka_Function_Check( PyObject *object ); static inline PyObject *Nuitka_Function_GetName( PyObject *object ); static inline bool Nuitka_Generator_Check( PyObject *object ); static inline PyObject *Nuitka_Generator_GetName( PyObject *object ); #include "nuitka/calling.hpp" NUITKA_MAY_BE_UNUSED static PyObject *TO_FLOAT( PyObject *value ) { PyObject *result; #if PYTHON_VERSION < 300 if ( PyString_CheckExact( value ) ) { result = PyFloat_FromString( value, NULL ); } #else if ( PyUnicode_CheckExact( value ) ) { result = PyFloat_FromString( value ); } #endif else { result = PyNumber_Float( value ); } if (unlikely( result == NULL )) { return NULL; } return result; } extern PyObject *const_int_0; NUITKA_MAY_BE_UNUSED static PyObject *TO_COMPLEX( PyObject *real, PyObject *imag ) { // TODO: Very lazy here, we should create the values ourselves, surely a // a lot of optimization can be had that way. if ( real == NULL ) { assert( imag != NULL ); real = const_int_0; } if ( imag == NULL) { PyObject *args[] = { real }; return CALL_FUNCTION_WITH_ARGS1( (PyObject *)&PyComplex_Type, args ); } else { PyObject *args[] = { real, imag }; return CALL_FUNCTION_WITH_ARGS2( (PyObject *)&PyComplex_Type, args ); } } NUITKA_MAY_BE_UNUSED static PyObject *TO_INT2( PyObject *value, PyObject *base ) { // TODO: Need to check if 3.4 is really the first version to do this. #if PYTHON_VERSION < 340 long base_int = PyInt_AsLong( base ); #else Py_ssize_t base_int = PyNumber_AsSsize_t( base, NULL ); #endif if (unlikely( base_int == -1 )) { PyObject *error = GET_ERROR_OCCURRED(); if (likely( error )) { #if PYTHON_VERSION >= 300 if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_OverflowError ) ) { PyErr_Format( PyExc_ValueError, #if PYTHON_VERSION < 324 "int() arg 2 must be >= 2 and <= 36" #else "int() base must be >= 2 and <= 36" #endif ); } #endif return NULL; } } #if PYTHON_VERSION >= 300 if (unlikely( ( base_int != 0 && base_int < 2 ) || base_int > 36 )) { PyErr_Format( PyExc_ValueError, #if PYTHON_VERSION < 324 "int() arg 2 must be >= 2 and <= 36" #else "int() base must be >= 2 and <= 36" #endif ); return NULL; } #endif #if PYTHON_VERSION < 300 if (unlikely( !Nuitka_String_Check( value ) && !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "int() can't convert non-string with explicit base" ); return NULL; } char *value_str = Nuitka_String_AsString( value ); if (unlikely( value_str == NULL )) { return NULL; } PyObject *result = PyInt_FromString( value_str, NULL, base_int ); if (unlikely( result == NULL )) { return NULL; } return result; #else if ( PyUnicode_Check( value ) ) { #if PYTHON_VERSION < 330 char *value_str = Nuitka_String_AsString( value ); if (unlikely( value_str == NULL )) { return NULL; } PyObject *result = PyInt_FromString( value_str, NULL, base_int ); if (unlikely( result == NULL )) { return NULL; } return result; #else return PyLong_FromUnicodeObject( value, (int)base_int ); #endif } else if ( PyBytes_Check( value ) || PyByteArray_Check( value ) ) { // Check for "NUL" as PyLong_FromString has no length parameter, Py_ssize_t size = Py_SIZE( value ); char *value_str; if ( PyByteArray_Check( value ) ) { value_str = PyByteArray_AS_STRING( value ); } else { value_str = PyBytes_AS_STRING( value ); } PyObject *result = NULL; if ( size != 0 && strlen( value_str ) == (size_t)size ) { result = PyInt_FromString( value_str, NULL, (int)base_int ); } if (unlikely( result == NULL )) { PyErr_Format( PyExc_ValueError, "invalid literal for int() with base %d: %R", base_int, value ); return NULL; } return result; } else { PyErr_Format( PyExc_TypeError, "int() can't convert non-string with explicit base" ); return NULL; } #endif } #if PYTHON_VERSION < 300 // Note: Python3 uses TO_INT2 function. NUITKA_MAY_BE_UNUSED static PyObject *TO_LONG2( PyObject *value, PyObject *base ) { long base_int = PyInt_AsLong( base ); if (unlikely( base_int == -1 )) { if (likely( ERROR_OCCURRED() )) { return NULL; } } if (unlikely( !Nuitka_String_Check( value ) && !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "long() can't convert non-string with explicit base" ); return NULL; } char *value_str = Nuitka_String_AsString( value ); if (unlikely( value_str == NULL )) { return NULL; } PyObject *result = PyLong_FromString( value_str, NULL, base_int ); if (unlikely( result == NULL )) { return NULL; } return result; } #endif NUITKA_MAY_BE_UNUSED static PyObject *TO_BOOL( PyObject *value ) { int res = CHECK_IF_TRUE( value ); if (unlikely( res == -1 )) return NULL; return BOOL_FROM( res != 0 ); } NUITKA_MAY_BE_UNUSED static PyObject *TO_UNICODE3( PyObject *value, PyObject *encoding, PyObject *errors ) { CHECK_OBJECT( value ); if ( encoding ) CHECK_OBJECT( encoding ); if ( errors ) CHECK_OBJECT( errors ); char *encoding_str; if ( encoding == NULL ) { encoding_str = NULL; } else if ( Nuitka_String_Check( encoding ) ) { encoding_str = Nuitka_String_AsString_Unchecked( encoding ); } #if PYTHON_VERSION < 300 else if ( PyUnicode_Check( encoding ) ) { PyObject *uarg2 = _PyUnicode_AsDefaultEncodedString( encoding, NULL ); CHECK_OBJECT( uarg2 ); encoding_str = Nuitka_String_AsString_Unchecked( uarg2 ); } #endif else { PyErr_Format( PyExc_TypeError, "unicode() argument 2 must be string, not %s", Py_TYPE( encoding )->tp_name ); return NULL; } char *errors_str; if ( errors == NULL ) { errors_str = NULL; } else if ( Nuitka_String_Check( errors ) ) { errors_str = Nuitka_String_AsString_Unchecked( errors ); } #if PYTHON_VERSION < 300 else if ( PyUnicode_Check( errors ) ) { PyObject *uarg3 = _PyUnicode_AsDefaultEncodedString( errors, NULL ); CHECK_OBJECT( uarg3 ); errors_str = Nuitka_String_AsString_Unchecked( uarg3 ); } #endif else { PyErr_Format( PyExc_TypeError, "unicode() argument 3 must be string, not %s", Py_TYPE( errors )->tp_name ); return NULL; } PyObject *result = PyUnicode_FromEncodedObject( value, encoding_str, errors_str ); if (unlikely( result == NULL )) { return NULL; } assert( PyUnicode_Check( result ) ); return result; } NUITKA_MAY_BE_UNUSED static PyObject *MAKE_STATIC_METHOD( PyObject *method ) { CHECK_OBJECT( method ); PyObject *attempt = PyStaticMethod_New( method ); if ( attempt ) { return attempt; } else { CLEAR_ERROR_OCCURRED(); return method; } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_VARS( PyObject *source ) { CHECK_OBJECT( source ); PyObject *result = PyObject_GetAttr( source, const_str_plain___dict__ ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_TypeError, "vars() argument must have __dict__ attribute" ); return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *IMPORT_NAME( PyObject *module, PyObject *import_name ) { CHECK_OBJECT( module ); CHECK_OBJECT( import_name ); PyObject *result = PyObject_GetAttr( module, import_name ); if (unlikely( result == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { #if PYTHON_VERSION < 340 PyErr_Format( PyExc_ImportError, "cannot import name %s", Nuitka_String_AsString( import_name )); #else PyErr_Format( PyExc_ImportError, "cannot import name '%s'", Nuitka_String_AsString( import_name )); #endif } return NULL; } return result; } #include "nuitka/helper/indexes.hpp" #include "nuitka/helper/subscripts.hpp" #include "nuitka/helper/slices.hpp" #include "nuitka/helper/attributes.hpp" #include "nuitka/helper/iterators.hpp" #include "nuitka/builtins.hpp" #include "nuitka/frame_stack.hpp" NUITKA_MAY_BE_UNUSED static PyObject *LIST_COPY( PyObject *list ) { CHECK_OBJECT( list ); assert( PyList_CheckExact( list ) ); Py_ssize_t size = PyList_GET_SIZE( list ); PyObject *result = PyList_New( size ); if (unlikely( result == NULL )) { return NULL; } for ( Py_ssize_t i = 0; i < size; i++ ) { PyList_SET_ITEM( result, i, INCREASE_REFCOUNT( PyList_GET_ITEM( list, i ) ) ); } return result; } // Compile source code given, pretending the file name was given. #if PYTHON_VERSION < 300 extern PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit ); #else extern PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit, PyObject *optimize ); #endif // For quicker built-in open() functionality. extern PyObject *BUILTIN_OPEN( PyObject *file_name, PyObject *mode, PyObject *buffering ); // For quicker built-in chr() functionality. extern PyObject *BUILTIN_CHR( PyObject *value ); // For quicker built-in ord() functionality. extern PyObject *BUILTIN_ORD( PyObject *value ); // For quicker built-in bin() functionality. extern PyObject *BUILTIN_BIN( PyObject *value ); // For quicker built-in oct() functionality. extern PyObject *BUILTIN_OCT( PyObject *value ); // For quicker built-in hex() functionality. extern PyObject *BUILTIN_HEX( PyObject *value ); // For quicker callable() functionality. extern PyObject *BUILTIN_CALLABLE( PyObject *value ); // For quicker iter() functionality if 2 arguments arg given. extern PyObject *BUILTIN_ITER2( PyObject *callable, PyObject *sentinel ); // For quicker type() functionality if 1 argument is given. extern PyObject *BUILTIN_TYPE1( PyObject *arg ); // For quicker type() functionality if 3 arguments are given (to build a new // type). extern PyObject *BUILTIN_TYPE3( PyObject *module_name, PyObject *name, PyObject *bases, PyObject *dict ); // For built-in built-in range() functionality. extern PyObject *BUILTIN_RANGE3( PyObject *low, PyObject *high, PyObject *step ); extern PyObject *BUILTIN_RANGE2( PyObject *low, PyObject *high ); extern PyObject *BUILTIN_RANGE( PyObject *boundary ); extern PyObject *BUILTIN_XRANGE( PyObject *low, PyObject *high, PyObject *step ); // For built-in built-in len() functionality. extern PyObject *BUILTIN_LEN( PyObject *boundary ); // For built-in built-in super() functionality. extern PyObject *BUILTIN_SUPER( PyObject *type, PyObject *object ); // For built-in isinstance() functionality. extern PyObject *BUILTIN_ISINSTANCE( PyObject *inst, PyObject *cls ); // The patched isinstance() functionality used for the built-in. extern int Nuitka_IsInstance( PyObject *inst, PyObject *cls ); // For built-in getattr() functionality. extern PyObject *BUILTIN_GETATTR( PyObject *object, PyObject *attribute, PyObject *default_value ); // For built-in setattr() functionality. extern PyObject *BUILTIN_SETATTR( PyObject *object, PyObject *attribute, PyObject *value ); // For built-in bytearray() functionality. extern PyObject *BUILTIN_BYTEARRAY( PyObject *value ); // For built-in hash() functionality. extern PyObject *BUILTIN_HASH( PyObject *value ); extern PyObject *const_str_plain___builtins__; // extern PyObject *EVAL_CODE( PyObject *code, PyObject *globals, PyObject *locals ); #include "nuitka/importing.hpp" // For the constant loading: // Call this to initialize all common constants pre-main. extern void createGlobalConstants( void ); // Call this to check of common constants are still intact. #ifndef __NUITKA_NO_ASSERT__ extern void checkGlobalConstants( void ); #endif #if defined(_WIN32) && defined(_NUITKA_EXE) #include extern const unsigned char* constant_bin; #else extern "C" const unsigned char constant_bin[]; #endif extern void UNSTREAM_INIT( void ); extern PyObject *UNSTREAM_CONSTANT( unsigned char const *buffer, Py_ssize_t size ); extern PyObject *UNSTREAM_STRING( unsigned char const *buffer, Py_ssize_t size, bool intern ); extern PyObject *UNSTREAM_CHAR( unsigned char value, bool intern ); #if PYTHON_VERSION < 300 extern PyObject *UNSTREAM_UNICODE( unsigned char const *buffer, Py_ssize_t size ); #else extern PyObject *UNSTREAM_BYTES( unsigned char const *buffer, Py_ssize_t size ); #endif extern PyObject *UNSTREAM_FLOAT( unsigned char const *buffer ); // Performance enhancements to Python types. extern void enhancePythonTypes( void ); // Setup meta path based loader if any. extern void setupMetaPathBasedLoader( void ); // Parse the command line parameters and provide it to "sys" built-in module. #if PYTHON_VERSION >= 300 typedef wchar_t ** argv_type_t; extern argv_type_t convertCommandLineParameters( int argc, char **argv ); #else typedef char ** argv_type_t; #endif extern bool setCommandLineParameters( int argc, argv_type_t argv, bool initial ); // Replace built-in functions with ones that accept compiled types too. extern void patchBuiltinModule( void ); // Replace type comparison with one that accepts compiled types too, will work // for "==" and "!=", but not for "is" checks. extern void patchTypeComparison( void ); #if PYTHON_VERSION < 300 // Initialize value for "tp_compare" default. extern void _initSlotCompare( void ); #endif #if PYTHON_VERSION >= 300 NUITKA_MAY_BE_UNUSED static PyObject *SELECT_METACLASS( PyObject *metaclass, PyObject *bases ) { CHECK_OBJECT( metaclass ); CHECK_OBJECT( bases ); if (likely( PyType_Check( metaclass ) )) { // Determine the proper metatype Py_ssize_t nbases = PyTuple_GET_SIZE( bases ); PyTypeObject *winner = (PyTypeObject *)metaclass; for ( int i = 0; i < nbases; i++ ) { PyObject *base = PyTuple_GET_ITEM( bases, i ); PyTypeObject *base_type = Py_TYPE( base ); if ( PyType_IsSubtype( winner, base_type ) ) { // Ignore if current winner is already a subtype. continue; } else if ( PyType_IsSubtype( base_type, winner ) ) { // Use if, if it's a subtype of the current winner. winner = base_type; continue; } else { PyErr_Format( PyExc_TypeError, "metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases" ); return NULL; } } if (unlikely( winner == NULL )) { return NULL; } return INCREASE_REFCOUNT( (PyObject *)winner ); } else { return INCREASE_REFCOUNT( metaclass ); } } #else NUITKA_MAY_BE_UNUSED static PyObject *SELECT_METACLASS( PyObject *bases, PyObject *metaclass_global ) { CHECK_OBJECT( bases ); PyObject *metaclass; assert( bases != Py_None ); if ( PyTuple_GET_SIZE( bases ) > 0 ) { PyObject *base = PyTuple_GET_ITEM( bases, 0 ); metaclass = PyObject_GetAttr( base, const_str_plain___class__ ); if ( metaclass == NULL ) { CLEAR_ERROR_OCCURRED(); metaclass = INCREASE_REFCOUNT( (PyObject *)Py_TYPE( base ) ); } } else if ( metaclass_global != NULL ) { metaclass = INCREASE_REFCOUNT( metaclass_global ); } else { // Default to old style class. metaclass = INCREASE_REFCOUNT( (PyObject *)&PyClass_Type ); } // Cannot fail on Python2. CHECK_OBJECT( metaclass ); return metaclass; } #endif extern PyObject *const_str_plain___name__; NUITKA_MAY_BE_UNUSED static PyObject *MODULE_NAME( PyObject *module ) { assert( PyModule_Check( module ) ); PyObject *module_dict = ((PyModuleObject *)module)->md_dict; return PyDict_GetItem( module_dict, const_str_plain___name__ ); } #if defined(_NUITKA_STANDALONE) || _NUITKA_FROZEN > 0 extern void prepareStandaloneEnvironment(); extern void restoreStandaloneEnvironment(); // Get the binary directory, translated to UTF8 or usable as a native path, // e.g. ANSI on Windows. extern char *getBinaryDirectoryUTF8Encoded(); extern char *getBinaryDirectoryHostEncoded(); #endif #if _NUITKA_STANDALONE extern void setEarlyFrozenModulesFileAttribute( void ); #endif // For making paths relative to where we got loaded from. extern PyObject *MAKE_RELATIVE_PATH( PyObject *relative ); #include NUITKA_MAY_BE_UNUSED static PyObject *MAKE_TUPLE( PyObject **elements, Py_ssize_t size ) { PyObject *result = PyTuple_New( size ); for( Py_ssize_t i = 0; i < size; i++ ) { PyTuple_SET_ITEM( result, i, INCREASE_REFCOUNT( elements[i] ) ); } return result; } // Make a deep copy of an object. extern PyObject *DEEP_COPY( PyObject *value ); // Force a garbage collection, for debugging purposes. NUITKA_MAY_BE_UNUSED static void forceGC() { PyObject_CallObject(PyObject_GetAttrString(PyImport_ImportModule("gc"), "collect"), NULL ); } #include "nuitka/helper/cells.hpp" #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/exceptions.hpp0000644000175000017500000004125212651072046023642 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_EXCEPTIONS_H__ #define __NUITKA_EXCEPTIONS_H__ // Exception helpers for generated code and compiled code helpers. // Did an error occur. NUITKA_MAY_BE_UNUSED static inline bool ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); return tstate->curexc_type != NULL; } // Get the error type occurred. NUITKA_MAY_BE_UNUSED static inline PyObject *GET_ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); return tstate->curexc_type; } // Clear error, which likely set. NUITKA_MAY_BE_UNUSED static inline void CLEAR_ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *old_type = tstate->curexc_type; PyObject *old_value = tstate->curexc_value; PyObject *old_tb = tstate->curexc_traceback; tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; Py_XDECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); } // Clear error, which is not likely set. This is about bugs from CPython, // use CLEAR_ERROR_OCCURRED is not sure. NUITKA_MAY_BE_UNUSED static inline void DROP_ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); if (unlikely( tstate->curexc_type != NULL )) { PyObject *old_type = tstate->curexc_type; PyObject *old_value = tstate->curexc_value; PyObject *old_tb = tstate->curexc_traceback; tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; Py_DECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); } } // Fetch the current error into object variables. NUITKA_MAY_BE_UNUSED static void FETCH_ERROR_OCCURRED( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_traceback) { PyThreadState *tstate = PyThreadState_GET(); *exception_type = tstate->curexc_type; *exception_value = tstate->curexc_value; *exception_traceback = (PyTracebackObject *)tstate->curexc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("FETCH_ERROR_OCCURRED:\n"); PRINT_EXCEPTION( tstate->curexc_type, tstate->curexc_value, tstate->curexc_traceback ); #endif tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; } // Fetch the current error into object variables. NUITKA_MAY_BE_UNUSED static void FETCH_ERROR_OCCURRED_UNTRACED( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_traceback) { PyThreadState *tstate = PyThreadState_GET(); *exception_type = tstate->curexc_type; *exception_value = tstate->curexc_value; *exception_traceback = (PyTracebackObject *)tstate->curexc_traceback; tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; } NUITKA_MAY_BE_UNUSED static void RESTORE_ERROR_OCCURRED( PyObject *exception_type, PyObject *exception_value, PyTracebackObject *exception_traceback ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *old_exception_type = tstate->curexc_type; PyObject *old_exception_value = tstate->curexc_value; PyObject *old_exception_traceback = tstate->curexc_traceback; tstate->curexc_type = exception_type; tstate->curexc_value = exception_value; tstate->curexc_traceback = (PyObject *)exception_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("RESTORE_ERROR_OCCURRED:\n"); PRINT_EXCEPTION( tstate->curexc_type, tstate->curexc_value, tstate->curexc_traceback ); #endif Py_XDECREF( old_exception_type ); Py_XDECREF( old_exception_value ); Py_XDECREF( old_exception_traceback ); } NUITKA_MAY_BE_UNUSED static void RESTORE_ERROR_OCCURRED_UNTRACED( PyObject *exception_type, PyObject *exception_value, PyTracebackObject *exception_traceback ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *old_exception_type = tstate->curexc_type; PyObject *old_exception_value = tstate->curexc_value; PyObject *old_exception_traceback = tstate->curexc_traceback; tstate->curexc_type = exception_type; tstate->curexc_value = exception_value; tstate->curexc_traceback = (PyObject *)exception_traceback; Py_XDECREF( old_exception_type ); Py_XDECREF( old_exception_value ); Py_XDECREF( old_exception_traceback ); } NUITKA_MAY_BE_UNUSED static inline PyTracebackObject *INCREASE_REFCOUNT( PyTracebackObject *traceback_object ) { Py_INCREF( traceback_object ); return traceback_object; } // Create a traceback for a given frame. TODO: Probably we ought to have a quick // cache for it, in case of repeated usage. NUITKA_MAY_BE_UNUSED static PyTracebackObject *MAKE_TRACEBACK( PyFrameObject *frame, int lineno ) { // assertFrameObject( frame ); PyTracebackObject *result = PyObject_GC_New( PyTracebackObject, &PyTraceBack_Type ); result->tb_next = NULL; result->tb_frame = frame; Py_INCREF( frame ); result->tb_lasti = 0; result->tb_lineno = lineno; Nuitka_GC_Track( result ); return result; } // Add a frame to an existing exception trace-back. NUITKA_MAY_BE_UNUSED static PyTracebackObject *ADD_TRACEBACK( PyTracebackObject *exception_tb, PyFrameObject *frame, int lineno ) { PyTracebackObject *traceback_new = MAKE_TRACEBACK( frame, lineno ); traceback_new->tb_next = exception_tb; return traceback_new; } #if PYTHON_VERSION < 300 extern PyObject *const_str_plain_exc_type, *const_str_plain_exc_value, *const_str_plain_exc_traceback; #endif // Helper that sets the current thread exception, releasing the current one, for // use in this file only. NUITKA_MAY_BE_UNUSED inline void SET_CURRENT_EXCEPTION( PyObject *exception_type, PyObject *exception_value, PyTracebackObject *exception_tb ) { PyThreadState *thread_state = PyThreadState_GET(); PyObject *old_type = thread_state->exc_type; PyObject *old_value = thread_state->exc_value; PyObject *old_tb = thread_state->exc_traceback; thread_state->exc_type = exception_type; thread_state->exc_value = exception_value; thread_state->exc_traceback = (PyObject *)exception_tb; #if _DEBUG_EXCEPTIONS PRINT_STRING("SET_CURRENT_EXCEPTION:\n"); PRINT_EXCEPTION( exception_type, exception_value, (PyObject *)exception_tb ); #endif Py_XDECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); #if PYTHON_VERSION < 300 // Set sys attributes in the fastest possible way. PyObject *sys_dict = thread_state->interp->sysdict; CHECK_OBJECT( sys_dict ); PyDict_SetItem( sys_dict, const_str_plain_exc_type, exception_type ? exception_type : Py_None ); PyDict_SetItem( sys_dict, const_str_plain_exc_value, exception_value ? exception_value : Py_None ); PyDict_SetItem( sys_dict, const_str_plain_exc_traceback, exception_tb ? (PyObject *)exception_tb : Py_None ); if ( exception_type ) assert( Py_REFCNT( exception_type ) >= 2 ); if ( exception_value ) assert( Py_REFCNT( exception_value ) >= 2 ); if ( exception_tb ) assert( Py_REFCNT( exception_tb ) >= 2 ); #endif } // Preserve the current exception as the frame to restore. NUITKA_MAY_BE_UNUSED static inline void PRESERVE_FRAME_EXCEPTION( PyFrameObject *frame_object ) { // Setting exception for frame if not already done. if ( frame_object->f_exc_type == NULL ) { PyThreadState *thread_state = PyThreadState_GET(); if ( thread_state->exc_type != NULL && thread_state->exc_type != Py_None ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("PRESERVE_FRAME_EXCEPTION: preserve thread exception\n"); #endif frame_object->f_exc_type = thread_state->exc_type; Py_INCREF( frame_object->f_exc_type ); frame_object->f_exc_value = thread_state->exc_value; Py_XINCREF( frame_object->f_exc_value ); frame_object->f_exc_traceback = thread_state->exc_traceback; Py_XINCREF( frame_object->f_exc_traceback ); } else { #if _DEBUG_EXCEPTIONS PRINT_STRING("PRESERVE_FRAME_EXCEPTION: no exception to preserve\n"); #endif frame_object->f_exc_type = Py_None; Py_INCREF( frame_object->f_exc_type ); frame_object->f_exc_value = NULL; frame_object->f_exc_traceback = NULL; } } #if _DEBUG_EXCEPTIONS else { PRINT_STRING("PRESERVE_FRAME_EXCEPTION: already preserving\n"); } PRINT_ITEM( (PyObject *)frame_object ); PRINT_NEW_LINE(); PRINT_EXCEPTION( frame_object->f_exc_type, frame_object->f_exc_value, frame_object->f_exc_traceback ); #endif } // Restore a previously preserved exception to the frame. NUITKA_MAY_BE_UNUSED static inline void RESTORE_FRAME_EXCEPTION( PyFrameObject *frame_object ) { if ( frame_object->f_exc_type ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("RESTORE_FRAME_EXCEPTION: restoring preserved\n"); PRINT_ITEM( (PyObject *)frame_object ); PRINT_NEW_LINE(); #endif SET_CURRENT_EXCEPTION( frame_object->f_exc_type, frame_object->f_exc_value, (PyTracebackObject *)frame_object->f_exc_traceback ); frame_object->f_exc_type = NULL; frame_object->f_exc_value = NULL; frame_object->f_exc_traceback = NULL; } #if _DEBUG_EXCEPTIONS else { PRINT_STRING("RESTORE_FRAME_EXCEPTION: nothing to restore\n"); PRINT_ITEM( (PyObject *)frame_object ); PRINT_NEW_LINE(); } #endif } // Publish an exception, erasing the values of the variables. NUITKA_MAY_BE_UNUSED static inline void PUBLISH_EXCEPTION( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("PUBLISH_EXCEPTION:\n"); #endif SET_CURRENT_EXCEPTION( *exception_type, *exception_value, *exception_tb ); *exception_type = NULL; *exception_value = NULL; *exception_tb = NULL; } // Normalize an exception. NUITKA_MAY_BE_UNUSED static inline void NORMALIZE_EXCEPTION( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("NORMALIZE_EXCEPTION:\n"); PRINT_EXCEPTION( *exception_type, *exception_value, (PyObject *)*exception_tb ); #endif if ( *exception_type != Py_None && *exception_type != NULL ) { PyErr_NormalizeException( exception_type, exception_value, (PyObject **)exception_tb ); } #if _DEBUG_EXCEPTIONS PRINT_STRING("normalized:\n"); PRINT_EXCEPTION( *exception_type, *exception_value, (PyObject *)*exception_tb ); #endif } NUITKA_MAY_BE_UNUSED static bool EXCEPTION_MATCH_GENERATOR( PyObject *exception_value ) { CHECK_OBJECT( exception_value ); // We need to check the class. if ( PyExceptionInstance_Check( exception_value ) ) { exception_value = PyExceptionInstance_Class( exception_value ); } // Lets be optimistic. If it matches, we would be wasting our time. if ( exception_value == PyExc_GeneratorExit || exception_value == PyExc_StopIteration ) { return true; } if ( PyExceptionClass_Check( exception_value ) ) { // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); int res = PyObject_IsSubclass( exception_value, PyExc_GeneratorExit ); // This function must not fail, so print the error here */ if (unlikely( res == -1 )) { PyErr_WriteUnraisable( exception_value ); } if (res == 1) return true; res = PyObject_IsSubclass( exception_value, PyExc_StopIteration ); // This function must not fail, so print the error here */ if (unlikely( res == -1 )) { PyErr_WriteUnraisable( exception_value ); } RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); return res == 1; } return false; } NUITKA_MAY_BE_UNUSED static bool EXCEPTION_MATCH_BOOL_SINGLE( PyObject *exception_value, PyObject *exception_checked ) { CHECK_OBJECT( exception_value ); CHECK_OBJECT( exception_checked ); // We need to check the class. if ( PyExceptionInstance_Check( exception_value ) ) { exception_value = PyExceptionInstance_Class( exception_value ); } // Lets be optimistic. If it matches, we would be wasting our time. if ( exception_value == exception_checked ) { return true; } if ( PyExceptionClass_Check( exception_value ) ) { // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); int res = PyObject_IsSubclass( exception_value, exception_checked ); // This function must not fail, so print the error here */ if (unlikely( res == -1 )) { PyErr_WriteUnraisable( exception_value ); } RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); return res == 1; } return false; } // This is for the actual comparison operation that is being done in the // node tree, no other code should use it. TODO: Then it's probably not // properly located here, and it could still in-line the code of // "PyErr_GivenExceptionMatches" to save on Python3 doing two tuple checks // and iterations. NUITKA_MAY_BE_UNUSED static inline int EXCEPTION_MATCH_BOOL( PyObject *exception_value, PyObject *exception_checked ) { CHECK_OBJECT( exception_value ); CHECK_OBJECT( exception_checked ); #if PYTHON_VERSION >= 300 if ( PyTuple_Check( exception_checked )) { Py_ssize_t length = PyTuple_Size( exception_checked ); for ( Py_ssize_t i = 0; i < length; i += 1 ) { PyObject *element = PyTuple_GET_ITEM( exception_checked, i ); if (unlikely( !PyExceptionClass_Check( element ) )) { PyErr_Format( PyExc_TypeError, "catching classes that do not inherit from BaseException is not allowed" ); return -1; } } } else if (unlikely( !PyExceptionClass_Check( exception_checked ) )) { PyErr_Format( PyExc_TypeError, "catching classes that do not inherit from BaseException is not allowed" ); return -1; } #endif return PyErr_GivenExceptionMatches( exception_value, exception_checked ); } #if PYTHON_VERSION >= 300 // Attach the exception context if necessary. NUITKA_MAY_BE_UNUSED static inline void ADD_EXCEPTION_CONTEXT( PyObject **exception_type, PyObject **exception_value ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *context = tstate->exc_value; if ( context != NULL ) { NORMALIZE_EXCEPTION( exception_type, exception_value, NULL ); Py_INCREF( context ); PyException_SetContext( *exception_value, context ); } } #endif // Special helper that checks for StopIteration and if so clears it, only // indicating if it was set. NUITKA_MAY_BE_UNUSED static bool CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED( void ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { return true; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) ) { CLEAR_ERROR_OCCURRED(); return true; } else { return false; } } // Special helper that checks for StopIteration and if so clears it, only // indicating if it was set. NUITKA_MAY_BE_UNUSED static bool CHECK_AND_CLEAR_ATTRIBUTE_ERROR_OCCURRED( void ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { return true; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); return true; } else { return false; } } #endif Nuitka-0.5.18.1/nuitka/build/include/nuitka/threading.hpp0000644000175000017500000000407312651071040023417 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_THREADING_H__ #define __NUITKA_THREADING_H__ #if PYTHON_VERSION < 300 // We share this with CPython bytecode main loop. PyAPI_DATA(volatile int) _Py_Ticker; #else extern volatile int _Py_Ticker; #define _Py_CheckInterval 20 #endif NUITKA_MAY_BE_UNUSED static inline bool CONSIDER_THREADING( void ) { // Decrease ticker if ( --_Py_Ticker < 0 ) { _Py_Ticker = _Py_CheckInterval; int res = Py_MakePendingCalls(); if (unlikely( res < 0 && ERROR_OCCURRED() )) { return false; } PyThreadState *tstate = PyThreadState_GET(); assert( tstate ); if ( PyEval_ThreadsInitialized() ) { // Release and acquire the GIL, it's very inefficient, because we // don't even know if it makes sense to do it. A controlling thread // should be used to determine if it's needed at all. PyEval_SaveThread(); PyEval_AcquireThread( tstate ); } if (unlikely( tstate->async_exc != NULL )) { PyObject *async_exc = INCREASE_REFCOUNT( tstate->async_exc ); tstate->async_exc = NULL; RESTORE_ERROR_OCCURRED( async_exc, NULL, NULL ); return false; } } return true; } #endif Nuitka-0.5.18.1/nuitka/build/SconsInterface.py0000644000175000017500000001356112651071040021305 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Scons interface. Interaction with scons. Find the binary, and run it with a set of given options. """ import contextlib import os import subprocess import sys from nuitka import Options, Tracing from nuitka.PythonVersions import python_version from nuitka.utils import Utils def getSconsDataPath(): return Utils.dirname(__file__) def getSconsInlinePath(): return Utils.joinpath(getSconsDataPath(), "inline_copy") def getSconsBinaryCall(): """ Return a way to execute Scons. Using potentially in-line copy if no system Scons is available or if we are on Windows. """ if Utils.getOS() != "Windows" and Utils.isFile("/usr/bin/scons"): return ["/usr/bin/scons"] else: return [ getPython2ExePath(), Utils.joinpath(getSconsInlinePath(), "bin", "scons.py") ] def _getPython2ExePathWindows(): # Shortcuts for the default installation directories, to avoid going to # registry at all unless necessary. Any Python2 will do for Scons, so it # can be avoided. if os.path.isfile(r"c:\Python27\python.exe"): return r"c:\Python27\python.exe" elif os.path.isfile(r"c:\Python26\python.exe"): return r"c:\Python26\python.exe" # Windows only code, pylint: disable=E0602,F0401,I0021 try: import _winreg as winreg except ImportError: import winreg # lint:ok for search in ("2.7", "2.6"): for hkey_branch in (winreg.HKEY_LOCAL_MACHINE, winreg.HKEY_CURRENT_USER): for arch_key in 0, winreg.KEY_WOW64_32KEY, winreg.KEY_WOW64_64KEY: try: key = winreg.OpenKey( hkey_branch, r"SOFTWARE\Python\PythonCore\%s\InstallPath" % search, 0, winreg.KEY_READ | arch_key ) return Utils.joinpath( winreg.QueryValue(key, ""), "python.exe" ) except WindowsError: # @UndefinedVariable pass def getPython2ExePath(): """ Find a way to call Python2. Scons needs it.""" if python_version < 300: return sys.executable elif Utils.getOS() == "Windows": python_exe = _getPython2ExePathWindows() if python_exe is not None: return python_exe else: sys.exit("""\ Error, need to find Python2 executable under C:\\Python26 or \ C:\\Python27 to execute scons which is not Python3 compatible.""") elif Utils.isFile("/usr/bin/python2.7"): return "/usr/bin/python2.7" elif Utils.isFile("/usr/bin/python2.6"): return "/usr/bin/python2.6" elif Utils.isFile("/usr/bin/python2"): return "/usr/bin/python2" else: return "python" @contextlib.contextmanager def setupSconsEnvironment(): # For the scons file to find the static C++ files and include path. The # scons file is unable to use __file__ for the task. os.environ["NUITKA_SCONS"] = getSconsDataPath() if Utils.getOS() == "Windows": # On Windows this Scons variable must be set by us. os.environ["SCONS_LIB_DIR"] = Utils.joinpath( getSconsInlinePath(), "lib", "scons-2.3.2" ) # Remove environment variables that can only harm if we have to switch # major Python versions, these cannot help Python2 to execute scons, this # is a bit of noise, but helpful. if python_version >= 300: if "PYTHONPATH" in os.environ: old_pythonpath = os.environ["PYTHONPATH"] del os.environ["PYTHONPATH"] else: old_pythonpath = None if "PYTHONHOME" in os.environ: old_pythonhome = os.environ["PYTHONHOME"] del os.environ["PYTHONHOME"] else: old_pythonhome = None yield if python_version >= 300: if old_pythonpath is not None: os.environ["PYTHONPATH"] = old_pythonpath if old_pythonhome is not None: os.environ["PYTHONHOME"] = old_pythonhome def buildSconsCommand(quiet, options): scons_command = getSconsBinaryCall() if quiet: scons_command.append("--quiet") scons_command += [ # The scons file "-f", Utils.joinpath(getSconsDataPath(), "SingleExe.scons"), # Parallel compilation. "--jobs", str(Options.getJobLimit()), # Do not warn about deprecation from Scons "--warn=no-deprecated", # Don't load "site_scons" at all. "--no-site-dir", ] if Options.isShowScons(): scons_command.append("--debug=explain") # Option values to provide to scons. Find these in the caller. for key, value in options.items(): scons_command += [key + '=' + value] return scons_command def runScons(options, quiet): with setupSconsEnvironment(): scons_command = buildSconsCommand(quiet, options) if Options.isShowScons(): Tracing.printLine("Scons command:", ' '.join(scons_command)) return subprocess.call(scons_command, shell = False) == 0 Nuitka-0.5.18.1/nuitka/build/static_src/0000755000175000017500000000000012651072347020170 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/static_src/InspectPatcher.cpp0000644000175000017500000002630212651071040023600 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // This is responsible for updating parts of CPython to better work with Nuitka // by replacing CPython implementations with enhanced versions. #include "nuitka/prelude.hpp" #if PYTHON_VERSION >= 300 extern PyObject *const_str_plain_inspect; extern PyObject *const_str_plain_site; extern PyObject *const_int_0; static PyObject *module_inspect; #if PYTHON_VERSION >= 350 static PyObject *module_types; extern PyObject *const_str_plain_types; #endif static char *kwlist[] = { (char *)"object", NULL }; static PyObject *old_getgeneratorstate = NULL; static PyObject *_inspect_getgeneratorstate_replacement( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *object; if ( !PyArg_ParseTupleAndKeywords( args, kwds, "O:getgeneratorstate", kwlist, &object, NULL )) { return NULL; } if ( Nuitka_Generator_Check( object ) ) { Nuitka_GeneratorObject *generator = (Nuitka_GeneratorObject *)object; if ( generator->m_running ) { return PyObject_GetAttrString( module_inspect, "GEN_RUNNING" ); } else if ( generator->m_status == status_Finished ) { return PyObject_GetAttrString( module_inspect, "GEN_CLOSED" ); } else if ( generator->m_status == status_Unused ) { return PyObject_GetAttrString( module_inspect, "GEN_CREATED" ); } else { return PyObject_GetAttrString( module_inspect, "GEN_SUSPENDED" ); } } else { return old_getgeneratorstate->ob_type->tp_call( old_getgeneratorstate, args, kwds ); } } #if PYTHON_VERSION >= 350 static PyObject *old_getcoroutinestate = NULL; static PyObject *_inspect_getcoroutinestate_replacement( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *object; if ( !PyArg_ParseTupleAndKeywords( args, kwds, "O:getcoroutinestate", kwlist, &object, NULL )) { return NULL; } if ( Nuitka_Coroutine_Check( object ) ) { Nuitka_CoroutineObject *coroutine = (Nuitka_CoroutineObject *)object; if ( coroutine->m_running ) { return PyObject_GetAttrString( module_inspect, "CORO_RUNNING" ); } else if ( coroutine->m_status == status_Finished ) { return PyObject_GetAttrString( module_inspect, "CORO_CLOSED" ); } else if ( coroutine->m_status == status_Unused ) { return PyObject_GetAttrString( module_inspect, "CORO_CREATED" ); } else { return PyObject_GetAttrString( module_inspect, "CORO_SUSPENDED" ); } } else { return old_getcoroutinestate->ob_type->tp_call( old_getcoroutinestate, args, kwds ); } } static PyObject *old_types_coroutine = NULL; static char *kwlist_func[] = { (char *)"func", NULL }; static PyObject *_types_coroutine_replacement( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *func; if ( !PyArg_ParseTupleAndKeywords( args, kwds, "O:getcoroutinestate", kwlist_func, &func, NULL )) { return NULL; } if ( Nuitka_Function_Check( func ) ) { Nuitka_FunctionObject *function = (Nuitka_FunctionObject *)func; if ( function->m_code_object->co_flags & CO_GENERATOR ) { function->m_code_object->co_flags |= 0x100; } } return old_types_coroutine->ob_type->tp_call( old_types_coroutine, args, kwds ); } #endif #endif #if PYTHON_VERSION >= 300 static PyMethodDef _method_def_inspect_getgeneratorstate_replacement = { "getgeneratorstate", (PyCFunction)_inspect_getgeneratorstate_replacement, METH_VARARGS | METH_KEYWORDS, NULL }; #if PYTHON_VERSION >= 350 static PyMethodDef _method_def_inspect_getcoroutinestate_replacement = { "getcoroutinestate", (PyCFunction)_inspect_getcoroutinestate_replacement, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_types_coroutine_replacement = { "coroutine", (PyCFunction)_types_coroutine_replacement, METH_VARARGS | METH_KEYWORDS, NULL }; #endif // Replace inspect functions with ones that accept compiled types too. static void patchInspectModule( void ) { #if PYTHON_VERSION >= 300 #ifdef _NUITKA_EXE // May need to import the "site" module, because otherwise the patching can // fail with it being unable to load it. if ( Py_NoSiteFlag == 0 ) { PyObject *site_module = IMPORT_MODULE( const_str_plain_site, Py_None, Py_None, const_tuple_empty, const_int_0 ); if ( site_module == NULL ) { // Ignore ImportError, site is not a must. CLEAR_ERROR_OCCURRED(); } } #endif module_inspect = IMPORT_MODULE( const_str_plain_inspect, Py_None, Py_None, const_tuple_empty, const_int_0 ); if ( module_inspect == NULL ) { PyErr_PrintEx( 0 ); Py_Exit( 1 ); } CHECK_OBJECT( module_inspect ); // Patch "inspect.getgeneratorstate" unless it is already patched. old_getgeneratorstate = PyObject_GetAttrString( module_inspect, "getgeneratorstate" ); CHECK_OBJECT( old_getgeneratorstate ); if ( PyFunction_Check( old_getgeneratorstate ) ) { PyObject *inspect_getgeneratorstate_replacement = PyCFunction_New( &_method_def_inspect_getgeneratorstate_replacement, NULL ); CHECK_OBJECT( inspect_getgeneratorstate_replacement ); PyObject_SetAttrString( module_inspect, "getgeneratorstate", inspect_getgeneratorstate_replacement ); } #if PYTHON_VERSION >= 350 // Patch "inspect.getcoroutinestate" unless it is already patched. old_getcoroutinestate = PyObject_GetAttrString( module_inspect, "getcoroutinestate" ); CHECK_OBJECT( old_getcoroutinestate ); if ( PyFunction_Check( old_getcoroutinestate ) ) { PyObject *inspect_getcoroutinestate_replacement = PyCFunction_New( &_method_def_inspect_getcoroutinestate_replacement, NULL ); CHECK_OBJECT( inspect_getcoroutinestate_replacement ); PyObject_SetAttrString( module_inspect, "getcoroutinestate", inspect_getcoroutinestate_replacement ); } module_types = IMPORT_MODULE( const_str_plain_types, Py_None, Py_None, const_tuple_empty, const_int_0 ); if ( module_types == NULL ) { PyErr_PrintEx( 0 ); Py_Exit( 1 ); } CHECK_OBJECT( module_types ); // Patch "types.coroutine" unless it is already patched. old_types_coroutine = PyObject_GetAttrString( module_types, "coroutine" ); CHECK_OBJECT( old_types_coroutine ); if ( PyFunction_Check( old_types_coroutine ) ) { PyObject *types_coroutine_replacement = PyCFunction_New( &_method_def_types_coroutine_replacement, NULL ); CHECK_OBJECT( types_coroutine_replacement ); PyObject_SetAttrString( module_types, "coroutine", types_coroutine_replacement ); } static char const *wrapper_enhancement_code = "\n\ import types\n\ _old_GeneratorWrapper = types._GeneratorWrapper\n\ class GeneratorWrapperEnhanced(_old_GeneratorWrapper):\n\ def __init__(self, gen):\n\ _old_GeneratorWrapper.__init__(self, gen)\n\ \n\ if hasattr(gen, 'gi_code'):\n\ if gen.gi_code.co_flags & 0x0020:\n\ self._GeneratorWrapper__isgen = True\n\ \n\ types._GeneratorWrapper = GeneratorWrapperEnhanced\ "; PyObject *wrapper_enhencement_codeobject = Py_CompileString( wrapper_enhancement_code, "", Py_file_input ); CHECK_OBJECT( wrapper_enhencement_codeobject ); PyImport_ExecCodeModuleEx( "_types_patch", wrapper_enhencement_codeobject, "" ); #endif #endif } #endif extern int Nuitka_IsInstance( PyObject *inst, PyObject *cls ); extern PyObject *original_isinstance; static PyObject *_builtin_isinstance_replacement( PyObject *self, PyObject *args ) { PyObject *inst, *cls; if (unlikely( PyArg_UnpackTuple(args, "isinstance", 2, 2, &inst, &cls) == 0 )) { return NULL; } int res = Nuitka_IsInstance( inst, cls ); if (unlikely( res < 0 )) { return NULL; } return PyBool_FromLong( res ); } static PyMethodDef _method_def_builtin_isinstance_replacement = { "isinstance", (PyCFunction)_builtin_isinstance_replacement, METH_VARARGS, NULL }; extern PyModuleObject *builtin_module; void patchBuiltinModule() { #if defined(_NUITKA_MODULE) static bool init_done = false; if (init_done == true) return; init_done = true; #endif CHECK_OBJECT( (PyObject *)builtin_module ); // Patch "inspect.isinstance" unless it is already patched. original_isinstance = PyObject_GetAttrString( (PyObject *)builtin_module, "isinstance" ); CHECK_OBJECT( original_isinstance ); // Copy the doc attribute over, needed for "inspect.signature" at least. if ( PyCFunction_Check( original_isinstance )) { _method_def_builtin_isinstance_replacement.ml_doc = ((PyCFunctionObject *)original_isinstance)->m_ml->ml_doc; } PyObject *builtin_isinstance_replacement = PyCFunction_New( &_method_def_builtin_isinstance_replacement, NULL ); CHECK_OBJECT( builtin_isinstance_replacement ); PyObject_SetAttrString( (PyObject *)builtin_module, "isinstance", builtin_isinstance_replacement ); #if PYTHON_VERSION >= 300 patchInspectModule(); #endif } static richcmpfunc original_PyType_tp_richcompare = NULL; static PyObject *Nuitka_type_tp_richcompare( PyObject *a, PyObject *b, int op ) { if (likely( op == Py_EQ || op == Py_NE )) { if ( a == (PyObject *)&Nuitka_Function_Type ) { a = (PyObject *)&PyFunction_Type; } else if ( a == (PyObject *)&Nuitka_Method_Type ) { a = (PyObject *)&PyMethod_Type; } else if ( a == (PyObject *)&Nuitka_Generator_Type ) { a = (PyObject *)&PyGen_Type; } if ( b == (PyObject *)&Nuitka_Function_Type ) { b = (PyObject *)&PyFunction_Type; } else if ( b == (PyObject *)&Nuitka_Method_Type ) { b = (PyObject *)&PyMethod_Type; } else if ( b == (PyObject *)&Nuitka_Generator_Type ) { b = (PyObject *)&PyGen_Type; } } CHECK_OBJECT( a ); CHECK_OBJECT( b ); assert( original_PyType_tp_richcompare ); return original_PyType_tp_richcompare( a, b, op ); } void patchTypeComparison() { if ( original_PyType_tp_richcompare == NULL ) { original_PyType_tp_richcompare = PyType_Type.tp_richcompare; PyType_Type.tp_richcompare = Nuitka_type_tp_richcompare; } } Nuitka-0.5.18.1/nuitka/build/static_src/x64_ucontext_src/0000755000175000017500000000000012651072347023411 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/static_src/x64_ucontext_src/swapfiber.S0000644000175000017500000000452312651071040025510 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch "swapFiber" for x64 #define oRBP 120 #define oRSP 160 #define oRBX 128 #define oR8 40 #define oR9 48 #define oR12 72 #define oR13 80 #define oR14 88 #define oR15 96 #define oRDI 104 #define oRSI 112 #define oRDX 136 #define oRCX 152 #define oRIP 168 #define oFPREGS 224 #define oFPREGSMEM 424 .globl _swapFiber _swapFiber: // Save the preserved used for passing args and the return address. movq %rbx, oRBX(%rdi) movq %rbp, oRBP(%rdi) movq %r12, oR12(%rdi) movq %r13, oR13(%rdi) movq %r14, oR14(%rdi) movq %r15, oR15(%rdi) movq %rdi, oRDI(%rdi) movq %rsi, oRSI(%rdi) movq %rdx, oRDX(%rdi) movq %rcx, oRCX(%rdi) movq %r8, oR8(%rdi) movq %r9, oR9(%rdi) movq (%rsp), %rcx movq %rcx, oRIP(%rdi) // Make room for return address. leaq 8(%rsp), %rcx movq %rcx, oRSP(%rdi) // Separate floating-point register content memory leaq oFPREGSMEM(%rdi), %rcx movq %rcx, oFPREGS(%rdi) // Load the new stack pointer and the preserved registers. movq oRSP(%rsi), %rsp movq oRBX(%rsi), %rbx movq oRBP(%rsi), %rbp movq oR12(%rsi), %r12 movq oR13(%rsi), %r13 movq oR14(%rsi), %r14 movq oR15(%rsi), %r15 // We should return to the address provided, put it on top of stack. movq oRIP(%rsi), %rcx pushq %rcx // Setup registers for arguments. movq oRDI(%rsi), %rdi movq oRDX(%rsi), %rdx movq oRCX(%rsi), %rcx movq oR8(%rsi), %r8 movq oR9(%rsi), %r9 // Setup %rsi. movq oRSI(%rsi), %rsi ret Nuitka-0.5.18.1/nuitka/build/static_src/x64_ucontext_src/fibers_x64.cpp0000644000175000017500000000422512651071040026060 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for x64. #include "nuitka/prelude.hpp" #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { to->f_context.uc_stack.ss_sp = NULL; to->f_context.uc_link = NULL; to->start_stack = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { #ifdef _NUITKA_MAKECONTEXT_INTS int ar[2]; memcpy( &ar[0], &arg, sizeof(arg) ); #endif int res = getcontext( &to->f_context ); if (unlikely( res != 0 )) { return 1; } to->f_context.uc_stack.ss_size = STACK_SIZE; to->f_context.uc_stack.ss_sp = last_stack ? last_stack : malloc( STACK_SIZE ); to->start_stack = to->f_context.uc_stack.ss_sp; to->f_context.uc_link = NULL; last_stack = NULL; #ifdef _NUITKA_MAKECONTEXT_INTS makecontext( &to->f_context, (void (*)())code, 2, ar[0], ar[1] ); #else makecontext( &to->f_context, (void (*)())code, 1, (unsigned long)arg ); #endif return 0; } void _releaseFiber( Fiber *to ) { if ( to->start_stack != NULL ) { if ( last_stack == NULL ) { last_stack = to->start_stack; } else { free( to->start_stack ); } to->start_stack = NULL; } } Nuitka-0.5.18.1/nuitka/build/static_src/CompiledFunctionType.cpp0000644000175000017500000020324312651072046025000 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.hpp" #include "nuitka/compiled_method.hpp" // Needed for offsetof #include // tp_descr_get slot, bind a function to an object. static PyObject *Nuitka_Function_descr_get( PyObject *function, PyObject *object, PyObject *klass ) { assert( Nuitka_Function_Check( function ) ); #if PYTHON_VERSION >= 300 if ( object == NULL || object == Py_None ) { return INCREASE_REFCOUNT( function ); } #endif return Nuitka_Method_New( (Nuitka_FunctionObject *)function, object == Py_None ? NULL : object, klass ); } // tp_repr slot, decide how compiled function shall be output to "repr" built-in static PyObject *Nuitka_Function_tp_repr( Nuitka_FunctionObject *function ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( #else return PyUnicode_FromFormat( #endif "", #if PYTHON_VERSION < 330 Nuitka_String_AsString( function->m_name ), #else Nuitka_String_AsString( function->m_qualname ), #endif function ); } static PyObject *Nuitka_Function_tp_call( Nuitka_FunctionObject *function, PyObject *tuple_args, PyObject *kw ) { CHECK_OBJECT( tuple_args ); assert( PyTuple_Check( tuple_args ) ); if ( kw == NULL ) { PyObject **args = &PyTuple_GET_ITEM( tuple_args, 0 ); Py_ssize_t args_size = PyTuple_GET_SIZE( tuple_args ); if ( function->m_args_simple && args_size == function->m_args_positional_count ) { for( Py_ssize_t i = 0; i < args_size; i++ ) { Py_INCREF( args[ i ] ); } return function->m_c_code( function, args ); } else if ( function->m_args_simple && args_size + function->m_defaults_given == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memcpy( python_pars, args, args_size * sizeof(PyObject *) ); memcpy( python_pars + args_size, &PyTuple_GET_ITEM( function->m_defaults, 0 ), function->m_defaults_given * sizeof(PyObject *) ); for( Py_ssize_t i = 0; i < function->m_args_overall_count; i++ ) { Py_INCREF( python_pars[ i ] ); } return function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsPos( function, python_pars, args, args_size )) { return function->m_c_code( function, python_pars ); } else { return NULL; } } } else { assert( PyDict_Check( kw ) ); return Nuitka_CallFunctionPosArgsKwArgs( function, &PyTuple_GET_ITEM( tuple_args, 0 ), PyTuple_GET_SIZE( tuple_args ), kw ); } } static long Nuitka_Function_tp_traverse( Nuitka_FunctionObject *function, visitproc visit, void *arg ) { // TODO: Identify the impact of not visiting other owned objects. It appears // to be mostly harmless, as these are strings. Py_VISIT( function->m_dict ); if ( function->m_closure ) { for( Py_ssize_t i = 0; i < function->m_closure_given; i++ ) { Py_VISIT( function->m_closure[i] ); } } return 0; } static long Nuitka_Function_tp_hash( Nuitka_FunctionObject *function ) { return function->m_counter; } static PyObject *Nuitka_Function_get_name( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( object->m_name ); } static int Nuitka_Function_set_name( Nuitka_FunctionObject *object, PyObject *value ) { #if PYTHON_VERSION < 300 if (unlikely( value == NULL || PyString_Check( value ) == 0 )) #else if (unlikely( value == NULL || PyUnicode_Check( value ) == 0 )) #endif { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *old = object->m_name; object->m_name = INCREASE_REFCOUNT( value ); Py_DECREF( old ); return 0; } #if PYTHON_VERSION >= 330 static PyObject *Nuitka_Function_get_qualname( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( object->m_qualname ); } static int Nuitka_Function_set_qualname( Nuitka_FunctionObject *object, PyObject *value ) { if (unlikely( value == NULL || PyUnicode_Check( value ) == 0 )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *old = object->m_qualname; object->m_qualname = INCREASE_REFCOUNT( value ); Py_DECREF( old ); return 0; } #endif static PyObject *Nuitka_Function_get_doc( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( object->m_doc ); } static int Nuitka_Function_set_doc( Nuitka_FunctionObject *object, PyObject *value ) { PyObject *old = object->m_doc; if ( value == NULL ) { value = Py_None; } object->m_doc = value; Py_INCREF( value ); Py_DECREF( old ); return 0; } static PyObject *Nuitka_Function_get_dict( Nuitka_FunctionObject *object ) { if ( object->m_dict == NULL ) { object->m_dict = PyDict_New(); } return INCREASE_REFCOUNT( object->m_dict ); } static int Nuitka_Function_set_dict( Nuitka_FunctionObject *object, PyObject *value ) { if (unlikely( value == NULL )) { PyErr_Format( PyExc_TypeError, "function's dictionary may not be deleted"); return -1; } if (likely( PyDict_Check( value ) )) { PyObject *old = object->m_dict; object->m_dict = INCREASE_REFCOUNT( value ); Py_XDECREF( old ); return 0; } else { PyErr_SetString( PyExc_TypeError, "setting function's dictionary to a non-dict" ); return -1; } } static PyObject *Nuitka_Function_get_code( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( (PyObject *)object->m_code_object ); } static int Nuitka_Function_set_code( Nuitka_FunctionObject *object, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "__code__ is not writable in Nuitka" ); return -1; } static PyObject *Nuitka_Function_get_closure( Nuitka_FunctionObject *object ) { if ( object->m_closure ) { PyObject *result = PyTuple_New( object->m_closure_given ); for( Py_ssize_t i = 0; i < object->m_closure_given; i++ ) { PyTuple_SET_ITEM( result, i, (PyObject *)object->m_closure[i] ); Py_INCREF( (PyObject *)object->m_closure[i] ); } return result; } else { Py_INCREF( Py_None ); return Py_None; } } static int Nuitka_Function_set_closure( Nuitka_FunctionObject *object, PyObject *value ) { PyErr_Format( #if PYTHON_VERSION < 300 PyExc_TypeError, #else PyExc_AttributeError, #endif "readonly attribute" ); return -1; } static PyObject *Nuitka_Function_get_defaults( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( (PyObject *)object->m_defaults ); } static void onUpdatedDefaultsValue( Nuitka_FunctionObject *function ) { if ( function->m_defaults == Py_None ) { function->m_defaults_given = 0; } else { function->m_defaults_given = PyTuple_GET_SIZE( function->m_defaults ); } } static int Nuitka_Function_set_defaults( Nuitka_FunctionObject *object, PyObject *value ) { if ( value == NULL ) { value = Py_None; } if (unlikely( value != Py_None && PyTuple_Check( value ) == false )) { PyErr_Format( PyExc_TypeError, "__defaults__ must be set to a tuple object" ); return -1; } if ( object->m_defaults == Py_None && value != Py_None ) { PyErr_Format( PyExc_TypeError, "Nuitka doesn't support __defaults__ size changes" ); return -1; } if ( object->m_defaults != Py_None && ( value == Py_None || PyTuple_Size( object->m_defaults ) != PyTuple_Size( value ) ) ) { PyErr_Format( PyExc_TypeError, "Nuitka doesn't support __defaults__ size changes" ); return -1; } PyObject *old = object->m_defaults; object->m_defaults = INCREASE_REFCOUNT( value ); Py_DECREF( old ); onUpdatedDefaultsValue( object ); return 0; } #if PYTHON_VERSION >= 300 static PyObject *Nuitka_Function_get_kwdefaults( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( (PyObject *)object->m_kwdefaults ); } static int Nuitka_Function_set_kwdefaults( Nuitka_FunctionObject *object, PyObject *value ) { if ( value == NULL ) { value = Py_None; } if (unlikely( value != Py_None && PyDict_Check( value ) == false )) { PyErr_Format( PyExc_TypeError, "__kwdefaults__ must be set to a dict object" ); return -1; } PyObject *old = object->m_kwdefaults; object->m_kwdefaults = INCREASE_REFCOUNT( value ); Py_DECREF( old ); return 0; } static PyObject *Nuitka_Function_get_annotations( Nuitka_FunctionObject *object ) { return INCREASE_REFCOUNT( (PyObject *)object->m_annotations ); } static int Nuitka_Function_set_annotations( Nuitka_FunctionObject *object, PyObject *value ) { // CPython silently converts None to empty dictionary. if ( value == Py_None || value == NULL ) { value = PyDict_New(); } if (unlikely( PyDict_Check( value ) == false )) { PyErr_Format( PyExc_TypeError, "__annotations__ must be set to a dict object" ); return -1; } PyObject *old = object->m_annotations; object->m_annotations = INCREASE_REFCOUNT( value ); Py_XDECREF( old ); return 0; } #endif static int Nuitka_Function_set_globals( Nuitka_FunctionObject *function, PyObject *value ) { PyErr_Format( PyExc_TypeError, "readonly attribute" ); return -1; } static PyObject *Nuitka_Function_get_globals( Nuitka_FunctionObject *function ) { return INCREASE_REFCOUNT( PyModule_GetDict( function->m_module ) ); } extern PyObject *const_str_plain___module__; static int Nuitka_Function_set_module( Nuitka_FunctionObject *object, PyObject *value ) { if ( object->m_dict == NULL ) { object->m_dict = PyDict_New(); } if ( value == NULL ) { value = Py_None; } return PyDict_SetItem( object->m_dict, const_str_plain___module__, value ); } static PyObject *Nuitka_Function_get_module( Nuitka_FunctionObject *object ) { if ( object->m_dict ) { PyObject *result = PyDict_GetItem( object->m_dict, const_str_plain___module__ ); if ( result != NULL ) { return INCREASE_REFCOUNT( result ); } } return INCREASE_REFCOUNT( MODULE_NAME( object->m_module ) ); } static PyGetSetDef Nuitka_Function_getset[] = { #if PYTHON_VERSION >= 330 { (char *)"__qualname__", (getter)Nuitka_Function_get_qualname, (setter)Nuitka_Function_set_qualname, NULL }, #endif #if PYTHON_VERSION < 300 { (char *)"func_name", (getter)Nuitka_Function_get_name, (setter)Nuitka_Function_set_name, NULL }, #endif { (char *)"__name__" , (getter)Nuitka_Function_get_name, (setter)Nuitka_Function_set_name, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_doc", (getter)Nuitka_Function_get_doc, (setter)Nuitka_Function_set_doc, NULL }, #endif { (char *)"__doc__" , (getter)Nuitka_Function_get_doc, (setter)Nuitka_Function_set_doc, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_dict", (getter)Nuitka_Function_get_dict, (setter)Nuitka_Function_set_dict, NULL }, #endif { (char *)"__dict__", (getter)Nuitka_Function_get_dict, (setter)Nuitka_Function_set_dict, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_code", (getter)Nuitka_Function_get_code, (setter)Nuitka_Function_set_code, NULL }, #endif { (char *)"__code__", (getter)Nuitka_Function_get_code, (setter)Nuitka_Function_set_code, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_defaults", (getter)Nuitka_Function_get_defaults, (setter)Nuitka_Function_set_defaults, NULL }, #endif { (char *)"__defaults__", (getter)Nuitka_Function_get_defaults, (setter)Nuitka_Function_set_defaults, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_globals", (getter)Nuitka_Function_get_globals, (setter)Nuitka_Function_set_globals, NULL }, #endif { (char *)"__closure__", (getter)Nuitka_Function_get_closure, (setter)Nuitka_Function_set_closure, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_closure", (getter)Nuitka_Function_get_closure, (setter)Nuitka_Function_set_closure, NULL }, #endif { (char *)"__globals__", (getter)Nuitka_Function_get_globals, (setter)Nuitka_Function_set_globals, NULL }, { (char *)"__module__", (getter)Nuitka_Function_get_module, (setter)Nuitka_Function_set_module, NULL }, #if PYTHON_VERSION >= 300 { (char *)"__kwdefaults__", (getter)Nuitka_Function_get_kwdefaults, (setter)Nuitka_Function_set_kwdefaults, NULL }, { (char *)"__annotations__", (getter)Nuitka_Function_get_annotations, (setter)Nuitka_Function_set_annotations, NULL }, #endif { NULL } }; static PyObject *Nuitka_Function_reduce( Nuitka_FunctionObject *function ) { #if PYTHON_VERSION < 330 return INCREASE_REFCOUNT( function->m_name ); #else return INCREASE_REFCOUNT( function->m_qualname ); #endif } static PyMethodDef Nuitka_Function_methods[] = { { "__reduce__", (PyCFunction)Nuitka_Function_reduce, METH_NOARGS, NULL }, { NULL } }; static void Nuitka_Function_tp_dealloc( Nuitka_FunctionObject *function ) { #ifndef __NUITKA_NO_ASSERT__ // Save the current exception, if any, we must to not corrupt it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); #endif Nuitka_GC_UnTrack( function ); if ( function->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)function ); } Py_DECREF( function->m_name ); #if PYTHON_VERSION >= 330 Py_DECREF( function->m_qualname ); #endif // These may actually re-surrect the object, not? Py_XDECREF( function->m_dict ); Py_DECREF( function->m_defaults ); Py_DECREF( function->m_doc ); #if PYTHON_VERSION >= 300 Py_DECREF( function->m_kwdefaults ); Py_DECREF( function->m_annotations ); #endif if ( function->m_closure ) { for( Py_ssize_t i = 0; i < function->m_closure_given; i++ ) { Py_DECREF( function->m_closure[i] ); } free( function->m_closure ); } PyObject_GC_Del( function ); #ifndef __NUITKA_NO_ASSERT__ PyThreadState *tstate = PyThreadState_GET(); assert( tstate->curexc_type == save_exception_type ); assert( tstate->curexc_value == save_exception_value ); assert( (PyTracebackObject *)tstate->curexc_traceback == save_exception_tb ); #endif } static const long tp_flags = Py_TPFLAGS_DEFAULT | #if PYTHON_VERSION < 300 Py_TPFLAGS_HAVE_WEAKREFS | #endif Py_TPFLAGS_HAVE_GC; PyTypeObject Nuitka_Function_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_function", /* tp_name */ sizeof(Nuitka_FunctionObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_Function_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)Nuitka_Function_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ (hashfunc)Nuitka_Function_tp_hash, /* tp_hash */ (ternaryfunc)Nuitka_Function_tp_call, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ tp_flags, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Function_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof( Nuitka_FunctionObject, m_weakrefs ), /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Nuitka_Function_methods, /* tp_methods */ 0, /* tp_members */ Nuitka_Function_getset, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ Nuitka_Function_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */ offsetof( Nuitka_FunctionObject, m_dict ), /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0 /* tp_version_tag */ #if PYTHON_VERSION >= 340 ,0 /* tp_finalizer */ #endif }; #if PYTHON_VERSION < 300 static inline PyObject *make_compiled_function( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ) #elif PYTHON_VERSION < 330 static inline PyObject *make_compiled_function( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given) #else static inline PyObject *make_compiled_function( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ) #endif { Nuitka_FunctionObject *result = PyObject_GC_New( Nuitka_FunctionObject, &Nuitka_Function_Type ); assert( result ); result->m_c_code = c_code; result->m_name = INCREASE_REFCOUNT( name ); #if PYTHON_VERSION >= 330 result->m_qualname = INCREASE_REFCOUNT( qualname ? qualname : name ); #endif if ( defaults == NULL ) { defaults = INCREASE_REFCOUNT( Py_None ); } CHECK_OBJECT( defaults ); assert( defaults == Py_None || ( PyTuple_Check( defaults ) && PyTuple_Size( defaults ) > 0 ) ); result->m_defaults = defaults; onUpdatedDefaultsValue( result ); #if PYTHON_VERSION >= 300 if ( kwdefaults == NULL ) { kwdefaults = INCREASE_REFCOUNT( Py_None ); } assert( kwdefaults == Py_None || ( PyDict_Check( kwdefaults ) && DICT_SIZE( kwdefaults ) > 0 ) ); result->m_kwdefaults = kwdefaults; assert( annotations == Py_None || PyDict_Check( annotations ) ); result->m_annotations = INCREASE_REFCOUNT( annotations ); #endif result->m_code_object = code_object; result->m_args_positional_count = code_object->co_argcount; result->m_args_keywords_count = result->m_args_positional_count; #if PYTHON_VERSION >= 300 result->m_args_keywords_count += code_object->co_kwonlyargcount; #endif result->m_args_overall_count = result->m_args_keywords_count + (( code_object->co_flags & CO_VARARGS ) ? 1 : 0) + (( code_object->co_flags & CO_VARKEYWORDS ) ? 1 : 0); result->m_args_simple = ( code_object->co_flags & (CO_VARARGS|CO_VARKEYWORDS) ) == 0; #if PYTHON_VERSION >= 300 if ( code_object->co_kwonlyargcount > 0 ) result->m_args_simple = false; #endif if ( ( code_object->co_flags & CO_VARARGS ) != 0 ) { result->m_args_star_list_index = result->m_args_keywords_count; } else { result->m_args_star_list_index = -1; } if ( ( code_object->co_flags & CO_VARKEYWORDS ) != 0 ) { result->m_args_star_dict_index = result->m_args_keywords_count; if ( code_object->co_flags & CO_VARARGS ) { result->m_args_star_dict_index += 1; } } else { result->m_args_star_dict_index = -1; } result->m_varnames = &PyTuple_GET_ITEM( code_object->co_varnames, 0 ); result->m_module = module; result->m_doc = INCREASE_REFCOUNT( doc ); result->m_dict = NULL; result->m_weakrefs = NULL; static long Nuitka_Function_counter = 0; result->m_counter = Nuitka_Function_counter++; result->m_closure = closure; result->m_closure_given = closure_given; Nuitka_GC_Track( result ); return (PyObject *)result; } // Make a function without closure. #if PYTHON_VERSION < 300 PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc ) { return make_compiled_function( c_code, name, code_object, defaults, module, doc, NULL, 0 ); } #elif PYTHON_VERSION < 330 PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc ) { return make_compiled_function( c_code, name, code_object, defaults, kwdefaults, annotations, module, doc, NULL, 0 ); } #else PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc ) { return make_compiled_function( c_code, name, qualname, code_object, defaults, kwdefaults, annotations, module, doc, NULL, 0 ); } #endif // Make a function with closure. #if PYTHON_VERSION < 300 PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ) { return make_compiled_function( c_code, name, code_object, defaults, module, doc, closure, closure_given ); } #elif PYTHON_VERSION < 330 PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ) { return make_compiled_function( c_code, name, code_object, defaults, kwdefaults, annotations, module, doc, closure, closure_given ); } #else PyObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, PyCellObject **closure, Py_ssize_t closure_given ) { return make_compiled_function( c_code, name, qualname, code_object, defaults, kwdefaults, annotations, module, doc, closure, closure_given ); } #endif static void formatErrorNoArgumentAllowed( Nuitka_FunctionObject const *function, #if PYTHON_VERSION >= 330 PyObject *kw, #endif Py_ssize_t given ) { char const *function_name = Nuitka_String_AsString( function->m_name ); #if PYTHON_VERSION < 330 PyErr_Format( PyExc_TypeError, "%s() takes no arguments (%zd given)", function_name, given ); #else if ( kw == NULL ) { PyErr_Format( PyExc_TypeError, "%s() takes 0 positional arguments but %zd was given", function_name, given ); } else { PyObject *tmp_iter = PyObject_GetIter( kw ); PyObject *tmp_arg_name = PyIter_Next( tmp_iter ); Py_DECREF( tmp_iter ); PyErr_Format( PyExc_TypeError, "%s() got an unexpected keyword argument '%s'", function_name, Nuitka_String_AsString( tmp_arg_name ) ); Py_DECREF( tmp_arg_name ); } #endif } static void formatErrorMultipleValuesGiven( Nuitka_FunctionObject const *function, Py_ssize_t index ) { char const *function_name = Nuitka_String_AsString( function->m_name ); PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 330 "%s() got multiple values for keyword argument '%s'", #else "%s() got multiple values for argument '%s'", #endif function_name, Nuitka_String_AsString( function->m_varnames[ index ] ) ); } #if PYTHON_VERSION < 330 static void formatErrorTooFewArguments( Nuitka_FunctionObject const *function, #if PYTHON_VERSION < 270 Py_ssize_t kw_size, #endif Py_ssize_t given ) { Py_ssize_t required_parameter_count = function->m_args_positional_count - function->m_defaults_given; char const *function_name = Nuitka_String_AsString( function->m_name ); char const *violation = ( function->m_defaults != Py_None || function->m_args_star_list_index != -1 ) ? "at least" : "exactly"; char const *plural = required_parameter_count == 1 ? "" : "s"; #if PYTHON_VERSION < 270 if ( kw_size > 0 ) { PyErr_Format( PyExc_TypeError, "%s() takes %s %zd non-keyword argument%s (%zd given)", function_name, violation, required_parameter_count, plural, given - function->m_defaults_given ); } else { PyErr_Format( PyExc_TypeError, "%s() takes %s %zd argument%s (%zd given)", function_name, violation, required_parameter_count, plural, given ); } #else PyErr_Format( PyExc_TypeError, "%s() takes %s %zd argument%s (%zd given)", function_name, violation, required_parameter_count, plural, given ); #endif } #else static void formatErrorTooFewArguments( Nuitka_FunctionObject const *function, PyObject **values ) { char const *function_name = Nuitka_String_AsString( function->m_name ); Py_ssize_t max_missing = 0; for( Py_ssize_t i = function->m_args_positional_count - 1 - function->m_defaults_given; i >= 0; --i ) { if ( values[ i ] == NULL ) { max_missing += 1; } } PyObject *list_str = PyUnicode_FromString( "" ); PyObject *comma_str = PyUnicode_FromString( ", " ); PyObject *and_str = PyUnicode_FromString( max_missing == 2 ? " and " : ", and " ); Py_ssize_t missing = 0; for( Py_ssize_t i = function->m_args_positional_count - 1 - function->m_defaults_given; i >= 0; --i ) { if ( values[ i ] == NULL ) { PyObject *current_str = function->m_varnames[ i ]; PyObject *current = PyObject_Repr( current_str ); if ( missing == 0 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( list_str, current ); Py_DECREF( old ); } else if ( missing == 1 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( and_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } else { PyObject *old = list_str; list_str = PyUnicode_Concat( comma_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } Py_DECREF( current ); missing += 1; } } Py_DECREF( comma_str ); Py_DECREF( and_str ); PyErr_Format( PyExc_TypeError, "%s() missing %zd required positional argument%s: %s", function_name, max_missing, max_missing > 1 ? "s" : "", Nuitka_String_AsString( list_str ) ); Py_DECREF( list_str ); } #endif static void formatErrorTooManyArguments( Nuitka_FunctionObject const *function, Py_ssize_t given #if PYTHON_VERSION < 270 , Py_ssize_t kw_size #endif #if PYTHON_VERSION >= 330 , Py_ssize_t kw_only #endif ) { Py_ssize_t top_level_parameter_count = function->m_args_positional_count; char const *function_name = Nuitka_String_AsString( function->m_name ); #if PYTHON_VERSION < 330 char const *violation = function->m_defaults != Py_None ? "at most" : "exactly"; #endif char const *plural = top_level_parameter_count == 1 ? "" : "s"; #if PYTHON_VERSION < 270 PyErr_Format( PyExc_TypeError, "%s() takes %s %zd %sargument%s (%zd given)", function_name, violation, top_level_parameter_count, kw_size > 0 ? "non-keyword " : "", plural, given ); #elif PYTHON_VERSION < 300 PyErr_Format( PyExc_TypeError, "%s() takes %s %zd argument%s (%zd given)", function_name, violation, top_level_parameter_count, plural, given ); #elif PYTHON_VERSION < 330 PyErr_Format( PyExc_TypeError, "%s() takes %s %zd positional argument%s (%zd given)", function_name, violation, top_level_parameter_count, plural, given ); #else char keyword_only_part[100]; if ( kw_only > 0 ) { sprintf( keyword_only_part, " positional argument%s (and %" PY_FORMAT_SIZE_T "d keyword-only argument%s)", given != 1 ? "s" : "", kw_only, kw_only != 1 ? "s" : "" ); } else { keyword_only_part[0] = 0; } if ( function->m_defaults_given == 0 ) { PyErr_Format( PyExc_TypeError, "%s() takes %zd positional argument%s but %zd%s were given", function_name, top_level_parameter_count, plural, given, keyword_only_part ); } else { PyErr_Format( PyExc_TypeError, "%s() takes from %zd to %zd positional argument%s but %zd%s were given", function_name, top_level_parameter_count - function->m_defaults_given, top_level_parameter_count, plural, given, keyword_only_part ); } #endif } #if PYTHON_VERSION >= 330 static void formatErrorTooFewKwOnlyArguments( Nuitka_FunctionObject const *function, PyObject **kw_vars ) { char const *function_name = Nuitka_String_AsString( function->m_name ); Py_ssize_t kwonlyargcount = function->m_code_object->co_kwonlyargcount; Py_ssize_t max_missing = 0; for( Py_ssize_t i = kwonlyargcount-1; i >= 0; --i ) { if ( kw_vars[ i ] == NULL ) { max_missing += 1; } } PyObject *list_str = PyUnicode_FromString( "" ); PyObject *comma_str = PyUnicode_FromString( ", " ); PyObject *and_str = PyUnicode_FromString( max_missing == 2 ? " and " : ", and " ); Py_ssize_t missing = 0; for( Py_ssize_t i = kwonlyargcount-1; i >= 0; --i ) { if ( kw_vars[ i ] == NULL ) { PyObject *current_str = function->m_varnames[ function->m_args_positional_count + i ]; PyObject *current = PyObject_Repr( current_str ); if ( missing == 0 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( list_str, current ); Py_DECREF( old ); } else if ( missing == 1 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( and_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } else { PyObject *old = list_str; list_str = PyUnicode_Concat( comma_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } Py_DECREF( current ); missing += 1; } } Py_DECREF( comma_str ); Py_DECREF( and_str ); PyErr_Format( PyExc_TypeError, "%s() missing %zd required keyword-only argument%s: %s", function_name, max_missing, max_missing > 1 ? "s" : "", Nuitka_String_AsString( list_str ) ); Py_DECREF( list_str ); } #endif #if PYTHON_VERSION < 300 static Py_ssize_t handleKeywordArgs( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw ) #else static Py_ssize_t handleKeywordArgs( Nuitka_FunctionObject const *function, PyObject **python_pars, Py_ssize_t *kw_only_found, PyObject *kw ) #endif { Py_ssize_t keywords_count = function->m_args_keywords_count; #if PYTHON_VERSION >= 300 Py_ssize_t keyword_after_index = function->m_args_positional_count; #endif assert( function->m_args_star_dict_index == -1 ); Py_ssize_t kw_found = 0; Py_ssize_t ppos = 0; PyObject *key, *value; while( PyDict_Next( kw, &ppos, &key, &value ) ) { #if PYTHON_VERSION < 300 if (unlikely( !PyString_Check( key ) && !PyUnicode_Check( key ) )) #else if (unlikely( !PyUnicode_Check( key ) )) #endif { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return -1; } NUITKA_MAY_BE_UNUSED bool found = false; Py_INCREF( key ); Py_INCREF( value ); for( Py_ssize_t i = 0; i < keywords_count; i++ ) { if ( function->m_varnames[ i ] == key ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = value; #if PYTHON_VERSION >= 300 if ( i >= keyword_after_index ) { *kw_only_found += 1; } #endif found = true; break; } } if ( found == false ) { PyObject **varnames = function->m_varnames; for( Py_ssize_t i = 0; i < keywords_count; i++ ) { if ( RICH_COMPARE_BOOL_EQ_NORECURSE( varnames[ i ], key ) ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = value; #if PYTHON_VERSION >= 300 if ( i >= keyword_after_index ) { *kw_only_found += 1; } #endif found = true; break; } } } if (unlikely( found == false )) { PyErr_Format( PyExc_TypeError, "%s() got an unexpected keyword argument '%s'", Nuitka_String_AsString( function->m_name ), Nuitka_String_Check( key ) ? Nuitka_String_AsString( key ) : "" ); Py_DECREF( key ); Py_DECREF( value ); return -1; } Py_DECREF( key ); kw_found += 1; } return kw_found; } static bool MAKE_STAR_DICT_DICTIONARY_COPY( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw ) { Py_ssize_t star_dict_index = function->m_args_star_dict_index; assert( star_dict_index != -1 ); if ( kw == NULL ) { python_pars[ star_dict_index ] = PyDict_New(); } else if ( ((PyDictObject *)kw)->ma_used > 0 ) { #if PYTHON_VERSION < 330 python_pars[ star_dict_index ] = _PyDict_NewPresized( ((PyDictObject *)kw)->ma_used ); for ( int i = 0; i <= ((PyDictObject *)kw)->ma_mask; i++ ) { PyDictEntry *entry = &((PyDictObject *)kw)->ma_table[ i ]; if ( entry->me_value != NULL ) { #if PYTHON_VERSION < 300 if (unlikely( !PyString_Check( entry->me_key ) && !PyUnicode_Check( entry->me_key ) )) #else if (unlikely( !PyUnicode_Check( entry->me_key ) )) #endif { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return false; } int res = PyDict_SetItem( python_pars[ star_dict_index ], entry->me_key, entry->me_value ); if (unlikely( res != 0 )) { return false; } } } #else if ( _PyDict_HasSplitTable( (PyDictObject *)kw) ) { PyDictObject *mp = (PyDictObject *)kw; PyObject **newvalues = PyMem_NEW( PyObject *, mp->ma_keys->dk_size ); assert( newvalues != NULL ); PyDictObject *split_copy = PyObject_GC_New( PyDictObject, &PyDict_Type ); assert( split_copy != NULL ); split_copy->ma_values = newvalues; split_copy->ma_keys = mp->ma_keys; split_copy->ma_used = mp->ma_used; mp->ma_keys->dk_refcnt += 1; Nuitka_GC_Track( split_copy ); Py_ssize_t size = mp->ma_keys->dk_size; for ( Py_ssize_t i = 0; i < size; i++ ) { PyDictKeyEntry *entry = &split_copy->ma_keys->dk_entries[ i ]; if ( ( entry->me_key != NULL ) && unlikely( !PyUnicode_Check( entry->me_key ) )) { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return false; } split_copy->ma_values[ i ] = mp->ma_values[ i ]; Py_XINCREF( split_copy->ma_values[ i ] ); } python_pars[ star_dict_index ] = (PyObject *)split_copy; } else { python_pars[ star_dict_index ] = PyDict_New(); PyDictObject *mp = (PyDictObject *)kw; Py_ssize_t size = mp->ma_keys->dk_size; for ( Py_ssize_t i = 0; i < size; i++ ) { PyDictKeyEntry *entry = &mp->ma_keys->dk_entries[i]; // TODO: One of these cases has been dealt with above. PyObject *value; if ( mp->ma_values ) { value = mp->ma_values[ i ]; } else { value = entry->me_value; } if ( value != NULL ) { if (unlikely( !PyUnicode_Check( entry->me_key ) )) { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return false; } int res = PyDict_SetItem( python_pars[ star_dict_index ], entry->me_key, value ); if (unlikely( res != 0 )) { return false; } } } } #endif } else { python_pars[ star_dict_index ] = PyDict_New(); } return true; } #if PYTHON_VERSION < 300 static Py_ssize_t handleKeywordArgsWithStarDict( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw ) #else static Py_ssize_t handleKeywordArgsWithStarDict( Nuitka_FunctionObject const *function, PyObject **python_pars, Py_ssize_t *kw_only_found, PyObject *kw ) #endif { assert( function->m_args_star_dict_index != -1 ); if (unlikely( MAKE_STAR_DICT_DICTIONARY_COPY( function, python_pars, kw ) == false )) { return -1; } Py_ssize_t kw_found = 0; Py_ssize_t keywords_count = function->m_args_keywords_count; #if PYTHON_VERSION >= 300 Py_ssize_t keyword_after_index = function->m_args_positional_count; #endif Py_ssize_t dict_star_index = function->m_args_star_dict_index; PyObject **varnames = function->m_varnames; for( Py_ssize_t i = 0; i < keywords_count; i++ ) { PyObject *arg_name = varnames[ i ]; PyObject *kw_arg_value = PyDict_GetItem( python_pars[ dict_star_index ], arg_name ); if ( kw_arg_value != NULL ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = kw_arg_value; Py_INCREF( kw_arg_value ); PyDict_DelItem( python_pars[ dict_star_index ], arg_name ); kw_found += 1; #if PYTHON_VERSION >= 300 if ( i >= keyword_after_index ) { *kw_only_found += 1; } #endif } } return kw_found; } static void makeStarListTupleCopy( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { assert( function->m_args_star_list_index != -1 ); Py_ssize_t list_star_index = function->m_args_star_list_index; // Copy left-over argument values to the star list parameter given. if ( args_size > function->m_args_positional_count ) { python_pars[ list_star_index ] = PyTuple_New( args_size - function->m_args_positional_count ); for( Py_ssize_t i = 0; i < args_size - function->m_args_positional_count; i++ ) { PyObject *value = args[ function->m_args_positional_count + i ]; PyTuple_SET_ITEM( python_pars[ list_star_index ], i, value ); Py_INCREF( value ); } } else { python_pars[ list_star_index ] = const_tuple_empty; Py_INCREF( const_tuple_empty ); } } static void makeStarListTupleCopyMethod( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { assert( function->m_args_star_list_index != -1 ); Py_ssize_t list_star_index = function->m_args_star_list_index; // Copy left-over argument values to the star list parameter given. if ( args_size + 1 > function->m_args_positional_count ) { python_pars[ list_star_index ] = PyTuple_New( args_size + 1 - function->m_args_positional_count ); for( Py_ssize_t i = 0; i < args_size + 1 - function->m_args_positional_count; i++ ) { PyObject *value = args[ function->m_args_positional_count + i - 1 ]; PyTuple_SET_ITEM( python_pars[ list_star_index ], i, value ); Py_INCREF( value ); } } else { python_pars[ list_star_index ] = const_tuple_empty; Py_INCREF( const_tuple_empty ); } } static bool _handleArgumentsPlainOnly( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { Py_ssize_t arg_count = function->m_args_positional_count; // Check if too many arguments were given in case of non list star arg. // For Python3.3 it's done only later, when more knowledge has // been gained. TODO: Could be done this way for improved mode // on all versions. #if PYTHON_VERSION < 330 if ( function->m_args_star_list_index == -1 ) { if (unlikely( args_size > arg_count )) { #if PYTHON_VERSION < 270 formatErrorTooManyArguments( function, args_size, 0 ); #else formatErrorTooManyArguments( function, args_size ); #endif return false; } } #endif #if PYTHON_VERSION >= 330 bool parameter_error = false; #endif Py_ssize_t defaults_given = function->m_defaults_given; if ( args_size + defaults_given < arg_count ) { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, 0, args_size ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size ); return false; #else parameter_error = true; #endif } for( Py_ssize_t i = 0; i < args_size; i++ ) { if ( i >= arg_count ) break; assert( python_pars[ i ] == NULL ); python_pars[ i ] = args[ i ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 if ( parameter_error == false ) { #endif PyObject **source = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = args_size; i < arg_count; i++ ) { assert( python_pars[ i ] == NULL ); assert( i + defaults_given >= arg_count ); python_pars[ i ] = source[ defaults_given + i - arg_count ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 } #endif #if PYTHON_VERSION >= 330 if (unlikely( parameter_error )) { formatErrorTooFewArguments( function, python_pars ); return false; } if ( function->m_args_star_list_index == -1 ) { // Check if too many arguments were given in case of non list star arg if (unlikely( args_size > arg_count )) { formatErrorTooManyArguments( function, args_size, 0 ); return false; } } #endif if ( function->m_args_star_list_index != -1 ) { makeStarListTupleCopy( function, python_pars, args, args_size ); } return true; } static bool handleMethodArgumentsPlainOnly( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *object, PyObject **args, Py_ssize_t args_size ) { Py_ssize_t arg_count = function->m_args_positional_count; // There may be no self, otherwise we can directly assign it. if ( arg_count >= 1 ) { python_pars[ 0 ] = object; Py_INCREF( object ); } else { // Without self, there can only be star list to get the object as its // first element. Or we complain about illegal arguments. if ( function->m_args_star_list_index == 0 ) { python_pars[ 0 ] = PyTuple_New( args_size+1 ); PyTuple_SET_ITEM( python_pars[ 0 ], 0, object ); Py_INCREF( object ); for( Py_ssize_t i = 0; i < args_size; i++ ) { PyTuple_SET_ITEM( python_pars[ 0 ], i + 1, args[ i ] ); Py_INCREF( args[ i ]); } return true; } } // Check if too many arguments were given in case of non list star arg. // For Python3.3 it's done only later, when more knowledge has // been gained. TODO: Could be done this way for improved mode // on all versions. #if PYTHON_VERSION < 330 if ( function->m_args_star_list_index == -1 ) { if (unlikely( args_size + 1 > arg_count )) { #if PYTHON_VERSION < 270 formatErrorTooManyArguments( function, args_size + 1, 0 ); #else formatErrorTooManyArguments( function, args_size + 1 ); #endif return false; } } #endif #if PYTHON_VERSION >= 330 bool parameter_error = false; #endif Py_ssize_t defaults_given = function->m_defaults_given; if ( args_size + 1 + defaults_given < arg_count ) { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, 0, args_size + 1 ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size + 1 ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size + 1 ); return false; #else parameter_error = true; #endif } for( Py_ssize_t i = 0; i < args_size; i++ ) { if ( i+1 >= arg_count ) break; assert( python_pars[ i+1 ] == NULL ); python_pars[ i+1 ] = args[ i ]; Py_INCREF( python_pars[ i+1 ] ); } #if PYTHON_VERSION >= 330 if ( parameter_error == false ) { #endif PyObject **source = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = args_size + 1; i < arg_count; i++ ) { assert( python_pars[ i ] == NULL ); assert( i + defaults_given >= arg_count ); python_pars[ i ] = source[ defaults_given + i - arg_count ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 } #endif #if PYTHON_VERSION >= 330 if (unlikely( parameter_error )) { formatErrorTooFewArguments( function, python_pars ); return false; } if ( function->m_args_star_list_index == -1 ) { // Check if too many arguments were given in case of non list star arg if (unlikely( args_size + 1 > arg_count )) { formatErrorTooManyArguments( function, args_size + 1, 0 ); return false; } } #endif if ( function->m_args_star_list_index != -1 ) { makeStarListTupleCopyMethod( function, python_pars, args, args_size ); } return true; } #if PYTHON_VERSION < 270 static bool handleArgumentsPlain( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw, PyObject **args, Py_ssize_t args_size, Py_ssize_t kw_found, Py_ssize_t kw_size ) #elif PYTHON_VERSION < 300 static bool handleArgumentsPlain( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw, PyObject **args, Py_ssize_t args_size, Py_ssize_t kw_found ) #else static bool handleArgumentsPlain( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw, PyObject **args, Py_ssize_t args_size, Py_ssize_t kw_found, Py_ssize_t kw_only_found ) #endif { Py_ssize_t arg_count = function->m_args_positional_count; // Check if too many arguments were given in case of non list star arg. // For Python3.3 it's done only later, when more knowledge has // been gained. TODO: Could be done this way for improved mode // on all versions. #if PYTHON_VERSION < 330 if ( function->m_args_star_list_index == -1 ) { if (unlikely( args_size > arg_count )) { #if PYTHON_VERSION < 270 formatErrorTooManyArguments( function, args_size, kw_size ); #else formatErrorTooManyArguments( function, args_size + kw_found ); #endif return false; } } #endif #if PYTHON_VERSION >= 330 bool parameter_error = false; #endif if ( kw_found > 0 ) { Py_ssize_t i; for( i = 0; i < (args_size < arg_count ? args_size : arg_count); i++ ) { if (unlikely( python_pars[ i ] != NULL )) { formatErrorMultipleValuesGiven( function, i ); return false; } python_pars[ i ] = args[ i ]; Py_INCREF( python_pars[ i ] ); } Py_ssize_t defaults_given = function->m_defaults_given; for( ; i < arg_count; i++ ) { if ( python_pars[ i ] == NULL ) { if ( i + defaults_given >= arg_count ) { python_pars[ i ] = PyTuple_GET_ITEM( function->m_defaults, defaults_given + i - arg_count ); Py_INCREF( python_pars[ i ] ); } else { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, kw_size, args_size + kw_found ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size + kw_found ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size + kw_found - kw_only_found ); return false; #else parameter_error = true; #endif } } } } else { Py_ssize_t usable = (args_size < arg_count ? args_size : arg_count); Py_ssize_t defaults_given = function->m_defaults_given; if ( defaults_given < arg_count - usable ) { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, kw_size, args_size + kw_found ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size + kw_found ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size + kw_found - kw_only_found ); return false; #else parameter_error = true; #endif } for( Py_ssize_t i = 0; i < usable; i++ ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = args[ i ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 if ( parameter_error == false ) { #endif PyObject **source = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = usable; i < arg_count; i++ ) { assert( python_pars[ i ] == NULL ); assert( i + defaults_given >= arg_count ); python_pars[ i ] = source[ defaults_given + i - arg_count ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 } #endif } #if PYTHON_VERSION >= 330 if (unlikely( parameter_error )) { formatErrorTooFewArguments( function, python_pars ); return false; } if ( function->m_args_star_list_index == -1 ) { // Check if too many arguments were given in case of non list star arg if (unlikely( args_size > arg_count )) { formatErrorTooManyArguments( function, args_size, kw_only_found ); return false; } } #endif if ( function->m_args_star_list_index != -1 ) { makeStarListTupleCopy( function, python_pars, args, args_size ); } return true; } // Release them all in case of an error. static void releaseParameters( Nuitka_FunctionObject const *function, PyObject **python_pars ) { Py_ssize_t arg_count = function->m_args_overall_count; for ( Py_ssize_t i = 0; i < arg_count; i++ ) { Py_XDECREF( python_pars[ i ] ); } } bool parseArgumentsPos( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { bool result; Py_ssize_t arg_count = function->m_args_positional_count; #if PYTHON_VERSION >= 330 bool kw_only_error; #endif if (unlikely( arg_count == 0 && function->m_args_simple && args_size != 0 )) { #if PYTHON_VERSION < 330 formatErrorNoArgumentAllowed( function, args_size ); #else formatErrorNoArgumentAllowed( function, NULL, args_size ); #endif goto error_exit; } result = _handleArgumentsPlainOnly( function, python_pars, args, args_size ); if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif if ( function->m_args_star_dict_index != -1 ) { python_pars[ function->m_args_star_dict_index ] = PyDict_New(); } return true; error_exit: releaseParameters( function, python_pars ); return false; } bool parseArgumentsMethodPos( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *object, PyObject **args, Py_ssize_t args_size ) { bool result; #if PYTHON_VERSION >= 330 bool kw_only_error; #endif result = handleMethodArgumentsPlainOnly( function, python_pars, object, args, args_size ); if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif if ( function->m_args_star_dict_index != -1 ) { python_pars[ function->m_args_star_dict_index ] = PyDict_New(); } return true; error_exit: releaseParameters( function, python_pars ); return false; } static bool parseArgumentsFull( Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size, PyObject *kw ) { Py_ssize_t kw_size = kw ? DICT_SIZE( kw ) : 0; Py_ssize_t kw_found; bool result; #if PYTHON_VERSION >= 300 Py_ssize_t kw_only_found; #endif #if PYTHON_VERSION >= 330 bool kw_only_error; #endif Py_ssize_t arg_count = function->m_args_keywords_count; if (unlikely( arg_count == 0 && function->m_args_simple && args_size + kw_size > 0 )) { #if PYTHON_VERSION < 330 formatErrorNoArgumentAllowed( function, args_size + kw_size ); #else formatErrorNoArgumentAllowed( function, kw_size > 0 ? kw : NULL, args_size ); #endif goto error_exit; } #if PYTHON_VERSION >= 300 kw_only_found = 0; #endif if ( function->m_args_star_dict_index != -1 ) { #if PYTHON_VERSION < 300 kw_found = handleKeywordArgsWithStarDict( function, python_pars, kw ); #else kw_found = handleKeywordArgsWithStarDict( function, python_pars, &kw_only_found, kw ); #endif if ( kw_found == -1 ) goto error_exit; } else if ( kw == NULL || DICT_SIZE( kw ) == 0 ) { kw_found = 0; } else { #if PYTHON_VERSION < 300 kw_found = handleKeywordArgs( function, python_pars, kw ); #else kw_found = handleKeywordArgs( function, python_pars, &kw_only_found, kw ); #endif if ( kw_found == -1 ) goto error_exit; } #if PYTHON_VERSION < 270 result = handleArgumentsPlain( function, python_pars, kw, args, args_size, kw_found, kw_size ); #elif PYTHON_VERSION < 300 result = handleArgumentsPlain( function, python_pars, kw, args, args_size, kw_found ); #else result = handleArgumentsPlain( function, python_pars, kw, args, args_size, kw_found, kw_only_found ); #endif if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif return true; error_exit: releaseParameters( function, python_pars ); return false; } PyObject *Nuitka_CallFunctionPosArgsKwArgs( Nuitka_FunctionObject const *function, PyObject **args, Py_ssize_t args_size, PyObject *kw ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if (!parseArgumentsFull( function, python_pars, args, args_size, kw )) return NULL; return function->m_c_code( function, python_pars ); } PyObject *Nuitka_CallMethodFunctionPosArgsKwArgs( Nuitka_FunctionObject const *function, PyObject *object, PyObject **args, Py_ssize_t args_size, PyObject *kw ) { #ifdef _MSC_VER PyObject **new_args = (PyObject **)_alloca( sizeof( PyObject * ) *( args_size + 1 ) ); #else PyObject *new_args[ args_size + 1 ]; #endif new_args[ 0 ] = object; memcpy( new_args + 1, args, args_size * sizeof( PyObject *) ); // TODO: Specialize implementation for massive gains. return Nuitka_CallFunctionPosArgsKwArgs( function, new_args, args_size + 1, kw ); } Nuitka-0.5.18.1/nuitka/build/static_src/MetaPathBasedLoader.cpp0000644000175000017500000004430412651071040024457 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // This implements the loading of C++ compiled modules and shared library // extension modules bundled for standalone mode. // This is achieved mainly by registered a "sys.meta_path" loader, that then // gets asked for module names, and responds if knows about one. It's fed by // a table created at compile time. // The nature and use of these 2 loaded module kinds is very different, but // having them as distinct loaders would only require to duplicate the search // and registering of stuff. #include #ifdef _WIN32 #undef SEP #define SEP '\\' #endif #include "nuitka/prelude.hpp" #include "nuitka/unfreezing.hpp" // For Python3.3, the loader is a module attribute, so we need to make it // accessible from this variable. #if PYTHON_VERSION < 330 static #endif PyObject *metapath_based_loader = NULL; static Nuitka_MetaPathBasedLoaderEntry *loader_entries = NULL; static char *_kwlist[] = { (char *)"fullname", (char *)"unused", NULL }; static bool hasFrozenModule( char const *name ) { for ( struct _frozen const *p = PyImport_FrozenModules; ; p++ ) { if ( p->name == NULL ) { return false; } if ( strcmp( p->name, name ) == 0 ) { break; } } return true; } static PyObject *_path_unfreezer_find_module( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; PyObject *unused; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O|O:find_module", _kwlist, &module_name, &unused ); if (unlikely( res == 0 )) { return NULL; } char const *name = Nuitka_String_AsString( module_name ); if ( Py_VerboseFlag ) { PySys_WriteStderr( "import %s # considering responsibility\n", name ); } struct Nuitka_MetaPathBasedLoaderEntry *current = loader_entries; while ( current->name != NULL ) { if ( strcmp( name, current->name ) == 0 ) { if ( Py_VerboseFlag ) { PySys_WriteStderr( "import %s # claimed responsibility (compiled)\n", name ); } return INCREASE_REFCOUNT( metapath_based_loader ); } current++; } if ( hasFrozenModule( name ) ) { if ( Py_VerboseFlag ) { PySys_WriteStderr( "import %s # claimed responsibility (frozen)\n", name ); } return INCREASE_REFCOUNT( metapath_based_loader ); } if ( Py_VerboseFlag ) { PySys_WriteStderr( "import %s # denied responsibility\n", name ); } return INCREASE_REFCOUNT( Py_None ); } #ifdef _NUITKA_STANDALONE #if PYTHON_VERSION < 300 typedef void (*entrypoint_t)( void ); #else typedef PyObject * (*entrypoint_t)( void ); #endif #ifndef _WIN32 // Shared libraries loading. #include #endif PyObject *callIntoShlibModule( const char *full_name, const char *filename ) { // Determine the package name and basename of the module to load. char const *dot = strrchr( full_name, '.' ); char const *name; char const *package; if ( dot == NULL ) { package = NULL; name = full_name; } else { package = (char *)full_name; name = dot+1; } char entry_function_name[1024]; snprintf( entry_function_name, sizeof( entry_function_name ), #if PYTHON_VERSION < 300 "init%s", #else "PyInit_%s", #endif name ); #ifdef _WIN32 unsigned int old_mode = SetErrorMode( SEM_FAILCRITICALERRORS ); if ( Py_VerboseFlag ) { PySys_WriteStderr( "import %s # LoadLibraryEx(\"%s\");\n", full_name, filename ); } HINSTANCE hDLL = LoadLibraryEx( filename, NULL, LOAD_WITH_ALTERED_SEARCH_PATH ); if (unlikely( hDLL == NULL )) { PyErr_Format( PyExc_ImportError, "LoadLibraryEx '%s' failed", filename ); return NULL; } entrypoint_t entrypoint = (entrypoint_t)GetProcAddress( hDLL, entry_function_name ); SetErrorMode( old_mode ); #else int dlopenflags = PyThreadState_GET()->interp->dlopenflags; if ( Py_VerboseFlag ) { PySys_WriteStderr( "import %s # dlopen(\"%s\", %x);\n", full_name, filename, dlopenflags ); } void *handle = dlopen( filename, dlopenflags ); if (unlikely( handle == NULL )) { const char *error = dlerror(); if (unlikely( error == NULL )) { error = "unknown dlopen() error"; } PyErr_SetString( PyExc_ImportError, error ); return NULL; } entrypoint_t entrypoint = (entrypoint_t)dlsym( handle, entry_function_name ); #endif assert( entrypoint ); char *old_context = _Py_PackageContext; _Py_PackageContext = (char *)package; // Finally call into the DLL. #if PYTHON_VERSION < 300 (*entrypoint)(); #else PyObject *module = (*entrypoint)(); #endif _Py_PackageContext = old_context; #if PYTHON_VERSION < 300 PyObject *module = PyDict_GetItemString( PyImport_GetModuleDict(), full_name ); #endif if (unlikely( module == NULL )) { PyErr_Format( PyExc_SystemError, "dynamic module not initialized properly" ); return NULL; } #if PYTHON_VERSION >= 300 struct PyModuleDef *def = PyModule_GetDef( module ); if (unlikely( def == NULL )) { PyErr_Format( PyExc_SystemError, "initialization of %s did not return an extension module", filename ); return NULL; } def->m_base.m_init = entrypoint; #endif // Set filename attribute int res = PyModule_AddStringConstant( module, "__file__", filename ); if (unlikely( res < 0 )) { // Might be refuted, which wouldn't be harmful. CLEAR_ERROR_OCCURRED(); } // Call the standard import fix-ups for extension modules. Their interface // changed over releases. #if PYTHON_VERSION < 300 PyObject *res2 = _PyImport_FixupExtension( (char *)full_name, (char *)filename ); if (unlikely( res2 == NULL )) { return NULL; } #elif PYTHON_VERSION < 330 PyObject *filename_obj = PyUnicode_DecodeFSDefault( filename ); CHECK_OBJECT( filename_obj ); res = _PyImport_FixupExtensionUnicode( module, (char *)full_name, filename_obj ); Py_DECREF( filename_obj ); if (unlikely( res == -1 )) { return NULL; } #else PyObject *full_name_obj = PyUnicode_FromString( full_name ); CHECK_OBJECT( full_name_obj ); PyObject *filename_obj = PyUnicode_DecodeFSDefault( filename ); CHECK_OBJECT( filename_obj ); res = _PyImport_FixupExtensionObject( module, full_name_obj, filename_obj ); Py_DECREF( full_name_obj ); Py_DECREF( filename_obj ); if (unlikely( res == -1 )) { return NULL; } #endif return module; } #endif static struct Nuitka_MetaPathBasedLoaderEntry *findEntry( char const *name ) { struct Nuitka_MetaPathBasedLoaderEntry *current = loader_entries; assert( current ); while ( current->name != NULL ) { if ( strcmp( name, current->name ) == 0 ) { return current; } current++; } return NULL; } static void loadTriggeredModule( char const *name, char const *trigger_name ) { char trigger_module_name[2048]; strcpy( trigger_module_name, name ); strcat( trigger_module_name, trigger_name); struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( trigger_module_name ); if ( entry != NULL ) { if ( Py_VerboseFlag ) { PySys_WriteStderr( "Loading %s\n", trigger_module_name ); } entry->python_initfunc(); if (unlikely( ERROR_OCCURRED() )) { PyErr_Print(); abort(); } } } static PyObject *loadModule( PyObject *module_name, Nuitka_MetaPathBasedLoaderEntry *entry ) { #ifdef _NUITKA_STANDALONE if ( ( entry->flags & NUITKA_SHLIB_MODULE ) != 0 ) { // Append the the entry name from full path module name with dots, // and translate these into directory separators. char filename[4096]; strcpy( filename, getBinaryDirectoryHostEncoded() ); char *d = filename; d += strlen( filename ); assert( *d == 0 ); *d++ = SEP; char *s = entry->name; while( *s ) { if ( *s == '.' ) { *d++ = SEP; s++; } else { *d++ = *s++; } } *d = 0; #ifdef _WIN32 strcat( filename, ".pyd" ); #else strcat( filename, ".so" ); #endif callIntoShlibModule( entry->name, filename ); } else #endif { assert( ( entry->flags & NUITKA_SHLIB_MODULE ) == 0 ); entry->python_initfunc(); } if (unlikely( ERROR_OCCURRED() )) { return NULL; } if ( Py_VerboseFlag ) { PySys_WriteStderr( "Loaded %s\n", entry->name ); } return LOOKUP_SUBSCRIPT( PyImport_GetModuleDict(), module_name ); } // Note: This may become an entry point for hard coded imports of compiled // stuff. PyObject *IMPORT_COMPILED_MODULE( PyObject *module_name, char const *name ) { struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); bool frozen_import = entry == NULL && hasFrozenModule( name ); if ( entry != NULL || frozen_import ) { // Execute the "preLoad" code produced for the module potentially. This // is from plug-ins typically, that want to modify things for the the // module before loading, to e.g. set a plug-in path, or do some monkey // patching in order to make things compatible. loadTriggeredModule( name, "-preLoad" ); } PyObject *result = NULL; if ( entry != NULL ) { result = loadModule( module_name, entry ); if ( result == NULL ) { return NULL; } } if ( frozen_import ) { int res = PyImport_ImportFrozenModule( (char *)name ); if (unlikely( res == -1 )) { return NULL; } if ( res == 1 ) { result = LOOKUP_SUBSCRIPT( PyImport_GetModuleDict(), module_name ); } } if ( result != NULL ) { // Execute the "postLoad" code produced for the module potentially. This // is from plug-ins typically, that want to modify the module immediately // after loading, to e.g. set a plug-in path, or do some monkey patching // in order to make things compatible. loadTriggeredModule( name, "-postLoad" ); return result; } return INCREASE_REFCOUNT( Py_None ); } static PyObject *_path_unfreezer_load_module( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; PyObject *unused; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O|O:load_module", _kwlist, &module_name, &unused ); if (unlikely( res == 0 )) { return NULL; } assert( module_name ); assert( Nuitka_String_Check( module_name ) ); char *const name = Nuitka_String_AsString( module_name ); if ( Py_VerboseFlag ) { PySys_WriteStderr( "Loading %s\n", name ); } return IMPORT_COMPILED_MODULE( module_name, name ); } static PyObject *_path_unfreezer_is_package( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O:is_package", _kwlist, &module_name ); if (unlikely( res == 0 )) { return NULL; } assert( module_name ); assert( Nuitka_String_Check( module_name ) ); char *name = Nuitka_String_AsString( module_name ); struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); if ( entry ) { PyObject *result = BOOL_FROM( ( entry->flags & NUITKA_COMPILED_PACKAGE ) != 0 ); return INCREASE_REFCOUNT( result ); } else { // TODO: Maybe needs to be an exception. return INCREASE_REFCOUNT( Py_None ); } } #if PYTHON_VERSION >= 340 static PyObject *_path_unfreezer_repr_module( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module; PyObject *unused; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O|O:module_repr", _kwlist, &module, &unused ); if (unlikely( res == 0 )) { return NULL; } return PyUnicode_FromFormat("", PyModule_GetName( module ), PyModule_GetFilename( module )); } static char *_kwlist2[] = { (char *)"fullname", (char *)"is_package", (char *)"path", NULL }; static PyObject *_path_unfreezer_find_spec( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; PyObject *unused1; PyObject *unused2; int res = PyArg_ParseTupleAndKeywords( args, kwds, "OO|O:find_spec", _kwlist2, &module_name, &unused1, &unused2 ); if (unlikely( res == 0 )) { return NULL; } assert( module_name ); assert( Nuitka_String_Check( module_name ) ); char *name = Nuitka_String_AsString( module_name ); struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); if ( entry == NULL ) { return INCREASE_REFCOUNT( Py_None ); } PyObject *importlib = PyImport_ImportModule( "importlib._bootstrap" ); if (unlikely( importlib == NULL )) { return NULL; } PyObject *module_spec_class = PyObject_GetAttrString( importlib, "ModuleSpec" ); if (unlikely( module_spec_class == NULL )) { Py_DECREF( importlib ); return NULL; } PyObject *result = PyObject_CallFunctionObjArgs( module_spec_class, module_name, metapath_based_loader, NULL ); if (unlikely( result == NULL )) { Py_DECREF( importlib ); Py_DECREF( module_spec_class ); return NULL; } return result; } #endif static PyMethodDef _method_def_loader_find_module = { "find_module", (PyCFunction)_path_unfreezer_find_module, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_load_module = { "load_module", (PyCFunction)_path_unfreezer_load_module, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_is_package = { "is_package", (PyCFunction)_path_unfreezer_is_package, METH_VARARGS | METH_KEYWORDS, NULL }; #if PYTHON_VERSION >= 340 static PyMethodDef _method_def_loader_repr_module = { "module_repr", (PyCFunction)_path_unfreezer_repr_module, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_find_spec = { "module_repr", (PyCFunction)_path_unfreezer_find_spec, METH_VARARGS | METH_KEYWORDS, NULL }; #endif void registerMetaPathBasedUnfreezer( struct Nuitka_MetaPathBasedLoaderEntry *_loader_entries ) { // Do it only once. if ( loader_entries ) { assert( _loader_entries == loader_entries ); return; } loader_entries = _loader_entries; // Build the dictionary of the "loader" object, which needs to have two // methods "find_module" where we acknowledge that we are capable of loading // the module, and "load_module" that does the actual thing. PyObject *method_dict = PyDict_New(); CHECK_OBJECT( method_dict ); PyObject *loader_find_module = PyCFunction_New( &_method_def_loader_find_module, NULL ); CHECK_OBJECT( loader_find_module ); PyDict_SetItemString( method_dict, "find_module", loader_find_module ); PyObject *loader_load_module = PyCFunction_New( &_method_def_loader_load_module, NULL ); CHECK_OBJECT( loader_load_module ); PyDict_SetItemString( method_dict, "load_module", loader_load_module ); PyObject *loader_is_package = PyCFunction_New( &_method_def_loader_is_package, NULL ); CHECK_OBJECT( loader_is_package ); PyDict_SetItemString( method_dict, "is_package", loader_is_package ); #if PYTHON_VERSION >= 340 PyObject *loader_repr_module = PyCFunction_New( &_method_def_loader_repr_module, NULL ); CHECK_OBJECT( loader_repr_module ); PyDict_SetItemString( method_dict, "module_repr", loader_repr_module ); PyObject *loader_find_spec = PyCFunction_New( &_method_def_loader_find_spec, NULL ); CHECK_OBJECT( loader_find_spec ); PyDict_SetItemString( method_dict, "find_spec", loader_find_spec ); #endif // Build the actual class. metapath_based_loader = PyObject_CallFunctionObjArgs( (PyObject *)&PyType_Type, #if PYTHON_VERSION < 300 PyString_FromString( "_nuitka_compiled_modules_loader" ), #else PyUnicode_FromString( "_nuitka_compiled_modules_loader" ), #endif const_tuple_empty, method_dict, NULL ); CHECK_OBJECT( metapath_based_loader ); if ( Py_VerboseFlag ) { PySys_WriteStderr( "setup nuitka compiled module/shlib importer\n" ); } // Register it as a meta path loader. int res = PyList_Insert( PySys_GetObject( ( char *)"meta_path" ), #if PYTHON_VERSION < 330 0, #else 2, #endif metapath_based_loader ); assert( res == 0 ); } Nuitka-0.5.18.1/nuitka/build/static_src/CompiledFrameType.cpp0000644000175000017500000004507412651071040024244 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.hpp" #include "structmember.h" #define OFF( x ) offsetof( PyFrameObject, x ) struct Nuitka_FrameObject { PyFrameObject m_frame; }; static PyMemberDef Nuitka_Frame_memberlist[] = { { (char *)"f_back", T_OBJECT, OFF( f_back ), READONLY | RESTRICTED }, { (char *)"f_code", T_OBJECT, OFF( f_code ), READONLY | RESTRICTED }, { (char *)"f_builtins", T_OBJECT, OFF( f_builtins ), READONLY | RESTRICTED }, { (char *)"f_globals", T_OBJECT, OFF( f_globals ), READONLY | RESTRICTED }, { (char *)"f_lasti", T_INT, OFF( f_lasti ), READONLY | RESTRICTED }, { NULL } }; #if PYTHON_VERSION < 300 static PyObject *Nuitka_Frame_get_exc_traceback( PyFrameObject *frame ) { if ( frame->f_exc_traceback != NULL ) { return INCREASE_REFCOUNT( frame->f_exc_traceback ); } else { return INCREASE_REFCOUNT( Py_None ); } } static int Nuitka_Frame_set_exc_traceback( PyFrameObject *frame, PyObject *traceback ) { Py_XDECREF( frame->f_exc_traceback ); if ( traceback == Py_None ) { frame->f_exc_traceback = NULL; } else { frame->f_exc_traceback = traceback; Py_XINCREF( traceback ); } return 0; } static PyObject *Nuitka_Frame_get_exc_type( PyFrameObject *frame ) { if ( frame->f_exc_type != NULL ) { return INCREASE_REFCOUNT( frame->f_exc_type ); } else { return INCREASE_REFCOUNT( Py_None ); } } static int Nuitka_Frame_set_exc_type( PyFrameObject *frame, PyObject *exception_type ) { Py_XDECREF( frame->f_exc_type ); if ( exception_type == Py_None ) { exception_type = NULL; } frame->f_exc_type = exception_type; Py_XINCREF( frame->f_exc_type ); return 0; } static PyObject *Nuitka_Frame_get_exc_value( PyFrameObject *frame ) { if ( frame->f_exc_value != NULL ) { return INCREASE_REFCOUNT( frame->f_exc_value ); } else { return INCREASE_REFCOUNT( Py_None ); } } static int Nuitka_Frame_set_exc_value( PyFrameObject *frame, PyObject *exception_value ) { if ( frame->f_exc_value != NULL ) { Py_DECREF( frame->f_exc_value ); } if ( exception_value == Py_None ) { exception_value = NULL; } frame->f_exc_value = exception_value; Py_XINCREF( exception_value ); return 0; } static PyObject *Nuitka_Frame_get_restricted( PyFrameObject *frame, void *closure ) { return INCREASE_REFCOUNT( Py_False ); } #endif static PyObject *Nuitka_Frame_getlocals( PyFrameObject *frame, void *closure ) { if ( frame->f_locals == NULL ) { frame->f_locals = PyDict_New(); } Py_INCREF( frame->f_locals ); return frame->f_locals; } static PyObject *Nuitka_Frame_getlineno( PyFrameObject *frame, void *closure ) { return PyInt_FromLong( frame->f_lineno ); } static PyObject *Nuitka_Frame_gettrace( PyFrameObject *frame, void *closure ) { return INCREASE_REFCOUNT( frame->f_trace ); } static int Nuitka_Frame_settrace( PyFrameObject *frame, PyObject* v, void *closure ) { PyErr_Format( PyExc_RuntimeError, "f_trace is not writable in Nuitka" ); return -1; } static PyGetSetDef Nuitka_Frame_getsetlist[] = { { (char *)"f_locals", (getter)Nuitka_Frame_getlocals, NULL, NULL }, { (char *)"f_lineno", (getter)Nuitka_Frame_getlineno, NULL, NULL }, { (char *)"f_trace", (getter)Nuitka_Frame_gettrace, (setter)Nuitka_Frame_settrace, NULL }, #if PYTHON_VERSION < 300 { (char *)"f_restricted", (getter)Nuitka_Frame_get_restricted, NULL, NULL }, { (char *)"f_exc_traceback", (getter)Nuitka_Frame_get_exc_traceback, (setter)Nuitka_Frame_set_exc_traceback, NULL }, { (char *)"f_exc_type", (getter)Nuitka_Frame_get_exc_type, (setter)Nuitka_Frame_set_exc_type, NULL }, { (char *)"f_exc_value", (getter)Nuitka_Frame_get_exc_value, (setter)Nuitka_Frame_set_exc_value, NULL }, #endif { NULL } }; // tp_repr slot, decide how a function shall be output static PyObject *Nuitka_Frame_tp_repr( Nuitka_FrameObject *nuitka_frame ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( #else return PyUnicode_FromFormat( #endif #if _DEBUG_FRAME || _DEBUG_REFRAME || _DEBUG_EXCEPTIONS "", Nuitka_String_AsString( nuitka_frame->m_frame.f_code->co_name ), nuitka_frame #else "", nuitka_frame #endif ); } static void Nuitka_Frame_tp_dealloc( Nuitka_FrameObject *nuitka_frame ) { #ifndef __NUITKA_NO_ASSERT__ // Save the current exception, if any, we must to not corrupt it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); #endif Nuitka_GC_UnTrack( nuitka_frame ); PyFrameObject *frame = &nuitka_frame->m_frame; // locals PyObject **valuestack = frame->f_valuestack; for ( PyObject **p = frame->f_localsplus; p < valuestack; p++ ) { Py_CLEAR( *p ); } // stack if any if ( frame->f_stacktop != NULL ) { for ( PyObject **p = valuestack; p < frame->f_stacktop; p++ ) { Py_XDECREF( *p ); } } Py_XDECREF( frame->f_back ); Py_DECREF( frame->f_builtins ); Py_DECREF( frame->f_globals ); Py_XDECREF( frame->f_locals ); Py_DECREF( frame->f_trace ); Py_XDECREF( frame->f_exc_type ); Py_XDECREF( frame->f_exc_value ); Py_XDECREF( frame->f_exc_traceback ); PyObject_GC_Del( nuitka_frame ); #ifndef __NUITKA_NO_ASSERT__ PyThreadState *tstate = PyThreadState_GET(); assert( tstate->curexc_type == save_exception_type ); assert( tstate->curexc_value == save_exception_value ); assert( (PyTracebackObject *)tstate->curexc_traceback == save_exception_tb ); #endif } static int Nuitka_Frame_tp_traverse( PyFrameObject *frame, visitproc visit, void *arg ) { Py_VISIT( frame->f_back ); Py_VISIT( frame->f_code ); Py_VISIT( frame->f_builtins ); Py_VISIT( frame->f_globals ); Py_VISIT( frame->f_locals ); Py_VISIT( frame->f_trace ); Py_VISIT( frame->f_exc_type ); Py_VISIT( frame->f_exc_value ); Py_VISIT( frame->f_exc_traceback ); // locals Py_ssize_t slots = frame->f_code->co_nlocals + PyTuple_GET_SIZE( frame->f_code->co_cellvars ) + PyTuple_GET_SIZE( frame->f_code->co_freevars ); PyObject **fastlocals = frame->f_localsplus; for ( Py_ssize_t i = slots; --i >= 0; ++fastlocals ) { Py_VISIT( *fastlocals ); } // stack if any if ( frame->f_stacktop != NULL ) { for ( PyObject **p = frame->f_valuestack; p < frame->f_stacktop; p++ ) { Py_VISIT( *p ); } } return 0; } static void Nuitka_Frame_tp_clear( PyFrameObject *frame ) { PyObject **oldtop = frame->f_stacktop; frame->f_stacktop = NULL; // locals Py_ssize_t slots = frame->f_code->co_nlocals + PyTuple_GET_SIZE( frame->f_code->co_cellvars ) + PyTuple_GET_SIZE( frame->f_code->co_freevars ); PyObject **fastlocals = frame->f_localsplus; for ( Py_ssize_t i = slots; --i >= 0; ++fastlocals ) { Py_CLEAR( *fastlocals ); } // stack if any if ( oldtop != NULL ) { for ( PyObject **p = frame->f_valuestack; p < oldtop; p++ ) { Py_CLEAR( *p ); } } } #if PYTHON_VERSION >= 340 extern PyObject *Nuitka_Generator_close( Nuitka_GeneratorObject *generator, PyObject *args ); static PyObject *Nuitka_Frame_clear( PyFrameObject *frame ) { if ( frame->f_executing ) { PyErr_Format( PyExc_RuntimeError, "cannot clear an executing frame" ); return NULL; } #if PYTHON_VERSION >= 340 // For frames that are closed, we also need to close the generator. if ( frame->f_gen != NULL ) { Py_INCREF( frame ); assert( Nuitka_Generator_Check( frame->f_gen ) ); Nuitka_GeneratorObject *generator = (Nuitka_GeneratorObject *)frame->f_gen; frame->f_gen = NULL; PyObject *close_result = Nuitka_Generator_close( generator, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)frame->f_gen ); } else { Py_DECREF( close_result ); } Py_DECREF( frame ); } #endif Nuitka_Frame_tp_clear( frame ); Py_RETURN_NONE; } #endif static PyObject *Nuitka_Frame_sizeof( PyFrameObject *frame ) { Py_ssize_t slots = frame->f_code->co_stacksize + frame->f_code->co_nlocals + PyTuple_GET_SIZE( frame->f_code->co_cellvars ) + PyTuple_GET_SIZE( frame->f_code->co_freevars ); return PyInt_FromSsize_t( sizeof( Nuitka_FrameObject ) + slots * sizeof(PyObject *) ); } static PyMethodDef Nuitka_Frame_methods[] = { #if PYTHON_VERSION >= 340 { "clear", (PyCFunction)Nuitka_Frame_clear, METH_NOARGS, "F.clear(): clear most references held by the frame" }, #endif { "__sizeof__", (PyCFunction)Nuitka_Frame_sizeof, METH_NOARGS, "F.__sizeof__() -> size of F in memory, in bytes" }, { NULL, NULL } }; PyTypeObject Nuitka_Frame_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_frame", sizeof(Nuitka_FrameObject), sizeof(PyObject *), (destructor)Nuitka_Frame_tp_dealloc, // tp_dealloc 0, // tp_print 0, // tp_getattr 0, // tp_setattr 0, // tp_compare (reprfunc)Nuitka_Frame_tp_repr, // tp_repr 0, // tp_as_number 0, // tp_as_sequence 0, // tp_as_mapping 0, // tp_hash 0, // tp_call 0, // tp_str PyObject_GenericGetAttr, // tp_getattro PyObject_GenericSetAttr, // tp_setattro 0, // tp_as_buffer Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, // tp_flags 0, // tp_doc (traverseproc)Nuitka_Frame_tp_traverse, // tp_traverse (inquiry)Nuitka_Frame_tp_clear, // tp_clear 0, // tp_richcompare 0, // tp_weaklistoffset 0, // tp_iter 0, // tp_iternext Nuitka_Frame_methods, // tp_methods Nuitka_Frame_memberlist, // tp_members Nuitka_Frame_getsetlist, // tp_getset 0, // tp_base 0, // tp_dict }; static void tb_dealloc( PyTracebackObject *tb ) { // printf( "dealloc TB %ld %lx FR %ld %lx\n", Py_REFCNT( tb ), (long)tb, Py_REFCNT( tb->tb_frame ), (long)tb->tb_frame ); PyObject_GC_UnTrack(tb); // Py_TRASHCAN_SAFE_BEGIN(tb) Py_XDECREF( tb->tb_next ); Py_XDECREF( tb->tb_frame ); PyObject_GC_Del( tb ); // Py_TRASHCAN_SAFE_END(tb) } extern PyObject *const_str_plain___module__; static PyFrameObject *MAKE_FRAME( PyCodeObject *code, PyObject *module, bool is_module ) { PyTraceBack_Type.tp_dealloc = (destructor)tb_dealloc; assertCodeObject( code ); PyObject *globals = ((PyModuleObject *)module)->md_dict; assert( PyDict_Check( globals ) ); Py_ssize_t ncells = PyTuple_GET_SIZE( code->co_cellvars ); Py_ssize_t nfrees = PyTuple_GET_SIZE( code->co_freevars ); Py_ssize_t extras = code->co_stacksize + code->co_nlocals + ncells + nfrees; Nuitka_FrameObject *result = PyObject_GC_NewVar( Nuitka_FrameObject, &Nuitka_Frame_Type, extras ); if (unlikely( result == NULL )) { return NULL; } PyFrameObject *frame = &result->m_frame; frame->f_code = code; extras = code->co_nlocals + ncells + nfrees; frame->f_valuestack = frame->f_localsplus + extras; for ( Py_ssize_t i = 0; i < extras; i++ ) { frame->f_localsplus[i] = NULL; } frame->f_locals = NULL; frame->f_trace = INCREASE_REFCOUNT( Py_None ); frame->f_exc_type = NULL; frame->f_exc_value = NULL; frame->f_exc_traceback = NULL; frame->f_stacktop = frame->f_valuestack; frame->f_builtins = INCREASE_REFCOUNT( (PyObject *)dict_builtin ); frame->f_back = NULL; frame->f_globals = INCREASE_REFCOUNT( globals ); if (likely( (code->co_flags & CO_OPTIMIZED ) == CO_OPTIMIZED )) { frame->f_locals = NULL; } else if (is_module) { frame->f_locals = INCREASE_REFCOUNT( globals ); } else { frame->f_locals = PyDict_New(); if (unlikely( frame->f_locals == NULL )) { Py_DECREF( result ); return NULL; } PyDict_SetItem( frame->f_locals, const_str_plain___module__, MODULE_NAME( module ) ); } #if PYTHON_VERSION < 340 frame->f_tstate = PyThreadState_GET(); #endif frame->f_lasti = -1; frame->f_lineno = code->co_firstlineno; frame->f_iblock = 0; #if PYTHON_VERSION >= 340 frame->f_gen = NULL; frame->f_executing = 0; #endif Nuitka_GC_Track( result ); return (PyFrameObject *)result; } PyFrameObject *MAKE_MODULE_FRAME( PyCodeObject *code, PyObject *module ) { return MAKE_FRAME( code, module, true ); } PyFrameObject *MAKE_FUNCTION_FRAME( PyCodeObject *code, PyObject *module ) { return MAKE_FRAME( code, module, false ); } extern PyObject *const_str_empty; extern PyObject *const_bytes_empty; #if PYTHON_VERSION < 300 PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int flags ) #else PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int kw_only_count, int flags ) #endif { CHECK_OBJECT( filename ); assert( Nuitka_String_Check( filename ) ); CHECK_OBJECT( function_name ); assert( Nuitka_String_Check( function_name ) ); CHECK_OBJECT( argnames ); assert( PyTuple_Check( argnames ) ); // The PyCode_New has funny code that interns, mutating the tuple that owns // it. Really serious non-immutable shit. We have triggered that changes // behind our back in the past. #ifndef __NUITKA_NO_ASSERT__ Py_hash_t hash = DEEP_HASH( argnames ); #endif // TODO: Consider using PyCode_NewEmpty PyCodeObject *result = PyCode_New( arg_count, // argcount #if PYTHON_VERSION >= 300 kw_only_count, // kw-only count #endif 0, // nlocals 0, // stacksize flags, // flags #if PYTHON_VERSION < 300 const_str_empty, // code (bytecode) #else const_bytes_empty, // code (bytecode) #endif const_tuple_empty, // consts (we are not going to be compatible) const_tuple_empty, // names (we are not going to be compatible) argnames, // varnames (we are not going to be compatible) const_tuple_empty, // freevars (we are not going to be compatible) const_tuple_empty, // cellvars (we are not going to be compatible) filename, // filename function_name, // name line, // firstlineno (offset of the code object) #if PYTHON_VERSION < 300 const_str_empty // lnotab (table to translate code object) #else const_bytes_empty // lnotab (table to translate code object) #endif ); assert( DEEP_HASH( argnames ) == hash ); if (unlikely( result == NULL )) { return NULL; } return result; } static PyFrameObject *duplicateFrame( PyFrameObject *old_frame, PyObject *locals ) { PyFrameObject *new_frame = PyObject_GC_NewVar( PyFrameObject, &PyFrame_Type, 0 ); // Allow only to detach only our tracing frames. assert( Py_TYPE( old_frame ) == &Nuitka_Frame_Type ); assert( old_frame->f_trace == Py_None ); new_frame->f_trace = Py_None; Py_INCREF( Py_None ); // Copy the back reference if any. new_frame->f_back = old_frame->f_back; Py_XINCREF( new_frame->f_back ); // Take a code reference as well. new_frame->f_code = old_frame->f_code; Py_XINCREF( new_frame->f_code ); // Copy attributes. new_frame->f_globals = INCREASE_REFCOUNT( old_frame->f_globals ); new_frame->f_locals = locals; new_frame->f_builtins = INCREASE_REFCOUNT( old_frame->f_builtins ); new_frame->f_exc_type = NULL; new_frame->f_exc_value = NULL; new_frame->f_exc_traceback = NULL; assert( old_frame->f_valuestack == old_frame->f_localsplus ); new_frame->f_valuestack = new_frame->f_localsplus; assert( old_frame->f_stacktop == old_frame->f_valuestack ); new_frame->f_stacktop = new_frame->f_valuestack; #if PYTHON_VERSION < 340 new_frame->f_tstate = old_frame->f_tstate; #endif new_frame->f_lasti = -1; new_frame->f_lineno = old_frame->f_lineno; assert( old_frame->f_iblock == 0 ); new_frame->f_iblock = 0; #if PYTHON_VERSION >= 340 new_frame->f_gen = NULL; new_frame->f_executing = 0; #endif Nuitka_GC_Track( new_frame ); return new_frame; } void detachFrame( PyTracebackObject *traceback, PyObject *locals ) { // Duplicate it. PyFrameObject *old_frame = traceback->tb_frame; PyFrameObject *new_frame = duplicateFrame( old_frame, locals ); // The new frame replaces it. traceback->tb_frame = new_frame; Py_DECREF( old_frame ); } Nuitka-0.5.18.1/nuitka/build/static_src/win32_ucontext_src/0000755000175000017500000000000012651072347023732 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/static_src/win32_ucontext_src/fibers_win32.cpp0000644000175000017500000000317312651071040026723 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for Win32 #include "nuitka/prelude.hpp" // Less than 1MB is ignored on Win32 apparently. #ifdef MS_WIN64 #define STACK_SIZE 2*(1024*1024) #else #define STACK_SIZE (1024*1024) #endif void _initFiber( Fiber *to ) { // Need to call this at least once per thread, so we have a main FIBER. ConvertThreadToFiber( NULL ); to->fiber = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { to->fiber = CreateFiber( STACK_SIZE, (LPFIBER_START_ROUTINE)code, (LPVOID)arg ); return to->fiber != NULL ? 0 : 1; } void _releaseFiber( Fiber *to ) { if ( to->fiber ) { DeleteFiber( to->fiber ); to->fiber = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { to->fiber = GetCurrentFiber(); assert( from->fiber != NULL ); SwitchToFiber( from->fiber ); } Nuitka-0.5.18.1/nuitka/build/static_src/gen_ucontext_src/0000755000175000017500000000000012651072347023541 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/static_src/gen_ucontext_src/fibers_gen.cpp0000644000175000017500000000421112651071040026333 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for generic targets. #include "nuitka/prelude.hpp" // TODO: Make stack size rational. #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { to->f_context.uc_stack.ss_sp = NULL; to->f_context.uc_link = NULL; to->start_stack = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { int res = getcontext( &to->f_context ); if (unlikely( res != 0 )) { return 1; } to->f_context.uc_stack.ss_size = STACK_SIZE; to->f_context.uc_stack.ss_sp = last_stack ? (char *)last_stack : (char *)malloc( STACK_SIZE ); to->start_stack = to->f_context.uc_stack.ss_sp; to->f_context.uc_link = NULL; last_stack = NULL; makecontext( &to->f_context, (void (*)())code, 1, (unsigned long)arg ); return 0; } void _releaseFiber( Fiber *to ) { if ( to->start_stack != NULL ) { if ( last_stack == NULL ) { last_stack = to->start_stack; } else { free( to->start_stack ); } to->start_stack = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { int res = swapcontext( &to->f_context, &from->f_context ); assert( res == 0 ); } Nuitka-0.5.18.1/nuitka/build/static_src/arm_ucontext_src/0000755000175000017500000000000012651072347023547 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/static_src/arm_ucontext_src/ucontext.cpp0000644000175000017500000000473512651070406026127 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ #include "ucontext.h" #include #include void makecontext(ucontext_t *uc, void (*fn)(void), int argc, ...) { int *sp; va_list arg; sp = (int*)uc->uc_stack.ss_sp+uc->uc_stack.ss_size/4; va_start(arg, argc); assert( argc == 1 ); // That allows me to hard code r0 uc->uc_mcontext.trap_no = va_arg(arg, unsigned long); va_end(arg); uc->uc_mcontext.arm_r10 = (unsigned long)sp; uc->uc_mcontext.arm_fp = (unsigned long)fn; } Nuitka-0.5.18.1/nuitka/build/static_src/arm_ucontext_src/fibers_arm.cpp0000644000175000017500000000712512651070406026363 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ // Implementation of process context switch for ARM #include "nuitka/prelude.hpp" extern "C" int getmcontext( mcontext_t *mcontext ); extern "C" void setmcontext( const mcontext_t *mcontext ); #define setcontext(u) setmcontext( &(u)->uc_mcontext ) #define getcontext(u) getmcontext( &(u)->uc_mcontext ) void makecontext( ucontext_t *uc, void (*fn)(void), int argc, ... ); #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { to->f_context.uc_stack.ss_sp = NULL; to->f_context.uc_link = NULL; to->start_stack = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { int res = getcontext( &to->f_context ); if (unlikely( res != 0 )) { return 1; } to->f_context.uc_stack.ss_size = STACK_SIZE; to->f_context.uc_stack.ss_sp = last_stack ? last_stack : malloc( STACK_SIZE ); to->start_stack = to->f_context.uc_stack.ss_sp; to->f_context.uc_link = NULL; last_stack = NULL; makecontext( &to->f_context, (void (*)())code, 1, (unsigned long)arg ); return 0; } void _releaseFiber( Fiber *to ) { if ( to->start_stack != NULL ) { if ( last_stack == NULL ) { last_stack = to->start_stack; } else { free( to->start_stack ); } to->start_stack = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { if ( getcontext( &to->f_context ) == 0 ) { setcontext( &from->f_context ); } } Nuitka-0.5.18.1/nuitka/build/static_src/arm_ucontext_src/getcontext.asm0000644000175000017500000000607312651070406026435 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ .globl getmcontext getmcontext: str r1, [r0,#4] str r2, [r0,#8] str r3, [r0,#12] str r4, [r0,#16] str r5, [r0,#20] str r6, [r0,#24] str r7, [r0,#28] str r8, [r0,#32] str r9, [r0,#36] str r10, [r0,#40] str r11, [r0,#44] str r12, [r0,#48] str r13, [r0,#52] str r14, [r0,#56] /* store 1 as r0-to-restore */ mov r1, #1 str r1, [r0] /* return 0 */ mov r0, #0 mov pc, lr .globl setmcontext setmcontext: ldr r1, [r0,#4] ldr r2, [r0,#8] ldr r3, [r0,#12] ldr r4, [r0,#16] ldr r5, [r0,#20] ldr r6, [r0,#24] ldr r7, [r0,#28] ldr r8, [r0,#32] ldr r9, [r0,#36] ldr r10, [r0,#40] ldr r11, [r0,#44] ldr r12, [r0,#48] ldr r13, [r0,#52] ldr r14, [r0,#56] ldr r0, [r0] mov pc, lr Nuitka-0.5.18.1/nuitka/build/static_src/arm_ucontext_src/ucontext.h0000644000175000017500000000463512651070406025573 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ #ifndef ARM_UCONTEXT_H #define ARM_UCONTEXT_H #include #include extern "C" int getmcontext(mcontext_t*); extern "C" void setmcontext(const mcontext_t*); int swapcontext(ucontext_t *oucp, const ucontext_t *ucp); void makecontext(ucontext_t *uc, void (*fn)(void), int argc, ...); #define setcontext(u) setmcontext(&(u)->uc_mcontext) #define getcontext(u) getmcontext(&(u)->uc_mcontext) #endif Nuitka-0.5.18.1/nuitka/build/static_src/CompiledGeneratorType.cpp0000644000175000017500000012117512651072076025147 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.hpp" static PyObject *Nuitka_Generator_tp_repr( Nuitka_GeneratorObject *generator ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( "", Nuitka_String_AsString( generator->m_name ), generator ); #else return PyUnicode_FromFormat( "", #if PYTHON_VERSION < 350 Nuitka_String_AsString( generator->m_name ), #else Nuitka_String_AsString( generator->m_qualname ), #endif generator ); #endif } static long Nuitka_Generator_tp_traverse( Nuitka_GeneratorObject *generator, visitproc visit, void *arg ) { // Not needed. // Py_VISIT( (PyObject *)generator->m_frame ); return 0; } static void Nuitka_Generator_release_closure( Nuitka_GeneratorObject *generator ) { if ( generator->m_closure ) { for( Py_ssize_t i = 0; i < generator->m_closure_given; i++ ) { Py_DECREF( generator->m_closure[ i ] ); } free( generator->m_closure ); generator->m_closure = NULL; } } // For the generator object fiber entry point, we may need to follow what // "makecontext" will support and that is only a list of integers, but we will need // to push a pointer through it, and so it's two of them, which might be fully // sufficient. #ifdef _NUITKA_MAKECONTEXT_INTS static void Nuitka_Generator_entry_point( int address_1, int address_2 ) { // Restore the pointer from integers should it be necessary, depending on // the platform. This requires pointers to be no larger that to "int" value. int addresses[2] = { address_1, address_2 }; Nuitka_GeneratorObject *generator = (Nuitka_GeneratorObject *)*(uintptr_t *)&addresses[0]; #else static void Nuitka_Generator_entry_point( Nuitka_GeneratorObject *generator ) { #endif ((generator_code)generator->m_code)( generator ); swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); } static PyObject *Nuitka_Generator_send( Nuitka_GeneratorObject *generator, PyObject *value ) { if ( generator->m_status == status_Unused && value != NULL && value != Py_None ) { PyErr_Format( PyExc_TypeError, "can't send non-None value to a just-started generator" ); return NULL; } if ( generator->m_status != status_Finished ) { PyThreadState *thread_state = PyThreadState_GET(); #if PYTHON_VERSION < 300 PyObject *saved_exception_type = thread_state->exc_type; Py_XINCREF( saved_exception_type ); PyObject *saved_exception_value = thread_state->exc_value; Py_XINCREF( saved_exception_value ); PyTracebackObject *saved_exception_traceback = (PyTracebackObject *)thread_state->exc_traceback; Py_XINCREF( saved_exception_traceback ); #endif if ( generator->m_running ) { PyErr_Format( PyExc_ValueError, "generator already executing" ); return NULL; } if ( generator->m_status == status_Unused ) { // Prepare the generator context to run. int res = prepareFiber( &generator->m_yielder_context, (void *)Nuitka_Generator_entry_point, (uintptr_t)generator ); if ( res != 0 ) { PyErr_Format( PyExc_MemoryError, "generator cannot be allocated" ); return NULL; } generator->m_status = status_Running; } generator->m_yielded = value; // Put the generator back on the frame stack. PyFrameObject *return_frame = thread_state->frame; #ifndef __NUITKA_NO_ASSERT__ if ( return_frame ) { assertFrameObject( return_frame ); } #endif if ( generator->m_frame ) { // It would be nice if our frame were still alive. Nobody had the // right to release it. assertFrameObject( generator->m_frame ); // It's not supposed to be on the top right now. assert( return_frame != generator->m_frame ); Py_XINCREF( return_frame ); generator->m_frame->f_back = return_frame; thread_state->frame = generator->m_frame; } // Continue the yielder function while preventing recursion. generator->m_running = true; swapFiber( &generator->m_caller_context, &generator->m_yielder_context ); generator->m_running = false; thread_state = PyThreadState_GET(); // Remove the generator from the frame stack. if ( generator->m_frame ) { assert( thread_state->frame == generator->m_frame ); assertFrameObject( generator->m_frame ); Py_CLEAR( generator->m_frame->f_back ); } thread_state->frame = return_frame; if ( generator->m_yielded == NULL ) { assert( ERROR_OCCURRED() ); generator->m_status = status_Finished; Py_XDECREF( generator->m_frame ); generator->m_frame = NULL; Nuitka_Generator_release_closure( generator ); assert( ERROR_OCCURRED() ); #if PYTHON_VERSION < 300 Py_XDECREF( saved_exception_type ); Py_XDECREF( saved_exception_value ); Py_XDECREF( saved_exception_traceback ); #endif #if PYTHON_VERSION >= 350 if ( generator->m_code_object->co_flags & CO_FUTURE_GENERATOR_STOP && GET_ERROR_OCCURRED() == PyExc_StopIteration ) { PyObject *saved_exception_type, *saved_exception_value; PyTracebackObject *saved_exception_tb; FETCH_ERROR_OCCURRED( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); NORMALIZE_EXCEPTION( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); PyErr_Format( PyExc_RuntimeError, "generator raised StopIteration" ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, saved_exception_value ); CHECK_OBJECT( exception_value ); CHECK_OBJECT( saved_exception_value ); Py_INCREF( saved_exception_value ); PyException_SetContext( exception_value, saved_exception_value ); Py_DECREF( saved_exception_type ); Py_XDECREF( saved_exception_tb ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } #endif return NULL; } else { #if PYTHON_VERSION < 300 SET_CURRENT_EXCEPTION( saved_exception_type, saved_exception_value, saved_exception_traceback ); #endif return generator->m_yielded; } } else { PyErr_SetObject( PyExc_StopIteration, (PyObject *)NULL ); return NULL; } } static PyObject *Nuitka_Generator_tp_iternext( Nuitka_GeneratorObject *generator ) { return Nuitka_Generator_send( generator, Py_None ); } #if PYTHON_VERSION < 340 static #endif PyObject *Nuitka_Generator_close( Nuitka_GeneratorObject *generator, PyObject *args ) { if ( generator->m_status == status_Running ) { generator->m_exception_type = INCREASE_REFCOUNT( PyExc_GeneratorExit ); generator->m_exception_value = NULL; generator->m_exception_tb = NULL; PyObject *result = Nuitka_Generator_send( generator, Py_None ); if (unlikely( result )) { Py_DECREF( result ); PyErr_Format( PyExc_RuntimeError, "generator ignored GeneratorExit" ); return NULL; } else { PyObject *error = GET_ERROR_OCCURRED(); assert( error != NULL ); if ( EXCEPTION_MATCH_GENERATOR( error ) ) { CLEAR_ERROR_OCCURRED(); return INCREASE_REFCOUNT( Py_None ); } return NULL; } } return INCREASE_REFCOUNT( Py_None ); } static PyObject *Nuitka_Generator_throw( Nuitka_GeneratorObject *generator, PyObject *args ) { assert( generator->m_exception_type == NULL ); assert( generator->m_exception_value == NULL ); assert( generator->m_exception_tb == NULL ); int res = PyArg_UnpackTuple( args, "throw", 1, 3, &generator->m_exception_type, &generator->m_exception_value, (PyObject **)&generator->m_exception_tb ); if (unlikely( res == 0 )) { generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } if ( (PyObject *)generator->m_exception_tb == Py_None ) { generator->m_exception_tb = NULL; } else if ( generator->m_exception_tb != NULL && !PyTraceBack_Check( generator->m_exception_tb ) ) { generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( PyExceptionClass_Check( generator->m_exception_type )) { Py_INCREF( generator->m_exception_type ); Py_XINCREF( generator->m_exception_value ); Py_XINCREF( generator->m_exception_tb ); NORMALIZE_EXCEPTION( &generator->m_exception_type, &generator->m_exception_value, &generator->m_exception_tb ); } else if ( PyExceptionInstance_Check( generator->m_exception_type ) ) { if ( generator->m_exception_value && generator->m_exception_value != Py_None ) { generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); return NULL; } generator->m_exception_value = generator->m_exception_type; Py_INCREF( generator->m_exception_value ); generator->m_exception_type = PyExceptionInstance_Class( generator->m_exception_type ); Py_INCREF( generator->m_exception_type ); Py_XINCREF( generator->m_exception_tb ); } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 300 "exceptions must be classes, or instances, not %s", #else "exceptions must be classes or instances deriving from BaseException, not %s", #endif Py_TYPE( generator->m_exception_type )->tp_name ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } if ( ( generator->m_exception_tb != NULL ) && ( (PyObject *)generator->m_exception_tb != Py_None ) && ( !PyTraceBack_Check( generator->m_exception_tb ) ) ) { PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( generator->m_status != status_Finished ) { PyObject *result = Nuitka_Generator_send( generator, Py_None ); return result; } else { RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } } #if PYTHON_VERSION >= 340 static void Nuitka_Generator_tp_del( Nuitka_GeneratorObject *generator ) { if ( generator->m_status != status_Running ) { return; } // Revive temporarily. PyObject *error_type, *error_value; PyTracebackObject *error_traceback; FETCH_ERROR_OCCURRED( &error_type, &error_value, &error_traceback ); PyObject *close_result = Nuitka_Generator_close( generator, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)generator ); } else { Py_DECREF( close_result ); } /* Restore the saved exception. */ RESTORE_ERROR_OCCURRED( error_type, error_value, error_traceback ); Py_DECREF( generator ); } #endif static void Nuitka_Generator_tp_dealloc( Nuitka_GeneratorObject *generator ) { assert( Py_REFCNT( generator ) == 0 ); Py_REFCNT( generator ) = 1; // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); PyObject *close_result = Nuitka_Generator_close( generator, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)generator ); } else { Py_DECREF( close_result ); } Nuitka_Generator_release_closure( generator ); Py_XDECREF( generator->m_frame ); assert( Py_REFCNT( generator ) == 1 ); Py_REFCNT( generator ) = 0; releaseFiber( &generator->m_yielder_context ); // Now it is safe to release references and memory for it. Nuitka_GC_UnTrack( generator ); if ( generator->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)generator ); assert( !ERROR_OCCURRED() ); } Py_DECREF( generator->m_name ); #if PYTHON_VERSION >= 350 Py_DECREF( generator->m_qualname ); #endif PyObject_GC_Del( generator ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } static PyObject *Nuitka_Generator_get_name( Nuitka_GeneratorObject *generator ) { return INCREASE_REFCOUNT( generator->m_name ); } #if PYTHON_VERSION >= 350 static int Nuitka_Generator_set_name( Nuitka_GeneratorObject *generator, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *tmp = generator->m_name; Py_INCREF( value ); generator->m_name = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Generator_get_qualname( Nuitka_GeneratorObject *generator ) { return INCREASE_REFCOUNT( generator->m_qualname ); } static int Nuitka_Generator_set_qualname( Nuitka_GeneratorObject *generator, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *tmp = generator->m_qualname; Py_INCREF( value ); generator->m_qualname = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Generator_get_yieldfrom( Nuitka_GeneratorObject *generator ) { if ( generator->m_yieldfrom ) { Py_INCREF( generator->m_yieldfrom ); return generator->m_yieldfrom; } else { Py_INCREF( Py_None ); return Py_None; } } #endif static PyObject *Nuitka_Generator_get_code( Nuitka_GeneratorObject *generator ) { return INCREASE_REFCOUNT( (PyObject *)generator->m_code_object ); } static int Nuitka_Generator_set_code( Nuitka_GeneratorObject *generator, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "gi_code is not writable in Nuitka" ); return -1; } static PyObject *Nuitka_Generator_get_frame( Nuitka_GeneratorObject *generator ) { if ( generator->m_frame ) { return INCREASE_REFCOUNT( (PyObject *)generator->m_frame ); } else { return INCREASE_REFCOUNT( Py_None ); } } static int Nuitka_Generator_set_frame( Nuitka_GeneratorObject *generator, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "gi_frame is not writable in Nuitka" ); return -1; } static PyGetSetDef Nuitka_Generator_getsetlist[] = { #if PYTHON_VERSION < 350 { (char *)"__name__", (getter)Nuitka_Generator_get_name, NULL, NULL }, #else { (char *)"__name__", (getter)Nuitka_Generator_get_name, (setter)Nuitka_Generator_set_name, NULL }, { (char *)"__qualname__", (getter)Nuitka_Generator_get_qualname, (setter)Nuitka_Generator_set_qualname, NULL }, { (char *)"gi_yieldfrom", (getter)Nuitka_Generator_get_yieldfrom, NULL, NULL }, #endif { (char *)"gi_code", (getter)Nuitka_Generator_get_code, (setter)Nuitka_Generator_set_code, NULL }, { (char *)"gi_frame", (getter)Nuitka_Generator_get_frame, (setter)Nuitka_Generator_set_frame, NULL }, { NULL } }; static PyMethodDef Nuitka_Generator_methods[] = { { "send", (PyCFunction)Nuitka_Generator_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_Generator_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_Generator_close, METH_NOARGS, NULL }, { NULL } }; #include static PyMemberDef Nuitka_Generator_members[] = { /* The type of "gi_running" changed in Python3. */ #if PYTHON_VERSION < 330 { (char *)"gi_running", T_INT, offsetof( Nuitka_GeneratorObject, m_running ), READONLY }, #else { (char *)"gi_running", T_BOOL, offsetof( Nuitka_GeneratorObject, m_running ), READONLY }, #endif { NULL } }; PyTypeObject Nuitka_Generator_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_generator", /* tp_name */ sizeof(Nuitka_GeneratorObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_Generator_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)Nuitka_Generator_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ #if PYTHON_VERSION < 340 Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, #else Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE, #endif /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Generator_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof( Nuitka_GeneratorObject, m_weakrefs ), /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_Generator_tp_iternext, /* tp_iternext */ Nuitka_Generator_methods, /* tp_methods */ Nuitka_Generator_members, /* tp_members */ Nuitka_Generator_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0 /* tp_version_tag */ #if PYTHON_VERSION >= 340 ,(destructor)Nuitka_Generator_tp_del /* tp_finalizer */ #endif }; #if PYTHON_VERSION < 350 PyObject *Nuitka_Generator_New( generator_code code, PyObject *name, PyCodeObject *code_object, PyCellObject **closure, Py_ssize_t closure_given ) #else PyObject *Nuitka_Generator_New( generator_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyCellObject **closure, Py_ssize_t closure_given ) #endif { Nuitka_GeneratorObject *result = PyObject_GC_New( Nuitka_GeneratorObject, &Nuitka_Generator_Type ); assert( result != NULL ); result->m_code = (void *)code; CHECK_OBJECT( name ); result->m_name = name; Py_INCREF( name ); #if PYTHON_VERSION >= 350 CHECK_OBJECT( qualname ); result->m_qualname = qualname; Py_INCREF( qualname ); result->m_yieldfrom = NULL; #endif // We take ownership of those and received the reference count from the // caller. result->m_closure = closure; result->m_closure_given = closure_given; result->m_weakrefs = NULL; result->m_status = status_Unused; result->m_running = false; result->m_exception_type = NULL; result->m_exception_value = NULL; result->m_exception_tb = NULL; result->m_yielded = NULL; result->m_frame = NULL; result->m_code_object = code_object; initFiber( &result->m_yielder_context ); Nuitka_GC_Track( result ); return (PyObject *)result; } #if PYTHON_VERSION >= 330 // This is for CPython iterator objects, the respective code is not exported as // API, so we need to redo it. This is an re-implementation that closely follows // what it does. It's unrelated to compiled generators. PyObject *PyGen_Send( PyGenObject *generator, PyObject *arg ) { if (unlikely( generator->gi_running )) { PyErr_SetString( PyExc_ValueError, "generator already executing" ); return NULL; } PyFrameObject *frame = generator->gi_frame; if ( frame == NULL || frame->f_stacktop == NULL ) { // Set exception if called from send() if ( arg != NULL ) { PyErr_SetNone( PyExc_StopIteration ); } return NULL; } if ( frame->f_lasti == -1 ) { if (unlikely( arg && arg != Py_None )) { PyErr_SetString( PyExc_TypeError, "can't send non-None value to a just-started generator" ); return NULL; } } else { // Put arg on top of the value stack PyObject *tmp = arg ? arg : Py_None; *(frame->f_stacktop++) = INCREASE_REFCOUNT( tmp ); } // Generators always return to their most recent caller, not necessarily // their creator. PyThreadState *tstate = PyThreadState_GET(); Py_XINCREF( tstate->frame ); assert( frame->f_back == NULL ); frame->f_back = tstate->frame; generator->gi_running = 1; PyObject *result = PyEval_EvalFrameEx( frame, 0 ); generator->gi_running = 0; // Don't keep the reference to f_back any longer than necessary. It // may keep a chain of frames alive or it could create a reference // cycle. assert( frame->f_back == tstate->frame ); Py_CLEAR( frame->f_back ); // If the generator just returned (as opposed to yielding), signal that the // generator is exhausted. if ( result && frame->f_stacktop == NULL ) { if ( result == Py_None ) { PyErr_SetNone( PyExc_StopIteration ); } else { PyObject *e = PyObject_CallFunctionObjArgs( PyExc_StopIteration, result, NULL ); if ( e != NULL ) { PyErr_SetObject( PyExc_StopIteration, e ); Py_DECREF( e ); } } Py_CLEAR( result ); } if ( result == NULL || frame->f_stacktop == NULL ) { // Generator is finished, remove exception from frame before releasing // it. PyObject *type = frame->f_exc_type; PyObject *value = frame->f_exc_value; PyObject *traceback = frame->f_exc_traceback; frame->f_exc_type = NULL; frame->f_exc_value = NULL; frame->f_exc_traceback = NULL; Py_XDECREF( type ); Py_XDECREF( value ); Py_XDECREF( traceback ); // Now release frame. generator->gi_frame = NULL; Py_DECREF( frame ); } return result; } #if PYTHON_VERSION >= 300 PyObject *ERROR_GET_STOP_ITERATION_VALUE() { assert( PyErr_ExceptionMatches( PyExc_StopIteration ) ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); Py_DECREF( exception_type ); Py_XDECREF( exception_tb ); PyObject *value = NULL; if ( exception_value ) { if ( EXCEPTION_MATCH_BOOL_SINGLE( exception_value, PyExc_StopIteration ) ) { value = ((PyStopIterationObject *)exception_value)->value; Py_XINCREF( value ); Py_DECREF( exception_value ); } else { value = exception_value; } } if ( value == NULL ) { value = INCREASE_REFCOUNT( Py_None ); } return value; } static void RAISE_GENERATOR_EXCEPTION( Nuitka_GeneratorObject *generator ) { CHECK_OBJECT( generator->m_exception_type ); RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } extern PyObject *ERROR_GET_STOP_ITERATION_VALUE(); extern PyObject *const_str_plain_send, *const_str_plain_throw, *const_str_plain_close; static PyObject *_YIELD_FROM( Nuitka_GeneratorObject *generator, PyObject *value ) { // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( generator->m_exception_type ) { // The yielding generator is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( generator->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL && !EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } } RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( generator->m_exception_type ); Py_CLEAR( generator->m_exception_value ); Py_CLEAR( generator->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { // Bug compatibility here, before 3.3 tuples were unrolled in calls, which is what // PyObject_CallMethod does. #if PYTHON_VERSION >= 340 retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); #else retval = PyObject_CallMethod( value, (char *)"send", (char *)"O", send_value ); #endif } // Check the sub-generator result if ( retval == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { return INCREASE_REFCOUNT( Py_None ) ; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { generator->m_yielded = retval; #if PYTHON_VERSION >= 350 generator->m_yieldfrom = value; #endif // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); #if PYTHON_VERSION >= 350 generator->m_yieldfrom = NULL; #endif send_value = generator->m_yielded; CHECK_OBJECT( send_value ); } } } PyObject *YIELD_FROM( Nuitka_GeneratorObject *generator, PyObject *target ) { PyObject *value; #if PYTHON_VERSION >= 350 if ( PyCoro_CheckExact( target ) || Nuitka_Coroutine_Check( target )) { if (unlikely( (generator->m_code_object->co_flags & CO_ITERABLE_COROUTINE) == 0 )) { PyErr_SetString( PyExc_TypeError, "cannot 'yield from' a coroutine object in a non-coroutine generator" ); return NULL; } return _YIELD_FROM( generator, target ); } else #endif { value = MAKE_ITERATOR( target ); if (unlikely( value == NULL )) { return NULL; } PyObject *result = _YIELD_FROM( generator, value ); Py_DECREF( value ); return result; } } // Note: This is copy if YIELD_FROM with changes only at the end. As it's not // easy to split up, we are going to tolerate the copy. static PyObject *_YIELD_FROM_IN_HANDLER( Nuitka_GeneratorObject *generator, PyObject *value ) { // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( generator->m_exception_type ) { // The yielding generator is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( generator->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else if ( !EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( generator->m_exception_type ); Py_CLEAR( generator->m_exception_value ); Py_CLEAR( generator->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { // Bug compatibility here, before 3.3 tuples were unrolled in calls, which is what // PyObject_CallMethod does. #if PYTHON_VERSION >= 340 retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); #else retval = PyObject_CallMethod( value, (char *)"send", (char *)"O", send_value ); #endif } // Check the sub-generator result if ( retval == NULL ) { if ( !ERROR_OCCURRED() ) { return INCREASE_REFCOUNT( Py_None ) ; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { generator->m_yielded = retval; // When yielding from an exception handler in Python3, the exception // preserved to the frame is restore, while the current one is put there. PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; #if PYTHON_VERSION >= 350 generator->m_yieldfrom = value; #endif // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); #if PYTHON_VERSION >= 350 generator->m_yieldfrom = NULL; #endif // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; send_value = generator->m_yielded; CHECK_OBJECT( send_value ); } } } PyObject *YIELD_FROM_IN_HANDLER( Nuitka_GeneratorObject *generator, PyObject *target ) { PyObject *value; #if PYTHON_VERSION >= 350 if ( PyCoro_CheckExact( target ) || Nuitka_Coroutine_Check( target )) { if (unlikely( (generator->m_code_object->co_flags & CO_ITERABLE_COROUTINE) == 0 )) { PyErr_SetString( PyExc_TypeError, "cannot 'yield from' a coroutine object in a non-coroutine generator" ); return NULL; } return _YIELD_FROM_IN_HANDLER( generator, target ); } else #endif { value = MAKE_ITERATOR( target ); if (unlikely( value == NULL )) { return NULL; } PyObject *result = _YIELD_FROM_IN_HANDLER( generator, value ); Py_DECREF( value ); return result; } } #endif #endif Nuitka-0.5.18.1/nuitka/build/static_src/CompiledCodeHelpers.cpp0000644000175000017500000031261612651072046024553 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementations of compiled code helpers. // The definition of a compiled code helper is that it's being used in // generated C++ code and provides part of the operations implementation. // Currently we also have standalone mode related code here, patches to CPython // runtime that we do, and e.g. the built-in module. TODO: Move these to their // own files for clarity. #include "nuitka/prelude.hpp" extern PyObject *const_str_plain_compile; static PythonBuiltin _python_builtin_compile( &const_str_plain_compile ); #if PYTHON_VERSION < 300 PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit ) #else PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit, PyObject *optimize ) #endif { // May be a source, but also could already be a compiled object, in which // case this should just return it. if ( PyCode_Check( source_code ) ) { return INCREASE_REFCOUNT( source_code ); } PyObject *pos_args = PyTuple_New(3); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( source_code ) ); PyTuple_SET_ITEM( pos_args, 1, INCREASE_REFCOUNT( file_name ) ); PyTuple_SET_ITEM( pos_args, 2, INCREASE_REFCOUNT( mode ) ); PyObject *kw_args = NULL; if ( flags != NULL ) { if ( kw_args == NULL ) kw_args = PyDict_New(); PyDict_SetItemString( kw_args, "flags", flags ); } if ( dont_inherit != NULL ) { if ( kw_args == NULL ) kw_args = PyDict_New(); PyDict_SetItemString( kw_args, "dont_inherit", dont_inherit ); } #if PYTHON_VERSION >= 300 if ( optimize != NULL ) { if ( kw_args == NULL ) kw_args = PyDict_New(); PyDict_SetItemString( kw_args, "optimize", optimize ); } #endif PyObject *result = CALL_FUNCTION( _python_builtin_compile.asObject0(), pos_args, kw_args ); Py_DECREF( pos_args ); Py_XDECREF( kw_args ); return result; } PyObject *EVAL_CODE( PyObject *code, PyObject *globals, PyObject *locals ) { CHECK_OBJECT( code ); CHECK_OBJECT( globals ); CHECK_OBJECT( locals ); if ( PyDict_Check( globals ) == 0 ) { PyErr_Format( PyExc_TypeError, "exec: arg 2 must be a dictionary or None" ); return NULL; } // TODO: Our re-formulation prevents this externally, doesn't it. if ( locals == Py_None ) { locals = globals; } if ( PyMapping_Check( locals ) == 0 ) { PyErr_Format( PyExc_TypeError, "exec: arg 3 must be a mapping or None" ); return NULL; } // Set the __builtins__ in globals, it is expected to be present. if ( PyDict_GetItem( globals, const_str_plain___builtins__ ) == NULL ) { if ( PyDict_SetItem( globals, const_str_plain___builtins__, (PyObject *)builtin_module ) != 0 ) { return NULL; } } #if PYTHON_VERSION < 300 PyObject *result = PyEval_EvalCode( (PyCodeObject *)code, globals, locals ); #else PyObject *result = PyEval_EvalCode( code, globals, locals ); #endif if (unlikely( result == NULL )) { return NULL; } return result; } extern PyObject *const_str_plain_open; static PythonBuiltin _python_builtin_open( &const_str_plain_open ); PyObject *BUILTIN_OPEN( PyObject *file_name, PyObject *mode, PyObject *buffering ) { if ( file_name == NULL ) { return CALL_FUNCTION_NO_ARGS( _python_builtin_open.asObject0() ); } else if ( mode == NULL ) { PyObject *args[] = { file_name }; return CALL_FUNCTION_WITH_ARGS1( _python_builtin_open.asObject0(), args ); } else if ( buffering == NULL ) { PyObject *args[] = { file_name, mode }; return CALL_FUNCTION_WITH_ARGS2( _python_builtin_open.asObject0(), args ); } else { PyObject *args[] = { file_name, mode, buffering }; return CALL_FUNCTION_WITH_ARGS3( _python_builtin_open.asObject0(), args ); } } PyObject *BUILTIN_CHR( unsigned char c ) { // TODO: A switch statement might be faster, because no object needs to be // created at all, this here is how CPython does it. char s[1]; s[0] = (char)c; #if PYTHON_VERSION < 300 return PyString_FromStringAndSize( s, 1 ); #else return PyUnicode_FromStringAndSize( s, 1 ); #endif } PyObject *BUILTIN_CHR( PyObject *value ) { long x = PyInt_AsLong( value ); #if PYTHON_VERSION < 300 if ( x < 0 || x >= 256 ) { PyErr_Format( PyExc_ValueError, "chr() arg not in range(256)" ); return NULL; } // TODO: A switch statement might be faster, because no object needs to be // created at all, this is how CPython does it. char s[1]; s[0] = (char)x; return PyString_FromStringAndSize( s, 1 ); #else PyObject *result = PyUnicode_FromOrdinal( x ); if (unlikely( result == NULL )) { return NULL; } assert( PyUnicode_Check( result )); return result; #endif } PyObject *BUILTIN_ORD( PyObject *value ) { long result; if (likely( PyBytes_Check( value ) )) { Py_ssize_t size = PyBytes_GET_SIZE( value ); if (likely( size == 1 )) { result = long( ((unsigned char *)PyBytes_AS_STRING( value ))[0] ); } else { PyErr_Format( PyExc_TypeError, "ord() expected a character, but string of length %zd found", size ); return NULL; } } else if ( PyByteArray_Check( value ) ) { Py_ssize_t size = PyByteArray_GET_SIZE( value ); if (likely( size == 1 )) { result = long( ((unsigned char *)PyByteArray_AS_STRING( value ))[0] ); } else { PyErr_Format( PyExc_TypeError, "ord() expected a character, but byte array of length %zd found", size ); return NULL; } } else if ( PyUnicode_Check( value ) ) { #if PYTHON_VERSION >= 330 if (unlikely( PyUnicode_READY( value ) == -1 )) { return NULL; } Py_ssize_t size = PyUnicode_GET_LENGTH( value ); #else Py_ssize_t size = PyUnicode_GET_SIZE( value ); #endif if (likely( size == 1 )) { #if PYTHON_VERSION >= 330 result = long( PyUnicode_READ_CHAR( value, 0 ) ); #else result = long( *PyUnicode_AS_UNICODE( value ) ); #endif } else { PyErr_Format( PyExc_TypeError, "ord() expected a character, but unicode string of length %zd found", size ); return NULL; } } else { PyErr_Format( PyExc_TypeError, "ord() expected string of length 1, but %s found", Py_TYPE( value )->tp_name ); return NULL; } return PyInt_FromLong( result ); } PyObject *BUILTIN_BIN( PyObject *value ) { // Note: I don't really know why ord and hex don't use this as well. PyObject *result = PyNumber_ToBase( value, 2 ); if ( unlikely( result == NULL )) { return NULL; } return result; } PyObject *BUILTIN_OCT( PyObject *value ) { #if PYTHON_VERSION >= 300 PyObject *result = PyNumber_ToBase( value, 8 ); if ( unlikely( result == NULL )) { return NULL; } return result; #else if (unlikely( value == NULL )) { PyErr_Format( PyExc_TypeError, "oct() argument can't be converted to oct" ); return NULL; } PyNumberMethods *nb = Py_TYPE( value )->tp_as_number; if (unlikely( nb == NULL || nb->nb_oct == NULL )) { PyErr_Format( PyExc_TypeError, "oct() argument can't be converted to oct" ); return NULL; } PyObject *result = (*nb->nb_oct)( value ); if ( result ) { if (unlikely( !PyString_Check( result ) )) { PyErr_Format( PyExc_TypeError, "__oct__ returned non-string (type %s)", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } } return result; #endif } PyObject *BUILTIN_HEX( PyObject *value ) { #if PYTHON_VERSION >= 300 PyObject *result = PyNumber_ToBase( value, 16 ); if ( unlikely( result == NULL )) { return NULL; } return result; #else if (unlikely( value == NULL )) { PyErr_Format( PyExc_TypeError, "hex() argument can't be converted to hex" ); return NULL; } PyNumberMethods *nb = Py_TYPE( value )->tp_as_number; if (unlikely( nb == NULL || nb->nb_hex == NULL )) { PyErr_Format( PyExc_TypeError, "hex() argument can't be converted to hex" ); return NULL; } PyObject *result = (*nb->nb_hex)( value ); if (likely( result )) { if (unlikely( !PyString_Check( result ) )) { PyErr_Format( PyExc_TypeError, "__hex__ returned non-string (type %s)", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } } return result; #endif } PyObject *BUILTIN_HASH( PyObject *value ) { Py_hash_t hash = PyObject_Hash( value ); if (unlikely( hash == -1 )) { return NULL; } #if PYTHON_VERSION < 300 return PyInt_FromLong( hash ); #else return PyLong_FromSsize_t( hash ); #endif } PyObject *BUILTIN_BYTEARRAY( PyObject *value ) { PyObject *result = PyByteArray_FromObject( value ); if ( unlikely( result == NULL )) { return NULL; } return result; } // From CPython: typedef struct { PyObject_HEAD PyObject *it_callable; PyObject *it_sentinel; } calliterobject; PyObject *BUILTIN_ITER2( PyObject *callable, PyObject *sentinel ) { calliterobject *result = PyObject_GC_New( calliterobject, &PyCallIter_Type ); if (unlikely( result == NULL )) { return NULL; } // Note: References were taken at call site already. result->it_callable = INCREASE_REFCOUNT( callable ); result->it_sentinel = INCREASE_REFCOUNT( sentinel ); Nuitka_GC_Track( result ); return (PyObject *)result; } PyObject *BUILTIN_TYPE1( PyObject *arg ) { return INCREASE_REFCOUNT( (PyObject *)Py_TYPE( arg ) ); } extern PyObject *const_str_plain___module__; PyObject *BUILTIN_TYPE3( PyObject *module_name, PyObject *name, PyObject *bases, PyObject *dict ) { PyObject *pos_args = PyTuple_New(3); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( name ) ); PyTuple_SET_ITEM( pos_args, 1, INCREASE_REFCOUNT( bases ) ); PyTuple_SET_ITEM( pos_args, 2, INCREASE_REFCOUNT( dict ) ); PyObject *result = PyType_Type.tp_new( &PyType_Type, pos_args, NULL ); if (unlikely( result == NULL )) { Py_DECREF( pos_args ); return NULL; } PyTypeObject *type = Py_TYPE( result ); if (likely( PyType_IsSubtype( type, &PyType_Type ) )) { if ( #if PYTHON_VERSION < 300 PyType_HasFeature( type, Py_TPFLAGS_HAVE_CLASS ) && #endif type->tp_init != NULL ) { int res = type->tp_init( result, pos_args, NULL ); if (unlikely( res < 0 )) { Py_DECREF( pos_args ); Py_DECREF( result ); return NULL; } } } Py_DECREF( pos_args ); int res = PyObject_SetAttr( result, const_str_plain___module__, module_name ); if ( res < 0 ) { return NULL; } return result; } Py_ssize_t ESTIMATE_RANGE( long low, long high, long step ) { if ( low >= high ) { return 0; } else { return ( high - low - 1 ) / step + 1; } } #if PYTHON_VERSION < 300 static PyObject *_BUILTIN_RANGE_INT3( long low, long high, long step ) { assert( step != 0 ); Py_ssize_t size; if ( step > 0 ) { size = ESTIMATE_RANGE( low, high, step ); } else { size = ESTIMATE_RANGE( high, low, -step ); } PyObject *result = PyList_New( size ); long current = low; for ( int i = 0; i < size; i++ ) { PyList_SET_ITEM( result, i, PyInt_FromLong( current ) ); current += step; } return result; } static PyObject *_BUILTIN_RANGE_INT2( long low, long high ) { return _BUILTIN_RANGE_INT3( low, high, 1 ); } static PyObject *_BUILTIN_RANGE_INT( long boundary ) { return _BUILTIN_RANGE_INT2( 0, boundary ); } static PyObject *TO_RANGE_ARG( PyObject *value, char const *name ) { if (likely( PyInt_Check( value ) || PyLong_Check( value ) )) { return INCREASE_REFCOUNT( value ); } PyTypeObject *type = Py_TYPE( value ); PyNumberMethods *tp_as_number = type->tp_as_number; // Everything that casts to int is allowed. if ( #if PYTHON_VERSION >= 270 PyFloat_Check( value ) || #endif tp_as_number == NULL || tp_as_number->nb_int == NULL ) { PyErr_Format( PyExc_TypeError, "range() integer %s argument expected, got %s.", name, type->tp_name ); return NULL; } PyObject *result = tp_as_number->nb_int( value ); if (unlikely( result == NULL )) { return NULL; } return result; } #endif extern PyObject *const_str_plain_range; static PythonBuiltin _python_builtin_range( &const_str_plain_range ); PyObject *BUILTIN_RANGE( PyObject *boundary ) { #if PYTHON_VERSION < 300 PyObject *boundary_temp = TO_RANGE_ARG( boundary, "end" ); if (unlikely( boundary_temp == NULL )) { return NULL; } long start = PyInt_AsLong( boundary_temp ); if ( start == -1 && ERROR_OCCURRED() ) { CLEAR_ERROR_OCCURRED(); PyObject *args[] = { boundary_temp }; PyObject *result = CALL_FUNCTION_WITH_ARGS1( _python_builtin_range.asObject0(), args ); Py_DECREF( boundary_temp ); return result; } Py_DECREF( boundary_temp ); return _BUILTIN_RANGE_INT( start ); #else PyObject *args[] = { boundary }; return CALL_FUNCTION_WITH_ARGS1( _python_builtin_range.asObject0(), args ); #endif } PyObject *BUILTIN_RANGE2( PyObject *low, PyObject *high ) { #if PYTHON_VERSION < 300 PyObject *low_temp = TO_RANGE_ARG( low, "start" ); if (unlikely( low_temp == NULL )) { return NULL; } PyObject *high_temp = TO_RANGE_ARG( high, "end" ); if (unlikely( high_temp == NULL )) { Py_DECREF( low_temp ); return NULL; } bool fallback = false; long start = PyInt_AsLong( low_temp ); if (unlikely( start == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } long end = PyInt_AsLong( high_temp ); if (unlikely( end == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } if ( fallback ) { PyObject *pos_args = PyTuple_New( 2 ); PyTuple_SET_ITEM( pos_args, 0, low_temp ); PyTuple_SET_ITEM( pos_args, 1, high_temp ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( _python_builtin_range.asObject0(), pos_args ); Py_DECREF( pos_args ); return result; } else { Py_DECREF( low_temp ); Py_DECREF( high_temp ); return _BUILTIN_RANGE_INT2( start, end ); } #else PyObject *pos_args = PyTuple_New( 2 ); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( low ) ); PyTuple_SET_ITEM( pos_args, 1, INCREASE_REFCOUNT( high ) ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( _python_builtin_range.asObject0(), pos_args ); Py_DECREF( pos_args ); return result; #endif } PyObject *BUILTIN_RANGE3( PyObject *low, PyObject *high, PyObject *step ) { #if PYTHON_VERSION < 300 PyObject *low_temp = TO_RANGE_ARG( low, "start" ); if (unlikely( low_temp == NULL )) { return NULL; } PyObject *high_temp = TO_RANGE_ARG( high, "end" ); if (unlikely( high_temp == NULL )) { Py_DECREF( low_temp ); return NULL; } PyObject *step_temp = TO_RANGE_ARG( step, "step" ); if (unlikely( high_temp == NULL )) { Py_DECREF( low_temp ); Py_DECREF( high_temp ); return NULL; } bool fallback = false; long start = PyInt_AsLong( low_temp ); if (unlikely( start == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } long end = PyInt_AsLong( high_temp ); if (unlikely( end == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } long step_long = PyInt_AsLong( step_temp ); if (unlikely( step_long == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } if ( fallback ) { PyObject *pos_args = PyTuple_New( 3 ); PyTuple_SET_ITEM( pos_args, 0, low_temp ); PyTuple_SET_ITEM( pos_args, 1, high_temp ); PyTuple_SET_ITEM( pos_args, 2, step_temp ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( _python_builtin_range.asObject0(), pos_args ); Py_DECREF( pos_args ); return result; } else { Py_DECREF( low_temp ); Py_DECREF( high_temp ); Py_DECREF( step_temp ); if (unlikely( step_long == 0 )) { PyErr_Format( PyExc_ValueError, "range() step argument must not be zero" ); return NULL; } return _BUILTIN_RANGE_INT3( start, end, step_long ); } #else PyObject *pos_args = PyTuple_New( 3 ); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( low ) ); PyTuple_SET_ITEM( pos_args, 1, INCREASE_REFCOUNT( high ) ); PyTuple_SET_ITEM( pos_args, 2, INCREASE_REFCOUNT( step ) ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( _python_builtin_range.asObject0(), pos_args ); Py_DECREF( pos_args ); return result; #endif } #if PYTHON_VERSION < 300 extern PyObject *const_str_plain_xrange; static PythonBuiltin _python_builtin_xrange( &const_str_plain_xrange ); PyObject *BUILTIN_XRANGE( PyObject *low, PyObject *high, PyObject *step ) { if ( step != NULL ) { PyObject *args[] = { low, high, step }; return CALL_FUNCTION_WITH_ARGS3( _python_builtin_xrange.asObject0(), args ); } else if ( high != NULL ) { PyObject *args[] = { low, high }; return CALL_FUNCTION_WITH_ARGS2( _python_builtin_xrange.asObject0(), args ); } else { PyObject *args[] = { low }; return CALL_FUNCTION_WITH_ARGS1( _python_builtin_xrange.asObject0(), args ); } } #endif PyObject *BUILTIN_LEN( PyObject *value ) { CHECK_OBJECT( value ); Py_ssize_t res = PyObject_Size( value ); if (unlikely( res < 0 && ERROR_OCCURRED() )) { return NULL; } return PyInt_FromSsize_t( res ); } extern PyObject *const_str_plain___import__; static PythonBuiltin _python_builtin_import( &const_str_plain___import__ ); PyObject *IMPORT_MODULE( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items, PyObject *level ) { CHECK_OBJECT( module_name ); CHECK_OBJECT( globals ); CHECK_OBJECT( locals ); CHECK_OBJECT( import_items ); CHECK_OBJECT( level ); PyObject *pos_args = PyTuple_New(5); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( module_name ) ); PyTuple_SET_ITEM( pos_args, 1, INCREASE_REFCOUNT( globals ) ); PyTuple_SET_ITEM( pos_args, 2, INCREASE_REFCOUNT( locals ) ); PyTuple_SET_ITEM( pos_args, 3, INCREASE_REFCOUNT( import_items ) ); PyTuple_SET_ITEM( pos_args, 4, INCREASE_REFCOUNT( level ) ); PyObject *import_result = CALL_FUNCTION_WITH_POSARGS( _python_builtin_import.asObject0(), pos_args ); Py_DECREF( pos_args ); return import_result; } extern PyObject *const_str_plain___all__; bool IMPORT_MODULE_STAR( PyObject *target, bool is_module, PyObject *module ) { // Check parameters. CHECK_OBJECT( module ); CHECK_OBJECT( target ); PyObject *iter; bool all_case; PyObject *all = PyObject_GetAttr( module, const_str_plain___all__ ); if ( all != NULL ) { iter = MAKE_ITERATOR( all ); Py_DECREF( all ); if (unlikely( iter == NULL )) { return false; } all_case = true; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); iter = MAKE_ITERATOR( PyModule_GetDict( module ) ); CHECK_OBJECT( iter ); all_case = false; } else { return false; } while ( PyObject *item = ITERATOR_NEXT( iter ) ) { assert( Nuitka_String_Check( item ) ); // TODO: Not yet clear, what happens with __all__ and "_" of its // contents. if ( all_case == false ) { if ( Nuitka_String_AsString_Unchecked( item )[0] == '_' ) { continue; } } // TODO: What if it isn't there, because of e.g. wrong __all__ value. PyObject *value = LOOKUP_ATTRIBUTE( module, item ); // TODO: Check if the reference is handled correctly if ( is_module ) { SET_ATTRIBUTE( target, item, value ); } else { SET_SUBSCRIPT( target, item, value ); } Py_DECREF( value ); Py_DECREF( item ); } Py_DECREF( iter ); return !ERROR_OCCURRED(); } // Helper functions for print. Need to play nice with Python softspace // behaviour. #if PYTHON_VERSION >= 300 extern PyObject *const_str_plain_print; extern PyObject *const_str_plain_end; extern PyObject *const_str_plain_file; extern PyObject *const_str_empty; static PythonBuiltin _python_builtin_print( &const_str_plain_print ); #endif bool PRINT_NEW_LINE_TO( PyObject *file ) { #if PYTHON_VERSION < 300 if ( file == NULL || file == Py_None ) { file = GET_STDOUT(); } // need to hold a reference to the file or else __getattr__ may release // "file" in the mean time. Py_INCREF( file ); if (unlikely( PyFile_WriteString( "\n", file ) == -1)) { Py_DECREF( file ); return false; } PyFile_SoftSpace( file, 0 ); CHECK_OBJECT( file ); Py_DECREF( file ); return true; #else if (likely( file == NULL )) { PyObject *result = CALL_FUNCTION_NO_ARGS( _python_builtin_print.asObject0() ); Py_XDECREF( result ); return result != NULL; } else { PyObject *kw_args = PyDict_New(); PyDict_SetItem( kw_args, const_str_plain_file, GET_STDOUT() ); PyObject *result = CALL_FUNCTION_WITH_KEYARGS( _python_builtin_print.asObject0(), kw_args ); Py_DECREF( kw_args ); Py_XDECREF( result ); return result != NULL; } #endif } bool PRINT_ITEM_TO( PyObject *file, PyObject *object ) { // The print built-in function cannot replace "softspace" behavior of CPython // print statement, so this code is really necessary. #if PYTHON_VERSION < 300 if ( file == NULL || file == Py_None ) { file = GET_STDOUT(); } CHECK_OBJECT( file ); CHECK_OBJECT( object ); // need to hold a reference to the file or else "__getattr__" code may // release "file" in the mean time. Py_INCREF( file ); // Check for soft space indicator if ( PyFile_SoftSpace( file, 0 ) ) { if (unlikely( PyFile_WriteString( " ", file ) == -1 )) { Py_DECREF( file ); return false; } } if ( unlikely( PyFile_WriteObject( object, file, Py_PRINT_RAW ) == -1 )) { Py_DECREF( file ); return false; } if ( PyString_Check( object ) ) { char *buffer; Py_ssize_t length; #ifndef __NUITKA_NO_ASSERT__ int status = #endif PyString_AsStringAndSize( object, &buffer, &length ); assert( status != -1 ); if ( length == 0 || !isspace( Py_CHARMASK(buffer[length-1])) || buffer[ length - 1 ] == ' ' ) { PyFile_SoftSpace( file, 1 ); } } else if ( PyUnicode_Check( object ) ) { Py_UNICODE *buffer = PyUnicode_AS_UNICODE( object ); Py_ssize_t length = PyUnicode_GET_SIZE( object ); if ( length == 0 || !Py_UNICODE_ISSPACE( buffer[length-1]) || buffer[length-1] == ' ' ) { PyFile_SoftSpace( file, 1 ); } } else { PyFile_SoftSpace( file, 1 ); } CHECK_OBJECT( file ); Py_DECREF( file ); return true; #else if (likely( file == NULL )) { PyObject *args[] = { object }; PyObject *result = CALL_FUNCTION_WITH_ARGS1( _python_builtin_print.asObject0(), args ); Py_XDECREF( result ); return result != NULL; } else { PyObject *print_kw = PyDict_New(); PyDict_SetItem( print_kw, const_str_plain_end, const_str_empty ); PyDict_SetItem( print_kw, const_str_plain_file, GET_STDOUT() ); PyObject *print_args = PyTuple_New( 1 ); PyTuple_SET_ITEM( print_args, 0, INCREASE_REFCOUNT( object ) ); PyObject *res = CALL_FUNCTION( _python_builtin_print.asObject0(), print_args, print_kw ); Py_DECREF( print_args ); Py_DECREF( print_kw ); Py_XDECREF( res ); return res != NULL; } #endif } void PRINT_REFCOUNT( PyObject *object ) { char buffer[ 1024 ]; sprintf( buffer, " refcnt %" PY_FORMAT_SIZE_T "d ", Py_REFCNT( object ) ); PRINT_STRING( buffer ); } bool PRINT_STRING( char const *str ) { int res = PyFile_WriteString( str, GET_STDOUT() ); return res != -1; } bool PRINT_REPR( PyObject *object ) { bool res; if ( object != NULL ) { PyObject *repr = PyObject_Repr( object ); res = PRINT_ITEM( repr ); Py_DECREF( repr ); } else { res = PRINT_NULL(); } return res; } bool PRINT_NULL( void ) { return PRINT_STRING(""); } void PRINT_EXCEPTION( PyObject *exception_type, PyObject *exception_value, PyObject *exception_tb ) { PRINT_REPR( exception_type ); PRINT_STRING("|"); PRINT_REPR( exception_value ); #if PYTHON_VERSION >= 300 if ( exception_value != NULL ) { PRINT_STRING(" <- "); PRINT_REPR( PyException_GetContext( exception_value ) ); } #endif PRINT_STRING("|"); PRINT_REPR( exception_tb ); PRINT_NEW_LINE(); } // TODO: Could be ported, the "printf" stuff would need to be split. On Python3 // the normal C print output gets lost. #if PYTHON_VERSION < 300 void PRINT_TRACEBACK( PyTracebackObject *traceback ) { PRINT_STRING("Dumping traceback:\n"); if ( traceback == NULL ) PRINT_STRING( "\n" ); while( traceback ) { printf( " line %d (frame object chain):\n", traceback->tb_lineno ); PyFrameObject *frame = traceback->tb_frame; while ( frame ) { printf( " Frame at %s\n", PyString_AsString( PyObject_Str( (PyObject *)frame->f_code ))); frame = frame->f_back; } assert( traceback->tb_next != traceback ); traceback = traceback->tb_next; } PRINT_STRING("End of Dump.\n"); } #endif PyObject *GET_STDOUT() { PyObject *result = PySys_GetObject( (char *)"stdout" ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_RuntimeError, "lost sys.stdout" ); return NULL; } return result; } PyObject *GET_STDERR() { PyObject *result = PySys_GetObject( (char *)"stderr" ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_RuntimeError, "lost sys.stderr" ); return NULL; } return result; } bool PRINT_NEW_LINE( void ) { PyObject *target = GET_STDOUT(); return target != NULL && PRINT_NEW_LINE_TO( target ); } bool PRINT_ITEM( PyObject *object ) { PyObject *target = GET_STDOUT(); return target != NULL && PRINT_ITEM_TO( target, object ); } PyObject *UNSTREAM_CONSTANT( unsigned char const *buffer, Py_ssize_t size ) { assert( buffer ); // We unstream difficult constant objects using the "pickle" module, this is // aimed at being the exception, e.g. unicode that doesn't fit into UTF-8 // will be dealt with like this. static PyObject *module_pickle = NULL; if ( module_pickle == NULL ) { #if PYTHON_VERSION < 300 module_pickle = PyImport_ImportModule( "cPickle" ); #else module_pickle = PyImport_ImportModule( "pickle" ); #endif if (unlikely( module_pickle == NULL )) { PyErr_Print(); } assert( module_pickle ); } static PyObject *function_pickle_loads = NULL; if ( function_pickle_loads == NULL ) { function_pickle_loads = PyObject_GetAttrString( module_pickle, "loads" ); if (unlikely( function_pickle_loads == NULL )) { PyErr_Print(); } assert( function_pickle_loads ); } PyObject *result = PyObject_CallFunction( function_pickle_loads, #if PYTHON_VERSION < 300 (char *)"(s#)", // TODO: Why the () #else (char *)"y#", #endif buffer, size ); if (unlikely( result == NULL )) { PyErr_Print(); } CHECK_OBJECT( result ); return result; } #if PYTHON_VERSION < 300 PyObject *UNSTREAM_UNICODE( unsigned char const *buffer, Py_ssize_t size ) { PyObject *result = PyUnicode_FromStringAndSize( (char const *)buffer, size ); assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); return result; } #endif PyObject *UNSTREAM_STRING( unsigned char const *buffer, Py_ssize_t size, bool intern ) { #if PYTHON_VERSION < 300 PyObject *result = PyString_FromStringAndSize( (char const *)buffer, size ); #else PyObject *result = PyUnicode_FromStringAndSize( (char const *)buffer, size ); #endif assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == size ); #endif if ( intern ) { Nuitka_StringIntern( &result ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == size ); #else assert( PyUnicode_GET_SIZE( result ) == size ); #endif } return result; } PyObject *UNSTREAM_CHAR( unsigned char value, bool intern ) { #if PYTHON_VERSION < 300 PyObject *result = PyString_FromStringAndSize( (char const *)&value, 1 ); #else PyObject *result = PyUnicode_FromStringAndSize( (char const *)&value, 1 ); #endif assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == 1 ); #else assert( PyUnicode_GET_SIZE( result ) == 1 ); #endif if ( intern ) { Nuitka_StringIntern( &result ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == 1 ); #else assert( PyUnicode_GET_SIZE( result ) == 1 ); #endif } return result; } PyObject *UNSTREAM_FLOAT( unsigned char const *buffer ) { double x = _PyFloat_Unpack8( buffer, 1 ); assert( x != -1.0 || !PyErr_Occurred() ); PyObject *result = PyFloat_FromDouble(x); assert( result != NULL ); return result; } #if PYTHON_VERSION >= 300 PyObject *UNSTREAM_BYTES( unsigned char const *buffer, Py_ssize_t size ) { PyObject *result = PyBytes_FromStringAndSize( (char const *)buffer, size ); assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( PyBytes_GET_SIZE( result ) == size ); return result; } #endif #if PYTHON_VERSION < 300 static void set_slot( PyObject **slot, PyObject *value ) { PyObject *temp = *slot; Py_XINCREF( value ); *slot = value; Py_XDECREF( temp ); } extern PyObject *const_str_plain___getattr__; extern PyObject *const_str_plain___setattr__; extern PyObject *const_str_plain___delattr__; static void set_attr_slots( PyClassObject *klass ) { set_slot( &klass->cl_getattr, FIND_ATTRIBUTE_IN_CLASS( klass, const_str_plain___getattr__ ) ); set_slot( &klass->cl_setattr, FIND_ATTRIBUTE_IN_CLASS( klass, const_str_plain___setattr__ ) ); set_slot( &klass->cl_delattr, FIND_ATTRIBUTE_IN_CLASS( klass, const_str_plain___delattr__ ) ); } static bool set_dict( PyClassObject *klass, PyObject *value ) { if ( value == NULL || !PyDict_Check( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__dict__ must be a dictionary object" ); return false; } else { set_slot( &klass->cl_dict, value ); set_attr_slots( klass ); return true; } } static bool set_bases( PyClassObject *klass, PyObject *value ) { if ( value == NULL || !PyTuple_Check( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__bases__ must be a tuple object" ); return false; } else { Py_ssize_t n = PyTuple_Size( value ); for ( Py_ssize_t i = 0; i < n; i++ ) { PyObject *base = PyTuple_GET_ITEM( value, i ); if (unlikely( !PyClass_Check( base ) )) { PyErr_SetString( PyExc_TypeError, (char *)"__bases__ items must be classes" ); return false; } if (unlikely( PyClass_IsSubclass( base, (PyObject *)klass) )) { PyErr_SetString( PyExc_TypeError, (char *)"a __bases__ item causes an inheritance cycle" ); return false; } } set_slot( &klass->cl_bases, value ); set_attr_slots( klass ); return true; } } static bool set_name( PyClassObject *klass, PyObject *value ) { if ( value == NULL || !PyDict_Check( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__name__ must be a string object" ); return false; } if ( strlen( PyString_AS_STRING( value )) != (size_t)PyString_GET_SIZE( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__name__ must not contain null bytes" ); return false; } set_slot( &klass->cl_name, value ); return true; } static int nuitka_class_setattr( PyClassObject *klass, PyObject *attr_name, PyObject *value ) { char *sattr_name = PyString_AsString( attr_name ); if ( sattr_name[0] == '_' && sattr_name[1] == '_' ) { Py_ssize_t n = PyString_Size( attr_name ); if ( sattr_name[ n-2 ] == '_' && sattr_name[ n-1 ] == '_' ) { if ( strcmp( sattr_name, "__dict__" ) == 0 ) { if ( set_dict( klass, value ) == false ) { return -1; } else { return 0; } } else if ( strcmp( sattr_name, "__bases__" ) == 0 ) { if ( set_bases( klass, value ) == false ) { return -1; } else { return 0; } } else if ( strcmp( sattr_name, "__name__" ) == 0 ) { if ( set_name( klass, value ) == false ) { return -1; } else { return 0; } } else if ( strcmp( sattr_name, "__getattr__" ) == 0 ) { set_slot( &klass->cl_getattr, value ); } else if ( strcmp(sattr_name, "__setattr__" ) == 0 ) { set_slot( &klass->cl_setattr, value ); } else if ( strcmp(sattr_name, "__delattr__" ) == 0 ) { set_slot( &klass->cl_delattr, value ); } } } if ( value == NULL ) { int status = PyDict_DelItem( klass->cl_dict, attr_name ); if ( status < 0 ) { PyErr_Format( PyExc_AttributeError, "class %s has no attribute '%s'", PyString_AS_STRING( klass->cl_name ), sattr_name ); } return status; } else { return PyDict_SetItem( klass->cl_dict, attr_name, value ); } } static PyObject *nuitka_class_getattr( PyClassObject *klass, PyObject *attr_name ) { char *sattr_name = PyString_AsString( attr_name ); if ( sattr_name[0] == '_' && sattr_name[1] == '_' ) { if ( strcmp( sattr_name, "__dict__" ) == 0 ) { return INCREASE_REFCOUNT( klass->cl_dict ); } else if ( strcmp(sattr_name, "__bases__" ) == 0 ) { return INCREASE_REFCOUNT( klass->cl_bases ); } else if ( strcmp(sattr_name, "__name__" ) == 0 ) { return klass->cl_name ? INCREASE_REFCOUNT( klass->cl_name ) : INCREASE_REFCOUNT( Py_None ); } } PyObject *value = FIND_ATTRIBUTE_IN_CLASS( klass, attr_name ); if (unlikely( value == NULL )) { PyErr_Format( PyExc_AttributeError, "class %s has no attribute '%s'", PyString_AS_STRING( klass->cl_name ), sattr_name ); return NULL; } PyTypeObject *type = Py_TYPE( value ); descrgetfunc tp_descr_get = PyType_HasFeature( type, Py_TPFLAGS_HAVE_CLASS ) ? type->tp_descr_get : NULL; if ( tp_descr_get == NULL ) { return INCREASE_REFCOUNT( value ); } else { return tp_descr_get( value, (PyObject *)NULL, (PyObject *)klass ); } } #endif void enhancePythonTypes( void ) { #if PYTHON_VERSION < 300 // Our own variant won't call PyEval_GetRestricted, saving quite some cycles // not doing that. PyClass_Type.tp_setattro = (setattrofunc)nuitka_class_setattr; PyClass_Type.tp_getattro = (getattrofunc)nuitka_class_getattr; #endif } #ifdef __APPLE__ extern "C" wchar_t* _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size); #endif #ifdef __FreeBSD__ #include #endif #include #if PYTHON_VERSION >= 300 argv_type_t convertCommandLineParameters( int argc, char **argv ) { #if _WIN32 int new_argc; argv_type_t result = CommandLineToArgvW( GetCommandLineW(), &new_argc ); assert( new_argc == argc ); return result; #else // Originally taken from CPython3: There seems to be no sane way to use static wchar_t **argv_copy; argv_copy = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*) * argc); // Temporarily disable locale for conversions to not use it. char *oldloc = strdup( setlocale( LC_ALL, NULL ) ); setlocale( LC_ALL, "" ); for ( int i = 0; i < argc; i++ ) { #ifdef __APPLE__ argv_copy[i] = _Py_DecodeUTF8_surrogateescape( argv[ i ], strlen( argv[ i ] ) ); #elif PYTHON_VERSION < 350 argv_copy[i] = _Py_char2wchar( argv[ i ], NULL ); #else argv_copy[i] = Py_DecodeLocale( argv[ i ], NULL ); #endif assert ( argv_copy[ i ] ); } setlocale( LC_ALL, oldloc ); free( oldloc ); return argv_copy; #endif } #endif bool setCommandLineParameters( int argc, argv_type_t argv, bool initial ) { bool is_multiprocessing_fork = false; if ( initial ) { // We need to skip what multiprocessing has told Python otherwise. for ( int i = 1; i < argc; i++ ) { #if PYTHON_VERSION < 300 if ((strcmp(argv[i], "--multiprocessing-fork")) == 0 && (i+1 < argc)) #else wchar_t constant_buffer[100]; mbstowcs( constant_buffer, "--multiprocessing-fork", 100 ); if ((wcscmp(argv[i], constant_buffer )) == 0 && (i+1 < argc)) #endif { is_multiprocessing_fork = true; break; } } } if ( initial ) { Py_SetProgramName( argv[0] ); } else { PySys_SetArgv( argc, argv ); } return is_multiprocessing_fork; } typedef struct { PyObject_HEAD PyTypeObject *type; PyObject *obj; PyTypeObject *obj_type; } superobject; extern PyObject *const_str_plain___class__; PyObject *BUILTIN_SUPER( PyObject *type, PyObject *object ) { CHECK_OBJECT( type ); superobject *result = PyObject_GC_New( superobject, &PySuper_Type ); assert( result ); if ( object == Py_None ) { object = NULL; } if (unlikely( PyType_Check( type ) == false )) { PyErr_Format( PyExc_RuntimeError, "super(): __class__ is not a type (%s)", Py_TYPE( type )->tp_name ); return NULL; } result->type = (PyTypeObject *)INCREASE_REFCOUNT( type ); if ( object ) { result->obj = INCREASE_REFCOUNT( object ); if ( PyType_Check( object ) && PyType_IsSubtype( (PyTypeObject *)object, (PyTypeObject *)type )) { result->obj_type = (PyTypeObject *)INCREASE_REFCOUNT( object ); } else if ( PyType_IsSubtype( Py_TYPE(object ), (PyTypeObject *)type) ) { result->obj_type = (PyTypeObject *)INCREASE_REFCOUNT( (PyObject *)Py_TYPE( object ) ); } else { PyObject *class_attr = PyObject_GetAttr( object, const_str_plain___class__); if (likely( class_attr != NULL && PyType_Check( class_attr ) && (PyTypeObject *)class_attr != Py_TYPE( object ) )) { result->obj_type = (PyTypeObject *)class_attr; } else { if ( class_attr == NULL ) { CLEAR_ERROR_OCCURRED(); } else { Py_DECREF( class_attr ); } PyErr_Format( PyExc_TypeError, "super(type, obj): obj must be an instance or subtype of type" ); return NULL; } } } else { result->obj = NULL; result->obj_type = NULL; } Nuitka_GC_Track( result ); CHECK_OBJECT( (PyObject *)result ); assert( Py_TYPE( result ) == &PySuper_Type ); return (PyObject *)result; } PyObject *BUILTIN_CALLABLE( PyObject *value ) { return PyBool_FromLong( (long)PyCallable_Check( value ) ); } PyObject *original_isinstance = NULL; // Note: Installed and used by "InspectPatcher". int Nuitka_IsInstance( PyObject *inst, PyObject *cls ) { CHECK_OBJECT( original_isinstance ); CHECK_OBJECT( inst ); CHECK_OBJECT( cls ); // Quick path. if ( Py_TYPE( inst ) == (PyTypeObject *)cls ) { return true; } // Our paths for the types we need to hook. if ( cls == (PyObject *)&PyFunction_Type && Nuitka_Function_Check( inst ) ) { return true; } if ( cls == (PyObject *)&PyGen_Type && Nuitka_Generator_Check( inst ) ) { return true; } if ( cls == (PyObject *)&PyMethod_Type && Nuitka_Method_Check( inst ) ) { return true; } if ( cls == (PyObject *)&PyFrame_Type && Nuitka_Frame_Check( inst ) ) { return true; } #if PYTHON_VERSION >= 350 if ( cls == (PyObject *)&PyCoro_Type && Nuitka_Coroutine_Check( inst ) ) { return true; } #endif // May need to be recursive for tuple arguments. if ( PyTuple_Check( cls ) ) { for ( Py_ssize_t i = 0, size = PyTuple_GET_SIZE( cls ); i < size; i++ ) { PyObject *element = PyTuple_GET_ITEM( cls, i ); if (unlikely( Py_EnterRecursiveCall( (char *)" in __instancecheck__" ) )) { return -1; } int res = Nuitka_IsInstance( inst, element ); Py_LeaveRecursiveCall(); if ( res != 0 ) { return res; } } return 0; } else { PyObject *args[] = { inst, cls }; PyObject *result = CALL_FUNCTION_WITH_ARGS2( original_isinstance, args ); if ( result == NULL ) { return -1; } int res = PyObject_IsTrue( result ); Py_DECREF( result ); return res; } } PyObject *BUILTIN_ISINSTANCE( PyObject *inst, PyObject *cls ) { int res = Nuitka_IsInstance( inst, cls ); if (unlikely( res < 0 )) { return NULL; } return BOOL_FROM( res != 0 ); } PyObject *BUILTIN_GETATTR( PyObject *object, PyObject *attribute, PyObject *default_value ) { #if PYTHON_VERSION < 300 if ( PyUnicode_Check( attribute ) ) { attribute = _PyUnicode_AsDefaultEncodedString( attribute, NULL ); if (unlikely( attribute == NULL )) { return NULL; } } if (unlikely( !PyString_Check( attribute ) )) { PyErr_Format( PyExc_TypeError, "getattr(): attribute name must be string" ); return NULL; } #else if (!PyUnicode_Check( attribute )) { PyErr_Format( PyExc_TypeError, "getattr(): attribute name must be string" ); return NULL; } #endif PyObject *result = PyObject_GetAttr( object, attribute ); if ( result == NULL ) { if ( default_value != NULL && EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); return INCREASE_REFCOUNT( default_value ); } else { return NULL; } } else { return result; } } PyObject *BUILTIN_SETATTR( PyObject *object, PyObject *attribute, PyObject *value ) { int res = PyObject_SetAttr( object, attribute, value ); if ( res < 0 ) { return NULL; } return BOOL_FROM( res == 0 ); } PyDictObject *dict_builtin = NULL; PyModuleObject *builtin_module = NULL; #define ASSIGN_BUILTIN( name ) _python_original_builtin_value_##name = LOOKUP_BUILTIN( const_str_plain_##name ); static PyTypeObject Nuitka_BuiltinModule_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_module", // tp_name sizeof(PyModuleObject), // tp_size }; extern PyObject *const_str_plain_open; int Nuitka_BuiltinModule_SetAttr( PyModuleObject *module, PyObject *name, PyObject *value ) { CHECK_OBJECT( (PyObject *)module ); CHECK_OBJECT( name ); // This is used for "del" as well. assert( value == NULL || Py_REFCNT( value ) > 0 ); // only checks the builtins that we can refresh at this time, if we have // many value to check maybe need create a dict first. bool found = false; int res = PyObject_RichCompareBool( name, const_str_plain_open, Py_EQ ); if (unlikely( res == -1 )) { return -1; } if ( res == 1 ) { _python_builtin_open.update( value ); found = true; } if ( found == false ) { res = PyObject_RichCompareBool( name, const_str_plain___import__, Py_EQ ); if (unlikely( res == -1 )) { return -1; } if ( res == 1 ) { _python_builtin_import.update( value ); found = true; } } #if PYTHON_VERSION >= 300 if ( found == false ) { res = PyObject_RichCompareBool( name, const_str_plain_print, Py_EQ ); if (unlikely( res == -1 )) { return -1; } if ( res == 1 ) { _python_builtin_print.update( value ); found = true; } } #endif return PyObject_GenericSetAttr( (PyObject *)module, name, value ); } #include #if defined(_WIN32) #include #elif defined(__APPLE__) #include #include #include #else #include #include #endif #if defined(_WIN32) && !defined(PATH_MAX) #define PATH_MAX MAXPATHLEN #endif #if defined( __FreeBSD__ ) #include #endif #if defined(_NUITKA_EXE) char *getBinaryDirectoryUTF8Encoded() { static char binary_directory[ PATH_MAX + 1 ]; static bool init_done = false; if ( init_done ) { return binary_directory; } #if defined(_WIN32) #if PYTHON_VERSION >= 300 WCHAR binary_directory2[ PATH_MAX + 1 ]; binary_directory2[0] = 0; DWORD res = GetModuleFileNameW( NULL, binary_directory2, PATH_MAX + 1 ); assert( res != 0 ); int res2 = WideCharToMultiByte( CP_UTF8, 0, binary_directory2, -1, binary_directory, PATH_MAX+1, NULL, NULL ); assert( res2 != 0 ); #else DWORD res = GetModuleFileName( NULL, binary_directory, PATH_MAX + 1 ); assert( res != 0 ); #endif PathRemoveFileSpec( binary_directory ); #elif defined(__APPLE__) uint32_t bufsize = PATH_MAX + 1; int res =_NSGetExecutablePath( binary_directory, &bufsize ); if (unlikely( res != 0 )) { abort(); } // On MacOS, the "dirname" call creates a separate internal string, we can // safely copy back. strncpy(binary_directory, dirname(binary_directory), PATH_MAX + 1); #elif defined( __FreeBSD__ ) // Not all of FreeBSD has /proc file system, so use the appropiate // "sysctl" instead. int mib[4]; mib[0] = CTL_KERN; mib[1] = KERN_PROC; mib[2] = KERN_PROC_PATHNAME; mib[3] = -1; size_t cb = sizeof(binary_directory); sysctl(mib, 4, binary_directory, &cb, NULL, 0); #else // The remaining platforms, mostly Linux. // The "readlink" does not terminate result, so fill zeros there, then // it is a proper C string right away. memset( binary_directory, 0, PATH_MAX + 1 ); ssize_t res = readlink( "/proc/self/exe", binary_directory, PATH_MAX ); if (unlikely( res == -1 )) { abort(); } strcpy( binary_directory, dirname( binary_directory ) ); #endif init_done = true; return binary_directory; } char *getBinaryDirectoryHostEncoded() { #if defined(_WIN32) static char binary_directory[ PATH_MAX + 1 ]; static bool init_done = false; if ( init_done ) { return binary_directory; } #if PYTHON_VERSION >= 300 WCHAR binary_directory2[ PATH_MAX + 1 ]; binary_directory2[0] = 0; DWORD res = GetModuleFileNameW( NULL, binary_directory2, PATH_MAX + 1 ); assert( res != 0 ); int res2 = WideCharToMultiByte( CP_ACP, 0, binary_directory2, -1, binary_directory, PATH_MAX+1, NULL, NULL ); assert( res2 != 0 ); #else DWORD res = GetModuleFileName( NULL, binary_directory, PATH_MAX + 1 ); assert( res != 0 ); #endif PathRemoveFileSpec( binary_directory ); init_done = true; return binary_directory; #else return getBinaryDirectoryUTF8Encoded(); #endif } static PyObject *getBinaryDirectoryObject() { static PyObject *binary_directory = NULL; if ( binary_directory != NULL ) { return binary_directory; } #if PYTHON_VERSION >= 300 binary_directory = PyUnicode_FromString( getBinaryDirectoryUTF8Encoded() ); #else binary_directory = PyString_FromString( getBinaryDirectoryUTF8Encoded() ); #endif if (unlikely( binary_directory == NULL )) { PyErr_Print(); abort(); } return binary_directory; } #else static char *getDllDirectory() { #if defined(_WIN32) static char path[ PATH_MAX ]; HMODULE hm = NULL; path[0] = '\0'; #if PYTHON_VERSION >= 300 WCHAR path2[ PATH_MAX + 1 ]; path2[0] = 0; int res = GetModuleHandleExA( GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR) &getDllDirectory, &hm ); assert( res != 0 ); int res2 = WideCharToMultiByte(CP_UTF8, 0, path2, -1, path, PATH_MAX+1, NULL, NULL); assert( res2 != 0 ); #else int res = GetModuleHandleExA( GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR) &getDllDirectory, &hm ); assert( res != 0 ); #endif PathRemoveFileSpec( path ); return path; #else Dl_info where; int res = dladdr( (void *)getDllDirectory, &where ); assert( res != 0 ); return dirname( (char *)where.dli_fname ); #endif } #endif void _initBuiltinModule() { #if _NUITKA_MODULE if ( builtin_module ) return; #else assert( builtin_module == NULL ); #endif #if PYTHON_VERSION < 300 builtin_module = (PyModuleObject *)PyImport_ImportModule( "__builtin__" ); #else builtin_module = (PyModuleObject *)PyImport_ImportModule( "builtins" ); #endif assert( builtin_module ); dict_builtin = (PyDictObject *)builtin_module->md_dict; assert( PyDict_Check( dict_builtin ) ); #ifdef _NUITKA_STANDALONE int res = PyDict_SetItemString( (PyObject *)dict_builtin, "__nuitka_binary_dir", getBinaryDirectoryObject() ); assert( res == 0 ); #endif // init Nuitka_BuiltinModule_Type, PyType_Ready wont copy all member from // base type, so we need copy all members from PyModule_Type manual for // safety. PyType_Ready will change tp_flags, we need define it again. Set // tp_setattro to Nuitka_BuiltinModule_SetAttr and we can detect value // change. Set tp_base to PyModule_Type and PyModule_Check will pass. Nuitka_BuiltinModule_Type.tp_dealloc = PyModule_Type.tp_dealloc; Nuitka_BuiltinModule_Type.tp_repr = PyModule_Type.tp_repr; Nuitka_BuiltinModule_Type.tp_setattro = (setattrofunc)Nuitka_BuiltinModule_SetAttr; Nuitka_BuiltinModule_Type.tp_getattro = PyModule_Type.tp_getattro; Nuitka_BuiltinModule_Type.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE; Nuitka_BuiltinModule_Type.tp_doc = PyModule_Type.tp_doc; Nuitka_BuiltinModule_Type.tp_traverse = PyModule_Type.tp_traverse; Nuitka_BuiltinModule_Type.tp_members = PyModule_Type.tp_members; Nuitka_BuiltinModule_Type.tp_base = &PyModule_Type; Nuitka_BuiltinModule_Type.tp_dictoffset = PyModule_Type.tp_dictoffset; Nuitka_BuiltinModule_Type.tp_init = PyModule_Type.tp_init; Nuitka_BuiltinModule_Type.tp_alloc = PyModule_Type.tp_alloc; Nuitka_BuiltinModule_Type.tp_new = PyModule_Type.tp_new; Nuitka_BuiltinModule_Type.tp_free = PyModule_Type.tp_free; int ret = PyType_Ready( &Nuitka_BuiltinModule_Type ); assert( ret == 0 ); // Replace type of builtin module to take over. ((PyObject *)builtin_module)->ob_type = &Nuitka_BuiltinModule_Type; assert( PyModule_Check( builtin_module ) == 1 ); } PyObject *callPythonFunction( PyObject *func, PyObject **args, int count ) { PyCodeObject *co = (PyCodeObject *)PyFunction_GET_CODE( func ); PyObject *globals = PyFunction_GET_GLOBALS( func ); PyObject *argdefs = PyFunction_GET_DEFAULTS( func ); #if PYTHON_VERSION >= 300 PyObject *kwdefs = PyFunction_GET_KW_DEFAULTS( func ); if ( kwdefs == NULL && argdefs == NULL && co->co_argcount == count && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #else if ( argdefs == NULL && co->co_argcount == count && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #endif { PyThreadState *tstate = PyThreadState_GET(); CHECK_OBJECT( globals ); PyFrameObject *frame = PyFrame_New( tstate, co, globals, NULL ); if (unlikely( frame == NULL )) { return NULL; }; for ( int i = 0; i < count; i++ ) { frame->f_localsplus[i] = args[i]; Py_INCREF( frame->f_localsplus[i] ); } PyObject *result = PyEval_EvalFrameEx( frame, 0 ); // Frame release protects against recursion as it may lead to variable // destruction. ++tstate->recursion_depth; Py_DECREF( frame ); --tstate->recursion_depth; return result; } PyObject **defaults = NULL; int nd = 0; if ( argdefs != NULL ) { defaults = &PyTuple_GET_ITEM( argdefs, 0 ); nd = int( Py_SIZE( argdefs ) ); } PyObject *result = PyEval_EvalCodeEx( #if PYTHON_VERSION >= 300 (PyObject *)co, #else co, // code object #endif globals, // globals NULL, // no locals args, // args count, // argcount NULL, // kwds 0, // kwcount defaults, // defaults nd, // defcount #if PYTHON_VERSION >= 300 kwdefs, #endif PyFunction_GET_CLOSURE( func ) ); return result; } static PyObject *_fast_function_noargs( PyObject *func ) { PyCodeObject *co = (PyCodeObject *)PyFunction_GET_CODE( func ); PyObject *globals = PyFunction_GET_GLOBALS( func ); PyObject *argdefs = PyFunction_GET_DEFAULTS( func ); #if PYTHON_VERSION >= 300 PyObject *kwdefs = PyFunction_GET_KW_DEFAULTS( func ); if ( kwdefs == NULL && argdefs == NULL && co->co_argcount == 0 && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #else if ( argdefs == NULL && co->co_argcount == 0 && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #endif { PyThreadState *tstate = PyThreadState_GET(); CHECK_OBJECT( globals ); PyFrameObject *frame = PyFrame_New( tstate, co, globals, NULL ); if (unlikely( frame == NULL )) { return NULL; }; PyObject *result = PyEval_EvalFrameEx( frame, 0 ); // Frame release protects against recursion as it may lead to variable // destruction. ++tstate->recursion_depth; Py_DECREF( frame ); --tstate->recursion_depth; return result; } PyObject **defaults = NULL; int nd = 0; if ( argdefs != NULL ) { defaults = &PyTuple_GET_ITEM( argdefs, 0 ); nd = int( Py_SIZE( argdefs ) ); } PyObject *result = PyEval_EvalCodeEx( #if PYTHON_VERSION >= 300 (PyObject *)co, #else co, // code object #endif globals, // globals NULL, // no locals NULL, // args 0, // argcount NULL, // kwds 0, // kwcount defaults, // defaults nd, // defcount #if PYTHON_VERSION >= 300 kwdefs, #endif PyFunction_GET_CLOSURE( func ) ); return result; } PyObject *CALL_FUNCTION_NO_ARGS( PyObject *called ) { CHECK_OBJECT( called ); if ( Nuitka_Function_Check( called ) ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } Nuitka_FunctionObject *function = (Nuitka_FunctionObject *)called; PyObject *result; if ( function->m_args_simple && 0 == function->m_args_positional_count ) { result = function->m_c_code( function, NULL ); } else if ( function->m_args_simple && function->m_defaults_given == function->m_args_positional_count ) { PyObject **python_pars = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = 0; i < function->m_defaults_given; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsPos( function, python_pars, NULL, 0 ) ) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } else if ( Nuitka_Method_Check( called ) ) { Nuitka_MethodObject *method = (Nuitka_MethodObject *)called; // Unbound method without arguments, let the error path be slow. if ( method->m_object != NULL ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } Nuitka_FunctionObject *function = method->m_function; PyObject *result; if ( function->m_args_simple && 1 == function->m_args_positional_count ) { PyObject *args[] = { method->m_object }; Py_INCREF( method->m_object ); result = function->m_c_code( function, args ); } else if ( function->m_args_simple && function->m_defaults_given == function->m_args_positional_count - 1 ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif python_pars[0] = method->m_object; memcpy( python_pars+1, &PyTuple_GET_ITEM( function->m_defaults, 0 ), sizeof(PyObject *) * function->m_defaults_given ); for( Py_ssize_t i = 0; i < function->m_args_positional_count; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsMethodPos( function, python_pars, method->m_object, NULL, 0 )) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } } else if ( PyFunction_Check( called ) ) { return _fast_function_noargs( called ); } return CALL_FUNCTION( called, const_tuple_empty, NULL ); } #if defined(_NUITKA_STANDALONE) || _NUITKA_FROZEN > 0 #if _NUITKA_FROZEN > 0 extern void copyFrozenModulesTo(void* destination); #endif #ifdef _NUITKA_STANDALONE extern PyObject *const_str_plain___file__; void setEarlyFrozenModulesFileAttribute( void ) { Py_ssize_t ppos = 0; PyObject *key, *value; PyObject *file_value = getBinaryDirectoryObject(); char sep[2] = { SEP, 0 }; #if PYTHON_VERSION < 300 file_value = PyNumber_InPlaceAdd( file_value, PyString_FromString( sep ) ); #else file_value = PyNumber_InPlaceAdd( file_value, PyUnicode_FromString( sep ) ); #endif assert( file_value ); #if PYTHON_VERSION < 300 file_value = PyNumber_InPlaceAdd( file_value, PyString_FromString( "not_present.py" ) ); #else file_value = PyNumber_InPlaceAdd( file_value, PyUnicode_FromString( "not_present.py" ) ); #endif assert( file_value ); while( PyDict_Next( PyImport_GetModuleDict(), &ppos, &key, &value ) ) { if ( key != NULL && value != NULL && PyModule_Check( value ) ) { if ( !PyObject_HasAttr( value, const_str_plain___file__ ) ) { PyObject_SetAttr( value, const_str_plain___file__, file_value ); } } } assert( !ERROR_OCCURRED() ); } #endif static char *original_home; static char *original_path; void prepareStandaloneEnvironment() { // Tell the CPython library to use our pre-compiled modules as frozen // modules. This for those modules/packages like "encoding" that will be // loaded during "Py_Initialize" already, for the others they may be // compiled. #if _NUITKA_FROZEN > 0 // The CPython library has some pre-existing frozen modules, we only append // to that. const _frozen *search = PyImport_FrozenModules; while( search->name ) { search++; } int pre_existing_count = int( search - PyImport_FrozenModules ); // Allocate new memory and merge the tables. _frozen *merged = new _frozen[ _NUITKA_FROZEN + pre_existing_count + 1 ]; memcpy( merged, PyImport_FrozenModules, pre_existing_count * sizeof( struct _frozen ) ); copyFrozenModulesTo(merged + pre_existing_count); PyImport_FrozenModules = merged; #endif #ifdef _NUITKA_STANDALONE // Setup environment variables to tell CPython that we would like it to use // the provided binary directory as the place to look for DLLs. char *binary_directory = getBinaryDirectoryHostEncoded(); #if defined( _WIN32 ) && defined( _MSC_VER ) SetDllDirectory( binary_directory ); #endif // get original value original_home = getenv( "PYTHONHOME" ); original_path = getenv( "PYTHONPATH" ); size_t original_home_size = ( original_home ) ? strlen( original_home ) : 0; size_t original_path_size = ( original_path ) ? strlen( original_path ) : 0; // get insert value size_t insert_size = strlen( binary_directory ) * 2 + 50; char *insert_path = (char *) malloc( insert_size ); #if defined( _WIN32 ) char const env_string[] = "%s;"; #else char const env_string[] = "%s:"; #endif memset( insert_path, 0, insert_size ); snprintf( insert_path, insert_size, env_string, binary_directory ); // set environment size_t python_home_size = original_home_size + insert_size; size_t python_path_size = original_path_size + insert_size; char *python_home = (char *) malloc( python_home_size ); char *python_path = (char *) malloc( python_path_size ); memset( python_home, 0, python_home_size ); memset( python_path, 0, python_path_size ); snprintf( python_home, python_home_size, "%s%s", insert_path, original_home ? original_home : "" ); snprintf( python_path, python_path_size, "%s%s", insert_path, original_path ? original_path : "" ); if ( !( original_home && strstr( original_home, insert_path ) ) ) { #if defined( _WIN32 ) SetEnvironmentVariable( "PYTHONHOME", python_home ); #else setenv( "PYTHONHOME", python_home, 1 ); #endif } if ( !( original_path && strstr( original_path, insert_path ) ) ) { #if defined( _WIN32 ) SetEnvironmentVariable( "PYTHONPATH", python_path ); #else setenv( "PYTHONPATH", python_path, 1 ); #endif } // clean up free( insert_path ); #endif } void restoreStandaloneEnvironment() { #if defined( _WIN32 ) SetEnvironmentVariable( "PYTHONHOME", original_home ); #else if ( original_home == NULL ) { unsetenv( "PYTHONHOME" ); } else { setenv( "PYTHONHOME", original_home, 1 ); } #endif #if defined( _WIN32 ) SetEnvironmentVariable( "PYTHONHOME", original_path ); #else if ( original_path == NULL ) { unsetenv( "PYTHONHOME" ); } else { setenv( "PYTHONHOME", original_path, 1 ); } #endif } #endif PyObject *MAKE_RELATIVE_PATH( PyObject *relative ) { static PyObject *our_path_object = NULL; if ( our_path_object == NULL ) { #if defined( _NUITKA_EXE ) our_path_object = getBinaryDirectoryObject(); #else #if PYTHON_VERSION >= 300 our_path_object = PyUnicode_FromString( getDllDirectory() ); #else our_path_object = PyString_FromString( getDllDirectory() ); #endif #endif } CHECK_OBJECT( our_path_object ); static PyObject *os_path_join = NULL; if ( os_path_join == NULL ) { PyObject *os_path = PyImport_ImportModule("os.path"); CHECK_OBJECT(os_path); os_path_join = PyObject_GetAttrString(os_path, "join"); CHECK_OBJECT(os_path_join); Py_DECREF(os_path); } PyObject *result = PyObject_CallFunctionObjArgs( os_path_join, our_path_object, relative, NULL ); if (unlikely( result == NULL )) { PyErr_Print(); abort(); } return result; } #ifdef _NUITKA_EXE #define DEFINE_BUILTIN( name ) extern PyObject *const_str_plain_##name; PyObject *_python_original_builtin_value_##name = NULL; DEFINE_BUILTIN( type ) DEFINE_BUILTIN( len ) DEFINE_BUILTIN( range ) DEFINE_BUILTIN( repr ) DEFINE_BUILTIN( int ) DEFINE_BUILTIN( iter ) #if PYTHON_VERSION < 300 DEFINE_BUILTIN( long ) #endif void _initBuiltinOriginalValues() { ASSIGN_BUILTIN( type ); ASSIGN_BUILTIN( len ); ASSIGN_BUILTIN( range ); ASSIGN_BUILTIN( repr ); ASSIGN_BUILTIN( int ); ASSIGN_BUILTIN( iter ); #if PYTHON_VERSION < 300 ASSIGN_BUILTIN( long ); #endif CHECK_OBJECT( _python_original_builtin_value_range ); } #endif // Used for threading. #if PYTHON_VERSION >= 300 volatile int _Py_Ticker = _Py_CheckInterval; #endif // Reverse operation mapping. static int const swapped_op[] = { Py_GT, Py_GE, Py_EQ, Py_NE, Py_LT, Py_LE }; #if PYTHON_VERSION >= 270 iternextfunc default_iternext; extern PyObject *const_str_plain___iter__; void _initSlotIternext() { PyObject *pos_args = PyTuple_New(1); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( (PyObject *)&PyBaseObject_Type ) ); PyObject *kw_args = PyDict_New(); PyDict_SetItem( kw_args, const_str_plain___iter__, Py_True ); PyObject *c = PyObject_CallFunctionObjArgs( (PyObject *)&PyType_Type, const_str_plain___iter__, pos_args, kw_args, NULL ); Py_DECREF( pos_args ); Py_DECREF( kw_args ); PyObject *r = PyObject_CallFunctionObjArgs( c, NULL ); Py_DECREF( c ); CHECK_OBJECT( r ); assert( Py_TYPE( r )->tp_iternext ); default_iternext = Py_TYPE(r)->tp_iternext; Py_DECREF( r ); } #endif #if PYTHON_VERSION < 300 extern PyObject *const_str_plain___cmp__; static cmpfunc default_tp_compare; void _initSlotCompare() { // Create a class with "__cmp__" attribute, to get a hand at the default // implementation of tp_compare. It's not part of the API and with shared // libraries it's not accessible. The name does not matter, nor does the // actual value used for "__cmp__". PyObject *pos_args = PyTuple_New(1); PyTuple_SET_ITEM( pos_args, 0, INCREASE_REFCOUNT( (PyObject *)&PyInt_Type ) ); PyObject *kw_args = PyDict_New(); PyDict_SetItem( kw_args, const_str_plain___cmp__, Py_True ); PyObject *c = PyObject_CallFunctionObjArgs( (PyObject *)&PyType_Type, const_str_plain___cmp__, pos_args, kw_args, NULL ); Py_DECREF( pos_args ); Py_DECREF( kw_args ); PyObject *r = PyObject_CallFunctionObjArgs( c, NULL ); Py_DECREF( c ); CHECK_OBJECT( r ); assert( Py_TYPE( r )->tp_compare ); default_tp_compare = Py_TYPE( r )->tp_compare; Py_DECREF( r ); } #define RICHCOMPARE( t ) ( PyType_HasFeature((t), Py_TPFLAGS_HAVE_RICHCOMPARE) ? (t)->tp_richcompare : NULL ) static inline int adjust_tp_compare( int c ) { if ( PyErr_Occurred() ) { return -2; } else if (c < -1 || c > 1) { return c < -1 ? -1 : 1; } else { return c; } } static inline int coerce_objects( PyObject **pa, PyObject **pb ) { PyObject *a = *pa; PyObject *b = *pb; // Shortcut only for old-style types if ( a->ob_type == b->ob_type && !PyType_HasFeature( a->ob_type, Py_TPFLAGS_CHECKTYPES )) { Py_INCREF( a ); Py_INCREF( b ); return 0; } if ( a->ob_type->tp_as_number && a->ob_type->tp_as_number->nb_coerce ) { int res = (*a->ob_type->tp_as_number->nb_coerce)( pa, pb ); if ( res <= 0 ) { return res; } } if ( b->ob_type->tp_as_number && b->ob_type->tp_as_number->nb_coerce ) { int res = (*b->ob_type->tp_as_number->nb_coerce)( pb, pa ); if ( res <= 0 ) { return res; } } return 1; } static int try_3way_compare( PyObject *a, PyObject *b ) { cmpfunc f1 = a->ob_type->tp_compare; cmpfunc f2 = b->ob_type->tp_compare; int c; // Same compares, just use it. if ( f1 != NULL && f1 == f2 ) { c = (*f1)( a, b ); return adjust_tp_compare( c ); } // If one slot is _PyObject_SlotCompare (which we got our hands on under a // different name in case it's a shared library), prefer it. if ( f1 == default_tp_compare || f2 == default_tp_compare ) { return default_tp_compare( a, b ); } // Try coercion. c = coerce_objects( &a, &b ); if (c < 0) { return -2; } if (c > 0) { return 2; } f1 = a->ob_type->tp_compare; if ( f1 != NULL && f1 == b->ob_type->tp_compare ) { c = (*f1)( a, b ); Py_DECREF( a ); Py_DECREF( b ); return adjust_tp_compare(c); } // No comparison defined. Py_DECREF( a ); Py_DECREF( b ); return 2; } PyObject *MY_RICHCOMPARE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); // TODO: Type a-ware rich comparison would be really nice, but this is what // CPython does, and should be even in "richcomparisons.hpp" as the first // thing, so it's even cheaper. if ( PyInt_CheckExact( a ) && PyInt_CheckExact( b )) { long aa, bb; #ifdef __NUITKA_NO_ASSERT__ bool res; #else bool res = false; #endif aa = PyInt_AS_LONG( a ); bb = PyInt_AS_LONG( b ); switch( op ) { case Py_LT: res = aa < bb; break; case Py_LE: res = aa <= bb; break; case Py_EQ: res = aa == bb; break; case Py_NE: res = aa != bb; break; case Py_GT: res = aa > bb; break; case Py_GE: res = aa >= bb; break; default: assert( false ); } return INCREASE_REFCOUNT( BOOL_FROM( res ) ); } // TODO: Get hint from recursion control if that's needed. if (unlikely( Py_EnterRecursiveCall((char *)" in cmp") )) { return NULL; } PyObject *result; // If the types are equal, we may get away immediately. if ( a->ob_type == b->ob_type && !PyInstance_Check( a ) ) { richcmpfunc frich = RICHCOMPARE( a->ob_type ); if ( frich != NULL ) { result = (*frich)( a, b, op ); if (result != Py_NotImplemented) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } // No rich comparison, but maybe compare works. cmpfunc fcmp = a->ob_type->tp_compare; if ( fcmp != NULL ) { int c = (*fcmp)( a, b ); c = adjust_tp_compare( c ); Py_LeaveRecursiveCall(); if ( c == -2 ) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } return INCREASE_REFCOUNT( BOOL_FROM( c != 0 ) ); } } // Fast path was not successful or not taken richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } } f = RICHCOMPARE( a->ob_type ); if ( f != NULL ) { result = (*f)( a, b, op ); if ( result != Py_NotImplemented ) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } int c; if ( PyInstance_Check( a ) ) { c = (*a->ob_type->tp_compare)( a, b ); } else if ( PyInstance_Check( b ) ) { c = (*b->ob_type->tp_compare)( a, b ); } else { c = try_3way_compare( a, b ); } if ( c >= 2 ) { if ( a->ob_type == b->ob_type ) { Py_uintptr_t aa = (Py_uintptr_t)a; Py_uintptr_t bb = (Py_uintptr_t)b; c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else if ( a == Py_None ) { // None is smaller than everything else c = -1; } else if ( b == Py_None ) { // None is smaller than everything else c = 1; } else if ( PyNumber_Check( a ) ) { // different type: compare type names but numbers are smaller than // others. if ( PyNumber_Check( b ) ) { // Both numbers, need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else { c = -1; } } else if ( PyNumber_Check( b ) ) { c = 1; } else { int s = strcmp( a->ob_type->tp_name, b->ob_type->tp_name ); if ( s < 0 ) { c = -1; } else if ( s > 0 ) { c = 1; } else { // Same type name need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } } } Py_LeaveRecursiveCall(); if (unlikely( c <= -2 )) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } return INCREASE_REFCOUNT( BOOL_FROM( c != 0 ) ); } PyObject *MY_RICHCOMPARE_NORECURSE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); // TODO: Type a-ware rich comparison would be really nice, but this is what // CPython does, and should be even in "richcomparisons.hpp" as the first // thing, so it's even cheaper. if ( PyInt_CheckExact( a ) && PyInt_CheckExact( b )) { long aa, bb; #ifdef __NUITKA_NO_ASSERT__ bool res; #else bool res = false; #endif aa = PyInt_AS_LONG( a ); bb = PyInt_AS_LONG( b ); switch( op ) { case Py_LT: res = aa < bb; break; case Py_LE: res = aa <= bb; break; case Py_EQ: res = aa == bb; break; case Py_NE: res = aa != bb; break; case Py_GT: res = aa > bb; break; case Py_GE: res = aa >= bb; break; default: assert( false ); } return INCREASE_REFCOUNT( BOOL_FROM( res ) ); } PyObject *result; // If the types are equal, we may get away immediately. if ( a->ob_type == b->ob_type && !PyInstance_Check( a ) ) { richcmpfunc frich = RICHCOMPARE( a->ob_type ); if ( frich != NULL ) { result = (*frich)( a, b, op ); if (result != Py_NotImplemented) { return result; } Py_DECREF( result ); } // No rich comparison, but maybe compare works. cmpfunc fcmp = a->ob_type->tp_compare; if ( fcmp != NULL ) { int c = (*fcmp)( a, b ); c = adjust_tp_compare( c ); if ( c == -2 ) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } return INCREASE_REFCOUNT( BOOL_FROM( c != 0 ) ); } } // Fast path was not successful or not taken richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { return result; } Py_DECREF( result ); } } f = RICHCOMPARE( a->ob_type ); if ( f != NULL ) { result = (*f)( a, b, op ); if ( result != Py_NotImplemented ) { return result; } Py_DECREF( result ); } f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { return result; } Py_DECREF( result ); } int c; if ( PyInstance_Check( a ) ) { c = (*a->ob_type->tp_compare)( a, b ); } else if ( PyInstance_Check( b ) ) { c = (*b->ob_type->tp_compare)( a, b ); } else { c = try_3way_compare( a, b ); } if ( c >= 2 ) { if ( a->ob_type == b->ob_type ) { Py_uintptr_t aa = (Py_uintptr_t)a; Py_uintptr_t bb = (Py_uintptr_t)b; c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else if ( a == Py_None ) { // None is smaller than everything else c = -1; } else if ( b == Py_None ) { // None is smaller than everything else c = 1; } else if ( PyNumber_Check( a ) ) { // different type: compare type names but numbers are smaller than // others. if ( PyNumber_Check( b ) ) { // Both numbers, need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else { c = -1; } } else if ( PyNumber_Check( b ) ) { c = 1; } else { int s = strcmp( a->ob_type->tp_name, b->ob_type->tp_name ); if ( s < 0 ) { c = -1; } else if ( s > 0 ) { c = 1; } else { // Same type name need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } } } if (unlikely( c <= -2 )) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } return INCREASE_REFCOUNT( BOOL_FROM( c != 0 ) ); } #else // Table for operation names as strings. static char const *op_strings[] = { "<", "<=", "==", "!=", ">", ">=" }; PyObject *MY_RICHCOMPARE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); if (unlikely( Py_EnterRecursiveCall( (char *)" in comparison" ) )) { return NULL; } bool checked_reverse_op = false; PyObject *result = NULL; richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { checked_reverse_op = true; result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { Py_LeaveRecursiveCall(); return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL ) { f = a->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( a, b, op ); if (unlikely( result == NULL )) { Py_LeaveRecursiveCall(); return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL && checked_reverse_op == false ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { Py_LeaveRecursiveCall(); return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } Py_LeaveRecursiveCall(); if ( result != NULL ) { return result; } // If it is not implemented, do identify checks as "==" and "!=" and // otherwise give an error if ( op == Py_EQ ) { return INCREASE_REFCOUNT( BOOL_FROM( a == b ) ); } else if ( op == Py_NE ) { return INCREASE_REFCOUNT( BOOL_FROM( a != b ) ); } else { PyErr_Format( PyExc_TypeError, "unorderable types: %s() %s %s()", a->ob_type->tp_name, op_strings[ op ], b->ob_type->tp_name ); return NULL; } } PyObject *MY_RICHCOMPARE_NORECURSE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); bool checked_reverse_op = false; PyObject *result = NULL; richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { checked_reverse_op = true; result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL ) { f = a->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( a, b, op ); if (unlikely( result == NULL )) { return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL && checked_reverse_op == false ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result != NULL ) { return result; } // If it is not implemented, do identify checks as "==" and "!=" and // otherwise give an error if ( op == Py_EQ ) { return INCREASE_REFCOUNT( BOOL_FROM( a == b ) ); } else if ( op == Py_NE ) { return INCREASE_REFCOUNT( BOOL_FROM( a != b ) ); } else { PyErr_Format( PyExc_TypeError, "unorderable types: %s() %s %s()", a->ob_type->tp_name, op_strings[ op ], b->ob_type->tp_name ); return NULL; } } #endif PyObject *DEEP_COPY( PyObject *value ) { if ( PyDict_Check( value ) ) { // For Python3.3, this can be done much faster in the same way as it is // done in parameter parsing. #if PYTHON_VERSION < 330 PyObject *result = _PyDict_NewPresized( ((PyDictObject *)value)->ma_used ); for ( Py_ssize_t i = 0; i <= ((PyDictObject *)value)->ma_mask; i++ ) { PyDictEntry *entry = &((PyDictObject *)value)->ma_table[ i ]; if ( entry->me_value != NULL ) { PyObject *deep_copy = DEEP_COPY( entry->me_value ); int res = PyDict_SetItem( result, entry->me_key, deep_copy ); Py_DECREF( deep_copy ); if (unlikely( res != 0 )) { return NULL; } } } return result; #else if ( _PyDict_HasSplitTable( (PyDictObject *)value) ) { PyDictObject *mp = (PyDictObject *)value; PyObject **newvalues = PyMem_NEW( PyObject *, mp->ma_keys->dk_size ); assert( newvalues != NULL ); PyDictObject *result = PyObject_GC_New( PyDictObject, &PyDict_Type ); assert( result != NULL ); result->ma_values = newvalues; result->ma_keys = mp->ma_keys; result->ma_used = mp->ma_used; mp->ma_keys->dk_refcnt += 1; Nuitka_GC_Track( result ); Py_ssize_t size = mp->ma_keys->dk_size; for ( Py_ssize_t i = 0; i < size; i++ ) { if ( mp->ma_values[ i ] ) { result->ma_values[ i ] = DEEP_COPY( mp->ma_values[ i ] ); } else { result->ma_values[ i ] = NULL; } } return (PyObject *)result; } else { PyObject *result = _PyDict_NewPresized( ((PyDictObject *)value)->ma_used ); PyDictObject *mp = (PyDictObject *)value; Py_ssize_t size = mp->ma_keys->dk_size; for ( Py_ssize_t i = 0; i < size; i++ ) { PyDictKeyEntry *entry = &mp->ma_keys->dk_entries[i]; PyObject *value2; if ( mp->ma_values ) { value2 = mp->ma_values[ i ]; } else { value2 = entry->me_value; } if ( value2 != NULL ) { PyObject *deep_copy = DEEP_COPY( value2 ); PyDict_SetItem( result, entry->me_key, deep_copy ); Py_DECREF( deep_copy ); } } return result; } #endif } else if ( PyTuple_Check( value ) ) { Py_ssize_t n = PyTuple_Size( value ); PyObject *result = PyTuple_New( n ); for( Py_ssize_t i = 0; i < n; i++ ) { PyTuple_SET_ITEM( result, i, DEEP_COPY( PyTuple_GET_ITEM( value, i ) ) ); } return result; } else if ( PyList_Check( value ) ) { Py_ssize_t n = PyList_GET_SIZE( value ); PyObject *result = PyList_New( n ); for( Py_ssize_t i = 0; i < n; i++ ) { PyList_SET_ITEM( result, i, DEEP_COPY( PyList_GET_ITEM( value, i ) ) ); } return result; } else if ( PySet_Check( value ) ) { // Sets cannot contain unhashable types, so they must be immutable. return PySet_New( value ); } else if ( #if PYTHON_VERSION < 300 PyString_Check( value ) || #endif PyUnicode_Check( value ) || #if PYTHON_VERSION < 300 PyInt_Check( value ) || #endif PyLong_Check( value ) || value == Py_None || PyBool_Check( value ) || PyFloat_Check( value ) || PyBytes_Check( value ) || #if PYTHON_VERSION >= 300 PyRange_Check( value ) || #endif PyType_Check( value ) || PySlice_Check( value ) || PyComplex_Check( value ) || value == Py_Ellipsis ) { return INCREASE_REFCOUNT( value ); } else { PyErr_Format( PyExc_TypeError, "DEEP_COPY does not implement: %s", value->ob_type->tp_name ); return NULL; } } #ifndef __NUITKA_NO_ASSERT__ static Py_hash_t DEEP_HASH_INIT( PyObject *value ) { // To avoid warnings about reduced sizes, we put an intermediate value // that is size_t. size_t value2 = (size_t)value; Py_hash_t result = Py_hash_t( value2 ); if ( Py_TYPE( value ) != &PyType_Type ) { result ^= DEEP_HASH( (PyObject *)Py_TYPE( value ) ); } return result; } static void DEEP_HASH_BLOB( Py_hash_t *hash, char const *s, Py_ssize_t size ) { while( size > 0 ) { *hash = ( 1000003 * (*hash) ) ^ Py_hash_t( *s++ ); size--; } } static void DEEP_HASH_CSTR( Py_hash_t *hash, char const *s ) { DEEP_HASH_BLOB( hash, s, strlen( s ) ); } // Hash function that actually verifies things done to the bit level. Can be // used to detect corruption. Py_hash_t DEEP_HASH( PyObject *value ) { assert( value != NULL ); if ( PyType_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); DEEP_HASH_CSTR( &result, ((PyTypeObject *)value)->tp_name ); return result; } else if ( PyDict_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t ppos = 0; PyObject *key, *dict_value; while( PyDict_Next( value, &ppos, &key, &dict_value ) ) { if ( key != NULL && value != NULL ) { result ^= DEEP_HASH( key ); result ^= DEEP_HASH( dict_value ); } } return result; } else if ( PyTuple_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t n = PyTuple_Size( value ); for( Py_ssize_t i = 0; i < n; i++ ) { result ^= DEEP_HASH( PyTuple_GET_ITEM( value, i ) ); } return result; } else if ( PyList_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t n = PyList_GET_SIZE( value ); for( Py_ssize_t i = 0; i < n; i++ ) { result ^= DEEP_HASH( PyList_GET_ITEM( value, i ) ); } return result; } else if ( PySet_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); PyObject *iterator = PyObject_GetIter( value ); CHECK_OBJECT( iterator ); while( true ) { PyObject *item = PyIter_Next( iterator ); if (!item) break; CHECK_OBJECT( item ); result ^= DEEP_HASH( item ); Py_DECREF( item ); } Py_DECREF( iterator ); return result; } else if ( PyLong_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); // Use string to hash the long value, which relies on that to not // use the object address. PyObject *str = PyObject_Str( value ); result ^= DEEP_HASH( str ); Py_DECREF( str ); RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return result; } else if ( PyUnicode_Check( value ) ) { Py_hash_t result = DEEP_HASH( (PyObject *)Py_TYPE( value ) ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); #if PYTHON_PYTHON >= 330 Py_ssize_t size; char *s = PyUnicode_AsUTF8AndSize( value, &size ); if ( s != NULL ) { DEEP_HASH_BLOB( &result, s, size ); } #elif PYTHON_VERSION >= 300 // Not done for Python3.2 really yet. #else PyObject *str = PyUnicode_AsUTF8String( value ); if ( str ) { result ^= DEEP_HASH( str ); } Py_DECREF( str ); #endif RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return result; } #if PYTHON_VERSION < 300 else if ( PyString_Check( value ) ) { Py_hash_t result = DEEP_HASH( (PyObject *)Py_TYPE( value ) ); Py_ssize_t size; char *s; int res = PyString_AsStringAndSize( value, &s, &size ); assert( res != -1 ); DEEP_HASH_BLOB( &result, s, size ); return result; } #else else if ( PyBytes_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size; char *s; int res = PyBytes_AsStringAndSize( value, &s, &size ); assert( res != -1 ); DEEP_HASH_BLOB( &result, s, size ); return result; } #endif else if ( value == Py_None || value == Py_Ellipsis ) { return DEEP_HASH_INIT( value ); } else if ( PyComplex_Check( value ) ) { Py_complex c = PyComplex_AsCComplex( value ); Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size = sizeof(c); char *s = (char *)&c; DEEP_HASH_BLOB( &result, s, size ); return result; } else if ( PyFloat_Check( value ) ) { double f = PyFloat_AsDouble( value ); Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size = sizeof(f); char *s = (char *)&f; DEEP_HASH_BLOB( &result, s, size ); return result; } else if ( #if PYTHON_VERSION < 300 PyInt_Check( value ) || #endif PyBool_Check( value ) || #if PYTHON_VERSION >= 300 PyRange_Check( value ) || #endif PySlice_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); #if 0 printf("Too simple deep hash: %s\n", Py_TYPE( value )->tp_name ); #endif return result; } else { assert( false ); return -1; } } #endif #if _NUITKA_PROFILE timespec diff(timespec start, timespec end); static timespec getTimespecDiff( timespec start, timespec end ) { timespec temp; if ( ( end.tv_nsec - start.tv_nsec ) < 0 ) { temp.tv_sec = end.tv_sec-start.tv_sec-1; temp.tv_nsec = 1000000000+end.tv_nsec-start.tv_nsec; } else { temp.tv_sec = end.tv_sec-start.tv_sec; temp.tv_nsec = end.tv_nsec-start.tv_nsec; } return temp; } static FILE *tempfile_profile; static PyObject *vmprof_module; static timespec time1, time2; void startProfiling( void ) { tempfile_profile = fopen("nuitka-performance.dat", "wb"); // Might be necessary to import "site" module to find "vmprof", lets just // hope we don't suffer too much from that. If we do, what might be done // is to try and just have the "PYTHONPATH" from it from out user. PyImport_ImportModule( "site" ); vmprof_module = PyImport_ImportModule( "vmprof" ); // Abort if it's not there. if ( vmprof_module == NULL ) { PyErr_Print(); abort(); } PyObject *result = CALL_FUNCTION_WITH_ARGS1( PyObject_GetAttrString( vmprof_module, "enable"), PyInt_FromLong( fileno( tempfile_profile ) ) ); if ( result == NULL ) { PyErr_Print(); abort(); } clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time1); } void stopProfiling( void ) { clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time2); // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); PyObject *result = CALL_FUNCTION_NO_ARGS( PyObject_GetAttrString( vmprof_module, "disable") ); if ( result == NULL ) PyErr_Clear(); fclose( tempfile_profile ); FILE *tempfile_times = fopen( "nuitka-times.dat", "wb" ); timespec diff = getTimespecDiff( time1, time2 ); long delta_ns = diff.tv_sec * 1000000000 + diff.tv_nsec; fprintf( tempfile_times, "%ld\n", delta_ns); fclose( tempfile_times ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } #endif Nuitka-0.5.18.1/nuitka/build/static_src/CompiledMethodType.cpp0000644000175000017500000005006712651071040024430 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.hpp" #include "nuitka/compiled_method.hpp" #include "structmember.h" static PyObject *Nuitka_Method_get__doc__( Nuitka_MethodObject *method, void *closure ) { return INCREASE_REFCOUNT( method->m_function->m_doc ); } static PyGetSetDef Nuitka_Method_getsets[] = { { (char *)"__doc__", (getter)Nuitka_Method_get__doc__, NULL, NULL }, { NULL } }; #define OFF( x ) offsetof( Nuitka_MethodObject, x ) static PyMemberDef Nuitka_Method_members[] = { { (char *)"im_class", T_OBJECT, OFF( m_class ), READONLY | RESTRICTED, (char *)"the class associated with a method"}, { (char *)"im_func", T_OBJECT, OFF( m_function ), READONLY | RESTRICTED, (char *)"the function (or other callable) implementing a method" }, { (char *)"__func__", T_OBJECT, OFF( m_function ), READONLY | RESTRICTED, (char *)"the function (or other callable) implementing a method" }, { (char *)"im_self", T_OBJECT, OFF( m_object ), READONLY | RESTRICTED, (char *)"the instance to which a method is bound; None for unbound method" }, { (char *)"__self__", T_OBJECT, OFF( m_object ), READONLY | RESTRICTED, (char *)"the instance to which a method is bound; None for unbound method" }, { NULL } }; static PyObject *Nuitka_Method_reduce( Nuitka_MethodObject *method ) { PyErr_Format( PyExc_TypeError, "Can't pickle instancemethod objects" ); return NULL; } static PyObject *Nuitka_Method_reduce_ex( Nuitka_MethodObject *method, PyObject *args ) { int proto; if ( !PyArg_ParseTuple(args, "|i:__reduce_ex__", &proto) ) { return NULL; } PyErr_Format( PyExc_TypeError, "Can't pickle instancemethod objects" ); return NULL; } static PyObject *Nuitka_Method_deepcopy( Nuitka_MethodObject *method, PyObject *memo ) { assert( Nuitka_Method_Check( (PyObject *)method )); static PyObject *module_copy = NULL; static PyObject *deepcopy_function = NULL; if ( module_copy == NULL ) { module_copy = PyImport_ImportModule( "copy" ); CHECK_OBJECT( module_copy ); deepcopy_function = PyObject_GetAttrString( module_copy, "deepcopy" ); CHECK_OBJECT( deepcopy_function ); } PyObject *object = PyObject_CallFunctionObjArgs( deepcopy_function, method->m_object, memo, NULL ); if (unlikely( object == NULL )) { return NULL; } return Nuitka_Method_New( method->m_function, object, method->m_class ); } static PyMethodDef Nuitka_Method_methods[] = { { "__reduce__", (PyCFunction)Nuitka_Method_reduce, METH_NOARGS, NULL }, { "__reduce_ex__", (PyCFunction)Nuitka_Method_reduce_ex, METH_VARARGS, NULL }, { "__deepcopy__", (PyCFunction)Nuitka_Method_deepcopy, METH_O, NULL }, { NULL } }; extern PyObject *const_str_plain___name__; static char const *GET_CLASS_NAME( PyObject *klass ) { if ( klass == NULL ) { return "?"; } else { #if PYTHON_VERSION < 300 if ( PyClass_Check( klass ) ) { return Nuitka_String_AsString( ((PyClassObject *)klass)->cl_name ); } #endif if ( !PyType_Check( klass ) ) { klass = (PyObject *)Py_TYPE( klass ); } return ((PyTypeObject *)klass)->tp_name; } } static char const *GET_INSTANCE_CLASS_NAME( PyObject *instance ) { // TODO: We have a constant for that already. PyObject *klass = PyObject_GetAttrString( instance, "__class__" ); // Fallback to type as this cannot fail. if ( klass == NULL ) { CLEAR_ERROR_OCCURRED(); klass = INCREASE_REFCOUNT( (PyObject *)Py_TYPE( instance ) ); } char const *result = GET_CLASS_NAME( klass ); Py_DECREF( klass ); return result; } static char const *GET_CALLABLE_DESC( PyObject *object ) { if ( Nuitka_Function_Check( object ) || Nuitka_Generator_Check( object ) || PyMethod_Check( object ) || PyFunction_Check( object ) || PyCFunction_Check( object ) ) { return "()"; } #if PYTHON_VERSION < 300 else if ( PyClass_Check( object ) ) { return " constructor"; } else if ( PyInstance_Check( object )) { return " instance"; } #endif else { return " object"; } } static char const *GET_CALLABLE_NAME( PyObject *object ) { if ( Nuitka_Function_Check( object ) ) { return Nuitka_String_AsString( Nuitka_Function_GetName( object ) ); } else if ( Nuitka_Generator_Check( object ) ) { return Nuitka_String_AsString( Nuitka_Generator_GetName( object ) ); } else if ( PyMethod_Check( object ) ) { return PyEval_GetFuncName( PyMethod_GET_FUNCTION( object ) ); } else if ( PyFunction_Check( object ) ) { return Nuitka_String_AsString( ((PyFunctionObject*)object)->func_name ); } #if PYTHON_VERSION < 300 else if ( PyInstance_Check( object ) ) { return Nuitka_String_AsString( ((PyInstanceObject*)object)->in_class->cl_name ); } else if ( PyClass_Check( object ) ) { return Nuitka_String_AsString( ((PyClassObject*)object)->cl_name ); } #endif else if ( PyCFunction_Check( object ) ) { return ((PyCFunctionObject*)object)->m_ml->ml_name; } else { return Py_TYPE( object )->tp_name; } } static PyObject *Nuitka_Method_tp_call( Nuitka_MethodObject *method, PyObject *args, PyObject *kw ) { Py_ssize_t arg_count = PyTuple_Size( args ); if ( method->m_object == NULL ) { if (unlikely( arg_count < 1 )) { PyErr_Format( PyExc_TypeError, "unbound compiled_method %s%s must be called with %s instance as first argument (got nothing instead)", GET_CALLABLE_NAME( (PyObject *)method->m_function ), GET_CALLABLE_DESC( (PyObject *)method->m_function ), GET_CLASS_NAME( method->m_class ) ); return NULL; } else { PyObject *self = PyTuple_GET_ITEM( args, 0 ); CHECK_OBJECT( self ); int result = PyObject_IsInstance( self, method->m_class ); if (unlikely( result < 0 )) { return NULL; } else if (unlikely( result == 0 )) { PyErr_Format( PyExc_TypeError, "unbound compiled_method %s%s must be called with %s instance as first argument (got %s instance instead)", GET_CALLABLE_NAME( (PyObject *)method->m_function ), GET_CALLABLE_DESC( (PyObject *)method->m_function ), GET_CLASS_NAME( method->m_class ), GET_INSTANCE_CLASS_NAME( (PyObject *)self ) ); return NULL; } } return Py_TYPE( method->m_function )->tp_call( (PyObject *)method->m_function, args, kw ); } else { return Nuitka_CallMethodFunctionPosArgsKwArgs( method->m_function, method->m_object, &PyTuple_GET_ITEM( args, 0 ), arg_count, kw ); } } static PyObject *Nuitka_Method_tp_descr_get( Nuitka_MethodObject *method, PyObject *object, PyObject *klass ) { // Don't rebind already bound methods. if ( method->m_object != NULL ) { return INCREASE_REFCOUNT( (PyObject *)method ); } if ( method->m_class != NULL && klass != NULL ) { // Quick subclass test, bound methods remain the same if the class is a sub class int result = PyObject_IsSubclass( klass, method->m_class ); if (unlikely( result < 0 )) { return NULL; } else if ( result == 0 ) { return INCREASE_REFCOUNT( (PyObject *)method ); } } return Nuitka_Method_New( method->m_function, object, klass ); } static PyObject *Nuitka_Method_tp_getattro( Nuitka_MethodObject *method, PyObject *name ) { PyObject *descr = _PyType_Lookup( &Nuitka_Method_Type, name ); if ( descr != NULL ) { if ( #if PYTHON_VERSION < 300 PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) && #endif ( Py_TYPE( descr )->tp_descr_get != NULL ) ) { return Py_TYPE( descr )->tp_descr_get( descr, (PyObject *)method, (PyObject *)Py_TYPE( method ) ); } else { return INCREASE_REFCOUNT( descr ); } } return PyObject_GetAttr( (PyObject *)method->m_function, name ); } static long Nuitka_Method_tp_traverse( Nuitka_MethodObject *method, visitproc visit, void *arg ) { Py_VISIT( method->m_function ); Py_VISIT( method->m_object ); Py_VISIT( method->m_class ); return 0; } // tp_repr slot, decide how a function shall be output static PyObject *Nuitka_Method_tp_repr( Nuitka_MethodObject *method ) { if ( method->m_object == NULL ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( "", GET_CLASS_NAME( method->m_class ), Nuitka_String_AsString( method->m_function->m_name ) ); #else return PyUnicode_FromFormat( "", Nuitka_String_AsString( method->m_function->m_name ), method->m_function ); #endif } else { // Note: CPython uses repr ob the object, although a comment despises // it, we do it for compatibility. PyObject *object_repr = PyObject_Repr( method->m_object ); if ( object_repr == NULL ) { return NULL; } #if PYTHON_VERSION < 300 else if ( !PyString_Check( object_repr ) ) { Py_DECREF( object_repr ); return NULL; } #else else if ( !PyUnicode_Check( object_repr ) ) { Py_DECREF( object_repr ); return NULL; } #endif #if PYTHON_VERSION < 300 PyObject *result = PyString_FromFormat( "", GET_CLASS_NAME( method->m_class ), Nuitka_String_AsString( method->m_function->m_name ), Nuitka_String_AsString_Unchecked( object_repr ) ); #elif PYTHON_VERSION < 350 PyObject *result = PyUnicode_FromFormat( "", GET_CLASS_NAME( method->m_class ), Nuitka_String_AsString( method->m_function->m_name ), Nuitka_String_AsString_Unchecked( object_repr ) ); #else PyObject *result = PyUnicode_FromFormat( "", Nuitka_String_AsString( method->m_function->m_qualname ), Nuitka_String_AsString_Unchecked( object_repr ) ); #endif Py_DECREF( object_repr ); return result; } } #if PYTHON_VERSION < 300 static int Nuitka_Method_tp_compare( Nuitka_MethodObject *a, Nuitka_MethodObject *b ) { if ( a->m_function->m_counter < b->m_function->m_counter ) { return -1; } else if ( a->m_function->m_counter > b->m_function->m_counter ) { return 1; } else if ( a->m_object == b->m_object ) { return 0; } else if ( a->m_object == NULL ) { return -1; } else if ( b->m_object == NULL ) { return 1; } else { return PyObject_Compare( a->m_object, b->m_object ); } } #endif static PyObject *Nuitka_Method_tp_richcompare( Nuitka_MethodObject *a, Nuitka_MethodObject *b, int op ) { if ( op != Py_EQ && op != Py_NE ) { return INCREASE_REFCOUNT( Py_NotImplemented ); } if ( Nuitka_Method_Check( (PyObject *)a ) == false || Nuitka_Method_Check( (PyObject *)b ) == false ) { return INCREASE_REFCOUNT( Py_NotImplemented ); } bool result = a->m_function->m_counter == b->m_function->m_counter; // If the underlying function objects are the same, check the objects, which // may be NULL in case of unbound methods, which would be the same again. if ( result ) { if ( a->m_object == NULL ) { result = b->m_object == NULL; } else if ( b->m_object == NULL ) { result = 0; } else { int res = PyObject_RichCompareBool( a->m_object, b->m_object, Py_EQ ); result = res != 0; } } if ( op == Py_EQ ) { return INCREASE_REFCOUNT( BOOL_FROM( result ) ); } else { return INCREASE_REFCOUNT( BOOL_FROM( !result ) ); } } static long Nuitka_Method_tp_hash( Nuitka_MethodObject *method ) { // Just give the hash of the method function, that ought to be good enough. return method->m_function->m_counter; } // Cache for method object, try to avoid malloc overhead. static Nuitka_MethodObject *method_cache_head = NULL; static int method_cache_size = 0; static const int max_method_cache_size = 4096; static void Nuitka_Method_tp_dealloc( Nuitka_MethodObject *method ) { #ifndef __NUITKA_NO_ASSERT__ // Save the current exception, if any, we must to not corrupt it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); #endif Nuitka_GC_UnTrack( method ); if ( method->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)method ); } Py_XDECREF( method->m_object ); Py_XDECREF( method->m_class ); Py_DECREF( (PyObject *)method->m_function ); if (likely( method_cache_size < max_method_cache_size )) { method->m_object = (PyObject *)method_cache_head; method_cache_head = method; method_cache_size += 1; } else { PyObject_GC_Del( method ); } #ifndef __NUITKA_NO_ASSERT__ PyThreadState *tstate = PyThreadState_GET(); assert( tstate->curexc_type == save_exception_type ); assert( tstate->curexc_value == save_exception_value ); assert( (PyTracebackObject *)tstate->curexc_traceback == save_exception_tb ); #endif } static PyObject *Nuitka_Method_tp_new( PyTypeObject* type, PyObject* args, PyObject *kw ) { PyObject *func; PyObject *self; PyObject *klass = NULL; if ( !_PyArg_NoKeywords( "instancemethod", kw ) ) { return NULL; } else if ( !PyArg_UnpackTuple( args, "compiled_method", 2, 3, &func, &self, &klass ) ) { return NULL; } else if ( !PyCallable_Check( func ) ) { PyErr_Format( PyExc_TypeError, "first argument must be callable" ); return NULL; } else { if ( self == Py_None ) { self = NULL; } if ( self == NULL && klass == NULL ) { PyErr_Format( PyExc_TypeError, "unbound methods must have non-NULL im_class" ); return NULL; } } assert( Nuitka_Function_Check( func ) ); return Nuitka_Method_New( (Nuitka_FunctionObject *)func, self, klass ); } static const long tp_flags = Py_TPFLAGS_DEFAULT | #if PYTHON_VERSION < 300 Py_TPFLAGS_HAVE_WEAKREFS | #endif Py_TPFLAGS_HAVE_GC; PyTypeObject Nuitka_Method_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_method", sizeof(Nuitka_MethodObject), 0, (destructor)Nuitka_Method_tp_dealloc, // tp_dealloc 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ #if PYTHON_VERSION < 300 (cmpfunc)Nuitka_Method_tp_compare, /* tp_compare */ #else 0, #endif (reprfunc)Nuitka_Method_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ (hashfunc)Nuitka_Method_tp_hash, /* tp_hash */ (ternaryfunc)Nuitka_Method_tp_call, /* tp_call */ 0, /* tp_str */ (getattrofunc)Nuitka_Method_tp_getattro, /* tp_getattro */ PyObject_GenericSetAttr, /* tp_setattro */ 0, /* tp_as_buffer */ tp_flags, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Method_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ (richcmpfunc)Nuitka_Method_tp_richcompare, /* tp_richcompare */ offsetof( Nuitka_MethodObject, m_weakrefs ), /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Nuitka_Method_methods, /* tp_methods */ Nuitka_Method_members, /* tp_members */ Nuitka_Method_getsets, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ (descrgetfunc)Nuitka_Method_tp_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ Nuitka_Method_tp_new, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0 /* tp_version_tag */ #if PYTHON_VERSION >= 340 ,0 /* tp_finalizer */ #endif }; PyObject *Nuitka_Method_New( Nuitka_FunctionObject *function, PyObject *object, PyObject *klass ) { Nuitka_MethodObject *result = method_cache_head; if ( result != NULL ) { method_cache_head = (Nuitka_MethodObject *)method_cache_head->m_object; method_cache_size -= 1; PyObject_INIT( result, &Nuitka_Method_Type ); } else { result = PyObject_GC_New( Nuitka_MethodObject, &Nuitka_Method_Type ); } if (unlikely( result == NULL )) { PyErr_Format( PyExc_RuntimeError, "cannot create method %s", Nuitka_String_AsString( function->m_name ) ); return NULL; } result->m_function = (Nuitka_FunctionObject * )INCREASE_REFCOUNT( (PyObject *)function ); result->m_object = object; Py_XINCREF( object ); result->m_class = klass; Py_XINCREF( klass ); result->m_weakrefs = NULL; Nuitka_GC_Track( result ); return (PyObject *)result; } Nuitka-0.5.18.1/nuitka/build/static_src/CompiledCoroutineType.cpp0000644000175000017500000011324512651071040025155 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.hpp" static PyObject *Nuitka_Coroutine_get_name( Nuitka_CoroutineObject *coroutine) { return INCREASE_REFCOUNT( coroutine->m_name ); } static int Nuitka_Coroutine_set_name( Nuitka_CoroutineObject *coroutine, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *tmp = coroutine->m_name; Py_INCREF( value ); coroutine->m_name = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Coroutine_get_qualname( Nuitka_CoroutineObject *coroutine ) { return INCREASE_REFCOUNT( coroutine->m_qualname ); } static int Nuitka_Coroutine_set_qualname( Nuitka_CoroutineObject *coroutine, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *tmp = coroutine->m_qualname; Py_INCREF( value ); coroutine->m_qualname = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Coroutine_get_cr_await( Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_yieldfrom ) { Py_INCREF( coroutine->m_yieldfrom ); return coroutine->m_yieldfrom; } else { Py_INCREF( Py_None ); return Py_None; } } static PyObject *Nuitka_Coroutine_get_code( Nuitka_CoroutineObject *coroutine ) { return INCREASE_REFCOUNT( (PyObject *)coroutine->m_code_object ); } static int Nuitka_Coroutine_set_code( Nuitka_CoroutineObject *coroutine, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "cr_code is not writable in Nuitka" ); return -1; } static PyObject *Nuitka_Coroutine_get_frame( Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_frame ) { return INCREASE_REFCOUNT( (PyObject *)coroutine->m_frame ); } else { return INCREASE_REFCOUNT( Py_None ); } } static int Nuitka_Coroutine_set_frame( Nuitka_CoroutineObject *coroutine, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "gi_frame is not writable in Nuitka" ); return -1; } static void Nuitka_Coroutine_release_closure( Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_closure ) { for( Py_ssize_t i = 0; i < coroutine->m_closure_given; i++ ) { Py_DECREF( coroutine->m_closure[ i ] ); } free( coroutine->m_closure ); coroutine->m_closure = NULL; } } // For the coroutine object fiber entry point, we may need to follow what // "makecontext" will support and that is only a list of integers, but we will need // to push a pointer through it, and so it's two of them, which might be fully // sufficient. #ifdef _NUITKA_MAKECONTEXT_INTS static void Nuitka_Coroutine_entry_point( int address_1, int address_2 ) { // Restore the pointer from integers should it be necessary, depending on // the platform. This requires pointers to be no larger that to "int" value. int addresses[2] = { address_1, address_2 }; Nuitka_CoroutineObject *coroutine = (Nuitka_CoroutineObject *)*(uintptr_t *)&addresses[0]; #else static void Nuitka_Coroutine_entry_point( Nuitka_CoroutineObject *coroutine ) { #endif ((coroutine_code)coroutine->m_code)( coroutine ); swapFiber( &coroutine->m_yielder_context, &coroutine->m_caller_context ); } static PyObject *Nuitka_Coroutine_send( Nuitka_CoroutineObject *coroutine, PyObject *value ) { if ( coroutine->m_status == status_Unused && value != NULL && value != Py_None ) { PyErr_Format( PyExc_TypeError, "can't send non-None value to a just-started coroutine" ); return NULL; } if ( coroutine->m_status != status_Finished ) { PyThreadState *thread_state = PyThreadState_GET(); #if PYTHON_VERSION < 300 PyObject *saved_exception_type = thread_state->exc_type; Py_XINCREF( saved_exception_type ); PyObject *saved_exception_value = thread_state->exc_value; Py_XINCREF( saved_exception_value ); PyTracebackObject *saved_exception_traceback = (PyTracebackObject *)thread_state->exc_traceback; Py_XINCREF( saved_exception_traceback ); #endif if ( coroutine->m_running ) { PyErr_Format( PyExc_ValueError, "coroutine already executing" ); return NULL; } if ( coroutine->m_status == status_Unused ) { // Prepare the generator context to run. int res = prepareFiber( &coroutine->m_yielder_context, (void *)Nuitka_Coroutine_entry_point, (uintptr_t)coroutine ); if ( res != 0 ) { PyErr_Format( PyExc_MemoryError, "coroutine cannot be allocated" ); return NULL; } coroutine->m_status = status_Running; } coroutine->m_yielded = value; // Put the generator back on the frame stack. PyFrameObject *return_frame = thread_state->frame; #ifndef __NUITKA_NO_ASSERT__ if ( return_frame ) { assertFrameObject( return_frame ); } #endif if ( coroutine->m_frame ) { // It would be nice if our frame were still alive. Nobody had the // right to release it. assertFrameObject( coroutine->m_frame ); // It's not supposed to be on the top right now. assert( return_frame != coroutine->m_frame ); Py_XINCREF( return_frame ); coroutine->m_frame->f_back = return_frame; thread_state->frame = coroutine->m_frame; } // Continue the yielder function while preventing recursion. coroutine->m_running = true; swapFiber( &coroutine->m_caller_context, &coroutine->m_yielder_context ); coroutine->m_running = false; thread_state = PyThreadState_GET(); // Remove the generator from the frame stack. if ( coroutine->m_frame ) { assert( thread_state->frame == coroutine->m_frame ); assertFrameObject( coroutine->m_frame ); Py_CLEAR( coroutine->m_frame->f_back ); } thread_state->frame = return_frame; if ( coroutine->m_returned != NULL ) { coroutine->m_status = status_Finished; Py_XDECREF( coroutine->m_frame ); coroutine->m_frame = NULL; Nuitka_Coroutine_release_closure( coroutine ); PyObject *result = coroutine->m_returned; if ( result == Py_None ) { PyErr_SetObject( PyExc_StopIteration, Py_None ); Py_DECREF( Py_None ); } else { PyObject *exc_result = PyObject_CallFunctionObjArgs( PyExc_StopIteration, result, NULL ); Py_DECREF( result ); if (likely( exc_result != NULL )) { PyErr_SetObject( PyExc_StopIteration, exc_result ); Py_DECREF( exc_result ); } } return NULL; } else if ( coroutine->m_yielded == NULL ) { assert( ERROR_OCCURRED() ); coroutine->m_status = status_Finished; Py_XDECREF( coroutine->m_frame ); coroutine->m_frame = NULL; Nuitka_Coroutine_release_closure( coroutine ); assert( ERROR_OCCURRED() ); if ( GET_ERROR_OCCURRED() == PyExc_StopIteration ) { PyObject *saved_exception_type, *saved_exception_value; PyTracebackObject *saved_exception_tb; FETCH_ERROR_OCCURRED( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); NORMALIZE_EXCEPTION( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); PyErr_Format( PyExc_RuntimeError, "coroutine raised StopIteration" ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, saved_exception_value ); CHECK_OBJECT( exception_value ); CHECK_OBJECT( saved_exception_value ); Py_INCREF( saved_exception_value ); PyException_SetContext( exception_value, saved_exception_value ); Py_DECREF( saved_exception_type ); Py_XDECREF( saved_exception_tb ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } return NULL; } else { return coroutine->m_yielded; } } else { PyErr_SetObject( PyExc_StopIteration, (PyObject *)NULL ); return NULL; } } PyObject *Nuitka_Coroutine_close( Nuitka_CoroutineObject *coroutine, PyObject *args ) { if ( coroutine->m_status == status_Running ) { coroutine->m_exception_type = INCREASE_REFCOUNT( PyExc_GeneratorExit ); coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; PyObject *result = Nuitka_Coroutine_send( coroutine, Py_None ); if (unlikely( result )) { Py_DECREF( result ); PyErr_Format( PyExc_RuntimeError, "coroutine ignored GeneratorExit" ); return NULL; } else { PyObject *error = GET_ERROR_OCCURRED(); assert( error != NULL ); if ( EXCEPTION_MATCH_GENERATOR( error ) ) { CLEAR_ERROR_OCCURRED(); return INCREASE_REFCOUNT( Py_None ); } return NULL; } } return INCREASE_REFCOUNT( Py_None ); } static PyObject *Nuitka_Coroutine_throw( Nuitka_CoroutineObject *coroutine, PyObject *args ) { assert( coroutine->m_exception_type == NULL ); assert( coroutine->m_exception_value == NULL ); assert( coroutine->m_exception_tb == NULL ); int res = PyArg_UnpackTuple( args, "throw", 1, 3, &coroutine->m_exception_type, &coroutine->m_exception_value, (PyObject **)&coroutine->m_exception_tb ); if (unlikely( res == 0 )) { coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } if ( (PyObject *)coroutine->m_exception_tb == Py_None ) { coroutine->m_exception_tb = NULL; } else if ( coroutine->m_exception_tb != NULL && !PyTraceBack_Check( coroutine->m_exception_tb ) ) { coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( PyExceptionClass_Check( coroutine->m_exception_type )) { Py_INCREF( coroutine->m_exception_type ); Py_XINCREF( coroutine->m_exception_value ); Py_XINCREF( coroutine->m_exception_tb ); NORMALIZE_EXCEPTION( &coroutine->m_exception_type, &coroutine->m_exception_value, &coroutine->m_exception_tb ); } else if ( PyExceptionInstance_Check( coroutine->m_exception_type ) ) { if ( coroutine->m_exception_value && coroutine->m_exception_value != Py_None ) { coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); return NULL; } coroutine->m_exception_value = coroutine->m_exception_type; Py_INCREF( coroutine->m_exception_value ); coroutine->m_exception_type = PyExceptionInstance_Class( coroutine->m_exception_type ); Py_INCREF( coroutine->m_exception_type ); Py_XINCREF( coroutine->m_exception_tb ); } else { PyErr_Format( PyExc_TypeError, "exceptions must be classes or instances deriving from BaseException, not %s", Py_TYPE( coroutine->m_exception_type )->tp_name ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } if ( ( coroutine->m_exception_tb != NULL ) && ( (PyObject *)coroutine->m_exception_tb != Py_None ) && ( !PyTraceBack_Check( coroutine->m_exception_tb ) ) ) { PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( coroutine->m_status != status_Finished ) { PyObject *result = Nuitka_Coroutine_send( coroutine, Py_None ); return result; } else { RESTORE_ERROR_OCCURRED( coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } } static void Nuitka_Coroutine_tp_del( Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_status != status_Running ) { return; } // Revive temporarily. assert( Py_REFCNT( coroutine ) == 0 ); Py_REFCNT( coroutine ) = 1; PyObject *error_type, *error_value; PyTracebackObject *error_traceback; FETCH_ERROR_OCCURRED( &error_type, &error_value, &error_traceback ); PyObject *close_result = Nuitka_Coroutine_close( coroutine, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)coroutine ); } else { Py_DECREF( close_result ); } /* Restore the saved exception. */ RESTORE_ERROR_OCCURRED( error_type, error_value, error_traceback ); assert( Py_REFCNT( coroutine ) > 0 ); Py_REFCNT( coroutine ) -= 1; Py_ssize_t refcnt = Py_REFCNT( coroutine ); if (unlikely( refcnt != 0 )) { _Py_NewReference( (PyObject *)coroutine ); Py_REFCNT( coroutine ) = refcnt; _Py_DEC_REFTOTAL; } } static void Nuitka_Coroutine_tp_dealloc( Nuitka_CoroutineObject *coroutine ) { assert( Py_REFCNT( coroutine ) == 0 ); Py_REFCNT( coroutine ) = 1; // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); PyObject *close_result = Nuitka_Coroutine_close( coroutine, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)coroutine ); } else { Py_DECREF( close_result ); } Nuitka_Coroutine_release_closure( coroutine ); Py_XDECREF( coroutine->m_frame ); assert( Py_REFCNT( coroutine ) == 1 ); Py_REFCNT( coroutine ) = 0; releaseFiber( &coroutine->m_yielder_context ); // Now it is safe to release references and memory for it. Nuitka_GC_UnTrack( coroutine ); if ( coroutine->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)coroutine ); assert( !ERROR_OCCURRED() ); } Py_DECREF( coroutine->m_name ); Py_DECREF( coroutine->m_qualname ); PyObject_GC_Del( coroutine ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } static PyObject *Nuitka_Coroutine_tp_repr( Nuitka_CoroutineObject *coroutine ) { return PyUnicode_FromFormat( "", Nuitka_String_AsString( coroutine->m_qualname ), coroutine ); } static long Nuitka_Coroutine_tp_traverse( PyObject *coroutine, visitproc visit, void *arg ) { // TODO: Identify the impact of not visiting owned objects and/or if it // could be NULL instead. The "methodobject" visits its self and module. I // understand this is probably so that back references of this function to // its upper do not make it stay in the memory. A specific test if that // works might be needed. return 0; } static PyObject *Nuitka_Coroutine_await( Nuitka_CoroutineObject *coroutine ) { #if _DEBUG_COROUTINE puts("Nuitka_Coroutine_await enter"); #endif Nuitka_CoroutineWrapperObject *result = PyObject_GC_New( Nuitka_CoroutineWrapperObject, &Nuitka_CoroutineWrapper_Type); if (unlikely(result == NULL)) { return NULL; } result->m_coroutine = coroutine; Py_INCREF( result->m_coroutine ); Nuitka_GC_Track( result ); return (PyObject *)result; } #include static PyMethodDef Nuitka_Coroutine_methods[] = { { "send", (PyCFunction)Nuitka_Coroutine_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_Coroutine_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_Coroutine_close, METH_NOARGS, NULL }, { NULL } }; static PyGetSetDef Nuitka_Coroutine_getsetlist[] = { { (char *)"__name__", (getter)Nuitka_Coroutine_get_name, (setter)Nuitka_Coroutine_set_name, NULL }, { (char *)"__qualname__", (getter)Nuitka_Coroutine_get_qualname, (setter)Nuitka_Coroutine_set_qualname, NULL }, { (char *)"cr_await", (getter)Nuitka_Coroutine_get_cr_await, (setter)NULL, NULL }, { (char *)"cr_code", (getter)Nuitka_Coroutine_get_code, (setter)Nuitka_Coroutine_set_code, NULL }, { (char *)"cr_frame", (getter)Nuitka_Coroutine_get_frame, (setter)Nuitka_Coroutine_set_frame, NULL }, { NULL } }; static PyMemberDef Nuitka_Coroutine_members[] = { { (char *)"cr_running", T_BOOL, offsetof( Nuitka_CoroutineObject, m_running ), READONLY }, { NULL } }; static PyAsyncMethods coro_as_async = { (unaryfunc)Nuitka_Coroutine_await, /* am_await */ 0, /* am_aiter */ 0 /* am_anext */ }; PyTypeObject Nuitka_Coroutine_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_coroutine", /* tp_name */ sizeof(Nuitka_CoroutineObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_Coroutine_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ &coro_as_async, /* tp_as_async */ (reprfunc)Nuitka_Coroutine_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Coroutine_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof( Nuitka_CoroutineObject, m_weakrefs ), /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Nuitka_Coroutine_methods, /* tp_methods */ Nuitka_Coroutine_members, /* tp_members */ Nuitka_Coroutine_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0, /* tp_version_tag */ (destructor)Nuitka_Coroutine_tp_del, /* tp_finalize */ }; static void Nuitka_CoroutineWrapper_tp_dealloc( Nuitka_CoroutineWrapperObject *cw ) { Nuitka_GC_UnTrack( (PyObject *)cw ); Py_DECREF( cw->m_coroutine ); cw->m_coroutine = NULL; PyObject_GC_Del( cw ); } static PyObject *Nuitka_CoroutineWrapper_tp_iternext( Nuitka_CoroutineWrapperObject *cw ) { return Nuitka_Coroutine_send( cw->m_coroutine, Py_None ); } static int Nuitka_CoroutineWrapper_tp_traverse( Nuitka_CoroutineWrapperObject *cw, visitproc visit, void *arg ) { Py_VISIT( (PyObject *)cw->m_coroutine ); return 0; } static PyObject *Nuitka_CoroutineWrapper_send( Nuitka_CoroutineWrapperObject *cw, PyObject *arg ) { return Nuitka_Coroutine_send( cw->m_coroutine, arg ); } static PyObject *Nuitka_CoroutineWrapper_throw( Nuitka_CoroutineWrapperObject *cw, PyObject *args ) { return Nuitka_Coroutine_throw( cw->m_coroutine, args ); } static PyObject *Nuitka_CoroutineWrapper_close( Nuitka_CoroutineWrapperObject *cw, PyObject *args ) { return Nuitka_Coroutine_close( cw->m_coroutine, args ); } static PyMethodDef Nuitka_CoroutineWrapper_methods[] = { { "send", (PyCFunction)Nuitka_CoroutineWrapper_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_CoroutineWrapper_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_CoroutineWrapper_close, METH_NOARGS, NULL }, { NULL } }; PyTypeObject Nuitka_CoroutineWrapper_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "compiled_coroutine_wrapper", sizeof(Nuitka_CoroutineWrapperObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_CoroutineWrapper_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_CoroutineWrapper_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_CoroutineWrapper_tp_iternext, /* tp_iternext */ Nuitka_CoroutineWrapper_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ PyObject_Del, /* tp_free */ }; PyObject *Nuitka_Coroutine_New( coroutine_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyCellObject **closure, Py_ssize_t closure_given ) { Nuitka_CoroutineObject *result = PyObject_GC_New( Nuitka_CoroutineObject, &Nuitka_Coroutine_Type ); assert( result != NULL ); result->m_code = (void *)code; CHECK_OBJECT( name ); result->m_name = name; Py_INCREF( name ); CHECK_OBJECT( qualname ); result->m_qualname = qualname; Py_INCREF( qualname ); // TODO: Makes no sense with coroutines maybe? result->m_yieldfrom = NULL; // We take ownership of those and received the reference count from the // caller. result->m_closure = closure; result->m_closure_given = closure_given; result->m_weakrefs = NULL; result->m_status = status_Unused; result->m_running = false; result->m_exception_type = NULL; result->m_exception_value = NULL; result->m_exception_tb = NULL; result->m_yielded = NULL; result->m_returned = NULL; result->m_frame = NULL; result->m_code_object = code_object; initFiber( &result->m_yielder_context ); Nuitka_GC_Track( result ); return (PyObject *)result; } extern PyObject *PyGen_Send( PyGenObject *gen, PyObject *arg ); extern PyObject *const_str_plain_send; static int gen_is_coroutine(PyObject *o) { if ( PyGen_CheckExact(o) ) { PyCodeObject *code = (PyCodeObject *)((PyGenObject*)o)->gi_code; if ( code->co_flags & CO_ITERABLE_COROUTINE ) { return 1; } } return 0; } static PyObject *PyCoro_GetAwaitableIter( PyObject *value ) { unaryfunc getter = NULL; if ( PyCoro_CheckExact( value ) || gen_is_coroutine( value ) ) { Py_INCREF( value ); return value; } if ( Py_TYPE( value )->tp_as_async != NULL ) { getter = Py_TYPE( value )->tp_as_async->am_await; } if ( getter != NULL ) { PyObject *result = (*getter)( value ); if ( result != NULL ) { if (unlikely( PyCoro_CheckExact( result ) || gen_is_coroutine( result ) || Nuitka_Coroutine_Check( result ) )) { Py_DECREF( result ); PyErr_Format( PyExc_TypeError, "__await__() returned a coroutine" ); return NULL; } if (unlikely( !HAS_ITERNEXT( result ) )) { PyErr_Format( PyExc_TypeError, "__await__() returned non-iterator of type '%s'", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } } return result; } PyErr_Format( PyExc_TypeError, "object %s can't be used in 'await' expression", Py_TYPE( value )->tp_name ); return NULL; } static void RAISE_GENERATOR_EXCEPTION( Nuitka_CoroutineObject *generator ) { CHECK_OBJECT( generator->m_exception_type ); RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } extern PyObject *ERROR_GET_STOP_ITERATION_VALUE(); extern PyObject *const_str_plain_send, *const_str_plain_throw, *const_str_plain_close; static PyObject *yieldFromCoroutine( Nuitka_CoroutineObject *generator, PyObject *value ) { CHECK_OBJECT( value ); // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( generator->m_exception_type ) { // The yielding generator is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( generator->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL && !EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } } RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( generator->m_exception_type ); Py_CLEAR( generator->m_exception_value ); Py_CLEAR( generator->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) || PyCoro_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); } // Check the sub-generator result if ( retval == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { Py_INCREF( Py_None ); return Py_None; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { generator->m_yielded = retval; generator->m_yieldfrom = value; // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); generator->m_yieldfrom = NULL; send_value = generator->m_yielded; CHECK_OBJECT( send_value ); } } } PyObject *AWAIT_COROUTINE( Nuitka_CoroutineObject *coroutine, PyObject *awaitable ) { #if _DEBUG_COROUTINE PRINT_STRING("AWAIT entry:"); PRINT_ITEM( awaitable ); PRINT_NEW_LINE(); #endif PyObject *awaitable_iter = PyCoro_GetAwaitableIter( awaitable ); if (unlikely( awaitable_iter == NULL )) { return NULL; } PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("AWAIT exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } PyObject *MAKE_ASYNC_ITERATOR( Nuitka_CoroutineObject *coroutine, PyObject *value ) { #if _DEBUG_COROUTINE PRINT_STRING("AITER entry:"); PRINT_ITEM( value ); PRINT_NEW_LINE(); #endif unaryfunc getter = NULL; if ( Py_TYPE( value )->tp_as_async ) { getter = Py_TYPE( value )->tp_as_async->am_aiter; } if (unlikely( getter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' requires an object with __aiter__ method, got %s", Py_TYPE( value )->tp_name ); return NULL; } PyObject *iter = (*getter)( value ); if (unlikely( iter == NULL )) { return NULL; } PyObject *awaitable_iter = PyCoro_GetAwaitableIter( iter ); if (unlikely( awaitable_iter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' received an invalid object from __aiter__: %s", Py_TYPE( iter )->tp_name ); Py_DECREF( iter ); return NULL; } Py_DECREF( iter ); PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("AITER exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } PyObject *ASYNC_ITERATOR_NEXT( Nuitka_CoroutineObject *coroutine, PyObject *value ) { #if _DEBUG_COROUTINE PRINT_STRING("ANEXT entry:"); PRINT_ITEM( value ); PRINT_NEW_LINE(); #endif unaryfunc getter = NULL; if ( Py_TYPE( value )->tp_as_async ) { getter = Py_TYPE( value )->tp_as_async->am_anext; } if (unlikely( getter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' requires an iterator with __anext__ method, got %s", Py_TYPE( value )->tp_name ); return NULL; } PyObject *next_value = (*getter)( value ); if (unlikely( next_value == NULL )) { return NULL; } PyObject *awaitable_iter = PyCoro_GetAwaitableIter( next_value ); if (unlikely( awaitable_iter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' received an invalid object from __anext__: %s", Py_TYPE( next_value )->tp_name ); Py_DECREF( next_value ); return NULL; } Py_DECREF( next_value ); PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("ANEXT exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } Nuitka-0.5.18.1/nuitka/build/static_src/libcoro_ucontext_src/0000755000175000017500000000000012651072347024421 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/build/static_src/libcoro_ucontext_src/fibers_coro.cpp0000644000175000017500000000361112651071040027407 0ustar hayenhayen00000000000000// Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for generic targets. #include "nuitka/prelude.hpp" extern "C" { #include "coro.h" } // TODO: Make stack size rational. #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { /* Not much to do. */ to->sptr = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { /* Need to allocate stack manually. */ to->sptr = last_stack ? (char *)last_stack : (char *)malloc( STACK_SIZE ); coro_create( &to->coro_ctx, (coro_func)code, (void *)arg, to->sptr, STACK_SIZE ); return 0; } void _releaseFiber( Fiber *to ) { if ( to->sptr != NULL ) { if ( last_stack == NULL && false ) { last_stack = to->sptr; } else { free( to->sptr ); } to->sptr = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { assert( to->sptr ); assert( from->sptr ); coro_transfer( &to->coro_ctx, &from->coro_ctx ); } Nuitka-0.5.18.1/nuitka/build/__init__.py0000644000175000017500000000150112651071040020125 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/finalizations/0000755000175000017500000000000012651072347017605 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/finalizations/Finalization.py0000644000175000017500000000326112651071040022575 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Finalizations. Last steps directly before code creation is called. Here the final tasks are executed. Things normally volatile during optimization can be computed here, so the code generation can be quick and doesn't have to check it many times. """ from nuitka.tree import Operations from .FinalizeClosureTaking import FinalizeClassClosure, FinalizeClosureTaking from .FinalizeMarkups import FinalizeMarkups def prepareCodeGeneration(tree): visitor = FinalizeMarkups() Operations.visitTree(tree, visitor) for function in tree.getUsedFunctions(): Operations.visitTree(function, visitor) visitor = FinalizeClassClosure() for function in tree.getUsedFunctions(): if function.hasFlag("has_super"): Operations.visitFunction(function, visitor) visitor = FinalizeClosureTaking() for function in tree.getUsedFunctions(): Operations.visitFunction(function, visitor) Nuitka-0.5.18.1/nuitka/finalizations/FinalizeClosureTaking.py0000644000175000017500000000476512651071040024414 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Finalize the closure. If a taker wants a variable, make sure that the closure taker in between all do forward it for this use or else it will not be available. We do this late so it is easier to remove closure variables and keep track of references, by not having it spoiled with these transitive only references. """ from nuitka.optimizations.Optimization import areEmptyTraces from .FinalizeBase import FinalizationVisitorBase class FinalizeClosureTaking(FinalizationVisitorBase): def onEnterNode(self, node): # print node, node.provider for variable in node.getClosureVariables(): assert not variable.isModuleVariable() current = node while current is not variable.getOwner(): if current.isParentVariableProvider(): if variable not in current.getClosureVariables(): current.addClosureVariable(variable) # Detect loops in the provider relationship assert current.getParentVariableProvider() is not current current = current.getParentVariableProvider() # Not found?! assert current is not None, variable class FinalizeClassClosure(FinalizationVisitorBase): def onEnterNode(self, function_body): for closure_variable in function_body.getClosureVariables(): if closure_variable.getName() not in ("__class__", "self"): continue variable_traces = function_body.constraint_collection.getVariableTraces( variable = closure_variable ) empty = areEmptyTraces(variable_traces) if empty: function_body.removeClosureVariable(closure_variable) Nuitka-0.5.18.1/nuitka/finalizations/FinalizeMarkups.py0000644000175000017500000001520112651071040023247 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Finalize the markups Set flags on functions and classes to indicate if a locals dict is really needed. Set a flag on loops if they really need to catch Continue and Break exceptions or if it can be more simple code. Set a flag on return statements and functions that require the use of "ReturnValue" exceptions, or if it can be more simple code. Set a flag on re-raises of exceptions if they can be simple throws or if they are in another context. """ from logging import warning from nuitka import Options, Tracing from nuitka.importing import StandardLibrary from nuitka.plugins.PluginBase import Plugins from nuitka.PythonVersions import python_version from .FinalizeBase import FinalizationVisitorBase def isWhiteListedImport(node): module = node.getParentModule() return StandardLibrary.isStandardLibraryPath(module.getFilename()) class FinalizeMarkups(FinalizationVisitorBase): def onEnterNode(self, node): try: self._onEnterNode(node) except Exception: Tracing.printError( "Problem with %r at %s" % ( node, node.getSourceReference().getAsString() ) ) raise def _onEnterNode(self, node): # This has many different things it deals with, so there need to be a # lot of branches and statements, pylint: disable=R0912,R0915 # Also all self specific things have been done on the outside, # pylint: disable=R0201 # Find nodes with only compile time constant children, these are # missing some obvious optimization potentially. if False: # For searching only, pylint: disable=W0125 if not node.isStatementReturn() and \ not node.isExpressionYield() and \ not node.isStatementRaiseException() and \ not node.isExpressionCall() and \ not node.isExpressionBuiltinIter1(): children = node.getVisitableNodes() if children: for child in children: if child.isStatement() or child.isStatementsSequence(): break if not child.isCompileTimeConstant(): break else: assert False, (node, node.parent, children) if node.isExpressionFunctionBody(): if node.isUnoptimized(): node.markAsLocalsDict() if node.needsLocalsDict(): provider = node.getParentVariableProvider() if not provider.isCompiledPythonModule(): provider.markAsLocalsDict() if node.isStatementReturn() or node.isStatementGeneratorReturn(): search = node in_tried_block = False # Search up to the containing function, and check for a try/finally # containing the "return" statement. search = search.getParentReturnConsumer() if search.isExpressionGeneratorObjectBody() or \ search.isExpressionCoroutineObjectBody(): if in_tried_block: search.markAsNeedsGeneratorReturnHandling(2) else: search.markAsNeedsGeneratorReturnHandling(1) if node.isExpressionBuiltinImport() and \ not Options.getShallFollowExtra() and \ not Options.getShallFollowExtraFilePatterns() and \ not Options.shallFollowNoImports() and \ not isWhiteListedImport(node) and \ not Plugins.suppressBuiltinImportWarning(node.getParentModule(), node.getSourceReference()): warning("""Unresolved '__import__' call at '%s' may require use \ of '--recurse-directory'.""" % ( node.getSourceReference().getAsString() ) ) if node.isExpressionFunctionCreation(): if not node.getParent().isExpressionFunctionCall() or \ node.getParent().getFunction() is not node: node.getFunctionRef().getFunctionBody().markAsNeedsCreation() if node.isExpressionFunctionCall(): node.getFunction().getFunctionRef().getFunctionBody().\ markAsDirectlyCalled() if node.isExpressionFunctionRef(): function_body = node.getFunctionBody() parent_module = function_body.getParentModule() node_module = node.getParentModule() if node_module is not parent_module: function_body.markAsCrossModuleUsed() node_module.addCrossUsedFunction(function_body) if node.isStatementAssignmentVariable(): target_var = node.getTargetVariableRef().getVariable() assign_source = node.getAssignSource() if assign_source.isExpressionOperationBinary(): left_arg = assign_source.getLeft() if left_arg.isExpressionVariableRef(): if assign_source.getLeft().getVariable().isModuleVariable(): assign_source.unmarkAsInplaceSuspect() elif assign_source.getLeft().getVariable() is target_var: if assign_source.isInplaceSuspect(): node.markAsInplaceSuspect() if node.isStatementPublishException(): node.getParentStatementsFrame().markAsFrameExceptionPreserving() if python_version >= 300: if node.isExpressionYield() or node.isExpressionYieldFrom(): search = node.getParent() while not search.isExpressionGeneratorObjectBody(): last_search = search search = search.getParent() if search.isStatementTry() and \ last_search == search.getBlockExceptHandler(): node.markAsExceptionPreserving() break Nuitka-0.5.18.1/nuitka/finalizations/__init__.py0000644000175000017500000000150112651071040021700 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/finalizations/FinalizeBase.py0000644000175000017500000000174712651071040022511 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Base for all finalization modules Provides a class that all finalization visitors should inherit from. """ from nuitka.tree import Operations class FinalizationVisitorBase(Operations.VisitorNoopMixin): pass Nuitka-0.5.18.1/nuitka/plugins/0000755000175000017500000000000012651072347016414 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/plugins/PluginBase.py0000644000175000017500000004070412651071040021011 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Plugins: Welcome to Nuitka! This is your shortest way to become part of it. This is to provide the base class for all plug-ins. Some of which are part of proper Nuitka, and some of which are waiting to be created and submitted for inclusion by you. The base class will serve as documentation. And it will point to examples of it being used. """ # This is heavily WIP. import sys from logging import info, warning from nuitka import Options from nuitka.ModuleRegistry import addUsedModule from nuitka.PythonVersions import python_version from nuitka.SourceCodeReferences import fromFilename from nuitka.utils import Utils pre_modules = {} post_modules = {} warned_unused_plugins = set() class NuitkaPluginBase: """ Nuitka base class for all plug-ins. Derive from "UserPlugin" please. The idea is it will allow to make differences for warnings, and traces of what is being done. For instance, the code that makes sure PyQt finds all his stuff, may want to do reports, but likely, you do not case about that enough to be visible by default. """ # You must provide this as a string which then can be used to enable the # plug-in. plugin_name = None def considerImplicitImports(self, module, signal_change): """ Consider module imports. You will most likely want to look at "module.getFullName()" to get the fully qualified module or package name. You do not want to overload this method, but rather the things it calls, as the "signal_change" part of this API is not to be cared about. Most prominently "getImplicitImports()". """ for full_name in self.getImplicitImports(module.getFullName()): module_name = full_name.split('.')[-1] module_package = '.'.join(full_name.split('.')[:-1]) or None module_filename = self.locateModule( importing = module, module_name = module_name, module_package = module_package, ) if module_filename is None: sys.exit( "Error, implicit module '%s' expected by '%s' not found." % ( full_name, module.getFullName() ) ) elif Utils.isDir(module_filename): module_kind = "py" elif module_filename.endswith(".py"): module_kind = "py" elif module_filename.endswith(".so") or \ module_filename.endswith(".pyd"): module_kind = "shlib" else: assert False, module_filename # TODO: This should get back to plug-ins, they should be allowed to # preempt or override the decision. decision, reason = self.decideRecursion( module_filename = module_filename, module_name = module_name, module_package = module_package, module_kind = module_kind ) if decision: self.recurseTo( module_package = module_package, module_filename = module_filename, module_kind = module_kind, reason = reason, signal_change = signal_change ) def getImplicitImports(self, full_name): # Virtual method, pylint: disable=R0201,W0613 return () # Provide fall-back for failed imports here. module_aliases = {} def considerFailedImportReferrals(self, module_name): return self.module_aliases.get(module_name, None) def onModuleSourceCode(self, module_name, source_code): # Virtual method, pylint: disable=R0201,W0613 return source_code @staticmethod def _createTriggerLoadedModule(module, trigger_name, code): from nuitka.tree.Building import createModuleTree from nuitka.nodes.ModuleNodes import CompiledPythonModule trigger_module = CompiledPythonModule( name = module.getName() + trigger_name, package_name = module.getPackage(), source_ref = fromFilename(module.getCompileTimeFilename() + trigger_name) ) createModuleTree( module = trigger_module, source_ref = module.getSourceReference(), source_code = code, is_main = False ) return trigger_module @staticmethod def createPreModuleLoadCode(module): # Virtual method, pylint: disable=W0613 return None, None @staticmethod def createPostModuleLoadCode(module): # Virtual method, pylint: disable=W0613 return None, None def onModuleDiscovered(self, module): pre_code, reason = self.createPreModuleLoadCode(module) full_name = module.getFullName() if pre_code: if full_name is pre_modules: sys.exit("Error, conflicting plug-ins for %s" % full_name) info( "Injecting plug-in based pre load code for module '%s':" % \ full_name ) for line in reason.split('\n'): info(" " + line) pre_modules[full_name] = self._createTriggerLoadedModule( module = module, trigger_name = "-preLoad", code = pre_code ) post_code, reason = self.createPostModuleLoadCode(module) if post_code: if full_name is post_modules: sys.exit("Error, conflicting plug-ins for %s" % full_name) info( "Injecting plug-in based post load code for module '%s':" % \ full_name ) for line in reason.split('\n'): info(" " + line) post_modules[full_name] = self._createTriggerLoadedModule( module = module, trigger_name = "-postLoad", code = post_code ) def onModuleEncounter(self, module_filename, module_name, module_package, module_kind): pass @staticmethod def locateModule(importing, module_name, module_package): from nuitka.importing import Importing _module_package, module_filename, _finding = Importing.findModule( importing = importing, module_name = module_name, parent_package = module_package, level = -1, warn = True ) return module_filename @staticmethod def decideRecursion(module_filename, module_name, module_package, module_kind): from nuitka.importing import Recursion decision, reason = Recursion.decideRecursion( module_filename = module_filename, module_name = module_name, module_package = module_package, module_kind = module_kind ) return decision, reason @staticmethod def recurseTo(module_package, module_filename, module_kind, reason, signal_change): from nuitka.importing import Recursion imported_module, added_flag = Recursion.recurseTo( module_package = module_package, module_filename = module_filename, module_relpath = Utils.relpath(module_filename), module_kind = module_kind, reason = reason ) addUsedModule(imported_module) if added_flag: signal_change( "new_code", imported_module.getSourceReference(), "Recursed to module." ) def considerExtraDlls(self, dist_dir, module): # Virtual method, pylint: disable=R0201,W0613 return [] def suppressBuiltinImportWarning(self, module_name, source_ref): # Virtual method, pylint: disable=R0201,W0613 return False def suppressUnknownImportWarning(self, importing, module_name, source_ref): # Virtual method, pylint: disable=R0201,W0613 return False def warnUnusedPlugin(self, message): if self.plugin_name not in warned_unused_plugins: warned_unused_plugins.add(self.plugin_name) warning( "Use --plugin-enable=%s for: %s" % ( self.plugin_name, message ) ) class NuitkaPluginPopularImplicitImports(NuitkaPluginBase): """ This is for implicit imports. When C extension modules import other modules, we cannot see this and need to be told that. This encodes the knowledge we have for various modules. Feel free to add to this and submit patches to make it more complete. """ def getImplicitImports(self, full_name): elements = full_name.split('.') if elements[0] in ("PyQt4", "PyQt5"): if python_version < 300: yield "atexit" yield "sip" child = elements[1] if len(elements) > 1 else None if child == "QtGui": yield elements[0] + ".QtCore" if child == "QtWidgets": yield elements[0] + ".QtGui" elif full_name == "lxml.etree": yield "gzip" yield "lxml._elementpath" elif full_name == "gtk._gtk": yield "pangocairo" yield "pango" yield "cairo" yield "gio" yield "atk" elif full_name == "reportlab.rl_config": yield "reportlab.rl_settings" elif full_name == "ctypes": yield "_ctypes" elif full_name == "gi._gi": yield "gi._error" module_aliases = { "requests.packages.urllib3" : "urllib3", "requests.packages.chardet" : "chardet" } def onModuleSourceCode(self, module_name, source_code): if module_name == "numexpr.cpuinfo": # We cannot intercept "is" tests, but need it to be "isinstance", # so we patch it on the file. TODO: This is only temporary, in # the future, we may use optimization that understands the right # hand size of the "is" argument well enough to allow for our # type too. return source_code.replace( "type(attr) is types.MethodType", "isinstance(attr, types.MethodType)" ) # Do nothing by default. return source_code def suppressBuiltinImportWarning(self, module_name, source_ref): if module_name == "setuptools": return True return False class UserPluginBase(NuitkaPluginBase): """ For user plug-ins. Check the base class methods for what you can do. """ # You must provide this as a string which then can be used to enable the # plug-in. plugin_name = None active_plugin_list = [ NuitkaPluginPopularImplicitImports(), ] from .standard.ConsiderPyLintAnnotationsPlugin import ( # isort:skip NuitkaPluginDetectorPylintEclipseAnnotations, NuitkaPluginPylintEclipseAnnotations ) from .standard.MultiprocessingPlugin import ( # isort:skip NuitkaPluginDetectorMultiprocessingWorkaorunds, NuitkaPluginMultiprocessingWorkaorunds ) from .standard.PySidePyQtPlugin import ( # isort:skip NuitkaPluginDetectorPyQtPySidePlugins, NuitkaPluginPyQtPySidePlugins ) # The standard plug-ins have their list hard-coded here. User plug-ins will # be scanned later, TODO. # List of optional plug-in classes. Until we have the meta class to do it, just # add your class here. The second one is a detector, which is supposed to give # a missing plug-in message, should it find the condition to make it useful. optional_plugin_classes = ( (NuitkaPluginMultiprocessingWorkaorunds, NuitkaPluginDetectorMultiprocessingWorkaorunds), (NuitkaPluginPyQtPySidePlugins, NuitkaPluginDetectorPyQtPySidePlugins), (NuitkaPluginPylintEclipseAnnotations, NuitkaPluginDetectorPylintEclipseAnnotations), ) plugin_name2plugin_classes = dict( (plugin[0].plugin_name, plugin) for plugin in optional_plugin_classes ) for plugin_name in Options.getPluginsEnabled() + Options.getPluginsDisabled(): if plugin_name not in plugin_name2plugin_classes: sys.exit("Error, unknown plug-in '%s' referenced." % plugin_name) if plugin_name in Options.getPluginsEnabled() and \ plugin_name in Options.getPluginsDisabled(): sys.exit("Error, conflicting enable/disable of plug-in '%s'." % plugin_name) for plugin_name, (plugin_class, plugin_detector) in plugin_name2plugin_classes.items(): if plugin_name in Options.getPluginsEnabled(): active_plugin_list.append( plugin_class( **Options.getPluginOptions(plugin_name) ) ) elif plugin_name not in Options.getPluginsDisabled(): if plugin_detector is not None \ and Options.shallDetectMissingPlugins() and \ plugin_detector.isRelevant(): active_plugin_list.append( plugin_detector() ) class Plugins: @staticmethod def considerImplicitImports(module, signal_change): for plugin in active_plugin_list: plugin.considerImplicitImports(module, signal_change) # Post load code may have been created, if so indicate it's used. full_name = module.getFullName() if full_name in post_modules: addUsedModule(post_modules[full_name]) if full_name in pre_modules: addUsedModule(pre_modules[full_name]) @staticmethod def considerExtraDlls(dist_dir, module): result = [] for plugin in active_plugin_list: for extra_dll in plugin.considerExtraDlls(dist_dir, module): assert Utils.isFile(extra_dll[0]) result.append(extra_dll) return result @staticmethod def onModuleDiscovered(module): for plugin in active_plugin_list: plugin.onModuleDiscovered(module) @staticmethod def onModuleSourceCode(module_name, source_code): assert type(module_name) is str assert type(source_code) is str for plugin in active_plugin_list: source_code = plugin.onModuleSourceCode(module_name, source_code) return source_code @staticmethod def onModuleEncounter(module_filename, module_name, module_package, module_kind): for plugin in active_plugin_list: plugin.onModuleEncounter( module_filename, module_name, module_package, module_kind ) @staticmethod def considerFailedImportReferrals(module_name): for plugin in active_plugin_list: new_module_name = plugin.considerFailedImportReferrals(module_name) if new_module_name is not None: return new_module_name return None @staticmethod def suppressBuiltinImportWarning(module_name, source_ref): for plugin in active_plugin_list: if plugin.suppressBuiltinImportWarning(module_name, source_ref): return True return False @staticmethod def suppressUnknownImportWarning(importing, module_name): if importing.isCompiledPythonModule(): importing_module = importing else: importing_module = importing.getParentModule() source_ref = importing.getSourceReference() for plugin in active_plugin_list: if plugin.suppressUnknownImportWarning(importing_module, module_name, source_ref): return True return False Nuitka-0.5.18.1/nuitka/plugins/user/0000755000175000017500000000000012651072347017372 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/plugins/user/__init__.py0000644000175000017500000000150112651071040021465 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/plugins/standard/0000755000175000017500000000000012651072347020214 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/plugins/standard/ConsiderPyLintAnnotationsPlugin.py0000644000175000017500000000632512651071040027064 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Standard plug-in to take advantage of pylint or PyDev annotations. Nuitka can detect some things that PyLint and PyDev will complain about too, and sometimes it's a false alarm, so people add disable markers into their source code. Nuitka does it itself. This tries to parse the code for these markers and uses hooks to prevent Nuitka from warning about things, disabled to PyLint or Eclipse. The idea is that we won't have another mechanism for Nuitka, but use existing ones instead. The code for this is very incomplete, barely good enough to cover Nuitka's own usage of PyLint markers. PyDev is still largely to be started. You are welcome to grow both. """ import re from nuitka.plugins.PluginBase import NuitkaPluginBase class NuitkaPluginPylintEclipseAnnotations(NuitkaPluginBase): plugin_name = "pylint-warnings" def __init__(self): self.line_annotations = {} def onModuleSourceCode(self, module_name, source_code): annotations = {} for count, line in enumerate(source_code.split('\n')): match = re.search(r"#.*pylint:\s*disable=\s*([\w,]+)", line) if match: comment_only = line[:line.find('#')-1].strip() == "" if comment_only: # TODO: Parse block wide annotations too. pass else: annotations[count+1] = set( match.strip() for match in match.group(1).split(',') ) # Only remember them if there were any. if annotations: self.line_annotations[module_name] = annotations # Do nothing to it. return source_code def suppressUnknownImportWarning(self, importing, module_name, source_ref): annotations = self.line_annotations.get(importing.getFullName(), {}) line_annotations = annotations.get(source_ref.getLineNumber(), ()) if "F0401" in line_annotations: return True return False class NuitkaPluginDetectorPylintEclipseAnnotations(NuitkaPluginBase): plugin_name = "pylint-warnings" @staticmethod def isRelevant(): return True def onModuleSourceCode(self, module_name, source_code): if re.search(r"#\s*pylint:\s*disable=\s*(\w+)", source_code): self.warnUnusedPlugin("Understand PyLint/PyDev annotations for warnings.") # Do nothing to it. return source_code Nuitka-0.5.18.1/nuitka/plugins/standard/MultiprocessingPlugin.py0000644000175000017500000001246512651071040025131 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Standard plug-in to make multiprocessing work well on Windows. On Windows, the multiprocessing modules forks new processes which then have to start from scratch. This won't work if there is no "sys.executable" to point to a "Python.exe" and won't use compiled code by default. The issue applies to accelerated and standalone mode alike. """ from nuitka import Options from nuitka.plugins.PluginBase import NuitkaPluginBase from nuitka.utils import Utils class NuitkaPluginMultiprocessingWorkaorunds(NuitkaPluginBase): """ This is to make multiprocess work with Nuitka and use compiled code. When running in accelerated mode, it's not good to fork a new Python instance to run other code, as that won't be accelerated. And when run in standalone mode, there may not even be a Python, but it's the same principle. So by default, this module is on and works around the behavior of the "multiprocess.forking" expectations. """ plugin_name = "multiprocessing" def __init__(self): self.multiprocessing_added = False @staticmethod def createPreModuleLoadCode(module): full_name = module.getFullName() if full_name == "multiprocessing.forking": code = """\ import sys sys.frozen = 1 sys.executable = sys.argv[0] """ return code, """\ Monkey patching "multiprocessing" load environment.""" return None, None @staticmethod def createPostModuleLoadCode(module): full_name = module.getFullName() if full_name == "multiprocessing.forking": code = """\ from multiprocessing.forking import ForkingPickler class C: def f(): pass def _reduce_compiled_method(m): if m.im_self is None: return getattr, (m.im_class, m.im_func.__name__) else: return getattr, (m.im_self, m.im_func.__name__) print type(_reduce_compiled_method) ForkingPickler.register(type(C.f), _reduce_compiled_method) """ return code, """\ Monkey patching "multiprocessing" for compiled methods.""" return None, None def onModuleEncounter(self, module_filename, module_name, module_package, module_kind): if module_name == "multiprocessing" and \ module_package is None \ and not self.multiprocessing_added: self.multiprocessing_added = True from nuitka.ModuleRegistry import getRootModules, addRootModule from nuitka.tree.Building import CompiledPythonModule, readSourceCodeFromFilename, createModuleTree for root_module in getRootModules(): if root_module.isMainModule(): # First, build the module node and then read again from the # source code. slave_main_module = CompiledPythonModule( name = "__parents_main__", package_name = None, source_ref = root_module.getSourceReference() ) source_code = readSourceCodeFromFilename( "__parents_main__", root_module.getFilename() ) # For the call stack, this may look bad or different to what # CPython does. Using the "__import__" built-in to not spoil # or use the module namespace. source_code += """ __import__("sys").modules["__main__"] = __import__("sys").modules[__name__] __import__("multiprocessing.forking").forking.main()""" createModuleTree( module = slave_main_module, source_ref = root_module.getSourceReference(), source_code = source_code, is_main = False ) # This is an alternative entry point of course. addRootModule(slave_main_module) break else: assert False class NuitkaPluginDetectorMultiprocessingWorkaorunds(NuitkaPluginBase): plugin_name = "multiprocessing" @staticmethod def isRelevant(): return Utils.getOS() == "Windows" and not Options.shallMakeModule() def onModuleSourceCode(self, module_name, source_code): if module_name == "__main__": if "multiprocessing" in source_code and "freeze_support" in source_code: self.warnUnusedPlugin("Multiprocessing workarounds for compiled code on Windows.") return source_code Nuitka-0.5.18.1/nuitka/plugins/standard/PySidePyQtPlugin.py0000644000175000017500000001117212651071040023747 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Standard plug-in to make PyQt and PySide work well in standalone mode. To run properly, these need the Qt plug-ins copied along, which have their own dependencies. """ import shutil import subprocess import sys from logging import info from nuitka import Options from nuitka.plugins.PluginBase import NuitkaPluginBase from nuitka.PythonVersions import python_version from nuitka.utils import Utils class NuitkaPluginPyQtPySidePlugins(NuitkaPluginBase): """ This is for plugins of PySide/PyQt4/PyQt5. When loads an image, it may use a plug-in, which in turn used DLLs, which for standalone mode, can cause issues of not having it. """ plugin_name = "qt-plugins" @staticmethod def getPyQtPluginDirs(qt_version): command = """\ from __future__ import print_function import PyQt%(qt_version)d.QtCore for v in PyQt%(qt_version)d.QtCore.QCoreApplication.libraryPaths(): print(v) import os guess_path = os.path.join(os.path.dirname(PyQt%(qt_version)d.__file__), "plugins") if os.path.exists(guess_path): print("GUESS:", guess_path) """ % { "qt_version" : qt_version } output = subprocess.check_output([sys.executable, "-c", command]) # May not be good for everybody, but we cannot have bytes in paths, or # else working with them breaks down. if python_version >= 300: output = output.decode("utf-8") result = [] for line in output.replace('\r', "").split('\n'): if not line: continue # Take the guessed path only if necessary. if line.startswith("GUESS: "): if result: continue line = line[len("GUESS: "):] result.append(Utils.normpath(line)) return result def considerExtraDlls(self, dist_dir, module): full_name = module.getFullName() if full_name in ("PyQt4", "PyQt5"): qt_version = int(full_name[-1]) plugin_dir, = self.getPyQtPluginDirs(qt_version) target_plugin_dir = Utils.joinpath( dist_dir, full_name, "qt-plugins" ) shutil.copytree( plugin_dir, target_plugin_dir ) info("Copying all Qt plug-ins to '%s'." % target_plugin_dir) return [ (filename, full_name) for filename in Utils.getFileList(target_plugin_dir) ] return () @staticmethod def createPostModuleLoadCode(module): """ Create code to load after a module was successfully imported. For Qt we need to set the library path to the distribution folder we are running from. The code is immediately run after the code and therefore makes sure it's updated properly. """ full_name = module.getFullName() if full_name in ("PyQt4.QtCore", "PyQt5.QtCore"): qt_version = int(full_name.split('.')[0][-1]) code = """\ from PyQt%(qt_version)d.QtCore import QCoreApplication import os QCoreApplication.setLibraryPaths( [ os.path.join( os.path.dirname(__file__), "qt-plugins" ) ] ) """ % { "qt_version" : qt_version } return code, """\ Setting Qt library path to distribution folder. Need to avoid loading target system Qt plug-ins, which may be from another Qt version.""" return None, None class NuitkaPluginDetectorPyQtPySidePlugins(NuitkaPluginBase): plugin_name = "qt-plugins" @staticmethod def isRelevant(): return Options.isStandaloneMode() def onModuleDiscovered(self, module): if module.getFullName() in ("PyQt4.QtCore", "PyQt5.QtCore", "PySide"): self.warnUnusedPlugin("Inclusion of Qt plugins.") Nuitka-0.5.18.1/nuitka/plugins/standard/__init__.py0000644000175000017500000000150112651071040022307 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/plugins/__init__.py0000644000175000017500000000150112651071040020507 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/Variables.py0000644000175000017500000001173312651072046017216 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Variables link the storage and use of a Python variable together. Different kinds of variables represent different scopes and owners types, and their links between each other, i.e. references as in closure or module variable references. """ from nuitka.utils import InstanceCounters, Utils class Variable: @InstanceCounters.counted_init def __init__(self, owner, variable_name): assert type(variable_name) is str, variable_name assert type(owner) not in (tuple, list), owner assert owner.getFullName self.variable_name = variable_name self.owner = owner self.read_only_indicator = None self.version_number = 0 __del__ = InstanceCounters.counted_del() def getName(self): return self.variable_name def getCodeName(self): var_name = self.variable_name var_name = var_name.replace('.', '$') var_name = Utils.encodeNonAscii(var_name) return var_name def getOwner(self): return self.owner def getReadOnlyIndicator(self): return self.read_only_indicator def setReadOnlyIndicator(self, value): assert value in (True, False) self.read_only_indicator = value def allocateTargetNumber(self): self.version_number += 1 return self.version_number # pylint: disable=R0201 def isLocalVariable(self): return False def isMaybeLocalVariable(self): return False def isParameterVariable(self): return False def isModuleVariable(self): return False def isTempVariable(self): return False # pylint: enable=R0201 def isSharedLogically(self): from nuitka.VariableRegistry import isSharedLogically return isSharedLogically(self) def isSharedTechnically(self): from nuitka.VariableRegistry import isSharedTechnically return isSharedTechnically(self) class LocalVariable(Variable): def __init__(self, owner, variable_name): Variable.__init__( self, owner = owner, variable_name = variable_name ) def __repr__(self): return "<%s '%s' of '%s'>" % ( self.__class__.__name__, self.variable_name, self.owner.getName() ) def isLocalVariable(self): return True class MaybeLocalVariable(Variable): def __init__(self, owner, maybe_variable): Variable.__init__( self, owner = owner, variable_name = maybe_variable.getName() ) self.maybe_variable = maybe_variable def __repr__(self): return "<%s '%s' of '%s' maybe '%s'" % ( self.__class__.__name__, self.variable_name, self.owner.getName(), self.maybe_variable ) def isMaybeLocalVariable(self): return True def getMaybeVariable(self): return self.maybe_variable class ParameterVariable(LocalVariable): def __init__(self, owner, parameter_name): LocalVariable.__init__( self, owner = owner, variable_name = parameter_name ) def isParameterVariable(self): return True class ModuleVariable(Variable): def __init__(self, module, variable_name): assert type(variable_name) is str, repr(variable_name) assert module.isCompiledPythonModule() Variable.__init__( self, owner = module, variable_name = variable_name ) self.module = module def __repr__(self): return "" % ( self.variable_name, self.getModule().getFullName() ) def isModuleVariable(self): return True def getModule(self): return self.module class TempVariable(Variable): def __init__(self, owner, variable_name): Variable.__init__( self, owner = owner, variable_name = variable_name ) def __repr__(self): return "" % ( self.getName(), self.getOwner() ) def isTempVariable(self): return True Nuitka-0.5.18.1/nuitka/containers/0000755000175000017500000000000012651072347017100 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/containers/__init__.py0000644000175000017500000000150112651071040021173 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/containers/oset.py0000644000175000017500000000610012651070406020413 0ustar hayenhayen00000000000000# Copyright 2009 Raymond Hettinger # # Permission is hereby granted, free of charge, to any person obtaining # a copy of this software and associated documentation files (the # "Software"), to deal in the Software without restriction, including # without limitation the rights to use, copy, modify, merge, publish, # distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so, subject to # the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. # # Note: Kay Hayen did some changes for Nuitka keeping this license. These # changes are not improvements, use the original source instead, not this # file. """ This module is only an abstraction of OrderedSet which is not present in Python at all. It was originally downloaded from http://code.activestate.com/recipes/576694/ """ # pylint: disable=W0221,W0622 import collections class OrderedSet(collections.MutableSet): def __init__(self, iterable = None): self.end = end = [] end += [None, end, end] # sentinel node for doubly linked list self.map = {} # key --> [key, prev, next] if iterable is not None: self |= iterable def __len__(self): return len(self.map) def __contains__(self, key): return key in self.map def add(self, key): if key not in self.map: end = self.end curr = end[1] curr[2] = end[1] = self.map[key] = [key, curr, end] def discard(self, key): if key in self.map: key, prev, next = self.map.pop(key) # @ReservedAssignment prev[2] = next next[1] = prev def __iter__(self): end = self.end curr = end[2] while curr is not end: yield curr[0] curr = curr[2] def __reversed__(self): end = self.end curr = end[1] while curr is not end: yield curr[0] curr = curr[1] def pop(self, last = True): if not self: raise KeyError("set is empty") key = self.end[1][0] if last else self.end[2][0] self.discard(key) return key def __repr__(self): if not self: return "%s()" % (self.__class__.__name__,) return "%s(%r)" % (self.__class__.__name__, list(self)) def __eq__(self, other): if isinstance(other, OrderedSet): return len(self) == len(other) and list(self) == list(other) return set(self) == set(other) Nuitka-0.5.18.1/nuitka/containers/odict.py0000644000175000017500000001413412651072046020553 0ustar hayenhayen00000000000000# :copyright: (c) 2008 by Armin Ronacher and PEP 273 authors. # :license: modified BSD license. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # * Neither the name of the nor the # names of its contributors may be used to endorse or promote products # derived from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE # DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY # DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES # (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; # LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # Kay Hayen did some changes for Nuitka, and put everything he added under the same # modified BSD license. """ This module is only an abstraction of OrderedDict as present in 2.7 and 3.1. It is not in 2.6, for this version we are using the odict.py as mentioned in the PEP-0372 This can be removed safely after the transition, note that the documentation was removed, as it's not interesting really, being redundant to Python 2.7 documentation. """ # pylint: disable=E0611,W0141 try: from collections import OrderedDict except ImportError: from itertools import izip, imap from copy import deepcopy missing = object() class OrderedDict(dict): def __init__(self, *args, **kwargs): dict.__init__(self) self._keys = [] self.update(*args, **kwargs) def __delitem__(self, key): dict.__delitem__(self, key) self._keys.remove(key) def __setitem__(self, key, item): if key not in self: self._keys.append(key) dict.__setitem__(self, key, item) def __deepcopy__(self, memo = None): if memo is None: memo = {} d = memo.get(id(self), missing) if d is not missing: return d memo[id(self)] = d = self.__class__() dict.__init__(d, deepcopy(self.items(), memo)) d._keys = self._keys[:] return d def __getstate__(self): return {"items": dict(self), "keys": self._keys} def __setstate__(self, d): self._keys = d["keys"] dict.update(d["items"]) def __reversed__(self): return reversed(self._keys) def __eq__(self, other): if isinstance(other, OrderedDict): if not dict.__eq__(self, other): return False return self.items() == other.items() return dict.__eq__(self, other) def __ne__(self, other): return not self.__eq__(other) def __cmp__(self, other): if isinstance(other, OrderedDict): return cmp(self.items(), other.items()) elif isinstance(other, dict): return dict.__cmp__(self, other) return NotImplemented @classmethod def fromkeys(cls, iterable, default = None): return cls((key, default) for key in iterable) def clear(self): del self._keys[:] dict.clear(self) def copy(self): return self.__class__(self) def items(self): return zip(self._keys, self.values()) def iteritems(self): return izip(self._keys, self.itervalues()) def keys(self): return self._keys[:] def iterkeys(self): return iter(self._keys) def pop(self, key, default = missing): if default is missing: return dict.pop(self, key) elif key not in self: return default self._keys.remove(key) return dict.pop(self, key, default) def popitem(self, key): self._keys.remove(key) return dict.popitem(key) def setdefault(self, key, default = None): if key not in self: self._keys.append(key) dict.setdefault(self, key, default) def update(self, *args, **kwargs): sources = [] if len(args) == 1: if hasattr(args[0], "iteritems"): sources.append(args[0].iteritems()) else: sources.append(iter(args[0])) elif args: raise TypeError("expected at most one positional argument") if kwargs: sources.append(kwargs.iteritems()) for iterable in sources: for key, val in iterable: self[key] = val def values(self): return map(self.get, self._keys) def itervalues(self): return imap(self.get, self._keys) def index(self, item): return self._keys.index(item) def byindex(self, item): key = self._keys[item] return (key, dict.__getitem__(self, key)) def reverse(self): self._keys.reverse() def sort(self, *args, **kwargs): self._keys.sort(*args, **kwargs) def __repr__(self): return "OrderedDict(%r)" % self.items() __copy__ = copy __iter__ = iterkeys Nuitka-0.5.18.1/nuitka/VariableRegistry.py0000644000175000017500000001346112651072046020564 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Registry of all variables. Makes information about variables available. Initially based solely on the level of what function took what closure, later on the basic of live traces giving information about what really needs to be closure taken. """ from nuitka.utils.InstanceCounters import counted_del, counted_init # The key is a plain variable object, not a reference. The value is a set # of information about that variable. variable_registry = {} class VariableInformation: @counted_init def __init__(self, variable): self.variable = variable self.users = set() self.active_users = set() def __repr__(self): return "<%s object for %s>" % ( self.__class__.__name__, self.variable ) __del__ = counted_del() def addUser(self, user): self.users.add(user) self.active_users.add(user) def removeUser(self, user): try: self.active_users.remove(user) except KeyError: raise KeyError(self, user) def getUsers(self): return self.users def getActiveUsers(self): return self.active_users def getTopOwner(self): return self.variable.getOwner() def addVariableUsage(variable, user): if variable.isMaybeLocalVariable(): addVariableUsage(variable.getMaybeVariable(), user) if variable not in variable_registry: variable_registry[variable] = VariableInformation(variable) variable_info = variable_registry[variable] variable_info.addUser(user) def removeVariableUsage(variable, user): variable_info = variable_registry[variable] variable_info.removeUser(user) # TODO: This seems practically unused and not needed. def isSharedLogically(variable): variable_info = variable_registry[variable] return len(variable_info.users) > 1 def isSharedAmongScopes(variable): variable_info = variable_registry[variable] if not variable.isParameterVariable(): return len(variable_info.users) > 1 count = 0 for user in variable_info.users: if user.isExpressionGeneratorObjectBody() or \ user.isExpressionCoroutineObjectBody(): if variable.getOwner() is user.getParentVariableProvider(): continue count += 1 if count > 1: return True return False def isSharedTechnically(variable): variable_info = variable_registry[variable] top_owner = variable_info.getTopOwner() variable_name = variable.getName() for user in variable_info.getActiveUsers(): # May have been optimized away, but then this would be false. assert user.hasVariableName(variable_name), (variable, user) while user is not top_owner and \ ( (user.isExpressionFunctionBody() and not user.needsCreation()) or \ user.isExpressionClassBody() ): user = user.getParentVariableProvider() if user is not top_owner: return True return False # The key is a variable name, and the value is a set of traces. variable_traces = {} class GlobalVariableTrace: @counted_init def __init__(self): self.traces = set() __del__ = counted_del() def add(self, variable_trace): self.traces.add(variable_trace) def remove(self, variable_trace): self.traces.remove(variable_trace) def hasDefiniteWrites(self): for trace in self.traces: if trace.isAssignTrace(): return True return False def getMatchingAssignTrace(self, assign_node): for trace in self.traces: if trace.isAssignTrace() and trace.getAssignNode() is assign_node: return trace return None def hasWritesOutsideOf(self, provider): for trace in self.traces: if trace.isAssignTrace(): if trace.getAssignNode().getParentVariableProvider() is not provider: return True return False def updateFromCollection(old_collection, new_collection): if old_collection is not None: for variable_trace in old_collection.getVariableTracesAll().values(): variable = variable_trace.getVariable() variable_traces[variable].remove(variable_trace) if new_collection is not None: for variable_trace in new_collection.getVariableTracesAll().values(): variable = variable_trace.getVariable() if variable not in variable_traces: variable_traces[variable] = GlobalVariableTrace() variable_traces[variable].add(variable_trace) # Release the memory, and prevent the "active" state from being ever # inspected, it's useless now. new_collection.variable_actives.clear() del new_collection.variable_actives complete = False def getGlobalVariableTrace(variable): # Global variable traces are not being handed out, before the first total # run was completed. if not complete: return None return variable_traces.get(variable, None) Nuitka-0.5.18.1/nuitka/gui/0000755000175000017500000000000012651072347015517 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/gui/TreeDisplay.py0000644000175000017500000002071512651071040020310 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module with functions to display a node tree. Useful to getting an idea of what the internal representation of Nuitka is about a source code. """ import sys from PyQt4 import QtCore, QtGui, uic # pylint: disable=F0401,I0021 from nuitka import SourceCodeReferences from nuitka.utils import Utils # The API requires a signature, sometimes we don't use it, pylint: disable=R0201 # Also using private stuff from classes, probably ok, pylint: disable=W0212 class NodeTreeModelItem: def __init__(self, node, parent = None): self.parent_treeitem = parent self.node = node self.children = None def appendChild(self, _item): assert False def _children(self): if self.children is None: self.children = [ NodeTreeModelItem(child, self) for child in self.node.getVisitableNodes() ] if self.node.isCompiledPythonModule(): self.children.extend( NodeTreeModelItem(child, self) for child in self.node.getUsedFunctions() ) return self.children def child(self, row): return self._children()[ row ] def childCount(self): return len(self._children()) def columnCount(self): return 2 def data(self, column): if column == 0: result = self.node.getDescription() elif column == 1: result = self.node.getDetail() else: assert False return QtCore.QVariant(result) def parent(self): return self.parent_treeitem def row(self): return self.parent_treeitem._children().index(self) if self.parent else 0 class NodeTreeModel(QtCore.QAbstractItemModel): def __init__(self, root, parent = None): QtCore.QAbstractItemModel.__init__(self, parent) self.root_node = root self.root_item = NodeTreeModelItem(root) def columnCount(self, _parent): return self.root_item.columnCount() def data(self, index, role): if not index.isValid(): return QtCore.QVariant() if role != QtCore.Qt.DisplayRole: return QtCore.QVariant() item = index.internalPointer() return QtCore.QVariant(item.data(index.column())) def flags(self, index): if not index.isValid(): return QtCore.Qt.ItemIsEnabled return QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable def headerData(self, section, orientation, role): if orientation == QtCore.Qt.Horizontal and role == QtCore.Qt.DisplayRole: if section == 0: return QtCore.QVariant("Node Type") elif section == 1: return QtCore.QVariant("Node Detail") return self.root_item.data(section) return QtCore.QVariant() def index(self, row, column, parent): if row < 0 or column < 0 or row >= self.rowCount(parent) or column >= self.columnCount(parent): return QtCore.QModelIndex() if not parent.isValid(): parent = self.root_item else: parent = parent.internalPointer() child = parent.child(row) if child: return self.createIndex(row, column, child) else: return QtCore.QModelIndex() def parent(self, index): if not index.isValid(): return QtCore.QModelIndex() child = index.internalPointer() parent = child.parent() if parent == self.root_item: return QtCore.QModelIndex() return self.createIndex(parent.row(), 0, parent) def rowCount(self, parent): if parent.column() > 0: return 0 if not parent.isValid(): parent = self.root_item else: parent = parent.internalPointer() return parent.childCount() def getNodeFromPath(self, tree_path): tree_path = list(tree_path) current = self.root_node while tree_path: index = tree_path[0] del tree_path[0] if current.isCompiledPythonModule(): if index == 0: current = current.getBody() else: current = tuple(current.getUsedFunctions())[index-1] else: current = current.getVisitableNodes()[index] return current def getItemFromSourceRef(self, source_ref): def check(item): if item.node.getSourceReference() == source_ref: return item for child in item._children(): result = check(child) if result is not None: return result return check(self.root_item) class InspectNodeTreeDialog(QtGui.QDialog): def __init__(self, *args): QtGui.QDialog.__init__(self, *args) ui_dir = Utils.dirname(__file__) ui_filename = Utils.joinpath(ui_dir, "dialogs", "InspectPythonTree.ui") uic.loadUi(ui_filename, self) self.treeview_nodes.setSelectionMode(self.treeview_nodes.SingleSelection) self.displayed = None self.source_code = None self.model = None self.moving = None def setModel(self, model): self.treeview_nodes.setModel(model) self.treeview_nodes.expandAll() @QtCore.pyqtSignature("on_treeview_nodes_clicked(QModelIndex)") def onTreeviewNodesClicked(self, item): tree_path = [] while item.isValid(): tree_path.insert(0, item.row()) item = item.parent() clicked_node = self.model.getNodeFromPath(tree_path) source_ref = clicked_node.getSourceReference() self.moving = True self.textedit_source.moveCursor(1, 0) for _i in range(1, source_ref.getLineNumber()): self.textedit_source.moveCursor(12, 0) self.textedit_source.setFocus() self.moving = False @QtCore.pyqtSignature("on_textedit_source_cursorPositionChanged()") def onTexteditSourceCursorMoved(self): if self.moving: return pos = self.textedit_source.textCursor().position() code = self.source_code[:pos] line = 1 for char in code: if char == '\n': line += 1 # print "Line", line item = self.model.getItemFromSourceRef( self.displayed.atLineNumber( line = line ) ) if item is not None: item_path = [] while item: item_path.insert(0, item) item = item.parent() index = QtCore.QModelIndex() parent = self.model.root_item for item in item_path[1:]: index = index.child(parent._children().index(item)+1, 1) parent = item # print self.treeview_nodes.visualRect( index ) def loadSource(self, filename): self.moving = True self.source_code = open(filename).read() self.textedit_source.setPlainText(self.source_code) self.moving = False self.displayed = SourceCodeReferences.fromFilename( filename = filename ) def displayTreeInspector(tree): app = QtGui.QApplication(sys.argv) model = NodeTreeModel(tree) dialog = InspectNodeTreeDialog() dialog.setModel(model) dialog.model = model from . import SyntaxHighlighting SyntaxHighlighting.addPythonHighlighter( document = dialog.textedit_source.document() ) dialog.loadSource(tree.getFilename()) dialog.setWindowFlags(QtCore.Qt.Window) dialog.show() import signal signal.signal(signal.SIGINT, signal.SIG_DFL) app.exec_() Nuitka-0.5.18.1/nuitka/gui/dialogs/0000755000175000017500000000000012651072347017141 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/gui/dialogs/InspectPythonTree.ui0000644000175000017500000000212012651070406023114 0ustar hayenhayen00000000000000 Dialog 0 0 1030 728 Nuitka Node Tree Inspector Node Tree Source Code Nuitka-0.5.18.1/nuitka/gui/SyntaxHighlighting.py0000644000175000017500000001616212651071040021700 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Syntax highlighting for Python. Inspired/copied from by http://diotavelli.net/PyQtWiki/Python%20syntax%20highlighting """ from PyQt4.QtCore import QRegExp # pylint: disable=F0401,I0021 from PyQt4.QtGui import ( # pylint: disable=F0401,I0021 QColor, QFont, QSyntaxHighlighter, QTextCharFormat ) def createTextFormat(color, style = ""): """Return a QTextCharFormat with the given attributes. """ _color = QColor() _color.setNamedColor(color) _format = QTextCharFormat() _format.setForeground(_color) if "bold" in style: _format.setFontWeight(QFont.Bold) if "italic" in style: _format.setFontItalic(True) return _format # Syntax styles that can be shared by all languages STYLES = { "keyword" : createTextFormat("blue"), "operator" : createTextFormat("red"), "brace" : createTextFormat("darkGray"), "defclass" : createTextFormat("black", "bold"), "string" : createTextFormat("magenta"), "string2" : createTextFormat("darkMagenta"), "comment" : createTextFormat("darkGreen", "italic"), "self" : createTextFormat("black", "italic"), "numbers" : createTextFormat("brown"), } class PythonHighlighter(QSyntaxHighlighter): """ Syntax highlighter for the Python language. """ # Python keywords keywords = [ "and", "assert", "break", "class", "continue", "def", "del", "elif", "else", "except", "exec", "finally", "for", "from", "global", "if", "import", "in", "is", "lambda", "not", "or", "pass", "print", "raise", "return", "try", "while", "with", "yield", "None", "True", "False", ] # Python operators operators = [ '=', # Comparison "==", "!=", '<', "<=", '>', ">=", # Arithmetic "\+", '-', "\*", '/', "//", "\%", "\*\*", # In-place "\+=", "-=", "\*=", "/=", "\%=", # Bitwise "\^", "\|", "\&", "\~", ">>", "<<", ] # Python braces braces = [ "\{", "\}", "\(", "\)", "\[", "\]", ] def __init__(self, document): QSyntaxHighlighter.__init__(self, document) # Multi-line strings (expression, flag, style) # The triple-quotes in these two lines will mess up the # syntax highlighting from this point onward self.tri_single = (QRegExp("'''"), 1, STYLES["string2"]) self.tri_double = (QRegExp('"""'), 2, STYLES["string2"]) rules = [] # Keyword, operator, and brace rules rules += [(r'\b%s\b' % w, 0, STYLES["keyword"]) for w in PythonHighlighter.keywords] rules += [(r'%s' % o, 0, STYLES["operator"]) for o in PythonHighlighter.operators] rules += [(r'%s' % b, 0, STYLES["brace"]) for b in PythonHighlighter.braces] # All other rules rules += [ # 'self' (r'\bself\b', 0, STYLES["self"]), # Double-quoted string, possibly containing escape sequences (r'"[^"\\]*(\\.[^"\\]*)*"', 0, STYLES["string"]), # Single-quoted string, possibly containing escape sequences (r"'[^'\\]*(\\.[^'\\]*)*'", 0, STYLES["string"]), # 'def' followed by an identifier (r'\bdef\b\s*(\w+)', 1, STYLES["defclass"]), # 'class' followed by an identifier (r'\bclass\b\s*(\w+)', 1, STYLES["defclass"]), # From '#' until a newline (r'#[^\n]*', 0, STYLES["comment"]), # Numeric literals (r'\b[+-]?[0-9]+[lL]?\b', 0, STYLES["numbers"]), (r'\b[+-]?0[xX][0-9A-Fa-f]+[lL]?\b', 0, STYLES["numbers"]), (r'\b[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?\b', 0, STYLES["numbers"]), ] # Build a QRegExp for each pattern self.rules = [(QRegExp(pat), index, fmt) for (pat, index, fmt) in rules] def highlightBlock(self, text): """Apply syntax highlighting to the given block of text. """ # Do other syntax formatting for expression, nth, display_format in self.rules: index = expression.indexIn(text, 0) while index >= 0: # We actually want the index of the nth match index = expression.pos(nth) length = expression.cap(nth).length() self.setFormat(index, length, display_format) index = expression.indexIn(text, index + length) self.setCurrentBlockState(0) # Do multi-line strings in_multiline = self.match_multiline(text, *self.tri_single) if not in_multiline: in_multiline = self.match_multiline(text, *self.tri_double) def match_multiline(self, text, delimiter, in_state, style): """Do highlighting of multi-line strings. ``delimiter`` should be a ``QRegExp`` for triple-single-quotes or triple-double-quotes, and ``in_state`` should be a unique integer to represent the corresponding state changes when inside those strings. Returns True if we're still inside a multi-line string when this function is finished. """ # If inside triple-single quotes, start at 0 if self.previousBlockState() == in_state: start = 0 add = 0 # Otherwise, look for the delimiter on this line else: start = delimiter.indexIn(text) # Move past this match add = delimiter.matchedLength() # As long as there's a delimiter match on this line... while start >= 0: # Look for the ending delimiter end = delimiter.indexIn(text, start + add) # Ending delimiter on this line? if end >= add: length = end - start + add + delimiter.matchedLength() self.setCurrentBlockState(0) # No; multi-line string else: self.setCurrentBlockState(in_state) length = text.length() - start + add # Apply formatting self.setFormat(start, length, style) # Look for the next match start = delimiter.indexIn(text, start + length) # Return True if still inside a multi-line string, False otherwise if self.currentBlockState() == in_state: return True else: return False def addPythonHighlighter(document): PythonHighlighter(document) Nuitka-0.5.18.1/nuitka/gui/__init__.py0000644000175000017500000000150112651071040017612 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/tree/0000755000175000017500000000000012651072347015672 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/tree/VariableClosure.py0000644000175000017500000003777712651071040021340 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Variable closure taking. This is the completion of variable object completion. The variables were not immediately resolved to be bound to actual scopes, but are only now. Only after this is executed, variable reference nodes can be considered complete. """ from nuitka import PythonVersions, Variables from nuitka.nodes.NodeMakingHelpers import makeConstantReplacementNode from nuitka.Options import isFullCompat from nuitka.PythonVersions import python_version from nuitka.tree import SyntaxErrors from nuitka.VariableRegistry import addVariableUsage, isSharedAmongScopes from .Operations import VisitorNoopMixin, visitTree from .ReformulationFunctionStatements import addFunctionVariableReleases # Note: We do the variable scope assignment, as an extra step from tree # building, because tree building creates the tree without any consideration of # evaluation order. And the ordered way these visitors are entered, will ensure # this order. # The main complexity is that there are two ways of visiting. One where variable # lookups are to be done immediately, and one where it is delayed. This is # basically class vs. function scope handling. class VariableClosureLookupVisitorPhase1(VisitorNoopMixin): """ Variable closure phase 1: Find assignments and early closure references. In class context, a reference to a variable must be obeyed immediately, so that "variable = variable" takes first "variable" as a closure and then adds a new local "variable" to override it from there on. For the not early closure case of a function, this will not be done and only assignments shall add local variables, and references will be ignored until phase 2. """ @staticmethod def _handleNonLocal(node): # Take closure variables for non-local declarations. for non_local_names, source_ref in node.getNonlocalDeclarations(): for non_local_name in non_local_names: variable = node.getClosureVariable( variable_name = non_local_name ) if variable.isModuleVariable(): SyntaxErrors.raiseSyntaxError( "no binding for nonlocal '%s' found" % ( non_local_name ), source_ref = None if isFullCompat() and \ python_version < 340 else source_ref, display_file = not isFullCompat() or \ python_version >= 340, display_line = not isFullCompat() or \ python_version >= 340 ) node.registerProvidedVariable(variable) addVariableUsage(variable, node) @staticmethod def _handleQualnameSetup(node): if node.qualname_setup is not None: if node.isExpressionClassBody(): class_assign, qualname_assign = node.qualname_setup class_variable = class_assign.getTargetVariableRef().getVariable() if class_variable.isModuleVariable(): qualname_node = qualname_assign.getAssignSource() qualname_node.replaceWith( makeConstantReplacementNode( constant = class_variable.getName(), node = qualname_node ) ) node.qualname_provider = node.getParentModule() else: function_variable_ref = node.qualname_setup function_variable = function_variable_ref.getVariable() if function_variable.isModuleVariable(): node.qualname_provider = node.getParentModule() # TODO: Actually for nested global classes, this approach # may not work, as their "qualname" will be wrong. In that # case a dedicated node for "qualname" references might be # needed. node.qualname_setup = None def onEnterNode(self, node): # Mighty complex code with lots of branches and statements, but it # couldn't be less without making it more difficult. # pylint: disable=R0912,R0915 if node.isExpressionTargetVariableRef(): provider = node.getParentVariableProvider() if node.getVariable() is None: variable_name = node.getVariableName() variable = provider.getVariableForAssignment( variable_name = variable_name ) node.setVariable(variable) addVariableUsage(node.getVariable(), provider) elif node.isExpressionTargetTempVariableRef(): provider = node.getParentVariableProvider() addVariableUsage(node.getVariable(), provider) elif node.isExpressionVariableRef(): if node.getVariable() is None: provider = node.getParentVariableProvider() if provider.isEarlyClosure(): variable = provider.getVariableForReference( variable_name = node.getVariableName() ) # Python3.4 version respects closure variables taken can be # overridden by writes to locals. It should be done for # globals too, on all versions, but for Python2 the locals # dictionary is avoided unless "exec" appears, so it's not # done. if variable.getOwner() is not provider: if python_version >= 340 or \ (python_version >= 300 and \ variable.isModuleVariable()): variable = Variables.MaybeLocalVariable( owner = provider, maybe_variable = variable ) node.setVariable(variable) elif node.isExpressionTempVariableRef(): if node.getVariable().getOwner() != node.getParentVariableProvider(): node.setVariable( node.getParentVariableProvider().addClosureVariable( node.getVariable() ) ) elif node.isExpressionGeneratorObjectBody(): self._handleNonLocal(node) # Python3.4 allows for class declarations to be made global, even # after they were declared, so we need to fix this up. # TODO: Then this may not even have to be here at all. if python_version >= 340: self._handleQualnameSetup(node) elif node.isExpressionCoroutineObjectBody(): self._handleNonLocal(node) # TODO: Then this may not even have to be here at all. self._handleQualnameSetup(node) elif node.isExpressionClassBody(): self._handleNonLocal(node) # Python3.4 allows for class declarations to be made global, even # after they were declared, so we need to fix this up. if python_version >= 340: self._handleQualnameSetup(node) elif node.isExpressionFunctionBody(): self._handleNonLocal(node) for variable in node.getParameters().getAllVariables(): addVariableUsage(variable, node) # Python3.4 allows for class declarations to be made global, even # after they were declared, so we need to fix this up. if python_version >= 340: self._handleQualnameSetup(node) # Attribute access of names of class functions should be mangled, if # they start with "__", but do not end in "__" as well. elif node.isExpressionAttributeLookup() or \ node.isStatementAssignmentAttribute() or \ node.isStatementDelAttribute(): attribute_name = node.getAttributeName() if attribute_name.startswith("__") and \ not attribute_name.endswith("__"): seen_function = False current = node while True: current = current.getParentVariableProvider() if current.isCompiledPythonModule(): break if current.isExpressionClassBody(): if seen_function: node.setAttributeName( "_%s%s" % ( current.getName().lstrip('_'), attribute_name ) ) break else: seen_function = True # Check if continue and break are properly in loops. If not, raise a # syntax error. elif node.isStatementLoopBreak() or node.isStatementLoopContinue(): current = node while True: if current.isParentVariableProvider(): if node.isStatementLoopContinue(): message = "'continue' not properly in loop" col_offset = 16 if python_version >= 300 else None display_line = True source_line = None else: message = "'break' outside loop" if isFullCompat(): col_offset = 2 if python_version >= 300 else None display_line = True source_line = "" if python_version >= 300 else None else: col_offset = 13 display_line = True source_line = None SyntaxErrors.raiseSyntaxError( message, source_ref = node.getSourceReference(), col_offset = col_offset, display_line = display_line, source_line = source_line ) current = current.getParent() if current.isStatementLoop(): break class VariableClosureLookupVisitorPhase2(VisitorNoopMixin): """ Variable closure phase 2: Find assignments and references. In class context, a reference to a variable must be obeyed immediately, so that "variable = variable" takes first "variable" as a closure and then adds a new local "variable" to override it from there on. So, assignments for early closure, accesses will already have a variable set now, the others, only in this phase. """ @staticmethod def _attachVariable(node, provider): # print "Late reference", node.getVariableName(), "for", provider, "caused at", node, "of", node.getParent() variable_name = node.getVariableName() was_taken = provider.hasTakenVariable(variable_name) variable = provider.getVariableForReference( variable_name = variable_name ) node.setVariable( variable ) # Need to catch functions with "exec" and closure variables not allowed. if python_version < 300 and \ not was_taken and \ provider.isExpressionFunctionBodyBase() and \ variable.getOwner() is not provider: parent_provider = provider.getParentVariableProvider() while parent_provider.isExpressionClassBody(): parent_provider = parent_provider.getParentVariableProvider() if parent_provider.isExpressionFunctionBody() and \ parent_provider.isUnqualifiedExec(): SyntaxErrors.raiseSyntaxError( reason = PythonVersions.\ getErrorMessageExecWithNestedFunction() % \ parent_provider.getName(), source_ref = parent_provider.getExecSourceRef(), col_offset = None, display_file = True, display_line = True, source_line = None ) def onEnterNode(self, node): if node.isExpressionVariableRef(): provider = node.getParentVariableProvider() if node.getVariable() is None: self._attachVariable(node, provider) addVariableUsage(node.getVariable(), provider) elif node.isExpressionTempVariableRef(): provider = node.getParentVariableProvider() addVariableUsage(node.getVariable(), provider) class VariableClosureLookupVisitorPhase3(VisitorNoopMixin): """ Variable closure phase 3: Find errors and complete frame variables. In this phase, we can do some fix-ups and find errors. We might e.g. detect that a "del" was executed on a shared variable, which is not allowed for Python 2.x, so it must be caught. The parsing wouldn't do that. Also, frame objects for functions should learn their variable names. """ def onEnterNode(self, node): if python_version < 300 and node.isStatementDelVariable(): variable = node.getTargetVariableRef().getVariable() if not variable.isModuleVariable() and \ isSharedAmongScopes(variable): SyntaxErrors.raiseSyntaxError( reason = """\ can not delete variable '%s' referenced in nested scope""" % ( variable.getName() ), source_ref = ( None if isFullCompat() else node.getSourceReference() ), display_file = not isFullCompat(), display_line = not isFullCompat() ) elif node.isStatementsFrame(): node.updateLocalNames() def completeVariableClosures(tree): visitors = ( VariableClosureLookupVisitorPhase1(), VariableClosureLookupVisitorPhase2(), VariableClosureLookupVisitorPhase3() ) for visitor in visitors: visitTree(tree, visitor) if tree.isCompiledPythonModule(): for function in tree.getFunctions(): visitTree(function, visitor) if tree.isCompiledPythonModule(): for function in tree.getFunctions(): addFunctionVariableReleases(function) # Python3 is influenced by the mere use of a variable named as # "super". So we need to prepare ability to take closure. if function.hasFlag("has_super"): if not function.hasVariableName("__class__"): class_var = function.getClosureVariable("__class__") function.registerProvidedVariable(class_var) addVariableUsage(class_var, function) if not function.hasVariableName("self"): self_var = function.getClosureVariable("self") function.registerProvidedVariable(self_var) addVariableUsage(self_var, function) Nuitka-0.5.18.1/nuitka/tree/ReformulationAssertStatements.py0000644000175000017500000000557412651071040024324 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of assert statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.BuiltinRefNodes import ExpressionBuiltinExceptionRef from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.PythonVersions import python_version from .Helpers import buildNode def buildAssertNode(provider, node, source_ref): # Build assert statements. These are re-formulated as described in the # developer manual too. They end up as conditional statement with raises of # AssertionError exceptions. # Underlying assumption: # # Assert x, y is the same as: # if not x: # raise AssertionError, y # Therefore assert statements are really just conditional statements with a # static raise contained. # exception_value = buildNode(provider, node.msg, source_ref, True) if exception_value is not None and python_version > 272: exception_value = ExpressionMakeTuple( elements = (exception_value,), source_ref = source_ref ) raise_statement = StatementRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = "AssertionError", source_ref = source_ref ), exception_value = exception_value, exception_trace = None, exception_cause = None, source_ref = source_ref ) return StatementConditional( condition = ExpressionOperationNOT( operand = buildNode(provider, node.test, source_ref), source_ref = source_ref ), yes_branch = StatementsSequence( statements = ( raise_statement, ), source_ref = source_ref ), no_branch = None, source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/tree/ReformulationComparisonExpressions.py0000644000175000017500000001566512651071040025372 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of comparison chain expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.NodeMakingHelpers import makeComparisonNode from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from .Helpers import buildNode, getKind, makeStatementsSequenceFromStatement from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildComparisonNode(provider, node, source_ref): assert len(node.comparators) == len(node.ops) # Comparisons are re-formulated as described in the developer manual. When # having multiple comparators, things require assignment expressions and # references of them to work properly. Then they can become normal "and" # code. # The operands are split out in two parts strangely. left = buildNode(provider, node.left, source_ref) rights = [ buildNode(provider, comparator, source_ref) for comparator in node.comparators ] comparators = [ getKind(comparator) for comparator in node.ops ] # Normal, and simple case, we only have one comparison, which is what our # node handles only. Then we can handle it if len(rights) == 1: return makeComparisonNode( left = left, right = rights[0], # TODO: The terminology of Nuitka might be messed up here. comparator = comparators[0], source_ref = source_ref ) return buildComplexComparisonNode(provider, left, rights, comparators, source_ref) def buildComplexComparisonNode(provider, left, rights, comparators, source_ref): # This is a bit complex, due to the many details, pylint: disable=R0914 outline_body = ExpressionOutlineBody( provider = provider, name = "comparison_chain", source_ref = source_ref ) variables = [ outline_body.allocateTempVariable( temp_scope = None, name = "operand_%d" % count ) for count in range(2, len(rights)+2) ] tmp_variable = outline_body.allocateTempVariable( temp_scope = None, name = "comparison_result" ) def makeTempAssignment(count, value): return StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = variables[count], source_ref = source_ref ), source = value, source_ref = source_ref, ) def makeReleaseStatement(count): return StatementReleaseVariable( variable = variables[count], source_ref = source_ref ) def makeValueComparisonReturn(left, right, comparator): yield StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable, source_ref = source_ref ), source = makeComparisonNode( left = left, right = right, comparator = comparator, source_ref = source_ref ), source_ref = source_ref, ) yield StatementConditional( condition = ExpressionOperationNOT( operand = ExpressionTempVariableRef( variable = tmp_variable, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_variable, source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ) statements = [] final = [] for count, value in enumerate(rights): if value is not rights[-1]: statements.append( makeTempAssignment(count, value) ) final.append( makeReleaseStatement(count) ) right = ExpressionTempVariableRef( variable = variables[count], source_ref = source_ref ) else: right = value if count != 0: left = ExpressionTempVariableRef( variable = variables[count-1], source_ref = source_ref ) comparator = comparators[count] if value is not rights[-1]: statements.extend( makeValueComparisonReturn( left, right, comparator ) ) else: statements.append( StatementReturn( expression = makeComparisonNode( left = left, right = right, comparator = comparator, source_ref = source_ref ), source_ref = source_ref ) ) final.append( StatementReleaseVariable( variable = tmp_variable, source_ref = source_ref ) ) outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outline_body, tried = statements, final = final, source_ref = source_ref ) ) ) return outline_body Nuitka-0.5.18.1/nuitka/tree/InternalModule.py0000644000175000017500000000352212651071040021155 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Internal module This is a container for helper functions that are shared across modules. It may not exist, and is treated specially in code generation. This avoids to own these functions to a random module. """ from nuitka.nodes.ModuleNodes import PythonInternalModule from nuitka.SourceCodeReferences import fromFilename from nuitka.VariableRegistry import addVariableUsage internal_module = None internal_source_ref = fromFilename("internal").atInternal() # Cache result. def once_decorator(func): func.cached_value = None def replacement(): if func.cached_value is None: func.cached_value = func() for variable in func.cached_value.getVariables(): addVariableUsage(variable, func.cached_value) return func.cached_value return replacement def getInternalModule(): # Using global here, as this is really a about the internal module as a # singleton, pylint: disable=W0603 global internal_module if internal_module is None: internal_module = PythonInternalModule() return internal_module Nuitka-0.5.18.1/nuitka/tree/ReformulationTryFinallyStatements.py0000644000175000017500000001637412651071040025160 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of try/finally statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.LoopNodes import StatementLoopBreak, StatementLoopContinue from nuitka.nodes.ReturnNodes import ExpressionReturnedValueRef, StatementReturn from nuitka.nodes.StatementNodes import ( StatementPreserveFrameException, StatementPublishException, StatementRestoreFrameException, StatementsSequence ) from nuitka.nodes.TryNodes import StatementTry from nuitka.Options import isDebug from nuitka.PythonVersions import python_version from .Helpers import ( buildStatementsNode, getStatementsAppended, getStatementsPrepended, makeReraiseExceptionStatement, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, mergeStatements, popBuildContext, pushBuildContext ) def makeTryFinallyStatement(provider, tried, final, source_ref, public_exc = False): # Complex handling, due to the many variants, pylint: disable=R0912,R0914 if type(tried) in (tuple, list): tried = makeStatementsSequenceFromStatements( *tried ) if type(final) in (tuple, list): final = StatementsSequence( statements = mergeStatements(final, False), source_ref = source_ref ) if tried is not None and not tried.isStatementsSequence(): tried = makeStatementsSequenceFromStatement(tried) if final is not None and not final.isStatementsSequence(): final = makeStatementsSequenceFromStatement(final) if tried is None: return final if final is None: return tried if provider is not None: tried.parent = provider final.parent = provider assert tried is not None, source_ref assert final is not None, source_ref if isDebug(): final2 = final.makeClone() final2.parent = provider import nuitka.TreeXML if nuitka.TreeXML.Element is not None: f1 = final.asXml() f2 = final2.asXml() def compare(a, b): for c1, c2 in zip(a, b): compare(c1, c2) assert a.attrib == b.attrib, (a.attrib, b.attrib) compare(f1, f2) if tried.mayRaiseException(BaseException): except_handler = final.makeClone() except_handler = getStatementsAppended( statement_sequence = except_handler, statements = makeReraiseExceptionStatement( source_ref = source_ref ) ) if public_exc: preserver_id = provider.allocatePreserverId() except_handler = getStatementsPrepended( statement_sequence = except_handler, statements = ( StatementPreserveFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref ) ) ) except_handler = makeTryFinallyStatement( provider = provider, tried = except_handler, final = StatementRestoreFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), public_exc = False, source_ref = source_ref, ) except_handler = makeStatementsSequenceFromStatement( statement = except_handler ) except_handler.parent = provider else: except_handler = None if tried.mayBreak(): break_handler = getStatementsAppended( statement_sequence = final.makeClone(), statements = StatementLoopBreak( source_ref = source_ref ) ) break_handler.parent = provider else: break_handler = None if tried.mayContinue(): continue_handler = getStatementsAppended( statement_sequence = final.makeClone(), statements = StatementLoopContinue( source_ref = source_ref ) ) continue_handler.parent = provider else: continue_handler = None if tried.mayReturn(): return_handler = getStatementsAppended( statement_sequence = final.makeClone(), statements = StatementReturn( expression = ExpressionReturnedValueRef( source_ref = source_ref ), source_ref = source_ref ) ) else: return_handler = None result = StatementTry( tried = tried, except_handler = except_handler, break_handler = break_handler, continue_handler = continue_handler, return_handler = return_handler, source_ref = source_ref ) if result.isStatementAborting(): return result else: return makeStatementsSequence( statements = ( result, final ), allow_none = False, source_ref = source_ref ) def buildTryFinallyNode(provider, build_tried, node, source_ref): if python_version < 300: # Prevent "continue" statements in the final blocks pushBuildContext("finally") final = buildStatementsNode( provider = provider, nodes = node.finalbody, source_ref = source_ref ) popBuildContext() return makeTryFinallyStatement( provider = provider, tried = build_tried(), final = final, source_ref = source_ref ) else: tried = build_tried() # Prevent "continue" statements in the final blocks, these have to # become "SyntaxError". pushBuildContext("finally") final = buildStatementsNode( provider = provider, nodes = node.finalbody, source_ref = source_ref ) popBuildContext() return makeTryFinallyStatement( provider = provider, tried = tried, final = final, public_exc = True, source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/tree/ReformulationClasses.py0000644000175000017500000007556612651072046022427 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of class statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTargetVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionBuiltinHasattr ) from nuitka.nodes.BuiltinRefNodes import ExpressionBuiltinRef from nuitka.nodes.CallNodes import ExpressionCall, ExpressionCallNoKeywords from nuitka.nodes.ClassNodes import ( ExpressionClassBody, ExpressionSelectMetaclass ) from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ComparisonNodes import ExpressionComparison from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ContainerOperationNodes import ( ExpressionDictOperationGet, StatementDictOperationRemove ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionQualnameRef, ExpressionFunctionRef ) from nuitka.nodes.GlobalsLocalsNodes import ( ExpressionBuiltinLocals, StatementSetLocals ) from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.SubscriptNodes import ExpressionSubscriptLookup from nuitka.nodes.TypeNodes import ExpressionBuiltinType1 from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .Helpers import ( buildNode, buildNodeList, buildStatementsNode, extractDocFromBody, getKind, makeDictCreationOrConstant, makeSequenceCreationOrConstant, makeStatementsSequence, makeStatementsSequenceFromStatement ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def _buildClassNode3(provider, node, source_ref): # Many variables, due to the huge re-formulation that is going on here, # which just has the complexity, pylint: disable=R0914 # This function is the Python3 special case with special re-formulation as # according to developer manual. class_statement_nodes, class_doc = extractDocFromBody(node) # We need a scope for the temporary variables, and they might be closured. temp_scope = provider.allocateTempScope( name = "class_creation", allow_closure = True ) tmp_bases = provider.allocateTempVariable( temp_scope = temp_scope, name = "bases" ) tmp_class_decl_dict = provider.allocateTempVariable( temp_scope = temp_scope, name = "class_decl_dict" ) tmp_metaclass = provider.allocateTempVariable( temp_scope = temp_scope, name = "metaclass" ) tmp_prepared = provider.allocateTempVariable( temp_scope = temp_scope, name = "prepared" ) class_creation_function = ExpressionClassBody( provider = provider, name = node.name, doc = class_doc, flags = set(), source_ref = source_ref ) if python_version >= 340 and False: # TODO: Temporarily reverted: tmp_class = class_creation_function.allocateTempVariable( temp_scope = None, name = "__class__" ) class_target_variable_ref = ExpressionTargetTempVariableRef( variable = tmp_class, source_ref = source_ref ) class_variable_ref = ExpressionTempVariableRef( variable = tmp_class, source_ref = source_ref ) else: class_variable = class_creation_function.getVariableForAssignment( "__class__" ) class_target_variable_ref = ExpressionTargetVariableRef( variable_name = "__class__", variable = class_variable, source_ref = source_ref ) class_variable_ref = ExpressionVariableRef( variable_name = "__class__", variable = class_variable, source_ref = source_ref ) code_object = CodeObjectSpec( code_name = node.name, code_kind = "Class", arg_names = (), kw_only_count = 0, has_starlist = False, has_stardict = False ) body = buildStatementsNode( provider = class_creation_function, nodes = class_statement_nodes, code_object = code_object, source_ref = source_ref ) source_ref_orig = source_ref if body is not None: # The frame guard has nothing to tell its line number to. body.source_ref = source_ref module_variable = class_creation_function.getVariableForAssignment( "__module__" ) statements = [ StatementSetLocals( new_locals = ExpressionTempVariableRef( variable = tmp_prepared, source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__module__", variable = module_variable, source_ref = source_ref ), source = ExpressionConstantRef( constant = provider.getParentModule().getFullName(), source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ] if class_doc is not None: doc_variable = class_creation_function.getVariableForAssignment( "__doc__" ) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__doc__", variable = doc_variable, source_ref = source_ref ), source = ExpressionConstantRef( constant = class_doc, source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ) # The "__qualname__" attribute is new in Python 3.3. if python_version >= 330: qualname = class_creation_function.getFunctionQualname() qualname_variable = class_creation_function.getVariableForAssignment( "__qualname__" ) if python_version < 340: qualname_ref = ExpressionConstantRef( constant = qualname, source_ref = source_ref, user_provided = True ) else: qualname_ref = ExpressionFunctionQualnameRef( function_body = class_creation_function, source_ref = source_ref, ) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__qualname__", variable = qualname_variable, source_ref = source_ref ), source = qualname_ref, source_ref = source_ref ) ) if python_version >= 340: qualname_assign = statements[-1] statements += [ body, StatementAssignmentVariable( variable_ref = class_target_variable_ref, source = ExpressionCall( called = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), args = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = ( ExpressionConstantRef( constant = node.name, source_ref = source_ref, user_provided = True ), ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), ExpressionBuiltinLocals( source_ref = source_ref ) ), source_ref = source_ref ), kw = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementReturn( expression = class_variable_ref, source_ref = source_ref ) ] body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) # The class body is basically a function that implicitly, at the end # returns its locals and cannot have other return statements contained. class_creation_function.setBody(body) class_creation_function.registerProvidedVariable(tmp_bases) class_creation_function.registerProvidedVariable(tmp_class_decl_dict) class_creation_function.registerProvidedVariable(tmp_metaclass) class_creation_function.registerProvidedVariable(tmp_prepared) # The class body is basically a function that implicitly, at the end # returns its created class and cannot have other return statements # contained. decorated_body = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = class_creation_function, source_ref = source_ref ), code_object = code_object, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = (), source_ref = source_ref ) for decorator in buildNodeList( provider, reversed(node.decorator_list), source_ref ): decorated_body = ExpressionCallNoKeywords( called = decorator, args = ExpressionMakeTuple( elements = ( decorated_body, ), source_ref = source_ref ), source_ref = decorator.getSourceReference() ) statements = ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_bases, source_ref = source_ref ), source = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = buildNodeList( provider, node.bases, source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source = makeDictCreationOrConstant( keys = [ ExpressionConstantRef( constant = keyword.arg, source_ref = source_ref, user_provided = True ) for keyword in node.keywords ], values = [ buildNode(provider, keyword.value, source_ref) for keyword in node.keywords ], source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), source = ExpressionSelectMetaclass( metaclass = ExpressionConditional( condition = ExpressionComparison( comparator = "In", left = ExpressionConstantRef( constant = "metaclass", source_ref = source_ref, user_provided = True ), right = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), expression_yes = ExpressionDictOperationGet( dict_arg = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), key = ExpressionConstantRef( constant = "metaclass", source_ref = source_ref, user_provided = True ), source_ref = source_ref ), expression_no = ExpressionConditional( condition = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), expression_no = ExpressionBuiltinRef( builtin_name = "type", source_ref = source_ref ), expression_yes = ExpressionBuiltinType1( value = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), subscript = ExpressionConstantRef( constant = 0, source_ref = source_ref, user_provided = True ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), bases = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref_orig ), StatementConditional( condition = ExpressionComparison( comparator = "In", left = ExpressionConstantRef( constant = "metaclass", source_ref = source_ref, user_provided = True ), right = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), no_branch = None, yes_branch = makeStatementsSequenceFromStatement( statement = StatementDictOperationRemove( dict_arg = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), key = ExpressionConstantRef( constant = "metaclass", source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_prepared, source_ref = source_ref ), source = ExpressionConditional( condition = ExpressionBuiltinHasattr( # pylint: disable=E1120,E1123 object = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), name = ExpressionConstantRef( constant = "__prepare__", source_ref = source_ref, user_provided = True ), source_ref = source_ref ), expression_no = ExpressionConstantRef( constant = {}, source_ref = source_ref, user_provided = True ), expression_yes = ExpressionCall( called = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), attribute_name = "__prepare__", source_ref = source_ref ), args = ExpressionMakeTuple( elements = ( ExpressionConstantRef( constant = node.name, source_ref = source_ref, user_provided = True ), ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ) ), source_ref = source_ref ), kw = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = node.name, source_ref = source_ref ), source = decorated_body, source_ref = source_ref ), ) if python_version >= 340: class_assign = statements[-1] class_creation_function.qualname_setup = class_assign, qualname_assign final = ( StatementReleaseVariable( variable = tmp_bases, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_class_decl_dict, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_metaclass, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_prepared, source_ref = source_ref ) ) return makeTryFinallyStatement( provider = provider, tried = statements, final = final, source_ref = source_ref ) def _buildClassNode2(provider, node, source_ref): # This function is the Python2 special case with special re-formulation as # according to developer manual, and it's very detailed, pylint: disable=R0914 class_statement_nodes, class_doc = extractDocFromBody(node) function_body = ExpressionClassBody( provider = provider, name = node.name, doc = class_doc, flags = set(), source_ref = source_ref ) code_object = CodeObjectSpec( code_name = node.name, code_kind = "Class", arg_names = (), kw_only_count = 0, has_starlist = False, has_stardict = False ) body = buildStatementsNode( provider = function_body, nodes = class_statement_nodes, code_object = code_object, source_ref = source_ref ) if body is not None: # The frame guard has nothing to tell its line number to. body.source_ref = source_ref.atInternal() # The class body is basically a function that implicitly, at the end # returns its locals and cannot have other return statements contained, and # starts out with a variables "__module__" and potentially "__doc__" set. statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__module__", source_ref = source_ref ), source = ExpressionConstantRef( constant = provider.getParentModule().getFullName(), source_ref = source_ref, user_provided = True ), source_ref = source_ref.atInternal() ) ] if class_doc is not None: statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__doc__", source_ref = source_ref ), source = ExpressionConstantRef( constant = class_doc, source_ref = source_ref, user_provided = True ), source_ref = source_ref.atInternal() ) ) statements += [ body, StatementReturn( expression = ExpressionBuiltinLocals( source_ref = source_ref ), source_ref = source_ref.atInternal() ) ] body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) # The class body is basically a function that implicitly, at the end # returns its locals and cannot have other return statements contained. function_body.setBody(body) temp_scope = provider.allocateTempScope("class_creation") tmp_bases = provider.allocateTempVariable(temp_scope, "bases") tmp_class_dict = provider.allocateTempVariable(temp_scope, "class_dict") tmp_metaclass = provider.allocateTempVariable(temp_scope, "metaclass") tmp_class = provider.allocateTempVariable(temp_scope, "class") statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_bases, source_ref = source_ref ), source = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = buildNodeList( provider, node.bases, source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ), source = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = (), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), source = ExpressionConditional( condition = ExpressionComparison( comparator = "In", left = ExpressionConstantRef( constant = "__metaclass__", source_ref = source_ref, user_provided = True ), right = ExpressionTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ), source_ref = source_ref ), expression_yes = ExpressionDictOperationGet( dict_arg = ExpressionTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ), key = ExpressionConstantRef( constant = "__metaclass__", source_ref = source_ref, user_provided = True ), source_ref = source_ref ), expression_no = ExpressionSelectMetaclass( metaclass = None, bases = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_class, source_ref = source_ref ), source = ExpressionCallNoKeywords( called = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), args = ExpressionMakeTuple( elements = ( ExpressionConstantRef( constant = node.name, source_ref = source_ref, user_provided = True ), ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), ExpressionTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ) ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), ] for decorator in buildNodeList( provider, reversed(node.decorator_list), source_ref ): statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_class, source_ref = source_ref ), source = ExpressionCallNoKeywords( called = decorator, args = ExpressionMakeTuple( elements = ( ExpressionTempVariableRef( variable = tmp_class, source_ref = source_ref ), ), source_ref = source_ref ), source_ref = decorator.getSourceReference() ), source_ref = decorator.getSourceReference() ) ) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = node.name, source_ref = source_ref ), source = ExpressionTempVariableRef( variable = tmp_class, source_ref = source_ref ), source_ref = source_ref ) ) final = ( StatementReleaseVariable( variable = tmp_class, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_bases, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_class_dict, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_metaclass, source_ref = source_ref ) ) return makeTryFinallyStatement( provider = function_body, tried = statements, final = final, source_ref = source_ref ) def buildClassNode(provider, node, source_ref): assert getKind(node) == "ClassDef" # There appears to be a inconsistency with the top level line number # not being the one really the class has, if there are bases, and a # decorator. if node.bases: source_ref = source_ref.atLineNumber(node.bases[-1].lineno) # Python2 and Python3 are similar, but fundamentally different, so handle # them in dedicated code. if python_version < 300: return _buildClassNode2(provider, node, source_ref) else: return _buildClassNode3(provider, node, source_ref) Nuitka-0.5.18.1/nuitka/tree/ReformulationFunctionStatements.py0000644000175000017500000006276212651071040024652 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of function statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTargetVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionSpecialUnpack, StatementSpecialUnpackCheck ) from nuitka.nodes.BuiltinRefNodes import ExpressionBuiltinRef from nuitka.nodes.CallNodes import ExpressionCallNoKeywords from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.CoroutineNodes import ( ExpressionCoroutineObjectBody, ExpressionMakeCoroutineObject ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.GeneratorNodes import ( ExpressionGeneratorObjectBody, ExpressionMakeGeneratorObject, StatementGeneratorReturn ) from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from nuitka.tree import SyntaxErrors from .Helpers import ( buildNode, buildNodeList, buildStatementsNode, detectFunctionBodyKind, extractDocFromBody, getKind, makeDictCreationOrConstant, makeStatementsSequenceFromStatement, mangleName ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def _insertFinalReturnStatement(function_statements_body, return_class, source_ref): return_statement = return_class( expression = ExpressionConstantRef( constant = None, user_provided = True, source_ref = source_ref ), source_ref = source_ref ) if function_statements_body is None: function_statements_body = makeStatementsSequenceFromStatement( statement = return_statement, ) elif not function_statements_body.isStatementAborting(): function_statements_body.setStatements( function_statements_body.getStatements() + ( return_statement, ) ) return function_statements_body def buildFunctionNode(provider, node, source_ref): # Functions have way too many details, pylint: disable=R0914 assert getKind(node) == "FunctionDef" function_statement_nodes, function_doc = extractDocFromBody(node) function_kind, flags, _written_variables, _non_local_declarations, _global_declarations = \ detectFunctionBodyKind( nodes = function_statement_nodes ) outer_body, function_body, code_object = buildFunctionWithParsing( provider = provider, function_kind = function_kind, name = node.name, function_doc = function_doc, flags = flags, node = node, source_ref = source_ref ) if function_kind == "Function": code_body = function_body elif function_kind == "Generator": code_body = ExpressionGeneratorObjectBody( provider = function_body, name = node.name, flags = flags, source_ref = source_ref ) for variable in function_body.getVariables(): code_body.getVariableForReference(variable.getName()) else: assert False, function_kind if function_kind == "Generator": function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionMakeGeneratorObject( generator_ref = ExpressionFunctionRef( function_body = code_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ) ) ) decorators = buildNodeList( provider = provider, nodes = reversed(node.decorator_list), source_ref = source_ref ) defaults = buildNodeList( provider = provider, nodes = node.args.defaults, source_ref = source_ref ) kw_defaults = buildParameterKwDefaults( provider = provider, node = node, function_body = function_body, source_ref = source_ref ) function_statements_body = buildStatementsNode( provider = code_body, nodes = function_statement_nodes, code_object = code_object, source_ref = source_ref ) if function_kind == "Function": # TODO: Generators might have to raise GeneratorExit instead. function_statements_body = _insertFinalReturnStatement( function_statements_body = function_statements_body, return_class = StatementReturn, source_ref = source_ref ) if function_statements_body.isStatementsFrame(): function_statements_body = makeStatementsSequenceFromStatement( statement = function_statements_body ) code_body.setBody( function_statements_body ) annotations = buildParameterAnnotations(provider, node, source_ref) function_creation = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = outer_body, source_ref = source_ref ), code_object = code_object, defaults = defaults, kw_defaults = kw_defaults, annotations = annotations, source_ref = source_ref ) # Add the "staticmethod" decorator to __new__ methods if not provided. # CPython made these optional, but secretly applies them when it does # "class __new__". We add them earlier, so our optimization will see it. if node.name == "__new__" and \ not decorators and \ provider.isExpressionClassBody(): decorators = ( ExpressionBuiltinRef( builtin_name = "staticmethod", source_ref = source_ref ), ) decorated_function = function_creation for decorator in decorators: decorated_function = ExpressionCallNoKeywords( called = decorator, args = ExpressionMakeTuple( elements = (decorated_function,), source_ref = source_ref ), source_ref = decorator.getSourceReference() ) result = StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = mangleName(node.name, provider), source_ref = source_ref ), source = decorated_function, source_ref = source_ref ) if python_version >= 340: function_body.qualname_setup = result.getTargetVariableRef() return result def buildAsyncFunctionNode(provider, node, source_ref): # We are creating a function here that creates coroutine objects, with # many details each, pylint: disable=R0914 assert getKind(node) == "AsyncFunctionDef" function_statement_nodes, function_doc = extractDocFromBody(node) creator_function_body, _, code_object = buildFunctionWithParsing( provider = provider, function_kind = "Coroutine", name = node.name, flags = set(), function_doc = function_doc, node = node, source_ref = source_ref ) function_body = ExpressionCoroutineObjectBody( provider = creator_function_body, name = node.name, flags = set(), source_ref = source_ref ) decorators = buildNodeList( provider = provider, nodes = reversed(node.decorator_list), source_ref = source_ref ) defaults = buildNodeList( provider = provider, nodes = node.args.defaults, source_ref = source_ref ) function_statements_body = buildStatementsNode( provider = function_body, nodes = function_statement_nodes, code_object = code_object, source_ref = source_ref ) function_statements_body = _insertFinalReturnStatement( function_statements_body = function_statements_body, return_class = StatementGeneratorReturn, source_ref = source_ref ) if function_statements_body.isStatementsFrame(): function_statements_body = makeStatementsSequenceFromStatement( statement = function_statements_body ) function_body.setBody( function_statements_body ) annotations = buildParameterAnnotations(provider, node, source_ref) kw_defaults = buildParameterKwDefaults( provider = provider, node = node, function_body = creator_function_body, source_ref = source_ref ) creator_function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionMakeCoroutineObject( coroutine_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ) ) ) function_creation = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = creator_function_body, source_ref = source_ref ), code_object = code_object, defaults = defaults, kw_defaults = kw_defaults, annotations = annotations, source_ref = source_ref ) decorated_function = function_creation for decorator in decorators: decorated_function = ExpressionCallNoKeywords( called = decorator, args = ExpressionMakeTuple( elements = (decorated_function,), source_ref = source_ref ), source_ref = decorator.getSourceReference() ) result = StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = mangleName(node.name, provider), source_ref = source_ref ), source = decorated_function, source_ref = source_ref ) function_body.qualname_setup = result.getTargetVariableRef() return result def buildParameterKwDefaults(provider, node, function_body, source_ref): # Build keyword only arguments default values. We are hiding here, that it # is a Python3 only feature. if python_version >= 300: kw_only_names = function_body.getParameters().getKwOnlyParameterNames() if kw_only_names: keys = [] values = [] for kw_only_name, kw_default in \ zip(kw_only_names, node.args.kw_defaults): if kw_default is not None: keys.append( ExpressionConstantRef( constant = kw_only_name, source_ref = source_ref ) ) values.append( buildNode(provider, kw_default, source_ref) ) kw_defaults = makeDictCreationOrConstant( keys = keys, values = values, source_ref = source_ref ) else: kw_defaults = None else: kw_defaults = None return kw_defaults def buildParameterAnnotations(provider, node, source_ref): # Too many branches, because there is too many cases, pylint: disable=R0912 # Build annotations. We are hiding here, that it is a Python3 only feature. if python_version < 300: return None # Starting with Python 3.4, the names of parameters are mangled in # annotations as well. if python_version < 340: mangle = lambda variable_name: variable_name else: mangle = lambda variable_name: mangleName(variable_name, provider) keys = [] values = [] def addAnnotation(key, value): keys.append( ExpressionConstantRef( constant = mangle(key), source_ref = source_ref, user_provided = True ) ) values.append(value) def extractArg(arg): if getKind(arg) == "Name": assert arg.annotation is None elif getKind(arg) == "arg": if arg.annotation is not None: addAnnotation( key = arg.arg, value = buildNode(provider, arg.annotation, source_ref) ) elif getKind(arg) == "Tuple": for arg in arg.elts: extractArg(arg) else: assert False, getKind(arg) for arg in node.args.args: extractArg(arg) for arg in node.args.kwonlyargs: extractArg(arg) if python_version < 340: if node.args.varargannotation is not None: addAnnotation( key = node.args.vararg, value = buildNode( provider, node.args.varargannotation, source_ref ) ) if node.args.kwargannotation is not None: addAnnotation( key = node.args.kwarg, value = buildNode( provider, node.args.kwargannotation, source_ref ) ) else: if node.args.vararg is not None: extractArg(node.args.vararg) if node.args.kwarg is not None: extractArg(node.args.kwarg) # Return value annotation (not there for lambdas) if hasattr(node, "returns") and node.returns is not None: addAnnotation( key = "return", value = buildNode( provider, node.returns, source_ref ) ) if keys: return makeDictCreationOrConstant( keys = keys, values = values, source_ref = source_ref ) else: return None def buildFunctionWithParsing(provider, function_kind, name, function_doc, flags, node, source_ref): # This contains a complex re-formulation for nested parameter functions. # pylint: disable=R0914 kind = getKind(node) assert kind in ("FunctionDef", "Lambda", "AsyncFunctionDef"), "unsupported for kind " + kind def extractArg(arg): if arg is None: return None elif type(arg) is str: return mangleName(arg, provider) elif getKind(arg) == "Name": return mangleName(arg.id, provider) elif getKind(arg) == "arg": return mangleName(arg.arg, provider) elif getKind(arg) == "Tuple": # These are to be re-formulated on the outside. assert False else: assert False, getKind(arg) special_args = {} def extractNormalArgs(args): normal_args = [] for arg in args: if type(arg) is not str and getKind(arg) == "Tuple": special_arg_name = ".%d" % (len(special_args) + 1) special_args[special_arg_name] = arg.elts normal_args.append(special_arg_name) else: normal_args.append(extractArg(arg)) return normal_args normal_args = extractNormalArgs(node.args.args) parameters = ParameterSpec( name = name, normal_args = normal_args, kw_only_args = [ extractArg(arg) for arg in node.args.kwonlyargs ] if python_version >= 300 else [], list_star_arg = extractArg(node.args.vararg), dict_star_arg = extractArg(node.args.kwarg), default_count = len(node.args.defaults) ) message = parameters.checkValid() if message is not None: SyntaxErrors.raiseSyntaxError( message, source_ref ) code_object = CodeObjectSpec( code_name = name, code_kind = function_kind, arg_names = parameters.getParameterNames(), kw_only_count = parameters.getKwOnlyParameterCount(), has_starlist = parameters.getStarListArgumentName() is not None, has_stardict = parameters.getStarDictArgumentName() is not None ) outer_body = ExpressionFunctionBody( provider = provider, name = name, flags = flags, doc = function_doc, parameters = parameters, source_ref = source_ref ) if special_args: inner_name = name.strip("<>") + "$inner" inner_arg_names = [] iter_vars = [] values = [] statements = [] def unpackFrom(source, arg_names): accesses = [] sub_special_index = 0 iter_var = outer_body.allocateTempVariable(None, "arg_iter_%d" % len(iter_vars)) iter_vars.append(iter_var) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = iter_var, source_ref = source_ref ), source = ExpressionBuiltinIter1( value = source, source_ref = source_ref ), source_ref = source_ref ) ) for element_index, arg_name in enumerate(arg_names): if getKind(arg_name) == "Name": inner_arg_names.append(arg_name.id) arg_var = outer_body.allocateTempVariable(None, "tmp_" + arg_name.id) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = arg_var, source_ref = source_ref ), source = ExpressionSpecialUnpack( value = ExpressionTempVariableRef( variable = iter_var, source_ref = source_ref ), count = element_index + 1, expected = len(arg_names), source_ref = source_ref ), source_ref = source_ref ) ) accesses.append( ExpressionTempVariableRef( variable = arg_var, source_ref = source_ref ) ) elif getKind(arg_name) == "Tuple": accesses.extend( unpackFrom( source = ExpressionSpecialUnpack( value = ExpressionTempVariableRef( variable = iter_var, source_ref = source_ref ), count = element_index + 1, expected = len(arg_names), source_ref = source_ref ), arg_names = arg_name.elts ) ) sub_special_index += 1 else: assert False, arg_name statements.append( StatementSpecialUnpackCheck( iterator = ExpressionTempVariableRef( variable = iter_var, source_ref = source_ref ), count = len(arg_names), source_ref = source_ref ) ) return accesses for arg_name in parameters.getParameterNames(): if arg_name.startswith('.'): source = ExpressionVariableRef( variable_name = arg_name, source_ref = source_ref ) values.extend( unpackFrom(source, special_args[arg_name]) ) else: values.append( ExpressionVariableRef( variable_name = arg_name, source_ref = source_ref ) ) inner_arg_names.append(arg_name) inner_parameters = ParameterSpec( name = inner_name, normal_args = inner_arg_names, kw_only_args = (), list_star_arg = None, dict_star_arg = None, default_count = None ) function_body = ExpressionFunctionBody( provider = outer_body, name = inner_name, flags = flags, doc = function_doc, parameters = inner_parameters, source_ref = source_ref ) statements.append( StatementReturn( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = code_object, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = values, source_ref = source_ref ), source_ref = source_ref ) ) outer_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider, tried = statements, final = [ StatementReleaseVariable( variable = variable, source_ref = source_ref ) for variable in outer_body.getTempVariables() ] , source_ref = source_ref, public_exc = False ) ) ) else: function_body = outer_body return outer_body, function_body, code_object def addFunctionVariableReleases(function): assert function.isExpressionFunctionBody() or \ function.isExpressionClassBody() or \ function.isExpressionGeneratorObjectBody() or \ function.isExpressionCoroutineObjectBody() releases = [] # We attach everything to the function definition source location. source_ref = function.getSourceReference() for variable in function.getLocalVariables(): # Shared variables are freed by function object attachment. if variable.getOwner() is not function: continue releases.append( StatementReleaseVariable( variable = variable, source_ref = source_ref ) ) if releases: body = function.getBody() if body.isStatementsFrame(): body = makeStatementsSequenceFromStatement( statement = body ) body = makeTryFinallyStatement( provider = function, tried = body, final = releases, source_ref = source_ref ) function.setBody( makeStatementsSequenceFromStatement( statement = body ) ) # assert body.isStatementAborting(), body.asXmlText() Nuitka-0.5.18.1/nuitka/tree/ReformulationNamespacePackages.py0000644000175000017500000001263512651071040024342 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Namespace packages of Python3.3 """ from nuitka import Options from nuitka.nodes.AssignNodes import ( ExpressionTargetVariableRef, StatementAssignmentVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.CallNodes import ExpressionCallNoKeywords from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ( ExpressionMakeList, ExpressionMakeTuple ) from nuitka.nodes.FutureSpecs import FutureSpec from nuitka.nodes.ImportNodes import ( ExpressionImportModule, ExpressionImportModuleHard, ExpressionImportName ) from nuitka.nodes.ModuleNodes import ( CompiledPythonPackage, ExpressionModuleFileAttributeRef ) from nuitka.PythonVersions import python_version from nuitka.SourceCodeReferences import SourceCodeReference from nuitka.utils.Utils import dirname from .Helpers import makeStatementsSequenceFromStatement from .VariableClosure import completeVariableClosures def createPathAssignment(source_ref): if Options.getFileReferenceMode() == "original": path_value = ExpressionConstantRef( constant = [ dirname(source_ref.getFilename()) ], source_ref = source_ref, user_provided = True ) else: path_value = ExpressionMakeList( elements = ( ExpressionCallNoKeywords( called = ExpressionAttributeLookup( source = ExpressionImportModuleHard( module_name = "os", import_name = "path", source_ref = source_ref ), attribute_name = "dirname", source_ref = source_ref ), args = ExpressionMakeTuple( elements = ( ExpressionModuleFileAttributeRef( source_ref = source_ref, ), ), source_ref = source_ref, ), source_ref = source_ref, ), ), source_ref = source_ref ) return StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__path__", source_ref = source_ref ), source = path_value, source_ref = source_ref ) def createPython3NamespacePath(package_name, module_relpath, source_ref): return StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__path__", source_ref = source_ref ), source = ExpressionCallNoKeywords( called = ExpressionImportName( module = ExpressionImportModule( module_name = "_frozen_importlib" if python_version < 350 else "_frozen_importlib_external", import_list = (), level = 0, source_ref = source_ref ), import_name = "_NamespacePath", source_ref = source_ref ), args = ExpressionConstantRef( constant = ( package_name, [module_relpath], None ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) def createNamespacePackage(package_name, module_relpath): parts = package_name.split('.') source_ref = SourceCodeReference.fromFilenameAndLine( filename = module_relpath, line = 1, future_spec = FutureSpec(), ) source_ref = source_ref.atInternal() package_package_name = '.'.join(parts[:-1]) or None package = CompiledPythonPackage( name = parts[-1], package_name = package_package_name, source_ref = source_ref, ) if python_version >= 300: statement = createPython3NamespacePath( package_name = package_name, module_relpath = module_relpath, source_ref = source_ref ) else: statement = createPathAssignment(source_ref) package.setBody( makeStatementsSequenceFromStatement( statement = statement ) ) completeVariableClosures(package) return source_ref, package Nuitka-0.5.18.1/nuitka/tree/Operations.py0000644000175000017500000000324612651071040020361 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Operations on the tree. This is mostly for the different kinds of visits that the node tree can have. You can visit a scope, a tree (module), or every scope of a tree (module). """ def visitTree(tree, visitor): visitor.onEnterNode(tree) for visitable in tree.getVisitableNodes(): if visitable is None: raise AssertionError( "'None' child encountered", tree, tree.source_ref ) visitTree(visitable, visitor) visitor.onLeaveNode(tree) def visitFunction(function, visitor): visitor.onEnterNode(function) visitor.onLeaveNode(function) class VisitorNoopMixin: def onEnterNode(self, node): """ Overloaded for operation before the node children were done. """ pass def onLeaveNode(self, node): """ Overloaded for operation after the node children were done. """ pass Nuitka-0.5.18.1/nuitka/tree/SyntaxErrors.py0000644000175000017500000000676012651071040020725 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Handling of syntax errors. Format SyntaxError/IndentationError exception for output, as well as raise it for the given source code reference. """ def formatOutput(e): if len(e.args) > 1: reason, (filename, lineno, colno, message) = e.args if message is None and colno is not None: colno = None else: reason, = e.args filename = None lineno = None colno = None message = None # On CPython3.4 at least, this attribute appears to override reason for # SyntaxErrors at least. if hasattr(e, "msg"): reason = e.msg if colno is not None: colno = colno - len(message) + len(message.lstrip()) return """\ File "%s", line %d %s %s^ %s: %s""" % ( filename, lineno, message.strip(), ' ' * (colno-1) if colno is not None else "", e.__class__.__name__, reason ) elif message is not None: return """\ File "%s", line %d %s %s: %s""" % ( filename, lineno, message.strip(), e.__class__.__name__, reason ) elif filename is not None: return """\ File "%s", line %s %s: %s""" % ( filename, lineno, e.__class__.__name__, reason ) else: return """\ %s: %s""" % ( e.__class__.__name__, reason ) def raiseSyntaxError(reason, source_ref, col_offset = None, display_file = True, display_line = True, source_line = None): def readSource(): import linecache return linecache.getline( filename = source_ref.getFilename(), lineno = source_ref.getLineNumber() ) if display_file and display_line: if source_line is None: source_line = readSource() raise SyntaxError( reason, ( source_ref.getFilename(), source_ref.getLineNumber(), col_offset, source_line ) ) else: if source_ref is not None: if source_line is None and display_line: source_line = readSource() raise SyntaxError( reason, ( source_ref.getFilename(), source_ref.getLineNumber(), None, source_line ) ) else: raise SyntaxError( reason, ( None, None, None, None ) ) Nuitka-0.5.18.1/nuitka/tree/SourceReading.py0000644000175000017500000001054712651071040020772 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Read source code from files. This is tremendously more complex than one might think, due to encoding issues and version differences of Python versions. """ import re from nuitka import Options, PythonVersions, SourceCodeReferences from nuitka.plugins.PluginBase import Plugins from nuitka.PythonVersions import python_version from nuitka.tree import SyntaxErrors def _readSourceCodeFromFilename3(source_filename): import tokenize try: with tokenize.open(source_filename) as source_file: # @UndefinedVariable return source_file.read() except SyntaxError as e: if Options.isFullCompat(): if PythonVersions.doShowUnknownEncodingName(): match = re.match("unknown encoding for '.*?': (.*)", e.args[0]) complaint = match.group(1) else: complaint = "with BOM" e.args = ( "encoding problem: %s" % complaint, (source_filename, 1, None, None) ) if hasattr(e, "msg"): e.msg = e.args[0] raise def _detectEncoding2(source_file): # Detect the encoding. encoding = "ascii" line1 = source_file.readline() if line1.startswith(b'\xef\xbb\xbf'): encoding = "utf-8" else: line1_match = re.search(b"coding[:=]\\s*([-\\w.]+)", line1) if line1_match: encoding = line1_match.group(1) else: line2 = source_file.readline() line2_match = re.search(b"coding[:=]\\s*([-\\w.]+)", line2) if line2_match: encoding = line2_match.group(1) source_file.seek(0) return encoding def _readSourceCodeFromFilename2(source_filename): # Detect the encoding. with open(source_filename, "rU") as source_file: encoding = _detectEncoding2(source_file) source_code = source_file.read() # Try and detect SyntaxError from missing or wrong encodings. if type(source_code) is not unicode and encoding == "ascii": try: _source_code = source_code.decode(encoding) except UnicodeDecodeError as e: lines = source_code.split('\n') so_far = 0 for count, line in enumerate(lines): so_far += len(line) + 1 if so_far > e.args[2]: break else: # Cannot happen, decode error implies non-empty. count = -1 wrong_byte = re.search( "byte 0x([a-f0-9]{2}) in position", str(e) ).group(1) SyntaxErrors.raiseSyntaxError( reason = """\ Non-ASCII character '\\x%s' in file %s on line %d, but no encoding declared; \ see http://python.org/dev/peps/pep-0263/ for details""" % ( wrong_byte, source_filename, count+1, ), source_ref = SourceCodeReferences.fromFilename( source_filename ).atLineNumber(count+1), display_line = False ) return source_code def readSourceCodeFromFilename(module_name, source_filename): if python_version < 300: source_code = _readSourceCodeFromFilename2(source_filename) else: source_code = _readSourceCodeFromFilename3(source_filename) # Allow plug-ins to mess with source code. source_code = Plugins.onModuleSourceCode(module_name, source_code) return source_code Nuitka-0.5.18.1/nuitka/tree/ReformulationDictionaryCreation.py0000644000175000017500000003073712651072046024613 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of dictionary creations. Dictionary creations might be directly translated to constants, or they might become nodes that build dictionaries. For Python3.5, unpacking can happen while creating dictionaries, these are being re-formulated to an internal function. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionBuiltinNext1 ) from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ContainerOperationNodes import ExpressionDictOperationUpdate from nuitka.nodes.DictionaryNodes import ( ExpressionKeyValuePair, ExpressionMakeDict ) from nuitka.nodes.ExceptionNodes import ( ExpressionBuiltinMakeException, StatementRaiseException ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.OperatorNodes import ExpressionOperationBinary from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.TypeNodes import ExpressionBuiltinType1 from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .Helpers import ( buildNode, buildNodeList, makeDictCreationOrConstant, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) from .InternalModule import ( getInternalModule, internal_source_ref, once_decorator ) from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildDictionaryNode(provider, node, source_ref): if python_version >= 350: for key in node.keys: if key is None: return buildDictionaryUnpacking( provider = provider, node = node, source_ref = source_ref ) return makeDictCreationOrConstant( keys = buildNodeList(provider, node.keys, source_ref), values = buildNodeList(provider, node.values, source_ref), source_ref = source_ref ) @once_decorator def getDictUnpackingHelper(): helper_name = "_unpack_dict" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = (), list_star_arg = "args", dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "keys") loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), makeTryExceptSingleHandlerNode( tried = StatementExpressionOnly( expression = ExpressionDictOperationUpdate( dict_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "AttributeError", handler_body = StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( ExpressionOperationBinary( operator = "Mod", left = ExpressionConstantRef( constant = """\ '%s' object is not a mapping""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), attribute_name = "__name__", source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) args_variable = result.getVariableForAssignment( variable_name = "args" ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = {}, source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementLoop( body = loop_body, source_ref = internal_source_ref ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result def buildDictionaryUnpackingArgs(provider, keys, values, source_ref): result = [] for key, value in zip(keys, values): # TODO: We could be a lot cleverer about the dictionaries for non-starred # arguments, but lets get this to work first. if key is None: result.append( buildNode(provider, value, source_ref), ) elif type(key) is str: result.append( ExpressionMakeDict( pairs = ( ExpressionKeyValuePair( key = ExpressionConstantRef( constant = key, source_ref = source_ref ), value = buildNode(provider, value, source_ref), source_ref = source_ref ), ), source_ref = source_ref ) ) else: result.append( ExpressionMakeDict( pairs = ( ExpressionKeyValuePair( key = buildNode(provider, key, source_ref), value = buildNode(provider, value, source_ref), source_ref = source_ref ), ), source_ref = source_ref ) ) return result def buildDictionaryUnpacking(provider, node, source_ref): helper_args = buildDictionaryUnpackingArgs(provider, node.keys, node.values, source_ref) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getDictUnpackingHelper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionMakeTuple( helper_args, source_ref ), ), source_ref = source_ref, ) result.setCompatibleSourceReference(helper_args[-1].getCompatibleSourceReference()) return result Nuitka-0.5.18.1/nuitka/tree/ReformulationContractionExpressions.py0000644000175000017500000004531412651072046025544 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of contraction expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionBuiltinNext1 ) from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerOperationNodes import ( ExpressionDictOperationSet, ExpressionListOperationAppend, ExpressionSetOperationAdd ) from nuitka.nodes.FrameNodes import StatementsFrame from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.GeneratorNodes import ( ExpressionGeneratorObjectBody, ExpressionMakeGeneratorObject ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.NodeMakingHelpers import ( makeVariableRefNode, makeVariableTargetRefNode ) from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import ( StatementExpressionOnly, StatementGeneratorEntry, StatementsSequence ) from nuitka.nodes.YieldNodes import ExpressionYield from nuitka.PythonVersions import python_version from .Helpers import ( buildNode, buildNodeList, getKind, makeStatementsSequenceFromStatement, mergeStatements ) from .ReformulationAssignmentStatements import buildAssignmentStatements from .ReformulationBooleanExpressions import buildAndNode from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildListContractionNode(provider, node, source_ref): # List contractions are dealt with by general code. return _buildContractionNode( provider = provider, node = node, name = "list_contraction" if python_version < 300 else "", emit_class = ExpressionListOperationAppend, start_value = ExpressionConstantRef( constant = [], source_ref = source_ref ), # Note: For Python3, the list contractions no longer assign to the outer # scope. assign_provider = python_version < 300, source_ref = source_ref ) def buildSetContractionNode(provider, node, source_ref): # Set contractions are dealt with by general code. return _buildContractionNode( provider = provider, node = node, name = "", emit_class = ExpressionSetOperationAdd, start_value = ExpressionConstantRef( constant = set(), source_ref = source_ref ), assign_provider = False, source_ref = source_ref ) def buildDictContractionNode(provider, node, source_ref): # Dict contractions are dealt with by general code. return _buildContractionNode( provider = provider, node = node, name = "", emit_class = ExpressionDictOperationSet, start_value = ExpressionConstantRef( constant = {}, source_ref = source_ref ), assign_provider = False, source_ref = source_ref ) def buildGeneratorExpressionNode(provider, node, source_ref): # Generator expressions are dealt with by general code. assert getKind(node) == "GeneratorExp" return _buildContractionNode( provider = provider, node = node, name = "", emit_class = ExpressionYield, start_value = None, assign_provider = False, source_ref = source_ref ) def _buildContractionBodyNode(provider, node, emit_class, start_value, container_tmp, iter_tmp, temp_scope, assign_provider, function_body, source_ref): # This uses lots of variables and branches. There is no good way # around that, and we deal with many cases, due to having generator # expressions sharing this code, pylint: disable=R0912,R0914 tmp_variables = [] if assign_provider: tmp_variables.append(iter_tmp) if container_tmp is not None: tmp_variables.append(container_tmp) # First assign the iterator if we are an outline. if assign_provider: statements = [ StatementAssignmentVariable( variable_ref = makeVariableTargetRefNode( variable = iter_tmp, source_ref = source_ref ), source = ExpressionBuiltinIter1( value = buildNode( provider = provider, node = node.generators[0].iter, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref.atInternal() ) ] else: statements = [] if start_value is not None: statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = container_tmp, source_ref = source_ref ), source = start_value, source_ref = source_ref.atInternal() ) ) if hasattr(node, "elt"): if start_value is not None: current_body = emit_class( ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), buildNode( provider = function_body, node = node.elt, source_ref = source_ref ), source_ref = source_ref ) else: assert emit_class is ExpressionYield current_body = emit_class( buildNode( provider = function_body, node = node.elt, source_ref = source_ref ), source_ref = source_ref ) else: assert emit_class is ExpressionDictOperationSet current_body = ExpressionDictOperationSet( dict_arg = ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), key = buildNode( provider = function_body, node = node.key, source_ref = source_ref, ), value = buildNode( provider = function_body, node = node.value, source_ref = source_ref, ), source_ref = source_ref ) current_body = StatementExpressionOnly( expression = current_body, source_ref = source_ref ) for count, qual in enumerate(reversed(node.generators)): tmp_value_variable = function_body.allocateTempVariable( temp_scope = temp_scope, name = "iter_value_%d" % count ) tmp_variables.append(tmp_value_variable) # The first iterated value is to be calculated outside of the function # and will be given as a parameter "_iterated", the others are built # inside the function. if qual is node.generators[0]: iterator_ref = makeVariableRefNode( variable = iter_tmp, source_ref = source_ref ) tmp_iter_variable = None nested_statements = [] else: # First create the iterator and store it, next should be loop body value_iterator = ExpressionBuiltinIter1( value = buildNode( provider = function_body, node = qual.iter, source_ref = source_ref ), source_ref = source_ref ) tmp_iter_variable = function_body.allocateTempVariable( temp_scope = temp_scope, name = "contraction_iter_%d" % count ) tmp_variables.append(tmp_iter_variable) nested_statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ), source = value_iterator, source_ref = source_ref ) ] iterator_ref = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ) loop_statements = [ makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_value_variable, source_ref = source_ref ), source = ExpressionBuiltinNext1( value = iterator_ref, source_ref = source_ref ), source_ref = source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = source_ref.atInternal() ), source_ref = source_ref ), buildAssignmentStatements( provider = provider if assign_provider else function_body, temp_provider = function_body, node = qual.target, source = ExpressionTempVariableRef( variable = tmp_value_variable, source_ref = source_ref ), source_ref = source_ref ) ] conditions = buildNodeList( provider = function_body, nodes = qual.ifs, source_ref = source_ref ) if len(conditions) >= 1: loop_statements.append( StatementConditional( condition = buildAndNode( values = conditions, source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = current_body ), no_branch = None, source_ref = source_ref ) ) else: loop_statements.append(current_body) nested_statements.append( StatementLoop( body = StatementsSequence( statements = mergeStatements(loop_statements), source_ref = source_ref ), source_ref = source_ref ) ) if tmp_iter_variable is not None: nested_statements.append( StatementReleaseVariable( variable = tmp_iter_variable, source_ref = source_ref ) ) current_body = StatementsSequence( statements = mergeStatements(nested_statements, False), source_ref = source_ref ) statements.append(current_body) statements = mergeStatements(statements) if emit_class is ExpressionYield: statements.insert( 0, StatementGeneratorEntry( source_ref = source_ref ) ) release_statements = [ StatementReleaseVariable( variable = tmp_variable, source_ref = source_ref ) for tmp_variable in tmp_variables ] return statements, release_statements def _buildContractionNode(provider, node, name, emit_class, start_value, assign_provider, source_ref): # The contraction nodes are reformulated to function bodies, with loops as # described in the developer manual. They use a lot of temporary names, # nested blocks, etc. and so a lot of variable names. # Note: The assign_provider is only to cover Python2 list contractions, # assigning one of the loop variables to the outside scope. if assign_provider: function_body = ExpressionOutlineBody( provider = provider, name = name, source_ref = source_ref ) iter_tmp = function_body.allocateTempVariable( temp_scope = None, name = ".0" ) else: # TODO: No function ought to be necessary. function_body = ExpressionFunctionBody( provider = provider, name = name, doc = None, parameters = ParameterSpec( name = name, normal_args = (".0",), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = source_ref ) iter_tmp = function_body.getVariableForAssignment( variable_name = ".0" ) assert iter_tmp.isParameterVariable() code_object = CodeObjectSpec( code_name = name, code_kind = "Generator" if emit_class is ExpressionYield else "Function", arg_names = () if assign_provider else function_body.getParameters().getParameterNames(), kw_only_count = 0, has_starlist = False, has_stardict = False, ) if emit_class is ExpressionYield: code_body = ExpressionGeneratorObjectBody( provider = function_body, name = "", flags = set(), source_ref = source_ref ) iter_tmp = code_body.getVariableForReference( variable_name = ".0" ) assert iter_tmp.isLocalVariable() function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionMakeGeneratorObject( generator_ref = ExpressionFunctionRef( function_body = code_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ) ) ) else: code_body = function_body if start_value is not None: container_tmp = code_body.allocateTempVariable( temp_scope = None, name = "contraction_result" ) else: container_tmp = None statements, release_statements = _buildContractionBodyNode( function_body = code_body, assign_provider = assign_provider, provider = provider, node = node, emit_class = emit_class, iter_tmp = iter_tmp, temp_scope = None, start_value = start_value, container_tmp = container_tmp, source_ref = source_ref, ) if start_value is not None: statements.append( StatementReturn( expression = ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), source_ref = source_ref ) ) statements = ( makeTryFinallyStatement( provider = function_body, tried = statements, final = release_statements, source_ref = source_ref.atInternal() ), ) code_body.setBody( makeStatementsSequenceFromStatement( statement = StatementsFrame( statements = mergeStatements(statements, False), guard_mode = "pass_through" if emit_class is not ExpressionYield else "generator", code_object = code_object, source_ref = source_ref ) ) ) if not assign_provider: return ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = code_object, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionBuiltinIter1( value = buildNode( provider = provider, node = node.generators[0].iter, source_ref = source_ref ), source_ref = source_ref ), ), source_ref = source_ref ) else: return function_body Nuitka-0.5.18.1/nuitka/tree/ReformulationLambdaExpressions.py0000644000175000017500000001636512651071040024436 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of lambda expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIsNOT from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.FrameNodes import StatementsFrame from nuitka.nodes.FunctionNodes import ( ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.GeneratorNodes import ( ExpressionGeneratorObjectBody, ExpressionMakeGeneratorObject ) from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.YieldNodes import ExpressionYield from nuitka.PythonVersions import python_version from .Helpers import ( buildNode, buildNodeList, detectFunctionBodyKind, getKind, makeStatementsSequenceFromStatement, mergeStatements ) from .ReformulationFunctionStatements import ( buildFunctionWithParsing, buildParameterAnnotations, buildParameterKwDefaults ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildLambdaNode(provider, node, source_ref): # Many details to deal with, pylint: disable=R0914 assert getKind(node) == "Lambda" function_kind, flags, _written_variables, _non_local_declarations, _global_declarations = \ detectFunctionBodyKind( nodes = (node.body,) ) outer_body, function_body, code_object = buildFunctionWithParsing( provider = provider, function_kind = function_kind, name = "", function_doc = None, flags = flags, node = node, source_ref = source_ref ) if function_kind == "Function": code_body = function_body else: code_body = ExpressionGeneratorObjectBody( provider = function_body, name = "", flags = set(), source_ref = source_ref ) if function_kind == "Generator": function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionMakeGeneratorObject( generator_ref = ExpressionFunctionRef( function_body = code_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ) ) ) defaults = buildNodeList(provider, node.args.defaults, source_ref) kw_defaults = buildParameterKwDefaults( provider = provider, node = node, function_body = function_body, source_ref = source_ref ) body = buildNode( provider = code_body, node = node.body, source_ref = source_ref, ) if function_kind == "Generator": if python_version < 270: tmp_return_value = code_body.allocateTempVariable( temp_scope = None, name = "yield_return" ) statements = ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_return_value, source_ref = source_ref, ), source = body, source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIsNOT( left = ExpressionTempVariableRef( variable = tmp_return_value, source_ref = source_ref, ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementExpressionOnly( expression = ExpressionYield( expression = ExpressionTempVariableRef( variable = tmp_return_value, source_ref = source_ref, ), source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ) ) body = makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_return_value, source_ref = source_ref ), source_ref = source_ref ) else: body = StatementExpressionOnly( expression = body, source_ref = source_ref ) else: body = StatementReturn( expression = body, source_ref = source_ref ) body = StatementsFrame( statements = mergeStatements( (body,) ), code_object = code_object, guard_mode = "generator" if function_kind == "Generator" else "full", source_ref = body.getSourceReference() ) body = makeStatementsSequenceFromStatement( statement = body, ) code_body.setBody(body) annotations = buildParameterAnnotations(provider, node, source_ref) return ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = outer_body, source_ref = source_ref ), code_object = code_object, defaults = defaults, kw_defaults = kw_defaults, annotations = annotations, source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/tree/ReformulationPrintStatements.py0000644000175000017500000001261112651071040024145 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of print statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ImportNodes import ExpressionImportModuleHard from nuitka.nodes.PrintNodes import StatementPrintNewline, StatementPrintValue from .Helpers import ( buildNode, buildNodeList, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildPrintNode(provider, node, source_ref): # "print" statements, should only occur with Python2. if node.dest is not None: temp_scope = provider.allocateTempScope("print") tmp_target_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "target" ) target_default_statement = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), source = ExpressionImportModuleHard( module_name = "sys", import_name = "stdout", source_ref = source_ref ), source_ref = source_ref ) statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), source = buildNode( provider = provider, node = node.dest, source_ref = source_ref ), source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = target_default_statement ), no_branch = None, source_ref = source_ref ) ] values = buildNodeList( provider = provider, nodes = node.values, source_ref = source_ref ) if node.dest is not None: print_statements = [ StatementPrintValue( dest = ExpressionTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), value = value, source_ref = source_ref ) for value in values ] if node.nl: print_statements.append( StatementPrintNewline( dest = ExpressionTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), source_ref = source_ref ) ) statements.append( makeTryFinallyStatement( provider = provider, tried = print_statements, final = StatementReleaseVariable( variable = tmp_target_variable, source_ref = source_ref ), source_ref = source_ref ) ) else: statements = [ StatementPrintValue( dest = None, value = value, source_ref = source_ref ) for value in values ] if node.nl: statements.append( StatementPrintNewline( dest = None, source_ref = source_ref ) ) return makeStatementsSequenceFromStatements( *statements ) Nuitka-0.5.18.1/nuitka/tree/ReformulationBooleanExpressions.py0000644000175000017500000000617412651071040024632 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of boolean and/or expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.ConditionalNodes import ( ExpressionConditionalAND, ExpressionConditionalOR ) from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from .Helpers import buildNode, buildNodeList, getKind def buildBoolOpNode(provider, node, source_ref): bool_op = getKind(node.op) if bool_op == "Or": # The "or" may be short circuit and is therefore not a plain operation. values = buildNodeList(provider, node.values, source_ref) for value in values[:-1]: value.setCompatibleSourceReference(values[-1].getSourceReference()) source_ref = values[-1].getSourceReference() return buildOrNode( values = values, source_ref = source_ref ) elif bool_op == "And": # The "and" may be short circuit and is therefore not a plain operation. values = buildNodeList(provider, node.values, source_ref) for value in values[:-1]: value.setCompatibleSourceReference(values[-1].getSourceReference()) source_ref = values[-1].getSourceReference() return buildAndNode( values = values, source_ref = source_ref ) elif bool_op == "Not": # The "not" is really only a unary operation and no special. return ExpressionOperationNOT( operand = buildNode(provider, node.operand, source_ref), source_ref = source_ref ) else: assert False, bool_op def buildOrNode(values, source_ref): values = list(values) result = values.pop() # When we encounter, "or", we expect it to be at least two values. assert values while values: result = ExpressionConditionalOR( left = values.pop(), right = result, source_ref = source_ref ) return result def buildAndNode(values, source_ref): values = list(values) result = values.pop() # Unlike "or", for "and", this is used with only one value. while values: result = ExpressionConditionalAND( left = values.pop(), right = result, source_ref = source_ref ) return result Nuitka-0.5.18.1/nuitka/tree/ReformulationCallExpressions.py0000644000175000017500000002777612651071040024141 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of call expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable ) from nuitka.nodes.CallNodes import makeExpressionCall from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.FunctionNodes import ( ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.PythonVersions import python_version from .ComplexCallHelperFunctions import ( getFunctionCallHelperDictionaryUnpacking, getFunctionCallHelperKeywordsStarDict, getFunctionCallHelperKeywordsStarList, getFunctionCallHelperKeywordsStarListStarDict, getFunctionCallHelperPosKeywordsStarDict, getFunctionCallHelperPosKeywordsStarList, getFunctionCallHelperPosKeywordsStarListStarDict, getFunctionCallHelperPosStarDict, getFunctionCallHelperPosStarList, getFunctionCallHelperPosStarListStarDict, getFunctionCallHelperStarDict, getFunctionCallHelperStarList, getFunctionCallHelperStarListStarDict ) from .Helpers import ( buildNode, buildNodeList, getKind, makeDictCreationOrConstant, makeSequenceCreationOrConstant, makeStatementsSequenceFromStatements ) from .ReformulationDictionaryCreation import buildDictionaryUnpackingArgs from .ReformulationSequenceCreation import buildListUnpacking def buildCallNode(provider, node, source_ref): called = buildNode(provider, node.func, source_ref) if python_version >= 350: list_star_arg = None dict_star_arg = None positional_args = [] # For Python3.5 compatibility, the error handling with star argument last # is the old one, only with a starred argument before that, things use the # new unpacking code. for node_arg in node.args[:-1]: if getKind(node_arg) == "Starred": assert python_version >= 350 list_star_arg = buildListUnpacking(provider, node.args, source_ref) positional_args = [] break else: if node.args and getKind(node.args[-1]) == "Starred": assert python_version >= 350 list_star_arg = buildNode(provider, node.args[-1].value, source_ref) positional_args = buildNodeList(provider, node.args[:-1], source_ref) else: positional_args = buildNodeList(provider, node.args, source_ref) # Only the values of keyword pairs have a real source ref, and those only # really matter, so that makes sense. keys = [] values = [] for keyword in node.keywords[:-1]: if keyword.arg is None: assert python_version >= 350 outline_body = ExpressionOutlineBody( provider = provider, name = "dict_unpacking_call", source_ref = source_ref ) tmp_called = outline_body.allocateTempVariable( temp_scope = None, name = "called" ) helper_args = [ ExpressionTempVariableRef( variable = tmp_called, source_ref = source_ref ), ExpressionMakeTuple( elements = buildDictionaryUnpackingArgs( provider = provider, keys = (keyword.arg for keyword in node.keywords), values = (keyword.value for keyword in node.keywords), source_ref = source_ref ), source_ref = source_ref ) ] dict_star_arg = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getFunctionCallHelperDictionaryUnpacking(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = helper_args, source_ref = source_ref, ) outline_body.setBody( makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_called, source_ref = source_ref ), source = called, source_ref = source_ref ), StatementReturn( expression = _makeCallNode( called = ExpressionTempVariableRef( variable = tmp_called, source_ref = source_ref ), positional_args = positional_args, keys = keys, values = values, list_star_arg = list_star_arg, dict_star_arg = dict_star_arg, source_ref = source_ref, ), source_ref = source_ref ) ) ) return outline_body # For Python3.5 compatibility, the error handling with star argument last # is the old one, only with a starred argument before that, things use the # new unpacking code. if node.keywords and node.keywords[-1].arg is None: assert python_version >= 350 dict_star_arg = buildNode(provider, node.keywords[-1].value, source_ref) keywords = node.keywords[:-1] else: keywords = node.keywords for keyword in keywords: keys.append( ExpressionConstantRef( constant = keyword.arg, source_ref = source_ref, user_provided = True ) ) values.append( buildNode(provider, keyword.value, source_ref) ) if python_version < 350: list_star_arg = buildNode(provider, node.starargs, source_ref, True) dict_star_arg = buildNode(provider, node.kwargs, source_ref, True) return _makeCallNode( called = called, positional_args = positional_args, keys = keys, values = values, list_star_arg = list_star_arg, dict_star_arg = dict_star_arg, source_ref = source_ref, ) def _makeCallNode(called, positional_args, keys, values, list_star_arg, dict_star_arg, source_ref): # Many variables, but only to cover the many complex call cases. if list_star_arg is None and dict_star_arg is None: result = makeExpressionCall( called = called, args = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = positional_args, source_ref = source_ref ), kw = makeDictCreationOrConstant( keys = keys, values = values, source_ref = source_ref ), source_ref = source_ref, ) if values: result.setCompatibleSourceReference( source_ref = values[-1].getCompatibleSourceReference() ) elif positional_args: result.setCompatibleSourceReference( source_ref = positional_args[-1].getCompatibleSourceReference() ) return result else: # Dispatch to complex helper function for each case. These do # re-formulation of complex calls according to developer manual. key = ( len(positional_args) > 0, len(keys) > 0, list_star_arg is not None, dict_star_arg is not None ) table = { (True, True, True, False) : getFunctionCallHelperPosKeywordsStarList, (True, False, True, False) : getFunctionCallHelperPosStarList, (False, True, True, False) : getFunctionCallHelperKeywordsStarList, (False, False, True, False) : getFunctionCallHelperStarList, (True, True, False, True) : getFunctionCallHelperPosKeywordsStarDict, (True, False, False, True) : getFunctionCallHelperPosStarDict, (False, True, False, True) : getFunctionCallHelperKeywordsStarDict, (False, False, False, True) : getFunctionCallHelperStarDict, (True, True, True, True) : getFunctionCallHelperPosKeywordsStarListStarDict, (True, False, True, True) : getFunctionCallHelperPosStarListStarDict, (False, True, True, True) : getFunctionCallHelperKeywordsStarListStarDict, (False, False, True, True) : getFunctionCallHelperStarListStarDict, } get_helper = table[key] helper_args = [called] if positional_args: helper_args.append( makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = positional_args, source_ref = source_ref ) ) # Order of evaluation changed in Python3.5. if python_version >= 350 and list_star_arg is not None: helper_args.append(list_star_arg) if keys: helper_args.append( makeDictCreationOrConstant( keys = keys, values = values, source_ref = source_ref ) ) # Order of evaluation changed in Python3.5. if python_version < 350 and list_star_arg is not None: helper_args.append(list_star_arg) if dict_star_arg is not None: helper_args.append(dict_star_arg) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = get_helper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = helper_args, source_ref = source_ref, ) result.setCompatibleSourceReference( source_ref = helper_args[-1].getCompatibleSourceReference() ) return result Nuitka-0.5.18.1/nuitka/tree/ReformulationExecStatements.py0000644000175000017500000005051412651071040023741 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of "exec" statements Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, ExpressionBuiltinExceptionRef ) from nuitka.nodes.CallNodes import ExpressionCallEmpty from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.ExecEvalNodes import StatementExec, StatementLocalsDictSync from nuitka.nodes.GlobalsLocalsNodes import ( ExpressionBuiltinGlobals, ExpressionBuiltinLocals ) from nuitka.nodes.TypeNodes import ExpressionBuiltinIsinstance from .Helpers import ( buildNode, getKind, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def _getLocalsClassNode(provider): if provider.isCompiledPythonModule(): return ExpressionBuiltinGlobals else: return ExpressionBuiltinLocals def wrapEvalGlobalsAndLocals(provider, globals_node, locals_node, temp_scope, source_ref): """ Wrap the locals and globals arguments for "eval". This is called from the outside, and when the node tree already exists. """ globals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "globals" ) locals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "locals" ) if locals_node is None: locals_node = ExpressionConstantRef( constant = None, source_ref = source_ref ) if globals_node is None: globals_node = ExpressionConstantRef( constant = None, source_ref = source_ref ) post_statements = [] if provider.isExpressionClassBody(): post_statements.append( StatementLocalsDictSync( locals_arg = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref, ), source_ref = source_ref.atInternal() ) ) post_statements += [ StatementReleaseVariable( variable = globals_keeper_variable, source_ref = source_ref ), StatementReleaseVariable( variable = locals_keeper_variable, source_ref = source_ref ) ] # The locals default is dependent on exec_mode, globals or locals. locals_default = ExpressionConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), expression_no = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), expression_yes = _getLocalsClassNode(provider)( source_ref = source_ref ), source_ref = source_ref ) pre_statements = [ # First assign globals and locals temporary the values given. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), source = globals_node, source_ref = source_ref, ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source = locals_node, source_ref = source_ref, ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source = locals_default, source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), source = ExpressionBuiltinGlobals( source_ref = source_ref ), source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ) ] return ( ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref if globals_node is None else globals_node.getSourceReference() ), ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref if locals_node is None else locals_node.getSourceReference() ), makeStatementsSequence(pre_statements, False, source_ref), makeStatementsSequence(post_statements, False, source_ref) ) def buildExecNode(provider, node, source_ref): # "exec" statements, should only occur with Python2. # This is using many variables, due to the many details this is # dealing with. The locals and globals need to be dealt with in # temporary variables, and we need handling of indicators, so # that is just the complexity, pylint: disable=R0914 exec_globals = node.globals exec_locals = node.locals body = node.body orig_globals = exec_globals # Handle exec(a,b,c) to be same as exec a, b, c if exec_locals is None and exec_globals is None and \ getKind(body) == "Tuple": parts = body.elts body = parts[0] if len(parts) > 1: exec_globals = parts[1] if len(parts) > 2: exec_locals = parts[2] else: return StatementRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = "TypeError", source_ref = source_ref ), exception_value = ExpressionConstantRef( constant = """\ exec: arg 1 must be a string, file, or code object""", source_ref = source_ref ), exception_trace = None, exception_cause = None, source_ref = source_ref ) if not provider.isCompiledPythonModule(): provider.markAsExecContaining() if orig_globals is None: provider.markAsUnqualifiedExecContaining(source_ref) temp_scope = provider.allocateTempScope("exec") locals_value = buildNode(provider, exec_locals, source_ref, True) if locals_value is None: locals_value = ExpressionConstantRef( constant = None, source_ref = source_ref ) globals_value = buildNode(provider, exec_globals, source_ref, True) if globals_value is None: globals_value = ExpressionConstantRef( constant = None, source_ref = source_ref ) source_code = buildNode(provider, body, source_ref) source_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "exec_source" ) globals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "globals" ) locals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "locals" ) plain_indicator_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "plain" ) tried = ( # First evaluate the source code expressions. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = source_variable, source_ref = source_ref ), source = source_code, source_ref = source_ref ), # Assign globals and locals temporary the values given, then fix it # up, taking note in the "plain" temporary variable, if it was an # "exec" statement with None arguments, in which case the copy back # will be necessary. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), source = globals_value, source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source = locals_value, source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = plain_indicator_variable, source_ref = source_ref ), source = ExpressionConstantRef( constant = False, source_ref = source_ref ), source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), source = ExpressionBuiltinGlobals( source_ref = source_ref ), source_ref = source_ref, ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source = _getLocalsClassNode(provider)( source_ref = source_ref ), source_ref = source_ref, ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = plain_indicator_variable, source_ref = source_ref ), source = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ), ), no_branch = makeStatementsSequenceFromStatements( StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = None, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ) ), source_ref = source_ref ), # Source needs some special treatment for not done for "eval", if it's a # file object, then must be read. StatementConditional( condition = ExpressionBuiltinIsinstance( instance = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "file", source_ref = source_ref, ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = source_variable, source_ref = source_ref ), source = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), attribute_name = "read", source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ), makeTryFinallyStatement( provider = provider, tried = StatementExec( source_code = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), globals_arg = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), locals_arg = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source_ref = source_ref ), final = StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = plain_indicator_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementLocalsDictSync( locals_arg = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref, ), source_ref = source_ref.atInternal() ) ), no_branch = None, source_ref = source_ref ), source_ref = source_ref ) ) final = ( StatementReleaseVariable( variable = source_variable, source_ref = source_ref ), StatementReleaseVariable( variable = globals_keeper_variable, source_ref = source_ref ), StatementReleaseVariable( variable = locals_keeper_variable, source_ref = source_ref ), StatementReleaseVariable( variable = plain_indicator_variable, source_ref = source_ref ), ) return makeTryFinallyStatement( provider = provider, tried = tried, final = final, source_ref = source_ref ) # This is here, to make sure it can register, pylint: disable=W0611 import nuitka.optimizations.OptimizeBuiltinCalls # isort:skip @UnusedImport Nuitka-0.5.18.1/nuitka/tree/ReformulationSequenceCreation.py0000644000175000017500000003755312651071040024252 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of sequence creations. Sequences might be directly translated to constants, or they might become nodes that build tuples, lists, or sets. For Python3.5, unpacking can happen while creating sequences, these are being re-formulated to an internal function. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionBuiltinNext1 ) from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinTuple from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ContainerOperationNodes import ( ExpressionListOperationExtend, ExpressionSetOperationUpdate ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from . import SyntaxErrors from .Helpers import ( buildNode, buildNodeList, getKind, makeSequenceCreationOrConstant, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) from .InternalModule import ( getInternalModule, internal_source_ref, once_decorator ) from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildSequenceCreationNode(provider, node, source_ref): if python_version >= 300: for element in node.elts: if getKind(element) == "Starred": if python_version < 350: SyntaxErrors.raiseSyntaxError( reason = """\ can use starred expression only as assignment target""", source_ref = source_ref, col_offset = element.col_offset ) else: return _buildSequenceUnpacking( provider = provider, node = node, source_ref = source_ref ) return makeSequenceCreationOrConstant( sequence_kind = getKind(node).upper(), elements = buildNodeList(provider, node.elts, source_ref), source_ref = source_ref ) @once_decorator def getListUnpackingHelper(): helper_name = "_unpack_list" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = (), list_star_arg = "args", dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "list") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "keys") loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementExpressionOnly( expression = ExpressionListOperationExtend( list_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) args_variable = result.getVariableForAssignment( variable_name = "args" ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = [], source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementLoop( body = loop_body, source_ref = internal_source_ref ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getSetUnpackingHelper(): helper_name = "_unpack_set" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = (), list_star_arg = "args", dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "set") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "keys") loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementExpressionOnly( expression = ExpressionSetOperationUpdate( set_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) args_variable = result.getVariableForAssignment( variable_name = "args" ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = set(), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementLoop( body = loop_body, source_ref = internal_source_ref ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result def buildListUnpacking(provider, elements, source_ref): helper_args = [] for element in elements: # TODO: We could be a lot cleverer about the tuples for non-starred # arguments, but lets get this to work first. if getKind(element) == "Starred": helper_args.append( buildNode(provider, element.value, source_ref), ) else: helper_args.append( ExpressionMakeTuple( elements = ( buildNode(provider, element, source_ref), ), source_ref = source_ref ) ) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getListUnpackingHelper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionMakeTuple( helper_args, source_ref ), ), source_ref = source_ref, ) result.setCompatibleSourceReference(helper_args[-1].getCompatibleSourceReference()) return result def _buildTupleUnpacking(provider, elements, source_ref): return ExpressionBuiltinTuple( value = buildListUnpacking(provider, elements, source_ref), source_ref = source_ref ) def _buildSetUnpacking(provider, elements, source_ref): helper_args = [] for element in elements: # TODO: We could be a lot cleverer about the tuples for non-starred # arguments, but lets get this to work first. if getKind(element) == "Starred": helper_args.append( buildNode(provider, element.value, source_ref), ) else: helper_args.append( ExpressionMakeTuple( elements = ( buildNode(provider, element, source_ref), ), source_ref = source_ref ) ) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getSetUnpackingHelper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionMakeTuple( helper_args, source_ref ), ), source_ref = source_ref, ) result.setCompatibleSourceReference(helper_args[-1].getCompatibleSourceReference()) return result def _buildSequenceUnpacking(provider, node, source_ref): kind = getKind(node) if kind == "List": return buildListUnpacking(provider, node.elts, source_ref) elif kind == "Tuple": return _buildTupleUnpacking(provider, node.elts, source_ref) elif kind == "Set": return _buildSetUnpacking(provider, node.elts, source_ref) else: assert False, kind Nuitka-0.5.18.1/nuitka/tree/Extractions.py0000644000175000017500000000445612651071040020545 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Extracting visitors. This is used for look-aheads supporting abstract execution. We need to e.g. know the variables written by a piece of code ahead of abstractly executing a loop. """ from .Operations import VisitorNoopMixin, visitTree class VariableWriteExtractor(VisitorNoopMixin): """ Extract variables written to. """ def __init__(self): self.written_to = set() def onEnterNode(self, node): if node.isExpressionTargetVariableRef() or \ node.isExpressionTargetTempVariableRef(): self.written_to.add(node.getVariable()) def getResult(self): return self.written_to def getVariablesWritten(node): visitor = VariableWriteExtractor() visitTree(node, visitor) return visitor.getResult() class VariableUsageUpdater(VisitorNoopMixin): def __init__(self, old_variable, new_variable): self.old_variable = old_variable self.new_variable = new_variable def onEnterNode(self, node): if node.isExpressionTargetVariableRef() or \ node.isExpressionTargetTempVariableRef() or \ node.isExpressionVariableRef() or \ node.isExpressionTempVariableRef() or \ node.isStatementReleaseVariable(): if node.getVariable() is self.old_variable: node.setVariable(self.new_variable) def updateVariableUsage(provider, old_variable, new_variable): visitor = VariableUsageUpdater( old_variable = old_variable, new_variable = new_variable ) visitTree(provider, visitor) Nuitka-0.5.18.1/nuitka/tree/ReformulationImportStatements.py0000644000175000017500000001725612651071040024335 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of import statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka import Options from nuitka.nodes.AssignNodes import ( ExpressionTargetVariableRef, StatementAssignmentVariable ) from nuitka.nodes.ImportNodes import ( ExpressionImportModule, ExpressionImportModuleHard, ExpressionImportName, StatementImportStar ) from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.PythonVersions import python_version from nuitka.tree import SyntaxErrors from .Helpers import mangleName # For checking afterwards, if __future__ imports really were at the beginning # of the file. _future_import_nodes = [] def checkFutureImportsOnlyAtStart(body): # Check if a __future__ imports really were at the beginning of the file. for node in body: if node in _future_import_nodes: _future_import_nodes.remove(node) else: if _future_import_nodes: SyntaxErrors.raiseSyntaxError( reason = """\ from __future__ imports must occur at the beginning of the file""", col_offset = 1 if python_version >= 300 or \ not Options.isFullCompat() else None, source_ref = _future_import_nodes[0].source_ref ) def buildImportFromNode(provider, node, source_ref): # "from .. import .." statements. This may trigger a star import, or # multiple names being looked up from the given module variable name. module_name = node.module if node.module is not None else "" level = node.level # Importing from "__future__" module may enable flags to the parser, # that we need to know. if module_name == "__future__": # Future imports we see are all legal, and known to work. if not provider.isCompiledPythonModule(): SyntaxErrors.raiseSyntaxError( reason = """\ from __future__ imports must occur at the beginning of the file""", col_offset = 8 if python_version >= 300 or \ not Options.isFullCompat() else None, source_ref = source_ref ) for import_desc in node.names: object_name, _local_name = import_desc.name, import_desc.asname enableFutureFeature( object_name = object_name, future_spec = source_ref.getFutureSpec(), source_ref = source_ref ) # Remember it for checks to be applied once module is complete. node.source_ref = source_ref _future_import_nodes.append(node) target_names = [] import_names = [] for import_desc in node.names: object_name, local_name = import_desc.name, import_desc.asname if object_name == '*': target_names.append(None) else: target_names.append( local_name if local_name is not None else object_name ) import_names.append(object_name) if None in target_names: # More than "*" is a syntax error in Python, need not care about this at # all, it's only allowed value for import list in this case. assert target_names == [None] # Python3 made this a syntax error unfortunately. if not provider.isCompiledPythonModule() and python_version >= 300: SyntaxErrors.raiseSyntaxError( "import * only allowed at module level", provider.getSourceReference() ) if provider.isExpressionFunctionBody(): provider.markAsStarImportContaining() return StatementImportStar( module_import = ExpressionImportModule( module_name = module_name, import_list = ('*',), level = level, source_ref = source_ref ), source_ref = source_ref ) else: # Make __future__ imports "hard" immediately, they cannot be any other # way. def makeImportName(import_name): if module_name == "__future__": return ExpressionImportModuleHard( module_name = "__future__", import_name = import_name, source_ref = source_ref ) else: # TODO: This ought to use a temporary variable for multiple # names, instead of importing multiple times. return ExpressionImportName( module = ExpressionImportModule( module_name = module_name, import_list = tuple(import_names), level = level, source_ref = source_ref ), import_name = import_name, source_ref = source_ref ) import_nodes = [] for target_name, import_name in zip(target_names, import_names): import_nodes.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = mangleName(target_name, provider), source_ref = source_ref ), source = makeImportName( import_name = import_name, ), source_ref = source_ref ) ) # Note: Each import is sequential. It can succeed, and the failure of a # later one is not changing one. We can therefore have a sequence of # imports that only import one thing therefore. return StatementsSequence( statements = import_nodes, source_ref = source_ref ) def enableFutureFeature(object_name, future_spec, source_ref): if object_name == "unicode_literals": future_spec.enableUnicodeLiterals() elif object_name == "absolute_import": future_spec.enableAbsoluteImport() elif object_name == "division": future_spec.enableFutureDivision() elif object_name == "print_function": future_spec.enableFuturePrint() elif object_name == "barry_as_FLUFL" and python_version >= 300: future_spec.enableBarry() elif object_name == "generator_stop": future_spec.enableGeneratorStop() elif object_name == "braces": SyntaxErrors.raiseSyntaxError( "not a chance", source_ref ) elif object_name in ("nested_scopes", "generators", "with_statement"): # These are enabled in all cases already. pass else: SyntaxErrors.raiseSyntaxError( "future feature %s is not defined" % object_name, source_ref ) Nuitka-0.5.18.1/nuitka/tree/ReformulationWithStatements.py0000644000175000017500000003523012651071040023766 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of with statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka import Options from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionAttributeLookupSpecial ) from nuitka.nodes.CallNodes import ExpressionCallEmpty, ExpressionCallNoKeywords from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.CoroutineNodes import ExpressionAsyncWait from nuitka.nodes.ExceptionNodes import ( ExpressionCaughtExceptionTracebackRef, ExpressionCaughtExceptionTypeRef, ExpressionCaughtExceptionValueRef ) from nuitka.nodes.StatementNodes import ( StatementExpressionOnly, StatementsSequence ) from nuitka.PythonVersions import python_version from nuitka.tree.Helpers import makeReraiseExceptionStatement from .Helpers import ( buildNode, buildStatementsNode, getKind, makeConditionalStatement, makeStatementsSequence, makeStatementsSequenceFromStatement ) from .ReformulationAssignmentStatements import buildAssignmentStatements from . \ ReformulationTryExceptStatements import makeTryExceptSingleHandlerNodeWithPublish from .ReformulationTryFinallyStatements import makeTryFinallyStatement def _buildWithNode(provider, context_expr, assign_target, body, body_lineno, sync, source_ref): # Many details, pylint: disable=R0914 with_source = buildNode(provider, context_expr, source_ref) if Options.isFullCompat(): source_ref = with_source.getCompatibleSourceReference() temp_scope = provider.allocateTempScope("with") tmp_source_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "source" ) tmp_exit_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "exit" ) tmp_enter_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "enter" ) tmp_indicator_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "indicator" ) statements = ( buildAssignmentStatements( provider = provider, node = assign_target, allow_none = True, source = ExpressionTempVariableRef( variable = tmp_enter_variable, source_ref = source_ref ), source_ref = source_ref ), body ) with_body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) if Options.isFullCompat(): if body: deepest = body while deepest.getVisitableNodes(): deepest = deepest.getVisitableNodes()[-1] body_lineno = deepest.getCompatibleSourceReference().getLineNumber() with_exit_source_ref = source_ref.atLineNumber(body_lineno) else: with_exit_source_ref = source_ref # The "__enter__" and "__exit__" were normal attribute lookups under # CPython2.6, but that changed with CPython2.7. if python_version < 270: attribute_lookup_class = ExpressionAttributeLookup else: attribute_lookup_class = ExpressionAttributeLookupSpecial enter_value = ExpressionCallEmpty( called = attribute_lookup_class( source = ExpressionTempVariableRef( variable = tmp_source_variable, source_ref = source_ref ), attribute_name = "__enter__" if sync else "__aenter__", source_ref = source_ref ), source_ref = source_ref ) exit_value_exception = ExpressionCallNoKeywords( called = ExpressionTempVariableRef( variable = tmp_exit_variable, source_ref = with_exit_source_ref ), args = ExpressionMakeTuple( elements = ( ExpressionCaughtExceptionTypeRef( source_ref = with_exit_source_ref ), ExpressionCaughtExceptionValueRef( source_ref = with_exit_source_ref ), ExpressionCaughtExceptionTracebackRef( source_ref = source_ref ), ), source_ref = source_ref ), source_ref = with_exit_source_ref ) exit_value_no_exception = ExpressionCallNoKeywords( called = ExpressionTempVariableRef( variable = tmp_exit_variable, source_ref = source_ref ), args = ExpressionConstantRef( constant = (None, None, None), source_ref = source_ref ), source_ref = with_exit_source_ref ) # For "async with", await the entered value and exit value must be awaited. if not sync: enter_value = ExpressionAsyncWait( expression = enter_value, source_ref = source_ref ) exit_value_exception = ExpressionAsyncWait( expression = exit_value_exception, source_ref = source_ref ) exit_value_no_exception = ExpressionAsyncWait( expression = exit_value_no_exception, source_ref = source_ref ) statements = [ # First assign the with context to a temporary variable. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_source_variable, source_ref = source_ref ), source = with_source, source_ref = source_ref ), # Next, assign "__enter__" and "__exit__" attributes to temporary # variables. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_exit_variable, source_ref = source_ref ), source = attribute_lookup_class( source = ExpressionTempVariableRef( variable = tmp_source_variable, source_ref = source_ref ), attribute_name = "__exit__" if sync else "__aexit__", source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_enter_variable, source_ref = source_ref ), source = enter_value, source_ref = source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_indicator_variable, source_ref = source_ref ), source = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), ] statements += [ makeTryFinallyStatement( provider = provider, tried = makeTryExceptSingleHandlerNodeWithPublish( provider = provider, tried = with_body, exception_name = "BaseException", handler_body = StatementsSequence( statements = ( # Prevents final block from calling __exit__ as # well. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_indicator_variable, source_ref = source_ref ), source = ExpressionConstantRef( constant = False, source_ref = source_ref ), source_ref = source_ref ), makeConditionalStatement( condition = exit_value_exception, no_branch = makeReraiseExceptionStatement( source_ref = with_exit_source_ref ), yes_branch = None, source_ref = with_exit_source_ref ), ), source_ref = source_ref ), public_exc = python_version >= 270, source_ref = source_ref ), final = StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_indicator_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementExpressionOnly( expression = exit_value_no_exception, source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ), source_ref = source_ref ) ] return makeTryFinallyStatement( provider = provider, tried = statements, final = ( StatementReleaseVariable( variable = tmp_source_variable, source_ref = with_exit_source_ref ), StatementReleaseVariable( variable = tmp_enter_variable, source_ref = with_exit_source_ref ), StatementReleaseVariable( variable = tmp_exit_variable, source_ref = with_exit_source_ref ), StatementReleaseVariable( variable = tmp_indicator_variable, source_ref = with_exit_source_ref ), ), source_ref = source_ref ) def buildWithNode(provider, node, source_ref): # "with" statements are re-formulated as described in the developer # manual. Catches exceptions, and provides them to "__exit__", while making # the "__enter__" value available under a given name. # Before Python3.3, multiple context managers are not visible in the parse # tree, now we need to handle it ourselves. if hasattr(node, "items"): context_exprs = [item.context_expr for item in node.items] assign_targets = [item.optional_vars for item in node.items] else: # Make it a list for before Python3.3 context_exprs = [node.context_expr] assign_targets = [node.optional_vars] # The body for the first context manager is the other things. body = buildStatementsNode(provider, node.body, source_ref) assert len(context_exprs) > 0 and len(context_exprs) == len(assign_targets) context_exprs.reverse() assign_targets.reverse() # For compatibility, we need to gather a line number for the body here # already, but only the full compatibility mode will use it. terminal_statement = node.body[-1] while getKind(terminal_statement) in ("With", "AsyncWith"): terminal_statement = terminal_statement.body[-1] body_lineno = terminal_statement.lineno for context_expr, assign_target in zip(context_exprs, assign_targets): body = _buildWithNode( provider = provider, body = body, body_lineno = body_lineno, context_expr = context_expr, assign_target = assign_target, sync = True, source_ref = source_ref ) return body def buildAsyncWithNode(provider, node, source_ref): # "with" statements are re-formulated as described in the developer # manual. Catches exceptions, and provides them to "__exit__", while making # the "__enter__" value available under a given name. # Before Python3.3, multiple context managers are not visible in the parse # tree, now we need to handle it ourselves. context_exprs = [item.context_expr for item in node.items] assign_targets = [item.optional_vars for item in node.items] # The body for the first context manager is the other things. body = buildStatementsNode(provider, node.body, source_ref) assert len(context_exprs) > 0 and len(context_exprs) == len(assign_targets) context_exprs.reverse() assign_targets.reverse() # For compatibility, we need to gather a line number for the body here # already, but only the full compatibility mode will use it. terminal_statement = node.body[-1] while getKind(terminal_statement) in ("With", "AsyncWith"): terminal_statement = terminal_statement.body[-1] body_lineno = terminal_statement.lineno for context_expr, assign_target in zip(context_exprs, assign_targets): body = _buildWithNode( provider = provider, body = body, body_lineno = body_lineno, context_expr = context_expr, assign_target = assign_target, sync = False, source_ref = source_ref ) return body Nuitka-0.5.18.1/nuitka/tree/ReformulationYieldExpressions.py0000644000175000017500000000521312651071040024312 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of "yield" and "yield from" expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ import ast from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.YieldNodes import ExpressionYield, ExpressionYieldFrom from nuitka.PythonVersions import python_version from nuitka.tree import SyntaxErrors from .Helpers import buildNode def _checkInsideGenerator(provider, node, source_ref): if provider.isCompiledPythonModule(): SyntaxErrors.raiseSyntaxError( "'yield' outside function", source_ref, None if python_version < 300 else node.col_offset ) if provider.isExpressionCoroutineObjectBody(): SyntaxErrors.raiseSyntaxError( "'%s' inside async function" % ( "yield" if node.__class__ is ast.Yield else "yield from", ), source_ref, node.col_offset+3 ) assert provider.isExpressionGeneratorObjectBody(), provider def buildYieldNode(provider, node, source_ref): _checkInsideGenerator(provider, node, source_ref) if node.value is not None: return ExpressionYield( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) else: return ExpressionYield( expression = ExpressionConstantRef( constant = None, source_ref = source_ref, user_provided = True ), source_ref = source_ref ) def buildYieldFromNode(provider, node, source_ref): assert python_version >= 330 _checkInsideGenerator(provider, node, source_ref) return ExpressionYieldFrom( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/tree/ReformulationTryExceptStatements.py0000644000175000017500000004075212651071040025007 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of try/except statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka import Options from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinRefNodes import ExpressionBuiltinExceptionRef from nuitka.nodes.ComparisonNodes import ( ExpressionComparisonExceptionMatch, ExpressionComparisonIs ) from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ExceptionNodes import ( ExpressionCaughtExceptionTypeRef, ExpressionCaughtExceptionValueRef ) from nuitka.nodes.StatementNodes import ( StatementPreserveFrameException, StatementPublishException, StatementRestoreFrameException, StatementsSequence ) from nuitka.nodes.TryNodes import StatementTry from nuitka.PythonVersions import python_version from nuitka.tree import SyntaxErrors from .Helpers import ( buildNode, buildStatementsNode, makeReraiseExceptionStatement, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, mergeStatements ) from .ReformulationAssignmentStatements import ( buildAssignmentStatements, buildDeleteStatementFromDecoded, decodeAssignTarget ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def makeTryExceptNoRaise(provider, temp_scope, tried, handling, no_raise, source_ref): # This helper executes the core re-formulation of "no_raise" blocks, which # are the "else" blocks of "try"/"except" statements. In order to limit the # execution, we use an indicator variable instead, which will signal that # the tried block executed up to the end. And then we make the else block be # a conditional statement checking that. tmp_handler_indicator_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "unhandled_indicator" ) statements = mergeStatements( ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_handler_indicator_variable, source_ref = source_ref.atInternal() ), source = ExpressionConstantRef( constant = False, source_ref = source_ref ), source_ref = no_raise.getSourceReference().atInternal() ), handling ), allow_none = True ) handling = StatementsSequence( statements = statements, source_ref = source_ref ) tried = ( StatementTry( tried = tried, except_handler = handling, break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_handler_indicator_variable, source_ref = source_ref ), right = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = no_raise, no_branch = None, source_ref = source_ref ) ) final = StatementReleaseVariable( variable = tmp_handler_indicator_variable, source_ref = source_ref.atInternal() ) return makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_handler_indicator_variable, source_ref = source_ref.atInternal() ), source = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref.atInternal() ), makeTryFinallyStatement( provider = provider, tried = tried, final = final, source_ref = source_ref ) ) def _makeTryExceptSingleHandlerNode(provider, public_exc, tried, exception_name, handler_body, source_ref): # No need to create this in the first place if nothing is tried. if tried is None: return None if public_exc: preserver_id = provider.allocatePreserverId() handling = [ StatementPreserveFrameException( preserver_id = preserver_id, source_ref = source_ref ), StatementPublishException( source_ref = source_ref ) ] else: handling = [] if not handler_body.isStatementsSequence(): handler_body = makeStatementsSequenceFromStatement( statement = handler_body ) if not tried.isStatementsSequence(): tried = makeStatementsSequenceFromStatement( statement = tried ) handling.append( StatementConditional( condition = ExpressionComparisonExceptionMatch( left = ExpressionCaughtExceptionTypeRef( source_ref = source_ref ), right = ExpressionBuiltinExceptionRef( exception_name = exception_name, source_ref = source_ref ), source_ref = source_ref ), yes_branch = handler_body, no_branch = makeReraiseExceptionStatement( source_ref = source_ref ), source_ref = source_ref ) ) if python_version >= 300 and public_exc: handling = ( makeTryFinallyStatement( provider = provider, tried = handling, final = StatementRestoreFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), source_ref = source_ref.atInternal() ), ) handling = makeStatementsSequenceFromStatements( *handling ) return StatementTry( tried = tried, except_handler = handling, break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ) def makeTryExceptSingleHandlerNode(tried, exception_name, handler_body, source_ref): return _makeTryExceptSingleHandlerNode( provider = None, public_exc = False, tried = tried, exception_name = exception_name, handler_body = handler_body, source_ref = source_ref ) def makeTryExceptSingleHandlerNodeWithPublish(provider, public_exc, tried, exception_name, handler_body, source_ref): return _makeTryExceptSingleHandlerNode( provider = provider, public_exc = public_exc, tried = tried, exception_name = exception_name, handler_body = handler_body, source_ref = source_ref ) def buildTryExceptionNode(provider, node, source_ref): # Try/except nodes. Re-formulated as described in the developer # manual. Exception handlers made the assignment to variables explicit. Same # for the "del" as done for Python3. Also catches always work a tuple of # exception types and hides away that they may be built or not. # Many variables and branches, due to the re-formulation that is going on # here, which just has the complexity, pylint: disable=R0912,R0914 tried = buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ) handlers = [] for handler in node.handlers: exception_expression, exception_assign, exception_block = ( handler.type, handler.name, handler.body ) if exception_assign is None: statements = [ buildStatementsNode( provider = provider, nodes = exception_block, source_ref = source_ref ) ] elif python_version < 300: statements = [ buildAssignmentStatements( provider = provider, node = exception_assign, source = ExpressionCaughtExceptionValueRef( source_ref = source_ref.atInternal() ), source_ref = source_ref.atInternal() ), buildStatementsNode( provider = provider, nodes = exception_block, source_ref = source_ref ) ] else: # Python3 requires temporary assignment of exception assignment. target_info = decodeAssignTarget( provider = provider, node = exception_assign, source_ref = source_ref, ) kind, detail = target_info assert kind == "Name", kind kind = "Name_Exception" statements = [ buildAssignmentStatements( provider = provider, node = exception_assign, source = ExpressionCaughtExceptionValueRef( source_ref = source_ref.atInternal() ), source_ref = source_ref.atInternal() ), makeTryFinallyStatement( provider = provider, tried = buildStatementsNode( provider = provider, nodes = exception_block, source_ref = source_ref ), final = buildDeleteStatementFromDecoded( kind = kind, detail = detail, source_ref = source_ref ), source_ref = source_ref ) ] handler_body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) exception_types = buildNode( provider = provider, node = exception_expression, source_ref = source_ref, allow_none = True ) # The exception types should be a tuple, so as to be most general. if exception_types is None: if handler is not node.handlers[-1]: SyntaxErrors.raiseSyntaxError( reason = "default 'except:' must be last", source_ref = source_ref.atLineNumber( handler.lineno-1 if Options.isFullCompat() else handler.lineno ) ) handlers.append( ( exception_types, handler_body, ) ) # Re-raise by default exception_handling = makeReraiseExceptionStatement( source_ref = source_ref ) for exception_type, handler in reversed(handlers): if exception_type is None: # A default handler was given, so use that indeed. exception_handling = handler else: exception_handling = StatementsSequence( statements = ( StatementConditional( condition = ExpressionComparisonExceptionMatch( left = ExpressionCaughtExceptionTypeRef( source_ref = exception_type.source_ref ), right = exception_type, source_ref = exception_type.source_ref ), yes_branch = handler, no_branch = exception_handling, source_ref = exception_type.source_ref ), ), source_ref = exception_type.source_ref ) if exception_handling is None: # For Python3, we need not publish at all, if all we do is to revert # that immediately. For Python2, the publish may release previously # published exception, which has side effects potentially. if python_version < 300: exception_handling = StatementsSequence( statements = ( StatementPreserveFrameException( preserver_id = 0, # unused with Python2 source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref.atInternal() ), ), source_ref = source_ref.atInternal() ) else: if python_version < 300: exception_handling.setStatements( ( StatementPreserveFrameException( preserver_id = 0, # unused with Python2 source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref.atInternal() ), ) + exception_handling.getStatements() ) else: preserver_id = provider.allocatePreserverId() exception_handling = makeStatementsSequenceFromStatements( StatementPreserveFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref.atInternal() ), makeTryFinallyStatement( provider = provider, tried = exception_handling, final = StatementRestoreFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), source_ref = source_ref ), ) no_raise = buildStatementsNode( provider = provider, nodes = node.orelse, source_ref = source_ref ) if no_raise is None: if tried is None: return None return StatementTry( tried = tried, except_handler = exception_handling, break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ) else: if tried is None: return no_raise return makeTryExceptNoRaise( provider = provider, temp_scope = provider.allocateTempScope("try_except"), handling = exception_handling, tried = tried, no_raise = no_raise, source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/tree/ReformulationWhileLoopStatements.py0000644000175000017500000001351412651071040024756 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of while loop statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.StatementNodes import StatementsSequence from .Helpers import ( buildNode, buildStatementsNode, makeStatementsSequence, mergeStatements, popBuildContext, pushBuildContext ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement def buildWhileLoopNode(provider, node, source_ref): # The while loop is re-formulated according to developer manual. The # condition becomes an early condition to break the loop. The else block is # taken if a while loop exits normally, i.e. because of condition not being # true. We do this by introducing an indicator variable. else_block = buildStatementsNode( provider = provider, nodes = node.orelse if node.orelse else None, source_ref = source_ref ) if else_block is not None: temp_scope = provider.allocateTempScope("while_loop") tmp_break_indicator = provider.allocateTempVariable( temp_scope = temp_scope, name = "break_indicator" ) statements = ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), source = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), StatementLoopBreak( source_ref = source_ref ) ) else: statements = ( StatementLoopBreak( source_ref = source_ref ), ) pushBuildContext("loop_body") loop_statements = buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ) popBuildContext() # The loop body contains a conditional statement at the start that breaks # the loop if it fails. loop_body = makeStatementsSequence( statements = ( StatementConditional( condition = ExpressionOperationNOT( operand = buildNode(provider, node.test, source_ref), source_ref = source_ref, ), yes_branch = StatementsSequence( statements = statements, source_ref = source_ref ), no_branch = None, source_ref = source_ref ), loop_statements ), allow_none = True, source_ref = source_ref ) loop_statement = StatementLoop( body = loop_body, source_ref = source_ref ) if else_block is None: return loop_statement else: statements = ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), source = ExpressionConstantRef( constant = False, source_ref = source_ref ), source_ref = source_ref ), loop_statement, StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), right = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = else_block, no_branch = None, source_ref = source_ref ) ) statements = ( makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_break_indicator, source_ref = source_ref ), source_ref = source_ref ), ) return StatementsSequence( statements = mergeStatements(statements, False), source_ref = source_ref ) Nuitka-0.5.18.1/nuitka/tree/__init__.py0000644000175000017500000000150112651071040017765 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/tree/Helpers.py0000644000175000017500000005226512651071040017645 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Helper functions for parsing the AST nodes and building the Nuitka node tree. """ import ast from logging import warning from nuitka import Constants, Options, Tracing from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ( ExpressionMakeList, ExpressionMakeSet, ExpressionMakeTuple ) from nuitka.nodes.DictionaryNodes import ( ExpressionKeyValuePair, ExpressionMakeDict ) from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.FrameNodes import StatementsFrame from nuitka.nodes.NodeBases import NodeBase from nuitka.nodes.NodeMakingHelpers import mergeStatements from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.StatementNodes import ( StatementGeneratorEntry, StatementsSequence ) from nuitka.PythonVersions import doShowUnknownEncodingName, python_version def dump(node): Tracing.printLine(ast.dump(node)) def getKind(node): return node.__class__.__name__.split('.')[-1] def extractDocFromBody(node): # Work around ast.get_docstring breakage. if len(node.body) > 0 and getKind(node.body[0]) == "Expr" and getKind(node.body[0].value) == "Str": return node.body[1:], node.body[0].value.s else: return node.body, None def _makeSyntaxErrorCompatible(e): # Encoding problems for Python happen here, for Python3, this was # already done when we read the source code. if Options.isFullCompat() and \ (e.args[0].startswith("unknown encoding:") or \ e.args[0].startswith("encoding problem:")): if doShowUnknownEncodingName(): complaint = e.args[0].split(':',2)[1] else: complaint = " with BOM" e.args = ( "encoding problem:%s" % complaint, (e.args[1][0], 1, None, None) ) if hasattr(e, "msg"): e.msg = e.args[0] def parseSourceCodeToAst(source_code, filename, line_offset): # Workaround: ast.parse cannot cope with some situations where a file is not # terminated by a new line. if not source_code.endswith('\n'): source_code = source_code + '\n' try: body = ast.parse(source_code, filename) except SyntaxError as e: _makeSyntaxErrorCompatible(e) raise e assert getKind(body) == "Module" if line_offset > 0: ast.increment_lineno(body, line_offset) return body def detectFunctionBodyKind(nodes): # This is a complex mess, following the scope means a lot of checks need # to be done. pylint: disable=R0912,R0915 indications = set() written_variables = set() non_local_declarations = set() global_declarations = set() flags = set() # print "Enter" def _check(node): # print "consider", node.__class__ node_class = node.__class__ if node_class is ast.Yield: indications.add("Generator") elif python_version >= 330 and node_class is ast.YieldFrom: # @UndefinedVariable indications.add("Generator") elif python_version >= 350 and node_class in (ast.Await, ast.AsyncWith): # @UndefinedVariable indications.add("Coroutine") # Detect assignments to variables, for functions we need to know that # to properly resolve closure. if node_class is ast.Assign: for target in node.targets: if type(target) is str: written_variables.add(target) elif target.__class__ is ast.Name: written_variables.add(target.id) # Detect global and nonlocal declarations ahead of time. if python_version >= 300 and node_class is ast.Nonlocal: # @UndefinedVariable non_local_declarations.update(set(node.names)) elif node_class is ast.Global: global_declarations.update(set(node.names)) # Recurse to children, but do not cross scope boundary doing so. if node_class is ast.ClassDef: for name, field in ast.iter_fields(node): if name in ("name", "body"): pass elif name in ("bases", "decorator_list", "keywords"): for child in field: _check(child) elif name == "starargs": if field is not None: _check(field) elif name == "kwargs": if field is not None: _check(field) else: assert False, (name, field, ast.dump(node)) elif node_class in (ast.FunctionDef, ast.Lambda) or \ (python_version >= 350 and node_class is ast.AsyncFunctionDef): # @UndefinedVariable for name, field in ast.iter_fields(node): if name in ("name", "body"): pass elif name in ("bases", "decorator_list"): for child in field: _check(child) elif name == "args": for child in field.defaults: _check(child) if python_version >= 300: for child in node.args.kw_defaults: if child is not None: _check(child) for child in node.args.args: if child.annotation is not None: _check(child.annotation) elif name == "returns": if field is not None: _check(field) else: assert False, (name, field, ast.dump(node)) elif node_class is ast.GeneratorExp: for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "elt"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif node_class is ast.ListComp and python_version >= 300: for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "elt"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif python_version >= 270 and node_class is ast.SetComp: for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "elt"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif python_version >= 270 and node_class is ast.DictComp: for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "key", "value"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif node_class is ast.Name: if python_version >= 300 and node.id == "super": flags.add("has_super") else: for child in ast.iter_child_nodes(node): _check(child) for node in nodes: _check(node) if indications: # If we found something, make sure we agree on all clues. assert len(indications) == 1 function_kind = indications.pop() else: function_kind = "Function" return function_kind, flags, written_variables, non_local_declarations, global_declarations build_nodes_args3 = None build_nodes_args2 = None build_nodes_args1 = None def setBuildingDispatchers(path_args3, path_args2, path_args1): # Using global here, as this is really a singleton, in the form of a module, # and this is to break the cyclic dependency it has, pylint: disable=W0603 global build_nodes_args3, build_nodes_args2, build_nodes_args1 build_nodes_args3 = path_args3 build_nodes_args2 = path_args2 build_nodes_args1 = path_args1 def buildNode(provider, node, source_ref, allow_none = False): if node is None and allow_none: return None try: kind = getKind(node) if hasattr(node, "lineno"): source_ref = source_ref.atLineNumber(node.lineno) else: source_ref = source_ref if kind in build_nodes_args3: result = build_nodes_args3[kind]( provider = provider, node = node, source_ref = source_ref ) elif kind in build_nodes_args2: result = build_nodes_args2[kind]( node = node, source_ref = source_ref ) elif kind in build_nodes_args1: result = build_nodes_args1[kind]( source_ref = source_ref ) elif kind == "Pass": result = None else: assert False, ast.dump(node) if result is None and allow_none: return None assert isinstance(result, NodeBase), result return result except SyntaxError: raise except RuntimeError: # Very likely the stack overflow, which we will turn into too complex # code exception, don't warn about it with a code dump then. raise except: warning("Problem at '%s' with %s." % (source_ref, ast.dump(node))) raise def buildNodeList(provider, nodes, source_ref, allow_none = False): if nodes is not None: result = [] for node in nodes: if hasattr(node, "lineno"): node_source_ref = source_ref.atLineNumber(node.lineno) else: node_source_ref = source_ref entry = buildNode(provider, node, node_source_ref, allow_none) if entry is not None: result.append(entry) return result else: return [] def makeModuleFrame(module, statements, source_ref): assert module.isCompiledPythonModule() if module.isMainModule(): code_name = "" else: code_name = module.getName() return StatementsFrame( statements = statements, guard_mode = "once", code_object = CodeObjectSpec( code_name = code_name, code_kind = "Module", arg_names = (), kw_only_count = 0, has_starlist = False, has_stardict = False, ), source_ref = source_ref ) def buildStatementsNode(provider, nodes, source_ref, code_object = None): # We are not creating empty statement sequences. if nodes is None: return None # Build as list of statements, throw away empty ones, and remove useless # nesting. statements = buildNodeList(provider, nodes, source_ref, allow_none = True) statements = mergeStatements(statements) # We are not creating empty statement sequences. Might be empty, because # e.g. a global node generates not really a statement, or pass statements. if not statements: return None # In case of a frame is desired, build it instead. if code_object: if provider.isExpressionGeneratorObjectBody(): # TODO: Could do this earlier and on the outside. statements.insert( 0, StatementGeneratorEntry( source_ref = source_ref ) ) result = StatementsFrame( statements = statements, guard_mode = "generator", code_object = code_object, source_ref = source_ref ) elif provider.isExpressionCoroutineObjectBody(): # TODO: That might be wrong result = StatementsFrame( statements = statements, guard_mode = "generator", code_object = code_object, source_ref = source_ref ) elif provider.isExpressionFunctionBody() or \ provider.isExpressionClassBody(): result = StatementsFrame( statements = statements, guard_mode = "full", code_object = code_object, source_ref = source_ref ) else: result = makeModuleFrame( module = provider, statements = statements, source_ref = source_ref ) else: result = StatementsSequence( statements = statements, source_ref = source_ref ) return result def makeStatementsSequenceOrStatement(statements, source_ref): """ Make a statement sequence, but only if more than one statement Useful for when we can unroll constructs already here, but are not sure if we actually did that. This avoids the branch or the pollution of doing it always. """ if len(statements) > 1: return StatementsSequence( statements = mergeStatements(statements), source_ref = source_ref ) else: return statements[0] def makeStatementsSequence(statements, allow_none, source_ref): if allow_none: statements = tuple( statement for statement in statements if statement is not None ) if statements: return StatementsSequence( statements = mergeStatements(statements), source_ref = source_ref ) else: return None def makeStatementsSequenceFromStatement(statement): return StatementsSequence( statements = mergeStatements( (statement,) ), source_ref = statement.getSourceReference() ) def makeStatementsSequenceFromStatements(*statements): assert statements assert None not in statements statements = mergeStatements(statements, allow_none = False) return StatementsSequence( statements = statements, source_ref = statements[0].getSourceReference() ) def makeSequenceCreationOrConstant(sequence_kind, elements, source_ref): # Sequence creation. Tries to avoid creations with only constant # elements. Would be caught by optimization, but would be useless churn. For # mutable constants we cannot do it though. # Due to the many sequence types, there is a lot of cases here # pylint: disable=R0912 for element in elements: if not element.isExpressionConstantRef(): constant = False break else: constant = True sequence_kind = sequence_kind.upper() # Note: This would happen in optimization instead, but lets just do it # immediately to save some time. if constant: if sequence_kind == "TUPLE": const_type = tuple elif sequence_kind == "LIST": const_type = list elif sequence_kind == "SET": const_type = set else: assert False, sequence_kind result = ExpressionConstantRef( constant = const_type( element.getConstant() for element in elements ), source_ref = source_ref, user_provided = True ) else: if sequence_kind == "TUPLE": result = ExpressionMakeTuple( elements = elements, source_ref = source_ref ) elif sequence_kind == "LIST": result = ExpressionMakeList( elements = elements, source_ref = source_ref ) elif sequence_kind == "SET": result = ExpressionMakeSet( elements = elements, source_ref = source_ref ) else: assert False, sequence_kind if elements: result.setCompatibleSourceReference( source_ref = elements[-1].getCompatibleSourceReference() ) return result def makeDictCreationOrConstant(keys, values, source_ref): # Create dictionary node. Tries to avoid it for constant values that are not # mutable. assert len(keys) == len(values) for key, value in zip(keys, values): if not key.isExpressionConstantRef() or not key.isKnownToBeHashable(): constant = False break if not value.isExpressionConstantRef(): constant = False break else: constant = True # Note: This would happen in optimization instead, but lets just do it # immediately to save some time. if constant: # Unless told otherwise, create the dictionary in its full size, so # that no growing occurs and the constant becomes as similar as possible # before being marshaled. result = ExpressionConstantRef( constant = Constants.createConstantDict( keys = [ key.getConstant() for key in keys ], values = [ value.getConstant() for value in values ] ), user_provided = True, source_ref = source_ref ) else: result = ExpressionMakeDict( pairs = [ ExpressionKeyValuePair( key = key, value = value, source_ref = key.getSourceReference() ) for key, value in zip(keys, values) ], source_ref = source_ref ) if values: result.setCompatibleSourceReference( source_ref = values[-1].getCompatibleSourceReference() ) return result def getStatementsAppended(statement_sequence, statements): return makeStatementsSequence( statements = (statement_sequence, statements), allow_none = False, source_ref = statement_sequence.getSourceReference() ) def getStatementsPrepended(statement_sequence, statements): return makeStatementsSequence( statements = (statements, statement_sequence), allow_none = False, source_ref = statement_sequence.getSourceReference() ) def makeReraiseExceptionStatement(source_ref): return StatementsSequence( statements = ( StatementRaiseException( exception_type = None, exception_value = None, exception_trace = None, exception_cause = None, source_ref = source_ref ), ), source_ref = source_ref ) def mangleName(variable_name, owner): if not variable_name.startswith("__") or variable_name.endswith("__"): return variable_name else: # The mangling of function variable names depends on being inside a # class. class_container = owner.getContainingClassDictCreation() if class_container is None: return variable_name else: return "_%s%s" % ( class_container.getName().lstrip('_'), variable_name ) def makeConditionalStatement(condition, yes_branch, no_branch, source_ref): """ Create conditional statement, with yes_branch not being empty. May have to invert condition to achieve that. """ if yes_branch is None: condition = ExpressionOperationNOT( operand = condition, source_ref = condition.getSourceReference() ) yes_branch, no_branch = no_branch, yes_branch if not yes_branch.isStatementsSequence(): yes_branch = makeStatementsSequenceFromStatement(yes_branch) if no_branch is not None and not no_branch.isStatementsSequence(): no_branch = makeStatementsSequenceFromStatement(no_branch) return StatementConditional( condition = condition, yes_branch = yes_branch, no_branch = no_branch, source_ref = source_ref ) build_contexts = [None] def pushBuildContext(value): build_contexts.append(value) def popBuildContext(): del build_contexts[-1] def getBuildContext(): return build_contexts[-1] Nuitka-0.5.18.1/nuitka/tree/ReformulationForLoopStatements.py0000644000175000017500000002152612651071040024436 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of for loop statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionAsyncIter, ExpressionAsyncNext, ExpressionBuiltinIter1, ExpressionBuiltinNext1 ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.StatementNodes import StatementsSequence from .Helpers import ( buildNode, buildStatementsNode, makeStatementsSequence, makeStatementsSequenceFromStatements, popBuildContext, pushBuildContext ) from .ReformulationAssignmentStatements import buildAssignmentStatements from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement def _buildForLoopNode(provider, node, sync, source_ref): # The for loop is re-formulated according to developer manual. An iterator # is created, and looped until it gives StopIteration. The else block is # taken if a for loop exits normally, i.e. because of iterator # exhaustion. We do this by introducing an indicator variable. # We handle async and sync both here, leading to cases, pylint: disable=R0914 source = buildNode(provider, node.iter, source_ref) # Temporary variables, we need one for the iterator, and one for the current # value. temp_scope = provider.allocateTempScope("for_loop") tmp_iter_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "for_iterator" ) tmp_value_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "iter_value" ) else_block = buildStatementsNode( provider = provider, nodes = node.orelse if node.orelse else None, source_ref = source_ref ) if else_block is not None: tmp_break_indicator = provider.allocateTempVariable( temp_scope = temp_scope, name = "break_indicator" ) statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), source = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ) ] else: statements = [] statements.append( StatementLoopBreak( source_ref = source_ref ) ) handler_body = makeStatementsSequence( statements = statements, allow_none = False, source_ref = source_ref ) if sync: next_node = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ), source_ref = source_ref ) else: next_node = ExpressionAsyncNext( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ), source_ref = source_ref ) statements = ( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_value_variable, source_ref = source_ref ), source = next_node, source_ref = source_ref ), exception_name = "StopIteration" if sync else "StopAsyncIteration", handler_body = handler_body, source_ref = source_ref ), buildAssignmentStatements( provider = provider, node = node.target, source = ExpressionTempVariableRef( variable = tmp_value_variable, source_ref = source_ref ), source_ref = source_ref ) ) pushBuildContext("loop_body") statements += ( buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ), ) popBuildContext() loop_body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) cleanup_statements = [ StatementReleaseVariable( variable = tmp_value_variable, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = source_ref ) ] if else_block is not None: statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), source = ExpressionConstantRef( constant = False, source_ref = source_ref ), source_ref = source_ref ) ] else: statements = [] if sync: iter_source = ExpressionBuiltinIter1( value = source, source_ref = source.getSourceReference() ) else: iter_source = ExpressionAsyncIter( value = source, source_ref = source.getSourceReference() ) statements += [ # First create the iterator and store it. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ), source = iter_source, source_ref = source_ref ), makeTryFinallyStatement( provider = provider, tried = StatementLoop( body = loop_body, source_ref = source_ref ), final = StatementsSequence( statements = cleanup_statements, source_ref = source_ref ), source_ref = source_ref ) ] if else_block is not None: statements += [ StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), right = ExpressionConstantRef( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = else_block, no_branch = None, source_ref = source_ref ) ] statements = ( makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_break_indicator, source_ref = source_ref ), source_ref = source_ref ), ) return makeStatementsSequenceFromStatements( *statements ) def buildForLoopNode(provider, node, source_ref): return _buildForLoopNode(provider, node, True, source_ref) def buildAsyncForLoopNode(provider, node, source_ref): return _buildForLoopNode(provider, node, False, source_ref) Nuitka-0.5.18.1/nuitka/tree/Building.py0000644000175000017500000010760012651071040017772 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Build the internal node tree from source code. Does all the Python parsing and puts it into a tree structure for use in later stages of the compilation process. In the "nuitka.tree.Helpers" module, the dispatching is happening. One function deals with every node kind as found in the AST. The parsing is centered around the module "ast" output. Many higher level language features and translated into lower level ones. In-place assignments, for loops, while loops, classes, complex calls, with statements, and even or/and etc. are all translated to simpler constructs. The output of this module is a node tree, which contains only relatively low level operations. A property of the output is also an overlaid tree of provider structure that indicates variable provision. Classes are handled in a separate module. They are re-formulated into functions producing dictionaries used to call the metaclass with. Try/except/else statements are handled in a separate module. They are re-formulated into using a temporary variable to track if the else branch should execute. Try/finally statements are handled in a separate module. They are re-formulated to use a nested try/finally for (un)publishing the exception for Python3. With statements are handled in a separate module. They are re-formulated into special attribute lookups for "__enter__" and "__exit__", calls of them, catching and passing in exceptions raised. """ import sys from nuitka import Options, SourceCodeReferences, Tracing from nuitka.__past__ import long, unicode # pylint: disable=W0622 from nuitka.importing import Importing from nuitka.importing.ImportCache import addImportedModule from nuitka.importing.PreloadedPackages import getPthImportedPackages from nuitka.nodes.AssignNodes import ( ExpressionTargetVariableRef, StatementAssignmentVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.CoroutineNodes import ExpressionAsyncWait from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.GeneratorNodes import StatementGeneratorReturn from nuitka.nodes.ImportNodes import ( ExpressionImportModule, ExpressionImportName ) from nuitka.nodes.LoopNodes import StatementLoopBreak, StatementLoopContinue from nuitka.nodes.ModuleNodes import ( CompiledPythonModule, CompiledPythonPackage, ExpressionModuleFileAttributeRef, PythonMainModule, PythonShlibModule ) from nuitka.nodes.OperatorNodes import ( ExpressionOperationBinary, ExpressionOperationUnary ) from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.VariableRefNodes import ExpressionVariableRef from nuitka.PythonVersions import python_version from nuitka.tree.ReformulationForLoopStatements import ( buildAsyncForLoopNode, buildForLoopNode ) from nuitka.tree.ReformulationWhileLoopStatements import buildWhileLoopNode from nuitka.utils import Utils from . import SyntaxErrors from .Helpers import ( buildNode, buildStatementsNode, extractDocFromBody, getBuildContext, getKind, makeModuleFrame, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceOrStatement, mangleName, mergeStatements, parseSourceCodeToAst, setBuildingDispatchers ) from .ReformulationAssertStatements import buildAssertNode from .ReformulationAssignmentStatements import ( buildAssignNode, buildDeleteNode, buildInplaceAssignNode ) from .ReformulationBooleanExpressions import buildBoolOpNode from .ReformulationCallExpressions import buildCallNode from .ReformulationClasses import buildClassNode from .ReformulationComparisonExpressions import buildComparisonNode from .ReformulationContractionExpressions import ( buildDictContractionNode, buildGeneratorExpressionNode, buildListContractionNode, buildSetContractionNode ) from .ReformulationDictionaryCreation import buildDictionaryNode from .ReformulationExecStatements import buildExecNode from .ReformulationFunctionStatements import ( buildAsyncFunctionNode, buildFunctionNode ) from .ReformulationImportStatements import ( buildImportFromNode, checkFutureImportsOnlyAtStart ) from .ReformulationLambdaExpressions import buildLambdaNode from .ReformulationNamespacePackages import ( createNamespacePackage, createPathAssignment ) from .ReformulationPrintStatements import buildPrintNode from .ReformulationSequenceCreation import buildSequenceCreationNode from .ReformulationSubscriptExpressions import buildSubscriptNode from .ReformulationTryExceptStatements import buildTryExceptionNode from .ReformulationTryFinallyStatements import buildTryFinallyNode from .ReformulationWithStatements import buildAsyncWithNode, buildWithNode from .ReformulationYieldExpressions import buildYieldFromNode, buildYieldNode from .SourceReading import readSourceCodeFromFilename from .VariableClosure import completeVariableClosures def buildVariableReferenceNode(provider, node, source_ref): return ExpressionVariableRef( variable_name = mangleName(node.id, provider), source_ref = source_ref ) # Python3.4 or higher, True and False, are not given as variables anymore. def buildNamedConstantNode(node, source_ref): return ExpressionConstantRef( constant = node.value, source_ref = source_ref ) def buildConditionNode(provider, node, source_ref): # Conditional statements may have one or two branches. We will never see an # "elif", because that's already dealt with by module "ast", which turns it # into nested conditional statements. return StatementConditional( condition = buildNode(provider, node.test, source_ref), yes_branch = buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ), no_branch = buildStatementsNode( provider = provider, nodes = node.orelse if node.orelse else None, source_ref = source_ref ), source_ref = source_ref ) def _buildTryFinallyNode(provider, node, source_ref): # Try/finally node statements of old style. return buildTryFinallyNode( provider = provider, build_tried = lambda : buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ), node = node, source_ref = source_ref ) def buildTryNode(provider, node, source_ref): # Note: This variant is used for Python3.3 or higher only, older stuff uses # the above ones, this one merges try/except with try/finally in the # "ast". We split it up again, as it's logically separated of course. # Shortcut missing try/finally. if not node.handlers: return _buildTryFinallyNode(provider, node, source_ref) if not node.finalbody: return buildTryExceptionNode( provider = provider, node = node, source_ref = source_ref ) return buildTryFinallyNode( provider = provider, build_tried = lambda : makeStatementsSequence( statements = mergeStatements( ( buildTryExceptionNode( provider = provider, node = node, source_ref = source_ref ), ), allow_none = True ), allow_none = True, source_ref = source_ref ), node = node, source_ref = source_ref ) def buildRaiseNode(provider, node, source_ref): # Raise statements. Under Python2 they may have type, value and traceback # attached, for Python3, you can only give type (actually value) and cause. if python_version < 300: exception_type = buildNode(provider, node.type, source_ref, allow_none = True) exception_value = buildNode(provider, node.inst, source_ref, allow_none = True) exception_trace = buildNode(provider, node.tback, source_ref, allow_none = True) exception_cause = None else: exception_type = buildNode(provider, node.exc, source_ref, allow_none = True) exception_value = None exception_trace = None exception_cause = buildNode(provider, node.cause, source_ref, allow_none = True) result = StatementRaiseException( exception_type = exception_type, exception_value = exception_value, exception_trace = exception_trace, exception_cause = exception_cause, source_ref = source_ref ) if exception_cause is not None: result.setCompatibleSourceReference( source_ref = exception_cause.getCompatibleSourceReference() ) elif exception_trace is not None: result.setCompatibleSourceReference( source_ref = exception_trace.getCompatibleSourceReference() ) elif exception_value is not None: result.setCompatibleSourceReference( source_ref = exception_value.getCompatibleSourceReference() ) elif exception_type is not None: result.setCompatibleSourceReference( source_ref = exception_type.getCompatibleSourceReference() ) return result def buildImportModulesNode(provider, node, source_ref): # Import modules statement. As described in the developer manual, these # statements can be treated as several ones. import_names = [ ( import_desc.name, import_desc.asname ) for import_desc in node.names ] import_nodes = [] for import_desc in import_names: module_name, local_name = import_desc module_topname = module_name.split('.')[0] # Note: The "level" of import is influenced by the future absolute # imports. level = 0 if source_ref.getFutureSpec().isAbsoluteImport() else -1 if local_name: # If is gets a local name, the real name must be used as a # temporary value only, being looked up recursively. import_node = ExpressionImportModule( module_name = module_name, import_list = None, level = level, source_ref = source_ref ) for import_name in module_name.split('.')[1:]: import_node = ExpressionImportName( module = import_node, import_name = import_name, source_ref = source_ref ) else: import_node = ExpressionImportModule( module_name = module_name, import_list = None, level = level, source_ref = source_ref ) # If a name was given, use the one provided, otherwise the import gives # the top level package name given for assignment of the imported # module. import_nodes.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = mangleName( local_name if local_name is not None else module_topname, provider ), source_ref = source_ref ), source = import_node, source_ref = source_ref ) ) # Note: Each import is sequential. It will potentially succeed, and the # failure of a later one is not changing that one bit . We can therefore # have a sequence of imports that only import one thing therefore. return makeStatementsSequenceOrStatement( statements = import_nodes, source_ref = source_ref ) def handleGlobalDeclarationNode(provider, node, source_ref): # On the module level, there is nothing to do. TODO: Probably a warning # would be warranted. if provider.isCompiledPythonModule(): return None # Need to catch the error of declaring a parameter variable as global # ourselves here. The AST parsing doesn't catch it, so we check here. if provider.isExpressionFunctionBody(): parameters = provider.getParameters() for variable_name in node.names: if variable_name in parameters.getParameterNames(): SyntaxErrors.raiseSyntaxError( reason = "name '%s' is %s and global" % ( variable_name, "local" if python_version < 300 else "parameter" ), source_ref = ( source_ref if not Options.isFullCompat() or \ python_version >= 340 else provider.getSourceReference() ) ) # The module the "global" statement refers to. module = provider.getParentModule() # Can give multiple names. for variable_name in node.names: closure_variable = None # Re-use already taken global variables, in order to avoid creating yet # another instance, esp. as the indications could then potentially not # be shared. if provider.hasTakenVariable(variable_name): closure_variable = provider.getTakenVariable(variable_name) # Only global variables count. Could have a closure reference to # a location of a parent function here. if not closure_variable.isModuleVariable(): closure_variable = None if closure_variable is None: module_variable = module.getVariableForAssignment( variable_name = variable_name ) closure_variable = provider.addClosureVariable( variable = module_variable ) assert closure_variable.isModuleVariable() if python_version < 340 and \ provider.isExpressionClassBody() and \ closure_variable.getName() == "__class__": SyntaxErrors.raiseSyntaxError( reason = "cannot make __class__ global", source_ref = source_ref ) provider.registerProvidedVariable( variable = closure_variable ) return None def handleNonlocalDeclarationNode(provider, node, source_ref): # Need to catch the error of declaring a parameter variable as global # ourselves here. The AST parsing doesn't catch it, but we can do it here. parameter_provider = provider while parameter_provider.isExpressionGeneratorObjectBody() or \ parameter_provider.isExpressionCoroutineObjectBody(): parameter_provider = parameter_provider.getParentVariableProvider() if parameter_provider.isExpressionClassBody(): parameter_names = () else: parameter_names = parameter_provider.getParameters().getParameterNames() for variable_name in node.names: if variable_name in parameter_names: SyntaxErrors.raiseSyntaxError( reason = "name '%s' is parameter and nonlocal" % ( variable_name ), source_ref = None if Options.isFullCompat() and \ python_version < 340 else source_ref, display_file = not Options.isFullCompat() or \ python_version >= 340, display_line = not Options.isFullCompat() or \ python_version >= 340 ) provider.addNonlocalsDeclaration(node.names, source_ref) return None def buildStringNode(node, source_ref): assert type(node.s) in (str, unicode) return ExpressionConstantRef( constant = node.s, source_ref = source_ref, user_provided = True ) def buildNumberNode(node, source_ref): assert type(node.n) in (int, long, float, complex), type(node.n) return ExpressionConstantRef( constant = node.n, source_ref = source_ref, user_provided = True ) def buildBytesNode(node, source_ref): return ExpressionConstantRef( constant = node.s, source_ref = source_ref, user_provided = True ) def buildEllipsisNode(source_ref): return ExpressionConstantRef( constant = Ellipsis, source_ref = source_ref, user_provided = True ) def buildStatementLoopContinue(node, source_ref): # Python forbids this, although technically it's probably not much of # an issue. if getBuildContext() == "finally": if not Options.isFullCompat() or python_version >= 300: col_offset = node.col_offset - 9 else: col_offset = None if python_version >= 300 and Options.isFullCompat(): source_line = "" else: source_line = None SyntaxErrors.raiseSyntaxError( "'continue' not supported inside 'finally' clause", source_ref, col_offset = col_offset, source_line = source_line ) return StatementLoopContinue( source_ref = source_ref ) def buildStatementLoopBreak(provider, node, source_ref): # A bit unusual, we need the provider, but not the node, # pylint: disable=W0613 return StatementLoopBreak( source_ref = source_ref ) def buildAttributeNode(provider, node, source_ref): return ExpressionAttributeLookup( source = buildNode(provider, node.value, source_ref), attribute_name = node.attr, source_ref = source_ref ) def buildReturnNode(provider, node, source_ref): if provider.isExpressionClassBody() or provider.isCompiledPythonModule(): SyntaxErrors.raiseSyntaxError( "'return' outside function", source_ref, None if python_version < 300 else ( node.col_offset if provider.isCompiledPythonModule() else node.col_offset+4 ) ) expression = buildNode(provider, node.value, source_ref, allow_none = True) if provider.isExpressionGeneratorObjectBody(): if expression is not None and python_version < 330: SyntaxErrors.raiseSyntaxError( "'return' with argument inside generator", source_ref = source_ref, ) if expression is None: expression = ExpressionConstantRef( constant = None, source_ref = source_ref, user_provided = True ) if provider.isExpressionGeneratorObjectBody(): return StatementGeneratorReturn( expression = expression, source_ref = source_ref ) else: return StatementReturn( expression = expression, source_ref = source_ref ) def buildExprOnlyNode(provider, node, source_ref): result = StatementExpressionOnly( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) result.setCompatibleSourceReference( result.getExpression().getCompatibleSourceReference() ) return result def buildUnaryOpNode(provider, node, source_ref): if getKind(node.op) == "Not": return buildBoolOpNode( provider = provider, node = node, source_ref = source_ref ) else: return ExpressionOperationUnary( operator = getKind(node.op), operand = buildNode(provider, node.operand, source_ref), source_ref = source_ref ) def buildBinaryOpNode(provider, node, source_ref): operator = getKind(node.op) if operator == "Div" and source_ref.getFutureSpec().isFutureDivision(): operator = "TrueDiv" left = buildNode(provider, node.left, source_ref) right = buildNode(provider, node.right, source_ref) result = ExpressionOperationBinary( operator = operator, left = left, right = right, source_ref = source_ref ) result.setCompatibleSourceReference( source_ref = right.getCompatibleSourceReference() ) return result def buildReprNode(provider, node, source_ref): return ExpressionOperationUnary( operator = "Repr", operand = buildNode(provider, node.value, source_ref), source_ref = source_ref ) def buildConditionalExpressionNode(provider, node, source_ref): return ExpressionConditional( condition = buildNode(provider, node.test, source_ref), expression_yes = buildNode(provider, node.body, source_ref), expression_no = buildNode(provider, node.orelse, source_ref), source_ref = source_ref ) def buildAwaitNode(provider, node, source_ref): return ExpressionAsyncWait( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) setBuildingDispatchers( path_args3 = { "Name" : buildVariableReferenceNode, "Assign" : buildAssignNode, "Delete" : buildDeleteNode, "Lambda" : buildLambdaNode, "GeneratorExp" : buildGeneratorExpressionNode, "If" : buildConditionNode, "While" : buildWhileLoopNode, "For" : buildForLoopNode, "AsyncFor" : buildAsyncForLoopNode, "Compare" : buildComparisonNode, "ListComp" : buildListContractionNode, "DictComp" : buildDictContractionNode, "SetComp" : buildSetContractionNode, "Dict" : buildDictionaryNode, "Set" : buildSequenceCreationNode, "Tuple" : buildSequenceCreationNode, "List" : buildSequenceCreationNode, "Global" : handleGlobalDeclarationNode, "Nonlocal" : handleNonlocalDeclarationNode, "TryExcept" : buildTryExceptionNode, "TryFinally" : _buildTryFinallyNode, "Try" : buildTryNode, "Raise" : buildRaiseNode, "Import" : buildImportModulesNode, "ImportFrom" : buildImportFromNode, "Assert" : buildAssertNode, "Exec" : buildExecNode, "With" : buildWithNode, "AsyncWith" : buildAsyncWithNode, "FunctionDef" : buildFunctionNode, "AsyncFunctionDef" : buildAsyncFunctionNode, "Await" : buildAwaitNode, "ClassDef" : buildClassNode, "Print" : buildPrintNode, "Call" : buildCallNode, "Subscript" : buildSubscriptNode, "BoolOp" : buildBoolOpNode, "Attribute" : buildAttributeNode, "Return" : buildReturnNode, "Yield" : buildYieldNode, "YieldFrom" : buildYieldFromNode, "Expr" : buildExprOnlyNode, "UnaryOp" : buildUnaryOpNode, "BinOp" : buildBinaryOpNode, "Repr" : buildReprNode, "AugAssign" : buildInplaceAssignNode, "IfExp" : buildConditionalExpressionNode, "Break" : buildStatementLoopBreak, }, path_args2 = { "NameConstant" : buildNamedConstantNode, "Str" : buildStringNode, "Num" : buildNumberNode, "Bytes" : buildBytesNode, "Continue" : buildStatementLoopContinue, }, path_args1 = { "Ellipsis" : buildEllipsisNode, } ) def buildParseTree(provider, source_code, source_ref, is_module, is_main): # There are a bunch of branches here, mostly to deal with version # differences for module default variables. body = parseSourceCodeToAst( source_code = source_code, filename = source_ref.getFilename(), line_offset = source_ref.getLineNumber() - 1 ) body, doc = extractDocFromBody(body) result = buildStatementsNode( provider = provider, nodes = body, source_ref = source_ref ) checkFutureImportsOnlyAtStart(body) internal_source_ref = source_ref.atInternal() statements = [] if is_module: # Add import of "site" module of main programs visibly in the node tree, # so recursion and optimization can pick it up, checking its effects. if is_main and "no_site" not in Options.getPythonFlags(): for path_imported_name in getPthImportedPackages(): statements.append( StatementExpressionOnly( expression = ExpressionImportModule( module_name = path_imported_name, import_list = (), level = 0, source_ref = source_ref, ), source_ref = source_ref ) ) statements.append( StatementExpressionOnly( expression = ExpressionImportModule( module_name = "site", import_list = (), level = 0, source_ref = source_ref, ), source_ref = source_ref ) ) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__doc__", source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = doc, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__file__", source_ref = internal_source_ref ), source = ExpressionModuleFileAttributeRef( source_ref = internal_source_ref, ), source_ref = internal_source_ref ) ) if provider.isCompiledPythonPackage(): # This assigns "__path__" value. statements.append( createPathAssignment(internal_source_ref) ) if python_version >= 300: statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__cached__", source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = None, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) if python_version >= 330: # For Python3.3, it's set for both packages and non-packages. statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__package__", source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = provider.getFullName() if provider.isCompiledPythonPackage() else provider.getPackage(), source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) needs__initializing__ = not provider.isMainModule() and \ (python_version >= 330 and python_version < 340) if needs__initializing__: # Set "__initializing__" at the beginning to True statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__initializing__", source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = True, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) # Now the module body if there is any at all. if result is not None: statements.extend( result.getStatements() ) if needs__initializing__: # Set "__initializing__" at the end to False statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = "__initializing__", source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = False, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) if is_module: return makeModuleFrame( module = provider, statements = statements, source_ref = source_ref ) else: assert False def decideModuleTree(filename, package, is_shlib, is_top, is_main): # Many variables, branches, due to the many cases, pylint: disable=R0912 assert package is None or type(package) is str assert filename is not None if is_main and Utils.isDir(filename): source_filename = Utils.joinpath(filename, "__main__.py") if not Utils.isFile(source_filename): sys.stderr.write( "%s: can't find '__main__' module in '%s'\n" % ( Utils.basename(sys.argv[0]), filename ) ) sys.exit(2) filename = source_filename main_added = True else: main_added = False if Utils.isFile(filename): source_filename = filename source_ref = SourceCodeReferences.fromFilename( filename = filename, ) if is_main: module_name = "__main__" else: module_name = Utils.basename(filename) if module_name.endswith(".py"): module_name = module_name[:-3] if is_shlib: module_name = module_name.split('.')[0] if '.' in module_name: sys.stderr.write( "Error, '%s' is not a proper python module name.\n" % ( module_name ) ) sys.exit(2) if is_shlib: result = PythonShlibModule( name = module_name, package_name = package, source_ref = source_ref ) elif is_main: result = PythonMainModule( main_added = main_added, source_ref = source_ref ) else: result = CompiledPythonModule( name = module_name, package_name = package, source_ref = source_ref ) elif Importing.isPackageDir(filename): if is_top: package_name = Utils.splitpath(filename)[-1] else: package_name = Utils.basename(filename) source_filename = Utils.joinpath(filename, "__init__.py") if not Utils.isFile(source_filename): source_ref, result = createNamespacePackage( package_name = package_name, module_relpath = filename ) source_filename = None else: source_ref = SourceCodeReferences.fromFilename( filename = Utils.abspath(source_filename), ) result = CompiledPythonPackage( name = package_name, package_name = package, source_ref = source_ref ) else: sys.stderr.write( "%s: can't open file '%s'.\n" % ( Utils.basename(sys.argv[0]), filename ) ) sys.exit(2) if not Options.shallHaveStatementLines(): source_ref = source_ref.atInternal() return result, source_ref, source_filename class CodeTooComplexCode(Exception): """ The code of the module is too complex. It cannot be compiled, with recursive code, and therefore the bytecode should be used instead. Example of this is "idnadata". """ pass def createModuleTree(module, source_ref, source_code, is_main): if Options.isShowProgress(): memory_watch = Utils.MemoryWatch() try: module_body = buildParseTree( provider = module, source_code = source_code, source_ref = source_ref, is_module = True, is_main = is_main ) except RuntimeError as e: if "maximum recursion depth" in e.args[0]: raise CodeTooComplexCode( module.getFullName(), module.getCompileTimeFilename() ) raise if module_body.isStatementsFrame(): module_body = makeStatementsSequenceFromStatement( statement = module_body, ) module.setBody(module_body) completeVariableClosures(module) if Options.isShowProgress(): memory_watch.finish() Tracing.printLine( "Memory usage changed loading module '%s': %s" % ( module.getFullName(), memory_watch.asStr() ) ) def buildModuleTree(filename, package, is_top, is_main): module, source_ref, source_filename = decideModuleTree( filename = filename, package = package, is_top = is_top, is_main = is_main, is_shlib = False ) # If there is source code associated (not the case for namespace packages of # Python3.3 or higher, then read it. if source_filename is not None: # Read source code. createModuleTree( module = module, source_ref = source_ref, source_code = readSourceCodeFromFilename(module.getFullName(), source_filename), is_main = is_main ) if not module.isMainModule(): addImportedModule( imported_module = module ) return module Nuitka-0.5.18.1/nuitka/tree/ReformulationSubscriptExpressions.py0000644000175000017500000001120112651071040025214 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of subscript into slicing. For Python2, there is a difference between x[a], x[a:b], x[a:b:c] whereas Python3 treats the later by making a slice object, Python2 tries to have special slice access, if available, or building a slice object only at the end. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.SliceNodes import ( ExpressionBuiltinSlice, ExpressionSliceLookup ) from nuitka.nodes.SubscriptNodes import ExpressionSubscriptLookup from nuitka.PythonVersions import python_version from .Helpers import buildNode, getKind from .ReformulationAssignmentStatements import buildExtSliceNode def buildSubscriptNode(provider, node, source_ref): # Subscript expression nodes. assert getKind(node.ctx) == "Load", source_ref # The subscript "[]" operator is one of many different things. This is # expressed by this kind, there are "slice" lookups (two values, even if one # is using default), and then "index" lookups. The form with three argument # is really an "index" lookup, with a slice object. And the "..." lookup is # also an index loop-up, with it as the argument. So this splits things into # two different operations, "subscript" with a single "subscript" object. Or # a slice lookup with a lower and higher boundary. These things should # behave similar, but they are different slots. kind = getKind(node.slice) if kind == "Index": return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = buildNode(provider, node.slice.value, source_ref), source_ref = source_ref ) elif kind == "Slice": lower = buildNode( provider = provider, node = node.slice.lower, source_ref = source_ref, allow_none = True ) upper = buildNode( provider = provider, node = node.slice.upper, source_ref = source_ref, allow_none = True ) step = buildNode( provider = provider, node = node.slice.step, source_ref = source_ref, allow_none = True ) # For Python3 there is no slicing operation, this is always done # with subscript using a slice object. For Python2, it is only done # if no "step" is provided. use_sliceobj = step is not None or python_version >= 300 if use_sliceobj: return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = ExpressionBuiltinSlice( start = lower, stop = upper, step = step, source_ref = source_ref ), source_ref = source_ref ) else: return ExpressionSliceLookup( expression = buildNode(provider, node.value, source_ref), lower = lower, upper = upper, source_ref = source_ref ) elif kind == "ExtSlice": return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = buildExtSliceNode(provider, node, source_ref), source_ref = source_ref ) elif kind == "Ellipsis": return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = ExpressionConstantRef( constant = Ellipsis, source_ref = source_ref ), source_ref = source_ref ) else: assert False, kind Nuitka-0.5.18.1/nuitka/tree/ComplexCallHelperFunctions.py0000644000175000017500000031226212651072046023502 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module is providing helper functions for complex call re-formulations. One for each type of call. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTargetVariableRef, StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.BuiltinDictNodes import ExpressionBuiltinDict from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionBuiltinNext1 ) from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, ExpressionBuiltinRef ) from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinTuple from nuitka.nodes.CallNodes import ( ExpressionCall, ExpressionCallEmpty, ExpressionCallKeywordsOnly, ExpressionCallNoKeywords ) from nuitka.nodes.ComparisonNodes import ExpressionComparison from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ContainerOperationNodes import ExpressionDictOperationSet from nuitka.nodes.ExceptionNodes import ( ExpressionBuiltinMakeException, StatementRaiseException ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.OperatorNodes import ( ExpressionOperationBinary, ExpressionOperationNOT ) from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.SubscriptNodes import ( ExpressionSubscriptLookup, StatementAssignmentSubscript ) from nuitka.nodes.TypeNodes import ( ExpressionBuiltinIsinstance, ExpressionBuiltinType1 ) from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .Helpers import ( makeConditionalStatement, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) from .InternalModule import ( getInternalModule, internal_source_ref, once_decorator ) from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement def orderArgs(*args): if python_version >= 350: def weight(arg): result = args.index(arg) if arg == "kw": result += 1.5 elif arg == "star_arg_list": result -= 1.5 return result return tuple( sorted(args, key = weight) ) return args @once_decorator def getCallableNameDescBody(): helper_name = "get_callable_name_desc" # Equivalent of: # # Note: The "called_type" is a temporary variable. # # called_type = type( BuiltinFunctionType ) # # if ininstance( called, (FunctionType, MethodType, BuiltinFunctionType) ): # return called.__name__ # elif python_version < 3 and isinstance( called, ClassType ): # return called_type.__name__ + " constructor" # elif python_version < 3 and isinstance( called, InstanceType ): # return called_type.__name__ + " instance" # else: # return called_type.__name__ + " object" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) def makeNameAttributeLookup(node, attribute_name = "__name__"): return ExpressionAttributeLookup( source = node, attribute_name = attribute_name, source_ref = internal_source_ref ) functions_case = makeStatementsSequenceFromStatement( statement = ( StatementReturn( expression = ExpressionOperationBinary( operator = "Add", right = ExpressionConstantRef( constant = "()", source_ref = internal_source_ref, user_provided = True ), left = makeNameAttributeLookup( ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) ) no_branch = StatementReturn( expression = ExpressionOperationBinary( operator = "Add", right = ExpressionConstantRef( constant = " object", source_ref = internal_source_ref, user_provided = True ), left = makeNameAttributeLookup( ExpressionBuiltinType1( value = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) if python_version < 300: instance_case = StatementReturn( expression = ExpressionOperationBinary( operator = "Add", right = ExpressionConstantRef( constant = " instance", source_ref = internal_source_ref, user_provided = True ), left = makeNameAttributeLookup( makeNameAttributeLookup( ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), attribute_name = "__class__", ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) no_branch = makeConditionalStatement( condition = ExpressionBuiltinIsinstance( instance = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "instance", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = instance_case, no_branch = no_branch, source_ref = internal_source_ref ) class_case = StatementReturn( expression = ExpressionOperationBinary( operator = "Add", right = ExpressionConstantRef( constant = " constructor", source_ref = internal_source_ref, user_provided = True ), left = makeNameAttributeLookup( ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) no_branch = makeConditionalStatement( condition = ExpressionBuiltinIsinstance( instance = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "classobj", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = class_case, no_branch = no_branch, source_ref = internal_source_ref ) if python_version < 300: normal_cases = ( "function", "builtin_function_or_method", "instancemethod" ) else: normal_cases = ( "function", "builtin_function_or_method" ) result.setBody( makeStatementsSequenceFromStatement( statement = makeConditionalStatement( condition = ExpressionBuiltinIsinstance( instance = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), classes = ExpressionMakeTuple( elements = tuple( ExpressionBuiltinAnonymousRef( builtin_name = builtin_name, source_ref = internal_source_ref ) for builtin_name in normal_cases ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = functions_case, no_branch = no_branch, source_ref = internal_source_ref ) ) ) return result def makeStarListArgumentErrorRaise(called_variable_ref, star_list_variable_ref): return StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( ExpressionOperationBinary( operator = "Mod", left = ExpressionConstantRef( constant = """\ %s argument after * must be a sequence, not %s""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getCallableNameDescBody(), source_ref = internal_source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = internal_source_ref ), values = ( called_variable_ref, ), source_ref = internal_source_ref ), ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = star_list_variable_ref.makeClone(), source_ref = internal_source_ref ), attribute_name = "__name__", source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ) def _makeStarListArgumentToTupleStatement(called_variable_ref, star_list_target_variable_ref, star_list_variable_ref): return makeConditionalStatement( condition = ExpressionOperationNOT( operand = ExpressionBuiltinIsinstance( instance = star_list_variable_ref.makeClone(), classes = ExpressionBuiltinRef( builtin_name = "tuple", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = star_list_target_variable_ref.makeClone(), source = ExpressionBuiltinTuple( value = star_list_variable_ref.makeClone(), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "TypeError", handler_body = makeStarListArgumentErrorRaise( called_variable_ref = called_variable_ref, star_list_variable_ref = star_list_variable_ref ), source_ref = internal_source_ref ), no_branch = None, source_ref = internal_source_ref ) def _makeRaiseExceptionMustBeMapping(called_variable_ref, star_dict_variable_ref): return StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( ExpressionOperationBinary( operator = "Mod", left = ExpressionConstantRef( constant = """\ %s argument after ** must be a mapping, not %s""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getCallableNameDescBody(), source_ref = internal_source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = internal_source_ref ), values = ( called_variable_ref.makeClone(), ), source_ref = internal_source_ref ), ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = ExpressionVariableRef( variable_name = star_dict_variable_ref.getVariableName(), variable = star_dict_variable_ref.getVariable(), source_ref = internal_source_ref ), source_ref = internal_source_ref ), attribute_name = "__name__", source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ) def _makeIteratingLoopStatement(tmp_iter_variable, tmp_item_variable, statements): loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), *statements ) return StatementLoop( body = loop_body, source_ref = internal_source_ref ) def _makeStarDictArgumentToDictStatement(result, called_variable_ref, star_dict_target_variable_ref, star_dict_variable_ref): temp_scope = result.allocateTempScope("mapping") tmp_dict_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_keys_variable = result.allocateTempVariable(temp_scope, "keys") tmp_key_variable = result.allocateTempVariable(temp_scope, "key") loop_body = ( StatementAssignmentSubscript( expression = ExpressionTempVariableRef( variable = tmp_dict_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = star_dict_variable_ref.makeClone(), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) mapping_case = makeStatementsSequenceFromStatements( # Initializing the temp variable outside of try/except, because code # generation does not yet detect that case properly. TODO: Can be # removed once code generation is apt enough. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = None, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = star_dict_variable_ref.makeClone(), attribute_name = "keys", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "AttributeError", handler_body = _makeRaiseExceptionMustBeMapping( called_variable_ref = called_variable_ref, star_dict_variable_ref = star_dict_target_variable_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_dict_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = {}, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_key_variable, statements = loop_body ), StatementAssignmentVariable( variable_ref = star_dict_target_variable_ref.makeClone(), source = ExpressionTempVariableRef( variable = tmp_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) tried = StatementConditional( condition = ExpressionOperationNOT( operand = ExpressionBuiltinIsinstance( instance = star_dict_variable_ref.makeClone(), classes = ExpressionBuiltinRef( builtin_name = "dict", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = mapping_case, no_branch = None, source_ref = internal_source_ref ) final = ( StatementReleaseVariable( variable = tmp_dict_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_keys_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref ) ) return makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) def _makeRaiseDuplicationItem(called_variable, tmp_key_variable): return StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( ExpressionOperationBinary( operator = "Mod", left = ExpressionConstantRef( constant = """\ %s got multiple values for keyword argument '%s'""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getCallableNameDescBody( ), source_ref = internal_source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = internal_source_ref ), values = ( ExpressionVariableRef( variable = called_variable, variable_name = called_variable.getName(), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ) def _makeStarDictArgumentMergeToKwStatement(result, called_variable_ref, kw_target_variable_ref, kw_variable_ref, star_dict_variable_ref): # This is plain terribly complex, pylint: disable=R0914 temp_scope = result.allocateTempScope("dict") tmp_dict_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_keys_variable = result.allocateTempVariable(temp_scope, "keys") tmp_key_variable = result.allocateTempVariable(temp_scope, "key_xxx") final = [ StatementReleaseVariable( variable = tmp_dict_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_keys_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref, ) ] mapping_loop_body = ( StatementConditional( condition = ExpressionComparison( comparator = "In", left = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), right = kw_variable_ref.makeClone(), source_ref = internal_source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = _makeRaiseDuplicationItem( called_variable = called_variable_ref.getVariable(), tmp_key_variable = tmp_key_variable ) ), no_branch = None, source_ref = internal_source_ref ), StatementAssignmentSubscript( expression = kw_variable_ref.makeClone(), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = star_dict_variable_ref.makeClone(), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) mapping_case = makeStatementsSequenceFromStatements( # Initializing the temp variable outside of try/except, because code # generation does not yet detect that case properly. TODO: Can be # removed once code generation is apt enough. StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = None, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = star_dict_variable_ref.makeClone(), attribute_name = "keys", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "AttributeError", handler_body = _makeRaiseExceptionMustBeMapping( called_variable_ref = called_variable_ref, star_dict_variable_ref = star_dict_variable_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_dict_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = {}, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_key_variable, statements = mapping_loop_body ) ) temp_scope = result.allocateTempScope("dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "item") tmp_key_variable = result.allocateTempVariable(temp_scope, "key") final += [ StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref, ) ] dict_loop_body = ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), subscript = ExpressionConstantRef( constant = 0, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementConditional( condition = ExpressionComparison( comparator = "In", left = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), right = kw_variable_ref.makeClone(), source_ref = internal_source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = _makeRaiseDuplicationItem( called_variable = called_variable_ref.getVariable(), tmp_key_variable = tmp_key_variable ) ), no_branch = None, source_ref = internal_source_ref ), StatementAssignmentSubscript( expression = kw_variable_ref.makeClone(), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), subscript = ExpressionConstantRef( constant = 1, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) dict_case = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = kw_target_variable_ref.makeClone(), source = ExpressionBuiltinDict( pos_arg = kw_variable_ref.makeClone(), pairs = (), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = star_dict_variable_ref.makeClone(), attribute_name = "iteritems" if python_version < 300 else "items", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_item_variable, statements = dict_loop_body, ), ) dict_case = makeStatementsSequenceFromStatement( statement = StatementConditional( condition = star_dict_variable_ref.makeClone(), yes_branch = dict_case, no_branch = None, source_ref = internal_source_ref ) ) tried = StatementConditional( condition = ExpressionOperationNOT( operand = ExpressionBuiltinIsinstance( instance = star_dict_variable_ref.makeClone(), classes = ExpressionBuiltinRef( builtin_name = "dict", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = mapping_case, no_branch = dict_case, source_ref = internal_source_ref ) return makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) @once_decorator def getFunctionCallHelperStarList(): helper_name = "complex_call_helper_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "star_arg_list" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCallNoKeywords( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperKeywordsStarList(): helper_name = "complex_call_helper_keywords_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = orderArgs( "called", "kw", "star_arg_list" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosStarList(): helper_name = "complex_call_helper_pos_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "args", "star_arg_list" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCallNoKeywords( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionOperationBinary( operator = "Add", left = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosKeywordsStarList(): helper_name = "complex_call_helper_pos_keywords_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = orderArgs( "called", "args", "kw", "star_arg_list" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionOperationBinary( operator = "Add", left = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperStarDict(): helper_name = "complex_call_helper_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # tmp_dict[ tmp_key ] = star_dict_arg[ tmp_key ) # # star_arg_dict = new # # return called( **star_arg_dict ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentToDictStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), star_dict_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCallKeywordsOnly( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosStarDict(): helper_name = "complex_call_helper_pos_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # tmp_dict[ tmp_key ] = star_dict_arg[ tmp_key ) # # star_arg_dict = new # # return called( args, **star_arg_dict ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "args", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentToDictStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), star_dict_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperKeywordsStarDict(): helper_name = "complex_call_helper_keywords_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. One goal is to avoid copying "kw" unless really # necessary, and to take the slow route only for non-dictionaries. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # if keys: # kw = dict( kw ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = star_dict_arg[ tmp_key ) # # elif star_arg_dict: # tmp_iter = star_arg_dict.iteritems() # # kw = dict( kw ) # while 1: # try: # tmp_key, tmp_value = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = tmp_value # # return called( **kw ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "kw", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), kw_variable_ref = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), kw_target_variable_ref = ExpressionTargetVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCallKeywordsOnly( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosKeywordsStarDict(): helper_name = "complex_call_helper_pos_keywords_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. One goal is to avoid copying "kw" unless really # necessary, and to take the slow route only for non-dictionaries. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # if keys: # kw = dict( kw ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = star_dict_arg[ tmp_key ) # # elif star_arg_dict: # tmp_iter = star_arg_dict.iteritems() # # kw = dict( kw ) # while 1: # try: # tmp_key, tmp_value = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = tmp_value # # return called( **kw ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "args", "kw", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), kw_variable_ref = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), kw_target_variable_ref = ExpressionTargetVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperStarListStarDict(): helper_name = "complex_call_helper_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "star_arg_list", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentToDictStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), star_dict_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosStarListStarDict(): helper_name = "complex_call_helper_pos_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", "args", "star_arg_list", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentToDictStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), star_dict_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionOperationBinary( operator = "Add", left = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperKeywordsStarListStarDict(): helper_name = "complex_call_helper_keywords_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = orderArgs( "called", "kw", "star_arg_list", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), kw_variable_ref = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), kw_target_variable_ref = ExpressionTargetVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosKeywordsStarListStarDict(): helper_name = "complex_call_helper_pos_keywords_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = orderArgs( "called", "args", "kw", "star_arg_list", "star_arg_dict" ), list_star_arg = None, dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), kw_variable_ref = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), kw_target_variable_ref = ExpressionTargetVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), star_dict_variable_ref = ExpressionVariableRef( variable_name = "star_arg_dict", variable = star_arg_dict_variable, source_ref = internal_source_ref ) ), _makeStarListArgumentToTupleStatement( called_variable_ref = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), star_list_variable_ref = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), star_list_target_variable_ref = ExpressionTargetVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ) ), StatementReturn( expression = ExpressionCall( called = ExpressionVariableRef( variable_name = "called", variable = called_variable, source_ref = internal_source_ref ), args = ExpressionOperationBinary( operator = "Add", left = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable_name = "star_arg_list", variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable_name = "kw", variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperDictionaryUnpacking(): helper_name = "complex_call_helper_dict_unpacking_checks" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( name = helper_name, normal_args = ( "called", ), list_star_arg = "args", dict_star_arg = None, default_count = 0, kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) args_variable = result.getVariableForAssignment( variable_name = "args" ) called_variable = result.getVariableForAssignment( variable_name = "called" ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "dicts_iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "args_item") tmp_iter2_variable = result.allocateTempVariable(temp_scope, "dict_iter") tmp_key_variable = result.allocateTempVariable(temp_scope, "dict_key") update_body = ( StatementConditional( condition = ExpressionComparison( comparator = "In", left = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), right = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = _makeRaiseDuplicationItem( called_variable = called_variable, tmp_key_variable = tmp_key_variable ) ), no_branch = None, source_ref = internal_source_ref ), StatementExpressionOnly( expression = ExpressionDictOperationSet( dict_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), key = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), value = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) loop_body = ( makeTryExceptSingleHandlerNode( tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter2_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionCallEmpty( called = ExpressionAttributeLookup( ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), attribute_name = "keys", source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter2_variable, tmp_item_variable = tmp_key_variable, statements = update_body ) ), exception_name = "AttributeError", handler_body = StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( ExpressionOperationBinary( operator = "Mod", left = ExpressionConstantRef( constant = """\ '%s' object is not a mapping""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), attribute_name = "__name__", source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter2_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable_name = "args", variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source = ExpressionConstantRef( constant = {}, source_ref = internal_source_ref ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_item_variable, statements = loop_body ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result Nuitka-0.5.18.1/nuitka/tree/ReformulationAssignmentStatements.py0000644000175000017500000011036412651071040025165 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of assignment statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTargetVariableRef, StatementAssignmentVariable, StatementDelVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, StatementAssignmentAttribute, StatementDelAttribute ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionSpecialUnpack, StatementSpecialUnpackCheck ) from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinList from nuitka.nodes.ComparisonNodes import ExpressionComparisonIsNOT from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ContainerOperationNodes import ExpressionListOperationPop from nuitka.nodes.OperatorNodes import ExpressionOperationBinaryInplace from nuitka.nodes.SliceNodes import ( ExpressionBuiltinSlice, ExpressionSliceLookup, StatementAssignmentSlice, StatementDelSlice ) from nuitka.nodes.SubscriptNodes import ( ExpressionSubscriptLookup, StatementAssignmentSubscript, StatementDelSubscript ) from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .Helpers import ( buildNode, getKind, makeSequenceCreationOrConstant, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, makeStatementsSequenceOrStatement, mangleName ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .SyntaxErrors import raiseSyntaxError def buildExtSliceNode(provider, node, source_ref): elements = [] for dim in node.slice.dims: dim_kind = getKind(dim) if dim_kind == "Slice": lower = buildNode(provider, dim.lower, source_ref, True) upper = buildNode(provider, dim.upper, source_ref, True) step = buildNode(provider, dim.step, source_ref, True) element = ExpressionBuiltinSlice( start = lower, stop = upper, step = step, source_ref = source_ref ) elif dim_kind == "Ellipsis": element = ExpressionConstantRef( constant = Ellipsis, source_ref = source_ref, user_provided = True ) elif dim_kind == "Index": element = buildNode( provider = provider, node = dim.value, source_ref = source_ref ) else: assert False, dim elements.append(element) return makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = elements, source_ref = source_ref ) def buildAssignmentStatementsFromDecoded(provider, kind, detail, source, source_ref): # This is using many variable names on purpose, so as to give names to the # unpacked detail values, and has many branches due to the many cases # dealt with, pylint: disable=R0912,R0914 if kind == "Name": variable_ref = detail return StatementAssignmentVariable( variable_ref = variable_ref, source = source, source_ref = source_ref ) elif kind == "Attribute": lookup_source, attribute_name = detail return StatementAssignmentAttribute( expression = lookup_source, attribute_name = attribute_name, source = source, source_ref = source_ref ) elif kind == "Subscript": subscribed, subscript = detail return StatementAssignmentSubscript( expression = subscribed, subscript = subscript, source = source, source_ref = source_ref ) elif kind == "Slice": lookup_source, lower, upper = detail # For Python3 there is no slicing operation, this is always done # with subscript using a slice object. For Python2, it is only done # if no "step" is provided. use_sliceobj = python_version >= 300 if use_sliceobj: return StatementAssignmentSubscript( expression = lookup_source, source = source, subscript = ExpressionBuiltinSlice( start = lower, stop = upper, step = None, source_ref = source_ref ), source_ref = source_ref ) else: return StatementAssignmentSlice( expression = lookup_source, lower = lower, upper = upper, source = source, source_ref = source_ref ) elif kind == "Tuple": temp_scope = provider.allocateTempScope("tuple_unpack") source_iter_var = provider.allocateTempVariable( temp_scope = temp_scope, name = "source_iter" ) statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = source_iter_var, source_ref = source_ref ), source = ExpressionBuiltinIter1( value = source, source_ref = source_ref ), source_ref = source_ref ) ] element_vars = [ provider.allocateTempVariable( temp_scope = temp_scope, name = "element_%d" % ( element_index + 1 ) ) for element_index in range(len(detail)) ] starred_list_var = None starred_index = None for element_index, element in enumerate(detail): element_var = element_vars[element_index] if starred_list_var is not None: if element[0] == "Starred": raiseSyntaxError( reason = "two starred expressions in assignment", col_offset = 0, source_ref = source_ref ) statements.insert( starred_index+2, StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = element_var, source_ref = source_ref ), source = ExpressionListOperationPop( list_arg = ExpressionTempVariableRef( variable = starred_list_var, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ) elif element[0] != "Starred": statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = element_var, source_ref = source_ref ), source = ExpressionSpecialUnpack( value = ExpressionTempVariableRef( variable = source_iter_var, source_ref = source_ref ), count = element_index + 1, expected = len(detail), source_ref = source_ref ), source_ref = source_ref ) ) else: starred_index = element_index starred_list_var = element_var statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = element_var, source_ref = source_ref ), source = ExpressionBuiltinList( value = ExpressionTempVariableRef( variable = source_iter_var, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ) if starred_list_var is None: statements.append( StatementSpecialUnpackCheck( iterator = ExpressionTempVariableRef( variable = source_iter_var, source_ref = source_ref ), count = len(detail), source_ref = source_ref ) ) # When all is done, copy over to the actual assignment targets, starred # or not makes no difference here anymore. for element_index, element in enumerate(detail): if element[0] == "Starred": element = element[1] element_var = element_vars[element_index] statements.append( buildAssignmentStatementsFromDecoded( provider = provider, kind = element[0], detail = element[1], source = ExpressionTempVariableRef( variable = element_var, source_ref = source_ref ), source_ref = source_ref ) ) final_statements = [] final_statements.append( StatementReleaseVariable( variable = source_iter_var, source_ref = source_ref ) ) # TODO: In that order, or reversed. for element_var in element_vars: final_statements.append( StatementReleaseVariable( variable = element_var, source_ref = source_ref ) ) return makeTryFinallyStatement( provider = provider, tried = statements, final = final_statements, source_ref = source_ref ) elif kind == "Starred": raiseSyntaxError( reason = "starred assignment target must be in a list or tuple", source_ref = source_ref, col_offset = 0 ) else: assert False, (kind, source_ref, detail) def buildAssignmentStatements(provider, node, source, source_ref, allow_none = False, temp_provider = None): if node is None and allow_none: return None if temp_provider is None: temp_provider = provider kind, detail = decodeAssignTarget( provider = provider, node = node, source_ref = source_ref ) return buildAssignmentStatementsFromDecoded( provider = temp_provider, kind = kind, detail = detail, source = source, source_ref = source_ref ) def decodeAssignTarget(provider, node, source_ref, allow_none = False): # Many cases to deal with, because of the different assign targets, # pylint: disable=R0911,R0912 if node is None and allow_none: return None if type(node) is str: return "Name", ExpressionTargetVariableRef( variable_name = mangleName(node, provider), source_ref = source_ref ) kind = getKind(node) if hasattr(node, "ctx"): assert getKind(node.ctx) in ("Store", "Del") if kind == "Name": return kind, ExpressionTargetVariableRef( variable_name = mangleName(node.id, provider), source_ref = source_ref ) elif kind == "Attribute": return kind, ( buildNode(provider, node.value, source_ref), node.attr ) elif kind == "Subscript": slice_kind = getKind(node.slice) if slice_kind == "Index": return "Subscript", ( buildNode(provider, node.value, source_ref), buildNode(provider, node.slice.value, source_ref) ) elif slice_kind == "Slice": lower = buildNode(provider, node.slice.lower, source_ref, True) upper = buildNode(provider, node.slice.upper, source_ref, True) if node.slice.step is not None: step = buildNode(provider, node.slice.step, source_ref) return "Subscript", ( buildNode(provider, node.value, source_ref), ExpressionBuiltinSlice( start = lower, stop = upper, step = step, source_ref = source_ref ) ) else: return "Slice", ( buildNode(provider, node.value, source_ref), lower, upper ) elif slice_kind == "ExtSlice": return "Subscript", ( buildNode(provider, node.value, source_ref), buildExtSliceNode(provider, node, source_ref) ) elif slice_kind == "Ellipsis": return "Subscript", ( buildNode(provider, node.value, source_ref), ExpressionConstantRef( constant = Ellipsis, source_ref = source_ref ) ) else: assert False, slice_kind elif kind in ("Tuple", "List"): return "Tuple", tuple( decodeAssignTarget( provider = provider, node = sub_node, source_ref = source_ref, allow_none = False ) for sub_node in node.elts ) elif kind == "Starred": return "Starred", decodeAssignTarget( provider = provider, node = node.value, source_ref = source_ref, allow_none = False ) else: assert False, (source_ref, kind) def buildAssignNode(provider, node, source_ref): assert len(node.targets) >= 1, source_ref # Evaluate the right hand side first, so it can get names provided # before the left hand side exists. source = buildNode(provider, node.value, source_ref) if len(node.targets) == 1: # Simple assignment case, one source, one target. return buildAssignmentStatements( provider = provider, node = node.targets[0], source = source, source_ref = source_ref ) else: # Complex assignment case, one source, but multiple targets. We keep the # source in a temporary variable, and then assign from it multiple # times. temp_scope = provider.allocateTempScope("assign_unpack") tmp_source = provider.allocateTempVariable( temp_scope = temp_scope, name = "assign_source" ) statements = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_source, source_ref = source_ref ), source = source, source_ref = source_ref ) ] for target in node.targets: statements.append( buildAssignmentStatements( provider = provider, node = target, source = ExpressionTempVariableRef( variable = tmp_source, source_ref = source_ref ), source_ref = source_ref ) ) return makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_source, source_ref = source_ref ), source_ref = source_ref ) def buildDeleteStatementFromDecoded(kind, detail, source_ref): if kind in ("Name", "Name_Exception"): # Note: Name_Exception is a "del" for exception handlers that doesn't # insist on the variable being defined, user code may do it too, and # that will be fine, so make that tolerant. variable_ref = detail return StatementDelVariable( variable_ref = variable_ref, tolerant = kind == "Name_Exception", source_ref = source_ref ) elif kind == "Attribute": lookup_source, attribute_name = detail return StatementDelAttribute( expression = lookup_source, attribute_name = attribute_name, source_ref = source_ref ) elif kind == "Subscript": subscribed, subscript = detail return StatementDelSubscript( expression = subscribed, subscript = subscript, source_ref = source_ref ) elif kind == "Slice": lookup_source, lower, upper = detail use_sliceobj = python_version >= 300 if use_sliceobj: return StatementDelSubscript( expression = lookup_source, subscript = ExpressionBuiltinSlice( start = lower, stop = upper, step = None, source_ref = source_ref ), source_ref = source_ref ) else: return StatementDelSlice( expression = lookup_source, lower = lower, upper = upper, source_ref = source_ref ) elif kind == "Tuple": result = [] for sub_node in detail: result.append( buildDeleteStatementFromDecoded( kind = sub_node[0], detail = sub_node[1], source_ref = source_ref ) ) return makeStatementsSequenceOrStatement( statements = result, source_ref = source_ref ) else: assert False, (kind, detail, source_ref) def buildDeleteNode(provider, node, source_ref): # Build "del" statements. # Note: Each delete is sequential. It can succeed, and the failure of a # later one does not prevent the former to succeed. We can therefore have a # simple sequence of "del" statements that each only delete one thing # therefore. In output tree "del" therefore only ever has single arguments. statements = [] for target in node.targets: kind, detail = decodeAssignTarget( provider = provider, node = target, source_ref = source_ref ) statements.append( buildDeleteStatementFromDecoded( kind = kind, detail = detail, source_ref = source_ref ) ) return makeStatementsSequenceOrStatement( statements = statements, source_ref = source_ref ) def _buildInplaceAssignVariableNode(variable_ref, operator, expression, source_ref): assert variable_ref.isExpressionTargetVariableRef(), variable_ref inplace_node = ExpressionOperationBinaryInplace( operator = operator, left = ExpressionVariableRef( variable_name = variable_ref.getVariableName(), source_ref = source_ref ), right = expression, source_ref = source_ref ) inplace_node.markAsInplaceSuspect() result = ( StatementAssignmentVariable( variable_ref = ExpressionTargetVariableRef( variable_name = variable_ref.getVariableName(), source_ref = source_ref ), source = inplace_node, source_ref = source_ref ), ) return result def _buildInplaceAssignAttributeNode(provider, lookup_source, attribute_name, tmp_variable1, tmp_variable2, operator, expression, source_ref): # First assign the target value to a temporary variable. preserve_to_tmp = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), source = ExpressionAttributeLookup( source = lookup_source.makeClone(), attribute_name = attribute_name, source_ref = source_ref ), source_ref = source_ref ) # Second assign the in-place result to a temporary variable inplace_to_tmp = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source = ExpressionOperationBinaryInplace( operator = operator, left = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) # Third, copy it over, if the reference values change, i.e. IsNot is true. copy_back_from_tmp = StatementConditional( condition = ExpressionComparisonIsNOT( left = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), right = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementAssignmentAttribute( expression = lookup_source.makeClone(), attribute_name = attribute_name, source = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ) copy_back_from_tmp = makeTryFinallyStatement( provider = provider, tried = copy_back_from_tmp, final = StatementReleaseVariable( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ) return ( preserve_to_tmp, # making sure the above temporary variable is deleted in any case. makeTryFinallyStatement( provider = provider, tried = ( inplace_to_tmp, copy_back_from_tmp, ), final = StatementReleaseVariable( variable = tmp_variable1, source_ref = source_ref ), source_ref = source_ref ) ) def _buildInplaceAssignSubscriptNode(provider, subscribed, subscript, tmp_variable1, tmp_variable2, operator, expression, source_ref): # First assign the subscribed value to a temporary variable. preserve_to_tmp1 = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), source = subscribed, source_ref = source_ref ) # Second assign the subscript value to a temporary variable preserve_to_tmp2 = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source = subscript, source_ref = source_ref ) execute_in_place = StatementAssignmentSubscript( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source = ExpressionOperationBinaryInplace( operator = operator, left = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) # Note: No copy back is happening, for subscripts that is implied. return ( preserve_to_tmp1, makeTryFinallyStatement( provider = provider, tried = ( preserve_to_tmp2, execute_in_place, ), final = ( StatementReleaseVariable( variable = tmp_variable1, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_variable2, source_ref = source_ref ) ), source_ref = source_ref ) ) def _buildInplaceAssignSliceNode(provider, lookup_source, lower, upper, tmp_variable1, tmp_variable2, tmp_variable3, operator, expression, source_ref): # Due to the 3 inputs, which we need to also put into temporary variables, # there are too many variables here, but they are needed. # pylint: disable=R0914 # First assign the target value, lower and upper to temporary variables. copy_to_tmp = StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), source = lookup_source, source_ref = source_ref ) final_statements = [ StatementReleaseVariable( variable = tmp_variable1, source_ref = source_ref ) ] statements = [] if lower is not None: statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source = lower, source_ref = source_ref ) ) final_statements.append( StatementReleaseVariable( variable = tmp_variable2, source_ref = source_ref ) ) lower_ref1 = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ) lower_ref2 = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ) else: assert tmp_variable2 is None lower_ref1 = lower_ref2 = None if upper is not None: statements.append( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = tmp_variable3, source_ref = source_ref ), source = upper, source_ref = source_ref ) ) final_statements.append( StatementReleaseVariable( variable = tmp_variable3, source_ref = source_ref ) ) upper_ref1 = ExpressionTempVariableRef( variable = tmp_variable3, source_ref = source_ref ) upper_ref2 = ExpressionTempVariableRef( variable = tmp_variable3, source_ref = source_ref ) else: assert tmp_variable3 is None upper_ref1 = upper_ref2 = None use_sliceobj = python_version >= 300 # Second assign the in-place result over the original value. if use_sliceobj: statements.append( StatementAssignmentSubscript( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionBuiltinSlice( start = lower_ref1, stop = upper_ref1, step = None, source_ref = source_ref ), source = ExpressionOperationBinaryInplace( operator = operator, left = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionBuiltinSlice( start = lower_ref2, stop = upper_ref2, step = None, source_ref = source_ref ), source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) ) else: statements.append( StatementAssignmentSlice( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), lower = lower_ref1, upper = upper_ref1, source = ExpressionOperationBinaryInplace( operator = operator, left = ExpressionSliceLookup( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), lower = lower_ref2, upper = upper_ref2, source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) ) return ( copy_to_tmp, makeTryFinallyStatement( provider = provider, tried = statements, final = final_statements, source_ref = source_ref ) ) def buildInplaceAssignNode(provider, node, source_ref): # There are many inplace assignment variables, and the detail is unpacked # into names, so we end up with a lot of variables, which is on purpose, # pylint: disable=R0914 operator = getKind(node.op) if operator == "Div" and source_ref.getFutureSpec().isFutureDivision(): operator = "TrueDiv" operator = 'I' + operator expression = buildNode(provider, node.value, source_ref) kind, detail = decodeAssignTarget( provider = provider, node = node.target, source_ref = source_ref ) if kind == "Name": variable_ref = detail statements = _buildInplaceAssignVariableNode( variable_ref = variable_ref, operator = operator, expression = expression, source_ref = source_ref ) elif kind == "Attribute": lookup_source, attribute_name = detail temp_scope = provider.allocateTempScope("inplace_assign_attr") tmp_variable1 = provider.allocateTempVariable( temp_scope = temp_scope, name = "start" ) tmp_variable2 = provider.allocateTempVariable( temp_scope = temp_scope, name = "end" ) statements = _buildInplaceAssignAttributeNode( provider = provider, lookup_source = lookup_source, attribute_name = attribute_name, tmp_variable1 = tmp_variable1, tmp_variable2 = tmp_variable2, operator = operator, expression = expression, source_ref = source_ref ) elif kind == "Subscript": subscribed, subscript = detail temp_scope = provider.allocateTempScope("inplace_assign_subscr") tmp_variable1 = provider.allocateTempVariable( temp_scope = temp_scope, name = "target" ) tmp_variable2 = provider.allocateTempVariable( temp_scope = temp_scope, name = "subscript" ) statements = _buildInplaceAssignSubscriptNode( provider = provider, subscribed = subscribed, subscript = subscript, tmp_variable1 = tmp_variable1, tmp_variable2 = tmp_variable2, operator = operator, expression = expression, source_ref = source_ref ) elif kind == "Slice": lookup_source, lower, upper = detail temp_scope = provider.allocateTempScope("inplace_assign_slice") tmp_variable1 = provider.allocateTempVariable( temp_scope = temp_scope, name = "target" ) if lower is not None: tmp_variable2 = provider.allocateTempVariable( temp_scope = temp_scope, name = "lower" ) else: tmp_variable2 = None if upper is not None: tmp_variable3 = provider.allocateTempVariable( temp_scope = temp_scope, name = "upper" ) else: tmp_variable3 = None statements = _buildInplaceAssignSliceNode( provider = provider, lookup_source = lookup_source, lower = lower, upper = upper, tmp_variable1 = tmp_variable1, tmp_variable2 = tmp_variable2, tmp_variable3 = tmp_variable3, operator = operator, expression = expression, source_ref = source_ref ) else: assert False, kind return makeStatementsSequenceFromStatements( *statements ) Nuitka-0.5.18.1/nuitka/__past__.py0000644000175000017500000000362512651071040017043 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module like __future__ for things that are changed between Python2 and Python3. These are here to provide compatible fall-backs. This is required to run the same code easily with both CPython2 and CPython3. Sometimes, we do not care about the actual types, or API, but would rather just check for something to be a "in (str, unicode)" rather than making useless version checks. """ # pylint: disable=C0103,W0622 # Work around for CPython 3.x renaming "long" to "int". try: long = long # @ReservedAssignment except NameError: long = int # @ReservedAssignment # Work around for CPython 3.x renaming "unicode" to "str". try: unicode = unicode # @ReservedAssignment except NameError: unicode = str # @ReservedAssignment def iterItems(d): try: return d.iteritems() except AttributeError: return d.items() if unicode is str: raw_input = input # @ReservedAssignment else: raw_input = raw_input # @ReservedAssignment try: from urllib.request import urlretrieve except ImportError: from urllib import urlretrieve # For PyLint to be happy. assert long assert unicode assert urlretrieve Nuitka-0.5.18.1/nuitka/__init__.py0000644000175000017500000000150112651071040017026 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/PythonOperators.py0000644000175000017500000000744312651071040020462 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Python operator tables These are mostly used to resolve the operator in the module operator and to know the list of operations allowed. """ import operator from nuitka.PythonVersions import python_version # TODO: Monkey patching this may cause issues for compile time assessments of # user code. This should be solved differently. if python_version >= 300: operator.div = operator.truediv operator.idiv = operator.itruediv binary_operator_functions = { "Add" : operator.add, "Sub" : operator.sub, "Pow" : operator.pow, "Mult" : operator.mul, "Div" : operator.div, "FloorDiv" : operator.floordiv, "TrueDiv" : operator.truediv, "Mod" : operator.mod, "LShift" : operator.lshift, "RShift" : operator.rshift, "BitAnd" : operator.and_, "BitOr" : operator.or_, "BitXor" : operator.xor, "IAdd" : operator.iadd, "ISub" : operator.isub, "IPow" : operator.ipow, "IMult" : operator.imul, "IDiv" : operator.idiv, "IFloorDiv" : operator.ifloordiv, "ITrueDiv" : operator.itruediv, "IMod" : operator.imod, "ILShift" : operator.ilshift, "IRShift" : operator.irshift, "IBitAnd" : operator.iand, "IBitOr" : operator.ior, "IBitXor" : operator.ixor, } # Python 3.5 only operator if python_version >= 350: binary_operator_functions["MatMult"] = operator.matmul # @UndefinedVariable binary_operator_functions["IMatMult"] = operator.imatmul # @UndefinedVariable unary_operator_functions = { "UAdd" : operator.pos, "USub" : operator.neg, "Invert" : operator.invert, "Repr" : repr, # Boolean not is treated an unary operator. "Not" : operator.not_, } rich_comparison_functions = { "Lt" : operator.lt, "LtE" : operator.le, "Eq" : operator.eq, "NotEq" : operator.ne, "Gt" : operator.gt, "GtE" : operator.ge } other_comparison_functions = { "Is" : operator.is_, "IsNot" : operator.is_not, "In" : lambda value1, value2: value1 in value2, "NotIn" : lambda value1, value2: value1 not in value2 } comparison_inversions = { "Is" : "IsNot", "IsNot" : "Is", "In" : "NotIn", "NotIn" : "In" } all_comparison_functions = dict(rich_comparison_functions) all_comparison_functions.update(other_comparison_functions) def matchException(left, right): if python_version >= 300: if type(right) is tuple: for element in right: if not isinstance(BaseException,element): raise TypeError("catching classes that do not inherit from BaseException is not allowed") elif not isinstance(BaseException,right): raise TypeError("catching classes that do not inherit from BaseException is not allowed") # This doesn't yet work, make it error exit. and silence PyLint for now. # pylint: disable=W0212 import os os._exit(16) assert False, left all_comparison_functions["exception_match"]=matchException Nuitka-0.5.18.1/nuitka/optimizations/0000755000175000017500000000000012651072347017644 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/optimizations/Tags.py0000644000175000017500000000410712651071040021103 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tags and set of it. Used by optimization to keep track of the current state of optimization, these tags trigger the execution of optimization steps, which in turn may emit these tags to execute other steps. """ allowed_tags = ( # New code means new statements. # Could be a new module, or an inlined exec statement. "new_code", # Added new import. "new_import", # New statements added, removed. "new_statements", # New expression added. "new_expression", # TODO: A bit unclear what this it, potentially a changed variable. "var_usage", # Detected module variable to be read only. "read_only_mvar", # New built-in reference detected. "new_builtin_ref", # New built-in call detected. "new_builtin", # New raise statement detected. "new_raise", # New constant introduced. "new_constant", ) class TagSet(set): def onSignal(self, signal): if type(signal) is str: signal = signal.split() for tag in signal: self.add(tag) def check(self, tags): for tag in tags.split(): assert tag in allowed_tags, tag if tag in self: return True return False def add(self, tag): assert tag in allowed_tags, tag set.add(self, tag) Nuitka-0.5.18.1/nuitka/optimizations/TraceCollections.py0000644000175000017500000006111712651072046023455 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Constraint collection At the core of value propagation there is the collection of constraints that allow to propagate knowledge forward or not. This is about collecting these constraints and to manage them. """ import contextlib from logging import debug from nuitka import Tracing, VariableRegistry from nuitka.__past__ import iterItems # Python3 compatibility. from nuitka.nodes.NodeMakingHelpers import getComputationResult from nuitka.PythonVersions import python_version from .VariableTraces import ( VariableTraceAssign, VariableTraceInit, VariableTraceLoopMerge, VariableTraceMerge, VariableTraceUninit, VariableTraceUnknown ) signalChange = None class VariableUsageTrackingMixin: def initVariable(self, variable): if variable.isParameterVariable(): result = self._initVariableInit(variable) elif variable.isLocalVariable(): result = self._initVariableUninit(variable) elif variable.isMaybeLocalVariable(): result = self._initVariableUnknown(variable) elif variable.isModuleVariable(): result = self._initVariableUnknown(variable) elif variable.isTempVariable(): result = self._initVariableUninit(variable) else: assert False, variable assert result.getVariable() is variable return result class CollectionTracingMixin: def __init__(self): # For functions, when we are in here, the currently active one, self.variable_actives = {} def getVariableCurrentTrace(self, variable): return self.getVariableTrace( variable = variable, version = self.getCurrentVariableVersion(variable) ) def markCurrentVariableTrace(self, variable, version): self.variable_actives[variable] = version def getCurrentVariableVersion(self, variable): try: return self.variable_actives[variable] except KeyError: # Initialize variables on the fly. if not self.hasVariableTrace(variable, 0): self.initVariable(variable) self.markCurrentVariableTrace(variable, 0) return self.variable_actives[variable] def getActiveVariables(self): return self.variable_actives.keys() def markActiveVariableAsUnknown(self, variable): current = self.getVariableCurrentTrace( variable = variable, ) if not current.isUnknownTrace(): version = variable.allocateTargetNumber() self.addVariableTrace( variable = variable, version = version, trace = VariableTraceUnknown( variable = variable, version = version, previous = current ) ) self.markCurrentVariableTrace(variable, version) def markActiveVariableAsLoopMerge(self, variable): current = self.getVariableCurrentTrace( variable = variable, ) version = variable.allocateTargetNumber() result = VariableTraceLoopMerge( variable = variable, version = version, previous = current ) self.addVariableTrace( variable = variable, version = version, trace = result ) self.markCurrentVariableTrace(variable, version) return result def markActiveVariablesAsUnknown(self): for variable in self.getActiveVariables(): self.markActiveVariableAsUnknown(variable) class CollectionStartpointMixin: def __init__(self): # Variable assignments performed in here, last issued number, only used # to determine the next number that should be used for a new assignment. self.variable_versions = {} # The full trace of a variable with a version for the function or module # this is. self.variable_traces = {} # Cannot mess with local variables that much, as "locals" and "eval" # calls may not yet be known. self.unclear_locals = False self.break_collections = None self.continue_collections = None self.return_collections = None self.exception_collections = None def getLoopBreakCollections(self): return self.break_collections def onLoopBreak(self, collection = None): if collection is None: collection = self self.break_collections.append( ConstraintCollectionBranch( parent = collection, name = "loop break" ) ) def getLoopContinueCollections(self): return self.continue_collections def onLoopContinue(self, collection = None): if collection is None: collection = self self.continue_collections.append( ConstraintCollectionBranch( parent = collection, name = "loop continue" ) ) def onFunctionReturn(self, collection = None): if collection is None: collection = self if self.return_collections is not None: self.return_collections.append( ConstraintCollectionBranch( parent = collection, name = "return" ) ) def onExceptionRaiseExit(self, raisable_exceptions, collection = None): # TODO: We might want to track per exception, pylint: disable=W0613 if collection is None: collection = self if self.exception_collections is not None: self.exception_collections.append( ConstraintCollectionBranch( parent = collection, name = "exception" ) ) def getFunctionReturnCollections(self): return self.return_collections def getExceptionRaiseCollections(self): return self.exception_collections def hasVariableTrace(self, variable, version): return (variable, version) in self.variable_traces def getVariableTrace(self, variable, version): return self.variable_traces[(variable, version)] def getVariableTraces(self, variable): result = [] for key, variable_trace in iterItems(self.variable_traces): candidate = key[0] if variable is candidate: result.append(variable_trace) return result def getVariableTracesAll(self): return self.variable_traces def addVariableTrace(self, variable, version, trace): key = variable, version assert key not in self.variable_traces, (key, self) self.variable_traces[key] = trace def addVariableMergeMultipleTrace(self, variable, traces): version = variable.allocateTargetNumber() trace_merge = VariableTraceMerge( variable = variable, version = version, traces = traces ) self.addVariableTrace(variable, version, trace_merge) return version def dumpTraces(self): debug("Constraint collection state: %s", self) for _variable_desc, variable_trace in sorted(iterItems(self.variable_traces)): # debug( "%r: %r", variable_trace ) variable_trace.dump() def dumpActiveTraces(self): Tracing.printSeparator() Tracing.printLine("Active are:") for variable, _version in sorted(self.variable_actives.iteritems()): self.getVariableCurrentTrace(variable).dump() Tracing.printSeparator() def _initVariableUnknown(self, variable): trace = VariableTraceUnknown( variable = variable, version = 0, previous = None ) self.addVariableTrace( variable = variable, version = 0, trace = trace ) return trace def _initVariableInit(self, variable): trace = VariableTraceInit( variable = variable, version = 0, ) self.addVariableTrace( variable = variable, version = 0, trace = trace ) return trace def _initVariableUninit(self, variable): trace = VariableTraceUninit( variable = variable, version = 0, previous = None ) self.addVariableTrace( variable = variable, version = 0, trace = trace ) return trace def assumeUnclearLocals(self): self.unclear_locals = True def hasUnclearLocals(self): return self.unclear_locals def updateFromCollection(self, old_collection): VariableRegistry.updateFromCollection(old_collection, self) @contextlib.contextmanager def makeAbortStackContext(self, catch_breaks, catch_continues, catch_returns, catch_exceptions): if catch_breaks: old_break_collections = self.break_collections self.break_collections = [] if catch_continues: old_continue_collections = self.continue_collections self.continue_collections = [] if catch_returns: old_return_collections = self.return_collections self.return_collections = [] if catch_exceptions: old_exception_collections = self.exception_collections self.exception_collections = [] yield if catch_breaks: self.break_collections = old_break_collections if catch_continues: self.continue_collections = old_continue_collections if catch_returns: self.return_collections = old_return_collections if catch_exceptions: self.exception_collections = old_exception_collections class ConstraintCollectionBase(CollectionTracingMixin): def __init__(self, name, parent): CollectionTracingMixin.__init__(self) self.parent = parent self.name = name # Trust variable_traces, should go away later on, for now we use it to # disable optimization. self.removes_knowledge = False def __repr__(self): return "<%s for %s %d>" % ( self.__class__.__name__, self.name, id(self) ) @staticmethod def signalChange(tags, source_ref, message): # This is monkey patches from another module, pylint: disable=E1102 signalChange(tags, source_ref, message) @staticmethod def mustAlias(a, b): if a.isExpressionVariableRef() and b.isExpressionVariableRef(): return a.getVariable() is b.getVariable() return False @staticmethod def mustNotAlias(a, b): # TODO: not yet really implemented, pylint: disable=W0613 return False def removeKnowledge(self, node): pass def onControlFlowEscape(self, node): # TODO: One day, we should trace which nodes exactly cause a variable # to be considered escaped, pylint: disable=W0613 for variable in self.getActiveVariables(): if variable.isModuleVariable(): # print variable self.markActiveVariableAsUnknown(variable) elif python_version >= 300 or variable.isSharedTechnically(): # print variable # TODO: Could be limited to shared variables that are actually # written to. Most of the time, that won't be the case. self.markActiveVariableAsUnknown(variable) def removeAllKnowledge(self): # Temporary, we don't have to have this anyway, this will just disable # all uses of variable traces for optimization. self.removes_knowledge = True self.markActiveVariablesAsUnknown() def assumeUnclearLocals(self): self.parent.assumeUnclearLocals() def getVariableTrace(self, variable, version): return self.parent.getVariableTrace(variable, version) def hasVariableTrace(self, variable, version): return self.parent.hasVariableTrace(variable, version) def addVariableTrace(self, variable, version, trace): self.parent.addVariableTrace(variable, version, trace) def addVariableMergeMultipleTrace(self, variable, traces): return self.parent.addVariableMergeMultipleTrace(variable, traces) def onVariableSet(self, assign_node): variable_ref = assign_node.getTargetVariableRef() version = variable_ref.getVariableVersion() variable = variable_ref.getVariable() variable_trace = VariableTraceAssign( assign_node = assign_node, variable = variable, version = version, previous = self.getVariableCurrentTrace( variable = variable ) ) self.addVariableTrace( variable = variable, version = version, trace = variable_trace ) # Make references point to it. self.markCurrentVariableTrace(variable, version) return variable_trace def onVariableDel(self, variable_ref): # Add a new trace, allocating a new version for the variable, and # remember the delete of the current variable = variable_ref.getVariable() version = variable_ref.getVariableVersion() old_trace = self.getVariableCurrentTrace(variable) variable_trace = VariableTraceUninit( variable = variable, version = version, previous = old_trace ) # Assign to not initialized again. self.addVariableTrace( variable = variable, version = version, trace = variable_trace ) # Make references point to it. self.markCurrentVariableTrace(variable, version) def onLocalsUsage(self): for variable in self.getActiveVariables(): # TODO: Currently this is a bit difficult to express in a positive # way, but we want to have only local variables. if not variable.isTempVariable() and \ not variable.isModuleVariable(): variable_trace = self.getVariableCurrentTrace( variable ) variable_trace.addNameUsage() def onVariableRelease(self, variable): current = self.getVariableCurrentTrace(variable) # Annotate that releases to the trace, it may be important knowledge. current.addRelease() return current def onVariableContentEscapes(self, variable): self.getVariableCurrentTrace(variable).onValueEscape() def onExpression(self, expression, allow_none = False): if expression is None and allow_none: return None assert expression.isExpression(), expression assert expression.parent, expression # Now compute this expression, allowing it to replace itself with # something else as part of a local peep hole optimization. r = expression.computeExpressionRaw( constraint_collection = self ) assert type(r) is tuple, expression new_node, change_tags, change_desc = r if change_tags is not None: # This is mostly for tracing and indication that a change occurred # and it may be interesting to look again. self.signalChange( change_tags, expression.getSourceReference(), change_desc ) if new_node is not expression: expression.replaceWith(new_node) if new_node.isExpressionVariableRef() or \ new_node.isExpressionTempVariableRef(): # Remember the reference for constraint collection. assert new_node.variable_trace.hasDefiniteUsages() # We add variable reference nodes late to their traces, only after they # are actually produced, and not resolved to something else, so we do # not have them dangling, and the code complexity inside of their own # "computeExpression" functions. if new_node.isExpressionVariableRef() or \ new_node.isExpressionTempVariableRef(): if new_node.getVariable().isMaybeLocalVariable(): variable_trace = self.getVariableCurrentTrace( variable = new_node.getVariable().getMaybeVariable() ) variable_trace.addUsage() return new_node def onStatement(self, statement): try: assert statement.isStatement(), statement new_statement, change_tags, change_desc = \ statement.computeStatement(self) # print new_statement, change_tags, change_desc if new_statement is not statement: self.signalChange( change_tags, statement.getSourceReference(), change_desc ) return new_statement except Exception: Tracing.printError( "Problem with statement at %s:" % statement.getSourceReference().getAsString() ) raise def mergeBranches(self, collection_yes, collection_no): """ Merge two alternative branches into this trace. This is mostly for merging conditional branches, or other ways of having alternative control flow. This deals with up to two alternative branches to both change this collection. """ # Refuse to do stupid work if collection_yes is None and collection_no is None: pass elif collection_yes is None or collection_no is None: # Handle one branch case, we need to merge versions backwards as # they may make themselves obsolete. self.mergeMultipleBranches( collections = (self, collection_yes or collection_no) ) else: self.mergeMultipleBranches( collections = (collection_yes,collection_no) ) def mergeMultipleBranches(self, collections): assert len(collections) > 0 # Optimize for length 1, which is trivial merge and needs not a # lot of work. if len(collections) == 1: self.replaceBranch(collections[0]) return variable_versions = {} for collection in collections: for variable, version in iterItems(collection.variable_actives): if variable not in variable_versions: variable_versions[variable] = set([version]) else: variable_versions[variable].add(version) for collection in collections: for variable, versions in iterItems(variable_versions): if variable not in collection.variable_actives: versions.add(0) self.variable_actives = {} for variable, versions in iterItems(variable_versions): if len(versions) == 1: version, = versions else: version = self.addVariableMergeMultipleTrace( variable = variable, traces = [ self.getVariableTrace(variable, version) for version in versions ] ) self.markCurrentVariableTrace(variable, version) def replaceBranch(self, collection_replace): self.variable_actives.update(collection_replace.variable_actives) collection_replace.variable_actives = None def onLoopBreak(self, collection = None): if collection is None: collection = self return self.parent.onLoopBreak(collection) def onLoopContinue(self, collection = None): if collection is None: collection = self return self.parent.onLoopContinue(collection) def onFunctionReturn(self, collection = None): if collection is None: collection = self return self.parent.onFunctionReturn(collection) def onExceptionRaiseExit(self, raisable_exceptions, collection = None): if collection is None: collection = self return self.parent.onExceptionRaiseExit(raisable_exceptions, collection) def getLoopBreakCollections(self): return self.parent.getLoopBreakCollections() def getLoopContinueCollections(self): return self.parent.getLoopContinueCollections() def getFunctionReturnCollections(self): return self.parent.getFunctionReturnCollections() def getExceptionRaiseCollections(self): return self.parent.getExceptionRaiseCollections() def makeAbortStackContext(self, catch_breaks, catch_continues, catch_returns, catch_exceptions): return self.parent.makeAbortStackContext( catch_breaks = catch_breaks, catch_continues = catch_continues, catch_returns = catch_returns, catch_exceptions = catch_exceptions ) def getCompileTimeComputationResult(self, node, computation, description): new_node, change_tags, message = getComputationResult( node = node, computation = computation, description = description ) if change_tags == "new_raise": self.onExceptionRaiseExit(BaseException) return new_node, change_tags, message class ConstraintCollectionBranch(ConstraintCollectionBase): def __init__(self, name, parent): ConstraintCollectionBase.__init__( self, name = name, parent = parent ) self.variable_actives = dict(parent.variable_actives) def computeBranch(self, branch): if branch.isStatementsSequence(): result = branch.computeStatementsSequence( constraint_collection = self ) if result is not branch: branch.replaceWith(result) else: self.onExpression( expression = branch ) def initVariable(self, variable): variable_trace = self.parent.initVariable(variable) assert variable_trace.getVersion() == 0 self.variable_actives[variable] = 0 return variable_trace def dumpTraces(self): Tracing.printSeparator() self.parent.dumpTraces() Tracing.printSeparator() def dumpActiveTraces(self): Tracing.printSeparator() Tracing.printLine("Active are:") for variable, _version in sorted(self.variable_actives.iteritems()): self.getVariableCurrentTrace(variable).dump() Tracing.printSeparator() class ConstraintCollectionFunction(CollectionStartpointMixin, ConstraintCollectionBase, VariableUsageTrackingMixin): def __init__(self, parent, function_body): assert function_body.isExpressionFunctionBody() or \ function_body.isExpressionClassBody() or \ function_body.isExpressionGeneratorObjectBody() or \ function_body.isExpressionCoroutineObjectBody(), function_body CollectionStartpointMixin.__init__(self) ConstraintCollectionBase.__init__( self, name = "function " + str(function_body), parent = parent ) # TODO: Useless cyclic dependency. self.function_body = function_body class ConstraintCollectionModule(CollectionStartpointMixin, ConstraintCollectionBase, VariableUsageTrackingMixin): def __init__(self): CollectionStartpointMixin.__init__(self) ConstraintCollectionBase.__init__( self, name = "module", parent = None ) Nuitka-0.5.18.1/nuitka/optimizations/OptimizeBuiltinCalls.py0000644000175000017500000011743012651071040024317 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Optimize calls to built-in references to specific built-in calls. For built-in name references, we check if it's one of the supported built-in types, and then specialize for the ones, where it makes sense. """ from logging import warning from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.nodes.AssignNodes import ( ExpressionTargetTempVariableRef, ExpressionTempVariableRef, StatementAssignmentVariable, StatementDelVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionBuiltinGetattr, ExpressionBuiltinHasattr, ExpressionBuiltinSetattr ) from nuitka.nodes.BuiltinDecodingNodes import ( ExpressionBuiltinChr, ExpressionBuiltinOrd, ExpressionBuiltinOrd0 ) from nuitka.nodes.BuiltinDictNodes import ExpressionBuiltinDict from nuitka.nodes.BuiltinFormatNodes import ( ExpressionBuiltinBin, ExpressionBuiltinHex, ExpressionBuiltinId, ExpressionBuiltinOct ) from nuitka.nodes.BuiltinHashNodes import ExpressionBuiltinHash from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionBuiltinIter2, ExpressionBuiltinLen, ExpressionBuiltinNext1, ExpressionBuiltinNext2 ) from nuitka.nodes.BuiltinOpenNodes import ExpressionBuiltinOpen from nuitka.nodes.BuiltinRangeNodes import ( ExpressionBuiltinRange0, ExpressionBuiltinRange1, ExpressionBuiltinRange2, ExpressionBuiltinRange3 ) from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, ExpressionBuiltinOriginalRef, ExpressionBuiltinRef ) from nuitka.nodes.BuiltinTypeNodes import ( ExpressionBuiltinBool, ExpressionBuiltinBytearray, ExpressionBuiltinComplex, ExpressionBuiltinFloat, ExpressionBuiltinInt, ExpressionBuiltinList, ExpressionBuiltinSet, ExpressionBuiltinStr, ExpressionBuiltinTuple ) from nuitka.nodes.BuiltinVarsNodes import ExpressionBuiltinVars from nuitka.nodes.CallNodes import ExpressionCallEmpty, ExpressionCallNoKeywords from nuitka.nodes.ClassNodes import ExpressionBuiltinType3 from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import ExpressionConstantRef from nuitka.nodes.ExecEvalNodes import ( ExpressionBuiltinCompile, ExpressionBuiltinEval ) from nuitka.nodes.GlobalsLocalsNodes import ( ExpressionBuiltinDir1, ExpressionBuiltinGlobals, ExpressionBuiltinLocals ) from nuitka.nodes.ImportNodes import ExpressionBuiltinImport from nuitka.nodes.NodeMakingHelpers import ( makeRaiseExceptionReplacementExpression, makeRaiseExceptionReplacementExpressionFromInstance, wrapExpressionWithSideEffects ) from nuitka.nodes.OperatorNodes import ( ExpressionOperationNOT, ExpressionOperationUnary ) from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.SliceNodes import ExpressionBuiltinSlice from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.nodes.TypeNodes import ( ExpressionBuiltinIsinstance, ExpressionBuiltinSuper, ExpressionBuiltinType1 ) from nuitka.nodes.VariableRefNodes import ExpressionVariableRef from nuitka.Options import isDebug, shallMakeModule from nuitka.PythonVersions import python_version from nuitka.tree.Helpers import ( makeStatementsSequence, makeStatementsSequenceFromStatement ) from nuitka.tree.ReformulationExecStatements import wrapEvalGlobalsAndLocals from nuitka.tree. \ ReformulationTryFinallyStatements import makeTryFinallyStatement from nuitka.VariableRegistry import addVariableUsage from . import BuiltinOptimization def dir_extractor(node): def buildDirEmptyCase(source_ref): if node.getParentVariableProvider().isCompiledPythonModule(): source = ExpressionBuiltinGlobals( source_ref = source_ref ) else: source = ExpressionBuiltinLocals( source_ref = source_ref ) result = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = source, attribute_name = "keys", source_ref = source_ref ), source_ref = source_ref ) # For Python3, keys doesn't really return values, but instead a handle # only. if python_version >= 300: result = ExpressionBuiltinList( value = result, source_ref = source_ref ) return result return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinDir1, builtin_spec = BuiltinOptimization.builtin_dir_spec, empty_special_class = buildDirEmptyCase ) def vars_extractor(node): def selectVarsEmptyClass(source_ref): if node.getParentVariableProvider().isCompiledPythonModule(): return ExpressionBuiltinGlobals( source_ref = source_ref ) else: return ExpressionBuiltinLocals( source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinVars, builtin_spec = BuiltinOptimization.builtin_vars_spec, empty_special_class = selectVarsEmptyClass ) def import_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinImport, builtin_spec = BuiltinOptimization.builtin_import_spec ) def type_extractor(node): args = node.getCallArgs() length = args.getIterationLength() if length == 1: return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinType1, builtin_spec = BuiltinOptimization.builtin_type1_spec ) else: return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinType3, builtin_spec = BuiltinOptimization.builtin_type3_spec ) def iter_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapIterCreation(callable_arg, sentinel, source_ref): if sentinel is None: return ExpressionBuiltinIter1( value = callable_arg, source_ref = source_ref ) else: return ExpressionBuiltinIter2( callable_arg = callable_arg, sentinel = sentinel, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapIterCreation, builtin_spec = BuiltinOptimization.builtin_iter_spec ) def next_extractor(node): # Split up next with and without defaults, they are not going to behave # really very similar. def selectNextBuiltinClass(iterator, default, source_ref): if default is None: return ExpressionBuiltinNext1( value = iterator, source_ref = source_ref ) else: return ExpressionBuiltinNext2( iterator = iterator, default = default, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectNextBuiltinClass, builtin_spec = BuiltinOptimization.builtin_iter_spec ) def dict_extractor(node): # The "dict" built-in is a bit strange in that it accepts a position # parameter, or not, but won't have a default value. def wrapExpressionBuiltinDictCreation(positional_args, dict_star_arg, source_ref): if len(positional_args) > 1: result = makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError( "dict expected at most 1 arguments, got %d" % ( len(positional_args) ) ) ) result = wrapExpressionWithSideEffects( side_effects = positional_args, old_node = node, new_node = result ) if dict_star_arg: result = wrapExpressionWithSideEffects( side_effects = dict_star_arg, old_node = node, new_node = result ) return result return ExpressionBuiltinDict( pos_arg = positional_args[0] if positional_args else None, pairs = dict_star_arg, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinDictCreation, builtin_spec = BuiltinOptimization.builtin_dict_spec ) def chr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinChr, builtin_spec = BuiltinOptimization.builtin_chr_spec ) def ord_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinOrd, builtin_spec = BuiltinOptimization.builtin_ord_spec, empty_special_class = ExpressionBuiltinOrd0 ) def bin_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinBin, builtin_spec = BuiltinOptimization.builtin_bin_spec ) def oct_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinOct, builtin_spec = BuiltinOptimization.builtin_oct_spec ) def hex_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinHex, builtin_spec = BuiltinOptimization.builtin_hex_spec ) def id_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinId, builtin_spec = BuiltinOptimization.builtin_id_spec ) def repr_extractor(node): def makeReprOperator(operand, source_ref): return ExpressionOperationUnary( operator = "Repr", operand = operand, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = makeReprOperator, builtin_spec = BuiltinOptimization.builtin_repr_spec ) def range_extractor(node): def selectRangeBuiltin(low, high, step, source_ref): if high is None: return ExpressionBuiltinRange1( low = low, source_ref = source_ref ) elif step is None: return ExpressionBuiltinRange2( low = low, high = high, source_ref = source_ref ) else: return ExpressionBuiltinRange3( low = low, high = high, step = step, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectRangeBuiltin, builtin_spec = BuiltinOptimization.builtin_range_spec, empty_special_class = ExpressionBuiltinRange0 ) if python_version < 300: from nuitka.nodes.BuiltinRangeNodes import ExpressionBuiltinXrange def xrange_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinXrange, builtin_spec = BuiltinOptimization.builtin_xrange_spec ) def len_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinLen, builtin_spec = BuiltinOptimization.builtin_len_spec ) def tuple_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinTuple, builtin_spec = BuiltinOptimization.builtin_tuple_spec ) def list_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinList, builtin_spec = BuiltinOptimization.builtin_list_spec ) def set_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinSet, builtin_spec = BuiltinOptimization.builtin_set_spec ) def float_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinFloat, builtin_spec = BuiltinOptimization.builtin_float_spec ) def complex_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinComplex, builtin_spec = BuiltinOptimization.builtin_complex_spec ) def str_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinStr, builtin_spec = BuiltinOptimization.builtin_str_spec ) if python_version < 300: from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinUnicode def unicode_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinUnicode, builtin_spec = BuiltinOptimization.builtin_unicode_spec ) def bool_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinBool, builtin_spec = BuiltinOptimization.builtin_bool_spec ) def int_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinInt, builtin_spec = BuiltinOptimization.builtin_int_spec ) if python_version < 300: from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinLong def long_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinLong, builtin_spec = BuiltinOptimization.builtin_long_spec ) def globals_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinGlobals, builtin_spec = BuiltinOptimization.builtin_globals_spec ) def locals_extractor(node): # Note: Locals on the module level is really globals. provider = node.getParentVariableProvider() if provider.isCompiledPythonModule(): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinGlobals, builtin_spec = BuiltinOptimization.builtin_globals_spec ) else: return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinLocals, builtin_spec = BuiltinOptimization.builtin_locals_spec ) if python_version < 300: from nuitka.nodes.ExecEvalNodes import ExpressionBuiltinExecfile def execfile_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapExpressionBuiltinExecfileCreation(filename, globals_arg, locals_arg, source_ref): outline_body = ExpressionOutlineBody( provider = node.getParentVariableProvider(), name = "execfile_call", source_ref = source_ref ) globals_ref, locals_ref, tried, final = wrapEvalGlobalsAndLocals( provider = node.getParentVariableProvider(), globals_node = globals_arg, locals_node = locals_arg, temp_scope = outline_body.getOutlineTempScope(), source_ref = source_ref ) tried = makeStatementsSequence( statements = ( tried, StatementReturn( expression = ExpressionBuiltinExecfile( source_code = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = ExpressionBuiltinOpen( filename = filename, mode = ExpressionConstantRef( constant = "rU", source_ref = source_ref ), buffering = None, source_ref = source_ref ), attribute_name = "read", source_ref = source_ref ), source_ref = source_ref ), globals_arg = globals_ref, locals_arg = locals_ref, source_ref = source_ref ), source_ref = source_ref ) ), allow_none = False, source_ref = source_ref ) outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outline_body, tried = tried, final = final, source_ref = source_ref ) ) ) return outline_body return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinExecfileCreation, builtin_spec = BuiltinOptimization.builtin_execfile_spec ) def eval_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapEvalBuiltin(source, globals_arg, locals_arg, source_ref): provider = node.getParentVariableProvider() outline_body = ExpressionOutlineBody( provider = node.getParentVariableProvider(), name = "eval_call", source_ref = source_ref ) globals_ref, locals_ref, tried, final = wrapEvalGlobalsAndLocals( provider = provider, globals_node = globals_arg, locals_node = locals_arg, temp_scope = outline_body.getOutlineTempScope(), source_ref = source_ref ) # The wrapping should not relocate to the "source_ref". assert globals_arg is None or \ globals_ref.getSourceReference() == \ globals_arg.getSourceReference() assert locals_arg is None or \ locals_ref.getSourceReference() == \ locals_arg.getSourceReference() source_variable = outline_body.allocateTempVariable( temp_scope = None, name = "source" ) final.setStatements( final.getStatements() + ( StatementDelVariable( variable_ref = ExpressionTargetTempVariableRef( variable = source_variable, source_ref = source_ref ), tolerant = True, source_ref = source_ref ), ) ) strip_choice = ExpressionConstantRef( constant = (" \t",), source_ref = source_ref ) if python_version >= 300: strip_choice = ExpressionConditional( condition = ExpressionComparisonIs( left = ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), source_ref = source_ref ), right = ExpressionBuiltinRef( builtin_name = "bytes", source_ref = source_ref ), source_ref = source_ref ), expression_yes = ExpressionConstantRef( constant = (b" \t",), source_ref = source_ref ), expression_no = strip_choice, source_ref = source_ref ) # Source needs some special treatment for eval, if it's a string, it # must be stripped. string_fixup = [ StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = source_variable, source_ref = source_ref ), source = ExpressionCallNoKeywords( called = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), attribute_name = "strip", source_ref = source_ref ), args = strip_choice, source_ref = source_ref ), source_ref = source_ref ) ] statements = ( StatementAssignmentVariable( variable_ref = ExpressionTargetTempVariableRef( variable = source_variable, source_ref = source_ref ), source = source, source_ref = source_ref, ), StatementConditional( condition = ExpressionOperationNOT( operand = ExpressionBuiltinIsinstance( instance = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "code", source_ref = source_ref, ), source_ref = source_ref ), source_ref = source_ref ), yes_branch = StatementsSequence( statements = string_fixup, source_ref = source_ref ), no_branch = None, source_ref = source_ref ), StatementReturn( expression = ExpressionBuiltinEval( source_code = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), globals_arg = globals_ref, locals_arg = locals_ref, source_ref = source_ref ), source_ref = source_ref ) ) tried = makeStatementsSequence( statements = ( tried, ) + statements, allow_none = False, source_ref = source_ref ) outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outline_body, tried = tried, final = final, source_ref = source_ref ) ) ) return outline_body return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapEvalBuiltin, builtin_spec = BuiltinOptimization.builtin_eval_spec ) if python_version >= 300: from nuitka.nodes.ExecEvalNodes import ExpressionBuiltinExec def exec_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapExpressionBuiltinExecCreation(source, globals_arg, locals_arg, source_ref): provider = node.getParentVariableProvider() outline_body = ExpressionOutlineBody( provider = provider, name = "exec_call", source_ref = source_ref ) globals_ref, locals_ref, tried, final = wrapEvalGlobalsAndLocals( provider = provider, globals_node = globals_arg, locals_node = locals_arg, temp_scope = outline_body.getOutlineTempScope(), source_ref = source_ref ) tried = makeStatementsSequence( statements = ( tried, StatementReturn( expression = ExpressionBuiltinExec( source_code = source, globals_arg = globals_ref, locals_arg = locals_ref, source_ref = source_ref ), source_ref = source_ref ), ), allow_none = False, source_ref = source_ref ) # Hack: Allow some APIs to work already tried.parent = outline_body outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = provider, tried = tried, final = final, source_ref = source_ref ) ) ) return outline_body return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinExecCreation, builtin_spec = BuiltinOptimization.builtin_eval_spec ) def compile_extractor(node): def wrapExpressionBuiltinCompileCreation(source_code, filename, mode, flags, dont_inherit, optimize = None, source_ref = None): return ExpressionBuiltinCompile( source_code = source_code, filename = filename, mode = mode, flags = flags, dont_inherit = dont_inherit, optimize = optimize, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinCompileCreation, builtin_spec = BuiltinOptimization.builtin_compile_spec ) def open_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinOpen, builtin_spec = BuiltinOptimization.builtin_open_spec ) def super_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapSuperBuiltin(type_arg, object_arg, source_ref): if type_arg is None and python_version >= 300: provider = node.getParentVariableProvider() type_arg = ExpressionVariableRef( variable_name = "__class__", source_ref = source_ref ) # Ought to be already closure taken. type_arg.setVariable( provider.getVariableForReference( variable_name = "__class__" ) ) # If we already have this as a local variable, then use that # instead. type_arg_owner = type_arg.getVariable().getOwner() if type_arg_owner is provider or \ not (type_arg_owner.isExpressionFunctionBody() or \ type_arg_owner.isExpressionClassBody()): type_arg = None else: addVariableUsage(type_arg.getVariable(), provider) if type_arg is None: return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "SystemError" if python_version < 331 else "RuntimeError", exception_value = "super(): __class__ cell not found", ) if object_arg is None: if provider.isExpressionGeneratorObjectBody(): parameter_provider = provider.getParentVariableProvider() else: parameter_provider = provider if parameter_provider.getParameters().getArgumentCount() > 0: par1_name = parameter_provider.getParameters().getArgumentNames()[0] object_arg = ExpressionVariableRef( variable_name = par1_name, source_ref = source_ref ) object_arg.setVariable( provider.getVariableForReference( variable_name = par1_name ) ) if not object_arg.getVariable().isParameterVariable(): return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "SystemError" if python_version < 330 else "RuntimeError", exception_value = "super(): __class__ cell not found", ) else: return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "RuntimeError", exception_value = "super(): no arguments" ) return ExpressionBuiltinSuper( super_type = type_arg, super_object = object_arg, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapSuperBuiltin, builtin_spec = BuiltinOptimization.builtin_super_spec ) def hasattr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinHasattr, builtin_spec = BuiltinOptimization.builtin_hasattr_spec ) def getattr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinGetattr, builtin_spec = BuiltinOptimization.builtin_getattr_spec ) def setattr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinSetattr, builtin_spec = BuiltinOptimization.builtin_setattr_spec ) def isinstance_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinIsinstance, builtin_spec = BuiltinOptimization.builtin_isinstance_spec ) def bytearray_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinBytearray, builtin_spec = BuiltinOptimization.builtin_bytearray_spec ) def slice_extractor(node): def wrapSlice(start, stop, step, source_ref): if start is not None and stop is None: # Default rules are strange. If one argument is given, it's the # second one then. stop = start start = None return ExpressionBuiltinSlice( start = start, stop = stop, step = step, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapSlice, builtin_spec = BuiltinOptimization.builtin_slice_spec ) def hash_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinHash, builtin_spec = BuiltinOptimization.builtin_hash_spec ) _dispatch_dict = { "compile" : compile_extractor, "globals" : globals_extractor, "locals" : locals_extractor, "eval" : eval_extractor, "dir" : dir_extractor, "vars" : vars_extractor, "__import__" : import_extractor, "chr" : chr_extractor, "ord" : ord_extractor, "bin" : bin_extractor, "oct" : oct_extractor, "hex" : hex_extractor, "id" : id_extractor, "type" : type_extractor, "iter" : iter_extractor, "next" : next_extractor, "range" : range_extractor, "tuple" : tuple_extractor, "list" : list_extractor, "dict" : dict_extractor, "set" : set_extractor, "float" : float_extractor, "complex" : complex_extractor, "str" : str_extractor, "bool" : bool_extractor, "int" : int_extractor, "repr" : repr_extractor, "len" : len_extractor, "super" : super_extractor, "hasattr" : hasattr_extractor, "getattr" : getattr_extractor, "setattr" : setattr_extractor, "isinstance" : isinstance_extractor, # TODO: Disabled for now, not handling all cases. # "bytearray" : bytearray_extractor, "slice" : slice_extractor, "hash" : hash_extractor } if python_version < 300: _dispatch_dict["long"] = long_extractor _dispatch_dict["unicode"] = unicode_extractor _dispatch_dict["execfile"] = execfile_extractor # The handling of 'open' built-in for Python3 is not yet correct. _dispatch_dict["open"] = open_extractor else: _dispatch_dict["exec"] = exec_extractor def check(): from nuitka.Builtins import builtin_names for builtin_name in _dispatch_dict: assert builtin_name in builtin_names, builtin_name check() _builtin_white_list = ( # Not supporting 'print', because it could be replaced, and is not # worth the effort yet. "print", # TODO: This could, and should be supported, as we could e.g. lower # types easily for it. "sorted", ) def computeBuiltinCall(call_node, called): # There is some dispatching for how to output various types of changes, # with lots of cases, pylint: disable=R0912 builtin_name = called.getBuiltinName() if builtin_name in _dispatch_dict: new_node = _dispatch_dict[builtin_name](call_node) # Lets just have this contract to return "None" when no change is meant # to be done. assert new_node is not call_node if new_node is None: return call_node, None, None # For traces, we are going to ignore side effects, and output traces # only based on the basis of it. inspect_node = new_node if inspect_node.isExpressionSideEffects(): inspect_node = inspect_node.getExpression() if inspect_node.isExpressionBuiltinImport(): tags = "new_import" message = """\ Replaced dynamic __import__ %s with static module import.""" % ( inspect_node.kind, ) elif inspect_node.isExpressionBuiltin() or \ inspect_node.isStatementExec(): tags = "new_builtin" message = "Replaced call to built-in '%s' with built-in call '%s'." % ( builtin_name, inspect_node.kind, ) elif inspect_node.isExpressionRaiseException(): tags = "new_raise" message = """\ Replaced call to built-in '%s' with exception raising call.""" % ( inspect_node.kind, ) elif inspect_node.isExpressionOperationUnary(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with unary operation '%s'.""" % ( inspect_node.kind, inspect_node.getOperator() ) elif inspect_node.isExpressionCall(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with call.""" % ( inspect_node.kind, ) elif inspect_node.isExpressionOutlineBody(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with outlined call.""" % builtin_name else: assert False, (builtin_name, "->", inspect_node) # TODO: One day, this should be enabled by default and call either the # original built-in or the optimized above one. That should be done, # once we can eliminate the condition for most cases. if False and isDebug() and not shallMakeModule() and builtin_name: source_ref = called.getSourceReference() new_node = ExpressionConditional( condition = ExpressionComparisonIs( left = ExpressionBuiltinRef( builtin_name = builtin_name, source_ref = source_ref ), right = ExpressionBuiltinOriginalRef( builtin_name = builtin_name, source_ref = source_ref ), source_ref = source_ref ), expression_yes = new_node, expression_no = makeRaiseExceptionReplacementExpression( exception_type = "RuntimeError", exception_value = "Built-in '%s' cannot be replaced." % ( builtin_name ), expression = call_node ), source_ref = source_ref ) assert tags != "" return new_node, tags, message else: if False and isDebug() and builtin_name not in _builtin_white_list: warning( "Not handling built-in '%s', consider support." % builtin_name ) return call_node, None, None Nuitka-0.5.18.1/nuitka/optimizations/Optimization.py0000644000175000017500000002537012651071040022700 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Control the flow of optimizations applied to node tree. Applies abstract execution on all so far known modules until no more optimization is possible. Every successful optimization to anything might make others possible. """ import inspect from logging import debug, info, warning from nuitka import ModuleRegistry, Options, VariableRegistry from nuitka.optimizations import TraceCollections from nuitka.plugins.PluginBase import Plugins from nuitka.Tracing import printLine from nuitka.utils import Utils from .Tags import TagSet _progress = Options.isShowProgress() def _attemptRecursion(module): new_modules = module.attemptRecursion() for new_module in new_modules: debug( "{source_ref} : {tags} : {message}".format( source_ref = new_module.getSourceReference().getAsString(), tags = "new_code", message = "Recursed to module package." ) ) tag_set = None def signalChange(tags, source_ref, message): """ Indicate a change to the optimization framework. """ if message is not None: debug( "{source_ref} : {tags} : {message}".format( source_ref = source_ref.getAsString(), tags = tags, message = message() if inspect.isfunction(message) else message ) ) tag_set.onSignal(tags) # Use this globally from there, without cyclic dependency. TraceCollections.signalChange = signalChange graph = None computation_counters = {} def optimizePythonModule(module): if _progress: printLine( "Doing module local optimizations for '{module_name}'.".format( module_name = module.getFullName() ) ) # The tag set is global, so it can react to changes without context. # pylint: disable=W0603 global tag_set tag_set = TagSet() touched = False if _progress: memory_watch = Utils.MemoryWatch() while True: tag_set.clear() try: module.computeModule() except BaseException: info("Interrupted while working on '%s'." % module) raise if not tag_set: break if graph is not None: computation_counters[module] = computation_counters.get(module, 0) + 1 module_graph = module.asGraph(computation_counters[module]) graph.subgraph(module_graph) touched = True if _progress: memory_watch.finish() printLine( "Memory usage changed during optimization of '%s': %s" % ( module.getFullName(), memory_watch.asStr() ) ) Plugins.considerImplicitImports(module, signal_change = signalChange) return touched or module.hasUnclearLocals() def optimizeShlibModule(module): # Pick up parent package if any. _attemptRecursion(module) # The tag set is global, so it can react to changes without context. # pylint: disable=W0603 global tag_set tag_set = TagSet() Plugins.considerImplicitImports(module, signal_change = signalChange) def areEmptyTraces(variable_traces): empty = True for variable_trace in variable_traces: if variable_trace.isAssignTrace(): empty = False break elif variable_trace.isInitTrace(): empty = False break elif variable_trace.isUninitTrace(): if variable_trace.getPrevious(): # A "del" statement can do this, and needs to prevent variable # from being removed. empty = False break elif variable_trace.hasDefiniteUsages(): # Checking definite is enough, the merges, we shall see # them as well. empty = False break elif variable_trace.isUnknownTrace(): if variable_trace.hasDefiniteUsages(): # Checking definite is enough, the merges, we shall see # them as well. empty = False break elif variable_trace.isMergeTrace(): if variable_trace.hasDefiniteUsages(): # Checking definite is enough, the merges, we shall see # them as well. empty = False break elif variable_trace.isEscaped(): assert False, variable_trace # If the value is escape, we still need to keep it for that # escape opportunity. This is only while that is not seen # as a definite usage. empty = False break else: assert False, variable_trace return empty def areReadOnlyTraces(variable_traces): read_only = True for variable_trace in variable_traces: if variable_trace.isAssignTrace(): read_only = False break elif variable_trace.isInitTrace(): read_only = False break return read_only def optimizeUnusedClosureVariables(function_body): for closure_variable in function_body.getClosureVariables(): # print "VAR", closure_variable # Need to take closure of if closure_variable.isParameterVariable() and \ function_body.isExpressionGeneratorObjectBody(): continue if function_body.hasFlag("has_super") and closure_variable.getName() in ("__class__", "self"): continue variable_traces = function_body.constraint_collection.getVariableTraces( variable = closure_variable ) empty = areEmptyTraces(variable_traces) if empty: signalChange( "var_usage", function_body.getSourceReference(), message = "Remove unused closure variable." ) function_body.removeClosureVariable(closure_variable) else: read_only = areReadOnlyTraces(variable_traces) if read_only: global_trace = VariableRegistry.getGlobalVariableTrace(closure_variable) if global_trace is not None: if not global_trace.hasWritesOutsideOf(function_body): function_body.demoteClosureVariable(closure_variable) signalChange( "var_usage", function_body.getSourceReference(), message = "Turn read-only usage of unassigned closure variable to local variable." ) def optimizeUnusedUserVariables(function_body): for local_variable in function_body.getUserLocalVariables(): variable_traces = function_body.constraint_collection.getVariableTraces( variable = local_variable ) empty = areEmptyTraces(variable_traces) if empty: function_body.removeUserVariable(local_variable) def optimizeUnusedTempVariables(provider): for temp_variable in provider.getTempVariables(): variable_traces = provider.constraint_collection.getVariableTraces( variable = temp_variable ) empty = areEmptyTraces(variable_traces) if empty: provider.removeTempVariable(temp_variable) def optimizeVariables(module): for function_body in module.getUsedFunctions(): constraint_collection = function_body.constraint_collection if constraint_collection.unclear_locals: continue optimizeUnusedUserVariables(function_body) optimizeUnusedClosureVariables(function_body) optimizeUnusedTempVariables(function_body) optimizeUnusedTempVariables(module) def optimize(): # This is somewhat complex with many cases, pylint: disable=R0912 # We maintain this globally to make it accessible, pylint: disable=W0603 global graph if Options.shouldCreateGraph(): try: from graphviz import Digraph # pylint: disable=F0401,I0021 graph = Digraph('G') except ImportError: warning("Cannot import graphviz module, no graphing capability.") while True: finished = True ModuleRegistry.startTraversal() while True: current_module = ModuleRegistry.nextModule() if current_module is None: break if _progress: printLine( """\ Optimizing module '{module_name}', {remaining:d} more modules to go \ after that. Memory usage {memory}:""".format( module_name = current_module.getFullName(), remaining = ModuleRegistry.remainingCount(), memory = Utils.getHumanReadableProcessMemoryUsage() ) ) if current_module.isPythonShlibModule(): optimizeShlibModule(current_module) else: changed = optimizePythonModule(current_module) if changed: finished = False # Unregister collection traces from now unused code. for current_module in ModuleRegistry.getDoneModules(): if not current_module.isPythonShlibModule(): for function in current_module.getUnusedFunctions(): VariableRegistry.updateFromCollection( old_collection = function.constraint_collection, new_collection = None ) function.constraint_collection = None if not VariableRegistry.complete: VariableRegistry.complete = True finished = False for current_module in ModuleRegistry.getDoneModules(): if not current_module.isPythonShlibModule(): optimizeVariables(current_module) if finished: break if graph is not None: graph.engine = "dot" graph.graph_attr["rankdir"] = "TB" graph.render("something.dot") printLine(graph.source) Nuitka-0.5.18.1/nuitka/optimizations/BuiltinOptimization.py0000644000175000017500000003611412651071040024225 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Optimizations of built-ins to built-in calls. """ import math import sys from nuitka.nodes.ParameterSpecs import ( ParameterSpec, TooManyArguments, matchCall ) from nuitka.PythonVersions import python_version class BuiltinParameterSpec(ParameterSpec): def __init__(self, name, arg_names, default_count, list_star_arg = None, dict_star_arg = None): ParameterSpec.__init__( self, name = name, normal_args = arg_names, list_star_arg = list_star_arg, dict_star_arg = dict_star_arg, default_count = default_count, kw_only_args = () ) self.builtin = __builtins__[name] def __repr__(self): return "" % self.name def getName(self): return self.name def isCompileTimeComputable(self, values): # By default, we make this dependent on the ability to compute the # arguments, which is of course a good start for most cases, so this # is for overloads, pylint: disable=R0201 for value in values: if value is not None and not value.isCompileTimeConstant(): return False return True def simulateCall(self, given_values): # Using star dict call for simulation and catch any exception as really # fatal, pylint: disable=W0703 try: given_normal_args = given_values[:len(self.normal_args)] if self.list_star_arg: given_list_star_args = given_values[len(self.normal_args)] else: given_list_star_args = None if self.dict_star_arg: given_dict_star_args = given_values[ -1 ] else: given_dict_star_args = None arg_dict = {} for arg_name, given_value in zip(self.normal_args, given_normal_args): assert type(given_value) not in (tuple, list), \ ("do not like a tuple %s" % (given_value,)) if given_value is not None: arg_dict[ arg_name ] = given_value.getCompileTimeConstant() if given_dict_star_args: for given_dict_star_arg in reversed(given_dict_star_args): arg_name = given_dict_star_arg.getKey().getCompileTimeConstant() arg_value = given_dict_star_arg.getValue().getCompileTimeConstant() arg_dict[arg_name] = arg_value except Exception as e: sys.exit("Fatal problem: %r" % e) if given_list_star_args: return self.builtin( *(value.getCompileTimeConstant() for value in given_list_star_args), **arg_dict ) else: return self.builtin(**arg_dict) class BuiltinParameterSpecNoKeywords(BuiltinParameterSpec): def allowsKeywords(self): return False def simulateCall(self, given_values): # Using star dict call for simulation and catch any exception as really fatal, # pylint: disable=W0703 try: if self.list_star_arg: given_list_star_arg = given_values[ len(self.normal_args) ] else: given_list_star_arg = None arg_list = [] refuse_more = False for _arg_name, given_value in zip(self.normal_args, given_values): assert type(given_value) not in (tuple, list), ("do not like tuple %s" % (given_value,)) if given_value is not None: if not refuse_more: arg_list.append(given_value.getCompileTimeConstant()) else: assert False else: refuse_more = True if given_list_star_arg is not None: arg_list += [ value.getCompileTimeConstant() for value in given_list_star_arg ] except Exception as e: print >> sys.stderr, "Fatal error: ", import traceback traceback.print_exc() sys.exit(repr(e)) return self.builtin(*arg_list) class BuiltinParameterSpecExceptions(BuiltinParameterSpec): def __init__(self, exception_name, default_count): # TODO: Parameter default_count makes no sense for exceptions probably. BuiltinParameterSpec.__init__( self, name = exception_name, arg_names = (), default_count = default_count, list_star_arg = "args" ) def allowsKeywords(self): return False def getKeywordRefusalText(self): return "exceptions.%s does not take keyword arguments" % self.name def getCallableName(self): return "exceptions." + self.getName() def makeBuiltinParameterSpec(exception_name): if exception_name == "ImportError" and python_version >= 330: # TODO: Create this beast, needs keyword only arguments to be supported, # currently user of this function must take care to not have them. pass return BuiltinParameterSpecExceptions( exception_name = exception_name, default_count = 0 ) builtin_int_spec = BuiltinParameterSpec("int", ('x', "base"), 2) # These builtins are only available for Python2 if python_version < 300: builtin_long_spec = BuiltinParameterSpec( "long", ('x', "base"), 2 ) builtin_execfile_spec = BuiltinParameterSpecNoKeywords( "execfile", ("filename", "globals", "locals"), 2 ) builtin_unicode_spec = BuiltinParameterSpec( "unicode", ("string", "encoding", "errors"), 3 ) builtin_xrange_spec = BuiltinParameterSpec( "xrange", ("start", "stop", "step"), 2 ) builtin_bool_spec = BuiltinParameterSpec("bool", ('x',), 1) builtin_float_spec = BuiltinParameterSpec("float", ('x',), 1) builtin_complex_spec = BuiltinParameterSpec("complex", ("real", "imag"), 2) # This built-in have variable parameters for Python2/3 if python_version < 300: builtin_str_spec = BuiltinParameterSpec("str", ("object",), 1) else: builtin_str_spec = BuiltinParameterSpec("str", ("object", "encoding", "errors"), 3) builtin_len_spec = BuiltinParameterSpecNoKeywords("len", ("object",), 0) builtin_dict_spec = BuiltinParameterSpec("dict", (), 0, "list_args", "dict_args") builtin_len_spec = BuiltinParameterSpecNoKeywords("len", ("object",), 0) builtin_tuple_spec = BuiltinParameterSpec("tuple", ("sequence",), 1) builtin_list_spec = BuiltinParameterSpec("list", ("sequence",), 1) builtin_set_spec = BuiltinParameterSpecNoKeywords("set", ("iterable",), 1) builtin_import_spec = BuiltinParameterSpec("__import__", ("name", "globals", "locals", "fromlist", "level"), 4) builtin_open_spec = BuiltinParameterSpec("open", ("name", "mode", "buffering"), 3) builtin_chr_spec = BuiltinParameterSpecNoKeywords("chr", ('i',), 0) builtin_ord_spec = BuiltinParameterSpecNoKeywords("ord", ('c',), 0) builtin_bin_spec = BuiltinParameterSpecNoKeywords("bin", ("number",), 0) builtin_oct_spec = BuiltinParameterSpecNoKeywords("oct", ("number",), 0) builtin_hex_spec = BuiltinParameterSpecNoKeywords("hex", ("number",), 0) builtin_id_spec = BuiltinParameterSpecNoKeywords("id", ("object",), 0) builtin_repr_spec = BuiltinParameterSpecNoKeywords("repr", ("object",), 0) builtin_dir_spec = BuiltinParameterSpecNoKeywords("dir", ("object",), 1) builtin_vars_spec = BuiltinParameterSpecNoKeywords("vars", ("object",), 1) builtin_locals_spec = BuiltinParameterSpecNoKeywords("locals", (), 0) builtin_globals_spec = BuiltinParameterSpecNoKeywords("globals", (), 0) builtin_eval_spec = BuiltinParameterSpecNoKeywords("eval", ("source", "globals", "locals"), 2) if python_version < 300: builtin_compile_spec = BuiltinParameterSpec( "compile", ("source", "filename", "mode", "flags", "dont_inherit"), 2 ) else: builtin_compile_spec = BuiltinParameterSpec( "compile", ("source", "filename", "mode", "flags", "dont_inherit", "optimize"), 3 ) if python_version >= 300: builtin_exec_spec = BuiltinParameterSpecNoKeywords( "exec", ("source", "globals", "locals"), 2 ) # Note: Iter in fact names its first argument if the default applies # "collection", fixed up in a wrapper. builtin_iter_spec = BuiltinParameterSpecNoKeywords("iter", ("callable", "sentinel"), 1) builtin_next_spec = BuiltinParameterSpecNoKeywords("next", ("iterator", "default"), 1) # Note: type with 1 and type with 3 arguments are too different. builtin_type1_spec = BuiltinParameterSpecNoKeywords("type", ("object",), 0) builtin_type3_spec = BuiltinParameterSpecNoKeywords("type", ("name", "bases", "dict"), 0) builtin_super_spec = BuiltinParameterSpecNoKeywords("super", ("type", "object"), 1 if python_version < 300 else 2) builtin_hasattr_spec = BuiltinParameterSpecNoKeywords("hasattr", ("object", "name"), 0) builtin_getattr_spec = BuiltinParameterSpecNoKeywords("getattr", ("object", "name", "default"), 1) builtin_setattr_spec = BuiltinParameterSpecNoKeywords("setattr", ("object", "name", "value"), 0) builtin_isinstance_spec = BuiltinParameterSpecNoKeywords("isinstance", ("instance", "classes"), 0) builtin_bytearray_spec = BuiltinParameterSpecNoKeywords("bytearray", ("iterable_of_ints",), 1) # Beware: One argument defines stop, not start. builtin_slice_spec = BuiltinParameterSpecNoKeywords("slice", ("start", "stop", "step"), 2) builtin_hash_spec = BuiltinParameterSpecNoKeywords("hash", ("object",), 0) class BuiltinRangeSpec(BuiltinParameterSpecNoKeywords): def __init__(self, *args): BuiltinParameterSpecNoKeywords.__init__(self, *args) def isCompileTimeComputable(self, values): # For ranges, we need have many cases that can prevent the ability # to pre-compute, pylint: disable=R0911,R0912 result = BuiltinParameterSpecNoKeywords.isCompileTimeComputable( self, values = values ) if result: arg_count = len(values) if arg_count == 1: low = values[0] # If it's not a number constant, we can compute the exception # that will be raised. if not low.isNumberConstant(): return True return low.getConstant() < 256 elif arg_count == 2: low, high = values # If it's not a number constant, we can compute the exception # that will be raised. if not low.isNumberConstant() or not high.isNumberConstant(): return True return high.getConstant() - low.getConstant() < 256 elif arg_count == 3: low, high, step = values if not low.isNumberConstant() or \ not high.isNumberConstant() or \ not step.isNumberConstant(): return True low = low.getConstant() high = high.getConstant() step = step.getConstant() # It's going to give a ZeroDivisionError in this case. if step == 0: return True if low < high: if step < 0: return True else: return math.ceil(float(high - low) / step) < 256 else: if step > 0: return True else: return math.ceil(float(high - low) / step) < 256 else: assert False else: return False builtin_range_spec = BuiltinRangeSpec("range", ("start", "stop", "step"), 2) def extractBuiltinArgs(node, builtin_spec, builtin_class, empty_special_class = None): try: kw = node.getCallKw() # TODO: Could check for too many / too few, even if they are unknown, we # might raise that error, but that need not be optimized immediately. if kw is not None: if not kw.isMappingWithConstantStringKeys(): return None pairs = kw.getMappingStringKeyPairs() if pairs and not builtin_spec.allowsKeywords(): raise TooManyArguments( TypeError(builtin_spec.getKeywordRefusalText()) ) else: pairs = () args = node.getCallArgs() if args: if not args.canPredictIterationValues(): return None positional = args.getIterationValues() else: positional = () if not positional and not pairs and empty_special_class is not None: return empty_special_class(source_ref = node.getSourceReference()) args_dict = matchCall( func_name = builtin_spec.getName(), args = builtin_spec.getArgumentNames(), star_list_arg = builtin_spec.getStarListArgumentName(), star_dict_arg = builtin_spec.getStarDictArgumentName(), num_defaults = builtin_spec.getDefaultCount(), positional = positional, pairs = pairs ) except TooManyArguments as e: from nuitka.nodes.NodeMakingHelpers import ( makeRaiseExceptionReplacementExpressionFromInstance, wrapExpressionWithSideEffects ) return wrapExpressionWithSideEffects( new_node = makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = e.getRealException() ), old_node = node, side_effects = node.extractPreCallSideEffects() ) args_list = [] for argument_name in builtin_spec.getArgumentNames(): args_list.append(args_dict[argument_name]) if builtin_spec.getStarListArgumentName() is not None: args_list.append(args_dict[builtin_spec.getStarListArgumentName()]) if builtin_spec.getStarDictArgumentName() is not None: args_list.append(args_dict[builtin_spec.getStarDictArgumentName()]) # Using list reference for passing the arguments without names, result = builtin_class( *args_list, source_ref = node.getSourceReference() ) result.setCompatibleSourceReference(node.getCompatibleSourceReference()) return result Nuitka-0.5.18.1/nuitka/optimizations/VariableTraces.py0000644000175000017500000003027112651072046023104 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Variable trace objects. Variable traces indicate the flow of variables and merges their versions for the SSA (Single State Assignment) form being used in Nuitka. Variable version can start as: * Unknown (maybe initialized, maybe not, we cannot know) * Uninit (definitely not initialized, first version, or after "del" statement) * Init (definitely initialized, e.g. parameter variables) * Merge (result of diverged code paths) """ from logging import debug from nuitka.utils import InstanceCounters class VariableTraceBase: # We are going to have many instance attributes, pylint: disable=R0902 @InstanceCounters.counted_init def __init__(self, variable, version, previous): self.variable = variable self.version = version # Definite usage indicator. self.usage_count = 0 # Potential usages indicator that an assignment value may be used. self.has_potential_usages = False # If False, this indicates the trace has no explicit releases. self.has_releases = False # If False, this indicates, the variable name needs to be assigned. self.has_name_usages = False # If False, this indicates that the value is not yet escaped. self.is_escaped = False # Previous trace this is replacing. self.previous = previous __del__ = InstanceCounters.counted_del() def getVariable(self): return self.variable def getVersion(self): return self.version def addUsage(self): self.usage_count += 1 def addPotentialUsage(self): self.has_potential_usages = True def addRelease(self): self.has_releases = True def addNameUsage(self): self.usage_count += 1 self.has_name_usages = True def onValueEscape(self): self.is_escaped = True def isEscaped(self): return self.is_escaped def hasDefiniteUsages(self): return self.usage_count > 0 def getDefiniteUsages(self): return self.usage_count def hasPotentialUsages(self): return self.has_potential_usages def hasNameUsages(self): return self.has_name_usages def getPrevious(self): return self.previous @staticmethod def isAssignTrace(): return False @staticmethod def isUninitTrace(): return False @staticmethod def isInitTrace(): return False @staticmethod def isUnknownTrace(): return False @staticmethod def isMergeTrace(): return False def mustHaveValue(self): # TODO: Temporarily disable far reaching of assumptions, until value # escaping can be trusted. if self.variable.isModuleVariable() or \ self.variable.isMaybeLocalVariable() or \ self.variable.isSharedTechnically(): return False # Merge traces have this overloaded. return self.isInitTrace() or self.isAssignTrace() def mustNotHaveValue(self): if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically(): return False return self.isUninitTrace() def getReplacementNode(self, usage): # Virtual method, pylint: disable=R0201,W0613 return None class VariableTraceUninit(VariableTraceBase): def __init__(self, variable, version, previous): VariableTraceBase.__init__( self, variable = variable, version = version, previous = previous ) def __repr__(self): return "".format( variable = self.variable, version = self.version ) @staticmethod def isUninitTrace(): return True def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug(" Starts out uninitialized") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") if self.has_releases: debug(" -> has released") class VariableTraceInit(VariableTraceBase): def __init__(self, variable, version): VariableTraceBase.__init__( self, variable = variable, version = version, previous = None ) def __repr__(self): return "".format( variable = self.variable, version = self.version ) def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug(" Starts initialized") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") if self.has_releases: debug(" -> has released") @staticmethod def isInitTrace(): return True class VariableTraceUnknown(VariableTraceBase): def __init__(self, variable, version, previous): VariableTraceBase.__init__( self, variable = variable, version = version, previous = previous ) def __repr__(self): return "".format( variable = self.variable, version = self.version ) def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug(" Starts unknown") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") if self.has_releases: debug(" -> has released") @staticmethod def isUnknownTrace(): return True def addUsage(self): self.usage_count += 1 if self.previous is not None: self.previous.addPotentialUsage() def addNameUsage(self): self.addUsage() if self.previous is not None: self.previous.addNameUsage() def addPotentialUsage(self): old = self.has_potential_usages if not old: self.has_potential_usages = True if self.previous is not None: self.previous.addPotentialUsage() class VariableTraceAssign(VariableTraceBase): def __init__(self, assign_node, variable, version, previous): VariableTraceBase.__init__( self, variable = variable, version = version, previous = previous ) self.assign_node = assign_node self.replace_it = None def __repr__(self): return """\ """.format( variable = self.variable, version = self.version, source_ref = self.assign_node.getSourceReference().getAsString() ) def dump(self): debug("Trace of %s %d:", self.variable, self.version) debug(" Starts assigned") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") if self.has_releases: debug(" -> has released") @staticmethod def isAssignTrace(): return True def getAssignNode(self): return self.assign_node def setReplacementNode(self, replacement): self.replace_it = replacement def getReplacementNode(self, usage): if self.replace_it is not None: return self.replace_it(usage) else: return None class VariableTraceMerge(VariableTraceBase): """ Merge of two or more traces. Happens at the end of conditional blocks. This is "phi" in SSA theory. Also used for merging multiple "return", "break" or "continue" exits. """ def __init__(self, variable, version, traces): VariableTraceBase.__init__( self, variable = variable, version = version, previous = tuple(traces) ) def __repr__(self): return """\ """.format( variable = self.variable, version = self.version, previous = tuple(previous.getVersion() for previous in self.previous) ) @staticmethod def isMergeTrace(): return True def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug( " Merge of %s", " <-> ".join(self.previous), ) def mustHaveValue(self): # TODO: Temporarily disable far reaching of assumptions, until value # escaping can be trusted. if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically(): return False for previous in self.previous: if not previous.isInitTrace() and not previous.isAssignTrace(): return False return True def mustNotHaveValue(self): if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically(): return False for previous in self.previous: if not previous.isUninitTrace(): return False return True def addUsage(self): self.usage_count += 1 for previous in self.previous: previous.addPotentialUsage() def addNameUsage(self): self.usage_count += 1 for previous in self.previous: previous.addPotentialUsage() previous.addNameUsage() def addPotentialUsage(self): old = self.has_potential_usages if not old: self.has_potential_usages = True for previous in self.previous: previous.addPotentialUsage() class VariableTraceLoopMerge(VariableTraceBase): """ Merge of loop wrap around with loop start value. Happens at the start of loop blocks. This is for loop closed SSA, to make it clear, that the entered value, cannot be trusted inside the loop. They will start out with just one previous, and later be updated with all of the variable versions at loop continue times. . """ def __init__(self, variable, version, previous): VariableTraceBase.__init__( self, variable = variable, version = version, previous = previous ) self.loop_finished = False previous.addPotentialUsage() def hasDefiniteUsages(self): if not self.loop_finished: return True return self.usage_count > 0 def hasPotentialUsages(self): if not self.loop_finished: return True return self.has_potential_usages def hasNameUsages(self): if not self.loop_finished: return True return self.has_name_usages def getPrevious(self): assert self.loop_finished return self.previous @staticmethod def isMergeTrace(): return True def addLoopContinueTraces(self, continue_traces): self.previous.addPotentialUsage() for continue_trace in continue_traces: continue_trace.addPotentialUsage() self.previous = (self.previous,) + tuple(continue_traces) Nuitka-0.5.18.1/nuitka/optimizations/FunctionInlining.py0000644000175000017500000000607112651071040023464 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ In-lining of functions. Done by assigning the argument values to variables, and producing an outline from the in-lined function. """ from nuitka.nodes.AssignNodes import StatementAssignmentVariable from nuitka.nodes.NodeMakingHelpers import makeVariableTargetRefNode from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.tree.Extractions import updateVariableUsage from nuitka.tree.Helpers import makeStatementsSequence def convertFunctionCallToOutline(provider, function_ref, values): # This has got to have pretty man details, pylint: disable=R0914 function_body = function_ref.getFunctionBody() # TODO: Use the call location source_ref = function_body.getSourceReference() outline_body = ExpressionOutlineBody( provider = provider, name = "inline", source_ref = source_ref ) clone = function_body.getBody().makeClone() temp_scope = outline_body.getOutlineTempScope() translation = {} for variable in function_body.getLocalVariables(): # TODO: Later we should be able to do that too. assert not variable.isSharedTechnically() new_variable = outline_body.allocateTempVariable( temp_scope = temp_scope, name = variable.getName() ) # TODO: Lets update all at once maybe, it would take less visits. updateVariableUsage( clone, old_variable = variable, new_variable = new_variable ) translation[variable.getName()] = new_variable statements = [] argument_names = function_body.getParameters().getParameterNames() assert len(argument_names) == len(values), (argument_names, values) for argument_name, value in zip(argument_names, values): statements.append( StatementAssignmentVariable( variable_ref = makeVariableTargetRefNode( variable = translation[argument_name], source_ref = source_ref ), source = value, source_ref = source_ref, ) ) body = makeStatementsSequence( statements = (statements, clone), allow_none = False, source_ref = source_ref ) outline_body.setBody(body) return outline_body Nuitka-0.5.18.1/nuitka/optimizations/__init__.py0000644000175000017500000000150112651071040021737 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/TreeXML.py0000644000175000017500000000464412651071040016562 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ XML node tree handling Means to create XML elements from Nuitka tree nodes and to convert the XML tree to ASCII or output it. """ from nuitka.PythonVersions import python_version from . import Tracing def indent(elem, level = 0, more_sibs = False): i = '\n' if level: i += (level-1) * " " num_kids = len(elem) if num_kids: if not elem.text or not elem.text.strip(): elem.text = i + " " if level: elem.text += " " count = 0 for kid in elem: indent(kid, level+1, count < num_kids - 1) count += 1 if not elem.tail or not elem.tail.strip(): elem.tail = i if more_sibs: elem.tail += " " else: if level and (not elem.tail or not elem.tail.strip()): elem.tail = i if more_sibs: elem.tail += " " return elem try: import lxml.etree # pylint: disable=F0401,I0021 xml_module = lxml.etree Element = xml_module.Element xml_tostring = lambda tree: lxml.etree.tostring(tree, pretty_print = True) except ImportError: try: import xml.etree.ElementTree xml_module = xml.etree.ElementTree Element = xml.etree.ElementTree.Element xml_tostring = lambda tree: xml_module.tostring(indent(tree)) except ImportError: xml_module = None Element = None xml_tostring = None def toString(tree): result = xml_tostring(tree) if python_version >= 300: result = result.decode("utf-8") return result def dump(tree): value = toString(tree).rstrip() Tracing.printLine(value) Nuitka-0.5.18.1/nuitka/Constants.py0000644000175000017500000001712212651071040017251 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module for constants in Nuitka. This contains tools to compare, classify and test constants. """ import math from nuitka.PythonVersions import python_version from .__past__ import iterItems, long, unicode # pylint: disable=W0622 from .Builtins import builtin_anon_names NoneType = type(None) def compareConstants(a, b): # Many many cases to deal with, pylint: disable=R0911,R0912 # Supposed fast path for comparison. if type(a) is not type(b): return False # Now it's either not the same, or it is a container that contains NaN or it # is a complex or float that is NaN, the other cases can use == at the end. if type(a) is complex: return compareConstants(a.imag, b.imag) and \ compareConstants(a.real, b.real) if type(a) is float: # Check sign first, -0.0 is not 0.0, or -nan is not nan, it has a # different sign for a start. if math.copysign(1.0, a) != math.copysign(1.0, b): return False if math.isnan(a) and math.isnan(b): return True return a == b if type(a) in (tuple, list): if len(a) != len(b): return False for ea, eb in zip(a, b): if not compareConstants(ea, eb): return False return True if type(a) is dict: if len(a) != len(b): return False for ea1, ea2 in iterItems(a): for eb1, eb2 in iterItems(b): if compareConstants(ea1, eb1) and \ compareConstants(ea2, eb2): break else: return False return True if type(a) in (frozenset, set): if len(a) != len(b): return False for ea in a: if ea not in b: # Due to NaN values, we need to compare each set element with # all the other set to be really sure. for eb in b: if compareConstants(ea, eb): break else: return False return True if type(a) is range: return str(a) == str(b) # The NaN values of float and complex may let this fail, even if the # constants are built in the same way. return a == b # These built-in type references are kind of constant too. TODO: The list is # totally not complete. constant_builtin_types = ( int, str, float, list, tuple, set, dict, slice, complex ) if python_version >= 300: constant_builtin_types += ( range, bytes, ) else: constant_builtin_types += ( unicode, long, # This has no name in Python, but the natural one in C-API. builtin_anon_names["instance"] ) def isConstant(constant): # Too many cases and all return, that is how we do it here, # pylint: disable=R0911 constant_type = type(constant) if constant_type is dict: for key, value in iterItems(constant): if not isConstant(key): return False if not isConstant(value): return False return True elif constant_type in (tuple, list): for element_value in constant: if not isConstant(element_value): return False return True elif constant_type in (str, unicode, complex, int, long, bool, float, NoneType, range, bytes, set): return True elif constant in (Ellipsis, NoneType): return True elif constant_type is type: return constant in constant_builtin_types elif constant_type is slice: return True else: return False def isMutable(constant): """ Is a constant mutable That means a user of a reference to it, can modify it. Strings are a prime example of mutable, dictionaries are mutable. """ constant_type = type(constant) if constant_type in (str, unicode, complex, int, long, bool, float, NoneType, range, bytes, slice): return False elif constant_type in (dict, list, set): return True elif constant_type is tuple: for value in constant: if isMutable(value): return True return False elif constant is Ellipsis: return False elif constant in constant_builtin_types: return True else: assert False, repr(constant) def isHashable(constant): """ Is a constant hashable That means a user of a reference to it, can use it for dicts and set keys. This is distinct from mutable, there is one types that is not mutable, and still not hashable: slices. """ # Too many cases and all return, that is how we do it here, # pylint: disable=R0911 constant_type = type(constant) if constant_type in (str, unicode, complex, int, long, bool, float, NoneType, range, bytes): return True elif constant_type in (dict, list, set): return False elif constant_type is tuple: for value in constant: if not isHashable(value): return False return True elif constant is Ellipsis: return True elif constant in constant_builtin_types: return True elif constant_type is slice: return False else: assert False, constant_type def isIterableConstant(constant): return type(constant) in ( str, unicode, list, tuple, set, frozenset, dict, range, bytes ) def getConstantIterationLength(constant): assert isIterableConstant(constant) return len(constant) def isNumberConstant(constant): return type(constant) in (int, long, float, bool) def isIndexConstant(constant): return type(constant) in (int, long, bool) def createConstantDict(keys, values): # Create it proper size immediately. constant_value = dict.fromkeys( keys, None ) for key, value in zip(keys, values): constant_value[key] = value return constant_value def getConstantWeight(constant): constant_type = type(constant) if constant_type is dict: result = 0 for key, value in iterItems(constant): result += getConstantWeight(key) result += getConstantWeight(value) return result elif constant_type in (tuple, list, set, frozenset): result = 0 for element_value in constant: result += getConstantWeight(element_value) return result else: return 1 def isCompileTimeConstantValue(value): """ Determine if a value will be usable at compile time. """ # This needs to match code in makeCompileTimeConstantReplacementNode if isConstant(value): return True elif type(value) is type: return True else: return False Nuitka-0.5.18.1/nuitka/freezer/0000755000175000017500000000000012651072347016375 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/nuitka/freezer/BytecodeModuleFreezer.py0000644000175000017500000000472312651071040023171 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Freezer for bytecode compiled modules. Not real C compiled modules. This is including modules as bytecode and mostly intended for modules, where we know compiling it useless or does not make much sense, or for standalone mode to access modules during CPython library init that cannot be avoided. The level of compatibility for C compiled stuff is so high that this is not needed except for technical reasons. """ from logging import info from nuitka import Options from nuitka.codegen import ConstantCodes from nuitka.codegen.Indentation import indented from nuitka.codegen.templates. \ CodeTemplatesFreezer import template_frozen_modules from nuitka.ModuleRegistry import getUncompiledModules stream_data = ConstantCodes.stream_data def generateBytecodeFrozenCode(): frozen_defs = [] for uncompiled_module in getUncompiledModules(): module_name = uncompiled_module.getFullName() code_data = uncompiled_module.getByteCode() is_package = uncompiled_module.isPackage() size = len(code_data) # Packages are indicated with negative size. if is_package: size = -size frozen_defs.append( """\ {{ (char *)"{module_name}", (unsigned char *){data}, {size} }},""".format( module_name = module_name, data = stream_data.getStreamDataCode( value = code_data, fixed_size = True ), size = size ) ) if Options.isShowInclusion(): info("Embedded as frozen module '%s'.", module_name) return template_frozen_modules % { "frozen_modules" : indented(frozen_defs, 2) } Nuitka-0.5.18.1/nuitka/freezer/Standalone.py0000644000175000017500000007152212651071040021033 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Pack and copy files for standalone mode. This is still under heavy evolution, but expected to work for MacOS, Windows, and Linux. Patches for other platforms are very welcome. """ import os import shutil import subprocess import sys from logging import debug, info, warning import marshal from nuitka import Options, SourceCodeReferences, Tracing from nuitka.__past__ import iterItems from nuitka.codegen.ConstantCodes import needsPickleInit from nuitka.importing import ImportCache from nuitka.importing.StandardLibrary import ( getStandardLibraryPaths, isStandardLibraryPath ) from nuitka.nodes.ModuleNodes import ( PythonShlibModule, makeUncompiledPythonModule ) from nuitka.PythonVersions import python_version from nuitka.tree.SourceReading import readSourceCodeFromFilename from nuitka.utils import Utils from .DependsExe import getDependsExePath def loadCodeObjectData(precompiled_filename): # Ignoring magic numbers, etc. which we don't have to care for much as # CPython already checked them (would have rejected it otherwise). return open(precompiled_filename, "rb").read()[8:] module_names = set() def _detectedPrecompiledFile(filename, module_name, result, user_provided): if filename.endswith(".pyc"): if Utils.isFile(filename[:-1]): return _detectedSourceFile( filename = filename[:-1], module_name = module_name, result = result, user_provided = user_provided ) if module_name in module_names: return debug( "Freezing module '%s' (from '%s').", module_name, filename ) result.append( makeUncompiledPythonModule( module_name = module_name, bytecode = loadCodeObjectData( precompiled_filename = filename ), is_package = "__init__" in filename, filename = filename, user_provided = user_provided ) ) module_names.add(module_name) def _detectedSourceFile(filename, module_name, result, user_provided): if module_name in module_names: return if module_name == "collections.abc": _detectedSourceFile( filename = filename, module_name = "_collections_abc", result = result, user_provided = user_provided ) source_code = readSourceCodeFromFilename(module_name, filename) if module_name == "site": if source_code.startswith("def ") or source_code.startswith("class "): source_code = '\n' + source_code source_code = """\ __file__ = (__nuitka_binary_dir + '%s%s') if '__nuitka_binary_dir' in dict(__builtins__ ) else '';%s""" % ( os.path.sep, Utils.basename(filename), source_code ) debug( "Freezing module '%s' (from '%s').", module_name, filename ) result.append( makeUncompiledPythonModule( module_name = module_name, bytecode = marshal.dumps( compile(source_code, filename, "exec") ), is_package = Utils.basename(filename) == "__init__.py", filename = filename, user_provided = user_provided ) ) module_names.add(module_name) def _detectedShlibFile(filename, module_name): # That is not a shared library, but looks like one. if module_name == "__main__": return from nuitka import ModuleRegistry if ModuleRegistry.hasRootModule(module_name): return parts = module_name.split('.') if len(parts) == 1: package_name = None name = module_name else: package_name = '.'.join(parts[:-1]) name = parts[-1] source_ref = SourceCodeReferences.fromFilename( filename = filename ) shlib_module = PythonShlibModule( name = name, package_name = package_name, source_ref = source_ref ) ModuleRegistry.addRootModule(shlib_module) ImportCache.addImportedModule(shlib_module) module_names.add(module_name) def _detectImports(command, user_provided): # This is pretty complicated stuff, with variants to deal with. # pylint: disable=R0912,R0914,R0915 # Print statements for stuff to show, the modules loaded. if python_version >= 300: command += '\nimport sys\nprint("\\n".join(sorted("import " + module.__name__ + " # sourcefile " + ' \ 'module.__file__ for module in sys.modules.values() if hasattr(module, "__file__") and ' \ 'module.__file__ != "")), file = sys.stderr)' # do not read it reduced_path = [ path_element for path_element in sys.path if not Utils.areSamePaths( path_element, '.' ) if not Utils.areSamePaths( path_element, Utils.dirname(sys.modules["__main__"].__file__) ) ] # Make sure the right import path (the one Nuitka binary is running with) # is used. command = ("import sys; sys.path = %s;" % repr(reduced_path)) + command import tempfile tmp_file, tmp_filename = tempfile.mkstemp() try: if python_version >= 300: command = command.encode("ascii") os.write(tmp_file, command) os.close(tmp_file) process = subprocess.Popen( args = [sys.executable, "-s", "-S", "-v", tmp_filename], stdout = subprocess.PIPE, stderr = subprocess.PIPE, ) _stdout, stderr = process.communicate() finally: os.unlink(tmp_filename) # Don't let errors here go unnoticed. if process.returncode != 0: warning("There is a problem with detecting imports, CPython said:") for line in stderr.split(b"\n"): Tracing.printLine(line) sys.exit("Error, please report the issue with above output.") result = [] debug("Detecting imports:") for line in stderr.replace(b"\r", b"").split(b"\n"): if line.startswith(b"import "): # print(line) parts = line.split(b" # ", 2) module_name = parts[0].split(b" ", 2)[1] origin = parts[1].split()[0] if python_version >= 300: module_name = module_name.decode("utf-8") if origin == b"precompiled": # This is a ".pyc" file that was imported, even before we have a # chance to do anything, we need to preserve it. filename = parts[1][len(b"precompiled from "):] if python_version >= 300: filename = filename.decode("utf-8") # Do not leave standard library when freezing. if not isStandardLibraryPath(filename): continue _detectedPrecompiledFile( filename = filename, module_name = module_name, result = result, user_provided = user_provided ) elif origin == b"sourcefile": filename = parts[1][len(b"sourcefile "):] if python_version >= 300: filename = filename.decode("utf-8") # Do not leave standard library when freezing. if not isStandardLibraryPath(filename): continue if filename.endswith(".py"): _detectedSourceFile( filename = filename, module_name = module_name, result = result, user_provided = user_provided ) elif not filename.endswith(""): _detectedShlibFile( filename = filename, module_name = module_name ) elif origin == b"dynamically": # Shared library in early load, happens on RPM based systems and # or self compiled Python installations. filename = parts[1][len(b"dynamically loaded from "):] if python_version >= 300: filename = filename.decode("utf-8") # Do not leave standard library when freezing. if not isStandardLibraryPath(filename): continue _detectedShlibFile( filename = filename, module_name = module_name ) return result def detectLateImports(): command = "" # When we are using pickle internally (for some hard constant cases we do), # we need to make sure it will be available as well. if needsPickleInit(): command += "import {pickle};".format( pickle = "pickle" if python_version >= 300 else "cPickle" ) # For Python3 we patch inspect without knowing if it is used. if python_version >= 300: command += "import inspect;" if command: result = _detectImports(command, True) debug("Finished detecting late imports.") return result else: return () # Some modules we want to blacklist. ignore_modules = [ "__main__.py", "__init__.py", "antigravity.py", ] if Utils.getOS() != "Windows": ignore_modules.append("wintypes.py") ignore_modules.append("cp65001.py") def scanStandardLibraryPath(stdlib_dir): # There is a lot of black-listing here, done in branches, so there # is many of them, but that's acceptable, pylint: disable=R0912 for root, dirs, filenames in os.walk(stdlib_dir): import_path = root[len(stdlib_dir):].strip("/\\") import_path = import_path.replace('\\', '.').replace('/','.') if import_path == "": if "site-packages" in dirs: dirs.remove("site-packages") if "dist-packages" in dirs: dirs.remove("dist-packages") if "test" in dirs: dirs.remove("test") if "idlelib" in dirs: dirs.remove("idlelib") if "turtledemo" in dirs: dirs.remove("turtledemo") if "ensurepip" in filenames: filenames.remove("ensurepip") if "ensurepip" in dirs: dirs.remove("ensurepip") if import_path in ("tkinter", "importlib", "ctypes", "unittest", "sqlite3", "distutils"): if "test" in dirs: dirs.remove("test") if import_path == "lib2to3": if "tests" in dirs: dirs.remove("tests") if python_version >= 340 and Utils.getOS() == "Windows": if import_path == "multiprocessing": filenames.remove("popen_fork.py") filenames.remove("popen_forkserver.py") filenames.remove("popen_spawn_posix.py") if Utils.getOS() == "NetBSD": if import_path == "xml.sax": filenames.remove("expatreader.py") for filename in filenames: if filename.endswith(".py") and filename not in ignore_modules: module_name = filename[:-3] if import_path == "": yield module_name else: yield import_path + '.' + module_name if python_version >= 300: if "__pycache__" in dirs: dirs.remove("__pycache__") for dirname in dirs: if import_path == "": yield dirname else: yield import_path + '.' + dirname def detectEarlyImports(): # TODO: Should recursively include all of encodings module. import_code = "import encodings.utf_8;import encodings.ascii;import encodings.idna;" if Utils.getOS() == "Windows": import_code += "import encodings.mbcs;import encodings.cp437;" # String method hex depends on it. if python_version < 300: import_code += "import encodings.hex_codec;" import_code += "import locale;" result = _detectImports(import_code, False) if Options.shallFreezeAllStdlib(): stdlib_modules = set() # Scan the standard library paths (multiple in case of virtualenv. for stdlib_dir in getStandardLibraryPaths(): for module_name in scanStandardLibraryPath(stdlib_dir): stdlib_modules.add(module_name) import_code = "imports = " + repr(sorted(stdlib_modules)) + '\n'\ "for imp in imports:\n" \ " try:\n" \ " __import__(imp)\n" \ " except (ImportError, SyntaxError):\n" \ " pass\n" early_names = [ module.getFullName() for module in result ] result += [ module for module in _detectImports(import_code, False) if module.getFullName() not in early_names ] debug("Finished detecting early imports.") return result def _detectBinaryPathDLLsLinuxBSD(binary_filename): # Ask "ldd" about the libraries being used by the created binary, these # are the ones that interest us. result = set() process = subprocess.Popen( args = [ "ldd", binary_filename ], stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout, _stderr = process.communicate() for line in stdout.split(b"\n"): if not line: continue if b"=>" not in line: continue part = line.split(b" => ", 2)[1] if b"(" in part: filename = part[:part.rfind(b"(")-1] else: filename = part if not filename: continue if python_version >= 300: filename = filename.decode("utf-8") # Sometimes might use stuff not found. if filename == "not found": continue result.add(filename) return result def _detectBinaryPathDLLsMacOS(binary_filename): result = set() process = subprocess.Popen( args = [ "otool", "-L", binary_filename ], stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout, _stderr = process.communicate() system_paths = (b"/usr/lib/", b"/System/Library/Frameworks/") for line in stdout.split(b"\n")[2:]: if not line: continue filename = line.split(b" (")[0].strip() stop = False for w in system_paths: if filename.startswith(w): stop = True break if not stop: if python_version >= 300: filename = filename.decode("utf-8") # print "adding", filename result.add(filename) return result def _makeBinaryPathPathDLLSearchEnv(package_name): # Put the PYTHONPATH into the system "PATH", DLLs frequently live in # the package directories. env = os.environ.copy() path = env.get("PATH","").split(os.pathsep) # Put the "Python.exe" first. At least for WinPython, they put the DLLs # there. path = [sys.prefix] + sys.path + path if package_name is not None: for element in sys.path: candidate = Utils.joinpath(element, package_name) if Utils.isDir(candidate): path.append(candidate) env["PATH"] = os.pathsep.join(path) return env def _detectBinaryPathDLLsWindows(binary_filename, package_name): result = set() depends_exe = getDependsExePath() # The search order by default prefers the system directory, where a # wrong "PythonXX.dll" might be living. with open(binary_filename + ".dwp", 'w') as dwp_file: dwp_file.write("""\ KnownDLLs SysPath AppDir 32BitSysDir 16BitSysDir OSDir AppPath SxS """) subprocess.call( ( depends_exe, "-c", "-ot%s" % binary_filename + ".depends", "-d:%s" % binary_filename + ".dwp", "-f1", "-pa1", "-ps1", binary_filename ), env = _makeBinaryPathPathDLLSearchEnv(package_name), ) inside = False first = False for line in open(binary_filename + ".depends"): if "| Module Dependency Tree |" in line: inside = True first = True continue if not inside: continue if "| Module List |" in line: break if ']' not in line: continue # Skip missing DLLs, apparently not needed anyway. if '?' in line[:line.find(']')]: continue # Skip DLLs that failed to load, apparently not needed anyway. if 'E' in line[:line.find(']')]: continue dll_filename = line[line.find(']')+2:-1] # The executable itself is of course exempted. We cannot check its path # because depends.exe mistreats unicode paths. if first: first = False continue assert Utils.isFile(dll_filename), dll_filename dll_name = Utils.basename(dll_filename).upper() # Win API can be assumed. if dll_name.startswith("API-MS-WIN-") or \ dll_name.startswith("EXT-MS-WIN-"): continue if dll_name in ("SHELL32.DLL", "USER32.DLL", "KERNEL32.DLL", "NTDLL.DLL", "NETUTILS.DLL", "LOGONCLI.DLL", "GDI32.DLL", "RPCRT4.DLL", "ADVAPI32.DLL", "SSPICLI.DLL", "SECUR32.DLL", "KERNELBASE.DLL", "WINBRAND.DLL", "DSROLE.DLL", "DNSAPI.DLL", "SAMCLI.DLL", "WKSCLI.DLL", "SAMLIB.DLL", "WLDAP32.DLL", "NTDSAPI.DLL", "CRYPTBASE.DLL", "W32TOPL", "WS2_32.DLL", "SPPC.DLL", "MSSIGN32.DLL", "CERTCLI.DLL", "WEBSERVICES.DLL", "AUTHZ.DLL", "CERTENROLL.DLL", "VAULTCLI.DLL", "REGAPI.DLL", "BROWCLI.DLL", "WINNSI.DLL", "DHCPCSVC6.DLL", "PCWUM.DLL", "CLBCATQ.DLL", "IMAGEHLP.DLL", "MSASN1.DLL", "DBGHELP.DLL", "DEVOBJ.DLL", "DRVSTORE.DLL", "CABINET.DLL", "SCECLI.DLL", "SPINF.DLL", "SPFILEQ.DLL", "GPAPI.DLL", "NETJOIN.DLL", "W32TOPL.DLL", "NETBIOS.DLL", "DXGI.DLL", "DWRITE.DLL", "D3D11.DLL", "WLANAPI.DLL", "WLANUTIL.DLL", "ONEX.DLL", "EAPPPRXY.DLL", "MFPLAT.DLL", "AVRT.DLL", "ELSCORE.DLL", "INETCOMM.DLL", "MSOERT2.DLL", "IEUI.DLL", "MSCTF.DLL", "MSFEEDS.DLL", "UIAUTOMATIONCORE.DLL", "PSAPI.DLL", "EFSADU.DLL", "MFC42U.DLL", "ODBC32.DLL", "OLEDLG.DLL", "NETAPI32.DLL", "LINKINFO.DLL", "DUI70.DLL", "ADVPACK.DLL", "NTSHRUI.DLL", "WINSPOOL.DRV", "EFSUTIL.DLL", "WINSCARD.DLL", "SHDOCVW.DLL", "IEFRAME.DLL", "D2D1.DLL", "GDIPLUS.DLL", "OCCACHE.DLL", "IEADVPACK.DLL", "MLANG.DLL", "MSI.DLL", "MSHTML.DLL", "COMDLG32.DLL", "PRINTUI.DLL", "PUIAPI.DLL", "ACLUI.DLL", "WTSAPI32.DLL", "FMS.DLL", "DFSCLI.DLL", "HLINK.DLL", "MSRATING.DLL", "PRNTVPT.DLL", "IMGUTIL.DLL", "MSLS31.DLL", "VERSION.DLL", "NORMALIZ.DLL", "IERTUTIL.DLL", "WININET.DLL", "WINTRUST.DLL", "XMLLITE.DLL", "APPHELP.DLL", "PROPSYS.DLL", "RSTRTMGR.DLL", "NCRYPT.DLL", "BCRYPT.DLL", "MMDEVAPI.DLL", "MSILTCFG.DLL", "DEVMGR.DLL", "DEVRTL.DLL", "NEWDEV.DLL", "VPNIKEAPI.DLL", "WINHTTP.DLL", "WEBIO.DLL", "NSI.DLL", "DHCPCSVC.DLL", "CRYPTUI.DLL", "ESENT.DLL", "DAVHLPR.DLL", "CSCAPI.DLL", "ATL.DLL", "OLEAUT32.DLL", "SRVCLI.DLL", "RASDLG.DLL", "MPRAPI.DLL", "RTUTILS.DLL", "RASMAN.DLL", "MPRMSG.DLL", "SLC.DLL", "CRYPTSP.DLL", "RASAPI32.DLL", "TAPI32.DLL", "EAPPCFG.DLL", "NDFAPI.DLL", "WDI.DLL", "COMCTL32.DLL", "UXTHEME.DLL", "IMM32.DLL", "OLEACC.DLL", "WINMM.DLL", "WINDOWSCODECS.DLL", "DWMAPI.DLL", "DUSER.DLL", "PROFAPI.DLL", "URLMON.DLL", "SHLWAPI.DLL", "LPK.DLL", "USP10.DLL", "CFGMGR32.DLL", "MSIMG32.DLL", "POWRPROF.DLL", "SETUPAPI.DLL", "WINSTA.DLL", "CRYPT32.DLL", "IPHLPAPI.DLL", "MPR.DLL", "CREDUI.DLL", "NETPLWIZ.DLL", "OLE32.DLL", "ACTIVEDS.DLL", "ADSLDPC.DLL", "USERENV.DLL", "APPREPAPI.DLL", "BCP47LANGS.DLL", "BCRYPTPRIMITIVES.DLL", "CERTCA.DLL", "CHARTV.DLL", "COMBASE.DLL", "COML2.DLL", "DCOMP.DLL", "DPAPI.DLL", "DSPARSE.DLL", "FECLIENT.DLL", "FIREWALLAPI.DLL", "FLTLIB.DLL", "MRMCORER.DLL", "NTASN1.DLL", "SECHOST.DLL", "SETTINGSYNCPOLICY.DLL", "SHCORE.DLL", "TBS.DLL", "TWINAPI.APPCORE.DLL", "TWINAPI.DLL", "VIRTDISK.DLL", "WEBSOCKET.DLL", "WEVTAPI.DLL", "WINMMBASE.DLL", "WMICLNT.DLL"): continue result.add( Utils.normcase(Utils.abspath(dll_filename)) ) Utils.deleteFile(binary_filename + ".depends", must_exist = True) Utils.deleteFile(binary_filename + ".dwp", must_exist = True) return result def detectBinaryDLLs(binary_filename, package_name): """ Detect the DLLs used by a binary. Using "ldd" (Linux), "depends.exe" (Windows), or "otool" (MacOS) the list of used DLLs is retrieved. """ if Utils.getOS() in ("Linux", "NetBSD"): # TODO: FreeBSD may work the same way, not tested. return _detectBinaryPathDLLsLinuxBSD( binary_filename = binary_filename ) elif Utils.getOS() == "Windows": return _detectBinaryPathDLLsWindows( binary_filename = binary_filename, package_name = package_name ) elif Utils.getOS() == "Darwin": return _detectBinaryPathDLLsMacOS( binary_filename = binary_filename ) else: # Support your platform above. assert False, Utils.getOS() def detectUsedDLLs(standalone_entry_points): result = {} for binary_filename, package_name in standalone_entry_points: used_dlls = detectBinaryDLLs( binary_filename = binary_filename, package_name = package_name ) for dll_filename in used_dlls: # We want these to be absolute paths. assert Utils.isAbsolutePath(dll_filename), dll_filename if dll_filename not in result: result[dll_filename] = [] result[dll_filename].append(binary_filename) return result def fixupBinaryDLLPaths(binary_filename, is_exe, dll_map): """ For MacOS, the binary needs to be told to use relative DLL paths """ # There may be nothing to do, in case there are no DLLs. if not dll_map: return command = [ "install_name_tool" ] for original_path, dist_path in dll_map: command += [ "-change", original_path, "@executable_path/" + dist_path, ] os.chmod(binary_filename, int("644", 8)) command.append(binary_filename) process = subprocess.Popen( args = command, stdout = subprocess.PIPE, stderr = subprocess.PIPE, ) _stdout, stderr = process.communicate() os.chmod(binary_filename, int("755" if is_exe else "444", 8)) # Don't let errors here go unnoticed. assert process.returncode == 0, stderr def removeSharedLibraryRPATH(filename): process = subprocess.Popen( ["readelf", "-d", filename], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) stdout, _stderr = process.communicate() retcode = process.poll() assert retcode == 0, filename for line in stdout.split(b"\n"): if b"RPATH" in line: if Options.isShowInclusion(): info("Removing 'RPATH' setting from '%s'.", filename) if not Utils.isExecutableCommand("chrpath"): sys.exit( """\ Error, needs 'chrpath' on your system, due to 'RPATH' settings in used shared libraries that need to be removed.""" ) os.chmod(filename, int("644", 8)) process = subprocess.Popen( ["chrpath", "-d", filename], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) process.communicate() retcode = process.poll() os.chmod(filename, int("444", 8)) assert retcode == 0, filename def copyUsedDLLs(dist_dir, standalone_entry_points): # This is terribly complex, because we check the list of used DLLs # trying to avoid duplicates, and detecting errors with them not # being binary identical, so we can report them. And then of course # we also need to handle OS specifics, pylint: disable=R0912 dll_map = [] used_dlls = detectUsedDLLs(standalone_entry_points) for dll_filename1, sources1 in tuple(iterItems(used_dlls)): for dll_filename2, sources2 in tuple(iterItems(used_dlls)): if dll_filename1 == dll_filename2: continue # Colliding basenames are an issue to us. if Utils.basename(dll_filename1) != Utils.basename(dll_filename2): continue # May already have been removed earlier if dll_filename1 not in used_dlls: continue if dll_filename2 not in used_dlls: continue dll_name = Utils.basename(dll_filename1) if Options.isShowInclusion(): info( """Colliding DLL names for %s, checking identity of \ '%s' <-> '%s'.""" % ( dll_name, dll_filename1, dll_filename2, ) ) # Check that if a DLL has the same name, if it's identical, # happens at least for OSC and Fedora 20. import filecmp if filecmp.cmp(dll_filename1, dll_filename2): del used_dlls[dll_filename2] continue # So we have conflicting DLLs, in which case we do not proceed. sys.exit( """Error, conflicting DLLs for '%s'. %s used by: %s different from %s used by %s""" % ( dll_name, dll_filename1, "\n ".join(sources1), dll_filename2, "\n ".join(sources2) ) ) for dll_filename, sources in iterItems(used_dlls): dll_name = Utils.basename(dll_filename) target_path = Utils.joinpath( dist_dir, dll_name ) shutil.copy( dll_filename, target_path ) dll_map.append( (dll_filename, dll_name) ) if Options.isShowInclusion(): info( "Included used shared library '%s' (used by %s)." % ( dll_filename, ", ".join(sources) ) ) if Utils.getOS() == "Darwin": # For MacOS, the binary and the DLLs needs to be changed to reflect # the relative DLL location in the ".dist" folder. for standalone_entry_point in standalone_entry_points: fixupBinaryDLLPaths( binary_filename = standalone_entry_point[0], is_exe = standalone_entry_point is standalone_entry_points[0], dll_map = dll_map ) for _original_path, dll_filename in dll_map: fixupBinaryDLLPaths( binary_filename = Utils.joinpath( dist_dir, dll_filename ), is_exe = False, dll_map = dll_map ) if Utils.getOS() == "Linux": # For Linux, the "rpath" of libraries may be an issue and must be # removed. for _original_path, dll_filename in dll_map: removeSharedLibraryRPATH( Utils.joinpath(dist_dir, dll_filename) ) for standalone_entry_point in standalone_entry_points[1:]: removeSharedLibraryRPATH( standalone_entry_point[0] ) Nuitka-0.5.18.1/nuitka/freezer/__init__.py0000644000175000017500000000150112651071040020470 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.18.1/nuitka/freezer/DependsExe.py0000644000175000017500000000666012651071040020770 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Interface to depends.exe on Windows. We use depends.exe to investigate needed DLLs of Python DLLs. """ import os import sys from logging import info from nuitka import Tracing from nuitka.__past__ import raw_input, urlretrieve # pylint: disable=W0622 from nuitka.utils import Utils def getDependsExePath(): """ Return the path of depends.exe (for Windows). Will prompt the user to download if not already cached in AppData directory for Nuitka. """ if Utils.getArchitecture() == "x86": depends_url = "http://dependencywalker.com/depends22_x86.zip" else: depends_url = "http://dependencywalker.com/depends22_x64.zip" if "APPDATA" not in os.environ: sys.exit("Error, standalone mode cannot find 'APPDATA' environment.") nuitka_app_dir = Utils.joinpath(os.environ["APPDATA"], "nuitka") if not Utils.isDir(nuitka_app_dir): Utils.makePath(nuitka_app_dir) nuitka_depends_zip = Utils.joinpath( nuitka_app_dir, Utils.basename(depends_url) ) if not Utils.isFile(nuitka_depends_zip): Tracing.printLine("""\ Nuitka will make use of Dependency Walker (http://dependencywalker.com) tool to analyze the dependencies of Python extension modules. Is it OK to download and put it in APPDATA (no installer needed, cached, one time question).""") reply = raw_input("Proceed and download? [Yes]/No ") if reply.lower() in ("no", 'n'): sys.exit("Nuitka does not work in --standalone on Windows without.") info("Downloading '%s'" % depends_url) urlretrieve( depends_url, nuitka_depends_zip ) nuitka_depends_dir = Utils.joinpath( nuitka_app_dir, Utils.getArchitecture() ) if not Utils.isDir(nuitka_depends_dir): os.makedirs(nuitka_depends_dir) depends_exe = os.path.join( nuitka_depends_dir, "depends.exe" ) if not Utils.isFile(depends_exe): info("Extracting to '%s'" % depends_exe) import zipfile try: depends_zip = zipfile.ZipFile(nuitka_depends_zip) depends_zip.extractall(nuitka_depends_dir) except Exception: # Catching anything zip throws, pylint:disable=W0703 info("Problem with the downloaded zip file, deleting it.") Utils.deleteFile(depends_exe, must_exist = False) Utils.deleteFile(nuitka_depends_zip, must_exist = True) sys.exit( "Error, need '%s' as extracted from '%s'." % ( depends_exe, depends_url ) ) assert Utils.isFile(depends_exe) return depends_exe Nuitka-0.5.18.1/images/0000755000175000017500000000000012651072347014705 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/images/Nuitka-Logo-Symbol.png0000644000175000017500000001054412651070406021005 0ustar hayenhayen00000000000000‰PNG  IHDRÆÇBÇ,øgAMA± üasRGB®Îé cHRMz&€„ú€èu0ê`:˜pœºQ< pHYsHHFÉk>bKGDùC»YIDATxÚíÝÛoõðïovv×ÆŽq(—%ÁÄ€Jqm!y€´i@ñ¥‘ÚªÞ<ö¡T¨U[©·'úTU«ÁVˆ–[…Z¨š‡¶`lǪJLl !NìÓo{wî;ç·;“÷2?ïþ>{Ι™ýA˜™™Ùè8Îv[ Ø  `Œ5ñö‚‡ýÔR?^dÍOkÿ]ÿÑ1À×®­6=¶Ç=Œ®Ò‘°g÷yÞÓ?JÈýÏÆÉÚ…ø_m S^nÀG–¼cŒ™ðr©TznÓ¦M$y"7Þ›#æàÁƒÛEäaÛ”9åãÅü¢`rÎK‰Bš„H  ìßǟ羋õåÉq~Ø$OÃíŸ`°“`8.¨ÿû3"r+€ï.//×fgg_ðØÀÀÀ3ƯeÓÓÓ7¿õÖ[¿‘›9ÍíE\¾÷†ZCqfDI¶Ýž(¸À½³³³oÌÌÌ<488øJ”° 7Æ™™™ùã8¯E£ÄOž)Šäh{ cþ2;;û³ýû÷—Á˜œœì}ÆóH]££QQ$ #"ßëëëûóáÇ/ˆUJMNNöº®û‚ˆÜÂišñDWB!b|·ã÷|™¢•mä ” Š»_™œœ¼khhèX`Æ‘’ëº{EN2E¬9©„¢ 1äºîÄääd%ÆðNÓ¼¡6¡(ŒŽ/¹®û _SSS7ø>§i‘PÀŠÂàxhjjjë"bÇyŒv>Q„—SD·!wç1)i¾§§§wcØWXl¾³EÔÛA‘ãæ»1®=pàÀ7üÆcÌCœÂyͱºo½L!…Êð<ïapÞ|óÍKÜÃéšcB™ÕSÆÜ:55µÅqçËì-òŽBˆ"[;\wHÿÈs… àŸ'Ê(rs®TÒ¸Ó‘ë9móŒ"ö(TP4®w\É©«ˆb­ÒN…r¥`=§¯•´¡ˆBˆ"Û¨:º8k‰"5Š‚ö§.oež§ŽB$¬ÖE‘ó³kÃáÔµÕg衈,i˜)£ciK)FP5£‡"²”RAQØãÌölEQ÷LFaQ€(ƒ(ˆ‚0rÛcè¡èh âaó­mÂuâ!d•}l¾é *­Lº`3at2‹¶ €l¾5:(œ/Eì1:#[ˆ>ŠÀs¥ôQˆxì1­–SJ(bd*L!…M„Qw (xä›Ñb¡†Bˆ‚0ˆ‚(Ø|Ɇ& 8ÀçYAÁæ›Ñ™(â6ß™£`)Åh½˜ÒC! F‘9 6ß ¢h~lA“{ m"PHŒÅô2{ FKyC Eð§¶2Š¦FP„ŸH¨T>ñ8£õrJ…ØEq4ßì1¬$ }á—3ÎzémŠ8‡\Î\˸F¦ˆõ¥ f –RçŠßùVCÁãŒE!š(pî¡ [-†6 QDq~ƒÍ·Õl¡ˆÂ„4ÀZåW ad’7Ô2…D¤+…ò‰«„0ò€BâŒ!Óò©ø}K) ¥”: I°°sFåÓêÛyƒ‘QæÈL(Åy¥÷pE÷³!Û–ˆ ÓøZž=â1 ¯-aä$öö5ʧª9†jåZ_Í0£‹dJ6{ßÞ‹ÎD‘²|Šœ¤9FÁæ» (Bk|…ò©è(Ø|ÛסƒÂD¬vžõÞ§‚£`)UQÛfù”êC¥”E z(âaù7SFPˆúiçç ö6Hˆ&Æjç,Ÿ}(0cèw2…Äîqˆ"Þc Ê eIÖ–"ŠX% ‹{¦¬£öiÏ0& {.ÔPÄ[»–(b?VØ|[¬¦4Q$\íœ("ÇÊŒa³ÑPBóÒ­D‘à¼1Â(4 E ÌVmh¢¢È ì1ÔMÔOäÓD!Qßà#ŠÄceÆ°—0ÔP0Sd‹‚0 ƒ"ê`Q$+aX•A9Að8†…ÃSGÞcEšÅ©™1ÚPNe"vÖ ŠX(£ å”] Š(î =BY_Ûƒ0¬öD‘l¾ „Âï}<ôÉVúø®s*#g\%$ç(VŽ©›€í%¾)‚¥”Õ3 *˜„Á—Àž=”‘y)ÅÃF¥‹B`À÷‘=F®bC·iZ![@MOO¦³éѽPÃ8ý²SS%'Á%dÃxü“ø–¢Í—Çÿ͉„T±þ{ÉšVq©ÿϼúê«ü¨QŽékØ;}5/{ð›"‘(¤Hð•eƒ¶¹êç$ ­A!ké"ÇàWBJ’× p›k×ÿba!®é/ad°Š’!Š< à•¬âp°k°‚½3'áyDÑÉ(Àí©ÌÃ--¥ÚË’î.¢³­¤÷•VÇäÿ¸ãK±ìuùÞûÚ%Œ¢Š½Ó'á Qt* À|tè'Ò×u4鮖ɶQZ¼_Àm‘ãkã_¬É¹ûpªÖøò½yt¹Žƒ(:EºUBˆ"ô~]¥y m|•ÒñÀ—ðº ]Œ\S…cˆ¢Q€“¨Ç ŠX÷ë*-àºþ}(;ÇWV;÷ùïº+ 9ˆ¢ãP$»p Q$º_·»€¡þ}¨”> Í÷VaŒ!ŠB!±K)¢Hu¿Ó8ÊN0ŽÏ_ä⾫+0ÆE‡ ˆwƒ(Z·;á+eUPÜpq;ë8ˆ¢ý(¢aE&ãïvahãPv_ê/\RÆ}[ª0kÎ !Šv €„•RD‘éøÏsçp]ÿ\s"°!¿á"_½ººòí ¢hŠàŒA*ãïqçê=Gpæ¸ñ’òYDÑâ ƒ(TÇßSžÃPÿÜ7]ZÁÎÁn¢h ŠæRŠ(¬Œ¿§<‡áþ ¸Î‰Ð̱} ‹(l£hj¾‰Âêø{ÊG1Ü¿¥žã–Ï–±} J6Q¬é1ˆ¢-ã_Á^VÝvYÛ¯î& åÓêçvˆ¢½ãï-ÅpÿS8*+eQ¨–Oð-¥ˆ¢mãï-ÏÕqœ ÄqûåUÜ;ÐMŠå“ ¢h÷øOgŽ’Y ì9n¿¼‚{¯ê& ¥ò) cE»Çß[>‚¡þ}(™S™ãŽ+ª¸çª.¢P(ŸB`E»ÇßWùÃý¡8îü\¶mî&ŠŒË§ðƒ(Ú>þ¾Êû‘8¶nêÂÖM]D‘aùÜcEÇŒ?Žm›»q÷¦n¢È¨|Zãô¥°ˆ¢óÆßWyÃýpp2°!߶¹kQ´\>­3ð‡r~÷Qtðø?>u ^ŸAMʾo¢°°è5®÷‡T•i(I‘Ñ’M}cÚ-¹Ì?þ¾Ê{ÞðÞ8æÃà‚n®—e8D‘ñ¯¯þ·~œc‰³ÖB¸D‘Ÿñ¯¯¬àx}nW`YÅÈ&Ìëÿø½TÜ) ¹$.‘x¿´6–4ã ž£—ö¼ÞÊœ*ê÷Žß€Ù…mœ½šãÃãóUÈ06tF/æT3]Ø ŒÌz Föa©ücèe ®vžO|ߘ1òDmGC-c|ZsP“ºÙLþ&Ik½z Ç[í?ùLs °®bˆ¢¨0^<Ú‡÷ƒwýIÙgÛËÙCïMWj8MÀç¿r ùâgŸ¿á´††ç\}¿ú0Cžßï«¡k·z¿³À†nÞX%Š¢Âý´% _§Ÿ?´ÎW>ΣM=Q„£ˆ®ß´>2¬Ã Š(‚p´ãˆ{ ñ<¢H…"¬Î×GÁC»”"ŠT(âíÒþæ C±”"Šä™"b‚E`EQˆzÑø&EÜ¿9¨Î·ƒ‚=†½RD‘E¼`¦È9 ¢H†Bâï®%Šü6ßD‘Eºƒ|¡`eaw-Qïù&ŠT($æI„ (x¡µÝµD‘E²™)òWJE*‘óÔÚZ¸ µæ›(Ò ¢(tA)Q˜ˆ:_{ +¥Q$ÍíDÁÌa!clé]ÄÅU§éE— %[üvW6^o- A•€óˆšç˜ø¬1&>CYý;sæ“üè§À켧ŒÂE¡a¬;™ãáߟߘó0;¯)bwßÙ£  }E¬U¥~-gM"&ú‹Jj(xÃÂ^©F¢c3iP´3Sad CEœoñat¤ ¢`°Çh˜RŠ(ê¥Zà¹RÊ(x®3Fút¡Š"äûV/yÌ ŒT6,£œÀDATL©¢6¢ ÂHËBEº¬Al¾Ûž'ê(D¼çKe‚Í73FëY#óL•1Qp×-at2 ! ÂÈkó­‹¢•ƒ|DÁ£¨($â‹JÊ(Øc0c¤¤¡‰"b f ÂèÐ|¡BÚ…‚H£eš(~Q‰(Øc´ÝDý8†. ²àš.Šèc( fŒ C9S$^ñ<³LA„‘A§¡‡BÚ‡‚6£%DÁ »(¤­(¨ƒÍwêæ[…„-ºf›ofŒpÙ] ’™‚0 „"Ñ:DAÁ i aÑ*M§Î×CÁƒ£z(’ä-f Âè¤.CEº Tf„‚Ù‚0Zm4ÔPH¢A{Œ0á‰o1\‰#ƒRJE¬ÅÔ2QFúÎ[1SŹ£Vdz(²:òMÚcÐW(¢BŽcˆ'VP°ÇÐB€#ͪ("K)5a!>tÌEúË)Ç Qä.fù{qû ¢`$cÌkŽˆ¼TLº(Òù&Šíwcà÷1ÆÆÆž/•J7øc®÷JYË`¦ÈO¼X«Õn{:ènØ£wíÚõ;Ÿxâ‰íƘGÜ’«¼-(@ÿ2Æ<2::º/êŽnœgÀs{÷îý¢çyß4ÆìpQEâ}c̳~;22òRýLÈ0i·¶gÏž×u‡lPé¤Wâíù¥ §, œýñ‚ñ4Ýäó›ÆèrK'îÙÒûÏÆß_³áÕáuÕ…þ8Ï‘6>Yꟛž»m’sÞ?<Ï[2Ƽ#"SãããÓižãÿ†ö`Âþz%tEXtdate:create2013-08-08T10:34:23+02:00íq‚#%tEXtdate:modify2013-08-08T10:34:23+02:00œ,:ŸIEND®B`‚Nuitka-0.5.18.1/images/Nuitka-Logo-Horizontal.png0000644000175000017500000001664112651070406021675 0ustar hayenhayen00000000000000‰PNG  IHDRÞaxuÀ×gAMA± üasRGB®Îé cHRMz&€„ú€èu0ê`:˜pœºQ< pHYsHHFÉk>bKGDùC»–IDATxÚíyœUµÇ¿ÝÓ3™%ë$˜@A’‚ ¸E@QÙƒKØ…ˆd‘M6‚ ø@(ûN@>AÀ‡²ˆ„$1d'û$™™¾ïS=éîé®{k¦«»ªû|?¶ºnuWߪ[¿:÷žEQzÏ8`*° hf—ýµk8 X˜¼W'ðТ]¤(Š¢¸rIAɼæÛiñªO`?í"EQÅ…,‚’ßA5ÞO3-}tˆ^JµCR»@Q”Ò üÜ¡ÝÇó´»EHzÓ“œ3gÎîÀÀ( ±’9}‰¹b}Úg $ iylØkDÃì†:ÓÙ§nEKcránmsm×?ëÍæK7t¶®íéqc:‰Ä`Nccãs#GŽlÓKL©bö†8¶œ£]¦(yÂûî»ï6¶··Ÿ6{öì³€Eå §/êàébbƈ|ƒÁ°Û°:ê“  Ù¬ñ™LÃ,‘Í›å1òþ¢ û²ÞônFÌxŸÕÖÖÖ6kÖ¬)©Têâ­·Þz‘^jJ²C€¶›­Àrí6¥Öéšjž;wî–/WGIt­"—'ºÞÿòZE·À^½¥ ˜ØÑÑñÖÌ™3÷ÖKM©æû‡#uÚeŠâ œ3f´¶··OK§ÓcŒ1DíHtó÷uÝðŽ0õí·ßÞ=Äóx=°ØHqçõÀëÀ®1½V¿,-òÛÖçêp.;s´]î?EQáH¥R·#—?Ñín¹º‰î¦m¥'‘H4MŸ>=ŒX½p 0¨÷i×<ôáµz"0¸È¶fà:œËÎ4$6Õ…{C`Š7á}ë­·ö4ÆEK·Õk]“eõfíä&º,澆766žÂ9ì+[ÄðZµ9ø5ép.;+‘^ Ë´»ÅÞD"qlµXº¢§&'GÑ ÏâÍ:þãBøØTö™|B/¥\Üà³}p '¾Š¢IcÌ—â`íºŠnî¾é@¢[†ß²ÝŒ3†•ø&z°Oà:½ü•1 8x øؼ \‰,o¼¦]¤(¹ÖÒˆغ΢[À¥¹â–nN‡§R£"òôÿ $MÝÓ: ”0Í{)Šâ`ñö¼ÅHt³­×tÑ-Ïoéèèè¡óþYŠ'†S<5ãGÀAÚEJ$„7Dt»…9Šn¹-߈ð)ÔX©¾JñèŒH^iE©8©bq²‘Þt0Ñë2[GÝE7ýQb.þ,Ñá Äœ–^nWµx7Éf0Ñíf/;‰n͆~¦CAQE-Þ<ítÝ®õTd×Ut-™²JE"‘ˆbŸÜ¼¢CBQE-^±x]E·j;ˆî¦¶UÉ;t/Ë”üR‡ƒ¢(ŠZ¼›¤×Ut³,Þî"k]þoggg%:ðMà$íb1ö¾Ü©ÃBQ¥Æ-^‚ˆnŽÕDt«~÷|ìyuŽ: (Š¢„+¼QáÍXÑÍηœ&º•ª´T&cÏ—;ÜhEQ¥–-Þ ¢›g*;Zº5áÕ|#0ËÒæGÀÖ:4EQÂ!q¼Y©]tÅÒ5ÙeDW2]Uyo;p&ð¸O›LçCtx(Š¢Ô¨ÅDt»—ãu]j'qÆH2{?¾¢ÃCQ¥f-^œE×PÀ«9€èÖHï$`üó4ßìtè0QE©9‹×]té¶Æë*º5•¹j&ð+K›€Suˆ(Š¢Ô¤Åë.ºÆ•¹*€èšò¬ñ¦Óé¨të¥À`3Ÿ6— q½Ku¨(Š¢Ô’Å@t [®vÑ­Á\Í+ -m¢yœEQjÏâ%€èšœ\Í&€èZS¯Ñ’|“ç'ßÈ:–‚ïülrÚ­M¡^½8Ø٧͉HçWu¸(E¸8h.°m’í/ÚMŠâ oLT×Yt3­!Ht1†¾©ùôå‚„ ù×û…ü\ÑéÆÖul8ø»eVäFà :\”"œ^Dt†zo*¼Š’¹©Æ!sUÚ¸ˆnÆÍÊeè•H–Zt#:ý,p¿¥ÍçoÕ虈¤ÚÌ;ùtÿô‹Ð±Ž>,p¬˜‹8Ì…As/·ìéÞáq|]oùÌý|Vök)ðµ*»®[‘±³E~÷GÀ6e<¦`ààà·ÀŸ—‡Ð÷eÀzoü­–ïÿ®ŽC"2"»”‹×ÕÒí¶V1Ñî¬þYޥɧÍÕÀ£ÞÅ^KœRD\“ÀîÀç°ÇE—‹ À°"Û>ü'&ý~06"Ç2سÚ¯’kz4ð$0ÂÒn#â  ÞÃâAÞÃýØ€šÔä½[ؾøp·wî6¨Å8DZ›èCý¢"º‘UÞ÷½'E?FP›yœ{¹]µg4éñ„Â探»Î{_Â1ìÜì}ö4d©âÓ!‚­ÀáÀÀàbÄa´òÂkÅ>½ÜõßùÎb‘ÝH;²ýøÀÒæGžå¤(Jüh¦:ˆn'pð¯ÿþÀsÞçNú—ñ··"á‘ïǨÅëbíµtM7Ñ5kíFIt!nê \œë`1ýBï_Š;À€]ÚžJi§Õ[¿"K2Ÿ¯p?´w÷Vr(F¯ÿôrv¬o¡}£ º1ˆ¾xÞÒæP$ݤ¢„I›OI¹Yã·qâÔTJ~ |9bý1Þ{諯ï ü¦—sld[½‘ÝXÄLŸŽx’ú1™Ø„¢)1å)àˆ šeÀïbÜ—ÇcŸÍ˜‚=©NOÑ~Ùx°÷²XÜ<]-]S°ÊŠn@^nG¼8‹±#ð}$¾WQÂ`Á½šˆHÑ4Ä[º–çhÁþønXžë­˜Ìñ^³½×"`-°Æûw-â¡ÜˆDlŽ„=íäYÖ{жý§«‹Ê*¼ñÈ\å.º¹¹šÓ*º=ã|àHü.îBó8+JTÙÖ³èê-í¦#Þ¿•p@yYO~xX`ßuÞkð⽌'Ä'"µÇ9|ÎyH ðËåúѱXã "ºÝDNE·',.·´ˆ¬Ý(Š=Z‘ÚÛ¶Lyï_V—ñØæ?A"$ÆxÂ÷d@ÑõãCïÞ´÷àa£‰êP‹·°ø:ˆn–Å»jãf¯úz–PfÍh˜üŽBS0ßrÁÿ6Ÿt$úÍ‹ÉÀ œì=5ã$$ï5½Ï)Jd¨÷g[K»€‘×0Ɉú_k=‘-Ë‘,X×!5ÈýØØ x¡, ÑõþßIt³DuCg6töÔoI&“«b2xÛ½‹õqËŒÉÀõ^§(‘áfdm× À!”'“Ùw<Ë{N…úã,dýwoK»‰eÞ8X¼É@¢køpu;M©„ÝY<}£±lÏÞP¸ëŒÏŸ†úT"Npž@¼KýR¾|¸GïwŠRqÎNp°eŽEÖUËey.¯`Ÿ¤‘¤¯ã?õ~„'¾ëÕâöÛºž…«;YµÑ.ºÆîxuMîºpVIÁ\ËÙ䊹Å{zS8SÄ]Ÿ±©¤`~E%c`Üöƒ¶Ù@žäMÃø9h\ pô˜¦¼¾UýEw“`FOtÁàÔÕÑb¦wÁþЧÍHàLjÄ¢(ågWàH†*?&#kžµÈ͈#—ŸÁ¹OY„wûaÏÓ'U )KÁ”‹Ì.Ÿõ^ŽÀÉSl,Û³ÞkëÀŒ¥Óžn¦µ)ÉÑcû0ez«6ÄStãâÈV€K‘Ê7C|Úœ ܆”èR¥|Œ@fœlåDBlj•H,÷A>mv)Ç'Š€è‚¡)µ‚O%•hÃCkc‚cÆ6ѯxŠ®8š¼ˆ›¿-£æqV”òÓ‚>ØÂÒîyïá9]ãýesž*K)ÊdTE7³Osj; ~”TRÖ»[›D|[ˆèLœ/Ø[°‡ †ÝsPQ”ÒÝ¿ïv¶´› |ƒ28 Å€Z¶ú”Wx#&º™m-õKÙqð£Ôe,ߌøÖ/Ñ5±Þ4p†C»ÉH@º¢(ár'¨~,D¢–kwâÙlc‹°"uÑͼגZÂèÁtY¾Cš“»sK®øF\tMü/Úgû-mF§èøV”P9ûzí$+Õ<í®.>rh3¸<ÂqÑÍ_KýRvl}¸ËòÜ”`ÂN-4¥ˆ‡èS îÙØK¤]VŽ‹WQj”}H?:¸ÔWµ»rèÄž3ô\ ÉnQÑͼ߷a©gùn`hK’cvn¡©>¡¢[æ!ißü„æqV”0Ø)`ËÁp2âÁ«tg¥e{c,Þøˆnæ½¾õ‹Ù±õQ’lÀÃЖ$G{–¯ŠnY¸ ˜ois’¦MQ”Ò0Iá:ÐÒîb¤´§RÜêõ#t•dÜD7ów¿†=øaêؼ_ÇìÒ—¦TBE7|Öç8\¼“uŒ+JIh>aiw+²Ô£D˜TE7sÌý2zðü±ô:MÃúÖñ­±Í<5k«™M9—3®MÝE—¬r‚…„“¬ÄYëÈyûv‰zÞ¾ù4¥Õ’VñnàTàs>m¾ŒîÓ¡¦(½âVàó–6O¢Ž1Þ˜Šnæïþ 3øÞXv(¦žQë9y÷‘íðD"1§Š?Ï^¿®ó5H€›7EéG[Úü8qªªF† õ{G ±¶Ã€!Óî@d}¶É{õAfÞ’Þ¿‘ sL­Yß©úbHqÑÍÞ^Tt±ˆ.>3÷ó›ëWRÇ|ס ôoø/Ÿ4•7—ŒñÕ€ÊÓÙÙYMƒáeà÷øWC…äq¾X˜ñاŽß¾¬­‚ßÛØø40 OÜèWUï› öŠìÁÝl*ƒß÷ÝŒ(·6¾G*¹žöt³Õòr>¶àWø8“ÇYã Å=€;ð/|° I±(¦¿1ì‰äO>qȬú<‘®Ç›½¶ê4ý³R&—#SÊÅhBBŽÔîR'FâÚ²É\5+†¿¯8øžgÑÖ©—÷e}:™#v›Jîeyìv9e•ò”rXdö5tÿ¬MYœ²‹È{›5Á^#²K¿škÎJ…˜ŒÄ úÕ><£Ý¥(¾ôC††ZÚý{Òÿ¨1©vö]ʨ"²ÂûÞº>¬éHf‰î¦Š?›<€³Beº½çýÝ¥‘¦K+³=~sÛæ}GVèÍV’ì9< ¥›ÿ=J¹iGÒ×Mµ´»)»Õ©]¦(©îAÖ7mŒCŠÚ_ƒß•D|A®Ä¿¼hM̵t{*ºYåùºö.ºÝ…³ÞÕJ¥x{¦œ1ÞÔ’¢(…¹ɯìÊ•ø‡ôEÅ‚ ©pVÑíD¦á× ÑQˆ”1&R¢kLº€øº‰n÷}£G"‘¨æ›Æ$¤ú‡_:»Ë`­–¢(¹œ œôÜ‹Ì$-‰àoú82¶c >Ë Ü3·€÷ zó½ûÉ ïe«9ü°e$„7–®1“è‘è¢SÌQà-àWÀ}Ú´"ÎX§jw)Jû#iV{ÂpàNÄ+8J…î‡">£z¸ððwd-û`U5œìTeD—"¢krÓ3vé­‹èÆc·Êâx q)0Á2¥4¸xCï·%%©][z»óà"oüEFà‘ˆn;²lõdéj]uÔŠˆ®)(ºÆä­Õ]%2¬@2íøQ‡8Z)%~Ö.¨JÞsæŸûF䘯> ý:¤¢Ùà0$/õºj=¡ÉòO/û‹®É[ïuÝü}£ùªn¦[ÚŒCãzð2”êb2 ý[ìÞËIdÊy‹ óÎȬ–+ ±¼‹kaœ$+¿¦›+ºÝq]%R¤‘<Î6®¥†cùBàcÚUÅ:$d&AÆyÀ?®{*<û1·eNàtïüÃ2_Åï9IÑ°ˆˆn&éFδ³£èµx#Æßû-mFa//u¢ä¦>¥Z辉$ÉÈÐå`~øY…Ž{7à‹í ’Dã—м–Ê[¼DOt»ž@Õ‹”q6öªDçÒs¯Ç(БãhŽy?*¹|ŸÂ ißÆî½|’N²ÜïØî§H•r3Œä‚NVÊ{¹¸¥›ÎÙ?ˆèªÅ9æa_—j"Ú™wl'­ODŽóS¨Wsµpð?>Ûÿ\â0sG[.êo9´{ )¬#¢p‚Sㆬ¡£KË6Ýcº[ž&ïdòBg»—Ý<4+íhéšîe ƒÔéU¢ÈUÞS°ß$ð%dz:j´[¶ŠÈqî©—ZUð;ÜJh^d¬ÚߧÍ@à>¯ÝÆ2ûXÇñp‘ø ‹1Q8É©ÍÃ=]ù›í¢›ÉùœS•ÈUtcbU¦ÓéZ»‘¬C¦“ï´´»©Áµ@çõ–íQqhú²jVìywoà4ðàU`¤O»Ý4”åHX³»C›÷±çt“]£p¢Ë35å*ºÆtÏ@å(ºJ¤¹ xÞáiùä»m: ëª ÀÞz™Åš—¢÷A<—!ÎV6ëñûˆ£VØìæÐæÉ ?\!Â[ÞõM»èæì“N]õjŽ8g`w¹œèLÝf°å¿Ý6Çx 2­XëÄy€]IÏF¼ˆ[dÀ-Àö!ÿ—øáT°G£kÄâ &º…ÇŽ]t•Èó2v/ÆÁTÎ颶ø©¼—äÑzyµ[nòàAK›¾Hx_sˆÇáòÐünûé𨜰ð-ÞtK7Ï;9˜èÊw©ÅiÎÇžäü{DÄÂã‡ÚØ ßVÀ!ª¹€Ä¹Ö*'³-mF¿ ñ\f]*Y•ì¤¨œ¬²¬ñ:‹n±z¼¢«ë¼±`¿çG’ù&*üÇ¡Í~<¾³#`qG…¶þí«€#úàXàÄŽÁÅs´R!oû!!w5bñ]“NçZ‡DW×xcÁd‡'ó/GhZèM‡6•²8·#ši•b­e{s•ÿþ×8´û°Sß¿Ô¡Í° õM¤–°ÊdñIÐAžÅëj骨ńÀ™í®%IÿßZÚ|–ÒúBø5Z‘(ßêó£ ôÁmÀí–6Èzo©ûc‰C›‘è“cq uª.‹7PV¬<Ë5ˆèÆÁâíèèÐÛ£ÔÛœfi³ÑÉãüW‡6g–ù˜NöéXW3¶õÃá52¾Nõ¬_?¶E’u”—e™¯”¹/†#qÌ‘¢Lóíî¢ÛÝru´t5_sܘ„ÝæÜ =!çó¤ãSu¹¬Þ/×Ù–®aáµÍL ‰Èõ6mÈz¯màܪˆ¹òœC›Ë8KÓ²c5ó¡ÃÃܸ_³qs¢ºØ£Dßù"ö®AHÆ­°éƒL§ïÖƒÔð…·,ö®£èv'µt«K°;dEå³Í¬AŠuÛø â¸{ ù¬[‹l¿À²µ¯w"…9ü8®†Æ×Ø#ê‘|έ%ø¾ÕÀSí®"ÜT«ƒ€?Ùþ'Ëjèã$Y /€èfÇâ]Í\CV:´»‘ÊWÞ¹Úñ o"p]Ç{Š'ºÅ¦ÍB’”جòjǶθ7Á×ÆãÌÙäÖô-Ä(à”f)Â¥¾î0OüZCø½»{ã`oK÷ǖϽÔgÙ,^'K7+‰FPÑU«7¶ÜL·´Ù™ÊOÎð¦®\˜„8mQ‚ïÝYcþ ÅK®Fê¯Ú¨¯ëéßm~GÏËÃÅÍ3ºÉ½ÌÒî«Ày%ø¾§·Úí ¼æ}o)ì= ¿ˆ)ÄÛ7,Ÿz4EyÖx]E×kkõh.$ºFãxcJ7(ˆÆ$ì+f"åÛz»8Ö»I¼ `i{n©økàzr)-¹%’3ص”b ð]ïq¥7û'>&8X'—•à× ^Õ.7»‘HE¦§Øýž‚;z3Ló¼qà÷o§;|fèñÞ)d]$Ü)¨¢[¸:‘]tãbñc4ž¨ðÍòÄË2Ê,ðîïÛ÷CÒdž <ü ø0XŒ8lÕyVÔ0ï&òdmj;Çï¸)xîB-ïóžu7ØA|_ðnü{û-òfzVÓnHøˈ‡l†=€›bÖ/O!Yã.òiSÜ ì‚ÝCÜ¿x34®¥÷÷^ ¼™¢W¼‡œÞƒîjdú·/2-þI$v~oÜ‹¬Å­EkØ'#eŒY l¢æ]³©¯éª]ë&ºq°*“Éä"ÕÙ‚œ|-âp‡wc †Q‡¬+–zmñosªáíðŒ~äØþ Š;ácpLûæ`/Ëu8Ìß}é]щs¼‡È ‰+¶ œLl‹‘)íw¢r~“ÈtXØfž“èc ®®–n,,ÞummmóP 1h„¹p&pk,»¯,?qc\KWn2þ¸ oø¶gIú1™vîսΛ)x­Â¿ùïaãå(ߤ1æ±ò¬óº‰nîo0ÑÁïÓãǯåDî.7Ìù1¹ŒLÝUâ‰oŠwS[p¿ZÞ¥H•«°ÎMœk/¾‰=ÞùÕœloo¿{ÍÑÞ¼®¢ÛÍ«¹j,]€t2™¼ Åö”|no.³2bõ²ý‘|Îå`%’|àX$¶¸ë- µÂýÞÌD×Ř÷Ísˆï $ÄhT/¿k5RðäTdµ¼¬ÿžHñxÝ >û7….¼ãÇ_cŒ9Ãáf¬rÝÜêDÁD7âÖîM|𫪭VîBœ^‚²±BÇûg¤ÜØyž5+ê*[{ýãÇMž~^AésóFÊQ¥ô«˜ûúq”¹xÔÁò»ÞGÄÙj;é7½ ì€ø?øq³Ï8¹=ìÎOvØa÷'‰ ²œ-Ý¢ŽQ±¶tžX²dɤ>{½C›¸MoŸFð¢æK+x¼ë‘iò‘HØÆã%èó Hüî÷‚?ÁmÝò $"‘÷ÜâLEo¶‡Écˆì¥Ø×6‹±Ö³ ¶zòÝÁ1zvŸ=z1 •Ïb$$o’õ¿=çùÌE‰öòŽõÇ™Ë}ÆÉ]aw|Wj¬C=ôʇzè}cÌu”0Ww¯f»èš¬úºAD7‚^Í€:;;/˜8qbgHß±Y˘@áŒ+Kp‰ ¯xâ{öõ–vÏ¢¹-ǽ¸Ó{5{7‚]‘˜Ü-½Î d­µw³]í½–y7ÁHìîó!Z¥æFd격ÈͶÒçfâÑ{9ð9ÄyèÓÀ'ˆŽoì¬õú|>0Ë;Ï"!`½ˆß¿@B̲I{Ÿÿ|úe…×—"qç[P8]âÀÎÇÍÞ«?¶õ$œn”÷;À»ž½q²ÒÛo>;I–2#ŽÓ ÝR„M™2¥¥¥¥e’íä“ÞEÙc:Ó0ù…Õ½‰d«èn5¨¾ã¨¬ØsÄÔ¾Cšç§\-Ýis_¹±³¹ÒÊ»ø¯1æÑD"qÛá‡>EQEñø‹$!ÄÊk%tEXtdate:create2013-08-13T07:50:42+02:00Q6íù%tEXtdate:modify2013-08-13T07:50:42+02:00 kUEIEND®B`‚Nuitka-0.5.18.1/images/Nuitka-Logo-Vertical.png0000644000175000017500000002326312651070406021313 0ustar hayenhayen00000000000000‰PNG  IHDRU?öcPgAMA± üasRGB®Îé cHRMz&€„ú€èu0ê`:˜pœºQ< pHYsHHFÉk>bKGDùC»%¨IDATxÚíyœU}ÿßϽ“™Ìd²M@ö … BD«ÕŠ‚((¶ S¢? H+¨XŠ²i«´?ÜpiíOÑŠ¿ÚbÕ €èÏV\jEIÌ"ûÈB $™ÌÌ=ýã<™äÞçœ{ç9Ïv?ï×k^YæÜ{Ï=÷yÞ÷{¶ï‰©qÇwôöööEÑkŒ1GDQ4Ï3˜ ô«…&²}Ôð½ßvüxc&ükâßwúç«æNaïÁÚ„ÇN©mfÆ”5IÏÞäy›¾pËrÆÔÙ8ò’"5ù“ÀðPEk€_FQôÃ:èv]é© &Ï]wÝuH£Ñ8x 0¤ñcËÿÚÑ¡PÍn’3M„j0¼}áTž]ŸðøžØ·ÿ‡­ÍîÒ4¦µxÍÄÇ4Lwo;§ ÅÀUccc_:ôÐCŸÒ•99zÔ³zõê=¢(ºblll)PS‹LBŒ„Š±åw}­–¯’PŸ©Q‡ï1c>Y«Õ.^½zõŇrÈ?DQ4¦+´3$‚Y»víqQÝ ¼]혓ˆ=„ÚÙxB:Bm¿¹3+Š¢Ï­]»ö¦Õ«W匿MRÍŒ5kÖœoŒ¹ØS­Qt¡zÊ5„PMi›ú¨(Šn]½zõKtÕ©ûŸE—ÿãƘ ÕÙvÿ;ªiÑýÏJ¨%éþ7c/cÌMk×®=aÞ¼y?ÕÕªH5”PßH¨%‰PMîª)u»GQ4`ŒY¾jÕªçë*”TCõÀ•j‰² 5ŸµìBÝ©ýg×jµëV­Z5]W£¤šk×®í®ÒpI•„JB­ŒXçEQt©®HI5Í‹ê|àµD …jâ2Éf•PÝœ{çwÎוéF‘—ƒ{î¹gÖððð_ª%Ê-ÔÝש62j‰'ªvsE­V»x³®PEª“bÇŽof¨%ª¡’M„j*­ž´jÕª}u•Jª“¢Ñh,U+H¨*ucÌ[u¥JªoCÕè2 Õ%· B­žX£(zƒ®VÇ8‰š ñF¹1FIg‚µox¡öþg TS½ˆõ¥7ÝtSÏ1Ç3ª+X‘j'RÕlg‰#Ô6ök)Bõg`ï½÷þ]¹’j§]穲 [Óê.Z3y µ²rÕ}!©vF£ÑЬé„jÚS•P;ì]ÌÒE+©v©NU+”9Bmg€„ÚÆ}¡S,ÐD•û[YÖ§) µùø¬1 Œ©5ù\ Õ•«¼÷„&o%UQØ4µ.‹èט|"T}«û/DnZ Ðå7>b .T‰U‘ª˜@£ÑP#„îþêòãu&Ux¡jøHR"§x5L—ß´3Y¥U¨û/$Ô" Ur•T…(¼P)PåTuÿ…ÈÁª#Tã®@0¡Êª’ª˜x»i’!pûf TV U2j¯¡(Ò2UuÿE±‚tù»®U(RUŽVÓŽPóªäªHUˆœ¢ÕpBM²«„*©fJ£ÑÐøQð(5d—?þwÓ1ÕF&B­â˜ª6ÅHª¢jØÕãˆêÔ…ªHUÝ!*(T¯üÿ¡„*¯JªBdïÔ0]þgžÛ¹L5¤PeUuÿ…È:L ¡J¨BR-àm¯ áÛ6¨PMÓ‰ª¦M]¨F׺ÿBd¬ªo¤@¨BR"§ºó¡NJ¨Î5ªªTE5Ç2* UríF4¦šxŸë¦È¤}ƒ µyb³Û±a„ª„*ŠT…¨P„jüâÅBÕ IUˆÜ¼4B•P…¤*$Ô…j$T‘)SM@ U·¯1™Õ´LRZ¨Õ\§ª¹Eª¢t7jʪïÂÿ”…*$U!ºH¨d T‰UR¢bBuk- PåTIUˆlŠ„**‡&ªÚŠ¤D˜¶ (TcZgþ.T£ÅÿŠT…È!\ (TÿP9}¡ IUˆ¼œšƒPÉ@¨«ºÿB]ÿŒÛ7´PM‹ƒÿ²ŠPuðŸ¤*D®ájÚBMŽC Õ1ÊÜþ=‡Z†ïѯiòÏV© ñ|žÖÏùtcîºf%U!¡&‹5@—?2cÌé]‘áƒcBÒñ1Æÿx˜FÔ·‡®ÛÖhLUäîÔàBuuýCLJ™,&Á²ªÆ‰%UQ ¯†ªÉT¨íJJBU÷¿ÛnvMTUF¨Mª„\å…j@óTŠTEñ»ÿÙE¨„\7Uq†¤*J¯j›4†*¡JªBBõÍÉš¶P»e U;ÅÚAcª®[^ãªÛ–L„šœ¤Z]þö…ªAUEª¢ÈjàÕ£û¯.?Úz+©Š š5ˆPMÚBíæ.¿„ªî¿P/­.¿„ªHUTM«…jüÆÔåïè E(Rmeã /ÓðBMHR­.GB5º/$UQüî(¡º{­êò·÷þ©ªû/J¶†ªG.Uuù%TIUH¨“*êòw,TuÿÕýïø^×7s`—êøÓ4ÿ,Õåo_¨º'©Š"k5¡&©êòK¨’ª¨hÈJ¨œ¨*¡J¨êþ u2BU—_B•T3¼çuAeѶ!…Úp&T‘PÛjC÷„ºÿ¢ Áj¡º¿ÕåW„*© 5¡ UB•TE%µšS„ª.¿„*©ŠJG¬„j$T 5[4Q•€ªd#ÒB5-ª<1<—_=~n×õÒy¶Ú¯ár]À’ªè¶Õ$Ê$ÂP×PK«û/D û4„P $U!¡J¨BÝÿêwME˜ö /Ôfcª¢s¢(R#Hª¢ˆL©ÁŒÞˆÍÃ&œP›æÎÇFZ™½ÅkîMïêå §µîv‚Á„?šþ®É³ìò%ƒ#éK³cMó—3-kC«ô‡Më`à€9SŸ««WR¤·ñö…}|íöažØf‚cX³~¤Åãš¿¦ÙML; µÉA…f£&þÛ{sCëaÝ3n59‚»ÅkúÔÁ˜ÝåÚ´ñß_UÚSWok4¦*reF_Ä ú˜5µD¨-‡s$ÔŽ„ª1jEª“¢Ñhhü(¦÷Âé z¹úöí<±MB-¼Pµ|[Rí”Þ-ý½=Û:x¤é°ˆ óZí–5“­SóÇídÛ謦¥göE,]0•«Wnç‰í µ Bµÿ–U%ÕyÎŒ{÷Ýwæš6]’´å/i©OÒdƒ™d¹Ýo––¯°þ†:«7Ǧí¶ë©óûøÚÊí<5l$ÔB UÝSM 5±^£2ûÌžz_Ë&꯱táT¦÷!¡J¨’ª„*¡ºêU‹2û‡Ìè}à™õ£»þ M8}ÁT¦÷FjÑ„*¯Jªjq„:þ÷Z4Ê †ndf߃-›tN¥ û™ÞI¨ª¤*¡J¨®rV¬ßOëPÄÒEý N‰$Ô‚ÕȬ’ª„Z<¡NŒX¿ïŽXõ3Ø[“P%TIUB•P]åjÑ/º™}µlê=jœ±¨ŸÁx(@BÍQ¨òª¤: #¡fP±^ÏŒÞ[N^Íé8ma?ýS" 5W¡­R•T¡Y¨;G¬‡9"ÖçL«±táS" 5'¡*P•T%Ôu¼þµh„̾.Q¬{ ÖXºhS" 5/¡*¢¤*¡–§þõÚh,Ö‡Åzú¢¦Ô$T UR•P%TWý­X¿ËŒÞ‡ZŽ±î5ÍŠµ¿ 5c¡ˆ)©J¨å«½6Êas®gzï£-?š½k,=b©S" Uª¤Z¡"¡æXÿ:ÃÌŸó]¦÷®Kë¢iô×# UB•TKjW 5Ãú×£>´<1bÝgzSM£· UB•T%T ÕUÿžÚº–Á)¶cÝoFÓ ÒW‹$T UR-`%ÔÌëßSÛÁü9ˇöŸQç´#¦ÑWGB•P%U UBuÕ¿§¶ƒùC×&Šõ€™=œºh¾žHB•P%U UBuÕ¿§¶ùCËœÒZ¬sgõpÚ¢Azë‘„*¡Jªª„êªOm˜s<ÄzÄ ½u$T UR-ŒN͘„ZÐú÷Ô¶³`Îr¦õ´ž¼š;³‡ÓŽ˜No-’PSjC‹ÿÑÁ^ª„ZÄú÷Ô¶³på¬\[Föjúñ8«‡sÿx&;ÆLËNHǽ—N:;i?wÇOÓyr”žzmåçdI5½«SB-RýmÄz-+7,æé‘=š~‚ƒ½ê¥Ûƒ3ÃjuÿVB-Jý§Ô¶±pη˜6e½.Q¡HUB•PÓ¨ÿ¸XW¬?…§Gæèr’jÙðô>oÛ1Ðæð@«¿“çØUH)œ¨jd#Ôä„*…j”P%4J¨¢HU´ìþ‡Ë›4†ª’ªÈ©û:ùv^B•Y…¤*rˆTà ÕSlª¨ SuÞ÷L¨Z¨¦yÛíú¼¡„ª1U¡HUdnÖÐÉ·¡ IUtWç?ôi¹ U‚’ªÈɪ¡‡ñ¨„„**‡ÆT¨âz<“Ùy[I‰M Ucª¡¯!µ¯"UÑ$PÍæÃì#TÝðBRù„«Õªœ*$U‘Cœþˆm:H¨BR T Õxf©’PEÑD•¯|$Ô¶„êL¨D¨¦²Ÿ›P¤*Šo×€j;‹ÿÓª’ª¨žPMžB•\…¤*²vj`¡úçT‘PEõИª3¨3|?¡…j‹ÿà UcªaÑiªŠTEŸ…ª"T!©ŠîëþªÿÁ!„*Á IUd?V¨^ÿªœ*rFcª T3¡J6BM^§V¨SÕ}¡HUdÜý,TŸî°UB’ªÈÞªa…êZS%¡ IµKÝcL…û9ªèPµÚ¨Z¡5SM橪®S -Ô¦cª “‰P5¦–F£ñ”ZA‘j§z¼šï+¼P=BÕ@ª„š(ŠS+Hª²ºª] UL"R]­VT;To©ªSƒ µ“}ÿjY'Ÿ|òµ„¤Ú+W®¼X_E«†ªq NB-­0jµ›Ô Éh¢*Ë.»¬±|ùòήP¤‘™P›„ª&ª‚^?ßT+(RFãê Þa…ê›TEjÙ¸ddä'jIuR,^¼øfàß«éÕ0BuçS•PK¥^¾dÉ’1µ„¤:iêõúyÀˆ„ê+T#¡V£££_R3¸Ñ˜ªo|ãïøÎw¾óÀ{+md"Ô– U2ªÆTS§EÑyŠR©¦ÊæÍ›/ˆ¢è7êÊe¡¶£"Ô¢EÑÇN:餟¨%$ÕT9óÌ3·ŒŒO6H¨¢kåªO<ñRµ„¤„%K–¹bÅŠ·-[¶lDw½¤œ7½éM¯_¿þ•Q} .iša„Š"Ôòp?pââÅ‹Ï¿ì²Ë”â_RÍŽeË–œ|òɇ_Ê13š«P‘P‹ËzcÌ…ccc‡.^¼ø{jŽÎÑ’ª†€¥Ë—/¿¤Ñhœœ ,*ô–„*,[ŸDQtÍèèèw–,Y²EM2y"5Aú|ó›ßÜ3Š¢…õz}0³HuÛ>Úèùñ]Ûÿp¢ÂÐh"5³ËÍþ§¯xÞàm³ûëÛvþ¿½ÖìsÀÌ»ñ}ŽNh4"sëº~¦«0©Õëõ5}}}+N8á„aµˆB!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!:ãhà:`° ¸ø0CMÀ»€'±g‡ïü3ü0MM$„ç²&²ÿ¹x¾šˆß$´‘ŽS !ÞáŸXgwy;­r´Ñ‰º”ŠCMM rbø;rÏ.Ts IUˆd^ìáYö45—T…Hæ°6Êî ©É„¤*Dz×^]M&ÊraxØAëðíÀJàE%}Ÿ ¬oñÞž.Ð¥9¿o£ìÆøó¢lÇ=;þs/0XÂ÷x­ã}= Ë sf›=¯»¿ïò¶ÒìÉ"Õ¾6Ê|¸„ïsªã÷ýº2g3vª‹Gª¹D™0mþl*Ù{üã=©k™ŸNø\ŽP)R-[¤Ú.}À§Ôt"%Þ¼.þâÛ «ËÀmj"QõHµŒ[ã© Eª¢°‘ê8Ÿ¦¨ EEØ/A^›€7¨‰Dh©¾øs5¡¨'Ð:yË,lž!‚JàR`O5£¨Ó&ù{!R‘êLàãjF!„HGªÄÝ¢#Õ”Bá'Õ»€†ã9>§¦B?©þøª£ÌËSÕœBIÕaÏÈIâïÐ`¾BRõâ1Üû¯÷‹å+„’ªŸÖ8Ê|ø5«BRu3¼ßQFy„’jÜ€ÝGŸÄ›€×ªi…’ªï‹£Ö$>ô¨y…’ª›UÀçeÎQó !$U?><î(sþG !DWKu3p‘£Ì,”@!©zóeÜYÙß ¼PÍ,¸{¢m³<¦Ǫ‰D·HµœëñüŸU3‹Þ ´øÝ^ñ³]!U€Ÿÿæ(ó à­]ھ˰Û{wÀÆ€[€éªëÑÀ#-"Æ»±“!˜äï[ÉxEüÅï{<ЧÏy<;´ø“Š]×CØmék´>-áà ë4-îŸü%ðÀ€[±“ë÷°—Ž[À½ÀÍØcì¯Þ,š¬]ŵŽÇÏ+˜ôtxs¤E^gTÝæxÝ×èFù²£®zÝ-“¼þšqŸ½âgù$Û¨HgTÍïg×{^GØö½Øsò®ŒÅ9’òg¶ø°»©)³H•øà G™ýéμS'ù{Õµ3úUŸ ìÜßÏIl£óÇÔáÅÀãçþa<|t$鯋Š…ú-àaì)'³²’*qWàG™ÏÓˆ‹¥d¸ÎC¨cÀŸÿ?å×?øyü¼Ë€w\†Í-½4+©n.ðˆt>©kSˆÒ_^äQöàú”…öcìÞ+rn‡!àkÀ5I=·ZŠ/ø/À/eNBKdDx¶©>©r9p²G¹a'ˆÒäo€c ÖKbц–*Øñ†£Ì•(/€Ë€Û±y³¸ªÄmy¦G/àjÂLfÎ/h»¼ øv3—¥-·_c^IZ[x8ð´~U„c°°ÍÇœGò²ªR¬ÕYp´gäù#ઃoàg€û€ßÇ?kãŸuØÍ%[â?Ÿ†ãîûtìäÛÁØeTÇ/mËÇa'°.)U°³üo&y ù#À7·”I19æÅ‘ØG¹ØYò‘êø[ìøíÏ€_O´ñØ­ñÏ:ìÒÇoÅÿ¿O¾˜íñ<b—ýݪûöè•¿v”™…+B!lîä!G¹û€§2¬ÛïK°+‰ÄR»±M¡&ñH즃ã/uì꧶Cëv¹2½“xp„®_! Å”X&óå6¯Ç®á ɸ° |^´Ýøu7bwg}Ú£ì±Ø¥ƒJu›ÌÚ5V¢qU!ŠÅ±c©I cwqÝ™A}NEzl‘fÍù±Ð], -Uð;zå(à-ºŽ…(ïp”1ØmÀ?˨Nã.^4° þ7:ÊB¼vµ¸B>G¯üoòÍ „°kÈ/÷ŒÜ®é²¶yø‚£ÌðÊ,¤ºÊ£2¥kZˆÜxð±;§’¸’î=-ù‹À¨£Ì±YHìò)×Ò©ÏÕµ-Dæì|Ï£·ømÜGÔW™‡±k•“xaVR}÷N å"{¦a“¤ìë(÷ à4Ü»%«Î9~¿0+©| ÷Ñ+'¯Öu.D&Ô°p\ËWoÄ&wîvnvü~/ /+©ú½v̦®ÏNˆà\Ë2‰G±[s7ª¹XéQfßZ†ò9ze>p¶>;!‚ònÜã£[°‹íïSs=Ã&2sjWꃸӠ}˜£ÏOˆ ‹{EÎ(vÝåoÔ\ý%·?k©Þ|ÂQf6Ê DÅ&q%Rz7î™îne³ã÷Sk9TêoeÞ…MÅ%„H‡=°\ç,]ŠMß)ZG«IÔóêV첉ÃNZ !&O/öD׃後~“ –Óëú½ò*ì±BˆÉñeÜç;݈&‰S!ÏcMÎ~åûØÅÉÛôQ ѧ;Êü76±ühEÛ`lþÕý±kI÷žƒ ™ÌÄ&ÕŸŠ=N¼è‹{̵øO丹yJõVàŸHΊ3›àRÝB´Íîü=ÀŸ`);ƒØãPŽÄ&°žM6=½["U°G¯œBòÑ+¾‚ÖË Ñ/ŧœ”$evqÿº’¾Ç{ßâ÷±ˆlªåüúëp½Ò{–bbï»$œMÝvúF`M ßßð¸î¿ˆƒ³#)ÈnÌZêàsôÊ)¸³‘ !lW÷zìØa7ãNR4fcO y0´.b%‹ ÕüRŠ}å"‰:ð¯ØñDGcN—öì5À_Ľ×BW¶\{Çà,Ý7B´äÓØýú¾\üq "ïïa3Ýí‘Áëa‡F¶`Wbñ¦§@÷>l˜¤:}»ÆUYs„˜È9q×îý 6¹òã|OÏÃ.©<<…ç2Ø•«€ßaOc}0þÙˆÍûü÷–Eª¿>œ—Pf;±uŽî!!žáxìÖîNØøgììy‘’Pïü;éÖ Û€Ÿ?ÅŽÿx²›ºÿãø½² ¿1#QîkAøs“›ox-pqÞÏTàÚ„:‚ÝŽ{rîCCp¦g¹¿Á&cÊš½ñ˜,²T}Ž^)z^עᾂÔóhö¿*|ø? ¿ÿà2Ô×°ëD³b ðVr¿Å/$ûû†³EÆçè•7cZ‘Çïg¤ž/“‹*ÁUø¥Éü€ofØ›Zèy?\ìq_…Âk)gÑ¥º5îâ»(j^€íŽßerèù¨ôÜ€ÿ¬y8xÀQîÅØ­¯Yð2÷Ó|X#+^T©|÷Ñ+ ±'@ טpÆ1{WËI¥æWØ„Ôcm¼w¢øùÀ߬ƒOo)ÏìtÞ kÊ$ÕuØõiIÔ±;2ŠÂeŽË±~D‰¿Çéæ{ŸÄž®ájƒ3€wªƒO†¬¼|uvÙaå¤ ~G¯ÓN¨˜ßz”É+R|>ÅœÜË ×i¢ÿ+?÷(÷yì{i³Þ£ÌÞ9µM[Êe“êüŽ^ùÅHr/ð¨£Ì‘NÞvˆ€/ ÌT»FkIÌè‚6ø ðUG™©ØñÕ´ÛÃ'Iö9´Éø-÷*­TÁïè•çRœ¼?ö(óþŒëôÀ±ˆ¸Ê¸Æëöë’/—sâ¨5‰yØiâ3TöÚŒÛb?:X§[Ö­‰ïÃ=±pANßl»r£ç·aVÑê++Zü®ÑÅRuõ(ö(ÈõšmØñUWä~ ~Ùä|ù¹G™×gØ»êÇ&Ênw™f­¬Rý]Ü}Mb AYr-î5„õ¸Ûú‚Y€MàÛlëáMŽ:V™G<¾¨î’hu-~RW/Mé5‰{YÛlìN°ÐôÅC/î ø¨]ôœôsmA?øYØqWý~à(³>p]¿æQO|1`^wq[½ö‹âvhõûOª×–]¿wÔå?rºÖW9êj²ó3×ì}سãÒàz×{„°Û»gÇŸs«×¿ÁqŸœXf©‚Ýëìªÿo€ÿ—³T¿Ý|Äú©Ã2gcó´zÍñÅßIËgº@ªßóø|Ží"©N‰#HW›ÜÒðÐñž÷ȧ(òy pwÂëŽÅ½½¤ûdIÙ¥Znóx;r–*Ø£€çÏ÷IçX‹yØ1ݤ×z’gS¼%],_è©^âñÙÜ‹g ¸&Ì(™TÁŽ#¯÷h—¥ðZ‘Ç{ÿ¹8!¥÷8›”iÌñš_ö¸O––]ª`Óþ™Iþd!Õ}ÍmÔéIlŠ¶NÖæ-ÄŽÑ{¼Î;=.éb¹ª ¤ê{-Ý‹ºÄiØ,õ+âÇž]2©Â³ÇW'Õa”tÆœm£Wg⡽c:ìÝ÷ ·x¼ÎJž]«œtŸœU©‚T.ºT‰ÖnÝF>ŒÝÙqpñô`Ô÷Œ»$o‰Ç=WµñÜ×ìR¿¤‹åŸ»@ª=žQÙøÏõØ C㱸ì*?Œåùì$åÎù§JlòkŸñÎ4较ûä!ì‡Ë°k¿çbç]êØ™ü=±óoÅn"º½çÞäyŸ|¨*R=»¤èR¿Áÿ,~~Ì­÷w­²TÁn ÙîוTªµøËÝõþnbò+E€[ rŸ¬c÷<ª‰ºU9Bã>Š±|ʇ÷ï46“¿þ”öö»wËÖKØÄsJÚ. ìùV;ÊGµ“ak<äp[Îïù.àåØ,yÞŸo­b7Ã%¹8ßM“GÏ«ã öé6×-R]‹…úlÊœ»ö±x˜Éµž÷B&?‰´ »QeyNïõ+ñ0Î]m>n¨*ÝÿqÞÖaˆÿxNõ}-6Gdݘ'âöIâÁ„Çÿ¨KºÿãœG{&í »”±û¿3ôxŸHçd‹{úÀ¦Œî“U¸zÈuŸTIªã]ÛN¹ób*ðWñØMˆ‹dSÜóY×÷avŸXŽ·t™TÁÓühŠŸÅÀ + UâÏÅõ~oÆ®uMƒç`÷á?è>¹»õÖ§÷~qÂ}ò6©þKɤz$öÜvtEêÝ‹Ý‚w]‹¬Ÿíص®Ë€™%èR&½—oä\¿Ø#êð³Ø‚=•ô$:?’ü7Ž×8>‡v™…Ý.îzÿø<Ί#‘IÞ'waW̤~šðU´^ÏøXNØd9+î~øl x€tC¤Á@Ü9?ýi|lŒ…;߬k°»^¾—0½DŸÕE _"ë€×¤žõxŒï’8J»=ÝOÆâ½ø:6¡ÏËH'ŸÃ»ZDhcq48˜S›ì…ÝZ}_‚à>ø ïÕq[_ ü$¾OÖcç Æâ/€a×¥f™ÀH!„Bˆ‚ð?WÛ²6i@%tEXtdate:create2013-08-13T07:50:42+02:00Q6íù%tEXtdate:modify2013-08-13T07:50:42+02:00 kUEIEND®B`‚Nuitka-0.5.18.1/bin/0000755000175000017500000000000012651072347014210 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/bin/nuitka-run0000777000175000017500000000000012651070406017435 2nuitkaustar hayenhayen00000000000000Nuitka-0.5.18.1/bin/nuitka0000755000175000017500000001457112651071040015426 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This is the main program of Nuitka, it checks the options and then translates one or more modules to a C-ish source code using Python C/API in a "*.build" directory and then compiles that to either an executable or an extension module or package, that can contain all used modules too. """ # Import as little as possible initially, because we might be re-executing # soon. import os import sys import warnings # LIBDIR trick start (marker for removal on platforms that don't need it) libdir = "@LIBDIR@" # Two cases: if libdir != '@' "LIBDIR" '@': # Changed by our "distutils" hook, then use the given path. if not os.path.isabs(libdir): libdir = os.path.join( os.path.dirname(os.path.realpath(__file__)), libdir ) libdir = os.path.abspath(libdir) sys.path.insert( 0, libdir ) else: # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), ".." ) ) ) # LIBDIR trick end (marker for removal on platforms that don't need it) if "NUITKA_PYTHONPATH" in os.environ: # Restore the PYTHONPATH gained from the site module, that we chose not # to have imported. pylint: disable=W0123 sys.path = eval(os.environ["NUITKA_PYTHONPATH"]) del os.environ["NUITKA_PYTHONPATH"] from nuitka import Options # isort:skip from nuitka.utils import Utils # isort:skip pylint:disable=E0611 import logging # isort:skip logging.basicConfig(format = "Nuitka:%(levelname)s:%(message)s") # We don't care, and these are triggered by run time calculations of "range" and # others, while on python2.7 they are disabled by default. warnings.simplefilter("ignore", DeprecationWarning) # We will run with the Python configuration as specified by the user, if it does # not match, we restart ourselves with matching configuration. needs_reexec = False current_version = "%d.%d" % (sys.version_info[0], sys.version_info[1]) # We support to execute with a specified version. intended_version = Options.getIntendedPythonVersion() if intended_version is None: intended_version = current_version # If it's a different version, we find it by guessing it, otherwise we use the # one previously used. if current_version != intended_version: if Utils.getOS() == "Windows": python_binary = r"C:\Python%s\python.exe" % \ intended_version.replace('.', "") else: python_binary = "/usr/bin/python" + intended_version needs_reexec = True else: python_binary = sys.executable python_flags = Options.getPythonFlags() if sys.flags.no_site == 0: needs_reexec = True # The hash randomization totally changes the created source code created, # changing it every single time Nuitka is run. This kills any attempt at # caching it, and comparing generated source code. While the created binary # actually may still use it, during compilation we don't want to. So lets # disable it. if os.environ.get("PYTHONHASHSEED", "-1") != '0': needs_reexec = True # In case we need to re-execute. if needs_reexec: if not Options.isAllowedToReexecute(): sys.exit("Error, not allowed to re-execute, but that would be needed.") # Execute with full path as the process name, so it can find itself and its # libraries. args = [ python_binary, python_binary, ] # Potentially give Python command line flags as necessary. args.append("-S") # Same arguments as before. args += sys.argv + list(Options.getMainArgs()) if current_version == intended_version: os.environ["NUITKA_PYTHONPATH"] = repr( sys.path ) # False alarm, pylint: disable=E0611 from nuitka.importing.PreloadedPackages import detectPreLoadedPackagePaths, detectPthImportedPackages os.environ["NUITKA_NAMESPACES"] = repr( detectPreLoadedPackagePaths() ) if "site" in sys.modules: os.environ["NUITKA_SITE_FILENAME"] = sys.modules["site"].__file__ os.environ["NUITKA_PTH_IMPORTED"] = repr(detectPthImportedPackages()) os.environ["NUITKA_SITE_FLAG"] = str(sys.flags.no_site) \ if "no_site" not in Options.getPythonFlags() \ else '1' os.environ["PYTHONHASHSEED"] = '0' Utils.callExec(args) # Inform the user about potential issues. if current_version not in Options.getSupportedPythonVersions(): # Do not disturb run of automatic tests, detected from the presence of # that environment variable. if "PYTHON" not in os.environ: logging.warning( "The version '%s' is not currently supported. Expect problems.", current_version ) if "NUITKA_NAMESPACES" in os.environ: # Restore the detected name space packages, that were force loaded in # site.py, and will need a free pass later on. pylint: disable=E0611,W0123 from nuitka.importing.PreloadedPackages import setPreloadedPackagePaths setPreloadedPackagePaths(eval(os.environ["NUITKA_NAMESPACES"])) del os.environ["NUITKA_NAMESPACES"] if "NUITKA_PTH_IMPORTED" in os.environ: # Restore the packages that the ".pth" files asked to import. # pylint: disable=E0611,W0123 from nuitka.importing.PreloadedPackages import setPthImportedPackages setPthImportedPackages(eval(os.environ["NUITKA_PTH_IMPORTED"])) del os.environ["NUITKA_PTH_IMPORTED"] # Now the main program. from nuitka import MainControl # isort:skip MainControl.main() Nuitka-0.5.18.1/bin/compare_with_cpython0000755000175000017500000004574412651072206020373 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import difflib import os import re import subprocess import sys import tempfile import time filename = sys.argv[1] args = sys.argv[2:] def hasArg(arg): if arg in args: args.remove(arg) return True else: return False # For output keep it arguments = list(args) silent_mode = hasArg("silent") ignore_stderr = hasArg("ignore_stderr") ignore_warnings = hasArg("ignore_warnings") ignore_infos = hasArg("ignore_infos") expect_success = hasArg("expect_success") expect_failure = hasArg("expect_failure") python_debug = hasArg("python_debug") module_mode = hasArg("module_mode") two_step_execution = hasArg("two_step_execution") binary_python_path = hasArg("binary_python_path") keep_python_path = hasArg("keep_python_path") trace_command = hasArg("trace_command") remove_output = hasArg("remove_output") standalone_mode = hasArg("standalone") no_site = hasArg("no_site") recurse_none = hasArg("recurse_none") recurse_all = hasArg("recurse_all") timing = hasArg("timing") coverage_mode = hasArg("coverage") original_file = hasArg("original_file") plugins_enabled = [] for count, arg in reversed(tuple(enumerate(args))): if arg.startswith("plugin_enable:"): plugins_enabled.append(arg[len("plugin_enable:"):]) del args[count] plugins_disabled = [] for count, arg in reversed(tuple(enumerate(args))): if arg.startswith("plugin_disable:"): plugins_disabled.append(arg[len("plugin_disable:"):]) del args[count] if args: sys.exit("Error, non understood mode(s) '%s'," % ','.join(args)) # In coverage mode, we don't want to execute, and to do this only in one mode, # we enable two step execution, which splits running the binary from the actual # compilation: if coverage_mode: two_step_execution = True # The coverage mode doesn't work with debug mode. if coverage_mode: python_debug = False comparison_mode = not coverage_mode assert not standalone_mode or not module_mode assert not recurse_all or not recurse_none if "PYTHONHASHSEED" not in os.environ: os.environ["PYTHONHASHSEED"] = '0' os.environ["PYTHONWARNINGS"] = "ignore" if "PYTHON" not in os.environ: os.environ["PYTHON"] = sys.executable extra_options = os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() if "--python-debug" in extra_options or "--python-dbg" in extra_options: python_debug = True if python_debug: if os.path.exists(os.path.join("/usr/bin/", os.environ["PYTHON"] + "-dbg")): os.environ["PYTHON"] += "-dbg" if os.environ["PYTHON"].endswith("-dbg"): python_debug = True # Make sure we flush after every print, the "-u" option does more than that # and this is easy enough. def my_print(*args, **kwargs): print(*args, **kwargs) sys.stdout.flush() if comparison_mode: my_print( """\ Comparing output of '{filename}' using '{python}' with flags {args} ...""". format( filename = filename, python = os.environ["PYTHON"], args = ", ".join(arguments) ) ) else: my_print( """\ Taking coverage of '{filename}' using '{python}' with flags {args} ...""". format( filename = filename, python = os.environ["PYTHON"], args = ", ".join(arguments) ) ) if comparison_mode and not silent_mode: my_print('*' * 80) my_print("CPython:") my_print('*' * 80) if two_step_execution: filename = os.path.abspath(filename) if module_mode: cpython_cmd = [ os.environ["PYTHON"], "-W", "ignore", "-c", "import sys; sys.path.append(%s); import %s" % ( repr(os.path.dirname(filename)), os.path.basename(filename) ) ] else: cpython_cmd = [ os.environ["PYTHON"], "-W", "ignore", filename ] if no_site: cpython_cmd.insert(1, "-S") if "NUITKA" in os.environ: # Would need to extract which "python" this is going to use. assert not coverage_mode, "Not implemented for binaries." nuitka_call = [os.environ["NUITKA"]] else: if comparison_mode: nuitka_call = [ os.environ["PYTHON"], os.path.abspath(os.path.join(os.path.dirname(__file__), "nuitka")) ] else: assert coverage_mode nuitka_call = [ os.environ["PYTHON"], "-S", "-m", "coverage", "run", "--rcfile", os.devnull, "-a", os.path.abspath(os.path.join(os.path.dirname(__file__), "nuitka")) ] if python_debug: extra_options.append("--python-debug") if remove_output: extra_options.append("--remove-output") if original_file: extra_options.append("--file-reference-choice=original") if coverage_mode: # Coverage modules hates Nuitka to re-execute, and so we must avoid # that. python_path = subprocess.check_output( [ os.environ["PYTHON"], "-c" "import sys, os; print(os.pathsep.join(sys.path))" ] ) if sys.version_info >= (3,): python_path = python_path.decode("utf8") os.environ["PYTHONPATH"] = python_path.strip() if binary_python_path: python_path = os.environ.get("PYTHONPATH", "") os.environ["PYTHONPATH"] = os.pathsep.join( python_path.split(os.pathsep) + \ [os.path.dirname(os.path.abspath(filename))] ) if keep_python_path or binary_python_path: extra_options.append("--keep-pythonpath") if recurse_none: extra_options.append("--recurse-none") if recurse_all: extra_options.append("--recurse-all") if coverage_mode: extra_options.append("--must-not-re-execute") extra_options.append("--generate-c++-only") for plugin_enabled in plugins_enabled: extra_options.append("--plugin-enable=" + plugin_enabled) for plugin_disabled in plugins_disabled: extra_options.append("--plugin-disable=" + plugin_disabled) # Now build the command to run Nuitka. if not two_step_execution: if module_mode: nuitka_cmd = nuitka_call + extra_options + \ ["--execute", "--python-flag=no_warnings", "--module", filename] elif standalone_mode: nuitka_cmd = nuitka_call + extra_options + \ ["--execute", "--python-flag=no_warnings", "--standalone", filename] else: nuitka_cmd = nuitka_call + extra_options + \ ["--execute", "--python-flag=no_warnings", filename] if no_site: nuitka_cmd.insert(len(nuitka_cmd) - 1, "--python-flag=-S") else: if module_mode: nuitka_cmd1 = nuitka_call + extra_options + \ ["--python-flag=no_warnings", "--module", os.path.abspath(filename)] elif standalone_mode: nuitka_cmd1 = nuitka_call + extra_options + \ ["--python-flag=no_warnings", "--standalone", filename] else: nuitka_cmd1 = nuitka_call + extra_options + \ ["--python-flag=no_warnings", filename] if no_site: nuitka_cmd1.insert(len(nuitka_cmd1) - 1, "--python-flag=-S") for extra_option in extra_options: dir_match = re.search(r"--output-dir=(.*?)(\s|$)", extra_option) if dir_match: output_dir = dir_match.group(1) break else: # The default. output_dir = '.' if module_mode: nuitka_cmd2 = [ os.environ["PYTHON"], "-W", "ignore", "-c", "import %s" % os.path.basename(filename) ] else: exe_filename = os.path.basename(filename) if filename.endswith(".py"): exe_filename = exe_filename[:-3] exe_filename = exe_filename.replace(')', "").replace('(', "") exe_filename += ".exe" nuitka_cmd2 = [ os.path.join(output_dir, exe_filename) ] if trace_command: my_print("CPython command:", *cpython_cmd) if comparison_mode: start_time = time.time() process = subprocess.Popen( args = cpython_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_cpython, stderr_cpython = process.communicate() exit_cpython = process.returncode cpython_time = time.time() - start_time def displayOutput(stdout, stderr): if type(stdout) is not str: stdout = stdout.decode("utf-8" if os.name != "nt" else "cp850") stderr = stderr.decode("utf-8" if os.name != "nt" else "cp850") my_print(stdout, end = ' ') if stderr: my_print(stderr) if comparison_mode and not silent_mode: displayOutput(stdout_cpython, stderr_cpython) if comparison_mode and not silent_mode: my_print('*' * 80) my_print("Nuitka:") my_print('*' * 80) if two_step_execution: if output_dir: os.chdir(output_dir) else: tmp_dir = tempfile.gettempdir() # Try to avoid RAM disk /tmp and use the disk one instead. if tmp_dir == "/tmp" and os.path.exists("/var/tmp"): tmp_dir = "/var/tmp" os.chdir(tmp_dir) if trace_command: my_print("Going to output directory", os.getcwd()) start_time = time.time() if not two_step_execution: if trace_command: my_print("Nuitka command:", nuitka_cmd) process = subprocess.Popen( args = nuitka_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_nuitka, stderr_nuitka = process.communicate() exit_nuitka = process.returncode else: if trace_command: my_print("Nuitka command 1:", nuitka_cmd1) process = subprocess.Popen( args = nuitka_cmd1, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_nuitka1, stderr_nuitka1 = process.communicate() exit_nuitka1 = process.returncode if exit_nuitka1 != 0: if not expect_failure and \ not comparison_mode and \ not os.path.exists(".coverage"): sys.exit( "Error, no coverage module installed for %s." % os.environ["PYTHON"] ) exit_nuitka = exit_nuitka1 stdout_nuitka, stderr_nuitka = stdout_nuitka1, stderr_nuitka1 else: # No execution second step for coverage mode. if comparison_mode: if trace_command: my_print("Nuitka command 2:", nuitka_cmd2) process = subprocess.Popen( args = nuitka_cmd2, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_nuitka2, stderr_nuitka2 = process.communicate() stdout_nuitka = stdout_nuitka1 + stdout_nuitka2 stderr_nuitka = stderr_nuitka1 + stderr_nuitka2 exit_nuitka = process.returncode else: exit_nuitka = exit_nuitka1 stdout_nuitka, stderr_nuitka = stdout_nuitka1, stderr_nuitka1 nuitka_time = time.time() - start_time if not silent_mode: displayOutput(stdout_nuitka, stderr_nuitka) if coverage_mode: assert not stdout_nuitka assert not stderr_nuitka ran_tests_re = re.compile(r"^(Ran \d+ tests? in )\d+\.\d+s$") instance_re = re.compile(r"at (?:0x)?[0-9a-fA-F]+(;?\s|\>)") thread_re = re.compile(r"Thread 0x[0-9a-fA-F]+") compiled_types_re = re.compile(r"compiled_(module|function|generator|method|frame|coroutine)") module_repr_re = re.compile(r"(\)") global_name_error_re = re.compile( r"global (name ')(.*?)(' is not defined)" ) non_ascii_error_rt = re.compile( r"(SyntaxError: Non-ASCII character.*? on line) \d+" ) python_win_lib_re = re.compile( r"[a-zA-Z]:\\\\?[Pp]ython(.*?\\\\?)[Ll]ib" ) local_port_re = re.compile( r"(127\.0\.0\.1):\d{2,5}" ) traceback_re = re.compile( r'(F|f)ile "(.*?)", line (\d+)' ) tempfile_re = re.compile( r'/tmp/tmp[a-z0-9_]*' ) def traceback_re_callback(match): return r'%sile "%s", line %s' % ( match.group(1), os.path.abspath(match.group(2)), match.group(3) ) def makeDiffable(output): result = [] # Fix import "readline" because output sometimes starts with "\x1b[?1034h" m = re.match(b'\\x1b\\[[^h]+h', output) if m: output = output[len(m.group()):] for line in output.split(b"\n"): if type(line) is not str: line = line.decode("utf-8" if os.name != "nt" else "cp850") if line.endswith('\r'): line = line[:-1] if line.startswith("REFCOUNTS"): first_value = line[line.find('[')+1:line.find(',')] last_value = line[line.rfind(' ')+1:line.rfind(']')] line = line.\ replace(first_value, "xxxxx").\ replace(last_value, "xxxxx") if line.startswith('[') and line.endswith("refs]"): continue if ignore_warnings and line.startswith("Nuitka:WARNING"): continue if ignore_infos and line.startswith("Nuitka:INFO"): continue if line.startswith("Nuitka:WARNING:Cannot recurse to import"): continue line = instance_re.sub(r"at 0xxxxxxxxx\1", line) line = thread_re.sub(r"Thread 0xXXXXXXXX", line) line = compiled_types_re.sub(r"\1", line) line = global_name_error_re.sub(r"\1\2\3", line) line = module_repr_re.sub(r"\1xxxxx\2", line) line = non_ascii_error_rt.sub(r"\1 xxxx", line) # Windows has a different "os.path", update according to it. line = line.replace("ntpath", "posixpath") # This URL is updated, and Nuitka outputs the new one, but we test # against versions that don't have that. line = line.replace( "http://www.python.org/peps/pep-0263.html", "http://python.org/dev/peps/pep-0263/", ) line = ran_tests_re.sub(r"\1x.xxxs", line) line = traceback_re.sub(traceback_re_callback, line) line = tempfile_re.sub(r"/tmp/tmpxxxxxxx", line) # This is a bug potentially, occurs only for CPython when re-directed, # we are going to ignore the issue as Nuitka is fine. if line == "Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in ignored": continue # This is also a bug potentially, but only visible under # CPython line = python_win_lib_re.sub(r"C:\\Python\1Lib", line) # Port numbers can be random, lets ignore them line = local_port_re.sub(r"\1:xxxxx", line) # This is a bug with clang potentially, can't find out why it says that. if line == "/usr/bin/ld: warning: .init_array section has zero size": continue # This is for NetBSD and OpenBSD, which seems to build "libpython" so # that it gives such warnings. if "() possibly used unsafely" in line or \ "() is almost always misused" in line: continue # This is for CentOS5, where the linker says this, and it's hard to # disable if "skipping incompatible /usr/lib/libpython2.6.so" in line: continue # This is for self compiled Python with default options, gives this # harmless option for every time we link to "libpython". if "is dangerous, better use `mkstemp'" in line or \ "In function `posix_tempnam'" in line or \ "In function `posix_tmpnam'" in line: continue result.append(line) return result def compareOutput(kind, out_cpython, out_nuitka): fromdate = "" todate = "" diff = difflib.unified_diff( makeDiffable(out_cpython), makeDiffable(out_nuitka), "{program} ({detail})".format( program = os.environ["PYTHON"], detail = kind ), "{program} ({detail})".format( program = "nuitka", detail = kind ), fromdate, todate, n = 3 ) result = list(diff) if result: for line in result: my_print(line, end = '\n' if not line.startswith("---") else "") return 1 else: return 0 if comparison_mode: exit_code_stdout = compareOutput("stdout", stdout_cpython, stdout_nuitka) if ignore_stderr: exit_code_stderr = 0 else: exit_code_stderr = compareOutput("stderr", stderr_cpython, stderr_nuitka) exit_code_return = exit_cpython != exit_nuitka if exit_code_return: my_print( """\ Exit codes {exit_cpython:d} (CPython) != {exit_nuitka:d} (Nuitka)""".format( exit_cpython = exit_cpython, exit_nuitka = exit_nuitka ) ) exit_code = exit_code_stdout or exit_code_stderr or exit_code_return if exit_code: sys.exit("Error, outputs differed.") if expect_success and exit_cpython != 0: if silent_mode: displayOutput(stdout_cpython, stderr_cpython) sys.exit("Unexpected error exit from CPython.") if expect_failure and exit_cpython == 0: sys.exit("Unexpected success exit from CPython.") if remove_output: if not module_mode: if os.path.exists(nuitka_cmd2[0]): if os.name == "nt": # It appears there is a tiny lock race that we randomly cause, # likely because --run spawns a subprocess that might still # be doing the cleanup work. os.rename(nuitka_cmd2[0], nuitka_cmd2[0]+".away") for i in range(10): try: os.unlink(nuitka_cmd2[0]+".away") except OSError: time.sleep(2) continue else: break assert not os.path.exists(nuitka_cmd2[0]+".away") else: os.unlink(nuitka_cmd2[0]) else: if os.name == "nt": module_filename = os.path.basename(filename) + ".pyd" else: module_filename = os.path.basename(filename) + ".so" if os.path.exists(module_filename): os.unlink(module_filename) if comparison_mode and timing: my_print( "CPython took %.2fs vs %0.2fs Nuitka." % ( cpython_time, nuitka_time ) ) if comparison_mode and not silent_mode: my_print("OK, same outputs.") Nuitka-0.5.18.1/bin/compare_with_xml0000755000175000017500000000625012651071040017467 0ustar hayenhayen00000000000000#!/usr/bin/env python # Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import difflib import os import subprocess import sys nuitka1 = sys.argv[1] nuitka2 = sys.argv[2] filename = sys.argv[3] print( """\ Comparing output of '{filename}' using '{nuitka1}' <-> '{nuitka2}' ...""". format( filename = filename, nuitka1 = nuitka1, nuitka2 = nuitka2 ) ) extra_options = os.environ.get("NUITKA_EXTRA_OPTIONS", "") nuitka1_cmd = "{nuitka1} --dump-xml {filename}".format( nuitka1 = nuitka1, filename = filename ) process = subprocess.Popen( args = nuitka1_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True ) stdout_nuitka1, stderr_nuitka1 = process.communicate() exit_nuitka1 = process.returncode nuitka2_cmd = "{nuitka2} --dump-xml {filename}".format( nuitka2 = nuitka2, filename = filename ) process = subprocess.Popen( args = nuitka2_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True ) stdout_nuitka2, stderr_nuitka2 = process.communicate() exit_nuitka2 = process.returncode def makeDiffable(output): result = [] for line in output.split(b"\n"): line = str(line) result.append(line) return result fromdate = None todate = None def compareOutput(kind, out1, out2): diff = difflib.unified_diff( makeDiffable(out1), makeDiffable(out2), "{program} ({detail})".format( program = "nuitka1 " + filename, detail = kind ), "{program} ({detail})".format( program = "nuitka2 " + filename, detail = kind ), fromdate, todate, n = 3 ) result = list(diff) if result: for line in result: print(line, end = '\n' if not line.startswith("---") else "") return 1 else: return 0 exit_code_stdout = compareOutput("stdout", stdout_nuitka1, stdout_nuitka2) exit_code_return = exit_nuitka1 != exit_nuitka2 if exit_code_return: print( """\ Exit codes {exit_nuitka1:d} ({nuitka1}) != {exit_nuitka2:d} ({nuitka2})""". format( exit_nuitka1 = exit_nuitka1, nuitka1 = nuitka1, exit_nuitka2 = exit_nuitka2, nuitka2 = nuitka2 ) ) exit_code = exit_code_stdout or exit_code_return if exit_code: sys.exit("Error, outputs differed.") print("OK, same outputs.") Nuitka-0.5.18.1/setup.py0000644000175000017500000001462012651071040015142 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os os.chdir(os.path.dirname(__file__) or '.') scripts = ["bin/nuitka", "bin/nuitka-run"] # For Windows, there are batch files to launch Nuitka. if os.name == "nt": scripts += ["misc/nuitka.bat", "misc/nuitka-run.bat"] # Detect the version of Nuitka from its source directly. Without calling it, we # don't mean to pollute with ".pyc" files and similar effects. def detectVersion(): version_line, = [ line for line in open("nuitka/Options.py") if line.startswith("Nuitka V") ] return version_line.split('V')[1].strip() version = detectVersion() # The MSI installer enforces a 3 digit version number, which is stupid, but no # way around it, so we map our number to it, in some way. if os.name == "nt" and "bdist_msi" in sys.argv: # Pre-releases are always smaller, official releases get the "1". middle = 1 if "pre" not in version else 0 version = version.replace("pre", "") parts = version.split('.') major, first, last = parts[:3] hotfix = parts[3] if len(parts) > 3 else 0 version = '.'.join( "%s" % value for value in ( int(major)*10+int(first), middle, int(last)*10+int(hotfix) ) ) def find_packages(): result = [] for root, _dirnames, filenames in os.walk("nuitka"): # Ignore the inline copy of scons, these are not packages of Nuitka. if "scons-" in root: continue # Packages must contain "__init__.py" or they are merely directories. if "__init__.py" not in filenames: continue result.append( root.replace(os.path.sep,'.') ) return result package = find_packages() from distutils.command.install_scripts import install_scripts class NuitkaInstallScripts(install_scripts): """ This is a specialization of install_scripts that replaces the @LIBDIR@ with the configured directory for modules. If possible, the path is made relative to the directory for scripts. """ def initialize_options(self): install_scripts.initialize_options(self) self.install_lib = None def finalize_options(self): install_scripts.finalize_options(self) self.set_undefined_options("install", ("install_lib", "install_lib")) def run(self): install_scripts.run(self) if os.path.splitdrive(self.install_dir)[0] != \ os.path.splitdrive(self.install_lib)[0]: # can't make relative paths from one drive to another, so use an # absolute path instead libdir = self.install_lib else: common = os.path.commonprefix( (self.install_dir, self.install_lib ) ) rest = self.install_dir[len(common):] uplevel = len([n for n in os.path.split(rest) if n ]) libdir = uplevel * (".." + os.sep) + self.install_lib[len(common):] patch_bats = os.path.exists( os.path.join(self.install_dir, "Python.exe") ) for outfile in self.outfiles: fp = open(outfile, "rb") data = fp.read() fp.close() # skip binary files if b'\0' in data: continue data = data.replace(b"@LIBDIR@", libdir.encode("unicode_escape")) if patch_bats and outfile.endswith(".bat"): data = data.replace(b"..\\",b"") fp = open(outfile, "wb") fp.write(data) fp.close() cmdclass = { "install_scripts" : NuitkaInstallScripts } if os.path.exists("/usr/bin/scons") and \ "sdist" not in sys.argv and \ "bdist_wininst" not in sys.argv and \ "bdist_msi" not in sys.argv: scons_files = [] else: scons_files = [ "inline_copy/*/*.py", "inline_copy/*/*/*.py", "inline_copy/*/*/*/*.py", "inline_copy/*/*/*/*/*.py", "inline_copy/*/*/*/*/*/*.py", ] # Have different project names for MSI installers, so 32 and 64 bit versions do # not conflict. if "bdist_msi" in sys.argv: project_name = "Nuitka%s" % (64 if "AMD64" in sys.version else 32) else: project_name = "Nuitka" # Lets hack the byte_compile function so it doesn't byte compile Scons built-in # copy with Python3. if sys.version_info >= (3,): from distutils import util real_byte_compile = util.byte_compile def byte_compile(py_files, *args, **kw): py_files = [ py_file for py_file in py_files if "inline_copy" not in py_file ] real_byte_compile(py_files, *args, **kw) util.byte_compile = byte_compile from distutils.core import setup setup( name = project_name, license = "Apache License, Version 2.0", version = version, packages = find_packages(), scripts = scripts, cmdclass = cmdclass, package_data = { # Include extra files "" : ["*.txt", "*.rst", "*.cpp", "*.hpp", "*.ui"], "nuitka.build" : [ "SingleExe.scons", "static_src/*.cpp", "static_src/*/*.cpp", "static_src/*/*.h", "static_src/*/*.asm", "static_src/*/*.S", "include/*.hpp", "include/*/*.hpp", "include/*/*/*.hpp", ] + scons_files, "nuitka.gui" : [ "dialogs/*.ui", ], }, # metadata for upload to PyPI author = "Kay Hayen", author_email = "Kay.Hayen@gmail.com", url = "http://nuitka.net", description = """\ Python compiler with full language support and CPython compatibility""", keywords = "compiler,python,nuitka", ) Nuitka-0.5.18.1/lib/0000755000175000017500000000000012651072347014206 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/lib/hints.py0000644000175000017500000000141512651071040015673 0ustar hayenhayen00000000000000# Copyright 2016, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.18.1/doc/0000755000175000017500000000000012651072347014205 5ustar hayenhayen00000000000000Nuitka-0.5.18.1/doc/nuitka-run.10000644000175000017500000002112512651072342016360 0ustar hayenhayen00000000000000.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. .TH NUITKA-RUN "1" "January 2016" "nuitka-run 0.5.18.1" "User Commands" .SH NAME nuitka-run \- the Python compiler .SH SYNOPSIS .B nuitka-run [\fI\,options\/\fR] \fI\,main_module.py\/\fR .SH OPTIONS .TP \fB\-\-version\fR show program's version number and exit .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP \fB\-\-module\fR Create an extension module executable instead of a program. Defaults to off. .TP \fB\-\-standalone\fR, \fB\-\-portable\fR Enable standalone mode in build. This allows you to transfer the created binary to other machines without it relying on an existing Python installation. It implies these options: "\-\-recurse\-all \fB\-\-recursestdlib\fR". You may also want to use "\-\-pythonflag=no_site" to avoid the "site.py" module. Defaults to off. .TP \fB\-\-nofreeze\-stdlib\fR In standalone mode by default all modules of standard library will be frozen as bytecode. This compiles them all and as a result compilation time will increase very much. .TP \fB\-\-python\-version\fR=\fI\,PYTHON_VERSION\/\fR Major version of Python to be used, one of '2.6', \&'2.7', '3.2', '3.3', '3.4', or '3.5'. Defaults to what you run Nuitka with (currently 2.7) .TP \fB\-\-python\-debug\fR, \fB\-\-python\-dbg\fR Use debug version or not. Default uses what you are using to run Nuitka, most likely a non\-debug version. .TP \fB\-\-python\-flag\fR=\fI\,PYTHON_FLAGS\/\fR Python flags to use. Default uses what you are using to run Nuitka, this enforces a specific mode. These are options that also exist to standard Python executable. Currently supported: "\-S" (alias nosite), "static_hashes" (not use Randomization), "no_warnings" (do not give Python runtime warnings). Default empty. .TP \fB\-\-warn\-implicit\-exceptions\fR Given warnings for implicit exceptions detected at compile time. .SS Control the recursion into imported modules: .BR .TP \fB\-\-recurse\-stdlib\fR Also descend into imported modules from standard library. Defaults to off. .TP \fB\-\-recurse\-none\fR When \fB\-\-recurse\-none\fR is used, do not descend into any imported modules at all, overrides all other recursion options. Defaults to off. .TP \fB\-\-recurse\-all\fR, \fB\-\-recurse\-on\fR When \fB\-\-recurse\-all\fR is used, attempt to descend into all imported modules. Defaults to off. .TP \fB\-\-recurse\-to\fR=\fI\,MODULE\/\fR/PACKAGE Recurse to that module, or if a package, to the whole package. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-not\-to\fR=\fI\,MODULE\/\fR/PACKAGE Do not recurse to that module, or if a package, to the whole package in any case, overrides all other options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-plugins\fR=\fI\,MODULE\/\fR/PACKAGE, \fB\-\-recurse\-directory\fR=\fI\,MODULE\/\fR/PACKAGE Recurse into that directory, no matter if it's used by the given main program in a visible form. Overrides all other recursion options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-files\fR=\fI\,PATTERN\/\fR, \fB\-\-recurse\-pattern\fR=\fI\,PATTERN\/\fR Recurse into files matching the PATTERN. Overrides all recursion other options. Can be given multiple times. Default empty. .SS Immediate execution after compilation: .BR .TP \fB\-\-run\fR, \fB\-\-execute\fR Execute immediately the created binary (or import the compiled module). Defaults to on. .TP \fB\-\-debugger\fR, \fB\-\-gdb\fR Execute inside "gdb" to automatically get a stack trace. Defaults to off. .TP \fB\-\-execute\-with\-pythonpath\fR, \fB\-\-keep\-pythonpath\fR When immediately executing the created binary (\fB\-\-execute\fR), don't reset PYTHONPATH. When all modules are successfully included, you ought to not need PYTHONPATH anymore. .SS Dump options for internal tree: .BR .TP \fB\-\-dump\-xml\fR, \fB\-\-xml\fR Dump the final result of optimization as XML, then exit. .TP \fB\-\-display\-tree\fR Display the final result of optimization in a GUI, then exit. .SS Code generation choices: .BR .TP \fB\-\-improved\fR, \fB\-\-enhanced\fR Allow minor deviations from CPython behavior, e.g. better tracebacks, which are not really incompatible, but different. .TP \fB\-\-file\-reference\-choice\fR=\fI\,FILE_REFERENCE_MODE\/\fR Select what value "__file__" is going to be. With "runtime" (default for standalone binary mode and module mode), the created binaries and modules, use the location of themselves to deduct the value of "__file__". Included packages pretend to be in directories below that location. This allows you to include data files in deployments. If you merely seek acceleration, it's better for you to use the "original" value, where the source files location will be used. With "frozen" a notation "" is used. For compatibility reasons, the "__file__" value will always have ".py" suffix independent of what it really is. .SS Output directory choices: .BR .TP \fB\-\-output\-dir\fR=\fI\,DIRECTORY\/\fR Specify where intermediate and final output files should be put. DIRECTORY will be populated with C++ files, object files, etc. Defaults to current directory. .TP \fB\-\-remove\-output\fR Removes the build directory after producing the module or exe file. Defaults to off. .SS Debug features: .BR .TP \fB\-\-debug\fR Executing all self checks possible to find errors in Nuitka, do not use for production. Defaults to off. .TP \fB\-\-unstripped\fR, \fB\-\-no\-strip\fR, \fB\-\-unstriped\fR Keep debug info in the resulting object file for better debugger interaction. Defaults to off. .TP \fB\-\-profile\fR Enable vmprof based profiling of time spent. Defaults to off. .TP \fB\-\-graph\fR Create graph of optimization process. Defaults to off. .TP \fB\-\-trace\-execution\fR Traced execution output, output the line of code before executing it. Defaults to off. .TP \fB\-\-recompile\-c\fR++\-only Take existing files and compile them again.Allows compiling edited C++ files with the C++ compiler for quick debugging changes to the generated source. Defaults to off. Depends on compiling Python source to determine which files it should look at. .TP \fB\-\-generate\-c\fR++\-only Generate only C++ source code, and do not compile it to binary or module. This is for debugging and code coverage analysis that doesn't waste CPU. Defaults to off. .TP \fB\-\-experimental\fR Use features declared as 'experimental'. May have no effect if no experimental features are present in the code. Defaults to off. .SS Backend C++ compiler choice: .BR .TP \fB\-\-clang\fR Enforce the use of clang (needs clang 3.2 or higher). Defaults to off. .TP \fB\-\-mingw\fR Enforce the use of MinGW on Windows. Defaults to off. .TP \fB\-\-msvc\fR=\fI\,MSVC\/\fR Enforce the use of specific MSVC version on Windows. Allowed values are e.g. 9.0, 9.0exp, specify an illegal value for a list of installed compilers. Defaults to the most recent version. .TP \fB\-j\fR N, \fB\-\-jobs\fR=\fI\,N\/\fR Specify the allowed number of parallel C++ compiler jobs. Defaults to the system CPU count. .TP \fB\-\-lto\fR Use link time optimizations if available and usable (g++ 4.6 and higher). Defaults to off. .SS Tracing features: .BR .TP \fB\-\-show\-scons\fR Operate Scons in non\-quiet mode, showing the executed commands. Defaults to off. .TP \fB\-\-show\-progress\fR Provide progress information and statistics. Defaults to off. .TP \fB\-\-show\-memory\fR Provide memory information and statistics. Defaults to off. .TP \fB\-\-show\-modules\fR Provide a final summary on included modules. Defaults to off. .TP \fB\-\-verbose\fR Output details of actions taken, esp. in optimizations. Can become a lot. Defaults to off. .SS Windows specific output control: .BR .TP \fB\-\-windows\-disable\-console\fR When compiling for Windows, disable the console window. Defaults to off. .TP \fB\-\-windows\-icon\fR=\fI\,ICON_PATH\/\fR, \fB\-\-icon\fR=\fI\,ICON_PATH\/\fR Add executable icon (Windows only). .SS Plugin control: .BR .TP \fB\-\-plugin\-enable\fR=\fI\,PLUGINS_ENABLED\/\fR, \fB\-\-enable\-plugin\fR=\fI\,PLUGINS_ENABLED\/\fR Enabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-disable\fR=\fI\,PLUGINS_DISABLED\/\fR, \fB\-\-disable\-plugin\fR=\fI\,PLUGINS_DISABLED\/\fR Disabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-no\-detection\fR Plugins can detect if they might be used, and the you can disable the warning via \fB\-\-plugin\-disable\fR=\fI\,pluginthat\-warned\/\fR, or you can use this option to disable the mechanism entirely, which also speeds up compilation slightly of course as this detection code is run in vain once you are certain of which plug\-ins to use. Defaults to off. Nuitka-0.5.18.1/doc/nuitka.10000644000175000017500000002536312651072342015566 0ustar hayenhayen00000000000000.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.3. .TH NUITKA "1" "January 2016" "nuitka 0.5.18.1" "User Commands" .SH NAME nuitka \- the Python compiler .SH SYNOPSIS .B nuitka [\fI\,--module\/\fR] [\fI\,--execute\/\fR] [\fI\,options\/\fR] \fI\,main_module.py\/\fR .SH OPTIONS .TP \fB\-\-version\fR show program's version number and exit .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP \fB\-\-module\fR Create an extension module executable instead of a program. Defaults to off. .TP \fB\-\-standalone\fR, \fB\-\-portable\fR Enable standalone mode in build. This allows you to transfer the created binary to other machines without it relying on an existing Python installation. It implies these options: "\-\-recurse\-all \fB\-\-recursestdlib\fR". You may also want to use "\-\-pythonflag=no_site" to avoid the "site.py" module. Defaults to off. .TP \fB\-\-nofreeze\-stdlib\fR In standalone mode by default all modules of standard library will be frozen as bytecode. This compiles them all and as a result compilation time will increase very much. .TP \fB\-\-python\-version\fR=\fI\,PYTHON_VERSION\/\fR Major version of Python to be used, one of '2.6', \&'2.7', '3.2', '3.3', '3.4', or '3.5'. Defaults to what you run Nuitka with (currently 2.7) .TP \fB\-\-python\-debug\fR, \fB\-\-python\-dbg\fR Use debug version or not. Default uses what you are using to run Nuitka, most likely a non\-debug version. .TP \fB\-\-python\-flag\fR=\fI\,PYTHON_FLAGS\/\fR Python flags to use. Default uses what you are using to run Nuitka, this enforces a specific mode. These are options that also exist to standard Python executable. Currently supported: "\-S" (alias nosite), "static_hashes" (not use Randomization), "no_warnings" (do not give Python runtime warnings). Default empty. .TP \fB\-\-warn\-implicit\-exceptions\fR Given warnings for implicit exceptions detected at compile time. .SS Control the recursion into imported modules: .BR .TP \fB\-\-recurse\-stdlib\fR Also descend into imported modules from standard library. Defaults to off. .TP \fB\-\-recurse\-none\fR When \fB\-\-recurse\-none\fR is used, do not descend into any imported modules at all, overrides all other recursion options. Defaults to off. .TP \fB\-\-recurse\-all\fR, \fB\-\-recurse\-on\fR When \fB\-\-recurse\-all\fR is used, attempt to descend into all imported modules. Defaults to off. .TP \fB\-\-recurse\-to\fR=\fI\,MODULE\/\fR/PACKAGE Recurse to that module, or if a package, to the whole package. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-not\-to\fR=\fI\,MODULE\/\fR/PACKAGE Do not recurse to that module, or if a package, to the whole package in any case, overrides all other options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-plugins\fR=\fI\,MODULE\/\fR/PACKAGE, \fB\-\-recurse\-directory\fR=\fI\,MODULE\/\fR/PACKAGE Recurse into that directory, no matter if it's used by the given main program in a visible form. Overrides all other recursion options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-files\fR=\fI\,PATTERN\/\fR, \fB\-\-recurse\-pattern\fR=\fI\,PATTERN\/\fR Recurse into files matching the PATTERN. Overrides all recursion other options. Can be given multiple times. Default empty. .SS Immediate execution after compilation: .BR .TP \fB\-\-run\fR, \fB\-\-execute\fR Execute immediately the created binary (or import the compiled module). Defaults to off. .TP \fB\-\-debugger\fR, \fB\-\-gdb\fR Execute inside "gdb" to automatically get a stack trace. Defaults to off. .TP \fB\-\-execute\-with\-pythonpath\fR, \fB\-\-keep\-pythonpath\fR When immediately executing the created binary (\fB\-\-execute\fR), don't reset PYTHONPATH. When all modules are successfully included, you ought to not need PYTHONPATH anymore. .SS Dump options for internal tree: .BR .TP \fB\-\-dump\-xml\fR, \fB\-\-xml\fR Dump the final result of optimization as XML, then exit. .TP \fB\-\-display\-tree\fR Display the final result of optimization in a GUI, then exit. .SS Code generation choices: .BR .TP \fB\-\-improved\fR, \fB\-\-enhanced\fR Allow minor deviations from CPython behavior, e.g. better tracebacks, which are not really incompatible, but different. .TP \fB\-\-file\-reference\-choice\fR=\fI\,FILE_REFERENCE_MODE\/\fR Select what value "__file__" is going to be. With "runtime" (default for standalone binary mode and module mode), the created binaries and modules, use the location of themselves to deduct the value of "__file__". Included packages pretend to be in directories below that location. This allows you to include data files in deployments. If you merely seek acceleration, it's better for you to use the "original" value, where the source files location will be used. With "frozen" a notation "" is used. For compatibility reasons, the "__file__" value will always have ".py" suffix independent of what it really is. .SS Output directory choices: .BR .TP \fB\-\-output\-dir\fR=\fI\,DIRECTORY\/\fR Specify where intermediate and final output files should be put. DIRECTORY will be populated with C++ files, object files, etc. Defaults to current directory. .TP \fB\-\-remove\-output\fR Removes the build directory after producing the module or exe file. Defaults to off. .SS Debug features: .BR .TP \fB\-\-debug\fR Executing all self checks possible to find errors in Nuitka, do not use for production. Defaults to off. .TP \fB\-\-unstripped\fR, \fB\-\-no\-strip\fR, \fB\-\-unstriped\fR Keep debug info in the resulting object file for better debugger interaction. Defaults to off. .TP \fB\-\-profile\fR Enable vmprof based profiling of time spent. Defaults to off. .TP \fB\-\-graph\fR Create graph of optimization process. Defaults to off. .TP \fB\-\-trace\-execution\fR Traced execution output, output the line of code before executing it. Defaults to off. .TP \fB\-\-recompile\-c\fR++\-only Take existing files and compile them again.Allows compiling edited C++ files with the C++ compiler for quick debugging changes to the generated source. Defaults to off. Depends on compiling Python source to determine which files it should look at. .TP \fB\-\-generate\-c\fR++\-only Generate only C++ source code, and do not compile it to binary or module. This is for debugging and code coverage analysis that doesn't waste CPU. Defaults to off. .TP \fB\-\-experimental\fR Use features declared as 'experimental'. May have no effect if no experimental features are present in the code. Defaults to off. .SS Backend C++ compiler choice: .BR .TP \fB\-\-clang\fR Enforce the use of clang (needs clang 3.2 or higher). Defaults to off. .TP \fB\-\-mingw\fR Enforce the use of MinGW on Windows. Defaults to off. .TP \fB\-\-msvc\fR=\fI\,MSVC\/\fR Enforce the use of specific MSVC version on Windows. Allowed values are e.g. 9.0, 9.0exp, specify an illegal value for a list of installed compilers. Defaults to the most recent version. .TP \fB\-j\fR N, \fB\-\-jobs\fR=\fI\,N\/\fR Specify the allowed number of parallel C++ compiler jobs. Defaults to the system CPU count. .TP \fB\-\-lto\fR Use link time optimizations if available and usable (g++ 4.6 and higher). Defaults to off. .SS Tracing features: .BR .TP \fB\-\-show\-scons\fR Operate Scons in non\-quiet mode, showing the executed commands. Defaults to off. .TP \fB\-\-show\-progress\fR Provide progress information and statistics. Defaults to off. .TP \fB\-\-show\-memory\fR Provide memory information and statistics. Defaults to off. .TP \fB\-\-show\-modules\fR Provide a final summary on included modules. Defaults to off. .TP \fB\-\-verbose\fR Output details of actions taken, esp. in optimizations. Can become a lot. Defaults to off. .SS Windows specific output control: .BR .TP \fB\-\-windows\-disable\-console\fR When compiling for Windows, disable the console window. Defaults to off. .TP \fB\-\-windows\-icon\fR=\fI\,ICON_PATH\/\fR, \fB\-\-icon\fR=\fI\,ICON_PATH\/\fR Add executable icon (Windows only). .SS Plugin control: .BR .TP \fB\-\-plugin\-enable\fR=\fI\,PLUGINS_ENABLED\/\fR, \fB\-\-enable\-plugin\fR=\fI\,PLUGINS_ENABLED\/\fR Enabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-disable\fR=\fI\,PLUGINS_DISABLED\/\fR, \fB\-\-disable\-plugin\fR=\fI\,PLUGINS_DISABLED\/\fR Disabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-no\-detection\fR Plugins can detect if they might be used, and the you can disable the warning via \fB\-\-plugin\-disable\fR=\fI\,pluginthat\-warned\/\fR, or you can use this option to disable the mechanism entirely, which also speeds up compilation slightly of course as this detection code is run in vain once you are certain of which plug\-ins to use. Defaults to off. .SH EXAMPLES Compile a python file "some_module.py" to a module "some_module.so": .IP \f(CW$ nuitka some_module.py\fR .PP Compile a python program "some_program.py" to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-exe some_program.py\fR .PP Compile a python program "some_program.py" and the package "some_package" it uses to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-exe \-\-recurse\-to=some_package some_program.py\fR .PP Compile a python program "some_program.py" and all the modules it uses to an executable "some_program.exe". Then execute it immediately when ready: .IP \f(CW$ nuitka \-\-exe \-\-execute \-\-recurse\-all some_program.py\fR .PP Compile a python program "some_program.py" and the modules it uses (even standard library) to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-recurse\-all \-\-recurse\-stdlib some_program.py \-\-exe\fR .PP Compile a python program "some_program.py" and the modules it uses to an executable "some_program.exe". Keep the debug information, so valrind, gdb, etc. work nice. Note: This will *not* degrade performance: .IP \f(CW$ nuitka \-\-unstriped \-\-recurse\-all some_program.py \-\-exe\fR .PP Compile a python program "some_program.py" and the modules it uses to an executable "some_program.exe". Perform all kinds of checks about correctness of the generated C++ and run\-time checks. Note: This will degrade performance and should only be used to debug Nuitka: .IP \f(CW$ nuitka \-\-debug \-\-recurse\-all some_program.py \-\-exe\fR .PP Compile a python program "some_program.py" and the modules it uses to an executable "some_program.exe". Perform all kinds of checks about correctness of the generated C++ and run\-time checks. Also use the debug Python library, which does its own checks. Note: This will degrade performance and should only be used to debug Nuitka: .IP \f(CW$ nuitka \-\-debug \-\-python-debug \-\-recurse\-all some_program.py \-\-exe\fR .PP Compile a python program "some_program.py" and the plugins modules it loads at run time to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-recurse\-all \-\-recurse\-directory=plugins_dir some_program.py \-\-exe\fR .PP