Nuitka-0.5.28.2/0000755000372000001440000000000013207540420013460 5ustar hayenusers00000000000000Nuitka-0.5.28.2/misc/0000755000372000001440000000000013207540420014413 5ustar hayenusers00000000000000Nuitka-0.5.28.2/misc/nuitka-run.bat0000644000372000001440000000157713112214770017213 0ustar hayenusers00000000000000@echo off rem Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com rem rem Part of "Nuitka", an optimizing Python compiler that is compatible and rem integrates with CPython, but also works on its own. rem rem Licensed under the Apache License, Version 2.0 (the "License"); rem you may not use this file except in compliance with the License. rem You may obtain a copy of the License at rem rem http://www.apache.org/licenses/LICENSE-2.0 rem rem Unless required by applicable law or agreed to in writing, software rem distributed under the License is distributed on an "AS IS" BASIS, rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. rem See the License for the specific language governing permissions and rem limitations under the License. rem setlocal "%~dp0nuitka.bat --run %* endlocal Nuitka-0.5.28.2/misc/print-all-sources.sh0000755000372000001440000000210013112214770020327 0ustar hayenusers00000000000000#!/bin/sh # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # cd `dirname $0`/.. find nuitka -name \*.py -a \! -path *inline_copy* find bin -name \*.py find nuitka/build/static_src -name \*.c find nuitka/build/include/nuitka -name \*.h find nuitka/build/ -name \*.scons find misc -name \*.sh find bin -name \*.sh echo Developer_Manual.rst echo Changelog.rst Nuitka-0.5.28.2/misc/nuitka.bat0000644000372000001440000000172513112214770016404 0ustar hayenusers00000000000000@echo off rem Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com rem rem Part of "Nuitka", an optimizing Python compiler that is compatible and rem integrates with CPython, but also works on its own. rem rem Licensed under the Apache License, Version 2.0 (the "License"); rem you may not use this file except in compliance with the License. rem You may obtain a copy of the License at rem rem http://www.apache.org/licenses/LICENSE-2.0 rem rem Unless required by applicable law or agreed to in writing, software rem distributed under the License is distributed on an "AS IS" BASIS, rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. rem See the License for the specific language governing permissions and rem limitations under the License. rem setlocal if exist "%~dp0..\python.exe" ( "%~dp0..\python" "%~dp0nuitka" %* ) else ( "%~dp0python" "%~dp0nuitka" %* ) endlocal Nuitka-0.5.28.2/setup.py0000644000372000001440000002204613207537242015206 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os import sys from distutils.command.install_scripts import install_scripts from distutils.core import setup os.chdir(os.path.dirname(__file__) or '.') scripts = ["bin/nuitka", "bin/nuitka-run"] # For Windows, there are batch files to launch Nuitka. if os.name == "nt": scripts += ["misc/nuitka.bat", "misc/nuitka-run.bat"] # Detect the version of Nuitka from its source directly. Without calling it, we # don't mean to pollute with ".pyc" files and similar effects. def detectVersion(): version_line, = [ line for line in open("nuitka/Version.py") if line.startswith("Nuitka V") ] return version_line.split('V')[1].strip() version = detectVersion() # The MSI installer enforces a 3 digit version number, which is stupid, but no # way around it, so we map our number to it, in some way. if os.name == "nt" and "bdist_msi" in sys.argv: # Pre-releases are always smaller, official releases get the "1". middle = 1 if "rc" not in version else 0 version = version.replace("rc", "") parts = version.split('.') major, first, last = parts[:3] hotfix = parts[3] if len(parts) > 3 else 0 version = '.'.join( "%s" % value for value in ( int(major)*10+int(first), middle, int(last)*10+int(hotfix) ) ) def findNuitkaPackages(): result = [] for root, dirnames, filenames in os.walk("nuitka"): # Ignore the inline copy of scons, these are not packages of Nuitka. if "scons-" in root: continue # Packages must contain "__init__.py" or they are merely directories # in Nuitka as we are Python2 compatible. if "__init__.py" not in filenames: continue # The "release" namespace is code used to release, but not itself for # release, same goes for "qualit"y. if "release" in dirnames: dirnames.remove("release") if "quality" in dirnames: dirnames.remove("quality") result.append( root.replace(os.path.sep,'.') ) return result class NuitkaInstallScripts(install_scripts): """ This is a specialization of install_scripts that replaces the @LIBDIR@ with the configured directory for modules. If possible, the path is made relative to the directory for scripts. """ def initialize_options(self): install_scripts.initialize_options(self) self.install_lib = None def finalize_options(self): install_scripts.finalize_options(self) self.set_undefined_options("install", ("install_lib", "install_lib")) def run(self): install_scripts.run(self) if os.path.splitdrive(self.install_dir)[0] != \ os.path.splitdrive(self.install_lib)[0]: # can't make relative paths from one drive to another, so use an # absolute path instead libdir = self.install_lib else: common = os.path.commonprefix( (self.install_dir, self.install_lib) ) rest = self.install_dir[len(common):] uplevel = len([n for n in os.path.split(rest) if n ]) libdir = uplevel * (".." + os.sep) + self.install_lib[len(common):] for outfile in self.outfiles: fp = open(outfile, "rb") data = fp.read() fp.close() # skip binary files if b'\0' in data: continue old_data = data data = data.replace(b"@LIBDIR@", libdir.encode("unicode_escape")) if data != old_data: fp = open(outfile, "wb") fp.write(data) fp.close() cmdclass = { "install_scripts" : NuitkaInstallScripts, } # Fix for "develop", where the generated scripts from easy install are not # capable of running in their re-executing, not finding pkg_resources anymore. if "develop" in sys.argv: try: import setuptools.command.easy_install except ImportError: pass else: orig_easy_install = setuptools.command.easy_install.easy_install class NuitkaEasyInstall(setuptools.command.easy_install.easy_install): @staticmethod def _load_template(dev_path): result = orig_easy_install._load_template(dev_path) result = result.replace( "__import__('pkg_resources')", "# __import__('pkg_resources')", ) return result setuptools.command.easy_install.easy_install = NuitkaEasyInstall if os.path.exists("/usr/bin/scons") and \ "sdist" not in sys.argv and \ "bdist_wininst" not in sys.argv and \ "bdist_msi" not in sys.argv: scons_files = [] else: scons_files = [ "inline_copy/*/*.py", "inline_copy/*/*/*.py", "inline_copy/*/*/*/*.py", "inline_copy/*/*/*/*/*.py", "inline_copy/*/*/*/*/*/*.py", ] # Have different project names for MSI installers, so 32 and 64 bit versions do # not conflict. if "bdist_msi" in sys.argv: project_name = "Nuitka%s" % (64 if "AMD64" in sys.version else 32) else: project_name = "Nuitka" # Lets hack the byte_compile function so it doesn't byte compile Scons built-in # copy with Python3. if sys.version_info >= (3,): from distutils import util real_byte_compile = util.byte_compile def byte_compile(py_files, *args, **kw): py_files = [ py_file for py_file in py_files if "inline_copy" not in py_file ] real_byte_compile(py_files, *args, **kw) util.byte_compile = byte_compile setup( name = project_name, license = "Apache License, Version 2.0", version = version, classifiers = [ # Nuitka is mature even "Development Status :: 5 - Production/Stable", # Indicate who Nuitka is for "Intended Audience :: Developers", "Intended Audience :: Science/Research", # Nuitka is a compiler and a build tool as such. "Topic :: Software Development :: Compilers", "Topic :: Software Development :: Build Tools", # Is has a weak subset of PyLint, but aims for more long term "Topic :: Software Development :: Quality Assurance", # Nuitka standalone mode aims at distribution "Topic :: System :: Software Distribution", # Python2 supported versions. "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", # Python3 supported versions. "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", # We depend on CPython. "Programming Language :: Python :: Implementation :: CPython", # We generate C intermediate code and implement part of the # run time environment in C. Actually C11. "Programming Language :: C", # Supported OSes are many "Operating System :: POSIX :: Linux", "Operating System :: POSIX :: BSD :: FreeBSD", "Operating System :: POSIX :: BSD :: NetBSD", "Operating System :: POSIX :: BSD :: OpenBSD", "Operating System :: Microsoft :: Windows", # License "License :: OSI Approved :: Apache Software License", ], packages = findNuitkaPackages(), scripts = scripts, cmdclass = cmdclass, package_data = { # Include extra files "" : ["*.txt", "*.rst", "*.c", "*.h", "*.ui"], "nuitka.build" : [ "SingleExe.scons", "static_src/*.c", "static_src/*/*.c", "static_src/*/*.h", "static_src/*/*.asm", "static_src/*/*.S", "include/*.h", "include/*/*.h", "include/*/*/*.h", ] + scons_files, "nuitka.gui" : [ "dialogs/*.ui", ], }, # metadata for upload to PyPI author = "Kay Hayen", author_email = "Kay.Hayen@gmail.com", url = "http://nuitka.net", description = """\ Python compiler with full language support and CPython compatibility""", keywords = "compiler,python,nuitka", ) Nuitka-0.5.28.2/MANIFEST.in0000644000372000001440000000204313207537242015225 0ustar hayenusers00000000000000include LICENSE.txt include MANIFEST.in include README.rst README.pdf include Changelog.rst Changelog.pdf include Developer_Manual.rst Developer_Manual.pdf include doc/*.1 include bin/compare_with_cpython include bin/compare_with_xml include bin/check-nuitka-with-pylint include bin/autoformat-nuitka-source include misc/*.sh include misc/*.bat include tests/run-tests # Logo with source include doc/Logo/Nuitka-Logo-Symbol.svg include doc/Logo/Nuitka-Logo-Vertical.svg include doc/Logo/Nuitka-Logo-Horizontal.svg include doc/images/Nuitka-Logo-Symbol.png include doc/images/Nuitka-Logo-Vertical.png include doc/images/Nuitka-Logo-Horizontal.png recursive-include lib *.py # Core tests are included along with Nuitka itself. recursive-include tests/basics *.py recursive-include tests/syntax *.py recursive-include tests/packages *.py recursive-include tests/programs *.py include tests/programs/pkgutil_usage/package/DATA_FILE.txt recursive-include tests/optimizations *.py recursive-include tests/standalone *.py recursive-include tests/reflected *.py Nuitka-0.5.28.2/bin/0000755000372000001440000000000013207540420014230 5ustar hayenusers00000000000000Nuitka-0.5.28.2/bin/check-nuitka-with-pylint0000755000372000001440000000224213207537242021022 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Launcher for pylint checker tool. """ import os import sys # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), "..", ) ) ) from nuitka.tools.quality.pylint.__main__ import main # isort:skip main() Nuitka-0.5.28.2/bin/nuitka0000755000372000001440000001565213134660221015463 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This is the main program of Nuitka, it checks the options and then translates one or more modules to a C-ish source code using Python C/API in a "*.build" directory and then compiles that to either an executable or an extension module or package, that can contain all used modules too. """ # Import as little as possible initially, because we might be re-executing # soon. import os import sys import warnings # LIBDIR trick start (marker for removal on platforms that don't need it) libdir = "@LIBDIR@" # Two cases: if libdir != '@' "LIBDIR" '@': # Changed by our "distutils" hook, then use the given path. if not os.path.isabs(libdir): libdir = os.path.join( os.path.dirname(os.path.realpath(__file__)), libdir ) libdir = os.path.abspath(libdir) sys.path.insert( 0, libdir ) else: # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), ".." ) ) ) # LIBDIR trick end (marker for removal on platforms that don't need it) # PyLint for Python3 thinks we import from ourselves if we really # import from package, pylint:disable=I0021,no-name-in-module if "NUITKA_PYTHONPATH" in os.environ: # Restore the PYTHONPATH gained from the site module, that we chose not # to have imported. pylint: disable=eval-used sys.path = eval(os.environ["NUITKA_PYTHONPATH"]) del os.environ["NUITKA_PYTHONPATH"] from nuitka import Options # isort:skip Options.parseArgs() from nuitka.utils import Utils, Execution # isort:skip import logging # isort:skip logging.basicConfig(format = "Nuitka:%(levelname)s:%(message)s") # We don't care, and these are triggered by run time calculations of "range" and # others, while on python2.7 they are disabled by default. warnings.simplefilter("ignore", DeprecationWarning) # We will run with the Python configuration as specified by the user, if it does # not match, we restart ourselves with matching configuration. needs_reexec = False current_version = "%d.%d" % (sys.version_info[0], sys.version_info[1]) # We support to execute with a specified version. intended_version = Options.getIntendedPythonVersion() if intended_version is None: intended_version = current_version # If it's a different version, we find it by guessing it, otherwise we use the # one previously used. if current_version != intended_version: if Utils.getOS() == "Windows": python_binary = Execution.getPythonExePathWindows( intended_version, Options.getIntendedPythonArch() ) else: python_binary = Execution.getExecutablePath("python" + intended_version) if python_binary is None: sys.exit( "Error, cannot find Python %s binary in PATH (%s)." % ( intended_version, os.environ.get("PATH", "") ) ) needs_reexec = True else: python_binary = sys.executable python_flags = Options.getPythonFlags() if sys.flags.no_site == 0: needs_reexec = True # The hash randomization totally changes the created source code created, # changing it every single time Nuitka is run. This kills any attempt at # caching it, and comparing generated source code. While the created binary # actually may still use it, during compilation we don't want to. So lets # disable it. if os.environ.get("PYTHONHASHSEED", "-1") != '0': needs_reexec = True # In case we need to re-execute. if needs_reexec: if not Options.isAllowedToReexecute(): sys.exit("Error, not allowed to re-execute, but that would be needed.") # Execute with full path as the process name, so it can find itself and its # libraries. args = [ python_binary, python_binary, ] # Potentially give Python command line flags as necessary. args.append("-S") # Same arguments as before. args += sys.argv + list(Options.getMainArgs()) if current_version == intended_version: os.environ["NUITKA_PYTHONPATH"] = repr( sys.path ) from nuitka.importing.PreloadedPackages import detectPreLoadedPackagePaths, detectPthImportedPackages os.environ["NUITKA_NAMESPACES"] = repr( detectPreLoadedPackagePaths() ) if "site" in sys.modules: os.environ["NUITKA_SITE_FILENAME"] = sys.modules["site"].__file__ os.environ["NUITKA_PTH_IMPORTED"] = repr(detectPthImportedPackages()) os.environ["NUITKA_SITE_FLAG"] = str(sys.flags.no_site) \ if "no_site" not in Options.getPythonFlags() \ else '1' os.environ["PYTHONHASHSEED"] = '0' Execution.callExec(args) if Options.isShowMemory(): from nuitka.utils import MemoryUsage MemoryUsage.startMemoryTracing() # Inform the user about potential issues. if current_version not in Options.getSupportedPythonVersions(): # Do not disturb run of automatic tests, detected from the presence of # that environment variable. if "PYTHON" not in os.environ: logging.warning( "The version '%s' is not currently supported. Expect problems.", current_version ) if "NUITKA_NAMESPACES" in os.environ: # Restore the detected name space packages, that were force loaded in # site.py, and will need a free pass later on. pylint: disable=eval-used from nuitka.importing.PreloadedPackages import setPreloadedPackagePaths setPreloadedPackagePaths(eval(os.environ["NUITKA_NAMESPACES"])) del os.environ["NUITKA_NAMESPACES"] if "NUITKA_PTH_IMPORTED" in os.environ: # Restore the packages that the ".pth" files asked to import. # pylint: disable=eval-used from nuitka.importing.PreloadedPackages import setPthImportedPackages setPthImportedPackages(eval(os.environ["NUITKA_PTH_IMPORTED"])) del os.environ["NUITKA_PTH_IMPORTED"] # Now the main program. from nuitka import MainControl # isort:skip MainControl.main() if Options.isShowMemory(): MemoryUsage.showMemoryTrace() Nuitka-0.5.28.2/bin/compare_with_cpython0000755000372000001440000000234513207537242020417 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Launcher for output comparison tool. """ import os import sys # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), "..", ) ) ) sys.path.insert( 1, "/usr/share/nuitka" ) from nuitka.tools.testing.compare_with_cpython.__main__ import main # isort:skip main() Nuitka-0.5.28.2/bin/nuitka-run0000777000372000037200000000000012711355343017440 2nuitkaustar hayenhayen00000000000000Nuitka-0.5.28.2/bin/autoformat-nuitka-source0000755000372000001440000000224213207537242021136 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Launcher for autoformat tool. """ import os import sys # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), "..", ) ) ) from nuitka.tools.quality.autoformat.__main__ import main # isort:skip main() Nuitka-0.5.28.2/bin/compare_with_xml0000755000372000001440000000634513112214770017530 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tool to compare XML outputs of two Nuitka versions. """ from __future__ import print_function import difflib import os import subprocess import sys nuitka1 = sys.argv[1] nuitka2 = sys.argv[2] filename = sys.argv[3] print( """\ Comparing output of '{filename}' using '{nuitka1}' <-> '{nuitka2}' ...""". format( filename = filename, nuitka1 = nuitka1, nuitka2 = nuitka2 ) ) extra_options = os.environ.get("NUITKA_EXTRA_OPTIONS", "") nuitka1_cmd = "{nuitka1} --dump-xml {filename}".format( nuitka1 = nuitka1, filename = filename ) process = subprocess.Popen( args = nuitka1_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True ) stdout_nuitka1, stderr_nuitka1 = process.communicate() exit_nuitka1 = process.returncode nuitka2_cmd = "{nuitka2} --dump-xml {filename}".format( nuitka2 = nuitka2, filename = filename ) process = subprocess.Popen( args = nuitka2_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True ) stdout_nuitka2, stderr_nuitka2 = process.communicate() exit_nuitka2 = process.returncode def makeDiffable(output): result = [] for line in output.split(b"\n"): line = str(line) result.append(line) return result fromdate = None todate = None def compareOutput(kind, out1, out2): diff = difflib.unified_diff( makeDiffable(out1), makeDiffable(out2), "{program} ({detail})".format( program = "nuitka1 " + filename, detail = kind ), "{program} ({detail})".format( program = "nuitka2 " + filename, detail = kind ), fromdate, todate, n = 3 ) result = list(diff) if result: for line in result: print(line, end = '\n' if not line.startswith("---") else "") return 1 else: return 0 exit_code_stdout = compareOutput("stdout", stdout_nuitka1, stdout_nuitka2) exit_code_return = exit_nuitka1 != exit_nuitka2 if exit_code_return: print( """\ Exit codes {exit_nuitka1:d} ({nuitka1}) != {exit_nuitka2:d} ({nuitka2})""". format( exit_nuitka1 = exit_nuitka1, nuitka1 = nuitka1, exit_nuitka2 = exit_nuitka2, nuitka2 = nuitka2 ) ) exit_code = exit_code_stdout or exit_code_return if exit_code: sys.exit("Error, outputs differed.") print("OK, same outputs.") Nuitka-0.5.28.2/nuitka/0000755000372000001440000000000013207540420014753 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/freezer/0000755000372000001440000000000013207540420016415 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/freezer/BytecodeModuleFreezer.py0000644000372000001440000000460613122472300023221 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Freezer for bytecode compiled modules. Not real C compiled modules. This is including modules as bytecode and mostly intended for modules, where we know compiling it useless or does not make much sense, or for standalone mode to access modules during CPython library init that cannot be avoided. The level of compatibility for C compiled stuff is so high that this is not needed except for technical reasons. """ from logging import info from nuitka import Options from nuitka.codegen import ConstantCodes from nuitka.codegen.Indentation import indented from nuitka.codegen.templates.CodeTemplatesFreezer import ( template_frozen_modules ) from nuitka.ModuleRegistry import getUncompiledTechnicalModules stream_data = ConstantCodes.stream_data def generateBytecodeFrozenCode(): frozen_defs = [] for uncompiled_module in getUncompiledTechnicalModules(): module_name = uncompiled_module.getFullName() code_data = uncompiled_module.getByteCode() is_package = uncompiled_module.isUncompiledPythonPackage() size = len(code_data) # Packages are indicated with negative size. if is_package: size = -size frozen_defs.append( """\ {{ "{module_name}", {start}, {size} }},""".format( module_name = module_name, start = stream_data.getStreamDataOffset(code_data), size = size ) ) if Options.isShowInclusion(): info("Embedded as frozen module '%s'.", module_name) return template_frozen_modules % { "frozen_modules" : indented(frozen_defs, 2) } Nuitka-0.5.28.2/nuitka/freezer/__init__.py0000644000372000001440000000150113112214770020524 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/freezer/Standalone.py0000644000372000001440000010706313207540035021070 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Pack and copy files for standalone mode. This is still under heavy evolution, but expected to work for MacOS, Windows, and Linux. Patches for other platforms are very welcome. """ import marshal import os import shutil import subprocess import sys from logging import debug, info, warning from nuitka import Options, SourceCodeReferences, Tracing from nuitka.__past__ import iterItems from nuitka.containers.odict import OrderedDict from nuitka.importing import ImportCache from nuitka.importing.StandardLibrary import ( getStandardLibraryPaths, isStandardLibraryPath ) from nuitka.nodes.ModuleNodes import ( PythonShlibModule, makeUncompiledPythonModule ) from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import python_version from nuitka.tree.SourceReading import readSourceCodeFromFilename from nuitka.utils import Utils from nuitka.utils.Execution import withEnvironmentPathAdded from nuitka.utils.FileOperations import ( areSamePaths, deleteFile, getSubDirectories, listDir ) from nuitka.utils.Timing import TimerReport from .DependsExe import getDependsExePath def loadCodeObjectData(precompiled_filename): # Ignoring magic numbers, etc. which we don't have to care for much as # CPython already checked them (would have rejected it otherwise). return open(precompiled_filename, "rb").read()[8:] module_names = set() def _detectedPrecompiledFile(filename, module_name, result, user_provided, technical): if filename.endswith(".pyc"): if os.path.isfile(filename[:-1]): return _detectedSourceFile( filename = filename[:-1], module_name = module_name, result = result, user_provided = user_provided, technical = technical ) if module_name in module_names: return debug( "Freezing module '%s' (from '%s').", module_name, filename ) uncompiled_module = makeUncompiledPythonModule( module_name = module_name, bytecode = loadCodeObjectData( precompiled_filename = filename ), is_package = "__init__" in filename, filename = filename, user_provided = user_provided, technical = technical ) ImportCache.addImportedModule(uncompiled_module) result.append(uncompiled_module) module_names.add(module_name) def _detectedSourceFile(filename, module_name, result, user_provided, technical): if module_name in module_names: return if module_name == "collections.abc": _detectedSourceFile( filename = filename, module_name = "_collections_abc", result = result, user_provided = user_provided, technical = technical ) source_code = readSourceCodeFromFilename(module_name, filename) if module_name == "site": if source_code.startswith("def ") or source_code.startswith("class "): source_code = '\n' + source_code source_code = """\ __file__ = (__nuitka_binary_dir + '%s%s') if '__nuitka_binary_dir' in dict(__builtins__ ) else '';%s""" % ( os.path.sep, os.path.basename(filename), source_code ) # Debian stretch site.py source_code = source_code.replace( "PREFIXES = [sys.prefix, sys.exec_prefix]", "PREFIXES = []" ) # Anaconda3 4.1.2 site.py source_code = source_code.replace( "def main():", "def main():return\n\nif 0:\n def _unused():", ) debug( "Freezing module '%s' (from '%s').", module_name, filename ) is_package = os.path.basename(filename) == "__init__.py" source_code = Plugins.onFrozenModuleSourceCode( module_name = module_name, is_package = is_package, source_code = source_code ) bytecode = compile(source_code, filename, "exec", dont_inherit = True) bytecode = Plugins.onFrozenModuleBytecode( module_name = module_name, is_package = is_package, bytecode = bytecode ) uncompiled_module = makeUncompiledPythonModule( module_name = module_name, bytecode = marshal.dumps( bytecode ), is_package = is_package, filename = filename, user_provided = user_provided, technical = technical ) ImportCache.addImportedModule(uncompiled_module) result.append(uncompiled_module) module_names.add(module_name) def _detectedShlibFile(filename, module_name): # That is not a shared library, but looks like one. if module_name == "__main__": return from nuitka import ModuleRegistry if ModuleRegistry.hasRootModule(module_name): return parts = module_name.split('.') if len(parts) == 1: package_name = None name = module_name else: package_name = '.'.join(parts[:-1]) name = parts[-1] source_ref = SourceCodeReferences.fromFilename( filename = filename ) shlib_module = PythonShlibModule( name = name, package_name = package_name, source_ref = source_ref ) ModuleRegistry.addRootModule(shlib_module) ImportCache.addImportedModule(shlib_module) module_names.add(module_name) def _detectImports(command, user_provided, technical): # This is pretty complicated stuff, with variants to deal with. # pylint: disable=too-many-branches,too-many-locals,too-many-statements # Print statements for stuff to show, the modules loaded. if python_version >= 300: command += '\nprint("\\n".join(sorted("import " + module.__name__ + " # sourcefile " + ' \ 'module.__file__ for module in sys.modules.values() if hasattr(module, "__file__") and ' \ 'module.__file__ != "")), file = sys.stderr)' # do not read it reduced_path = [ path_element for path_element in sys.path if not areSamePaths( path_element, '.' ) if not areSamePaths( path_element, os.path.dirname(sys.modules["__main__"].__file__) ) ] # Make sure the right import path (the one Nuitka binary is running with) # is used. command = ("import sys; sys.path = %s; sys.real_prefix = sys.prefix;" % repr(reduced_path)) + command import tempfile tmp_file, tmp_filename = tempfile.mkstemp() try: if python_version >= 300: command = command.encode("utf8") os.write(tmp_file, command) os.close(tmp_file) process = subprocess.Popen( args = [sys.executable, "-s", "-S", "-v", tmp_filename], stdout = subprocess.PIPE, stderr = subprocess.PIPE, ) _stdout, stderr = process.communicate() finally: os.unlink(tmp_filename) # Don't let errors here go unnoticed. if process.returncode != 0: warning("There is a problem with detecting imports, CPython said:") for line in stderr.split(b"\n"): Tracing.printError(line) sys.exit("Error, please report the issue with above output.") result = [] debug("Detecting imports:") detections = [] for line in stderr.replace(b"\r", b"").split(b"\n"): if line.startswith(b"import "): # print(line) parts = line.split(b" # ", 2) module_name = parts[0].split(b" ", 2)[1] origin = parts[1].split()[0] if python_version >= 300: module_name = module_name.decode("utf-8") if origin == b"precompiled": # This is a ".pyc" file that was imported, even before we have a # chance to do anything, we need to preserve it. filename = parts[1][len(b"precompiled from "):] if python_version >= 300: filename = filename.decode("utf-8") # Do not leave standard library when freezing. if not isStandardLibraryPath(filename): continue detections.append( (module_name, 3, "precompiled", filename) ) elif origin == b"sourcefile": filename = parts[1][len(b"sourcefile "):] if python_version >= 300: filename = filename.decode("utf-8") # Do not leave standard library when freezing. if not isStandardLibraryPath(filename): continue if filename.endswith(".py"): detections.append( (module_name, 2, "sourcefile", filename) ) elif not filename.endswith(""): # Python3 started lying in "__name__" for the "_decimal" # calls itself "decimal", which then is wrong and also # clashes with "decimal" proper if python_version >= 300: if module_name == "decimal": module_name = "_decimal" detections.append( (module_name, 2, "shlib", filename) ) elif origin == b"dynamically": # Shared library in early load, happens on RPM based systems and # or self compiled Python installations. filename = parts[1][len(b"dynamically loaded from "):] if python_version >= 300: filename = filename.decode("utf-8") # Do not leave standard library when freezing. if not isStandardLibraryPath(filename): continue detections.append( (module_name, 1, "shlib", filename) ) for module_name, _prio, kind, filename in sorted(detections): if kind == "precompiled": _detectedPrecompiledFile( filename = filename, module_name = module_name, result = result, user_provided = user_provided, technical = technical ) elif kind == "sourcefile": _detectedSourceFile( filename = filename, module_name = module_name, result = result, user_provided = user_provided, technical = technical ) elif kind == "shlib": _detectedShlibFile( filename = filename, module_name = module_name ) else: assert False, kind return result # Some modules we want to blacklist. ignore_modules = [ "__main__.py", "__init__.py", "antigravity.py", ] if Utils.getOS() != "Windows": ignore_modules.append("wintypes.py") ignore_modules.append("cp65001.py") def scanStandardLibraryPath(stdlib_dir): # There is a lot of black-listing here, done in branches, so there # is many of them, but that's acceptable, pylint: disable=too-many-branches for root, dirs, filenames in os.walk(stdlib_dir): import_path = root[len(stdlib_dir):].strip("/\\") import_path = import_path.replace('\\', '.').replace('/','.') if import_path == "": if "site-packages" in dirs: dirs.remove("site-packages") if "dist-packages" in dirs: dirs.remove("dist-packages") if "test" in dirs: dirs.remove("test") if "idlelib" in dirs: dirs.remove("idlelib") if "turtledemo" in dirs: dirs.remove("turtledemo") if "ensurepip" in filenames: filenames.remove("ensurepip") if "ensurepip" in dirs: dirs.remove("ensurepip") # Ignore "lib-dynload" and "lib-tk" and alikes. dirs[:] = [ dirname for dirname in dirs if not dirname.startswith("lib-") if dirname != "Tools" ] if import_path in ("tkinter", "importlib", "ctypes", "unittest", "sqlite3", "distutils", "email", "bsddb"): if "test" in dirs: dirs.remove("test") if import_path in ("lib2to3", "json", "distutils"): if "tests" in dirs: dirs.remove("tests") if python_version >= 340 and Utils.getOS() == "Windows": if import_path == "multiprocessing": filenames.remove("popen_fork.py") filenames.remove("popen_forkserver.py") filenames.remove("popen_spawn_posix.py") if Utils.getOS() == "NetBSD": if import_path == "xml.sax": filenames.remove("expatreader.py") for filename in filenames: if filename.endswith(".py") and filename not in ignore_modules: module_name = filename[:-3] if import_path == "": yield module_name else: yield import_path + '.' + module_name if python_version >= 300: if "__pycache__" in dirs: dirs.remove("__pycache__") for dirname in dirs: if import_path == "": yield dirname else: yield import_path + '.' + dirname def detectEarlyImports(): encoding_names = [ filename[:-3] for _path, filename in listDir(os.path.dirname(sys.modules["encodings"].__file__)) if filename.endswith(".py") if "__init__" not in filename ] if Utils.getOS() != "Windows": for encoding_name in ("mbcs", "cp65001", "oem"): if encoding_name in encoding_names: encoding_names.remove(encoding_name) import_code = ';'.join( "import encodings.%s" % encoding_name for encoding_name in encoding_names ) import_code += ";import locale;" # For Python3 we patch inspect without knowing if it is used. if python_version >= 300: import_code += "import inspect;" result = _detectImports( command = import_code, user_provided = False, technical = True ) if Options.shallFreezeAllStdlib(): stdlib_modules = set() # Scan the standard library paths (multiple in case of virtualenv. for stdlib_dir in getStandardLibraryPaths(): for module_name in scanStandardLibraryPath(stdlib_dir): stdlib_modules.add(module_name) # Put here ones that should be imported first. first_ones = ( "Tkinter", ) # We have to fight zombie modules in this, some things, e.g. Tkinter # on newer Python 2.7, comes back after failure without a second error # being raised, leading to other issues. So we fight it after each # module that was tried, and prevent re-try by adding a meta path # based loader that will never load it again, and remove it from the # "sys.modules" over and over, once it sneaks back. The root cause is # that extension modules sometimes only raise an error when first # imported, not the second time around. # Otherwise this just makes imports of everything so we can see where # it comes from and what it requires. import_code = """ imports = %r failed = set() class ImportBlocker(object): def find_module(self, fullname, path = None): if fullname in failed: return self return None def load_module(self, name): raise ImportError("%%s has failed before" %% name) sys.meta_path.insert(0, ImportBlocker()) for imp in imports: try: __import__(imp) except (ImportError, SyntaxError): failed.add(imp) for fail in failed: if fail in sys.modules: del sys.modules[fail] """ % sorted( stdlib_modules, key = lambda name: (name not in first_ones, name) ) early_names = [ module.getFullName() for module in result ] result += [ module for module in _detectImports( command = import_code, user_provided = False, technical = False ) if module.getFullName() not in early_names ] debug("Finished detecting early imports.") return result _detected_python_rpath = None ldd_result_cache = {} def _detectBinaryPathDLLsLinuxBSD(dll_filename): # Ask "ldd" about the libraries being used by the created binary, these # are the ones that interest us. result = set() # This is the rpath of the Python binary, which will be effective when # loading the other DLLs too. This happens at least for Python installs # on Travis. pylint: disable=global-statement global _detected_python_rpath if _detected_python_rpath is None: _detected_python_rpath = getSharedLibraryRPATH(sys.executable) or False if _detected_python_rpath: _detected_python_rpath = _detected_python_rpath.replace( b"$ORIGIN", os.path.dirname(sys.executable).encode("utf-8") ) if dll_filename not in ldd_result_cache: with withEnvironmentPathAdded("LD_LIBRARY_PATH", _detected_python_rpath): process = subprocess.Popen( args = [ "ldd", dll_filename ], stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout, _stderr = process.communicate() for line in stdout.split(b"\n"): if not line: continue if b"=>" not in line: continue part = line.split(b" => ", 2)[1] if b"(" in part: filename = part[:part.rfind(b"(")-1] else: filename = part if not filename: continue if python_version >= 300: filename = filename.decode("utf-8") # Sometimes might use stuff not found. if filename == "not found": continue # Do not include kernel specific libraries. if os.path.basename(filename).startswith( ( "libc.so.", "libpthread.so.", "libm.so.", "libdl.so." ) ): continue result.add(filename) # Allow plugins to prevent inclusion. blocked = Plugins.removeDllDependencies( dll_filename = dll_filename, dll_filenames = result ) for to_remove in blocked: result.discard(to_remove) ldd_result_cache[dll_filename] = result else: result = ldd_result_cache[dll_filename] sub_result = set(result) for sub_dll_filename in result: sub_result = sub_result.union(_detectBinaryPathDLLsLinuxBSD(sub_dll_filename)) return sub_result def _detectBinaryPathDLLsMacOS(original_dir, binary_filename): result = set() process = subprocess.Popen( args = [ "otool", "-L", binary_filename ], stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout, _stderr = process.communicate() system_paths = (b"/usr/lib/", b"/System/Library/Frameworks/") for line in stdout.split(b"\n")[2:]: if not line: continue filename = line.split(b" (")[0].strip() stop = False for w in system_paths: if filename.startswith(w): stop = True break if not stop: if python_version >= 300: filename = filename.decode("utf-8") if filename.startswith("@rpath/"): filename = os.path.join(original_dir, filename[7:]) elif filename.startswith("@loader_path/"): filename = os.path.join(original_dir, filename[13:]) # print "adding", filename result.add(filename) return result def _makeBinaryPathPathDLLSearchEnv(package_name): # Put the PYTHONPATH into the system "PATH", DLLs frequently live in # the package directories. env = os.environ.copy() path = env.get("PATH","").split(os.pathsep) # Put the "Python.exe" first. At least for WinPython, they put the DLLs # there. path = [sys.prefix] + sys.path + path if package_name is not None: for element in sys.path: candidate = os.path.join(element, package_name) if os.path.isdir(candidate): path.append(candidate) env["PATH"] = os.pathsep.join(path) return env def _detectBinaryPathDLLsWindows(original_dir, binary_filename, package_name): result = set() depends_exe = getDependsExePath() # The search order by default prefers the system directory, where a # wrong "PythonXX.dll" might be living. with open(binary_filename + ".dwp", 'w') as dwp_file: dwp_file.write("""\ KnownDLLs SysPath AppDir {original_dirs} 32BitSysDir 16BitSysDir OSDir AppPath SxS """.format( original_dirs = ( '\n'.join( "UserDir %s" % dirname for dirname in [original_dir] + getSubDirectories(original_dir) if not os.path.basename(dirname) == "__pycache__" if any(entry[1].lower().endswith(".dll") for entry in listDir(dirname)) ) if original_dir is not None else "" ) ) ) subprocess.call( ( depends_exe, "-c", "-ot%s" % binary_filename + ".depends", "-d:%s" % binary_filename + ".dwp", "-f1", "-pa1", "-ps1", binary_filename ), env = _makeBinaryPathPathDLLSearchEnv(package_name), ) inside = False first = False for line in open(binary_filename + ".depends"): if "| Module Dependency Tree |" in line: inside = True first = True continue if not inside: continue if "| Module List |" in line: break if ']' not in line: continue # Skip missing DLLs, apparently not needed anyway. if '?' in line[:line.find(']')]: continue # Skip DLLs that failed to load, apparently not needed anyway. if 'E' in line[:line.find(']')]: continue dll_filename = line[line.find(']')+2:-1] # The executable itself is of course exempted. We cannot check its path # because depends.exe mistreats unicode paths. if first: first = False continue assert os.path.isfile(dll_filename), dll_filename dll_name = os.path.basename(dll_filename).upper() # Win API can be assumed. if dll_name.startswith("API-MS-WIN-") or \ dll_name.startswith("EXT-MS-WIN-"): continue if dll_name in ("SHELL32.DLL", "USER32.DLL", "KERNEL32.DLL", "NTDLL.DLL", "NETUTILS.DLL", "LOGONCLI.DLL", "GDI32.DLL", "RPCRT4.DLL", "ADVAPI32.DLL", "SSPICLI.DLL", "SECUR32.DLL", "KERNELBASE.DLL", "WINBRAND.DLL", "DSROLE.DLL", "DNSAPI.DLL", "SAMCLI.DLL", "WKSCLI.DLL", "SAMLIB.DLL", "WLDAP32.DLL", "NTDSAPI.DLL", "CRYPTBASE.DLL", "W32TOPL", "WS2_32.DLL", "SPPC.DLL", "MSSIGN32.DLL", "CERTCLI.DLL", "WEBSERVICES.DLL", "AUTHZ.DLL", "CERTENROLL.DLL", "VAULTCLI.DLL", "REGAPI.DLL", "BROWCLI.DLL", "WINNSI.DLL", "DHCPCSVC6.DLL", "PCWUM.DLL", "CLBCATQ.DLL", "IMAGEHLP.DLL", "MSASN1.DLL", "DBGHELP.DLL", "DEVOBJ.DLL", "DRVSTORE.DLL", "CABINET.DLL", "SCECLI.DLL", "SPINF.DLL", "SPFILEQ.DLL", "GPAPI.DLL", "NETJOIN.DLL", "W32TOPL.DLL", "NETBIOS.DLL", "DXGI.DLL", "DWRITE.DLL", "D3D11.DLL", "WLANAPI.DLL", "WLANUTIL.DLL", "ONEX.DLL", "EAPPPRXY.DLL", "MFPLAT.DLL", "AVRT.DLL", "ELSCORE.DLL", "INETCOMM.DLL", "MSOERT2.DLL", "IEUI.DLL", "MSCTF.DLL", "MSFEEDS.DLL", "UIAUTOMATIONCORE.DLL", "PSAPI.DLL", "EFSADU.DLL", "MFC42U.DLL", "ODBC32.DLL", "OLEDLG.DLL", "NETAPI32.DLL", "LINKINFO.DLL", "DUI70.DLL", "ADVPACK.DLL", "NTSHRUI.DLL", "WINSPOOL.DRV", "EFSUTIL.DLL", "WINSCARD.DLL", "SHDOCVW.DLL", "IEFRAME.DLL", "D2D1.DLL", "GDIPLUS.DLL", "OCCACHE.DLL", "IEADVPACK.DLL", "MLANG.DLL", "MSI.DLL", "MSHTML.DLL", "COMDLG32.DLL", "PRINTUI.DLL", "PUIAPI.DLL", "ACLUI.DLL", "WTSAPI32.DLL", "FMS.DLL", "DFSCLI.DLL", "HLINK.DLL", "MSRATING.DLL", "PRNTVPT.DLL", "IMGUTIL.DLL", "MSLS31.DLL", "VERSION.DLL", "NORMALIZ.DLL", "IERTUTIL.DLL", "WININET.DLL", "WINTRUST.DLL", "XMLLITE.DLL", "APPHELP.DLL", "PROPSYS.DLL", "RSTRTMGR.DLL", "NCRYPT.DLL", "BCRYPT.DLL", "MMDEVAPI.DLL", "MSILTCFG.DLL", "DEVMGR.DLL", "DEVRTL.DLL", "NEWDEV.DLL", "VPNIKEAPI.DLL", "WINHTTP.DLL", "WEBIO.DLL", "NSI.DLL", "DHCPCSVC.DLL", "CRYPTUI.DLL", "ESENT.DLL", "DAVHLPR.DLL", "CSCAPI.DLL", "ATL.DLL", "OLEAUT32.DLL", "SRVCLI.DLL", "RASDLG.DLL", "MPRAPI.DLL", "RTUTILS.DLL", "RASMAN.DLL", "MPRMSG.DLL", "SLC.DLL", "CRYPTSP.DLL", "RASAPI32.DLL", "TAPI32.DLL", "EAPPCFG.DLL", "NDFAPI.DLL", "WDI.DLL", "COMCTL32.DLL", "UXTHEME.DLL", "IMM32.DLL", "OLEACC.DLL", "WINMM.DLL", "WINDOWSCODECS.DLL", "DWMAPI.DLL", "DUSER.DLL", "PROFAPI.DLL", "URLMON.DLL", "SHLWAPI.DLL", "LPK.DLL", "USP10.DLL", "CFGMGR32.DLL", "MSIMG32.DLL", "POWRPROF.DLL", "SETUPAPI.DLL", "WINSTA.DLL", "CRYPT32.DLL", "IPHLPAPI.DLL", "MPR.DLL", "CREDUI.DLL", "NETPLWIZ.DLL", "OLE32.DLL", "ACTIVEDS.DLL", "ADSLDPC.DLL", "USERENV.DLL", "APPREPAPI.DLL", "BCP47LANGS.DLL", "BCRYPTPRIMITIVES.DLL", "CERTCA.DLL", "CHARTV.DLL", "COMBASE.DLL", "COML2.DLL", "DCOMP.DLL", "DPAPI.DLL", "DSPARSE.DLL", "FECLIENT.DLL", "FIREWALLAPI.DLL", "FLTLIB.DLL", "MRMCORER.DLL", "NTASN1.DLL", "SECHOST.DLL", "SETTINGSYNCPOLICY.DLL", "SHCORE.DLL", "TBS.DLL", "TWINAPI.APPCORE.DLL", "TWINAPI.DLL", "VIRTDISK.DLL", "WEBSOCKET.DLL", "WEVTAPI.DLL", "WINMMBASE.DLL", "WMICLNT.DLL"): continue result.add( os.path.normcase(os.path.abspath(dll_filename)) ) deleteFile(binary_filename + ".depends", must_exist = True) deleteFile(binary_filename + ".dwp", must_exist = True) return result def detectBinaryDLLs(original_filename, binary_filename, package_name): """ Detect the DLLs used by a binary. Using "ldd" (Linux), "depends.exe" (Windows), or "otool" (MacOS) the list of used DLLs is retrieved. """ if Utils.getOS() in ("Linux", "NetBSD", "FreeBSD"): return _detectBinaryPathDLLsLinuxBSD( dll_filename = original_filename ) elif Utils.getOS() == "Windows": with TimerReport("Running depends.exe for %s took %%.2f seconds" % binary_filename): return _detectBinaryPathDLLsWindows( original_dir = os.path.dirname(original_filename), binary_filename = binary_filename, package_name = package_name ) elif Utils.getOS() == "Darwin": return _detectBinaryPathDLLsMacOS( original_dir = os.path.dirname(original_filename), binary_filename = original_filename ) else: # Support your platform above. assert False, Utils.getOS() def detectUsedDLLs(standalone_entry_points): result = OrderedDict() for original_filename, binary_filename, package_name in standalone_entry_points: used_dlls = detectBinaryDLLs( original_filename = original_filename, binary_filename = binary_filename, package_name = package_name ) for dll_filename in used_dlls: # We want these to be absolute paths. assert os.path.isabs(dll_filename), dll_filename if dll_filename not in result: result[dll_filename] = [] result[dll_filename].append(binary_filename) return result def fixupBinaryDLLPaths(binary_filename, is_exe, dll_map): """ For MacOS, the binary needs to be told to use relative DLL paths """ # There may be nothing to do, in case there are no DLLs. if not dll_map: return command = [ "install_name_tool" ] for original_path, dist_path in dll_map: command += [ "-change", original_path, "@executable_path/" + dist_path, ] os.chmod(binary_filename, int("644", 8)) command.append(binary_filename) process = subprocess.Popen( args = command, stdout = subprocess.PIPE, stderr = subprocess.PIPE, ) _stdout, stderr = process.communicate() os.chmod(binary_filename, int("755" if is_exe else "444", 8)) # Don't let errors here go unnoticed. assert process.returncode == 0, stderr def getSharedLibraryRPATH(filename): process = subprocess.Popen( ["readelf", "-d", filename], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) stdout, stderr = process.communicate() retcode = process.poll() if retcode != 0: sys.exit( "Error reading shared library path for %s, tool said %r" % ( filename, stderr ) ) for line in stdout.split(b"\n"): if b"RPATH" in line or b"RUNPATH" in line: return line[line.find(b'[')+1:line.rfind(b']')] return None def removeSharedLibraryRPATH(filename): rpath = getSharedLibraryRPATH(filename) if rpath is not None: if Options.isShowInclusion(): info("Removing 'RPATH' setting from '%s'.", filename) if not Utils.isExecutableCommand("chrpath"): sys.exit( """\ Error, needs 'chrpath' on your system, due to 'RPATH' settings in used shared libraries that need to be removed.""" ) os.chmod(filename, int("644", 8)) process = subprocess.Popen( ["chrpath", "-d", filename], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) process.communicate() retcode = process.poll() os.chmod(filename, int("444", 8)) assert retcode == 0, filename def copyUsedDLLs(dist_dir, standalone_entry_points): # This is terribly complex, because we check the list of used DLLs # trying to avoid duplicates, and detecting errors with them not # being binary identical, so we can report them. And then of course # we also need to handle OS specifics, pylint: disable=too-many-branches used_dlls = detectUsedDLLs(standalone_entry_points) # Fist make checks and remove some. for dll_filename1, sources1 in tuple(iterItems(used_dlls)): for dll_filename2, sources2 in tuple(iterItems(used_dlls)): if dll_filename1 == dll_filename2: continue # Colliding basenames are an issue to us. if os.path.basename(dll_filename1) != \ os.path.basename(dll_filename2): continue # May already have been removed earlier if dll_filename1 not in used_dlls: continue if dll_filename2 not in used_dlls: continue dll_name = os.path.basename(dll_filename1) if Options.isShowInclusion(): info( """Colliding DLL names for %s, checking identity of \ '%s' <-> '%s'.""" % ( dll_name, dll_filename1, dll_filename2, ) ) # Check that if a DLL has the same name, if it's identical, # happens at least for OSC and Fedora 20. import filecmp if filecmp.cmp(dll_filename1, dll_filename2): del used_dlls[dll_filename2] continue # So we have conflicting DLLs, in which case we do not proceed. sys.exit( """Error, conflicting DLLs for '%s'. %s used by: %s different from %s used by %s""" % ( dll_name, dll_filename1, "\n ".join(sources1), dll_filename2, "\n ".join(sources2) ) ) dll_map = [] for dll_filename, sources in iterItems(used_dlls): dll_name = os.path.basename(dll_filename) target_path = os.path.join( dist_dir, dll_name ) shutil.copy( dll_filename, target_path ) dll_map.append( (dll_filename, dll_name) ) if Options.isShowInclusion(): info( "Included used shared library '%s' (used by %s)." % ( dll_filename, ", ".join(sources) ) ) if Utils.getOS() == "Darwin": # For MacOS, the binary and the DLLs needs to be changed to reflect # the relative DLL location in the ".dist" folder. for standalone_entry_point in standalone_entry_points: fixupBinaryDLLPaths( binary_filename = standalone_entry_point[1], is_exe = standalone_entry_point is standalone_entry_points[0], dll_map = dll_map ) for _original_path, dll_filename in dll_map: fixupBinaryDLLPaths( binary_filename = os.path.join( dist_dir, dll_filename ), is_exe = False, dll_map = dll_map ) if Utils.getOS() == "Linux": # For Linux, the "rpath" of libraries may be an issue and must be # removed. for standalone_entry_point in standalone_entry_points[1:]: removeSharedLibraryRPATH( standalone_entry_point[1] ) for _original_path, dll_filename in dll_map: removeSharedLibraryRPATH( os.path.join(dist_dir, dll_filename) ) Nuitka-0.5.28.2/nuitka/freezer/DependsExe.py0000644000372000001440000000734513134660221021025 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Interface to depends.exe on Windows. We use depends.exe to investigate needed DLLs of Python DLLs. """ import os import sys from logging import info from nuitka import Tracing from nuitka.__past__ import ( # pylint: disable=I0021,redefined-builtin raw_input, urlretrieve ) from nuitka.utils import Utils from nuitka.utils.AppDirs import getAppDir from nuitka.utils.FileOperations import deleteFile, makePath def getDependsExePath(): """ Return the path of depends.exe (for Windows). Will prompt the user to download if not already cached in AppData directory for Nuitka. """ if Utils.getArchitecture() == "x86": depends_url = "http://dependencywalker.com/depends22_x86.zip" else: depends_url = "http://dependencywalker.com/depends22_x64.zip" nuitka_app_dir = getAppDir() nuitka_depends_dir = os.path.join( nuitka_app_dir, Utils.getArchitecture() ) nuitka_depends_zip = os.path.join( nuitka_depends_dir, os.path.basename(depends_url) ) depends_exe = os.path.join( nuitka_depends_dir, "depends.exe" ) makePath(nuitka_depends_dir) if not os.path.isfile(nuitka_depends_zip) and not os.path.isfile(depends_exe): Tracing.printLine( """\ Nuitka will make use of Dependency Walker (http://dependencywalker.com) tool to analyze the dependencies of Python extension modules. Is it OK to download and put it in "%s". No installer needed, cached, one time question. Proceed and download? [Yes]/No """ % ( nuitka_app_dir ) ) Tracing.flushStdout() reply = raw_input() if reply.lower() in ("no", 'n'): sys.exit("Nuitka does not work in --standalone on Windows without.") info("Downloading '%s'" % depends_url) try: urlretrieve( depends_url, nuitka_depends_zip ) except Exception: # Any kind of error, pylint: disable=broad-except sys.exit( """Failed to download '%s'.\ Contents should manually be extracted to '%s'.""" % ( depends_url, nuitka_depends_dir ) ) if not os.path.isfile(depends_exe): info("Extracting to '%s'" % depends_exe) import zipfile try: depends_zip = zipfile.ZipFile(nuitka_depends_zip) depends_zip.extractall(nuitka_depends_dir) except Exception: # Catching anything zip throws, pylint:disable=W0703 info("Problem with the downloaded zip file, deleting it.") deleteFile(depends_exe, must_exist = False) deleteFile(nuitka_depends_zip, must_exist = True) sys.exit( "Error, need '%s' as extracted from '%s'." % ( depends_exe, depends_url ) ) assert os.path.isfile(depends_exe) return depends_exe Nuitka-0.5.28.2/nuitka/PythonOperators.py0000644000372000001440000000750213134660221020512 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Python operator tables These are mostly used to resolve the operator in the module operator and to know the list of operations allowed. """ import operator from nuitka.PythonVersions import python_version binary_operator_functions = { "Add" : operator.add, "Sub" : operator.sub, "Pow" : operator.pow, "Mult" : operator.mul, "FloorDiv" : operator.div if python_version < 300 else operator.floordiv, "TrueDiv" : operator.truediv, "Mod" : operator.mod, "LShift" : operator.lshift, "RShift" : operator.rshift, "BitAnd" : operator.and_, "BitOr" : operator.or_, "BitXor" : operator.xor, "Divmod" : divmod, "IAdd" : operator.iadd, "ISub" : operator.isub, "IPow" : operator.ipow, "IMult" : operator.imul, "IFloorDiv" : operator.idiv if python_version < 300 else operator.ifloordiv, "ITrueDiv" : operator.itruediv, "IMod" : operator.imod, "ILShift" : operator.ilshift, "IRShift" : operator.irshift, "IBitAnd" : operator.iand, "IBitOr" : operator.ior, "IBitXor" : operator.ixor, } # Python 2 only operator if python_version < 300: binary_operator_functions["Div"] = operator.div # @UndefinedVariable binary_operator_functions["IDiv"] = operator.idiv # @UndefinedVariable # Python 3.5 only operator if python_version >= 350: binary_operator_functions["MatMult"] = operator.matmul # @UndefinedVariable binary_operator_functions["IMatMult"] = operator.imatmul # @UndefinedVariable unary_operator_functions = { "UAdd" : operator.pos, "USub" : operator.neg, "Invert" : operator.invert, "Repr" : repr, # Boolean not is treated an unary operator. "Not" : operator.not_, } rich_comparison_functions = { "Lt" : operator.lt, "LtE" : operator.le, "Eq" : operator.eq, "NotEq" : operator.ne, "Gt" : operator.gt, "GtE" : operator.ge } other_comparison_functions = { "Is" : operator.is_, "IsNot" : operator.is_not, "In" : lambda value1, value2: value1 in value2, "NotIn" : lambda value1, value2: value1 not in value2 } comparison_inversions = { "Is" : "IsNot", "IsNot" : "Is", "In" : "NotIn", "NotIn" : "In" } all_comparison_functions = dict(rich_comparison_functions) all_comparison_functions.update(other_comparison_functions) def matchException(left, right): if python_version >= 300: if type(right) is tuple: for element in right: if not isinstance(BaseException,element): raise TypeError("catching classes that do not inherit from BaseException is not allowed") elif not isinstance(BaseException,right): raise TypeError("catching classes that do not inherit from BaseException is not allowed") # This doesn't yet work, make it error exit. and silence PyLint for now. # pylint: disable=protected-access import os os._exit(16) assert False, left all_comparison_functions["exception_match"]=matchException Nuitka-0.5.28.2/nuitka/codegen/0000755000372000001440000000000013207540420016357 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/codegen/IntegerCodes.py0000644000372000001440000000702513207537242021320 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Integer related codes, long and int. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateChildExpressionsCode from .ErrorCodes import getErrorExitCode, getReleaseCodes from .PythonAPICodes import generateCAPIObjectCode def generateBuiltinLongCode(to_name, expression, emit, context): assert python_version < 300 value = expression.getValue() base = expression.getBase() assert value is not None if base is None: generateCAPIObjectCode( to_name = to_name, capi = "PyNumber_Long", arg_desc = ( ("long_arg", value), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) else: value_name, base_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = TO_LONG2( %s, %s );" % ( to_name, value_name, base_name ) ) getReleaseCodes( release_names = (value_name, base_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinIntCode(to_name, expression, emit, context): value = expression.getValue() base = expression.getBase() assert value is not None if base is None: generateCAPIObjectCode( to_name = to_name, capi = "PyNumber_Int", arg_desc = ( ("int_arg", value), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, ) else: value_name, base_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = TO_INT2( %s, %s );" % ( to_name, value_name, base_name ) ) getReleaseCodes( release_names = (value_name, base_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/BuiltinCodes.py0000644000372000001440000003076013207537242021333 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in codes This is code generation for built-in references, and some built-ins like range, bin, etc. """ from nuitka import Builtins from nuitka.PythonVersions import python_version from .CodeHelpers import generateChildExpressionsCode from .ErrorCodes import getAssertionCode, getErrorExitCode, getReleaseCodes from .PythonAPICodes import generateCAPIObjectCode, generateCAPIObjectCode0 def generateBuiltinRefCode(to_name, expression, emit, context): builtin_name = expression.getBuiltinName() emit( "%s = LOOKUP_BUILTIN( %s );" % ( to_name, context.getConstantCode( constant = builtin_name, ) ) ) getAssertionCode( check = "%s != NULL" % to_name, emit = emit ) # Gives no reference def generateBuiltinAnonymousRefCode(to_name, expression, emit, context): # Functions used for generation all accept context, but this one does # not use it. pylint: disable=unused-argument builtin_name = expression.getBuiltinName() emit( "%s = (PyObject *)%s;" % ( to_name, Builtins.builtin_anon_codes[builtin_name] ) ) def generateBuiltinType1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_TYPE1", arg_desc = ( ("type_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinType3Code(to_name, expression, emit, context): type_name, bases_name, dict_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = BUILTIN_TYPE3( %s, %s, %s, %s );" % ( to_name, context.getConstantCode( constant = context.getModuleName(), ), type_name, bases_name, dict_name ), ) getReleaseCodes( release_names = (type_name, bases_name, dict_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinOpenCode(to_name, expression, emit, context): arg_desc = ( ("open_filename", expression.getFilename()), ("open_mode", expression.getMode()), ("open_buffering", expression.getBuffering()) ) if python_version >= 300: arg_desc += ( ("open_encoding", expression.getEncoding()), ("open_errors", expression.getErrors()), ("open_newline", expression.getNewline()), ("open_closefd", expression.getCloseFd()), ("open_opener", expression.getOpener()) ) generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_OPEN", arg_desc = arg_desc, may_raise = expression.mayRaiseException(BaseException), none_null = True, source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinSum1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_SUM1", arg_desc = ( ("sum_sequence", expression.getSequence()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinSum2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_SUM2", arg_desc = ( ("sum_sequence", expression.getSequence()), ("sum_start", expression.getStart()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinRange1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_RANGE", arg_desc = ( ("range_arg", expression.getLow()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinRange2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_RANGE2", arg_desc = ( ("range2_low", expression.getLow()), ("range2_high", expression.getHigh()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinRange3Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_RANGE3", arg_desc = ( ("range3_low", expression.getLow()), ("range3_high", expression.getHigh()), ("range3_step", expression.getStep()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinXrange1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_XRANGE1", arg_desc = ( ("xrange_low", expression.getLow()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, none_null = True ) def generateBuiltinXrange2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_XRANGE2", arg_desc = ( ("xrange_low", expression.getLow()), ("xrange_high", expression.getHigh()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, none_null = True, ) def generateBuiltinXrange3Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_XRANGE3", arg_desc = ( ("xrange_low", expression.getLow()), ("xrange_high", expression.getHigh()), ("xrange_step", expression.getStep()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, none_null = True, ) def generateBuiltinFloatCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "TO_FLOAT", arg_desc = ( ("float_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinComplexCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "TO_COMPLEX", arg_desc = ( ("real_arg", expression.getReal()), ("imag_arg", expression.getImag()) ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinBoolCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "TO_BOOL", arg_desc = ( ("bool_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinBinCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_BIN", arg_desc = ( ("bin_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinOctCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_OCT", arg_desc = ( ("oct_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinHexCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_HEX", arg_desc = ( ("hex_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinBytearray1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_BYTEARRAY1", arg_desc = ( ("bytearray_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinBytearray3Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_BYTEARRAY3", arg_desc = ( ("bytearray_string", expression.getStringArg()), ("bytearray_encoding", expression.getEncoding()), ("bytearray_errors", expression.getErrors()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinStaticmethodCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_STATICMETHOD", arg_desc = ( ("staticmethod_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinClassmethodCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_CLASSMETHOD", arg_desc = ( ("classmethod_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/LineNumberCodes.py0000644000372000001440000000457013122472300021752 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Generate code that updates the source code line. """ def getCurrentLineNumberCode(context): frame_handle = context.getFrameHandle() if frame_handle is None: return "" else: source_ref = context.getCurrentSourceCodeReference() if source_ref.isInternal(): return "" else: return str(source_ref.getLineNumber()) def getLineNumberUpdateCode(context): lineno_value = getCurrentLineNumberCode(context) if lineno_value: frame_handle = context.getFrameHandle() return "%s->m_frame.f_lineno = %s;" % ( frame_handle, lineno_value ) else: return "" def getErrorLineNumberUpdateCode(context): lineno_value = getCurrentLineNumberCode(context) if lineno_value: return "exception_lineno = %s;" % ( lineno_value ) else: return "" def emitErrorLineNumberUpdateCode(emit, context): update_code = getErrorLineNumberUpdateCode(context) if update_code: emit(update_code) def emitLineNumberUpdateCode(emit, context): code = getLineNumberUpdateCode(context) if code: emit(code) def getSetLineNumberCodeRaw(to_name, emit, context): assert context.getFrameHandle() is not None emit( "%s->m_frame.f_lineno = %s;" % ( context.getFrameHandle(), to_name ) ) def getLineNumberCode(to_name, emit, context): assert context.getFrameHandle() is not None emit( "%s = %s->m_frame.f_lineno;" % ( to_name, context.getFrameHandle() ) ) Nuitka-0.5.28.2/nuitka/codegen/StringCodes.py0000644000372000001440000001543313207537242021173 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ String related codes, str and unicode. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateExpressionCode from .ErrorCodes import getErrorExitCode, getReleaseCode, getReleaseCodes from .PythonAPICodes import generateCAPIObjectCode from .TupleCodes import getTupleCreationCode def generateBuiltinBytesCode(to_name, expression, emit, context): encoding = expression.getEncoding() errors = expression.getErrors() generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_BYTES3", arg_desc = ( ("bytes_arg", expression.getValue()), ("bytes_encoding", encoding), ("bytes_errors", errors), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinUnicodeCode(to_name, expression, emit, context): encoding = expression.getEncoding() errors = expression.getErrors() if encoding is None and errors is None: generateCAPIObjectCode( to_name = to_name, capi = "PyObject_Unicode", arg_desc = ( ( "str_arg" if python_version < 300 \ else "unicode_arg", expression.getValue() ), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) else: generateCAPIObjectCode( to_name = to_name, capi = "TO_UNICODE3", arg_desc = ( ("unicode_arg", expression.getValue()), ("unicode_encoding", encoding), ("unicode_errors", errors), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinStrCode(to_name, expression, emit, context): if python_version < 300: generateCAPIObjectCode( to_name = to_name, capi = "PyObject_Str", arg_desc = ( ("str_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) else: return generateBuiltinUnicodeCode(to_name, expression, emit, context) def generateBuiltinChrCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_CHR", arg_desc = ( ("chr_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinOrdCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_ORD", arg_desc = ( ("ord_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateStringContenationCode(to_name, expression, emit, context): values = expression.getValues() # TODO: These are bad things. tuple_temp_name = context.allocateTempName("string_concat_values") getTupleCreationCode( to_name = tuple_temp_name, elements = values, emit = emit, context = context ) emit( "%s = PyUnicode_Join( %s, %s );" % ( to_name, context.getConstantCode(""), tuple_temp_name ) ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) getReleaseCode( release_name = tuple_temp_name, emit = emit, context = context ) def generateBuiltinFormatCode(to_name, expression, emit, context): value_name = context.allocateTempName("format_value") generateExpressionCode( to_name = value_name, expression = expression.getValue(), emit = emit, context = context ) format_spec_name = context.allocateTempName("format_spec") format_spec = expression.getFormatSpec() if format_spec is None: emit( "%s = %s;" % ( format_spec_name, context.getConstantCode("") ) ) else: generateExpressionCode( to_name = format_spec_name, expression = format_spec, emit = emit, context = context ) emit( "%s = BUILTIN_FORMAT( %s, %s );" % ( to_name, value_name, format_spec_name ) ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) getReleaseCodes( release_names = (value_name, format_spec_name), emit = emit, context = context ) def generateBuiltinAsciiCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PyObject_ASCII", arg_desc = ( ("ascii_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/FrameCodes.py0000644000372000001440000004121013207537242020747 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Frame codes This is about frame stacks and their management. There are different kinds of frames for different uses. """ from nuitka.PythonVersions import python_version from . import Emission from .CodeHelpers import _generateStatementSequenceCode from .ErrorCodes import getFrameVariableTypeDescriptionCode from .ExceptionCodes import getTracebackMakingIdentifier from .Indentation import indented from .LabelCodes import getGotoCode, getLabelCode from .ModuleCodes import getModuleAccessCode from .templates.CodeTemplatesFrames import ( template_frame_attach_locals, template_frame_guard_cache_decl, template_frame_guard_frame_decl, template_frame_guard_full_block, template_frame_guard_full_exception_handler, template_frame_guard_full_return_handler, template_frame_guard_generator, template_frame_guard_generator_exception_handler, template_frame_guard_generator_return_handler, template_frame_guard_once ) def getFrameLocalsStorageSize(type_descriptions): candidates = set() for type_description in type_descriptions: candidate = '+'.join( getTypeSizeOf(type_indicator) for type_indicator in sorted(type_description) ) candidates.add(candidate) if not candidates: return '0' candidates = list(sorted(candidates)) result = candidates.pop() while candidates: # assert False, (type_descriptions, context.frame_variables_stack[-1]) result = "MAX( %s, %s )" % (result, candidates.pop()) return result def generateStatementsFrameCode(statement_sequence, emit, context): # This is a wrapper that provides also handling of frames, which got a # lot of variants and details, therefore lots of branches. # pylint: disable=too-many-branches context.pushCleanupScope() guard_mode = statement_sequence.getGuardMode() code_object = statement_sequence.getCodeObject() code_identifier = context.getCodeObjectHandle( code_object = code_object ) parent_exception_exit = context.getExceptionEscape() # For nested frames, make sure to restore set the type description. if context.getFrameHandle() is not None: real_parent_exception_exit = parent_exception_exit parent_exception_exit = context.allocateLabel("nested_frame_exit") if statement_sequence.hasStructureMember(): frame_identifier = "%s->m_frame" % context.getContextObjectName() else: frame_identifier = code_identifier.replace("codeobj_", "frame_") # Allow stacking of frame handles. frame_identifier = context.pushFrameHandle( frame_identifier ) context.setExceptionEscape( context.allocateLabel("frame_exception_exit") ) needs_preserve = statement_sequence.needsFrameExceptionPreserving() if statement_sequence.mayReturn(): parent_return_exit = context.getReturnTarget() context.setReturnTarget( context.allocateLabel("frame_return_exit") ) else: parent_return_exit = None # We need to define that all the variables needs to be pushed. We do not # have a flag that says "always NULL" for variables. With efficient NULL # passing however (not at all, TODO), that doesn't matter much. local_variables = statement_sequence.getParentVariableProvider().getLocalVariables() def search(variable_name): for local_variable in local_variables: if local_variable.getName() == variable_name: return local_variable return None context.pushFrameVariables( tuple( search(variable_name) for variable_name in code_object.getVarNames() ) ) # Now generate the statements code into a local buffer, to we can wrap # the frame stuff around it. local_emit = Emission.SourceCodeCollector() _generateStatementSequenceCode( statement_sequence = statement_sequence, emit = local_emit, context = context ) if statement_sequence.mayRaiseException(BaseException): frame_exception_exit = context.getExceptionEscape() else: frame_exception_exit = None if parent_return_exit is not None: frame_return_exit = context.getReturnTarget() else: frame_return_exit = None type_descriptions = context.getFrameVariableTypeDescriptions() if guard_mode == "generator": # TODO: This case should care about "needs_preserve", as for # Python3 it is actually not a stub of empty code. getFrameGuardLightCode( code_identifier = code_identifier, type_descriptions = type_descriptions, codes = local_emit.codes, parent_exception_exit = parent_exception_exit, frame_exception_exit = frame_exception_exit, parent_return_exit = parent_return_exit, frame_return_exit = frame_return_exit, emit = emit, context = context ) elif guard_mode == "full": getFrameGuardHeavyCode( frame_identifier = context.getFrameHandle(), code_identifier = code_identifier, type_descriptions = type_descriptions, parent_exception_exit = parent_exception_exit, parent_return_exit = parent_return_exit, frame_exception_exit = frame_exception_exit, frame_return_exit = frame_return_exit, codes = local_emit.codes, needs_preserve = needs_preserve, emit = emit, context = context ) elif guard_mode == "once": getFrameGuardOnceCode( frame_identifier = context.getFrameHandle(), code_identifier = code_identifier, parent_exception_exit = parent_exception_exit, parent_return_exit = parent_return_exit, frame_exception_exit = frame_exception_exit, frame_return_exit = frame_return_exit, codes = local_emit.codes, needs_preserve = needs_preserve, emit = emit, context = context ) else: assert False, guard_mode context.popFrameVariables() context.popFrameHandle() # For nested frames, make sure to restore set the type description. if context.getFrameHandle() is not None: label = context.allocateLabel("skip_nested_handling") getGotoCode(label, emit) getLabelCode(parent_exception_exit, emit) emit(getFrameVariableTypeDescriptionCode(context)) getGotoCode(real_parent_exception_exit, emit) getLabelCode(label, emit) parent_exception_exit = real_parent_exception_exit context.popCleanupScope() context.setExceptionEscape(parent_exception_exit) if frame_return_exit is not None: context.setReturnTarget(parent_return_exit) def getTypeSizeOf(type_indicator): if type_indicator in ('O', 'o', 'N', 'c'): return "sizeof(void *)" elif type_indicator == 'b': return "sizeof(nuitka_bool)" else: assert False, type_indicator def getFrameAttachLocalsCode(context, frame_identifier): frame_variable_codes = context.getFrameVariableCodeNames() frame_variable_codes = ",\n ".join(frame_variable_codes) if frame_variable_codes: frame_variable_codes = ",\n " + frame_variable_codes return template_frame_attach_locals % { "frame_identifier" : frame_identifier, "type_description" : context.getFrameVariableTypeDescriptionName(), "frame_variable_refs" : frame_variable_codes } def getFrameGuardHeavyCode(frame_identifier, code_identifier, codes, type_descriptions, needs_preserve, parent_exception_exit, parent_return_exit, frame_exception_exit, frame_return_exit, emit, context): # We really need this many parameters here. no_exception_exit = context.allocateLabel("frame_no_exception") context.addFrameDeclaration( template_frame_guard_cache_decl % { "frame_identifier" : frame_identifier, } ) context.addFrameDeclaration( template_frame_guard_frame_decl % { "frame_identifier" : frame_identifier, } ) emit( template_frame_guard_full_block % { "frame_identifier" : frame_identifier, "code_identifier" : code_identifier, "locals_size" : getFrameLocalsStorageSize(type_descriptions), "codes" : indented(codes, 0), "module_identifier" : getModuleAccessCode(context), "no_exception_exit" : no_exception_exit, "needs_preserve" : 1 if needs_preserve else 0, } ) if frame_return_exit is not None: emit( template_frame_guard_full_return_handler % { "frame_identifier" : frame_identifier, "return_exit" : parent_return_exit, "frame_return_exit" : frame_return_exit, "needs_preserve" : 1 if needs_preserve else 0, } ) if frame_exception_exit is not None: emit( template_frame_guard_full_exception_handler % { "frame_identifier" : frame_identifier, "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = "exception_lineno" ), "parent_exception_exit" : parent_exception_exit, "frame_exception_exit" : frame_exception_exit, "attach_locals" : getFrameAttachLocalsCode(context, frame_identifier), "needs_preserve" : 1 if needs_preserve else 0, } ) emit("%s:;\n" % no_exception_exit) def getFrameGuardOnceCode(frame_identifier, code_identifier, codes, parent_exception_exit, parent_return_exit, frame_exception_exit, frame_return_exit, needs_preserve, emit, context): # We really need this many parameters here. # Used for modules only currently, but that ought to change. assert parent_return_exit is None and frame_return_exit is None context.addFrameDeclaration( template_frame_guard_frame_decl % { "frame_identifier" : frame_identifier, } ) emit( template_frame_guard_once % { "frame_identifier" : frame_identifier, "code_identifier" : code_identifier, "codes" : indented(codes, 0), "module_identifier" : getModuleAccessCode(context), "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = "exception_lineno" ), "parent_exception_exit" : parent_exception_exit, "frame_exception_exit" : frame_exception_exit, "no_exception_exit" : context.allocateLabel( "frame_no_exception" ), "needs_preserve" : 1 if needs_preserve else 0 } ) def getFrameGuardLightCode(code_identifier, codes, parent_exception_exit, type_descriptions, parent_return_exit, frame_exception_exit, frame_return_exit, emit, context): context.markAsNeedsExceptionVariables() context_identifier = context.getContextObjectName() context.addFrameDeclaration( template_frame_guard_cache_decl % { "frame_identifier" : "frame_" + context_identifier, } ) no_exception_exit = context.allocateLabel("frame_no_exception") frame_identifier = context_identifier + "->m_frame" emit( template_frame_guard_generator % { "context_identifier" : context_identifier, "frame_cache_identifier" : "cache_frame_" + context_identifier, "code_identifier" : code_identifier, "locals_size" : getFrameLocalsStorageSize(type_descriptions), "codes" : indented(codes, 0), "module_identifier" : getModuleAccessCode(context), "no_exception_exit" : no_exception_exit, } ) if frame_return_exit is not None: emit( template_frame_guard_generator_return_handler % { "frame_identifier" : "%s->m_frame" % context_identifier, "return_exit" : parent_return_exit, "frame_return_exit" : frame_return_exit, } ) if frame_exception_exit is not None: emit( template_frame_guard_generator_exception_handler % { "frame_identifier" : "%s->m_frame" % context_identifier, "frame_cache_identifier" : "cache_frame_" + context_identifier, "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = "exception_lineno" ), "attach_locals" : indented( getFrameAttachLocalsCode(context, frame_identifier) ), "frame_exception_exit" : frame_exception_exit, "parent_exception_exit" : parent_exception_exit } ) emit("%s:;\n" % no_exception_exit) def generateFramePreserveExceptionCode(statement, emit, context): emit("// Preserve existing published exception.") if python_version < 300: emit( "PRESERVE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: preserver_id = statement.getPreserverId() if preserver_id == 0 and python_version < 300: emit( "PRESERVE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: context.addExceptionPreserverVariables(preserver_id) emit( """\ exception_preserved_type_%(preserver_id)d = PyThreadState_GET()->exc_type; Py_XINCREF( exception_preserved_type_%(preserver_id)d ); exception_preserved_value_%(preserver_id)d = PyThreadState_GET()->exc_value; Py_XINCREF( exception_preserved_value_%(preserver_id)d ); exception_preserved_tb_%(preserver_id)d = (PyTracebackObject *)PyThreadState_GET()->exc_traceback; Py_XINCREF( exception_preserved_tb_%(preserver_id)d ); """ % { "preserver_id" : preserver_id, } ) def generateFrameRestoreExceptionCode(statement, emit, context): emit("// Restore previous exception.") if python_version < 300: emit( "RESTORE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: preserver_id = statement.getPreserverId() if preserver_id == 0 and python_version < 300: emit( "RESTORE_FRAME_EXCEPTION( %(frame_identifier)s );" % { "frame_identifier" : context.getFrameHandle() } ) else: emit( """\ SET_CURRENT_EXCEPTION( exception_preserved_type_%(preserver_id)d, \ exception_preserved_value_%(preserver_id)d, \ exception_preserved_tb_%(preserver_id)d );""" % { "preserver_id" : preserver_id, } ) Nuitka-0.5.28.2/nuitka/codegen/ClassCodes.py0000644000372000001440000000736513207537242020777 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Codes for classes. Most the class specific stuff is solved in re-formulation. Only the selection of the metaclass remains as specific. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateChildExpressionsCode from .ErrorCodes import getErrorExitCode, getReleaseCode, getReleaseCodes from .PythonAPICodes import generateCAPIObjectCode0 def _getMetaclassVariableCode(context): assert python_version < 300 return "GET_STRING_DICT_VALUE( moduledict_%s, (Nuitka_StringObject *)%s )" % ( context.getModuleCodeName(), context.getConstantCode( constant = "__metaclass__" ) ) def generateSelectMetaclassCode(to_name, expression, emit, context): metaclass_name, bases_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) if python_version < 300: assert metaclass_name is None args = [ bases_name, _getMetaclassVariableCode(context = context) ] else: args = [ metaclass_name, bases_name ] emit( "%s = SELECT_METACLASS( %s );" % ( to_name, ", ".join(args) ) ) # Can only fail with Python3. if python_version >= 300: getErrorExitCode( check_name = to_name, emit = emit, context = context ) getReleaseCodes( release_names = args, emit = emit, context = context ) else: getReleaseCode( release_name = bases_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinSuperCode(to_name, expression, emit, context): type_name, object_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = BUILTIN_SUPER( %s, %s );" % ( to_name, type_name if type_name is not None else "NULL", object_name if object_name is not None else "NULL" ) ) getReleaseCodes( release_names = (type_name, object_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinIsinstanceCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "BUILTIN_ISINSTANCE", arg_desc = ( ("isinstance_inst", expression.getInstance()), ("isinstance_cls", expression.getCls()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/PythonAPICodes.py0000644000372000001440000000702513207537242021536 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for standard CPython/API calls. This is generic stuff. """ from .CodeHelpers import generateExpressionCode from .ErrorCodes import getErrorExitCode, getReleaseCodes def generateCAPIObjectCodeCommon(to_name, capi, arg_desc, may_raise, ref_count, source_ref, emit, context, none_null = False): arg_names = [] for arg_name, arg_expression in arg_desc: if arg_expression is None and none_null: arg_names.append("NULL") else: arg_name = context.allocateTempName(arg_name) generateExpressionCode( to_name = arg_name, expression = arg_expression, emit = emit, context = context ) arg_names.append(arg_name) context.setCurrentSourceCodeReference(source_ref) getCAPIObjectCode( to_name = to_name, capi = capi, arg_names = arg_names, may_raise = may_raise, ref_count = ref_count, emit = emit, context = context ) def generateCAPIObjectCode(to_name, capi, arg_desc, may_raise, source_ref, emit, context, none_null = False): generateCAPIObjectCodeCommon( to_name = to_name, capi = capi, arg_desc = arg_desc, may_raise = may_raise, ref_count = 1, source_ref = source_ref, emit = emit, context = context, none_null = none_null ) def generateCAPIObjectCode0(to_name, capi, arg_desc, may_raise, source_ref, emit, context, none_null = False): generateCAPIObjectCodeCommon( to_name = to_name, capi = capi, arg_desc = arg_desc, may_raise = may_raise, ref_count = 0, source_ref = source_ref, emit = emit, context = context, none_null = none_null ) def getCAPIObjectCode(to_name, capi, arg_names, may_raise, ref_count, emit, context): emit( "%s = %s( %s );" % ( to_name, capi, ", ".join(arg_names) ) ) getReleaseCodes( release_names = ( arg_name for arg_name in arg_names if arg_name != "NULL" ), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = may_raise, emit = emit, context = context ) if ref_count: context.addCleanupTempName(to_name) def getReferenceExportCode(base_name, emit, context): if not context.needsCleanup(base_name): emit("Py_INCREF( %s );" % base_name) Nuitka-0.5.28.2/nuitka/codegen/YieldCodes.py0000644000372000001440000000764613207537242021002 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Yield related codes. The normal "yield", and the Python 3.3 or higher "yield from" variant. """ from nuitka import Options from .CodeHelpers import generateChildExpressionsCode from .ErrorCodes import getErrorExitCode, getReleaseCode from .PythonAPICodes import getReferenceExportCode def generateYieldCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # In handlers, we must preserve/restore the exception. preserve_exception = expression.isExceptionPreserving() # This will produce GENERATOR_YIELD, COROUTINE_YIELD or ASYNCGEN_YIELD. getReferenceExportCode(value_name, emit, context) if Options.isExperimental("generator_goto"): yield_return_label = context.allocateLabel("yield_return") yield_return_index = yield_return_label.split('_')[-1] emit( """ %(context_object_name)s->m_yield_return_index = %(yield_return_index)s; return %(yielded_value)s; %(yield_return_label)s: %(to_name)s = yield_return_value; """ % { "context_object_name" : context.getContextObjectName(), "yield_return_index" : yield_return_index, "yielded_value" : value_name, "yield_return_label" : yield_return_label, "to_name" : to_name } ) else: emit( "%s = %s_%s( %s, %s );" % ( to_name, context.getContextObjectName().upper(), "YIELD" if not preserve_exception else "YIELD_IN_HANDLER", context.getContextObjectName(), value_name ) ) if context.needsCleanup(value_name): context.removeCleanupTempName(value_name) getErrorExitCode( check_name = to_name, emit = emit, context = context ) # Comes as only borrowed. # context.addCleanupTempName(to_name) def generateYieldFromCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # In handlers, we must preserve/restore the exception. preserve_exception = expression.isExceptionPreserving() # This will produce GENERATOR_YIELD_FROM, COROUTINE_YIELD_FROM or # ASYNCGEN_YIELD_FROM. getReferenceExportCode(value_name, emit, context) emit( "%s = %s_%s( %s, %s );" % ( to_name, context.getContextObjectName().upper(), "YIELD_FROM" if not preserve_exception else "YIELD_FROM_IN_HANDLER", context.getContextObjectName(), value_name ) ) if not context.needsCleanup(value_name): context.addCleanupTempName(value_name) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/IndexCodes.py0000644000372000001440000000317613112214770020766 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for index values. This is not for actual subscripts (see SubscriptCodes), but to convert generic values to indexes. Also the maximum and minimum index values are abstracted here. """ from .ErrorCodes import getErrorExitBoolCode def getMaxIndexCode(to_name, emit): emit( "%s = PY_SSIZE_T_MAX;" % to_name ) def getMinIndexCode(to_name, emit): emit( "%s = 0;" % to_name ) def getIndexCode(to_name, value_name, emit, context): emit( "%s = CONVERT_TO_INDEX( %s );" % ( to_name, value_name, ) ) getErrorExitBoolCode( condition = "%s == -1 && ERROR_OCCURRED()" % to_name, emit = emit, context = context ) def getIndexValueCode(to_name, value, emit): emit( "%s = %d;" % ( to_name, value ) ) Nuitka-0.5.28.2/nuitka/codegen/OperationCodes.py0000644000372000001440000001351113207537242021660 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Codes for operations. There are unary and binary operations. Many of them have specializations and of course types could play into it. Then there is also the added difficulty of in-place assignments, which have other operation variants. """ from . import OperatorCodes from .CodeHelpers import generateChildExpressionsCode from .ErrorCodes import getErrorExitBoolCode, getErrorExitCode, getReleaseCode def generateOperationBinaryCode(to_name, expression, emit, context): left_arg_name, right_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) inplace = expression.isInplaceSuspect() assert not inplace or not expression.getLeft().isCompileTimeConstant(), \ expression getOperationCode( to_name = to_name, operator = expression.getOperator(), arg_names = (left_arg_name, right_arg_name), in_place = inplace, emit = emit, context = context ) def generateOperationUnaryCode(to_name, expression, emit, context): arg_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) inplace = expression.isInplaceSuspect() assert not inplace or not expression.getOperand().isCompileTimeConstant(), \ expression getOperationCode( to_name = to_name, operator = expression.getOperator(), arg_names = (arg_name,), in_place = inplace, emit = emit, context = context ) def getOperationCode(to_name, operator, arg_names, in_place, emit, context): # This needs to have one case per operation of Python, and there are many # of these, # pylint: disable=too-many-branches,too-many-statements prefix_args = () ref_count = 1 if operator == "Pow": helper = "POWER_OPERATION" elif operator == "IPow" and in_place: helper = "POWER_OPERATION_INPLACE" elif operator == "IPow": helper = "POWER_OPERATION2" elif operator == "Add": helper = "BINARY_OPERATION_ADD" elif operator == "IAdd" and in_place: helper = "BINARY_OPERATION_ADD_INPLACE" elif operator == "IMult" and in_place: helper = "BINARY_OPERATION_MUL_INPLACE" elif operator == "Sub": helper = "BINARY_OPERATION_SUB" elif operator == "Div": helper = "BINARY_OPERATION_DIV" elif operator == "FloorDiv": helper = "BINARY_OPERATION_FLOORDIV" elif operator == "TrueDiv": helper = "BINARY_OPERATION_TRUEDIV" elif operator == "Mult": helper = "BINARY_OPERATION_MUL" elif operator == "Mod": helper = "BINARY_OPERATION_REMAINDER" elif operator == "Divmod": helper = "BUILTIN_DIVMOD" elif len(arg_names) == 2: helper = "BINARY_OPERATION" prefix_args = ( OperatorCodes.binary_operator_codes[ operator ], ) elif len(arg_names) == 1: impl_helper, ref_count = OperatorCodes.unary_operator_codes[ operator ] helper = "UNARY_OPERATION" prefix_args = ( impl_helper, ) else: assert False, operator # We must assume to write to a variable is "in_place" is active, not e.g. # a constant reference. That was asserted before calling us. if in_place: res_name = context.getBoolResName() # We may have not specialized this one yet, so lets use generic in-place # code, or the helper specified. if helper == "BINARY_OPERATION": emit( "%s = BINARY_OPERATION_INPLACE( %s, &%s, %s );" % ( res_name, OperatorCodes.binary_operator_codes[ operator ], arg_names[0], arg_names[1], ) ) else: emit( "%s = %s( &%s, %s );" % ( res_name, helper, arg_names[0], arg_names[1], ) ) ref_count = 0 emit( "%s = %s;" % ( to_name, arg_names[0] ) ) for arg_name in arg_names: getReleaseCode( arg_name, emit, context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) if ref_count: context.addCleanupTempName(to_name) else: emit( "%s = %s( %s );" % ( to_name, helper, ", ".join(prefix_args + arg_names) ) ) for arg_name in arg_names: getReleaseCode( arg_name, emit, context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) if ref_count: context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/OperatorCodes.py0000644000372000001440000000572713122472300021512 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Operator code tables These are mostly used to look up the Python C/API from operations or a wrapper used. """ from nuitka.PythonVersions import python_version binary_operator_codes = { # Those commented out in this section have fully specialized variants already. # "Add" : "PyNumber_Add", # "Sub" : "PyNumber_Subtract", # "Div" : "PyNumber_Divide", # "Mult" : "PyNumber_Multiply", # "Mod" : "PyNumber_Remainder", # "Div" : "PyNumber_Divide", # "FloorDiv" : "PyNumber_FloorDivide", # "TrueDiv" : "PyNumber_TrueDivide", # These have their own variants only to make sure the generic code is in-lined # but the CPython code is not in-lined. # "Pow" : "PyNumber_Power", # "IPow" : "PyNumber_InPlacePower", # The others are generic code and would be faster if they had a specialized variant too. "LShift" : "PyNumber_Lshift", "RShift" : "PyNumber_Rshift", "BitAnd" : "PyNumber_And", "BitOr" : "PyNumber_Or", "BitXor" : "PyNumber_Xor", "IAdd" : "PyNumber_InPlaceAdd", "ISub" : "PyNumber_InPlaceSubtract", "IMult" : "PyNumber_InPlaceMultiply", "IDiv" : "PyNumber_InPlaceDivide", "IFloorDiv" : "PyNumber_InPlaceFloorDivide", "ITrueDiv" : "PyNumber_InPlaceTrueDivide", "IMod" : "PyNumber_InPlaceRemainder", "ILShift" : "PyNumber_InPlaceLshift", "IRShift" : "PyNumber_InPlaceRshift", "IBitAnd" : "PyNumber_InPlaceAnd", "IBitOr" : "PyNumber_InPlaceOr", "IBitXor" : "PyNumber_InPlaceXor", } # Python 3.5 only operator if python_version >= 350: binary_operator_codes["MatMult"] = "PyNumber_MatrixMultiply" binary_operator_codes["IMatMult"] = "PyNumber_InPlaceMatrixMultiply" unary_operator_codes = { "UAdd" : ("PyNumber_Positive", 1), "USub" : ("PyNumber_Negative", 1), "Invert" : ("PyNumber_Invert", 1), "Repr" : ("PyObject_Repr", 1), "Not" : ("UNARY_NOT", 0) } rich_comparison_codes = { "Lt" : "LT", "LtE" : "LE", "Eq" : "EQ", "NotEq" : "NE", "Gt" : "GT", "GtE" : "GE" } normal_comparison_codes = { "In" : "SEQUENCE_CONTAINS", "NotIn" : "SEQUENCE_CONTAINS_NOT" } Nuitka-0.5.28.2/nuitka/codegen/c_types/0000755000372000001440000000000013207540420020025 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/codegen/c_types/CTypePyObjectPtrs.py0000644000372000001440000002654413134660221023750 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ CType classes for PyObject *, PyObject **, and struct Nuitka_CellObject * """ from nuitka.codegen.ErrorCodes import ( getAssertionCode, getCheckObjectCode, getLocalVariableReferenceErrorCode ) from nuitka.codegen.templates.CodeTemplatesVariables import ( template_del_local_intolerant, template_del_local_known, template_del_local_tolerant, template_del_shared_intolerant, template_del_shared_known, template_del_shared_tolerant, template_read_local, template_read_shared_unclear, template_release_clear, template_release_unclear, template_write_local_clear_ref0, template_write_local_clear_ref1, template_write_local_empty_ref0, template_write_local_empty_ref1, template_write_local_inplace, template_write_local_unclear_ref0, template_write_local_unclear_ref1, template_write_shared_clear_ref0, template_write_shared_clear_ref1, template_write_shared_inplace, template_write_shared_unclear_ref0, template_write_shared_unclear_ref1 ) from .CTypeBases import CTypeBase class CPythonPyObjectPtrBase(CTypeBase): @classmethod def getLocalVariableAssignCode(cls, variable_code_name, needs_release, tmp_name, ref_count, in_place): if in_place: # Releasing is not an issue here, local variable reference never # gave a reference, and the in-place code deals with possible # replacement/release. template = template_write_local_inplace else: if ref_count: if needs_release is False: template = template_write_local_empty_ref0 elif needs_release is True: template = template_write_local_clear_ref0 else: template = template_write_local_unclear_ref0 else: if needs_release is False: template = template_write_local_empty_ref1 elif needs_release is True: template = template_write_local_clear_ref1 else: template = template_write_local_unclear_ref1 return template % { "identifier" : variable_code_name, "tmp_name" : tmp_name } @classmethod def getVariableObjectAccessCode(cls, to_name, needs_check, variable_code_name, variable, emit, context): template = template_read_local emit( template % { "tmp_name" : to_name, "identifier" : cls.getLocalVariableObjectAccessCode(variable_code_name) } ) if needs_check: getLocalVariableReferenceErrorCode( variable = variable, condition = "%s == NULL" % to_name, emit = emit, context = context ) else: getCheckObjectCode( check_name = to_name, emit = emit ) @classmethod def getReleaseCode(cls, variable_code_name, needs_check, emit): if needs_check: template = template_release_unclear else: template = template_release_clear emit( template % { "identifier" : variable_code_name } ) class CTypePyObjectPtr(CPythonPyObjectPtrBase): c_type = "PyObject *" @classmethod def getInitValue(cls, init_from): if init_from is None: return "NULL" else: return init_from @classmethod def getVariableArgDeclarationCode(cls, variable_code_name): return "PyObject *%s" % variable_code_name @classmethod def getVariableArgReferencePassingCode(cls, variable_code_name): return '&' + variable_code_name @classmethod def getLocalVariableObjectAccessCode(cls, variable_code_name): """ Code to access value as object. """ return variable_code_name @classmethod def getLocalVariableInitTestCode(cls, variable_code_name): return "%s != NULL" % variable_code_name @classmethod def getCellObjectAssignmentCode(cls, target_cell_code, variable_code_name, emit): emit( "%s = PyCell_NEW0( %s );" % ( target_cell_code, variable_code_name ) ) @classmethod def getDeleteObjectCode(cls, variable_code_name, needs_check, tolerant, variable, emit, context): if not needs_check: emit( template_del_local_known % { "identifier" : variable_code_name } ) elif tolerant: emit( template_del_local_tolerant % { "identifier" : variable_code_name } ) else: res_name = context.getBoolResName() emit( template_del_local_intolerant % { "identifier" : variable_code_name, "result" : res_name } ) if variable.isLocalVariable(): getLocalVariableReferenceErrorCode( variable = variable, condition = "%s == false" % res_name, emit = emit, context = context ) else: getAssertionCode( check = "%s != false" % res_name, emit = emit ) class CTypePyObjectPtrPtr(CPythonPyObjectPtrBase): c_type = "PyObject **" @classmethod def getVariableArgDeclarationCode(cls, variable_code_name): return "PyObject **%s" % variable_code_name @classmethod def getVariableArgReferencePassingCode(cls, variable_code_name): return variable_code_name @classmethod def getLocalVariableObjectAccessCode(cls, variable_code_name): return '*' + variable_code_name @classmethod def getLocalVariableInitTestCode(cls, variable_code_name): return "*%s != NULL" % variable_code_name class CTypeCellObject(CTypeBase): c_type = "struct Nuitka_CellObject *" @classmethod def getInitValue(cls, init_from): # TODO: Single out "init_from" only user, so it becomes sure that we # get a reference transferred here in these cases. if init_from is not None: return "PyCell_NEW1( %s )" % init_from else: return "PyCell_EMPTY()" @classmethod def getCellObjectAssignmentCode(cls, target_cell_code, variable_code_name, emit): emit( "%s = %s;" % ( target_cell_code, variable_code_name ) ) emit( "Py_INCREF( %s );" % ( target_cell_code ) ) @classmethod def getLocalVariableAssignCode(cls, variable_code_name, needs_release, tmp_name, ref_count, in_place): if in_place: # Releasing is not an issue here, local variable reference never # gave a reference, and the in-place code deals with possible # replacement/release. template = template_write_shared_inplace else: if ref_count: if needs_release is False: template = template_write_shared_clear_ref0 else: template = template_write_shared_unclear_ref0 else: if needs_release is False: template = template_write_shared_clear_ref1 else: template = template_write_shared_unclear_ref1 return template % { "identifier" : variable_code_name, "tmp_name" : tmp_name } @classmethod def getVariableObjectAccessCode(cls, to_name, needs_check, variable_code_name, variable, emit, context): template = template_read_shared_unclear emit( template % { "tmp_name" : to_name, "identifier" : variable_code_name } ) if needs_check: getLocalVariableReferenceErrorCode( variable = variable, condition = "%s == NULL" % to_name, emit = emit, context = context ) else: getCheckObjectCode( check_name = to_name, emit = emit ) @classmethod def getVariableArgDeclarationCode(cls, variable_code_name): return "struct Nuitka_CellObject *%s" % variable_code_name @classmethod def getVariableArgReferencePassingCode(cls, variable_code_name): return variable_code_name @classmethod def getLocalVariableObjectAccessCode(cls,variable_code_name): return variable_code_name + "->ob_ref" @classmethod def getLocalVariableInitTestCode(cls, variable_code_name): return "%s->ob_ref != NULL" % variable_code_name @classmethod def getDeleteObjectCode(cls, variable_code_name, needs_check, tolerant, variable, emit, context): if not needs_check: emit( template_del_shared_known % { "identifier" : variable_code_name } ) elif tolerant: emit( template_del_shared_tolerant % { "identifier" : variable_code_name } ) else: res_name = context.getBoolResName() emit( template_del_shared_intolerant % { "identifier" : variable_code_name, "result" : res_name } ) if variable.isLocalVariable(): getLocalVariableReferenceErrorCode( variable = variable, condition = "%s == false" % res_name, emit = emit, context = context ) else: getAssertionCode( check = "%s != false" % res_name, emit = emit ) @classmethod def getReleaseCode(cls, variable_code_name, needs_check, emit): if needs_check: template = template_release_unclear else: template = template_release_clear emit( template % { "identifier" : variable_code_name } ) Nuitka-0.5.28.2/nuitka/codegen/c_types/CTypeBases.py0000644000372000001440000000746213134660221022413 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Base class for all C types. Defines the interface to use by code generation on C types. Different types then have to overload the class methods. """ type_indicators = { "PyObject *" : 'o', "PyObject **" : 'O', "struct Nuitka_CellObject *" : 'c', "nuitka_bool" : 'b' } class CTypeBase(object): # For overload. c_type = None @classmethod def getTypeIndicator(cls): return type_indicators[cls.c_type] @classmethod def getInitValue(cls, init_from): """ Convert to init value for the type. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getVariableInitCode(cls, variable_code_name, init_from): init_value = cls.getInitValue(init_from) return "%s%s%s = %s;" % ( cls.c_type, ' ' if cls.c_type[-1] not in '*' else "", variable_code_name, init_value ) @classmethod def getLocalVariableInitTestCode(cls, variable_code_name): """ Get code to test for uninitialized. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getLocalVariableAssignCode(cls, variable_code_name, needs_release, tmp_name, ref_count, in_place): """ Get code to assign local variable. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getDeleteObjectCode(cls, variable_code_name, needs_check, tolerant, variable, emit, context): """ Get code to delete (del) local variable. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getVariableArgReferencePassingCode(cls, variable_code_name): """ Get code to pass variable as reference argument. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getVariableArgDeclarationCode(cls, variable_code_name): """ Get variable declaration code with given name. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getCellObjectAssignmentCode(cls, target_cell_code, variable_code_name, emit): """ Get assignment code to given cell object from object. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type @classmethod def getReleaseCode(cls, variable_code_name, needs_check, emit): """ Get release code for given object. """ # Need to overload this for each type it is used for, pylint: disable=unused-argument assert False, cls.c_type Nuitka-0.5.28.2/nuitka/codegen/c_types/__init__.py0000644000372000001440000000150113122472300022130 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/codegen/c_types/CTypeNuitkaBools.py0000644000372000001440000001066413134660221023606 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ CType classes for nuitka_bool, an enum to represent True, False, unassigned. """ from nuitka.codegen.ErrorCodes import ( getAssertionCode, getLocalVariableReferenceErrorCode ) from .CTypeBases import CTypeBase class CTypeNuitkaBoolEnum(CTypeBase): c_type = "nuitka_bool" @classmethod def getLocalVariableAssignCode(cls, variable_code_name, needs_release, tmp_name, ref_count, in_place): assert not in_place assert not ref_count return """\ if (%(tmp_name)s == Py_True) { %(variable_code_name)s = NUITKA_BOOL_TRUE; } else { %(variable_code_name)s = NUITKA_BOOL_FALSE; } """ % { "variable_code_name" : variable_code_name, "tmp_name" : tmp_name, } @classmethod def getVariableObjectAccessCode(cls, to_name, needs_check, variable_code_name, variable, emit, context): emit( """\ switch (%(variable_code_name)s) { case NUITKA_BOOL_TRUE: { %(to_name)s = Py_True; break; } case NUITKA_BOOL_FALSE: { %(to_name)s = Py_False; break; } // case NUITKA_BOOL_UNASSIGNED: (MSVC wants default to believe it). We may // try to add an illegal default instead, but that may trigger warnings // from better compilers. default: { #if %(needs_check)s %(to_name)s = NULL; #else NUITKA_CANNOT_GET_HERE(%(identifier)s); #endif break; } }""" % { "variable_code_name" : variable_code_name, "to_name" : to_name, "identifier" : context.getOwner().getCodeName(), "needs_check" : '1' if needs_check else '0' } ) if 0: # Future work, pylint: disable=using-constant-test context.reportObjectConversion(variable) @classmethod def getLocalVariableInitTestCode(cls, variable_code_name): return "%s != NUITKA_BOOL_UNASSIGNED" % variable_code_name @classmethod def getLocalVariableObjectAccessCode(cls, variable_code_name): return "%s == NUITKA_BOOL_TRUE ? Py_True : Py_False" % variable_code_name @classmethod def getInitValue(cls, init_from): if init_from is None: return "NUITKA_BOOL_UNASSIGNED" else: assert False, init_from return init_from @classmethod def getReleaseCode(cls, variable_code_name, needs_check, emit): # TODO: Allow optimization to not get here. pass @classmethod def getDeleteObjectCode(cls, variable_code_name, needs_check, tolerant, variable, emit, context): if not needs_check: emit( "%s = NUITKA_BOOL_UNASSIGNED;" % variable_code_name ) elif tolerant: emit( "%s = NUITKA_BOOL_UNASSIGNED;" % variable_code_name ) else: res_name = context.getBoolResName() emit( "%s = %s == NUITKA_BOOL_UNASSIGNED;" % ( res_name, variable_code_name, ) ) emit( "%s = NUITKA_BOOL_UNASSIGNED;" % variable_code_name ) if variable.isLocalVariable(): getLocalVariableReferenceErrorCode( variable = variable, condition = "%s == false" % res_name, emit = emit, context = context ) else: getAssertionCode( check = "%s != false" % res_name, emit = emit ) Nuitka-0.5.28.2/nuitka/codegen/RaisingCodes.py0000644000372000001440000002763113207537242021324 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for implicit and explicit exception raises. Exceptions from other operations are consider ErrorCodes domain. """ from nuitka.Options import isDebug from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import getFrameVariableTypeDescriptionCode from .LabelCodes import getGotoCode from .LineNumberCodes import ( emitErrorLineNumberUpdateCode, getErrorLineNumberUpdateCode ) from .PythonAPICodes import getReferenceExportCode def generateRaiseCode(statement, emit, context): exception_type = statement.getExceptionType() exception_value = statement.getExceptionValue() exception_tb = statement.getExceptionTrace() exception_cause = statement.getExceptionCause() context.markAsNeedsExceptionVariables() # Exception cause is only possible with simple raise form. if exception_cause is not None: assert exception_type is not None assert exception_value is None assert exception_tb is None raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) raise_cause_name = context.allocateTempName("raise_cause") generateExpressionCode( to_name = raise_cause_name, expression = exception_cause, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(exception_cause.getSourceReference()) getRaiseExceptionWithCauseCode( raise_type_name = raise_type_name, raise_cause_name = raise_cause_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif exception_type is None: assert exception_cause is None assert exception_value is None assert exception_tb is None old_source_ref = context.setCurrentSourceCodeReference( value = statement.getCompatibleSourceReference() ) getReRaiseExceptionCode( emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif exception_value is None and exception_tb is None: raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value = exception_type.getCompatibleSourceReference() ) getRaiseExceptionWithTypeCode( raise_type_name = raise_type_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif exception_tb is None: raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) raise_value_name = context.allocateTempName("raise_value") generateExpressionCode( to_name = raise_value_name, expression = exception_value, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(exception_value.getSourceReference()) context.setCurrentSourceCodeReference( statement.getCompatibleSourceReference() ) getRaiseExceptionWithValueCode( raise_type_name = raise_type_name, raise_value_name = raise_value_name, implicit = statement.isImplicit(), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: raise_type_name = context.allocateTempName("raise_type") generateExpressionCode( to_name = raise_type_name, expression = exception_type, emit = emit, context = context ) raise_value_name = context.allocateTempName("raise_value") generateExpressionCode( to_name = raise_value_name, expression = exception_value, emit = emit, context = context ) raise_tb_name = context.allocateTempName("raise_tb") generateExpressionCode( to_name = raise_tb_name, expression = exception_tb, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(exception_tb.getSourceReference()) getRaiseExceptionWithTracebackCode( raise_type_name = raise_type_name, raise_value_name = raise_value_name, raise_tb_name = raise_tb_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateRaiseExpressionCode(to_name, expression, emit, context): arg_names = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # Missed optimization opportunity, please report it, this should not # normally happen. We are supposed to propagate this upwards. if isDebug(): parent = expression.parent assert parent.isExpressionSideEffects() or \ parent.isExpressionConditional(), \ (expression, expression.parent) # That's how we indicate exception to the surrounding world. emit("%s = NULL;" % to_name) getRaiseExceptionWithValueCode( raise_type_name = arg_names[0], raise_value_name = arg_names[1], implicit = True, emit = emit, context = context ) def getReRaiseExceptionCode(emit, context): keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is None: emit( """\ %(bool_res_name)s = RERAISE_EXCEPTION( &exception_type, &exception_value, &exception_tb ); if (unlikely( %(bool_res_name)s == false )) { %(update_code)s } """ % { "bool_res_name" : context.getBoolResName(), "update_code" : getErrorLineNumberUpdateCode(context) } ) frame_handle = context.getFrameHandle() if frame_handle: emit( """\ if (exception_tb && exception_tb->tb_frame == &%(frame_identifier)s->m_frame) \ %(frame_identifier)s->m_frame.f_lineno = exception_tb->tb_lineno;""" % { "frame_identifier" : context.getFrameHandle() } ) emit( getFrameVariableTypeDescriptionCode(context) ) else: keeper_type, keeper_value, keeper_tb, keeper_lineno = context.getExceptionKeeperVariables() emit( """\ // Re-raise. exception_type = %(keeper_type)s; exception_value = %(keeper_value)s; exception_tb = %(keeper_tb)s; exception_lineno = %(keeper_lineno)s; """ % { "keeper_type" : keeper_type, "keeper_value" : keeper_value, "keeper_tb" : keeper_tb, "keeper_lineno" : keeper_lineno } ) getGotoCode(context.getExceptionEscape(), emit) def getRaiseExceptionWithCauseCode(raise_type_name, raise_cause_name, emit, context): context.markAsNeedsExceptionVariables() emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode(raise_type_name, emit, context) emit("exception_value = NULL;") getReferenceExportCode(raise_cause_name, emit, context) emitErrorLineNumberUpdateCode(emit, context) emit( """\ RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, \ %s );""" % raise_cause_name ) emit( getFrameVariableTypeDescriptionCode(context) ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) if context.needsCleanup(raise_cause_name): context.removeCleanupTempName(raise_cause_name) def getRaiseExceptionWithTypeCode(raise_type_name, emit, context): context.markAsNeedsExceptionVariables() emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode(raise_type_name, emit, context) emitErrorLineNumberUpdateCode(emit, context) emit( "RAISE_EXCEPTION_WITH_TYPE( &exception_type, &exception_value, &exception_tb );" ) emit( getFrameVariableTypeDescriptionCode(context) ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) def getRaiseExceptionWithValueCode(raise_type_name, raise_value_name, implicit, emit, context): emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode(raise_type_name, emit, context) emit( "exception_value = %s;" % raise_value_name ) getReferenceExportCode(raise_value_name, emit, context) emitErrorLineNumberUpdateCode(emit, context) emit( "RAISE_EXCEPTION_%s( &exception_type, &exception_value, &exception_tb );" % ( ("IMPLICIT" if implicit else "WITH_VALUE") ) ) emit( getFrameVariableTypeDescriptionCode(context) ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) if context.needsCleanup(raise_value_name): context.removeCleanupTempName(raise_value_name) def getRaiseExceptionWithTracebackCode(raise_type_name, raise_value_name, raise_tb_name, emit, context): emit( "exception_type = %s;" % raise_type_name ) getReferenceExportCode(raise_type_name, emit, context) emit( "exception_value = %s;" % raise_value_name ) getReferenceExportCode(raise_value_name, emit, context) emit( "exception_tb = (PyTracebackObject *)%s;" % raise_tb_name ) getReferenceExportCode(raise_tb_name, emit, context) emit( "RAISE_EXCEPTION_WITH_TRACEBACK( &exception_type, &exception_value, &exception_tb);" ) # If anything is wrong, that will be used. emitErrorLineNumberUpdateCode(emit, context) emit( getFrameVariableTypeDescriptionCode(context) ) getGotoCode(context.getExceptionEscape(), emit) if context.needsCleanup(raise_type_name): context.removeCleanupTempName(raise_type_name) if context.needsCleanup(raise_value_name): context.removeCleanupTempName(raise_value_name) if context.needsCleanup(raise_tb_name): context.removeCleanupTempName(raise_tb_name) Nuitka-0.5.28.2/nuitka/codegen/ListCodes.py0000644000372000001440000001200313207537242020626 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for lists. Right now only the creation is done here. But more should be added later on. """ from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .PythonAPICodes import generateCAPIObjectCode def generateListCreationCode(to_name, expression, emit, context): elements = expression.getElements() emit( "%s = PyList_New( %d );" % ( to_name, len(elements) ) ) context.addCleanupTempName(to_name) element_name = context.allocateTempName("list_element") for count, element in enumerate(elements): generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) if not context.needsCleanup(element_name): emit("Py_INCREF( %s );" % element_name) else: context.removeCleanupTempName(element_name) emit( "PyList_SET_ITEM( %s, %d, %s );" % ( to_name, count, element_name ) ) def generateListOperationAppendCode(statement, emit, context): list_arg_name = context.allocateTempName("append_list") generateExpressionCode( to_name = list_arg_name, expression = statement.getList(), emit = emit, context = context ) value_arg_name = context.allocateTempName("append_value") generateExpressionCode( to_name = value_arg_name, expression = statement.getValue(), emit = emit, context = context ) context.setCurrentSourceCodeReference(statement.getSourceReference()) res_name = context.getIntResName() emit("assert( PyList_Check( %s ) );" % list_arg_name) emit( "%s = PyList_Append( %s, %s );" % ( res_name, list_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (list_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) def generateListOperationExtendCode(to_name, expression, emit, context): list_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PyList_Check( %s ) );" % list_arg_name) emit( "%s = _PyList_Extend( (PyListObject *)%s, %s );" % ( to_name, list_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (list_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateListOperationPopCode(to_name, expression, emit, context): list_arg_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # TODO: Have a dedicated helper instead, this could be more efficient. emit("assert( PyList_Check( %s ) );" % list_arg_name) emit( '%s = PyObject_CallMethod( %s, (char *)"pop", NULL );' % ( to_name, list_arg_name ) ) getReleaseCode( release_name = list_arg_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateBuiltinListCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PySequence_List", arg_desc = ( ("list_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/LoaderCodes.py0000644000372000001440000001051113112214770021114 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with module loaders. This is for generating the look-up table for the modules included in a binary or distribution folder. """ from nuitka.ModuleRegistry import getUncompiledNonTechnicalModules from . import ConstantCodes from .Indentation import indented from .templates.CodeTemplatesLoader import ( template_metapath_loader_body, template_metapath_loader_bytecode_module_entry, template_metapath_loader_compiled_module_entry, template_metapath_loader_compiled_package_entry, template_metapath_loader_shlib_module_entry ) def getModuleMetapathLoaderEntryCode(module_name, module_identifier, is_shlib, is_package): if is_shlib: assert module_name != "__main__" assert not is_package return template_metapath_loader_shlib_module_entry % { "module_name" : module_name } elif is_package: return template_metapath_loader_compiled_package_entry % { "module_name" : module_name, "module_identifier" : module_identifier, } else: return template_metapath_loader_compiled_module_entry % { "module_name" : module_name, "module_identifier" : module_identifier, } stream_data = ConstantCodes.stream_data def getMetapathLoaderBodyCode(other_modules): metapath_loader_inittab = [] metapath_module_decls = [] for other_module in other_modules: if other_module.isUncompiledPythonModule(): code_data = other_module.getByteCode() is_package = other_module.isUncompiledPythonPackage() flags = ["NUITKA_BYTECODE_FLAG"] if is_package: flags.append("NUITKA_PACKAGE_FLAG") metapath_loader_inittab.append( template_metapath_loader_bytecode_module_entry % { "module_name" : other_module.getFullName(), "bytecode" : stream_data.getStreamDataOffset(code_data), "size" : len(code_data), "flags" : " | ".join(flags) } ) else: metapath_loader_inittab.append( getModuleMetapathLoaderEntryCode( module_name = other_module.getFullName(), module_identifier = other_module.getCodeName(), is_shlib = other_module.isPythonShlibModule(), is_package = other_module.isCompiledPythonPackage() ) ) if other_module.isCompiledPythonModule(): metapath_module_decls.append( "MOD_INIT_DECL( %s );" % other_module.getCodeName() ) for uncompiled_module in getUncompiledNonTechnicalModules(): code_data = uncompiled_module.getByteCode() is_package = uncompiled_module.isUncompiledPythonPackage() flags = ["NUITKA_BYTECODE_FLAG"] if is_package: flags.append("NUITKA_PACKAGE_FLAG") metapath_loader_inittab.append( template_metapath_loader_bytecode_module_entry % { "module_name" : uncompiled_module.getFullName(), "bytecode" : stream_data.getStreamDataOffset(code_data), "size" : len(code_data), "flags" : " | ".join(flags) } ) return template_metapath_loader_body % { "metapath_module_decls" : indented(metapath_module_decls, 0), "metapath_loader_inittab" : indented(metapath_loader_inittab) } Nuitka-0.5.28.2/nuitka/codegen/AttributeCodes.py0000644000372000001440000002625113207537242021670 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Attribute related codes. Attribute lookup, setting. """ from nuitka import Options from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .LabelCodes import getBranchingCode from .PythonAPICodes import generateCAPIObjectCode, generateCAPIObjectCode0 def generateAssignmentAttributeCode(statement, emit, context): lookup_source = statement.getLookupSource() attribute_name = statement.getAttributeName() value = statement.getAssignSource() value_name = context.allocateTempName("assattr_name") generateExpressionCode( to_name = value_name, expression = value, emit = emit, context = context ) target_name = context.allocateTempName("assattr_target") generateExpressionCode( to_name = target_name, expression = lookup_source, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) if attribute_name == "__dict__": getAttributeAssignmentDictSlotCode( target_name = target_name, value_name = value_name, emit = emit, context = context ) elif attribute_name == "__class__": getAttributeAssignmentClassSlotCode( target_name = target_name, value_name = value_name, emit = emit, context = context ) else: getAttributeAssignmentCode( target_name = target_name, value_name = value_name, attribute_name = context.getConstantCode( constant = attribute_name ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateDelAttributeCode(statement, emit, context): target_name = context.allocateTempName("attrdel_target") generateExpressionCode( to_name = target_name, expression = statement.getLookupSource(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( statement.getLookupSource().getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getAttributeDelCode( target_name = target_name, attribute_name = context.getConstantCode( constant = statement.getAttributeName() ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateAttributeLookupCode(to_name, expression, emit, context): source_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) attribute_name = expression.getAttributeName() getAttributeLookupCode( to_name = to_name, source_name = source_name, attribute_name = attribute_name, needs_check = expression.getLookupSource().mayRaiseExceptionAttributeLookup( exception_type = BaseException, attribute_name = attribute_name ), emit = emit, context = context ) def getAttributeLookupCode(to_name, source_name, attribute_name, needs_check, emit, context): if attribute_name == "__dict__": emit( "%s = LOOKUP_ATTRIBUTE_DICT_SLOT( %s );" % ( to_name, source_name ) ) elif attribute_name == "__class__": emit( "%s = LOOKUP_ATTRIBUTE_CLASS_SLOT( %s );" % ( to_name, source_name ) ) else: emit( "%s = LOOKUP_ATTRIBUTE( %s, %s );" % ( to_name, source_name, context.getConstantCode( constant = attribute_name ) ) ) getReleaseCode( release_name = source_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getAttributeCheckBoolCode(source_name, attr_name, needs_check, emit, context): res_name = context.getIntResName() emit( "%s = PyObject_HasAttr( %s, %s );" % ( res_name, source_name, attr_name ) ) getReleaseCodes( release_names = (source_name, attr_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, needs_check = needs_check, emit = emit, context = context ) getBranchingCode("%s == 1" % res_name, emit, context) def getAttributeAssignmentCode(target_name, attribute_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_ATTRIBUTE( %s, %s, %s );" % ( res_name, target_name, attribute_name, value_name ) ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) getReleaseCodes( release_names = (value_name, target_name, attribute_name), emit = emit, context = context ) def getAttributeAssignmentDictSlotCode(target_name, value_name, emit, context): """ Code for special case target.__dict__ = value """ res_name = context.getBoolResName() emit( "%s = SET_ATTRIBUTE_DICT_SLOT( %s, %s );" % ( res_name, target_name, value_name ) ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) getReleaseCodes( release_names = (value_name, target_name), emit = emit, context = context ) def getAttributeAssignmentClassSlotCode(target_name, value_name, emit, context): """ Get code for special case target.__class__ = value """ res_name = context.getBoolResName() emit( "%s = SET_ATTRIBUTE_CLASS_SLOT( %s, %s );" % ( res_name, target_name, value_name ) ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) getReleaseCodes( release_names = (value_name, target_name), emit = emit, context = context ) def getAttributeDelCode(target_name, attribute_name, emit, context): res_name = context.getIntResName() emit( "%s = PyObject_DelAttr( %s, %s );" % ( res_name, target_name, attribute_name ) ) getReleaseCodes( release_names = (target_name, attribute_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) def generateAttributeLookupSpecialCode(to_name, expression, emit, context): source_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) attribute_name = expression.getAttributeName() getAttributeLookupSpecialCode( to_name = to_name, source_name = source_name, attr_name = context.getConstantCode( constant = attribute_name ), needs_check = expression.getLookupSource().mayRaiseExceptionAttributeLookupSpecial( exception_type = BaseException, attribute_name = attribute_name ), emit = emit, context = context ) def getAttributeLookupSpecialCode(to_name, source_name, attr_name, needs_check, emit, context): emit( "%s = LOOKUP_SPECIAL( %s, %s );" % ( to_name, source_name, attr_name, ) ) getReleaseCodes( release_names = (source_name, attr_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) def generateBuiltinHasattrCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "BUILTIN_HASATTR", arg_desc = ( ("hasattr_value", expression.getLookupSource()), ("hasattr_attr", expression.getAttribute()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinGetattrCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_GETATTR", arg_desc = ( ("getattr_target", expression.getLookupSource()), ("getattr_attr", expression.getAttribute()), ("getattr_default", expression.getDefault()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), none_null = True, emit = emit, context = context ) def generateBuiltinSetattrCode(to_name, expression, emit, context): generateCAPIObjectCode0( to_name = to_name, capi = "BUILTIN_SETATTR", arg_desc = ( ("setattr_target", expression.getLookupSource()), ("setattr_attr", expression.getAttribute()), ("setattr_value", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context, ) Nuitka-0.5.28.2/nuitka/codegen/TryCodes.py0000644000372000001440000002563713207537242020512 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Try statement and related code generation. For Nuitka, all try/except and try/finally are dealt with this, where the finally block gets duplicated into handlers. So this is a common low level structure used, where exception handling and everything is made explicit. """ from nuitka import Options from .CodeHelpers import generateExpressionCode, generateStatementSequenceCode from .ErrorCodes import getMustNotGetHereCode from .ExceptionCodes import getExceptionUnpublishedReleaseCode from .IteratorCodes import getBuiltinLoopBreakNextCode from .LabelCodes import getGotoCode, getLabelCode from .VariableCodes import getVariableAssignmentCode def generateTryCode(statement, emit, context): # The try construct is the most complex for code generation. We may need to # react on break, continue, return, raise in the handlers. For exception # and return handlers, we need to be able to re-raise or re-return. # So this is full of detail stuff, pylint: disable=too-many-branches,too-many-locals,too-many-statements if generateTryNextExceptStopIterationCode(statement, emit, context): return # Get the statement sequences involved. All except the tried block can be # None. For the tried block it would be a missed optimization. Also not all # the handlers must be None, then it's also a missed optimization. tried_block = statement.getBlockTry() except_handler = statement.getBlockExceptHandler() continue_handler = statement.getBlockContinueHandler() break_handler = statement.getBlockBreakHandler() return_handler = statement.getBlockReturnHandler() tried_block_may_raise = tried_block.mayRaiseException(BaseException) assert tried_block_may_raise or \ continue_handler is not None or \ break_handler is not None or \ return_handler is not None, statement.asXmlText() # The tried statements might raise, for which we define an escape. tried_handler_escape = context.allocateLabel("try_except_handler") if tried_block_may_raise: old_exception_escape = context.setExceptionEscape(tried_handler_escape) # The tried statements might continue, for which we define an escape. continue_handler_escape = context.allocateLabel("try_continue_handler") if continue_handler is not None: old_continue_target = context.setLoopContinueTarget(continue_handler_escape) # The tried statements might break, for which we define an escape. break_handler_escape = context.allocateLabel("try_break_handler") if break_handler is not None: old_break_target = context.setLoopBreakTarget(break_handler_escape) # The tried statements might return, for which we define an escape. return_handler_escape = context.allocateLabel("try_return_handler") if return_handler is not None: old_return_target = context.setReturnTarget(return_handler_escape) # Now the tried block can be generated, cannot be "None" or else the # optimization failed. emit("// Tried code:") generateStatementSequenceCode( statement_sequence = tried_block, emit = emit, allow_none = False, context = context ) # Restore the old escape targets as preserved above, during the handlers, # the parent handlers should be back in effect. if tried_block_may_raise: context.setExceptionEscape(old_exception_escape) if continue_handler: context.setLoopContinueTarget(old_continue_target) if break_handler: context.setLoopBreakTarget(old_break_target) if return_handler: context.setReturnTarget(old_return_target) post_label = None if not tried_block.isStatementAborting(): if post_label is None: post_label = context.allocateLabel("try_end") getGotoCode(post_label, emit) else: getMustNotGetHereCode( reason = "tried codes exits in all cases", context = context, emit = emit ) if return_handler is not None: assert tried_block.mayReturn() emit("// Return handler code:") getLabelCode(return_handler_escape, emit) # During the return value, the value being returned is in a variable, # and therefore needs to be released before being updated. old_return_value_release = context.setReturnReleaseMode(True) generateStatementSequenceCode( statement_sequence = return_handler, emit = emit, allow_none = False, context = context ) context.setReturnReleaseMode(old_return_value_release) assert return_handler.isStatementAborting() if tried_block_may_raise: emit("// Exception handler code:") getLabelCode(tried_handler_escape, emit) # Need to preserve exception state. keeper_type, keeper_value, keeper_tb, keeper_lineno = \ context.allocateExceptionKeeperVariables() old_keepers = context.setExceptionKeeperVariables( (keeper_type, keeper_value, keeper_tb, keeper_lineno) ) assert keeper_type is not None # TODO: That normalization and chaining is only necessary if the # exception is published. emit( """\ %(keeper_type)s = exception_type; %(keeper_value)s = exception_value; %(keeper_tb)s = exception_tb; %(keeper_lineno)s = exception_lineno; exception_type = NULL; exception_value = NULL; exception_tb = NULL; exception_lineno = 0; """ % { "keeper_type" : keeper_type, "keeper_value" : keeper_value, "keeper_tb" : keeper_tb, "keeper_lineno" : keeper_lineno } ) generateStatementSequenceCode( statement_sequence = except_handler, emit = emit, allow_none = True, context = context ) if except_handler is None or not except_handler.isStatementAborting(): getExceptionUnpublishedReleaseCode(emit, context) if post_label is None: post_label = context.allocateLabel("try_end") getGotoCode(post_label, emit) getMustNotGetHereCode( reason = "exception handler codes exits in all cases", context = context, emit = emit ) context.setExceptionKeeperVariables(old_keepers) else: assert except_handler is None if break_handler is not None: assert tried_block.mayBreak() emit("// try break handler code:") getLabelCode(break_handler_escape, emit) generateStatementSequenceCode( statement_sequence = break_handler, emit = emit, allow_none = False, context = context ) assert break_handler.isStatementAborting() if continue_handler is not None: assert tried_block.mayContinue() emit("// try continue handler code:") getLabelCode(continue_handler_escape, emit) generateStatementSequenceCode( statement_sequence = continue_handler, emit = emit, allow_none = False, context = context ) assert continue_handler.isStatementAborting() emit("// End of try:") if post_label is not None: getLabelCode(post_label, emit) def generateTryNextExceptStopIterationCode(statement, emit, context): # This has many branches which mean this optimized code generation is not # applicable, we return each time. pylint: disable=too-many-branches,too-many-return-statements except_handler = statement.getBlockExceptHandler() if except_handler is None: return False if statement.getBlockBreakHandler() is not None: return False if statement.getBlockContinueHandler() is not None: return False if statement.getBlockReturnHandler() is not None: return False tried_statements = statement.getBlockTry().getStatements() if len(tried_statements) != 1: return False handling_statements = except_handler.getStatements() if len(handling_statements) != 1: return False tried_statement = tried_statements[0] if not tried_statement.isStatementAssignmentVariable(): return False assign_source = tried_statement.getAssignSource() if not assign_source.isExpressionBuiltinNext1(): return False handling_statement = handling_statements[0] if not handling_statement.isStatementConditional(): return False yes_statements = handling_statement.getBranchYes().getStatements() no_statements = handling_statement.getBranchNo().getStatements() if len(yes_statements) != 1: return False if not yes_statements[0].isStatementLoopBreak(): return False if len(no_statements) != 1: return False if not no_statements[0].isStatementReraiseException() or \ not no_statements[0].isStatementReraiseException(): return False tmp_name = context.allocateTempName("next_source") generateExpressionCode( expression = assign_source.getValue(), to_name = tmp_name, emit = emit, context = context ) tmp_name2 = context.allocateTempName("assign_source") old_source_ref = context.setCurrentSourceCodeReference( assign_source.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getBuiltinLoopBreakNextCode( to_name = tmp_name2, value = tmp_name, emit = emit, context = context ) getVariableAssignmentCode( tmp_name = tmp_name2, variable = tried_statement.getVariable(), version = tried_statement.getVariableVersion(), needs_release = None, in_place = False, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) if context.needsCleanup(tmp_name2): context.removeCleanupTempName(tmp_name2) return True Nuitka-0.5.28.2/nuitka/codegen/LabelCodes.py0000644000372000001440000000430413112214770020730 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ C labels, small helpers. Much things are handled with "goto" statements in the generated code, error exits, finally blocks, etc. this provides just the means to emit a label or the goto statement itself. """ from nuitka.utils.CStrings import encodePythonStringToC def getGotoCode(label, emit): assert label is not None emit( "goto %s;" % label ) def getLabelCode(label, emit): assert label is not None emit( "%s:;" % label ) def getBranchingCode(condition, emit, context): true_target = context.getTrueBranchTarget() false_target = context.getFalseBranchTarget() if true_target is not None and false_target is None: emit( "if ( %s ) goto %s;" % ( condition, true_target ) ) elif true_target is None and false_target is not None: emit( "if (!( %s )) goto %s;" % ( condition, false_target ) ) else: assert true_target is not None and false_target is not None emit( """\ if ( %s ) { goto %s; } else { goto %s; }""" % ( condition, true_target, false_target ) ) def getStatementTrace(source_desc, statement_repr): return 'NUITKA_PRINT_TRACE( "Execute: " %s );' % ( encodePythonStringToC(source_desc + b" " + statement_repr), ) Nuitka-0.5.28.2/nuitka/codegen/Indentation.py0000644000372000001440000000260213112214770021206 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Indentation of code. Language independent, the amount of the spaces is not configurable, as it needs to be the same as in templates. """ def _indentedCode(codes, count): return '\n'.join( ' ' * count + line if (line and not line.startswith('#')) else line for line in codes ) def indented(codes, level = 1, vert_block = False): if type(codes) is str: codes = codes.split('\n') if vert_block and codes != [""]: codes.insert(0, "") codes.append("") return _indentedCode(codes, level * 4) def getCommentCode(comment, emit): emit("// " + comment) Nuitka-0.5.28.2/nuitka/codegen/TupleCodes.py0000644000372000001440000000604013207537242021010 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tuple codes """ from .CodeHelpers import generateExpressionCode from .ConstantCodes import getConstantAccess from .PythonAPICodes import generateCAPIObjectCode def _areConstants(expressions): for expression in expressions: if not expression.isExpressionConstantRef(): return False if expression.isMutable(): return False return True def generateTupleCreationCode(to_name, expression, emit, context): return getTupleCreationCode( to_name = to_name, elements = expression.getElements(), emit = emit, context = context ) def getTupleCreationCode(to_name, elements, emit, context): if _areConstants(elements): getConstantAccess( to_name = to_name, constant = tuple( element.getConstant() for element in elements ), emit = emit, context = context ) else: emit( "%s = PyTuple_New( %d );" % ( to_name, len(elements) ) ) context.addCleanupTempName(to_name) element_name = context.allocateTempName("tuple_element") for count, element in enumerate(elements): generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) if not context.needsCleanup(element_name): emit("Py_INCREF( %s );" % element_name) else: context.removeCleanupTempName(element_name) emit( "PyTuple_SET_ITEM( %s, %d, %s );" % ( to_name, count, element_name ) ) def generateBuiltinTupleCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PySequence_Tuple", arg_desc = ( ("tuple_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/Namify.py0000644000372000001440000001614013134660221020157 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Namify constants. This determines the identifier names of constants in the generated code. We try to have readable names where possible, and resort to hash codes only when it is really necessary. """ import hashlib import math import re from logging import warning from nuitka.__past__ import ( # pylint: disable=I0021,redefined-builtin long, unicode, xrange ) class ExceptionCannotNamify(Exception): pass def namifyConstant(constant): # Many branches, statements and every case has a return, this is a huge case # statement, that encodes the naming policy of constants, with often complex # conditions, pylint: disable=too-many-branches,too-many-return-statements,too-many-statements if type(constant) is int: if constant == 0: result = "int_0" elif constant > 0: result = "int_pos_%d" % constant else: result = "int_neg_%d" % abs(constant) if len(result) > 32: result = _digest(result) return result elif type(constant) is long: if constant == 0: result = "long_0" elif constant > 0: result = "long_pos_%d" % constant else: result = "long_neg_%d" % abs(constant) if len(result) > 32: result = _digest(result) return result elif constant is None: return "none" elif constant is True: return "true" elif constant is False: return "false" elif constant is Ellipsis: return "ellipsis" elif type(constant) is str: return "str_" + _namifyString(constant) elif type(constant) is bytes: return "bytes_" + _namifyString(constant) elif type(constant) is unicode: if _isAscii(constant): return "unicode_" + _namifyString(str(constant)) else: # Others are better digested to not cause compiler trouble return "unicode_digest_" + _digest(repr(constant)) elif type(constant) is float: if math.isnan(constant): return "float_%s_nan" % ( "minus" if math.copysign(1, constant) < 0 else "plus" ) return "float_%s" % repr(constant).replace('.', '_').\ replace('-', "_minus_").replace('+', "") elif type(constant) is complex: value = "%s__%s" % (constant.real, constant.imag) value = value.replace('+', 'p').replace('-', 'm').\ replace('.', '_') if value.startswith('(') and value.endswith(')'): value = value[1:-1] return "complex_%s" % value elif type(constant) is dict: if constant == {}: return "dict_empty" else: return "dict_" + _digest(repr(constant)) elif type(constant) is set: if constant == set(): return "set_empty" else: return "set_" + _digest(repr(constant)) elif type(constant) is frozenset: if constant == frozenset(): return "frozenset_empty" else: return "frozenset_" + _digest(repr(constant)) elif type(constant) is tuple: if constant == (): return "tuple_empty" else: try: result = '_'.join( namifyConstant(value) for value in constant ) if len(result) > 60: result = _digest(repr(constant)) return "tuple_" + result + "_tuple" except ExceptionCannotNamify: warning("Couldn't namify '%r'" % (constant,)) return "tuple_" + _digest(repr(constant)) elif type(constant) is list: if constant == []: return "list_empty" else: try: result = '_'.join( namifyConstant(value) for value in constant ) if len(result) > 60: result = _digest(repr(constant)) return "list_" + result + "_list" except ExceptionCannotNamify: warning("Couldn't namify '%r'" % value) return "list_" + _digest(repr(constant)) elif type(constant) is bytearray: return "bytearray_" + _digest(repr(constant)) elif type(constant) is xrange: return "xrange_%s" % ( str(constant)[7 if str is bytes else 6:-1].replace(' ', "").replace(',', '_').replace('-', "neg") ) elif type(constant) is slice: return "slice_%s_%s_%s" % ( namifyConstant(constant.start), namifyConstant(constant.stop), namifyConstant(constant.step) ) elif type(constant) is type: return "type_%s" % constant.__name__ raise ExceptionCannotNamify("%r" % constant, type(constant)) _re_str_needs_no_digest = re.compile(r"^([a-z]|[A-Z]|[0-9]|_){1,40}$", re.S) def _namifyString(string): # Many branches case has a return, encodes the naming policy of strings # constants, with often complex decisions to make, pylint: disable=too-many-return-statements if string in ("", b""): return "empty" elif string == ' ': return "space" elif string == '.': return "dot" elif string == '\n': return "newline" elif type(string) is str and \ _re_str_needs_no_digest.match(string) and \ '\n' not in string: # Some strings can be left intact for source code readability. return "plain_" + string elif len(string) == 1: return "chr_%d" % ord(string) elif len(string) > 2 and string[0] == '<' and string[-1] == '>' and \ _re_str_needs_no_digest.match(string[1:-1]) and \ '\n' not in string: return "angle_" + string[1:-1] else: # Others are better digested to not cause compiler trouble return "digest_" + _digest(repr(string)) def _isAscii(string): try: _unused = str(string) return True except UnicodeEncodeError: return False def _digest(value): if str is bytes: # Python2 is simple return hashlib.md5(value).hexdigest() else: # Python3 needs to encode the string is it is one. if type(value) is bytes: return hashlib.md5(value).hexdigest() else: return hashlib.md5(value.encode("utf-8")).hexdigest() Nuitka-0.5.28.2/nuitka/codegen/SetCodes.py0000644000372000001440000001550613207537242020461 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for sets. Right now only the creation, and set add code is done here. But more should be added later on. """ from nuitka.PythonVersions import needsSetLiteralReverseInsertion from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import getErrorExitBoolCode, getReleaseCodes from .PythonAPICodes import generateCAPIObjectCode def generateSetCreationCode(to_name, expression, emit, context): emit( "%s = PySet_New( NULL );" % ( to_name, ) ) context.addCleanupTempName(to_name) element_name = context.allocateTempName("set_element") for element in expression.getElements(): generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) if element.isKnownToBeHashable(): emit( "PySet_Add( %s, %s );" % ( to_name, element_name ) ) else: res_name = context.getIntResName() emit( "%s = PySet_Add( %s, %s );" % ( res_name, to_name, element_name ) ) getErrorExitBoolCode( condition = "%s != 0" % res_name, emit = emit, context = context ) if context.needsCleanup(element_name): emit("Py_DECREF( %s );" % element_name) context.removeCleanupTempName(element_name) def generateSetLiteralCreationCode(to_name, expression, emit, context): if not needsSetLiteralReverseInsertion(): return generateSetCreationCode(to_name, expression, emit, context) emit( "%s = PySet_New( NULL );" % ( to_name, ) ) context.addCleanupTempName(to_name) elements = expression.getElements() element_names = [] for count, element in enumerate(elements): element_name = context.allocateTempName( "set_element_%d" % (count+1) ) element_names.append(element_name) generateExpressionCode( to_name = element_name, expression = element, emit = emit, context = context ) for count, element in enumerate(elements): element_name = element_names[len(elements)-count-1] if element.isKnownToBeHashable(): emit( "PySet_Add( %s, %s );" % ( to_name, element_name ) ) else: res_name = context.getIntResName() emit( "%s = PySet_Add( %s, %s );" % ( res_name, to_name, element_name ) ) getErrorExitBoolCode( condition = "%s != 0" % res_name, emit = emit, context = context ) if context.needsCleanup(element_name): emit("Py_DECREF( %s );" % element_name) context.removeCleanupTempName(element_name) def generateSetOperationAddCode(statement, emit, context): set_arg_name = context.allocateTempName("append_list") generateExpressionCode( to_name = set_arg_name, expression = statement.getSet(), emit = emit, context = context ) value_arg_name = context.allocateTempName("append_value") generateExpressionCode( to_name = value_arg_name, expression = statement.getValue(), emit = emit, context = context ) context.setCurrentSourceCodeReference(statement.getSourceReference()) res_name = context.getIntResName() emit("assert( PySet_Check( %s ) );" % set_arg_name) emit( "%s = PySet_Add( %s, %s );" % ( res_name, set_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (set_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) def generateSetOperationUpdateCode(to_name, expression, emit, context): res_name = context.getIntResName() set_arg_name, value_arg_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit("assert( PySet_Check( %s ) );" % set_arg_name) emit( "%s = _PySet_Update( %s, %s );" % ( res_name, set_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (set_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) # Only assign if necessary. if context.isUsed(to_name): emit( "%s = Py_None;" % to_name ) else: context.forgetTempName(to_name) def generateBuiltinSetCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PySet_New", arg_desc = ( ("set_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinFrozensetCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PyFrozenSet_New", arg_desc = ( ("frozenset_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/CoroutineCodes.py0000644000372000001440000001571613207537242021700 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled coroutine objects. """ from .CodeHelpers import ( generateChildExpressionsCode, generateStatementSequenceCode ) from .Emission import SourceCodeCollector from .ErrorCodes import getErrorExitCode, getReleaseCode from .FunctionCodes import ( finalizeFunctionLocalVariables, setupFunctionLocalVariables ) from .GeneratorCodes import getClosureCopyCode from .Indentation import indented from .LineNumberCodes import emitLineNumberUpdateCode from .PythonAPICodes import getReferenceExportCode from .templates.CodeTemplatesCoroutines import ( template_coroutine_exception_exit, template_coroutine_noexception_exit, template_coroutine_object_body_template, template_coroutine_object_decl_template, template_coroutine_return_exit, template_make_coroutine_template ) def getCoroutineObjectDeclCode(function_identifier): return template_coroutine_object_decl_template % { "function_identifier" : function_identifier, } def getCoroutineObjectCode(context, function_identifier, closure_variables, user_variables, outline_variables, temp_variables, needs_exception_exit, needs_generator_return): function_locals, function_cleanup = setupFunctionLocalVariables( context = context, parameters = None, closure_variables = closure_variables, user_variables = user_variables + outline_variables, temp_variables = temp_variables ) function_codes = SourceCodeCollector() generateStatementSequenceCode( statement_sequence = context.getOwner().getBody(), allow_none = True, emit = function_codes, context = context ) finalizeFunctionLocalVariables(context, function_locals, function_cleanup) if needs_exception_exit: generator_exit = template_coroutine_exception_exit % { "function_identifier" : function_identifier, "function_cleanup" : indented(function_cleanup) } else: generator_exit = template_coroutine_noexception_exit % { "function_identifier" : function_identifier, "function_cleanup" : indented(function_cleanup) } if needs_generator_return: generator_exit += template_coroutine_return_exit % {} return template_coroutine_object_body_template % { "function_identifier" : function_identifier, "function_body" : indented(function_codes.codes), "function_var_inits" : indented(function_locals), "coroutine_exit" : generator_exit } def generateMakeCoroutineObjectCode(to_name, expression, emit, context): coroutine_object_body = expression.getCoroutineRef().getFunctionBody() closure_variables = expression.getClosureVariableVersions() closure_copy = getClosureCopyCode( to_name = to_name, closure_type = "struct Nuitka_CoroutineObject *", closure_variables = closure_variables, context = context ) emit( template_make_coroutine_template % { "closure_copy" : indented(closure_copy, 0, True), "coroutine_identifier" : coroutine_object_body.getCodeName(), "to_name" : to_name, "code_identifier" : context.getCodeObjectHandle( code_object = expression.getCodeObject(), ), "closure_count" : len(closure_variables) } ) context.addCleanupTempName(to_name) def generateAsyncWaitCode(to_name, expression, emit, context): emitLineNumberUpdateCode(emit, context) value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) # In handlers, we must preserve/restore the exception. preserve_exception = expression.isExceptionPreserving() context_identifier = context.getContextObjectName() # This produces AWAIT_COROUTINE or AWAIT_ASYNCGEN calls. getReferenceExportCode(value_name, emit, context) emit( "%s = %s_%s( %s, %s );" % ( to_name, context_identifier.upper(), "AWAIT" if not preserve_exception else "AWAIT_IN_HANDLER", context_identifier, value_name ) ) if not context.needsCleanup(value_name): context.addCleanupTempName(value_name) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateAsyncIterCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = %s_ASYNC_MAKE_ITERATOR( %s, %s );" % ( to_name, context.getContextObjectName().upper(), context.getContextObjectName(), value_name ) ) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateAsyncNextCode(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = %s_ASYNC_ITERATOR_NEXT( %s, %s );" % ( to_name, context.getContextObjectName().upper(), context.getContextObjectName(), value_name, ) ) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, quick_exception = "StopAsyncIteration", emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/SubscriptCodes.py0000644000372000001440000001671213207537242021704 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Subscript related code generation. There is special handling for integer indexes, which can be dealt with much faster than general subscript lookups. """ from nuitka import Options from nuitka.Constants import isIndexConstant from .CodeHelpers import ( generateChildExpressionsCode, generateExpressionCode, generateExpressionsCode ) from .ErrorCodes import getErrorExitBoolCode, getErrorExitCode, getReleaseCodes def generateAssignmentSubscriptCode(statement, emit, context): subscribed = statement.getSubscribed() subscript = statement.getSubscript() value = statement.getAssignSource() integer_subscript = False if subscript.isExpressionConstantRef(): constant = subscript.getConstant() if isIndexConstant(constant): constant_value = int(constant) if abs(constant_value) < 2**31: integer_subscript = True value_name = context.allocateTempName("ass_subvalue") generateExpressionCode( to_name = value_name, expression = value, emit = emit, context = context ) subscribed_name = context.allocateTempName("ass_subscribed") generateExpressionCode( to_name = subscribed_name, expression = subscribed, emit = emit, context = context ) subscript_name = context.allocateTempName("ass_subscript") generateExpressionCode( to_name = subscript_name, expression = subscript, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) if integer_subscript: getIntegerSubscriptAssignmentCode( subscribed_name = subscribed_name, subscript_name = subscript_name, subscript_value = constant_value, value_name = value_name, emit = emit, context = context ) else: getSubscriptAssignmentCode( target_name = subscribed_name, subscript_name = subscript_name, value_name = value_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateDelSubscriptCode(statement, emit, context): subscribed = statement.getSubscribed() subscript = statement.getSubscript() target_name, subscript_name = generateExpressionsCode( expressions = (subscribed, subscript), names = ("delsubscr_target", "delsubscr_subscript"), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( subscript.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSubscriptDelCode( target_name = target_name, subscript_name = subscript_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateSubscriptLookupCode(to_name, expression, emit, context): subscribed_name, subscript_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) return getSubscriptLookupCode( to_name = to_name, subscribed_name = subscribed_name, subscript_name = subscript_name, emit = emit, context = context ) def getIntegerSubscriptLookupCode(to_name, target_name, subscript_name, subscript_value, emit, context): emit( "%s = LOOKUP_SUBSCRIPT_CONST( %s, %s, %s );" % ( to_name, target_name, subscript_name, subscript_value ) ) getReleaseCodes( release_names = (target_name, subscript_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getSubscriptLookupCode(to_name, subscript_name, subscribed_name, emit, context): emit( "%s = LOOKUP_SUBSCRIPT( %s, %s );" % ( to_name, subscribed_name, subscript_name, ) ) getReleaseCodes( release_names = (subscribed_name, subscript_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getIntegerSubscriptAssignmentCode(subscribed_name, subscript_name, subscript_value, value_name, emit, context): assert abs(subscript_value) < 2**31 res_name = context.allocateTempName("ass_subscript_res", "int") emit( "%s = SET_SUBSCRIPT_CONST( %s, %s, %s, %s );" % ( res_name, subscribed_name, subscript_name, subscript_value, value_name, ) ) getReleaseCodes( release_names = (subscribed_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSubscriptAssignmentCode(target_name, subscript_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_SUBSCRIPT( %s, %s, %s );" % ( res_name, target_name, subscript_name, value_name, ) ) getReleaseCodes( release_names = (target_name, subscript_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSubscriptDelCode(target_name, subscript_name, emit, context): res_name = context.getBoolResName() emit( "%s = DEL_SUBSCRIPT( %s, %s );" % ( res_name, target_name, subscript_name, ) ) getReleaseCodes( release_names = (target_name, subscript_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/ComparisonCodes.py0000644000372000001440000002131113207537242022027 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Comparison related codes. Rich comparisons, "in", and "not in", also "is", and "is not", and the "isinstance" check as used in conditions, as well as exception matching. """ from . import OperatorCodes from .CodeHelpers import generateExpressionCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .LabelCodes import getBranchingCode def generateComparisonExpressionCode(to_name, expression, emit, context): left_name = context.allocateTempName("compexpr_left") right_name = context.allocateTempName("compexpr_right") generateExpressionCode( to_name = left_name, expression = expression.getLeft(), emit = emit, context = context ) generateExpressionCode( to_name = right_name, expression = expression.getRight(), emit = emit, context = context ) comparator = expression.getComparator() if comparator in OperatorCodes.normal_comparison_codes: needs_check = expression.getRight().mayRaiseExceptionIn( BaseException, expression.getLeft() ) helper = OperatorCodes.normal_comparison_codes[ comparator ] assert helper.startswith("SEQUENCE_CONTAINS") emit( "%s = %s( %s, %s );" % ( to_name, helper, left_name, right_name ) ) getReleaseCode( release_name = left_name, emit = emit, context = context ) getReleaseCode( release_name = right_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) elif comparator in OperatorCodes.rich_comparison_codes: needs_check = expression.mayRaiseExceptionBool(BaseException) helper = "RICH_COMPARE_%s" % ( OperatorCodes.rich_comparison_codes[ comparator ] ) if not context.mayRecurse() and comparator == "Eq": helper += "_NORECURSE" emit( "%s = %s( %s, %s );" % ( to_name, helper, left_name, right_name ) ) getReleaseCodes( release_names = (left_name, right_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) elif comparator == "Is": emit( "%s = BOOL_FROM( %s == %s );" % ( to_name, left_name, right_name ) ) getReleaseCodes( release_names = (left_name, right_name), emit = emit, context = context ) elif comparator == "IsNot": emit( "%s = BOOL_FROM( %s != %s );" % ( to_name, left_name, right_name ) ) getReleaseCodes( release_names = (left_name, right_name), emit = emit, context = context ) elif comparator == "exception_match": needs_check = expression.mayRaiseExceptionBool(BaseException) operator_res_name = context.allocateTempName( "cmp_exception_match", "int" ) emit( "%s = EXCEPTION_MATCH_BOOL( %s, %s );" % ( operator_res_name, left_name, right_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, needs_check = needs_check, emit = emit, context = context ) emit( "%s = BOOL_FROM( %s != 0 );" % ( to_name, operator_res_name ) ) else: assert False, comparator def getComparisonExpressionBoolCode(comparator, left_name, right_name, needs_check, emit, context): if comparator in OperatorCodes.normal_comparison_codes: operator_res_name = context.allocateTempName("cmp_" + comparator, "int") emit( "%s = PySequence_Contains( %s, %s );" % ( operator_res_name, right_name, # sequence goes first. left_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, emit = emit, needs_check = needs_check, context = context ) condition = "%s == %d" % ( operator_res_name, 1 if comparator == "In" else 0 ) elif comparator in OperatorCodes.rich_comparison_codes: operator_res_name = context.allocateTempName("cmp_" + comparator, "int") helper = OperatorCodes.rich_comparison_codes[comparator] if not context.mayRecurse() and comparator == "Eq": helper += "_NORECURSE" emit( "%s = RICH_COMPARE_BOOL_%s( %s, %s );" % ( operator_res_name, helper, left_name, right_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, needs_check = needs_check, emit = emit, context = context ) condition = "%s == 1" % ( operator_res_name, ) elif comparator == "Is": operator_res_name = context.allocateTempName("is", "bool") emit( "%s = ( %s == %s );" % ( operator_res_name, left_name, right_name ) ) condition = operator_res_name elif comparator == "IsNot": operator_res_name = context.allocateTempName("isnot", "bool") emit( "%s = ( %s != %s );" % ( operator_res_name, left_name, right_name ) ) condition = operator_res_name elif comparator == "exception_match": operator_res_name = context.allocateTempName("exc_match_" + comparator, "int") emit( "%s = EXCEPTION_MATCH_BOOL( %s, %s );" % ( operator_res_name, left_name, right_name ) ) getErrorExitBoolCode( condition = "%s == -1" % operator_res_name, needs_check = needs_check, emit = emit, context = context ) condition = "%s == 1" % ( operator_res_name ) else: assert False, comparator getReleaseCode( release_name = left_name, emit = emit, context = context ) getReleaseCode( release_name = right_name, emit = emit, context = context ) getBranchingCode(condition, emit, context) def getBuiltinIsinstanceBoolCode(inst_name, cls_name, emit, context): res_name = context.getIntResName() emit( "%s = Nuitka_IsInstance( %s, %s );" % ( res_name, inst_name, cls_name ) ) getReleaseCodes( release_names = (inst_name, cls_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, emit = emit, context = context ) getBranchingCode( condition = "%s == 1" % res_name, emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/GlobalsLocalsCodes.py0000644000372000001440000001767513207537242022460 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for locals and globals handling. This also includes writing back to locals for exec statements. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateExpressionCode from .ErrorCodes import getErrorExitBoolCode from .ModuleCodes import getModuleAccessCode from .PythonAPICodes import generateCAPIObjectCode, getReferenceExportCode from .templates.CodeTemplatesVariables import ( template_set_locals_dict_value, template_set_locals_mapping_value, template_update_locals_dict_value, template_update_locals_mapping_value ) from .VariableCodes import getLocalVariableCodeType def generateBuiltinLocalsCode(to_name, expression, emit, context): provider = expression.getParentVariableProvider() getLoadLocalsCode( to_name = to_name, variables = expression.getVariableVersions(), provider = provider, updated = expression.isExpressionBuiltinLocalsUpdated(), emit = emit, context = context ) def generateBuiltinGlobalsCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=unused-argument getLoadGlobalsCode( to_name = to_name, emit = emit, context = context ) def getLoadGlobalsCode(to_name, emit, context): assert type(to_name) is str emit( "%(to_name)s = (PyObject *)moduledict_%(module_identifier)s;" % { "to_name" : to_name, "module_identifier" : context.getModuleCodeName() }, ) def _getLocalVariableList(provider): if provider.isExpressionFunctionBody(): include_closure = not provider.isUnoptimized() elif provider.isExpressionClassBody(): include_closure = False else: include_closure = True return [ variable for variable in provider.getVariables() if not variable.isModuleVariable() if (include_closure or variable.getOwner() is provider) ] def _getVariableDictUpdateCode(target_name, variable, version, initial, is_dict, emit, context): # TODO: Variable could known to be set here, get a hand at that # information. variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, version) test_code = variable_c_type.getLocalVariableInitTestCode(variable_code_name) access_code = variable_c_type.getLocalVariableObjectAccessCode(variable_code_name) if is_dict: if initial: template = template_set_locals_dict_value else: template = template_update_locals_dict_value emit( template % { "dict_name" : target_name, "var_name" : context.getConstantCode( constant = variable.getName() ), "test_code" : test_code, "access_code" : access_code } ) else: if initial: template = template_set_locals_mapping_value else: template = template_update_locals_mapping_value res_name = context.getBoolResName() emit( template % { "mapping_name" : target_name, "var_name" : context.getConstantCode( constant = variable.getName() ), "test_code" : test_code, "access_code" : access_code, "tmp_name" : res_name } ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getLoadLocalsCode(to_name, variables, provider, updated, emit, context): # Locals is sorted of course. def _sorted(variables): locals_owner = context.getOwner() if locals_owner.isExpressionOutlineBody(): locals_owner = locals_owner.getParentVariableProvider() all_variables = tuple( locals_owner.getVariables() ) return sorted( variables, key = lambda variable_desc: all_variables.index(variable_desc[0]), ) # Optimization will have made this "globals", and it wouldn't be # about local variables at all. assert not provider.isCompiledPythonModule(), provider if not context.hasLocalsDict(): assert not updated # Need to create the dictionary first. emit( "%s = PyDict_New();" % ( to_name, ) ) context.addCleanupTempName(to_name) is_dict = True initial = True elif updated: emit( """\ %s = %s; Py_INCREF( %s );""" % ( to_name, context.getLocalsDictName(), to_name ) ) context.addCleanupTempName(to_name) # For Python3 it may not really be a dictionary. is_dict = python_version < 300 or \ not context.getOwner().isExpressionClassBody() initial = False else: emit( "%s = PyDict_Copy( %s );" % ( to_name, context.getLocalsDictName(), ) ) context.addCleanupTempName(to_name) is_dict = True initial = False for local_var, version in _sorted(variables): _getVariableDictUpdateCode( target_name = to_name, variable = local_var, version = version, is_dict = is_dict, initial = initial, emit = emit, context = context ) def generateSetLocalsCode(statement, emit, context): new_locals_name = context.allocateTempName("set_locals", unique = True) generateExpressionCode( to_name = new_locals_name, expression = statement.getNewLocals(), emit = emit, context = context ) emit( """\ Py_DECREF(%(locals_dict)s); %(locals_dict)s = %(locals_value)s;""" % { "locals_dict" : context.getLocalsDictName(), "locals_value" : new_locals_name } ) getReferenceExportCode(new_locals_name, emit, context) if context.needsCleanup(new_locals_name): context.removeCleanupTempName(new_locals_name) def generateBuiltinDir1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PyObject_Dir", arg_desc = ( ("dir_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinVarsCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "LOOKUP_VARS", arg_desc = ( ("vars_arg", expression.getSource()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/AsyncgenCodes.py0000644000372000001440000001016713207537242021473 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled asyncgen objects. """ from .CodeHelpers import generateStatementSequenceCode from .Emission import SourceCodeCollector from .FunctionCodes import ( finalizeFunctionLocalVariables, setupFunctionLocalVariables ) from .GeneratorCodes import getClosureCopyCode from .Indentation import indented from .templates.CodeTemplatesAsyncgens import ( template_asyncgen_exception_exit, template_asyncgen_noexception_exit, template_asyncgen_object_body_template, template_asyncgen_object_decl_template, template_asyncgen_return_exit, template_make_asyncgen_template ) def getAsyncgenObjectDeclCode(function_identifier): return template_asyncgen_object_decl_template % { "function_identifier" : function_identifier, } def getAsyncgenObjectCode(context, function_identifier, closure_variables, user_variables, outline_variables, temp_variables, needs_exception_exit, needs_generator_return): function_locals, function_cleanup = setupFunctionLocalVariables( context = context, parameters = None, closure_variables = closure_variables, user_variables = user_variables + outline_variables, temp_variables = temp_variables ) function_codes = SourceCodeCollector() generateStatementSequenceCode( statement_sequence = context.getOwner().getBody(), allow_none = True, emit = function_codes, context = context ) finalizeFunctionLocalVariables(context, function_locals, function_cleanup) if needs_exception_exit: generator_exit = template_asyncgen_exception_exit % { "function_identifier" : function_identifier, "function_cleanup" : indented(function_cleanup) } else: generator_exit = template_asyncgen_noexception_exit % { "function_identifier" : function_identifier, "function_cleanup" : indented(function_cleanup) } if needs_generator_return: generator_exit += template_asyncgen_return_exit % {} return template_asyncgen_object_body_template % { "function_identifier" : function_identifier, "function_body" : indented(function_codes.codes), "function_var_inits" : indented(function_locals), "asyncgen_exit" : generator_exit } def generateMakeAsyncgenObjectCode(to_name, expression, emit, context): asyncgen_object_body = expression.getAsyncgenRef().getFunctionBody() closure_variables = expression.getClosureVariableVersions() closure_copy = getClosureCopyCode( to_name = to_name, closure_type = "struct Nuitka_AsyncgenObject *", closure_variables = closure_variables, context = context ) emit( template_make_asyncgen_template % { "closure_copy" : indented(closure_copy, 0, True), "asyncgen_identifier" : asyncgen_object_body.getCodeName(), "to_name" : to_name, "code_identifier" : context.getCodeObjectHandle( code_object = expression.getCodeObject(), ), "closure_count" : len(closure_variables) } ) context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/IdCodes.py0000644000372000001440000000331313112214770020244 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Codes for id and hash """ from .PythonAPICodes import generateCAPIObjectCode def generateBuiltinIdCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "PyLong_FromVoidPtr", arg_desc = ( ("id_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinHashCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_HASH", arg_desc = ( ("hash_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/ExpressionCodes.py0000644000372000001440000000425713207537242022066 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Expression codes, side effects, or statements that are an unused expression. When you write "f()", i.e. you don't use the return value, that is an expression only statement. """ from .CodeHelpers import generateExpressionCode from .ErrorCodes import getReleaseCode def generateExpressionOnlyCode(statement, emit, context): return getStatementOnlyCode( value = statement.getExpression(), emit = emit, context = context ) def getStatementOnlyCode(value, emit, context): tmp_name = context.allocateTempName( base_name = "unused", type_name = "NUITKA_MAY_BE_UNUSED PyObject *", unique = True ) # An error of the expression is dealt inside of this, not necessary here. generateExpressionCode( expression = value, to_name = tmp_name, emit = emit, context = context ) getReleaseCode( release_name = tmp_name, emit = emit, context = context ) def generateSideEffectsCode(to_name, expression, emit, context): for side_effect in expression.getSideEffects(): getStatementOnlyCode( value = side_effect, emit = emit, context = context ) generateExpressionCode( to_name = to_name, expression = expression.getExpression(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/ConditionalCodes.py0000644000372000001440000003233113207537242022164 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Conditional statements related codes. Branches, conditions, truth checks. """ from nuitka import Options from .AttributeCodes import getAttributeCheckBoolCode from .CodeHelpers import generateExpressionCode from .ComparisonCodes import ( getBuiltinIsinstanceBoolCode, getComparisonExpressionBoolCode ) from .Emission import SourceCodeCollector from .ErrorCodes import getErrorExitBoolCode, getReleaseCode from .LabelCodes import getBranchingCode, getGotoCode, getLabelCode def generateConditionCode(condition, emit, context): # The complexity is needed to avoid unnecessary complex generated C # pylint: disable=too-many-locals,too-many-statements if condition.isExpressionComparison(): left_name = context.allocateTempName("compare_left") generateExpressionCode( to_name = left_name, expression = condition.getLeft(), emit = emit, context = context ) right_name = context.allocateTempName("compare_right") generateExpressionCode( to_name = right_name, expression = condition.getRight(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(condition.getSourceReference()) getComparisonExpressionBoolCode( comparator = condition.getComparator(), left_name = left_name, right_name = right_name, needs_check = condition.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif condition.isExpressionOperationNOT(): # Lets just switch the targets temporarily to get at "NOT" without # any effort really. true_target = context.getTrueBranchTarget() false_target = context.getFalseBranchTarget() context.setTrueBranchTarget(false_target) context.setFalseBranchTarget(true_target) generateConditionCode( condition = condition.getOperand(), emit = emit, context = context ) context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) elif condition.isExpressionConditional(): expression_yes = condition.getExpressionYes() expression_no = condition.getExpressionNo() condition = condition.getCondition() old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() select_true = context.allocateLabel("select_true") select_false = context.allocateLabel("select_false") # TODO: Could be avoided in some cases. select_end = context.allocateLabel("select_end") context.setTrueBranchTarget(select_true) context.setFalseBranchTarget(select_false) generateConditionCode( condition = condition, emit = emit, context = context, ) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) getLabelCode(select_true,emit) generateConditionCode( condition = expression_yes, emit = emit, context = context, ) getGotoCode(select_end, emit) getLabelCode(select_false,emit) generateConditionCode( condition = expression_no, emit = emit, context = context, ) getLabelCode(select_end,emit) elif condition.isExpressionBuiltinHasattr(): source_name = context.allocateTempName("hasattr_source") attr_name = context.allocateTempName("hasattr_attr") generateExpressionCode( to_name = source_name, expression = condition.getLookupSource(), emit = emit, context = context ) generateExpressionCode( to_name = attr_name, expression = condition.getAttribute(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( condition.getAttribute().getSourceReference() if Options.isFullCompat() else condition.getSourceReference() ) getAttributeCheckBoolCode( source_name = source_name, attr_name = attr_name, needs_check = condition.getLookupSource().mayRaiseExceptionAttributeCheckObject( exception_type = BaseException, attribute = condition.getAttribute() ), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif condition.isExpressionBuiltinIsinstance(): inst_name = context.allocateTempName("isinstance_inst") cls_name = context.allocateTempName("isinstance_cls") generateExpressionCode( to_name = inst_name, expression = condition.getInstance(), emit = emit, context = context ) generateExpressionCode( to_name = cls_name, expression = condition.getCls(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(condition.getSourceReference()) getBuiltinIsinstanceBoolCode( inst_name = inst_name, cls_name = cls_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) elif condition.isCompileTimeConstant(): getBranchingCode( condition = '1' if condition.getCompileTimeConstant() else '0', emit = emit, context = context ) else: condition_name = context.allocateTempName("cond_value") truth_name = context.allocateTempName("cond_truth", "int") generateExpressionCode( to_name = condition_name, expression = condition, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(condition.getSourceReference()) getConditionCheckTrueCode( to_name = truth_name, value_name = condition_name, needs_check = condition.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) getReleaseCode( release_name = condition_name, emit = emit, context = context ) getBranchingCode( condition = "%s == 1" % truth_name, emit = emit, context = context ) def getConditionCheckTrueCode(to_name, value_name, needs_check, emit, context): emit( "%s = CHECK_IF_TRUE( %s );" % ( to_name, value_name ) ) getErrorExitBoolCode( condition = "%s == -1" % to_name, needs_check = needs_check, emit = emit, context = context ) def generateConditionalAndOrCode(to_name, expression, emit, context): # This is a complex beast, handling both "or" and "and" expressions, # and it needs to micro manage details. # pylint: disable=too-many-locals if expression.isExpressionConditionalOR(): prefix = "or_" else: prefix = "and_" true_target = context.allocateLabel(prefix + "left") false_target = context.allocateLabel(prefix + "right") end_target = context.allocateLabel(prefix + "end") old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() truth_name = context.allocateTempName(prefix + "left_truth", "int") left_name = context.allocateTempName(prefix + "left_value") right_name = context.allocateTempName(prefix + "right_value") left_value = expression.getLeft() generateExpressionCode( to_name = left_name, expression = left_value, emit = emit, context = context ) # We need to treat this mostly manually here. We remember to release # this, and we better do this manually later. needs_ref1 = context.needsCleanup(left_name) getConditionCheckTrueCode( to_name = truth_name, value_name = left_name, needs_check = left_value.mayRaiseExceptionBool(BaseException), emit = emit, context = context ) if expression.isExpressionConditionalOR(): context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) else: context.setTrueBranchTarget(false_target) context.setFalseBranchTarget(true_target) getBranchingCode( condition = "%s == 1" % truth_name, emit = emit, context = context ) getLabelCode(false_target,emit) # So it's not the left value, then lets release that one right away, it # is not needed, but we remember if it should be added above. getReleaseCode( release_name = left_name, emit = emit, context = context ) # Evaluate the "right" value then. generateExpressionCode( to_name = right_name, expression = expression.getRight(), emit = emit, context = context ) # Again, remember the reference count to manage it manually. needs_ref2 = context.needsCleanup(right_name) if needs_ref2: context.removeCleanupTempName(right_name) if not needs_ref2 and needs_ref1: emit("Py_INCREF( %s );" % right_name) emit( "%s = %s;" % ( to_name, right_name ) ) getGotoCode(end_target, emit) getLabelCode(true_target, emit) if not needs_ref1 and needs_ref2: emit("Py_INCREF( %s );" % left_name) emit( "%s = %s;" % ( to_name, left_name ) ) getLabelCode(end_target, emit) if needs_ref1 or needs_ref2: context.addCleanupTempName(to_name) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) def generateConditionalCode(to_name, expression, emit, context): true_target = context.allocateLabel("condexpr_true") false_target = context.allocateLabel("condexpr_false") end_target = context.allocateLabel("condexpr_end") old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) generateConditionCode( condition = expression.getCondition(), emit = emit, context = context ) getLabelCode(true_target,emit) generateExpressionCode( to_name = to_name, expression = expression.getExpressionYes(), emit = emit, context = context ) needs_ref1 = context.needsCleanup(to_name) # Must not clean this up in other expression. if needs_ref1: context.removeCleanupTempName(to_name) real_emit = emit emit = SourceCodeCollector() generateExpressionCode( to_name = to_name, expression = expression.getExpressionNo(), emit = emit, context = context ) needs_ref2 = context.needsCleanup(to_name) # TODO: Need to buffer generated code, so we can emit extra reference if # not same. if needs_ref1 and not needs_ref2: getGotoCode(end_target, real_emit) getLabelCode(false_target, real_emit) for line in emit.codes: real_emit(line) emit = real_emit emit("Py_INCREF( %s );" % to_name) context.addCleanupTempName(to_name) elif not needs_ref1 and needs_ref2: real_emit("Py_INCREF( %s );" % to_name) getGotoCode(end_target, real_emit) getLabelCode(false_target, real_emit) for line in emit.codes: real_emit(line) emit = real_emit else: getGotoCode(end_target, real_emit) getLabelCode(false_target, real_emit) for line in emit.codes: real_emit(line) emit = real_emit getLabelCode(end_target,emit) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) Nuitka-0.5.28.2/nuitka/codegen/GeneratorCodes.py0000644000372000001440000001606213207537242021652 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled function objects. """ from nuitka import Options from nuitka.PythonVersions import python_version from .CodeHelpers import generateStatementSequenceCode from .Emission import SourceCodeCollector from .FunctionCodes import ( finalizeFunctionLocalVariables, setupFunctionLocalVariables ) from .Indentation import indented from .ModuleCodes import getModuleAccessCode from .templates.CodeTemplatesGeneratorFunction import ( template_generator_exception_exit, template_generator_making, template_generator_noexception_exit, template_generator_return_exit, template_genfunc_yielder_body_template, template_genfunc_yielder_decl_template ) from .VariableCodes import getLocalVariableCodeType def getGeneratorObjectDeclCode(function_identifier): return template_genfunc_yielder_decl_template % { "function_identifier" : function_identifier, } def getGeneratorObjectCode(context, function_identifier, closure_variables, user_variables, outline_variables, temp_variables, needs_exception_exit, needs_generator_return): # Due to the current experimental code, pylint: disable=too-many-locals function_locals, function_cleanup = setupFunctionLocalVariables( context = context, parameters = None, closure_variables = closure_variables, user_variables = user_variables + outline_variables, temp_variables = temp_variables ) function_codes = SourceCodeCollector() generateStatementSequenceCode( statement_sequence = context.getOwner().getBody(), allow_none = True, emit = function_codes, context = context ) finalizeFunctionLocalVariables(context, function_locals, function_cleanup) if needs_exception_exit: generator_exit = template_generator_exception_exit % { "function_cleanup" : indented(function_cleanup) } else: generator_exit = template_generator_noexception_exit % { "function_cleanup" : indented(function_cleanup) } if needs_generator_return: generator_exit += template_generator_return_exit % {} function_dispatch = [ "case %(index)d: goto yield_return_%(index)d;" % { "index" : yield_index } for yield_index in range(context.getLabelCount("yield_return"), 0, -1) ] if function_dispatch: function_dispatch.insert(0, "switch(generator->m_yield_return_index) {") function_dispatch.append('}') local_type_decl = [] local_type_init = [] local_reals = [] for decl in function_locals: if decl.startswith("NUITKA_MAY_BE_UNUSED "): decl = decl[21:] if decl.startswith("static"): local_reals.append(decl) continue if decl in ("char const *type_description;", "PyObject *tmp_unused;"): local_reals.append(decl) continue parts = decl.split('=') if len(parts) == 1: local_type_decl.append(decl) else: type_decl = parts[0].strip() var_name = type_decl.split('*')[-1] var_name = var_name.split(' ')[-1] local_type_decl.append(type_decl) local_type_init.append( "local_variables->" + var_name + " =" + parts[1] ) if Options.isExperimental("generator_goto"): function_locals = local_reals + local_type_init return template_genfunc_yielder_body_template % { "function_identifier" : function_identifier, "function_body" : indented(function_codes.codes), "function_local_types" : indented(local_type_decl), "function_var_inits" : indented(function_locals), "function_dispatch" : indented(function_dispatch), "generator_exit" : generator_exit } def getClosureCopyCode(to_name, closure_variables, closure_type, context): """ Get code to copy closure variables storage. This gets used by generator/coroutine/asyncgen with varying "closure_type". """ closure_copy = [] for count, (variable, version) in enumerate(closure_variables): variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, version) target_cell_code = "((%s)%s)->m_closure[%d]" % ( closure_type, to_name, count ) variable_c_type.getCellObjectAssignmentCode( target_cell_code = target_cell_code, variable_code_name = variable_code_name, emit = closure_copy.append ) closure_copy.append( "assert( Py_SIZE( %s ) >= %s ); " % ( to_name, len(closure_variables) ) ) return closure_copy def generateMakeGeneratorObjectCode(to_name, expression, emit, context): generator_object_body = expression.getGeneratorRef().getFunctionBody() generator_name_obj = context.getConstantCode( constant = generator_object_body.getFunctionName() ) if python_version < 350: generator_qualname_obj = "NULL" else: generator_qualname_obj = context.getConstantCode( constant = generator_object_body.getFunctionQualname() ) closure_variables = expression.getClosureVariableVersions() closure_copy = getClosureCopyCode( to_name = to_name, closure_type = "struct Nuitka_GeneratorObject *", closure_variables = closure_variables, context = context ) emit( template_generator_making % { "closure_copy" : indented(closure_copy, 0, True), "to_name" : to_name, "generator_identifier" : generator_object_body.getCodeName(), "generator_module" : getModuleAccessCode(context), "generator_name_obj" : generator_name_obj, "generator_qualname_obj" : generator_qualname_obj, "code_identifier" : context.getCodeObjectHandle( code_object = expression.getCodeObject() ), "closure_count" : len(closure_variables) } ) context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/VariableCodes.py0000644000372000001440000003227713207537242021457 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Low level variable code generation. """ from .CodeHelpers import generateExpressionCode from .Emission import SourceCodeCollector from .ErrorCodes import getCheckObjectCode, getNameReferenceErrorCode from .Indentation import indented from .templates.CodeTemplatesVariables import ( template_del_global_unclear, template_read_maybe_local_unclear, template_read_mvar_unclear ) def generateAssignmentVariableCode(statement, emit, context): tmp_name = context.allocateTempName("assign_source") generateExpressionCode( expression = statement.getAssignSource(), to_name = tmp_name, emit = emit, context = context ) getVariableAssignmentCode( tmp_name = tmp_name, variable = statement.getVariable(), version = statement.getVariableVersion(), needs_release = statement.needsReleasePreviousValue(), in_place = statement.inplace_suspect, emit = emit, context = context ) # Ownership of that reference must have been transfered. assert not context.needsCleanup(tmp_name) def generateDelVariableCode(statement, emit, context): old_source_ref = context.setCurrentSourceCodeReference( statement.getSourceReference() ) getVariableDelCode( variable = statement.getVariable(), new_version = statement.variable_trace.getVersion(), old_version = statement.previous_trace.getVersion(), tolerant = statement.isTolerant(), needs_check = statement.isTolerant() or \ statement.mayRaiseException(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateVariableReleaseCode(statement, emit, context): variable = statement.getVariable() if variable.isSharedTechnically(): # TODO: We might start to not allocate the cell object, then a check # would be due. But currently we always allocate it. needs_check = False else: needs_check = not statement.variable_trace.mustHaveValue() getVariableReleaseCode( variable = statement.getVariable(), version = statement.getVariableVersion(), needs_check = needs_check, emit = emit, context = context ) def generateVariableReferenceCode(to_name, expression, emit, context): getVariableAccessCode( to_name = to_name, variable = expression.getVariable(), version = expression.getVariableVersion(), needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) def getVariableCodeName(in_context, variable): if in_context: # Closure case: return "closure_" + variable.getCodeName() elif variable.isParameterVariable(): return "par_" + variable.getCodeName() elif variable.isTempVariable(): return "tmp_" + variable.getCodeName() else: return "var_" + variable.getCodeName() def getLocalVariableCodeType(context, variable, version): # Now must be local or temporary variable. user = context.getOwner() owner = variable.getOwner() user = user.getEntryPoint() prefix = "" if owner.isExpressionOutlineFunction() or owner.isExpressionClassBody(): entry_point = owner.getEntryPoint() prefix = "outline_%d_" % entry_point.getTraceCollection().getOutlineFunctions().index(owner) owner = entry_point variable_trace = user.getTraceCollection().getVariableTrace(variable, version) c_type = variable_trace.getPickedCType(context) if owner is user: result = getVariableCodeName( in_context = False, variable = variable ) result = prefix + result elif context.isForDirectCall(): if user.isExpressionGeneratorObjectBody(): closure_index = user.getClosureVariableIndex(variable) result = "generator->m_closure[%d]" % closure_index elif user.isExpressionCoroutineObjectBody(): closure_index = user.getClosureVariableIndex(variable) result = "coroutine->m_closure[%d]" % closure_index elif user.isExpressionAsyncgenObjectBody(): closure_index = user.getClosureVariableIndex(variable) result = "asyncgen->m_closure[%d]" % closure_index else: result = getVariableCodeName( in_context = True, variable = variable ) result = prefix + result else: closure_index = user.getClosureVariableIndex(variable) if user.isExpressionGeneratorObjectBody(): result = "generator->m_closure[%d]" % closure_index elif user.isExpressionCoroutineObjectBody(): result = "coroutine->m_closure[%d]" % closure_index elif user.isExpressionAsyncgenObjectBody(): result = "asyncgen->m_closure[%d]" % closure_index else: # TODO: If this were context.getContextObjectName() this would be # a one liner. result = "self->m_closure[%d]" % closure_index return result, c_type def getVariableCode(context, variable, version): # Modules are simple. if variable.isModuleVariable(): return getVariableCodeName( in_context = False, variable = variable ) variable_code_name, _variable_c_type = getLocalVariableCodeType(context, variable, version) return variable_code_name def getLocalVariableInitCode(context, variable, version, init_from): assert not variable.isModuleVariable() variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, version) if variable.isLocalVariable(): context.setVariableType(variable, variable_code_name, variable_c_type) return variable_c_type.getVariableInitCode(variable_code_name, init_from) def getVariableAssignmentCode(context, emit, variable, version, tmp_name, needs_release, in_place): # For transfer of ownership. if context.needsCleanup(tmp_name): ref_count = 1 else: ref_count = 0 if variable.isModuleVariable(): emit( "UPDATE_STRING_DICT%s( moduledict_%s, (Nuitka_StringObject *)%s, %s );" % ( ref_count, context.getModuleCodeName(), context.getConstantCode( constant = variable.getName(), ), tmp_name ) ) if ref_count: context.removeCleanupTempName(tmp_name) else: variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, version) if variable.isLocalVariable(): context.setVariableType(variable, variable_code_name, variable_c_type) # TODO: this was not handled previously, do not overlook when it # occurs. assert not in_place or not variable.isTempVariable() emit( variable_c_type.getLocalVariableAssignCode( variable_code_name = variable_code_name, needs_release = needs_release, tmp_name = tmp_name, ref_count = ref_count, in_place = in_place ) ) if ref_count: context.removeCleanupTempName(tmp_name) def _generateModuleVariableAccessCode(to_name, variable_name, needs_check, emit, context): emit( template_read_mvar_unclear % { "module_identifier" : context.getModuleCodeName(), "tmp_name" : to_name, "var_name" : context.getConstantCode( constant = variable_name ) } ) if needs_check: getNameReferenceErrorCode( variable_name = variable_name, condition = "%s == NULL" % to_name, emit = emit, context = context ) else: getCheckObjectCode(to_name, emit) def generateLocalsDictVariableRefCode(to_name, expression, emit, context): variable_name = expression.getVariableName() fallback_emit = SourceCodeCollector() getVariableAccessCode( to_name = to_name, variable = expression.getFallbackVariable(), version = expression.getFallbackVariableVersion(), needs_check = True, emit = fallback_emit, context = context ) emit( template_read_maybe_local_unclear % { "locals_dict" : context.getLocalsDictName(), "fallback" : indented(fallback_emit.codes), "tmp_name" : to_name, "var_name" : context.getConstantCode( constant = variable_name ) } ) def getVariableAccessCode(to_name, variable, version, needs_check, emit, context): if variable.isModuleVariable(): _generateModuleVariableAccessCode( to_name = to_name, variable_name = variable.getName(), needs_check = needs_check, emit = emit, context = context ) else: variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, version) variable_c_type.getVariableObjectAccessCode( to_name = to_name, variable_code_name = variable_code_name, variable = variable, needs_check = needs_check, emit = emit, context = context ) def getVariableDelCode(variable, old_version, new_version, tolerant, needs_check, emit, context): if variable.isModuleVariable(): check = not tolerant res_name = context.getIntResName() emit( template_del_global_unclear % { "module_identifier" : context.getModuleCodeName(), "res_name" : res_name, "var_name" : context.getConstantCode( constant = variable.getName() ) } ) # TODO: Apply needs_check for module variables too. if check: getNameReferenceErrorCode( variable_name = variable.getName(), condition = "%s == -1" % res_name, emit = emit, context = context ) elif variable.isLocalVariable(): variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, old_version) variable_code_name_new, variable_c_new_type = getLocalVariableCodeType(context, variable, new_version) # TODO: We need to split this operation in two parts. Release and init # are not one thing. assert variable_c_type == variable_c_new_type context.setVariableType(variable, variable_code_name_new, variable_c_new_type) variable_c_type.getDeleteObjectCode( variable_code_name = variable_code_name, tolerant = tolerant, needs_check = needs_check, variable = variable, emit = emit, context = context ) elif variable.isTempVariable(): variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, old_version) _variable_code_name, variable_c_new_type = getLocalVariableCodeType(context, variable, new_version) # TODO: We need to split this operation in two parts. Release and init # are not one thing. assert variable_c_type is variable_c_new_type variable_c_type.getDeleteObjectCode( variable_code_name = variable_code_name, tolerant = tolerant, needs_check = needs_check, variable = variable, emit = emit, context = context ) else: assert False, variable def getVariableReleaseCode(variable, version, needs_check, emit, context): assert not variable.isModuleVariable() variable_code_name, variable_c_type = getLocalVariableCodeType(context, variable, version) variable_c_type.getReleaseCode( variable_code_name = variable_code_name, needs_check = needs_check, emit = emit ) Nuitka-0.5.28.2/nuitka/codegen/ConstantCodes.py0000644000372000001440000010640613134660221021510 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Low level constant code generation. This deals with constants, there creation, there access, and some checks about them. Even mutable constants should not change during the course of the program. There are shared constants, which are created for multiple modules to use, you can think of them as globals. And there are module local constants, which are for a single module only. """ import ctypes import marshal import re import struct import sys from logging import warning from nuitka import Options from nuitka.__past__ import ( # pylint: disable=I0021,redefined-builtin iterItems, long, unicode, xrange ) from nuitka.Constants import compareConstants, getConstantWeight, isMutable from nuitka.PythonVersions import python_version from .BlobCodes import StreamData from .Emission import SourceCodeCollector from .Indentation import indented from .templates.CodeTemplatesConstants import template_constants_reading def generateConstantReferenceCode(to_name, expression, emit, context): """ Assign the constant behind the expression to to_name.""" getConstantAccess( to_name = to_name, constant = expression.getConstant(), emit = emit, context = context ) def generateConstantNoneReferenceCode(to_name, expression, emit, context): """ Assign 'None' to to_name.""" # No context or other knowledge needed, pylint: disable=unused-argument emit( "%s = Py_None;" % to_name ) def generateConstantTrueReferenceCode(to_name, expression, emit, context): """ Assign 'True' to to_name.""" # No context or other knowledge needed, pylint: disable=unused-argument emit( "%s = Py_True;" % to_name ) def generateConstantFalseReferenceCode(to_name, expression, emit, context): """ Assign 'False' to to_name.""" # No context or other knowledge needed, pylint: disable=unused-argument emit( "%s = Py_False;" % to_name ) def generateConstantEllipsisReferenceCode(to_name, expression, emit, context): """ Assign 'Ellipsis' to to_name.""" # No context or other knowledge needed, pylint: disable=unused-argument emit( "%s = Py_Ellipsis;" % to_name ) # One global stream of constant information. In the future it might make # sense to have per module ones, for better locality of indexes within it, # but we don't do this yet. stream_data = StreamData() # TODO: The determination of this should already happen in Building or in a # helper not during code generation. _match_attribute_names = re.compile(r"[a-zA-Z_][a-zA-Z0-9_]*$") def _isAttributeName(value): # TODO: The exception is to make sure we intern the ".0" argument name # used for generator expressions, iterator value. return _match_attribute_names.match(value) or value == ".0" sizeof_long = ctypes.sizeof(ctypes.c_long) max_unsigned_long = 2**(sizeof_long*8)-1 # The gcc gives a warning for -2**sizeof_long*8-1, which is still an "int", but # seems to not work (without warning) as literal, so avoid it. min_signed_long = -(2**(sizeof_long*8-1)-1) done = set() def _getConstantInitValueCode(constant_value, constant_type): """ Return code, if possible, to create a constant. It's only used for module local constants, like error messages, and provides no caching of the values. When it returns "None", it is in error. """ # This function is a case driven by returns, pylint: disable=too-many-return-statements if constant_type is unicode: # Python3: Strings that can be encoded as UTF-8 are done more or less # directly. When they cannot be expressed as UTF-8, that is rare not we # can indeed use pickling. try: encoded = constant_value.encode("utf-8") if str is bytes: return "UNSTREAM_UNICODE( %s )" % ( stream_data.getStreamDataCode(encoded) ) else: return "UNSTREAM_STRING( %s, %d, %d )" % ( stream_data.getStreamDataCode(encoded, fixed_size = True), len(constant_value), 1 if _isAttributeName(constant_value) else 0 ) except UnicodeEncodeError: # TODO: try and use "surrogateescape" for this return None elif constant_type is str: assert str is bytes if len(constant_value) == 1: return "UNSTREAM_CHAR( %d, %d )" % ( ord(constant_value[0]), 1 if _isAttributeName(constant_value) else 0 ) else: return "UNSTREAM_STRING( %s, %d )" % ( stream_data.getStreamDataCode(constant_value), 1 if _isAttributeName(constant_value) else 0 ) elif constant_type is bytes: assert str is not bytes return "UNSTREAM_BYTES( %s )" % ( stream_data.getStreamDataCode(constant_value) ) else: return None def decideMarshal(constant_value): """ Decide of a constant can be created using "marshal" module methods. This is not the case for everything. A prominent exception is types, they are constants, but the "marshal" module refuses to work with them. """ # Many cases to deal with, pylint: disable=too-many-return-statements constant_type = type(constant_value) if constant_type is type: # Types cannot be marshaled, there is no choice about it. return False elif constant_type is dict: # Look at all the keys an values, if one of it cannot be marshaled, # or should not, that is it. for key, value in iterItems(constant_value): if not decideMarshal(key): return False if not decideMarshal(value): return False elif constant_type in (tuple, list, set, frozenset): for element_value in constant_value: if not decideMarshal(element_value): return False elif constant_type is xrange: return False elif constant_type is slice: return False return True def isMarshalConstant(constant_value): """ Decide if we want to use marshal to create a constant. The reason we do this, is because creating dictionaries with 700 elements creates a lot of C code, while gaining usually no performance at all. The MSVC compiler is especially notorious about hanging like forever with this active, due to its optimizer not scaling. Therefore we use a constant "weight" (how expensive it is), and apply that to decide. If marshal is not possible, or constant "weight" is too large, we don't do it. Also, for some constants, marshal can fail, and return other values. Check that too. In that case, we have to create it. """ if not decideMarshal(constant_value): return False if getConstantWeight(constant_value) < 20: return False marshal_value = marshal.dumps(constant_value) restored = marshal.loads(marshal_value) r = compareConstants(constant_value, restored) if not r: pass # TODO: Potentially warn about these, where that is not the case. return r def getMarshalCode(constant_identifier, constant_value, emit): """ Force the marshal of a value. """ marshal_value = marshal.dumps(constant_value) restored = marshal.loads(marshal_value) assert compareConstants(constant_value, restored) emit( "%s = PyMarshal_ReadObjectFromString( (char *)%s );" % ( constant_identifier, stream_data.getStreamDataCode(marshal_value) ) ) def attemptToMarshal(constant_identifier, constant_value, emit): """ Try and marshal a value, if so decided. Indicate with return value. See above for why marshal is only used in problematic cases. """ if not isMarshalConstant(constant_value): return False marshal_value = marshal.dumps(constant_value) restored = marshal.loads(marshal_value) # TODO: The check in isMarshalConstant is currently preventing this from # happening. if not compareConstants(constant_value, restored): warning("Problem with marshal of constant %r", constant_value) return False emit( "%s = PyMarshal_ReadObjectFromString( (char *)%s );" % ( constant_identifier, stream_data.getStreamDataCode(marshal_value) ) ) return True def _addConstantInitCode(context, emit, check, constant_type, constant_value, constant_identifier, module_level): """ Emit code for a specific constant to be prepared during init. This may be module or global init. Code makes sure that nested constants belong into the same scope. """ if constant_value is None: return if constant_value is False: return if constant_value is True: return if constant_value is Ellipsis: return if type(constant_value) is type: return # Do not repeat ourselves. if constant_identifier in done: return if Options.shallTraceExecution(): emit("""NUITKA_PRINT_TRACE("Creating constant: %s");""" % constant_identifier) # Then it's a real named constant not yet created. __addConstantInitCode(context, emit, check, constant_type, constant_value, constant_identifier, module_level) if Options.isDebug(): emit( """\ hash_%(constant_identifier)s = DEEP_HASH( %(constant_identifier)s );""" % { "constant_identifier" : constant_identifier } ) check( """\ CHECK_OBJECT( %(constant_identifier)s ); assert( hash_%(constant_identifier)s == DEEP_HASH( %(constant_identifier)s ) );""" % { "constant_identifier" : constant_identifier } ) def __addConstantInitCode(context, emit, check, constant_type, constant_value, constant_identifier, module_level): """ Emit code for a specific constant to be prepared during init. This may be module or global init. Code makes sure that nested constants belong into the same scope. """ # This has many cases, that all return, and do a lot. # pylint: disable=too-many-branches,too-many-locals,too-many-return-statements,too-many-statements # For the module level, we only mean to create constants that are used only # inside of it. For the global level, it must must be single use. if module_level: if context.global_context.getConstantUseCount(constant_identifier) != 1: return else: if context.getConstantUseCount(constant_identifier) == 1: return # Adding it to "done". We cannot have recursive constants, so this is OK # to be done now. done.add(constant_identifier) # Use shortest code for ints and longs. if constant_type is long: # See above, same for long values. Note: These are of course not # existent with Python3 which would have covered it before. if constant_value >= 0 and constant_value <= max_unsigned_long: emit ( "%s = PyLong_FromUnsignedLong( %sul );" % ( constant_identifier, constant_value ) ) return elif constant_value < 0 and constant_value >= min_signed_long: emit ( "%s = PyLong_FromLong( %sl );" % ( constant_identifier, constant_value ) ) return elif constant_value == min_signed_long-1: # There are compilers out there, that give warnings for the literal # MININT when used. We work around that warning here. emit( """\ %s = PyLong_FromLong( %sl ); // To be corrected with -1 in-place next lines. CHECK_OBJECT( const_int_pos_1 ); %s = PyNumber_InPlaceSubtract( %s, PyLong_FromLong( 1 ) );""" % ( constant_identifier, min_signed_long, constant_identifier, constant_identifier ) ) return else: getMarshalCode( constant_identifier = constant_identifier, constant_value = constant_value, emit = emit ) return elif constant_type is int: if constant_value >= min_signed_long: emit( "%s = PyInt_FromLong( %sl );" % ( constant_identifier, constant_value ) ) return else: # There are compilers out there, that give warnings for the literal # MININT when used. We work around that warning here. assert constant_value == min_signed_long-1 emit( """\ %s = PyInt_FromLong( %sl ); // To be corrected in next line. %s = PyNumber_InPlaceSubtract( %s, PyInt_FromLong( 1 ) );""" % ( constant_identifier, min_signed_long, constant_identifier, constant_identifier ) ) return if constant_type is unicode: try: encoded = constant_value.encode("utf-8") if str is bytes: emit( "%s = UNSTREAM_UNICODE( %s );" % ( constant_identifier, stream_data.getStreamDataCode(encoded) ) ) else: emit( "%s = UNSTREAM_STRING( %s, %d );" % ( constant_identifier, stream_data.getStreamDataCode(encoded), 1 if _isAttributeName(constant_value) else 0 ) ) return except UnicodeEncodeError: getMarshalCode( constant_identifier = constant_identifier, constant_value = constant_value, emit = emit ) return elif constant_type is str: # Python3: Strings that can be encoded as UTF-8 are done more or less # directly. When they cannot be expressed as UTF-8, that is rare not we # can indeed use pickling. assert str is bytes if len(constant_value) == 1: emit( "%s = UNSTREAM_CHAR( %d, %d );" % ( constant_identifier, ord(constant_value[0]), 1 if _isAttributeName(constant_value) else 0 ) ) else: emit( "%s = UNSTREAM_STRING( %s, %d );" % ( constant_identifier, stream_data.getStreamDataCode(constant_value), 1 if _isAttributeName(constant_value) else 0 ) ) return elif constant_type is bytes: # Python3 only, for Python2, bytes do not happen. assert str is not bytes emit( "%s = UNSTREAM_BYTES( %s );" % ( constant_identifier, stream_data.getStreamDataCode(constant_value) ) ) return if constant_type is float: emit( "%s = UNSTREAM_FLOAT( %s );" % ( constant_identifier, stream_data.getStreamDataCode( value = struct.pack("= 330: # For Python2 ranges, we use C long values directly. considerForDeferral(constant_value.start) considerForDeferral(constant_value.step) considerForDeferral(constant_value.stop) else: parts = [ int(value) for value in str(constant_value)[6:-1].split(',') ] if len(parts) <= 1: parts.append(0) if len(parts) <= 2: parts.append(1) for value in parts: considerForDeferral(value) for constant_identifier in set(module_context.getConstants()): constant_value = module_context.global_context.constants[ constant_identifier ] constant_type = type(constant_value) if constant_type in (tuple, dict, list, set, frozenset, slice, xrange): considerForDeferral(constant_value) def getConstantsDefinitionCode(context): """ Create the code code "__constants.c" file. This needs to create code to make all global constants (used in more than one module) and create them. """ constant_inits, constant_checks = getConstantsInitCode( context = context ) constant_declarations = getConstantsDeclCode( context = context ) if Options.shallMakeModule(): sys_executable = None else: sys_executable = context.getConstantCode(sys.executable) return template_constants_reading % { "constant_declarations" : '\n'.join(constant_declarations), "constant_inits" : indented(constant_inits), "constant_checks" : indented(constant_checks), "sys_executable" : sys_executable } Nuitka-0.5.28.2/nuitka/codegen/CodeObjectCodes.py0000644000372000001440000001147413122472300021714 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for code objects. Right now only the creation is done here. But more should be added later on. """ import os from nuitka import Options from nuitka.PythonVersions import python_version def getCodeObjectsDeclCode(context): statements = [] for _code_object_key, code_identifier in context.getCodeObjects(): declaration = "static PyCodeObject *%s;" % code_identifier statements.append(declaration) if context.getOwner().getFullName() == "__main__": statements.append('/* For use in "MainProgram.c". */') statements.append("PyCodeObject *codeobj_main = NULL;") return statements def getCodeObjectsInitCode(context): # There is a bit of details to this, code objects have many flags to deal # with, and we are making some optimizations as well as customization to # what path should be put there, pylint: disable=too-many-branches statements = [] code_objects = context.getCodeObjects() # Create the always identical, but dynamic filename first thing. if code_objects: context.markAsNeedsModuleFilenameObject() filename_code = "module_filename_obj" if context.needsModuleFilenameObject(): module_filename = context.getOwner().getRunTimeFilename() # We do not care about release of this object, as code object live # forever anyway. if Options.getFileReferenceMode() == "frozen" or \ os.path.isabs(module_filename): template = "module_filename_obj = %s;" else: template = "module_filename_obj = MAKE_RELATIVE_PATH( %s );" statements.append( template % ( context.getConstantCode( constant = module_filename ) ) ) for code_object_key, code_identifier in code_objects: co_flags = [] # Make sure the filename is always identical. assert code_object_key[0] == module_filename if code_object_key[6] in ("Module", "Class", "Function"): pass elif code_object_key[6] == "Generator": co_flags.append("CO_GENERATOR") elif code_object_key[6] == "Coroutine": co_flags.append("CO_COROUTINE") elif code_object_key[6] == "Asyncgen": co_flags.append("CO_ASYNC_GENERATOR") else: assert False, code_object_key[6] if code_object_key[7]: co_flags.append("CO_OPTIMIZED") if code_object_key[8]: co_flags.append("CO_NEWLOCALS") if code_object_key[9]: co_flags.append("CO_VARARGS") if code_object_key[10]: co_flags.append("CO_VARKEYWORDS") if not code_object_key[11]: co_flags.append("CO_NOFREE") co_flags.extend(code_object_key[12]) if python_version < 300: code = "%s = MAKE_CODEOBJ( %s, %s, %d, %s, %d, %s );" % ( code_identifier, filename_code, context.getConstantCode( constant = code_object_key[1] ), code_object_key[2], context.getConstantCode( constant = code_object_key[3] ), code_object_key[4], " | ".join(co_flags) or '0', ) else: code = "%s = MAKE_CODEOBJ( %s, %s, %d, %s, %d, %d, %s );" % ( code_identifier, filename_code, context.getConstantCode( constant = code_object_key[1] ), code_object_key[2], context.getConstantCode( constant = code_object_key[3] ), code_object_key[4], code_object_key[5], " | ".join(co_flags) or '0', ) statements.append(code) if context.getOwner().getFullName() == "__main__": if code_object_key[1] == "": statements.append("codeobj_main = %s;" % code_identifier) return statements Nuitka-0.5.28.2/nuitka/codegen/BranchCodes.py0000644000372000001440000000426313207537242021121 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Branch related codes. """ from .CodeHelpers import generateStatementSequenceCode from .ConditionalCodes import generateConditionCode from .LabelCodes import getGotoCode, getLabelCode def generateBranchCode(statement, emit, context): true_target = context.allocateLabel("branch_yes") false_target = context.allocateLabel("branch_no") end_target = context.allocateLabel("branch_end") old_true_target = context.getTrueBranchTarget() old_false_target = context.getFalseBranchTarget() context.setTrueBranchTarget(true_target) context.setFalseBranchTarget(false_target) generateConditionCode( condition = statement.getCondition(), emit = emit, context = context ) context.setTrueBranchTarget(old_true_target) context.setFalseBranchTarget(old_false_target) getLabelCode(true_target, emit) generateStatementSequenceCode( statement_sequence = statement.getBranchYes(), emit = emit, context = context ) if statement.getBranchNo() is not None: getGotoCode(end_target, emit) getLabelCode(false_target, emit) generateStatementSequenceCode( statement_sequence = statement.getBranchNo(), emit = emit, context = context ) getLabelCode(end_target, emit) else: getLabelCode(false_target, emit) Nuitka-0.5.28.2/nuitka/codegen/FunctionCodes.py0000644000372000001440000006071713207537242021517 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled function objects. """ from nuitka.PythonVersions import python_version from .c_types.CTypePyObjectPtrs import CTypeCellObject, CTypePyObjectPtrPtr from .CodeHelpers import generateExpressionCode, generateStatementSequenceCode from .Contexts import PythonFunctionOutlineContext from .Emission import SourceCodeCollector from .ErrorCodes import ( getErrorExitCode, getErrorVariableDeclarations, getExceptionKeeperVariableNames, getExceptionPreserverVariableNames, getMustNotGetHereCode, getReleaseCode ) from .Indentation import indented from .LabelCodes import getGotoCode, getLabelCode from .LineNumberCodes import emitErrorLineNumberUpdateCode from .ModuleCodes import getModuleAccessCode from .PythonAPICodes import getReferenceExportCode from .templates.CodeTemplatesFunction import ( function_dict_setup, function_direct_body_template, template_function_body, template_function_direct_declaration, template_function_exception_exit, template_function_make_declaration, template_function_return_exit, template_make_function_template ) from .TupleCodes import getTupleCreationCode from .VariableCodes import ( getLocalVariableCodeType, getLocalVariableInitCode, getVariableCode, getVariableCodeName ) def getClosureVariableProvisionCode(context, closure_variables): result = [] for variable, version in closure_variables: result.append( getVariableCode( context = context, variable = variable, version = version ) ) return result def _getFunctionCreationArgs(defaults_name, kw_defaults_name, annotations_name, closure_variables): result = [] if defaults_name is not None: result.append("PyObject *defaults") if kw_defaults_name is not None: result.append("PyObject *kw_defaults") if annotations_name is not None: result.append("PyObject *annotations") for closure_variable in closure_variables: result.append( "struct Nuitka_CellObject *%s" % ( getVariableCodeName( variable = closure_variable, in_context = True ) ) ) return result def getFunctionMakerDecl(function_identifier, defaults_name, kw_defaults_name, annotations_name, closure_variables): function_creation_arg_spec = _getFunctionCreationArgs( defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = closure_variables ) return template_function_make_declaration % { "function_identifier" : function_identifier, "function_creation_arg_spec" : ", ".join( function_creation_arg_spec ) } def getFunctionEntryPointIdentifier(function_identifier): return "impl_" + function_identifier def getFunctionMakerCode(function_name, function_qualname, function_identifier, code_identifier, closure_variables, defaults_name, kw_defaults_name, annotations_name, function_doc, context): # We really need this many parameters here and functions have many details, # that we express as variables, pylint: disable=too-many-locals function_creation_args = _getFunctionCreationArgs( defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = closure_variables ) if python_version < 330 or function_qualname == function_name: function_qualname_obj = "NULL" else: function_qualname_obj = context.getConstantCode( constant = function_qualname ) closure_copy = [] for count, closure_variable in enumerate(closure_variables): closure_copy.append( "result->m_closure[%d] = %s;" % ( count, getVariableCodeName( True, closure_variable ) ) ) closure_copy.append( "Py_INCREF( result->m_closure[%d] );" %count ) result = template_make_function_template % { "function_name_obj" : context.getConstantCode( constant = function_name ), "function_qualname_obj" : function_qualname_obj, "function_identifier" : function_identifier, "function_impl_identifier" : getFunctionEntryPointIdentifier( function_identifier = function_identifier, ), "function_creation_args" : ", ".join( function_creation_args ), "code_identifier" : code_identifier, "closure_copy" : indented(closure_copy, 0, True), "function_doc" : context.getConstantCode( constant = function_doc ), "defaults" : "defaults" if defaults_name else "NULL", "kw_defaults" : "kw_defaults" if kw_defaults_name else "NULL", "annotations" : "annotations" if annotations_name else context.getConstantCode({}), "closure_count" : len(closure_variables), "module_identifier" : getModuleAccessCode( context = context ), } return result def generateFunctionCreationCode(to_name, expression, emit, context): # This is about creating functions, which is detail ridden stuff, # pylint: disable=too-many-locals function_body = expression.getFunctionRef().getFunctionBody() code_object = expression.getCodeObject() defaults = expression.getDefaults() kw_defaults = expression.getKwDefaults() annotations = expression.getAnnotations() defaults_first = not expression.kw_defaults_before_defaults assert function_body.needsCreation(), function_body def handleKwDefaults(): if kw_defaults: kw_defaults_name = context.allocateTempName("kw_defaults") assert not kw_defaults.isExpressionConstantRef() or \ kw_defaults.getConstant() != {}, kw_defaults.getConstant() generateExpressionCode( to_name = kw_defaults_name, expression = kw_defaults, emit = emit, context = context ) else: kw_defaults_name = None return kw_defaults_name def handleDefaults(): if defaults: defaults_name = context.allocateTempName("defaults") getTupleCreationCode( to_name = defaults_name, elements = defaults, emit = emit, context = context ) else: defaults_name = None return defaults_name if defaults_first: defaults_name = handleDefaults() kw_defaults_name = handleKwDefaults() else: kw_defaults_name = handleKwDefaults() defaults_name = handleDefaults() if annotations: annotations_name = context.allocateTempName("annotations") generateExpressionCode( to_name = annotations_name, expression = annotations, emit = emit, context = context, ) else: annotations_name = None function_identifier = function_body.getCodeName() # Creation code needs to be done only once. if not context.hasHelperCode(function_identifier): maker_code = getFunctionMakerCode( function_name = function_body.getFunctionName(), function_qualname = function_body.getFunctionQualname(), function_identifier = function_identifier, code_identifier = context.getCodeObjectHandle( code_object = code_object, ), closure_variables = function_body.getClosureVariables(), defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, function_doc = function_body.getDoc(), context = context ) context.addHelperCode(function_identifier, maker_code) function_decl = getFunctionMakerDecl( function_identifier = function_body.getCodeName(), defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = function_body.getClosureVariables() ) context.addDeclaration(function_identifier, function_decl) getFunctionCreationCode( to_name = to_name, function_identifier = function_body.getCodeName(), defaults_name = defaults_name, kw_defaults_name = kw_defaults_name, annotations_name = annotations_name, closure_variables = expression.getClosureVariableVersions(), emit = emit, context = context ) getReleaseCode( release_name = annotations_name, emit = emit, context = context ) def getFunctionCreationCode(to_name, function_identifier, defaults_name, kw_defaults_name, annotations_name, closure_variables, emit, context): args = [] if defaults_name is not None: getReferenceExportCode(defaults_name, emit, context) args.append(defaults_name) if kw_defaults_name is not None: args.append(kw_defaults_name) if annotations_name is not None: args.append(annotations_name) args += getClosureVariableProvisionCode( context = context, closure_variables = closure_variables ) emit( "%s = MAKE_FUNCTION_%s( %s );" % ( to_name, function_identifier, ", ".join(args) ) ) if context.needsCleanup(defaults_name): context.removeCleanupTempName(defaults_name) if context.needsCleanup(kw_defaults_name): context.removeCleanupTempName(kw_defaults_name) # No error checks, this supposedly, cannot fail. context.addCleanupTempName(to_name) def getDirectFunctionCallCode(to_name, function_identifier, arg_names, closure_variables, needs_check, emit, context): function_identifier = getFunctionEntryPointIdentifier( function_identifier = function_identifier ) suffix_args = [] for closure_variable, closure_variable_version in closure_variables: variable_code_name, variable_c_type = getLocalVariableCodeType( context = context, variable = closure_variable, version = closure_variable_version ) suffix_args.append( variable_c_type.getVariableArgReferencePassingCode(variable_code_name) ) # TODO: We ought to not assume references for direct calls, or make a # profile if an argument needs a reference at all. Most functions don't # bother to release a called argument by "del" or assignment to it. We # could well know that ahead of time. for arg_name in arg_names: if context.needsCleanup(arg_name): context.removeCleanupTempName(arg_name) else: emit("Py_INCREF( %s );" % arg_name) if arg_names: emit( """ { PyObject *dir_call_args[] = {%s}; %s = %s( dir_call_args%s%s ); }""" % ( ", ".join( arg_names ), to_name, function_identifier, ", " if suffix_args else "", ", ".join(suffix_args) ) ) else: emit( "%s = %s( NULL%s%s );" % ( to_name, function_identifier, ", " if suffix_args else "", ", ".join(suffix_args) ) ) # Arguments are owned to the called in direct function call. for arg_name in arg_names: if context.needsCleanup(arg_name): context.removeCleanupTempName(arg_name) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) def getFunctionDirectDecl(function_identifier, closure_variables, file_scope, context): parameter_objects_decl = [ "PyObject **python_pars" ] for closure_variable in closure_variables: variable_code_name, variable_c_type = getLocalVariableCodeType( context = context, variable = closure_variable, version = 0 ) parameter_objects_decl.append( variable_c_type.getVariableArgDeclarationCode(variable_code_name) ) result = template_function_direct_declaration % { "file_scope" : file_scope, "function_identifier" : function_identifier, "direct_call_arg_spec" : ", ".join(parameter_objects_decl), } return result def setupFunctionLocalVariables(context, parameters, closure_variables, user_variables, temp_variables): function_locals = [] function_cleanup = [] if context.hasLocalsDict(): context.allocateLocalsDictName() if parameters is not None: for count, variable in enumerate(parameters.getAllVariables()): function_locals.append( getLocalVariableInitCode( context = context, variable = variable, version = 0, init_from = "python_pars[ %d ]" % count ) ) # User local variable initializations function_locals += [ getLocalVariableInitCode( context = context, variable = variable, version = 0, init_from = None ) for variable in user_variables + tuple( variable for variable in sorted( temp_variables, key = lambda variable: variable.getName() ) ) ] for closure_variable in closure_variables: # Temporary variable closures are to be ignored. if closure_variable.isTempVariable(): continue variable_code_name, variable_c_type = getLocalVariableCodeType( context = context, variable = closure_variable, version = 0 ) if variable_c_type in (CTypeCellObject, CTypePyObjectPtrPtr): context.setVariableType(closure_variable, variable_code_name, variable_c_type) else: assert False, (variable_code_name, variable_c_type) return function_locals, function_cleanup def finalizeFunctionLocalVariables(context, function_locals, function_cleanup): if context.needsExceptionVariables(): function_locals.extend(getErrorVariableDeclarations()) for keeper_index in range(1, context.getKeeperVariableCount()+1): function_locals.extend(getExceptionKeeperVariableNames(keeper_index)) for preserver_id in context.getExceptionPreserverCounts(): function_locals.extend(getExceptionPreserverVariableNames(preserver_id)) tmp_infos = context.getTempNameInfos() function_locals += [ "%s%s%s;" % ( tmp_type, ' ' if not tmp_type.endswith('*') else "", tmp_name ) for tmp_name, tmp_type in tmp_infos ] function_locals += context.getFrameDeclarations() # TODO: Could avoid this unless try/except or try/finally with returns # occur. if context.hasTempName("return_value"): function_locals.append("tmp_return_value = NULL;") if context.hasTempName("generator_return"): function_locals.append("tmp_generator_return = false;") for tmp_name, _tmp_type in tmp_infos: if tmp_name.startswith("tmp_outline_return_value_"): function_locals.append("%s = NULL;" % tmp_name) for locals_dict_name in context.getLocalsDictNames(): function_locals.append( function_dict_setup % { "locals_dict" : locals_dict_name } ) function_cleanup.append( "Py_DECREF( %(locals_dict)s );\n" % { "locals_dict" : locals_dict_name } ) def getFunctionCode(context, function_identifier, parameters, closure_variables, user_variables, outline_variables, temp_variables, function_doc, file_scope, needs_exception_exit): # Functions have many details, that we express as variables, with many # branches to decide, pylint: disable=too-many-locals function_locals, function_cleanup = setupFunctionLocalVariables( context = context, parameters = parameters, closure_variables = closure_variables, user_variables = user_variables + outline_variables, temp_variables = temp_variables ) function_codes = SourceCodeCollector() generateStatementSequenceCode( statement_sequence = context.getOwner().getBody(), allow_none = True, emit = function_codes, context = context ) finalizeFunctionLocalVariables(context, function_locals, function_cleanup) function_doc = context.getConstantCode( constant = function_doc ) result = "" emit = SourceCodeCollector() getMustNotGetHereCode( reason = "Return statement must have exited already.", context = context, emit = emit ) function_exit = indented(emit.codes) + "\n\n" del emit if needs_exception_exit: function_exit += template_function_exception_exit % { "function_cleanup" : indented(function_cleanup), } if context.hasTempName("return_value"): function_exit += indented( template_function_return_exit % { "function_cleanup" : indented(function_cleanup), } ) if context.isForCreatedFunction(): parameter_objects_decl = ["struct Nuitka_FunctionObject const *self"] else: parameter_objects_decl = [] parameter_objects_decl += [ "PyObject **python_pars" ] if context.isForDirectCall(): for closure_variable in closure_variables: variable_code_name, variable_c_type = getLocalVariableCodeType( context = context, variable = closure_variable, version = 0 ) parameter_objects_decl.append( variable_c_type.getVariableArgDeclarationCode(variable_code_name) ) result += function_direct_body_template % { "file_scope" : file_scope, "function_identifier" : function_identifier, "direct_call_arg_spec" : ", ".join( parameter_objects_decl ), "function_locals" : indented(function_locals), "function_body" : indented(function_codes.codes), "function_exit" : function_exit } else: result += template_function_body % { "function_identifier" : function_identifier, "parameter_objects_decl" : ", ".join(parameter_objects_decl), "function_locals" : indented(function_locals), "function_body" : indented(function_codes.codes), "function_exit" : function_exit } return result def getExportScopeCode(cross_module): if cross_module: return "NUITKA_CROSS_MODULE" else: return "NUITKA_LOCAL_MODULE" def generateFunctionCallCode(to_name, expression, emit, context): assert expression.getFunction().isExpressionFunctionCreation() function_body = expression.getFunction().getFunctionRef().getFunctionBody() function_identifier = function_body.getCodeName() argument_values = expression.getArgumentValues() arg_names = [] for count, arg_value in enumerate(argument_values): arg_name = context.allocateTempName("dircall_arg%d" % (count+1)) generateExpressionCode( to_name = arg_name, expression = arg_value, emit = emit, context = context ) arg_names.append(arg_name) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getDirectFunctionCallCode( to_name = to_name, function_identifier = function_identifier, arg_names = arg_names, closure_variables = expression.getClosureVariableVersions(), needs_check = expression.getFunction().getFunctionRef().\ getFunctionBody().mayRaiseException(BaseException), emit = emit, context = context ) def generateFunctionOutlineCode(to_name, expression, emit, context): assert expression.isExpressionOutlineBody() or \ expression.isExpressionOutlineFunction() or \ expression.isExpressionClassBody() if expression.isExpressionOutlineFunctionBodyBase(): context = PythonFunctionOutlineContext( parent = context, outline = expression ) locals_dict_handling = True else: locals_dict_handling = False if locals_dict_handling: if expression.hasLocalsDict(): context.allocateLocalsDictName() # Need to set return target, to assign to_name from. old_return_release_mode = context.getReturnReleaseMode() return_target = context.allocateLabel("outline_result") old_return_target = context.setReturnTarget(return_target) return_value_name = context.allocateTempName("outline_return_value") old_return_value_name = context.setReturnValueName(return_value_name) if expression.isExpressionOutlineFunctionBodyBase() and \ expression.getBody().mayRaiseException(BaseException): exception_target = context.allocateLabel("outline_exception") old_exception_target = context.setExceptionEscape(exception_target) else: exception_target = None generateStatementSequenceCode( statement_sequence = expression.getBody(), emit = emit, context = context, allow_none = False ) getMustNotGetHereCode( reason = "Return statement must have exited already.", context = context, emit = emit ) if exception_target is not None: getLabelCode(exception_target, emit) context.setCurrentSourceCodeReference(expression.getSourceReference()) emitErrorLineNumberUpdateCode(emit, context) getGotoCode(old_exception_target, emit) context.setExceptionEscape(old_exception_target) getLabelCode(return_target, emit) emit( "%s = %s;" % ( to_name, return_value_name ) ) context.addCleanupTempName(to_name) # Restore previous "return" handling. context.setReturnTarget(old_return_target) context.setReturnReleaseMode(old_return_release_mode) context.setReturnValueName(old_return_value_name) if locals_dict_handling: if expression.hasLocalsDict(): context.endLocalsDictName() Nuitka-0.5.28.2/nuitka/codegen/ExceptionCodes.py0000644000372000001440000001560613207537242021665 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Exception handling. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateExpressionCode from .templates.CodeTemplatesExceptions import ( template_publish_exception_to_handler ) def getExceptionIdentifier(exception_type): assert "PyExc" not in exception_type, exception_type if exception_type == "NotImplemented": return "Py_NotImplemented" return "PyExc_%s" % exception_type def generateExceptionRefCode(to_name, expression, emit, context): exception_type = expression.getExceptionName() emit( "%s = %s;" % ( to_name, getExceptionIdentifier(exception_type), ) ) # Lets have context in the API for consistency with everything else. assert context def getTracebackMakingIdentifier(context, lineno_name): frame_handle = context.getFrameHandle() assert frame_handle is not None return "MAKE_TRACEBACK( %s, %s )" % ( frame_handle, lineno_name ) def generateExceptionCaughtTypeCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=unused-argument keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is None: emit( "%s = PyThreadState_GET()->exc_type;" % ( to_name, ) ) else: emit( "%s = %s;" % ( to_name, keeper_variables[0] ) ) def generateExceptionCaughtValueCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=unused-argument keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[1] is None: emit( "%s = PyThreadState_GET()->exc_value;" % ( to_name, ) ) else: if python_version >= 270: emit( "%s = %s;" % ( to_name, keeper_variables[1] ) ) else: emit( "%s = %s ? %s : Py_None;" % ( to_name, keeper_variables[1], keeper_variables[1] ) ) def generateExceptionCaughtTracebackCode(to_name, expression, emit, context): # Functions used for generation all accept expression, but this one does # not use it. pylint: disable=unused-argument keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[2] is None: emit( "%s = PyThreadState_GET()->exc_traceback;" % ( to_name, ) ) else: emit( """\ if ( %(keeper_tb)s != NULL ) { %(to_name)s = (PyObject *)%(keeper_tb)s; Py_INCREF( %(to_name)s ); } else { %(to_name)s = (PyObject *)%(tb_making)s; } """ % { "to_name" : to_name, "keeper_tb" : keeper_variables[2], "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = keeper_variables[3] ) } ) context.addCleanupTempName(to_name) def getExceptionUnpublishedReleaseCode(emit, context): keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is not None: emit("Py_DECREF( %s );" % keeper_variables[0]) emit("Py_XDECREF( %s );" % keeper_variables[1]) emit("Py_XDECREF( %s );" % keeper_variables[2]) def generateExceptionPublishCode(statement, emit, context): # This statement has no attributes really, pylint: disable=unused-argument # TODO: Should this be necessary, something else would have required # them already, or it's wrong. context.markAsNeedsExceptionVariables() # Current variables cannot be used anymore now. keeper_type, keeper_value, keeper_tb, keeper_lineno = context.setExceptionKeeperVariables( (None, None, None, None) ) emit( template_publish_exception_to_handler % { "tb_making" : getTracebackMakingIdentifier( context = context, lineno_name = keeper_lineno ), "keeper_tb" : keeper_tb, "keeper_lineno" : keeper_lineno, "frame_identifier" : context.getFrameHandle(), } ) emit( "NORMALIZE_EXCEPTION( &%s, &%s, &%s );" % ( keeper_type, keeper_value, keeper_tb ) ) if python_version >= 300: emit( "PyException_SetTraceback( %s, (PyObject *)%s );" % ( keeper_value, keeper_tb ) ) emit( "PUBLISH_EXCEPTION( &%s, &%s, &%s );" % ( keeper_type, keeper_value, keeper_tb ) ) def generateBuiltinMakeExceptionCode(to_name, expression, emit, context): from .CallCodes import getCallCodeNoArgs, getCallCodePosArgsQuick exception_arg_names = [] for exception_arg in expression.getArgs(): exception_arg_name = context.allocateTempName("make_exception_arg") generateExpressionCode( to_name = exception_arg_name, expression = exception_arg, emit = emit, context = context ) exception_arg_names.append(exception_arg_name) exception_type = expression.getExceptionName() if exception_arg_names: getCallCodePosArgsQuick( to_name = to_name, called_name = getExceptionIdentifier(exception_type), arg_names = exception_arg_names, needs_check = False, emit = emit, context = context ) else: getCallCodeNoArgs( to_name = to_name, called_name = getExceptionIdentifier(exception_type), needs_check = False, emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/templates/0000755000372000001440000000000013207540420020355 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesVariables.py0000644000372000001440000001513313134660221025315 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the variable handling. """ template_write_local_unclear_ref0 = """\ { PyObject *old = %(identifier)s; %(identifier)s = %(tmp_name)s; Py_XDECREF( old ); } """ template_write_local_unclear_ref1 = """\ { PyObject *old = %(identifier)s; %(identifier)s = %(tmp_name)s; Py_INCREF( %(identifier)s ); Py_XDECREF( old ); } """ template_write_local_empty_ref0 = """\ assert( %(identifier)s == NULL ); %(identifier)s = %(tmp_name)s; """ template_write_local_empty_ref1 = """\ assert( %(identifier)s == NULL ); Py_INCREF( %(tmp_name)s ); %(identifier)s = %(tmp_name)s; """ template_write_local_clear_ref0 = """\ { PyObject *old = %(identifier)s; assert( old != NULL ); %(identifier)s = %(tmp_name)s; Py_DECREF( old ); } """ template_write_local_inplace = """\ %(identifier)s = %(tmp_name)s; """ template_write_shared_inplace = """\ PyCell_SET( %(identifier)s, %(tmp_name)s ); """ template_write_local_clear_ref1 = """\ { PyObject *old = %(identifier)s; assert( old != NULL ); %(identifier)s = %(tmp_name)s; Py_INCREF( %(identifier)s ); Py_DECREF( old ); } """ template_write_shared_unclear_ref0 = """\ { PyObject *old = PyCell_GET( %(identifier)s ); PyCell_SET( %(identifier)s, %(tmp_name)s ); Py_XDECREF( old ); } """ template_write_shared_unclear_ref1 = """\ { PyObject *old = PyCell_GET( %(identifier)s ); PyCell_SET( %(identifier)s, %(tmp_name)s ); Py_INCREF( %(tmp_name)s ); Py_XDECREF( old ); } """ template_write_shared_clear_ref0 = """\ assert( PyCell_GET( %(identifier)s ) == NULL ); PyCell_SET( %(identifier)s, %(tmp_name)s ); """ template_write_shared_clear_ref1 = """\ assert( PyCell_GET( %(identifier)s ) == NULL ); Py_INCREF( %(tmp_name)s ); PyCell_SET( %(identifier)s, %(tmp_name)s ); """ template_read_local = """\ %(tmp_name)s = %(identifier)s; """ template_del_local_tolerant = """\ Py_XDECREF( %(identifier)s ); %(identifier)s = NULL; """ template_del_shared_tolerant = """\ if ( %(identifier)s ) { Py_XDECREF( PyCell_GET( %(identifier)s )); PyCell_SET( %(identifier)s, NULL ); } """ template_del_local_intolerant = """\ %(result)s = %(identifier)s != NULL; if ( %(result)s == true ) { Py_DECREF( %(identifier)s ); %(identifier)s = NULL; } """ template_del_shared_intolerant = """\ assert( %(identifier)s != NULL ); %(result)s = PyCell_GET( %(identifier)s ) != NULL; if ( %(result)s == true ) { Py_DECREF( PyCell_GET( %(identifier)s ) ); PyCell_SET( %(identifier)s, NULL ); } """ template_del_local_known = """\ CHECK_OBJECT( %(identifier)s ); Py_DECREF( %(identifier)s ); %(identifier)s = NULL; """ template_del_shared_known = """\ Py_DECREF( PyCell_GET( %(identifier)s ) ); PyCell_SET( %(identifier)s, NULL ); """ # TODO: We could know, if we could loop, and only set the # variable to NULL then, using a different template. template_release_unclear = """\ Py_XDECREF( %(identifier)s ); %(identifier)s = NULL; """ template_release_clear = """\ CHECK_OBJECT( (PyObject *)%(identifier)s ); Py_DECREF( %(identifier)s ); %(identifier)s = NULL; """ # TODO: Storage will not be NULL. template_read_shared_unclear = """\ if ( %(identifier)s == NULL ) { %(tmp_name)s = NULL; } else { %(tmp_name)s = PyCell_GET( %(identifier)s ); } """ template_read_shared_known = """\ %(tmp_name)s = PyCell_GET( %(identifier)s ); """ # For module variable values, need to lookup in module dictionary or in # built-in dictionary. # TODO: Only provide fallback for known actually possible values. Do this # by keeping track of things that were added by "site.py" mechanisms. Then # we can avoid the second call entirely for most cases. template_read_mvar_unclear = """\ %(tmp_name)s = GET_STRING_DICT_VALUE( moduledict_%(module_identifier)s, (Nuitka_StringObject *)%(var_name)s ); if (unlikely( %(tmp_name)s == NULL )) { %(tmp_name)s = GET_STRING_DICT_VALUE( dict_builtin, (Nuitka_StringObject *)%(var_name)s ); } """ template_read_maybe_local_unclear = """\ %(tmp_name)s = PyDict_GetItem( %(locals_dict)s, %(var_name)s ); if ( %(tmp_name)s == NULL ) { %(fallback)s } """ template_del_global_unclear = """\ %(res_name)s = PyDict_DelItem( (PyObject *)moduledict_%(module_identifier)s, %(var_name)s ); if ( %(res_name)s == -1 ) CLEAR_ERROR_OCCURRED(); """ template_update_locals_dict_value = """\ if ( %(test_code)s ) { UPDATE_STRING_DICT0( (PyDictObject *)%(dict_name)s, (Nuitka_StringObject *)%(var_name)s, %(access_code)s ); } else { int res = PyDict_DelItem( %(dict_name)s, %(var_name)s ); if ( res != 0 ) { CLEAR_ERROR_OCCURRED(); } } """ template_set_locals_dict_value = """\ if ( %(test_code)s ) { int res = PyDict_SetItem( %(dict_name)s, %(var_name)s, %(access_code)s ); assert( res == 0 ); } """ template_update_locals_mapping_value = """\ if ( %(test_code)s ) { int res = PyObject_SetItem( %(mapping_name)s, %(var_name)s, %(access_code)s ); %(tmp_name)s = res == 0; } else { PyObject *test_value = PyObject_GetItem( %(mapping_name)s, %(var_name)s ); if ( test_value ) { Py_DECREF( test_value ); int res = PyObject_DelItem( %(mapping_name)s, %(var_name)s ); %(tmp_name)s = res == 0; } else { CLEAR_ERROR_OCCURRED(); %(tmp_name)s = true; } } """ template_set_locals_mapping_value = """\ if ( %(test_code)s ) { %(tmp_name)s = SET_SUBSCRIPT( %(mapping_name)s, %(var_name)s, %(access_code)s ); } else { %(tmp_name)s = true; } """ # TODO: Unused now. template_assign_from_frame_locals = """\ if ( %(frame_identifier)s->f_locals == NULL ) { %(frame_identifier)s->f_locals = PyDict_New(); } %(to_name)s = %(frame_identifier)s->f_locals; """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesCoroutines.py0000644000372000001440000000450613134660221025541 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Coroutines function (await/async) related templates. """ template_coroutine_object_decl_template = """\ static void %(function_identifier)s( struct Nuitka_CoroutineObject *coroutine ); """ template_coroutine_object_body_template = """ static void %(function_identifier)s( struct Nuitka_CoroutineObject *coroutine ) { CHECK_OBJECT( (PyObject *)coroutine ); assert( Nuitka_Coroutine_Check( (PyObject *)coroutine ) ); // Local variable initialization %(function_var_inits)s // Actual function code. %(function_body)s %(coroutine_exit)s } """ template_coroutine_exception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); function_exception_exit: %(function_cleanup)s\ assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); coroutine->m_yielded = NULL; return; """ template_coroutine_noexception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); %(function_cleanup)s\ coroutine->m_yielded = NULL; return; """ template_coroutine_return_exit = """\ function_return_exit:; coroutine->m_yielded = NULL; coroutine->m_returned = tmp_return_value; return; """ template_make_coroutine_template = """ %(to_name)s = Nuitka_Coroutine_New( %(coroutine_identifier)s, self->m_name, self->m_qualname, %(code_identifier)s, %(closure_count)d ); %(closure_copy)s """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesCalls.py0000644000372000001440000005235113134660221024446 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for calling functions with positional args only very quickly. """ template_call_function_with_args_decl = """\ extern PyObject *CALL_FUNCTION_WITH_ARGS%(args_count)d( PyObject *called, PyObject **args );""" template_call_function_with_args_impl = """\ PyObject *CALL_FUNCTION_WITH_ARGS%(args_count)d( PyObject *called, PyObject **args ) { CHECK_OBJECT( called ); // Check if arguments are valid objects in debug mode. #ifndef __NUITKA_NO_ASSERT__ for( size_t i = 0; i < %(args_count)d; i++ ) { CHECK_OBJECT( args[ i ] ); } #endif if ( Nuitka_Function_Check( called ) ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } struct Nuitka_FunctionObject *function = (struct Nuitka_FunctionObject *)called; PyObject *result; if ( function->m_args_simple && %(args_count)d == function->m_args_positional_count ) { for( Py_ssize_t i = 0; i < %(args_count)d; i++ ) { Py_INCREF( args[ i ] ); } result = function->m_c_code( function, args ); } else if ( function->m_args_simple && %(args_count)d + function->m_defaults_given == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_positional_count ); #else PyObject *python_pars[ function->m_args_positional_count ]; #endif memcpy( python_pars, args, %(args_count)d * sizeof(PyObject *) ); memcpy( python_pars + %(args_count)d, &PyTuple_GET_ITEM( function->m_defaults, 0 ), function->m_defaults_given * sizeof(PyObject *) ); for( Py_ssize_t i = 0; i < function->m_args_positional_count; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsPos( function, python_pars, args, %(args_count)d )) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } else if ( Nuitka_Method_Check( called ) ) { struct Nuitka_MethodObject *method = (struct Nuitka_MethodObject *)called; // Unbound method without arguments, let the error path be slow. if ( method->m_object != NULL ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } struct Nuitka_FunctionObject *function = method->m_function; PyObject *result; if ( function->m_args_simple && %(args_count)d + 1 == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_positional_count ); #else PyObject *python_pars[ function->m_args_positional_count ]; #endif python_pars[ 0 ] = method->m_object; Py_INCREF( method->m_object ); for( Py_ssize_t i = 0; i < %(args_count)d; i++ ) { python_pars[ i + 1 ] = args[ i ]; Py_INCREF( args[ i ] ); } result = function->m_c_code( function, python_pars ); } else if ( function->m_args_simple && %(args_count)d + 1 + function->m_defaults_given == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_positional_count ); #else PyObject *python_pars[ function->m_args_positional_count ]; #endif python_pars[ 0 ] = method->m_object; Py_INCREF( method->m_object ); memcpy( python_pars+1, args, %(args_count)d * sizeof(PyObject *) ); memcpy( python_pars+1 + %(args_count)d, &PyTuple_GET_ITEM( function->m_defaults, 0 ), function->m_defaults_given * sizeof(PyObject *) ); for( Py_ssize_t i = 1; i < function->m_args_overall_count; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsMethodPos( function, python_pars, method->m_object, args, %(args_count)d ) ) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } } else if ( PyCFunction_Check( called ) ) { // Try to be fast about wrapping the arguments. int flags = PyCFunction_GET_FLAGS( called ) & ~(METH_CLASS | METH_STATIC | METH_COEXIST); if ( flags & METH_NOARGS ) { #if %(args_count)d == 0 PyCFunction method = PyCFunction_GET_FUNCTION( called ); PyObject *self = PyCFunction_GET_SELF( called ); // Recursion guard is not strictly necessary, as we already have // one on our way to here. #ifdef _NUITKA_FULL_COMPAT if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } #endif PyObject *result = (*method)( self, NULL ); #ifdef _NUITKA_FULL_COMPAT Py_LeaveRecursiveCall(); #endif if ( result != NULL ) { // Some buggy C functions do set an error, but do not indicate it // and Nuitka inner workings can get upset/confused from it. DROP_ERROR_OCCURRED(); return result; } else { // Other buggy C functions do this, return NULL, but with // no error set, not allowed. if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in PyObject_Call" ); } return NULL; } #else PyErr_Format( PyExc_TypeError, "%%s() takes no arguments (%(args_count)d given)", ((PyCFunctionObject *)called)->m_ml->ml_name ); return NULL; #endif } else if ( flags & METH_O ) { #if %(args_count)d == 1 PyCFunction method = PyCFunction_GET_FUNCTION( called ); PyObject *self = PyCFunction_GET_SELF( called ); // Recursion guard is not strictly necessary, as we already have // one on our way to here. #ifdef _NUITKA_FULL_COMPAT if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } #endif PyObject *result = (*method)( self, args[0] ); #ifdef _NUITKA_FULL_COMPAT Py_LeaveRecursiveCall(); #endif if ( result != NULL ) { // Some buggy C functions do set an error, but do not indicate it // and Nuitka inner workings can get upset/confused from it. DROP_ERROR_OCCURRED(); return result; } else { // Other buggy C functions do this, return NULL, but with // no error set, not allowed. if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in PyObject_Call" ); } return NULL; } #else PyErr_Format(PyExc_TypeError, "%%s() takes exactly one argument (%(args_count)d given)", ((PyCFunctionObject *)called)->m_ml->ml_name ); return NULL; #endif } else if ( flags & METH_VARARGS ) { PyCFunction method = PyCFunction_GET_FUNCTION( called ); PyObject *self = PyCFunction_GET_SELF( called ); PyObject *pos_args = MAKE_TUPLE( args, %(args_count)d ); PyObject *result; // Recursion guard is not strictly necessary, as we already have // one on our way to here. #ifdef _NUITKA_FULL_COMPAT if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } #endif #if PYTHON_VERSION < 360 if ( flags & METH_KEYWORDS ) { result = (*(PyCFunctionWithKeywords)method)( self, pos_args, NULL ); } else { result = (*method)( self, pos_args ); } #else if ( flags == ( METH_VARARGS | METH_KEYWORDS ) ) { result = (*(PyCFunctionWithKeywords)method)( self, pos_args, NULL ); } else if ( flags == METH_FASTCALL ) { result = (*(_PyCFunctionFast)method)( self, &PyTuple_GET_ITEM( pos_args, 0 ), %(args_count)d, NULL );; } else { result = (*method)( self, pos_args ); } #endif #ifdef _NUITKA_FULL_COMPAT Py_LeaveRecursiveCall(); #endif if ( result != NULL ) { // Some buggy C functions do set an error, but do not indicate it // and Nuitka inner workings can get upset/confused from it. DROP_ERROR_OCCURRED(); Py_DECREF( pos_args ); return result; } else { // Other buggy C functions do this, return NULL, but with // no error set, not allowed. if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in PyObject_Call" ); } Py_DECREF( pos_args ); return NULL; } } } else if ( PyFunction_Check( called ) ) { return callPythonFunction( called, args, %(args_count)d ); } PyObject *pos_args = MAKE_TUPLE( args, %(args_count)d ); PyObject *result = CALL_FUNCTION( called, pos_args, NULL ); Py_DECREF( pos_args ); return result; } """ template_call_method_with_args_decl = """\ extern PyObject *CALL_METHOD_WITH_ARGS%(args_count)d( PyObject *source, PyObject *attr_name, PyObject **args );\ """ template_call_method_with_args_impl = """\ PyObject *CALL_METHOD_WITH_ARGS%(args_count)d( PyObject *source, PyObject *attr_name, PyObject **args ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); // Check if arguments are valid objects in debug mode. #ifndef __NUITKA_NO_ASSERT__ for( size_t i = 0; i < %(args_count)d; i++ ) { CHECK_OBJECT( args[ i ] ); } #endif PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro == PyObject_GenericGetAttr ) { // Unfortunately this is required, although of cause rarely necessary. if (unlikely( type->tp_dict == NULL )) { if (unlikely( PyType_Ready( type ) < 0 )) { return NULL; } } PyObject *descr = _PyType_Lookup( type, attr_name ); descrgetfunc func = NULL; if ( descr != NULL ) { Py_INCREF( descr ); #if PYTHON_VERSION < 300 if ( PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) ) { #endif func = Py_TYPE( descr )->tp_descr_get; if ( func != NULL && PyDescr_IsData( descr ) ) { PyObject *called_object = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); Py_DECREF( called_object ); return result; } #if PYTHON_VERSION < 300 } #endif } Py_ssize_t dictoffset = type->tp_dictoffset; PyObject *dict = NULL; if ( dictoffset != 0 ) { // Negative dictionary offsets have special meaning. if ( dictoffset < 0 ) { Py_ssize_t tsize; size_t size; tsize = ((PyVarObject *)source)->ob_size; if (tsize < 0) tsize = -tsize; size = _PyObject_VAR_SIZE( type, tsize ); dictoffset += (long)size; } PyObject **dictptr = (PyObject **) ((char *)source + dictoffset); dict = *dictptr; } if ( dict != NULL ) { CHECK_OBJECT( dict ); Py_INCREF( dict ); PyObject *called_object = PyDict_GetItem( dict, attr_name ); if ( called_object != NULL ) { Py_INCREF( called_object ); Py_XDECREF( descr ); Py_DECREF( dict ); PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); Py_DECREF( called_object ); return result; } Py_DECREF( dict ); } if ( func != NULL ) { if ( func == Nuitka_Function_Type.tp_descr_get ) { PyObject *result = Nuitka_CallMethodFunctionPosArgs( (struct Nuitka_FunctionObject const *)descr, source, args, %(args_count)d ); Py_DECREF( descr ); return result; } else { PyObject *called_object = func( descr, source, (PyObject *)type ); CHECK_OBJECT( called_object ); Py_DECREF( descr ); PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); Py_DECREF( called_object ); return result; } } if ( descr != NULL ) { CHECK_OBJECT( descr ); return CALL_FUNCTION_WITH_ARGS%(args_count)d( descr, args ); } #if PYTHON_VERSION < 300 PyErr_Format( PyExc_AttributeError, "'%%s' object has no attribute '%%s'", type->tp_name, PyString_AS_STRING( attr_name ) ); #else PyErr_Format( PyExc_AttributeError, "'%%s' object has no attribute '%%U'", type->tp_name, attr_name ); #endif return NULL; } #if PYTHON_VERSION < 300 else if ( type == &PyInstance_Type ) { PyInstanceObject *source_instance = (PyInstanceObject *)source; // The special cases have their own variant on the code generation level // as we are called with constants only. assert( attr_name != const_str_plain___dict__ ); assert( attr_name != const_str_plain___class__ ); // Try the instance dict first. PyObject *called_object = GET_STRING_DICT_VALUE( (PyDictObject *)source_instance->in_dict, (PyStringObject *)attr_name ); // Note: The "called_object" was found without taking a reference, // so we need not release it in this branch. if ( called_object != NULL ) { return CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); } // Then check the class dictionaries. called_object = FIND_ATTRIBUTE_IN_CLASS( source_instance->in_class, attr_name ); // Note: The "called_object" was found without taking a reference, // so we need not release it in this branch. if ( called_object != NULL ) { descrgetfunc descr_get = Py_TYPE( called_object )->tp_descr_get; if ( descr_get == Nuitka_Function_Type.tp_descr_get ) { return Nuitka_CallMethodFunctionPosArgs( (struct Nuitka_FunctionObject const *)called_object, source, args, %(args_count)d ); } else if ( descr_get != NULL ) { PyObject *method = descr_get( called_object, source, (PyObject *)source_instance->in_class ); if (unlikely( method == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( method, args ); Py_DECREF( method ); return result; } else { return CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); } } else if (unlikely( source_instance->in_class->cl_getattr == NULL )) { PyErr_Format( PyExc_AttributeError, "%%s instance has no attribute '%%s'", PyString_AS_STRING( source_instance->in_class->cl_name ), PyString_AS_STRING( attr_name ) ); return NULL; } else { // Finally allow the "__getattr__" override to provide it or else // it's an error. PyObject *args2[] = { source, attr_name }; called_object = CALL_FUNCTION_WITH_ARGS2( source_instance->in_class->cl_getattr, args2 ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); Py_DECREF( called_object ); return result; } } #endif else if ( type->tp_getattro != NULL ) { PyObject *called_object = (*type->tp_getattro)( source, attr_name ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); Py_DECREF( called_object ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *called_object = (*type->tp_getattr)( source, Nuitka_String_AsString_Unchecked( attr_name ) ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_ARGS%(args_count)d( called_object, args ); Py_DECREF( called_object ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%%s' object has no attribute '%%s'", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return NULL; } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesConstants.py0000644000372000001440000000473513112214770025367 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the constants handling. """ template_constants_reading = """ #include "nuitka/prelude.h" // Sentinel PyObject to be used for all our call iterator endings. It will // become a PyCObject pointing to NULL. It's address is unique, and that's // enough for us to use it as sentinel value. PyObject *_sentinel_value = NULL; %(constant_declarations)s static void _createGlobalConstants( void ) { NUITKA_MAY_BE_UNUSED PyObject *exception_type, *exception_value; NUITKA_MAY_BE_UNUSED PyTracebackObject *exception_tb; #ifdef _MSC_VER // Prevent unused warnings in case of simple programs, the attribute // NUITKA_MAY_BE_UNUSED doesn't work for MSVC. (void *)exception_type; (void *)exception_value; (void *)exception_tb; #endif %(constant_inits)s #if _NUITKA_EXE /* Set the "sys.executable" path to the original CPython executable. */ PySys_SetObject( (char *)"executable", %(sys_executable)s ); #endif } // In debug mode we can check that the constants were not tampered with in any // given moment. We typically do it at program exit, but we can add extra calls // for sanity. #ifndef __NUITKA_NO_ASSERT__ void checkGlobalConstants( void ) { %(constant_checks)s } #endif void createGlobalConstants( void ) { if ( _sentinel_value == NULL ) { #if PYTHON_VERSION < 300 _sentinel_value = PyCObject_FromVoidPtr( NULL, NULL ); #else // The NULL value is not allowed for a capsule, so use something else. _sentinel_value = PyCapsule_New( (void *)27, "sentinel", NULL ); #endif assert( _sentinel_value ); _createGlobalConstants(); } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesModules.py0000644000372000001440000002142713207537242025027 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Main module code templates This for the main program in case of executables, the module templates and stuff related to importing, and of course the generated code license. """ template_global_copyright = """\ /* Generated code for Python source for module '%(name)s' * created by Nuitka version %(version)s * * This code is in part copyright %(year)s Kay Hayen. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ """ template_module_body_template = """ #include "nuitka/prelude.h" #include "__helpers.h" /* The _module_%(module_identifier)s is a Python object pointer of module type. */ /* Note: For full compatibility with CPython, every module variable access * needs to go through it except for cases where the module cannot possibly * have changed in the mean time. */ PyObject *module_%(module_identifier)s; PyDictObject *moduledict_%(module_identifier)s; /* The module constants used, if any. */ %(constant_decl_codes)s static bool constants_created = false; static void createModuleConstants( void ) { %(constant_init_codes)s constants_created = true; } #ifndef __NUITKA_NO_ASSERT__ void checkModuleConstants_%(module_identifier)s( void ) { // The module may not have been used at all. if (constants_created == false) return; %(constant_check_codes)s } #endif // The module code objects. %(module_code_objects_decl)s static void createModuleCodeObjects(void) { %(module_code_objects_init)s } // The module function declarations. %(module_functions_decl)s // The module function definitions. %(module_functions_code)s #if PYTHON_VERSION >= 300 static struct PyModuleDef mdef_%(module_identifier)s = { PyModuleDef_HEAD_INIT, "%(module_name)s", /* m_name */ NULL, /* m_doc */ -1, /* m_size */ NULL, /* m_methods */ NULL, /* m_reload */ NULL, /* m_traverse */ NULL, /* m_clear */ NULL, /* m_free */ }; #endif #if PYTHON_VERSION >= 300 extern PyObject *metapath_based_loader; #endif #if PYTHON_VERSION >= 330 extern PyObject *const_str_plain___loader__; #endif extern void _initCompiledCellType(); extern void _initCompiledGeneratorType(); extern void _initCompiledFunctionType(); extern void _initCompiledMethodType(); extern void _initCompiledFrameType(); #if PYTHON_VERSION >= 350 extern void _initCompiledCoroutineTypes(); #endif #if PYTHON_VERSION >= 360 extern void _initCompiledAsyncgenTypes(); #endif // The exported interface to CPython. On import of the module, this function // gets called. It has to have an exact function name, in cases it's a shared // library export. This is hidden behind the MOD_INIT_DECL. MOD_INIT_DECL( %(module_identifier)s ) { #if defined(_NUITKA_EXE) || PYTHON_VERSION >= 300 static bool _init_done = false; // Modules might be imported repeatedly, which is to be ignored. if ( _init_done ) { return MOD_RETURN_VALUE( module_%(module_identifier)s ); } else { _init_done = true; } #endif #ifdef _NUITKA_MODULE // In case of a stand alone extension module, need to call initialization // the init here because that's the first and only time we are going to get // called here. // Initialize the constant values used. _initBuiltinModule(); createGlobalConstants(); /* Initialize the compiled types of Nuitka. */ _initCompiledCellType(); _initCompiledGeneratorType(); _initCompiledFunctionType(); _initCompiledMethodType(); _initCompiledFrameType(); #if PYTHON_VERSION >= 350 _initCompiledCoroutineTypes(); #endif #if PYTHON_VERSION >= 360 _initCompiledAsyncgenTypes(); #endif #if PYTHON_VERSION < 300 _initSlotCompare(); #endif #if PYTHON_VERSION >= 270 _initSlotIternext(); #endif patchBuiltinModule(); patchTypeComparison(); // Enable meta path based loader if not already done. setupMetaPathBasedLoader(); #if PYTHON_VERSION >= 300 patchInspectModule(); #endif #endif /* The constants only used by this module are created now. */ #ifdef _NUITKA_TRACE puts("%(module_name)s: Calling createModuleConstants()."); #endif createModuleConstants(); /* The code objects used by this module are created now. */ #ifdef _NUITKA_TRACE puts("%(module_name)s: Calling createModuleCodeObjects()."); #endif createModuleCodeObjects(); // puts( "in init%(module_identifier)s" ); // Create the module object first. There are no methods initially, all are // added dynamically in actual code only. Also no "__doc__" is initially // set at this time, as it could not contain NUL characters this way, they // are instead set in early module code. No "self" for modules, we have no // use for it. #if PYTHON_VERSION < 300 module_%(module_identifier)s = Py_InitModule4( "%(module_name)s", // Module Name NULL, // No methods initially, all are added // dynamically in actual module code only. NULL, // No __doc__ is initially set, as it could // not contain NUL this way, added early in // actual code. NULL, // No self for modules, we don't use it. PYTHON_API_VERSION ); #else module_%(module_identifier)s = PyModule_Create( &mdef_%(module_identifier)s ); #endif moduledict_%(module_identifier)s = MODULE_DICT( module_%(module_identifier)s ); CHECK_OBJECT( module_%(module_identifier)s ); // Seems to work for Python2.7 out of the box, but for Python3, the module // doesn't automatically enter "sys.modules", so do it manually. #if PYTHON_VERSION >= 300 { int r = PyObject_SetItem( PySys_GetObject( (char *)"modules" ), %(module_name_obj)s, module_%(module_identifier)s ); assert( r != -1 ); } #endif // For deep importing of a module we need to have "__builtins__", so we set // it ourselves in the same way than CPython does. Note: This must be done // before the frame object is allocated, or else it may fail. if ( GET_STRING_DICT_VALUE( moduledict_%(module_identifier)s, (Nuitka_StringObject *)const_str_plain___builtins__ ) == NULL ) { PyObject *value = (PyObject *)builtin_module; // Check if main module, not a dict then but the module itself. #if !defined(_NUITKA_EXE) || !%(is_main_module)s value = PyModule_GetDict( value ); #endif UPDATE_STRING_DICT0( moduledict_%(module_identifier)s, (Nuitka_StringObject *)const_str_plain___builtins__, value ); } #if PYTHON_VERSION >= 330 UPDATE_STRING_DICT0( moduledict_%(module_identifier)s, (Nuitka_StringObject *)const_str_plain___loader__, metapath_based_loader ); #endif // Temp variables if any %(temps_decl)s // Module code. %(module_code)s return MOD_RETURN_VALUE( module_%(module_identifier)s ); %(module_exit)s """ template_module_exception_exit = """\ module_exception_exit: RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return MOD_RETURN_VALUE( NULL ); }""" template_module_noexception_exit = """\ }""" template_helper_impl_decl = """\ // This file contains helper functions that are automatically created from // templates. #include "nuitka/prelude.h" extern PyObject *callPythonFunction( PyObject *func, PyObject **args, int count ); """ template_header_guard = """\ #ifndef %(header_guard_name)s #define %(header_guard_name)s %(header_body)s #endif """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesFreezer.py0000644000372000001440000000411413112214770025004 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for code to load frozen bytecode. """ template_frozen_modules = """\ // This provides the frozen (compiled bytecode) files that are included if // any. #include #include "nuitka/constants_blob.h" // Blob from which modules are unstreamed. #define stream_data constant_bin // These modules should be loaded as bytecode. They may e.g. have to be loadable // during "Py_Initialize" already, or for irrelevance, they are only included // in this un-optimized form. These are not compiled by Nuitka, and therefore // are not accelerated at all, merely bundled with the binary or module, so // that CPython library can start out finding them. struct frozen_desc { char const *name; ssize_t start; int size; }; void copyFrozenModulesTo( struct _frozen *destination ) { struct frozen_desc frozen_modules[] = { %(frozen_modules)s { NULL, 0, 0 } }; struct frozen_desc *current = frozen_modules; for(;;) { destination->name = (char *)current->name; destination->code = (unsigned char *)&constant_bin[ current->start ]; destination->size = current->size; if (destination->name == NULL) break; current += 1; destination += 1; }; } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesFunction.py0000644000372000001440000000606113207537242025201 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Normal function (no generator, not yielding) related templates. """ template_function_make_declaration = """\ static PyObject *MAKE_FUNCTION_%(function_identifier)s( %(function_creation_arg_spec)s ); """ template_function_direct_declaration = """\ %(file_scope)s PyObject *impl_%(function_identifier)s( %(direct_call_arg_spec)s ); """ template_make_function_template = """ static PyObject *MAKE_FUNCTION_%(function_identifier)s( %(function_creation_args)s ) { struct Nuitka_FunctionObject *result = Nuitka_Function_New( %(function_impl_identifier)s, %(function_name_obj)s, #if PYTHON_VERSION >= 330 %(function_qualname_obj)s, #endif %(code_identifier)s, %(defaults)s, #if PYTHON_VERSION >= 300 %(kw_defaults)s, %(annotations)s, #endif %(module_identifier)s, %(function_doc)s, %(closure_count)d ); %(closure_copy)s return (PyObject *)result; } """ template_function_body = """\ static PyObject *impl_%(function_identifier)s( %(parameter_objects_decl)s ) { // Preserve error status for checks #ifndef __NUITKA_NO_ASSERT__ NUITKA_MAY_BE_UNUSED bool had_error = ERROR_OCCURRED(); #endif // Local variable declarations. %(function_locals)s // Actual function code. %(function_body)s %(function_exit)s } """ template_function_exception_exit = """\ function_exception_exit: %(function_cleanup)s\ assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return NULL; """ template_function_return_exit = """\ function_return_exit: %(function_cleanup)s CHECK_OBJECT( tmp_return_value ); assert( had_error || !ERROR_OCCURRED() ); return tmp_return_value; """ function_direct_body_template = """\ %(file_scope)s PyObject *impl_%(function_identifier)s( %(direct_call_arg_spec)s ) { #ifndef __NUITKA_NO_ASSERT__ NUITKA_MAY_BE_UNUSED bool had_error = ERROR_OCCURRED(); assert(!had_error); // Do not enter inlined functions with error set. #endif // Local variable declarations. %(function_locals)s // Actual function code. %(function_body)s %(function_exit)s } """ function_dict_setup = """\ // Locals dictionary setup. PyObject *%(locals_dict)s = PyDict_New(); """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesAsyncgens.py0000644000372000001440000000446313134660221025343 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Async generator (await/async + yield) related templates. """ template_asyncgen_object_decl_template = """\ static void %(function_identifier)s( struct Nuitka_AsyncgenObject *asyncgen ); """ template_asyncgen_object_body_template = """ static void %(function_identifier)s( struct Nuitka_AsyncgenObject *asyncgen ) { CHECK_OBJECT( (PyObject *)asyncgen ); assert( Nuitka_Asyncgen_Check( (PyObject *)asyncgen ) ); // Local variable initialization %(function_var_inits)s // Actual function code. %(function_body)s %(asyncgen_exit)s } """ template_asyncgen_exception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); function_exception_exit: %(function_cleanup)s\ assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); asyncgen->m_yielded = NULL; return; """ template_asyncgen_noexception_exit = """\ // Return statement must be present. NUITKA_CANNOT_GET_HERE( %(function_identifier)s ); %(function_cleanup)s\ asyncgen->m_yielded = NULL; return; """ template_asyncgen_return_exit = """\ function_return_exit:; asyncgen->m_yielded = NULL; asyncgen->m_status = status_Finished; return; """ template_make_asyncgen_template = """ %(to_name)s = Nuitka_Asyncgen_New( %(asyncgen_identifier)s, self->m_name, self->m_qualname, %(code_identifier)s, %(closure_count)d ); %(closure_copy)s """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/TemplateDebugWrapper.py0000644000372000001440000000467713122472300025025 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nuitka templates can have more checks that the normal '%' operation. This wraps strings with a class derived from "str" that does more checks. """ from nuitka.__past__ import iterItems from nuitka.Options import isDebug def enableDebug(globals_dict): templates = dict(globals_dict) class TemplateWrapper(object): """ Wrapper around templates. To better trace and control template usage. """ def __init__(self, name, value): self.name = name self.value = value def __str__(self): return self.value def __mod__(self, other): assert type(other) is dict, self.name for key in other.keys(): if "%%(%s)" % key not in self.value: from logging import warning warning( "Extra value '%s' provided to template '%s'.", key, self.name ) try: return self.value % other except KeyError as e: raise KeyError(self.name, *e.args) def split(self, sep): return self.value.split(sep) for template_name, template_value in iterItems(templates): # Ignore internal attribute like "__name__" that the module will also # have of course. if template_name.startswith('_'): continue if type(template_value) is str: globals_dict[template_name] = TemplateWrapper( template_name, template_value ) def checkDebug(globals_dict): if isDebug(): enableDebug(globals_dict) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesGeneratorFunction.py0000644000372000001440000000723713207537242027056 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Generator function (with yield) related templates. """ template_genfunc_yielder_decl_template = """\ #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO static PyObject *%(function_identifier)s_context( struct Nuitka_GeneratorObject *generator, PyObject *yield_return_value ); #else static void %(function_identifier)s_context( struct Nuitka_GeneratorObject *generator ); #endif """ template_genfunc_yielder_body_template = """ #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO struct %(function_identifier)s_locals { %(function_local_types)s }; #endif #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO static PyObject *%(function_identifier)s_context( struct Nuitka_GeneratorObject *generator, PyObject *yield_return_value ) #else static void %(function_identifier)s_context( struct Nuitka_GeneratorObject *generator ) #endif { CHECK_OBJECT( (PyObject *)generator ); assert( Nuitka_Generator_Check( (PyObject *)generator ) ); // Local variable initialization %(function_var_inits)s // Dispatch to yield based on return label index: %(function_dispatch)s // Actual function code. %(function_body)s %(generator_exit)s } """ template_generator_exception_exit = """\ %(function_cleanup)s\ #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO return NULL; #else generator->m_yielded = NULL; return; #endif function_exception_exit: %(function_cleanup)s\ assert( exception_type ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO return NULL; #else generator->m_yielded = NULL; return; #endif """ template_generator_noexception_exit = """\ // Return statement need not be present. %(function_cleanup)s\ #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO return NULL; #else generator->m_yielded = NULL; return; #endif """ template_generator_return_exit = """\ // The above won't return, but we need to make it clear to the compiler // as well, or else it will complain and/or generate inferior code. assert(false); return; function_return_exit: #if PYTHON_VERSION >= 330 if ( tmp_return_value != Py_None ) { PyObject *args[1] = { tmp_return_value }; PyObject *stop_value = CALL_FUNCTION_WITH_ARGS1( PyExc_StopIteration, args ); RESTORE_ERROR_OCCURRED( PyExc_StopIteration, stop_value, NULL ); Py_INCREF( PyExc_StopIteration ); } else { Py_DECREF( tmp_return_value ); } #endif #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO return NULL; #else generator->m_yielded = NULL; return; #endif """ template_generator_making = """\ %(to_name)s = Nuitka_Generator_New( %(generator_identifier)s_context, %(generator_module)s, %(generator_name_obj)s, #if PYTHON_VERSION >= 350 %(generator_qualname_obj)s, #endif %(code_identifier)s, %(closure_count)d ); %(closure_copy)s """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesExceptions.py0000644000372000001440000000404113134660221025522 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for handling exceptions. """ template_publish_exception_to_handler = """\ if ( %(keeper_tb)s == NULL ) { %(keeper_tb)s = %(tb_making)s; } else if ( %(keeper_lineno)s != 0 ) { %(keeper_tb)s = ADD_TRACEBACK( %(keeper_tb)s, %(frame_identifier)s, %(keeper_lineno)s ); } """ template_error_catch_quick_exception = """\ if ( %(condition)s ) { if ( !ERROR_OCCURRED() ) { exception_type = %(quick_exception)s; Py_INCREF( exception_type ); exception_value = NULL; exception_tb = NULL; } else { FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); } %(release_temps)s %(var_description_code)s %(line_number_code)s goto %(exception_exit)s; }""" template_error_catch_exception = """\ if ( %(condition)s ) { assert( ERROR_OCCURRED() ); FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(release_temps)s %(line_number_code)s %(var_description_code)s goto %(exception_exit)s; }""" template_error_format_string_exception = """\ if ( %(condition)s ) { %(release_temps)s %(set_exception)s %(line_number_code)s %(var_description_code)s goto %(exception_exit)s; } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesIterators.py0000644000372000001440000000467413134660221025371 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the iterator handling. """ template_iterator_check = """\ // Check if iterator has left-over elements. CHECK_OBJECT( %(iterator_name)s ); assert( HAS_ITERNEXT( %(iterator_name)s ) ); %(attempt_name)s = (*Py_TYPE( %(iterator_name)s )->tp_iternext)( %(iterator_name)s ); if (likely( %(attempt_name)s == NULL )) { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL ) { if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration )) { CLEAR_ERROR_OCCURRED(); } else { FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(release_temps_1)s %(var_description_code_1)s %(line_number_code_1)s goto %(exception_exit)s; } } } else { Py_DECREF( %(attempt_name)s ); // TODO: Could avoid PyErr_Format. #if PYTHON_VERSION < 300 PyErr_Format( PyExc_ValueError, "too many values to unpack" ); #else PyErr_Format( PyExc_ValueError, "too many values to unpack (expected %(count)d)" ); #endif FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(release_temps_2)s %(var_description_code_2)s %(line_number_code_2)s goto %(exception_exit)s; }""" template_loop_break_next = """\ if ( %(to_name)s == NULL ) { if ( CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED() ) { %(break_indicator_code)s goto %(break_target)s; } else { %(release_temps)s FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); %(var_description_code)s %(line_number_code)s goto %(exception_target)s; } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesFrames.py0000644000372000001440000001704413207537242024634 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code templates for frames of all kinds. """ template_frame_guard_cache_decl = """\ static struct Nuitka_FrameObject *cache_%(frame_identifier)s = NULL; """ template_frame_guard_frame_decl = """\ struct Nuitka_FrameObject *%(frame_identifier)s; """ # Frame in a function template_frame_guard_full_block = """\ MAKE_OR_REUSE_FRAME( cache_%(frame_identifier)s, %(code_identifier)s, %(module_identifier)s, %(locals_size)s ); %(frame_identifier)s = cache_%(frame_identifier)s; // Push the new frame as the currently active one. pushFrameStack( %(frame_identifier)s ); // Mark the frame object as in use, ref count 1 will be up for reuse. assert( Py_REFCNT( %(frame_identifier)s ) == 2 ); // Frame stack // Framed code: %(codes)s #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif // Put the previous frame back on top. popFrameStack(); goto %(no_exception_exit)s; """ template_frame_guard_full_return_handler = """\ %(frame_return_exit)s:; #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif // Put the previous frame back on top. popFrameStack(); goto %(return_exit)s; """ template_frame_attach_locals = """\ Nuitka_Frame_AttachLocals( (struct Nuitka_FrameObject *)%(frame_identifier)s, %(type_description)s%(frame_variable_refs)s ); """ template_frame_guard_full_exception_handler = """\ %(frame_exception_exit)s:; #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif if ( exception_tb == NULL ) { exception_tb = %(tb_making)s; } else if ( exception_tb->tb_frame != &%(frame_identifier)s->m_frame ) { exception_tb = ADD_TRACEBACK( exception_tb, %(frame_identifier)s, exception_lineno ); } // Attachs locals to frame if any. %(attach_locals)s // Release cached frame. if ( %(frame_identifier)s == cache_%(frame_identifier)s ) { Py_DECREF( %(frame_identifier)s ); } cache_%(frame_identifier)s = NULL; assertFrameObject( %(frame_identifier)s ); // Put the previous frame back on top. popFrameStack(); // Return the error. goto %(parent_exception_exit)s; """ # Frame for a module. TODO: Use it for functions called only once. # TODO: The once guard need not take a reference count in its frame class. template_frame_guard_once = """\ // Frame without reuse. %(frame_identifier)s = MAKE_MODULE_FRAME( %(code_identifier)s, %(module_identifier)s ); // Push the new frame as the currently active one, and we should be exclusively // owning it. pushFrameStack( %(frame_identifier)s ); assert( Py_REFCNT( %(frame_identifier)s ) == 2 ); // Framed code: %(codes)s // Restore frame exception if necessary. #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif popFrameStack(); assertFrameObject( %(frame_identifier)s ); goto %(no_exception_exit)s; %(frame_exception_exit)s:; #if %(needs_preserve)d RESTORE_FRAME_EXCEPTION( %(frame_identifier)s ); #endif if ( exception_tb == NULL ) { exception_tb = %(tb_making)s; } else if ( exception_tb->tb_frame != &%(frame_identifier)s->m_frame ) { exception_tb = ADD_TRACEBACK( exception_tb, %(frame_identifier)s, exception_lineno ); } // Put the previous frame back on top. popFrameStack(); // Return the error. goto %(parent_exception_exit)s; %(no_exception_exit)s:;""" # Frame in a generator, coroutine or asyncgen. template_frame_guard_generator = """\ MAKE_OR_REUSE_FRAME( %(frame_cache_identifier)s, %(code_identifier)s, %(module_identifier)s, %(locals_size)s ); %(context_identifier)s->m_frame = %(frame_cache_identifier)s; // Mark the frame object as in use, ref count 1 will be up for reuse. Py_INCREF( %(context_identifier)s->m_frame ); assert( Py_REFCNT( %(context_identifier)s->m_frame ) == 2 ); // Frame stack #if PYTHON_VERSION >= 340 %(context_identifier)s->m_frame->m_frame.f_gen = (PyObject *)%(context_identifier)s; #endif Py_CLEAR( %(context_identifier)s->m_frame->m_frame.f_back ); %(context_identifier)s->m_frame->m_frame.f_back = PyThreadState_GET()->frame; Py_INCREF( %(context_identifier)s->m_frame->m_frame.f_back ); PyThreadState_GET()->frame = &%(context_identifier)s->m_frame->m_frame; Py_INCREF( %(context_identifier)s->m_frame ); Nuitka_Frame_MarkAsExecuting( %(context_identifier)s->m_frame ); #if PYTHON_VERSION >= 300 // Accept currently existing exception as the one to publish again when we // yield or yield from. PyThreadState *thread_state = PyThreadState_GET(); %(context_identifier)s->m_frame->m_frame.f_exc_type = thread_state->exc_type; if ( %(context_identifier)s->m_frame->m_frame.f_exc_type == Py_None ) %(context_identifier)s->m_frame->m_frame.f_exc_type = NULL; Py_XINCREF( %(context_identifier)s->m_frame->m_frame.f_exc_type ); %(context_identifier)s->m_frame->m_frame.f_exc_value = thread_state->exc_value; Py_XINCREF( %(context_identifier)s->m_frame->m_frame.f_exc_value ); %(context_identifier)s->m_frame->m_frame.f_exc_traceback = thread_state->exc_traceback; Py_XINCREF( %(context_identifier)s->m_frame->m_frame.f_exc_traceback ); #endif // Framed code: %(codes)s Nuitka_Frame_MarkAsNotExecuting( %(context_identifier)s->m_frame ); #if PYTHON_VERSION >= 300 Py_CLEAR( %(context_identifier)s->m_frame->m_frame.f_exc_type ); Py_CLEAR( %(context_identifier)s->m_frame->m_frame.f_exc_value ); Py_CLEAR( %(context_identifier)s->m_frame->m_frame.f_exc_traceback ); #endif // Allow re-use of the frame again. Py_DECREF( %(context_identifier)s->m_frame ); goto %(no_exception_exit)s; """ # TODO: This cannot happen, can it? template_frame_guard_generator_return_handler = """\ %(frame_return_exit)s:; #if PYTHON_VERSION >= 300 Py_CLEAR( %(frame_identifier)s->m_frame.f_exc_type ); Py_CLEAR( %(frame_identifier)s->m_frame.f_exc_value ); Py_CLEAR( %(frame_identifier)s->m_frame.f_exc_traceback ); #endif Py_DECREF( %(frame_identifier)s ); goto %(return_exit)s; """ template_frame_guard_generator_exception_handler = """\ %(frame_exception_exit)s:; // If it's not an exit exception, consider and create a traceback for it. if ( !EXCEPTION_MATCH_GENERATOR( exception_type ) ) { if ( exception_tb == NULL ) { exception_tb = %(tb_making)s; } else if ( exception_tb->tb_frame != &%(frame_identifier)s->m_frame ) { exception_tb = ADD_TRACEBACK( exception_tb, %(frame_identifier)s, exception_lineno ); } %(attach_locals)s // Release cached frame. if ( %(frame_identifier)s == %(frame_cache_identifier)s ) { Py_DECREF( %(frame_identifier)s ); } %(frame_cache_identifier)s = NULL; assertFrameObject( %(frame_identifier)s ); } #if PYTHON_VERSION >= 300 Py_CLEAR( %(frame_identifier)s->m_frame.f_exc_type ); Py_CLEAR( %(frame_identifier)s->m_frame.f_exc_value ); Py_CLEAR( %(frame_identifier)s->m_frame.f_exc_traceback ); #endif Py_DECREF( %(frame_identifier)s ); // Return the error. goto %(parent_exception_exit)s; """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/templates/__init__.py0000644000372000001440000000150113112214770022464 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/codegen/templates/CodeTemplatesLoader.py0000644000372000001440000000424313112214770024613 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Templates for the loading of embedded modules. """ template_metapath_loader_compiled_module_entry = """\ { (char *)"%(module_name)s", MOD_INIT_NAME( %(module_identifier)s ), 0, 0, NUITKA_COMPILED_MODULE },""" template_metapath_loader_compiled_package_entry = """\ { (char *)"%(module_name)s", MOD_INIT_NAME( %(module_identifier)s ), 0, 0, NUITKA_PACKAGE_FLAG },""" template_metapath_loader_shlib_module_entry = """\ { (char *)"%(module_name)s", NULL, 0, 0, NUITKA_SHLIB_FLAG },""" template_metapath_loader_bytecode_module_entry = """\ { (char *)"%(module_name)s", NULL, %(bytecode)s, %(size)d, %(flags)s },""" template_metapath_loader_body = """\ /* Code to register embedded modules for meta path based loading if any. */ #include "nuitka/unfreezing.h" /* Table for lookup to find compiled or bytecode modules included in this * binary or module, or put along this binary as extension modules. We do * our own loading for each of these. */ %(metapath_module_decls)s static struct Nuitka_MetaPathBasedLoaderEntry meta_path_loader_entries[] = { %(metapath_loader_inittab)s { NULL, NULL, 0, 0, 0 } }; void setupMetaPathBasedLoader( void ) { static bool init_done = false; if ( init_done == false ) { registerMetaPathBasedUnfreezer( meta_path_loader_entries ); init_done = true; } } """ from . import TemplateDebugWrapper # isort:skip TemplateDebugWrapper.checkDebug(globals()) Nuitka-0.5.28.2/nuitka/codegen/Emission.py0000644000372000001440000000247713122472300020526 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Emission of source code. Code generation is driven via "emit", which is to receive lines of code and this is to collect them, providing the emit implementation. Sometimes nested use of these will occur. """ class SourceCodeCollector(object): def __init__(self): self.codes = [] def __call__(self, code): self.emit(code) def emit(self,code): for line in code.split('\n'): self.codes.append(line) def emitTo(self, emit): for code in self.codes: emit(code) self.codes = None Nuitka-0.5.28.2/nuitka/codegen/CodeGeneration.py0000644000372000001440000010044513207537242021633 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ The code generation. No language specifics at all are supposed to be present here. Instead it is using primitives from the given generator to build code sequences (list of strings). As such this is the place that knows how to take a condition and two code branches and make a code block out of it. But it doesn't contain any target language syntax. """ from nuitka.__past__ import iterItems from . import Contexts, Emission from .AsyncgenCodes import ( generateMakeAsyncgenObjectCode, getAsyncgenObjectCode, getAsyncgenObjectDeclCode ) from .AttributeCodes import ( generateAssignmentAttributeCode, generateAttributeLookupCode, generateAttributeLookupSpecialCode, generateBuiltinGetattrCode, generateBuiltinHasattrCode, generateBuiltinSetattrCode, generateDelAttributeCode ) from .BranchCodes import generateBranchCode from .BuiltinCodes import ( generateBuiltinAnonymousRefCode, generateBuiltinBinCode, generateBuiltinBoolCode, generateBuiltinBytearray1Code, generateBuiltinBytearray3Code, generateBuiltinClassmethodCode, generateBuiltinComplexCode, generateBuiltinFloatCode, generateBuiltinHexCode, generateBuiltinOctCode, generateBuiltinOpenCode, generateBuiltinRange1Code, generateBuiltinRange2Code, generateBuiltinRange3Code, generateBuiltinRefCode, generateBuiltinStaticmethodCode, generateBuiltinSum1Code, generateBuiltinSum2Code, generateBuiltinType1Code, generateBuiltinType3Code, generateBuiltinXrange1Code, generateBuiltinXrange2Code, generateBuiltinXrange3Code ) from .CallCodes import generateCallCode, getCallsCode, getCallsDecls from .ClassCodes import ( generateBuiltinIsinstanceCode, generateBuiltinSuperCode, generateSelectMetaclassCode ) from .CodeHelpers import ( generateStatementSequenceCode, setExpressionDispatchDict, setStatementDispatchDict ) from .ComparisonCodes import generateComparisonExpressionCode from .ConditionalCodes import ( generateConditionalAndOrCode, generateConditionalCode ) from .ConstantCodes import ( generateConstantEllipsisReferenceCode, generateConstantFalseReferenceCode, generateConstantNoneReferenceCode, generateConstantReferenceCode, generateConstantTrueReferenceCode ) from .CoroutineCodes import ( generateAsyncIterCode, generateAsyncNextCode, generateAsyncWaitCode, generateMakeCoroutineObjectCode, getCoroutineObjectCode, getCoroutineObjectDeclCode ) from .DictCodes import ( generateBuiltinDictCode, generateDictionaryCreationCode, generateDictOperationGetCode, generateDictOperationInCode, generateDictOperationRemoveCode, generateDictOperationSetCode, generateDictOperationUpdateCode ) from .EvalCodes import ( generateBuiltinCompileCode, generateEvalCode, generateExecCode, generateExecfileCode, generateLocalsDictSyncCode ) from .ExceptionCodes import ( generateBuiltinMakeExceptionCode, generateExceptionCaughtTracebackCode, generateExceptionCaughtTypeCode, generateExceptionCaughtValueCode, generateExceptionPublishCode, generateExceptionRefCode ) from .ExpressionCodes import ( generateExpressionOnlyCode, generateSideEffectsCode ) from .FrameCodes import ( generateFramePreserveExceptionCode, generateFrameRestoreExceptionCode ) from .FunctionCodes import ( generateFunctionCallCode, generateFunctionCreationCode, generateFunctionOutlineCode, getExportScopeCode, getFunctionCode, getFunctionDirectDecl ) from .GeneratorCodes import ( generateMakeGeneratorObjectCode, getGeneratorObjectCode, getGeneratorObjectDeclCode ) from .GlobalsLocalsCodes import ( generateBuiltinDir1Code, generateBuiltinGlobalsCode, generateBuiltinLocalsCode, generateBuiltinVarsCode, generateSetLocalsCode ) from .IdCodes import generateBuiltinHashCode, generateBuiltinIdCode from .ImportCodes import ( generateBuiltinImportCode, generateImportModuleHardCode, generateImportModuleNameHardCode, generateImportNameCode, generateImportStarCode ) from .IntegerCodes import generateBuiltinIntCode, generateBuiltinLongCode from .IteratorCodes import ( generateBuiltinIter1Code, generateBuiltinIter2Code, generateBuiltinLenCode, generateBuiltinNext1Code, generateBuiltinNext2Code, generateSpecialUnpackCode, generateUnpackCheckCode ) from .ListCodes import ( generateBuiltinListCode, generateListCreationCode, generateListOperationAppendCode, generateListOperationExtendCode, generateListOperationPopCode ) from .LoaderCodes import getMetapathLoaderBodyCode from .LoopCodes import ( generateLoopBreakCode, generateLoopCode, generateLoopContinueCode ) from .ModuleCodes import ( generateModuleFileAttributeCode, generateModuleLoaderRefCode, getModuleCode, getModuleValues ) from .OperationCodes import ( generateOperationBinaryCode, generateOperationUnaryCode ) from .PrintCodes import generatePrintNewlineCode, generatePrintValueCode from .RaisingCodes import generateRaiseCode, generateRaiseExpressionCode from .ReturnCodes import ( generateGeneratorReturnNoneCode, generateGeneratorReturnValueCode, generateReturnCode, generateReturnConstantCode, generateReturnedValueRefCode ) from .SetCodes import ( generateBuiltinFrozensetCode, generateBuiltinSetCode, generateSetCreationCode, generateSetLiteralCreationCode, generateSetOperationAddCode, generateSetOperationUpdateCode ) from .SliceCodes import ( generateAssignmentSliceCode, generateBuiltinSliceCode, generateDelSliceCode, generateSliceLookupCode ) from .StringCodes import ( generateBuiltinAsciiCode, generateBuiltinBytesCode, generateBuiltinChrCode, generateBuiltinFormatCode, generateBuiltinOrdCode, generateBuiltinStrCode, generateBuiltinUnicodeCode, generateStringContenationCode ) from .SubscriptCodes import ( generateAssignmentSubscriptCode, generateDelSubscriptCode, generateSubscriptLookupCode ) from .TryCodes import generateTryCode from .TupleCodes import generateBuiltinTupleCode, generateTupleCreationCode from .VariableCodes import ( generateAssignmentVariableCode, generateDelVariableCode, generateLocalsDictVariableRefCode, generateVariableReferenceCode, generateVariableReleaseCode ) from .YieldCodes import generateYieldCode, generateYieldFromCode _generated_functions = {} def generateFunctionBodyCode(function_body, context): function_identifier = function_body.getCodeName() if function_identifier in _generated_functions: return _generated_functions[function_identifier] # TODO: Generate both codes, and base direct/etc. decisions on context. if function_body.isExpressionGeneratorObjectBody(): function_context = Contexts.PythonGeneratorObjectContext( parent = context, function = function_body ) elif function_body.isExpressionClassBody(): function_context = Contexts.PythonFunctionDirectContext( parent = context, function = function_body ) elif function_body.isExpressionCoroutineObjectBody(): function_context = Contexts.PythonCoroutineObjectContext( parent = context, function = function_body ) elif function_body.isExpressionAsyncgenObjectBody(): function_context = Contexts.PythonAsyncgenObjectContext( parent = context, function = function_body ) elif function_body.needsCreation(): function_context = Contexts.PythonFunctionCreatedContext( parent = context, function = function_body ) else: function_context = Contexts.PythonFunctionDirectContext( parent = context, function = function_body ) needs_exception_exit = function_body.mayRaiseException(BaseException) if function_body.isExpressionGeneratorObjectBody(): function_code = getGeneratorObjectCode( context = function_context, function_identifier = function_identifier, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), outline_variables = function_body.getOutlineLocalVariables(), temp_variables = function_body.getTempVariables(), needs_exception_exit = needs_exception_exit, needs_generator_return = function_body.needsGeneratorReturnExit() ) elif function_body.isExpressionCoroutineObjectBody(): function_code = getCoroutineObjectCode( context = function_context, function_identifier = function_identifier, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), outline_variables = function_body.getOutlineLocalVariables(), temp_variables = function_body.getTempVariables(), needs_exception_exit = needs_exception_exit, needs_generator_return = function_body.needsGeneratorReturnExit() ) elif function_body.isExpressionAsyncgenObjectBody(): function_code = getAsyncgenObjectCode( context = function_context, function_identifier = function_identifier, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), outline_variables = function_body.getOutlineLocalVariables(), temp_variables = function_body.getTempVariables(), needs_exception_exit = needs_exception_exit, needs_generator_return = function_body.needsGeneratorReturnExit() ) elif function_body.isExpressionClassBody(): function_code = getFunctionCode( context = function_context, function_identifier = function_identifier, parameters = None, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), outline_variables = function_body.getOutlineLocalVariables(), temp_variables = function_body.getTempVariables(), function_doc = function_body.getDoc(), needs_exception_exit = needs_exception_exit, file_scope = getExportScopeCode( cross_module = False ) ) else: parameters = function_body.getParameters() function_code = getFunctionCode( context = function_context, function_identifier = function_identifier, parameters = parameters, closure_variables = function_body.getClosureVariables(), user_variables = function_body.getUserLocalVariables(), outline_variables = function_body.getOutlineLocalVariables(), temp_variables = function_body.getTempVariables(), function_doc = function_body.getDoc(), needs_exception_exit = needs_exception_exit, file_scope = getExportScopeCode( cross_module = function_body.isCrossModuleUsed() ) ) return function_code, function_context def _generateFunctionDeclCode(function_body, context): if function_body.isExpressionGeneratorObjectBody(): return getGeneratorObjectDeclCode( function_identifier = function_body.getCodeName(), ) elif function_body.isExpressionCoroutineObjectBody(): return getCoroutineObjectDeclCode( function_identifier = function_body.getCodeName(), ) elif function_body.isExpressionAsyncgenObjectBody(): return getAsyncgenObjectDeclCode( function_identifier = function_body.getCodeName(), ) elif function_body.isExpressionClassBody(): return getFunctionDirectDecl( function_identifier = function_body.getCodeName(), closure_variables = function_body.getClosureVariables(), file_scope = getExportScopeCode( cross_module = False ), context = context ) elif function_body.needsDirectCall(): return getFunctionDirectDecl( function_identifier = function_body.getCodeName(), closure_variables = function_body.getClosureVariables(), file_scope = getExportScopeCode( cross_module = function_body.isCrossModuleUsed() ), context = context ) else: return None def prepareModuleCode(global_context, module, module_name): # As this not only creates all modules, but also functions, it deals # also with its functions. assert module.isCompiledPythonModule(), module context = Contexts.PythonModuleContext( module = module, module_name = module_name, code_name = module.getCodeName(), filename = module.getFilename(), global_context = global_context ) context.setExceptionEscape("module_exception_exit") statement_sequence = module.getBody() codes = Emission.SourceCodeCollector() generateStatementSequenceCode( statement_sequence = statement_sequence, emit = codes, allow_none = True, context = context, ) function_decl_codes = [] function_body_codes = [] for function_body in module.getUsedFunctions(): function_code, function_context = generateFunctionBodyCode( function_body = function_body, context = context ) assert type(function_code) is str, type(function_code) function_body_codes.append(function_code) function_decl = _generateFunctionDeclCode( function_body = function_body, context = function_context ) if function_decl is not None: function_decl_codes.append(function_decl) # These are for functions used from other modules. Due to cyclic # dependencies, we cannot rely on those to be already created. for function_body in module.getCrossUsedFunctions(): assert function_body.isCrossModuleUsed() function_decl = getFunctionDirectDecl( function_identifier = function_body.getCodeName(), closure_variables = function_body.getClosureVariables(), file_scope = getExportScopeCode( cross_module = function_body.isCrossModuleUsed() ), context = Contexts.PythonFunctionDirectContext( parent = context, function = function_body ) ) function_decl_codes.append(function_decl) for _identifier, code in sorted(iterItems(context.getHelperCodes())): function_body_codes.append(code) for _identifier, code in sorted(iterItems(context.getDeclarations())): function_decl_codes.append(code) function_body_codes = "\n\n".join(function_body_codes) function_decl_codes = "\n\n".join(function_decl_codes) template_values = getModuleValues( module_name = module_name, module_identifier = module.getCodeName(), codes = codes.codes, function_decl_codes = function_decl_codes, function_body_codes = function_body_codes, temp_variables = module.getTempVariables(), outline_variables = module.getOutlineLocalVariables(), is_main_module = module.isMainModule(), is_internal_module = module.isInternalModule(), context = context ) return template_values, context def generateModuleCode(module_context, template_values): return getModuleCode( module_context = module_context, template_values = template_values ) def generateHelpersCode(other_modules): calls_decl_code = getCallsDecls() loader_code = getMetapathLoaderBodyCode(other_modules) calls_body_code = getCallsCode() return calls_decl_code, calls_body_code + loader_code def makeGlobalContext(): return Contexts.PythonGlobalContext() setExpressionDispatchDict( { "EXPRESSION_ATTRIBUTE_LOOKUP" : generateAttributeLookupCode, "EXPRESSION_ATTRIBUTE_LOOKUP_SPECIAL" : generateAttributeLookupSpecialCode, "EXPRESSION_BUILTIN_SLICE" : generateBuiltinSliceCode, "EXPRESSION_BUILTIN_HASH" : generateBuiltinHashCode, "EXPRESSION_BUILTIN_ID" : generateBuiltinIdCode, "EXPRESSION_BUILTIN_COMPILE" : generateBuiltinCompileCode, "EXPRESSION_BUILTIN_EXECFILE" : generateExecfileCode, "EXPRESSION_BUILTIN_EVAL" : generateEvalCode, "EXPRESSION_BUILTIN_EXEC" : generateEvalCode, "EXPRESSION_BUILTIN_ITER1" : generateBuiltinIter1Code, "EXPRESSION_BUILTIN_ITER2" : generateBuiltinIter2Code, "EXPRESSION_BUILTIN_NEXT1" : generateBuiltinNext1Code, "EXPRESSION_BUILTIN_NEXT2" : generateBuiltinNext2Code, "EXPRESSION_BUILTIN_SUM1" : generateBuiltinSum1Code, "EXPRESSION_BUILTIN_SUM2" : generateBuiltinSum2Code, "EXPRESSION_BUILTIN_TYPE1" : generateBuiltinType1Code, "EXPRESSION_BUILTIN_TYPE3" : generateBuiltinType3Code, "EXPRESSION_BUILTIN_IMPORT" : generateBuiltinImportCode, "EXPRESSION_BUILTIN_BOOL" : generateBuiltinBoolCode, "EXPRESSION_BUILTIN_BYTEARRAY1" : generateBuiltinBytearray1Code, "EXPRESSION_BUILTIN_BYTEARRAY3" : generateBuiltinBytearray3Code, "EXPRESSION_BUILTIN_INT" : generateBuiltinIntCode, "EXPRESSION_BUILTIN_LONG" : generateBuiltinLongCode, "EXPRESSION_BUILTIN_FLOAT" : generateBuiltinFloatCode, "EXPRESSION_BUILTIN_COMPLEX" : generateBuiltinComplexCode, "EXPRESSION_BUILTIN_LEN" : generateBuiltinLenCode, "EXPRESSION_BUILTIN_STR" : generateBuiltinStrCode, "EXPRESSION_BUILTIN_BYTES" : generateBuiltinBytesCode, "EXPRESSION_BUILTIN_UNICODE" : generateBuiltinUnicodeCode, "EXPRESSION_BUILTIN_CHR" : generateBuiltinChrCode, "EXPRESSION_BUILTIN_ORD" : generateBuiltinOrdCode, "EXPRESSION_BUILTIN_BIN" : generateBuiltinBinCode, "EXPRESSION_BUILTIN_OCT" : generateBuiltinOctCode, "EXPRESSION_BUILTIN_HEX" : generateBuiltinHexCode, "EXPRESSION_BUILTIN_TUPLE" : generateBuiltinTupleCode, "EXPRESSION_BUILTIN_LIST" : generateBuiltinListCode, "EXPRESSION_BUILTIN_SET" : generateBuiltinSetCode, "EXPRESSION_BUILTIN_FROZENSET" : generateBuiltinFrozensetCode, "EXPRESSION_BUILTIN_DICT" : generateBuiltinDictCode, "EXPRESSION_BUILTIN_LOCALS_UPDATED" : generateBuiltinLocalsCode, "EXPRESSION_BUILTIN_LOCALS_COPY" : generateBuiltinLocalsCode, "EXPRESSION_BUILTIN_GLOBALS" : generateBuiltinGlobalsCode, "EXPRESSION_BUILTIN_SUPER" : generateBuiltinSuperCode, "EXPRESSION_BUILTIN_ISINSTANCE" : generateBuiltinIsinstanceCode, "EXPRESSION_BUILTIN_DIR1" : generateBuiltinDir1Code, "EXPRESSION_BUILTIN_VARS" : generateBuiltinVarsCode, "EXPRESSION_BUILTIN_HASATTR" : generateBuiltinHasattrCode, "EXPRESSION_BUILTIN_GETATTR" : generateBuiltinGetattrCode, "EXPRESSION_BUILTIN_SETATTR" : generateBuiltinSetattrCode, "EXPRESSION_BUILTIN_OPEN" : generateBuiltinOpenCode, "EXPRESSION_BUILTIN_STATICMETHOD" : generateBuiltinStaticmethodCode, "EXPRESSION_BUILTIN_CLASSMETHOD" : generateBuiltinClassmethodCode, "EXPRESSION_BUILTIN_RANGE1" : generateBuiltinRange1Code, "EXPRESSION_BUILTIN_RANGE2" : generateBuiltinRange2Code, "EXPRESSION_BUILTIN_RANGE3" : generateBuiltinRange3Code, "EXPRESSION_BUILTIN_XRANGE1" : generateBuiltinXrange1Code, "EXPRESSION_BUILTIN_XRANGE2" : generateBuiltinXrange2Code, "EXPRESSION_BUILTIN_XRANGE3" : generateBuiltinXrange3Code, "EXPRESSION_BUILTIN_MAKE_EXCEPTION" : generateBuiltinMakeExceptionCode, "EXPRESSION_BUILTIN_REF" : generateBuiltinRefCode, "EXPRESSION_BUILTIN_EXCEPTION_REF" : generateExceptionRefCode, "EXPRESSION_BUILTIN_ANONYMOUS_REF" : generateBuiltinAnonymousRefCode, "EXPRESSION_CAUGHT_EXCEPTION_TYPE_REF" : generateExceptionCaughtTypeCode, "EXPRESSION_CAUGHT_EXCEPTION_VALUE_REF" : generateExceptionCaughtValueCode, "EXPRESSION_CAUGHT_EXCEPTION_TRACEBACK_REF" : generateExceptionCaughtTracebackCode, "EXPRESSION_CALL_EMPTY" : generateCallCode, "EXPRESSION_CALL_KEYWORDS_ONLY" : generateCallCode, "EXPRESSION_CALL_NO_KEYWORDS" : generateCallCode, "EXPRESSION_CALL" : generateCallCode, "EXPRESSION_CONSTANT_NONE_REF" : generateConstantNoneReferenceCode, "EXPRESSION_CONSTANT_TRUE_REF" : generateConstantTrueReferenceCode, "EXPRESSION_CONSTANT_FALSE_REF" : generateConstantFalseReferenceCode, "EXPRESSION_CONSTANT_STR_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_UNICODE_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_BYTES_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_INT_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_LONG_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_FLOAT_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_COMPLEX_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_ELLIPSIS_REF" : generateConstantEllipsisReferenceCode, "EXPRESSION_CONSTANT_DICT_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_DICT_EMPTY_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_TUPLE_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_TUPLE_EMPTY_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_LIST_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_LIST_EMPTY_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_SET_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_SET_EMPTY_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_FROZENSET_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_FROZENSET_EMPTY_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_SLICE_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_XRANGE_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_TYPE_REF" : generateConstantReferenceCode, "EXPRESSION_CONSTANT_BYTEARRAY_REF" : generateConstantReferenceCode, "EXPRESSION_CONDITIONAL" : generateConditionalCode, "EXPRESSION_CONDITIONAL_OR" : generateConditionalAndOrCode, "EXPRESSION_CONDITIONAL_AND" : generateConditionalAndOrCode, "EXPRESSION_COMPARISON" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_IS" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_IS_NOT" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_IN" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_NOT_IN" : generateComparisonExpressionCode, "EXPRESSION_COMPARISON_EXCEPTION_MATCH" : generateComparisonExpressionCode, "EXPRESSION_DICT_OPERATION_GET" : generateDictOperationGetCode, "EXPRESSION_DICT_OPERATION_IN" : generateDictOperationInCode, "EXPRESSION_DICT_OPERATION_NOT_IN" : generateDictOperationInCode, "EXPRESSION_FUNCTION_CREATION" : generateFunctionCreationCode, "EXPRESSION_FUNCTION_CALL" : generateFunctionCallCode, "EXPRESSION_IMPORT_MODULE_HARD" : generateImportModuleHardCode, "EXPRESSION_IMPORT_MODULE_NAME_HARD" : generateImportModuleNameHardCode, "EXPRESSION_IMPORT_NAME" : generateImportNameCode, "EXPRESSION_LIST_OPERATION_EXTEND" : generateListOperationExtendCode, "EXPRESSION_LIST_OPERATION_POP" : generateListOperationPopCode, "EXPRESSION_MODULE_FILE_ATTRIBUTE_REF" : generateModuleFileAttributeCode, "EXPRESSION_MODULE_LOADER_REF" : generateModuleLoaderRefCode, "EXPRESSION_MAKE_GENERATOR_OBJECT" : generateMakeGeneratorObjectCode, "EXPRESSION_MAKE_COROUTINE_OBJECT" : generateMakeCoroutineObjectCode, "EXPRESSION_MAKE_ASYNCGEN_OBJECT" : generateMakeAsyncgenObjectCode, "EXPRESSION_MAKE_SET" : generateSetCreationCode, "EXPRESSION_MAKE_SET_LITERAL" : generateSetLiteralCreationCode, "EXPRESSION_MAKE_TUPLE" : generateTupleCreationCode, "EXPRESSION_MAKE_LIST" : generateListCreationCode, "EXPRESSION_MAKE_DICT" : generateDictionaryCreationCode, "EXPRESSION_OPERATION_BINARY" : generateOperationBinaryCode, "EXPRESSION_OPERATION_BINARY_ADD" : generateOperationBinaryCode, "EXPRESSION_OPERATION_BINARY_MULT" : generateOperationBinaryCode, "EXPRESSION_OPERATION_BINARY_DIVMOD" : generateOperationBinaryCode, "EXPRESSION_OPERATION_BINARY_INPLACE" : generateOperationBinaryCode, "EXPRESSION_OPERATION_UNARY" : generateOperationUnaryCode, "EXPRESSION_OPERATION_NOT" : generateOperationUnaryCode, "EXPRESSION_OUTLINE_BODY" : generateFunctionOutlineCode, "EXPRESSION_OUTLINE_FUNCTION" : generateFunctionOutlineCode, # TODO: Rename to make more clear it is an outline "EXPRESSION_CLASS_BODY" : generateFunctionOutlineCode, "EXPRESSION_RETURNED_VALUE_REF" : generateReturnedValueRefCode, "EXPRESSION_SUBSCRIPT_LOOKUP" : generateSubscriptLookupCode, "EXPRESSION_SLICE_LOOKUP" : generateSliceLookupCode, "EXPRESSION_SET_OPERATION_UPDATE" : generateSetOperationUpdateCode, "EXPRESSION_SIDE_EFFECTS" : generateSideEffectsCode, "EXPRESSION_SPECIAL_UNPACK" : generateSpecialUnpackCode, "EXPRESSION_TEMP_VARIABLE_REF" : generateVariableReferenceCode, "EXPRESSION_VARIABLE_REF" : generateVariableReferenceCode, "EXPRESSION_YIELD" : generateYieldCode, "EXPRESSION_YIELD_FROM" : generateYieldFromCode, "EXPRESSION_SELECT_METACLASS" : generateSelectMetaclassCode, "EXPRESSION_ASYNC_WAIT" : generateAsyncWaitCode, "EXPRESSION_ASYNC_ITER" : generateAsyncIterCode, "EXPRESSION_ASYNC_NEXT" : generateAsyncNextCode, "EXPRESSION_STRING_CONCATENATION" : generateStringContenationCode, "EXPRESSION_BUILTIN_FORMAT" : generateBuiltinFormatCode, "EXPRESSION_BUILTIN_ASCII" : generateBuiltinAsciiCode, "EXPRESSION_LOCALS_VARIABLE_REF" : generateLocalsDictVariableRefCode, "EXPRESSION_RAISE_EXCEPTION" : generateRaiseExpressionCode, } ) setStatementDispatchDict( { "STATEMENT_ASSIGNMENT_VARIABLE" : generateAssignmentVariableCode, "STATEMENT_ASSIGNMENT_ATTRIBUTE" : generateAssignmentAttributeCode, "STATEMENT_ASSIGNMENT_SUBSCRIPT" : generateAssignmentSubscriptCode, "STATEMENT_ASSIGNMENT_SLICE" : generateAssignmentSliceCode, "STATEMENT_DEL_VARIABLE" : generateDelVariableCode, "STATEMENT_DEL_ATTRIBUTE" : generateDelAttributeCode, "STATEMENT_DEL_SUBSCRIPT" : generateDelSubscriptCode, "STATEMENT_DEL_SLICE" : generateDelSliceCode, "STATEMENT_DICT_OPERATION_REMOVE" : generateDictOperationRemoveCode, "STATEMENT_DICT_OPERATION_UPDATE" : generateDictOperationUpdateCode, "STATEMENT_RELEASE_VARIABLE" : generateVariableReleaseCode, "STATEMENT_EXPRESSION_ONLY" : generateExpressionOnlyCode, "STATEMENT_RETURN" : generateReturnCode, "STATEMENT_RETURN_TRUE" : generateReturnConstantCode, "STATEMENT_RETURN_FALSE" : generateReturnConstantCode, "STATEMENT_RETURN_NONE" : generateReturnConstantCode, "STATEMENT_RETURN_CONSTANT" : generateReturnConstantCode, "STATEMENT_GENERATOR_RETURN" : generateGeneratorReturnValueCode, "STATEMENT_GENERATOR_RETURN_NONE" : generateGeneratorReturnNoneCode, "STATEMENT_CONDITIONAL" : generateBranchCode, "STATEMENT_TRY" : generateTryCode, "STATEMENT_PRINT_VALUE" : generatePrintValueCode, "STATEMENT_PRINT_NEWLINE" : generatePrintNewlineCode, "STATEMENT_IMPORT_STAR" : generateImportStarCode, "STATEMENT_LIST_OPERATION_APPEND" : generateListOperationAppendCode, "STATEMENT_SET_OPERATION_ADD" : generateSetOperationAddCode, "STATEMENT_DICT_OPERATION_SET" : generateDictOperationSetCode, "STATEMENT_LOOP" : generateLoopCode, "STATEMENT_LOOP_BREAK" : generateLoopBreakCode, "STATEMENT_LOOP_CONTINUE" : generateLoopContinueCode, "STATEMENT_RAISE_EXCEPTION" : generateRaiseCode, "STATEMENT_RAISE_EXCEPTION_IMPLICIT" : generateRaiseCode, "STATEMENT_SPECIAL_UNPACK_CHECK" : generateUnpackCheckCode, "STATEMENT_EXEC" : generateExecCode, "STATEMENT_LOCALS_DICT_SYNC" : generateLocalsDictSyncCode, "STATEMENT_SET_LOCALS" : generateSetLocalsCode, "STATEMENT_PRESERVE_FRAME_EXCEPTION" : generateFramePreserveExceptionCode, "STATEMENT_RESTORE_FRAME_EXCEPTION" : generateFrameRestoreExceptionCode, "STATEMENT_PUBLISH_EXCEPTION" : generateExceptionPublishCode } ) Nuitka-0.5.28.2/nuitka/codegen/CodeHelpers.py0000644000372000001440000001653713207537242021152 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Helpers for code generation. This dispatch building of expressions and statements, as well as providing typical support functions to building parts. """ from nuitka.Options import shallTraceExecution from nuitka.PythonVersions import python_version from nuitka.Tracing import printError from .LabelCodes import getStatementTrace expression_dispatch_dict = {} def setExpressionDispatchDict(dispatch_dict): # Using global here, as this is really a singleton, in the form of a module, # and this is to break the cyclic dependency it has, pylint: disable=global-statement # Please call us only once. global expression_dispatch_dict assert not expression_dispatch_dict expression_dispatch_dict = dispatch_dict def generateExpressionCode(to_name, expression, emit, context, allow_none = False): try: _generateExpressionCode( to_name = to_name, expression = expression, emit = emit, context = context, allow_none = allow_none ) except Exception: printError( "Problem with %r at %s" % ( expression, "" if expression is None else expression.getSourceReference().getAsString() ) ) raise def _generateExpressionCode(to_name, expression, emit, context, allow_none = False): # This is a dispatching function for every expression. if expression is None and allow_none: return None # Make sure we don't generate code twice for any node, this uncovers bugs # where nodes are shared in the tree, which is not allowed. assert not hasattr(expression, "code_generated"), expression expression.code_generated = True old_source_ref = context.setCurrentSourceCodeReference(expression.getSourceReference()) if not expression.isExpression(): printError("No expression %r" % expression) expression.dump() assert False, expression expression_dispatch_dict[expression.kind]( to_name = to_name, expression = expression, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateExpressionsCode(names, expressions, emit, context): assert len(names) == len(expressions) result = [] for name, expression in zip(names, expressions): if expression is not None: to_name = context.allocateTempName(name) generateExpressionCode( to_name = to_name, expression = expression, emit = emit, context = context ) else: to_name = None result.append(to_name) return result def generateChildExpressionsCode(expression, emit, context): value_names = [] for child_name in expression.named_children: child_value = expression.getChild(child_name) # Allocate anyway, so names are aligned. value_name = context.allocateTempName(child_name + "_name") if child_value is not None: generateExpressionCode( to_name = value_name, expression = child_value, emit = emit, context = context ) value_names.append(value_name) else: context.forgetTempName(value_name) value_names.append(None) return value_names def generateChildExpressionCode(expression, emit, context, child_name = None): assert expression is not None if child_name is None: child_name = expression.getChildName() # Allocate anyway, so names are aligned. value_name = context.allocateTempName(child_name + "_name") generateExpressionCode( to_name = value_name, expression = expression, emit = emit, context = context ) return value_name statement_dispatch_dict = {} def setStatementDispatchDict(dispatch_dict): # Using global here, as this is really a singleton, in the form of a module, # and this is to break the cyclic dependency it has, pylint: disable=global-statement # Please call us only once. global statement_dispatch_dict assert not statement_dispatch_dict statement_dispatch_dict = dispatch_dict def generateStatementCode(statement, emit, context): try: statement_dispatch_dict[statement.kind]( statement = statement, emit = emit, context = context ) # Complain if any temporary was not dealt with yet. assert not context.getCleanupTempnames(), \ context.getCleanupTempnames() except Exception: printError( "Problem with %r at %s" % ( statement, statement.getSourceReference().getAsString() ) ) raise def _generateStatementSequenceCode(statement_sequence, emit, context): if statement_sequence is None: return for statement in statement_sequence.getStatements(): if shallTraceExecution(): source_ref = statement.getSourceReference() statement_repr = repr(statement) source_repr = source_ref.getAsString() if python_version >= 300: statement_repr = statement_repr.encode("utf8") source_repr = source_repr.encode("utf8") emit( getStatementTrace( source_repr, statement_repr ) ) # Might contain frame statement sequences as children. if statement.isStatementsFrame(): from .FrameCodes import generateStatementsFrameCode generateStatementsFrameCode( statement_sequence = statement, emit = emit, context = context ) else: generateStatementCode( statement = statement, emit = emit, context = context ) def generateStatementSequenceCode(statement_sequence, emit, context, allow_none = False): if allow_none and statement_sequence is None: return None assert statement_sequence.kind == "STATEMENTS_SEQUENCE", statement_sequence context.pushCleanupScope() _generateStatementSequenceCode( statement_sequence = statement_sequence, emit = emit, context = context ) context.popCleanupScope() Nuitka-0.5.28.2/nuitka/codegen/Contexts.py0000644000372000001440000010604413207537242020555 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation contexts. """ import hashlib import sys from abc import ABCMeta, abstractmethod from nuitka import Options from nuitka.__past__ import iterItems from nuitka.PythonVersions import python_version from nuitka.utils.InstanceCounters import counted_del, counted_init from .Namify import namifyConstant class ContextMetaClass(ABCMeta): pass # For Python2/3 compatible source, we create a base class that has the metaclass # used and doesn't require making a choice. ContextMetaClassBase = ContextMetaClass( "ContextMetaClassBase", (object,), {} ) # Many methods won't use self, but it's the interface. pylint: disable=no-self-use class TempMixin(object): # Lots of details, everything gets to store bits here, to indicate # code generation states, and there are many, pylint: disable=too-many-instance-attributes def __init__(self): self.tmp_names = {} self.tmp_types = {} self.forgotten_names = set() self.labels = {} # For exception handling self.needs_exception_variables = False self.exception_escape = None self.loop_continue = None self.loop_break = None # For branches self.true_target = None self.false_target = None self.keeper_variable_count = 0 self.exception_keepers = (None, None, None, None) self.preserver_variable_counts = set() self.cleanup_names = [] def formatTempName(self, base_name, number): if number is None: return "tmp_{name}".format( name = base_name ) else: return "tmp_{name}_{number:d}".format( name = base_name, number = number ) def allocateTempName(self, base_name, type_name = "PyObject *", unique = False): if unique: number = None else: number = self.tmp_names.get(base_name, 0) number += 1 self.tmp_names[base_name] = number if base_name not in self.tmp_types: self.tmp_types[base_name] = type_name else: assert self.tmp_types[base_name] == type_name, \ (self.tmp_types[base_name], type_name) return self.formatTempName( base_name = base_name, number = number ) def getIntResName(self): return self.allocateTempName("res", "int", unique = True) def getBoolResName(self): return self.allocateTempName("result", "bool", unique = True) def hasTempName(self, base_name): return base_name in self.tmp_names def forgetTempName(self, tmp_name): self.forgotten_names.add(tmp_name) def getTempNameInfos(self): result = [] for base_name, count in sorted(iterItems(self.tmp_names)): if count is not None: for number in range(1,count+1): tmp_name = self.formatTempName( base_name = base_name, number = number ) if tmp_name not in self.forgotten_names: result.append( ( tmp_name, self.tmp_types[base_name] ) ) else: tmp_name = self.formatTempName( base_name = base_name, number = None ) if tmp_name not in self.forgotten_names: result.append( ( tmp_name, self.tmp_types[base_name] ) ) return result def getExceptionEscape(self): return self.exception_escape def setExceptionEscape(self, label): result = self.exception_escape self.exception_escape = label return result def getLoopBreakTarget(self): return self.loop_break def setLoopBreakTarget(self, label): result = self.loop_break self.loop_break = label return result def getLoopContinueTarget(self): return self.loop_continue def setLoopContinueTarget(self, label): result = self.loop_continue self.loop_continue = label return result def allocateLabel(self, label): result = self.labels.get(label, 0) result += 1 self.labels[label] = result return "{name}_{number:d}".format( name = label, number = result ) def getLabelCount(self, label): return self.labels.get(label, 0) def needsExceptionVariables(self): return self.needs_exception_variables def markAsNeedsExceptionVariables(self): self.needs_exception_variables = True def allocateExceptionKeeperVariables(self): self.keeper_variable_count += 1 return ( "exception_keeper_type_%d" % self.keeper_variable_count, "exception_keeper_value_%d" % self.keeper_variable_count, "exception_keeper_tb_%d" % self.keeper_variable_count, "exception_keeper_lineno_%s" % self.keeper_variable_count ) def getKeeperVariableCount(self): return self.keeper_variable_count def getExceptionKeeperVariables(self): return self.exception_keepers def setExceptionKeeperVariables(self, keeper_vars): result = self.exception_keepers self.exception_keepers = tuple(keeper_vars) return result def getExceptionPreserverCounts(self): return self.preserver_variable_counts def addExceptionPreserverVariables(self, count): assert count != 0 self.preserver_variable_counts.add(count) def getTrueBranchTarget(self): return self.true_target def getFalseBranchTarget(self): return self.false_target def setTrueBranchTarget(self, label): self.true_target = label def setFalseBranchTarget(self, label): self.false_target = label def getCleanupTempnames(self): return self.cleanup_names[-1] def addCleanupTempName(self, tmp_name): assert tmp_name not in self.cleanup_names[-1], tmp_name self.cleanup_names[-1].append(tmp_name) def removeCleanupTempName(self, tmp_name): assert tmp_name in self.cleanup_names[-1], tmp_name self.cleanup_names[-1].remove(tmp_name) def needsCleanup(self, tmp_name): return tmp_name in self.cleanup_names[-1] def pushCleanupScope(self): self.cleanup_names.append([]) def popCleanupScope(self): assert not self.cleanup_names[-1] del self.cleanup_names[-1] class CodeObjectsMixin(object): def __init__(self): # Code objects needed made unique by a key. self.code_objects = {} def getCodeObjects(self): return sorted(iterItems(self.code_objects)) def getCodeObjectHandle(self, code_object): key = ( code_object.getFilename(), code_object.getCodeObjectName(), code_object.getLineNumber(), code_object.getVarNames(), code_object.getArgumentCount(), code_object.getKwOnlyParameterCount(), code_object.getCodeObjectKind(), code_object.getFlagIsOptimizedValue(), code_object.getFlagNewLocalsValue(), code_object.hasStarListArg(), code_object.hasStarDictArg(), code_object.getFlagHasClosureValue(), code_object.getFutureSpec().asFlags() ) if key not in self.code_objects: self.code_objects[key] = "codeobj_%s" % self._calcHash(key) return self.code_objects[key] if python_version < 300: def _calcHash(self, key): hash_value = hashlib.md5( "%s%s%d%s%d%d%s%s%s%s%s%s%s" % key ) return hash_value.hexdigest() else: def _calcHash(self, key): hash_value = hashlib.md5( ("%s%s%d%s%d%d%s%s%s%s%s%s%s" % key).encode("utf-8") ) return hash_value.hexdigest() class PythonContextBase(ContextMetaClassBase): @counted_init def __init__(self): self.source_ref = None self.current_source_ref = None self.last_source_ref = None __del__ = counted_del() def getCurrentSourceCodeReference(self): return self.current_source_ref def setCurrentSourceCodeReference(self, value): result = self.current_source_ref self.current_source_ref = value if value is not None: self.last_source_ref = result return result def getLastSourceCodeReference(self): result = self.last_source_ref # self.last_source_ref = None return result def isUsed(self, tmp_name): if tmp_name.startswith("tmp_unused_"): return False else: return True @abstractmethod def getConstantCode(self, constant): pass @abstractmethod def getModuleCodeName(self): pass @abstractmethod def getModuleName(self): pass @abstractmethod def addHelperCode(self, key, code): pass @abstractmethod def hasHelperCode(self, key): pass @abstractmethod def addDeclaration(self, key, code): pass @abstractmethod def pushFrameVariables(self, frame_variables): pass @abstractmethod def popFrameVariables(self): pass @abstractmethod def getFrameVariableTypeDescriptions(self): pass @abstractmethod def getFrameVariableTypeDescription(self): pass @abstractmethod def getFrameVariableTypeDescriptionName(self): pass @abstractmethod def getFrameVariableCodeNames(self): pass @abstractmethod def getLocalsDictName(self): pass @abstractmethod def allocateLocalsDictName(self): pass @abstractmethod def endLocalsDictName(self): pass @abstractmethod def allocateTempName(self, base_name, type_name = "PyObject *", unique = False): pass @abstractmethod def getIntResName(self): pass @abstractmethod def getBoolResName(self): pass @abstractmethod def hasTempName(self, base_name): pass @abstractmethod def forgetTempName(self, tmp_name): pass @abstractmethod def getTempNameInfos(self): pass @abstractmethod def getExceptionEscape(self): pass @abstractmethod def setExceptionEscape(self, label): pass @abstractmethod def getLoopBreakTarget(self): pass @abstractmethod def setLoopBreakTarget(self, label): pass @abstractmethod def getLoopContinueTarget(self): pass @abstractmethod def setLoopContinueTarget(self, label): pass @abstractmethod def allocateLabel(self, label): pass @abstractmethod def needsExceptionVariables(self): pass @abstractmethod def markAsNeedsExceptionVariables(self): pass @abstractmethod def allocateExceptionKeeperVariables(self): pass @abstractmethod def getExceptionKeeperVariables(self): pass @abstractmethod def setExceptionKeeperVariables(self, keeper_vars): pass @abstractmethod def addExceptionPreserverVariables(self, count): pass @abstractmethod def getTrueBranchTarget(self): pass @abstractmethod def getFalseBranchTarget(self): pass @abstractmethod def setTrueBranchTarget(self, label): pass @abstractmethod def setFalseBranchTarget(self, label): pass @abstractmethod def getCleanupTempnames(self): pass @abstractmethod def addCleanupTempName(self, tmp_name): pass @abstractmethod def removeCleanupTempName(self, tmp_name): pass @abstractmethod def needsCleanup(self, tmp_name): pass @abstractmethod def pushCleanupScope(self): pass @abstractmethod def popCleanupScope(self): pass class PythonChildContextBase(PythonContextBase): # Base classes can be abstract, pylint: disable=abstract-method def __init__(self, parent): PythonContextBase.__init__(self) self.parent = parent def getConstantCode(self, constant): return self.parent.getConstantCode(constant) def getModuleCodeName(self): return self.parent.getModuleCodeName() def getModuleName(self): return self.parent.getModuleName() def addHelperCode(self, key, code): return self.parent.addHelperCode(key, code) def hasHelperCode(self, key): return self.parent.hasHelperCode(key) def addDeclaration(self, key, code): self.parent.addDeclaration(key, code) def pushFrameVariables(self, frame_variables): return self.parent.pushFrameVariables(frame_variables) def popFrameVariables(self): return self.parent.popFrameVariables() def getFrameVariableTypeDescriptions(self): return self.parent.getFrameVariableTypeDescriptions() def getFrameVariableTypeDescription(self): return self.parent.getFrameVariableTypeDescription() def getFrameVariableTypeDescriptionName(self): return self.parent.getFrameVariableTypeDescriptionName() def getFrameVariableCodeNames(self): return self.parent.getFrameVariableCodeNames() def getLocalsDictName(self): return self.parent.getLocalsDictName() def allocateLocalsDictName(self): return self.parent.allocateLocalsDictName() def endLocalsDictName(self): return self.parent.endLocalsDictName() def _getConstantDefaultPopulation(): # Note: Can't work with set here, because we need to put in some values that # cannot be hashed. result = [ # Basic values that the helper code uses all the times. (), {}, "", True, False, 0, 1, # For Python3 empty bytes, no effect for Python2, same as "", used for # code objects. b"", # Python mechanics, used in various helpers. "__module__", "__class__", "__name__", "__metaclass__", "__dict__", "__doc__", "__file__", "__path__", "__enter__", "__exit__", "__builtins__", "__all__", "__cmp__", "__iter__", # Patched module name. "inspect", # Names of built-ins used in helper code. "compile", "range", "open", "sum", "format", "__import__", "bytearray", "staticmethod", "classmethod", # Arguments of __import__ built-in used in helper code. "name", "globals", "locals", "fromlist", "level", # Meta path based loader. "read", "rb" ] if python_version >= 300: # For Python3 modules result += ( "__cached__", "__loader__", ) # For Python3 print result += ( "print", "end", "file" ) # For Python3 bytes built-in. result += ( "bytes", ) if python_version >= 330: # Modules have that attribute starting with 3.3 result.append( "__loader__" ) if python_version >= 340: result.append( # YIELD_FROM uses this starting 3.4, with 3.3 other code is used. "send" ) if python_version >= 330: result += ( # YIELD_FROM uses this "throw", "close", ) if python_version < 300: # For patching Python2 internal class type result += ( "__getattr__", "__setattr__", "__delattr__", ) # For patching Python2 "sys" attributes for current exception result += ( "exc_type", "exc_value", "exc_traceback" ) # The xrange built-in is Python2 only. if python_version < 300: result.append( "xrange" ) # Executables only if not Options.shallMakeModule(): result.append( "__main__" ) # The "site" module is referenced in inspect patching. result.append( "site" ) # Built-in original values if not Options.shallMakeModule(): result += [ "type", "len", "range", "repr", "int", "iter", ] if python_version < 300: result.append( "long", ) # Disabling warnings at startup if "no_warnings" in Options.getPythonFlags(): result.append( "ignore" ) if python_version >= 350: # Patching the types module. result.append( "types" ) if not Options.shallMakeModule(): result.append( sys.executable ) return result class PythonGlobalContext(object): def __init__(self): self.constants = {} self.constant_use_count = {} for constant in _getConstantDefaultPopulation(): code = self.getConstantCode(constant) # Force them to be global. self.countConstantUse(code) self.countConstantUse(code) self.needs_exception_variables = False def getConstantCode(self, constant): # Use in user code, or for constants building code itself, many # constant types get special code immediately. # pylint: disable=too-many-branches if constant is None: key = "Py_None" elif constant is True: key = "Py_True" elif constant is False: key = "Py_False" elif constant is Ellipsis: key = "Py_Ellipsis" elif type(constant) is type: # TODO: Maybe make this a mapping in nuitka.Builtins if constant is None: key = "(PyObject *)Py_TYPE( Py_None )" elif constant is object: key = "(PyObject *)&PyBaseObject_Type" elif constant is staticmethod: key = "(PyObject *)&PyStaticMethod_Type" elif constant is classmethod: key = "(PyObject *)&PyClassMethod_Type" elif constant is bytearray: key = "(PyObject *)&PyByteArray_Type" elif constant is enumerate: key = "(PyObject *)&PyEnum_Type" elif constant is frozenset: key = "(PyObject *)&PyFrozenSet_Type" elif python_version >= 270 and constant is memoryview: key = "(PyObject *)&PyMemoryView_Type" elif python_version < 300 and constant is basestring: # pylint: disable=I0021,undefined-variable key = "(PyObject *)&PyBaseString_Type" elif python_version < 300 and constant is xrange: # pylint: disable=I0021,undefined-variable key = "(PyObject *)&PyRange_Type" else: type_name = constant.__name__ if constant is int and python_version >= 300: type_name = "long" elif constant is str: type_name = "string" if python_version < 300 else "unicode" key = "(PyObject *)&Py%s_Type" % type_name.title() else: key = "const_" + namifyConstant(constant) if key not in self.constants: self.constants[key] = constant return key def countConstantUse(self, constant): if constant not in self.constant_use_count: self.constant_use_count[constant] = 0 self.constant_use_count[constant] += 1 def getConstantUseCount(self, constant): return self.constant_use_count[constant] def getConstants(self): return self.constants class FrameDeclarationsMixin(object): def __init__(self): self.frame_declarations = [] # Frame is active or not, default not. self.frame_variables_stack = [""] # Type descriptions of the current frame. self.frame_type_descriptions = [()] # Types of variables for current frame. self.frame_variable_types = {} self.frames_used = 0 # Currently active frame stack inside the context. self.frame_stack = [None] self.locals_dict_used = 0 self.locals_dict_names = [] def getFrameHandle(self): return self.frame_stack[-1] def pushFrameHandle(self, frame_handle): self.frames_used += 1 if self.frames_used > 1: frame_handle += "_%d" % self.frames_used self.frame_stack.append(frame_handle) return self.frame_stack[-1] def popFrameHandle(self): result = self.frame_stack[-1] del self.frame_stack[-1] return result def getFramesCount(self): return self.frames_used def addFrameDeclaration(self, frame_decl): self.frame_declarations.append(frame_decl) def getFrameDeclarations(self): return self.frame_declarations + [ "NUITKA_MAY_BE_UNUSED char const *type_description_%d = NULL;" % (i+1) for i in range(self.getFramesCount()) ] def pushFrameVariables(self, frame_variables): """ Set current the frame variables. """ self.frame_variables_stack.append(frame_variables) self.frame_type_descriptions.append(set()) def popFrameVariables(self): """ End of frame, restore previous ones. """ del self.frame_variables_stack[-1] del self.frame_type_descriptions[-1] def setVariableType(self, variable, variable_code_name, variable_c_type): assert variable.isLocalVariable(), variable self.frame_variable_types[variable] = variable_code_name, variable_c_type.getTypeIndicator() def getFrameVariableTypeDescriptions(self): return self.frame_type_descriptions[-1] def getFrameVariableTypeDescriptionName(self): return "type_description_%d" % (len(self.frame_stack) - 1) def getFrameVariableTypeDescription(self): result = "".join( self.frame_variable_types.get(variable, ("NULL", 'N'))[1] for variable in self.frame_variables_stack[-1] ) if result: self.frame_type_descriptions[-1].add(result) return result def getFrameVariableCodeNames(self): result = [] for variable in self.frame_variables_stack[-1]: variable_code_name, variable_code_type = self.frame_variable_types.get(variable, ("NULL", 'N')) if variable_code_type in ('b',): result.append("(int)" + variable_code_name) else: result.append(variable_code_name) return result def getLocalsDictName(self): return self.locals_dict_names[-1] def allocateLocalsDictName(self): self.locals_dict_used += 1 self.locals_dict_names.append( "locals_dict_%d" % self.locals_dict_used ) def endLocalsDictName(self): del self.locals_dict_names[-1] def getLocalsDictNames(self): return [ "locals_dict_%d" % (locals_dict_index + 1) for locals_dict_index in range(0, self.locals_dict_used) ] class ReturnReleaseModeMixin(object): def __init__(self): self.return_release_mode = False self.return_exit = None def setReturnReleaseMode(self, value): result = self.return_release_mode self.return_release_mode = value return result def getReturnReleaseMode(self): return self.return_release_mode def setReturnTarget(self, label): result = self.return_exit self.return_exit = label return result def getReturnTarget(self): return self.return_exit class ReturnValueNameMixin(object): def __init__(self): self.return_name = None def getReturnValueName(self): if self.return_name is None: self.return_name = self.allocateTempName("return_value", unique = True) return self.return_name def setReturnValueName(self, value): result = self.return_name self.return_name = value return result class PythonModuleContext(TempMixin, CodeObjectsMixin, FrameDeclarationsMixin, ReturnReleaseModeMixin, ReturnValueNameMixin, PythonContextBase): # Plenty of attributes, because it's storing so many different things. # pylint: disable=too-many-instance-attributes def __init__(self, module, module_name, code_name, filename, global_context): PythonContextBase.__init__(self) TempMixin.__init__(self) CodeObjectsMixin.__init__(self) FrameDeclarationsMixin.__init__(self) ReturnReleaseModeMixin.__init__(self) # TODO: For outlines bodies. ReturnValueNameMixin.__init__(self) self.module = module self.name = module_name self.code_name = code_name self.filename = filename self.global_context = global_context self.declaration_codes = {} self.helper_codes = {} self.constants = set() self.frame_handle = None self.needs_module_filename_object = False def __repr__(self): return "" % self.filename def getOwner(self): return self.module def getEntryPoint(self): return self.module def isCompiledPythonModule(self): return True def hasLocalsDict(self): return False def hasForeignLocalsDict(self): return False def getName(self): return self.name def getFilename(self): return self.filename def mayRaiseException(self): body = self.module.getBody() return body is not None and body.mayRaiseException(BaseException) getModuleName = getName def getModuleCodeName(self): return self.code_name def setFrameGuardMode(self, guard_mode): assert guard_mode == "once" def addHelperCode(self, key, code): assert key not in self.helper_codes, key self.helper_codes[ key ] = code def hasHelperCode(self, key): return key in self.helper_codes def getHelperCodes(self): return self.helper_codes def addDeclaration(self, key, code): assert key not in self.declaration_codes self.declaration_codes[ key ] = code def getDeclarations(self): return self.declaration_codes def mayRecurse(self): return False def getConstantCode(self, constant): result = self.global_context.getConstantCode(constant) if result not in self.constants: self.constants.add(result) self.global_context.countConstantUse(result) return result def getConstants(self): return self.constants def markAsNeedsModuleFilenameObject(self): self.needs_module_filename_object = True def needsModuleFilenameObject(self): return self.needs_module_filename_object class PythonFunctionContext(FrameDeclarationsMixin, TempMixin, ReturnReleaseModeMixin, ReturnValueNameMixin, PythonChildContextBase): def __init__(self, parent, function): PythonChildContextBase.__init__( self, parent = parent ) TempMixin.__init__(self) FrameDeclarationsMixin.__init__(self) ReturnReleaseModeMixin.__init__(self) ReturnValueNameMixin.__init__(self) self.function = function self.setExceptionEscape("function_exception_exit") self.setReturnTarget("function_return_exit") self.frame_handle = None def __repr__(self): return "" % ( "function" if not self.function.isExpressionClassBody() else "class", self.function.getName() ) def getFunction(self): return self.function def getOwner(self): return self.function def getEntryPoint(self): return self.function def hasLocalsDict(self): return self.function.hasLocalsDict() def hasForeignLocalsDict(self): return self.function.hasForeignLocalsDict() def mayRecurse(self): # TODO: Determine this at compile time for enhanced optimizations. return True def getCodeObjectHandle(self, code_object): return self.parent.getCodeObjectHandle(code_object) class PythonFunctionDirectContext(PythonFunctionContext): def isForDirectCall(self): return True def isForCrossModuleUsage(self): return self.function.isCrossModuleUsed() def isForCreatedFunction(self): return False class PythonGeneratorObjectContext(PythonFunctionContext): def isForDirectCall(self): return False def isForCrossModuleUsage(self): return self.function.isCrossModuleUsed() def isForCreatedFunction(self): return False def getContextObjectName(self): return "generator" def getGeneratorReturnValueName(self): if python_version >= 330: return self.allocateTempName( "return_value", "PyObject *", unique = True ) else: return self.allocateTempName( "generator_return", "bool", unique = True ) class PythonCoroutineObjectContext(PythonGeneratorObjectContext): def getContextObjectName(self): return "coroutine" class PythonAsyncgenObjectContext(PythonGeneratorObjectContext): def getContextObjectName(self): return "asyncgen" class PythonFunctionCreatedContext(PythonFunctionContext): def isForDirectCall(self): return False def isForCreatedFunction(self): return True class PythonFunctionOutlineContext(ReturnReleaseModeMixin, ReturnValueNameMixin, PythonChildContextBase): def __init__(self, parent, outline): PythonChildContextBase.__init__( self, parent = parent ) ReturnReleaseModeMixin.__init__(self) ReturnValueNameMixin.__init__(self) self.outline = outline def getOwner(self): return self.outline def getEntryPoint(self): return self.outline.getEntryPoint() def allocateLabel(self, label): return self.parent.allocateLabel(label) def allocateTempName(self, base_name, type_name = "PyObject *", unique = False): return self.parent.allocateTempName(base_name, type_name, unique) def hasTempName(self, base_name): return self.parent.hasTempName(base_name) def getCleanupTempnames(self): return self.parent.getCleanupTempnames() def addCleanupTempName(self, tmp_name): self.parent.addCleanupTempName(tmp_name) def removeCleanupTempName(self, tmp_name): self.parent.removeCleanupTempName(tmp_name) def needsCleanup(self, tmp_name): return self.parent.needsCleanup(tmp_name) def pushCleanupScope(self): return self.parent.pushCleanupScope() def popCleanupScope(self): self.parent.popCleanupScope() def getCodeObjectHandle(self, code_object): return self.parent.getCodeObjectHandle(code_object) def getExceptionEscape(self): return self.parent.getExceptionEscape() def setExceptionEscape(self, label): return self.parent.setExceptionEscape(label) def getLoopBreakTarget(self): return self.parent.getLoopBreakTarget() def setLoopBreakTarget(self, label): return self.parent.setLoopBreakTarget(label) def getLoopContinueTarget(self): return self.parent.getLoopContinueTarget() def setLoopContinueTarget(self, label): return self.parent.setLoopContinueTarget(label) def getTrueBranchTarget(self): return self.parent.getTrueBranchTarget() def getFalseBranchTarget(self): return self.parent.getFalseBranchTarget() def setTrueBranchTarget(self, label): self.parent.setTrueBranchTarget(label) def setFalseBranchTarget(self, label): self.parent.setFalseBranchTarget(label) def getFrameHandle(self): return self.parent.getFrameHandle() def pushFrameHandle(self, frame_handle): return self.parent.pushFrameHandle(frame_handle) def popFrameHandle(self): return self.parent.popFrameHandle() def getExceptionKeeperVariables(self): return self.parent.getExceptionKeeperVariables() def setExceptionKeeperVariables(self, keeper_vars): return self.parent.setExceptionKeeperVariables(keeper_vars) def setVariableType(self, variable, variable_code_name, variable_c_type): self.parent.setVariableType(variable, variable_code_name, variable_c_type) def getIntResName(self): return self.parent.getIntResName() def getBoolResName(self): return self.parent.getBoolResName() def needsExceptionVariables(self): return self.parent.needsExceptionVariables() def markAsNeedsExceptionVariables(self): self.parent.markAsNeedsExceptionVariables() def allocateExceptionKeeperVariables(self): return self.parent.allocateExceptionKeeperVariables() def addFrameDeclaration(self, frame_decl): self.parent.addFrameDeclaration(frame_decl) def isForDirectCall(self): return self.parent.isForDirectCall() def mayRecurse(self): # TODO: In modules, there could be loops with outlines, we could detect # that. return True def hasLocalsDict(self): return self.outline.hasLocalsDict() def hasForeignLocalsDict(self): return self.outline.hasForeignLocalsDict() def addExceptionPreserverVariables(self, count): self.parent.addExceptionPreserverVariables(count) def getTempNameInfos(self): return self.parent.getTempNameInfos() def forgetTempName(self, tmp_name): self.parent.forgetTempName(tmp_name) def getContextObjectName(self): return self.parent.getContextObjectName() Nuitka-0.5.28.2/nuitka/codegen/IteratorCodes.py0000644000372000001440000001765213207537242021523 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Iteration related codes. Next variants and unpacking with related checks. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import ( getErrorExitCode, getErrorExitReleaseCode, getFrameVariableTypeDescriptionCode, getReleaseCode ) from .Indentation import indented from .LineNumberCodes import getErrorLineNumberUpdateCode from .PythonAPICodes import generateCAPIObjectCode from .templates.CodeTemplatesIterators import ( template_iterator_check, template_loop_break_next ) def generateBuiltinNext1Code(to_name, expression, emit, context): value_name, = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = %s;" % ( to_name, "ITERATOR_NEXT( %s )" % value_name, ) ) getReleaseCode( release_name = value_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, quick_exception = "StopIteration", emit = emit, context = context ) context.addCleanupTempName(to_name) def getBuiltinLoopBreakNextCode(to_name, value, emit, context): emit( "%s = %s;" % ( to_name, "ITERATOR_NEXT( %s )" % value, ) ) getReleaseCode( release_name = value, emit = emit, context = context ) break_target = context.getLoopBreakTarget() if type(break_target) is tuple: break_indicator_code = "%s = true;" % break_target[1] break_target = break_target[0] else: break_indicator_code = "" emit( template_loop_break_next % { "to_name" : to_name, "break_indicator_code" : break_indicator_code, "break_target" : break_target, "release_temps" : indented( getErrorExitReleaseCode(context), 2 ), "var_description_code" : indented( getFrameVariableTypeDescriptionCode(context), 2 ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context), 2 ), "exception_target" : context.getExceptionEscape() } ) context.addCleanupTempName(to_name) def getUnpackNextCode(to_name, value, expected, count, emit, context): if python_version < 350: emit( "%s = UNPACK_NEXT( %s, %s );" % ( to_name, value, count - 1 ) ) else: emit( "%s = UNPACK_NEXT( %s, %s, %s );" % ( to_name, value, count - 1, expected ) ) getErrorExitCode( check_name = to_name, quick_exception = "StopIteration", emit = emit, context = context ) getReleaseCode( release_name = value, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateSpecialUnpackCode(to_name, expression, emit, context): value_name = context.allocateTempName("unpack") generateExpressionCode( to_name = value_name, expression = expression.getValue(), emit = emit, context = context ) getUnpackNextCode( to_name = to_name, value = value_name, count = expression.getCount(), expected = expression.getExpected(), emit = emit, context = context ) def generateUnpackCheckCode(statement, emit, context): iterator_name = context.allocateTempName("iterator_name") generateExpressionCode( to_name = iterator_name, expression = statement.getIterator(), emit = emit, context = context ) # These variable cannot collide, as it's used very locally. attempt_name = context.allocateTempName("iterator_attempt", unique = True) release_code = getErrorExitReleaseCode(context) var_description_code = getFrameVariableTypeDescriptionCode(context) old_source_ref = context.setCurrentSourceCodeReference( statement.getSourceReference() ) emit( template_iterator_check % { "iterator_name" : iterator_name, "attempt_name" : attempt_name, "count" : statement.getCount(), "exception_exit" : context.getExceptionEscape(), "release_temps_1" : indented(release_code, 3), "line_number_code_1" : indented(getErrorLineNumberUpdateCode(context),3), "var_description_code_1" : indented(var_description_code, 3), "release_temps_2" : indented(release_code), "var_description_code_2" : indented(var_description_code), "line_number_code_2" : indented(getErrorLineNumberUpdateCode(context)), } ) getReleaseCode( release_name = iterator_name, emit = emit, context = context ) context.setCurrentSourceCodeReference( old_source_ref ) def generateBuiltinNext2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_NEXT2", arg_desc = ( ("next_arg", expression.getIterator()), ("next_default", expression.getDefault()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinIter1Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "MAKE_ITERATOR", arg_desc = ( ("iter_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinIter2Code(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_ITER2", arg_desc = ( ("iter_callable", expression.getCallable()), ("iter_sentinel", expression.getSentinel()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) def generateBuiltinLenCode(to_name, expression, emit, context): generateCAPIObjectCode( to_name = to_name, capi = "BUILTIN_LEN", arg_desc = ( ("len_arg", expression.getValue()), ), may_raise = expression.mayRaiseException(BaseException), source_ref = expression.getCompatibleSourceReference(), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/ModuleCodes.py0000644000372000001440000001151013207537242021142 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code to generate and interact with compiled module objects. """ from nuitka.Version import getNuitkaVersion, getNuitkaVersionYear from .CodeObjectCodes import getCodeObjectsDeclCode, getCodeObjectsInitCode from .ConstantCodes import allocateNestedConstants, getConstantInitCodes from .Indentation import indented from .templates.CodeTemplatesModules import ( template_global_copyright, template_module_body_template, template_module_exception_exit, template_module_noexception_exit ) def getModuleAccessCode(context): return "module_%s" % context.getModuleCodeName() def getModuleValues(context, module_name, module_identifier, codes, function_decl_codes, function_body_codes, outline_variables, temp_variables, is_main_module, is_internal_module): # For the module code, lots of arguments and attributes come together. # pylint: disable=too-many-locals # Temporary variable initializations # TODO: Move that to a place outside of functions. from .FunctionCodes import setupFunctionLocalVariables, finalizeFunctionLocalVariables local_var_inits, cleanup = setupFunctionLocalVariables( context = context, parameters = None, closure_variables = (), user_variables = outline_variables, temp_variables = temp_variables ) finalizeFunctionLocalVariables(context, local_var_inits, cleanup) if context.needsExceptionVariables(): module_exit = template_module_exception_exit else: module_exit = template_module_noexception_exit module_body_template_values = { "module_name" : module_name, "module_name_obj" : context.getConstantCode( constant = module_name ), "is_main_module" : 1 if is_main_module else 0, "module_identifier" : module_identifier, "module_functions_decl" : function_decl_codes, "module_functions_code" : function_body_codes, "temps_decl" : indented(local_var_inits), "module_code" : indented(codes), "module_exit" : module_exit, "module_code_objects_decl" : indented( getCodeObjectsDeclCode(context), 0 ), "module_code_objects_init" : indented( getCodeObjectsInitCode(context), 1 ) } allocateNestedConstants(context) # Force internal module to not need constants init, by making all its # constants be shared. if is_internal_module: for constant in context.getConstants(): context.global_context.countConstantUse(constant) return module_body_template_values def getModuleCode(module_context, template_values): header = template_global_copyright % { "name" : module_context.getName(), "version" : getNuitkaVersion(), "year" : getNuitkaVersionYear() } decls, inits, checks = getConstantInitCodes(module_context) if module_context.needsModuleFilenameObject(): decls.append("static PyObject *module_filename_obj;") template_values["constant_decl_codes"] = indented( decls, 0 ) template_values["constant_init_codes"] = indented( inits, 1 ) template_values["constant_check_codes"] = indented( checks, 1 ) return header + template_module_body_template % template_values def generateModuleFileAttributeCode(to_name, expression, emit, context): # The expression doesn't really matter, but it is part of the API for # the expression registry, pylint: disable=unused-argument emit( "%s = module_filename_obj;" % ( to_name, ) ) context.markAsNeedsModuleFilenameObject() def generateModuleLoaderRefCode(to_name, expression, emit, context): # The expression doesn't really matter, but it is part of the API for # the expression registry, pylint: disable=unused-argument emit( "%s = metapath_based_loader;" % ( to_name, ) ) Nuitka-0.5.28.2/nuitka/codegen/LoopCodes.py0000644000372000001440000000627213207537242020637 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Loop codes. Code generation for loops, breaking them, or continuing them. In Nuitka, there are no for-loops or while-loops at this point. They have been re-formulated in a simpler loop without a condition, and statements there-in that break under certain conditions. See Developer Manual for how the CPython loops are mapped to these nodes. """ from .CodeHelpers import generateStatementSequenceCode from .ErrorCodes import getErrorExitBoolCode from .ExceptionCodes import getExceptionUnpublishedReleaseCode from .LabelCodes import getGotoCode, getLabelCode def generateLoopBreakCode(statement, emit, context): # Functions used for generation all accept statement, but this one does # not use it. pylint: disable=unused-argument getExceptionUnpublishedReleaseCode(emit, context) break_target = context.getLoopBreakTarget() getGotoCode(break_target, emit) def generateLoopContinueCode(statement, emit, context): # Functions used for generation all accept statement, but this one does # not use it. pylint: disable=unused-argument getExceptionUnpublishedReleaseCode(emit, context) continue_target = context.getLoopContinueTarget() getGotoCode(continue_target, emit) def generateLoopCode(statement, emit, context): loop_start_label = context.allocateLabel("loop_start") if not statement.isStatementAborting(): loop_end_label = context.allocateLabel("loop_end") else: loop_end_label = None getLabelCode(loop_start_label, emit) old_loop_break = context.setLoopBreakTarget(loop_end_label) old_loop_continue = context.setLoopContinueTarget(loop_start_label) generateStatementSequenceCode( statement_sequence = statement.getLoopBody(), allow_none = True, emit = emit, context = context ) context.setLoopBreakTarget(old_loop_break) context.setLoopContinueTarget(old_loop_continue) # Note: We are using the wrong line here, but it's an exception, it's unclear what line it would be anyway. old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) getErrorExitBoolCode( condition = "CONSIDER_THREADING() == false", emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) getGotoCode(loop_start_label, emit) if loop_end_label is not None: getLabelCode(loop_end_label, emit) Nuitka-0.5.28.2/nuitka/codegen/__init__.py0000644000372000001440000000150113112214770020466 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/codegen/BlobCodes.py0000644000372000001440000000322613134660221020571 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Blob codes for storing binary data semi-efficiently. This module offers means to store and encode binary blobs in C semi efficiently. The "StreamData" class is used in two places, for constants and for freezing of bytecode. """ class StreamData(object): def __init__(self): self.stream_data = bytes() def getStreamDataCode(self, value, fixed_size = False): offset = self.getStreamDataOffset(value) if fixed_size: return "&constant_bin[ %d ]" % offset else: return "&constant_bin[ %d ], %d" % ( offset, len(value) ) def getStreamDataOffset(self, value): offset = self.stream_data.find(value) if offset == -1: offset = len(self.stream_data) self.stream_data += value return offset def getBytes(self): return self.stream_data Nuitka-0.5.28.2/nuitka/codegen/SliceCodes.py0000644000372000001440000003405713207537242020767 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for slicing. This is about slice lookups, assignments, and deletions. There is also a special case, for using index values instead of objects. The slice objects are also created here, and can be used for indexing. """ from nuitka import Options from nuitka.Constants import isNumberConstant from nuitka.PythonVersions import python_version from .CodeHelpers import ( generateChildExpressionsCode, generateExpressionCode, generateExpressionsCode ) from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .IndexCodes import ( getIndexCode, getIndexValueCode, getMaxIndexCode, getMinIndexCode ) def generateSliceRangeIdentifier(lower, upper, scope, emit, context): lower_name = context.allocateTempName( scope + "slicedel_index_lower", "Py_ssize_t" ) upper_name = context.allocateTempName( scope + "_index_upper", "Py_ssize_t" ) def isSmallNumberConstant(node): value = node.getConstant() if isNumberConstant(value): return abs(int(value)) < 2**63-1 else: return False if lower is None: getMinIndexCode( to_name = lower_name, emit = emit ) elif lower.isExpressionConstantRef() and isSmallNumberConstant(lower): getIndexValueCode( to_name = lower_name, value = int(lower.getConstant()), emit = emit ) else: value_name = context.allocateTempName(scope + "_lower_index_value") generateExpressionCode( to_name = value_name, expression = lower, emit = emit, context = context ) getIndexCode( to_name = lower_name, value_name = value_name, emit = emit, context = context ) if upper is None: getMaxIndexCode( to_name = upper_name, emit = emit ) elif upper.isExpressionConstantRef() and isSmallNumberConstant(upper): getIndexValueCode( to_name = upper_name, value = int(upper.getConstant()), emit = emit ) else: value_name = context.allocateTempName(scope + "_upper_index_value") generateExpressionCode( to_name = value_name, expression = upper, emit = emit, context = context ) getIndexCode( to_name = upper_name, value_name = value_name, emit = emit, context = context ) return lower_name, upper_name def _decideSlicing(lower, upper): return (lower is None or lower.isIndexable()) and \ (upper is None or upper.isIndexable()) def generateSliceLookupCode(to_name, expression, emit, context): assert python_version < 300 lower = expression.getLower() upper = expression.getUpper() if _decideSlicing(lower, upper): lower_name, upper_name = generateSliceRangeIdentifier( lower = lower, upper = upper, scope = "slice", emit = emit, context = context ) source_name = context.allocateTempName("slice_source") generateExpressionCode( to_name = source_name, expression = expression.getLookupSource(), emit = emit, context = context ) getSliceLookupIndexesCode( to_name = to_name, source_name = source_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) else: source_name, lower_name, upper_name = generateExpressionsCode( names = ("slice_source", "slice_lower", "slice_upper"), expressions = ( expression.getLookupSource(), expression.getLower(), expression.getUpper() ), emit = emit, context = context ) getSliceLookupCode( to_name = to_name, source_name = source_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) def generateAssignmentSliceCode(statement, emit, context): assert python_version < 300 lookup_source = statement.getLookupSource() lower = statement.getLower() upper = statement.getUpper() value = statement.getAssignSource() value_name = context.allocateTempName("sliceass_value") generateExpressionCode( to_name = value_name, expression = value, emit = emit, context = context ) target_name = context.allocateTempName("sliceass_target") generateExpressionCode( to_name = target_name, expression = lookup_source, emit = emit, context = context ) if _decideSlicing(lower, upper): lower_name, upper_name = generateSliceRangeIdentifier( lower = lower, upper = upper, scope = "sliceass", emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceAssignmentIndexesCode( target_name = target_name, lower_name = lower_name, upper_name = upper_name, value_name = value_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: lower_name, upper_name = generateExpressionsCode( names = ( "sliceass_lower", "sliceass_upper" ), expressions = ( lower, upper ), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( value.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceAssignmentCode( target_name = target_name, upper_name = upper_name, lower_name = lower_name, value_name = value_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateDelSliceCode(statement, emit, context): assert python_version < 300 target = statement.getLookupSource() lower = statement.getLower() upper = statement.getUpper() target_name = context.allocateTempName("slicedel_target") generateExpressionCode( to_name = target_name, expression = target, emit = emit, context = context ) if _decideSlicing(lower, upper): lower_name, upper_name = generateSliceRangeIdentifier( lower = lower, upper = upper, scope = "slicedel", emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( (upper or lower or statement).getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceDelIndexesCode( target_name = target_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) else: lower_name, upper_name = generateExpressionsCode( names = ( "slicedel_lower", "slicedel_upper" ), expressions = ( lower, upper ), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( (upper or lower or target).getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) getSliceDelCode( target_name = target_name, lower_name = lower_name, upper_name = upper_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateBuiltinSliceCode(to_name, expression, emit, context): lower_name, upper_name, step_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = MAKE_SLICEOBJ3( %s, %s, %s );" % ( to_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None", step_name if step_name is not None else "Py_None", ) ) getReleaseCodes( release_names = (lower_name, upper_name, step_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = False, # Note: Cannot fail emit = emit, context = context ) context.addCleanupTempName(to_name) def getSliceLookupCode(to_name, source_name, lower_name, upper_name, emit, context): emit( "%s = LOOKUP_SLICE( %s, %s, %s );" % ( to_name, source_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None" ) ) getReleaseCodes( release_names = (source_name, lower_name, upper_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getSliceLookupIndexesCode(to_name, lower_name, upper_name, source_name, emit, context): emit( "%s = LOOKUP_INDEX_SLICE( %s, %s, %s );" % ( to_name, source_name, lower_name, upper_name, ) ) getReleaseCode( release_name = source_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getSliceAssignmentIndexesCode(target_name, lower_name, upper_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_INDEX_SLICE( %s, %s, %s, %s );" % ( res_name, target_name, lower_name, upper_name, value_name ) ) getReleaseCodes( release_names = (value_name, target_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSliceAssignmentCode(target_name, lower_name, upper_name, value_name, emit, context): res_name = context.getBoolResName() emit( "%s = SET_SLICE( %s, %s, %s, %s );" % ( res_name, target_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None", value_name ) ) getReleaseCodes( release_names = (target_name, lower_name, upper_name, value_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSliceDelIndexesCode(target_name, lower_name, upper_name, emit, context): res_name = context.getBoolResName() emit( "%s = DEL_INDEX_SLICE( %s, %s, %s );" % ( res_name, target_name, lower_name, upper_name ) ) getReleaseCode( release_name = target_name, emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) def getSliceDelCode(target_name, lower_name, upper_name, emit, context): res_name = context.getBoolResName() emit( "%s = DEL_SLICE( %s, %s, %s );" % ( res_name, target_name, lower_name if lower_name is not None else "Py_None", upper_name if upper_name is not None else "Py_None" ) ) getReleaseCodes( release_names = (target_name, lower_name, upper_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/CallCodes.py0000644000372000001440000005135113207537242020577 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for calls. The different kinds of calls get dedicated code. Most notable, calls with only positional arguments, are attempted through helpers that might be able to execute them without creating the argument dictionary at all. """ from .CodeHelpers import generateChildExpressionCode, generateExpressionCode from .ConstantCodes import getConstantAccess from .ErrorCodes import getErrorExitCode, getReleaseCode, getReleaseCodes from .LineNumberCodes import emitLineNumberUpdateCode from .templates.CodeTemplatesCalls import ( template_call_function_with_args_decl, template_call_function_with_args_impl, template_call_method_with_args_decl, template_call_method_with_args_impl ) from .templates.CodeTemplatesModules import ( template_header_guard, template_helper_impl_decl ) def _generateCallCodePosOnly(to_name, expression, called_name, called_attribute_name, emit, context): # We have many variants for this to deal with, pylint: disable=too-many-branches assert called_name is not None # TODO: Not yet specialized for method calls. # assert called_attribute_name is None call_args = expression.getCallArgs() if call_args is None or call_args.isExpressionConstantRef(): context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) if call_args is not None: call_args_value = call_args.getConstant() else: call_args_value = () assert type(call_args_value) is tuple if call_args is not None and call_args.isMutable(): call_arg_names = [] for call_arg_element in call_args_value: call_arg_name = context.allocateTempName("call_arg_element") getConstantAccess( to_name = call_arg_name, constant = call_arg_element, emit = emit, context = context, ) call_arg_names.append(call_arg_name) if called_attribute_name is None: getCallCodePosArgsQuick( to_name = to_name, called_name = called_name, arg_names = call_arg_names, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: getInstanceCallCodePosArgsQuick( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, arg_names = call_arg_names, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) elif call_args_value: if called_attribute_name is None: getCallCodeFromTuple( to_name = to_name, called_name = called_name, arg_tuple = context.getConstantCode( constant = call_args_value ), arg_size = len(call_args_value), needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: getInstanceCallCodeFromTuple( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, arg_tuple = context.getConstantCode( constant = call_args_value ), arg_size = len(call_args_value), needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: if called_attribute_name is None: getCallCodeNoArgs( to_name = to_name, called_name = called_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: getInstanceCallCodeNoArgs( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) elif call_args.isExpressionMakeTuple(): call_arg_names = [] for call_arg_element in call_args.getElements(): call_arg_name = generateChildExpressionCode( child_name = call_args.getChildName() + "_element", expression = call_arg_element, emit = emit, context = context, ) call_arg_names.append(call_arg_name) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) if called_attribute_name is None: getCallCodePosArgsQuick( to_name = to_name, called_name = called_name, arg_names = call_arg_names, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: getInstanceCallCodePosArgsQuick( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, arg_names = call_arg_names, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: args_name = generateChildExpressionCode( expression = call_args, emit = emit, context = context ) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) if called_attribute_name is None: getCallCodePosArgs( to_name = to_name, called_name = called_name, args_name = args_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) else: getInstanceCallCodePosArgs( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, args_name = args_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) def _generateCallCodeKwOnly(to_name, expression, call_kw, called_name, called_attribute_name, emit, context): # TODO: Not yet specialized for method calls. assert called_name is not None assert called_attribute_name is None call_kw_name = generateChildExpressionCode( expression = call_kw, emit = emit, context = context ) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) _getCallCodeKeywordArgs( to_name = to_name, called_name = called_name, call_kw_name = call_kw_name, emit = emit, context = context ) def generateCallCode(to_name, expression, emit, context): # There is a whole lot of different cases, for each of which, we create # optimized code, constant, with and without positional or keyword arguments # each, so there is lots of branches involved. called = expression.getCalled() call_kw = expression.getCallKw() call_args = expression.getCallArgs() # TODO: Make this work for all cases. Currently, the method calls that do # a combined lookup and call, do a re-ordering of things, and therefore it # must be disabled until this is solved. if called.isExpressionAttributeLookup() and \ not called.isExpressionAttributeLookupSpecial() and \ called.getAttributeName() not in ("__class__", "__dict__") and \ (call_args is None or \ not call_args.mayHaveSideEffects() or \ not called.mayHaveSideEffects() ) and \ call_kw is None: called_name = context.allocateTempName("called_instance") generateExpressionCode( to_name = called_name, expression = called.getLookupSource(), emit = emit, context = context ) called_attribute_name = context.getConstantCode( constant = called.getAttributeName() ) else: called_attribute_name = None called_name = generateChildExpressionCode( expression = called, emit = emit, context = context ) if call_kw is None or \ (call_kw.isExpressionConstantRef() and call_kw.getConstant() == {}): _generateCallCodePosOnly( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, expression = expression, emit = emit, context = context ) else: call_args = expression.getCallArgs() if call_args is None or \ (call_args.isExpressionConstantRef() and \ call_args.getConstant() == ()): _generateCallCodeKwOnly( to_name = to_name, called_name = called_name, called_attribute_name = called_attribute_name, expression = expression, call_kw = call_kw, emit = emit, context = context ) else: call_args_name = generateChildExpressionCode( expression = call_args, emit = emit, context = context ) call_kw_name = generateChildExpressionCode( expression = call_kw, emit = emit, context = context ) context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getCallCodePosKeywordArgs( to_name = to_name, called_name = called_name, call_args_name = call_args_name, call_kw_name = call_kw_name, emit = emit, context = context ) def getCallCodeNoArgs(to_name, called_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION_NO_ARGS( %s );" % ( to_name, called_name ) ) getReleaseCode( release_name = called_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) def getInstanceCallCodeNoArgs(to_name, called_name, called_attribute_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_METHOD_NO_ARGS( %s, %s );" % ( to_name, called_name, called_attribute_name ) ) getReleaseCodes( release_names = ( called_name, called_attribute_name ), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, needs_check = needs_check, context = context ) context.addCleanupTempName(to_name) # Outside helper code relies on some quick call to be present. quick_calls_used = set([1, 2, 3, 4, 5]) quick_instance_calls_used = set() def getInstanceCallCodePosArgsQuick(to_name, called_name, called_attribute_name, arg_names, needs_check, emit, context): arg_size = len(arg_names) quick_instance_calls_used.add(arg_size) # For 0 arguments, NOARGS is supposed to be used. assert arg_size > 0 emitLineNumberUpdateCode(emit, context) emit( """\ { PyObject *call_args[] = { %s }; %s = CALL_METHOD_WITH_ARGS%d( %s, %s, call_args ); } """ % ( ", ".join(arg_names), to_name, arg_size, called_name, called_attribute_name ) ) getReleaseCodes( release_names = [called_name] + arg_names, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodePosArgsQuick(to_name, called_name, arg_names, needs_check, emit, context): arg_size = len(arg_names) quick_calls_used.add(arg_size) # For 0 arguments, NOARGS is supposed to be used. assert arg_size > 0 emitLineNumberUpdateCode(emit, context) emit( """\ { PyObject *call_args[] = { %s }; %s = CALL_FUNCTION_WITH_ARGS%d( %s, call_args ); } """ % ( ", ".join(arg_names), to_name, arg_size, called_name, ) ) getReleaseCodes( release_names = [called_name] + arg_names, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getInstanceCallCodeFromTuple(to_name, called_name, called_attribute_name, arg_tuple, arg_size, needs_check, emit, context): quick_instance_calls_used.add(arg_size) # For 0 arguments, NOARGS is supposed to be used. assert arg_size > 0 emitLineNumberUpdateCode(emit, context) emit( """\ %s = CALL_METHOD_WITH_ARGS%d( %s, %s, &PyTuple_GET_ITEM( %s, 0 ) ); """ % ( to_name, arg_size, called_name, called_attribute_name, arg_tuple, ) ) getReleaseCodes( release_names = ( called_name, called_attribute_name ), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodeFromTuple(to_name, called_name, arg_tuple, arg_size, needs_check, emit, context): quick_calls_used.add(arg_size) # For 0 arguments, NOARGS is supposed to be used. assert arg_size > 0 emitLineNumberUpdateCode(emit, context) emit( """\ %s = CALL_FUNCTION_WITH_ARGS%d( %s, &PyTuple_GET_ITEM( %s, 0 ) ); """ % ( to_name, arg_size, called_name, arg_tuple, ) ) getReleaseCode( release_name = called_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getInstanceCallCodePosArgs(to_name, called_name, called_attribute_name, args_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_METHOD_WITH_POSARGS( %s, %s, %s );" % ( to_name, called_name, called_attribute_name, args_name ) ) getReleaseCodes( release_names = (called_name, args_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodePosArgs(to_name, called_name, args_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION_WITH_POSARGS( %s, %s );" % ( to_name, called_name, args_name ) ) getReleaseCodes( release_names = (called_name, args_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def _getCallCodeKeywordArgs(to_name, called_name, call_kw_name, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION_WITH_KEYARGS( %s, %s );" % ( to_name, called_name, call_kw_name ) ) getReleaseCodes( release_names = (called_name, call_kw_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallCodePosKeywordArgs(to_name, called_name, call_args_name, call_kw_name, emit, context): emitLineNumberUpdateCode(emit, context) emit( "%s = CALL_FUNCTION( %s, %s, %s );" % ( to_name, called_name, call_args_name, call_kw_name ) ) getReleaseCodes( release_names = (called_name, call_args_name, call_kw_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getCallsDecls(): result = [] for quick_call_used in sorted(quick_calls_used.union(quick_instance_calls_used)): result.append( template_call_function_with_args_decl % { "args_count" : quick_call_used } ) for quick_call_used in sorted(quick_instance_calls_used): result.append( template_call_method_with_args_decl % { "args_count" : quick_call_used } ) return template_header_guard % { "header_guard_name" : "__NUITKA_CALLS_H__", "header_body" : '\n'.join(result) } def getCallsCode(): result = [] result.append( template_helper_impl_decl % {} ) for quick_call_used in sorted(quick_calls_used.union(quick_instance_calls_used)): result.append( template_call_function_with_args_impl % { "args_count" : quick_call_used } ) for quick_call_used in sorted(quick_instance_calls_used): result.append( template_call_method_with_args_impl % { "args_count" : quick_call_used } ) return '\n'.join(result) Nuitka-0.5.28.2/nuitka/codegen/EvalCodes.py0000644000372000001440000003022213207537242020605 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Eval/exec/execfile/compile built-in related codes. """ from nuitka import Options from nuitka.PythonVersions import python_version from .CodeHelpers import generateExpressionCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .VariableCodes import getVariableAssignmentCode def getStoreLocalsCode(locals_name, variables, is_foreign, emit, context): for variable, version in variables: if not variable.isModuleVariable(): key_name = context.getConstantCode( constant = variable.getName() ) value_name = context.allocateTempName("locals_value", unique = True) if is_foreign: emit( "%s = PyObject_GetItem( %s, %s );" % ( value_name, locals_name, key_name, ) ) getErrorExitBoolCode( condition = """\ %s == NULL && !EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_KeyError )""" % value_name, emit = emit, context = context ) emit("CLEAR_ERROR_OCCURRED();") context.addCleanupTempName(value_name) else: emit( "%s = PyDict_GetItem( %s, %s );" % ( value_name, locals_name, key_name, ) ) emit("if ( %s != NULL )" % value_name) emit('{') getVariableAssignmentCode( variable = variable, version = version, tmp_name = value_name, needs_release = None, # TODO: Could be known maybe. in_place = False, emit = emit, context = context ) emit('}') def generateBuiltinCompileCode(to_name, expression, emit, context): source_name = context.allocateTempName("compile_source") filename_name = context.allocateTempName("compile_filename") mode_name = context.allocateTempName("compile_mode") generateExpressionCode( to_name = source_name, expression = expression.getSourceCode(), emit = emit, context = context ) generateExpressionCode( to_name = filename_name, expression = expression.getFilename(), emit = emit, context = context ) generateExpressionCode( to_name = mode_name, expression = expression.getMode(), emit = emit, context = context ) if expression.getFlags() is not None: flags_name = context.allocateTempName("compile_flags") generateExpressionCode( to_name = flags_name, expression = expression.getFlags(), emit = emit, context = context ) else: flags_name = "NULL" if expression.getDontInherit() is not None: dont_inherit_name = context.allocateTempName("compile_dont_inherit") generateExpressionCode( to_name = dont_inherit_name, expression = expression.getDontInherit(), emit = emit, context = context ) else: dont_inherit_name = "NULL" if expression.getOptimize() is not None: optimize_name = context.allocateTempName("compile_dont_inherit") generateExpressionCode( to_name = optimize_name, expression = expression.getOptimize(), emit = emit, context = context ) else: optimize_name = "NULL" context.setCurrentSourceCodeReference( expression.getCompatibleSourceReference() ) getBuiltinCompileCode( to_name = to_name, source_name = source_name, filename_name = filename_name, mode_name = mode_name, flags_name = flags_name, dont_inherit_name = dont_inherit_name, optimize_name = optimize_name, emit = emit, context = context ) def getBuiltinCompileCode(to_name, source_name, filename_name, mode_name, flags_name, dont_inherit_name, optimize_name, emit, context): if python_version < 300: args = ( source_name, filename_name, mode_name, flags_name, dont_inherit_name ) else: args = ( source_name, filename_name, mode_name, flags_name, dont_inherit_name, optimize_name ) emit( "%s = COMPILE_CODE( %s );" % ( to_name, ", ".join(args) ) ) getReleaseCodes( release_names = (source_name, filename_name, mode_name, flags_name, dont_inherit_name, optimize_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def getBuiltinEvalCode(to_name, source_name, filename_name, globals_name, locals_name, mode_name, emit, context): compiled_name = context.allocateTempName("eval_compiled") getBuiltinCompileCode( to_name = compiled_name, source_name = source_name, filename_name = filename_name, mode_name = mode_name, flags_name = "NULL", dont_inherit_name = "NULL", optimize_name = "NULL", emit = emit, context = context ) emit( "%s = EVAL_CODE( %s, %s, %s );" % ( to_name, compiled_name, globals_name, locals_name ) ) getReleaseCodes( release_names = (compiled_name, globals_name, locals_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateExecCode(statement, emit, context): source_arg = statement.getSourceCode() globals_arg = statement.getGlobals() locals_arg = statement.getLocals() source_name = context.allocateTempName("exec_source") globals_name = context.allocateTempName("exec_globals") locals_name = context.allocateTempName("exec_locals") generateExpressionCode( to_name = source_name, expression = source_arg, emit = emit, context = context ) generateExpressionCode( to_name = globals_name, expression = globals_arg, emit = emit, context = context ) generateExpressionCode( to_name = locals_name, expression = locals_arg, emit = emit, context = context ) source_ref = statement.getSourceReference() # Filename with origin in improved mode. if Options.isFullCompat(): filename_name = context.getConstantCode( constant = "" ) else: filename_name = context.getConstantCode( constant = "" % source_ref.getAsString() ) old_source_ref = context.setCurrentSourceCodeReference( locals_arg.getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) compiled_name = context.allocateTempName("exec_compiled") getBuiltinCompileCode( to_name = compiled_name, source_name = source_name, filename_name = filename_name, mode_name = context.getConstantCode( constant = "exec" ), flags_name = "NULL", dont_inherit_name = "NULL", optimize_name = "NULL", emit = emit, context = context ) to_name = context.allocateTempName("exec_result") emit( "%s = EVAL_CODE( %s, %s, %s );" % ( to_name, compiled_name, globals_name, locals_name ) ) getReleaseCodes( release_names = (compiled_name, globals_name, locals_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) # Immediately release the exec result. context.addCleanupTempName(to_name) getReleaseCode( release_name = to_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def _generateEvalCode(to_name, node, emit, context): source_name = context.allocateTempName("eval_source") globals_name = context.allocateTempName("eval_globals") locals_name = context.allocateTempName("eval_locals") generateExpressionCode( to_name = source_name, expression = node.getSourceCode(), emit = emit, context = context ) generateExpressionCode( to_name = globals_name, expression = node.getGlobals(), emit = emit, context = context ) generateExpressionCode( to_name = locals_name, expression = node.getLocals(), emit = emit, context = context ) if node.isExpressionBuiltinEval() or \ (python_version >= 300 and node.isExpressionBuiltinExec()): filename = "" else: filename = "" getBuiltinEvalCode( to_name = to_name, source_name = source_name, globals_name = globals_name, locals_name = locals_name, filename_name = context.getConstantCode( constant = filename ), mode_name = context.getConstantCode( constant = "eval" if node.isExpressionBuiltinEval() else "exec" ), emit = emit, context = context ) def generateEvalCode(to_name, expression, emit, context): return _generateEvalCode( to_name = to_name, node = expression, emit = emit, context = context ) def generateExecfileCode(to_name, expression, emit, context): assert python_version < 300 return _generateEvalCode( to_name = to_name, node = expression, emit = emit, context = context ) def generateLocalsDictSyncCode(statement, emit, context): locals_arg = statement.getLocals() locals_name = context.allocateTempName("sync_locals") generateExpressionCode( to_name = locals_name, expression = locals_arg, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( statement.getSourceReference() ) getStoreLocalsCode( locals_name = locals_name, variables = statement.previous_traces, is_foreign = context.hasForeignLocalsDict(), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) Nuitka-0.5.28.2/nuitka/codegen/ImportCodes.py0000644000372000001440000001650113207537242021174 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Import related codes. That is import as expression, and star import. """ from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import ( getErrorExitBoolCode, getErrorExitCode, getReleaseCode, getReleaseCodes ) from .LineNumberCodes import emitLineNumberUpdateCode from .ModuleCodes import getModuleAccessCode def generateBuiltinImportCode(to_name, expression, emit, context): # We know that 5 expressions are created, pylint: disable=W0632 module_name, globals_name, locals_name, import_list_name, level_name = \ generateChildExpressionsCode( expression = expression, emit = emit, context = context ) getBuiltinImportCode( to_name = to_name, module_name = module_name, globals_name = globals_name, locals_name = locals_name, import_list_name = import_list_name, level_name = level_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) def getCountedArgumentsHelperCallCode(helper_prefix, to_name, args, min_args, needs_check, emit, context): orig_args = args args = list(args) while args[-1] is None: del args[-1] if None in args: emit( "%s = %s_KW( %s );" % ( to_name, helper_prefix, ", ".join( "NULL" if arg is None else arg for arg in orig_args ) ) ) else: # Check that no following arguments are not None. assert len(args) >= min_args emit( "%s = %s%d( %s );" % ( to_name, helper_prefix, len(args), ", ".join(args) ) ) getReleaseCodes( release_names = args, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) context.addCleanupTempName(to_name) def getBuiltinImportCode(to_name, module_name, globals_name, locals_name, import_list_name, level_name, needs_check, emit, context): emitLineNumberUpdateCode(emit, context) getCountedArgumentsHelperCallCode( helper_prefix = "IMPORT_MODULE", to_name = to_name, args = ( module_name, globals_name, locals_name, import_list_name, level_name ), min_args = 1, needs_check = needs_check, emit = emit, context = context ) def generateImportModuleHardCode(to_name, expression, emit, context): module_name = expression.getModuleName() needs_check = expression.mayRaiseException(BaseException) emitLineNumberUpdateCode(emit, context) emit( """%s = PyImport_ImportModule("%s");""" % ( to_name, module_name ) ) getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) def generateImportModuleNameHardCode(to_name, expression, emit, context): module_name = expression.getModuleName() import_name = expression.getImportName() needs_check = expression.mayRaiseException(BaseException) if module_name == "sys": emit( """%s = PySys_GetObject( (char *)"%s" );""" % ( to_name, import_name ) ) elif module_name in ("os", "__future__", "importlib._bootstrap"): emitLineNumberUpdateCode(emit, context) emit( """\ { PyObject *module = PyImport_ImportModule("%(module_name)s"); if (likely( module != NULL )) { %(to_name)s = PyObject_GetAttr( module, %(import_name)s ); } else { %(to_name)s = NULL; } } """ % { "to_name" : to_name, "module_name" : module_name, "import_name" : context.getConstantCode(import_name) } ) else: assert False, module_name getErrorExitCode( check_name = to_name, needs_check = needs_check, emit = emit, context = context ) def generateImportStarCode(statement, emit, context): module_name = context.allocateTempName("star_imported") generateExpressionCode( to_name = module_name, expression = statement.getSourceModule(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) res_name = context.getBoolResName() if not context.hasLocalsDict(): emit( "%s = IMPORT_MODULE_STAR( %s, true, %s );" % ( res_name, getModuleAccessCode( context = context ), module_name ) ) else: emit( "%s = IMPORT_MODULE_STAR( %s, false, %s );" % ( res_name, context.getLocalsDictName(), module_name ) ) getReleaseCode( release_name = module_name, emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generateImportNameCode(to_name, expression, emit, context): from_arg_name = context.allocateTempName("import_name_from") generateExpressionCode( to_name = from_arg_name, expression = expression.getModule(), emit = emit, context = context ) emit( "%s = IMPORT_NAME( %s, %s );" % ( to_name, from_arg_name, context.getConstantCode( constant = expression.getImportName() ) ) ) getReleaseCode( release_name = from_arg_name, emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) context.addCleanupTempName(to_name) Nuitka-0.5.28.2/nuitka/codegen/ErrorCodes.py0000644000372000001440000002632613207537242021021 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Error codes These are the helper functions that will emit the error exit codes. They can abstractly check conditions or values directly. The release of statement temporaries from context is automatic. Also formatting errors is done here, avoiding PyErr_Format as much as possible. And releasing of values, as this is what the error case commonly does. """ from nuitka import Options from nuitka.PythonVersions import python_version from .ExceptionCodes import getExceptionIdentifier from .Indentation import getCommentCode, indented from .LineNumberCodes import getErrorLineNumberUpdateCode from .templates.CodeTemplatesExceptions import ( template_error_catch_exception, template_error_catch_quick_exception, template_error_format_string_exception ) def getErrorExitReleaseCode(context): temp_release = '\n'.join( "Py_DECREF( %s );" % tmp_name for tmp_name in context.getCleanupTempnames() ) keeper_variables = context.getExceptionKeeperVariables() if keeper_variables[0] is not None: temp_release += "\nPy_DECREF( %s );" % keeper_variables[0] temp_release += "\nPy_XDECREF( %s );" % keeper_variables[1] temp_release += "\nPy_XDECREF( %s );" % keeper_variables[2] return temp_release def getFrameVariableTypeDescriptionCode(context): type_description = context.getFrameVariableTypeDescription() if type_description: return '%s = "%s";' % ( context.getFrameVariableTypeDescriptionName(), type_description, ) else: return "" def getErrorExitBoolCode(condition, emit, context, needs_check = True, quick_exception = None): assert not condition.endswith(';') if not needs_check: getAssertionCode("!(%s)" % condition, emit) return context.markAsNeedsExceptionVariables() if quick_exception: emit( indented( template_error_catch_quick_exception % { "condition" : condition, "exception_exit" : context.getExceptionEscape(), "quick_exception" : getExceptionIdentifier(quick_exception), "release_temps" : indented( getErrorExitReleaseCode(context) ), "var_description_code" : indented( getFrameVariableTypeDescriptionCode(context), ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context) ) }, 0 ) ) else: emit( indented( template_error_catch_exception % { "condition" : condition, "exception_exit" : context.getExceptionEscape(), "release_temps" : indented( getErrorExitReleaseCode(context) ), "var_description_code": indented( getFrameVariableTypeDescriptionCode(context), ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context) ) }, 0 ) ) def getErrorExitCode(check_name, emit, context, quick_exception = None, needs_check = True): if needs_check: getErrorExitBoolCode( condition = "%s == NULL" % check_name, quick_exception = quick_exception, emit = emit, context = context ) else: getAssertionCode("%s != NULL" % check_name, emit) def getErrorFormatExitBoolCode(condition, exception, args, emit, context): assert not condition.endswith(';') context.markAsNeedsExceptionVariables() if len(args) == 1 and type(args[0]) is str: from .ConstantCodes import getModuleConstantCode set_exception = [ "exception_type = %s;" % exception, "Py_INCREF( exception_type );", "exception_value = %s;" % getModuleConstantCode( constant = args[0], ), "exception_tb = NULL;" ] else: set_exception = [ "exception_type = %s;" % exception, "Py_INCREF( exception_type );", "exception_value = Py%s_FromFormat( %s );" % ( "String" if python_version < 300 else "Unicode", ", ".join( '"%s"' % arg for arg in args ) ), "exception_tb = NULL;" ] if python_version >= 300: keeper_vars = context.getExceptionKeeperVariables() if keeper_vars[0] is not None: set_exception.append( "ADD_EXCEPTION_CONTEXT( &%s, &%s );" % ( keeper_vars[0], keeper_vars[1] ) ) else: set_exception.append( "NORMALIZE_EXCEPTION( &exception_type, &exception_value, &exception_tb );" ) set_exception.append( "CHAIN_EXCEPTION( exception_value );" ) emit( template_error_format_string_exception % { "condition" : condition, "exception_exit" : context.getExceptionEscape(), "set_exception" : indented(set_exception), "release_temps" : indented( getErrorExitReleaseCode(context) ), "var_description_code" : indented( getFrameVariableTypeDescriptionCode(context), ), "line_number_code" : indented( getErrorLineNumberUpdateCode(context) ) } ) def getErrorVariableDeclarations(): return ( "PyObject *exception_type = NULL;", "PyObject *exception_value = NULL;", "PyTracebackObject *exception_tb = NULL;", "NUITKA_MAY_BE_UNUSED int exception_lineno = 0;" ) def getExceptionKeeperVariableNames(keeper_index): # For finally handlers of Python3, which have conditions on assign and # use. debug = Options.isDebug() and python_version >= 300 if debug: keeper_obj_init = " = NULL" else: keeper_obj_init = "" return ( "PyObject *exception_keeper_type_%d%s;" % ( keeper_index, keeper_obj_init ), "PyObject *exception_keeper_value_%d%s;" % ( keeper_index, keeper_obj_init ), "PyTracebackObject *exception_keeper_tb_%d%s;" % ( keeper_index, keeper_obj_init ), "NUITKA_MAY_BE_UNUSED int exception_keeper_lineno_%d%s;" % ( keeper_index, " = 0" if debug else "" ) ) def getExceptionPreserverVariableNames(preserver_id): # For finally handlers of Python3, which have conditions on assign and # use. debug = Options.isDebug() and python_version >= 300 if debug: preserver_obj_init = " = NULL" else: preserver_obj_init = "" return ( "PyObject *exception_preserved_type_%d%s;" % ( preserver_id, preserver_obj_init ), "PyObject *exception_preserved_value_%d%s;" % ( preserver_id, preserver_obj_init ), "PyTracebackObject *exception_preserved_tb_%d%s;" % ( preserver_id, preserver_obj_init ), ) def getErrorFormatExitCode(check_name, exception, args, emit, context): getErrorFormatExitBoolCode( condition = "%s == NULL" % check_name, exception = exception, args = args, emit = emit, context = context ) def getReleaseCode(release_name, emit, context): assert release_name is None or len(release_name) > 2 if context.needsCleanup(release_name): emit("Py_DECREF( %s );" % release_name) context.removeCleanupTempName(release_name) def getReleaseCodes(release_names, emit, context): for release_name in release_names: getReleaseCode( release_name = release_name, emit = emit, context = context ) def getMustNotGetHereCode(reason, context, emit): getCommentCode(reason, emit) provider = context.getEntryPoint() emit( "NUITKA_CANNOT_GET_HERE( %(function_identifier)s );" % { "function_identifier" : provider.getCodeName() } ) if provider.isExpressionGeneratorObjectBody(): emit("return;") elif provider.isExpressionCoroutineObjectBody(): emit("return;") elif provider.isExpressionAsyncgenObjectBody(): emit("return;") elif provider.isCompiledPythonModule(): emit("return MOD_RETURN_VALUE( NULL );") else: emit("return NULL;") def getAssertionCode(check, emit): emit("assert( %s );" % check) def getCheckObjectCode(check_name, emit): emit("CHECK_OBJECT( %s );" % check_name) def getLocalVariableReferenceErrorCode(variable, condition, emit, context): if variable.getOwner() is not context.getOwner(): getErrorFormatExitBoolCode( condition = condition, exception = "PyExc_NameError", args = ( """\ free variable '%s' referenced before assignment in enclosing scope""", variable.getName() ), emit = emit, context = context ) else: getErrorFormatExitBoolCode( condition = condition, exception = "PyExc_UnboundLocalError", args = ( """\ local variable '%s' referenced before assignment""", variable.getName() ), emit = emit, context = context ) def getNameReferenceErrorCode(variable_name, condition, emit, context): if python_version < 340: owner = context.getOwner() if not owner.isCompiledPythonModule() and \ not owner.isExpressionClassBody(): error_message = "global name '%s' is not defined" else: error_message = "name '%s' is not defined" else: error_message = "name '%s' is not defined" getErrorFormatExitBoolCode( condition = condition, exception = "PyExc_NameError", args = ( error_message, variable_name ), emit = emit, context = context ) Nuitka-0.5.28.2/nuitka/codegen/PrintCodes.py0000644000372000001440000000657413207537242021027 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Print related codes. This is broken down to to level on printing one individual item, and a new line potentially. The heavy lifting for 'softspace', etc. is happening in the C helper functions. """ from .CodeHelpers import generateExpressionCode from .ErrorCodes import getErrorExitBoolCode, getReleaseCode, getReleaseCodes def generatePrintValueCode(statement, emit, context): destination = statement.getDestination() value = statement.getValue() if destination is not None: dest_name = context.allocateTempName("print_dest", unique = True) generateExpressionCode( expression = destination, to_name = dest_name, emit = emit, context = context ) else: dest_name = None value_name = context.allocateTempName("print_value", unique = True) generateExpressionCode( expression = value, to_name = value_name, emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) if dest_name is not None: print_code = "PRINT_ITEM_TO( %s, %s ) == false" % ( dest_name, value_name ) else: print_code = "PRINT_ITEM( %s ) == false" % ( value_name, ) getErrorExitBoolCode( condition = print_code, emit = emit, context = context ) getReleaseCodes( release_names = (dest_name, value_name), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) def generatePrintNewlineCode(statement, emit, context): destination = statement.getDestination() if destination is not None: dest_name = context.allocateTempName("print_dest", unique = True) generateExpressionCode( expression = destination, to_name = dest_name, emit = emit, context = context ) else: dest_name = None old_source_ref = context.setCurrentSourceCodeReference(statement.getSourceReference()) if dest_name is not None: print_code = "PRINT_NEW_LINE_TO( %s ) == false" % ( dest_name, ) else: print_code = "PRINT_NEW_LINE() == false" getErrorExitBoolCode( condition = print_code, emit = emit, context = context ) getReleaseCode( release_name = dest_name, emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) Nuitka-0.5.28.2/nuitka/codegen/DictCodes.py0000644000372000001440000002472113207537242020610 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code generation for dictionaries. """ from nuitka import Options from nuitka.PythonVersions import python_version from .CodeHelpers import generateChildExpressionsCode, generateExpressionCode from .ErrorCodes import getErrorExitBoolCode, getErrorExitCode, getReleaseCodes def generateBuiltinDictCode(to_name, expression, emit, context): if expression.getPositionalArgument(): seq_name = context.allocateTempName("dict_seq") generateExpressionCode( to_name = seq_name, expression = expression.getPositionalArgument(), emit = emit, context = context, allow_none = True ) else: seq_name = None if expression.getNamedArgumentPairs(): # If there is no sequence to mix in, then directly generate # into to_name. if seq_name is None: getDictionaryCreationCode( to_name = to_name, pairs = expression.getNamedArgumentPairs(), emit = emit, context = context ) dict_name = None else: dict_name = context.allocateTempName("dict_arg") getDictionaryCreationCode( to_name = dict_name, pairs = expression.getNamedArgumentPairs(), emit = emit, context = context ) else: dict_name = None if seq_name is not None: emit( "%s = TO_DICT( %s, %s );" % ( to_name, seq_name, "NULL" if dict_name is None else dict_name ) ) getReleaseCodes( release_names = (seq_name, dict_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, emit = emit, context = context ) context.addCleanupTempName(to_name) def generateDictionaryCreationCode(to_name, expression, emit, context): getDictionaryCreationCode( to_name = to_name, pairs = expression.getPairs(), emit = emit, context = context ) def getDictionaryCreationCode(to_name, pairs, emit, context): emit( "%s = _PyDict_NewPresized( %d );" % ( to_name, len(pairs) ) ) context.addCleanupTempName(to_name) def generateValueCode(dict_value_name, pair): generateExpressionCode( to_name = dict_value_name, expression = pair.getValue(), emit = emit, context = context ) def generateKeyCode(dict_key_name, pair): generateExpressionCode( to_name = dict_key_name, expression = pair.getKey(), emit = emit, context = context ) # Strange as it is, CPython evaluates the key/value pairs strictly in order, # but for each pair, the value first. for pair in pairs: dict_key_name = context.allocateTempName("dict_key") dict_value_name = context.allocateTempName("dict_value") if python_version < 350: generateValueCode(dict_value_name, pair) generateKeyCode(dict_key_name, pair) else: generateKeyCode(dict_key_name, pair) generateValueCode(dict_value_name, pair) needs_check = not pair.getKey().isKnownToBeHashable() res_name = context.getIntResName() emit( "%s = PyDict_SetItem( %s, %s, %s );" % ( res_name, to_name, dict_key_name, dict_value_name ) ) if context.needsCleanup(dict_value_name): emit("Py_DECREF( %s );" % dict_value_name) context.removeCleanupTempName(dict_value_name) if context.needsCleanup(dict_key_name): emit("Py_DECREF( %s );" % dict_key_name) context.removeCleanupTempName(dict_key_name) getErrorExitBoolCode( condition = "%s != 0" % res_name, needs_check = needs_check, emit = emit, context = context ) def generateDictOperationUpdateCode(statement, emit, context): value_arg_name = context.allocateTempName("dictupdate_value", unique = True) generateExpressionCode( to_name = value_arg_name, expression = statement.getValue(), emit = emit, context = context ) dict_arg_name = context.allocateTempName("dictupdate_dict", unique = True) generateExpressionCode( to_name = dict_arg_name, expression = statement.getDict(), emit = emit, context = context ) res_name = context.getIntResName() emit("assert( PyDict_Check( %s ) );" % dict_arg_name) emit( "%s = PyDict_Update( %s, %s );" % ( res_name, dict_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (dict_arg_name, value_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s != 0" % res_name, needs_check = statement.mayRaiseException(BaseException), emit = emit, context = context ) def generateDictOperationGetCode(to_name, expression, emit, context): dict_name, key_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) emit( "%s = DICT_GET_ITEM( %s, %s );" % ( to_name, dict_name, key_name ) ) getReleaseCodes( release_names = (dict_name, key_name), emit = emit, context = context ) getErrorExitCode( check_name = to_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) context.addCleanupTempName(to_name) def generateDictOperationInCode(to_name, expression, emit, context): inverted = expression.isExpressionDictOperationNOTIn() dict_name, key_name = generateChildExpressionsCode( expression = expression, emit = emit, context = context ) res_name = context.getIntResName() emit( "%s = PyDict_Contains( %s, %s );" % ( res_name, key_name, dict_name ) ) getReleaseCodes( release_names = (dict_name, key_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == -1" % res_name, needs_check = expression.mayRaiseException(BaseException), emit = emit, context = context ) emit( "%s = BOOL_FROM( %s == %s );" % ( to_name, res_name, '1' if not inverted else '0' ) ) def generateDictOperationSetCode(statement, emit, context): value_arg_name = context.allocateTempName("dictset_value", unique = True) generateExpressionCode( to_name = value_arg_name, expression = statement.getValue(), emit = emit, context = context ) dict_arg_name = context.allocateTempName("dictset_dict", unique = True) generateExpressionCode( to_name = dict_arg_name, expression = statement.getDict(), emit = emit, context = context ) key_arg_name = context.allocateTempName("dictset_key", unique = True) generateExpressionCode( to_name = key_arg_name, expression = statement.getKey(), emit = emit, context = context ) context.setCurrentSourceCodeReference(statement.getSourceReference()) res_name = context.getIntResName() emit( "%s = PyDict_SetItem( %s, %s, %s );" % ( res_name, dict_arg_name, key_arg_name, value_arg_name ) ) getReleaseCodes( release_names = (value_arg_name, dict_arg_name, key_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s != 0" % res_name, emit = emit, needs_check = not statement.getKey().isKnownToBeHashable(), context = context ) def generateDictOperationRemoveCode(statement, emit, context): dict_arg_name = context.allocateTempName("dictdel_dict", unique = True) generateExpressionCode( to_name = dict_arg_name, expression = statement.getDict(), emit = emit, context = context ) key_arg_name = context.allocateTempName("dictdel_key", unique = True) generateExpressionCode( to_name = key_arg_name, expression = statement.getKey(), emit = emit, context = context ) old_source_ref = context.setCurrentSourceCodeReference( statement.getKey().getSourceReference() if Options.isFullCompat() else statement.getSourceReference() ) res_name = context.getBoolResName() emit( "%s = DICT_REMOVE_ITEM( %s, %s );" % ( res_name, dict_arg_name, key_arg_name ) ) getReleaseCodes( release_names = (dict_arg_name, key_arg_name), emit = emit, context = context ) getErrorExitBoolCode( condition = "%s == false" % res_name, needs_check = statement.mayRaiseException(BaseException), emit = emit, context = context ) context.setCurrentSourceCodeReference(old_source_ref) Nuitka-0.5.28.2/nuitka/codegen/ReturnCodes.py0000644000372000001440000001243213207537242021200 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Return codes This handles code generation for return statements of normal functions and of generator functions. Also the value currently being returned, and intercepted by a try statement is accessible this way. """ from nuitka.PythonVersions import python_version from .CodeHelpers import generateExpressionCode from .ConstantCodes import getConstantAccess from .ExceptionCodes import getExceptionUnpublishedReleaseCode from .LabelCodes import getGotoCode def generateReturnCode(statement, emit, context): getExceptionUnpublishedReleaseCode(emit, context) return_value = statement.getExpression() if not return_value.isExpressionReturnedValueRef(): return_value_name = context.getReturnValueName() if context.getReturnReleaseMode(): emit("Py_DECREF( %s );" % return_value_name) generateExpressionCode( to_name = return_value_name, expression = return_value, emit = emit, context = context ) if context.needsCleanup(return_value_name): context.removeCleanupTempName(return_value_name) else: emit( "Py_INCREF( %s );" % return_value_name ) getGotoCode( label = context.getReturnTarget(), emit = emit ) def generateReturnConstantCode(statement, emit, context): getExceptionUnpublishedReleaseCode(emit, context) return_value_name = context.getReturnValueName() if context.getReturnReleaseMode(): emit("Py_DECREF( %s );" % return_value_name) getConstantAccess( to_name = return_value_name, constant = statement.getConstant(), emit = emit, context = context ) if context.needsCleanup(return_value_name): context.removeCleanupTempName(return_value_name) else: emit( "Py_INCREF( %s );" % return_value_name ) getGotoCode( label = context.getReturnTarget(), emit = emit ) def generateReturnedValueRefCode(to_name, expression, emit, context): # We don't need the expression, pylint: disable=unused-argument return_value_name = context.getReturnValueName() emit( "%s = %s;" % ( to_name, return_value_name ) ) def generateGeneratorReturnValueCode(statement, emit, context): if context.getOwner().isExpressionAsyncgenObjectBody(): pass elif python_version >= 330: return_value_name = context.getGeneratorReturnValueName() expression = statement.getExpression() if context.getReturnReleaseMode(): emit("Py_DECREF( %s );" % return_value_name) generateExpressionCode( to_name = return_value_name, expression = expression, emit = emit, context = context ) if context.needsCleanup(return_value_name): context.removeCleanupTempName(return_value_name) else: emit( "Py_INCREF( %s );" % return_value_name ) elif statement.getParentVariableProvider().needsGeneratorReturnHandling(): return_value_name = context.getGeneratorReturnValueName() generator_return_name = context.allocateTempName( "generator_return", "bool", unique = True ) emit("%s = true;" % generator_return_name) getGotoCode(context.getReturnTarget(), emit) def generateGeneratorReturnNoneCode(statement, emit, context): if context.getOwner().isExpressionAsyncgenObjectBody(): pass elif python_version >= 330: return_value_name = context.getGeneratorReturnValueName() if context.getReturnReleaseMode(): emit("Py_DECREF( %s );" % return_value_name) getConstantAccess( to_name = return_value_name, constant = None, emit = emit, context = context ) if context.needsCleanup(return_value_name): context.removeCleanupTempName(return_value_name) else: emit( "Py_INCREF( %s );" % return_value_name ) elif statement.getParentVariableProvider().needsGeneratorReturnHandling(): return_value_name = context.getGeneratorReturnValueName() generator_return_name = context.allocateTempName( "generator_return", "bool", unique = True ) emit("%s = true;" % generator_return_name) getGotoCode(context.getReturnTarget(), emit) Nuitka-0.5.28.2/nuitka/Builtins.py0000644000372000001440000001513113134660221017120 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-ins module. Information about built-ins of the running Python. """ import functools import sys from types import BuiltinFunctionType, FunctionType, GeneratorType from nuitka.__past__ import iterItems from nuitka.PythonVersions import python_version def _getBuiltinExceptionNames(): def isExceptionName(builtin_name): if builtin_name.endswith("Error") or \ builtin_name.endswith("Exception"): return True elif builtin_name in ("StopIteration", "GeneratorExit", "SystemExit", "NotImplemented", "KeyboardInterrupt", "StopAsyncIteration"): return True else: return False # Hide Python3 changes for built-in exception names try: import exceptions names = [ str(name) for name in dir(exceptions) if isExceptionName(name) ] values = {} for key in names: values[key] = getattr(exceptions, key) for key in dir(sys.modules["__builtin__"]): name = str(key) if isExceptionName(name): names.append(key) values[name] = getattr(sys.modules["__builtin__"], key) except ImportError: exceptions = {} for key, value in sys.modules["builtins"].__dict__.items(): if isExceptionName(key): exceptions[key] = value names = [ key for key, value in exceptions.items() ] values = {} for key, value in exceptions.items(): values[key] = value return names, values builtin_exception_names, builtin_exception_values = _getBuiltinExceptionNames() # Just to make sure it's covering these cases correctly. assert "TypeError" in builtin_exception_names assert "ValueError" in builtin_exception_names assert "StopIteration" in builtin_exception_names assert "GeneratorExit" in builtin_exception_names assert "AssertionError" in builtin_exception_names assert "BaseException" in builtin_exception_names assert "Exception" in builtin_exception_names assert "NotImplemented" in builtin_exception_names assert "StopAsyncIteration" in builtin_exception_names or python_version < 350 def _getBuiltinNames(): names = [ str(x) for x in __builtins__.keys() ] for builtin_exception_name in builtin_exception_names: if builtin_exception_name in names: names.remove(builtin_exception_name) names.remove("__doc__") names.remove("__name__") names.remove("__package__") if "__loader__" in names: names.remove("__loader__") warnings = [] for builtin_name in names: if builtin_name.endswith("Warning"): warnings.append(builtin_name) for builtin_name in warnings: names.remove(builtin_name) return names, warnings builtin_names, builtin_warnings = _getBuiltinNames() assert "__import__" in builtin_names assert "int" in builtin_names assert "__doc__" not in builtin_names assert "sys" not in builtin_names builtin_all_names = builtin_names + builtin_exception_names + builtin_warnings def getBuiltinTypeNames(): result = [] for builtin_name in builtin_names: if isinstance(__builtins__[builtin_name], type): result.append(builtin_name) return tuple(sorted(result)) builtin_type_names = getBuiltinTypeNames() def _getAnonBuiltins(): with open(sys.executable) as any_file: anon_names = { # Strangely not Python3 types module "NoneType" : type(None), "ellipsis" : type(Ellipsis), # see above "NotImplementedType" : type(NotImplemented), "function" : FunctionType, "builtin_function_or_method" : BuiltinFunctionType, # Can't really have it any better way. "compiled_function" : BuiltinFunctionType, "generator" : GeneratorType, "compiled_generator" : GeneratorType, # see above "code" : type(_getAnonBuiltins.__code__), "file" : type(any_file) } anon_codes = { "NoneType" : "Py_TYPE( Py_None )", "ellipsis" : "&PyEllipsis_Type", "NotImplementedType" : "Py_TYPE( Py_NotImplemented )", "function" : "&PyFunction_Type", "builtin_function_or_method" : "&PyCFunction_Type", "compiled_function" : "&Nuitka_Function_Type", "compiled_generator" : "&Nuitka_Generator_Type", "code" : "&PyCode_Type", "file" : "&PyFile_Type" } if python_version < 300: # There are only there for Python2, # pylint: disable=I0021,no-name-in-module from types import ClassType, InstanceType, MethodType anon_names["classobj"] = ClassType anon_codes["classobj"] = "&PyClass_Type" anon_names["instance"] = InstanceType anon_codes["instance"] = "&PyInstance_Type" anon_names["instancemethod"] = MethodType anon_codes["instancemethod"] = "&PyMethod_Type" return anon_names, anon_codes builtin_anon_names, builtin_anon_codes = _getAnonBuiltins() def calledWithBuiltinArgumentNamesDecorator(f): """ Allow a function to be called with an "_arg" if a built-in name. This avoids using built-in names in Nuitka source, while enforcing a policy how to make them pretty. """ @functools.wraps(f) def wrapper(*args, **kw): new_kw = {} for key, value in iterItems(kw): if key in builtin_all_names: key = key + "_arg" new_kw[key] = value return f(*args, **new_kw) return wrapper Nuitka-0.5.28.2/nuitka/Version.py0000644000372000001440000000200513207540035016751 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nuitka version related stuff. """ version_string = """\ Nuitka V0.5.28.2 Copyright (C) 2017 Kay Hayen.""" def getNuitkaVersion(): return version_string.split()[1][1:] def getNuitkaVersionYear(): return int(version_string.split()[4]) Nuitka-0.5.28.2/nuitka/finalizations/0000755000372000001440000000000013207540420017625 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/finalizations/Finalization.py0000644000372000001440000000231013207537242022632 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Finalizations. Last steps directly before code creation is called. Here the final tasks are executed. Things normally volatile during optimization can be computed here, so the code generation can be quick and doesn't have to check it many times. """ from nuitka.tree import Operations from .FinalizeMarkups import FinalizeMarkups def prepareCodeGeneration(tree): visitor = FinalizeMarkups() Operations.visitTree(tree, visitor) Nuitka-0.5.28.2/nuitka/finalizations/FinalizeMarkups.py0000644000372000001440000001511013207537242023311 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Finalize the markups Set flags on functions and classes to indicate if a locals dict is really needed. Set a flag on loops if they really need to catch Continue and Break exceptions or if it can be more simple code. Set a flag on return statements and functions that require the use of "ReturnValue" exceptions, or if it can be more simple code. Set a flag on re-raises of exceptions if they can be simple throws or if they are in another context. """ from logging import warning from nuitka import Options, Tracing from nuitka.__past__ import unicode # pylint: disable=I0021,redefined-builtin from nuitka.importing.Importing import isWhiteListedImport from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import python_version from .FinalizeBase import FinalizationVisitorBase imported_names = set() def getImportedNames(): return imported_names class FinalizeMarkups(FinalizationVisitorBase): def onEnterNode(self, node): try: self._onEnterNode(node) except Exception: Tracing.printError( "Problem with %r at %s" % ( node, node.getSourceReference().getAsString() ) ) raise def _onEnterNode(self, node): # This has many different things it deals with, so there need to be a # lot of branches and statements, pylint: disable=too-many-branches,too-many-statements # Also all self specific things have been done on the outside, # pylint: disable=no-self-use if node.isExpressionFunctionBody(): if node.isUnoptimized(): node.markAsLocalsDict() if node.needsLocalsDict(): provider = node.getParentVariableProvider() provider.markAsLocalsDict() if node.isStatementSetLocals(): provider.markAsForeignLocalsDict() if node.isStatementReturn() or node.isStatementGeneratorReturn(): search = node in_tried_block = False # Search up to the containing function, and check for a try/finally # containing the "return" statement. search = search.getParentReturnConsumer() if search.isExpressionGeneratorObjectBody() or \ search.isExpressionCoroutineObjectBody() or \ search.isExpressionAsyncgenObjectBody(): if in_tried_block: search.markAsNeedsGeneratorReturnHandling(2) else: search.markAsNeedsGeneratorReturnHandling(1) if node.isExpressionBuiltinImport() and \ not Options.getShallFollowExtra() and \ not Options.getShallFollowExtraFilePatterns() and \ not Options.shallFollowNoImports() and \ not isWhiteListedImport(node) and \ not node.recurse_attempted and \ not Plugins.suppressBuiltinImportWarning(node.getParentModule(), node.getSourceReference()): warning("""Unresolved '__import__' call at '%s' may require use \ of '--recurse-directory'.""" % ( node.getSourceReference().getAsString() ) ) if node.isExpressionBuiltinImport() and \ node.recurse_attempted: module_name = node.getImportName() if module_name.isCompileTimeConstant(): imported_module_name = module_name.getCompileTimeConstant() if type(imported_module_name) in (str, unicode): if imported_module_name: imported_names.add(imported_module_name) if node.isExpressionFunctionCreation(): if not node.getParent().isExpressionFunctionCall() or \ node.getParent().getFunction() is not node: node.getFunctionRef().getFunctionBody().markAsNeedsCreation() if node.isExpressionFunctionCall(): node.getFunction().getFunctionRef().getFunctionBody().\ markAsDirectlyCalled() if node.isExpressionFunctionRef(): function_body = node.getFunctionBody() parent_module = function_body.getParentModule() node_module = node.getParentModule() if node_module is not parent_module: function_body.markAsCrossModuleUsed() node_module.addCrossUsedFunction(function_body) if node.isStatementAssignmentVariable(): target_var = node.getVariable() assign_source = node.getAssignSource() if assign_source.isExpressionOperationBinary(): left_arg = assign_source.getLeft() if left_arg.isExpressionVariableRef(): if assign_source.getLeft().getVariable().isModuleVariable(): assign_source.unmarkAsInplaceSuspect() elif assign_source.getLeft().getVariable() is target_var: if assign_source.isInplaceSuspect(): node.markAsInplaceSuspect() if node.isStatementPublishException(): node.getParentStatementsFrame().markAsFrameExceptionPreserving() if python_version >= 300: if node.isExpressionYield() or \ node.isExpressionYieldFrom() or \ node.isExpressionAsyncWait(): search = node.getParent() while not search.isExpressionGeneratorObjectBody() and \ not search.isExpressionCoroutineObjectBody() and \ not search.isExpressionAsyncgenObjectBody(): last_search = search search = search.getParent() if search.isStatementTry() and \ last_search == search.getBlockExceptHandler(): node.markAsExceptionPreserving() break Nuitka-0.5.28.2/nuitka/finalizations/FinalizeBase.py0000644000372000001440000000174713112214770022545 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Base for all finalization modules Provides a class that all finalization visitors should inherit from. """ from nuitka.tree import Operations class FinalizationVisitorBase(Operations.VisitorNoopMixin): pass Nuitka-0.5.28.2/nuitka/finalizations/__init__.py0000644000372000001440000000150113112214770021734 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/__past__.py0000644000372000001440000000605413134660221017076 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module like __future__ for things that are changed between Python2 and Python3. These are here to provide compatible fall-backs. This is required to run the same code easily with both CPython2 and CPython3. Sometimes, we do not care about the actual types, or API, but would rather just check for something to be a "in (str, unicode)" rather than making useless version checks. """ import sys # pylint: disable=I0021,invalid-name,redefined-builtin # Work around for CPython 3.x renaming "long" to "int". if str is bytes: long = long # @ReservedAssignment pylint: disable=I0021,undefined-variable else: long = int # @ReservedAssignment # Work around for CPython 3.x renaming "unicode" to "str". if str is bytes: unicode = unicode # @ReservedAssignment pylint: disable=I0021,undefined-variable else: unicode = str # @ReservedAssignment def iterItems(d): try: return d.iteritems() except AttributeError: return d.items() if str is not bytes: raw_input = input # @ReservedAssignment else: raw_input = raw_input # @ReservedAssignment if str is bytes: xrange = xrange # @ReservedAssignment pylint: disable=I0021,undefined-variable else: xrange = range # @ReservedAssignment try: from urllib.request import urlretrieve except ImportError: from urllib import urlretrieve try: from cStringIO import StringIO except ImportError: from io import StringIO try: from functools import total_ordering except ImportError: # Lame replacement for functools.total_ordering, which does not exist on # Python2.6, this requires "<" and "=" and adds all other operations. def total_ordering(cls): cls.__ne__ = lambda self, other: not self == other cls.__le__ = lambda self, other: self == other or self < other cls.__gt__ = lambda self, other: self != other and not self < other cls.__ge__ = lambda self, other: self == other and not self < other return cls if str is bytes: intern = intern # @ReservedAssignment pylint: disable=I0021,undefined-variable else: intern = sys.intern # @ReservedAssignment @UndefinedVariable # For PyLint to be happy. assert long assert unicode assert urlretrieve assert StringIO assert type(xrange) is type, xrange assert total_ordering assert intern Nuitka-0.5.28.2/nuitka/utils/0000755000372000001440000000000013207540420016113 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/utils/Shebang.py0000644000372000001440000000304113112214770020033 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Utils to work with shebang lines. """ import os import re def getShebangFromSource(source_code): if source_code.startswith("#!"): shebang = re.match(r"^#!\s*(.*?)\n", source_code) if shebang is not None: shebang = shebang.group(0).rstrip('\n') else: shebang = None return shebang def getShebangFromFile(filename): return getShebangFromSource(open(filename).readline()) def parseShebang(shebang): parts = shebang.split() if os.path.basename(parts[0]) == "env": # This attempts to handle env with arguments and options. del parts[0] while parts[0].startswith('-'): del parts[0] while '=' in parts[0]: del parts[0] return parts[0][2:].lstrip(), parts[1:] Nuitka-0.5.28.2/nuitka/utils/CStrings.py0000644000372000001440000000533613134660221020231 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ C string encoding This contains the code to create string literals for C to represent the given values and little more. """ import re def _encodePythonStringToC(value): """ Encode a string, so that it gives a C string literal. This doesn't handle limits. """ assert type(value) is bytes, type(value) result = "" octal = False for c in value: if str is bytes: cv = ord(c) else: cv = c if c in b'\\\t\r\n"?': result += r'\%o' % cv octal = True elif cv >= 32 and cv <= 127: if octal and c in b'0123456789': result += '" "' result += chr(cv) octal = False else: result += r'\%o' % cv octal = True result = result.replace('" "\\', '\\') return '"%s"' % result def encodePythonStringToC(value): """ Encode a string, so that it gives a C string literal. """ # Not all compilers allow arbitrary large C strings, therefore split it up # into chunks. That changes nothing to the meanings, but is easier on the # parser. Currently only MSVC is known to have this issue, but the # workaround can be used universally. result = _encodePythonStringToC(value[:16000 ]) value = value[16000:] while value: result += ' ' result += _encodePythonStringToC(value[:16000 ]) value = value[16000:] return result def encodePythonIdentifierToC(value): """ Encode an identifier from a given Python string. """ # Python identifiers allow almost of characters except a very # few, much more than C identifiers support. This attempts to # be bi-directional, so we can reverse it. def r(match): c = match.group() if c == '.': return '$' else: return "$$%d$" % ord(c) return "".join( re.sub("[^a-zA-Z0-9_]", r ,c) for c in value ) Nuitka-0.5.28.2/nuitka/utils/AppDirs.py0000644000372000001440000000427013207537242020042 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Wrapper around appdir from PyPI We do not assume to be installed and fallback to an inline copy and if that is not installed, we use our own code for best effort. """ from __future__ import absolute_import import os import sys import tempfile from .FileOperations import makePath try: import appdirs except ImportError: sys.path.append( os.path.join( os.path.dirname(__file__), "..", "build", "inline_copy", "appdirs" ) ) # Handle case without inline copy too. try: import appdirs except ImportError: appdirs = None def getCacheDir(): if appdirs is not None: result = appdirs.user_cache_dir("Nuitka", None) else: result = os.path.join(os.path.expanduser('~'), ".cache", "Nuitka") # For people that build with HOME set this, e.g. Debian. if result.startswith(("/nonexistent/", "/sbuild-nonexistent/")): result = os.path.join(tempfile.gettempdir(), "Nuitka") makePath(result) return result def getAppDir(): if appdirs is not None: result = appdirs.user_data_dir("Nuitka", None) else: result = os.path.join(os.path.expanduser('~'), ".config", "Nuitka") # For people that build with HOME set this, e.g. Debian. if result.startswith(("/nonexistent/", "/sbuild-nonexistent/")): result = os.path.join(tempfile.gettempdir(), "Nuitka") makePath(result) return result Nuitka-0.5.28.2/nuitka/utils/MemoryUsage.py0000644000372000001440000001054113134660221020724 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tools for tracing memory usage at compiled time. """ from nuitka.Tracing import printLine from .Utils import getOS def getOwnProcessMemoryUsage(): """ Memory usage of own process in bytes. """ if getOS() == "Windows": # adapted from http://code.activestate.com/recipes/578513 import ctypes.wintypes # Lets allow this to match Windows API it reflects, # pylint: disable=invalid-name class PROCESS_MEMORY_COUNTERS_EX(ctypes.Structure): _fields_ = [ ("cb", ctypes.wintypes.DWORD), ("PageFaultCount", ctypes.wintypes.DWORD), ("PeakWorkingSetSize", ctypes.c_size_t), ("WorkingSetSize", ctypes.c_size_t), ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), ("QuotaPagedPoolUsage", ctypes.c_size_t), ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), ("QuotaNonPagedPoolUsage", ctypes.c_size_t), ("PagefileUsage", ctypes.c_size_t), ("PeakPagefileUsage", ctypes.c_size_t), ("PrivateUsage", ctypes.c_size_t), ] GetProcessMemoryInfo = ctypes.windll.psapi.GetProcessMemoryInfo GetProcessMemoryInfo.argtypes = [ ctypes.wintypes.HANDLE, ctypes.POINTER(PROCESS_MEMORY_COUNTERS_EX), ctypes.wintypes.DWORD, ] GetProcessMemoryInfo.restype = ctypes.wintypes.BOOL counters = PROCESS_MEMORY_COUNTERS_EX() rv = GetProcessMemoryInfo( ctypes.windll.kernel32.GetCurrentProcess(), # @UndefinedVariable ctypes.byref(counters), ctypes.sizeof(counters) ) if not rv: raise ctypes.WinError() return counters.PrivateUsage else: # Posix only code, pylint: disable=I0021,import-error import resource # @UnresolvedImport # The value is from "getrusage", which has OS dependent scaling, at least # MacOS and Linux are different. Others maybe too. if getOS() == "Darwin": factor = 1 else: factor = 1024 return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * factor def getHumanReadableProcessMemoryUsage(value = None): if value is None: value = getOwnProcessMemoryUsage() if abs(value) < 1024*1014: return "%.2f KB (%d bytes)" % ( value / 1024.0, value ) elif abs(value) < 1024*1014*1024: return "%.2f MB (%d bytes)" % ( value / (1024*1024.0), value ) elif abs(value) < 1024*1014*1024*1024: return "%.2f GB (%d bytes)" % ( value / (1024*1024*1024.0), value ) else: return "%d bytes" % value class MemoryWatch(object): def __init__(self): self.start = getOwnProcessMemoryUsage() self.stop = None def finish(self): self.stop = getOwnProcessMemoryUsage() def asStr(self): return getHumanReadableProcessMemoryUsage(self.stop - self.start) def startMemoryTracing(): try: import tracemalloc # @UnresolvedImport except ImportError: pass else: tracemalloc.start() def showMemoryTrace(): try: import tracemalloc # @UnresolvedImport except ImportError: pass else: snapshot = tracemalloc.take_snapshot() stats = snapshot.statistics("lineno") printLine("Top 50 memory allocations:") for count, stat in enumerate(stats): if count == 50: break printLine(stat) Nuitka-0.5.28.2/nuitka/utils/SharedLibraries.py0000644000372000001440000000331513112214770021533 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module deals with finding and information about shared libraries. """ from sys import getfilesystemencoding from nuitka.PythonVersions import python_version def locateDLL(dll_name): import ctypes.util dll_name = ctypes.util.find_library(dll_name) import subprocess process = subprocess.Popen( args = ["/sbin/ldconfig", "-p"], stdout = subprocess.PIPE, stderr = subprocess.PIPE, ) stdout, _stderr = process.communicate() dll_map = {} for line in stdout.splitlines()[1:]: assert line.count(b"=>") == 1, line left, right = line.strip().split(b" => ") assert b" (" in left, line left = left[:left.rfind(b" (")] if python_version >= 300: left = left.decode(getfilesystemencoding()) right = right.decode(getfilesystemencoding()) if left not in dll_map: dll_map[left] = right return dll_map[dll_name] Nuitka-0.5.28.2/nuitka/utils/InstanceCounters.py0000644000372000001440000000452713112214770021765 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Instance counter primitives We don't use a meta class as it's unnecessary complex, and portable meta classes have their difficulties, and want to count classes, who already have a meta class. This is going to expanded with time. """ from nuitka.Options import isShowMemory from nuitka.Tracing import printIndented, printLine counted_inits = {} counted_dels = {} def counted_init(init): if isShowMemory(): def wrapped_init(self, *args, **kw): name = self.__class__.__name__ assert type(name) is str if name not in counted_inits: counted_inits[name] = 0 counted_inits[name] += 1 init(self, *args, **kw) return wrapped_init else: return init empty_del = lambda x : 0 def counted_del(del_func = empty_del): if isShowMemory(): def wrapped_del(self): # This cannot be necessary, because in program finalization, the # global variables were assign to None. if counted_dels is None: return name = self.__class__.__name__ assert type(name) is str if name not in counted_dels: counted_dels[name] = 0 counted_dels[name] += 1 if del_func is not empty_del: del_func(self) return wrapped_del else: return empty_del def printStats(): printLine("Init/del calls:") for name, count in sorted(counted_inits.items()): dels = counted_dels.get(name, 0) printIndented(1, name, count, dels, count - dels) Nuitka-0.5.28.2/nuitka/utils/FileOperations.py0000644000372000001440000001026013207537706021424 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Utils for file and directory operations. This provides enhanced and more error resilient forms of standard stuff. It will also frequently add sorting. """ import os import shutil import tempfile import time from contextlib import contextmanager from .Utils import getOS def areSamePaths(path1, path2): """ Are two paths the same. Same meaning here, that case differences ignored on platforms where that is the norm, and with normalized, and turned absolut paths, there is no differences. """ path1 = os.path.normcase(os.path.abspath(os.path.normpath(path1))) path2 = os.path.normcase(os.path.abspath(os.path.normpath(path2))) return path1 == path2 def relpath(path): """ Relative path, if possible. """ try: return os.path.relpath(path) except ValueError: # On Windows, paths on different devices prevent it to work. Use that # full path then. if getOS() == "Windows": return os.path.abspath(path) raise def makePath(path): """ Create a directory if it doesn'T exist.""" if not os.path.isdir(path): os.makedirs(path) def listDir(path): """ Give a sorted path, base filename pairs of a directory.""" return sorted( [ ( os.path.join(path, filename), filename ) for filename in os.listdir(path) ] ) def getFileList(path): result = [] for root, _dirnames, filenames in os.walk(path): for filename in filenames: result.append(os.path.join(root, filename)) return result def getSubDirectories(path): result = [] for root, dirnames, _filenames in os.walk(path): for dirname in dirnames: result.append( os.path.join(root, dirname) ) result.sort() return result def deleteFile(path, must_exist): if must_exist or os.path.isfile(path): os.unlink(path) def splitPath(path): """ Split path, skipping empty elements. """ return tuple( element for element in os.path.split(path) if element ) def hasFilenameExtension(path, extensions): extension = os.path.splitext(os.path.normcase(path))[1] return extension in extensions def removeDirectory(path, ignore_errors): """ Remove a directory recursively. On Windows, it happens that operations fail, and succeed when reried, so added a retry and small delay, then another retry. Should make it much more stable during tests. All kinds of programs that scan files might cause this, but they do it hopefully only briefly. """ def onError(func, path, exc_info): # Try again immediately, ignore what happened, pylint: disable=unused-argument try: func(path) except OSError: time.sleep(0.1) func(path) if os.path.exists(path): try: shutil.rmtree( path, ignore_errors = False, onerror = onError ) except OSError: if ignore_errors: shutil.rmtree( path, ignore_errors = ignore_errors ) else: raise @contextmanager def withTemporaryFilename(): with tempfile.NamedTemporaryFile() as temp_file: yield temp_file.name Nuitka-0.5.28.2/nuitka/utils/Timing.py0000644000372000001440000000357313207537242017734 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Time taking. Mostly for measurements of Nuitka of itself, e.g. how long did it take to call an external tool. """ from logging import info from timeit import default_timer as timer from nuitka.Options import isShowProgress class StopWatch(object): __slots__ = ("start_time", "end_time") def __init__(self): self.start_time = None self.end_time = None def start(self): self.start_time = timer() def end(self): self.end_time = timer() def delta(self): return self.end_time - self.start_time class TimerReport(object): """ Timer that reports how long things took. Mostly intended as a wrapper for external process calls. """ __slots__ = ("message", "timer") def __init__(self, message): self.message = message self.timer = None def __enter__(self): self.timer = StopWatch() self.timer.start() def __exit__(self, exception_type, exception_value, exception_tb): self.timer.end() if exception_type is None and isShowProgress(): info(self.message % self.timer.delta()) Nuitka-0.5.28.2/nuitka/utils/Execution.py0000644000372000001440000001215213134660221020432 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Program execution related stuff. Basically a layer for os, subprocess, shutil to come together. It can find binaries (needed for exec) and run them capturing outputs. """ import os import subprocess import sys from contextlib import contextmanager from .Utils import getArchitecture, getOS def callExec(args): """ Do exec in a portable way preserving exit code. On Windows, unfortunately there is no real exec, so we have to spawn a new process instead. """ # On Windows os.execl does not work properly if getOS() != "Windows": # The star arguments is the API of execl os.execl(*args) else: args = list(args) del args[1] try: sys.exit( subprocess.call(args) ) except KeyboardInterrupt: # There was a more relevant stack trace already, so abort this # right here, pylint: disable=protected-access os._exit(2) def getExecutablePath(filename): """ Find an execute in PATH environment. """ # Append ".exe" suffix on Windows if not already present. if getOS() == "Windows" and not filename.lower().endswith(".exe"): filename += ".exe" # Search in PATH environment. search_path = os.environ.get("PATH", "") # Now check in each path element, much like the shell will. path_elements = search_path.split(os.pathsep) for path_element in path_elements: path_element = path_element.strip('"') full = os.path.join(path_element, filename) if os.path.exists(full): return full return None def getPythonExePathWindows(search, arch): """ Find Python on Windows. """ # Shortcuts for the default installation directories, to avoid going to # registry at all unless necessary. Any Python2 will do for Scons, so it # might be avoided entirely. # Windows only code, pylint: disable=I0021,import-error,undefined-variable try: import _winreg as winreg except ImportError: import winreg # lint:ok if arch is None: if getArchitecture() == "x86": arches = (winreg.KEY_WOW64_32KEY, winreg.KEY_WOW64_64KEY) else: arches = (winreg.KEY_WOW64_64KEY, winreg.KEY_WOW64_32KEY) elif arch == "x86": arches = (winreg.KEY_WOW64_32KEY,) elif arch == "x86_64": arches = (winreg.KEY_WOW64_64KEY,) else: assert False, arch for hkey_branch in (winreg.HKEY_LOCAL_MACHINE, winreg.HKEY_CURRENT_USER): for arch_key in arches: try: key = winreg.OpenKey( hkey_branch, r"SOFTWARE\Python\PythonCore\%s\InstallPath" % search, 0, winreg.KEY_READ | arch_key ) candidate = os.path.join( winreg.QueryValue(key, ""), "python.exe" ) except WindowsError: # @UndefinedVariable continue if os.path.exists(candidate): return candidate def check_output(*popenargs, **kwargs): """ Call a process and check result code. This is for Python 2.6 compatibility, which doesn't have that in its standard library. """ if "stdout" in kwargs: raise ValueError("stdout argument not allowed, it will be overridden.") process = subprocess.Popen( stdout = subprocess.PIPE, *popenargs, **kwargs ) output, _unused_err = process.communicate() retcode = process.poll() if retcode: cmd = kwargs.get("args") if cmd is None: cmd = popenargs[0] raise subprocess.CalledProcessError(retcode, cmd, output = output) return output @contextmanager def withEnvironmentPathAdded(env_var_name, path): if type(path) in (tuple, list): path = os.pathsep.join(path) if path: if str is not bytes: path = path.decode("utf-8") if env_var_name in os.environ: old_path = os.environ[env_var_name] os.environ[env_var_name] += os.pathsep + path else: old_path = None os.environ[env_var_name] = path yield if path: if old_path is None: del os.environ[env_var_name] else: os.environ[env_var_name] = old_path Nuitka-0.5.28.2/nuitka/utils/__init__.py0000644000372000001440000000150113112214770020222 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/utils/Utils.py0000644000372000001440000000604713134660221017575 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Utility module. Here the small things that fit nowhere else and don't deserve their own module. """ import imp import os import sys from nuitka.PythonVersions import python_version def getOS(): if os.name == "nt": return "Windows" elif os.name == "posix": return os.uname()[0] # @UndefinedVariable else: assert False, os.name def getArchitecture(): if getOS() == "Windows": if "AMD64" in sys.version: return "x86_64" else: return "x86" else: return os.uname()[4] # @UndefinedVariable def getSharedLibrarySuffix(): result = None for suffix, _mode, module_type in imp.get_suffixes(): if module_type != imp.C_EXTENSION: continue if result is None or len(suffix) < len(result): result = suffix return result def getCoreCount(): cpu_count = 0 # Try to sum up the CPU cores, if the kernel shows them. try: # Try to get the number of logical processors with open("/proc/cpuinfo") as cpuinfo_file: cpu_count = cpuinfo_file.read().count("processor\t:") except IOError: pass if not cpu_count: import multiprocessing cpu_count = multiprocessing.cpu_count() return cpu_count def encodeNonAscii(var_name): """ Encode variable name that is potentially not ASCII to ASCII only. For Python3, unicode identifiers can be used, but these are not possible in C, so we need to replace them. """ if python_version < 300: return var_name else: # Using a escaping here, because that makes it safe in terms of not # to occur in the encoding escape sequence for unicode use. var_name = var_name.replace("$$", "$_$") var_name = var_name.encode("ascii", "xmlcharrefreplace") var_name = var_name.decode("ascii") return var_name.replace("&#", "$$").replace(';', "") def isExecutableCommand(command): path = os.environ["PATH"] suffixes = (".exe",) if getOS() == "Windows" else ("",) for part in path.split(os.pathsep): if not part: continue for suffix in suffixes: if os.path.isfile(os.path.join(part, command + suffix)): return True return False Nuitka-0.5.28.2/nuitka/Tracing.py0000644000372000001440000000331413134660221016716 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Outputs to the user. Printing with intends or plain, mostly a compensation for the print strangeness. We want to avoid "from __future__ import print_function" in every file out there, which makes adding another debug print rather tedious. This should cover all calls/uses of "print" we have to do, and the make it easy to simply to "print for_debug" without much hassle (braces). """ from __future__ import print_function import sys def printIndented(level, *what): print(" " * level, *what) def printSeparator(level = 0): print(" " * level, '*' * 10) def printLine(*what): print(*what) def printError(message): print( message, file = sys.stderr ) def flushStdout(): sys.stdout.flush() def my_print(*args, **kwargs): """ Make sure we flush after every print. Not even the "-u" option does more than that and this is easy enough. """ print(*args, **kwargs) flushStdout() Nuitka-0.5.28.2/nuitka/tools/0000755000372000001440000000000013207540420016113 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/tools/testing/0000755000372000001440000000000013207540420017570 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/tools/testing/Constructs.py0000644000372000001440000000271213134660221022314 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tools for construct tests. """ def generateConstructCases(construct_source_code): inside = False case = 0 case_1 = [] case_2 = [] for line in construct_source_code.splitlines(): if not inside or case == 1: case_1.append(line) else: case_1.append("") if "# construct_end" in line: inside = False if "# construct_alternative" in line: case = 2 if not inside or case == 2: case_2.append(line) else: case_2.append("") if "# construct_begin" in line: inside = True case = 1 return '\n'.join(case_1), '\n'.join(case_2) Nuitka-0.5.28.2/nuitka/tools/testing/SearchModes.py0000644000372000001440000001055513207537242022355 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Search modes for Nuitka's test runner. The test runner can handle found errors, skip tests, etc. with search modes, which are implemented here. """ import hashlib import os import sys from nuitka.utils.FileOperations import areSamePaths class SearchModeBase(object): def __init__(self): self.may_fail = [] def consider(self, dirname, filename): # Virtual method, pylint: disable=no-self-use,unused-argument return True def finish(self): pass def abortOnFinding(self, dirname, filename): for candidate in self.may_fail: if self._match(dirname, filename, candidate): return False return True def getExtraFlags(self, dirname, filename): # Virtual method, pylint: disable=no-self-use,unused-argument return [] def mayFailFor(self, *names): self.may_fail += names @classmethod def _match(cls, dirname, filename, candidate): parts = [dirname, filename] while None in parts: parts.remove(None) assert parts path = os.path.join(*parts) candidates = ( dirname, filename, filename.replace(".py", ""), filename.split('.')[0], path, path.replace(".py", ""), ) candidate2 = os.path.relpath( candidate, os.path.dirname(sys.modules["__main__"].__file__) ) return candidate.rstrip('/') in candidates or candidate2 in candidates def isCoverage(self): # Virtual method, pylint: disable=no-self-use return False class SearchModeByPattern(SearchModeBase): def __init__(self, start_at): SearchModeBase.__init__(self) self.active = False self.start_at = start_at def consider(self, dirname, filename): if self.active: return True self.active = self._match(dirname, filename, self.start_at) return self.active def finish(self): if not self.active: sys.exit("Error, became never active.") class SearchModeResume(SearchModeBase): def __init__(self, tests_path): SearchModeBase.__init__(self) tests_path = os.path.normcase(os.path.abspath(tests_path)) version = sys.version if str is not bytes: tests_path = tests_path.encode("utf8") version = version.encode("utf8") case_hash = hashlib.md5(tests_path) case_hash.update(version) from .Common import getTestingCacheDir cache_filename = os.path.join( getTestingCacheDir(), case_hash.hexdigest() ) self.cache_filename = cache_filename if os.path.exists(cache_filename): self.resume_from = open(cache_filename, 'r').read() or None else: self.resume_from = None self.active = not self.resume_from def consider(self, dirname, filename): parts = [dirname, filename] while None in parts: parts.remove(None) assert parts path = os.path.join(*parts) if self.active: open(self.cache_filename, 'w').write(path) return True if areSamePaths(path, self.resume_from): self.active = True return self.active def finish(self): if not self.active: sys.exit("Error, became never active.") os.unlink(self.cache_filename) class SearchModeCoverage(SearchModeBase): def getExtraFlags(self, dirname, filename): return ["coverage"] def isCoverage(self): return True Nuitka-0.5.28.2/nuitka/tools/testing/Valgrind.py0000644000372000001440000000655313134660221021722 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Valgrind tool usage. We are using it for benchmarking purposes, as it's an analysis tool at the same time and gives deterministic results. """ import shutil import subprocess import sys from nuitka.Tracing import my_print from nuitka.utils.Execution import check_output from nuitka.utils.FileOperations import withTemporaryFilename def runValgrind(descr, tool, args, include_startup, save_logfilename = None): if descr: my_print(descr, tool, file = sys.stderr, end = "... ") with withTemporaryFilename() as log_filename: command = [ "valgrind", "-q", ] if tool == "callgrind": command += [ "--tool=callgrind", "--callgrind-out-file=%s" % log_filename ] elif tool == "massif": command += [ "--tool=massif", "--massif-out-file=%s" % log_filename ] else: sys.exit("Error, no support for tool '%s' yet." % tool) # Do not count things before main module starts its work. if not include_startup: command += [ "--zero-before=init__main__()", "--zero-before=init__main__", "--zero-before=PyInit___main__", "--zero-before=PyInit___main__()" ] command.extend(args) process = subprocess.Popen( args = command, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) _stdout_valgrind, stderr_valgrind = process.communicate() exit_valgrind = process.returncode assert exit_valgrind == 0, stderr_valgrind if descr: my_print("OK", file = sys.stderr) if save_logfilename is not None: shutil.copy(log_filename, save_logfilename) max_mem = None for line in open(log_filename): if tool == "callgrind" and line.startswith("summary:"): return int(line.split()[1]) elif tool == "massif" and line.startswith("mem_heap_B="): mem = int(line.split('=')[1]) if max_mem is None: max_mem = 0 max_mem = max(mem, max_mem) if tool == "massif" and max_mem is not None: return max_mem sys.exit("Error, didn't parse Valgrind log file successfully.") def getBinarySizes(filename): command = [ "size", filename ] sizes = check_output(command).strip() sizes = sizes.split(b'\n')[-1].replace(b'\t', b"").split() return int(sizes[0]), int(sizes[1]) Nuitka-0.5.28.2/nuitka/tools/testing/compare_with_cpython/0000755000372000001440000000000013207540420024015 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/tools/testing/compare_with_cpython/__main__.py0000755000372000001440000006033413207537242026130 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tool to compare output of CPython and Nuitka. """ from __future__ import print_function import difflib import os import re import subprocess import sys import tempfile import time ran_tests_re = re.compile(r"^(Ran \d+ tests? in )\-?\d+\.\d+s$") instance_re = re.compile(r"at (?:0x)?[0-9a-fA-F]+(;?\s|\>)") thread_re = re.compile(r"Thread 0x[0-9a-fA-F]+") compiled_types_re = re.compile(r"compiled_(module|function|generator|method|frame|coroutine)") module_repr_re = re.compile(r"(\)") global_name_error_re = re.compile( r"global (name ')(.*?)(' is not defined)" ) non_ascii_error_rt = re.compile( r"(SyntaxError: Non-ASCII character.*? on line) \d+" ) python_win_lib_re = re.compile( r"[a-zA-Z]:\\\\?[Pp]ython(.*?\\\\?)[Ll]ib" ) local_port_re = re.compile( r"(127\.0\.0\.1):\d{2,5}" ) traceback_re = re.compile( r'(F|f)ile "(.*?)", line (\d+)' ) tempfile_re = re.compile( r'/tmp/tmp[a-z0-9_]*' ) def traceback_re_callback(match): return r'%sile "%s", line %s' % ( match.group(1), os.path.realpath(os.path.abspath(match.group(2))), match.group(3) ) # Make sure we flush after every print, the "-u" option does more than that # and this is easy enough. def my_print(*args, **kwargs): print(*args, **kwargs) sys.stdout.flush() def displayOutput(stdout, stderr): if type(stdout) is not str: stdout = stdout.decode("utf-8" if os.name != "nt" else "cp850") stderr = stderr.decode("utf-8" if os.name != "nt" else "cp850") my_print(stdout, end = ' ') if stderr: my_print(stderr) def compareOutput(kind, out_cpython, out_nuitka, ignore_warnings, ignore_infos, syntax_errors): fromdate = "" todate = "" diff = difflib.unified_diff( makeDiffable(out_cpython, ignore_warnings, ignore_infos, syntax_errors), makeDiffable(out_nuitka, ignore_warnings, ignore_infos, syntax_errors), "{program} ({detail})".format( program = os.environ["PYTHON"], detail = kind ), "{program} ({detail})".format( program = "nuitka", detail = kind ), fromdate, todate, n = 3 ) result = list(diff) if result: for line in result: my_print(line, end = '\n' if not line.startswith("---") else "") return 1 else: return 0 def makeDiffable(output, ignore_warnings, ignore_infos, syntax_errors): # Of course many cases to deal with, pylint: disable=too-many-branches result = [] # Fix import "readline" because output sometimes starts with "\x1b[?1034h" m = re.match(b'\\x1b\\[[^h]+h', output) if m: output = output[len(m.group()):] lines = output.split(b"\n") if syntax_errors: for line in lines: if line.startswith(b"SyntaxError:"): lines = [line] break for line in lines: if type(line) is not str: line = line.decode("utf-8" if os.name != "nt" else "cp850") if line.endswith('\r'): line = line[:-1] if line.startswith("REFCOUNTS"): first_value = line[line.find('[')+1:line.find(',')] last_value = line[line.rfind(' ')+1:line.rfind(']')] line = line.\ replace(first_value, "xxxxx").\ replace(last_value, "xxxxx") if line.startswith('[') and line.endswith("refs]"): continue if ignore_warnings and line.startswith("Nuitka:WARNING"): continue if ignore_infos and line.startswith("Nuitka:INFO"): continue if line.startswith("Nuitka:WARNING:Cannot recurse to import"): continue line = instance_re.sub(r"at 0xxxxxxxxx\1", line) line = thread_re.sub(r"Thread 0xXXXXXXXX", line) line = compiled_types_re.sub(r"\1", line) line = global_name_error_re.sub(r"\1\2\3", line) line = module_repr_re.sub(r"\1xxxxx\2", line) line = non_ascii_error_rt.sub(r"\1 xxxx", line) # Windows has a different "os.path", update according to it. line = line.replace("ntpath", "posixpath") # This URL is updated, and Nuitka outputs the new one, but we test # against versions that don't have that. line = line.replace( "http://www.python.org/peps/pep-0263.html", "http://python.org/dev/peps/pep-0263/", ) line = ran_tests_re.sub(r"\1x.xxxs", line) line = traceback_re.sub(traceback_re_callback, line) line = tempfile_re.sub(r"/tmp/tmpxxxxxxx", line) # This is a bug potentially, occurs only for CPython when re-directed, # we are going to ignore the issue as Nuitka is fine. if line == """\ Exception RuntimeError: 'maximum recursion depth \ exceeded while calling a Python object' in \ ignored""": continue # This is also a bug potentially, but only visible under # CPython line = python_win_lib_re.sub(r"C:\\Python\1Lib", line) # Port numbers can be random, lets ignore them line = local_port_re.sub(r"\1:xxxxx", line) # This is a bug with clang potentially, can't find out why it says that. if line == "/usr/bin/ld: warning: .init_array section has zero size": continue # This is for NetBSD and OpenBSD, which seems to build "libpython" so # that it gives such warnings. if "() possibly used unsafely" in line or \ "() is almost always misused" in line: continue # This is for CentOS5, where the linker says this, and it's hard to # disable if "skipping incompatible /usr/lib/libpython2.6.so" in line: continue # This is for self compiled Python with default options, gives this # harmless option for every time we link to "libpython". if "is dangerous, better use `mkstemp'" in line or \ "In function `posix_tempnam'" in line or \ "In function `posix_tmpnam'" in line: continue result.append(line) return result def main(): # Of course many cases to deal with, pylint: disable=too-many-branches,too-many-locals,too-many-statements from nuitka.utils.Execution import check_output filename = sys.argv[1] args = sys.argv[2:] def hasArg(arg): if arg in args: args.remove(arg) return True else: return False # For output keep it arguments = list(args) silent_mode = hasArg("silent") ignore_stderr = hasArg("ignore_stderr") ignore_warnings = hasArg("ignore_warnings") ignore_infos = hasArg("ignore_infos") expect_success = hasArg("expect_success") expect_failure = hasArg("expect_failure") python_debug = hasArg("python_debug") module_mode = hasArg("module_mode") two_step_execution = hasArg("two_step_execution") binary_python_path = hasArg("binary_python_path") keep_python_path = hasArg("keep_python_path") trace_command = hasArg("trace_command") remove_output = hasArg("remove_output") standalone_mode = hasArg("standalone") no_site = hasArg("no_site") recurse_none = hasArg("recurse_none") recurse_all = hasArg("recurse_all") timing = hasArg("timing") coverage_mode = hasArg("coverage") original_file = hasArg("original_file") no_warnings = not hasArg("warnings") full_compat = not hasArg("improved") syntax_errors = hasArg("syntax_errors") plugins_enabled = [] for count, arg in reversed(tuple(enumerate(args))): if arg.startswith("plugin_enable:"): plugins_enabled.append(arg[len("plugin_enable:"):]) del args[count] plugins_disabled = [] for count, arg in reversed(tuple(enumerate(args))): if arg.startswith("plugin_disable:"): plugins_disabled.append(arg[len("plugin_disable:"):]) del args[count] recurse_not = [] for count, arg in reversed(tuple(enumerate(args))): if arg.startswith("recurse_not:"): recurse_not.append(arg[len("recurse_not:"):]) del args[count] if args: sys.exit("Error, non understood mode(s) '%s'," % ','.join(args)) # In coverage mode, we don't want to execute, and to do this only in one mode, # we enable two step execution, which splits running the binary from the actual # compilation: if coverage_mode: two_step_execution = True # The coverage mode doesn't work with debug mode. if coverage_mode: python_debug = False comparison_mode = not coverage_mode assert not standalone_mode or not module_mode assert not recurse_all or not recurse_none if "PYTHONHASHSEED" not in os.environ: os.environ["PYTHONHASHSEED"] = '0' os.environ["PYTHONWARNINGS"] = "ignore" if "PYTHON" not in os.environ: os.environ["PYTHON"] = sys.executable extra_options = os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() if "--python-debug" in extra_options or "--python-dbg" in extra_options: python_debug = True if python_debug: if os.path.exists(os.path.join("/usr/bin/", os.environ["PYTHON"] + "-dbg")): os.environ["PYTHON"] += "-dbg" if os.name == "nt": if os.path.exists(os.environ["PYTHON"][:-4]+"_d.exe"): os.environ["PYTHON"] = os.environ["PYTHON"][:-4]+"_d.exe" if os.environ["PYTHON"].endswith("-dbg"): python_debug = True if os.environ["PYTHON"].lower().endswith("_d.exe"): python_debug = True if comparison_mode: my_print( """\ Comparing output of '{filename}' using '{python}' with flags {args} ...""". format( filename = filename, python = os.environ["PYTHON"], args = ", ".join(arguments) ) ) else: my_print( """\ Taking coverage of '{filename}' using '{python}' with flags {args} ...""". format( filename = filename, python = os.environ["PYTHON"], args = ", ".join(arguments) ) ) if comparison_mode and not silent_mode: my_print('*' * 80) my_print("CPython:") my_print('*' * 80) if two_step_execution: filename = os.path.abspath(filename) if module_mode: if no_warnings: cpython_cmd = [ os.environ["PYTHON"], "-W", "ignore", "-c", "import sys; sys.path.append(%s); import %s" % ( repr(os.path.dirname(filename)), os.path.basename(filename) ) ] else: cpython_cmd = [ os.environ["PYTHON"], "-c", "import sys; sys.path.append(%s); import %s" % ( repr(os.path.dirname(filename)), os.path.basename(filename) ) ] else: if no_warnings: cpython_cmd = [ os.environ["PYTHON"], "-W", "ignore", filename ] else: cpython_cmd = [ os.environ["PYTHON"], filename ] if no_site: cpython_cmd.insert(1, "-S") if "NUITKA" in os.environ: # Would need to extract which "python" this is going to use. assert not coverage_mode, "Not implemented for binaries." nuitka_call = [os.environ["NUITKA"]] else: if comparison_mode: nuitka_call = [ os.environ["PYTHON"], os.path.abspath( os.path.join( os.path.dirname(__file__), "..", "..", "..", "..", "bin", "nuitka" ) ) ] else: assert coverage_mode nuitka_call = [ os.environ["PYTHON"], "-S", "-m", "coverage", "run", "--rcfile", os.devnull, "-a", os.path.abspath( os.path.join( os.path.dirname(__file__), "..", "..", "..", "..", "bin", "nuitka" ) ) ] if python_debug: extra_options.append("--python-debug") if no_warnings: extra_options.append("--python-flag=no_warnings") if remove_output: extra_options.append("--remove-output") if original_file: extra_options.append("--file-reference-choice=original") if full_compat: extra_options.append("--full-compat") if coverage_mode: # Coverage modules hates Nuitka to re-execute, and so we must avoid # that. python_path = check_output( [ os.environ["PYTHON"], "-c" "import sys, os; print(os.pathsep.join(sys.path))" ] ) if sys.version_info >= (3,): python_path = python_path.decode("utf8") os.environ["PYTHONPATH"] = python_path.strip() if binary_python_path: python_path = os.environ.get("PYTHONPATH", "") os.environ["PYTHONPATH"] = os.pathsep.join( python_path.split(os.pathsep) + \ [os.path.dirname(os.path.abspath(filename))] ) if keep_python_path or binary_python_path: extra_options.append("--keep-pythonpath") if recurse_none: extra_options.append("--recurse-none") if recurse_all: extra_options.append("--recurse-all") if recurse_not: extra_options.extend("--recurse-not-to=" + v for v in recurse_not) if coverage_mode: extra_options.append("--must-not-re-execute") extra_options.append("--generate-c-only") for plugin_enabled in plugins_enabled: extra_options.append("--plugin-enable=" + plugin_enabled) for plugin_disabled in plugins_disabled: extra_options.append("--plugin-disable=" + plugin_disabled) # Now build the command to run Nuitka. if not two_step_execution: if module_mode: nuitka_cmd = nuitka_call + extra_options + \ ["--execute", "--module", filename] elif standalone_mode: nuitka_cmd = nuitka_call + extra_options + \ ["--execute", "--standalone", filename] else: nuitka_cmd = nuitka_call + extra_options + \ ["--execute", filename] if no_site: nuitka_cmd.insert(len(nuitka_cmd) - 1, "--python-flag=-S") else: if module_mode: nuitka_cmd1 = nuitka_call + extra_options + \ ["--module", os.path.abspath(filename)] elif standalone_mode: nuitka_cmd1 = nuitka_call + extra_options + \ ["--standalone", filename] else: nuitka_cmd1 = nuitka_call + extra_options + \ [filename] if no_site: nuitka_cmd1.insert(len(nuitka_cmd1) - 1, "--python-flag=-S") for extra_option in extra_options: dir_match = re.search(r"--output-dir=(.*?)(\s|$)", extra_option) if dir_match: output_dir = dir_match.group(1) break else: # The default. output_dir = '.' if module_mode: nuitka_cmd2 = [ os.environ["PYTHON"], "-W", "ignore", "-c", "import %s" % os.path.basename(filename) ] else: exe_filename = os.path.basename(filename) if filename.endswith(".py"): exe_filename = exe_filename[:-3] exe_filename = exe_filename.replace(')', "").replace('(', "") exe_filename += ".exe" nuitka_cmd2 = [ os.path.join(output_dir, exe_filename) ] pdb_filename = exe_filename[:-4] + ".pdb" if trace_command: my_print("CPython command:", *cpython_cmd) if comparison_mode: start_time = time.time() process = subprocess.Popen( args = cpython_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_cpython, stderr_cpython = process.communicate() exit_cpython = process.returncode cpython_time = time.time() - start_time if comparison_mode and not silent_mode: displayOutput(stdout_cpython, stderr_cpython) if comparison_mode and not silent_mode: my_print('*' * 80) my_print("Nuitka:") my_print('*' * 80) if two_step_execution: if output_dir: os.chdir(output_dir) else: tmp_dir = tempfile.gettempdir() # Try to avoid RAM disk /tmp and use the disk one instead. if tmp_dir == "/tmp" and os.path.exists("/var/tmp"): tmp_dir = "/var/tmp" os.chdir(tmp_dir) if trace_command: my_print("Going to output directory", os.getcwd()) start_time = time.time() if not two_step_execution: if trace_command: my_print("Nuitka command:", nuitka_cmd) process = subprocess.Popen( args = nuitka_cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_nuitka, stderr_nuitka = process.communicate() exit_nuitka = process.returncode else: if trace_command: my_print("Nuitka command 1:", nuitka_cmd1) process = subprocess.Popen( args = nuitka_cmd1, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_nuitka1, stderr_nuitka1 = process.communicate() exit_nuitka1 = process.returncode if exit_nuitka1 != 0: if not expect_failure and \ not comparison_mode and \ not os.path.exists(".coverage"): sys.exit( """\ Error, failed to take coverage with '%s'. Stderr was: %s """ % ( os.environ["PYTHON"], stderr_nuitka1 ) ) exit_nuitka = exit_nuitka1 stdout_nuitka, stderr_nuitka = stdout_nuitka1, stderr_nuitka1 else: # No execution second step for coverage mode. if comparison_mode: if trace_command: my_print("Nuitka command 2:", nuitka_cmd2) process = subprocess.Popen( args = nuitka_cmd2, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) stdout_nuitka2, stderr_nuitka2 = process.communicate() stdout_nuitka = stdout_nuitka1 + stdout_nuitka2 stderr_nuitka = stderr_nuitka1 + stderr_nuitka2 exit_nuitka = process.returncode else: exit_nuitka = exit_nuitka1 stdout_nuitka, stderr_nuitka = stdout_nuitka1, stderr_nuitka1 nuitka_time = time.time() - start_time if not silent_mode: displayOutput(stdout_nuitka, stderr_nuitka) if coverage_mode: assert not stdout_nuitka assert not stderr_nuitka if comparison_mode: exit_code_stdout = compareOutput( "stdout", stdout_cpython, stdout_nuitka, ignore_warnings, ignore_infos, syntax_errors ) if ignore_stderr: exit_code_stderr = 0 else: exit_code_stderr = compareOutput( "stderr", stderr_cpython, stderr_nuitka, ignore_warnings, ignore_infos, syntax_errors ) exit_code_return = exit_cpython != exit_nuitka if exit_code_return: my_print( """\ Exit codes {exit_cpython:d} (CPython) != {exit_nuitka:d} (Nuitka)""".format( exit_cpython = exit_cpython, exit_nuitka = exit_nuitka ) ) # In case of segfault, also output the call stack by entering debugger # without stdin forwarded. if exit_code_return and exit_nuitka == -11 and sys.platform != "nt": nuitka_cmd.insert(len(nuitka_cmd) - 1, "--debugger") process = subprocess.Popen( args = nuitka_cmd, stdin = subprocess.PIPE ) process.communicate() exit_code = exit_code_stdout or exit_code_stderr or exit_code_return if exit_code: sys.exit("Error, outputs differed.") if expect_success and exit_cpython != 0: if silent_mode: displayOutput(stdout_cpython, stderr_cpython) sys.exit("Unexpected error exit from CPython.") if expect_failure and exit_cpython == 0: sys.exit("Unexpected success exit from CPython.") if remove_output: if not module_mode: if os.path.exists(nuitka_cmd2[0]): if os.name == "nt": # It appears there is a tiny lock race that we randomly cause, # likely because --run spawns a subprocess that might still # be doing the cleanup work. for _i in range(10): try: os.rename(nuitka_cmd2[0], nuitka_cmd2[0]+".away") except OSError: time.sleep(0.1) continue for _i in range(10): try: os.unlink(nuitka_cmd2[0]+".away") except OSError: time.sleep(2) continue else: break assert not os.path.exists(nuitka_cmd2[0]+".away") if os.path.exists(pdb_filename): os.unlink(pdb_filename) else: os.unlink(nuitka_cmd2[0]) else: if os.name == "nt": module_filename = os.path.basename(filename) + ".pyd" else: module_filename = os.path.basename(filename) + ".so" if os.path.exists(module_filename): os.unlink(module_filename) if comparison_mode and timing: my_print( "CPython took %.2fs vs %0.2fs Nuitka." % ( cpython_time, nuitka_time ) ) if comparison_mode and not silent_mode: my_print("OK, same outputs.") if __name__ == "__main__": # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), "..", "..", "..", ) ) ) sys.path.insert( 1, "/usr/share/nuitka" ) main() Nuitka-0.5.28.2/nuitka/tools/testing/compare_with_cpython/__init__.py0000644000372000001440000000150113207537242026133 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/tools/testing/run_nuitka_tests/0000755000372000001440000000000013207540420023171 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/tools/testing/run_nuitka_tests/__main__.py0000755000372000001440000006171413207537242025307 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Main front-end to the tests of Nuitka. Has many options, read --help output. """ from __future__ import print_function import os import subprocess import sys import tempfile from optparse import OptionParser from nuitka.tools.Basics import goHome from nuitka.utils.Execution import ( check_output, getExecutablePath, getPythonExePathWindows ) def main(): # There are freaking many options to honor, pylint: disable=too-many-branches # Lets honor this Debian option here. if "nocheck" in os.environ.get("DEB_BUILD_OPTIONS", "").split(): print("Skipped all tests as per DEB_BUILD_OPTIONS environment.") sys.exit(0) goHome() parser = OptionParser() parser.add_option( "--skip-basic-tests", action = "store_false", dest = "basic_tests", default = True, help = """\ The basic tests, execute these to check if Nuitka is healthy. Default is %default.""" ) parser.add_option( "--skip-syntax-tests", action = "store_false", dest = "syntax_tests", default = True, help = """\ The syntax tests, execute these to check if Nuitka handles Syntax errors fine. Default is %default.""" ) parser.add_option( "--skip-program-tests", action = "store_false", dest = "program_tests", default = True, help = """\ The programs tests, execute these to check if Nuitka handles programs, e.g. import recursions, etc. fine. Default is %default.""" ) parser.add_option( "--skip-package-tests", action = "store_false", dest = "package_tests", default = True, help = """\ The packages tests, execute these to check if Nuitka handles packages, e.g. import recursions, etc. fine. Default is %default.""" ) parser.add_option( "--skip-optimizations-tests", action = "store_false", dest = "optimization_tests", default = True, help = """\ The optimization tests, execute these to check if Nuitka does optimize certain constructs fully away. Default is %default.""" ) parser.add_option( "--skip-standalone-tests", action = "store_false", dest = "standalone_tests", default = os.name != "posix" or os.uname()[0] != "NetBSD", # @UndefinedVariable help = """\ The standalone tests, execute these to check if Nuitka standalone mode, e.g. not referring to outside, important 3rd library packages like PyQt fine. Default is %default.""" ) parser.add_option( "--skip-reflection-test", action = "store_false", dest = "reflection_test", default = True, help = """\ The reflection test compiles Nuitka with Nuitka, and then Nuitka with the compile Nuitka and compares the outputs. Default is %default.""" ) parser.add_option( "--skip-cpython26-tests", action = "store_false", dest = "cpython26", default = True, help = """\ The standard CPython2.6 test suite. Execute this for all corner cases to be covered. With Python 2.7 this covers exception behavior quite well. Default is %default.""" ) parser.add_option( "--skip-cpython27-tests", action = "store_false", dest = "cpython27", default = True, help = """\ The standard CPython2.7 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython32-tests", action = "store_false", dest = "cpython32", default = True, help = """\ The standard CPython3.2 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython33-tests", action = "store_false", dest = "cpython33", default = True, help = """\ The standard CPython3.3 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython34-tests", action = "store_false", dest = "cpython34", default = True, help = """\ The standard CPython3.4 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython35-tests", action = "store_false", dest = "cpython35", default = True, help = """\ The standard CPython3.5 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is %default.""" ) parser.add_option( "--skip-cpython36-tests", action = "store_false", dest = "cpython36", default = True, help = """\ The standard CPython3.6 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is %default.""" ) parser.add_option( "--skip-other-cpython-tests", action = "store_true", dest = "cpython_no_other", default = False, help = """\ Do not execute any CPython test suite other than the one matching the running Python. Default is %default.""" ) parser.add_option( "--skip-all-cpython-tests", action = "store_true", dest = "cpython_none", default = False, help = """\ Do not execute any CPython test suite other than the one matching the running Python. Default is %default.""" ) parser.add_option( "--no-other-python", action = "store_true", dest = "no_other", default = False, help = """\ Do not use any other Python than the one running, even if available on the system. Default is %default.""" ) parser.add_option( "--no-python2.6", action = "store_true", dest = "no26", default = False, help = """\ Do not use Python2.6 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python2.7", action = "store_true", dest = "no27", default = False, help = """\ Do not use Python2.7 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.2", action = "store_true", dest = "no32", default = False, help = """\ Do not use Python3.2 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.3", action = "store_true", dest = "no33", default = False, help = """\ Do not use Python3.3 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.4", action = "store_true", dest = "no34", default = False, help = """\ Do not use Python3.4 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.5", action = "store_true", dest = "no35", default = False, help = """\ Do not use Python3.5 even if available on the system. Default is %default.""" ) parser.add_option( "--no-python3.6", action = "store_true", dest = "no36", default = False, help = """\ Do not use Python3.6 even if available on the system. Default is %default.""" ) parser.add_option( "--coverage", action = "store_true", dest = "coverage", default = False, help = """\ Make a coverage analysis, that does not really check. Default is %default.""" ) options, positional_args = parser.parse_args() if positional_args: parser.print_help() sys.exit("\nError, no positional argument allowed.") if options.no_other: if sys.version_info[0:2] != (2,6): options.no26 = True if sys.version_info[0:2] != (2,7): options.no27 = True if sys.version_info[0:2] != (3,2): options.no32 = True if sys.version_info[0:2] != (3,3): options.no33 = True if sys.version_info[0:2] != (3,4): options.no34 = True if sys.version_info[0:2] != (3,5): options.no35 = True if sys.version_info[0:2] != (3,6): options.no36 = True if options.cpython_no_other: if sys.version_info[0:2] != (2,6): options.cpython26 = False if sys.version_info[0:2] != (2,7): options.cpython27 = False if sys.version_info[0:2] != (3,2): options.cpython32 = False if sys.version_info[0:2] != (3,3): options.cpython33 = False if sys.version_info[0:2] != (3,4): options.cpython34 = False if sys.version_info[0:2] != (3,5): options.cpython35 = False if sys.version_info[0:2] != (3,6): options.cpython36 = False if options.cpython_none: options.cpython26 = False options.cpython27 = False options.cpython32 = False options.cpython33 = False options.cpython34 = False options.cpython35 = False options.cpython36 = False if options.coverage and os.path.exists(".coverage"): os.unlink(".coverage") # Add the local bin directory to search path start. os.environ["PATH"] = \ os.path.join( os.getcwd(), "bin" ) + \ os.pathsep + \ os.environ["PATH"] def checkExecutableCommand(command): """ Check if a command is executable. """ # Many cases, pylint: disable=too-many-branches,too-many-return-statements # Do respect given options to disable specific Python versions if command == "python2.6" and options.no26: return False if command == "python2.7" and options.no27: return False if command == "python3.2" and options.no32: return False if command == "python3.3" and options.no33: return False if command == "python3.4" and options.no34: return False if command == "python3.5" and options.no35: return False if command == "python3.6" and options.no36: return False # Shortcuts for python versions, also needed for Windows as it won't have # the version number in the Python binaries at all. if command == "python2.6" and sys.version_info[0:2] == (2,6): return True if command == "python2.7" and sys.version_info[0:2] == (2,7): return True if command == "python3.2" and sys.version_info[0:2] == (3,2): return True if command == "python3.3" and sys.version_info[0:2] == (3,3): return True if command == "python3.4" and sys.version_info[0:2] == (3,4): return True if command == "python3.5" and sys.version_info[0:2] == (3,5): return True if command == "python3.6" and sys.version_info[0:2] == (3,6): return True path = os.environ[ "PATH" ] suffixes = (".exe",) if os.name == "nt" else ("",) for part in path.split(os.pathsep): if not part: continue for suffix in suffixes: if os.path.exists(os.path.join(part, command + suffix)): return True if os.name == "nt": if command.startswith("python"): remainder = command[6:] if len(remainder) == 3 and remainder[1] == '.': command = getPythonExePathWindows( search = remainder, arch = None ) return True return False def setExtraFlags(where, name, flags): if where is not None: tmp_dir = tempfile.gettempdir() # Try to avoid RAM disk /tmp and use the disk one instead. if tmp_dir == "/tmp" and os.path.exists("/var/tmp"): tmp_dir = "/var/tmp" where = os.path.join(tmp_dir, name, where) if not os.path.exists(where): os.makedirs(where) os.environ[ "NUITKA_EXTRA_OPTIONS" ] = flags + " --output-dir=" + where else: os.environ[ "NUITKA_EXTRA_OPTIONS" ] = flags def executeSubTest(command, hide_output = False): if options.coverage and "search" in command: command = command.replace("search", "coverage") parts = command.split() parts[0] = parts[0].replace('/', os.path.sep) # The running Python will be good enough, on some platforms there is no # "python", and we need to pass this alone then. parts.insert(0, sys.executable) print("Run '%s' in '%s'." % (' '.join(parts), os.getcwd())) sys.stdout.flush() if hide_output: result = subprocess.call( parts, stdout = open(os.devnull, 'w') ) else: result = subprocess.call( parts ) if result != 0: sys.exit(result) def execute_tests(where, use_python, flags): # Many cases, pylint: disable=too-many-branches,too-many-statements print( "Executing test case called %s with CPython %s and extra flags '%s'." % ( where, use_python, flags ) ) intended_version = use_python[6:] if sys.version.startswith(intended_version): os.environ[ "PYTHON" ] = sys.executable else: if os.name == "nt": os.environ[ "PYTHON" ] = getPythonExePathWindows( search = intended_version, arch = None ) else: os.environ[ "PYTHON" ] = getExecutablePath(use_python) if options.basic_tests: print("Running the basic tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "basics", flags) executeSubTest("./tests/basics/run_all.py search") if options.syntax_tests: print("Running the syntax tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "syntax", flags) executeSubTest("./tests/syntax/run_all.py search") if options.program_tests: print("Running the program tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "programs", flags) executeSubTest("./tests/programs/run_all.py search") if options.package_tests: print("Running the package tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "packages", flags) executeSubTest("./tests/packages/run_all.py search") # At least one Debian Jessie, these versions won't have lxml installed, so # don't run them there. Also these won't be very version dependent in their # results. if use_python != "python2.6" and use_python != "python3.2": if options.optimization_tests: print("Running the optimizations tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "optimizations", flags) executeSubTest("./tests/optimizations/run_all.py search") if options.standalone_tests and not options.coverage: print("Running the standalone tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(None, "standalone", flags) executeSubTest("./tests/standalone/run_all.py search") if options.reflection_test and not options.coverage: print("Running the reflection test with options '%s' with %s:" % (flags, use_python)) setExtraFlags(None, "reflected", flags) executeSubTest("./tests/reflected/compile_itself.py search") if not use_python.startswith("python3"): if os.path.exists("./tests/CPython26/run_all.py"): if options.cpython26: print("Running the CPython 2.6 tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "26tests", flags) executeSubTest("./tests/CPython26/run_all.py search") else: print("The CPython2.6 tests are not present, not run.") # Running the Python 2.7 test suite with CPython 2.6 gives little # insight, because "importlib" will not be there and that's it. if use_python != "python2.6": if os.path.exists("./tests/CPython27/run_all.py"): if options.cpython27: print("Running the CPython 2.7 tests with options '%s' with %s:" % (flags, use_python)) setExtraFlags(where, "27tests", flags) executeSubTest("./tests/CPython27/run_all.py search") else: print("The CPython2.7 tests are not present, not run.") if "--debug" not in flags: # Not running the Python 3.2 test suite with CPython2.6, as that's about # the same as CPython2.7 and won't have any new insights. if use_python != "python2.6" and \ use_python != "python2.7" or not options.coverage: if os.path.exists("./tests/CPython32/run_all.py"): if options.cpython32: setExtraFlags(where, "32tests", flags) executeSubTest("./tests/CPython32/run_all.py search") else: print("The CPython3.2 tests are not present, not run.") # Running the Python 3.3 test suite only with CPython3.x. if not use_python.startswith("python2"): if os.path.exists("./tests/CPython33/run_all.py"): if options.cpython33: setExtraFlags(where, "33tests", flags) executeSubTest("./tests/CPython33/run_all.py search") else: print("The CPython3.3 tests are not present, not run.") # Running the Python 3.4 test suite only with CPython3.x. if not use_python.startswith("python2"): if os.path.exists("./tests/CPython34/run_all.py"): if options.cpython34: setExtraFlags(where, "34tests", flags) executeSubTest("./tests/CPython34/run_all.py search") else: print("The CPython3.4 tests are not present, not run.") # Running the Python 3.4 test suite only with CPython3.x. if not use_python.startswith("python2"): if os.path.exists("./tests/CPython35/run_all.py"): if options.cpython35: setExtraFlags(where, "35tests", flags) executeSubTest("./tests/CPython35/run_all.py search") else: print("The CPython3.5 tests are not present, not run.") # Running the Python 3.4 test suite only with CPython3.x. if not use_python.startswith("python2"): if os.path.exists("./tests/CPython36/run_all.py"): if options.cpython36: setExtraFlags(where, "36tests", flags) executeSubTest("./tests/CPython36/run_all.py search") else: print("The CPython3.6 tests are not present, not run.") if "NUITKA_EXTRA_OPTIONS" in os.environ: del os.environ[ "NUITKA_EXTRA_OPTIONS" ] assert checkExecutableCommand("python2.6") or \ checkExecutableCommand("python2.7") or \ checkExecutableCommand("python3.2") or \ checkExecutableCommand("python3.3") or \ checkExecutableCommand("python3.4") or \ checkExecutableCommand("python3.5") or \ checkExecutableCommand("python3.6") # Just the quick syntax test, full tests are run later. if checkExecutableCommand("python3.2"): executeSubTest( "./bin/nuitka --python-version=3.2 --version", hide_output = True ) if checkExecutableCommand("python2.6"): execute_tests("python2.6-debug", "python2.6", "--debug") else: print("Cannot execute tests with Python 2.6, disabled or not installed.") if checkExecutableCommand("python2.7"): execute_tests("python2.7-debug", "python2.7", "--debug") else: print("Cannot execute tests with Python 2.7, disabled or not installed.") if checkExecutableCommand("python3.2"): execute_tests("python3.2-debug", "python3.2", "--debug") else: print("Cannot execute tests with Python 3.2, disabled or not installed.") if checkExecutableCommand("python2.6"): execute_tests("python2.6-nodebug", "python2.6", "") else: print("Cannot execute tests with Python 2.6, disabled or not installed.") if checkExecutableCommand("python2.7"): execute_tests("python2.7-nodebug", "python2.7", "") else: print("Cannot execute tests with Python 2.7, disabled or not installed.") if checkExecutableCommand("python3.2"): execute_tests("python3.2-nodebug", "python3.2", "") else: print("Cannot execute tests with Python 3.2, disabled or not installed.") if checkExecutableCommand("python3.3"): execute_tests("python3.3-nodebug", "python3.3", "") else: print("Cannot execute tests with Python 3.3, disabled or not installed.") if checkExecutableCommand("python3.4"): execute_tests("python3.4-nodebug", "python3.4", "") else: print("Cannot execute tests with Python 3.4, disabled or not installed.") if checkExecutableCommand("python3.5"): execute_tests("python3.5-nodebug", "python3.5", "") else: print("Cannot execute tests with Python 3.5, disabled or not installed.") if checkExecutableCommand("python3.6"): execute_tests("python3.6-nodebug", "python3.6", "") else: print("Cannot execute tests with Python 3.6, disabled or not installed.") if options.coverage: def copyToGlobalCoverageData(source, target): coverage_dir = os.environ.get("COVERAGE_DIR", None) if coverage_dir is None: return assert subprocess.call( ( "C:\\MinGW\\msys\\1.0\\bin\\scp.exe" if os.name == "nt" else "scp", source, os.path.join( coverage_dir, target ) ) ) == 0 if os.name == "nt": suffix = "win" else: import platform suffix = platform.uname()[0] + '.' + platform.uname()[4] with open("data.coverage", 'w') as data_file: source_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__))) nuitka_id = check_output( "cd '%s'; git rev-parse HEAD" % source_dir, shell = True ) nuitka_id = nuitka_id.strip() if sys.version_info > (3,): nuitka_id = nuitka_id.decode() data_file.write("NUITKA_COMMIT='%s'\n" % nuitka_id) copyToGlobalCoverageData("data.coverage", "data.coverage." + suffix) def makeCoverageRelative(filename): """ Normalize coverage data. """ with open(filename) as input_file: data = input_file.read() data = data.replace(os.path.abspath('.') + os.path.sep, "") if os.path.sep != '/': data.replace(os.path.sep, '/') with open(filename, 'w') as output_file: output_file.write(data) coverage_file = os.environ.get("COVERAGE_FILE", ".coverage") makeCoverageRelative(coverage_file) copyToGlobalCoverageData(coverage_file, ".coverage." + suffix) print("OK.") if __name__ == "__main__": main() Nuitka-0.5.28.2/nuitka/tools/testing/run_nuitka_tests/__init__.py0000644000372000001440000000150113134660221025300 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/tools/testing/Common.py0000644000372000001440000007565013207537242021417 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Common test infrastructure functions. To be used by test runners. """ from __future__ import print_function import ast import atexit import os import re import shutil import subprocess import sys import tempfile from contextlib import contextmanager from nuitka.Tracing import my_print from nuitka.utils.AppDirs import getCacheDir from nuitka.utils.Execution import check_output from nuitka.utils.FileOperations import makePath, removeDirectory from .SearchModes import ( SearchModeBase, SearchModeByPattern, SearchModeCoverage, SearchModeResume ) def check_result(*popenargs, **kwargs): if "stdout" in kwargs: raise ValueError("stdout argument not allowed, it will be overridden.") process = subprocess.Popen( stdout = subprocess.PIPE, *popenargs, **kwargs ) _unused_output, _unused_err = process.communicate() retcode = process.poll() if retcode: return False else: return True _python_version = None _python_arch = None _python_executable = None def setup(needs_io_encoding = False, silent = False): # Go its own directory, to have it easy with path knowledge. os.chdir( os.path.dirname( os.path.abspath(sys.modules[ "__main__" ].__file__) ) ) if "PYTHON" not in os.environ: os.environ["PYTHON"] = sys.executable # Allow providing 33, 27, and expand that to python2.7 if len(os.environ["PYTHON"]) == 2 and \ os.environ["PYTHON"].isdigit() and \ os.name != "nt": os.environ["PYTHON"] = "python%s.%s" % ( os.environ["PYTHON"][0], os.environ["PYTHON"][1] ) if needs_io_encoding and "PYTHONIOENCODING" not in os.environ: os.environ["PYTHONIOENCODING"] = "utf-8" version_output = check_output( ( os.environ["PYTHON"], "-c", """\ import sys, os;\ print(".".join(str(s) for s in list(sys.version_info)[:3]));\ print(("x86_64" if "AMD64" in sys.version else "x86") if os.name == "nt" else os.uname()[4]);\ print(sys.executable);\ """, ), stderr = subprocess.STDOUT ) global _python_version, _python_arch, _python_executable # singleton, pylint: disable=global-statement _python_version = version_output.split(b"\n")[0].strip() _python_arch = version_output.split(b"\n")[1].strip() _python_executable = version_output.split(b"\n")[2].strip() if sys.version.startswith('3'): _python_arch = _python_arch.decode("utf-8") _python_version = _python_version.decode("utf-8") _python_executable = _python_executable.decode("utf-8") if not silent: my_print("Using concrete python", _python_version, "on", _python_arch) assert type(_python_version) is str, repr(_python_version) assert type(_python_arch) is str, repr(_python_arch) assert type(_python_executable) is str, repr(_python_executable) if "COVERAGE_FILE" not in os.environ: os.environ["COVERAGE_FILE"] = os.path.join( os.path.dirname(__file__), "..", "..", "..", ".coverage" ) return _python_version tmp_dir = None def getTempDir(): # Create a temporary directory to work in, automatically remove it in case # it is empty in the end. global tmp_dir # singleton, pylint: disable=global-statement if tmp_dir is None: tmp_dir = tempfile.mkdtemp( prefix = os.path.basename( os.path.dirname( os.path.abspath(sys.modules[ "__main__" ].__file__) ) ) + '-', dir = tempfile.gettempdir() if not os.path.exists("/var/tmp") else "/var/tmp" ) def removeTempDir(): removeDirectory( path = tmp_dir, ignore_errors = True ) atexit.register(removeTempDir) return tmp_dir def convertUsing2to3(path, force = False): command = [ os.environ["PYTHON"], "-m", "py_compile", path ] if not force: with open(path) as source_file: if "xrange" not in source_file.read(): with open(os.devnull, 'w') as stderr: if check_result(command, stderr = stderr): return path, False filename = os.path.basename(path) new_path = os.path.join(getTempDir(), filename) # This may already be a temp file, e.g. because of construct creation. try: shutil.copy(path, new_path) except shutil.SameFileError: # @UndefinedVariable pass # On Windows, we cannot rely on 2to3 to be in the path. if os.name == "nt": command = [ sys.executable, os.path.join( os.path.dirname(sys.executable), "Tools/Scripts/2to3.py" ) ] else: command = [ "2to3" ] command += [ "-w", "-n", "--no-diffs", new_path ] with open(os.devnull, 'w') as devnull: check_output( command, stderr = devnull ) with open(new_path) as result_file: data = result_file.read() with open(new_path, 'w') as result_file: result_file.write("__file__ = %r\n" % os.path.abspath(path)) result_file.write(data) return new_path, True def decideFilenameVersionSkip(filename): """ Make decision whether to skip based on filename and Python version. This codifies certain rules that files can have as suffixes or prefixes to make them be part of the set of tests executed for a version or not. Generally, an ening of ".py" indicates that it must be that Python version or higher. There is no need for ending in "26.py" as this is the minimum version anyway. The "_2.py" indicates a maxmimum version of 2.7, i.e. not Python 3.x, for language syntax no more supported. """ # This will make many decisions with immediate returns. # pylint: disable=too-many-return-statements assert type(filename) is str assert type(_python_version) is str # Skip runner scripts by default. if filename.startswith("run_"): return False # Skip tests that require Python 2.7 at least. if filename.endswith("27.py") and _python_version.startswith("2.6"): return False if filename.endswith("_2.py") and _python_version.startswith('3'): return False # Skip tests that require Python 3.2 at least. if filename.endswith("32.py") and _python_version < "3.2": return False # Skip tests that require Python 3.3 at least. if filename.endswith("33.py") and _python_version < "3.3": return False # Skip tests that require Python 3.4 at least. if filename.endswith("34.py") and _python_version < "3.4": return False # Skip tests that require Python 3.5 at least. if filename.endswith("35.py") and _python_version < "3.5": return False # Skip tests that require Python 3.6 at least. if filename.endswith("36.py") and _python_version < "3.6": return False return True def _removeCPythonTestSuiteDir(): # Cleanup, some tests apparently forget that. try: if os.path.isdir("@test"): removeDirectory("@test", ignore_errors = False) elif os.path.isfile("@test"): os.unlink("@test") except OSError: # TODO: Move this into removeDirectory maybe. Doing an external # call as last resort could be a good idea. # This seems to work for broken "lnk" files. if os.name == "nt": os.system("rmdir /S /Q @test") if os.path.exists("@test"): raise def compareWithCPython(dirname, filename, extra_flags, search_mode, needs_2to3): """ Call the comparison tool. For a given directory filename. The search mode decides if the test case aborts on error or gets extra flags that are exceptions. """ if dirname is None: path = filename else: path = os.path.join(dirname, filename) # Apply 2to3 conversion if necessary. if needs_2to3: path, converted = convertUsing2to3(path) else: converted = False command = [ sys.executable, os.path.join("..", "..", "bin", "compare_with_cpython"), path, "silent" ] if extra_flags is not None: command += extra_flags command += search_mode.getExtraFlags(dirname, filename) # Cleanup before and after test stage directory. _removeCPythonTestSuiteDir() try: result = subprocess.call( command ) except KeyboardInterrupt: result = 2 # Cleanup before and after test stage directory. _removeCPythonTestSuiteDir() if result != 0 and \ result != 2 and \ search_mode.abortOnFinding(dirname, filename): my_print("Error exit!", result) sys.exit(result) if converted: os.unlink(path) if result == 2: sys.stderr.write("Interrupted, with CTRL-C\n") sys.exit(2) def checkCompilesNotWithCPython(dirname, filename, search_mode): if dirname is None: path = filename else: path = os.path.join(dirname, filename) command = [ _python_executable, "-mcompileall", path ] try: result = subprocess.call( command ) except KeyboardInterrupt: result = 2 if result != 1 and \ result != 2 and \ search_mode.abortOnFinding(dirname, filename): my_print("Error exit!", result) sys.exit(result) def checkSucceedsWithCPython(filename): command = [ _python_executable, filename ] result = subprocess.call( command, stdout = open(os.devnull,'w'), stderr = subprocess.STDOUT ) return result == 0 def hasDebugPython(): # On Debian systems, these work. debug_python = os.path.join("/usr/bin/", os.environ["PYTHON"] + "-dbg") if os.path.exists(debug_python): return True # On Windows systems, these work. debug_python = os.environ["PYTHON"] if debug_python.lower().endswith(".exe"): debug_python = debug_python[:-4] debug_python = debug_python + "_d.exe" if os.path.exists(debug_python): return True # For other Python, if it's the one also executing the runner, which is # very probably the case, we check that. We don't check the provided # binary here, this could be done as well. if sys.executable == os.environ["PYTHON"] and \ hasattr(sys, "gettotalrefcount"): return True # Otherwise no. return False def getArchitecture(): if os.name == "nt": if "AMD64" in sys.version: return "x86_64" else: return "x86" else: return os.uname()[4] # @UndefinedVariable def getDependsExePath(): if "APPDATA" not in os.environ: sys.exit("Error, standalone mode cannot find 'APPDATA' environment.") nuitka_app_dir = os.path.join(os.environ["APPDATA"],"nuitka") depends_dir = os.path.join( nuitka_app_dir, _python_arch, ) depends_exe = os.path.join( depends_dir, "depends.exe" ) assert os.path.exists(depends_exe), depends_exe return depends_exe def isExecutableCommand(command): path = os.environ["PATH"] suffixes = (".exe",) if os.name == "nt" else ("",) for part in path.split(os.pathsep): if not part: continue for suffix in suffixes: if os.path.isfile(os.path.join(part, command + suffix)): return True return False def getRuntimeTraceOfLoadedFiles(path, trace_error = True): """ Returns the files loaded when executing a binary. """ # This will make a crazy amount of work, pylint: disable=too-many-branches,too-many-statements result = [] if os.name == "posix": if sys.platform == "darwin" or \ sys.platform.startswith("freebsd"): if not isExecutableCommand("dtruss"): sys.exit( """\ Error, needs 'dtruss' on your system to scan used libraries.""" ) if not isExecutableCommand("sudo"): sys.exit( """\ Error, needs 'sudo' on your system to scan used libraries.""" ) args = ( "sudo", "dtruss", "-t", "open", path ) else: if not isExecutableCommand("strace"): sys.exit( """\ Error, needs 'strace' on your system to scan used libraries.""" ) args = ( "strace", "-e", "file", "-s4096", # Some paths are truncated otherwise. path ) process = subprocess.Popen( args = args, stdout = subprocess.PIPE, stderr = subprocess.PIPE ) _stdout_strace, stderr_strace = process.communicate() open(path+".strace","wb").write(stderr_strace) for line in stderr_strace.split(b"\n"): if process.returncode != 0 and trace_error: my_print(line) if not line: continue # Don't consider files not found. The "site" module checks lots # of things. if b"ENOENT" in line: continue if line.startswith(b"stat(") and b"S_IFDIR" in line: continue # Allow stats on the python binary, and stuff pointing to the # standard library, just not uses of it. It will search there # for stuff. if line.startswith(b"lstat(") or \ line.startswith(b"stat(") or \ line.startswith(b"readlink("): filename = line[line.find(b"(")+2:line.find(b", ")-1] binary_path = _python_executable if str is not bytes: binary_path = binary_path.encode("utf-8") found = False while binary_path: if filename == binary_path: found = True break if binary_path == os.path.dirname(binary_path): break binary_path = os.path.dirname(binary_path) if filename == os.path.join(binary_path, b"python" + _python_version[:3].encode("utf8")): found = True continue if found: continue result.extend( os.path.abspath(match) for match in re.findall(b'"(.*?)(?:\\\\0)?"', line) ) if sys.version.startswith('3'): result = [s.decode("utf-8") for s in result] elif os.name == "nt": subprocess.call( ( getDependsExePath(), "-c", "-ot%s" % path + ".depends", "-f1", "-pa1", "-ps1", "-pp0", "-pl1", path ) ) inside = False for line in open(path + ".depends"): if "| Module Dependency Tree |" in line: inside = True continue if not inside: continue if "| Module List |" in line: break if ']' not in line: continue # Skip missing DLLs, apparently not needed anyway. if '?' in line[:line.find(']')]: continue dll_filename = line[line.find(']')+2:-1] assert os.path.isfile(dll_filename), dll_filename # The executable itself is of course exempted. if os.path.normcase(dll_filename) == \ os.path.normcase(os.path.abspath(path)): continue dll_filename = os.path.normcase(dll_filename) result.append(dll_filename) os.unlink(path + ".depends") result = list(sorted(set(result))) return result def checkRuntimeLoadedFilesForOutsideAccesses(loaded_filenames, white_list): result = [] for loaded_filename in loaded_filenames: loaded_filename = os.path.normpath(loaded_filename) loaded_filename = os.path.normcase(loaded_filename) loaded_basename = os.path.basename(loaded_filename) ok = False for entry in white_list: if loaded_filename.startswith(entry): ok = True while entry: old_entry = entry entry = os.path.dirname(entry) if old_entry == entry: break if loaded_filename == entry: ok = True break if ok: continue if loaded_filename.startswith("/etc/"): continue if loaded_filename.startswith("/proc/") or loaded_filename == "/proc": continue if loaded_filename.startswith("/dev/"): continue if loaded_filename.startswith("/tmp/"): continue if loaded_filename.startswith("/run/"): continue if loaded_filename.startswith("/sys/"): continue if loaded_filename.startswith("/usr/lib/locale/"): continue if loaded_filename.startswith("/usr/share/locale/"): continue if loaded_filename.startswith("/usr/share/X11/locale/"): continue # Themes may of course be loaded. if loaded_filename.startswith("/usr/share/themes"): continue if "gtk" in loaded_filename and "/engines/" in loaded_filename: continue # Terminal info files are OK too. if loaded_filename.startswith("/lib/terminfo/"): continue # System C libraries are to be expected. if loaded_basename.startswith(( "libc.so.", "libpthread.so.", "libdl.so.", "libm.so.", )): continue # Taking these from system is harmless and desirable if loaded_basename.startswith(( "libz.so", "libutil.so", "libgcc_s.so", )): continue # TODO: Unclear, loading gconv from filesystem of installed system # may be OK or not. I think it should be. if loaded_basename == "gconv-modules.cache": continue if "/gconv/" in loaded_filename: continue if loaded_basename.startswith("libicu"): continue result.append(loaded_filename) return result def hasModule(module_name): result = subprocess.call( ( os.environ["PYTHON"], "-c" "import %s" % module_name ), stdout = open(os.devnull,'w'), stderr = subprocess.STDOUT ) return result == 0 m1 = {} m2 = {} def snapObjRefCntMap(before): import gc if before: m = m1 else: m = m2 for x in gc.get_objects(): if x is m1: continue if x is m2: continue m[ str(x) ] = sys.getrefcount(x) def checkReferenceCount(checked_function, max_rounds = 10): assert sys.exc_info() == (None, None, None), sys.exc_info() print(checked_function.__name__ + ": ", end = "") sys.stdout.flush() ref_count1 = 17 ref_count2 = 17 explain = False import gc assert max_rounds > 0 for count in range(max_rounds): gc.collect() ref_count1 = sys.gettotalrefcount() # @UndefinedVariable if explain and count == max_rounds - 1: snapObjRefCntMap(True) checked_function() # Not allowed, but happens when bugs occur. assert sys.exc_info() == (None, None, None), sys.exc_info() gc.collect() if explain and count == max_rounds - 1: snapObjRefCntMap(False) ref_count2 = sys.gettotalrefcount() # @UndefinedVariable if ref_count1 == ref_count2: result = True print("PASSED") break # print count, ref_count1, ref_count2 else: result = False print("FAILED", ref_count1, ref_count2, "leaked", ref_count2 - ref_count1) if explain: assert m1 assert m2 for key in m1: if key not in m2: print('*' * 80) print(key) elif m1[key] != m2[key]: print('*' * 80) print(key) else: pass # print m1[key] assert sys.exc_info() == (None, None, None), sys.exc_info() gc.collect() sys.stdout.flush() return result def createSearchMode(): search_mode = len(sys.argv) > 1 and sys.argv[1] == "search" resume_mode = len(sys.argv) > 1 and sys.argv[1] == "resume" start_at = sys.argv[2] if len(sys.argv) > 2 else None coverage_mode = len(sys.argv) > 1 and sys.argv[1] == "coverage" if coverage_mode: return SearchModeCoverage() elif resume_mode: return SearchModeResume( sys.modules["__main__"].__file__ ) elif search_mode and start_at: start_at = start_at.replace('/', os.path.sep) return SearchModeByPattern(start_at) else: class SearchModeImmediate(SearchModeBase): def abortOnFinding(self, dirname, filename): return search_mode and \ SearchModeBase.abortOnFinding(self, dirname, filename) return SearchModeImmediate() def reportSkip(reason, dirname, filename): case = os.path.join(dirname, filename) case = os.path.normpath(case) my_print("Skipped, %s (%s)." % (case, reason)) def executeReferenceChecked(prefix, names, tests_skipped, tests_stderr): import gc gc.disable() extract_number = lambda name: int(name.replace(prefix, "")) # Find the function names. matching_names = tuple( name for name in names if name.startswith(prefix) and name[-1].isdigit() ) old_stderr = sys.stderr # Everything passed result = True for name in sorted(matching_names, key = extract_number): number = extract_number(name) # print(tests_skipped) if number in tests_skipped: my_print(name + ": SKIPPED (%s)" % tests_skipped[number]) continue # Avoid unraisable output. try: if number in tests_stderr: sys.stderr = open(os.devnull, "wb") except OSError: # Windows if not checkReferenceCount(names[name]): result = False else: if not checkReferenceCount(names[name]): result = False if number in tests_stderr: new_stderr = sys.stderr sys.stderr = old_stderr new_stderr.close() gc.enable() return result def checkDebugPython(): if not hasattr(sys, "gettotalrefcount"): my_print("Warning, using non-debug Python makes this test ineffective.") sys.gettotalrefcount = lambda : 0 @contextmanager def withPythonPathChange(python_path): if type(python_path) in (tuple, list): python_path = os.pathsep.join(python_path) if python_path: if "PYTHONPATH" in os.environ: old_path = os.environ["PYTHONPATH"] os.environ["PYTHONPATH"] += os.pathsep + python_path else: old_path = None os.environ["PYTHONPATH"] = python_path yield if python_path: if old_path is None: del os.environ["PYTHONPATH"] else: os.environ["PYTHONPATH"] = old_path @contextmanager def withExtendedExtraOptions(*args): assert args old_value = os.environ.get("NUITKA_EXTRA_OPTIONS", None) value = old_value for arg in args: if value is None: value = arg else: value += ' ' + arg os.environ[ "NUITKA_EXTRA_OPTIONS" ] = value yield if old_value is None: del os.environ[ "NUITKA_EXTRA_OPTIONS" ] else: os.environ[ "NUITKA_EXTRA_OPTIONS" ] = old_value def indentedCode(codes, count): """ Indent code, used for generating test codes. """ return '\n'.join( ' ' * count + line if line else "" for line in codes ) def convertToPython(doctests, line_filter = None): """ Convert give doctest string to static Python code. """ # This is convoluted, but it just needs to work, pylint: disable=too-many-branches import doctest code = doctest.script_from_examples(doctests) if code.endswith('\n'): code += "#\n" else: assert False output = [] inside = False def getPrintPrefixed(evaluated, line_number): try: node = ast.parse(evaluated.lstrip(), "eval") except SyntaxError: return evaluated if node.body[0].__class__.__name__ == "Expr": count = 0 while evaluated.startswith(' ' * count): count += 1 if sys.version_info < (3,): modified = (count-1) * ' ' + "print " + evaluated return (count-1) * ' ' + ("print 'Line %d'" % line_number) + '\n' + modified else: modified = (count-1) * ' ' + "print(" + evaluated + "\n)\n" return (count-1) * ' ' + ("print('Line %d'" % line_number) + ")\n" + modified else: return evaluated def getTried(evaluated, line_number): if sys.version_info < (3,): return """ try: %(evaluated)s except Exception as __e: print "Occurred", type(__e), __e """ % { "evaluated" : indentedCode(getPrintPrefixed(evaluated, line_number).split('\n'), 4) } else: return """ try: %(evaluated)s except Exception as __e: print("Occurred", type(__e), __e) """ % { "evaluated" : indentedCode(getPrintPrefixed(evaluated, line_number).split('\n'), 4) } def isOpener(evaluated): evaluated = evaluated.lstrip() if evaluated == "": return False return evaluated.split()[0] in ( "def", "class", "for", "while", "try:", "except", "except:", "finally:", "else:" ) chunk = None for line_number, line in enumerate(code.split('\n')): # print "->", inside, line if line_filter is not None and line_filter(line): continue if inside and line and line[0].isalnum() and not isOpener(line): output.append(getTried('\n'.join(chunk), line_number)) # @UndefinedVariable chunk = [] inside = False if inside and not (line.startswith('#') and line.find("SyntaxError:") != -1): chunk.append(line) elif line.startswith('#'): if line.find("SyntaxError:") != -1: # print "Syntax error detected" if inside: # print "Dropping chunk", chunk chunk = [] inside = False else: del output[-1] elif isOpener(line): inside = True chunk = [line] elif line.strip() == "": output.append(line) else: output.append(getTried(line, line_number)) return '\n'.join(output).rstrip() + '\n' def compileLibraryPath(search_mode, path, stage_dir, decide, action): my_print("Checking standard library path:", path) for root, dirnames, filenames in os.walk(path): dirnames_to_remove = [ dirname for dirname in dirnames if '-' in dirname ] for dirname in dirnames_to_remove: dirnames.remove(dirname) dirnames.sort() filenames = [ filename for filename in filenames if decide(root, filename) ] for filename in sorted(filenames): if not search_mode.consider(root, filename): continue full_path = os.path.join(root, filename) my_print(full_path, ':', end = ' ') sys.stdout.flush() action(stage_dir, path, full_path) def compileLibraryTest(search_mode, stage_dir, decide, action): if not os.path.exists(stage_dir): os.makedirs(stage_dir) my_dirname = os.path.join(os.path.dirname(__file__), "../../..") my_dirname = os.path.normpath(my_dirname) paths = [ path for path in sys.path if not path.startswith(my_dirname) ] my_print("Using standard library paths:") for path in paths: my_print(path) for path in paths: print("Checking path:", path) compileLibraryPath( search_mode = search_mode, path = path, stage_dir = stage_dir, decide = decide, action = action ) def run_async(coro): """ Execute a coroutine until it's done. """ values = [] result = None while True: try: values.append(coro.send(None)) except StopIteration as ex: result = ex.args[0] if ex.args else None break return values, result def async_iterate(g): """ Execute async generator until it's done. """ # Test code for Python3, catches all kinds of exceptions. # pylint: disable=broad-except # Also Python3 only, pylint: disable=I0021,undefined-variable res = [] while True: try: g.__anext__().__next__() except StopAsyncIteration: # @UndefinedVariable res.append("STOP") break except StopIteration as ex: if ex.args: res.append("ex arg %s" % ex.args[0]) else: res.append("EMPTY StopIteration") break except Exception as ex: res.append(str(type(ex))) return res def getTestingCacheDir(): cache_dir = getCacheDir() result = os.path.join(cache_dir, "tests_state") makePath(result) return result Nuitka-0.5.28.2/nuitka/tools/testing/__init__.py0000644000372000001440000000150113122472300021673 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/tools/Basics.py0000644000372000001440000000324313112214770017674 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Basics for Nuitka tools. """ import os import sys def goHome(): """ Go its own directory, to have it easy with path knowledge. """ os.chdir( getHomePath() ) def getHomePath(): return os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) def setupPATH(): """ Make sure installed tools are in PATH. For Windows, add this to the PATH, so pip installed PyLint will be found near the Python executing this script. """ os.environ["PATH"] = os.environ["PATH"] + \ os.pathsep + \ os.path.join(os.path.dirname(sys.executable),"scripts") def addPYTHONPATH(path): python_path = os.environ.get("PYTHONPATH", "") os.environ["PYTHONPATH"] = os.pathsep.join( python_path.split(os.pathsep) + \ [path] ) Nuitka-0.5.28.2/nuitka/tools/__init__.py0000644000372000001440000000150113112214770020222 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/tools/profiler/0000755000372000001440000000000013207540420017735 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/tools/profiler/__main__.py0000644000372000001440000000504013134660221022027 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Profiling for Nuitka and CPython. This provides the capability of comparing performance results of Nuitka and CPython relatively to one another. """ # Note: This is currently severely broken. from __future__ import print_function import runpy import sys import tempfile import vmprof # @UnresolvedImport pylint: disable=I0021,import-error def _namelen(e): if e.startswith("py:"): return len(e.split(':')[1]) else: return len(e) def show(stats): p = stats.top_profile() if not p: print("no stats") return p.sort(key = lambda x: x[1], reverse = True) top = p[0][1] max_len = max([_namelen(e[0]) for e in p]) print(" vmprof output:") print(" %: name:" + ' ' * (max_len - 3) + "location:") for k, v in p: v = "%.1f%%" % (float(v) / top * 100) if v == "0.0%": v = "<0.1%" if k.startswith("py:"): _, func_name, lineno, filename = k.split(':', 3) lineno = int(lineno) print(" %s %s %s:%d" % (v.ljust(7), func_name.ljust(max_len + 1), filename, lineno)) else: print(" %s %s" % (v.ljust(7), k.ljust(max_len + 1))) def main(): with tempfile.NamedTemporaryFile() as prof_file: vmprof.enable(prof_file.fileno(), 0.001) try: program = sys.argv[1] del sys.argv[1] # sys.argv = [args.program] + args.args runpy.run_path(program, run_name = "__main__") except BaseException as e: if not isinstance(e, (KeyboardInterrupt, SystemExit)): raise vmprof.disable() stats = vmprof.read_profile( prof_file.name, virtual_only = True ) show(stats) if __name__ == "__main__": main() Nuitka-0.5.28.2/nuitka/tools/profiler/__init__.py0000644000372000001440000000150113122472300022040 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/nodes/0000755000372000001440000000000013207540420016063 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/nodes/ExceptionNodes.py0000644000372000001440000002300313122472300021357 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes related to raising and making exceptions. """ from .ExpressionBases import ExpressionBase, ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions class StatementRaiseException(StatementChildrenHavingBase): kind = "STATEMENT_RAISE_EXCEPTION" named_children = ( "exception_type", "exception_value", "exception_trace", "exception_cause" ) def __init__(self, exception_type, exception_value, exception_trace, exception_cause, source_ref): if exception_type is None: assert exception_value is None if exception_value is None: assert exception_trace is None StatementChildrenHavingBase.__init__( self, values = { "exception_type" : exception_type, "exception_value" : exception_value, "exception_trace" : exception_trace, "exception_cause" : exception_cause }, source_ref = source_ref ) self.reraise_finally = False getExceptionType = StatementChildrenHavingBase.childGetter( "exception_type" ) getExceptionValue = StatementChildrenHavingBase.childGetter( "exception_value" ) getExceptionTrace = StatementChildrenHavingBase.childGetter( "exception_trace" ) getExceptionCause = StatementChildrenHavingBase.childGetter( "exception_cause" ) def isStatementReraiseException(self): return self.getExceptionType() is None def isStatementAborting(self): return True @staticmethod def isImplicit(): return False def computeStatement(self, trace_collection): trace_collection.onExpression( expression = self.getExceptionType(), allow_none = True ) exception_type = self.getExceptionType() # TODO: Limit by type. trace_collection.onExceptionRaiseExit(BaseException) if exception_type is not None and \ exception_type.willRaiseException(BaseException): from .NodeMakingHelpers import makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = exception_type, node = self ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception type.""" trace_collection.onExpression( expression = self.getExceptionValue(), allow_none = True ) exception_value = self.getExceptionValue() if exception_value is not None and exception_value.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( exception_type, exception_value ) ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception value.""" trace_collection.onExpression( expression = self.getExceptionTrace(), allow_none = True ) exception_trace = self.getExceptionTrace() if exception_trace is not None and \ exception_trace.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( exception_type, exception_value, exception_trace ) ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception traceback.""" trace_collection.onExpression( expression = self.getExceptionCause(), allow_none = True ) exception_cause = self.getExceptionCause() if exception_cause is not None and \ exception_cause.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( exception_type, exception_cause, ) ) return result, "new_raise", """ Explicit raise already raises implicitly building exception cause.""" return self, None, None def needsFrame(self): return not self.isStatementReraiseException() class StatementRaiseExceptionImplicit(StatementRaiseException): kind = "STATEMENT_RAISE_EXCEPTION_IMPLICIT" @staticmethod def isStatementRaiseException(): return True @staticmethod def isImplicit(): return True class ExpressionRaiseException(ExpressionChildrenHavingBase): """ This node type is only produced via optimization. CPython only knows exception raising as a statement, but often the raising of exceptions can be predicted to occur as part of an expression, which it replaces then. """ kind = "EXPRESSION_RAISE_EXCEPTION" named_children = ( "exception_type", "exception_value" ) def __init__(self, exception_type, exception_value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "exception_type" : exception_type, "exception_value" : exception_value }, source_ref = source_ref ) def willRaiseException(self, exception_type): # One thing is clear, it will raise. TODO: Match exception_type more # closely if it is predictable. return exception_type is BaseException getExceptionType = ExpressionChildrenHavingBase.childGetter( "exception_type" ) getExceptionValue = ExpressionChildrenHavingBase.childGetter( "exception_value" ) def computeExpression(self, trace_collection): return self, None, None def computeExpressionDrop(self, statement, trace_collection): result = StatementRaiseExceptionImplicit( exception_type = self.getExceptionType(), exception_value = self.getExceptionValue(), exception_trace = None, exception_cause = None, source_ref = self.getSourceReference() ) return result, "new_raise", """\ Propagated implicit raise expression to raise statement.""" class ExpressionBuiltinMakeException(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_MAKE_EXCEPTION" named_children = ( "args", ) def __init__(self, exception_name, args, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "args" : tuple(args), }, source_ref = source_ref ) self.exception_name = exception_name def getDetails(self): return { "exception_name" : self.exception_name } def getExceptionName(self): return self.exception_name getArgs = ExpressionChildrenHavingBase.childGetter("args") def computeExpression(self, trace_collection): return self, None, None def mayRaiseException(self, exception_type): for arg in self.getArgs(): if arg.mayRaiseException(exception_type): return True return False class ExpressionCaughtExceptionTypeRef(ExpressionBase): kind = "EXPRESSION_CAUGHT_EXCEPTION_TYPE_REF" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def computeExpressionRaw(self, trace_collection): # TODO: Might be predictable based on the exception handler this is in. return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False class ExpressionCaughtExceptionValueRef(ExpressionBase): kind = "EXPRESSION_CAUGHT_EXCEPTION_VALUE_REF" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def computeExpressionRaw(self, trace_collection): # TODO: Might be predictable based on the exception handler this is in. return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False class ExpressionCaughtExceptionTracebackRef(ExpressionBase): kind = "EXPRESSION_CAUGHT_EXCEPTION_TRACEBACK_REF" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def computeExpressionRaw(self, trace_collection): return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False Nuitka-0.5.28.2/nuitka/nodes/BuiltinDecodingNodes.py0000644000372000001440000000323013134660221022470 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in ord/chr nodes These are good for optimizations, as they give a very well known result. In the case of 'chr', it's one of 256 strings, and in case of 'ord' it's one of 256 numbers, so these can answer quite a few questions at compile time. """ from nuitka.optimizations import BuiltinOptimization from .ExpressionBases import ExpressionBuiltinSingleArgBase class ExpressionBuiltinOrd(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ORD" builtin_spec = BuiltinOptimization.builtin_ord_spec def isKnownToBeIterable(self, count): return False class ExpressionBuiltinChr(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_CHR" builtin_spec = BuiltinOptimization.builtin_chr_spec def isKnownToBeIterable(self, count): if self.mayRaiseException(BaseException): return None return count is None or count == 1 Nuitka-0.5.28.2/nuitka/nodes/BuiltinVarsNodes.py0000644000372000001440000000317713134660221021701 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Builtin vars node. Not used much, esp. not in the form with arguments. Maybe used in some meta programming, and hopefully can be predicted, because at run time, it is hard to support. """ from .ExpressionBases import ExpressionChildrenHavingBase class ExpressionBuiltinVars(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_VARS" named_children = ( "source", ) def __init__(self, source, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source, }, source_ref = source_ref ) getSource = ExpressionChildrenHavingBase.childGetter("source") def computeExpression(self, trace_collection): # TODO: Should be possible to predict this. return self, None, None def mayBeNone(self): return False Nuitka-0.5.28.2/nuitka/nodes/AttributeNodes.py0000644000372000001440000003245013122472300021372 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Attribute nodes Knowing attributes of an object is very important, esp. when it comes to 'self' and objects and classes. There will be a methods "computeExpression*Attribute" to aid predicting them, with many variants for setting, deleting, and accessing. Also there is some complication in the form of special lookups, that won't go through the normal path, but just check slots. Due to ``getattr`` and ``setattr`` built-ins, there is also a different in the computations for objects and for compile time known strings. This reflects what CPython also does with "tp_getattr" and "tp_getattro". These nodes are therefore mostly delegating the work to expressions they work on, and let them decide and do the heavy lifting of optimization and annotation is happening in the nodes that implement these compute slots. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import wrapExpressionWithNodeSideEffects class StatementAssignmentAttribute(StatementChildrenHavingBase): """ Assignment to an attribute. Typically from code like: source.attribute_name = expression Both source and expression may be complex expressions, the source is evaluated first. Assigning to an attribute has its on slot on the source, which gets to decide if it knows it will work or not, and what value it will be. """ kind = "STATEMENT_ASSIGNMENT_ATTRIBUTE" named_children = ( "source", "expression" ) def __init__(self, expression, attribute_name, source, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression, "source" : source, }, source_ref = source_ref ) self.attribute_name = attribute_name def getDetails(self): return { "attribute_name" : self.attribute_name } def getDetail(self): return "to attribute %s" % self.attribute_name def getAttributeName(self): return self.attribute_name def setAttributeName(self, attribute_name): self.attribute_name = attribute_name getLookupSource = StatementChildrenHavingBase.childGetter("expression") getAssignSource = StatementChildrenHavingBase.childGetter("source") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc return self.getLookupSource().computeExpressionSetAttribute( set_node = self, attribute_name = self.attribute_name, value_node = self.getAssignSource(), trace_collection = trace_collection ) def getStatementNiceName(self): return "attribute assignment statement" class StatementDelAttribute(StatementChildrenHavingBase): """ Deletion of an attribute. Typically from code like: del source.attribute_name The source may be complex expression. Deleting an attribute has its on slot on the source, which gets to decide if it knows it will work or not, and what value it will be. """ kind = "STATEMENT_DEL_ATTRIBUTE" named_children = ( "expression", ) def __init__(self, expression, attribute_name, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.attribute_name = attribute_name def getDetails(self): return { "attribute_name" : self.attribute_name } def getDetail(self): return "to attribute %s" % self.attribute_name def getAttributeName(self): return self.attribute_name def setAttributeName(self, attribute_name): self.attribute_name = attribute_name getLookupSource = StatementChildrenHavingBase.childGetter("expression") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection, ) if result is not self: return result, change_tags, change_desc return self.getLookupSource().computeExpressionDelAttribute( set_node = self, attribute_name = self.attribute_name, trace_collection = trace_collection ) def getStatementNiceName(self): return "attribute del statement" class ExpressionAttributeLookup(ExpressionChildrenHavingBase): """ Looking up an attribute of an object. Typically code like: source.attribute_name """ kind = "EXPRESSION_ATTRIBUTE_LOOKUP" named_children = ( "source", ) def __init__(self, source, attribute_name, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source }, source_ref = source_ref ) self.attribute_name = attribute_name def getAttributeName(self): return self.attribute_name def setAttributeName(self, attribute_name): self.attribute_name = attribute_name def getDetails(self): return { "attribute_name" : self.getAttributeName() } def getDetail(self): return "attribute %s from %s" % ( self.getAttributeName(), self.getLookupSource() ) getLookupSource = ExpressionChildrenHavingBase.childGetter( "source" ) def computeExpression(self, trace_collection): return self.getLookupSource().computeExpressionAttribute( lookup_node = self, attribute_name = self.getAttributeName(), trace_collection = trace_collection ) def mayRaiseException(self, exception_type): return self.getLookupSource().mayRaiseExceptionAttributeLookup( exception_type = exception_type, attribute_name = self.getAttributeName() ) def isKnownToBeIterable(self, count): # TODO: Could be known. We would need for computeExpressionAttribute to # either return a new node, or a decision maker. return None class ExpressionAttributeLookupSpecial(ExpressionAttributeLookup): """ Special lookup up an attribute of an object. Typically from code like this: with source: pass These directly go to slots, and are performed for with statements of Python2.7 or higher. """ kind = "EXPRESSION_ATTRIBUTE_LOOKUP_SPECIAL" def computeExpression(self, trace_collection): return self.getLookupSource().computeExpressionAttributeSpecial( lookup_node = self, attribute_name = self.getAttributeName(), trace_collection = trace_collection ) class ExpressionBuiltinGetattr(ExpressionChildrenHavingBase): """ Built-in "getattr". Typical code like this: getattr(source, attribute, default) The default is optional, but computed before the lookup is done. """ kind = "EXPRESSION_BUILTIN_GETATTR" named_children = ("source", "attribute", "default") @calledWithBuiltinArgumentNamesDecorator def __init__(self, object_arg, name, default, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : object_arg, "attribute" : name, "default" : default }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("source") getAttribute = ExpressionChildrenHavingBase.childGetter("attribute") getDefault = ExpressionChildrenHavingBase.childGetter("default") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) default = self.getDefault() if default is None or not default.mayHaveSideEffects(): attribute = self.getAttribute() attribute_name = attribute.getStringValue() if attribute_name is not None: source = self.getLookupSource() if source.isKnownToHaveAttribute(attribute_name): # If source has side effects, they must be evaluated, before # the lookup, meaning, a temporary variable should be assigned. # For now, we give up in this case. side_effects = source.extractSideEffects() if not side_effects: result = ExpressionAttributeLookup( source = source, attribute_name = attribute_name, source_ref = self.source_ref ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = attribute ) return ( result, "new_expression", """Replaced call to built-in 'getattr' with constant \ attribute '%s' to mere attribute lookup""" % attribute_name ) return self, None, None class ExpressionBuiltinSetattr(ExpressionChildrenHavingBase): """ Built-in "setattr". Typical code like this: setattr(source, attribute, value) """ kind = "EXPRESSION_BUILTIN_SETATTR" named_children = ("source", "attribute", "value") @calledWithBuiltinArgumentNamesDecorator def __init__(self, object_arg, name, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : object_arg, "attribute" : name, "value" : value }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("source") getAttribute = ExpressionChildrenHavingBase.childGetter("attribute") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) # Note: Might be possible to predict or downgrade to mere attribute set. return self, None, None class ExpressionBuiltinHasattr(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_HASATTR" named_children = ("source", "attribute") @calledWithBuiltinArgumentNamesDecorator def __init__(self, object_arg, name, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : object_arg, "attribute" : name, }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("source") getAttribute = ExpressionChildrenHavingBase.childGetter("attribute") def computeExpression(self, trace_collection): # We do at least for compile time constants optimization here, but more # could be done, were we to know shapes. source = self.getLookupSource() if source.isCompileTimeConstant(): attribute = self.getAttribute() attribute_name = attribute.getStringValue() if attribute_name is not None: # If source or attribute have side effects, they must be # evaluated, before the lookup. result, tags, change_desc = trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : hasattr( source.getCompileTimeConstant(), attribute_name ), description = "Call to 'hasattr' pre-computed." ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = attribute ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = source ) return result, tags, change_desc trace_collection.onExceptionRaiseExit(BaseException) return self, None, None Nuitka-0.5.28.2/nuitka/nodes/SubscriptNodes.py0000644000372000001440000001112113122472300021375 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Subscript node. Subscripts are important when working with lists and dictionaries. Tracking them can allow to achieve more compact code, or predict results at compile time. There is be a method "computeExpressionSubscript" to aid predicting them in the other nodes. """ from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase class StatementAssignmentSubscript(StatementChildrenHavingBase): kind = "STATEMENT_ASSIGNMENT_SUBSCRIPT" named_children = ( "source", "expression", "subscript" ) def __init__(self, expression, subscript, source, source_ref): StatementChildrenHavingBase.__init__( self, values = { "source" : source, "expression" : expression, "subscript" : subscript }, source_ref = source_ref ) getSubscribed = StatementChildrenHavingBase.childGetter("expression") getSubscript = StatementChildrenHavingBase.childGetter("subscript") getAssignSource = StatementChildrenHavingBase.childGetter("source") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc return self.getSubscribed().computeExpressionSetSubscript( set_node = self, subscript = self.getSubscript(), value_node = self.getAssignSource(), trace_collection = trace_collection ) def getStatementNiceName(self): return "subscript assignment statement" class StatementDelSubscript(StatementChildrenHavingBase): kind = "STATEMENT_DEL_SUBSCRIPT" named_children = ( "expression", "subscript" ) def __init__(self, expression, subscript, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression, "subscript" : subscript }, source_ref = source_ref ) getSubscribed = StatementChildrenHavingBase.childGetter("expression") getSubscript = StatementChildrenHavingBase.childGetter("subscript") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc return self.getSubscribed().computeExpressionDelSubscript( del_node = self, subscript = self.getSubscript(), trace_collection = trace_collection ) def getStatementNiceName(self): return "subscript del statement" class ExpressionSubscriptLookup(ExpressionChildrenHavingBase): kind = "EXPRESSION_SUBSCRIPT_LOOKUP" named_children = ( "subscribed", "subscript" ) def __init__(self, subscribed, subscript, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "subscribed" : subscribed, "subscript" : subscript }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("subscribed") getSubscript = ExpressionChildrenHavingBase.childGetter("subscript") def computeExpression(self, trace_collection): return self.getLookupSource().computeExpressionSubscript( lookup_node = self, subscript = self.getSubscript(), trace_collection = trace_collection ) def isKnownToBeIterable(self, count): return None Nuitka-0.5.28.2/nuitka/nodes/ReturnNodes.py0000644000372000001440000001377013134660221020716 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Return node This one exits functions. The only other exit is the default exit of functions with 'None' value, if no return is done. """ from abc import abstractmethod from .ExpressionBases import ExpressionBase from .NodeBases import NodeBase, StatementChildrenHavingBase from .NodeMakingHelpers import makeConstantReplacementNode class StatementReturn(StatementChildrenHavingBase): kind = "STATEMENT_RETURN" named_children = ("expression",) nice_children = ("return value",) def __init__(self, expression, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) getExpression = StatementChildrenHavingBase.childGetter( "expression" ) def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return self.getExpression().mayRaiseException(exception_type) def computeStatement(self, trace_collection): trace_collection.onExpression(self.getExpression()) expression = self.getExpression() if expression.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) if expression.willRaiseException(BaseException): from .NodeMakingHelpers import makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = expression, node = self ) return result, "new_raise", """\ Return statement raises in returned expression, removed return.""" trace_collection.onFunctionReturn() if expression.isExpressionConstantRef(): result = makeStatementReturnConstant( constant = expression.getCompileTimeConstant(), source_ref = self.source_ref ) return result, "new_statements", """\ Return value is always constant.""" return self, None, None class StatementReturnConstantBase(NodeBase): __slots__ = () def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) @staticmethod def isStatementReturn(): return True def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return False def computeStatement(self, trace_collection): trace_collection.onFunctionReturn() return self, None, None @abstractmethod def getConstant(self): """ The returned constant value. """ def getExpression(self): return makeConstantReplacementNode( node = self, constant = self.getConstant() ) class StatementReturnNone(StatementReturnConstantBase): kind = "STATEMENT_RETURN_NONE" __slots__ = () def __init__(self, source_ref): StatementReturnConstantBase.__init__( self, source_ref = source_ref ) def getConstant(self): return None class StatementReturnFalse(StatementReturnConstantBase): kind = "STATEMENT_RETURN_FALSE" __slots__ = () def __init__(self, source_ref): StatementReturnConstantBase.__init__( self, source_ref = source_ref ) def getConstant(self): return False class StatementReturnTrue(StatementReturnConstantBase): kind = "STATEMENT_RETURN_TRUE" __slots__ = () def __init__(self, source_ref): StatementReturnConstantBase.__init__( self, source_ref = source_ref ) def getConstant(self): return True class StatementReturnConstant(StatementReturnConstantBase): kind = "STATEMENT_RETURN_CONSTANT" __slots__ = ("constant",) def __init__(self, constant, source_ref): StatementReturnConstantBase.__init__( self, source_ref = source_ref ) self.constant = constant def getConstant(self): return self.constant def getDetails(self): return { "constant" : self.constant } def makeStatementReturnConstant(constant, source_ref): if constant is None: return StatementReturnNone( source_ref = source_ref ) elif constant is True: return StatementReturnTrue( source_ref = source_ref ) elif constant is False: return StatementReturnFalse( source_ref = source_ref ) else: return StatementReturnConstant( constant = constant, source_ref = source_ref ) class ExpressionReturnedValueRef(ExpressionBase): kind = "EXPRESSION_RETURNED_VALUE_REF" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def computeExpressionRaw(self, trace_collection): # TODO: Might be predictable based on the exception handler this is in. return self, None, None def mayHaveSideEffects(self): # Referencing the expression type has no side effect return False def mayRaiseException(self, exception_type): return False Nuitka-0.5.28.2/nuitka/nodes/BuiltinDecoratorNodes.py0000644000372000001440000000542113134660221022702 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in staticmethod/classmethod nodes These are good for optimizations, as they give a very well known result, changing only the way a class member is being called. Being able to avoid going through a C call to the built-ins resulting wrapper, will speed up things. """ from .ExpressionBases import ExpressionChildrenHavingBase from .shapes.BuiltinTypeShapes import ( ShapeTypeClassmethod, ShapeTypeStaticmethod ) class ExpressionBuiltinStaticmethod(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_STATICMETHOD" named_children = ( "value", ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def __init__(self, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "value" : value }, source_ref = source_ref ) def computeExpression(self, trace_collection): # TODO: Consider shape and predict exception raise or not. trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def isKnownToBeIterable(self, count): return False def getTypeShape(self): return ShapeTypeStaticmethod class ExpressionBuiltinClassmethod(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_CLASSMETHOD" named_children = ( "value", ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def __init__(self, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "value" : value }, source_ref = source_ref ) def computeExpression(self, trace_collection): # TODO: Consider shape and predict exception raise or not. trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def isKnownToBeIterable(self, count): return False def getTypeShape(self): return ShapeTypeClassmethod Nuitka-0.5.28.2/nuitka/nodes/AsyncgenNodes.py0000644000372000001440000001053313207537242021207 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for async generator objects and their creations. Async generator are turned into normal functions that create generator objects, whose implementation lives here. The creation itself also lives here. """ from .Checkers import checkStatementsSequenceOrNone from .ExpressionBases import ExpressionChildrenHavingBase from .FunctionNodes import ExpressionFunctionEntryPointBase from .IndicatorMixins import MarkLocalsDictIndicatorMixin class ExpressionMakeAsyncgenObject(ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_ASYNCGEN_OBJECT" named_children = ( "asyncgen_ref", ) getAsyncgenRef = ExpressionChildrenHavingBase.childGetter("asyncgen_ref") def __init__(self, asyncgen_ref, code_object, source_ref): assert asyncgen_ref.getFunctionBody().isExpressionAsyncgenObjectBody() ExpressionChildrenHavingBase.__init__( self, values = { "asyncgen_ref" : asyncgen_ref, }, source_ref = source_ref ) self.code_object = code_object self.variable_closure_traces = [] def getDetails(self): return { "code_object" : self.code_object } def getCodeObject(self): return self.code_object def getDetailsForDisplay(self): return { "asyncgen" : self.getAsyncgenRef().getFunctionBody().getCodeName() } def computeExpression(self, trace_collection): self.variable_closure_traces = [] for closure_variable in self.getAsyncgenRef().getFunctionBody().getClosureVariables(): trace = trace_collection.getVariableCurrentTrace(closure_variable) trace.addClosureUsage() self.variable_closure_traces.append(trace) # TODO: Asyncgen body may know something too. return self, None, None def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False def getClosureVariableVersions(self): return [ (trace.getVariable(), trace.getVersion()) for trace in self.variable_closure_traces ] class ExpressionAsyncgenObjectBody(MarkLocalsDictIndicatorMixin, ExpressionFunctionEntryPointBase): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, pylint: disable=R0901 kind = "EXPRESSION_ASYNCGEN_OBJECT_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } qualname_setup = None def __init__(self, provider, name, flags, source_ref): ExpressionFunctionEntryPointBase.__init__( self, provider = provider, name = name, code_prefix = "asyncgen", flags = flags, source_ref = source_ref ) MarkLocalsDictIndicatorMixin.__init__(self) self.needs_generator_return_exit = False def getFunctionName(self): return self.name def markAsNeedsGeneratorReturnHandling(self, value): self.needs_generator_return_exit = max( self.needs_generator_return_exit, value ) def needsGeneratorReturnHandling(self): return self.needs_generator_return_exit == 2 def needsGeneratorReturnExit(self): return bool(self.needs_generator_return_exit) @staticmethod def needsCreation(): return False @staticmethod def isUnoptimized(): return False Nuitka-0.5.28.2/nuitka/nodes/ComparisonNodes.py0000644000372000001440000002372113134660221021546 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for comparisons. """ from nuitka import PythonOperators from .ExpressionBases import ExpressionChildrenHavingBase from .NodeMakingHelpers import ( makeComparisonNode, makeConstantReplacementNode, wrapExpressionWithSideEffects ) class ExpressionComparisonBase(ExpressionChildrenHavingBase): named_children = ( "left", "right" ) def __init__(self, left, right, comparator, source_ref): assert left.isExpression() assert right.isExpression() assert comparator in PythonOperators.all_comparison_functions, comparator ExpressionChildrenHavingBase.__init__( self, values = { "left" : left, "right" : right }, source_ref = source_ref ) self.comparator = comparator if comparator in ("Is", "IsNot", "In", "NotIn"): assert self.__class__ is not ExpressionComparison def getOperands(self): return ( self.getLeft(), self.getRight() ) getLeft = ExpressionChildrenHavingBase.childGetter("left") getRight = ExpressionChildrenHavingBase.childGetter("right") def getComparator(self): return self.comparator def getDetails(self): return { "comparator" : self.comparator } @staticmethod def isExpressionComparison(): return True def getSimulator(self): return PythonOperators.all_comparison_functions[self.comparator] def computeExpression(self, trace_collection): left = self.getLeft() right = self.getRight() if left.isCompileTimeConstant() and right.isCompileTimeConstant(): left_value = left.getCompileTimeConstant() right_value = right.getCompileTimeConstant() return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( left_value, right_value ), description = "Comparison of constant arguments." ) # The value of these nodes escaped and could change its contents. trace_collection.removeKnowledge(left) trace_collection.removeKnowledge(right) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def computeExpressionOperationNot(self, not_node, trace_collection): if self.comparator in PythonOperators.comparison_inversions: left, right = self.getOperands() result = makeComparisonNode( left = left, right = right, comparator = PythonOperators.comparison_inversions[ self.comparator ], source_ref = self.source_ref ) return result, "new_expression", """\ Replaced negated comparison with inverse comparison.""" return not_node, None, None class ExpressionComparison(ExpressionComparisonBase): kind = "EXPRESSION_COMPARISON" def __init__(self, left, right, comparator, source_ref): ExpressionComparisonBase.__init__( self, left = left, right = right, comparator = comparator, source_ref = source_ref ) class ExpressionComparisonIsIsNotBase(ExpressionComparisonBase): def __init__(self, left, right, comparator, source_ref): ExpressionComparisonBase.__init__( self, left = left, right = right, comparator = comparator, source_ref = source_ref ) assert comparator in ("Is", "IsNot") self.match_value = comparator == "Is" def getDetails(self): return {} def mayRaiseException(self, exception_type): return self.getLeft().mayRaiseException(exception_type) or \ self.getRight().mayRaiseException(exception_type) def mayRaiseExceptionBool(self, exception_type): return False def computeExpression(self, trace_collection): left, right = self.getOperands() if trace_collection.mustAlias(left, right): result = makeConstantReplacementNode( constant = self.match_value, node = self ) if left.mayHaveSideEffects() or right.mayHaveSideEffects(): result = wrapExpressionWithSideEffects( side_effects = self.extractSideEffects(), old_node = self, new_node = result ) return result, "new_constant", """\ Determined values to alias and therefore result of %s comparison.""" % ( self.comparator ) if trace_collection.mustNotAlias(left, right): result = makeConstantReplacementNode( constant = not self.match_value, node = self ) if left.mayHaveSideEffects() or right.mayHaveSideEffects(): result = wrapExpressionWithSideEffects( side_effects = self.extractSideEffects(), old_node = self, new_node = result ) return result, "new_constant", """\ Determined values to not alias and therefore result of '%s' comparison.""" % ( self.comparator ) return ExpressionComparisonBase.computeExpression( self, trace_collection = trace_collection ) def extractSideEffects(self): left, right = self.getOperands() return left.extractSideEffects() + right.extractSideEffects() def computeExpressionDrop(self, statement, trace_collection): from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions result = makeStatementOnlyNodesFromExpressions( expressions = self.getOperands() ) return result, "new_statements", """\ Removed %s comparison for unused result.""" % self.comparator class ExpressionComparisonIs(ExpressionComparisonIsIsNotBase): kind = "EXPRESSION_COMPARISON_IS" def __init__(self, left, right, source_ref): ExpressionComparisonIsIsNotBase.__init__( self, left = left, right = right, comparator = "Is", source_ref = source_ref ) class ExpressionComparisonIsNOT(ExpressionComparisonIsIsNotBase): kind = "EXPRESSION_COMPARISON_IS_NOT" def __init__(self, left, right, source_ref): ExpressionComparisonIsIsNotBase.__init__( self, left = left, right = right, comparator = "IsNot", source_ref = source_ref ) class ExpressionComparisonExceptionMatch(ExpressionComparisonBase): kind = "EXPRESSION_COMPARISON_EXCEPTION_MATCH" def __init__(self, left, right, source_ref): ExpressionComparisonBase.__init__( self, left = left, right = right, comparator = "exception_match", source_ref = source_ref ) def getDetails(self): return {} def getSimulator(self): assert False return PythonOperators.all_comparison_functions[self.comparator] class ExpressionComparisonInNotInBase(ExpressionComparisonBase): def __init__(self, left, right, comparator, source_ref): ExpressionComparisonBase.__init__( self, left = left, right = right, comparator = comparator, source_ref = source_ref ) assert comparator in ("In", "NotIn") def getDetails(self): return {} def mayRaiseException(self, exception_type): left = self.getLeft() if left.mayRaiseException(exception_type): return True right = self.getRight() if right.mayRaiseException(exception_type): return True return right.mayRaiseExceptionIn(exception_type, left) def mayRaiseExceptionBool(self, exception_type): return False def computeExpression(self, trace_collection): return self.getRight().computeExpressionComparisonIn( in_node = self, value_node = self.getLeft(), trace_collection = trace_collection ) class ExpressionComparisonIn(ExpressionComparisonInNotInBase): kind = "EXPRESSION_COMPARISON_IN" def __init__(self, left, right, source_ref): ExpressionComparisonInNotInBase.__init__( self, left = left, right = right, comparator = "In", source_ref = source_ref ) class ExpressionComparisonNOTIn(ExpressionComparisonInNotInBase): kind = "EXPRESSION_COMPARISON_NOT_IN" def __init__(self, left, right, source_ref): ExpressionComparisonInNotInBase.__init__( self, left = left, right = right, comparator = "NotIn", source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/nodes/ImportNodes.py0000644000372000001440000004335013207537706020722 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes related to importing modules or names. Normally imports are mostly relatively static, but Nuitka also attempts to cover the uses of "__import__" built-in and other import techniques, that allow dynamic values. If other optimizations make it possible to predict these, the compiler can go deeper that what it normally could. The import expression node can recurse. """ import os from logging import warning from nuitka.__past__ import ( # pylint: disable=I0021,redefined-builtin long, unicode ) from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.importing.Importing import ( findModule, getModuleNameAndKindFromFilename ) from nuitka.importing.Recursion import decideRecursion, recurseTo from nuitka.importing.Whitelisting import getModuleWhiteList from nuitka.utils.FileOperations import relpath from .ExpressionBases import ExpressionBase, ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase from .shapes.BuiltinTypeShapes import ShapeTypeBuiltinModule, ShapeTypeModule class ExpressionImportModuleHard(ExpressionBase): """ Hard coded import names, e.g. of "__future__" These are directly created for some Python mechanics, but also due to compile time optimization for imports of statically known modules. """ kind = "EXPRESSION_IMPORT_MODULE_HARD" __slots__ = "module_name", def __init__(self, module_name, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) self.module_name = module_name def getDetails(self): return { "module_name" : self.module_name } def getModuleName(self): return self.module_name def computeExpressionRaw(self, trace_collection): return self, None, None def mayHaveSideEffects(self): if self.module_name == "sys": return False elif self.module_name == "__future__": return False else: return True def mayRaiseException(self, exception_type): return self.mayHaveSideEffects() class ExpressionImportModuleNameHard(ExpressionBase): """ Hard coded import names, e.g. of "os.path.dirname" These are directly created for some Python mechanics. """ kind = "EXPRESSION_IMPORT_MODULE_NAME_HARD" __slots__ = "module_name", "import_name" def __init__(self, module_name, import_name, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) self.module_name = module_name self.import_name = import_name def getDetails(self): return { "module_name" : self.module_name, "import_name" : self.import_name } def getModuleName(self): return self.module_name def getImportName(self): return self.import_name def computeExpressionRaw(self, trace_collection): # TODO: May return a module reference of some sort in the future with # embedded modules. return self, None, None def mayHaveSideEffects(self): if self.module_name == "sys" and self.import_name == "stdout": return False elif self.module_name == "__future__": return False else: return True def mayRaiseException(self, exception_type): return self.mayHaveSideEffects() class ExpressionBuiltinImport(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_IMPORT" named_children = ( "name", "globals", "locals", "fromlist", "level" ) _warned_about = set() @calledWithBuiltinArgumentNamesDecorator def __init__(self, name, globals_arg, locals_arg, fromlist, level, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "name" : name, "globals" : globals_arg, "locals" : locals_arg, "fromlist" : fromlist, "level" : level }, source_ref = source_ref ) self.recurse_attempted = False # The module actually referenced in that import. self.imported_module = None # The fromlist imported modules if any. self.import_list_modules = [] # For "package.sub_package.module" we also need to import the package, # because the imported_module not be found, as it's not a module, e.g. # in the case of "os.path" or "six.moves". self.package_modules = None self.finding = None self.type_shape = ShapeTypeModule self.builtin_module = None getImportName = ExpressionChildrenHavingBase.childGetter("name") getFromList = ExpressionChildrenHavingBase.childGetter("fromlist") getGlobals = ExpressionChildrenHavingBase.childGetter("globals") getLocals = ExpressionChildrenHavingBase.childGetter("locals") getLevel = ExpressionChildrenHavingBase.childGetter("level") def _consider(self, trace_collection, module_filename, module_package): assert module_package is None or \ (type(module_package) is str and module_package != ""), repr(module_package) module_filename = os.path.normpath(module_filename) module_name, module_kind = getModuleNameAndKindFromFilename(module_filename) if module_kind is not None: decision, reason = decideRecursion( module_filename = module_filename, module_name = module_name, module_package = module_package, module_kind = module_kind ) if decision: module_relpath = relpath(module_filename) imported_module, added_flag = recurseTo( module_package = module_package, module_filename = module_filename, module_relpath = module_relpath, module_kind = module_kind, reason = reason ) if added_flag: trace_collection.signalChange( "new_code", imported_module.getSourceReference(), "Recursed to module." ) return imported_module elif decision is None and module_kind == "py": if module_package is None: module_fullpath = module_name else: module_fullpath = module_package + '.' + module_name if module_filename not in self._warned_about and \ module_fullpath not in getModuleWhiteList(): self._warned_about.add(module_filename) warning( """\ Not recursing to '%(full_path)s' (%(filename)s), please specify \ --recurse-none (do not warn), \ --recurse-all (recurse to all), \ --recurse-not-to=%(full_path)s (ignore it), \ --recurse-to=%(full_path)s (recurse to it) to change.""" % { "full_path" : module_fullpath, "filename" : module_filename } ) def _attemptRecursion(self, trace_collection, module_name): # Complex stuff, pylint: disable=too-many-branches parent_module = self.getParentModule() if parent_module.isCompiledPythonPackage(): parent_package = parent_module.getFullName() else: parent_package = self.getParentModule().getPackage() level = self.getLevel() if level is None: level = 0 if parent_module.getFutureSpec().isAbsoluteImport() else -1 elif not level.isCompileTimeConstant(): return else: level = level.getCompileTimeConstant() # TODO: Catch this as a static error maybe. if type(level) not in (int, long): return module_package, module_filename, self.finding = findModule( importing = self, module_name = module_name, parent_package = parent_package, level = level, warn = True ) if module_filename is not None: self.imported_module = self._consider( trace_collection = trace_collection, module_filename = module_filename, module_package = module_package ) if self.imported_module is not None: import_list = self.getFromList() if import_list is not None: if import_list.isCompileTimeConstant(): import_list = import_list.getCompileTimeConstant() if type(import_list) not in (tuple, list): import_list = None if import_list and \ self.imported_module.isCompiledPythonPackage(): for import_item in import_list: if import_item == '*': continue module_package, module_filename, _finding = findModule( importing = self, module_name = import_item, parent_package = self.imported_module.getFullName(), level = -1, # Relative import, so child is used. warn = False ) if module_filename is not None: sub_imported_module = self._consider( trace_collection = trace_collection, module_filename = module_filename, module_package = module_package ) if sub_imported_module is not None: self.import_list_modules.append( sub_imported_module.getFullName() ) else: while "." in module_name: module_name = ".".join(module_name.split(".")[:-1]) module_package, module_filename, _finding = findModule( importing = self, module_name = module_name, parent_package = parent_package, level = level, warn = True ) if module_filename is not None: package_module = self._consider( trace_collection = trace_collection, module_filename = module_filename, module_package = module_package ) if package_module is not None: if self.package_modules is None: self.package_modules = [] self.package_modules.append(package_module) def _addUsedModules(self, trace_collection): if self.finding != "not-found": if self.imported_module is not None: trace_collection.onUsedModule(self.imported_module.getFullName()) for import_list_module in self.import_list_modules: trace_collection.onUsedModule(import_list_module) # These are added in any case. if self.package_modules is not None: for package_module in self.package_modules: trace_collection.onUsedModule(package_module.getFullName()) def computeExpression(self, trace_collection): # TODO: In fact, if the module is not a package, we don't have to insist # on the "fromlist" that much, but normally it's not used for anything # but packages, so it will be rare. self._addUsedModules(trace_collection) # Attempt to recurse if not already done. if self.recurse_attempted: if self.finding == "not-found": # Importing and not finding, may raise an exception obviously. trace_collection.onExceptionRaiseExit(BaseException) else: # If we know it exists, only RuntimeError shall occur. trace_collection.onExceptionRaiseExit(RuntimeError) # We stay here. return self, None, None module_name = self.getImportName() if module_name.isCompileTimeConstant(): imported_module_name = module_name.getCompileTimeConstant() if type(imported_module_name) in (str, unicode): # TODO: This is not handling decoding errors all that well. if str is not unicode and type(imported_module_name) is unicode: imported_module_name = str(imported_module_name) self._attemptRecursion( trace_collection = trace_collection, module_name = imported_module_name ) self.recurse_attempted = True if self.finding == "built-in": self.type_shape = ShapeTypeBuiltinModule self.builtin_module = __import__(imported_module_name) self._addUsedModules(trace_collection) else: # TODO: This doesn't preserve side effects. # Non-strings is going to raise an error. new_node, change_tags, message = trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : __import__(module_name.getConstant()), description = "Replaced '__import__' call with non-string module name argument." ) # Must fail, must not go on when it doesn't. assert change_tags == "new_raise", module_name return new_node, change_tags, message # Importing may raise an exception obviously, unless we know it will # not. if self.finding != "built-in": trace_collection.onExceptionRaiseExit(BaseException) # TODO: May return a module or module variable reference of some sort in # the future with embedded modules. return self, None, None # TODO: Add computeExpressionImportName def mayRaiseException(self, exception_type): return self.finding != "built-in" def mayRaiseExceptionImportName(self, exception_type, import_name): if self.finding == "built-in": return not hasattr(self.builtin_module, import_name) else: return True def getTypeShape(self): return self.type_shape class StatementImportStar(StatementChildrenHavingBase): kind = "STATEMENT_IMPORT_STAR" named_children = ("module",) def __init__(self, module_import, source_ref): StatementChildrenHavingBase.__init__( self, values = { "module" : module_import }, source_ref = source_ref ) getSourceModule = StatementChildrenHavingBase.childGetter("module") def computeStatement(self, trace_collection): trace_collection.onExpression(self.getSourceModule()) # Need to invalidate everything, and everything could be assigned to # something else now. trace_collection.removeAllKnowledge() # We could always encounter that __all__ is a strange beast and causes # the exception. trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): # Not done. TODO: Later we can try and check for "__all__" if it # really can be that way. return True class ExpressionImportName(ExpressionChildrenHavingBase): kind = "EXPRESSION_IMPORT_NAME" named_children = ( "module", ) def __init__(self, module, import_name, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "module" : module }, source_ref = source_ref ) self.import_name = import_name assert module is not None def getImportName(self): return self.import_name def getDetails(self): return { "import_name" : self.getImportName() } def getDetail(self): return "import %s from %s" % ( self.getImportName(), self.getModule().getModuleName() ) getModule = ExpressionChildrenHavingBase.childGetter("module") def computeExpression(self, trace_collection): return self.getModule().computeExpressionImportName( import_node = self, import_name = self.import_name, trace_collection = trace_collection ) def mayRaiseException(self, exception_type): return self.getModule().mayRaiseExceptionImportName( exception_type = exception_type, import_name = self.import_name ) Nuitka-0.5.28.2/nuitka/nodes/TypeNodes.py0000644000372000001440000001235513134660221020356 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ The type1 node. This one just determines types. It's great for optimization. We may be able to predict its value, but knowing it. In that case, we have a built-in name reference for that type to convert to, or when checking the result of it, we will then know it's limited after the fact. """ from nuitka.Builtins import builtin_names from .BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, makeExpressionBuiltinRef ) from .ExpressionBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase ) class ExpressionBuiltinType1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_TYPE1" def computeExpression(self, trace_collection): value = self.getValue() if value.isCompileTimeConstant(): value = value.getCompileTimeConstant() type_name = value.__class__.__name__ if type_name in builtin_names: new_node = makeExpressionBuiltinRef( builtin_name = type_name, source_ref = self.getSourceReference() ) else: new_node = ExpressionBuiltinAnonymousRef( builtin_name = type_name, source_ref = self.getSourceReference() ) return ( new_node, "new_builtin", "Replaced predictable type lookup with builtin type '%s'." % ( type_name ) ) return self, None, None def computeExpressionDrop(self, statement, trace_collection): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = self.getValue(), node = statement ) return result, "new_statements", """\ Removed type taking for unused result.""" def mayRaiseException(self, exception_type): return self.getValue().mayRaiseException(exception_type) def mayHaveSideEffects(self): return self.getValue().mayHaveSideEffects() class ExpressionBuiltinSuper(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_SUPER" named_children = ( "type", "object" ) def __init__(self, super_type, super_object, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "type" : super_type, "object" : super_object }, source_ref = source_ref ) getType = ExpressionChildrenHavingBase.childGetter("type") getObject = ExpressionChildrenHavingBase.childGetter("object") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) # TODO: Quite some cases should be possible to predict. return self, None, None class ExpressionBuiltinIsinstance(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_ISINSTANCE" named_children = ( "instance", "classes" ) def __init__(self, instance, classes, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "instance" : instance, "classes" : classes }, source_ref = source_ref ) getInstance = ExpressionChildrenHavingBase.childGetter("instance") getCls = ExpressionChildrenHavingBase.childGetter("classes") def computeExpression(self, trace_collection): # TODO: Quite some cases should be possible to predict. instance = self.getInstance() # TODO: Should be possible to query run time type instead, but we don't # have that method yet. Later this will be essential. if not instance.isCompileTimeConstant(): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None cls = self.getCls() if not cls.isCompileTimeConstant(): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None # So if both are compile time constant, we are able to compute it. return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : isinstance( instance.getCompileTimeConstant(), cls.getCompileTimeConstant() ), description = "Built-in call to 'isinstance' computed." ) Nuitka-0.5.28.2/nuitka/nodes/ExpressionBases.py0000644000372000001440000007542313207537242021575 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Expression base classes. These classes provide the generic base classes available for expressions. They have a richer interface, mostly related to abstract execution, and different from statements. """ from abc import abstractmethod from nuitka.Constants import isCompileTimeConstantValue from .NodeBases import ChildrenHavingMixin, NodeBase from .NodeMakingHelpers import ( getComputationResult, makeConstantReplacementNode, makeRaiseTypeErrorExceptionReplacementFromTemplateAndValue, wrapExpressionWithNodeSideEffects, wrapExpressionWithSideEffects ) from .shapes.BuiltinTypeShapes import ( ShapeTypeDict, ShapeTypeStr, ShapeTypeUnicode ) from .shapes.StandardShapes import ShapeUnknown class ExpressionBase(NodeBase): # TODO: Maybe we can do this only for debug mode. __slots__ = ("code_generated",) def getTypeShape(self): # Virtual method, pylint: disable=no-self-use return ShapeUnknown def getValueShape(self): return self def isCompileTimeConstant(self): """ Has a value that we can use at compile time. Yes or no. If it has such a value, simulations can be applied at compile time and e.g. operations or conditions, or even calls may be executed against it. """ # Virtual method, pylint: disable=no-self-use return False def getCompileTimeConstant(self): assert self.isCompileTimeConstant(), self assert False def getTruthValue(self): """ Return known truth value. The "None" value indicates unknown. """ if self.isCompileTimeConstant(): return bool(self.getCompileTimeConstant()) else: return None def mayBeNone(self): """ Could this evaluate to be "None". Yes or no. Defaults to pessimistic yes.""" # For overload, pylint: disable=no-self-use return True def isKnownToBeIterable(self, count): """ Can be iterated at all (count is None) or exactly count times. Yes or no. If it can be iterated a known number of times, it may be asked to unpack itself. """ # Virtual method, pylint: disable=no-self-use,unused-argument return False def isKnownToBeIterableAtMin(self, count): # Virtual method, pylint: disable=no-self-use,unused-argument return False def isKnownToBeIterableAtMax(self, count): # Virtual method, pylint: disable=no-self-use,unused-argument return False def getIterationLength(self): """ Value that "len" or "PyObject_Size" would give, if known. Otherwise it is "None" to indicate unknown. """ # Virtual method, pylint: disable=no-self-use return None def getIterationMinLength(self): """ Value that "len" or "PyObject_Size" would give at minimum, if known. Otherwise it is "None" to indicate unknown. """ return self.getIterationLength() def getIterationMaxLength(self): """ Value that "len" or "PyObject_Size" would give at maximum, if known. Otherwise it is "None" to indicate unknown. """ return self.getIterationLength() def getStringValue(self): """ Node as string value, if possible.""" # Virtual method, pylint: disable=no-self-use return None def getStrValue(self): """ Value that "str" or "PyObject_Str" would give, if known. Otherwise it is "None" to indicate unknown. Users must not forget to take side effects into account, when replacing a node with its string value. """ string_value = self.getStringValue() if string_value is not None: return makeConstantReplacementNode( node = self, constant = string_value ) return None def getTypeValue(self): """ Type of the node. """ from .TypeNodes import ExpressionBuiltinType1 return ExpressionBuiltinType1( value = self.makeClone(), source_ref = self.getSourceReference() ) def isKnownToBeHashable(self): """ Is the value hashable, i.e. suitable for dictionary/set keying.""" # Virtual method, pylint: disable=no-self-use # Unknown by default. return None def extractUnhashableNode(self): # Virtual method, pylint: disable=no-self-use # Not available by default. return None def onRelease(self, trace_collection): # print "onRelease", self pass def isKnownToHaveAttribute(self, attribute_name): # Virtual method, pylint: disable=no-self-use,unused-argument return None @abstractmethod def computeExpressionRaw(self, trace_collection): """ Replace this node with computation result. """ def computeExpressionAttribute(self, lookup_node, attribute_name, trace_collection): # By default, an attribute lookup may change everything about the lookup # source. trace_collection.removeKnowledge(self) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) if not self.isKnownToHaveAttribute(attribute_name): trace_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionAttributeSpecial(self, lookup_node, attribute_name, trace_collection): # By default, an attribute lookup may change everything about the lookup # source. Virtual method, pylint: disable=unused-argument trace_collection.removeKnowledge(lookup_node) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) trace_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionImportName(self, import_node, import_name, trace_collection): if self.mayRaiseExceptionImportName(BaseException, import_name): trace_collection.onExceptionRaiseExit(BaseException) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return import_node, None, None def computeExpressionSetAttribute(self, set_node, attribute_name, value_node, trace_collection): # By default, an attribute lookup may change everything about the lookup # source. Virtual method, pylint: disable=unused-argument trace_collection.removeKnowledge(self) trace_collection.removeKnowledge(value_node) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) trace_collection.onExceptionRaiseExit(BaseException) # Better mechanics? return set_node, None, None def computeExpressionDelAttribute(self, set_node, attribute_name, trace_collection): # By default, an attribute lookup may change everything about the lookup # source. Virtual method, pylint: disable=unused-argument trace_collection.removeKnowledge(self) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) trace_collection.onExceptionRaiseExit(BaseException) # Better mechanics? return set_node, None, None def computeExpressionSubscript(self, lookup_node, subscript, trace_collection): # By default, an subscript can execute any code and change all values # that escaped. This is a virtual method that may consider the subscript # but generally we don't know what to do. pylint: disable=unused-argument trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSetSubscript(self, set_node, subscript, value_node, trace_collection): # By default, an subscript can execute any code and change all values # that escaped. This is a virtual method that may consider the subscript # but generally we don't know what to do. pylint: disable=unused-argument trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionDelSubscript(self, del_node, subscript, trace_collection): # By default, an subscript can execute any code and change all values # that escaped. This is a virtual method that may consider the subscript # but generally we don't know what to do. pylint: disable=unused-argument trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return del_node, None, None def computeExpressionSlice(self, lookup_node, lower, upper, trace_collection): # By default, a slicing may change everything about the lookup source. trace_collection.removeKnowledge(self) trace_collection.removeKnowledge(lower) trace_collection.removeKnowledge(upper) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSetSlice(self, set_node, lower, upper, value_node, trace_collection): # By default, an subscript may change everything about the lookup # source. trace_collection.removeKnowledge(self) trace_collection.removeKnowledge(lower) trace_collection.removeKnowledge(upper) trace_collection.removeKnowledge(value_node) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionDelSlice(self, set_node, lower, upper, trace_collection): # By default, an subscript may change everything about the lookup # source. trace_collection.removeKnowledge(self) trace_collection.removeKnowledge(lower) trace_collection.removeKnowledge(upper) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return set_node, None, None def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): # The called and the arguments escape for good. self.onContentEscapes(trace_collection) if call_args is not None: call_args.onContentEscapes(trace_collection) if call_kw is not None: call_kw.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return call_node, None, None def computeExpressionLen(self, len_node, trace_collection): shape = self.getValueShape() has_len = shape.hasShapeSlotLen() if has_len is False: return makeRaiseTypeErrorExceptionReplacementFromTemplateAndValue( template = "object of type '%s' has no len()", operation = "len", original_node = len_node, value_node = self ) elif has_len is True: iter_length = self.getIterationLength() if iter_length is not None: from .ConstantRefNodes import makeConstantRefNode result = makeConstantRefNode( constant = int(iter_length), # make sure to downcast long source_ref = len_node.getSourceReference() ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = self ) return result, "new_constant", "Predicted 'len' result from value shape." self.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return len_node, None, None def computeExpressionIter1(self, iter_node, trace_collection): shape = self.getTypeShape() assert shape is not None, self if shape.hasShapeSlotIter() is False: return makeRaiseTypeErrorExceptionReplacementFromTemplateAndValue( template = "'%s' object is not iterable", operation = "iter", original_node = iter_node, value_node = self ) self.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return iter_node, None, None def computeExpressionNext1(self, next_node, trace_collection): self.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return next_node, None, None def computeExpressionAsyncIter(self, iter_node, trace_collection): self.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return iter_node, None, None def computeExpressionOperationNot(self, not_node, trace_collection): # Virtual method, pylint: disable=no-self-use # The value of that node escapes and could change its contents. trace_collection.removeKnowledge(not_node) # Any code could be run, note that. trace_collection.onControlFlowEscape(not_node) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return not_node, None, None def computeExpressionComparisonIn(self, in_node, value_node, trace_collection): # Virtual method, pylint: disable=unused-argument shape = self.getTypeShape() assert shape is not None, self if shape.hasShapeSlotContains() is False: return makeRaiseTypeErrorExceptionReplacementFromTemplateAndValue( template = "argument of type '%s' object is not iterable", operation = "in", original_node = in_node, value_node = self ) # Any code could be run, note that. trace_collection.onControlFlowEscape(in_node) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return in_node, None, None def computeExpressionDrop(self, statement, trace_collection): if not self.mayHaveSideEffects(): return None, "new_statements", lambda : "Removed %s without effect." % self.getDescription() return statement, None, None def onContentEscapes(self, trace_collection): pass def mayRaiseExceptionBool(self, exception_type): """ Unless we are told otherwise, everything may raise being checked. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionIn(self, exception_type, checked_value): """ Unless we are told otherwise, everything may raise being iterated. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionAttributeLookup(self, exception_type, attribute_name): """ Unless we are told otherwise, everything may raise for attribute access. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionAttributeLookupSpecial(self, exception_type, attribute_name): """ Unless we are told otherwise, everything may raise for attribute access. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionAttributeLookupObject(self, exception_type, attribute): """ Unless we are told otherwise, everything may raise for attribute access. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionAttributeCheck(self, exception_type, attribute_name): """ Unless we are told otherwise, everything may raise for attribute check. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionAttributeCheckObject(self, exception_type, attribute): """ Unless we are told otherwise, everything may raise for attribute check. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayRaiseExceptionImportName(self, exception_type, import_name): """ Unless we are told otherwise, everything may raise for name import. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayHaveSideEffectsBool(self): """ Unless we are told otherwise, everything may have a side effect for bool check. """ # Virtual method, pylint: disable=no-self-use return True def hasShapeSlotLen(self): """ The type shape tells us, if "len" is available. """ return self.getTypeShape().hasShapeSlotLen() def hasShapeSlotIter(self): """ The type shape tells us, if "iter" is available. """ return self.getTypeShape().hasShapeSlotIter() def hasShapeSlotNext(self): """ The type shape tells us, if "next" is available. """ return self.getTypeShape().hasShapeSlotNext() # TODO: Maybe this is a shape slot thing. def isIndexable(self): """ Unless we are told otherwise, it's not indexable. """ # Virtual method, pylint: disable=no-self-use return False # TODO: The ought to be a type shape check for that too. def getIntegerValue(self): """ Node as integer value, if possible.""" # Virtual method, pylint: disable=no-self-use return None def getIntValue(self): """ Value that "int" or "PyNumber_Int" (sp) would give, if known. Otherwise it is "None" to indicate unknown. Users must not forget to take side effects into account, when replacing a node with its string value. """ # Virtual method, pylint: disable=no-self-use return None def hasShapeDictionaryExact(self): """ Does a node have exactly a dictionary shape. """ return self.getTypeShape() is ShapeTypeDict def hasShapeStrExact(self): """ Does an expression have exactly a string shape. """ return self.getTypeShape() is ShapeTypeStr def hasShapeUnicodeExact(self): """ Does an expression have exactly a unicode shape. """ return self.getTypeShape() is ShapeTypeUnicode class CompileTimeConstantExpressionBase(ExpressionBase): # Base classes can be abstract, pylint: disable=abstract-method # TODO: Do this for all computations, do this in the base class of all # nodes. __slots__ = ("computed_attribute",) def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) self.computed_attribute = None def isCompileTimeConstant(self): """ Has a value that we can use at compile time. Yes or no. If it has such a value, simulations can be applied at compile time and e.g. operations or conditions, or even calls may be executed against it. """ return True def isMutable(self): # Virtual method, pylint: disable=no-self-use return False def mayHaveSideEffects(self): # Virtual method overload return False def mayHaveSideEffectsBool(self): # Virtual method overload return False def mayRaiseException(self, exception_type): # Virtual method overload return False def mayRaiseExceptionBool(self, exception_type): # Virtual method overload return False def mayRaiseExceptionAttributeLookup(self, exception_type, attribute_name): # Virtual method overload # We remember it from our computation. return not self.computed_attribute def mayRaiseExceptionAttributeLookupSpecial(self, exception_type, attribute_name): # Virtual method overload # We remember it from our computation. return not self.computed_attribute def mayRaiseExceptionAttributeCheck(self, exception_type, attribute_name): # Virtual method overload # Checking attributes of compile time constants never raises. return False def mayBeNone(self): # Only the "None" compile time constant is going to allow that, and # it overload this again. return False def computeExpressionOperationNot(self, not_node, trace_collection): return trace_collection.getCompileTimeComputationResult( node = not_node, computation = lambda : not self.getCompileTimeConstant(), description = """\ Compile time constant negation truth value pre-computed.""" ) def computeExpressionLen(self, len_node, trace_collection): return trace_collection.getCompileTimeComputationResult( node = len_node, computation = lambda : len(self.getCompileTimeConstant()), description = """\ Compile time constant len value pre-computed.""" ) def isKnownToHaveAttribute(self, attribute_name): if self.computed_attribute is None: self.computed_attribute = hasattr(self.getCompileTimeConstant(), attribute_name) return self.computed_attribute def computeExpressionAttribute(self, lookup_node, attribute_name, trace_collection): value = self.getCompileTimeConstant() if self.computed_attribute is None: self.computed_attribute = hasattr(value, attribute_name) # If it raises, or the attribute itself is a compile time constant, # then do execute it. if not self.computed_attribute or \ isCompileTimeConstantValue(getattr(value, attribute_name)): return trace_collection.getCompileTimeComputationResult( node = lookup_node, computation = lambda : getattr(value, attribute_name), description = "Attribute lookup to '%s' pre-computed." % ( attribute_name ) ) return lookup_node, None, None def computeExpressionSubscript(self, lookup_node, subscript, trace_collection): if subscript.isCompileTimeConstant(): return trace_collection.getCompileTimeComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ subscript.getCompileTimeConstant() ], description = "Subscript of constant with constant value." ) # TODO: Look-up of subscript to index may happen. trace_collection.onExceptionRaiseExit(BaseException) return lookup_node, None, None def computeExpressionSlice(self, lookup_node, lower, upper, trace_collection): # TODO: Could be happy with predictable index values and not require # constants. if lower is not None: if upper is not None: if lower.isCompileTimeConstant() and upper.isCompileTimeConstant(): return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ lower.getCompileTimeConstant() : upper.getCompileTimeConstant() ], description = """\ Slicing of constant with constant indexes.""" ) else: if lower.isCompileTimeConstant(): return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ lower.getCompileTimeConstant() : ], description = """\ Slicing of constant with constant lower index only.""" ) else: if upper is not None: if upper.isCompileTimeConstant(): return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ : upper.getCompileTimeConstant() ], description = """\ Slicing of constant with constant upper index only.""" ) else: return getComputationResult( node = lookup_node, computation = lambda : self.getCompileTimeConstant()[ : ], description = "Slicing of constant with no indexes." ) return lookup_node, None, None def computeExpressionComparisonIn(self, in_node, value_node, trace_collection): if value_node.isCompileTimeConstant(): return getComputationResult( node = in_node, computation = lambda : in_node.getSimulator()( value_node.getCompileTimeConstant(), self.getCompileTimeConstant() ), description = """\ Predicted '%s' on compiled time constant values.""" % in_node.comparator ) # Look-up of __contains__ on compile time constants does mostly nothing. trace_collection.onExceptionRaiseExit(BaseException) return in_node, None, None class ExpressionChildrenHavingBase(ChildrenHavingMixin, ExpressionBase): def __init__(self, values, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) ChildrenHavingMixin.__init__( self, values = values ) def computeExpressionRaw(self, trace_collection): """ Compute an expression. Default behavior is to just visit the child expressions first, and then the node "computeExpression". For a few cases this needs to be overloaded, e.g. conditional expressions. """ # First apply the sub-expressions, as they are evaluated before. sub_expressions = self.getVisitableNodes() for count, sub_expression in enumerate(sub_expressions): assert sub_expression.isExpression(), (self, sub_expression) expression = trace_collection.onExpression( expression = sub_expression ) if expression.willRaiseException(BaseException): wrapped_expression = wrapExpressionWithSideEffects( side_effects = sub_expressions[:count], old_node = sub_expression, new_node = expression ) return ( wrapped_expression, "new_raise", "For '%s' the expression '%s' will raise." % ( self.getChildNameNice(), expression.getChildNameNice() ) ) # Then ask ourselves to work on it. return self.computeExpression( trace_collection = trace_collection ) class ExpressionSpecBasedComputationBase(ExpressionChildrenHavingBase): builtin_spec = None def computeBuiltinSpec(self, trace_collection, given_values): assert self.builtin_spec is not None, self for value in given_values: if value is not None and not value.isCompileTimeConstant(): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None if not self.builtin_spec.isCompileTimeComputable(given_values): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_spec.simulateCall(given_values), description = "Built-in call to '%s' pre-computed." % ( self.builtin_spec.getName() ) ) class ExpressionBuiltinSingleArgBase(ExpressionSpecBasedComputationBase): named_children = ( "value", ) def __init__(self, value, source_ref): ExpressionSpecBasedComputationBase.__init__( self, values = { "value" : value, }, source_ref = source_ref ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, trace_collection): value = self.getValue() if value is None: return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = () ) else: return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = (value,) ) Nuitka-0.5.28.2/nuitka/nodes/BuiltinHashNodes.py0000644000372000001440000000365613122472300021647 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'hash' built-in. This is a specific thing, which must be calculated at run time, but we can predict things about its type, and the fact that it won't raise an exception for some types, so it is still useful. Also calls to it can be accelerated slightly. """ from .ExpressionBases import ExpressionChildrenHavingBase class ExpressionBuiltinHash(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_HASH" named_children = ( "value", ) def __init__(self, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "value" : value, }, source_ref = source_ref ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, trace_collection): value = self.getValue() # TODO: Have a computation slot for hashing. if not value.isKnownToBeHashable(): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): return not self.getValue().isKnownToBeHashable() Nuitka-0.5.28.2/nuitka/nodes/IndicatorMixins.py0000644000372000001440000000613513207537242021556 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module for node class mixins that indicate runtime determined node facts. These come into play after finalization only. All of the these attributes (and we could use properties instead) are determined once or from a default and then used like this. """ class MarkLocalsDictIndicatorMixin(object): def __init__(self): self.needs_locals_dict = False self.foreign_locals_dict = False def hasLocalsDict(self): return self.needs_locals_dict def markAsLocalsDict(self): self.needs_locals_dict = True def markAsForeignLocalsDict(self): self.foreign_locals_dict = True def hasForeignLocalsDict(self): return self.foreign_locals_dict class MarkUnoptimizedFunctionIndicatorMixin(object): """ Mixin for indication that a function contains an exec or star import. These do not access global variables directly, but check a locals dictionary first, because they do. """ def __init__(self, flags): self.unoptimized_locals = "has_exec" in flags self.unqualified_exec = "has_unqualified_exec" in flags self.exec_source_ref = None def markAsUnqualifiedExecContaining(self, source_ref): assert self.unqualified_exec # Let the first one win. if self.exec_source_ref is None: self.exec_source_ref = source_ref def isUnoptimized(self): return self.unoptimized_locals def isUnqualifiedExec(self): return self.unoptimized_locals and self.unqualified_exec def getExecSourceRef(self): return self.exec_source_ref class MarkNeedsAnnotationsMixin(object): def __init__(self): self.needs_annotations_dict = False def markAsNeedsAnnotationsDictionary(self): """ For use during building only. Indicate "__annotations__" need. """ self.needs_annotations_dict = True def needsAnnotationsDictionary(self): """ For use during building only. Indicate "__annotations__" need. """ return self.needs_annotations_dict class EntryPointMixin(object): def __init__(self): self.trace_collection = None def setTraceCollection(self, trace_collection): previous = self.trace_collection self.trace_collection = trace_collection return previous def getTraceCollection(self): return self.trace_collection Nuitka-0.5.28.2/nuitka/nodes/Checkers.py0000644000372000001440000000221513112214770020165 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node children checkers. The role of checkers is to make sure that node children have specific value types only. """ def checkStatementsSequenceOrNone(value): assert value is None or value.kind == "STATEMENTS_SEQUENCE", value return value def checkStatementsSequence(value): assert value is not None and value.kind == "STATEMENTS_SEQUENCE", value return value Nuitka-0.5.28.2/nuitka/nodes/BuiltinOpenNodes.py0000644000372000001440000000727713134660221021674 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'open' built-in. This is a rather two sided beast, as it may be read or write. And we would like to be able to track it, so we can include files into the executable, or write more efficiently. """ from nuitka.PythonVersions import python_version from .ExpressionBases import ExpressionChildrenHavingBase from .shapes.BuiltinTypeShapes import ShapeTypeFile class ExpressionBuiltinOpenMixin(object): getFilename = ExpressionChildrenHavingBase.childGetter("filename") getMode = ExpressionChildrenHavingBase.childGetter("mode") getBuffering = ExpressionChildrenHavingBase.childGetter("buffering") @staticmethod def getTypeShape(): return ShapeTypeFile def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) # Note: Quite impossible to predict without further assumptions, but we could look # at the arguments at least. return self, None, None if python_version < 300: class ExpressionBuiltinOpen(ExpressionBuiltinOpenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_OPEN" named_children = ( "filename", "mode", "buffering" ) def __init__(self, filename, mode, buffering, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "filename" : filename, "mode" : mode, "buffering" : buffering }, source_ref = source_ref ) else: class ExpressionBuiltinOpen(ExpressionBuiltinOpenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_OPEN" named_children = ( "filename", "mode", "buffering", "encoding", "errors", "newline", "closefd", "opener" ) def __init__(self, filename, mode, buffering, encoding, errors, newline, closefd, opener, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "filename" : filename, "mode" : mode, "buffering" : buffering, "encoding" : encoding, "errors" : errors, "newline" : newline, "closefd" : closefd, "opener" : opener }, source_ref = source_ref ) getEncoding = ExpressionChildrenHavingBase.childGetter("encoding") getErrors = ExpressionChildrenHavingBase.childGetter("errors") getNewline = ExpressionChildrenHavingBase.childGetter("newline") getCloseFd = ExpressionChildrenHavingBase.childGetter("closefd") getOpener = ExpressionChildrenHavingBase.childGetter("opener") Nuitka-0.5.28.2/nuitka/nodes/BuiltinRefNodes.py0000644000372000001440000001701213207537242021502 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tree nodes for built-in references. There is 2 major types of built-in references. One is the values from built-ins, the other is built-in exceptions. They work differently and mean different things, but they have similar origin, that is, access to variables only ever read. """ from nuitka.Builtins import ( builtin_anon_names, builtin_exception_names, builtin_exception_values, builtin_names, builtin_type_names ) from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ConstantRefNodes import makeConstantRefNode from .ExceptionNodes import ExpressionBuiltinMakeException from .ExpressionBases import CompileTimeConstantExpressionBase class ExpressionBuiltinRefBase(CompileTimeConstantExpressionBase): # Base classes can be abstract, pylint: disable=abstract-method __slots__ = ("builtin_name",) def __init__(self, builtin_name, source_ref): CompileTimeConstantExpressionBase.__init__( self, source_ref = source_ref ) self.builtin_name = builtin_name def getDetails(self): return { "builtin_name" : self.builtin_name } def getBuiltinName(self): return self.builtin_name def isKnownToBeHashable(self): return True def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False def getStrValue(self): return makeConstantRefNode( constant = str(self.getCompileTimeConstant()), user_provided = True, source_ref = self.getSourceReference(), ) def makeExpressionBuiltinRef(builtin_name, source_ref): assert builtin_name in builtin_names, builtin_name quick_names = { "None" : None, "True" : True, "False" : False, "__debug__" : __debug__, "Ellipsis" : Ellipsis, } if builtin_name in quick_names: return makeConstantRefNode( constant = quick_names[builtin_name], source_ref = source_ref ) elif builtin_name in builtin_type_names: return makeConstantRefNode( constant = __builtins__[builtin_name], source_ref = source_ref ) else: return ExpressionBuiltinRef( builtin_name = builtin_name, source_ref = source_ref ) class ExpressionBuiltinRef(ExpressionBuiltinRefBase): kind = "EXPRESSION_BUILTIN_REF" __slots__ = () def __init__(self, builtin_name, source_ref): ExpressionBuiltinRefBase.__init__( self, builtin_name = builtin_name, source_ref = source_ref ) def isCompileTimeConstant(self): return True def getCompileTimeConstant(self): return __builtins__[self.builtin_name] def computeExpressionRaw(self, trace_collection): return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): from nuitka.optimizations.OptimizeBuiltinCalls import computeBuiltinCall # Anything may happen. On next pass, if replaced, we might be better # but not now. trace_collection.onExceptionRaiseExit(BaseException) new_node, tags, message = computeBuiltinCall( builtin_name = self.builtin_name, call_node = call_node ) if self.builtin_name in ("dir", "eval", "exec", "execfile", "locals", "vars"): # Just inform the collection that all has escaped. trace_collection.onLocalsUsage(self.getParentVariableProvider()) return new_node, tags, message def getStringValue(self): return repr(self.getCompileTimeConstant()) def isKnownToBeIterable(self, count): # TODO: Why yes, some may be, could be told here. return None class ExpressionBuiltinAnonymousRef(ExpressionBuiltinRefBase): kind = "EXPRESSION_BUILTIN_ANONYMOUS_REF" __slots__ = () def __init__(self, builtin_name, source_ref): assert builtin_name in builtin_anon_names, builtin_name ExpressionBuiltinRefBase.__init__( self, builtin_name = builtin_name, source_ref = source_ref ) def isCompileTimeConstant(self): return True def getCompileTimeConstant(self): return builtin_anon_names[self.builtin_name] def computeExpressionRaw(self, trace_collection): return self, None, None def getStringValue(self): return repr(self.getCompileTimeConstant()) class ExpressionBuiltinExceptionRef(ExpressionBuiltinRefBase): kind = "EXPRESSION_BUILTIN_EXCEPTION_REF" __slots__ = () def __init__(self, exception_name, source_ref): assert exception_name in builtin_exception_names, exception_name ExpressionBuiltinRefBase.__init__( self, builtin_name = exception_name, source_ref = source_ref ) def getDetails(self): return { "exception_name" : self.builtin_name } getExceptionName = ExpressionBuiltinRefBase.getBuiltinName def isCompileTimeConstant(self): return True def mayRaiseException(self, exception_type): return False def getCompileTimeConstant(self): return builtin_exception_values[self.builtin_name] def computeExpressionRaw(self, trace_collection): # Not much that can be done here. return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): exception_name = self.getExceptionName() # TODO: Keyword only arguments of it, are not properly handled yet by # the built-in call code. if exception_name == "ImportError" and python_version >= 330: if call_kw is not None and \ (not call_kw.isExpressionConstantRef() or call_kw.getConstant() != {}): return call_node, None, None def createBuiltinMakeException(args, source_ref): return ExpressionBuiltinMakeException( exception_name = exception_name, args = args, source_ref = source_ref ) new_node = BuiltinOptimization.extractBuiltinArgs( node = call_node, builtin_class = createBuiltinMakeException, builtin_spec = BuiltinOptimization.makeBuiltinExceptionParameterSpec( exception_name = exception_name ) ) # TODO: Don't allow this to happen. if new_node is None: return call_node, None, None return new_node, "new_expression", "Detected built-in exception making." Nuitka-0.5.28.2/nuitka/nodes/ClassNodes.py0000644000372000001440000001370013207537242020504 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for classes and their creations. The classes are are at the core of the language and have their complexities. """ from nuitka.PythonVersions import python_version from .Checkers import checkStatementsSequenceOrNone from .ExpressionBases import ExpressionChildrenHavingBase from .IndicatorMixins import ( MarkLocalsDictIndicatorMixin, MarkNeedsAnnotationsMixin ) from .OutlineNodes import ExpressionOutlineFunction class ExpressionClassBody(MarkLocalsDictIndicatorMixin, MarkNeedsAnnotationsMixin, ExpressionOutlineFunction): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, pylint: disable=R0901 kind = "EXPRESSION_CLASS_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } def __init__(self, provider, name, doc, source_ref): ExpressionOutlineFunction.__init__( self, provider = provider, name = name, code_prefix = "class", source_ref = source_ref ) MarkNeedsAnnotationsMixin.__init__(self) MarkLocalsDictIndicatorMixin.__init__(self) self.doc = doc self.has_annotations = False def getDetail(self): return "named %s" % self.getFunctionName() def getDetails(self): return { "name" : self.getFunctionName(), "provider" : self.provider.getCodeName(), "doc" : self.doc, "flags" : self.flags } def getDetailsForDisplay(self): result = { "name" : self.getFunctionName(), "provider" : self.provider.getCodeName(), "flags" : "" if self.flags is None else ','.join(sorted(self.flags)) } if self.doc is not None: result["doc"] = self.doc return result @classmethod def fromXML(cls, provider, source_ref, **args): return cls( provider = provider, source_ref = source_ref, **args ) def getDoc(self): return self.doc @staticmethod def isEarlyClosure(): return True def getVariableForClosure(self, variable_name): # print( "getVariableForClosure", self, variable_name ) # The class bodies provide no closure, except under CPython3.x, there # they provide "__class__" but nothing else. if variable_name == "__class__": if python_version < 300: return self.provider.getVariableForClosure( "__class__" ) else: return ExpressionOutlineFunction.getVariableForClosure( self, variable_name = "__class__" ) else: result = self.provider.getVariableForClosure(variable_name) self.taken.add(result) return result def markAsDirectlyCalled(self): pass def markAsExecContaining(self): pass def markAsUnqualifiedExecContaining(self, source_ref): pass def mayHaveSideEffects(self): # The function definition has no side effects, calculating the defaults # would be, but that is done outside of this. return False def mayRaiseException(self, exception_type): return self.getBody().mayRaiseException(exception_type) def isUnoptimized(self): return True class ExpressionSelectMetaclass(ExpressionChildrenHavingBase): kind = "EXPRESSION_SELECT_METACLASS" named_children = ( "metaclass", "bases" ) def __init__(self, metaclass, bases, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "metaclass" : metaclass, "bases" : bases }, source_ref = source_ref ) def computeExpression(self, trace_collection): # TODO: Meta class selection is very computable, and should be done. return self, None, None getMetaclass = ExpressionChildrenHavingBase.childGetter("metaclass") getBases = ExpressionChildrenHavingBase.childGetter("bases") class ExpressionBuiltinType3(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_TYPE3" named_children = ("type_name", "bases", "dict") def __init__(self, type_name, bases, type_dict, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "type_name" : type_name, "bases" : bases, "dict" : type_dict }, source_ref = source_ref ) getTypeName = ExpressionChildrenHavingBase.childGetter("type_name") getBases = ExpressionChildrenHavingBase.childGetter("bases") getDict = ExpressionChildrenHavingBase.childGetter("dict") def computeExpression(self, trace_collection): # TODO: Should be compile time computable if bases and dict are. return self, None, None Nuitka-0.5.28.2/nuitka/nodes/ConstantRefNodes.py0000644000372000001440000007170313134660221021665 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for constant expressions. Can be all common built-in types. """ from logging import warning from nuitka.__past__ import ( # pylint: disable=I0021,redefined-builtin iterItems, long, unicode, xrange ) from nuitka.Constants import ( getConstantIterationLength, getUnhashableConstant, isConstant, isHashable, isIndexConstant, isIterableConstant, isMutable, isNumberConstant ) from nuitka.Options import isDebug from .ExpressionBases import CompileTimeConstantExpressionBase from .NodeMakingHelpers import ( makeRaiseExceptionReplacementExpression, wrapExpressionWithSideEffects ) from .shapes.BuiltinTypeShapes import ( ShapeTypeBool, ShapeTypeBytearray, ShapeTypeBytes, ShapeTypeDict, ShapeTypeEllipsisType, ShapeTypeFloat, ShapeTypeFrozenset, ShapeTypeInt, ShapeTypeList, ShapeTypeLong, ShapeTypeNoneType, ShapeTypeSet, ShapeTypeSlice, ShapeTypeStr, ShapeTypeTuple, ShapeTypeType, ShapeTypeUnicode, ShapeTypeXrange ) class ExpressionConstantRefBase(CompileTimeConstantExpressionBase): __slots__ = "constant", "user_provided" def __init__(self, constant, source_ref, user_provided = False): CompileTimeConstantExpressionBase.__init__( self, source_ref = source_ref ) assert isConstant(constant), repr(constant) self.constant = constant # Memory saving method, have the attribute only where necessary. self.user_provided = user_provided if not user_provided and isDebug(): try: if type(constant) in (str, unicode): max_size = 1000 elif type(constant) is xrange: max_size = None else: max_size = 256 if max_size is not None and len(constant) > max_size: warning( "Too large constant (%s %d) encountered at %s.", type(constant), len(constant), source_ref.getAsString() ) except TypeError: pass def __repr__(self): return "" % ( self.kind, self.constant, self.source_ref.getAsString(), self.user_provided ) def getDetails(self): return { "constant" : self.constant, "user_provided" : self.user_provided } def getDetailsForDisplay(self): result = self.getDetails() if "constant" in result: result["constant"] = repr(result["constant"]) return result def getDetail(self): return repr(self.constant) @staticmethod def isExpressionConstantRef(): return True def computeExpressionRaw(self, trace_collection): # Cannot compute any further, this is already the best. return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): # The arguments don't matter. All constant values cannot be called, and # we just need to make and error out of that. new_node = wrapExpressionWithSideEffects( new_node = makeRaiseExceptionReplacementExpression( expression = self, exception_type = "TypeError", exception_value = "'%s' object is not callable" % type(self.constant).__name__ ), old_node = call_node, side_effects = call_node.extractSideEffectsPreCall() ) trace_collection.onExceptionRaiseExit(TypeError) return new_node, "new_raise", "Predicted call of constant value to exception raise." def getCompileTimeConstant(self): return self.constant getConstant = getCompileTimeConstant def isMutable(self): return isMutable(self.constant) def isKnownToBeHashable(self): return isHashable(self.constant) def extractUnhashableNode(self): value = getUnhashableConstant(self.constant) if value is not None: return makeConstantRefNode( constant = value, source_ref = self.source_ref ) def isNumberConstant(self): return isNumberConstant(self.constant) def isIndexConstant(self): return isIndexConstant(self.constant) def isIndexable(self): return self.constant is None or self.isNumberConstant() def isKnownToBeIterable(self, count): if isIterableConstant(self.constant): return count is None or \ getConstantIterationLength(self.constant) == count else: return False def isKnownToBeIterableAtMin(self, count): length = self.getIterationLength() return length is not None and length >= count def canPredictIterationValues(self): return self.isKnownToBeIterable(None) def getIterationValue(self, count): assert count < len(self.constant) return makeConstantRefNode( constant = self.constant[count], source_ref = self.source_ref ) def getIterationValueRange(self, start, stop): return [ makeConstantRefNode( constant = value, source_ref = self.source_ref ) for value in self.constant[start:stop] ] def getIterationValues(self): source_ref = self.getSourceReference() return tuple( makeConstantRefNode( constant = value, source_ref = source_ref, user_provided = self.user_provided ) for value in self.constant ) def isMapping(self): return type(self.constant) is dict def isMappingWithConstantStringKeys(self): assert self.isMapping() for key in self.constant: if type(key) not in (str, unicode): return False return True def getMappingPairs(self): assert self.isMapping() pairs = [] source_ref = self.getSourceReference() for key, value in iterItems(self.constant): pairs.append( makeConstantRefNode( constant = key, source_ref = source_ref ), makeConstantRefNode( constant = value, source_ref = source_ref ) ) return pairs def getMappingStringKeyPairs(self): assert self.isMapping() pairs = [] source_ref = self.getSourceReference() for key, value in iterItems(self.constant): pairs.append( ( key, makeConstantRefNode( constant = value, source_ref = source_ref ) ) ) return pairs def isBoolConstant(self): return type(self.constant) is bool def mayHaveSideEffects(self): # Constants have no side effects return False def extractSideEffects(self): # Constants have no side effects return () def getIntegerValue(self): if self.isNumberConstant(): return int(self.constant) else: return None def getIndexValue(self): if self.isIndexConstant(): return int(self.constant) else: return None def getStringValue(self): if self.isStringConstant(): return self.constant else: return None def getIterationLength(self): if isIterableConstant(self.constant): return getConstantIterationLength(self.constant) else: return None def isIterableConstant(self): return isIterableConstant(self.constant) def isUnicodeConstant(self): return type(self.constant) is unicode def isStringConstant(self): return type(self.constant) is str def getStrValue(self): if type(self.constant) is str: # Nothing to do. return self else: try: return makeConstantRefNode( constant = str(self.constant), user_provided = self.user_provided, source_ref = self.getSourceReference(), ) except UnicodeEncodeError: # Unicode constants may not be possible to encode. return None def computeExpressionIter1(self, iter_node, trace_collection): if type(self.constant) in (list, set, frozenset, dict): result = makeConstantRefNode( constant = tuple(self.constant), user_provided = self.user_provided, source_ref = self.getSourceReference() ) self.replaceWith(result) return ( iter_node, "new_constant", """\ Iteration over constant %s changed to tuple.""" % type(self.constant).__name__ ) if not isIterableConstant(self.constant): # Any exception may be raised. trace_collection.onExceptionRaiseExit(TypeError) return iter_node, None, None class ExpressionConstantNoneRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_NONE_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = None, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} def getTypeShape(self): return ShapeTypeNoneType def mayBeNone(self): return True class ExpressionConstantTrueRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_TRUE_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = True, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} def getTypeShape(self): return ShapeTypeBool class ExpressionConstantFalseRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_FALSE_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = False, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} def getTypeShape(self): return ShapeTypeBool class ExpressionConstantEllipsisRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_ELLIPSIS_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = Ellipsis, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} def getTypeShape(self): return ShapeTypeEllipsisType class ExpressionConstantDictRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_DICT_REF" def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantDictRef(): return True def getTypeShape(self): return ShapeTypeDict def hasShapeDictionaryExact(self): return True class ExpressionConstantTupleRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_TUPLE_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantTupleRef(): return True def getTypeShape(self): return ShapeTypeTuple the_empty_tuple = () class ExpressionConstantTupleEmptyRef(ExpressionConstantTupleRef): kind = "EXPRESSION_CONSTANT_TUPLE_EMPTY_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantTupleRef.__init__( self, constant = the_empty_tuple, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} class ExpressionConstantListRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_LIST_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantListRef(): return True def getTypeShape(self): return ShapeTypeList the_empty_list = [] class ExpressionConstantListEmptyRef(ExpressionConstantListRef): kind = "EXPRESSION_CONSTANT_LIST_EMPTY_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantListRef.__init__( self, constant = the_empty_list, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} class ExpressionConstantSetRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_SET_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantSetRef(): return True def getTypeShape(self): return ShapeTypeSet the_empty_set = set() class ExpressionConstantSetEmptyRef(ExpressionConstantSetRef): kind = "EXPRESSION_CONSTANT_SET_EMPTY_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantSetRef.__init__( self, constant = the_empty_set, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} class ExpressionConstantFrozensetRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_FROZENSET_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantFrozensetRef(): return True def getTypeShape(self): return ShapeTypeFrozenset the_empty_frozenset = frozenset() class ExpressionConstantFrozensetEmptyRef(ExpressionConstantFrozensetRef): kind = "EXPRESSION_CONSTANT_FROZENSET_EMPTY_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantFrozensetRef.__init__( self, constant = the_empty_frozenset, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} class ExpressionConstantIntRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_INT_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantIntRef(): return True def getTypeShape(self): return ShapeTypeInt class ExpressionConstantLongRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_LONG_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantLongRef(): return True def getTypeShape(self): return ShapeTypeLong class ExpressionConstantStrRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_STR_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantStrRef(): return True def getTypeShape(self): return ShapeTypeStr class ExpressionConstantUnicodeRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_UNICODE_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantUnicodeRef(): return True def getTypeShape(self): return ShapeTypeUnicode class ExpressionConstantBytesRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_BYTES_REF" def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantBytesRef(): return True def getTypeShape(self): return ShapeTypeBytes class ExpressionConstantBytearrayRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_BYTEARRAY_REF" def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantBytearrayRef(): return True def getTypeShape(self): return ShapeTypeBytearray class ExpressionConstantFloatRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_FLOAT_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantFloatRef(): return True def getTypeShape(self): return ShapeTypeFloat class ExpressionConstantComplexRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_COMPLEX_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantComplexRef(): return True class ExpressionConstantSliceRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_SLICE_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantSliceRef(): return True def getTypeShape(self): return ShapeTypeSlice class ExpressionConstantXrangeRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_XRANGE_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantXrangeRef(): return True def getTypeShape(self): return ShapeTypeXrange class ExpressionConstantTypeRef(ExpressionConstantRefBase): kind = "EXPRESSION_CONSTANT_TYPE_REF" __slots__ = () def __init__(self, source_ref, constant, user_provided = False): ExpressionConstantRefBase.__init__( self, constant = constant, user_provided = user_provided, source_ref = source_ref ) @staticmethod def isExpressionConstantTypeRef(): return True def getTypeShape(self): return ShapeTypeType def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): from nuitka.optimizations.OptimizeBuiltinCalls import computeBuiltinCall # Anything may happen. On next pass, if replaced, we might be better # but not now. trace_collection.onExceptionRaiseExit(BaseException) new_node, tags, message = computeBuiltinCall( builtin_name = self.constant.__name__, call_node = call_node ) return new_node, tags, message the_empty_dict = {} class ExpressionConstantDictEmptyRef(ExpressionConstantDictRef): kind = "EXPRESSION_CONSTANT_DICT_EMPTY_REF" __slots__ = () def __init__(self, source_ref, user_provided = False): ExpressionConstantDictRef.__init__( self, constant = the_empty_dict, user_provided = user_provided, source_ref = source_ref ) def getDetails(self): return {} def makeConstantRefNode(constant, source_ref, user_provided = False): # This is dispatching per constant value and types, every case # to be a return statement, pylint: disable=too-many-branches,too-many-return-statements # Dispatch based on constants first. if constant is None: return ExpressionConstantNoneRef( source_ref = source_ref, user_provided = user_provided ) elif constant is True: return ExpressionConstantTrueRef( source_ref = source_ref, user_provided = user_provided ) elif constant is False: return ExpressionConstantFalseRef( source_ref = source_ref, user_provided = user_provided ) elif constant is Ellipsis: return ExpressionConstantEllipsisRef( source_ref = source_ref, user_provided = user_provided ) else: # Next, dispatch based on type. constant_type = type(constant) if constant_type is int: return ExpressionConstantIntRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is str: return ExpressionConstantStrRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is float: return ExpressionConstantFloatRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is long: return ExpressionConstantLongRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is unicode: return ExpressionConstantUnicodeRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is bytes: return ExpressionConstantBytesRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is dict: if constant: return ExpressionConstantDictRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) else: return ExpressionConstantDictEmptyRef( source_ref = source_ref, user_provided = user_provided ) elif constant_type is tuple: if constant: return ExpressionConstantTupleRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) else: return ExpressionConstantTupleEmptyRef( source_ref = source_ref, user_provided = user_provided ) elif constant_type is list: if constant: return ExpressionConstantListRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) else: return ExpressionConstantListEmptyRef( source_ref = source_ref, user_provided = user_provided ) elif constant_type is set: if constant: return ExpressionConstantSetRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) else: return ExpressionConstantSetEmptyRef( source_ref = source_ref, user_provided = user_provided ) elif constant_type is frozenset: if constant: return ExpressionConstantFrozensetRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) else: return ExpressionConstantFrozensetEmptyRef( source_ref = source_ref, user_provided = user_provided ) elif constant_type is complex: return ExpressionConstantComplexRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is slice: return ExpressionConstantSliceRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is type: return ExpressionConstantTypeRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is xrange: return ExpressionConstantXrangeRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) elif constant_type is bytearray: return ExpressionConstantBytearrayRef( source_ref = source_ref, constant = constant, user_provided = user_provided ) else: # Missing constant type, ought to not happen, please report. assert False, constant_type Nuitka-0.5.28.2/nuitka/nodes/DictionaryNodes.py0000644000372000001440000004020313207537242021542 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes that build dictionaries. The "pair" is a sub-structure of the dictionary, representing a key/value pair that is the child of the dictionary creation. """ from nuitka import Constants from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.PythonVersions import python_version from .AttributeNodes import ExpressionAttributeLookup from .BuiltinHashNodes import ExpressionBuiltinHash from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import ( SideEffectsFromChildrenMixin, StatementChildrenHavingBase ) from .NodeMakingHelpers import ( makeConstantReplacementNode, makeRaiseExceptionExpressionFromTemplate, makeStatementOnlyNodesFromExpressions, wrapExpressionWithSideEffects ) from .TypeNodes import ExpressionBuiltinType1 class ExpressionKeyValuePair(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_KEY_VALUE_PAIR" if python_version < 350: named_children = ( "value", "key" ) else: named_children = ( "key", "value" ) def __init__(self, key, value, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "key" : key, "value" : value }, source_ref = source_ref ) getKey = ExpressionChildrenHavingBase.childGetter("key") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, trace_collection): key = self.getKey() hashable = key.isKnownToBeHashable() # If not known to be hashable, that can raise an exception. if not hashable: trace_collection.onExceptionRaiseExit( TypeError ) if hashable is False: # TODO: If it's not hashable, we should turn it into a raise, it's # just difficult to predict the exception value precisely, as it # could be e.g. (2, []), and should then complain about the list. pass return self, None, None def mayRaiseException(self, exception_type): key = self.getKey() return key.mayRaiseException(exception_type) or \ key.isKnownToBeHashable() is not True or \ self.getValue().mayRaiseException(exception_type) def extractSideEffects(self): if self.subnode_key.isKnownToBeHashable() is True: key_part = self.subnode_key.extractSideEffects() else: key_part = ( ExpressionBuiltinHash( value = self.subnode_key, source_ref = self.subnode_key.source_ref ), ) if python_version < 350: return self.subnode_value.extractSideEffects() + key_part else: return key_part + self.subnode_value.extractSideEffects() class ExpressionMakeDict(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_DICT" named_children = ( "pairs", ) def __init__(self, pairs, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "pairs" : tuple(pairs), }, source_ref = source_ref ) getPairs = ExpressionChildrenHavingBase.childGetter("pairs") def computeExpression(self, trace_collection): pairs = self.getPairs() is_constant = True for pair in pairs: key = pair.getKey() if key.isKnownToBeHashable() is False: side_effects = [] for pair2 in pairs: side_effects.extend(pair2.extractSideEffects()) if pair2 is pair: break result = makeRaiseExceptionExpressionFromTemplate( exception_type = "TypeError", template = "unhashable type: '%s'", template_args = ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = key.extractUnhashableNode(), source_ref = key.source_ref ), attribute_name = "__name__", source_ref = key.source_ref ), source_ref = key.source_ref ) result = wrapExpressionWithSideEffects( side_effects = side_effects, old_node = key, new_node = result ) return ( result, "new_raise", "Dictionary key is known to not be hashable." ) if is_constant: if not key.isExpressionConstantRef(): is_constant = False else: value = pair.getValue() if not value.isExpressionConstantRef(): is_constant = False if not is_constant: return self, None, None constant_value = Constants.createConstantDict( keys = [ pair.getKey().getConstant() for pair in pairs ], values = [ pair.getValue().getConstant() for pair in pairs ] ) new_node = makeConstantReplacementNode( constant = constant_value, node = self ) return new_node, "new_constant", """\ Created dictionary found to be constant.""" def mayRaiseException(self, exception_type): for pair in self.getPairs(): if pair.mayRaiseException(exception_type): return True return False def mayHaveSideEffectsBool(self): return False def isKnownToBeIterable(self, count): return count is None or count == len(self.getPairs()) def getIterationLength(self): pair_count = len(self.getPairs()) # Hashing may consume elements. if pair_count >= 2: return None else: return pair_count def getIterationMinLength(self): pair_count = len(self.getPairs()) if pair_count == 0: return 0 else: return 1 def getIterationMaxLength(self): return len(self.getPairs()) def canPredictIterationValues(self): # Dictionaries are fully predictable, pylint: disable=no-self-use # TODO: For some things, that may not be true, when key collisions # happen for example. We will have to check that then. return True def getIterationValue(self, count): return self.getPairs()[count].getKey() def getTruthValue(self): return self.getIterationLength() > 0 def mayBeNone(self): return False def isMapping(self): # Dictionaries are always mappings, but this is a virtual method, # pylint: disable=no-self-use return True def isMappingWithConstantStringKeys(self): for pair in self.getPairs(): key = pair.getKey() if not key.isExpressionConstantRef() or not key.isStringConstant(): return False return True def getMappingStringKeyPairs(self): return [ ( pair.getKey().getConstant(), pair.getValue() ) for pair in self.getPairs() ] def getMappingPairs(self): return self.getPairs() # TODO: Missing computeExpressionIter1 here. For now it would require us to # add lots of temporary variables for keys, which then becomes the tuple, # but for as long as we don't have efficient forward propagation of these, # we won't do that. Otherwise we loose execution order of values with them # remaining as side effects. We could limit ourselves to cases where # isMappingWithConstantStringKeys is true, or keys had no side effects, but # that feels wasted effort as we are going to have full propagation. def computeExpressionDrop(self, statement, trace_collection): expressions = [] for pair in self.getPairs(): expressions.extend(pair.extractSideEffects()) result = makeStatementOnlyNodesFromExpressions( expressions = expressions ) return result, "new_statements", """\ Removed sequence creation for unused sequence.""" def computeExpressionIter1(self, iter_node, trace_collection): return iter_node, None, None def hasShapeDictionaryExact(self): return True class StatementDictOperationSet(StatementChildrenHavingBase): kind = "STATEMENT_DICT_OPERATION_SET" named_children = ( "value", "dict", "key", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, key, value, source_ref): assert dict_arg is not None assert key is not None assert value is not None StatementChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, "value" : value }, source_ref = source_ref ) getDict = StatementChildrenHavingBase.childGetter("dict") getKey = StatementChildrenHavingBase.childGetter("key") getValue = StatementChildrenHavingBase.childGetter("value") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc key = self.getKey() if not key.isKnownToBeHashable(): # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): key = self.getKey() if not key.isKnownToBeHashable(): return True if key.mayRaiseException(exception_type): return True value = self.getValue() if value.mayRaiseException(exception_type): return True return False class StatementDictOperationRemove(StatementChildrenHavingBase): kind = "STATEMENT_DICT_OPERATION_REMOVE" named_children = ( "dict", "key" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, key, source_ref): assert dict_arg is not None assert key is not None StatementChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, }, source_ref = source_ref ) getDict = StatementChildrenHavingBase.childGetter("dict") getKey = StatementChildrenHavingBase.childGetter("key") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): key = self.getKey() if not key.isKnownToBeHashable(): return True if key.mayRaiseException(exception_type): return True # TODO: Could check dict for knowledge about keys. return True class ExpressionDictOperationGet(ExpressionChildrenHavingBase): kind = "EXPRESSION_DICT_OPERATION_GET" named_children = ( "dict", "key" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, key, source_ref): assert dict_arg is not None assert key is not None ExpressionChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, }, source_ref = source_ref ) getDict = ExpressionChildrenHavingBase.childGetter("dict") getKey = ExpressionChildrenHavingBase.childGetter("key") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None class StatementDictOperationUpdate(StatementChildrenHavingBase): kind = "STATEMENT_DICT_OPERATION_UPDATE" named_children = ( "dict", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, dict_arg, value, source_ref): assert dict_arg is not None assert value is not None StatementChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "value" : value }, source_ref = source_ref ) getDict = StatementChildrenHavingBase.childGetter( "dict" ) getValue = StatementChildrenHavingBase.childGetter( "value" ) def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc trace_collection.onExceptionRaiseExit(BaseException) return self, None, None class ExpressionDictOperationIn(ExpressionChildrenHavingBase): kind = "EXPRESSION_DICT_OPERATION_IN" # Follow the reversed nature of "in", with the dictionary on the right # side of things. named_children = ( "key", "dict" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, key, dict_arg, source_ref): assert dict_arg is not None assert key is not None ExpressionChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, }, source_ref = source_ref ) getDict = ExpressionChildrenHavingBase.childGetter("dict") getKey = ExpressionChildrenHavingBase.childGetter("key") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None class ExpressionDictOperationNOTIn(ExpressionChildrenHavingBase): kind = "EXPRESSION_DICT_OPERATION_NOT_IN" # Follow the reversed nature of "in", with the dictionary on the right # side of things. named_children = ( "key", "dict" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, key, dict_arg, source_ref): assert dict_arg is not None assert key is not None ExpressionChildrenHavingBase.__init__( self, values = { "dict" : dict_arg, "key" : key, }, source_ref = source_ref ) getDict = ExpressionChildrenHavingBase.childGetter("dict") getKey = ExpressionChildrenHavingBase.childGetter("key") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None Nuitka-0.5.28.2/nuitka/nodes/NodeBases.py0000644000372000001440000007635313207537242020326 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node base classes. These classes provide the generic base classes available for nodes, statements or expressions alike. There is a dedicated module for expression only stuff. """ from abc import ABCMeta from nuitka import Options, Tracing, TreeXML, Variables from nuitka.__past__ import iterItems from nuitka.containers.odict import OrderedDict from nuitka.PythonVersions import python_version from nuitka.SourceCodeReferences import SourceCodeReference from nuitka.utils.InstanceCounters import counted_del, counted_init from .FutureSpecs import fromFlags from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions class NodeCheckMetaClass(ABCMeta): kinds = {} def __new__(cls, name, bases, dictionary): # This is in conflict with either PyDev or Pylint, pylint: disable=C0204 assert len(bases) == len(set(bases)), bases last_mixin = None for base in bases: base_name = base.__name__ is_mixin = base_name.endswith("Mixin") if is_mixin and last_mixin is False: assert False, (name, bases) last_mixin = is_mixin if "__slots__" not in dictionary: dictionary["__slots__"] = () return ABCMeta.__new__(cls, name, bases, dictionary) def __init__(cls, name, bases, dictionary): # @NoSelf if not name.endswith("Base"): assert ("kind" in dictionary), name kind = dictionary["kind"] assert type(kind) is str, name assert kind not in NodeCheckMetaClass.kinds, name NodeCheckMetaClass.kinds[kind] = cls NodeCheckMetaClass.kinds[name] = cls def convert(value): if value in ("AND", "OR", "NOT"): return value else: return value.title() kind_to_name_part = "".join( [convert(x) for x in kind.split('_')] ) assert name.endswith(kind_to_name_part), \ (name, kind_to_name_part) # Automatically add checker methods for everything to the common # base class checker_method = "is" + kind_to_name_part def checkKind(self): return self.kind == kind if not hasattr(NodeBase, checker_method): setattr(NodeBase, checker_method, checkKind) ABCMeta.__init__(cls, name, bases, dictionary) # For every node type, there is a test, and then some more members, # For Python2/3 compatible source, we create a base class that has the metaclass # used and doesn't require making a choice. NodeMetaClassBase = NodeCheckMetaClass( "NodeMetaClassBase", (object,), {"__slots__" : () } ) class NodeBase(NodeMetaClassBase): __slots__ = "parent", "source_ref", "effective_source_ref" # String to identify the node class, to be consistent with its name. kind = None @counted_init def __init__(self, source_ref): # The base class has no __init__ worth calling. # Check source reference to meet basic standards, so we note errors # when they occur. assert source_ref is not None assert source_ref.line is not None self.parent = None self.source_ref = source_ref __del__ = counted_del() def __repr__(self): return "" % (self.getDescription()) def getDescription(self): """ Description of the node, intended for use in __repr__ and graphical display. """ details = self.getDetails() if details: return "'%s' with %s" % (self.kind, self.getDetails()) else: return "'%s'" % self.kind def getDetails(self): """ Details of the node, intended for re-creation. We are not using the pickle mechanisms, but this is basically part of what the constructor call needs. Real children will also be added. """ # Virtual method, pylint: disable=no-self-use return {} def getDetailsForDisplay(self): """ Details of the node, intended for use in __repr__ and dumps. This is also used for XML. """ return self.getDetails() def getDetail(self): """ Details of the node, intended for use in __repr__ and graphical display. """ return str(self.getDetails())[1:-1] def makeClone(self): try: # Using star dictionary arguments here for generic use. result = self.__class__( source_ref = self.source_ref, **self.getDetails() ) except TypeError: print("Problem cloning", self.__class__) raise effective_source_ref = self.getCompatibleSourceReference() if effective_source_ref is not self.source_ref: result.setCompatibleSourceReference(effective_source_ref) return result def makeCloneAt(self, source_ref): result = self.makeClone() result.source_ref = source_ref return result def getParent(self): """ Parent of the node. Every node except modules have to have a parent. """ if self.parent is None and not self.isCompiledPythonModule(): # print self.getVisitableNodesNamed() assert False, (self, self.source_ref) return self.parent def getChildName(self): """ Return the role in the current parent, subject to changes. """ parent = self.getParent() for key in parent.named_children: value = parent.getChild(key) if self is value: return key if type(value) is tuple: if self in value: return key, value.index(self) # TODO: Not checking tuples yet return None def getChildNameNice(self): child_name = self.getChildName() if hasattr(self.parent, "nice_children"): return self.parent.nice_children[self.parent.named_children.index(child_name)] else: return child_name def getParentFunction(self): """ Return the parent that is a function. """ parent = self.getParent() while parent is not None and not parent.isExpressionFunctionBodyBase(): parent = parent.getParent() return parent def getParentModule(self): """ Return the parent that is module. """ parent = self while not parent.isCompiledPythonModule(): if hasattr(parent, "provider"): # After we checked, we can use it, will be much faster route # to take. parent = parent.provider else: parent = parent.getParent() return parent def isParentVariableProvider(self): # Check if it's a closure giver, in which cases it can provide variables, return isinstance(self, ClosureGiverNodeMixin) def getParentVariableProvider(self): parent = self.getParent() while not parent.isParentVariableProvider(): parent = parent.getParent() return parent def getParentReturnConsumer(self): parent = self.getParent() while not parent.isParentVariableProvider() and \ not parent.isExpressionOutlineBody(): parent = parent.getParent() return parent def getParentStatementsFrame(self): current = self.getParent() while True: if current.isStatementsFrame(): return current if current.isParentVariableProvider(): return None if current.isExpressionOutlineBody(): return None current = current.getParent() def getSourceReference(self): return self.source_ref def setCompatibleSourceReference(self, source_ref): """ Bug compatible line numbers information. As CPython outputs the last bit of bytecode executed, and not the line of the operation. For example calls, output the line of the last argument, as opposed to the line of the operation start. For tests, we wants to be compatible. In improved more, we are not being fully compatible, and just drop it altogether. """ # Getting the same source reference can be dealt with quickly, so do # this first. if self.source_ref is not source_ref and \ Options.isFullCompat() and \ self.source_ref != source_ref: # An attribute outside of "__init__", so we save one memory for the # most cases. Very few cases involve splitting across lines. # pylint: disable=W0201 self.effective_source_ref = source_ref def getCompatibleSourceReference(self): """ Bug compatible line numbers information. See above. """ return getattr(self, "effective_source_ref", self.source_ref) def asXml(self): line = self.getSourceReference().getLineNumber() result = TreeXML.Element( "node", kind = self.__class__.__name__, line = "%s" % line ) compat_line = self.getCompatibleSourceReference().getLineNumber() if compat_line != line: result.attrib["compat_line"] = str(compat_line) for key, value in iterItems(self.getDetailsForDisplay()): result.set(key, str(value)) for name, children in self.getVisitableNodesNamed(): role = TreeXML.Element( "role", name = name ) result.append(role) if children is None: role.attrib["type"] = "none" elif type(children) not in (list, tuple): role.append( children.asXml() ) else: role.attrib["type"] = "list" for child in children: role.append( child.asXml() ) return result @classmethod def fromXML(cls, provider, source_ref, **args): # Only some things need a provider, pylint: disable=unused-argument return cls(source_ref = source_ref, **args) def asXmlText(self): xml = self.asXml() return TreeXML.toString(xml) def dump(self, level = 0): Tracing.printIndented(level, self) Tracing.printSeparator(level) for visitable in self.getVisitableNodes(): visitable.dump(level + 1) Tracing.printSeparator(level) @staticmethod def isStatementsFrame(): return False @staticmethod def isCompiledPythonModule(): # For overload by module nodes return False def isExpression(self): return self.kind.startswith("EXPRESSION_") def isStatement(self): return self.kind.startswith("STATEMENT_") def isExpressionBuiltin(self): return self.kind.startswith("EXPRESSION_BUILTIN_") @staticmethod def isExpressionConstantRef(): return False @staticmethod def isExpressionOperationBinary(): return False def isExpressionSideEffects(self): # Virtual method, pylint: disable=no-self-use # We need to provide this, as these node kinds are only imported if # necessary, but we test against them. return False def isStatementReraiseException(self): # Virtual method, pylint: disable=no-self-use return False def isExpressionMakeSequence(self): # Virtual method, pylint: disable=no-self-use return False def isNumberConstant(self): # Virtual method, pylint: disable=no-self-use return False def isExpressionCall(self): # Virtual method, pylint: disable=no-self-use return False def isExpressionFunctionBodyBase(self): # Virtual method, pylint: disable=no-self-use return False def isExpressionOutlineFunctionBodyBase(self): # Virtual method, pylint: disable=no-self-use return False def visit(self, context, visitor): visitor(self) for visitable in self.getVisitableNodes(): visitable.visit(context, visitor) def getVisitableNodes(self): # Virtual method, pylint: disable=no-self-use return () def getVisitableNodesNamed(self): """ Named children dictionary. For use in debugging and XML output. """ # Virtual method, pylint: disable=no-self-use return () def replaceWith(self, new_node): self.parent.replaceChild( old_node = self, new_node = new_node ) def getName(self): # Virtual method, pylint: disable=no-self-use return None def mayHaveSideEffects(self): """ Unless we are told otherwise, everything may have a side effect. """ # Virtual method, pylint: disable=no-self-use return True def isOrderRelevant(self): return self.mayHaveSideEffects() def extractSideEffects(self): """ Unless defined otherwise, the expression is the side effect. """ return (self,) def mayRaiseException(self, exception_type): """ Unless we are told otherwise, everything may raise everything. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def mayReturn(self): return "_RETURN" in self.kind def mayBreak(self): # For overload, pylint: disable=no-self-use return False def mayContinue(self): # For overload, pylint: disable=no-self-use return False def needsFrame(self): """ Unless we are tolder otherwise, this depends on exception raise. """ return self.mayRaiseException(BaseException) def willRaiseException(self, exception_type): """ Unless we are told otherwise, nothing may raise anything. """ # Virtual method, pylint: disable=no-self-use,unused-argument return False def isStatementAborting(self): """ Is the node aborting, control flow doesn't continue after this node. """ assert self.isStatement(), self.kind return False def needsLocalsDict(self): """ Node requires a locals dictionary by provider. """ # Virtual method, pylint: disable=no-self-use return False class CodeNodeMixin(object): def __init__(self, name, code_prefix): assert name is not None self.name = name self.code_prefix = code_prefix # The code name is determined on demand only. self.code_name = None # The "UID" values of children kinds are kept here. self.uids = {} def getName(self): return self.name def getCodeName(self): if self.code_name is None: provider = self.getParentVariableProvider().getEntryPoint() parent_name = provider.getCodeName() uid = "_%d" % provider.getChildUID(self) assert isinstance(self, CodeNodeMixin) if self.name: name = uid + '_' + self.name.strip("<>") else: name = uid self.code_name = "%s$$$%s%s" % ( parent_name, self.code_prefix, name ) return self.code_name def getChildUID(self, node): if node.kind not in self.uids: self.uids[node.kind] = 0 self.uids[node.kind] += 1 return self.uids[node.kind] class ChildrenHavingMixin(object): named_children = () checkers = {} def __init__(self, values): assert type(self.named_children) is tuple and self.named_children # Check for completeness of given values, everything should be there # but of course, might be put to None. assert set(values.keys()) == set(self.named_children) for name, value in values.items(): if name in self.checkers: value = self.checkers[name](value) assert type(value) is not list, name if type(value) is tuple: assert None not in value, name for val in value: val.parent = self elif value is not None: value.parent = self elif value is None: pass else: assert False, type(value) attr_name = "subnode_" + name setattr(self, attr_name, value) def setChild(self, name, value): """ Set a child value. Do not overload, provider self.checkers instead. """ # Only accept legal child names assert name in self.named_children, name # Lists as inputs are OK, but turn them into tuples. if type(value) is list: value = tuple(value) if name in self.checkers: value = self.checkers[name](value) # Re-parent value to us. if type(value) is tuple: for val in value: val.parent = self elif value is not None: value.parent = self attr_name = "subnode_" + name # Determine old value, and inform it about loosing its parent. old_value = getattr(self, attr_name) assert old_value is not value, value setattr(self, attr_name, value) def getChild(self, name): # Only accept legal child names attr_name = "subnode_" + name return getattr(self, attr_name) @staticmethod def childGetter(name): attr_name = "subnode_" + name def getter(self): return getattr(self, attr_name) return getter @staticmethod def childSetter(name): def setter(self, value): self.setChild(name, value) return setter def getVisitableNodes(self): # TODO: Consider if a generator would be faster. result = [] for name in self.named_children: attr_name = "subnode_" + name value = getattr(self, attr_name) if value is None: pass elif type(value) is tuple: result += list(value) elif isinstance(value, NodeBase): result.append(value) else: raise AssertionError( self, "has illegal child", name, value, value.__class__ ) return tuple(result) def getVisitableNodesNamed(self): """ Named children dictionary. For use in debugging and XML output. """ for name in self.named_children: attr_name = "subnode_" + name value = getattr(self, attr_name) yield name, value def replaceChild(self, old_node, new_node): if new_node is not None and not isinstance(new_node, NodeBase): raise AssertionError( "Cannot replace with", new_node, "old", old_node, "in", self ) # Find the replaced node, as an added difficulty, what might be # happening, is that the old node is an element of a tuple, in which we # may also remove that element, by setting it to None. for key in self.named_children: value = self.getChild(key) if value is None: pass elif type(value) is tuple: if old_node in value: if new_node is not None: self.setChild( key, tuple( (val if val is not old_node else new_node) for val in value ) ) else: self.setChild( key, tuple( val for val in value if val is not old_node ) ) return key elif isinstance(value, NodeBase): if old_node is value: self.setChild(key, new_node) return key else: assert False, (key, value, value.__class__) raise AssertionError( "Didn't find child", old_node, "in", self ) def makeClone(self): values = {} for key in self.named_children: value = self.getChild(key) assert type(value) is not list, key if value is None: values[key] = None elif type(value) is tuple: values[key] = tuple( v.makeClone() for v in value ) else: values[key] = value.makeClone() values.update( self.getDetails() ) try: # Using star dictionary arguments here for generic use, # pylint: disable=E1123 result = self.__class__( source_ref = self.source_ref, **values ) except TypeError: print("Problem cloning", self.__class__) raise effective_source_ref = self.getCompatibleSourceReference() if effective_source_ref is not self.source_ref: result.setCompatibleSourceReference(effective_source_ref) return result class ClosureGiverNodeMixin(CodeNodeMixin): """ Blass class for nodes that provide variables for closure takers. """ def __init__(self, name, code_prefix): CodeNodeMixin.__init__( self, name = name, code_prefix = code_prefix ) # TODO: Only Python3 classes need this to be an ordered dict, the order # of it should come from elsewhere though. self.providing = OrderedDict() self.temp_variables = {} self.temp_scopes = {} self.preserver_id = 0 def hasProvidedVariable(self, variable_name): return variable_name in self.providing def getProvidedVariable(self, variable_name): if variable_name not in self.providing: self.providing[variable_name] = self.createProvidedVariable( variable_name = variable_name ) return self.providing[variable_name] def createProvidedVariable(self, variable_name): # Virtual method, pylint: disable=no-self-use assert type(variable_name) is str return None def registerProvidedVariable(self, variable): assert variable is not None self.providing[variable.getName()] = variable def allocateTempScope(self, name): self.temp_scopes[name] = self.temp_scopes.get(name, 0) + 1 return "%s_%d" % ( name, self.temp_scopes[name] ) def allocateTempVariable(self, temp_scope, name): if temp_scope is not None: full_name = "%s__%s" % ( temp_scope, name ) else: assert name != "result" full_name = name # No duplicates please. assert full_name not in self.temp_variables, full_name result = self.createTempVariable( temp_name = full_name ) return result def createTempVariable(self, temp_name): if temp_name in self.temp_variables: return self.temp_variables[ temp_name ] result = Variables.TempVariable( owner = self, variable_name = temp_name ) self.temp_variables[temp_name] = result return result def getTempVariable(self, temp_scope, name): if temp_scope is not None: full_name = "%s__%s" % (temp_scope, name) else: full_name = name return self.temp_variables[full_name] def getTempVariables(self): return self.temp_variables.values() def removeTempVariable(self, variable): del self.temp_variables[variable.getName()] def allocatePreserverId(self): if python_version >= 300: self.preserver_id += 1 return self.preserver_id class ClosureTakerMixin(object): """ Mixin for nodes that accept variables from closure givers. """ def __init__(self, provider): self.provider = provider self.taken = set() def getParentVariableProvider(self): return self.provider def getClosureVariable(self, variable_name): result = self.provider.getVariableForClosure( variable_name = variable_name ) assert result is not None, variable_name if not result.isModuleVariable(): self.addClosureVariable(result) return result def addClosureVariable(self, variable): self.taken.add(variable) return variable def getClosureVariables(self): return tuple( sorted( [ take for take in self.taken if not take.isModuleVariable() ], key = lambda x : x.getName() ) ) def getClosureVariableIndex(self, variable): closure_variables = self.getClosureVariables() for count, closure_variable in enumerate(closure_variables): if variable is closure_variable: return count raise IndexError(variable) def hasTakenVariable(self, variable_name): for variable in self.taken: if variable.getName() == variable_name: return True return False def getTakenVariable(self, variable_name): for variable in self.taken: if variable.getName() == variable_name: return variable return None class StatementChildrenHavingBase(ChildrenHavingMixin, NodeBase): def __init__(self, values, source_ref): NodeBase.__init__(self, source_ref = source_ref) ChildrenHavingMixin.__init__( self, values = values ) def computeStatementSubExpressions(self, trace_collection): """ Compute a statement. Default behavior is to just visit the child expressions first, and then the node "computeStatement". For a few cases this needs to be overloaded. """ expressions = self.getVisitableNodes() for count, expression in enumerate(expressions): assert expression.isExpression(), (self, expression) expression = trace_collection.onExpression( expression = expression ) if expression.willRaiseException(BaseException): wrapped_expression = makeStatementOnlyNodesFromExpressions( expressions[:count+1] ) assert wrapped_expression is not None return ( wrapped_expression, "new_raise", lambda : "For %s the expression '%s' will raise." % ( self.getStatementNiceName(), expression.getChildNameNice() ) ) return self, None, None def getStatementNiceName(self): # Virtual method, pylint: disable=no-self-use return "undescribed statement" class SideEffectsFromChildrenMixin(object): def mayHaveSideEffects(self): for child in self.getVisitableNodes(): if child.mayHaveSideEffects(): return True return False def extractSideEffects(self): # No side effects at all but from the children. result = [] for child in self.getVisitableNodes(): result.extend( child.extractSideEffects() ) return tuple(result) def makeChild(provider, child, source_ref): child_type = child.attrib.get("type") if child_type == "list": return [ fromXML( provider = provider, xml = sub_child, source_ref = source_ref ) for sub_child in child ] elif child_type == "none": return None else: return fromXML( provider = provider, xml = child[0], source_ref = source_ref ) def getNodeClassFromName(kind): return NodeCheckMetaClass.kinds[kind] def extractKindAndArgsFromXML(xml, source_ref): kind = xml.attrib["kind"] args = dict(xml.attrib) del args["kind"] if source_ref is None: source_ref = SourceCodeReference.fromFilenameAndLine( args["filename"], int(args["line"]) ) del args["filename"] del args["line"] else: source_ref = source_ref.atLineNumber(int(args["line"])) del args["line"] node_class = getNodeClassFromName(kind) return kind, node_class, args, source_ref def fromXML(provider, xml, source_ref = None): assert xml.tag == "node", xml kind, node_class, args, source_ref = extractKindAndArgsFromXML(xml, source_ref) if "constant" in args: # TODO: Try and reduce/avoid this, use marshal and/or pickle from a file # global stream instead. For now, this will do. pylint: disable=eval-used args["constant"] = eval(args["constant"]) if kind in ("ExpressionFunctionBody", "PythonMainModule", "PythonCompiledModule", "PythonCompiledPackage", "PythonInternalModule"): delayed = node_class.named_children if "code_flags" in args: args["future_spec"] = fromFlags(args["code_flags"]) else: delayed = () for child in xml: assert child.tag == "role", child.tag child_name = child.attrib["name"] # Might want to want until provider is updated with some # children. In these cases, we pass the XML node, rather # than a Nuitka node. if child_name not in delayed: args[child_name] = makeChild(provider, child, source_ref) else: args[child_name] = child try: return node_class.fromXML( provider = provider, source_ref = source_ref, **args ) except (TypeError, AttributeError): Tracing.printLine(node_class, args, source_ref) raise Nuitka-0.5.28.2/nuitka/nodes/StatementNodes.py0000644000372000001440000002323113207537242021403 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for statements. """ from .NodeBases import NodeBase, StatementChildrenHavingBase def checkStatements(value): """ Check that statements list value property. Must not be None, must not contain None, and of course only statements, may be empty. """ assert value is not None assert None not in value for statement in value: assert statement.isStatement() or statement.isStatementsFrame(), \ statement.asXmlText() return tuple(value) class StatementsSequence(StatementChildrenHavingBase): kind = "STATEMENTS_SEQUENCE" named_children = ( "statements", ) checkers = { "statements" : checkStatements } def __init__(self, statements, source_ref): StatementChildrenHavingBase.__init__( self, values = { "statements" : statements }, source_ref = source_ref ) getStatements = StatementChildrenHavingBase.childGetter("statements") setStatements = StatementChildrenHavingBase.childSetter("statements") # Overloading name based automatic check, so that derived ones know it too. def isStatementsSequence(self): # Virtual method, pylint: disable=no-self-use return True def trimStatements(self, statement): assert statement.parent is self old_statements = list(self.getStatements()) assert statement in old_statements, \ (statement, self) new_statements = old_statements[ : old_statements.index(statement)+1 ] self.setChild("statements", new_statements) def removeStatement(self, statement): assert statement.parent is self statements = list(self.getStatements()) statements.remove(statement) self.setChild("statements", statements) def mergeStatementsSequence(self, statement_sequence): assert statement_sequence.parent is self old_statements = list(self.getStatements()) assert statement_sequence in old_statements, \ (statement_sequence, self) merge_index = old_statements.index(statement_sequence) new_statements = tuple(old_statements[ : merge_index ]) + \ statement_sequence.getStatements() + \ tuple(old_statements[ merge_index+1 : ]) self.setChild("statements", new_statements) def mayHaveSideEffects(self): # Statement sequences have a side effect if one of the statements does. for statement in self.getStatements(): if statement.mayHaveSideEffects(): return True return False def mayRaiseException(self, exception_type): for statement in self.getStatements(): if statement.mayRaiseException(exception_type): return True return False def needsFrame(self): for statement in self.getStatements(): if statement.needsFrame(): return True return False def mayReturn(self): for statement in self.getStatements(): if statement.mayReturn(): return True return False def mayBreak(self): for statement in self.getStatements(): if statement.mayBreak(): return True return False def mayContinue(self): for statement in self.getStatements(): if statement.mayContinue(): return True return False def mayRaiseExceptionOrAbort(self, exception_type): return self.mayRaiseException(exception_type) or \ self.mayReturn() or \ self.mayBreak() or \ self.mayContinue() def isStatementAborting(self): return self.getStatements()[-1].isStatementAborting() def computeStatement(self, trace_collection): # Don't want to be called like this. assert False, self def computeStatementsSequence(self, trace_collection): new_statements = [] statements = self.getStatements() assert statements, self for count, statement in enumerate(statements): # May be frames embedded. if statement.isStatementsFrame(): new_statement = statement.computeStatementsSequence( trace_collection ) else: new_statement = trace_collection.onStatement( statement = statement ) if new_statement is not None: if new_statement.isStatementsSequence() and \ not new_statement.isStatementsFrame(): new_statements.extend( new_statement.getStatements() ) else: new_statements.append( new_statement ) if statement is not statements[-1] and \ new_statement.isStatementAborting(): trace_collection.signalChange( "new_statements", statements[count+1].getSourceReference(), "Removed dead statements." ) break if statements != new_statements: if new_statements: self.setStatements(new_statements) return self else: return None else: return self class StatementExpressionOnly(StatementChildrenHavingBase): kind = "STATEMENT_EXPRESSION_ONLY" named_children = ( "expression", ) def __init__(self, expression, source_ref): assert expression.isExpression() StatementChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) def getDetail(self): return "expression %s" % self.getExpression() def mayHaveSideEffects(self): return self.getExpression().mayHaveSideEffects() def mayRaiseException(self, exception_type): return self.getExpression().mayRaiseException(exception_type) getExpression = StatementChildrenHavingBase.childGetter( "expression" ) def computeStatement(self, trace_collection): trace_collection.onExpression( expression = self.getExpression() ) expression = self.getExpression() if expression.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) result, change_tags, change_desc = expression.computeExpressionDrop( statement = self, trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc return self, None, None class StatementPreserveFrameException(NodeBase): kind = "STATEMENT_PRESERVE_FRAME_EXCEPTION" __slots__ = ("preserver_id",) def __init__(self, preserver_id, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.preserver_id = preserver_id def getDetails(self): return { "preserver_id" : self.preserver_id } def getPreserverId(self): return self.preserver_id def computeStatement(self, trace_collection): # For Python2 generators, it's not necessary to preserve, the frame # decides it. TODO: This check makes only sense once. if self.getParentStatementsFrame().needsExceptionFramePreservation(): return self, None, None else: return ( None, "new_statements", "Removed frame preservation for generators." ) def mayRaiseException(self, exception_type): return False def needsFrame(self): return True class StatementRestoreFrameException(NodeBase): kind = "STATEMENT_RESTORE_FRAME_EXCEPTION" __slots__ = ("preserver_id",) def __init__(self, preserver_id, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.preserver_id = preserver_id def getDetails(self): return { "preserver_id" : self.preserver_id } def getPreserverId(self): return self.preserver_id def computeStatement(self, trace_collection): return self, None, None def mayRaiseException(self, exception_type): return False class StatementPublishException(NodeBase): kind = "STATEMENT_PUBLISH_EXCEPTION" def __init__(self, source_ref): NodeBase.__init__( self, source_ref = source_ref ) def computeStatement(self, trace_collection): # TODO: Determine the need for it. return self, None, None def mayRaiseException(self, exception_type): return False Nuitka-0.5.28.2/nuitka/nodes/BuiltinSumNodes.py0000644000372000001440000000746613122472300021533 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'sum' built-in. This is a rather challenging case for optimization, as it has C code behind it that could be in-lined sometimes for more static analysis. """ from nuitka.optimizations import BuiltinOptimization from .ExpressionBases import ExpressionChildrenHavingBase class ExpressionBuiltinSumBase(ExpressionChildrenHavingBase): builtin_spec = BuiltinOptimization.builtin_sum_spec def __init__(self, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = values, source_ref = source_ref ) def computeBuiltinSpec(self, trace_collection, given_values): assert self.builtin_spec is not None, self if not self.builtin_spec.isCompileTimeComputable(given_values): trace_collection.onExceptionRaiseExit(BaseException) # TODO: Raise exception known step 0. return self, None, None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_spec.simulateCall( given_values ), description = "Built-in call to '%s' computed." % ( self.builtin_spec.getName() ) ) class ExpressionBuiltinSum1(ExpressionBuiltinSumBase): kind = "EXPRESSION_BUILTIN_SUM1" named_children = ( "sequence", ) def __init__(self, sequence, source_ref): assert sequence is not None ExpressionBuiltinSumBase.__init__( self, values = { "sequence" : sequence, }, source_ref = source_ref ) getSequence = ExpressionChildrenHavingBase.childGetter("sequence") def computeExpression(self, trace_collection): sequence = self.getSequence() # TODO: Protect against large xrange constants return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( sequence, ) ) class ExpressionBuiltinSum2(ExpressionBuiltinSumBase): kind = "EXPRESSION_BUILTIN_SUM2" named_children = ( "sequence", "start" ) def __init__(self, sequence, start, source_ref): assert sequence is not None assert start is not None ExpressionBuiltinSumBase.__init__( self, values = { "sequence" : sequence, "start" : start }, source_ref = source_ref ) getSequence = ExpressionChildrenHavingBase.childGetter("sequence") getStart = ExpressionChildrenHavingBase.childGetter("start") def computeExpression(self, trace_collection): sequence = self.getSequence() start = self.getStart() # TODO: Protect against large xrange constants return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( sequence, start ) ) Nuitka-0.5.28.2/nuitka/nodes/ContainerOperationNodes.py0000644000372000001440000001275013122472300023233 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Operations on Containers. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase class StatementListOperationAppend(StatementChildrenHavingBase): kind = "STATEMENT_LIST_OPERATION_APPEND" named_children = ( "list", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, list_arg, value, source_ref): assert list_arg is not None assert value is not None StatementChildrenHavingBase.__init__( self, values = { "list" : list_arg, "value" : value }, source_ref = source_ref ) getList = StatementChildrenHavingBase.childGetter("list") getValue = StatementChildrenHavingBase.childGetter("value") def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc trace_collection.removeKnowledge(self.getList()) return self, None, None class ExpressionListOperationExtend(ExpressionChildrenHavingBase): kind = "EXPRESSION_LIST_OPERATION_EXTEND" named_children = ( "list", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, list_arg, value, source_ref): assert list_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "list" : list_arg, "value" : value }, source_ref = source_ref ) getList = ExpressionChildrenHavingBase.childGetter("list") getValue = ExpressionChildrenHavingBase.childGetter("value") def computeExpression(self, trace_collection): trace_collection.removeKnowledge(self.getList()) return self, None, None class ExpressionListOperationPop(ExpressionChildrenHavingBase): kind = "EXPRESSION_LIST_OPERATION_POP" named_children = ( "list", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, list_arg, source_ref): assert list_arg is not None ExpressionChildrenHavingBase.__init__( self, values = { "list" : list_arg, }, source_ref = source_ref ) getList = ExpressionChildrenHavingBase.childGetter("list") def computeExpression(self, trace_collection): # We might be able to tell that element, or know that it cannot exist # and raise an exception instead. trace_collection.removeKnowledge(self.getList()) return self, None, None class StatementSetOperationAdd(StatementChildrenHavingBase): kind = "STATEMENT_SET_OPERATION_ADD" named_children = ( "set", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, set_arg, value, source_ref): assert set_arg is not None assert value is not None StatementChildrenHavingBase.__init__( self, values = { "set" : set_arg, "value" : value }, source_ref = source_ref ) getSet = StatementChildrenHavingBase.childGetter( "set" ) getValue = StatementChildrenHavingBase.childGetter( "value" ) def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc trace_collection.removeKnowledge(self.getSet()) return self, None, None class ExpressionSetOperationUpdate(ExpressionChildrenHavingBase): kind = "EXPRESSION_SET_OPERATION_UPDATE" named_children = ( "set", "value" ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, set_arg, value, source_ref): assert set_arg is not None assert value is not None ExpressionChildrenHavingBase.__init__( self, values = { "set" : set_arg, "value" : value }, source_ref = source_ref ) getSet = ExpressionChildrenHavingBase.childGetter( "set" ) getValue = ExpressionChildrenHavingBase.childGetter( "value" ) def computeExpression(self, trace_collection): trace_collection.removeKnowledge(self.getSet()) return self, None, None Nuitka-0.5.28.2/nuitka/nodes/BuiltinRangeNodes.py0000644000372000001440000004601213122472300022011 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node the calls to the 'range' built-in. This is a rather complex beast as it has many cases, is difficult to know if it's sizable enough to compute, and there are complex cases, where the bad result of it can be predicted still, and these are interesting for warnings. """ import math from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ExpressionBases import ExpressionChildrenHavingBase from .NodeMakingHelpers import makeConstantReplacementNode from .shapes.BuiltinTypeShapes import ShapeTypeList, ShapeTypeXrange class ExpressionBuiltinRangeBase(ExpressionChildrenHavingBase): """ Base class for range nodes with 1/2/3 arguments. """ builtin_spec = BuiltinOptimization.builtin_range_spec def __init__(self, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = values, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeList def getTruthValue(self): length = self.getIterationLength() if length is None: return None else: return length > 0 def mayHaveSideEffects(self): for child in self.getVisitableNodes(): if child.mayHaveSideEffects(): return True if child.getIntegerValue() is None: return True if python_version >= 270 and \ child.isExpressionConstantRef() and \ type(child.getConstant()) is float: return True return False def mayRaiseException(self, exception_type): for child in self.getVisitableNodes(): if child.mayRaiseException(exception_type): return True # TODO: Should take exception_type value into account here. if child.getIntegerValue() is None: return True if python_version >= 270 and \ child.isExpressionConstantRef() and \ type(child.getConstant()) is float: return True step = self.getStep() # false alarm, pylint: disable=assignment-from-none # A step of 0 will raise. if step is not None and step.getIntegerValue() == 0: return True return False def computeBuiltinSpec(self, trace_collection, given_values): assert self.builtin_spec is not None, self if not self.builtin_spec.isCompileTimeComputable(given_values): trace_collection.onExceptionRaiseExit(BaseException) # TODO: Raise exception known step 0. return self, None, None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_spec.simulateCall( given_values ), description = "Built-in call to '%s' computed." % ( self.builtin_spec.getName() ) ) def computeExpressionIter1(self, iter_node, trace_collection): assert python_version < 300 iteration_length = self.getIterationLength() if iteration_length is not None and iteration_length > 256: result = makeExpressionBuiltinXrange( low = self.getLow(), high = self.getHigh(), step = self.getStep(), source_ref = self.getSourceReference() ) self.replaceWith(result) return ( iter_node, "new_expression", "Replaced 'range' with 'xrange' built-in call." ) # No exception will be raised on ranges. return iter_node, None, None def canPredictIterationValues(self): return self.getIterationLength() is not None @staticmethod def getLow(): return None @staticmethod def getHigh(): return None @staticmethod def getStep(): return None def mayBeNone(self): return False class ExpressionBuiltinRange1(ExpressionBuiltinRangeBase): kind = "EXPRESSION_BUILTIN_RANGE1" named_children = ( "low", ) def __init__(self, low, source_ref): assert low is not None ExpressionBuiltinRangeBase.__init__( self, values = { "low" : low, }, source_ref = source_ref ) getLow = ExpressionChildrenHavingBase.childGetter("low") def computeExpression(self, trace_collection): assert python_version < 300 low = self.getLow() return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( low, ) ) def getIterationLength(self): low = self.getLow().getIntegerValue() if low is None: return None return max(0, low) def getIterationValue(self, element_index): length = self.getIterationLength() if length is None: return None if element_index > length: return None # TODO: Make sure to cast element_index to what CPython will give, for # now a downcast will do. return makeConstantReplacementNode( constant = int(element_index), node = self ) def isKnownToBeIterable(self, count): return count is None or count == self.getIterationLength() class ExpressionBuiltinRange2(ExpressionBuiltinRangeBase): kind = "EXPRESSION_BUILTIN_RANGE2" named_children = ("low", "high") def __init__(self, low, high, source_ref): ExpressionBuiltinRangeBase.__init__( self, values = { "low" : low, "high" : high }, source_ref = source_ref ) getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") builtin_spec = BuiltinOptimization.builtin_range_spec def computeExpression(self, trace_collection): assert python_version < 300 low = self.getLow() high = self.getHigh() return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( low, high ) ) def getIterationLength(self): low = self.getLow() high = self.getHigh() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None return max(0, high - low) def getIterationValue(self, element_index): low = self.getLow() high = self.getHigh() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None result = low + element_index if result >= high: return None else: return makeConstantReplacementNode( constant = result, node = self ) def isKnownToBeIterable(self, count): return count is None or count == self.getIterationLength() class ExpressionBuiltinRange3(ExpressionBuiltinRangeBase): kind = "EXPRESSION_BUILTIN_RANGE3" named_children = ( "low", "high", "step" ) def __init__(self, low, high, step, source_ref): ExpressionBuiltinRangeBase.__init__( self, values = { "low" : low, "high" : high, "step" : step }, source_ref = source_ref ) getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") getStep = ExpressionChildrenHavingBase.childGetter("step") builtin_spec = BuiltinOptimization.builtin_range_spec def computeExpression(self, trace_collection): assert python_version < 300 low = self.getLow() high = self.getHigh() step = self.getStep() return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( low, high, step ) ) def getIterationLength(self): low = self.getLow() high = self.getHigh() step = self.getStep() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None step = step.getIntegerValue() if step is None: return None # Give up on this, will raise ValueError. if step == 0: return None if low < high: if step < 0: estimate = 0 else: estimate = math.ceil(float(high - low) / step) else: if step > 0: estimate = 0 else: estimate = math.ceil(float(high - low) / step) estimate = round(estimate) assert estimate >= 0 return int(estimate) def canPredictIterationValues(self): return self.getIterationLength() is not None def getIterationValue(self, element_index): low = self.getLow().getIntegerValue() if low is None: return None high = self.getHigh().getIntegerValue() if high is None: return None step = self.getStep().getIntegerValue() result = low + step * element_index if result >= high: return None else: return makeConstantReplacementNode( constant = result, node = self ) def isKnownToBeIterable(self, count): return count is None or count == self.getIterationLength() class ExpressionBuiltinXrangeBase(ExpressionChildrenHavingBase): """ Base class for xrange nodes with 1/2/3 arguments. """ builtin_spec = BuiltinOptimization.builtin_xrange_spec def __init__(self, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = values, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeXrange def canPredictIterationValues(self): return self.getIterationLength() is not None def getTruthValue(self): length = self.getIterationLength() if length is None: return None else: return length > 0 def mayHaveSideEffects(self): for child in self.getVisitableNodes(): if child.mayHaveSideEffects(): return True if child.getIntegerValue() is None: return True return False def mayRaiseException(self, exception_type): for child in self.getVisitableNodes(): if child.mayRaiseException(exception_type): return True # TODO: Should take exception_type value into account here. if child.getIntegerValue() is None: return True step = self.getStep() # false alarm, pylint: disable=assignment-from-none # A step of 0 will raise. if step is not None and step.getIntegerValue() == 0: return True return False def computeBuiltinSpec(self, trace_collection, given_values): assert self.builtin_spec is not None, self if not self.builtin_spec.isCompileTimeComputable(given_values): trace_collection.onExceptionRaiseExit(BaseException) # TODO: Raise exception known step 0. return self, None, None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin_spec.simulateCall( given_values ), description = "Built-in call to '%s' computed." % ( self.builtin_spec.getName() ) ) def computeExpressionIter1(self, iter_node, trace_collection): # No exception will be raised on xrange iteration. return iter_node, None, None @staticmethod def getLow(): return None @staticmethod def getHigh(): return None @staticmethod def getStep(): return None def mayBeNone(self): return False class ExpressionBuiltinXrange1(ExpressionBuiltinXrangeBase): kind = "EXPRESSION_BUILTIN_XRANGE1" named_children = ( "low", ) def __init__(self, low, source_ref): ExpressionBuiltinXrangeBase.__init__( self, values = { "low" : low, }, source_ref = source_ref ) def computeExpression(self, trace_collection): low = self.getLow() return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( low, ) ) getLow = ExpressionChildrenHavingBase.childGetter("low") def getIterationLength(self): low = self.getLow().getIntegerValue() if low is None: return None return max(0, low) def getIterationValue(self, element_index): length = self.getIterationLength() if length is None: return None if element_index > length: return None # TODO: Make sure to cast element_index to what CPython will give, for # now a downcast will do. return makeConstantReplacementNode( constant = int(element_index), node = self ) class ExpressionBuiltinXrange2(ExpressionBuiltinXrangeBase): kind = "EXPRESSION_BUILTIN_XRANGE2" named_children = ( "low", "high", ) def __init__(self, low, high, source_ref): ExpressionBuiltinXrangeBase.__init__( self, values = { "low" : low, "high" : high, }, source_ref = source_ref ) def computeExpression(self, trace_collection): low = self.getLow() high = self.getHigh() return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( low, high ) ) def getIterationLength(self): low = self.getLow() high = self.getHigh() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None return max(0, high - low) def getIterationValue(self, element_index): low = self.getLow() high = self.getHigh() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None result = low + element_index if result >= high: return None else: return makeConstantReplacementNode( constant = result, node = self ) getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") class ExpressionBuiltinXrange3(ExpressionBuiltinXrangeBase): kind = "EXPRESSION_BUILTIN_XRANGE3" named_children = ( "low", "high", "step" ) def __init__(self, low, high, step, source_ref): ExpressionBuiltinXrangeBase.__init__( self, values = { "low" : low, "high" : high, "step" : step }, source_ref = source_ref ) def computeExpression(self, trace_collection): low = self.getLow() high = self.getHigh() step = self.getStep() return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = ( low, high, step ) ) getLow = ExpressionChildrenHavingBase.childGetter("low") getHigh = ExpressionChildrenHavingBase.childGetter("high") getStep = ExpressionChildrenHavingBase.childGetter("step") def getIterationLength(self): low = self.getLow() high = self.getHigh() step = self.getStep() low = low.getIntegerValue() if low is None: return None high = high.getIntegerValue() if high is None: return None step = step.getIntegerValue() if step is None: return None # Give up on this, will raise ValueError. if step == 0: return None if low < high: if step < 0: estimate = 0 else: estimate = math.ceil(float(high - low) / step) else: if step > 0: estimate = 0 else: estimate = math.ceil(float(high - low) / step) estimate = round(estimate) assert estimate >= 0 return int(estimate) def getIterationValue(self, element_index): low = self.getLow().getIntegerValue() if low is None: return None high = self.getHigh().getIntegerValue() if high is None: return None step = self.getStep().getIntegerValue() result = low + step * element_index if result >= high: return None else: return makeConstantReplacementNode( constant = result, node = self ) def makeExpressionBuiltinXrange(low, high, step, source_ref): if high is None: return ExpressionBuiltinXrange1( low = low, source_ref = source_ref ) elif step is None: return ExpressionBuiltinXrange2( low = low, high = high, source_ref = source_ref ) else: return ExpressionBuiltinXrange3( low = low, high = high, step = step, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/nodes/BuiltinIteratorNodes.py0000644000372000001440000002652213207540035022557 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in iterator nodes. These play a role in for loops, and in unpacking. They can something be predicted to succeed or fail, in which case, code can become less complex. The length of things is an important optimization issue for these to be good. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.PythonVersions import python_version from .BuiltinRefNodes import ExpressionBuiltinExceptionRef from .ConstantRefNodes import makeConstantRefNode from .ExceptionNodes import StatementRaiseExceptionImplicit from .ExpressionBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase ) from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import wrapExpressionWithSideEffects from .shapes.StandardShapes import ShapeIterator class ExpressionBuiltinIter1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ITER1" def computeExpression(self, trace_collection): trace_collection.initIteratorValue(self) value = self.getValue() return value.computeExpressionIter1( iter_node = self, trace_collection = trace_collection ) def computeExpressionIter1(self, iter_node, trace_collection): # Iteration over an iterator is that iterator. return ( self, "new_builtin", "Eliminated useless iterator creation." ) def getTypeShape(self): return self.getValue().getTypeShape().getShapeIter() def computeExpressionNext1(self, next_node, trace_collection): value = self.getValue() if value.isKnownToBeIterableAtMin(1) and \ value.canPredictIterationValues(): result = wrapExpressionWithSideEffects( new_node = value.getIterationValue(0), old_node = value, side_effects = value.getIterationValueRange(1,None) ) return result, "new_expression", "Pridicted 'next' value from iteration." self.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return next_node, None, None def isKnownToBeIterable(self, count): if count is None: return True iter_length = self.getValue().getIterationLength() return iter_length == count def isKnownToBeIterableAtMin(self, count): assert type(count) is int iter_length = self.getValue().getIterationMinLength() return iter_length is not None and count <= iter_length def isKnownToBeIterableAtMax(self, count): assert type(count) is int iter_length = self.getValue().getIterationMaxLength() return iter_length is not None and count <= iter_length def getIterationLength(self): return self.getValue().getIterationLength() def canPredictIterationValues(self): return self.getValue().canPredictIterationValues() def getIterationValue(self, element_index): return self.getValue().getIterationValue(element_index) def extractSideEffects(self): # Iterator making is the side effect itself. value = self.getValue() if value.isCompileTimeConstant() and value.isKnownToBeIterable(None): return () else: return (self,) def mayHaveSideEffects(self): if self.getValue().isCompileTimeConstant(): return not self.getValue().isKnownToBeIterable(None) return True def mayRaiseException(self, exception_type): value = self.getValue() if value.mayRaiseException(exception_type): return True if value.isKnownToBeIterable(None): return False return True def onRelease(self, trace_collection): # print "onRelease", self pass class StatementSpecialUnpackCheck(StatementChildrenHavingBase): kind = "STATEMENT_SPECIAL_UNPACK_CHECK" named_children = ( "iterator", ) def __init__(self, iterator, count, source_ref): StatementChildrenHavingBase.__init__( self, values = { "iterator" : iterator }, source_ref = source_ref ) self.count = int(count) def getDetails(self): return { "count" : self.getCount(), } def getCount(self): return self.count getIterator = StatementChildrenHavingBase.childGetter("iterator") def computeStatement(self, trace_collection): trace_collection.onExpression(self.getIterator()) iterator = self.getIterator() if iterator.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if iterator.willRaiseException(BaseException): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = iterator, node = self ) return result, "new_raise", """\ Explicit raise already raises implicitly building exception type.""" if iterator.isExpressionTempVariableRef() and \ iterator.variable_trace.isAssignTrace(): iterator = iterator.variable_trace.getAssignNode().getAssignSource() current_index = trace_collection.getIteratorNextCount(iterator) else: current_index = None if current_index is not None: iter_length = iterator.getIterationLength() if iter_length is not None: # Remove the check if it can be decided at compile time. if current_index == iter_length: return None, "new_statements", """\ Determined iteration end check to be always true.""" else: source_ref = self.source_ref result = StatementRaiseExceptionImplicit( exception_type = ExpressionBuiltinExceptionRef( exception_name = "ValueError", source_ref = source_ref ), exception_value = makeConstantRefNode( constant = "too many values to unpack" if python_version < 300 else "too many values to unpack (expected %d)" % self.getCount(), source_ref = source_ref ), exception_cause = None, exception_trace = None, source_ref = source_ref ) trace_collection.onExceptionRaiseExit( TypeError ) return result, "new_raise", """\ Determined iteration end check to always raise.""" trace_collection.onExceptionRaiseExit( BaseException ) return self, None, None class ExpressionBuiltinIter2(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_ITER2" named_children = ( "callable", "sentinel", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, callable_arg, sentinel, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "callable" : callable_arg, "sentinel" : sentinel, }, source_ref = source_ref ) getCallable = ExpressionChildrenHavingBase.childGetter("callable") getSentinel = ExpressionChildrenHavingBase.childGetter("sentinel") def getTypeShape(self): # TODO: This could be more specific. return ShapeIterator def computeExpression(self, trace_collection): # TODO: The "callable" should be investigated here, maybe it is not # really callable, or raises an exception. return self, None, None def computeExpressionIter1(self, iter_node, trace_collection): return self, "new_builtin", "Eliminated useless iterator creation." class ExpressionAsyncIter(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_ASYNC_ITER" def computeExpression(self, trace_collection): value = self.getValue() return value.computeExpressionAsyncIter( iter_node = self, trace_collection = trace_collection ) def isKnownToBeIterable(self, count): if count is None: return True # TODO: Should ask value if it is. return None def getIterationLength(self): return self.getValue().getIterationLength() def extractSideEffects(self): # Iterator making is the side effect itself. if self.getValue().isCompileTimeConstant(): return () else: return (self,) def mayHaveSideEffects(self): if self.getValue().isCompileTimeConstant(): return self.getValue().isKnownToBeIterable(None) return True def mayRaiseException(self, exception_type): value = self.getValue() if value.mayRaiseException(exception_type): return True if value.isKnownToBeIterable(None): return False return True def isKnownToBeIterableAtMin(self, count): assert type(count) is int iter_length = self.getValue().getIterationMinLength() return iter_length is not None and iter_length < count def isKnownToBeIterableAtMax(self, count): assert type(count) is int iter_length = self.getValue().getIterationMaxLength() return iter_length is not None and count <= iter_length def onRelease(self, trace_collection): # print "onRelease", self pass class ExpressionAsyncNext(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_ASYNC_NEXT" def __init__(self, value, source_ref): ExpressionBuiltinSingleArgBase.__init__( self, value = value, source_ref = source_ref ) def computeExpression(self, trace_collection): # TODO: Predict iteration result if possible via SSA variable trace of # the iterator state. # Assume exception is possible. TODO: We might query the next from the # source with a computeExpressionAsyncNext slot, but we delay that. trace_collection.onExceptionRaiseExit(BaseException) return self, None, None Nuitka-0.5.28.2/nuitka/nodes/OperatorNodes.py0000644000372000001440000004314713134660221021233 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for unary and binary operations. No short-circuit involved, boolean 'not' is an unary operation like '-' is, no real difference. """ import math from nuitka import PythonOperators from .ExpressionBases import ExpressionChildrenHavingBase from .shapes.BuiltinTypeShapes import ShapeTypeTuple from .shapes.StandardShapes import ( ShapeLargeConstantValuePredictable, ShapeUnknown, vshape_unknown ) class ExpressionOperationBase(ExpressionChildrenHavingBase): inplace_suspect = False def __init__(self, operator, simulator, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = values, source_ref = source_ref ) self.operator = operator self.simulator = simulator def markAsInplaceSuspect(self): self.inplace_suspect = True def unmarkAsInplaceSuspect(self): self.inplace_suspect = False def isInplaceSuspect(self): return self.inplace_suspect def getDetail(self): return self.operator def getDetails(self): return { "operator" : self.operator } def getOperator(self): return self.operator def getSimulator(self): return self.simulator def isKnownToBeIterable(self, count): # TODO: Could be true, if the arguments said so return None class ExpressionOperationBinary(ExpressionOperationBase): kind = "EXPRESSION_OPERATION_BINARY" named_children = ("left", "right") nice_children = tuple(child_name + " operand" for child_name in named_children) def __init__(self, operator, left, right, source_ref): assert left.isExpression() and right.isExpression, (left, right) ExpressionOperationBase.__init__( self, operator = operator, simulator = PythonOperators.binary_operator_functions[ operator ], values = { "left" : left, "right" : right }, source_ref = source_ref ) @staticmethod def isExpressionOperationBinary(): return True def computeExpression(self, trace_collection): operator = self.getOperator() assert operator not in ("Mult", "Add") left = self.subnode_left right = self.subnode_right if left.isCompileTimeConstant() and right.isCompileTimeConstant(): left_value = left.getCompileTimeConstant() right_value = right.getCompileTimeConstant() return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( left_value, right_value ), description = "Operator '%s' with constant arguments." % operator ) # TODO: May go down to MemoryError for compile time constant overflow # ones. trace_collection.onExceptionRaiseExit(BaseException) # The value of these nodes escaped and could change its contents. trace_collection.removeKnowledge(left) trace_collection.removeKnowledge(right) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return self, None, None def getOperands(self): return (self.subnode_left, self.subnode_right) getLeft = ExpressionChildrenHavingBase.childGetter("left") getRight = ExpressionChildrenHavingBase.childGetter("right") class ExpressionOperationBinaryAdd(ExpressionOperationBinary): kind = "EXPRESSION_OPERATION_BINARY_ADD" def __init__(self, left, right, source_ref): ExpressionOperationBinary.__init__( self, operator = "Add", left = left, right = right, source_ref = source_ref ) def getDetails(self): return {} def computeExpression(self, trace_collection): # TODO: May go down to MemoryError for compile time constant overflow # ones. trace_collection.onExceptionRaiseExit(BaseException) operator = self.getOperator() left = self.subnode_left right = self.subnode_right if left.isCompileTimeConstant() and right.isCompileTimeConstant(): left_value = left.getCompileTimeConstant() right_value = right.getCompileTimeConstant() if left.isKnownToBeIterable(None) and \ right.isKnownToBeIterable(None): iter_length = left.getIterationLength() + \ right.getIterationLength() if iter_length > 256: return self, None, None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( left_value, right_value ), description = "Operator '%s' with constant arguments." % operator ) # The value of these nodes escaped and could change its contents. trace_collection.removeKnowledge(left) trace_collection.removeKnowledge(right) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return self, None, None class ExpressionOperationBinaryMult(ExpressionOperationBinary): kind = "EXPRESSION_OPERATION_BINARY_MULT" def __init__(self, left, right, source_ref): ExpressionOperationBinary.__init__( self, operator = "Mult", left = left, right = right, source_ref = source_ref ) self.shape = None def getDetails(self): return {} def getValueShape(self): if self.shape is not None: return self.shape else: return vshape_unknown def getTypeShape(self): if self.shape is not None: return self.shape.getTypeShape() else: return ShapeUnknown def getIterationLength(self): left_length = self.getLeft().getIterationLength() if left_length is not None: right_value = self.getRight().getIntegerValue() if right_value is not None: return left_length * right_value right_length = self.getRight().getIterationLength() if right_length is not None: left_value = self.getLeft().getIntegerValue() if left_value is not None: return right_length * left_value return ExpressionOperationBase.getIterationLength(self) def computeExpression(self, trace_collection): # TODO: May go down to MemoryError for compile time constant overflow # ones. trace_collection.onExceptionRaiseExit(BaseException) # Nothing to do anymore for large constants. if self.shape is not None and self.shape.isConstant(): return self, None, None left = self.subnode_left right = self.subnode_right if left.isCompileTimeConstant() and right.isCompileTimeConstant(): left_value = left.getCompileTimeConstant() right_value = right.getCompileTimeConstant() if right.isNumberConstant(): iter_length = left.getIterationLength() if iter_length is not None: size = iter_length * right_value if size > 256: self.shape = ShapeLargeConstantValuePredictable( size = size, predictor = None, # predictValuesFromRightAndLeftValue, shape = left.getTypeShape() ) return self, None, None if left.isNumberConstant(): if left.isIndexConstant() and right.isIndexConstant(): # Estimate with logarithm, if the result of number # calculations is computable with acceptable effort, # otherwise, we will have to do it at runtime. if left_value != 0 and right_value != 0: if math.log10(abs(left_value)) + math.log10(abs(right_value)) > 20: return self, None, None elif left.isNumberConstant(): iter_length = right.getIterationLength() if iter_length is not None: size = iter_length * left_value if iter_length * left_value > 256: self.shape = ShapeLargeConstantValuePredictable( size = size, predictor = None, # predictValuesFromRightAndLeftValue, shape = right.getTypeShape() ) return self, None, None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( left_value, right_value ), description = "Operator '*' with constant arguments." ) # The value of these nodes escaped and could change its contents. trace_collection.removeKnowledge(left) trace_collection.removeKnowledge(right) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return self, None, None def extractSideEffects(self): left_length = self.getLeft().getIterationLength() if left_length is not None: right_value = self.getRight().getIntegerValue() if right_value is not None: return self.getLeft().extractSideEffects() + self.getRight().extractSideEffects() right_length = self.getRight().getIterationLength() if right_length is not None: left_value = self.getLeft().getIntegerValue() if left_value is not None: return self.getLeft().extractSideEffects() + self.getRight().extractSideEffects() return ExpressionOperationBinary.extractSideEffects(self) class ExpressionOperationBinaryDivmod(ExpressionOperationBinary): kind = "EXPRESSION_OPERATION_BINARY_DIVMOD" def __init__(self, left, right, source_ref): ExpressionOperationBinary.__init__( self, operator = "Divmod", left = left, right = right, source_ref = source_ref ) self.shape = None # TODO: Value shape is two elemented tuple of int or float both. def getTypeShape(self): return ShapeTypeTuple def makeBinaryOperationNode(operator, left, right, source_ref): if operator == "Add": return ExpressionOperationBinaryAdd( left = left, right = right, source_ref = source_ref ) elif operator == "Mult": return ExpressionOperationBinaryMult( left = left, right = right, source_ref = source_ref ) else: # TODO: Add more specializations for common operators. return ExpressionOperationBinary( operator = operator, left = left, right = right, source_ref = source_ref ) class ExpressionOperationUnary(ExpressionOperationBase): kind = "EXPRESSION_OPERATION_UNARY" named_children = ("operand",) def __init__(self, operator, operand, source_ref): assert operand.isExpression(), operand ExpressionOperationBase.__init__( self, operator = operator, simulator = PythonOperators.unary_operator_functions[ operator ], values = { "operand" : operand }, source_ref = source_ref ) def computeExpression(self, trace_collection): operator = self.getOperator() operand = self.subnode_operand if operand.isCompileTimeConstant(): operand_value = operand.getCompileTimeConstant() return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.getSimulator()( operand_value, ), description = "Operator '%s' with constant argument." % operator ) else: # TODO: May go down to MemoryError for compile time constant overflow # ones. trace_collection.onExceptionRaiseExit(BaseException) # The value of that node escapes and could change its contents. trace_collection.removeKnowledge(operand) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return self, None, None getOperand = ExpressionChildrenHavingBase.childGetter("operand") def getOperands(self): return (self.getOperand(),) @staticmethod def isExpressionOperationUnary(): return True class ExpressionOperationNOT(ExpressionOperationUnary): kind = "EXPRESSION_OPERATION_NOT" def __init__(self, operand, source_ref): ExpressionOperationUnary.__init__( self, operator = "Not", operand = operand, source_ref = source_ref ) def getDetails(self): return {} def computeExpression(self, trace_collection): return self.getOperand().computeExpressionOperationNot( not_node = self, trace_collection = trace_collection ) def mayRaiseException(self, exception_type): return self.getOperand().mayRaiseException(exception_type) or \ self.getOperand().mayRaiseExceptionBool(exception_type) def mayRaiseExceptionBool(self, exception_type): return self.getOperand().mayRaiseExceptionBool(exception_type) def getTruthValue(self): result = self.getOperand().getTruthValue() # Need to invert the truth value of operand of course here. return None if result is None else not result def mayHaveSideEffects(self): operand = self.getOperand() if operand.mayHaveSideEffects(): return True return operand.mayHaveSideEffectsBool() def mayHaveSideEffectsBool(self): return self.getOperand().mayHaveSideEffectsBool() def extractSideEffects(self): operand = self.getOperand() # TODO: Find the common ground of these, and make it an expression # method. if operand.isExpressionMakeSequence(): return operand.extractSideEffects() if operand.isExpressionMakeDict(): return operand.extractSideEffects() return (self,) class ExpressionOperationBinaryInplace(ExpressionOperationBinary): kind = "EXPRESSION_OPERATION_BINARY_INPLACE" def __init__(self, operator, left, right, source_ref): ExpressionOperationBinary.__init__( self, operator = operator, left = left, right = right, source_ref = source_ref ) @staticmethod def isExpressionOperationBinary(): return True def computeExpression(self, trace_collection): # In-place operation requires extra care to avoid corruption of # values. left = self.getLeft() right = self.getRight() if left.isCompileTimeConstant(): # Then we made a mistake currently. assert not left.isMutable(), self source_ref = self.getSourceReference() result = makeBinaryOperationNode( operator = self.getOperator()[1:], left = left, right = right, source_ref = source_ref ) trace_collection.signalChange( tags = "new_expression", source_ref = source_ref, message = """\ Lowered in-place binary operation of compile time constant to binary operation.""" ) return result.computeExpression(trace_collection) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) # The value of these nodes escaped and could change its contents. trace_collection.removeKnowledge(left) trace_collection.removeKnowledge(right) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return self, None, None def makeExpressionOperationBinaryInplace(operator, left, right, source_ref): # TODO: Add more specializations for common operators. return ExpressionOperationBinaryInplace( operator = operator, left = left, right = right, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/nodes/TryNodes.py0000644000372000001440000004221213122472300020202 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for try/except/finally handling. This is the unified low level solution to trying a block, and executing code when it returns, break, continues, or raises an exception. See Developer Manual for how this maps to try/finally and try/except as in Python. """ from nuitka.optimizations.TraceCollections import TraceCollectionBranch from .Checkers import checkStatementsSequence, checkStatementsSequenceOrNone from .NodeBases import StatementChildrenHavingBase from .StatementNodes import StatementsSequence class StatementTry(StatementChildrenHavingBase): kind = "STATEMENT_TRY" named_children = ( "tried", "except_handler", "break_handler", "continue_handler", "return_handler" ) checkers = { "tried" : checkStatementsSequence, "except_handler" : checkStatementsSequenceOrNone, "break_handler" : checkStatementsSequenceOrNone, "continue_handler" : checkStatementsSequenceOrNone, "return_handler" : checkStatementsSequenceOrNone } def __init__(self, tried, except_handler, break_handler, continue_handler, return_handler, source_ref): StatementChildrenHavingBase.__init__( self, values = { "tried" : tried, "except_handler" : except_handler, "break_handler" : break_handler, "continue_handler" : continue_handler, "return_handler" : return_handler }, source_ref = source_ref ) getBlockTry = StatementChildrenHavingBase.childGetter( "tried" ) setBlockTry = StatementChildrenHavingBase.childSetter( "tried" ) getBlockExceptHandler = StatementChildrenHavingBase.childGetter( "except_handler" ) setBlockExceptHandler = StatementChildrenHavingBase.childSetter( "except_handler" ) getBlockBreakHandler = StatementChildrenHavingBase.childGetter( "break_handler" ) setBlockBreakHandler = StatementChildrenHavingBase.childSetter( "break_handler" ) getBlockContinueHandler = StatementChildrenHavingBase.childGetter( "continue_handler" ) setBlockContinueHandler = StatementChildrenHavingBase.childSetter( "continue_handler" ) getBlockReturnHandler = StatementChildrenHavingBase.childGetter( "return_handler" ) setBlockReturnHandler = StatementChildrenHavingBase.childSetter( "return_handler" ) def computeStatement(self, trace_collection): # This node has many children to handle, pylint: disable=too-many-branches,too-many-locals tried = self.getBlockTry() except_handler = self.getBlockExceptHandler() break_handler = self.getBlockBreakHandler() continue_handler = self.getBlockContinueHandler() return_handler = self.getBlockReturnHandler() # The tried block must be considered as a branch, if it is not empty # already. collection_start = TraceCollectionBranch( parent = trace_collection, name = "try start" ) abort_context = trace_collection.makeAbortStackContext( catch_breaks = break_handler is not None, catch_continues = continue_handler is not None, catch_returns = return_handler is not None, catch_exceptions = True, ) with abort_context: # As a branch point for the many types of handlers. result = tried.computeStatementsSequence( trace_collection = trace_collection ) # We might be done entirely already. if result is None: return None, "new_statements", "Removed now empty try statement." # Might be changed. if result is not tried: self.setBlockTry(result) tried = result break_collections = trace_collection.getLoopBreakCollections() continue_collections = trace_collection.getLoopContinueCollections() return_collections = trace_collection.getFunctionReturnCollections() exception_collections = trace_collection.getExceptionRaiseCollections() tried_may_raise = tried.mayRaiseException(BaseException) # Exception handling is useless if no exception is to be raised. # TODO: signal the change. if not tried_may_raise: if except_handler is not None: self.setBlockExceptHandler(None) except_handler = None # If tried may raise, even empty exception handler has a meaning to # ignore that exception. if tried_may_raise: collection_exception_handling = TraceCollectionBranch( parent = collection_start, name = "except handler" ) if not exception_collections: for statement in tried.getStatements(): if statement.mayRaiseException(BaseException): assert False, statement.asXmlText() assert False collection_exception_handling.mergeMultipleBranches(exception_collections) if except_handler is not None: result = except_handler.computeStatementsSequence( trace_collection = collection_exception_handling ) # Might be changed. if result is not except_handler: self.setBlockExceptHandler(result) except_handler = result if break_handler is not None: if not tried.mayBreak(): self.setBlockBreakHandler(None) break_handler = None if break_handler is not None: collection_break = TraceCollectionBranch( parent = collection_start, name = "break handler" ) collection_break.mergeMultipleBranches(break_collections) result = break_handler.computeStatementsSequence( trace_collection = collection_break ) # Might be changed. if result is not break_handler: self.setBlockBreakHandler(result) break_handler = result if continue_handler is not None: if not tried.mayContinue(): self.setBlockContinueHandler(None) continue_handler = None if continue_handler is not None: collection_continue = TraceCollectionBranch( parent = collection_start, name = "continue handler" ) collection_continue.mergeMultipleBranches(continue_collections) result = continue_handler.computeStatementsSequence( trace_collection = collection_continue ) # Might be changed. if result is not continue_handler: self.setBlockContinueHandler(result) continue_handler = result if return_handler is not None: if not tried.mayReturn(): self.setBlockReturnHandler(None) return_handler = None if return_handler is not None: collection_return = TraceCollectionBranch( parent = collection_start, name = "return handler" ) collection_return.mergeMultipleBranches(return_collections) result = return_handler.computeStatementsSequence( trace_collection = collection_return ) # Might be changed. if result is not return_handler: self.setBlockReturnHandler(result) return_handler = result if return_handler is not None: if return_handler.getStatements()[0].isStatementReturn() and \ return_handler.getStatements()[0].getExpression().isExpressionReturnedValueRef(): self.setBlockReturnHandler(None) return_handler = None # Merge exception handler only if it is used. Empty means it is not # aborting, as it swallows the exception. if tried_may_raise and ( except_handler is None or \ not except_handler.isStatementAborting() ): trace_collection.mergeBranches( collection_yes = collection_exception_handling, collection_no = None ) # An empty exception handler means we have to swallow exception. if not tried_may_raise and \ break_handler is None and \ continue_handler is None and \ return_handler is None: return tried, "new_statements", "Removed useless try, all handlers removed." tried_statements = tried.getStatements() pre_statements = [] while tried_statements: tried_statement = tried_statements[0] if tried_statement.mayRaiseException(BaseException): break if break_handler is not None and \ tried_statement.mayBreak(): break if continue_handler is not None and \ tried_statement.mayContinue(): break if return_handler is not None and \ tried_statement.mayReturn(): break pre_statements.append(tried_statement) tried_statements = list(tried_statements) del tried_statements[0] post_statements = [] if except_handler is not None and except_handler.isStatementAborting(): while tried_statements: tried_statement = tried_statements[-1] if tried_statement.mayRaiseException(BaseException): break if break_handler is not None and \ tried_statement.mayBreak(): break if continue_handler is not None and \ tried_statement.mayContinue(): break if return_handler is not None and \ tried_statement.mayReturn(): break post_statements.insert(0, tried_statement) tried_statements = list(tried_statements) del tried_statements[-1] if pre_statements or post_statements: assert tried_statements # Should be dealt with already tried.setStatements(tried_statements) result = StatementsSequence( statements = pre_statements + [self] + post_statements, source_ref = self.getSourceReference() ) def explain(): # TODO: We probably don't want to say this for re-formulation ones. result = "Reduced scope of tried block." if pre_statements: result += " Leading statements at %s." % ( ','.join( x.getSourceReference().getAsString() + '/' + str(x) for x in pre_statements ) ) if post_statements: result += " Trailing statements at %s." % ( ','.join( x.getSourceReference().getAsString() + '/' + str(x) for x in post_statements ) ) return result return ( result, "new_statements", explain ) return self, None, None def mayReturn(self): # TODO: If we optimized return handler away, this would be not needed # or even non-optimal. if self.getBlockTry().mayReturn(): return True except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.mayReturn(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.mayReturn(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.mayReturn(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.mayReturn(): return True return False def mayBreak(self): # TODO: If we optimized return handler away, this would be not needed # or even non-optimal. if self.getBlockTry().mayBreak(): return True except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.mayBreak(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.mayBreak(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.mayBreak(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.mayBreak(): return True return False def mayContinue(self): # TODO: If we optimized return handler away, this would be not needed # or even non-optimal. if self.getBlockTry().mayContinue(): return True except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.mayContinue(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.mayContinue(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.mayContinue(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.mayContinue(): return True return False def isStatementAborting(self): except_handler = self.getBlockExceptHandler() if except_handler is None or not except_handler.isStatementAborting(): return False break_handler = self.getBlockBreakHandler() if break_handler is not None and not break_handler.isStatementAborting(): return False continue_handler = self.getBlockContinueHandler() if continue_handler is not None and not continue_handler.isStatementAborting(): return False return_handler = self.getBlockReturnHandler() if return_handler is not None and not return_handler.isStatementAborting(): return False return self.getBlockTry().isStatementAborting() def mayRaiseException(self, exception_type): tried = self.getBlockTry() if tried.mayRaiseException(exception_type): except_handler = self.getBlockExceptHandler() if except_handler is not None and \ except_handler.mayRaiseException(exception_type): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and \ break_handler.mayRaiseException(exception_type): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and \ continue_handler.mayRaiseException(exception_type): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and \ return_handler.mayRaiseException(exception_type): return True return False def needsFrame(self): except_handler = self.getBlockExceptHandler() if except_handler is not None and except_handler.needsFrame(): return True break_handler = self.getBlockBreakHandler() if break_handler is not None and break_handler.needsFrame(): return True continue_handler = self.getBlockContinueHandler() if continue_handler is not None and continue_handler.needsFrame(): return True return_handler = self.getBlockReturnHandler() if return_handler is not None and return_handler.needsFrame(): return True return self.getBlockTry().needsFrame() Nuitka-0.5.28.2/nuitka/nodes/ParameterSpecs.py0000644000372000001440000003612513134660221021363 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module maintains the parameter specification classes. These are used for function, lambdas, generators. They are also a factory for the respective variable objects. One of the difficulty of Python and its parameter parsing is that they are allowed to be nested like this: (a,b), c Much like in assignments, which are very similar to parameters, except that parameters may also be assigned from a dictionary, they are no less flexible. """ from nuitka import Variables from nuitka.PythonVersions import python_version class TooManyArguments(Exception): def __init__(self, real_exception): Exception.__init__(self) self.real_exception = real_exception def getRealException(self): return self.real_exception class ParameterSpec(object): # These got many attributes, in part duplicating name and instance of # variables, pylint: disable=too-many-instance-attributes __slots__ = ( "name", "owner", "normal_args", "normal_variables", "list_star_arg", "dict_star_arg", "list_star_variable", "dict_star_variable", "default_count", "kw_only_args", "kw_only_variables" ) def __init__(self, ps_name, ps_normal_args, ps_kw_only_args, ps_list_star_arg, ps_dict_star_arg, ps_default_count): if type(ps_normal_args) is str: if ps_normal_args == "": ps_normal_args = () else: ps_normal_args = ps_normal_args.split(',') if type(ps_kw_only_args) is str: if ps_kw_only_args == "": ps_kw_only_args = () else: ps_kw_only_args = ps_kw_only_args.split(',') assert None not in ps_normal_args self.owner = None self.name = ps_name self.normal_args = tuple(ps_normal_args) self.normal_variables = None assert ps_list_star_arg is None or type(ps_list_star_arg) is str, \ ps_list_star_arg assert ps_dict_star_arg is None or type(ps_dict_star_arg) is str, \ ps_dict_star_arg self.list_star_arg = ps_list_star_arg if ps_list_star_arg else None self.dict_star_arg = ps_dict_star_arg if ps_dict_star_arg else None self.list_star_variable = None self.dict_star_variable = None self.default_count = ps_default_count self.kw_only_args = tuple(ps_kw_only_args) self.kw_only_variables = None def makeClone(self): return ParameterSpec( ps_name = self.name, ps_normal_args = self.normal_args, ps_kw_only_args = self.kw_only_args, ps_list_star_arg = self.list_star_arg, ps_dict_star_arg = self.dict_star_arg, ps_default_count = self.default_count ) def getDetails(self): return { "ps_name" : self.name, "ps_normal_args" : ','.join(self.normal_args), "ps_kw_only_args" : ','.join(self.kw_only_args), "ps_list_star_arg" : self.list_star_arg if self.list_star_arg is not None else "", "ps_dict_star_arg" : self.dict_star_arg if self.dict_star_arg is not None else "", "ps_default_count" : self.default_count } def checkParametersValid(self): arg_names = self.getParameterNames() # Check for duplicate arguments, could happen. for arg_name in arg_names: if arg_names.count(arg_name) != 1: return "duplicate argument '%s' in function definition" % arg_name return None def __repr__(self): parts = [ str(normal_arg) for normal_arg in self.normal_args ] if self.list_star_arg is not None: parts.append("*%s" % self.list_star_arg) if self.dict_star_variable is not None: parts.append("**%s" % self.dict_star_variable) if parts: return "" % ','.join(parts) else: return "" def getArgumentCount(self): return len(self.normal_args) def setOwner(self, owner): if self.owner is not None: return self.owner = owner self.normal_variables = [] for normal_arg in self.normal_args: if type(normal_arg) is str: normal_variable = Variables.ParameterVariable( owner = self.owner, parameter_name = normal_arg ) else: assert False, normal_arg self.normal_variables.append(normal_variable) if self.list_star_arg: self.list_star_variable = Variables.ParameterVariable( owner = owner, parameter_name = self.list_star_arg ) else: self.list_star_variable = None if self.dict_star_arg: self.dict_star_variable = Variables.ParameterVariable( owner = owner, parameter_name = self.dict_star_arg ) else: self.dict_star_variable = None self.kw_only_variables = [ Variables.ParameterVariable( owner = self.owner, parameter_name = kw_only_arg ) for kw_only_arg in self.kw_only_args ] def getDefaultCount(self): return self.default_count def hasDefaultParameters(self): return self.getDefaultCount() > 0 def getTopLevelVariables(self): return self.normal_variables + self.kw_only_variables def getAllVariables(self): result = list(self.normal_variables) result += self.kw_only_variables if self.list_star_variable is not None: result.append(self.list_star_variable) if self.dict_star_variable is not None: result.append(self.dict_star_variable) return result def getParameterNames(self): result = list(self.normal_args) result += self.kw_only_args if self.list_star_arg is not None: result.append(self.list_star_arg) if self.dict_star_arg is not None: result.append(self.dict_star_arg) return result def getStarListArgumentName(self): return self.list_star_arg def getListStarArgVariable(self): return self.list_star_variable def getStarDictArgumentName(self): return self.dict_star_arg def getDictStarArgVariable(self): return self.dict_star_variable def getKwOnlyVariables(self): return self.kw_only_variables def allowsKeywords(self): # Abstract method, pylint: disable=no-self-use return True def getKeywordRefusalText(self): return "%s() takes no keyword arguments" % self.name def getArgumentNames(self): return self.normal_args def getKwOnlyParameterNames(self): return self.kw_only_args def getKwOnlyParameterCount(self): return len(self.kw_only_args) # Note: Based loosely on "inspect.getcallargs" with corrections. def matchCall(func_name, args, star_list_arg, star_dict_arg, num_defaults, positional, pairs, improved = False): # This is of incredible code complexity, but there really is no other way to # express this with less statements, branches, or variables. # pylint: disable=too-many-branches,too-many-locals,too-many-statements assert type(positional) is tuple, positional assert type(pairs) in (tuple, list), pairs # Make a copy, we are going to modify it. pairs = list(pairs) result = {} assigned_tuple_params = [] def assign(arg, value): if type(arg) is str: # Normal case: result[ arg ] = value else: # Tuple argument case: assigned_tuple_params.append(arg) value = iter(value.getIterationValues()) for i, subarg in enumerate(arg): try: subvalue = next(value) except StopIteration: raise TooManyArguments( ValueError( "need more than %d %s to unpack" % ( i, "values" if i > 1 else "value" ) ) ) # Recurse into tuple argument values, could be more tuples. assign(subarg, subvalue) # Check that not too many values we provided. try: next(value) except StopIteration: pass else: raise TooManyArguments( ValueError("too many values to unpack") ) def isAssigned(arg): if type(arg) is str: return arg in result return arg in assigned_tuple_params num_pos = len(positional) num_total = num_pos + len(pairs) num_args = len(args) for arg, value in zip(args, positional): assign(arg, value) # Python3 does this check earlier. if python_version >= 300 and not star_dict_arg: for pair in pairs: if pair[0] not in args: message = "'%s' is an invalid keyword argument for this function" % pair[0] raise TooManyArguments( TypeError(message) ) if star_list_arg: if num_pos > num_args: assign(star_list_arg, positional[ -(num_pos-num_args) : ]) else: assign(star_list_arg, ()) elif 0 < num_args < num_total: if num_defaults == 0: if num_args == 1: raise TooManyArguments( TypeError( "%s() takes exactly one argument (%d given)" % ( func_name, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s expected %d arguments, got %d" % ( func_name, num_args, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s() takes at most %d %s (%d given)" % ( func_name, num_args, "argument" if num_args == 1 else "arguments", num_total ) ) ) elif num_args == 0 and num_total: if star_dict_arg: if num_pos: # Could use num_pos, but Python also uses num_total. raise TooManyArguments( TypeError( "%s() takes exactly 0 arguments (%d given)" % ( func_name, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s() takes no arguments (%d given)" % ( func_name, num_total ) ) ) named_argument_names = [ pair[0] for pair in pairs ] for arg in args: if type(arg) is str and arg in named_argument_names: if isAssigned(arg): raise TooManyArguments( TypeError( "%s() got multiple values for keyword argument '%s'" % ( func_name, arg ) ) ) else: new_pairs = [] for pair in pairs: if arg == pair[0]: assign(arg, pair[1]) else: new_pairs.append(pair) assert len(new_pairs) == len(pairs) - 1 pairs = new_pairs # Fill in any missing values with the None to indicate "default". if num_defaults > 0: for arg in args[ -num_defaults : ]: if not isAssigned(arg): assign(arg, None) if star_dict_arg: assign(star_dict_arg, pairs) elif pairs: unexpected = next(iter(dict(pairs))) if improved: message = "%s() got an unexpected keyword argument '%s'" % ( func_name, unexpected ) else: message = "'%s' is an invalid keyword argument for this function" % unexpected raise TooManyArguments( TypeError(message) ) unassigned = num_args - len( [ arg for arg in args if isAssigned(arg) ] ) if unassigned: num_required = num_args - num_defaults if num_required > 0 or improved: if num_defaults == 0 and num_args != 1: raise TooManyArguments( TypeError( "%s expected %d arguments, got %d" % ( func_name, num_args, num_total ) ) ) if num_required == 1: arg_desc = "1 argument" if python_version < 350 else "one argument" else: arg_desc = "%d arguments" % num_required raise TooManyArguments( TypeError( "%s() takes %s %s (%d given)" % ( func_name, "at least" if num_defaults > 0 else "exactly", arg_desc, num_total ) ) ) else: raise TooManyArguments( TypeError( "%s expected %s%s, got %d" % ( func_name, ( "at least " if python_version < 300 else "" ) if num_defaults > 0 else "exactly ", "%d arguments" % num_required, num_total ) ) ) return result Nuitka-0.5.28.2/nuitka/nodes/VariableRefNodes.py0000644000372000001440000005227113207537242021627 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for variable references. These represent all variable references in the node tree. Can be in assignments and its expressions, changing the meaning of course dramatically. """ from nuitka import Builtins, Variables from nuitka.ModuleRegistry import getOwnerFromCodeName from .ConstantRefNodes import makeConstantRefNode from .DictionaryNodes import ( ExpressionDictOperationGet, ExpressionDictOperationIn, ExpressionDictOperationNOTIn, StatementDictOperationRemove, StatementDictOperationSet ) from .ExpressionBases import ExpressionBase from .NodeMakingHelpers import makeRaiseExceptionReplacementExpression from .shapes.StandardShapes import ShapeUnknown class ExpressionVariableNameRef(ExpressionBase): """ These are used before the actual variable object is known from VariableClosure. """ kind = "EXPRESSION_VARIABLE_NAME_REF" __slots__ = ("variable_name",) def __init__(self, variable_name, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) self.variable_name = variable_name def getDetails(self): return { "variable_name" : self.variable_name } def getVariableName(self): return self.variable_name def computeExpressionRaw(self, trace_collection): return self, None, None class ExpressionVariableRefBase(ExpressionBase): # Base classes can be abstract, pylint: disable=abstract-method __slots__ = "variable", "variable_trace" def __init__(self, variable, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) self.variable = variable self.variable_trace = None def getVariableName(self): return self.variable.getName() def getVariable(self): return self.variable def getVariableVersion(self): return self.variable_trace.getVersion() class ExpressionVariableRef(ExpressionVariableRefBase): kind = "EXPRESSION_VARIABLE_REF" __slots__ = () def __init__(self, variable, source_ref): assert variable is not None ExpressionVariableRefBase.__init__( self, variable = variable, source_ref = source_ref ) def getDetails(self): return { "variable" : self.variable } def getDetailsForDisplay(self): return { "variable_name" : self.variable.getName(), "owner" : self.variable.getOwner().getCodeName() } @classmethod def fromXML(cls, provider, source_ref, **args): assert cls is ExpressionVariableRef, cls owner = getOwnerFromCodeName(args["owner"]) variable = owner.getProvidedVariable(args["variable_name"]) return cls( variable = variable, source_ref = source_ref ) def getDetail(self): if self.variable is None: return self.variable_name else: return repr(self.variable) @staticmethod def isTargetVariableRef(): return False def getVariableName(self): return self.variable_name def getVariable(self): return self.variable def setVariable(self, variable): assert isinstance(variable, Variables.Variable), repr(variable) self.variable = variable def getTypeShape(self): if self.variable_trace.isAssignTrace(): return self.variable_trace.getAssignNode().getAssignSource().getTypeShape() else: return ShapeUnknown def computeExpressionRaw(self, trace_collection): variable = self.variable assert variable is not None self.variable_trace = trace_collection.getVariableCurrentTrace( variable = variable ) replacement = self.variable_trace.getReplacementNode(self) if replacement is not None: trace_collection.signalChange( "new_expression", self.source_ref, "Value propagated for '%s' from '%s'." % ( variable.getName(), replacement.getSourceReference().getAsString() ) ) # Need to compute the replacement still. return replacement.computeExpressionRaw(trace_collection) if not self.variable_trace.mustHaveValue(): # TODO: This could be way more specific surely. trace_collection.onExceptionRaiseExit( BaseException ) if variable.isModuleVariable() and \ variable.hasDefiniteWrites() is False: variable_name = self.variable.getName() if variable_name in Builtins.builtin_exception_names: from .BuiltinRefNodes import ExpressionBuiltinExceptionRef new_node = ExpressionBuiltinExceptionRef( exception_name = self.variable.getName(), source_ref = self.getSourceReference() ) change_tags = "new_builtin_ref" change_desc = """\ Module variable '%s' found to be built-in exception reference.""" % ( variable_name ) elif variable_name in Builtins.builtin_names and \ variable_name != "pow": from .BuiltinRefNodes import makeExpressionBuiltinRef new_node = makeExpressionBuiltinRef( builtin_name = variable_name, source_ref = self.getSourceReference() ) change_tags = "new_builtin_ref" change_desc = """\ Module variable '%s' found to be built-in reference.""" % ( variable_name ) elif variable_name == "__name__": new_node = makeConstantRefNode( constant = variable.getOwner().getParentModule().\ getFullName(), source_ref = self.getSourceReference() ) change_tags = "new_constant" change_desc = """\ Replaced read-only module attribute '__name__' with constant value.""" else: self.variable_trace.addUsage() # Probably should give a warning once about it. new_node = self change_tags = None change_desc = None return new_node, change_tags, change_desc self.variable_trace.addUsage() if self.variable_trace.mustNotHaveValue(): assert self.variable.isLocalVariable(), self.variable variable_name = self.variable.getName() result = makeRaiseExceptionReplacementExpression( expression = self, exception_type = "UnboundLocalError", exception_value = """\ local variable '%s' referenced before assignment""" % variable_name ) return result, "new_raise", "Variable access of not initialized variable '%s'" % variable_name return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) trace_collection.onControlFlowEscape(self) if not Variables.complete and \ self.variable.getName() in ("dir", "eval", "exec", "execfile", "locals", "vars") and \ self.variable.isModuleVariable(): # Just inform the collection that all escaped. trace_collection.onLocalsUsage(self.getParentVariableProvider()) return call_node, None, None def computeExpressionSetSubscript(self, set_node, subscript, value_node, trace_collection): tags = None message = None # By default, an subscript may change everything about the lookup # source. if self.variable_trace.hasShapeDictionaryExact(): set_node = StatementDictOperationSet( dict_arg = self, key = subscript, value = value_node, source_ref = set_node.getSourceReference() ) tags = "new_statements" message = """\ Subscript assignment to dictionary lowered to dictionary assignment.""" # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception might be raised. if set_node.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) return set_node, tags, message def computeExpressionDelSubscript(self, del_node, subscript, trace_collection): tags = None message = None # By default, an subscript may change everything about the lookup # source. # Any code could be run, note that. trace_collection.onControlFlowEscape(self) if self.variable_trace.hasShapeDictionaryExact(): del_node = StatementDictOperationRemove( dict_arg = self, key = subscript, source_ref = del_node.getSourceReference() ) tags = "new_statements" message = """\ Subscript del to dictionary lowered to dictionary del.""" # Any exception might be raised. if del_node.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) return del_node, tags, message def computeExpressionSubscript(self, lookup_node, subscript, trace_collection): tags = None message = None # Any code could be run, note that. trace_collection.onControlFlowEscape(self) if self.variable_trace.hasShapeDictionaryExact(): lookup_node = ExpressionDictOperationGet( dict_arg = self, key = subscript, source_ref = lookup_node.getSourceReference() ) tags = "new_expression" message = """\ Subscript look-up to dictionary lowered to dictionary look-up.""" # Any exception might be raised. if lookup_node.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) return lookup_node, tags, message def computeExpressionComparisonIn(self, in_node, value_node, trace_collection): tags = None message = None # Any code could be run, note that. trace_collection.onControlFlowEscape(in_node) if self.variable_trace.hasShapeDictionaryExact(): tags = "new_expression" message = """\ Check '%s' on dictionary lowered to dictionary '%s'.""" % ( in_node.comparator, in_node.comparator ) if in_node.comparator == "In": in_node = ExpressionDictOperationIn( key = value_node, dict_arg = self, source_ref = in_node.getSourceReference() ) else: in_node = ExpressionDictOperationNOTIn( key = value_node, dict_arg = self, source_ref = in_node.getSourceReference() ) # Any exception may be raised. if in_node.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) return in_node, tags, message def hasShapeDictionaryExact(self): return self.variable_trace.hasShapeDictionaryExact() def onContentEscapes(self, trace_collection): trace_collection.onVariableContentEscapes(self.variable) def isKnownToBeIterable(self, count): return None def mayHaveSideEffects(self): return not self.variable_trace.mustHaveValue() def mayRaiseException(self, exception_type): variable_trace = self.variable_trace return variable_trace is None or not self.variable_trace.mustHaveValue() class ExpressionTempVariableRef(ExpressionVariableRefBase): kind = "EXPRESSION_TEMP_VARIABLE_REF" def __init__(self, variable, source_ref): assert variable.isTempVariable() ExpressionVariableRefBase.__init__( self, variable = variable, source_ref = source_ref ) def getDetailsForDisplay(self): return { "temp_name" : self.variable.getName(), "owner" : self.variable.getOwner().getCodeName() } def getDetails(self): return { "variable" : self.variable } @classmethod def fromXML(cls, provider, source_ref, **args): assert cls is ExpressionTempVariableRef, cls owner = getOwnerFromCodeName(args["owner"]) variable = owner.getTempVariable(None, args["temp_name"]) return cls( variable = variable, source_ref = source_ref ) def getDetail(self): return self.variable.getName() @staticmethod def isTargetVariableRef(): return False def getTypeShape(self): if self.variable_trace is None: return ShapeUnknown elif self.variable_trace.isAssignTrace(): return self.variable_trace.getAssignNode().getAssignSource().getTypeShape() else: return ShapeUnknown def computeExpressionRaw(self, trace_collection): self.variable_trace = trace_collection.getVariableCurrentTrace( variable = self.variable ) replacement = self.variable_trace.getReplacementNode(self) if replacement is not None: return replacement, "new_expression", "Value propagated for temp '%s'." % self.variable.getName() self.variable_trace.addUsage() # Nothing to do here. return self, None, None def computeExpressionNext1(self, next_node, trace_collection): if self.variable_trace.isAssignTrace(): value = self.variable_trace.getAssignNode().getAssignSource() current_index = trace_collection.getIteratorNextCount(value) trace_collection.onIteratorNext(value) if value.hasShapeSlotNext(): if current_index is not None and \ value.isKnownToBeIterableAtMin(current_index+1) and \ value.canPredictIterationValues(): # TODO: Make use of this, pylint: disable=W0125 candidate = value.getIterationValue(current_index) if False: return candidate, "new_expression", "Predicted 'next' value from iteration." else: # TODO: Could ask it about exception predictability for that case # or warn about it at least. pass # assert False, value self.onContentEscapes(trace_collection) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Any exception may be raised. trace_collection.onExceptionRaiseExit(BaseException) return next_node, None, None def onContentEscapes(self, trace_collection): trace_collection.onVariableContentEscapes(self.variable) def mayHaveSideEffects(self): # Can't happen with temporary variables. return False def mayRaiseException(self, exception_type): # Can't happen with temporary variables. return False def mayRaiseExceptionImportName(self, exception_type, import_name): if self.variable_trace is not None and \ self.variable_trace.isAssignTrace(): return self.variable_trace.getAssignNode().\ getAssignSource().mayRaiseExceptionImportName(exception_type, import_name) else: return ExpressionBase.mayRaiseExceptionImportName(self, exception_type, import_name) def isKnownToBeIterableAtMin(self, count): # TODO: See through the variable current trace. return None def isKnownToBeIterableAtMax(self, count): # TODO: See through the variable current trace. return None class ExpressionLocalsVariableRef(ExpressionBase): kind = "EXPRESSION_LOCALS_VARIABLE_REF" __slots__ = "variable_name", "fallback_variable", "variable_trace" def __init__(self, variable_name, fallback_variable, source_ref): self.variable_name = variable_name ExpressionBase.__init__( self, source_ref = source_ref ) self.fallback_variable = fallback_variable self.variable_trace = None def getDetails(self): return { "variable_name" : self.variable_name, } def getVariableName(self): return self.variable_name def getFallbackVariable(self): return self.fallback_variable def getFallbackVariableVersion(self): return self.variable_trace.getVersion() def computeExpressionRaw(self, trace_collection): # TODO: Use dictionary tracings for locals dict self.variable_trace = trace_collection.getVariableCurrentTrace( variable = self.fallback_variable ) if self.fallback_variable.isModuleVariable() and \ self.fallback_variable.hasDefiniteWrites() is False: if self.variable_name in Builtins.builtin_exception_names: from .BuiltinRefNodes import ExpressionBuiltinExceptionRef new_node = ExpressionBuiltinExceptionRef( exception_name = self.variable_name, source_ref = self.getSourceReference() ) change_tags = "new_builtin_ref" change_desc = """\ Module variable '%s' found to be built-in exception reference.""" % ( self.variable_name ) elif self.variable_name in Builtins.builtin_names and \ self.variable_name != "pow": from .BuiltinRefNodes import makeExpressionBuiltinRef new_node = makeExpressionBuiltinRef( builtin_name = self.variable_name, source_ref = self.getSourceReference() ) change_tags = "new_builtin_ref" change_desc = """\ Module variable '%s' found to be built-in reference.""" % ( self.variable_name ) elif self.variable_name == "__name__": new_node = makeConstantRefNode( constant = self.getParentModule().getFullName(), source_ref = self.getSourceReference() ) change_tags = "new_constant" change_desc = """\ Replaced read-only module attribute '__name__' with constant value.""" elif self.variable_name == "__package__": new_node = makeConstantRefNode( constant = self.getParentModule().getPackage(), source_ref = self.getSourceReference() ) change_tags = "new_constant" change_desc = """\ Replaced read-only module attribute '__package__' with constant value.""" else: # Probably should give a warning once about it. new_node = self change_tags = None change_desc = None if new_node is not self: return new_node, change_tags, change_desc # TODO: This could be way more specific surely, using dictionary tracing if not self.variable_trace.mustHaveValue(): trace_collection.onExceptionRaiseExit( BaseException ) self.variable_trace.addUsage() return self, None, None def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) trace_collection.onControlFlowEscape(self) if not Variables.complete and \ self.variable_name in ("dir", "eval", "exec", "execfile", "locals", "vars") and \ self.fallback_variable.isModuleVariable(): # Just inform the collection that all escaped. trace_collection.onLocalsUsage(self.getParentVariableProvider()) return call_node, None, None def mayRaiseException(self, exception_type): return self.variable_trace is None or not self.variable_trace.mustHaveValue() Nuitka-0.5.28.2/nuitka/nodes/LoopNodes.py0000644000372000001440000002162713112214770020350 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Loop nodes. There are for and loop nodes, but both are reduced to loops with break/continue statements for it. These re-formulations require that optimization of loops has to be very general, yet the node type for loop, becomes very simple. """ from nuitka.optimizations.TraceCollections import TraceCollectionBranch from nuitka.tree.Extractions import getVariablesWritten from .Checkers import checkStatementsSequenceOrNone from .NodeBases import NodeBase, StatementChildrenHavingBase class StatementLoop(StatementChildrenHavingBase): kind = "STATEMENT_LOOP" named_children = ( "body", ) checkers = { "body" : checkStatementsSequenceOrNone } def __init__(self, body, source_ref): StatementChildrenHavingBase.__init__( self, values = { "body" : body }, source_ref = source_ref ) self.loop_variables = None getLoopBody = StatementChildrenHavingBase.childGetter("body") setLoopBody = StatementChildrenHavingBase.childSetter("body") def mayReturn(self): loop_body = self.getLoopBody() if loop_body is not None and loop_body.mayReturn(): return True return False def mayBreak(self): # The loop itself may never break another loop. return False def mayContinue(self): # The loop itself may never continue another loop. return False def isStatementAborting(self): loop_body = self.getLoopBody() if loop_body is None: return True else: return not loop_body.mayBreak() def mayRaiseException(self, exception_type): # Loops can only raise, if their body does, but they also issue the # async exceptions, so we must make them do it all the time. return True # loop_body = self.getLoopBody() # return loop_body is not None and \ # self.getLoopBody().mayRaiseException(exception_type) def computeLoopBody(self, trace_collection): abort_context = trace_collection.makeAbortStackContext( catch_breaks = True, catch_continues = True, catch_returns = False, catch_exceptions = False, ) with abort_context: loop_body = self.getLoopBody() if loop_body is not None: # Look ahead. what will be written and degrade about that if we # are in the first iteration, later we will have more precise # knowledge. if self.loop_variables is None: self.loop_variables = getVariablesWritten( loop_body ) loop_entry_traces = set() # Mark all variables as loop wrap around that are written in # the loop and hit a 'continue'. for variable in self.loop_variables: loop_entry_traces.add( trace_collection.markActiveVariableAsLoopMerge( variable = variable ) ) # Forget all iterator and other value status. trace_collection.resetValueStates() result = loop_body.computeStatementsSequence( trace_collection = trace_collection ) # Might be changed. if result is not loop_body: self.setLoopBody(result) loop_body = result if loop_body is not None: # Emulate terminal continue if not aborting. if not loop_body.isStatementAborting(): trace_collection.onLoopContinue() continue_collections = trace_collection.getLoopContinueCollections() self.loop_variables = set() for loop_entry_trace in loop_entry_traces: variable = loop_entry_trace.getVariable() loop_end_traces = set() for continue_collection in continue_collections: loop_end_trace = continue_collection.getVariableCurrentTrace(variable) if loop_end_trace is not loop_entry_trace: loop_end_traces.add(loop_end_trace) if loop_end_traces: loop_entry_trace.addLoopContinueTraces(loop_end_traces) self.loop_variables.add(variable) # If we break, the outer collections becomes a merge of all those breaks # or just the one, if there is only one. break_collections = trace_collection.getLoopBreakCollections() return loop_body, break_collections def computeStatement(self, trace_collection): outer_trace_collection = trace_collection trace_collection = TraceCollectionBranch( parent = trace_collection, name = "loop" ) loop_body, break_collections = self.computeLoopBody(trace_collection) # Consider trailing "continue" statements, these have no effect, so we # can remove them. if loop_body is not None: assert loop_body.isStatementsSequence() statements = loop_body.getStatements() assert statements # Cannot be empty # If the last statement is a "continue" statement, it can simply # be discarded. last_statement = statements[-1] if last_statement.isStatementLoopContinue(): if len(statements) == 1: self.setLoopBody(None) loop_body = None else: last_statement.replaceWith(None) trace_collection.signalChange( "new_statements", last_statement.getSourceReference(), """\ Removed useless terminal 'continue' as last statement of loop.""" ) if break_collections: outer_trace_collection.mergeMultipleBranches(break_collections) # Consider leading "break" statements, they should be the only, and # should lead to removing the whole loop statement. Trailing "break" # statements could also be handled, but that would need to consider if # there are other "break" statements too. Numbering loop exits is # nothing we have yet. if loop_body is not None: assert loop_body.isStatementsSequence() statements = loop_body.getStatements() assert statements # Cannot be empty if len(statements) == 1 and statements[-1].isStatementLoopBreak(): return None, "new_statements", """\ Removed useless loop with immediate 'break' statement.""" # Also consider the threading intermission. TODO: We ought to make it # explicit, so we can see it potentially disrupting and changing the # global variables. It may also raise. outer_trace_collection.onExceptionRaiseExit(BaseException) return self, None, None class StatementLoopContinue(NodeBase): kind = "STATEMENT_LOOP_CONTINUE" def __init__(self, source_ref): NodeBase.__init__(self, source_ref = source_ref) def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return False def mayContinue(self): return True def computeStatement(self, trace_collection): # This statement being aborting, will already tell everything. trace_collection.onLoopContinue() return self, None, None class StatementLoopBreak(NodeBase): kind = "STATEMENT_LOOP_BREAK" def __init__(self, source_ref): NodeBase.__init__(self, source_ref = source_ref) def isStatementAborting(self): return True def mayRaiseException(self, exception_type): return False def mayBreak(self): return True def computeStatement(self, trace_collection): # This statement being aborting, will already tell everything. trace_collection.onLoopBreak() return self, None, None Nuitka-0.5.28.2/nuitka/nodes/PrintNodes.py0000644000372000001440000001435713112214770020535 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Print nodes. Right now there is only the print statement, but in principle, there should also be the print function here. These perform output, which can be combined if possible, and could be detected to fail, which would be perfect. Predicting the behavior of 'print' is not trivial at all, due to many special cases. """ from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, makeStatementsSequenceReplacementNode, wrapStatementWithSideEffects ) class StatementPrintValue(StatementChildrenHavingBase): kind = "STATEMENT_PRINT_VALUE" named_children = ( "dest", "value" ) def __init__(self, dest, value, source_ref): StatementChildrenHavingBase.__init__( self, values = { "value" : value, "dest" : dest }, source_ref = source_ref ) assert value is not None getDestination = StatementChildrenHavingBase.childGetter( "dest" ) getValue = StatementChildrenHavingBase.childGetter( "value" ) setValue = StatementChildrenHavingBase.childSetter( "value" ) def computeStatement(self, trace_collection): trace_collection.onExpression( expression = self.getDestination(), allow_none = True ) dest = self.getDestination() if dest is not None and dest.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if dest is not None and dest.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = dest, node = self ) return result, "new_raise", """\ Exception raise in 'print' statement destination converted to explicit raise.""" trace_collection.onExpression( expression = self.getValue() ) value = self.getValue() if value.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if value.willRaiseException(BaseException): if dest is not None: result = wrapStatementWithSideEffects( new_node = makeStatementExpressionOnlyReplacementNode( expression = value, node = self ), old_node = dest ) else: result = makeStatementExpressionOnlyReplacementNode( expression = value, node = self ) return result, "new_raise", """\ Exception raise in 'print' statement arguments converted to explicit raise.""" trace_collection.onExceptionRaiseExit( BaseException ) if dest is None: if value.isExpressionSideEffects(): self.setValue(value.getExpression()) statements = [ makeStatementExpressionOnlyReplacementNode( side_effect, self ) for side_effect in value.getSideEffects() ] statements.append(self) result = makeStatementsSequenceReplacementNode( statements = statements, node = self, ) return result, "new_statements", """\ Side effects printed item promoted to statements.""" if value.isCompileTimeConstant(): if not (value.isExpressionConstantRef() and \ value.isUnicodeConstant()): new_value = value.getStrValue() assert new_value is not None, value if value is not new_value: self.setValue(new_value) return self, None, None def mayRaiseException(self, exception_type): return True class StatementPrintNewline(StatementChildrenHavingBase): kind = "STATEMENT_PRINT_NEWLINE" named_children = ( "dest", ) def __init__(self, dest, source_ref): StatementChildrenHavingBase.__init__( self, values = { "dest" : dest }, source_ref = source_ref ) getDestination = StatementChildrenHavingBase.childGetter( "dest" ) def computeStatement(self, trace_collection): # TODO: Reactivate below optimizations for prints. trace_collection.onExpression( expression = self.getDestination(), allow_none = True ) dest = self.getDestination() if dest is not None and dest.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if dest is not None and dest.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = dest, node = self ) return result, "new_raise", """\ Exception raise in 'print' statement destination converted to explicit raise.""" trace_collection.onExceptionRaiseExit( BaseException ) return self, None, None def mayRaiseException(self, exception_type): return True Nuitka-0.5.28.2/nuitka/nodes/SideEffectNodes.py0000644000372000001440000000745413122472300021436 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node that models side effects. Sometimes, the effect of an expression needs to be had, but the value itself does not matter at all. """ from .ExpressionBases import ExpressionChildrenHavingBase from .NodeMakingHelpers import makeStatementOnlyNodesFromExpressions def checkSideEffects(value): real_value = [] for child in value: if child.isExpressionSideEffects(): real_value.extend(child.getSideEffects()) real_value.append(child.getExpression()) else: assert child.isExpression() real_value.append(child) return tuple(real_value) class ExpressionSideEffects(ExpressionChildrenHavingBase): kind = "EXPRESSION_SIDE_EFFECTS" named_children = ( "side_effects", "expression" ) checkers = { "side_effects" : checkSideEffects } def __init__(self, side_effects, expression, source_ref): # We expect to be not used without there actually being side effects. assert side_effects ExpressionChildrenHavingBase.__init__( self, values = { "side_effects" : tuple(side_effects), "expression" : expression }, source_ref = source_ref ) getSideEffects = ExpressionChildrenHavingBase.childGetter("side_effects") setSideEffects = ExpressionChildrenHavingBase.childSetter("side_effects") getExpression = ExpressionChildrenHavingBase.childGetter("expression") def isExpressionSideEffects(self): return True def computeExpression(self, trace_collection): side_effects = self.getSideEffects() new_side_effects = [] for side_effect in side_effects: if side_effect.mayHaveSideEffects(): new_side_effects.append(side_effect) expression = self.getExpression() if expression.isExpressionSideEffects(): new_side_effects.extend( expression.getSideEffects() ) expression.setSideEffects(new_side_effects) return expression, "new_expression", "Remove nested side effects." if not new_side_effects: return expression, "new_expression", "Removed empty side effects." if new_side_effects != side_effects: self.setSideEffects(new_side_effects) return self, None, None def willRaiseException(self, exception_type): for child in self.getVisitableNodes(): if child.willRaiseException(exception_type): return True return False def getTruthValue(self): return self.getExpression().getTruthValue() def computeExpressionDrop(self, statement, trace_collection): # Side effects can become statements. expressions = self.getSideEffects() + (self.getExpression(),) result = makeStatementOnlyNodesFromExpressions( expressions = expressions ) return result, "new_statements", """\ Turned side effects of expression only statement into statements.""" Nuitka-0.5.28.2/nuitka/nodes/SliceNodes.py0000644000372000001440000002427413122472300020473 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Slice nodes. Slices are important when working with lists. Tracking them can allow to achieve more compact code, or predict results at compile time. There will be a method "computeExpressionSlice" to aid predicting them. """ from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ConstantRefNodes import ExpressionConstantNoneRef from .ExpressionBases import ( ExpressionChildrenHavingBase, ExpressionSpecBasedComputationBase ) from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import ( convertNoneConstantToNone, makeStatementExpressionOnlyReplacementNode, makeStatementOnlyNodesFromExpressions ) class StatementAssignmentSlice(StatementChildrenHavingBase): kind = "STATEMENT_ASSIGNMENT_SLICE" named_children = ( "source", "expression", "lower", "upper" ) def __init__(self, expression, lower, upper, source, source_ref): assert python_version < 300 StatementChildrenHavingBase.__init__( self, values = { "source" : source, "expression" : expression, "lower" : lower, "upper" : upper }, source_ref = source_ref ) getLookupSource = StatementChildrenHavingBase.childGetter("expression") getLower = StatementChildrenHavingBase.childGetter("lower") getUpper = StatementChildrenHavingBase.childGetter("upper") getAssignSource = StatementChildrenHavingBase.childGetter("source") def computeStatement(self, trace_collection): trace_collection.onExpression(self.getAssignSource()) source = self.getAssignSource() # No assignment will occur, if the assignment source raises, so strip it # away. if source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_raise", """\ Slice assignment raises exception in assigned value, removed assignment.""" trace_collection.onExpression(self.getLookupSource()) lookup_source = self.getLookupSource() if lookup_source.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, lookup_source ) ) return result, "new_raise", """\ Slice assignment raises exception in sliced value, removed assignment.""" trace_collection.onExpression(self.getLower(), allow_none = True) lower = self.getLower() if lower is not None and lower.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, lookup_source, lower ) ) return result, "new_raise", """\ Slice assignment raises exception in lower slice boundary value, removed \ assignment.""" trace_collection.onExpression(self.getUpper(), allow_none = True) upper = self.getUpper() if upper is not None and upper.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source, lookup_source, lower, upper ) ) return result, "new_raise", """\ Slice assignment raises exception in upper slice boundary value, removed \ assignment.""" return lookup_source.computeExpressionSetSlice( set_node = self, lower = lower, upper = upper, value_node = source, trace_collection = trace_collection ) class StatementDelSlice(StatementChildrenHavingBase): kind = "STATEMENT_DEL_SLICE" named_children = ( "expression", "lower", "upper" ) def __init__(self, expression, lower, upper, source_ref): StatementChildrenHavingBase.__init__( self, values = { "expression" : expression, "lower" : lower, "upper" : upper }, source_ref = source_ref ) getLookupSource = StatementChildrenHavingBase.childGetter("expression") getLower = StatementChildrenHavingBase.childGetter("lower") getUpper = StatementChildrenHavingBase.childGetter("upper") def computeStatement(self, trace_collection): trace_collection.onExpression(self.getLookupSource()) lookup_source = self.getLookupSource() if lookup_source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = lookup_source, node = self ) return result, "new_raise", """\ Slice del raises exception in sliced value, removed del""" trace_collection.onExpression(self.getLower(), allow_none = True) lower = self.getLower() if lower is not None and lower.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( lookup_source, lower ) ) return result, "new_raise", """ Slice del raises exception in lower slice boundary value, removed del""" trace_collection.onExpression(self.getUpper(), allow_none = True) upper = self.getUpper() if upper is not None and upper.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( lookup_source, lower, upper ) ) return result, "new_raise", """ Slice del raises exception in upper slice boundary value, removed del""" return lookup_source.computeExpressionDelSlice( set_node = self, lower = lower, upper = upper, trace_collection = trace_collection ) class ExpressionSliceLookup(ExpressionChildrenHavingBase): kind = "EXPRESSION_SLICE_LOOKUP" named_children = ( "expression", "lower", "upper" ) checkers = { "upper" : convertNoneConstantToNone, "lower" : convertNoneConstantToNone } def __init__(self, expression, lower, upper, source_ref): assert python_version < 300 ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression, "upper" : upper, "lower" : lower }, source_ref = source_ref ) getLookupSource = ExpressionChildrenHavingBase.childGetter("expression") getLower = ExpressionChildrenHavingBase.childGetter("lower") setLower = ExpressionChildrenHavingBase.childSetter("lower") getUpper = ExpressionChildrenHavingBase.childGetter("upper") setUpper = ExpressionChildrenHavingBase.childSetter("upper") def computeExpression(self, trace_collection): lookup_source = self.getLookupSource() return lookup_source.computeExpressionSlice( lookup_node = self, lower = self.getLower(), upper = self.getUpper(), trace_collection = trace_collection ) def isKnownToBeIterable(self, count): # TODO: Should ask SliceRegistry return None class ExpressionBuiltinSlice(ExpressionSpecBasedComputationBase): kind = "EXPRESSION_BUILTIN_SLICE" named_children = ( "start", "stop", "step" ) builtin_spec = BuiltinOptimization.builtin_slice_spec def __init__(self, start, stop, step, source_ref): if start is None: start = ExpressionConstantNoneRef( source_ref = source_ref ) if stop is None: stop = ExpressionConstantNoneRef( source_ref = source_ref ) if step is None: step = ExpressionConstantNoneRef( source_ref = source_ref ) ExpressionSpecBasedComputationBase.__init__( self, values = { "start" : start, "stop" : stop, "step" : step }, source_ref = source_ref ) def computeExpression(self, trace_collection): start = self.getStart() stop = self.getStop() step = self.getStep() args = ( start, stop, step ) return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = args ) def mayRaiseException(self, exception_type): return self.getStart().mayRaiseException(exception_type) or \ self.getStop().mayRaiseException(exception_type) or \ self.getStep().mayRaiseException(exception_type) getStart = ExpressionSpecBasedComputationBase.childGetter("start") getStop = ExpressionSpecBasedComputationBase.childGetter("stop") getStep = ExpressionSpecBasedComputationBase.childGetter("step") Nuitka-0.5.28.2/nuitka/nodes/BuiltinFormatNodes.py0000644000372000001440000001033513122472300022204 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Format related nodes format/bin/oct/hex/ascii. These will most often be used for outputs, and the hope is, the type prediction or the result prediction will help to be smarter, but generally these should not be that much about performance critical. """ from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ExpressionBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase ) from .shapes.BuiltinTypeShapes import ( ShapeTypeIntOrLong, ShapeTypeStr, ShapeTypeStrOrUnicode ) class ExpressionBuiltinFormat(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_FORMAT" named_children = ( "value", "format_spec" ) def __init__(self, value, format_spec, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "value" : value, "format_spec" : format_spec }, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeStrOrUnicode def computeExpression(self, trace_collection): # TODO: Can use the format built-in on compile time constants at least. value = self.getValue() format_spec = self.getFormatSpec() # Go to default way if possible. if format_spec is not None and \ format_spec.isExpressionConstantStrRef() and \ not format_spec.getCompileTimeConstant(): self.setFormatSpec(None) format_spec = None # Strings format themselves as what they are. if format_spec is None: if value.hasShapeStrExact() or value.hasShapeUnicodeExact(): return ( value, "new_expression", """\ Removed useless 'format' on '%s' value.""" % value.getTypeShape().getTypeName() ) return self, None, None getValue = ExpressionChildrenHavingBase.childGetter("value") getFormatSpec = ExpressionChildrenHavingBase.childGetter("format_spec") setFormatSpec = ExpressionChildrenHavingBase.childSetter("format_spec") class ExpressionBuiltinAscii(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ASCII" if python_version >= 300: builtin_spec = BuiltinOptimization.builtin_ascii_spec def getTypeShape(self): return ShapeTypeStr class ExpressionBuiltinBin(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_BIN" builtin_spec = BuiltinOptimization.builtin_bin_spec def getTypeShape(self): return ShapeTypeStr class ExpressionBuiltinOct(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_OCT" builtin_spec = BuiltinOptimization.builtin_oct_spec def getTypeShape(self): return ShapeTypeStr class ExpressionBuiltinHex(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_HEX" builtin_spec = BuiltinOptimization.builtin_hex_spec def getTypeShape(self): return ShapeTypeStr class ExpressionBuiltinId(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_ID" builtin_spec = BuiltinOptimization.builtin_id_spec def computeExpression(self, trace_collection): # Note: Quite impossible to predict the pointer value or anything, but # we know the result will be a long. return self, None, None def getIntValue(self): return self def getTypeShape(self): return ShapeTypeIntOrLong Nuitka-0.5.28.2/nuitka/nodes/OutlineNodes.py0000644000372000001440000002575513207537242021073 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Outline nodes. We use them for re-formulations and for in-lining of code. They are expressions that get their value from return statements in their code body. They do not own anything by themselves. It's just a way of having try/finally for the expressions, or multiple returns, without running in a too different context. """ from .ExceptionNodes import ExpressionRaiseException from .ExpressionBases import ExpressionChildrenHavingBase from .FunctionNodes import ExpressionFunctionBodyBase class ExpressionOutlineBody(ExpressionChildrenHavingBase): """ Outlined expression code. This is for a call to a piece of code to be executed in a specific context. It contains an exclusively owned function body, that has no other references, and can be considered part of the calling context. It must return a value, to use as expression value. """ kind = "EXPRESSION_OUTLINE_BODY" named_children = ( "body", ) @staticmethod def isExpressionOutlineBody(): return True def __init__(self, provider, name, source_ref, body = None): assert name != "" ExpressionChildrenHavingBase.__init__( self, values = { "body" : body }, source_ref = source_ref ) self.provider = provider self.name = name self.temp_scope = None # Hack: This allows some APIs to work although this is not yet # officially a child yet. Important during building. self.parent = provider def getDetails(self): return { "provider" : self.provider, "name" : self.name } getBody = ExpressionChildrenHavingBase.childGetter("body") setBody = ExpressionChildrenHavingBase.childSetter("body") def getOutlineTempScope(self): # We use our own name as a temp_scope, cached from the parent, if the # scope is None. if self.temp_scope is None: self.temp_scope = self.provider.allocateTempScope(self.name) return self.temp_scope def allocateTempVariable(self, temp_scope, name): if temp_scope is None: temp_scope = self.getOutlineTempScope() return self.provider.allocateTempVariable( temp_scope = temp_scope, name = name ) def allocateTempScope(self, name): # Let's scope the temporary scopes by the outline they come from. return self.provider.allocateTempScope( name = self.name + '$' + name ) def getContainingClassDictCreation(self): return self.getParentVariableProvider().getContainingClassDictCreation() def computeExpressionRaw(self, trace_collection): owning_module = self.getParentModule() # Make sure the owning module is added to the used set. This is most # important for helper functions, or modules, which otherwise have # become unused. from nuitka.ModuleRegistry import addUsedModule addUsedModule(owning_module) abort_context = trace_collection.makeAbortStackContext( catch_breaks = False, catch_continues = False, catch_returns = True, catch_exceptions = False ) with abort_context: body = self.getBody() result = body.computeStatementsSequence( trace_collection = trace_collection ) if result is not body: self.setBody(result) body = result return_collections = trace_collection.getFunctionReturnCollections() if return_collections: trace_collection.mergeMultipleBranches(return_collections) first_statement = body.getStatements()[0] if first_statement.isStatementReturn(): return ( first_statement.getExpression(), "new_expression", "Outline '%s' is now simple return, use directly." % self.name ) if first_statement.isStatementRaiseException(): result = ExpressionRaiseException( exception_type = first_statement.getExceptionType(), exception_value = first_statement.getExceptionValue(), source_ref = first_statement.getSourceReference() ) return ( result, "new_expression", "Outline is now exception raise, use directly." ) # TODO: Function outline may become too trivial to outline and return # collections may tell us something. return self, None, None def mayRaiseException(self, exception_type): return self.getBody().mayRaiseException(exception_type) def willRaiseException(self, exception_type): return self.getBody().willRaiseException(exception_type) def getEntryPoint(self): """ Entry point for code. Normally ourselves. Only outlines will refer to their parent which technically owns them. """ return self.provider.getEntryPoint() def getCodeName(self): return self.provider.getCodeName() class ExpressionOutlineFunctionBodyBase(ExpressionFunctionBodyBase): """ Outlined function code. This is for a call to a function to be called in-line to be executed in a specific context. It contains an exclusively owned function body, that has no other references, and can be considered part of the calling context. As normal function it must return a value, to use as expression value, but we know we only exist once. Once this has no frame, it can be changed to a mere outline expression. """ def __init__(self, provider, name, source_ref, code_prefix = "outline", body = None): assert name != "" ExpressionFunctionBodyBase.__init__( self, provider = provider, name = name, code_prefix = code_prefix, flags = None, body = body, source_ref = source_ref ) self.temp_scope = None getBody = ExpressionFunctionBodyBase.childGetter("body") setBody = ExpressionFunctionBodyBase.childSetter("body") def isExpressionOutlineFunctionBodyBase(self): return True def getDetails(self): return { "name" : self.name, "provider" : self.provider.getCodeName() } def computeExpressionRaw(self, trace_collection): # Keep track of these, so they can provide what variables are to be # setup. trace_collection.addOutlineFunction(self) abort_context = trace_collection.makeAbortStackContext( catch_breaks = False, catch_continues = False, catch_returns = True, catch_exceptions = False ) with abort_context: body = self.getBody() result = body.computeStatementsSequence( trace_collection = trace_collection ) if result is not body: self.setBody(result) body = result return_collections = trace_collection.getFunctionReturnCollections() if return_collections: trace_collection.mergeMultipleBranches(return_collections) first_statement = body.getStatements()[0] if first_statement.isStatementReturn(): return ( first_statement.getExpression(), "new_expression", "Outline function '%s' is now simple return, use directly." % self.name ) if first_statement.isStatementRaiseException(): result = ExpressionRaiseException( exception_type = first_statement.getExceptionType(), exception_value = first_statement.getExceptionValue(), source_ref = first_statement.getSourceReference() ) return ( result, "new_expression", "Outline function is now exception raise, use directly." ) # TODO: Function outline may become too trivial to outline and return # collections may tell us something. return self, None, None def mayRaiseException(self, exception_type): return self.getBody().mayRaiseException(exception_type) def willRaiseException(self, exception_type): return self.getBody().willRaiseException(exception_type) def getTraceCollection(self): return self.provider.getTraceCollection() def getOutlineTempScope(self): # We use our own name as a temp_scope, cached from the parent, if the # scope is None. if self.temp_scope is None: self.temp_scope = self.provider.allocateTempScope(self.name) return self.temp_scope def allocateTempVariable(self, temp_scope, name): if temp_scope is None: temp_scope = self.getOutlineTempScope() return self.provider.allocateTempVariable( temp_scope = temp_scope, name = name ) def allocateTempScope(self, name): # Let's scope the temporary scopes by the outline they come from. return self.provider.allocateTempScope( name = self.name + '$' + name ) def getEntryPoint(self): """ Entry point for code. Normally ourselves. Only outlines will refer to their parent which technically owns them. """ return self.provider.getEntryPoint() def getCodeName(self): return self.provider.getCodeName() def getClosureVariable(self, variable_name): # Simply try and get from our parent. return self.provider.getVariableForReference( variable_name = variable_name ) class ExpressionOutlineFunction(ExpressionOutlineFunctionBodyBase): kind = "EXPRESSION_OUTLINE_FUNCTION" named_children = ( "body", ) def isEarlyClosure(self): return self.provider.isEarlyClosure() def hasLocalsDict(self): return self.provider.hasLocalsDict() def markAsLocalsDict(self): return self.provider.markAsLocalsDict() Nuitka-0.5.28.2/nuitka/nodes/NodeMakingHelpers.py0000644000372000001440000003252713207540035022007 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ These are just helpers to create nodes, often to replace existing nodes These are for use in optimizations and computations, and therefore cover mostly exceptions and constants. Often cyclic dependencies kicks in, which is why this module is mostly only imported locally. Note: It's intended to be reversed, this module will make the local imports instead, as these local imports look ugly everywhere else, making it more difficult to use. """ from logging import warning from nuitka.Builtins import builtin_names from nuitka.Constants import isConstant from nuitka.Options import isDebug, shallWarnImplicitRaises def makeConstantReplacementNode(constant, node): from .ConstantRefNodes import makeConstantRefNode return makeConstantRefNode( constant = constant, source_ref = node.getSourceReference() ) def makeRaiseExceptionReplacementExpression(expression, exception_type, exception_value): from .ExceptionNodes import ExpressionRaiseException from .BuiltinRefNodes import ExpressionBuiltinExceptionRef source_ref = expression.getSourceReference() assert type(exception_type) is str if shallWarnImplicitRaises(): warning( '%s: Will always raise exception: "%s(%s)"', expression.getSourceReference().getAsString(), exception_type, exception_value ) result = ExpressionRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = exception_type, source_ref = source_ref ), exception_value = makeConstantReplacementNode( constant = exception_value, node = expression ), source_ref = source_ref ) return result def makeRaiseExceptionReplacementExpressionFromInstance(expression, exception): assert isinstance(exception, Exception) args = exception.args if type(args) is tuple and len(args) == 1: value = args[0] else: assert type(args) is tuple value = args return makeRaiseExceptionReplacementExpression( expression = expression, exception_type = exception.__class__.__name__, exception_value = value ) def makeRaiseExceptionExpressionFromTemplate(exception_type, template, template_args, source_ref): from .ExceptionNodes import ExpressionRaiseException from .BuiltinRefNodes import ExpressionBuiltinExceptionRef from .OperatorNodes import makeBinaryOperationNode from .ConstantRefNodes import makeConstantRefNode return ExpressionRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = exception_type, source_ref = source_ref ), exception_value = makeBinaryOperationNode( operator = "Mod", left = makeConstantRefNode( constant = template, source_ref = source_ref, user_provided = True ), right = template_args, source_ref = source_ref ), source_ref = source_ref ) def makeRaiseTypeErrorExceptionReplacementFromTemplateAndValue(template, operation, original_node, value_node): shape = value_node.getTypeShape() type_name = shape.getTypeName() if type_name is not None: result = makeRaiseExceptionReplacementExpressionFromInstance( expression = original_node, exception = TypeError(template % type_name) ) result = wrapExpressionWithNodeSideEffects( new_node = result, old_node = value_node ) else: from .AttributeNodes import ExpressionAttributeLookup from .TypeNodes import ExpressionBuiltinType1 source_ref = original_node.getSourceReference() result = makeRaiseExceptionExpressionFromTemplate( exception_type = "TypeError", template = "object of type '%s' has no len()", template_args = ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = value_node.makeClone(), source_ref = source_ref ), attribute_name = "__name__", source_ref = source_ref ), source_ref = source_ref ) type_name = shape.__name__ return result, "new_raise", "Raising for use of '%s' on %s '%s'." % ( operation, "type" if type_name is not None else "shape", type_name ) def makeCompileTimeConstantReplacementNode(value, node): # This needs to match code in isCompileTimeConstantValue if isConstant(value): return makeConstantReplacementNode( constant = value, node = node ) elif type(value) is type: if value.__name__ in builtin_names: from .BuiltinRefNodes import makeExpressionBuiltinRef return makeExpressionBuiltinRef( builtin_name = value.__name__, source_ref = node.getSourceReference() ) else: return node else: return node def getComputationResult(node, computation, description): """ With a computation function, execute it and return constant result or exception node. """ # Try and turn raised exceptions into static raises. pylint: disable=broad-except try: result = computation() except Exception as e: new_node = makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = e ) change_tags = "new_raise" change_desc = description + " Predicted to raise an exception." else: new_node = makeCompileTimeConstantReplacementNode( value = result, node = node ) if isDebug(): assert new_node is not node, (node, result) if new_node is not node: change_tags = "new_constant" change_desc = description + " Predicted constant result." else: change_tags = None change_desc = None return new_node, change_tags, change_desc def makeStatementExpressionOnlyReplacementNode(expression, node): from .StatementNodes import StatementExpressionOnly return StatementExpressionOnly( expression = expression, source_ref = node.getSourceReference() ) def mergeStatements(statements, allow_none = False): """ Helper function that merges nested statement sequences. """ merged_statements = [] for statement in statements: if statement is None and allow_none: pass elif type(statement) in (tuple, list): merged_statements += mergeStatements(statement, allow_none) elif statement.isStatement() or statement.isStatementsFrame(): merged_statements.append(statement) elif statement.isStatementsSequence(): merged_statements.extend(mergeStatements(statement.getStatements())) else: assert False, statement return merged_statements def makeStatementsSequenceReplacementNode(statements, node): from .StatementNodes import StatementsSequence return StatementsSequence( statements = mergeStatements(statements), source_ref = node.getSourceReference() ) def convertNoneConstantToNone(node): if node is None: return None elif node.isExpressionConstantRef() and node.getConstant() is None: return None else: return node def wrapExpressionWithSideEffects(side_effects, old_node, new_node): assert new_node.isExpression() from .SideEffectNodes import ExpressionSideEffects if side_effects: side_effects = sum( ( side_effect.extractSideEffects() for side_effect in side_effects if side_effect.mayHaveSideEffects() ), () ) if side_effects: new_node = ExpressionSideEffects( expression = new_node, side_effects = side_effects, source_ref = old_node.getSourceReference() ) return new_node def wrapExpressionWithNodeSideEffects(new_node, old_node): return wrapExpressionWithSideEffects( side_effects = old_node.extractSideEffects(), old_node = old_node, new_node = new_node ) def wrapStatementWithSideEffects(new_node, old_node, allow_none = False): assert new_node is not None or allow_none side_effects = old_node.extractSideEffects() if side_effects: from .StatementNodes import StatementExpressionOnly side_effects = tuple( StatementExpressionOnly( expression = side_effect, source_ref = side_effect.getSourceReference() ) for side_effect in side_effects ) if new_node is not None: new_node = makeStatementsSequenceReplacementNode( statements = side_effects + (new_node,), node = old_node ) else: new_node = makeStatementsSequenceReplacementNode( statements = side_effects, node = old_node ) return new_node def makeStatementOnlyNodesFromExpressions(expressions): from .StatementNodes import StatementExpressionOnly, StatementsSequence statements = tuple( StatementExpressionOnly( expression = expression, source_ref = expression.getSourceReference() ) for expression in expressions ) if not statements: return None elif len(statements) == 1: return statements[0] else: return StatementsSequence( statements = statements, source_ref = statements[0].getSourceReference() ) def makeComparisonNode(left, right, comparator, source_ref): from .ComparisonNodes import ( ExpressionComparison, ExpressionComparisonIs, ExpressionComparisonIsNOT, ExpressionComparisonIn, ExpressionComparisonNOTIn ) if comparator == "Is": result = ExpressionComparisonIs( left = left, right = right, source_ref = source_ref ) elif comparator == "IsNot": result = ExpressionComparisonIsNOT( left = left, right = right, source_ref = source_ref ) elif comparator == "In": result = ExpressionComparisonIn( left = left, right = right, source_ref = source_ref ) elif comparator == "NotIn": result = ExpressionComparisonNOTIn( left = left, right = right, source_ref = source_ref ) else: result = ExpressionComparison( left = left, right = right, comparator = comparator, source_ref = source_ref ) result.setCompatibleSourceReference( source_ref = right.getCompatibleSourceReference() ) return result def makeVariableRefNode(variable, source_ref): if variable.isTempVariable(): from .VariableRefNodes import ExpressionTempVariableRef return ExpressionTempVariableRef( variable = variable, source_ref = source_ref ) else: from .VariableRefNodes import ExpressionVariableRef return ExpressionVariableRef( variable = variable, source_ref = source_ref ) def makeExpressionBuiltinLocals(provider, source_ref): while provider.isExpressionOutlineFunction(): provider = provider.getParentVariableProvider() if provider.isCompiledPythonModule(): from .GlobalsLocalsNodes import ExpressionBuiltinGlobals return ExpressionBuiltinGlobals( source_ref = source_ref ) else: from .GlobalsLocalsNodes import ExpressionBuiltinLocalsUpdated, ExpressionBuiltinLocalsCopy if provider.isLocalsUpdatedMode(): locals_class = ExpressionBuiltinLocalsUpdated else: locals_class = ExpressionBuiltinLocalsCopy return locals_class( source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/nodes/BuiltinNextNodes.py0000644000372000001440000000574713122472300021705 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for the calls to the 'next' built-in and unpacking special next. The unpacking next has only special that it raises a different exception text, explaining things about its context. """ from .ExpressionBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase ) class ExpressionBuiltinNext1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_NEXT1" def __init__(self, value, source_ref): ExpressionBuiltinSingleArgBase.__init__( self, value = value, source_ref = source_ref ) def computeExpression(self, trace_collection): return self.getValue().computeExpressionNext1( next_node = self, trace_collection = trace_collection ) class ExpressionSpecialUnpack(ExpressionBuiltinNext1): kind = "EXPRESSION_SPECIAL_UNPACK" def __init__(self, value, count, expected, source_ref): ExpressionBuiltinNext1.__init__( self, value = value, source_ref = source_ref ) self.count = int(count) self.expected = int(expected) def getDetails(self): result = ExpressionBuiltinNext1.getDetails(self) result["count"] = self.getCount() result["expected"] = self.getExpected() return result def getCount(self): return self.count def getExpected(self): return self.expected class ExpressionBuiltinNext2(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_NEXT2" named_children = ( "iterator", "default" ) def __init__(self, iterator, default, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "iterator" : iterator, "default" : default, }, source_ref = source_ref ) getIterator = ExpressionChildrenHavingBase.childGetter("iterator") getDefault = ExpressionChildrenHavingBase.childGetter("default") def computeExpression(self, trace_collection): # TODO: The "iterator" should be investigated here, if it is iterable, # or if the default is raising. return self, None, None Nuitka-0.5.28.2/nuitka/nodes/ModuleNodes.py0000644000372000001440000006372313207537706020703 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module/Package nodes The top of the tree. Packages are also modules. Modules are what hold a program together and cross-module optimizations are the most difficult to tackle. """ import os from nuitka import Options, Variables from nuitka.containers.oset import OrderedSet from nuitka.importing.Importing import ( findModule, getModuleNameAndKindFromFilename ) from nuitka.importing.Recursion import decideRecursion, recurseTo from nuitka.ModuleRegistry import getModuleByName, getOwnerFromCodeName from nuitka.optimizations.TraceCollections import TraceCollectionModule from nuitka.PythonVersions import python_version from nuitka.SourceCodeReferences import SourceCodeReference, fromFilename from nuitka.utils.CStrings import encodePythonIdentifierToC from nuitka.utils.FileOperations import relpath from .Checkers import checkStatementsSequenceOrNone from .ConstantRefNodes import makeConstantRefNode from .ExpressionBases import ExpressionBase from .FutureSpecs import FutureSpec, fromFlags from .IndicatorMixins import EntryPointMixin, MarkNeedsAnnotationsMixin from .NodeBases import ( ChildrenHavingMixin, ClosureGiverNodeMixin, NodeBase, extractKindAndArgsFromXML, fromXML ) class PythonModuleBase(NodeBase): __slots__ = "name", "package_name", "package" def __init__(self, name, package_name, source_ref): assert type(name) is str, type(name) assert '.' not in name, name assert package_name is None or \ (type(package_name) is str and package_name != "") NodeBase.__init__( self, source_ref = source_ref ) self.name = name self.package_name = package_name self.package = None def getName(self): return self.name def getPackage(self): return self.package_name def getFullName(self): if self.package_name: return self.package_name + '.' + self.getName() else: return self.getName() @staticmethod def isMainModule(): return False @staticmethod def isInternalModule(): return False def attemptRecursion(self): # Make sure the package is recursed to. # Return the list of newly added modules. result = [] if self.package_name is not None and self.package is None: self.package = getModuleByName(self.package_name) if self.package_name is not None and self.package is None: package_package, package_filename, finding = findModule( importing = self, module_name = self.package_name, parent_package = None, level = 1, warn = python_version < 330 ) # TODO: Temporary, if we can't find the package for Python3.3 that # is semi-OK, maybe. if python_version >= 330 and not package_filename: return [] if self.package_name == "uniconvertor.app.modules": return [] assert package_filename is not None, (self.package_name, finding) _package_name, package_kind = getModuleNameAndKindFromFilename(package_filename) # assert _package_name == self.package_name, (package_filename, _package_name, self.package_name) decision, _reason = decideRecursion( module_filename = package_filename, module_name = self.package_name, module_package = package_package, module_kind = package_kind ) if decision is not None: self.package, is_added = recurseTo( module_package = package_package, module_filename = package_filename, module_relpath = relpath(package_filename), module_kind = "py", reason = "Containing package of recursed module '%s'." % self.getFullName(), ) if is_added: result.append(self.package) if self.package: from nuitka.ModuleRegistry import addUsedModule addUsedModule(self.package) # print "Recursed to package", self.package_name result.extend(self.package.attemptRecursion()) return result def getCodeName(self): # Abstract method, pylint: disable=no-self-use return None def getCompileTimeFilename(self): return os.path.abspath(self.getSourceReference().getFilename()) def getRunTimeFilename(self): # TODO: Don't look at such things this late, push this into building. reference_mode = Options.getFileReferenceMode() if reference_mode == "original": return self.getCompileTimeFilename() elif reference_mode == "frozen": return "" % self.getFullName() else: filename = self.getCompileTimeFilename() full_name = self.getFullName() result = os.path.basename(filename) current = filename levels = full_name.count('.') if self.isCompiledPythonPackage(): levels += 1 for _i in range(levels): current = os.path.dirname(current) result = os.path.join( os.path.basename(current), result ) return result class CompiledPythonModule(ChildrenHavingMixin, ClosureGiverNodeMixin, MarkNeedsAnnotationsMixin, EntryPointMixin, PythonModuleBase): """ Compiled Python Module """ kind = "COMPILED_PYTHON_MODULE" named_children = ( "body", "functions" ) checkers = { "body": checkStatementsSequenceOrNone } def __init__(self, name, package_name, mode, future_spec, source_ref): PythonModuleBase.__init__( self, name = name, package_name = package_name, source_ref = source_ref ) ClosureGiverNodeMixin.__init__( self, name = name, code_prefix = "module" ) ChildrenHavingMixin.__init__( self, values = { "body" : None, # delayed "functions" : (), }, ) MarkNeedsAnnotationsMixin.__init__(self) EntryPointMixin.__init__(self) self.mode = mode self.variables = {} self.active_functions = OrderedSet() self.cross_used_functions = OrderedSet() # Often "None" until tree building finishes its part. self.future_spec = future_spec def getDetails(self): return { "filename" : self.source_ref.getFilename(), "package" : self.package_name, "name" : self.name } def getDetailsForDisplay(self): result = self.getDetails() if self.future_spec is not None: result["code_flags"] = ','.join(self.future_spec.asFlags()) return result @classmethod def fromXML(cls, provider, source_ref, **args): # Modules are not having any provider, must not be used, assert False def getFutureSpec(self): return self.future_spec def asGraph(self, computation_counter): from graphviz import Digraph # @UnresolvedImport pylint: disable=I0021,import-error graph = Digraph("cluster_%d" % computation_counter, comment = "Graph for %s" % self.getName()) graph.body.append("style=filled") graph.body.append("color=lightgrey") graph.body.append("label=Iteration_%d" % computation_counter) def makeTraceNodeName(variable_trace): return "%d/ %s %s %s" % ( computation_counter, variable_trace.getVariable(), variable_trace.getVersion(), variable_trace.__class__.__name__ ) for function_body in self.active_functions: trace_collection = function_body.trace_collection for (_variable, _version), variable_trace in trace_collection.getVariableTracesAll().items(): node = makeTraceNodeName(variable_trace) previous = variable_trace.getPrevious() if variable_trace.hasDefiniteUsages(): graph.attr("node", style = "filled", color = "blue") elif variable_trace.hasPotentialUsages(): graph.attr("node", style = "filled", color = "yellow") else: graph.attr("node", style = "filled", color = "red") graph.node(node) if type(previous) is tuple: for previous in previous: graph.edge(makeTraceNodeName(previous), node) elif previous is not None: graph.edge(makeTraceNodeName(previous), node) return graph getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") getFunctions = ChildrenHavingMixin.childGetter("functions") setFunctions = ChildrenHavingMixin.childSetter("functions") @staticmethod def isCompiledPythonModule(): return True def getParent(self): # We have never have a parent return None def getParentVariableProvider(self): # We have never have a provider return None def hasVariableName(self, variable_name): return variable_name in self.variables or variable_name in self.temp_variables def getVariables(self): return self.variables.values() def getFilename(self): return self.source_ref.getFilename() def getVariableForAssignment(self, variable_name): return self.getProvidedVariable(variable_name) def getVariableForReference(self, variable_name): return self.getProvidedVariable(variable_name) def getVariableForClosure(self, variable_name): return self.getProvidedVariable( variable_name = variable_name ) def createProvidedVariable(self, variable_name): assert variable_name not in self.variables result = Variables.ModuleVariable( module = self, variable_name = variable_name ) self.variables[variable_name] = result return result @staticmethod def getContainingClassDictCreation(): return None @staticmethod def isEarlyClosure(): # Modules should immediately closure variables on use. return True def getEntryPoint(self): return self def getCodeName(self): # For code name of modules, we need to translate to C identifiers, # removing characters illegal for that. return encodePythonIdentifierToC(self.getFullName()) def addFunction(self, function_body): functions = self.getFunctions() assert function_body not in functions functions += (function_body,) self.setFunctions(functions) def startTraversal(self): self.active_functions = OrderedSet() def addUsedFunction(self, function_body): assert function_body in self.getFunctions() assert function_body.isExpressionFunctionBody() or \ function_body.isExpressionClassBody() or \ function_body.isExpressionGeneratorObjectBody() or \ function_body.isExpressionCoroutineObjectBody() or \ function_body.isExpressionAsyncgenObjectBody() if function_body not in self.active_functions: self.active_functions.add(function_body) def getUsedFunctions(self): return self.active_functions def getUnusedFunctions(self): for function in self.getFunctions(): if function not in self.active_functions: yield function def addCrossUsedFunction(self, function_body): if function_body not in self.cross_used_functions: self.cross_used_functions.add(function_body) def getCrossUsedFunctions(self): return self.cross_used_functions def getFunctionFromCodeName(self, code_name): for function in self.getFunctions(): if function.getCodeName() == code_name: return function def getOutputFilename(self): main_filename = self.getFilename() if main_filename.endswith(".py"): result = main_filename[:-3] elif main_filename.endswith(".pyw"): result = main_filename[:-4] else: result = main_filename # There are some characters that somehow are passed to shell, by # Scons or unknown, so lets avoid them for now. return result.replace(')',"").replace('(',"") def computeModule(self): old_collection = self.trace_collection self.trace_collection = TraceCollectionModule(self) module_body = self.getBody() if module_body is not None: result = module_body.computeStatementsSequence( trace_collection = self.trace_collection ) if result is not module_body: self.setBody(result) new_modules = self.attemptRecursion() for new_module in new_modules: self.trace_collection.signalChange( source_ref = new_module.getSourceReference(), tags = "new_code", message = "Recursed to module package." ) self.trace_collection.updateVariablesFromCollection(old_collection) def getTraceCollections(self): yield self.trace_collection for function in self.getUsedFunctions(): yield function.trace_collection def markAsLocalsDict(self): # Does not apply to modules. pass def hasLocalsDict(self): return False def isUnoptimized(self): return False def getLocalVariables(self): return () def getUserLocalVariables(self): return () def getOutlineLocalVariables(self): outlines = self.getTraceCollection().getOutlineFunctions() if outlines is None: return () result = [] for outline in outlines: result.extend(outline.getUserLocalVariables()) return tuple(result) def hasClosureVariable(self, variable): return False def removeUserVariable(self, variable): outlines = self.getTraceCollection().getOutlineFunctions() for outline in outlines: user_locals = outline.getUserLocalVariables() if variable in user_locals: outline.removeUserVariable(variable) break class CompiledPythonPackage(CompiledPythonModule): kind = "COMPILED_PYTHON_PACKAGE" def __init__(self, name, package_name, mode, future_spec, source_ref): assert name, source_ref CompiledPythonModule.__init__( self, name = name, package_name = package_name, mode = mode, future_spec = future_spec, source_ref = source_ref ) def getOutputFilename(self): return os.path.dirname(self.getFilename()) @staticmethod def canHaveExternalImports(): return True def makeUncompiledPythonModule(module_name, filename, bytecode, is_package, user_provided, technical): parts = module_name.rsplit('.', 1) name = parts[-1] package_name = parts[0] if len(parts) == 2 else None source_ref = fromFilename(filename) if is_package: return UncompiledPythonPackage( name = name, package_name = package_name, bytecode = bytecode, filename = filename, user_provided = user_provided, technical = technical, source_ref = source_ref ) else: return UncompiledPythonModule( name = name, package_name = package_name, bytecode = bytecode, filename = filename, user_provided = user_provided, technical = technical, source_ref = source_ref ) class UncompiledPythonModule(PythonModuleBase): """ Compiled Python Module """ kind = "UNCOMPILED_PYTHON_MODULE" __slots__ = "bytecode", "filename", "user_provided", "technical", "used_modules" def __init__(self, name, package_name, bytecode, filename, user_provided, technical, source_ref): PythonModuleBase.__init__( self, name = name, package_name = package_name, source_ref = source_ref ) self.bytecode = bytecode self.filename = filename self.user_provided = user_provided self.technical = technical self.used_modules = () @staticmethod def isUncompiledPythonModule(): return True def isUserProvided(self): return self.user_provided def isTechnical(self): """ Must be bytecode as it's used in CPython library initialization. """ return self.technical def getByteCode(self): return self.bytecode def getFilename(self): return self.filename def getUsedModules(self): return self.used_modules def setUsedModules(self, used_modules): self.used_modules = used_modules def startTraversal(self): pass class UncompiledPythonPackage(UncompiledPythonModule): kind = "UNCOMPILED_PYTHON_PACKAGE" class PythonMainModule(CompiledPythonModule): kind = "PYTHON_MAIN_MODULE" def __init__(self, main_added, mode, future_spec, source_ref): CompiledPythonModule.__init__( self, name = "__main__", package_name = None, mode = mode, future_spec = future_spec, source_ref = source_ref ) self.main_added = main_added def getDetails(self): return { "filename" : self.source_ref.getFilename(), "main_added" : self.main_added, "mode" : self.mode } @classmethod def fromXML(cls, provider, source_ref, **args): if "code_flags" in args: future_spec = fromFlags(args["code_flags"]) result = cls( main_added = args["main_added"] == "True", mode = args["mode"], future_spec = future_spec, source_ref = source_ref ) from nuitka.ModuleRegistry import addRootModule addRootModule(result) function_work = [] for xml in args["functions"]: _kind, node_class, func_args, source_ref = extractKindAndArgsFromXML(xml, source_ref) if "provider" in func_args: func_args["provider"] = getOwnerFromCodeName(func_args["provider"]) else: func_args["provider"] = result if "flags" in args: func_args["flags"] = set(func_args["flags"].split(',')) if "doc" not in args: func_args["doc"] = None function = node_class.fromXML( source_ref = source_ref, **func_args ) # Could do more checks for look up of body here, but so what... function_work.append( (function, iter(iter(xml).next()).next()) ) for function, xml in function_work: function.setChild( "body", fromXML( provider = function, xml = xml, source_ref = function.getSourceReference() ) ) result.setChild( "body", fromXML( provider = result, xml = args["body"][0], source_ref = source_ref ) ) return result @staticmethod def isMainModule(): return True def getOutputFilename(self): if self.main_added: return os.path.dirname(self.getFilename()) else: return CompiledPythonModule.getOutputFilename(self) class PythonInternalModule(CompiledPythonModule): kind = "PYTHON_INTERNAL_MODULE" def __init__(self): CompiledPythonModule.__init__( self, name = "__internal__", package_name = None, mode = "compiled", source_ref = SourceCodeReference.fromFilenameAndLine( filename = "internal", line = 0 ), future_spec = FutureSpec() ) @staticmethod def isInternalModule(): return True def getOutputFilename(self): return "__internal" class PythonShlibModule(PythonModuleBase): kind = "PYTHON_SHLIB_MODULE" __slots__ = ("used_modules",) avoid_duplicates = set() def __init__(self, name, package_name, source_ref): PythonModuleBase.__init__( self, name = name, package_name = package_name, source_ref = source_ref ) # That would be a mistake we just made. assert os.path.basename(source_ref.getFilename()) != "" # That is too likely a bug. assert name != "__main__" # Duplicates should be avoided by us caching elsewhere before creating # the object. assert self.getFullName() not in self.avoid_duplicates, self.getFullName() self.avoid_duplicates.add(self.getFullName()) self.used_modules = None def getDetails(self): return { "name" : self.name, "package_name" : self.package_name } def getFilename(self): return self.getSourceReference().getFilename() def startTraversal(self): pass def getPyIFilename(self): """ Get Python type description filename. """ path = self.getFilename() filename = os.path.basename(path) dirname = os.path.dirname(path) return os.path.join(dirname, filename.split('.')[0]) + ".pyi" def _readPyPIFile(self): """ Read the .pyi file if present and scan for dependencies. """ if self.used_modules is None: pyi_filename = self.getPyIFilename() if os.path.exists(pyi_filename): pyi_deps = OrderedSet() for line in open(pyi_filename): line = line.strip() if line.startswith("import "): imported = line[7:] pyi_deps.add(imported) elif line.startswith("from "): parts = line.split(None, 3) assert parts[0] == "from" assert parts[2] == "import" if parts[1] == "typing": continue pyi_deps.add(parts[1]) imported = parts[3] if imported.startswith('('): # No multiline imports please assert imported.endswith(')') imported = imported[1:-1] assert imported if imported == '*': continue for name in imported.split(','): name = name.strip() pyi_deps.add(parts[1] + '.' + name) if "typing" in pyi_deps: pyi_deps.discard("typing") self.used_modules = tuple(pyi_deps) else: self.used_modules = () def getUsedModules(self): self._readPyPIFile() return self.used_modules def getParentModule(self): return self class ExpressionModuleFileAttributeRef(ExpressionBase): kind = "EXPRESSION_MODULE_FILE_ATTRIBUTE_REF" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def mayRaiseException(self, exception_type): return False def computeExpressionRaw(self, trace_collection): # There is not a whole lot to do here, the path will change at run # time if Options.getFileReferenceMode() != "runtime": result = makeConstantRefNode( constant = self.getRunTimeFilename(), source_ref = self.getSourceReference() ) return result, "new_expression", "Using original '__file__' value." return self, None, None def getCompileTimeFilename(self): return self.getParentModule().getCompileTimeFilename() def getRunTimeFilename(self): return self.getParentModule().getRunTimeFilename() class ExpressionModuleLoaderRef(ExpressionBase): kind = "EXPRESSION_MODULE_LOADER_REF" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def mayRaiseException(self, exception_type): return False def computeExpressionRaw(self, trace_collection): return self, None, None Nuitka-0.5.28.2/nuitka/nodes/FunctionNodes.py0000644000372000001440000007651113207537242021235 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for functions and their creations. Lambdas are functions too. The functions are at the core of the language and have their complexities. Creating a CPython function object is an optional thing. Some things might only be used to be called directly, while knowing exactly what it is. So the "ExpressionFunctionCreation" might be used to provide that kind of CPython reference, and may escape. Coroutines and generators live in their dedicated module and share base classes. """ from nuitka import Options, Variables from nuitka.PythonVersions import python_version from nuitka.tree.Extractions import updateVariableUsage from .Checkers import checkStatementsSequenceOrNone from .CodeObjectSpecs import CodeObjectSpec from .ExpressionBases import ( CompileTimeConstantExpressionBase, ExpressionBase, ExpressionChildrenHavingBase ) from .FutureSpecs import fromFlags from .IndicatorMixins import ( EntryPointMixin, MarkLocalsDictIndicatorMixin, MarkUnoptimizedFunctionIndicatorMixin ) from .NodeBases import ( ChildrenHavingMixin, ClosureGiverNodeMixin, ClosureTakerMixin, SideEffectsFromChildrenMixin ) from .NodeMakingHelpers import ( makeConstantReplacementNode, makeRaiseExceptionReplacementExpressionFromInstance, wrapExpressionWithSideEffects ) from .ParameterSpecs import ParameterSpec, TooManyArguments, matchCall class MaybeLocalVariableUsage(Exception): pass class ExpressionFunctionBodyBase(ClosureTakerMixin, ClosureGiverNodeMixin, ExpressionChildrenHavingBase): def __init__(self, provider, name, code_prefix, flags, source_ref, body): while provider.isExpressionOutlineBody(): provider = provider.getParentVariableProvider() ExpressionChildrenHavingBase.__init__( self, values = { "body" : body # Might be None initially in some cases. }, source_ref = source_ref ) ClosureTakerMixin.__init__( self, provider = provider ) ClosureGiverNodeMixin.__init__( self, name = name, code_prefix = code_prefix ) # Special things, "has_super" indicates presence of "super" in variable # usage, which modifies some behaviors. self.flags = flags or None # Hack: This allows some APIs to work although this is not yet # officially a child yet. Important during building. self.parent = provider # Python3.4: Might be overridden by global statement on the class name. # TODO: Make this class only code. if python_version >= 340: self.qualname_provider = provider # Non-local declarations. self.non_local_declarations = [] # Register ourselves immediately with the module. # TODO: Have a base class that is between outline function and entry # point functions. if not self.isExpressionOutlineFunction() and \ not self.isExpressionClassBody(): provider.getParentModule().addFunction(self) @staticmethod def isExpressionFunctionBodyBase(): return True def getEntryPoint(self): """ Entry point for code. Normally ourselves. Only outlines will refer to their parent which technically owns them. """ return self def getContainingClassDictCreation(self): current = self while not current.isCompiledPythonModule(): if current.isExpressionClassBody(): return current current = current.getParentVariableProvider() return None def hasFlag(self, flag): return self.flags is not None and flag in self.flags def discardFlag(self, flag): if self.flags is not None: self.flags.discard(flag) @staticmethod def isEarlyClosure(): """ Early closure taking means immediate binding of references. Normally it's good to lookup name references immediately, but not for functions. In case of a function body it is not allowed to do that, because a later assignment needs to be queried first. Nodes need to indicate via this if they would like to resolve references at the same time as assignments. """ return False def isLocalsUpdatedMode(self): return python_version >= 300 or \ self.isEarlyClosure() or \ self.isUnoptimized() def hasVariableName(self, variable_name): return variable_name in self.providing or variable_name in self.temp_variables def getVariables(self): return self.providing.values() def getLocalVariables(self): return [ variable for variable in self.providing.values() if variable.isLocalVariable() ] def getLocalVariableNames(self): return [ variable.getName() for variable in self.providing.values() if variable.isLocalVariable() ] def getUserLocalVariables(self): return tuple( variable for variable in self.providing.values() if variable.isLocalVariable() and not variable.isParameterVariable() if variable.getOwner() is self ) def getOutlineLocalVariables(self): outlines = self.getTraceCollection().getOutlineFunctions() if outlines is None: return () result = [] for outline in outlines: result.extend(outline.getUserLocalVariables()) return tuple(result) def removeClosureVariable(self, variable): assert variable in self.providing.values(), (self.providing, variable) del self.providing[variable.getName()] assert not variable.isParameterVariable() or \ variable.getOwner() is not self self.taken.remove(variable) def demoteClosureVariable(self, variable): assert variable.isLocalVariable() self.taken.remove(variable) assert variable.getOwner() is not self new_variable = Variables.LocalVariable( owner = self, variable_name = variable.getName() ) for variable_trace in variable.traces: if variable_trace.getOwner() is self: new_variable.addTrace(variable_trace) new_variable.updateUsageState() self.providing[variable.getName()] = new_variable updateVariableUsage( provider = self, old_variable = variable, new_variable = new_variable ) def hasClosureVariable(self, variable): return variable in self.taken def removeUserVariable(self, variable): assert variable in self.providing.values(), (self.providing, variable) del self.providing[variable.getName()] assert not variable.isParameterVariable() or \ variable.getOwner() is not self def getVariableForAssignment(self, variable_name): # print("ASS func", self, variable_name) if self.hasTakenVariable(variable_name): result = self.getTakenVariable(variable_name) else: result = self.getProvidedVariable(variable_name) return result def getVariableForReference(self, variable_name): # print( "REF func", self, variable_name ) if self.hasProvidedVariable(variable_name): result = self.getProvidedVariable(variable_name) else: result = self.getClosureVariable( variable_name = variable_name ) # Remember that we need that closure variable for something, so # we don't create it again all the time. if not result.isModuleVariable(): self.registerProvidedVariable(result) entry_point = self.getEntryPoint() # For "exec" containing/star import containing, we raise this exception to indicate # that instead of merely a variable, to be assigned, we need to replace with locals # dict access. if python_version < 300 and \ not entry_point.isExpressionClassBody() and \ not entry_point.isPythonMainModule() and \ result.isModuleVariable() and \ entry_point.isUnoptimized(): raise MaybeLocalVariableUsage return result def getVariableForClosure(self, variable_name): # print( "getVariableForClosure", self.getCodeName(), variable_name, self.isUnoptimized() ) if self.hasProvidedVariable(variable_name): return self.getProvidedVariable(variable_name) return self.takeVariableForClosure(variable_name) def takeVariableForClosure(self, variable_name): result = self.provider.getVariableForClosure(variable_name) self.taken.add(result) return result def createProvidedVariable(self, variable_name): # print("createProvidedVariable", self, variable_name) return Variables.LocalVariable( owner = self, variable_name = variable_name ) def addNonlocalsDeclaration(self, names, source_ref): self.non_local_declarations.append( (names, source_ref) ) def consumeNonlocalDeclarations(self): result = self.non_local_declarations self.non_local_declarations = () return result def getFunctionName(self): return self.name def getFunctionQualname(self): """ Function __qualname__ new in CPython3.3 Should contain some kind of full name descriptions for the closure to recognize and will be used for outputs. """ function_name = self.getFunctionName() if python_version < 340: provider = self.getParentVariableProvider() else: provider = self.qualname_provider if provider.isCompiledPythonModule(): return function_name elif provider.isExpressionClassBody(): return provider.getFunctionQualname() + '.' + function_name else: return provider.getFunctionQualname() + ".." + function_name def computeExpression(self, trace_collection): assert False # Function body is quite irreplaceable. return self, None, None def mayRaiseException(self, exception_type): body = self.getBody() if body is None: return False else: return self.getBody().mayRaiseException(exception_type) class ExpressionFunctionEntryPointBase(EntryPointMixin, ExpressionFunctionBodyBase): def __init__(self, provider, name, code_prefix, flags, source_ref): ExpressionFunctionBodyBase.__init__( self, provider = provider, name = name, code_prefix = code_prefix, flags = flags, body = None, source_ref = source_ref ) EntryPointMixin.__init__(self) getBody = ChildrenHavingMixin.childGetter("body") setBody = ChildrenHavingMixin.childSetter("body") class ExpressionFunctionBody(MarkLocalsDictIndicatorMixin, MarkUnoptimizedFunctionIndicatorMixin, ExpressionFunctionEntryPointBase): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, leading to many methods, pylint: disable=R0901 kind = "EXPRESSION_FUNCTION_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } if python_version >= 340: qualname_setup = None def __init__(self, provider, name, doc, parameters, flags, source_ref): if name == "": assert python_version >= 300 ExpressionFunctionEntryPointBase.__init__( self, provider = provider, name = name, code_prefix = "function", flags = flags, source_ref = source_ref ) MarkLocalsDictIndicatorMixin.__init__(self) MarkUnoptimizedFunctionIndicatorMixin.__init__(self, flags) self.doc = doc # Indicator if the return value exception might be required. self.return_exception = False # Indicator if the function needs to be created as a function object. self.needs_creation = False # Indicator if the function is called directly. self.needs_direct = False # Indicator if the function is used outside of where it's defined. self.cross_module_use = False self.parameters = parameters self.parameters.setOwner(self) for variable in self.parameters.getAllVariables(): self.registerProvidedVariable(variable) def getDetails(self): return { "name" : self.getFunctionName(), "ref_name" : self.getCodeName(), "parameters" : self.getParameters(), "provider" : self.provider.getCodeName(), "doc" : self.doc } def getDetailsForDisplay(self): result = { "name" : self.getFunctionName(), "provider" : self.provider.getCodeName(), "flags" : self.flags } result.update(self.parameters.getDetails()) if self.doc is not None: result["doc"] = self.doc return result @classmethod def fromXML(cls, provider, source_ref, **args): assert provider is not None parameter_spec_args = {} other_args = {} for key, value in args.items(): if key.startswith("ps_"): parameter_spec_args[key] = value elif key == "code_flags": source_ref.future_spec = fromFlags(value) else: other_args[key] = value parameters = ParameterSpec(**parameter_spec_args) # The empty doc string and no doc string are distinguished by presence. The # most common case is going to be not present. if "doc" not in other_args: other_args["doc"] = None return cls( provider = provider, parameters = parameters, source_ref = source_ref, **other_args ) def getDetail(self): return "named %s with %s" % (self.getFunctionName(), self.parameters) def getParent(self): assert False def getDoc(self): return self.doc def getParameters(self): return self.parameters def needsCreation(self): return self.needs_creation def markAsNeedsCreation(self): self.needs_creation = True def needsDirectCall(self): return self.needs_direct def markAsDirectlyCalled(self): self.needs_direct = True def isCrossModuleUsed(self): return self.cross_module_use def markAsCrossModuleUsed(self): self.cross_module_use = True def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): # TODO: Until we have something to re-order the arguments, we need to # skip this. For the immediate need, we avoid this complexity, as a # re-ordering will be needed. assert False, self def isCompileTimeConstant(self): # TODO: It's actually pretty much compile time accessible maybe. return None def mayHaveSideEffects(self): # The function definition has no side effects, calculating the defaults # would be, but that is done outside of this. return False def mayRaiseException(self, exception_type): return self.getBody().mayRaiseException(exception_type) def markAsExceptionReturnValue(self): self.return_exception = True def needsExceptionReturnValue(self): return self.return_exception def convertNoneConstantOrEmptyDictToNone(node): if node is None: return None elif node.isExpressionConstantRef() and node.getConstant() is None: return None elif node.isExpressionConstantRef() and node.getConstant() == {}: return None else: return node # TODO: Function direct call node ought to be here too. class ExpressionFunctionCreation(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): kind = "EXPRESSION_FUNCTION_CREATION" # Note: The order of evaluation for these is a bit unexpected, but # true. Keyword defaults go first, then normal defaults, and annotations of # all kinds go last. # A bug of CPython3.x not fixed before version 3.4, see bugs.python.org/issue16967 kw_defaults_before_defaults = python_version < 340 if kw_defaults_before_defaults: named_children = ( "kw_defaults", "defaults", "annotations", "function_ref" ) else: named_children = ( "defaults", "kw_defaults", "annotations", "function_ref" ) checkers = { "kw_defaults" : convertNoneConstantOrEmptyDictToNone, } def __init__(self, function_ref, code_object, defaults, kw_defaults, annotations, source_ref): assert kw_defaults is None or kw_defaults.isExpression() assert annotations is None or annotations.isExpression() assert function_ref.isExpressionFunctionRef() ExpressionChildrenHavingBase.__init__( self, values = { "function_ref" : function_ref, "defaults" : tuple(defaults), "kw_defaults" : kw_defaults, "annotations" : annotations }, source_ref = source_ref ) self.code_object = code_object self.variable_closure_traces = None def getName(self): return self.getFunctionRef().getName() def getCodeObject(self): return self.code_object def getDetails(self): return { "code_object" : self.code_object } def getDetailsForDisplay(self): if self.code_object: return self.code_object.getDetails() else: return {} @classmethod def fromXML(cls, provider, source_ref, **args): code_object_args = {} other_args = {} for key, value in args.items(): if key.startswith("co_"): code_object_args[key] = value elif key == "code_flags": code_object_args["future_spec"] = fromFlags(args["code_flags"]) else: other_args[key] = value if code_object_args: code_object = CodeObjectSpec(**code_object_args) else: code_object = None return cls( code_object = code_object, source_ref = source_ref, **other_args ) def computeExpression(self, trace_collection): self.variable_closure_traces = [] for closure_variable in self.getFunctionRef().getFunctionBody().getClosureVariables(): trace = trace_collection.getVariableCurrentTrace(closure_variable) trace.addClosureUsage() self.variable_closure_traces.append(trace) # TODO: Function body may know something too. return self, None, None getFunctionRef = ExpressionChildrenHavingBase.childGetter("function_ref") getDefaults = ExpressionChildrenHavingBase.childGetter("defaults") getKwDefaults = ExpressionChildrenHavingBase.childGetter("kw_defaults") getAnnotations = ExpressionChildrenHavingBase.childGetter("annotations") def mayRaiseException(self, exception_type): for default in self.getDefaults(): if default.mayRaiseException(exception_type): return True kw_defaults = self.getKwDefaults() if kw_defaults is not None and kw_defaults.mayRaiseException(exception_type): return True annotations = self.getAnnotations() if annotations is not None and annotations.mayRaiseException(exception_type): return True return False def computeExpressionCall(self, call_node, call_args, call_kw, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) # TODO: Until we have something to re-order the keyword arguments, we # need to skip this. For the immediate need, we avoid this complexity, # as a re-ordering will be needed. if call_kw is not None and \ (not call_kw.isExpressionConstantRef() or call_kw.getConstant() != {}): return call_node, None, None if call_kw is not None: return call_node, None, None if call_args is None: args_tuple = () else: assert call_args.isExpressionConstantRef() or \ call_args.isExpressionMakeTuple() args_tuple = call_args.getIterationValues() function_body = self.getFunctionRef().getFunctionBody() # TODO: Actually the above disables it entirely, as it is at least # the empty dictionary node in any case. We will need some enhanced # interfaces for "matchCall" to work on. call_spec = function_body.getParameters() try: args_dict = matchCall( func_name = self.getName(), args = call_spec.getArgumentNames(), star_list_arg = call_spec.getStarListArgumentName(), star_dict_arg = call_spec.getStarDictArgumentName(), num_defaults = call_spec.getDefaultCount(), positional = args_tuple, pairs = () ) values = [ args_dict[name] for name in call_spec.getParameterNames() ] result = ExpressionFunctionCall( function = self, values = values, source_ref = call_node.getSourceReference() ) return ( result, "new_statements", # TODO: More appropriate tag maybe. """\ Replaced call to created function body '%s' with direct \ function call.""" % self.getName() ) except TooManyArguments as e: result = wrapExpressionWithSideEffects( new_node = makeRaiseExceptionReplacementExpressionFromInstance( expression = call_node, exception = e.getRealException() ), old_node = call_node, side_effects = call_node.extractSideEffectsPreCall() ) return ( result, "new_raise", # TODO: More appropriate tag maybe. """Replaced call to created function body '%s' to argument \ error""" % self.getName() ) def getCallCost(self, values): # TODO: Ought to use values. If they are all constant, how about we # assume no cost, pylint: disable=unused-argument function_body = self.getFunctionRef().getFunctionBody() if function_body.isExpressionClassBody(): if function_body.getBody().getStatements()[0].isStatementReturn(): return 0 return None if True or not Options.isExperimental("function_inlining"): return None if function_body.isExpressionGeneratorObjectBody(): # TODO: That's not even allowed, is it? assert False return None # TODO: Lying for the demo, this is too limiting, but needs frames to # be allowed twice in a context. if function_body.mayRaiseException(BaseException): return 60 return 20 def createOutlineFromCall(self, provider, values): from nuitka.optimizations.FunctionInlining import convertFunctionCallToOutline return convertFunctionCallToOutline( provider = provider, function_ref = self.getFunctionRef(), values = values ) def getClosureVariableVersions(self): return [ (trace.getVariable(), trace.getVersion()) for trace in self.variable_closure_traces ] class ExpressionFunctionRef(ExpressionBase): kind = "EXPRESSION_FUNCTION_REF" __slots__ = "function_body", "code_name" def __init__(self, source_ref, function_body = None, code_name = None): assert function_body is not None or code_name is not None assert code_name != "None" ExpressionBase.__init__( self, source_ref = source_ref ) self.function_body = function_body self.code_name = code_name def getName(self): return self.function_body.getName() def getDetails(self): return { "function_body" : self.function_body } def getDetailsForDisplay(self): return { "code_name" : self.getFunctionBody().getCodeName() } def getFunctionBody(self): if self.function_body is None: module_code_name, _ = self.code_name.split("$$$", 1) from nuitka.ModuleRegistry import getModuleFromCodeName module = getModuleFromCodeName(module_code_name) self.function_body = module.getFunctionFromCodeName(self.code_name) return self.function_body def computeExpressionRaw(self, trace_collection): function_body = self.getFunctionBody() owning_module = function_body.getParentModule() # Make sure the owning module is added to the used set. This is most # important for helper functions, or modules, which otherwise have # become unused. from nuitka.ModuleRegistry import addUsedModule addUsedModule(owning_module) owning_module.addUsedFunction(function_body) from nuitka.optimizations.TraceCollections import \ TraceCollectionFunction # TODO: Doesn't this mean, we can do this multiple times by doing it # in the reference. We should do it in the body, and there we should # limit us to only doing it once per module run, e.g. being based on # presence in used functions of the module already. trace_collection = TraceCollectionFunction( parent = trace_collection, function_body = function_body ) old_collection = function_body.setTraceCollection(trace_collection) statements_sequence = function_body.getBody() if statements_sequence is not None and \ not statements_sequence.getStatements(): function_body.setStatements(None) statements_sequence = None if statements_sequence is not None: result = statements_sequence.computeStatementsSequence( trace_collection = trace_collection ) if result is not statements_sequence: function_body.setBody(result) trace_collection.updateVariablesFromCollection(old_collection) # TODO: Function collection may now know something. return self, None, None def mayHaveSideEffects(self): # Using a function has no side effects. return False class ExpressionFunctionCall(ExpressionChildrenHavingBase): """ Shared function call. This is for calling created function bodies with multiple users. Not clear if such a thing should exist. But what this will do is to have respect for the fact that there are multiple such calls. """ kind = "EXPRESSION_FUNCTION_CALL" named_children = ( "function", "values" ) def __init__(self, function, values, source_ref): assert function.isExpressionFunctionCreation() ExpressionChildrenHavingBase.__init__( self, values = { "function" : function, "values" : tuple(values), }, source_ref = source_ref ) self.variable_closure_traces = None def computeExpression(self, trace_collection): function = self.getFunction() values = self.getArgumentValues() # TODO: This needs some design. cost = function.getCallCost(values) function_body = function.getFunctionRef().getFunctionBody() if function_body.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) if cost is not None and cost < 50: result = function.createOutlineFromCall( provider = self.getParentVariableProvider(), values = values ) return ( result, "new_statements", "Function call to '%s' in-lined." % function_body.getCodeName() ) self.variable_closure_traces = [] for closure_variable in function_body.getClosureVariables(): trace = trace_collection.getVariableCurrentTrace(closure_variable) trace.addClosureUsage() self.variable_closure_traces.append(trace) return self, None, None def mayRaiseException(self, exception_type): function = self.getFunction() if function.getFunctionRef().getFunctionBody().mayRaiseException(exception_type): return True values = self.getArgumentValues() for value in values: if value.mayRaiseException(exception_type): return True return False getFunction = ExpressionChildrenHavingBase.childGetter("function") getArgumentValues = ExpressionChildrenHavingBase.childGetter("values") def getClosureVariableVersions(self): return [ (trace.getVariable(), trace.getVersion()) for trace in self.variable_closure_traces ] # Needed for Python3.3 and higher class ExpressionFunctionQualnameRef(CompileTimeConstantExpressionBase): kind = "EXPRESSION_FUNCTION_QUALNAME_REF" __slots__ = ("function_body",) def __init__(self, function_body, source_ref): CompileTimeConstantExpressionBase.__init__( self, source_ref = source_ref ) self.function_body = function_body def computeExpressionRaw(self, trace_collection): result = makeConstantReplacementNode( node = self, constant = self.function_body.getFunctionQualname() ) return ( result, "new_constant", "Executed '__qualname__' resolution to '%s'." % self.function_body.getFunctionQualname() ) Nuitka-0.5.28.2/nuitka/nodes/FrameNodes.py0000644000372000001440000002557613207537242020507 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Frame nodes. The frame attaches name and other frame properties to a scope, where it is optional. For use in tracebacks, their created frame objects, potentially cached are essential. Otherwise, they are similar to statement sequences, so they inherit from them. """ from nuitka.PythonVersions import python_version from .CodeObjectSpecs import CodeObjectSpec from .FutureSpecs import fromFlags from .StatementNodes import StatementsSequence def checkFrameStatements(value): """ Check that frames statements list value proper. Must not be None, must not contain None, and of course only statements sequences, or statements, may be empty. """ assert value is not None assert None not in value for statement in value: assert statement.isStatement() or statement.isStatementsFrame(), \ statement return tuple(value) class StatementsFrameBase(StatementsSequence): checkers = { "statements" : checkFrameStatements } def __init__(self, statements, guard_mode, code_object, source_ref): StatementsSequence.__init__( self, statements = statements, source_ref = source_ref ) # TODO: Why not have multiple classes for this. self.guard_mode = guard_mode self.code_object = code_object self.needs_frame_exception_preserve = False def isStatementsFrame(self): return True def getDetails(self): result = { "code_object" : self.code_object } result.update(StatementsSequence.getDetails(self)) return result def getDetailsForDisplay(self): result = StatementsSequence.getDetails(self) result.update() result.update(self.code_object.getDetails()) return result @classmethod def fromXML(cls, provider, source_ref, **args): code_object_args = {} other_args = {} for key, value in args.items(): if key.startswith("co_"): code_object_args[key] = value elif key == "code_flags": code_object_args["future_spec"] = fromFlags(args["code_flags"]) else: other_args[key] = value code_object = CodeObjectSpec(**code_object_args) return cls( code_object = code_object, source_ref = source_ref, **other_args ) def getGuardMode(self): return self.guard_mode def needsExceptionFramePreservation(self): if python_version < 300: return self.guard_mode != "generator" else: return True def getVarNames(self): return self.code_object.getVarNames() def updateLocalNames(self): """ For use during variable closure phase. Finalize attributes. """ provider = self.getParentVariableProvider() if not provider.isCompiledPythonModule(): self.code_object.updateLocalNames( [ variable.getName() for variable in provider.getLocalVariables() ] ) entry_point = provider.getEntryPoint() is_optimized = not entry_point.isCompiledPythonModule() and \ not entry_point.isExpressionClassBody() and \ not entry_point.hasLocalsDict() self.code_object.setFlagIsOptimizedValue(is_optimized) new_locals = not provider.isCompiledPythonModule() and \ (python_version < 340 or ( not provider.isExpressionClassBody() and \ not provider.hasLocalsDict())) self.code_object.setFlagNewLocalsValue(new_locals) if provider.isExpressionGeneratorObjectBody() or \ provider.isExpressionCoroutineObjectBody() or \ provider.isExpressionAsyncgenObjectBody(): closure_provider = provider.getParentVariableProvider() else: closure_provider = provider has_closure = closure_provider.isExpressionFunctionBody() and \ closure_provider.getClosureVariables() != () and \ not closure_provider.isExpressionClassBody() self.code_object.setFlagHasClosureValue(has_closure) def markAsFrameExceptionPreserving(self): self.needs_frame_exception_preserve = True def needsFrameExceptionPreserving(self): return self.needs_frame_exception_preserve def getCodeObject(self): return self.code_object def computeStatementsSequence(self, trace_collection): # The extraction of parts of the frame that can be moved before or after # the frame scope, takes it toll to complexity, pylint: disable=too-many-branches new_statements = [] statements = self.getStatements() for count, statement in enumerate(statements): # May be frames embedded. if statement.isStatementsFrame(): new_statement = statement.computeStatementsSequence( trace_collection = trace_collection ) else: new_statement = trace_collection.onStatement( statement = statement ) if new_statement is not None: if new_statement.isStatementsSequence() and \ not new_statement.isStatementsFrame(): new_statements.extend(new_statement.getStatements()) else: new_statements.append(new_statement) if statement is not statements[-1] and \ new_statement.isStatementAborting(): trace_collection.signalChange( "new_statements", statements[count+1].getSourceReference(), "Removed dead statements." ) break if not new_statements: trace_collection.signalChange( "new_statements", self.getSourceReference(), "Removed empty frame object of '%s'." % self.code_object.getCodeObjectName() ) return None # If our statements changed just now, they are not immediately usable, # so do this in two steps. Next time we can reduce the frame scope just # as well. if statements != tuple(new_statements): self.setStatements(new_statements) return self # Determine statements inside the frame, that need not be in a frame, # because they wouldn't raise an exception. outside_pre = [] while new_statements and \ not new_statements[0].needsFrame(): outside_pre.append(new_statements[0]) del new_statements[0] outside_post = [] while new_statements and \ not new_statements[-1].needsFrame(): outside_post.insert(0, new_statements[-1]) del new_statements[-1] if outside_pre or outside_post: from .NodeMakingHelpers import makeStatementsSequenceReplacementNode if new_statements: self.setStatements(new_statements) return makeStatementsSequenceReplacementNode( statements = outside_pre + [self] + \ outside_post, node = self ) else: trace_collection.signalChange( "new_statements", self.getSourceReference(), "Removed useless frame object of '%s'." % self.code_object.getCodeObjectName() ) return makeStatementsSequenceReplacementNode( statements = outside_pre + outside_post, node = self ) else: if statements != new_statements: self.setStatements(new_statements) return self class StatementsFrameModule(StatementsFrameBase): kind = "STATEMENTS_FRAME_MODULE" def __init__(self, statements, code_object, source_ref): StatementsFrameBase.__init__( self, statements = statements, code_object = code_object, guard_mode = "once", source_ref = source_ref ) @staticmethod def hasStructureMember(): return False class StatementsFrameFunction(StatementsFrameBase): kind = "STATEMENTS_FRAME_FUNCTION" def __init__(self, statements, code_object, source_ref): StatementsFrameBase.__init__( self, statements = statements, code_object = code_object, guard_mode = "full", source_ref = source_ref ) @staticmethod def hasStructureMember(): return False class StatementsFrameGenerator(StatementsFrameBase): kind = "STATEMENTS_FRAME_GENERATOR" def __init__(self, statements, code_object, source_ref): StatementsFrameBase.__init__( self, statements = statements, code_object = code_object, guard_mode = "generator", source_ref = source_ref ) @staticmethod def hasStructureMember(): return True class StatementsFrameCoroutine(StatementsFrameBase): kind = "STATEMENTS_FRAME_COROUTINE" def __init__(self, statements, code_object, source_ref): StatementsFrameBase.__init__( self, statements = statements, code_object = code_object, guard_mode = "generator", source_ref = source_ref ) @staticmethod def hasStructureMember(): return True class StatementsFrameAsyncgen(StatementsFrameBase): kind = "STATEMENTS_FRAME_ASYNCGEN" def __init__(self, statements, code_object, source_ref): StatementsFrameBase.__init__( self, statements = statements, code_object = code_object, guard_mode = "generator", source_ref = source_ref ) @staticmethod def hasStructureMember(): return True Nuitka-0.5.28.2/nuitka/nodes/ContainerMakingNodes.py0000644000372000001440000001711713122472300022503 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes that build containers. """ import functools from nuitka.PythonVersions import needsSetLiteralReverseInsertion from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import SideEffectsFromChildrenMixin from .NodeMakingHelpers import ( getComputationResult, makeStatementOnlyNodesFromExpressions, wrapExpressionWithSideEffects ) from .shapes.BuiltinTypeShapes import ( ShapeTypeList, ShapeTypeSet, ShapeTypeTuple ) class ExpressionMakeSequenceBase(SideEffectsFromChildrenMixin, ExpressionChildrenHavingBase): named_children = ( "elements", ) def __init__(self, sequence_kind, elements, source_ref): assert sequence_kind in ("TUPLE", "LIST", "SET"), sequence_kind for element in elements: assert element.isExpression(), element self.sequence_kind = sequence_kind.lower() ExpressionChildrenHavingBase.__init__( self, values = { "elements" : tuple(elements), }, source_ref = source_ref ) def isExpressionMakeSequence(self): return True def getSequenceKind(self): return self.sequence_kind getElements = ExpressionChildrenHavingBase.childGetter("elements") def getSimulator(self): # Abstract method, pylint: disable=no-self-use return None def computeExpression(self, trace_collection): elements = self.getElements() for count, element in enumerate(elements): if element.willRaiseException(BaseException): result = wrapExpressionWithSideEffects( side_effects = elements[:count], new_node = element, old_node = self ) return result, "new_raise", "Sequence creation raises exception" # TODO: CompileTimeConstant should be good enough. for element in elements: if not element.isExpressionConstantRef(): return self, None, None simulator = self.getSimulator() assert simulator is not None # The simulator is in fact callable if not None, pylint: disable=not-callable return getComputationResult( node = self, computation = lambda : simulator( element.getConstant() for element in self.getElements() ), description = "%s with constant arguments." % simulator.__name__.title() ) def mayHaveSideEffectsBool(self): return False def isKnownToBeIterable(self, count): return count is None or count == len(self.getElements()) def isKnownToBeIterableAtMin(self, count): return count <= len(self.getElements()) def getIterationValue(self, count): return self.getElements()[count] def getIterationValueRange(self, start, stop): return self.getElements()[start:stop] @staticmethod def canPredictIterationValues(): return True def getIterationValues(self): return self.getElements() def getTruthValue(self): return self.getIterationLength() > 0 def mayRaiseException(self, exception_type): for element in self.getElements(): if element.mayRaiseException(exception_type): return True return False def mayBeNone(self): return False def computeExpressionDrop(self, statement, trace_collection): result = makeStatementOnlyNodesFromExpressions( expressions = self.getElements() ) return result, "new_statements", """\ Removed sequence creation for unused sequence.""" class ExpressionMakeTuple(ExpressionMakeSequenceBase): kind = "EXPRESSION_MAKE_TUPLE" def __init__(self, elements, source_ref): ExpressionMakeSequenceBase.__init__( self, sequence_kind = "TUPLE", elements = elements, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeTuple def getSimulator(self): return tuple def getIterationLength(self): return len(self.getElements()) class ExpressionMakeList(ExpressionMakeSequenceBase): kind = "EXPRESSION_MAKE_LIST" def __init__(self, elements, source_ref): ExpressionMakeSequenceBase.__init__( self, sequence_kind = "LIST", elements = elements, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeList def getSimulator(self): return list def getIterationLength(self): return len(self.getElements()) def computeExpressionIter1(self, iter_node, trace_collection): result = ExpressionMakeTuple( elements = self.getElements(), source_ref = self.source_ref ) self.replaceWith(result) return iter_node, "new_expression", """\ Iteration over list reduced to tuple.""" class ExpressionMakeSet(ExpressionMakeSequenceBase): kind = "EXPRESSION_MAKE_SET" def __init__(self, elements, source_ref): ExpressionMakeSequenceBase.__init__( self, sequence_kind = "SET", elements = elements, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeSet def getSimulator(self): return set def getIterationLength(self): element_count = len(self.getElements()) # Hashing may consume elements. if element_count >= 2: return None else: return element_count def getIterationMinLength(self): element_count = len(self.getElements()) if element_count == 0: return 0 else: return 1 def getIterationMaxLength(self): return len(self.getElements()) def mayRaiseException(self, exception_type): for element in self.getElements(): if not element.isKnownToBeHashable(): return True if element.mayRaiseException(exception_type): return True return False def computeExpressionIter1(self, iter_node, trace_collection): result = ExpressionMakeTuple( elements = self.getElements(), source_ref = self.source_ref ) self.replaceWith(result) return iter_node, "new_expression", """\ Iteration over set reduced to tuple.""" class ExpressionMakeSetLiteral(ExpressionMakeSet): kind = "EXPRESSION_MAKE_SET_LITERAL" def getSimulator(self): if needsSetLiteralReverseInsertion(): @functools.wraps(set) def mySet(value): return set(reversed(tuple(value))) return mySet else: return set Nuitka-0.5.28.2/nuitka/nodes/StringConcatenationNodes.py0000644000372000001440000000356313122472300023406 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node dedicated to "".join() code pattern. This is used for Python 3.6 fstrings re-formulation and has pretty direct code alternative to actually looking up that method from the empty string object, so it got a dedicated node, also to perform optimizations specific to this. """ from .ExpressionBases import ExpressionChildrenHavingBase from .shapes.BuiltinTypeShapes import ShapeTypeStrOrUnicode class ExpressionStringConcatenation(ExpressionChildrenHavingBase): kind = "EXPRESSION_STRING_CONCATENATION" named_children = ( "values", ) def __init__(self, values, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "values" : tuple(values), }, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeStrOrUnicode def computeExpression(self, trace_collection): # TODO: Could remove itself if only one argument or merge arguments # of mergable types. return self, None, None getValues = ExpressionChildrenHavingBase.childGetter("values") Nuitka-0.5.28.2/nuitka/nodes/AssignNodes.py0000644000372000001440000005666513134660221020675 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Assignment related nodes. The most simple assignment statement ``a = b`` is what we have here. All others are either re-formulated using temporary variables, e.g. ``a, b = c`` or are attribute, slice, subscript assignments. The deletion is a separate node unlike in CPython where assigning to ``NULL`` is internally what deletion is. But deleting is something entirely different to us during code generation, which is why we keep them separate. Tracing assignments in SSA form is the core of optimization for which we use the traces. """ from nuitka import Options from nuitka.ModuleRegistry import getOwnerFromCodeName from .NodeBases import NodeBase, StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, makeStatementsSequenceReplacementNode ) class StatementAssignmentVariableName(StatementChildrenHavingBase): """ Precursor of StatementAssignmentVariable used during tree building phase """ kind = "STATEMENT_ASSIGNMENT_VARIABLE_NAME" named_children = ( "source", ) def __init__(self, source, variable_name, source_ref): assert source is not None, source_ref StatementChildrenHavingBase.__init__( self, values = { "source" : source, }, source_ref = source_ref ) self.variable_name = variable_name def getDetails(self): return { "variable_name" : self.variable_name } def getVariableName(self): return self.variable_name def computeStatement(self, trace_collection): # Only for abc, pylint: disable=no-self-use assert False getAssignSource = StatementChildrenHavingBase.childGetter( "source" ) class StatementDelVariableName(NodeBase): """ Precursor of StatementDelVariable used during tree building phase """ kind = "STATEMENT_DEL_VARIABLE_NAME" __slots__ = "variable_name", "tolerant" def __init__(self, variable_name, tolerant, source_ref): NodeBase.__init__( self, source_ref = source_ref ) self.variable_name = variable_name self.tolerant = tolerant def getDetails(self): return { "variable_name" : self.variable_name, "tolerant" : self.tolerant } def getVariableName(self): return self.variable_name def computeStatement(self, trace_collection): # Only for abc, pylint: disable=no-self-use assert False class StatementAssignmentVariable(StatementChildrenHavingBase): """ Assignment to a variable from an expression. All assignment forms that are not to attributes, slices, subscripts use this. The source might be a complex expression. The target can be any kind of variable, temporary, local, global, etc. Assigning a variable is something we trace in a new version, this is hidden behind target variable reference, which has this version once it can be determined. """ kind = "STATEMENT_ASSIGNMENT_VARIABLE" named_children = ( "source", ) inplace_suspect = None def __init__(self, source, variable, source_ref, version = None): assert source is not None, source_ref if variable is not None: if version is None: version = variable.allocateTargetNumber() self.variable = variable self.variable_version = version StatementChildrenHavingBase.__init__( self, values = { "source" : source, }, source_ref = source_ref ) self.variable_trace = None def getDetail(self): if self.variable is not None: return "to variable %s" % self.variable else: return "to variable %s" % self.variable_name def getDetails(self): return { "variable" : self.variable, } def getDetailsForDisplay(self): return { "variable_name" : self.getVariableName(), "is_temp" : self.variable.isTempVariable(), "owner" : self.variable.getOwner().getCodeName(), } @classmethod def fromXML(cls, provider, source_ref, **args): assert cls is StatementAssignmentVariable, cls owner = getOwnerFromCodeName(args["owner"]) if args["is_temp"] == "True": variable = owner.createTempVariable(args["variable_name"]) else: variable = owner.getProvidedVariable(args["variable_name"]) del args["is_temp"] del args["owner"] version = variable.allocateTargetNumber() return cls( variable = variable, version = version, source_ref = source_ref, **args ) def makeClone(self): if self.variable is not None: version = self.variable.allocateTargetNumber() else: version = None return StatementAssignmentVariable( source = self.getAssignSource().makeClone(), variable = self.variable, version = version, source_ref = self.source_ref ) getAssignSource = StatementChildrenHavingBase.childGetter( "source" ) setAssignSource = StatementChildrenHavingBase.childSetter( "source" ) def getVariableName(self): return self.variable.getName() def getVariable(self): return self.variable def setVariable(self, variable): self.variable = variable self.variable_version = variable.allocateTargetNumber() def getVariableVersion(self): return self.variable_version def markAsInplaceSuspect(self): self.inplace_suspect = True def isInplaceSuspect(self): return self.inplace_suspect def mayRaiseException(self, exception_type): return self.getAssignSource().mayRaiseException(exception_type) def computeStatement(self, trace_collection): # This is very complex stuff, pylint: disable=too-many-branches,too-many-return-statements # TODO: Way too ugly to have global trace kinds just here, and needs to # be abstracted somehow. But for now we let it live here: pylint: disable=too-many-statements source = self.getAssignSource() if source.isExpressionSideEffects(): # If the assignment source has side effects, we can put them into a # sequence and compute that instead. statements = [ makeStatementExpressionOnlyReplacementNode( side_effect, self ) for side_effect in source.getSideEffects() ] statements.append(self) # Remember out parent, we will assign it for the sequence to use. parent = self.parent # Need to update ourselves to no longer reference the side effects, # but go to the wrapped thing. self.setAssignSource(source.getExpression()) result = makeStatementsSequenceReplacementNode( statements = statements, node = self, ) result.parent = parent return ( result.computeStatementsSequence(trace_collection), "new_statements", """\ Side effects of assignments promoted to statements.""" ) # Let assignment source may re-compute first. trace_collection.onExpression(self.getAssignSource()) source = self.getAssignSource() # No assignment will occur, if the assignment source raises, so give up # on this, and return it as the only side effect. if source.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_raise", """\ Assignment raises exception in assigned value, removed assignment.""" variable = self.variable # Not allowed anymore at this point. assert variable is not None # Assigning from and to the same variable, can be optimized away # immediately, there is no point in doing it. Exceptions are of course # module variables that collide with built-in names. # TODO: The variable type checks ought to become unnecessary, as they # are to be a feature of the trace. Assigning from known assigned is # supposed to be possible to eliminate. If we get that wrong, we are # doing it wrong. if not variable.isModuleVariable() and \ source.isExpressionVariableRef() and \ source.getVariable() is variable: # A variable access that has a side effect, must be preserved, # so it can e.g. raise an exception, otherwise we can be fully # removed. if source.mayHaveSideEffects(): result = makeStatementExpressionOnlyReplacementNode( expression = source, node = self ) return result, "new_statements", """\ Reduced assignment of %s from itself to mere access of it.""" % variable.getDescription() else: return None, "new_statements", """\ Removed assignment of %s from itself which is known to be defined.""" % variable.getDescription() # Set-up the trace to the trace collection, so future references will # find this assignment. self.variable_trace = trace_collection.onVariableSet( assign_node = self ) provider = self.getParentVariableProvider() if variable.hasAccessesOutsideOf(provider) is False: last_trace = variable.getMatchingAssignTrace(self) if last_trace is not None: if source.isCompileTimeConstant(): # Can safely forward propagate only non-mutable constants. if not source.isMutable(): if last_trace.hasDefiniteUsages() and not last_trace.getNameUsageCount(): self.variable_trace.setReplacementNode( lambda usage : source.makeClone() ) propagated = True else: propagated = False if not variable.isModuleVariable(): if not last_trace.hasPotentialUsages() and not last_trace.getNameUsageCount(): if not last_trace.getPrevious().isUninitTrace(): # TODO: We could well decide, if that's even necessary, but for now # the "StatementDelVariable" is tasked with that. result = StatementDelVariable( variable = self.variable, version = self.variable_version, tolerant = True, source_ref = self.getSourceReference() ) else: result = None return ( result, "new_statements", "Dropped %s assignment statement to '%s'." % ( "propagated" if propagated else "dead", self.getVariableName() ) ) else: # Something might be possible still. pass elif False and Options.isExperimental("unnamed_yet") and \ source.isExpressionFunctionCreation() and \ source.getFunctionRef().getFunctionBody().isExpressionFunctionBody() and \ not source.getDefaults() and \ not source.getKwDefaults() and \ not source.getAnnotations(): # TODO: These are very mutable, right? provider = self.getParentVariableProvider() if variable.isTempVariable() or \ (not provider.isUnoptimized() and \ not provider.isExpressionClassBody()): if last_trace.getDefiniteUsages() <= 1 and \ not last_trace.hasPotentialUsages() and \ not last_trace.getNameUsageCount(): if last_trace.getDefiniteUsages() == 1: self.variable_trace.setReplacementNode( lambda usage : source.makeClone() ) propagated = True else: propagated = False if not last_trace.getPrevious().isUninitTrace(): # TODO: We could well decide, if that's even necessary. result = StatementDelVariable( variable = self.variable, version = self.variable_version, tolerant = True, source_ref = self.getSourceReference() ) else: result = None return ( result, "new_statements", "Dropped %s assignment statement to '%s'." % ( "propagated" if propagated else "dead", self.getVariableName() ) ) else: # More cases thinkable. pass return self, None, None def needsReleasePreviousValue(self): previous = self.variable_trace.getPrevious() if previous.mustNotHaveValue(): return False elif previous.mustHaveValue(): return True else: return None class StatementDelVariable(NodeBase): """ Deleting a variable. All del forms that are not to attributes, slices, subscripts use this. The target can be any kind of variable, temporary, local, global, etc. Deleting a variable is something we trace in a new version, this is hidden behind target variable reference, which has this version once it can be determined. Tolerance means that the value might be unset. That can happen with re-formulation of ours, and Python3 exception variables. """ kind = "STATEMENT_DEL_VARIABLE" __slots__ = "variable", "variable_version", "variable_trace", "previous_trace", "tolerant" def __init__(self, tolerant, source_ref, variable, version = None): if type(tolerant) is str: tolerant = tolerant == "True" assert tolerant is True or tolerant is False, repr(tolerant) if variable is not None: if version is None: version = variable.allocateTargetNumber() self.variable = variable self.variable_version = version NodeBase.__init__( self, source_ref = source_ref ) self.variable_trace = None self.previous_trace = None self.tolerant = tolerant def getDetail(self): if self.variable is not None: return "to variable %s" % self.variable else: return "to variable %s" % self.variable_name def getDetails(self): return { "variable" : self.variable, "version" : self.variable_version, "tolerant" : self.tolerant } def getDetailsForDisplay(self): return { "variable_name" : self.getVariableName(), "is_temp" : self.variable.isTempVariable(), "owner" : self.variable.getOwner().getCodeName(), "tolerant" : self.tolerant } @classmethod def fromXML(cls, provider, source_ref, **args): assert cls is StatementDelVariable, cls owner = getOwnerFromCodeName(args["owner"]) if args["is_temp"] == "True": variable = owner.createTempVariable(args["variable_name"]) else: variable = owner.getProvidedVariable(args["variable_name"]) del args["is_temp"] del args["owner"] version = variable.allocateTargetNumber() variable.version_number = max(variable.version_number, version) return cls( variable = variable, source_ref = source_ref, **args ) def makeClone(self): if self.variable is not None: version = self.variable.allocateTargetNumber() else: version = None return StatementDelVariable( variable = self.variable, version = version, tolerant = self.tolerant, source_ref = self.source_ref ) # TODO: Value propagation needs to make a difference based on this. def isTolerant(self): return self.tolerant def getVariableName(self): return self.variable.getName() def getVariable(self): return self.variable def setVariable(self, variable): assert self.variable_version is None self.variable = variable self.variable_version = variable.allocateTargetNumber() def getVariableVersion(self): return self.variable_version def computeStatement(self, trace_collection): variable = self.variable self.previous_trace = trace_collection.getVariableCurrentTrace(variable) # First eliminate us entirely if we can. if self.tolerant and self.previous_trace.isUninitTrace(): return ( None, "new_statements", "Removed tolerant 'del' statement of '%s' without effect." % ( self.getVariableName(), ) ) # The "del" is a potential use of a value. TODO: This could be made more # beautiful indication, as it's not any kind of usage. self.previous_trace.addPotentialUsage() # If not tolerant, we may exception exit now during the __del__ if not self.tolerant and not self.previous_trace.mustHaveValue(): trace_collection.onExceptionRaiseExit(BaseException) # Record the deletion, needs to start a new version then. trace_collection.onVariableDel( variable = variable, version = self.variable_version ) trace_collection.onVariableContentEscapes(variable) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # Need to fetch the potentially invalidated variable. A "del" on a # or shared value, may easily assign the global variable in "__del__". self.variable_trace = trace_collection.getVariableCurrentTrace(variable) return self, None, None def mayHaveSideEffects(self): return True def mayRaiseException(self, exception_type): if self.tolerant: return False else: if self.variable_trace is not None: variable = self.getVariable() # Temporary variables deletions won't raise, just because we # don't create them that way. We can avoid going through SSA in # these cases. if variable.isTempVariable(): return False # If SSA knows, that's fine. if self.previous_trace is not None and \ self.previous_trace.mustHaveValue(): return False return True class StatementReleaseVariable(NodeBase): """ Releasing a variable. Just release the value, which of course is not to be used afterwards. Typical code: Function exit, try/finally release of temporary variables. """ kind = "STATEMENT_RELEASE_VARIABLE" __slots__ = "variable", "variable_trace" def __init__(self, variable, source_ref): assert variable is not None, source_ref NodeBase.__init__( self, source_ref = source_ref ) self.variable = variable self.variable_trace = None def getDetail(self): return "of variable %s" % self.variable def getDetails(self): return { "variable" : self.variable } def getDetailsForDisplay(self): return { "variable_name" : self.variable.getName(), "owner" : self.variable.getOwner().getCodeName() } @classmethod def fromXML(cls, provider, source_ref, **args): assert cls is StatementReleaseVariable, cls owner = getOwnerFromCodeName(args["owner"]) assert owner is not None, args["owner"] variable = owner.getProvidedVariable(args["variable_name"]) return cls( variable = variable, source_ref = source_ref ) def getVariable(self): return self.variable def getVariableVersion(self): return self.variable_trace.getVersion() def setVariable(self, variable): self.variable = variable def computeStatement(self, trace_collection): self.variable_trace = trace_collection.getVariableCurrentTrace(self.variable) if self.variable_trace.isUninitTrace(): return ( None, "new_statements", "Uninitialized %s is not released." % ( self.variable.getDescription() ) ) trace_collection.onVariableContentEscapes(self.variable) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) # TODO: We might be able to remove ourselves based on the trace # we belong to. return self, None, None def mayHaveSideEffects(self): # May execute __del__ code, it would be sweet to be able to predict # that another reference will still be active for a value though. return True def mayRaiseException(self, exception_type): # By default, __del__ is not allowed to raise an exception. return False Nuitka-0.5.28.2/nuitka/nodes/ExecEvalNodes.py0000644000372000001440000002627513207537242021146 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes concern with exec and eval builtins. These are the dynamic codes, and as such rather difficult. We would like to eliminate or limit their impact as much as possible, but it's difficult to do. """ from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.PythonVersions import python_version from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import ( convertNoneConstantToNone, makeStatementOnlyNodesFromExpressions ) class ExpressionBuiltinEval(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_EVAL" named_children = ( "source", "globals", "locals" ) def __init__(self, source_code, globals_arg, locals_arg, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source_code, "globals" : globals_arg, "locals" : locals_arg, }, source_ref = source_ref ) getSourceCode = ExpressionChildrenHavingBase.childGetter("source") getGlobals = ExpressionChildrenHavingBase.childGetter("globals") getLocals = ExpressionChildrenHavingBase.childGetter("locals") def computeExpression(self, trace_collection): # TODO: Attempt for constant values to do it. return self, None, None # Note: Python3 only so far. if python_version >= 300: class ExpressionBuiltinExec(ExpressionBuiltinEval): kind = "EXPRESSION_BUILTIN_EXEC" def __init__(self, source_code, globals_arg, locals_arg, source_ref): ExpressionBuiltinEval.__init__( self, source_code = source_code, globals_arg = globals_arg, locals_arg = locals_arg, source_ref = source_ref ) def needsLocalsDict(self): return True def computeExpression(self, trace_collection): # TODO: Attempt for constant values to do it. return self, None, None def computeExpressionDrop(self, statement, trace_collection): if self.getParentVariableProvider().isEarlyClosure(): result = StatementExec( source_code = self.getSourceCode(), globals_arg = self.getGlobals(), locals_arg = self.getLocals(), source_ref = self.getSourceReference() ) return result, "new_statements", """\ Replaced built-in exec call to exec statement in early closure context.""" else: return statement, None, None # Note: Python2 only if python_version < 300: class ExpressionBuiltinExecfile(ExpressionBuiltinEval): kind = "EXPRESSION_BUILTIN_EXECFILE" named_children = ("source", "globals", "locals") def __init__(self, source_code, globals_arg, locals_arg, source_ref): ExpressionBuiltinEval.__init__( self, source_code = source_code, globals_arg = globals_arg, locals_arg = locals_arg, source_ref = source_ref ) def needsLocalsDict(self): return True def computeExpressionDrop(self, statement, trace_collection): # In this case, the copy-back must be done and will only be done # correctly by the code for exec statements. provider = self.getParentVariableProvider() if provider.isExpressionClassBody(): result = StatementExec( source_code = self.getSourceCode(), globals_arg = self.getGlobals(), locals_arg = self.getLocals(), source_ref = self.getSourceReference() ) return result, "new_statements", """\ Changed 'execfile' with unused result to 'exec' on class level.""" else: return statement, None, None class StatementExec(StatementChildrenHavingBase): kind = "STATEMENT_EXEC" named_children = ( "source", "globals", "locals" ) def __init__(self, source_code, globals_arg, locals_arg, source_ref): StatementChildrenHavingBase.__init__( self, values = { "globals" : globals_arg, "locals" : locals_arg, "source" : source_code }, source_ref = source_ref, ) def setChild(self, name, value): if name in ("globals", "locals"): value = convertNoneConstantToNone(value) return StatementChildrenHavingBase.setChild(self, name, value) getSourceCode = StatementChildrenHavingBase.childGetter("source") getGlobals = StatementChildrenHavingBase.childGetter("globals") getLocals = StatementChildrenHavingBase.childGetter("locals") def needsLocalsDict(self): return self.getLocals().mayBeNone() def computeStatement(self, trace_collection): trace_collection.onExpression( expression = self.getSourceCode() ) source_code = self.getSourceCode() if source_code.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if source_code.willRaiseException(BaseException): result = source_code return ( result, "new_raise", """\ Exec statement raises implicitly when determining source code argument.""" ) trace_collection.onExpression( expression = self.getGlobals(), allow_none = True ) globals_arg = self.getGlobals() if globals_arg is not None and globals_arg.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if globals_arg is not None and \ globals_arg.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source_code, globals_arg ) ) return ( result, "new_raise", """\ Exec statement raises implicitly when determining globals argument.""" ) trace_collection.onExpression( expression = self.getLocals(), allow_none = True ) locals_arg = self.getLocals() if locals_arg is not None and locals_arg.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) if locals_arg is not None and \ locals_arg.willRaiseException(BaseException): result = makeStatementOnlyNodesFromExpressions( expressions = ( source_code, globals_arg, locals_arg ) ) return ( result, "new_raise", """\ Exec statement raises implicitly when determining locals argument.""" ) trace_collection.onExceptionRaiseExit( BaseException ) str_value = self.getSourceCode().getStrValue() if False and str_value is not None: # TODO: Don't forget to consider side effects of source code, # locals_arg, and globals_arg. # TODO: This needs to be re-done. exec_body = None return ( exec_body, "new_statements", "In-lined constant exec statement." ) return self, None, None class StatementLocalsDictSync(StatementChildrenHavingBase): kind = "STATEMENT_LOCALS_DICT_SYNC" named_children = ( "locals", ) @calledWithBuiltinArgumentNamesDecorator def __init__(self, locals_arg, source_ref): StatementChildrenHavingBase.__init__( self, values = { "locals" : locals_arg, }, source_ref = source_ref, ) self.previous_traces = None self.variable_traces = None def computeStatement(self, trace_collection): result, change_tags, change_desc = self.computeStatementSubExpressions( trace_collection = trace_collection ) if result is not self: return result, change_tags, change_desc if self.getParentVariableProvider().isCompiledPythonModule(): return None, "new_statements", "Removed sync back to locals without locals." self.previous_traces = trace_collection.onLocalsUsage(self.getParentVariableProvider()) trace_collection.removeAllKnowledge() self.variable_traces = trace_collection.onLocalsUsage(self.getParentVariableProvider()) return self, None, None getLocals = ExpressionChildrenHavingBase.childGetter("locals") def mayRaiseException(self, exception_type): return False class ExpressionBuiltinCompile(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_COMPILE" named_children = ( "source", "filename", "mode", "flags", "dont_inherit", "optimize" ) def __init__(self, source_code, filename, mode, flags, dont_inherit, optimize, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "source" : source_code, "filename" : filename, "mode" : mode, "flags" : flags, "dont_inherit" : dont_inherit, "optimize" : optimize }, source_ref = source_ref ) getSourceCode = ExpressionChildrenHavingBase.childGetter("source") getFilename = ExpressionChildrenHavingBase.childGetter("filename") getMode = ExpressionChildrenHavingBase.childGetter("mode") getFlags = ExpressionChildrenHavingBase.childGetter("flags") getDontInherit = ExpressionChildrenHavingBase.childGetter("dont_inherit") getOptimize = ExpressionChildrenHavingBase.childGetter("optimize") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) # TODO: Attempt for constant values to do it. return self, None, None Nuitka-0.5.28.2/nuitka/nodes/BuiltinTypeNodes.py0000644000372000001440000003025613134660221021705 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in type nodes tuple/list/float/int etc. These are all very simple and have predictable properties, because we know their type and that should allow some important optimizations. """ from nuitka.__past__ import long # pylint: disable=I0021,redefined-builtin from nuitka.optimizations import BuiltinOptimization from nuitka.PythonVersions import python_version from .ConstantRefNodes import makeConstantRefNode from .ExpressionBases import ( ExpressionBuiltinSingleArgBase, ExpressionChildrenHavingBase, ExpressionSpecBasedComputationBase ) from .NodeMakingHelpers import ( makeConstantReplacementNode, wrapExpressionWithNodeSideEffects ) from .shapes.BuiltinTypeShapes import ( ShapeTypeBool, ShapeTypeBytearray, ShapeTypeBytes, ShapeTypeIntOrLong, ShapeTypeLong, ShapeTypeStr, ShapeTypeUnicode ) class ExpressionBuiltinTypeBase(ExpressionBuiltinSingleArgBase): pass class ExpressionBuiltinContainerBase(ExpressionSpecBasedComputationBase): builtin_spec = None named_children = ( "value", ) def __init__(self, value, source_ref): ExpressionSpecBasedComputationBase.__init__( self, values = { "value" : value, }, source_ref = source_ref ) getValue = ExpressionSpecBasedComputationBase.childGetter( "value" ) def computeExpression(self, trace_collection): value = self.getValue() if value is None: return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = () ) elif value.isExpressionConstantXrangeRef(): if value.getIterationLength() <= 256: return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = (value,) ) else: return self, None, None else: return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = (value,) ) class ExpressionBuiltinTuple(ExpressionBuiltinContainerBase): kind = "EXPRESSION_BUILTIN_TUPLE" builtin_spec = BuiltinOptimization.builtin_tuple_spec class ExpressionBuiltinList(ExpressionBuiltinContainerBase): kind = "EXPRESSION_BUILTIN_LIST" builtin_spec = BuiltinOptimization.builtin_list_spec class ExpressionBuiltinSet(ExpressionBuiltinContainerBase): kind = "EXPRESSION_BUILTIN_SET" builtin_spec = BuiltinOptimization.builtin_set_spec class ExpressionBuiltinFrozenset(ExpressionBuiltinContainerBase): kind = "EXPRESSION_BUILTIN_FROZENSET" builtin_spec = BuiltinOptimization.builtin_frozenset_spec class ExpressionBuiltinFloat(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_FLOAT" builtin_spec = BuiltinOptimization.builtin_float_spec class ExpressionBuiltinBool(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_BOOL" builtin_spec = BuiltinOptimization.builtin_bool_spec def computeExpression(self, trace_collection): value = self.getValue() if value is not None: truth_value = self.getValue().getTruthValue() if truth_value is not None: result = wrapExpressionWithNodeSideEffects( new_node = makeConstantReplacementNode( constant = truth_value, node = self, ), old_node = self.getValue() ) return ( result, "new_constant", "Predicted truth value of built-in bool argument" ) return ExpressionBuiltinTypeBase.computeExpression(self, trace_collection) def getTypeShape(self): return ShapeTypeBool class ExpressionBuiltinIntLongBase(ExpressionSpecBasedComputationBase): named_children = ("value", "base") # Note: Version specific, may be allowed or not. try: int(base = 2) except TypeError: base_only_value = False else: base_only_value = True # To be overloaded by child classes with int/long. builtin = int def __init__(self, value, base, source_ref): if value is None and self.base_only_value: value = makeConstantRefNode( constant = '0', source_ref = source_ref, user_provided = True ) ExpressionSpecBasedComputationBase.__init__( self, values = { "value" : value, "base" : base }, source_ref = source_ref ) getValue = ExpressionSpecBasedComputationBase.childGetter("value") getBase = ExpressionSpecBasedComputationBase.childGetter("base") def computeExpression(self, trace_collection): value = self.getValue() base = self.getBase() given_values = [] if value is None: if base is not None: if not self.base_only_value: return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : self.builtin(base = 2), description = """\ %s built-in call with only base argument""" % self.builtin.__name__ ) given_values = () elif base is None: given_values = (value,) else: given_values = (value, base) return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = given_values ) class ExpressionBuiltinInt(ExpressionBuiltinIntLongBase): kind = "EXPRESSION_BUILTIN_INT" builtin_spec = BuiltinOptimization.builtin_int_spec builtin = int def getTypeShape(self): return ShapeTypeIntOrLong class ExpressionBuiltinUnicodeBase(ExpressionSpecBasedComputationBase): named_children = ( "value", "encoding", "errors" ) def __init__(self, value, encoding, errors, source_ref): ExpressionSpecBasedComputationBase.__init__( self, values = { "value" : value, "encoding" : encoding, "errors" : errors }, source_ref = source_ref ) getValue = ExpressionSpecBasedComputationBase.childGetter("value") getEncoding = ExpressionSpecBasedComputationBase.childGetter("encoding") getErrors = ExpressionSpecBasedComputationBase.childGetter("errors") def computeExpression(self, trace_collection): args = [ self.getValue(), self.getEncoding(), self.getErrors() ] while args and args[-1] is None: del args[-1] for arg in args: # The value of that node escapes and could change its contents. trace_collection.removeKnowledge(arg) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = tuple(args) ) if python_version < 300: class ExpressionBuiltinStr(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_STR" builtin_spec = BuiltinOptimization.builtin_str_spec def computeExpression(self, trace_collection): new_node, change_tags, change_desc = ExpressionBuiltinTypeBase.computeExpression( self, trace_collection ) if new_node is self: str_value = self.getValue().getStrValue() if str_value is not None: new_node = wrapExpressionWithNodeSideEffects( new_node = str_value, old_node = self.getValue() ) change_tags = "new_expression" change_desc = "Predicted 'str' built-in result" return new_node, change_tags, change_desc def getTypeShape(self): return ShapeTypeStr class ExpressionBuiltinLong(ExpressionBuiltinIntLongBase): kind = "EXPRESSION_BUILTIN_LONG" builtin_spec = BuiltinOptimization.builtin_long_spec builtin = long def getTypeShape(self): return ShapeTypeLong class ExpressionBuiltinUnicode(ExpressionBuiltinUnicodeBase): kind = "EXPRESSION_BUILTIN_UNICODE" builtin_spec = BuiltinOptimization.builtin_unicode_spec def getTypeShape(self): return ShapeTypeUnicode else: class ExpressionBuiltinStr(ExpressionBuiltinUnicodeBase): kind = "EXPRESSION_BUILTIN_STR" builtin_spec = BuiltinOptimization.builtin_str_spec def getTypeShape(self): return ShapeTypeStr class ExpressionBuiltinBytes(ExpressionBuiltinUnicodeBase): kind = "EXPRESSION_BUILTIN_BYTES" builtin_spec = BuiltinOptimization.builtin_bytes_spec def getTypeShape(self): return ShapeTypeBytes class ExpressionBuiltinBytearray1(ExpressionBuiltinTypeBase): kind = "EXPRESSION_BUILTIN_BYTEARRAY1" builtin_spec = BuiltinOptimization.builtin_bytearray_spec def __init__(self, value, source_ref): ExpressionBuiltinTypeBase.__init__( self, value = value, source_ref = source_ref ) def getTypeShape(self): return ShapeTypeBytearray class ExpressionBuiltinBytearray3(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_BYTEARRAY3" named_children = ("string", "encoding", "errors") builtin_spec = BuiltinOptimization.builtin_bytearray_spec def __init__(self, string, encoding, errors, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "string" : string, "encoding" : encoding, "errors" : errors }, source_ref = source_ref ) getStringArg = ExpressionChildrenHavingBase.childGetter("string") getEncoding = ExpressionChildrenHavingBase.childGetter("encoding") getErrors = ExpressionChildrenHavingBase.childGetter("errors") def computeExpression(self, trace_collection): trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def getTypeShape(self): return ShapeTypeBytearray class ExpressionBuiltinComplex(ExpressionSpecBasedComputationBase): kind = "EXPRESSION_BUILTIN_COMPLEX" named_children = ( "real", "imag", ) builtin_spec = BuiltinOptimization.builtin_complex_spec def __init__(self, real, imag, source_ref): ExpressionSpecBasedComputationBase.__init__( self, values = { "real" : real, "imag" : imag, }, source_ref = source_ref ) def computeExpression(self, trace_collection): start = self.getReal() stop = self.getImag() args = ( start, stop, ) return self.computeBuiltinSpec( trace_collection = trace_collection, given_values = args ) getReal = ExpressionSpecBasedComputationBase.childGetter("real") getImag = ExpressionSpecBasedComputationBase.childGetter("imag") Nuitka-0.5.28.2/nuitka/nodes/__init__.py0000644000372000001440000000150113112214770020172 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/nodes/GeneratorNodes.py0000644000372000001440000001434313207537242021371 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for generator objects and their creations. Generators are turned into normal functions that create generator objects, whose implementation lives here. The creation itself also lives here. """ from nuitka.PythonVersions import python_version from .Checkers import checkStatementsSequenceOrNone from .ExpressionBases import ExpressionChildrenHavingBase from .FunctionNodes import ExpressionFunctionEntryPointBase from .IndicatorMixins import ( MarkLocalsDictIndicatorMixin, MarkUnoptimizedFunctionIndicatorMixin ) from .ReturnNodes import StatementReturn, StatementReturnNone class ExpressionMakeGeneratorObject(ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_GENERATOR_OBJECT" named_children = ( "generator_ref", ) getGeneratorRef = ExpressionChildrenHavingBase.childGetter("generator_ref") def __init__(self, generator_ref, code_object, source_ref): assert generator_ref.getFunctionBody().isExpressionGeneratorObjectBody() ExpressionChildrenHavingBase.__init__( self, values = { "generator_ref" : generator_ref, }, source_ref = source_ref ) self.code_object = code_object self.variable_closure_traces = None def getDetails(self): return { "code_object" : self.code_object } def getCodeObject(self): return self.code_object def computeExpression(self, trace_collection): self.variable_closure_traces = [] for closure_variable in self.getGeneratorRef().getFunctionBody().getClosureVariables(): trace = trace_collection.getVariableCurrentTrace(closure_variable) trace.addClosureUsage() self.variable_closure_traces.append(trace) # TODO: Generator body may know something too. return self, None, None def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False def getClosureVariableVersions(self): return [ (trace.getVariable(), trace.getVersion()) for trace in self.variable_closure_traces ] class ExpressionGeneratorObjectBody(MarkLocalsDictIndicatorMixin, MarkUnoptimizedFunctionIndicatorMixin, ExpressionFunctionEntryPointBase): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, pylint: disable=R0901 kind = "EXPRESSION_GENERATOR_OBJECT_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } if python_version >= 340: qualname_setup = None def __init__(self, provider, name, flags, source_ref): ExpressionFunctionEntryPointBase.__init__( self, provider = provider, name = name, code_prefix = "genexpr" if name == "" else "genobj", flags = flags, source_ref = source_ref ) MarkLocalsDictIndicatorMixin.__init__(self) MarkUnoptimizedFunctionIndicatorMixin.__init__(self, flags) self.needs_generator_return_exit = False self.trace_collection = None def getFunctionName(self): return self.name def markAsNeedsGeneratorReturnHandling(self, value): self.needs_generator_return_exit = max( self.needs_generator_return_exit, value ) def needsGeneratorReturnHandling(self): return self.needs_generator_return_exit == 2 def needsGeneratorReturnExit(self): return bool(self.needs_generator_return_exit) @staticmethod def needsCreation(): return False class StatementGeneratorReturn(StatementReturn): kind = "STATEMENT_GENERATOR_RETURN" def __init__(self, expression, source_ref): StatementReturn.__init__( self, expression = expression, source_ref = source_ref ) def computeStatement(self, trace_collection): trace_collection.onExpression(self.getExpression()) expression = self.getExpression() if expression.mayRaiseException(BaseException): trace_collection.onExceptionRaiseExit(BaseException) if expression.willRaiseException(BaseException): from .NodeMakingHelpers import makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = expression, node = self ) return result, "new_raise", """\ Return statement raises in returned expression, removed return.""" trace_collection.onFunctionReturn() if expression.isExpressionConstantNoneRef(): result = StatementGeneratorReturnNone( source_ref = self.source_ref ) return result, "new_statements", """\ Generator return value is always None.""" return self, None, None @staticmethod def isStatementGeneratorReturn(): return True class StatementGeneratorReturnNone(StatementReturnNone): kind = "STATEMENT_GENERATOR_RETURN_NONE" __slots__ = () def __init__(self, source_ref): StatementReturnNone.__init__( self, source_ref = source_ref ) @staticmethod def isStatementGeneratorReturn(): return True Nuitka-0.5.28.2/nuitka/nodes/BuiltinDictNodes.py0000644000372000001440000001261613134660221021647 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Node for the calls to the 'dict' built-in. """ from nuitka.optimizations.BuiltinOptimization import builtin_dict_spec from .BuiltinIteratorNodes import ExpressionBuiltinIter1 from .ConstantRefNodes import makeConstantRefNode from .DictionaryNodes import ExpressionKeyValuePair, ExpressionMakeDict from .ExpressionBases import ExpressionChildrenHavingBase from .NodeMakingHelpers import wrapExpressionWithNodeSideEffects class ExpressionBuiltinDict(ExpressionChildrenHavingBase): kind = "EXPRESSION_BUILTIN_DICT" named_children = ( "pos_arg", "pairs" ) def __init__(self, pos_arg, pairs, source_ref): assert type(pos_arg) not in (tuple, list), source_ref assert type(pairs) in (tuple, list), source_ref ExpressionChildrenHavingBase.__init__( self, values = { "pos_arg" : pos_arg, "pairs" : tuple( ExpressionKeyValuePair( makeConstantRefNode(key, source_ref), value, value.getSourceReference() ) for key, value in pairs ) }, source_ref = source_ref ) getPositionalArgument = ExpressionChildrenHavingBase.childGetter("pos_arg") getNamedArgumentPairs = ExpressionChildrenHavingBase.childGetter("pairs") def hasOnlyConstantArguments(self): pos_arg = self.getPositionalArgument() if pos_arg is not None and not pos_arg.isCompileTimeConstant(): return False for arg_pair in self.getNamedArgumentPairs(): if not arg_pair.getKey().isCompileTimeConstant(): return False if not arg_pair.getValue().isCompileTimeConstant(): return False return True def computeExpression(self, trace_collection): pos_arg = self.getPositionalArgument() pairs = self.getNamedArgumentPairs() if pos_arg is None: new_node = ExpressionMakeDict( pairs = self.getNamedArgumentPairs(), source_ref = self.source_ref ) # This cannot raise anymore than its arguments, as the keys will # be known as hashable, due to being Python parameters before. return ( new_node, "new_expression", "Replace 'dict' built-in call dictionary creation from arguments." ) pos_iteration_length = pos_arg.getIterationLength() if pos_iteration_length == 0: new_node = ExpressionMakeDict( pairs = self.getNamedArgumentPairs(), source_ref = self.source_ref ) # Maintain potential side effects from the positional arguments. new_node = wrapExpressionWithNodeSideEffects( old_node = ExpressionBuiltinIter1( value = pos_arg, source_ref = self.source_ref ), new_node = new_node ) # Just in case, the iteration may do that. if not pos_arg.hasShapeSlotIter(): trace_collection.onExceptionRaiseExit(BaseException) return ( new_node, "new_expression", "Replace 'dict' built-in call dictionary creation from arguments." ) if pos_iteration_length is not None and \ pos_iteration_length + len(pairs) < 256 and \ self.hasOnlyConstantArguments(): if pos_arg is not None: pos_args = ( pos_arg, ) else: pos_args = None return trace_collection.getCompileTimeComputationResult( node = self, computation = lambda : builtin_dict_spec.simulateCall( (pos_args, self.getNamedArgumentPairs()) ), description = "Replace 'dict' call with constant arguments." ) else: trace_collection.onExceptionRaiseExit(BaseException) return self, None, None def mayRaiseException(self, exception_type): pos_arg = self.getPositionalArgument() # TODO: Determining if it's sufficient is not easy but possible. if pos_arg is not None: return True for arg_pair in self.getNamedArgumentPairs(): if arg_pair.mayRaiseException(exception_type): return True return False def hasShapeDictionaryExact(self): return True Nuitka-0.5.28.2/nuitka/nodes/shapes/0000755000372000001440000000000013207540420017346 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/nodes/shapes/BuiltinTypeShapes.py0000644000372000001440000003561713134660221023351 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Shapes for Python built-in types. """ from nuitka.codegen.c_types.CTypeNuitkaBools import CTypeNuitkaBoolEnum from nuitka.PythonVersions import python_version from .StandardShapes import ShapeBase, ShapeIterator class ShapeTypeNoneType(ShapeBase): @staticmethod def getTypeName(): return "NoneType" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeBool(ShapeBase): @staticmethod def getTypeName(): return "bool" @staticmethod def getCType(): # enum: "0: False", "1": True, "2": unassigned return CTypeNuitkaBoolEnum @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return True @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeInt(ShapeBase): @staticmethod def getTypeName(): return "int" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return True @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeLong(ShapeBase): @staticmethod def getTypeName(): return "long" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return True @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False if python_version < 300: class ShapeTypeIntOrLong(ShapeBase): @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return True @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False else: ShapeTypeIntOrLong = ShapeTypeInt class ShapeTypeFloat(ShapeBase): @staticmethod def getTypeName(): return "float" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return True @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeTuple(ShapeBase): @staticmethod def getTypeName(): return "tuple" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeTupleIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeTupleIterator(ShapeIterator): @staticmethod def getTypeName(): return "tupleiterator" if python_version < 300 else "tuple_iterator" @staticmethod def hasShapeSlotLen(): return False class ShapeTypeList(ShapeBase): @staticmethod def getTypeName(): return "list" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeListIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeListIterator(ShapeIterator): @staticmethod def getTypeName(): return "listiterator" if python_version < 300 else "list_iterator" @staticmethod def hasShapeSlotLen(): return False class ShapeTypeSet(ShapeBase): @staticmethod def getTypeName(): return "set" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def getShapeIter(): return ShapeTypeSetIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeSetIterator(ShapeIterator): @staticmethod def getTypeName(): return "setiterator" if python_version < 300 else "set_iterator" @staticmethod def hasShapeSlotLen(): return False class ShapeTypeFrozenset(ShapeBase): @staticmethod def getTypeName(): return "frozenset" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def getShapeIter(): return ShapeTypeSetIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeDict(ShapeBase): @staticmethod def getTypeName(): return "dict" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeDictIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeDictIterator(ShapeIterator): @staticmethod def getTypeName(): return "dictionary-keyiterator" if python_version < 300 else "dictkey_iterator" @staticmethod def hasShapeSlotLen(): return False class ShapeTypeStr(ShapeBase): @staticmethod def getTypeName(): return "str" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeStrIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeStrIterator(ShapeIterator): @staticmethod def getTypeName(): return "iterator" if python_version < 300 else "str_iterator" @staticmethod def hasShapeSlotLen(): return False if python_version < 300: class ShapeTypeUnicode(ShapeBase): @staticmethod def getTypeName(): return "unicode" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeUnicodeIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeUnicodeIterator(ShapeIterator): @staticmethod def getTypeName(): return "iterator" @staticmethod def hasShapeSlotLen(): return False else: ShapeTypeUnicode = ShapeTypeStr ShapeTypeUnicodeIterator = ShapeTypeStrIterator if python_version < 300: class ShapeTypeStrOrUnicode(ShapeBase): @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return True else: ShapeTypeStrOrUnicode = ShapeTypeStr if python_version >= 300: class ShapeTypeBytes(ShapeBase): @staticmethod def getTypeName(): return "bytes" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeBytesIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeBytesIterator(ShapeIterator): @staticmethod def getTypeName(): return "bytes_iterator" @staticmethod def hasShapeSlotLen(): return False else: ShapeTypeBytes = ShapeTypeStr ShapeTypeBytesIterator = ShapeTypeStrIterator class ShapeTypeBytearray(ShapeBase): @staticmethod def getTypeName(): return "bytearray" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeBytearrayIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeBytearrayIterator(ShapeIterator): @staticmethod def getTypeName(): return "bytearray_iterator" @staticmethod def hasShapeSlotLen(): return False class ShapeTypeEllipsisType(ShapeBase): @staticmethod def getTypeName(): return "ellipsis" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return True class ShapeTypeSlice(ShapeBase): @staticmethod def getTypeName(): return "slice" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeXrange(ShapeBase): @staticmethod def getTypeName(): return "xrange" if python_version < 300 else "range" @staticmethod def hasShapeSlotLen(): return True @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return False @staticmethod def getShapeIter(): return ShapeTypeXrangeIterator @staticmethod def hasShapeSlotContains(): return True class ShapeTypeXrangeIterator(ShapeIterator): @staticmethod def getTypeName(): return "rangeiterator" if python_version < 300 else "range_iterator" @staticmethod def hasShapeSlotLen(): return False class ShapeTypeType(ShapeBase): @staticmethod def getTypeName(): return "type" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeModule(ShapeBase): @staticmethod def getTypeName(): return "module" @staticmethod def hasShapeModule(): return True @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeBuiltinModule(ShapeTypeModule): pass class ShapeTypeFile(ShapeBase): @staticmethod def getTypeName(): return "file" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return True @staticmethod def hasShapeSlotContains(): return True class ShapeTypeStaticmethod(ShapeBase): @staticmethod def getTypeName(): return "staticmethod" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False class ShapeTypeClassmethod(ShapeBase): @staticmethod def getTypeName(): return "classmethod" @staticmethod def hasShapeSlotLen(): return False @staticmethod def hasShapeSlotInt(): return False @staticmethod def hasShapeSlotIter(): return False @staticmethod def hasShapeSlotNext(): return False @staticmethod def hasShapeSlotContains(): return False Nuitka-0.5.28.2/nuitka/nodes/shapes/StandardShapes.py0000644000372000001440000000557513207537242022650 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Standard shapes that commonly appear. """ from nuitka.codegen.c_types.CTypePyObjectPtrs import CTypePyObjectPtr class ShapeBase(object): @staticmethod def getShapeIter(): return ShapeUnknown @staticmethod def getTypeName(): return None @staticmethod def hasShapeModule(): return False @staticmethod def getCType(): return CTypePyObjectPtr class ShapeUnknown(ShapeBase): @staticmethod def hasShapeSlotLen(): return None @staticmethod def hasShapeSlotInt(): return None @staticmethod def hasShapeSlotIter(): return None @staticmethod def hasShapeSlotNext(): return None @staticmethod def hasShapeSlotContains(): return None class ValueShapeBase(object): __slots__ = () def hasShapeSlotLen(self): return self.getTypeShape().hasShapeSlotLen() class ValueShapeUnknown(ValueShapeBase): __slots__ = () @staticmethod def getTypeShape(): return ShapeUnknown # Singleton value for sharing. vshape_unknown = ValueShapeUnknown() class ShapeLargeConstantValue(object): __slots__ = "shape", "size" def __init__(self, size, shape): self.size = size self.shape = shape def getTypeShape(self): return self.shape @staticmethod def isConstant(): return True def hasShapeSlotLen(self): return self.shape.hasShapeSlotLen() class ShapeLargeConstantValuePredictable(ShapeLargeConstantValue): __slots__ = ("predictor",) def __init__(self, size, predictor, shape): ShapeLargeConstantValue.__init__(self, size, shape) self.predictor = predictor class ShapeIterator(ShapeBase): @staticmethod def hasShapeSlotLen(): return None @staticmethod def hasShapeSlotInt(): return None @staticmethod def hasShapeSlotIter(): return True @staticmethod def hasShapeSlotNext(): return True @staticmethod def getShapeIter(): return ShapeIterator @staticmethod def hasShapeSlotContains(): return True Nuitka-0.5.28.2/nuitka/nodes/shapes/__init__.py0000644000372000001440000000150113112214770021455 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/nodes/GlobalsLocalsNodes.py0000644000372000001440000001617313207537242022167 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Globals/locals/dir1 nodes These nodes give access to variables, highly problematic, because using them, the code may change or access anything about them, so nothing can be trusted anymore, if we start to not know where their value goes. The "dir()" call without arguments is reformulated to locals or globals calls. """ from nuitka.PythonVersions import python_version from .ConstantRefNodes import makeConstantRefNode from .DictionaryNodes import ExpressionKeyValuePair, ExpressionMakeDict from .ExpressionBases import ExpressionBase, ExpressionBuiltinSingleArgBase from .NodeBases import StatementChildrenHavingBase from .VariableRefNodes import ExpressionVariableRef class ExpressionBuiltinGlobals(ExpressionBase): kind = "EXPRESSION_BUILTIN_GLOBALS" def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) def computeExpressionRaw(self, trace_collection): return self, None, None def mayHaveSideEffects(self): return False def mayRaiseException(self, exception_type): return False def mayBeNone(self): return False class ExpressionBuiltinLocalsBase(ExpressionBase): # Base classes can be abstract, pylint: disable=abstract-method __slots__ = ("variable_versions",) def __init__(self, source_ref): ExpressionBase.__init__( self, source_ref = source_ref ) self.variable_versions = None def mayHaveSideEffects(self): if python_version < 300: return False provider = self.getParentVariableProvider() assert not provider.isCompiledPythonModule() # TODO: That's not true. return self.getParentVariableProvider().isExpressionClassBody() def mayRaiseException(self, exception_type): return self.mayHaveSideEffects() def mayBeNone(self): return False def getVariableVersions(self): return self.variable_versions class ExpressionBuiltinLocalsUpdated(ExpressionBuiltinLocalsBase): kind = "EXPRESSION_BUILTIN_LOCALS_UPDATED" def needsLocalsDict(self): return not self.getParent().isStatementReturn() def computeExpressionRaw(self, trace_collection): # Just inform the collection that all escaped. self.variable_versions = trace_collection.onLocalsUsage(self.getParentVariableProvider()) if self.getParent().isStatementReturn(): result = ExpressionBuiltinLocalsCopy( source_ref = self.getSourceReference() ) return result, "new_expression", "Locals does not escape, no write back needed." return self, None, None class ExpressionBuiltinLocalsCopy(ExpressionBuiltinLocalsBase): kind = "EXPRESSION_BUILTIN_LOCALS_COPY" def needsLocalsDict(self): return False def computeExpressionRaw(self, trace_collection): # Just inform the collection that all escaped. self.variable_versions = trace_collection.onLocalsUsage(self.getParentVariableProvider()) for variable, version in self.variable_versions: trace = trace_collection.getVariableTrace(variable, version) if not trace.mustHaveValue() and not trace.mustNotHaveValue(): return self, None, None # Other locals elsewhere. if trace.getNameUsageCount() > 1: return self, None, None pairs = [] for variable, version in self.variable_versions: trace = trace_collection.getVariableTrace(variable, version) if trace.mustHaveValue(): pairs.append( ExpressionKeyValuePair( key = makeConstantRefNode( constant = variable.getName(), user_provided = True, source_ref = self.source_ref ), value = ExpressionVariableRef( variable = variable, source_ref = self.source_ref ), source_ref = self.source_ref ) ) # Locals is sorted of course. def _sorted(pairs): names = self.getParentVariableProvider().getLocalVariableNames() return sorted( pairs, key = lambda pair: names.index( pair.getKey().getCompileTimeConstant() ), ) result = ExpressionMakeDict( pairs = _sorted(pairs), source_ref = self.source_ref ) return result, "new_expression", "Statically predicted locals dictionary." class StatementSetLocals(StatementChildrenHavingBase): kind = "STATEMENT_SET_LOCALS" named_children = ( "new_locals", ) def __init__(self, new_locals, source_ref): StatementChildrenHavingBase.__init__( self, values = { "new_locals" : new_locals, }, source_ref = source_ref ) def needsLocalsDict(self): return True def mayRaiseException(self, exception_type): return self.getNewLocals().mayRaiseException(exception_type) getNewLocals = StatementChildrenHavingBase.childGetter("new_locals") def computeStatement(self, trace_collection): # Make sure that we don't even assume "unset" of things not set yet for # anything. trace_collection.removeAllKnowledge() trace_collection.onExpression(self.getNewLocals()) new_locals = self.getNewLocals() if new_locals.willRaiseException(BaseException): from .NodeMakingHelpers import \ makeStatementExpressionOnlyReplacementNode result = makeStatementExpressionOnlyReplacementNode( expression = new_locals, node = self ) return result, "new_raise", """\ Setting locals already raises implicitly building new locals.""" return self, None, None class ExpressionBuiltinDir1(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_DIR1" def computeExpression(self, trace_collection): # TODO: Quite some cases should be possible to predict. return self, None, None def mayBeNone(self): return False Nuitka-0.5.28.2/nuitka/nodes/FutureSpecs.py0000644000372000001440000001072513134660221020713 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Specification record for future flags. A source reference also implies a specific set of future flags in use by the parser at that location. Can be different inside a module due to e.g. the in-lining of "exec" statements with their own future imports, or in-lining of code from other modules. """ from nuitka.PythonVersions import python_version from nuitka.utils.InstanceCounters import counted_del, counted_init # These defaults have changed with Python versions. _future_division_default = python_version >= 300 _future_absolute_import_default = python_version >= 300 _future_generator_stop_default = python_version >= 370 class FutureSpec(object): @counted_init def __init__(self): self.future_division = _future_division_default self.unicode_literals = False self.absolute_import = _future_absolute_import_default self.future_print = False self.barry_bdfl = False self.generator_stop = _future_generator_stop_default __del__ = counted_del() def __repr__(self): return "" % ','.join(self.asFlags()) def clone(self): result = FutureSpec() result.future_division = self.future_division result.unicode_literals = self.unicode_literals result.absolute_import = self.absolute_import result.future_print = self.future_print result.barry_bdfl = self.barry_bdfl return result def isFutureDivision(self): return self.future_division def enableFutureDivision(self): self.future_division = True def isFuturePrint(self): return self.future_print def enableFuturePrint(self): self.future_print = True def enableUnicodeLiterals(self): self.unicode_literals = True def enableAbsoluteImport(self): self.absolute_import = True def enableBarry(self): self.barry_bdfl = True def enableGeneratorStop(self): self.generator_stop = True def isAbsoluteImport(self): return self.absolute_import def isGeneratorStop(self): return self.generator_stop def asFlags(self): """ Create a list of C identifiers to represent the flag values. This is for use in code generation and to restore from saved modules. """ result = [] if python_version < 300 and self.future_division: result.append("CO_FUTURE_DIVISION") if self.unicode_literals: result.append("CO_FUTURE_UNICODE_LITERALS") if python_version < 300 and self.absolute_import: result.append("CO_FUTURE_ABSOLUTE_IMPORT") if python_version < 300 and self.future_print: result.append("CO_FUTURE_PRINT_FUNCTION") if python_version >= 300 and self.barry_bdfl: result.append("CO_FUTURE_BARRY_AS_BDFL") if python_version >= 350 and self.generator_stop: result.append("CO_FUTURE_GENERATOR_STOP") return tuple(result) def fromFlags(flags): flags = flags.split(',') result = FutureSpec() if "CO_FUTURE_DIVISION" in flags: result.enableFutureDivision() if "CO_FUTURE_UNICODE_LITERALS" in flags: result.enableUnicodeLiterals() if "CO_FUTURE_ABSOLUTE_IMPORT" in flags: result.enableAbsoluteImport() if "CO_FUTURE_PRINT_FUNCTION" in flags: result.enableFuturePrint() if "CO_FUTURE_BARRY_AS_BDFL" in flags: result.enableBarry() if "CO_FUTURE_GENERATOR_STOP" in flags: result.enableGeneratorStop() # Check if we are going to give similar results than what we got. assert tuple(result.asFlags()) == tuple(flags), (result, result.asFlags(), flags) return result Nuitka-0.5.28.2/nuitka/nodes/CoroutineNodes.py0000644000372000001440000001216013207537242021405 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nodes for coroutine objects and their creations. Coroutines are turned into normal functions that create generator objects, whose implementation lives here. The creation itself also lives here. """ from .Checkers import checkStatementsSequenceOrNone from .ExpressionBases import ExpressionChildrenHavingBase from .FunctionNodes import ExpressionFunctionEntryPointBase from .IndicatorMixins import MarkLocalsDictIndicatorMixin class ExpressionMakeCoroutineObject(ExpressionChildrenHavingBase): kind = "EXPRESSION_MAKE_COROUTINE_OBJECT" named_children = ( "coroutine_ref", ) getCoroutineRef = ExpressionChildrenHavingBase.childGetter("coroutine_ref") def __init__(self, coroutine_ref, code_object, source_ref): assert coroutine_ref.getFunctionBody().isExpressionCoroutineObjectBody() ExpressionChildrenHavingBase.__init__( self, values = { "coroutine_ref" : coroutine_ref, }, source_ref = source_ref ) self.code_object = code_object self.variable_closure_traces = None def getDetails(self): return { "code_object" : self.code_object } def getCodeObject(self): return self.code_object def getDetailsForDisplay(self): return { "coroutine" : self.getCoroutineRef().getFunctionBody().getCodeName() } def computeExpression(self, trace_collection): self.variable_closure_traces = [] for closure_variable in self.getCoroutineRef().getFunctionBody().getClosureVariables(): trace = trace_collection.getVariableCurrentTrace(closure_variable) trace.addClosureUsage() self.variable_closure_traces.append(trace) # TODO: Coroutine body may know something too. return self, None, None def mayRaiseException(self, exception_type): return False def mayHaveSideEffects(self): return False def getClosureVariableVersions(self): return [ (trace.getVariable(), trace.getVersion()) for trace in self.variable_closure_traces ] class ExpressionCoroutineObjectBody(MarkLocalsDictIndicatorMixin, ExpressionFunctionEntryPointBase): # We really want these many ancestors, as per design, we add properties via # base class mix-ins a lot, pylint: disable=R0901 kind = "EXPRESSION_COROUTINE_OBJECT_BODY" named_children = ( "body", ) checkers = { # TODO: Is "None" really an allowed value. "body" : checkStatementsSequenceOrNone } qualname_setup = None def __init__(self, provider, name, flags, source_ref): ExpressionFunctionEntryPointBase.__init__( self, provider = provider, name = name, code_prefix = "coroutine", flags = flags, source_ref = source_ref ) MarkLocalsDictIndicatorMixin.__init__(self) self.needs_generator_return_exit = False def getFunctionName(self): return self.name def markAsNeedsGeneratorReturnHandling(self, value): self.needs_generator_return_exit = max( self.needs_generator_return_exit, value ) def needsGeneratorReturnHandling(self): return self.needs_generator_return_exit == 2 def needsGeneratorReturnExit(self): return bool(self.needs_generator_return_exit) @staticmethod def needsCreation(): return False @staticmethod def isUnoptimized(): return False class ExpressionAsyncWait(ExpressionChildrenHavingBase): kind = "EXPRESSION_ASYNC_WAIT" named_children = ("expression",) def __init__(self, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.exception_preserving = False def markAsExceptionPreserving(self): self.exception_preserving = True def isExceptionPreserving(self): return self.exception_preserving def computeExpression(self, trace_collection): # TODO: Might be predictable based awaitable analysis or for constants. return self, None, None getValue = ExpressionChildrenHavingBase.childGetter("expression") Nuitka-0.5.28.2/nuitka/nodes/CallNodes.py0000644000372000001440000001476713207537242020330 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Call node Function calls and generally calling expressions are the same thing. This is very important, because it allows to predict most things, and avoid expensive operations like parameter parsing at run time. There will be a method "computeExpressionCall" to aid predicting them in other nodes. """ from .ExpressionBases import ExpressionChildrenHavingBase class ExpressionCall(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL" named_children = ( "called", "args", "kw" ) def __init__(self, called, args, kw, source_ref): assert called.isExpression() assert args.isExpression() assert kw.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, "args" : args, "kw" : kw, }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") getCallArgs = ExpressionChildrenHavingBase.childGetter("args") getCallKw = ExpressionChildrenHavingBase.childGetter("kw") def isExpressionCall(self): return True def computeExpression(self, trace_collection): called = self.getCalled() return called.computeExpressionCall( call_node = self, call_args = self.getCallArgs(), call_kw = self.getCallKw(), trace_collection = trace_collection ) def extractSideEffectsPreCall(self): args = self.getCallArgs() kw = self.getCallKw() return args.extractSideEffects() + kw.extractSideEffects() class ExpressionCallNoKeywords(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL_NO_KEYWORDS" named_children = ( "called", "args" ) def __init__(self, called, args, source_ref): assert called.isExpression() assert args.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, "args" : args }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") getCallArgs = ExpressionChildrenHavingBase.childGetter("args") def computeExpression(self, trace_collection): return self.getCalled().computeExpressionCall( call_node = self, call_args = self.getCallArgs(), call_kw = None, trace_collection = trace_collection ) @staticmethod def getCallKw(): return None def isExpressionCall(self): return True def extractSideEffectsPreCall(self): args = self.getCallArgs() return args.extractSideEffects() class ExpressionCallKeywordsOnly(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL_KEYWORDS_ONLY" named_children = ( "called", "kw" ) def __init__(self, called, kw, source_ref): assert called.isExpression() assert kw.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, "kw" : kw }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") getCallKw = ExpressionChildrenHavingBase.childGetter("kw") def computeExpression(self, trace_collection): called = self.getCalled() return called.computeExpressionCall( call_node = self, call_args = None, call_kw = self.getCallKw(), trace_collection = trace_collection ) @staticmethod def getCallArgs(): return None def isExpressionCall(self): return True def extractSideEffectsPreCall(self): kw = self.getCallKw() return kw.extractSideEffects() class ExpressionCallEmpty(ExpressionChildrenHavingBase): kind = "EXPRESSION_CALL_EMPTY" named_children = ( "called", ) def __init__(self, called, source_ref): assert called.isExpression() ExpressionChildrenHavingBase.__init__( self, values = { "called" : called, }, source_ref = source_ref ) getCalled = ExpressionChildrenHavingBase.childGetter("called") def computeExpression(self, trace_collection): called = self.getCalled() return called.computeExpressionCall( call_node = self, call_args = None, call_kw = None, trace_collection = trace_collection ) @staticmethod def getCallKw(): return None @staticmethod def getCallArgs(): return None def isExpressionCall(self): return True @staticmethod def extractSideEffectsPreCall(): return () def makeExpressionCall(called, args, kw, source_ref): """ Make the most simple call node possible. By avoiding the more complex classes, we can achieve that there is less work to do for analysis. """ has_kw = kw is not None and \ (not kw.isExpressionConstantRef() or kw.getConstant() != {}) has_args = args is not None and \ (not args.isExpressionConstantRef() or args.getConstant() != ()) if has_kw: if has_args: return ExpressionCall(called, args, kw, source_ref) else: return ExpressionCallKeywordsOnly(called, kw, source_ref) else: if has_args: return ExpressionCallNoKeywords(called, args, source_ref) else: return ExpressionCallEmpty(called, source_ref) Nuitka-0.5.28.2/nuitka/nodes/BuiltinLenNodes.py0000644000372000001440000000376113122472300021477 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Built-in iterator nodes. These play a role in for loops, and in unpacking. They can something be predicted to succeed or fail, in which case, code can become less complex. The length of things is an important optimization issue for these to be good. """ from nuitka.optimizations import BuiltinOptimization from .ExpressionBases import ExpressionBuiltinSingleArgBase from .shapes.BuiltinTypeShapes import ShapeTypeIntOrLong class ExpressionBuiltinLen(ExpressionBuiltinSingleArgBase): kind = "EXPRESSION_BUILTIN_LEN" builtin_spec = BuiltinOptimization.builtin_len_spec def getIntegerValue(self): value = self.getValue() if value.hasShapeSlotLen(): return value.getIterationLength() else: return None def computeExpression(self, trace_collection): return self.getValue().computeExpressionLen( len_node = self, trace_collection = trace_collection ) def getTypeShape(self): return ShapeTypeIntOrLong def mayRaiseException(self, exception_type): value = self.getValue() if value.mayRaiseException(exception_type): return True return not value.getTypeShape().hasShapeSlotLen() Nuitka-0.5.28.2/nuitka/nodes/ConditionalNodes.py0000644000372000001440000005570713134660221021710 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Conditional nodes. These is the conditional expression '(a if b else c)' and the conditional statement, 'if a: ... else: ...' and there is no 'elif', because that is expressed via nesting of conditional statements. """ from nuitka.optimizations.TraceCollections import TraceCollectionBranch from .BuiltinTypeNodes import ExpressionBuiltinBool from .Checkers import checkStatementsSequenceOrNone from .ExpressionBases import ExpressionChildrenHavingBase from .NodeBases import StatementChildrenHavingBase from .NodeMakingHelpers import ( makeStatementExpressionOnlyReplacementNode, wrapExpressionWithNodeSideEffects, wrapStatementWithSideEffects ) class ExpressionConditional(ExpressionChildrenHavingBase): kind = "EXPRESSION_CONDITIONAL" named_children = ( "condition", "expression_yes", "expression_no" ) def __init__(self, condition, expression_yes, expression_no, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "condition" : condition, "expression_yes" : expression_yes, "expression_no" : expression_no }, source_ref = source_ref ) self.merge_traces = None def getBranches(self): return ( self.getExpressionYes(), self.getExpressionNo() ) getExpressionYes = ExpressionChildrenHavingBase.childGetter( "expression_yes" ) getExpressionNo = ExpressionChildrenHavingBase.childGetter( "expression_no" ) getCondition = ExpressionChildrenHavingBase.childGetter( "condition" ) def computeExpressionRaw(self, trace_collection): # Query the truth value after the expression is evaluated, once it is # evaluated in onExpression, it is known. trace_collection.onExpression( expression = self.getCondition() ) condition = self.getCondition() condition_may_raise = condition.mayRaiseException(BaseException) if condition_may_raise: trace_collection.onExceptionRaiseExit( BaseException ) # No need to look any further, if the condition raises, the branches do # not matter at all. if condition.willRaiseException(BaseException): return condition, "new_raise", """\ Conditional expression already raises implicitly in condition, removing \ branches.""" if not condition_may_raise and condition.mayRaiseExceptionBool(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) # Decide this based on truth value of condition. truth_value = condition.getTruthValue() # TODO: We now know that condition evaluates to true for the yes branch # and to not true for no branch, the branch should know that. yes_branch = self.getExpressionYes() # Continue to execute for yes branch unless we know it's not going to be # relevant. if truth_value is not False: branch_yes_collection = TraceCollectionBranch( parent = trace_collection, name = "conditional expression yes branch" ) branch_yes_collection.computeBranch( branch = yes_branch ) # May have just gone away, so fetch it again. yes_branch = self.getExpressionYes() # If it's aborting, it doesn't contribute to merging. if yes_branch.willRaiseException(BaseException): branch_yes_collection = None else: branch_yes_collection = None no_branch = self.getExpressionNo() # Continue to execute for yes branch. if truth_value is not True: branch_no_collection = TraceCollectionBranch( parent = trace_collection, name = "conditional expression no branch" ) branch_no_collection.computeBranch( branch = no_branch ) # May have just gone away, so fetch it again. no_branch = self.getExpressionNo() # If it's aborting, it doesn't contribute to merging. if no_branch.willRaiseException(BaseException): branch_no_collection = None else: branch_no_collection = None # Merge into parent execution. self.merge_traces = trace_collection.mergeBranches( branch_yes_collection, branch_no_collection ) if truth_value is True: return ( wrapExpressionWithNodeSideEffects( new_node = self.getExpressionYes(), old_node = condition ), "new_expression", "Conditional expression predicted to yes case" ) elif truth_value is False: return ( wrapExpressionWithNodeSideEffects( new_node = self.getExpressionNo(), old_node = condition ), "new_expression", "Conditional expression predicted to no case" ) else: return self, None, None def mayHaveSideEffectsBool(self): if self.getCondition().mayHaveSideEffectsBool(): return True if self.getExpressionYes().mayHaveSideEffectsBool(): return True if self.getExpressionNo().mayHaveSideEffectsBool(): return True return False def mayRaiseException(self, exception_type): condition = self.getCondition() if condition.mayRaiseException(exception_type): return True if condition.mayRaiseExceptionBool(exception_type): return True yes_branch = self.getExpressionYes() # Handle branches that became empty behind our back if yes_branch is not None and \ yes_branch.mayRaiseException(exception_type): return True no_branch = self.getExpressionNo() # Handle branches that became empty behind our back if no_branch is not None and \ no_branch.mayRaiseException(exception_type): return True return False def mayRaiseExceptionBool(self, exception_type): if self.getCondition().mayRaiseExceptionBool(exception_type): return True if self.getExpressionYes().mayRaiseExceptionBool(exception_type): return True if self.getExpressionNo().mayRaiseExceptionBool(exception_type): return True return False class ExpressionConditionalBoolBase(ExpressionChildrenHavingBase): named_children = ( "left", "right" ) def __init__(self, left, right, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "left" : left, "right" : right, }, source_ref = source_ref ) self.merge_traces = None getLeft = ExpressionChildrenHavingBase.childGetter( "left" ) getRight = ExpressionChildrenHavingBase.childGetter( "right" ) def computeExpressionRaw(self, trace_collection): # Query the truth value after the expression is evaluated, once it is # evaluated in onExpression, it is known. trace_collection.onExpression( expression = self.getLeft() ) left = self.getLeft() left_may_raise = left.mayRaiseException(BaseException) if left_may_raise: trace_collection.onExceptionRaiseExit( BaseException ) # No need to look any further, if the condition raises, the branches do # not matter at all. if left.willRaiseException(BaseException): return left, "new_raise", """\ Conditional %s statements already raises implicitly in condition, removing \ branches.""" % self.conditional_kind if not left_may_raise and left.mayRaiseExceptionBool(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) # Decide this based on truth value of condition. truth_value = left.getTruthValue() truth_value_use_left = self.conditional_kind == "or" truth_value_use_right = not truth_value_use_left right = self.getRight() # Continue to execute for yes branch unless we know it's not going to be # relevant. if truth_value is not truth_value_use_left: # TODO: We now know that left evaluates and we should tell the # branch that. branch_yes_collection = TraceCollectionBranch( parent = trace_collection, name = "boolean %s right branch" % self.conditional_kind ) branch_yes_collection.computeBranch( branch = right ) # May have just gone away, so fetch it again. right = self.getRight() # If it's aborting, it doesn't contribute to merging. if right.willRaiseException(BaseException): branch_yes_collection = None else: branch_yes_collection = None if branch_yes_collection: # Merge into parent execution. self.merge_traces = trace_collection.mergeBranches( branch_yes_collection, None ) else: self.merge_traces = None if truth_value is truth_value_use_left: return ( left, "new_expression", "Conditional '%s' expression predicted to left value." % self.conditional_kind ) elif truth_value is truth_value_use_right: return ( wrapExpressionWithNodeSideEffects( new_node = right, old_node = left ), "new_expression", "Conditional '%s' expression predicted right value." % self.conditional_kind ) else: return self, None, None def mayRaiseException(self, exception_type): left = self.getLeft() if left.mayRaiseException(exception_type): return True if left.mayRaiseExceptionBool(exception_type): return True right = self.getRight() if right.mayRaiseException(exception_type): return True return False def mayRaiseExceptionBool(self, exception_type): if self.getLeft().mayRaiseExceptionBool(exception_type): return True if self.getRight().mayRaiseExceptionBool(exception_type): return True return False def mayHaveSideEffectsBool(self): if self.getLeft().mayHaveSideEffectsBool(): return True if self.getRight().mayHaveSideEffectsBool(): return True return False class ExpressionConditionalOR(ExpressionConditionalBoolBase): kind = "EXPRESSION_CONDITIONAL_OR" def __init__(self, left, right, source_ref): ExpressionConditionalBoolBase.__init__( self, left = left, right = right, source_ref = source_ref ) self.conditional_kind = "or" class ExpressionConditionalAND(ExpressionConditionalBoolBase): kind = "EXPRESSION_CONDITIONAL_AND" def __init__(self, left, right, source_ref): ExpressionConditionalBoolBase.__init__( self, left = left, right = right, source_ref = source_ref ) self.conditional_kind = "and" class StatementConditional(StatementChildrenHavingBase): kind = "STATEMENT_CONDITIONAL" named_children = ( "condition", "yes_branch", "no_branch" ) checkers = { "yes_branch" : checkStatementsSequenceOrNone, "no_branch" : checkStatementsSequenceOrNone, } def __init__(self, condition, yes_branch, no_branch, source_ref): StatementChildrenHavingBase.__init__( self, values = { "condition" : condition, "yes_branch" : yes_branch, "no_branch" : no_branch }, source_ref = source_ref ) self.merge_traces = None getCondition = StatementChildrenHavingBase.childGetter("condition") getBranchYes = StatementChildrenHavingBase.childGetter("yes_branch") setBranchYes = StatementChildrenHavingBase.childSetter("yes_branch") getBranchNo = StatementChildrenHavingBase.childGetter("no_branch") setBranchNo = StatementChildrenHavingBase.childSetter("no_branch") def isStatementAborting(self): yes_branch = self.getBranchYes() if yes_branch is not None: if yes_branch.isStatementAborting(): no_branch = self.getBranchNo() if no_branch is not None: return no_branch.isStatementAborting() else: return False else: return False else: return False def mayRaiseException(self, exception_type): condition = self.getCondition() if condition.mayRaiseException(exception_type): return True if condition.mayRaiseExceptionBool(exception_type): return True yes_branch = self.getBranchYes() # Handle branches that became empty behind our back if yes_branch is not None and \ yes_branch.mayRaiseException(exception_type): return True no_branch = self.getBranchNo() # Handle branches that became empty behind our back if no_branch is not None and \ no_branch.mayRaiseException(exception_type): return True return False def needsFrame(self): condition = self.getCondition() if condition.mayRaiseException(BaseException): return True if condition.mayRaiseExceptionBool(BaseException): return True yes_branch = self.getBranchYes() # Handle branches that became empty behind our back if yes_branch is not None and \ yes_branch.needsFrame(): return True no_branch = self.getBranchNo() # Handle branches that became empty behind our back if no_branch is not None and \ no_branch.needsFrame(): return True return False def computeStatement(self, trace_collection): # This is rather complex stuff, pylint: disable=too-many-branches,too-many-statements trace_collection.onExpression( expression = self.getCondition() ) condition = self.getCondition() condition_may_raise = condition.mayRaiseException(BaseException) if condition_may_raise: trace_collection.onExceptionRaiseExit( BaseException ) # No need to look any further, if the condition raises, the branches do # not matter at all. if condition.willRaiseException(BaseException): result = makeStatementExpressionOnlyReplacementNode( expression = condition, node = self ) return result, "new_raise", """\ Conditional statements already raises implicitly in condition, removing \ branches.""" if not condition_may_raise and condition.mayRaiseExceptionBool(BaseException): trace_collection.onExceptionRaiseExit( BaseException ) # Query the truth value after the expression is evaluated, once it is # evaluated in onExpression, it is known. truth_value = condition.getTruthValue() # TODO: We now know that condition evaluates to true for the yes branch # and to not true for no branch, the branch collection should know that. yes_branch = self.getBranchYes() no_branch = self.getBranchNo() # Handle branches that became empty behind our back. if yes_branch is not None: if not yes_branch.getStatements(): yes_branch = None if no_branch is not None: if not no_branch.getStatements(): no_branch = None # Consider to not remove branches that we know won't be taken. if yes_branch is not None and truth_value is False: trace_collection.signalChange( tags = "new_statements", source_ref = yes_branch.source_ref, message = "Removed conditional branch not taken due to false condition value." ) self.setBranchYes(None) yes_branch = None if no_branch is not None and truth_value is True: trace_collection.signalChange( tags = "new_statements", source_ref = no_branch.source_ref, message = "Removed 'else' branch not taken due to true condition value." ) self.setBranchNo(None) no_branch = None # Continue to execute for yes branch unless we know it's not going to be # relevant. if yes_branch is not None: branch_yes_collection = TraceCollectionBranch( parent = trace_collection, name = "conditional yes branch", ) branch_yes_collection.computeBranch( branch = yes_branch ) # May have just gone away, so fetch it again. yes_branch = self.getBranchYes() # If it's aborting, it doesn't contribute to merging. if yes_branch is None or yes_branch.isStatementAborting(): branch_yes_collection = None else: branch_yes_collection = None # Continue to execute for yes branch. if no_branch is not None: branch_no_collection = TraceCollectionBranch( parent = trace_collection, name = "conditional no branch" ) branch_no_collection.computeBranch( branch = no_branch ) # May have just gone away, so fetch it again. no_branch = self.getBranchNo() # If it's aborting, it doesn't contribute to merging. if no_branch is None or no_branch.isStatementAborting(): branch_no_collection = None else: branch_no_collection = None # Merge into parent execution. self.merge_traces = trace_collection.mergeBranches( branch_yes_collection, branch_no_collection ) # Both branches may have become empty, which case, the statement needs # not remain. if yes_branch is None and no_branch is None: # Need to keep the boolean check. if truth_value is None: condition = ExpressionBuiltinBool( value = condition, source_ref = condition.getSourceReference() ) if condition.mayHaveSideEffects(): # With both branches eliminated, the condition remains as a side # effect. result = makeStatementExpressionOnlyReplacementNode( expression = condition, node = self ) return result, "new_statements", """\ Both branches have no effect, reduced to evaluate condition.""" else: return None, "new_statements", """\ Removed conditional statement without effect.""" # Note: Checking the condition late, so that the surviving branch got # processed already. Returning without doing that, will corrupt the SSA # results. TODO: Could pretend the other branch didn't exist to save # complexity the merging of processing. if truth_value is not None: if truth_value is True: choice = "true" new_statement = self.getBranchYes() else: choice = "false" new_statement = self.getBranchNo() new_statement = wrapStatementWithSideEffects( new_node = new_statement, old_node = condition, allow_none = True # surviving branch may empty ) return new_statement, "new_statements", """\ Condition for branch was predicted to be always %s.""" % choice # If there is no "yes" branch, remove that. Maybe a bad idea though. if yes_branch is None: # Would be eliminated already, if there wasn't any "no" branch # either. assert no_branch is not None from .OperatorNodes import ExpressionOperationNOT new_statement = StatementConditional( condition = ExpressionOperationNOT( operand = condition, source_ref = condition.getSourceReference() ), yes_branch = no_branch, no_branch = None, source_ref = self.getSourceReference() ) return new_statement, "new_statements", """\ Empty 'yes' branch for conditional statement treated with inverted condition check.""" return self, None, None def mayReturn(self): yes_branch = self.getBranchYes() if yes_branch is not None and yes_branch.mayReturn(): return True no_branch = self.getBranchNo() if no_branch is not None and no_branch.mayReturn(): return True return False def mayBreak(self): yes_branch = self.getBranchYes() if yes_branch is not None and yes_branch.mayBreak(): return True no_branch = self.getBranchNo() if no_branch is not None and no_branch.mayBreak(): return True return False def mayContinue(self): yes_branch = self.getBranchYes() if yes_branch is not None and yes_branch.mayContinue(): return True no_branch = self.getBranchNo() if no_branch is not None and no_branch.mayContinue(): return True return False Nuitka-0.5.28.2/nuitka/nodes/CodeObjectSpecs.py0000644000372000001440000001243513134660221021442 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Code object specifications. For code objects that will be attached to module, function, and generator objects, as well as tracebacks. They might be shared. """ from nuitka.utils.InstanceCounters import counted_del, counted_init class CodeObjectSpec(object): # One attribute for each code object aspect, and even flags, # pylint: disable=too-many-arguments,too-many-instance-attributes __slots__ = ( "co_name", "co_kind", "co_varnames", "co_argcount", "co_kwonlyargcount", "co_has_starlist", "co_has_stardict", "filename", "line_number", "future_spec", "new_locals", "has_closure", "is_optimized" ) @counted_init def __init__(self, co_name, co_kind, co_varnames, co_argcount, co_kwonlyargcount, co_has_starlist, co_has_stardict, co_filename, co_lineno, future_spec, co_new_locals = None, co_has_closure = None, co_is_optimized = None): self.co_name = co_name self.co_kind = co_kind self.future_spec = future_spec assert future_spec # Strings happens from XML parsing, make sure to convert them. if type(co_varnames) is str: if co_varnames == "": co_varnames = () else: co_varnames = co_varnames.split(',') if type(co_has_starlist) is not bool: co_has_starlist = co_has_starlist != "False" if type(co_has_stardict) is not bool: co_has_stardict = co_has_stardict != "False" self.co_varnames = tuple(co_varnames) self.co_argcount = int(co_argcount) self.co_kwonlyargcount = int(co_kwonlyargcount) self.co_has_starlist = co_has_starlist self.co_has_stardict = co_has_stardict self.filename = co_filename self.line_number = int(co_lineno) if type(co_has_starlist) is not bool: co_new_locals = co_new_locals != "False" if type(co_has_starlist) is not bool: co_has_closure = co_has_closure != "False" if type(co_has_starlist) is not bool: co_is_optimized = co_is_optimized != "False" self.new_locals = co_new_locals self.has_closure = co_has_closure self.is_optimized = co_is_optimized __del__ = counted_del() def __repr__(self): return """\ """ % self.getDetails() def getDetails(self): return { "co_name" : self.co_name, "co_kind" : self.co_kind, "co_varnames" : ','.join(self.co_varnames), "co_argcount" : self.co_argcount, "co_kwonlyargcount" : self.co_kwonlyargcount, "co_has_starlist" : self.co_has_starlist, "co_has_stardict" : self.co_has_stardict, "co_filename" : self.filename, "co_lineno" : self.line_number, "co_new_locals" : self.new_locals, "co_has_closure" : self.has_closure, "co_is_optimized" : self.is_optimized, "code_flags" : ','.join(self.future_spec.asFlags()) } def getCodeObjectKind(self): return self.co_kind def updateLocalNames(self, local_names): """ Add detected local variables after closure has been decided. """ self.co_varnames += tuple( local_name for local_name in local_names if local_name not in self.co_varnames ) def setFlagIsOptimizedValue(self, value): self.is_optimized = value def getFlagIsOptimizedValue(self): return self.is_optimized def setFlagNewLocalsValue(self, value): self.new_locals = value def getFlagNewLocalsValue(self): return self.new_locals def setFlagHasClosureValue(self, value): self.has_closure = value def getFlagHasClosureValue(self): return self.has_closure def getFutureSpec(self): return self.future_spec def getVarNames(self): return self.co_varnames def getArgumentCount(self): return self.co_argcount def getKwOnlyParameterCount(self): return self.co_kwonlyargcount def getCodeObjectName(self): return self.co_name def hasStarListArg(self): return self.co_has_starlist def hasStarDictArg(self): return self.co_has_stardict def getFilename(self): return self.filename def getLineNumber(self): return self.line_number Nuitka-0.5.28.2/nuitka/nodes/YieldNodes.py0000644000372000001440000000773013122472300020500 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Yield node. The yield node returns to the caller of the generator and therefore may execute absolutely arbitrary code, from the point of view of this code. It then returns something, which may often be 'None', but doesn't have to be. Often it will be used as a statement, which should also be reflected in a dedicated node. """ from .ExpressionBases import ExpressionChildrenHavingBase class ExpressionYield(ExpressionChildrenHavingBase): """ Yielding an expression. Typical code: yield expression Can only happen in a generator. Kind of explicitly suspends and resumes the execution. The user may inject any kind of exception or give any return value. The value of "None" is the most common though, esp. if it's not used. """ kind = "EXPRESSION_YIELD" named_children = ("expression",) def __init__(self, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.exception_preserving = False def markAsExceptionPreserving(self): self.exception_preserving = True def isExceptionPreserving(self): return self.exception_preserving getExpression = ExpressionChildrenHavingBase.childGetter("expression") def computeExpression(self, trace_collection): trace_collection.removeKnowledge(self.getExpression()) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) trace_collection.onExceptionRaiseExit(BaseException) # Nothing possible really here. return self, None, None class ExpressionYieldFrom(ExpressionChildrenHavingBase): """ Yielding from an expression. Typical code: yield from expression (Python3) Can only happen in a generator and only in Python3. Similar to yield, but implies a loop and exception propagation to the yield from generator if such. Kind of explicitly suspends and resumes the execution. The user may inject any kind of exception or give any return value. Having a return value is what makes Python3 generators special, and with yield from, that value is the expression result. """ kind = "EXPRESSION_YIELD_FROM" named_children = ("expression",) def __init__(self, expression, source_ref): ExpressionChildrenHavingBase.__init__( self, values = { "expression" : expression }, source_ref = source_ref ) self.exception_preserving = False def markAsExceptionPreserving(self): self.exception_preserving = True def isExceptionPreserving(self): return self.exception_preserving getExpression = ExpressionChildrenHavingBase.childGetter("expression") def computeExpression(self, trace_collection): trace_collection.removeKnowledge(self.getExpression()) # Any code could be run, note that. trace_collection.onControlFlowEscape(self) trace_collection.onExceptionRaiseExit(BaseException) # Nothing possible really here. return self, None, None Nuitka-0.5.28.2/nuitka/plugins/0000755000372000001440000000000013207540420016434 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/plugins/PluginBase.py0000644000372000001440000002630113207537706021056 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Plugins: Welcome to Nuitka! This is your shortest way to become part of it. This is to provide the base class for all plug-ins. Some of which are part of proper Nuitka, and some of which are waiting to be created and submitted for inclusion by you. The base class will serve as documentation. And it will point to examples of it being used. """ import os # This is heavily WIP. import sys from logging import info, warning from nuitka import Options from nuitka.ModuleRegistry import addUsedModule from nuitka.SourceCodeReferences import fromFilename from nuitka.utils.FileOperations import relpath pre_modules = {} post_modules = {} warned_unused_plugins = set() class NuitkaPluginBase(object): """ Nuitka base class for all plug-ins. Derive from "UserPlugin" please. The idea is it will allow to make differences for warnings, and traces of what is being done. For instance, the code that makes sure PyQt finds all his stuff, may want to do reports, but likely, you do not case about that enough to be visible by default. """ # You must provide this as a string which then can be used to enable the # plug-in. plugin_name = None def getPluginOptionBool(self, option_name, default_value): plugin_options = Options.getPluginOptions(self.plugin_name) return plugin_options.get(option_name, default_value) def considerImplicitImports(self, module, signal_change): """ Consider module imports. You will most likely want to look at "module.getFullName()" to get the fully qualified module or package name. You do not want to overload this method, but rather the things it calls, as the "signal_change" part of this API is not to be cared about. Most prominently "getImplicitImports()". """ for full_name in self.getImplicitImports(module): module_name = full_name.split('.')[-1] module_package = '.'.join(full_name.split('.')[:-1]) or None required = self.isRequiredImplicitImport(module, full_name) module_filename = self.locateModule( importing = module, module_name = module_name, module_package = module_package, warn = required ) if module_filename is None: if required: sys.exit( "Error, implicit module '%s' expected by '%s' not found." % ( full_name, module.getFullName() ) ) else: continue elif os.path.isdir(module_filename): module_kind = "py" elif module_filename.endswith(".py"): module_kind = "py" elif module_filename.endswith(".so") or \ module_filename.endswith(".pyd"): module_kind = "shlib" else: assert False, module_filename # TODO: This should get back to plug-ins, they should be allowed to # preempt or override the decision. decision, reason = self.decideRecursion( module_filename = module_filename, module_name = module_name, module_package = module_package, module_kind = module_kind ) if decision: self.recurseTo( module_package = module_package, module_filename = module_filename, module_kind = module_kind, reason = reason, signal_change = signal_change ) def isRequiredImplicitImport(self, module, full_name): """ By default, if given as an implicit import, require it. """ # Virtual method, pylint: disable=no-self-use,unused-argument return True def getImplicitImports(self, module): # Virtual method, pylint: disable=no-self-use,unused-argument return () # Provide fall-back for failed imports here. module_aliases = {} def considerFailedImportReferrals(self, module_name): return self.module_aliases.get(module_name, None) def onModuleSourceCode(self, module_name, source_code): # Virtual method, pylint: disable=no-self-use,unused-argument return source_code def onFrozenModuleSourceCode(self, module_name, is_package, source_code): # Virtual method, pylint: disable=no-self-use,unused-argument return source_code def onFrozenModuleBytecode(self, module_name, is_package, bytecode): # Virtual method, pylint: disable=no-self-use,unused-argument return bytecode @staticmethod def _createTriggerLoadedModule(module, trigger_name, code): from nuitka.tree.Building import createModuleTree from nuitka.nodes.ModuleNodes import CompiledPythonModule from .Plugins import Plugins module_name = module.getName() + trigger_name source_ref = fromFilename(module.getCompileTimeFilename() + trigger_name) mode = Plugins.decideCompilation(module_name, source_ref) trigger_module = CompiledPythonModule( name = module_name, package_name = module.getPackage(), mode = mode, future_spec = None, source_ref = source_ref ) createModuleTree( module = trigger_module, source_ref = module.getSourceReference(), source_code = code, is_main = False ) return trigger_module @staticmethod def createPreModuleLoadCode(module): # Virtual method, pylint: disable=unused-argument return None, None @staticmethod def createPostModuleLoadCode(module): # Virtual method, pylint: disable=unused-argument return None, None def onModuleDiscovered(self, module): pre_code, reason = self.createPreModuleLoadCode(module) full_name = module.getFullName() if pre_code: if full_name is pre_modules: sys.exit("Error, conflicting plug-ins for %s" % full_name) info( "Injecting plug-in based pre load code for module '%s':" % \ full_name ) for line in reason.split('\n'): info(" " + line) pre_modules[full_name] = self._createTriggerLoadedModule( module = module, trigger_name = "-preLoad", code = pre_code ) post_code, reason = self.createPostModuleLoadCode(module) if post_code: if full_name is post_modules: sys.exit("Error, conflicting plug-ins for %s" % full_name) info( "Injecting plug-in based post load code for module '%s':" % \ full_name ) for line in reason.split('\n'): info(" " + line) post_modules[full_name] = self._createTriggerLoadedModule( module = module, trigger_name = "-postLoad", code = post_code ) def onModuleEncounter(self, module_filename, module_name, module_package, module_kind): pass @staticmethod def locateModule(importing, module_name, module_package, warn): from nuitka.importing import Importing _module_package, module_filename, _finding = Importing.findModule( importing = importing, module_name = module_name, parent_package = module_package, level = -1, warn = warn ) return module_filename @staticmethod def decideRecursion(module_filename, module_name, module_package, module_kind): from nuitka.importing import Recursion decision, reason = Recursion.decideRecursion( module_filename = module_filename, module_name = module_name, module_package = module_package, module_kind = module_kind ) return decision, reason @staticmethod def recurseTo(module_package, module_filename, module_kind, reason, signal_change): from nuitka.importing import Recursion imported_module, added_flag = Recursion.recurseTo( module_package = module_package, module_filename = module_filename, module_relpath = relpath(module_filename), module_kind = module_kind, reason = reason ) addUsedModule(imported_module) if added_flag: signal_change( "new_code", imported_module.getSourceReference(), "Recursed to module." ) def considerExtraDlls(self, dist_dir, module): # Virtual method, pylint: disable=no-self-use,unused-argument return () def removeDllDependencies(self, dll_filename, dll_filenames): # Virtual method, pylint: disable=no-self-use,unused-argument return () def considerDataFiles(self, module): # Virtual method, pylint: disable=no-self-use,unused-argument return () def suppressBuiltinImportWarning(self, module_name, source_ref): # Virtual method, pylint: disable=no-self-use,unused-argument return False def suppressUnknownImportWarning(self, importing, module_name, source_ref): # Virtual method, pylint: disable=no-self-use,unused-argument return False def decideCompilation(self, module_name, source_ref): # Virtual method, pylint: disable=no-self-use,unused-argument return None def warnUnusedPlugin(self, message): if self.plugin_name not in warned_unused_plugins: warned_unused_plugins.add(self.plugin_name) warning( "Use '--plugin-enable=%s' for: %s" % ( self.plugin_name, message ) ) class UserPluginBase(NuitkaPluginBase): """ For user plug-ins. Check the base class methods for what you can do. """ # You must provide this as a string which then can be used to enable the # plug-in. plugin_name = None Nuitka-0.5.28.2/nuitka/plugins/standard/0000755000372000001440000000000013207540420020234 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/plugins/standard/PmwPlugin.py0000644000372000001440000001413713207537242022546 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Plugin to pre-process PMW for inclusion. """ import os import re import sys from nuitka import Options from nuitka.__past__ import StringIO from nuitka.plugins.PluginBase import NuitkaPluginBase from nuitka.utils.FileOperations import listDir # The main logic of this is from StackOverflow answer: # http://stackoverflow.com/questions/6772916/python-pmw-and-cx-freeze # The order of these files is significant. Files which reference # other files must appear later. Files may be deleted if they are not # used. files = [ "Dialog", "TimeFuncs", "Balloon", "ButtonBox", "EntryField", "Group", "LabeledWidget", "MainMenuBar", "MenuBar", "MessageBar", "MessageDialog", "NoteBook", "OptionMenu", "PanedWidget", "PromptDialog", "RadioSelect", "ScrolledCanvas", "ScrolledField", "ScrolledFrame", "ScrolledListBox", "ScrolledText", "HistoryText", "SelectionDialog", "TextDialog", "TimeCounter", "AboutDialog", "ComboBox", "ComboBoxDialog", "Counter", "CounterDialog", ] # Work out which version is being bundled. class NuitkaPluginPmw(NuitkaPluginBase): plugin_name = "pmw-freezer" def onModuleSourceCode(self, module_name, source_code): if module_name == "Pmw": pmw_path = self.locateModule( importing = None, module_name = "Pmw", module_package = None, warn = True ) return self._packagePmw(pmw_path) return source_code def _packagePmw(self, pmw_path): # From the "__init__.py" of Pwm: def _hasLoader(dirname): # @ReservedAssignment # Only accept Pmw_V_R_P with single digits, since ordering will # not work correctly with multiple digits (for example, Pmw_10_0 # will be before Pmw_9_9). if re.search("^Pmw_[0-9]_[0-9](_[0-9])?$", dirname) is not None: for suffix in (".py", ".pyc", ".pyo"): path = os.path.join(pmw_path, dirname, "lib", "PmwLoader" + suffix) if os.path.isfile(path): return 1 return 0 # This mimicks the scan the __init__.py does. candidates = [] for _fullpath, candidate in listDir(pmw_path): if _hasLoader(candidate): candidates.append(candidate) candidates.sort() candidates.reverse() if not candidates: sys.exit("Error, cannot find any Pmw versions.") candidate = os.path.join(pmw_path, candidates[0], "lib") version = candidates[0][4:].replace('_', '.') return self._packagePmw2(candidate, version) def _packagePmw2(self, srcdir, version): def mungeFile(filename): # Read the filename and modify it so that it can be bundled with the # other Pmw files. filename = "Pmw" + filename + ".py" text = open(os.path.join(srcdir, filename)).read() text = re.sub(r"import Pmw\>", "", text) text = re.sub("INITOPT = Pmw.INITOPT", "", text) text = re.sub(r"\ 1 else None if child in ("QtGui", "QtAssistant", "QtDBus", "QtDeclarative", "QtSql", "QtDesigner", "QtHelp", "QtNetwork", "QtScript", "QtQml", "QtScriptTools", "QtSvg", "QtTest", "QtWebKit", "QtOpenGL", "QtXml", "QtXmlPatterns", "QtPrintSupport", "QtNfc", "QtWebKitWidgets", "QtBluetooth", "QtMultimediaWidgets", "QtQuick", "QtWebChannel", "QtWebSockets", "QtX11Extras", "_QOpenGLFunctions_2_0", "_QOpenGLFunctions_2_1", "_QOpenGLFunctions_4_1_Core"): yield elements[0] + ".QtCore" if child in ("QtDeclarative", "QtWebKit", "QtXmlPatterns", "QtQml", "QtPrintSupport", "QtWebKitWidgets", "QtMultimedia", "QtMultimediaWidgets", "QtQuick", "QtQuickWidgets", "QtWebSockets"): yield elements[0] + ".QtNetwork" if child == "QtScriptTools": yield elements[0] + ".QtScript" if child in ("QtWidgets", "QtDeclarative", "QtDesigner", "QtHelp", "QtScriptTools", "QtSvg", "QtTest", "QtWebKit", "QtPrintSupport", "QtWebKitWidgets", "QtMultimedia", "QtMultimediaWidgets", "QtOpenGL", "QtQuick", "QtQuickWidgets", "QtSql", "_QOpenGLFunctions_2_0", "_QOpenGLFunctions_2_1", "_QOpenGLFunctions_4_1_Core"): yield elements[0] + ".QtGui" if full_name in ("PyQt5.QtDesigner", "PyQt5.QtHelp", "PyQt5.QtTest", "PyQt5.QtPrintSupport", "PyQt5.QtSvg", "PyQt5.QtOpenGL", "PyQt5.QtWebKitWidgets", "PyQt5.QtMultimediaWidgets", "PyQt5.QtQuickWidgets", "PyQt5.QtSql"): yield "PyQt5.QtWidgets" if full_name in ("PyQt5.QtPrintSupport",): yield "PyQt5.QtSvg" if full_name in ("PyQt5.QtWebKitWidgets",): yield "PyQt5.QtWebKit" yield "PyQt5.QtPrintSupport" if full_name in ("PyQt5.QtMultimediaWidgets",): yield "PyQt5.QtMultimedia" if full_name in ("PyQt5.QtQuick", "PyQt5.QtQuickWidgets"): yield "PyQt5.QtQml" if full_name == "PyQt5.QtQuickWidgets": yield "PyQt5.QtQuick" elif full_name == "PySide.QtDeclarative": yield "PySide.QtGui" elif full_name == "PySide.QtHelp": yield "PySide.QtGui" elif full_name == "PySide.QtOpenGL": yield "PySide.QtGui" elif full_name == "PySide.QtScriptTools": yield "PySide.QtScript" yield "PySide.QtGui" elif full_name == "PySide.QtSql": yield "PySide.QtGui" elif full_name == "PySide.QtSvg": yield "PySide.QtGui" elif full_name == "PySide.QtTest": yield "PySide.QtGui" elif full_name == "PySide.QtUiTools": yield "PySide.QtGui" yield "PySide.QtXml" elif full_name == "PySide.QtWebKit": yield "PySide.QtGui" elif full_name == "PySide.phonon": yield "PySide.QtGui" elif full_name == "lxml.etree": yield "gzip" yield "lxml._elementpath" elif full_name == "gtk._gtk": yield "pangocairo" yield "pango" yield "cairo" yield "gio" yield "atk" elif full_name == "atk": yield "gobject" elif full_name == "gtkunixprint": yield "gobject" yield "cairo" yield "gtk" elif full_name == "pango": yield "gobject" elif full_name == "pangocairo": yield "pango" yield "cairo" elif full_name == "reportlab.rl_config": yield "reportlab.rl_settings" elif full_name == "socket": yield "_socket" elif full_name == "ctypes": yield "_ctypes" elif full_name == "gi._gi": yield "gi._error" elif full_name == "gi._gi_cairo": yield "cairo" elif full_name == "cairo._cairo": yield "gi._gobject" elif full_name in ("Tkinter", "tkinter"): yield "_tkinter" elif full_name in ("cryptography.hazmat.bindings._openssl", "cryptography.hazmat.bindings._constant_time", "cryptography.hazmat.bindings._padding"): yield "_cffi_backend" elif full_name.startswith("cryptography._Cryptography_cffi_"): yield "_cffi_backend" elif full_name == "bcrypt._bcrypt": yield "_cffi_backend" elif full_name == "nacl._sodium": yield "_cffi_backend" elif full_name == "_dbus_glib_bindings": yield "_dbus_bindings" elif full_name == "_mysql": yield "_mysql_exceptions" elif full_name == "lxml.objectify": yield "lxml.etree" elif full_name == "_yaml": yield "yaml" elif full_name == "apt_inst": yield "apt_pkg" elif full_name == "PIL._imagingtk": yield "PIL._tkinter_finder" module_aliases = { # Python2 "six.moves.builtins" : "__builtin__" if python_version < 300 else "builtins", "six.moves.configparser" : "ConfigParser" if python_version < 300 else "configparser", "six.moves.copyreg" : "copy_reg" if python_version < 300 else "copyreg", "six.moves.dbm_gnu" : "gdbm" if python_version < 300 else "dbm.gnu", "six.moves._dummy_thread" : "dummy_thread" if python_version < 300 else "_dummy_thread", "six.moves.http_cookiejar" : "cookielib" if python_version < 300 else "http.cookiejar", "six.moves.http_cookies" : "Cookie" if python_version < 300 else "http.cookies", "six.moves.html_entities" : "htmlentitydefs" if python_version < 300 else "html.entities", "six.moves.html_parser" : "HTMLParser" if python_version < 300 else "html.parser", "six.moves.http_client" : "httplib" if python_version < 300 else "http.client", "six.moves.email_mime_multipart" : "email.MIMEMultipart" if python_version < 300 else "email.mime.multipart", "six.moves.email_mime_nonmultipart" : "email.MIMENonMultipart" if python_version < 300 else "email.mime.nonmultipart", "six.moves.email_mime_text" : "email.MIMEText" if python_version < 300 else "email.mime.text", "six.moves.email_mime_base" : "email.MIMEBase" if python_version < 300 else "email.mime.base", "six.moves.BaseHTTPServer" : "BaseHTTPServer" if python_version < 300 else "http.server", "six.moves.CGIHTTPServer" : "CGIHTTPServer" if python_version < 300 else "http.server", "six.moves.SimpleHTTPServer" : "SimpleHTTPServer" if python_version < 300 else "http.server", "six.moves.cPickle" : "cPickle" if python_version < 300 else "pickle", "six.moves.queue" : "Queue" if python_version < 300 else "queue", "six.moves.reprlib" :"repr" if python_version < 300 else "reprlib", "six.moves.socketserver" : "SocketServer" if python_version < 300 else "socketserver", "six.moves._thread" : "thread" if python_version < 300 else "_thread", "six.moves.tkinter" :"Tkinter" if python_version < 300 else "tkinter", "six.moves.tkinter_dialog" : "Dialog" if python_version < 300 else "tkinter.dialog", "six.moves.tkinter_filedialog" : "FileDialog" if python_version < 300 else "tkinter.filedialog", "six.moves.tkinter_scrolledtext" : "ScrolledText" if python_version < 300 else "tkinter.scrolledtext", "six.moves.tkinter_simpledialog" : "SimpleDialog" if python_version < 300 else "tkinter.simpledialog", "six.moves.tkinter_tix" : "Tix" if python_version < 300 else "tkinter.tix", "six.moves.tkinter_ttk" :"ttk" if python_version < 300 else "tkinter.ttk", "six.moves.tkinter_constants" :"Tkconstants" if python_version < 300 else "tkinter.constants", "six.moves.tkinter_dnd" : "Tkdnd" if python_version < 300 else "tkinter.dnd", "six.moves.tkinter_colorchooser" : "tkColorChooser" if python_version < 300 else "tkinter_colorchooser", "six.moves.tkinter_commondialog" : "tkCommonDialog" if python_version < 300 else "tkinter_commondialog", "six.moves.tkinter_tkfiledialog" : "tkFileDialog" if python_version < 300 else "tkinter.filedialog", "six.moves.tkinter_font" : "tkFont" if python_version < 300 else "tkinter.font", "six.moves.tkinter_messagebox" : "tkMessageBox" if python_version < 300 else "tkinter.messagebox", "six.moves.tkinter_tksimpledialog" :"tkSimpleDialog" if python_version < 300 else "tkinter_tksimpledialog", "six.moves.urllib_parse" : None if python_version < 300 else "urllib.parse", "six.moves.urllib_error" : None if python_version < 300 else "urllib.error", "six.moves.urllib_robotparser" :"robotparser" if python_version < 300 else "urllib.robotparser", "six.moves.xmlrpc_client" :"xmlrpclib" if python_version < 300 else "xmlrpc.client", "six.moves.xmlrpc_server" :"SimpleXMLRPCServer" if python_version < 300 else "xmlrpc.server", "six.moves.winreg" : "_winreg" if python_version < 300 else "winreg", "requests.packages.urllib3" : "urllib3", "requests.packages.urllib3.contrib" : "urllib3.contrib", "requests.packages.urllib3.contrib.pyopenssl" : "urllib3.contrib.pyopenssl", "requests.packages.urllib3.contrib.ntlmpool" : "urllib3.contrib.ntlmpool", "requests.packages.urllib3.contrib.socks" : "urllib3.contrib.socks", "requests.packages.urllib3.exceptions" : "urllib3.exceptions", 'requests.packages.urllib3._collections' : 'urllib3._collections', "requests.packages.chardet" : "chardet", "requests.packages.idna" : "idna", 'requests.packages.urllib3.packages' : 'urllib3.packages', 'requests.packages.urllib3.packages.ordered_dict' : 'urllib3.packages.ordered_dict', 'requests.packages.urllib3.packages.ssl_match_hostname' : 'urllib3.packages.ssl_match_hostname', 'requests.packages.urllib3.packages.ssl_match_hostname._implementation' : 'urllib3.packages.ssl_match_hostname._implementation', 'requests.packages.urllib3.connectionpool' : 'urllib3.connectionpool', 'requests.packages.urllib3.connection' : 'urllib3.connection', 'requests.packages.urllib3.filepost' : 'urllib3.filepost', 'requests.packages.urllib3.request' : 'urllib3.request', 'requests.packages.urllib3.response' : 'urllib3.response', 'requests.packages.urllib3.fields' : 'urllib3.fields', 'requests.packages.urllib3.poolmanager' : 'urllib3.poolmanager', 'requests.packages.urllib3.util' : 'urllib3.util', 'requests.packages.urllib3.util.connection' : 'urllib3.util.connection', 'requests.packages.urllib3.util.request' : 'urllib3.util.request', 'requests.packages.urllib3.util.response' : 'urllib3.util.response', 'requests.packages.urllib3.util.retry' : 'urllib3.util.retry', 'requests.packages.urllib3.util.ssl_' : 'urllib3.util.ssl_', 'requests.packages.urllib3.util.timeout' : 'urllib3.util.timeout', 'requests.packages.urllib3.util.url' : 'urllib3.util.url', } def onModuleSourceCode(self, module_name, source_code): if module_name == "numexpr.cpuinfo": # We cannot intercept "is" tests, but need it to be "isinstance", # so we patch it on the file. TODO: This is only temporary, in # the future, we may use optimization that understands the right # hand size of the "is" argument well enough to allow for our # type too. return source_code.replace( "type(attr) is types.MethodType", "isinstance(attr, types.MethodType)" ) # Do nothing by default. return source_code def suppressBuiltinImportWarning(self, module, source_ref): if module.getFullName() in ("setuptools", "six"): return True return False def considerExtraDlls(self, dist_dir, module): full_name = module.getFullName() if getOS() == "Linux" and full_name == "uuid": uuid_dll_path = locateDLL("uuid") dist_dll_path = os.path.join(dist_dir, os.path.basename(uuid_dll_path)) shutil.copy(uuid_dll_path, dist_dir) return ( (uuid_dll_path, dist_dll_path, None), ) return () unworthy_namespaces = ( "setuptools", # Not performance relevant. "distutils", # Not performance relevant. "pkg_resources", # Not performance relevant. "numpy.distutils", # Largely unused, and a lot of modules. "numpy.f2py", # Mostly unused, only numpy.distutils import it. "numpy.testing", # Useless. "nose", # Not performance relevant. "coverage", # Not performance relevant. "docutils", # Not performance relevant. "pexpect", # Not performance relevant. "Cython", # Mostly unused, and a lot of modules. "cython", "pyximport", "IPython", # Mostly unused, and a lot of modules. "wx._core", # Too large generated code ) def decideCompilation(self, module_name, source_ref): for unworthy_namespace in self.unworthy_namespaces: if module_name == unworthy_namespace or \ module_name.startswith(unworthy_namespace + '.'): return "bytecode" Nuitka-0.5.28.2/nuitka/plugins/__init__.py0000644000372000001440000000150113112214770020543 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/plugins/Plugins.py0000644000372000001440000002055013207540035020433 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Plugins: Welcome to Nuitka! This is your shortest way to become part of it. This is to provide the base class for all plug-ins. Some of which are part of proper Nuitka, and some of which are waiting to be created and submitted for inclusion by you. The base class in PluginBase will serve as documentation of available. """ from __future__ import print_function import os import sys from nuitka import Options from nuitka.ModuleRegistry import addUsedModule from .PluginBase import post_modules, pre_modules from .standard.DataFileCollectorPlugin import ( NuitkaPluginDataFileCollector, NuitkaPluginDetectorDataFileCollector ) from .standard.ImplicitImports import NuitkaPluginPopularImplicitImports from .standard.PmwPlugin import NuitkaPluginDetectorPmw, NuitkaPluginPmw from .standard.ConsiderPyLintAnnotationsPlugin import ( # isort:skip NuitkaPluginDetectorPylintEclipseAnnotations, NuitkaPluginPylintEclipseAnnotations ) from .standard.MultiprocessingPlugin import ( # isort:skip NuitkaPluginDetectorMultiprocessingWorkaorunds, NuitkaPluginMultiprocessingWorkaorunds ) from .standard.PySidePyQtPlugin import ( # isort:skip NuitkaPluginDetectorPyQtPySidePlugins, NuitkaPluginPyQtPySidePlugins ) # The standard plug-ins have their list hard-coded here. User plug-ins will # be scanned later, TODO. active_plugin_list = [ NuitkaPluginPopularImplicitImports(), ] # List of optional plug-in classes. Until we have the meta class to do it, just # add your class here. The second one is a detector, which is supposed to give # a missing plug-in message, should it find the condition to make it useful. optional_plugin_classes = ( (NuitkaPluginMultiprocessingWorkaorunds, NuitkaPluginDetectorMultiprocessingWorkaorunds), (NuitkaPluginPyQtPySidePlugins, NuitkaPluginDetectorPyQtPySidePlugins), (NuitkaPluginPylintEclipseAnnotations, NuitkaPluginDetectorPylintEclipseAnnotations), (NuitkaPluginDataFileCollector, NuitkaPluginDetectorDataFileCollector), (NuitkaPluginPmw, NuitkaPluginDetectorPmw), ) plugin_name2plugin_classes = dict( (plugin[0].plugin_name, plugin) for plugin in optional_plugin_classes ) class Plugins(object): @staticmethod def considerImplicitImports(module, signal_change): for plugin in active_plugin_list: plugin.considerImplicitImports(module, signal_change) # Post load code may have been created, if so indicate it's used. full_name = module.getFullName() if full_name in post_modules: addUsedModule(post_modules[full_name]) if full_name in pre_modules: addUsedModule(pre_modules[full_name]) @staticmethod def considerExtraDlls(dist_dir, module): result = [] for plugin in active_plugin_list: for extra_dll in plugin.considerExtraDlls(dist_dir, module): assert os.path.isfile(extra_dll[1]), extra_dll[1] result.append(extra_dll) return result @staticmethod def removeDllDependencies(dll_filename, dll_filenames): dll_filenames = tuple(sorted(dll_filenames)) result = [] for plugin in active_plugin_list: for removed_dll in plugin.removeDllDependencies(dll_filename, dll_filenames): result.append(removed_dll) return result @staticmethod def considerDataFiles(module): for plugin in active_plugin_list: for value in plugin.considerDataFiles(module): yield value @staticmethod def onModuleDiscovered(module): for plugin in active_plugin_list: plugin.onModuleDiscovered(module) @staticmethod def onModuleSourceCode(module_name, source_code): assert type(module_name) is str assert type(source_code) is str for plugin in active_plugin_list: source_code = plugin.onModuleSourceCode(module_name, source_code) assert type(source_code) is str return source_code @staticmethod def onFrozenModuleSourceCode(module_name, is_package, source_code): assert type(module_name) is str assert type(source_code) is str for plugin in active_plugin_list: source_code = plugin.onFrozenModuleSourceCode(module_name, is_package, source_code) assert type(source_code) is str return source_code @staticmethod def onFrozenModuleBytecode(module_name, is_package, bytecode): assert type(module_name) is str assert bytecode.__class__.__name__ == "code" for plugin in active_plugin_list: bytecode = plugin.onFrozenModuleBytecode(module_name, is_package, bytecode) assert bytecode.__class__.__name__ == "code" return bytecode @staticmethod def onModuleEncounter(module_filename, module_name, module_package, module_kind): for plugin in active_plugin_list: plugin.onModuleEncounter( module_filename, module_name, module_package, module_kind ) @staticmethod def considerFailedImportReferrals(module_name): for plugin in active_plugin_list: new_module_name = plugin.considerFailedImportReferrals(module_name) if new_module_name is not None: return new_module_name return None @staticmethod def suppressBuiltinImportWarning(module_name, source_ref): for plugin in active_plugin_list: if plugin.suppressBuiltinImportWarning(module_name, source_ref): return True return False @staticmethod def suppressUnknownImportWarning(importing, module_name): if importing.isCompiledPythonModule() or importing.isPythonShlibModule(): importing_module = importing else: importing_module = importing.getParentModule() source_ref = importing.getSourceReference() for plugin in active_plugin_list: if plugin.suppressUnknownImportWarning(importing_module, module_name, source_ref): return True return False @staticmethod def decideCompilation(module_name, source_ref): for plugin in active_plugin_list: value = plugin.decideCompilation(module_name, source_ref) if value is not None: assert value in ("compiled", "bytecode") return value return "compiled" def listPlugins(): for plugin_name in sorted(plugin_name2plugin_classes): print(plugin_name) sys.exit(0) def initPlugins(): for plugin_name in Options.getPluginsEnabled() + Options.getPluginsDisabled(): if plugin_name not in plugin_name2plugin_classes: sys.exit("Error, unknown plug-in '%s' referenced." % plugin_name) if plugin_name in Options.getPluginsEnabled() and \ plugin_name in Options.getPluginsDisabled(): sys.exit("Error, conflicting enable/disable of plug-in '%s'." % plugin_name) for plugin_name, (plugin_class, plugin_detector) in plugin_name2plugin_classes.items(): if plugin_name in Options.getPluginsEnabled(): active_plugin_list.append( plugin_class( **Options.getPluginOptions(plugin_name) ) ) elif plugin_name not in Options.getPluginsDisabled(): if plugin_detector is not None \ and Options.shallDetectMissingPlugins() and \ plugin_detector.isRelevant(): active_plugin_list.append( plugin_detector() ) initPlugins() Nuitka-0.5.28.2/nuitka/plugins/user/0000755000372000001440000000000013207540420017412 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/plugins/user/__init__.py0000644000372000001440000000150113112214770021521 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/distutils/0000755000372000001440000000000013207540420016777 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/distutils/__init__.py0000644000372000001440000000150113207537242021115 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/distutils/bdist_nuitka.py0000644000372000001440000001065013207537242022043 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Nuitka distutils integration. """ import os import shutil import subprocess import sys import wheel.bdist_wheel # @UnresolvedImport # Class name enforced by distutils, must match the command name. # pylint: disable=C0103 class bdist_nuitka(wheel.bdist_wheel.bdist_wheel): # Our wheel base class is very unpure, so are we # pylint: disable=attribute-defined-outside-init def run(self): self.compile_packages = self.distribution.packages self.main_package = self.compile_packages[0] # Exclude packages from the wheel, we deal with them by compiling # into single package. # self.distribution.packages = [] wheel.bdist_wheel.bdist_wheel.run(self) def _buildPackage(self, build_lib): # Nuitka wants the main package by filename, probably we should stop # needing that. from nuitka.importing.Importing import findModule, setMainScriptDirectory from nuitka.utils.Execution import getExecutablePath old_dir = os.getcwd() os.chdir(build_lib) # Search in the build directory preferrably. setMainScriptDirectory('.') package, main_filename, finding = findModule( importing = None, module_name = self.main_package, parent_package = None, level = 0, warn = False ) # Check expectations, e.g. do not compile built-in modules. assert finding == "absolute", finding assert package is None, package nuitka_binary = getExecutablePath("nuitka") if nuitka_binary is None: sys.exit("Error, cannot find nuitka binary in PATH.") command = [ sys.executable, nuitka_binary, "--module", "--plugin-enable=pylint-warnings", "--output-dir=%s" % build_lib, "--recurse-to={%s}" % ','.join(self.compile_packages), "--recurse-dir=%s" % self.main_package, "--recurse-not-to=*.tests", "--show-modules", "--remove-output", main_filename, ] subprocess.check_call( command ) os.chdir(old_dir) self.build_lib = build_lib def run_command(self, command): if command == "install": command_obj = self.distribution.get_command_obj(command) basedir_observed = "" if os.name == "nt": # win32 barfs if any of these are ''; could be '.'? # (distutils.command.install:change_roots bug) basedir_observed = os.path.normpath(os.path.join(self.data_dir, "..")) setattr(command_obj, "install_platlib", basedir_observed) setattr(command_obj, "install_purelib", None) shutil.rmtree(os.path.join(self.build_lib, self.main_package)) result = wheel.bdist_wheel.bdist_wheel.run_command(self, command) # After building, we are ready to build the extension module, from # what "build_py" command has done. if command == "build": command_obj = self.distribution.get_command_obj(command) command_obj.ensure_finalized() self._buildPackage(command_obj.build_lib) return result def write_wheelfile(self, wheelfile_base, generator = None): self.root_is_pure = False if generator is None: from nuitka.Version import getNuitkaVersion generator = "Nuitka (%s)" % getNuitkaVersion() wheel.bdist_wheel.bdist_wheel.write_wheelfile( self, wheelfile_base = wheelfile_base, generator = generator ) Nuitka-0.5.28.2/nuitka/TreeXML.py0000644000372000001440000000541313134660221016611 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ XML node tree handling Means to create XML elements from Nuitka tree nodes and to convert the XML tree to ASCII or output it. """ from nuitka.__past__ import StringIO from nuitka.PythonVersions import python_version from . import Tracing def indent(elem, level = 0, more_sibs = False): i = '\n' if level: i += (level-1) * " " num_kids = len(elem) if num_kids: if not elem.text or not elem.text.strip(): elem.text = i + " " if level: elem.text += " " count = 0 for kid in elem: indent(kid, level+1, count < num_kids - 1) count += 1 if not elem.tail or not elem.tail.strip(): elem.tail = i if more_sibs: elem.tail += " " else: if level and (not elem.tail or not elem.tail.strip()): elem.tail = i if more_sibs: elem.tail += " " return elem try: import lxml.etree # pylint: disable=I0021,import-error xml_module = lxml.etree Element = xml_module.Element xml_tostring = lambda tree: lxml.etree.tostring(tree, pretty_print = True) except ImportError: try: import xml.etree.ElementTree xml_module = xml.etree.ElementTree Element = xml.etree.ElementTree.Element xml_tostring = lambda tree: xml_module.tostring(indent(tree)) except ImportError: xml_module = None Element = None xml_tostring = None # TODO: Use the writer to create the XML we output. That should be more # scalable and/or faster. try: import lxml.xmlfile # pylint: disable=I0021,import-error xml_writer = lxml.xmlfile except ImportError: xml_writer = None def toString(tree): result = xml_tostring(tree) if python_version >= 300: result = result.decode("utf-8") return result def fromString(text): return xml_module.parse(StringIO(text)).getroot() def dump(tree): value = toString(tree).rstrip() Tracing.printLine(value) Nuitka-0.5.28.2/nuitka/gui/0000755000372000001440000000000013207540420015537 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/gui/dialogs/0000755000372000001440000000000013207540420017161 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/gui/dialogs/InspectPythonTree.ui0000644000372000037200000000212012711355343023117 0ustar hayenhayen00000000000000 Dialog 0 0 1030 728 Nuitka Node Tree Inspector Node Tree Source Code Nuitka-0.5.28.2/nuitka/gui/SyntaxHighlighting.py0000644000372000001440000001661713134660221021741 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Syntax highlighting for Python. Inspired/copied from by http://diotavelli.net/PyQtWiki/Python%20syntax%20highlighting """ from PyQt5.QtCore import \ QRegExp # @UnresolvedImport pylint: disable=I0021,import-error from PyQt5.QtGui import QColor # @UnresolvedImport pylint: disable=I0021,import-error from PyQt5.QtGui import QFont # @UnresolvedImport pylint: disable=I0021,import-error from PyQt5.QtGui import \ QSyntaxHighlighter # @UnresolvedImport pylint: disable=I0021,import-error from PyQt5.QtGui import \ QTextCharFormat # @UnresolvedImport pylint: disable=I0021,import-error def createTextFormat(color, style = ""): """Return a QTextCharFormat with the given attributes. """ _color = QColor() _color.setNamedColor(color) _format = QTextCharFormat() _format.setForeground(_color) if "bold" in style: _format.setFontWeight(QFont.Bold) if "italic" in style: _format.setFontItalic(True) return _format # Syntax styles that can be shared by all languages STYLES = { "keyword" : createTextFormat("blue"), "operator" : createTextFormat("red"), "brace" : createTextFormat("darkGray"), "defclass" : createTextFormat("black", "bold"), "string" : createTextFormat("magenta"), "string2" : createTextFormat("darkMagenta"), "comment" : createTextFormat("darkGreen", "italic"), "self" : createTextFormat("black", "italic"), "numbers" : createTextFormat("brown"), } class PythonHighlighter(QSyntaxHighlighter): """ Syntax highlighter for the Python language. """ # Python keywords keywords = [ "and", "assert", "break", "class", "continue", "def", "del", "elif", "else", "except", "exec", "finally", "for", "from", "global", "if", "import", "in", "is", "lambda", "not", "or", "pass", "print", "raise", "return", "try", "while", "with", "yield", "None", "True", "False", ] # Python operators operators = [ '=', # Comparison "==", "!=", '<', "<=", '>', ">=", # Arithmetic "\+", '-', "\*", '/', "//", "\%", "\*\*", # In-place "\+=", "-=", "\*=", "/=", "\%=", # Bitwise "\^", "\|", "\&", "\~", ">>", "<<", ] # Python braces braces = [ "\{", "\}", "\(", "\)", "\[", "\]", ] def __init__(self, document): QSyntaxHighlighter.__init__(self, document) # Multi-line strings (expression, flag, style) # The triple-quotes in these two lines will mess up the # syntax highlighting from this point onward self.tri_single = (QRegExp("'''"), 1, STYLES["string2"]) self.tri_double = (QRegExp('"""'), 2, STYLES["string2"]) rules = [] # Keyword, operator, and brace rules rules += [(r'\b%s\b' % w, 0, STYLES["keyword"]) for w in PythonHighlighter.keywords] rules += [(r'%s' % o, 0, STYLES["operator"]) for o in PythonHighlighter.operators] rules += [(r'%s' % b, 0, STYLES["brace"]) for b in PythonHighlighter.braces] # All other rules rules += [ # 'self' (r'\bself\b', 0, STYLES["self"]), # Double-quoted string, possibly containing escape sequences (r'"[^"\\]*(\\.[^"\\]*)*"', 0, STYLES["string"]), # Single-quoted string, possibly containing escape sequences (r"'[^'\\]*(\\.[^'\\]*)*'", 0, STYLES["string"]), # 'def' followed by an identifier (r'\bdef\b\s*(\w+)', 1, STYLES["defclass"]), # 'class' followed by an identifier (r'\bclass\b\s*(\w+)', 1, STYLES["defclass"]), # From '#' until a newline (r'#[^\n]*', 0, STYLES["comment"]), # Numeric literals (r'\b[+-]?[0-9]+[lL]?\b', 0, STYLES["numbers"]), (r'\b[+-]?0[xX][0-9A-Fa-f]+[lL]?\b', 0, STYLES["numbers"]), (r'\b[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?\b', 0, STYLES["numbers"]), ] # Build a QRegExp for each pattern self.rules = [(QRegExp(pat), index, fmt) for (pat, index, fmt) in rules] def highlightBlock(self, text): """Apply syntax highlighting to the given block of text. """ # Do other syntax formatting for expression, nth, display_format in self.rules: index = expression.indexIn(text, 0) while index >= 0: # We actually want the index of the nth match index = expression.pos(nth) length = expression.cap(nth).length() self.setFormat(index, length, display_format) index = expression.indexIn(text, index + length) self.setCurrentBlockState(0) # Do multi-line strings in_multiline = self.match_multiline(text, *self.tri_single) if not in_multiline: in_multiline = self.match_multiline(text, *self.tri_double) def match_multiline(self, text, delimiter, in_state, style): """Do highlighting of multi-line strings. ``delimiter`` should be a ``QRegExp`` for triple-single-quotes or triple-double-quotes, and ``in_state`` should be a unique integer to represent the corresponding state changes when inside those strings. Returns True if we're still inside a multi-line string when this function is finished. """ # If inside triple-single quotes, start at 0 if self.previousBlockState() == in_state: start = 0 add = 0 # Otherwise, look for the delimiter on this line else: start = delimiter.indexIn(text) # Move past this match add = delimiter.matchedLength() # As long as there's a delimiter match on this line... while start >= 0: # Look for the ending delimiter end = delimiter.indexIn(text, start + add) # Ending delimiter on this line? if end >= add: length = end - start + add + delimiter.matchedLength() self.setCurrentBlockState(0) # No; multi-line string else: self.setCurrentBlockState(in_state) length = text.length() - start + add # Apply formatting self.setFormat(start, length, style) # Look for the next match start = delimiter.indexIn(text, start + length) # Return True if still inside a multi-line string, False otherwise if self.currentBlockState() == in_state: return True else: return False def addPythonHighlighter(document): PythonHighlighter(document) Nuitka-0.5.28.2/nuitka/gui/__init__.py0000644000372000001440000000150113112214770017646 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/gui/TreeDisplay.py0000644000372000001440000002111313134660221020335 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module with functions to display a node tree. Useful to getting an idea of what the internal representation of Nuitka is about a source code. """ import os import sys from nuitka import SourceCodeReferences from PyQt5 import ( # isort:skip pylint: disable=I0021,import-error QtCore, # @UnresolvedImport QtGui, # @UnresolvedImport uic ) # The API requires a signature, sometimes we don't use it, pylint: disable=no-self-use # Also using private stuff from classes, probably ok, pylint: disable=protected-access class NodeTreeModelItem(object): def __init__(self, node, parent = None): self.parent_treeitem = parent self.node = node self.children = None def appendChild(self, _item): assert False def _children(self): if self.children is None: self.children = [ NodeTreeModelItem(child, self) for child in self.node.getVisitableNodes() ] if self.node.isCompiledPythonModule(): self.children.extend( NodeTreeModelItem(child, self) for child in self.node.getUsedFunctions() ) return self.children def child(self, row): return self._children()[ row ] def childCount(self): return len(self._children()) def columnCount(self): return 2 def data(self, column): if column == 0: result = self.node.getDescription() elif column == 1: result = self.node.getDetail() else: assert False return QtCore.QVariant(result) def parent(self): return self.parent_treeitem def row(self): return self.parent_treeitem._children().index(self) if self.parent else 0 class NodeTreeModel(QtCore.QAbstractItemModel): def __init__(self, root, parent = None): QtCore.QAbstractItemModel.__init__(self, parent) self.root_node = root self.root_item = NodeTreeModelItem(root) def columnCount(self, _parent): return self.root_item.columnCount() def data(self, index, role): if not index.isValid(): return QtCore.QVariant() if role != QtCore.Qt.DisplayRole: return QtCore.QVariant() item = index.internalPointer() return QtCore.QVariant(item.data(index.column())) def flags(self, index): if not index.isValid(): return QtCore.Qt.ItemIsEnabled return QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable def headerData(self, section, orientation, role): if orientation == QtCore.Qt.Horizontal and role == QtCore.Qt.DisplayRole: if section == 0: return QtCore.QVariant("Node Type") elif section == 1: return QtCore.QVariant("Node Detail") return self.root_item.data(section) return QtCore.QVariant() def index(self, row, column, parent): if row < 0 or column < 0 or row >= self.rowCount(parent) or column >= self.columnCount(parent): return QtCore.QModelIndex() if not parent.isValid(): parent = self.root_item else: parent = parent.internalPointer() child = parent.child(row) if child: return self.createIndex(row, column, child) else: return QtCore.QModelIndex() def parent(self, index): if not index.isValid(): return QtCore.QModelIndex() child = index.internalPointer() parent = child.parent() if parent == self.root_item: return QtCore.QModelIndex() return self.createIndex(parent.row(), 0, parent) def rowCount(self, parent): if parent.column() > 0: return 0 if not parent.isValid(): parent = self.root_item else: parent = parent.internalPointer() return parent.childCount() def getNodeFromPath(self, tree_path): tree_path = list(tree_path) current = self.root_node while tree_path: index = tree_path[0] del tree_path[0] if current.isCompiledPythonModule(): if index == 0: current = current.getBody() else: current = tuple(current.getUsedFunctions())[index-1] else: current = current.getVisitableNodes()[index] return current def getItemFromSourceRef(self, source_ref): def check(item): if item.node.getSourceReference() == source_ref: return item for child in item._children(): result = check(child) if result is not None: return result return check(self.root_item) class InspectNodeTreeDialog(QtGui.QDialog): def __init__(self, *args): QtGui.QDialog.__init__(self, *args) ui_dir = os.path.dirname(__file__) ui_filename = os.path.join( ui_dir, "dialogs", "InspectPythonTree.ui" ) uic.loadUi(ui_filename, self) self.treeview_nodes.setSelectionMode(self.treeview_nodes.SingleSelection) self.displayed = None self.source_code = None self.model = None self.moving = None def setModel(self, model): self.treeview_nodes.setModel(model) self.treeview_nodes.expandAll() @QtCore.pyqtSignature("on_treeview_nodes_clicked(QModelIndex)") def onTreeviewNodesClicked(self, item): tree_path = [] while item.isValid(): tree_path.insert(0, item.row()) item = item.parent() clicked_node = self.model.getNodeFromPath(tree_path) source_ref = clicked_node.getSourceReference() self.moving = True self.textedit_source.moveCursor(1, 0) for _i in range(1, source_ref.getLineNumber()): self.textedit_source.moveCursor(12, 0) self.textedit_source.setFocus() self.moving = False @QtCore.pyqtSignature("on_textedit_source_cursorPositionChanged()") def onTexteditSourceCursorMoved(self): if self.moving: return pos = self.textedit_source.textCursor().position() code = self.source_code[:pos] line = 1 for char in code: if char == '\n': line += 1 # print "Line", line item = self.model.getItemFromSourceRef( self.displayed.atLineNumber( line = line ) ) if item is not None: item_path = [] while item: item_path.insert(0, item) item = item.parent() index = QtCore.QModelIndex() parent = self.model.root_item for item in item_path[1:]: index = index.child(parent._children().index(item)+1, 1) parent = item # print self.treeview_nodes.visualRect( index ) def loadSource(self, filename): self.moving = True self.source_code = open(filename).read() self.textedit_source.setPlainText(self.source_code) self.moving = False self.displayed = SourceCodeReferences.fromFilename( filename = filename ) def displayTreeInspector(tree): app = QtGui.QApplication(sys.argv) model = NodeTreeModel(tree) dialog = InspectNodeTreeDialog() dialog.setModel(model) dialog.model = model from . import SyntaxHighlighting SyntaxHighlighting.addPythonHighlighter( document = dialog.textedit_source.document() ) dialog.loadSource(tree.getFilename()) dialog.setWindowFlags(QtCore.Qt.Window) dialog.show() import signal signal.signal(signal.SIGINT, signal.SIG_DFL) app.exec_() Nuitka-0.5.28.2/nuitka/optimizations/0000755000372000001440000000000013207540420017664 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/optimizations/Optimization.py0000644000372000001440000003306613207537242022744 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Control the flow of optimizations applied to node tree. Applies abstract execution on all so far known modules until no more optimization is possible. Every successful optimization to anything might make others possible. """ import inspect from logging import debug, info from nuitka import ModuleRegistry, Options, Variables from nuitka.importing import ImportCache from nuitka.plugins.Plugins import Plugins from nuitka.Tracing import printLine from nuitka.utils import MemoryUsage from . import Graphs, TraceCollections from .BytecodeDemotion import demoteCompiledModuleToBytecode from .Tags import TagSet _progress = Options.isShowProgress() def _attemptRecursion(module): new_modules = module.attemptRecursion() for new_module in new_modules: debug( "{source_ref} : {tags} : {message}".format( source_ref = new_module.getSourceReference().getAsString(), tags = "new_code", message = "Recursed to module package." ) ) tag_set = None def signalChange(tags, source_ref, message): """ Indicate a change to the optimization framework. """ if message is not None: debug( "{source_ref} : {tags} : {message}".format( source_ref = source_ref.getAsString(), tags = tags, message = message() if inspect.isfunction(message) else message ) ) tag_set.onSignal(tags) # Use this globally from there, without cyclic dependency. TraceCollections.signalChange = signalChange def optimizeCompiledPythonModule(module): if _progress: info( "Doing module local optimizations for '{module_name}'.".format( module_name = module.getFullName() ) ) touched = False if _progress and Options.isShowMemory(): memory_watch = MemoryUsage.MemoryWatch() while True: tag_set.clear() try: module.computeModule() except BaseException: info("Interrupted while working on '%s'." % module) raise Graphs.onModuleOptimizationStep(module) # Search for local change tags. for tag in tag_set: if tag == "new_code": continue break else: break # Otherwise we did stuff, so note that for return value. touched = True if _progress and Options.isShowMemory(): memory_watch.finish() info( "Memory usage changed during optimization of '%s': %s" % ( module.getFullName(), memory_watch.asStr() ) ) Plugins.considerImplicitImports( module = module, signal_change = signalChange ) return touched def optimizeUncompiledPythonModule(module): if _progress: info( "Doing module dependency considerations for '{module_name}':".format( module_name = module.getFullName() ) ) for used_module_name in module.getUsedModules(): used_module = ImportCache.getImportedModuleByName(used_module_name) ModuleRegistry.addUsedModule(used_module) package_name = module.getPackage() if package_name is not None: used_module = ImportCache.getImportedModuleByName(package_name) ModuleRegistry.addUsedModule(used_module) Plugins.considerImplicitImports( module = module, signal_change = signalChange ) def optimizeShlibModule(module): # Pick up parent package if any. _attemptRecursion(module) Plugins.considerImplicitImports( module = module, signal_change = signalChange ) def optimizeModule(module): if module.isPythonShlibModule(): optimizeShlibModule(module) changed = False elif module.isCompiledPythonModule(): changed = optimizeCompiledPythonModule(module) else: optimizeUncompiledPythonModule(module) changed = False return changed def areEmptyTraces(variable_traces): empty = True for variable_trace in variable_traces: if variable_trace.isAssignTrace(): empty = False break elif variable_trace.isInitTrace(): empty = False break elif variable_trace.isUninitTrace(): if variable_trace.getPrevious(): # A "del" statement can do this, and needs to prevent variable # from being removed. empty = False break elif variable_trace.hasDefiniteUsages(): # Checking definite is enough, the merges, we shall see # them as well. empty = False break elif variable_trace.isUnknownTrace(): if variable_trace.hasDefiniteUsages(): # Checking definite is enough, the merges, we shall see # them as well. empty = False break elif variable_trace.isMergeTrace(): if variable_trace.hasDefiniteUsages(): # Checking definite is enough, the merges, we shall see # them as well. empty = False break else: assert False, variable_trace return empty def optimizeUnusedClosureVariables(function_body): changed = False for closure_variable in function_body.getClosureVariables(): # print "VAR", closure_variable # Need to take closure of those. if closure_variable.isParameterVariable() and \ function_body.isExpressionGeneratorObjectBody(): continue variable_traces = function_body.trace_collection.getVariableTraces( variable = closure_variable ) empty = areEmptyTraces(variable_traces) if empty: changed = True signalChange( "var_usage", function_body.getSourceReference(), message = "Remove unused closure variable '%s'." % closure_variable.getName() ) function_body.removeClosureVariable(closure_variable) return changed def optimizeUnusedUserVariables(function_body): changed = False for local_variable in function_body.getUserLocalVariables() + function_body.getOutlineLocalVariables(): variable_traces = function_body.trace_collection.getVariableTraces( variable = local_variable ) empty = areEmptyTraces(variable_traces) if empty: signalChange( "var_usage", function_body.getSourceReference(), message = "Remove unused local variable '%s'." % local_variable.getName() ) function_body.removeUserVariable(local_variable) changed = True return changed def optimizeUnusedTempVariables(provider): remove = None for temp_variable in provider.getTempVariables(): variable_traces = provider.trace_collection.getVariableTraces( variable = temp_variable ) empty = areEmptyTraces(variable_traces) if empty: if remove is None: remove = [] remove.append(temp_variable) if remove: for temp_variable in remove: provider.removeTempVariable(temp_variable) return True else: return False def optimizeVariables(module): changed = False try: if Variables.complete: try: for function_body in module.getUsedFunctions(): if optimizeUnusedUserVariables(function_body): changed = True if optimizeUnusedClosureVariables(function_body): changed = True if optimizeUnusedTempVariables(function_body): changed = True except Exception: print("Problem with", function_body) raise# if optimizeUnusedUserVariables(module): changed = True if optimizeUnusedTempVariables(module): changed = True except Exception: print("Problem with", module) raise return changed def _traceProgress(current_module): output = """\ Optimizing module '{module_name}', {remaining:d} more modules to go \ after that.""".format( module_name = current_module.getFullName(), remaining = ModuleRegistry.remainingCount(), ) if Options.isShowMemory(): output += "Memory usage {memory}:".format( memory = MemoryUsage.getHumanReadableProcessMemoryUsage() ) info(output) def restoreFromXML(text): from nuitka.TreeXML import fromString from nuitka.nodes.NodeBases import fromXML xml = fromString(text) module = fromXML( provider = None, xml = xml ) return module def makeOptimizationPass(initial_pass): """ Make a single pass for optimization, indication potential completion. """ # Controls complex optimization, pylint: disable=too-many-branches finished = True ModuleRegistry.startTraversal() if _progress: if initial_pass: info("Initial optimization pass.") else: info("Next global optimization pass.") while True: current_module = ModuleRegistry.nextModule() if current_module is None: break if _progress: _traceProgress(current_module) # The tag set is global, so it can react to changes without context. # pylint: disable=global-statement global tag_set tag_set = TagSet() changed = optimizeModule(current_module) if changed: finished = False # Unregister collection traces from now unused code, dropping the trace # collections of functions no longer used. for current_module in ModuleRegistry.getDoneModules(): if current_module.isCompiledPythonModule(): for function in current_module.getUnusedFunctions(): Variables.updateVariablesFromCollection( old_collection = function.trace_collection, new_collection = None ) function.trace_collection = None for current_module in ModuleRegistry.getDoneModules(): if current_module.isCompiledPythonModule(): if optimizeVariables(current_module): finished = False used_functions = current_module.getUsedFunctions() for unused_function in current_module.getUnusedFunctions(): unused_function.trace_collection = None used_functions = tuple( function for function in current_module.getFunctions() if function in used_functions ) current_module.setFunctions(used_functions) return finished def _checkXMLPersistence(): new_roots = ModuleRegistry.root_modules.__class__() # @UndefinedVariable for module in tuple(ModuleRegistry.getDoneModules()): ModuleRegistry.root_modules.remove(module) if module.isPythonShlibModule(): continue text = module.asXmlText() open("out.xml", 'w').write(text) restored = restoreFromXML(text) retext = restored.asXmlText() open("out2.xml", 'w').write(retext) assert module.getOutputFilename() == restored.getOutputFilename(), \ (module.getOutputFilename(),restored.getOutputFilename()) # The variable versions give diffs. if True: # To manually enable, pylint: disable=W0125 import difflib diff = difflib.unified_diff( text.splitlines(), retext.splitlines(), "xml orig", "xml reloaded" ) for line in diff: printLine(line) new_roots.add(restored) ModuleRegistry.root_modules = new_roots ModuleRegistry.startTraversal() def optimize(): Graphs.startGraph() # First pass. if _progress: info("PASS 1:") makeOptimizationPass(False) Variables.complete = True finished = makeOptimizationPass(False) if Options.isExperimental("check_xml_persistence"): _checkXMLPersistence() # Demote compiled modules to bytecode, now that imports had a chance to be resolved, and # dependencies were handled. for module in ModuleRegistry.getDoneUserModules(): if module.isCompiledPythonModule() and \ module.mode == "bytecode": demoteCompiledModuleToBytecode(module) if _progress: info("PASS 2 ... :") # Second, "endless" pass. while not finished: finished = makeOptimizationPass(True) Graphs.endGraph() Nuitka-0.5.28.2/nuitka/optimizations/Graphs.py0000644000372000001440000000360713134660221021471 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Graph optimization states. These are not the graphs you might be thinking of. This is for rending the progress of optimization into images. """ from logging import warning from nuitka import Options from nuitka.Tracing import printLine graph = None computation_counters = {} def onModuleOptimizationStep(module): # Update the graph if active. if graph is not None: computation_counters[module] = computation_counters.get(module, 0) + 1 module_graph = module.asGraph(computation_counters[module]) graph.subgraph(module_graph) def startGraph(): # We maintain this globally to make it accessible, pylint: disable=global-statement global graph if Options.shouldCreateGraph(): try: from graphviz import Digraph # pylint: disable=I0021,import-error graph = Digraph('G') except ImportError: warning("Cannot import graphviz module, no graphing capability.") def endGraph(): if graph is not None: graph.engine = "dot" graph.graph_attr["rankdir"] = "TB" graph.render("something.dot") printLine(graph.source) Nuitka-0.5.28.2/nuitka/optimizations/Tags.py0000644000372000001440000000410713112214770021137 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Tags and set of it. Used by optimization to keep track of the current state of optimization, these tags trigger the execution of optimization steps, which in turn may emit these tags to execute other steps. """ allowed_tags = ( # New code means new statements. # Could be a new module, or an inlined exec statement. "new_code", # Added new import. "new_import", # New statements added, removed. "new_statements", # New expression added. "new_expression", # TODO: A bit unclear what this it, potentially a changed variable. "var_usage", # Detected module variable to be read only. "read_only_mvar", # New built-in reference detected. "new_builtin_ref", # New built-in call detected. "new_builtin", # New raise statement detected. "new_raise", # New constant introduced. "new_constant", ) class TagSet(set): def onSignal(self, signal): if type(signal) is str: signal = signal.split() for tag in signal: self.add(tag) def check(self, tags): for tag in tags.split(): assert tag in allowed_tags, tag if tag in self: return True return False def add(self, tag): assert tag in allowed_tags, tag set.add(self, tag) Nuitka-0.5.28.2/nuitka/optimizations/FunctionInlining.py0000644000372000001440000000602213207537242023523 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ In-lining of functions. Done by assigning the argument values to variables, and producing an outline from the in-lined function. """ from nuitka.nodes.AssignNodes import StatementAssignmentVariable from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.tree.Extractions import updateVariableUsage from nuitka.tree.TreeHelpers import makeStatementsSequence def convertFunctionCallToOutline(provider, function_ref, values): # This has got to have pretty man details, pylint: disable=too-many-locals function_body = function_ref.getFunctionBody() call_source_ref = function_ref.getSourceReference() function_source_ref = function_body.getSourceReference() outline_body = ExpressionOutlineBody( provider = provider, name = "inline", source_ref = function_source_ref ) clone = function_body.getBody().makeClone() temp_scope = outline_body.getOutlineTempScope() translation = {} for variable in function_body.getLocalVariables(): # TODO: Later we should be able to do that too. assert variable.isSharedTechnically() is False new_variable = outline_body.allocateTempVariable( temp_scope = temp_scope, name = variable.getName() ) # TODO: Lets update all at once maybe, it would take less visits. updateVariableUsage( clone, old_variable = variable, new_variable = new_variable ) translation[variable.getName()] = new_variable statements = [] if function_body.isExpressionClassBody(): argument_names = () else: argument_names = function_body.getParameters().getParameterNames() assert len(argument_names) == len(values), (argument_names, values) for argument_name, value in zip(argument_names, values): statements.append( StatementAssignmentVariable( variable = translation[argument_name], source = value, source_ref = call_source_ref, ) ) body = makeStatementsSequence( statements = (statements, clone), allow_none = False, source_ref = function_source_ref ) outline_body.setBody(body) return outline_body Nuitka-0.5.28.2/nuitka/optimizations/OptimizeBuiltinCalls.py0000644000372000001440000013527413207537242024370 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Optimize calls to built-in references to specific built-in calls. For built-in name references, we check if it's one of the supported built-in types, and then specialize for the ones, where it makes sense. """ from logging import warning from nuitka.Builtins import calledWithBuiltinArgumentNamesDecorator from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementDelVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionBuiltinGetattr, ExpressionBuiltinHasattr, ExpressionBuiltinSetattr ) from nuitka.nodes.BuiltinDecodingNodes import ( ExpressionBuiltinChr, ExpressionBuiltinOrd ) from nuitka.nodes.BuiltinDecoratorNodes import ( ExpressionBuiltinClassmethod, ExpressionBuiltinStaticmethod ) from nuitka.nodes.BuiltinDictNodes import ExpressionBuiltinDict from nuitka.nodes.BuiltinFormatNodes import ( ExpressionBuiltinAscii, ExpressionBuiltinBin, ExpressionBuiltinFormat, ExpressionBuiltinHex, ExpressionBuiltinId, ExpressionBuiltinOct ) from nuitka.nodes.BuiltinHashNodes import ExpressionBuiltinHash from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, ExpressionBuiltinIter2 ) from nuitka.nodes.BuiltinLenNodes import ExpressionBuiltinLen from nuitka.nodes.BuiltinNextNodes import ( ExpressionBuiltinNext1, ExpressionBuiltinNext2 ) from nuitka.nodes.BuiltinOpenNodes import ExpressionBuiltinOpen from nuitka.nodes.BuiltinRangeNodes import ( ExpressionBuiltinRange1, ExpressionBuiltinRange2, ExpressionBuiltinRange3, ExpressionBuiltinXrange1, ExpressionBuiltinXrange2, ExpressionBuiltinXrange3 ) from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, makeExpressionBuiltinRef ) from nuitka.nodes.BuiltinSumNodes import ( ExpressionBuiltinSum1, ExpressionBuiltinSum2 ) from nuitka.nodes.BuiltinTypeNodes import ( ExpressionBuiltinBool, ExpressionBuiltinBytearray1, ExpressionBuiltinBytearray3, ExpressionBuiltinComplex, ExpressionBuiltinFloat, ExpressionBuiltinFrozenset, ExpressionBuiltinInt, ExpressionBuiltinList, ExpressionBuiltinSet, ExpressionBuiltinStr, ExpressionBuiltinTuple ) from nuitka.nodes.BuiltinVarsNodes import ExpressionBuiltinVars from nuitka.nodes.CallNodes import makeExpressionCall from nuitka.nodes.ClassNodes import ExpressionBuiltinType3 from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ExecEvalNodes import ( ExpressionBuiltinCompile, ExpressionBuiltinEval ) from nuitka.nodes.GlobalsLocalsNodes import ( ExpressionBuiltinDir1, ExpressionBuiltinGlobals ) from nuitka.nodes.ImportNodes import ExpressionBuiltinImport from nuitka.nodes.NodeMakingHelpers import ( makeExpressionBuiltinLocals, makeRaiseExceptionReplacementExpression, makeRaiseExceptionReplacementExpressionFromInstance, wrapExpressionWithSideEffects ) from nuitka.nodes.OperatorNodes import ( ExpressionOperationBinaryDivmod, ExpressionOperationNOT, ExpressionOperationUnary ) from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.SliceNodes import ExpressionBuiltinSlice from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.nodes.TypeNodes import ( ExpressionBuiltinIsinstance, ExpressionBuiltinSuper, ExpressionBuiltinType1 ) from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from nuitka.tree.ReformulationExecStatements import wrapEvalGlobalsAndLocals from nuitka.tree.ReformulationTryFinallyStatements import ( makeTryFinallyStatement ) from nuitka.tree.TreeHelpers import ( makeCallNode, makeSequenceCreationOrConstant, makeStatementsSequence, makeStatementsSequenceFromStatement ) from . import BuiltinOptimization def dir_extractor(node): def buildDirEmptyCase(source_ref): source = makeExpressionBuiltinLocals( provider = node.getParentVariableProvider(), source_ref = source_ref ) result = makeCallNode( ExpressionAttributeLookup( source = source, attribute_name = "keys", source_ref = source_ref ), source_ref ) # For Python3, keys doesn't really return values, but instead a handle # only, but we want it to be a list. if python_version >= 300: result = ExpressionBuiltinList( value = result, source_ref = source_ref ) return result return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinDir1, builtin_spec = BuiltinOptimization.builtin_dir_spec, empty_special_class = buildDirEmptyCase ) def vars_extractor(node): def selectVarsEmptyClass(source_ref): return makeExpressionBuiltinLocals( provider = node.getParentVariableProvider(), source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinVars, builtin_spec = BuiltinOptimization.builtin_vars_spec, empty_special_class = selectVarsEmptyClass ) def import_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinImport, builtin_spec = BuiltinOptimization.builtin_import_spec ) def type_extractor(node): args = node.getCallArgs() if args is None: iter_length = 0 else: iter_length = args.getIterationLength() if iter_length == 1: return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinType1, builtin_spec = BuiltinOptimization.builtin_type1_spec ) elif iter_length == 3: return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinType3, builtin_spec = BuiltinOptimization.builtin_type3_spec ) else: return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("type() takes 1 or 3 arguments") ) def iter_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapIterCreation(callable_arg, sentinel, source_ref): if sentinel is None: return ExpressionBuiltinIter1( value = callable_arg, source_ref = source_ref ) else: return ExpressionBuiltinIter2( callable_arg = callable_arg, sentinel = sentinel, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapIterCreation, builtin_spec = BuiltinOptimization.builtin_iter_spec ) def next_extractor(node): # Split up next with and without defaults, they are not going to behave # really very similar. def selectNextBuiltinClass(iterator, default, source_ref): if default is None: return ExpressionBuiltinNext1( value = iterator, source_ref = source_ref ) else: return ExpressionBuiltinNext2( iterator = iterator, default = default, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectNextBuiltinClass, builtin_spec = BuiltinOptimization.builtin_next_spec ) def sum_extractor(node): # Split up sumwith and without start value, one is much easier. def selectSumBuiltinClass(sequence, start, source_ref): if start is None: return ExpressionBuiltinSum1( sequence = sequence, source_ref = source_ref ) else: return ExpressionBuiltinSum2( sequence = sequence, start = start, source_ref = source_ref ) def makeSum0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("sum expected at least 1 arguments, got 0") ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectSumBuiltinClass, builtin_spec = BuiltinOptimization.builtin_sum_spec, empty_special_class = makeSum0 ) def dict_extractor(node): # The "dict" built-in is a bit strange in that it accepts a position # parameter, or not, but won't have a default value. def wrapExpressionBuiltinDictCreation(positional_args, dict_star_arg, source_ref): if len(positional_args) > 1: result = makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError( "dict expected at most 1 arguments, got %d" % ( len(positional_args) ) ) ) result = wrapExpressionWithSideEffects( side_effects = positional_args, old_node = node, new_node = result ) if dict_star_arg: result = wrapExpressionWithSideEffects( side_effects = dict_star_arg, old_node = node, new_node = result ) return result return ExpressionBuiltinDict( pos_arg = positional_args[0] if positional_args else None, pairs = dict_star_arg, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinDictCreation, builtin_spec = BuiltinOptimization.builtin_dict_spec ) def chr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinChr, builtin_spec = BuiltinOptimization.builtin_chr_spec ) def ord_extractor(node): def makeOrd0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("ord() takes exactly one argument (0 given)") ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinOrd, builtin_spec = BuiltinOptimization.builtin_ord_spec, empty_special_class = makeOrd0 ) def bin_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinBin, builtin_spec = BuiltinOptimization.builtin_bin_spec ) def oct_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinOct, builtin_spec = BuiltinOptimization.builtin_oct_spec ) def hex_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinHex, builtin_spec = BuiltinOptimization.builtin_hex_spec ) def id_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinId, builtin_spec = BuiltinOptimization.builtin_id_spec ) def repr_extractor(node): def makeReprOperator(operand, source_ref): return ExpressionOperationUnary( operator = "Repr", operand = operand, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = makeReprOperator, builtin_spec = BuiltinOptimization.builtin_repr_spec ) if python_version >= 300: def ascii_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinAscii, builtin_spec = BuiltinOptimization.builtin_repr_spec ) def range_extractor(node): def selectRangeBuiltin(low, high, step, source_ref): if high is None: return ExpressionBuiltinRange1( low = low, source_ref = source_ref ) elif step is None: return ExpressionBuiltinRange2( low = low, high = high, source_ref = source_ref ) else: return ExpressionBuiltinRange3( low = low, high = high, step = step, source_ref = source_ref ) def makeRange0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("range expected at least 1 arguments, got 0") ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectRangeBuiltin, builtin_spec = BuiltinOptimization.builtin_range_spec, empty_special_class = makeRange0 ) def xrange_extractor(node): def selectXrangeBuiltin(low, high, step, source_ref): if high is None: return ExpressionBuiltinXrange1( low = low, source_ref = source_ref ) elif step is None: return ExpressionBuiltinXrange2( low = low, high = high, source_ref = source_ref ) else: return ExpressionBuiltinXrange3( low = low, high = high, step = step, source_ref = source_ref ) def makeXrange0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError( "xrange requires 1-3 int arguments" if python_version < 300 else "range expected 1 arguments, got 0" ) ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectXrangeBuiltin, builtin_spec = BuiltinOptimization.builtin_xrange_spec, empty_special_class = makeXrange0 ) def len_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinLen, builtin_spec = BuiltinOptimization.builtin_len_spec ) def tuple_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinTuple, builtin_spec = BuiltinOptimization.builtin_tuple_spec ) def list_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinList, builtin_spec = BuiltinOptimization.builtin_list_spec ) def set_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinSet, builtin_spec = BuiltinOptimization.builtin_set_spec ) def frozenset_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinFrozenset, builtin_spec = BuiltinOptimization.builtin_frozenset_spec ) def float_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinFloat, builtin_spec = BuiltinOptimization.builtin_float_spec ) def complex_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinComplex, builtin_spec = BuiltinOptimization.builtin_complex_spec ) def str_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinStr, builtin_spec = BuiltinOptimization.builtin_str_spec ) if python_version < 300: from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinUnicode def unicode_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinUnicode, builtin_spec = BuiltinOptimization.builtin_unicode_spec ) else: from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinBytes def bytes_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinBytes, builtin_spec = BuiltinOptimization.builtin_bytes_spec ) def bool_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinBool, builtin_spec = BuiltinOptimization.builtin_bool_spec ) def int_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinInt, builtin_spec = BuiltinOptimization.builtin_int_spec ) if python_version < 300: from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinLong def long_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinLong, builtin_spec = BuiltinOptimization.builtin_long_spec ) def globals_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinGlobals, builtin_spec = BuiltinOptimization.builtin_globals_spec ) def locals_extractor(node): # Note: Locals on the module level is really globals. provider = node.getParentVariableProvider() def makeLocalsNode(source_ref): return makeExpressionBuiltinLocals( provider = provider, source_ref = source_ref ) if provider.isCompiledPythonModule(): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinGlobals, builtin_spec = BuiltinOptimization.builtin_globals_spec ) else: return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = makeLocalsNode, builtin_spec = BuiltinOptimization.builtin_locals_spec ) if python_version < 300: from nuitka.nodes.ExecEvalNodes import ExpressionBuiltinExecfile def execfile_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapExpressionBuiltinExecfileCreation(filename, globals_arg, locals_arg, source_ref): outline_body = ExpressionOutlineBody( provider = node.getParentVariableProvider(), name = "execfile_call", source_ref = source_ref ) globals_ref, locals_ref, tried, final = wrapEvalGlobalsAndLocals( provider = node.getParentVariableProvider(), globals_node = globals_arg, locals_node = locals_arg, temp_scope = outline_body.getOutlineTempScope(), source_ref = source_ref ) tried = makeStatementsSequence( statements = ( tried, StatementReturn( expression = ExpressionBuiltinExecfile( source_code = makeCallNode( ExpressionAttributeLookup( source = ExpressionBuiltinOpen( filename = filename, mode = makeConstantRefNode( constant = "rU", source_ref = source_ref ), buffering = None, source_ref = source_ref ), attribute_name = "read", source_ref = source_ref ), source_ref ), globals_arg = globals_ref, locals_arg = locals_ref, source_ref = source_ref ), source_ref = source_ref ) ), allow_none = False, source_ref = source_ref ) outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outline_body, tried = tried, final = final, source_ref = source_ref ) ) ) return outline_body return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinExecfileCreation, builtin_spec = BuiltinOptimization.builtin_execfile_spec ) def eval_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapEvalBuiltin(source, globals_arg, locals_arg, source_ref): provider = node.getParentVariableProvider() outline_body = ExpressionOutlineBody( provider = node.getParentVariableProvider(), name = "eval_call", source_ref = source_ref ) globals_ref, locals_ref, tried, final = wrapEvalGlobalsAndLocals( provider = provider, globals_node = globals_arg, locals_node = locals_arg, temp_scope = outline_body.getOutlineTempScope(), source_ref = source_ref ) # The wrapping should not relocate to the "source_ref". assert globals_arg is None or \ globals_ref.getSourceReference() == \ globals_arg.getSourceReference() assert locals_arg is None or \ locals_ref.getSourceReference() == \ locals_arg.getSourceReference() source_variable = outline_body.allocateTempVariable( temp_scope = None, name = "source" ) final.setStatements( final.getStatements() + ( StatementDelVariable( variable = source_variable, tolerant = True, source_ref = source_ref ), ) ) strip_choice = makeConstantRefNode( constant = (" \t",), source_ref = source_ref ) if python_version >= 300: strip_choice = ExpressionConditional( condition = ExpressionComparisonIs( left = ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), source_ref = source_ref ), right = makeExpressionBuiltinRef( builtin_name = "bytes", source_ref = source_ref ), source_ref = source_ref ), expression_yes = makeConstantRefNode( constant = (b" \t",), source_ref = source_ref ), expression_no = strip_choice, source_ref = source_ref ) # Source needs some special treatment for eval, if it's a string, it # must be stripped. string_fixup = [ StatementAssignmentVariable( variable = source_variable, source = makeExpressionCall( called = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), attribute_name = "strip", source_ref = source_ref ), args = strip_choice, # This is a tuple kw = None, source_ref = source_ref ), source_ref = source_ref ) ] acceptable_builtin_types = [ ExpressionBuiltinAnonymousRef( builtin_name = "code", source_ref = source_ref, ) ] if python_version >= 270: acceptable_builtin_types.append( makeExpressionBuiltinRef( builtin_name = "memoryview", source_ref = source_ref, ) ) statements = ( StatementAssignmentVariable( variable = source_variable, source = source, source_ref = source_ref, ), StatementConditional( condition = ExpressionOperationNOT( operand = ExpressionBuiltinIsinstance( instance = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), classes = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = acceptable_builtin_types, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), yes_branch = StatementsSequence( statements = string_fixup, source_ref = source_ref ), no_branch = None, source_ref = source_ref ), StatementReturn( expression = ExpressionBuiltinEval( source_code = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), globals_arg = globals_ref, locals_arg = locals_ref, source_ref = source_ref ), source_ref = source_ref ) ) tried = makeStatementsSequence( statements = ( tried, ) + statements, allow_none = False, source_ref = source_ref ) outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outline_body, tried = tried, final = final, source_ref = source_ref ) ) ) return outline_body return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapEvalBuiltin, builtin_spec = BuiltinOptimization.builtin_eval_spec ) if python_version >= 300: from nuitka.nodes.ExecEvalNodes import ExpressionBuiltinExec def exec_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapExpressionBuiltinExecCreation(source, globals_arg, locals_arg, source_ref): provider = node.getParentVariableProvider() outline_body = ExpressionOutlineBody( provider = provider, name = "exec_call", source_ref = source_ref ) globals_ref, locals_ref, tried, final = wrapEvalGlobalsAndLocals( provider = provider, globals_node = globals_arg, locals_node = locals_arg, temp_scope = outline_body.getOutlineTempScope(), source_ref = source_ref ) tried = makeStatementsSequence( statements = ( tried, StatementReturn( expression = ExpressionBuiltinExec( source_code = source, globals_arg = globals_ref, locals_arg = locals_ref, source_ref = source_ref ), source_ref = source_ref ), ), allow_none = False, source_ref = source_ref ) # Hack: Allow some APIs to work already tried.parent = outline_body outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = provider, tried = tried, final = final, source_ref = source_ref ) ) ) return outline_body return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinExecCreation, builtin_spec = BuiltinOptimization.builtin_eval_spec ) def compile_extractor(node): def wrapExpressionBuiltinCompileCreation(source_code, filename, mode, flags, dont_inherit, optimize = None, source_ref = None): return ExpressionBuiltinCompile( source_code = source_code, filename = filename, mode = mode, flags = flags, dont_inherit = dont_inherit, optimize = optimize, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapExpressionBuiltinCompileCreation, builtin_spec = BuiltinOptimization.builtin_compile_spec ) def open_extractor(node): def makeOpen0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError( "Required argument 'name' (pos 1) not found" if python_version < 300 else "Required argument 'file' (pos 1) not found" ) ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinOpen, builtin_spec = BuiltinOptimization.builtin_open_spec, empty_special_class = makeOpen0 ) def super_extractor(node): @calledWithBuiltinArgumentNamesDecorator def wrapSuperBuiltin(type_arg, object_arg, source_ref): if type_arg is None and python_version >= 300: if provider.isCompiledPythonModule(): return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "RuntimeError", exception_value = "super(): no arguments", ) type_arg = ExpressionVariableRef( # Ought to be already closure taken due to "super" flag in # tree building. variable = provider.getVariableForReference( variable_name = "__class__" ), source_ref = source_ref ) # If we already have this as a local variable, then use that # instead. type_arg_owner = type_arg.getVariable().getOwner() if type_arg_owner is provider or \ not (type_arg_owner.isExpressionFunctionBody() or \ type_arg_owner.isExpressionClassBody()): type_arg = None if type_arg is None: return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "SystemError" if python_version < 331 else "RuntimeError", exception_value = "super(): __class__ cell not found", ) if object_arg is None: if provider.isExpressionGeneratorObjectBody() or \ provider.isExpressionCoroutineObjectBody() or \ provider.isExpressionAsyncgenObjectBody(): parameter_provider = provider.getParentVariableProvider() else: parameter_provider = provider if parameter_provider.getParameters().getArgumentCount() > 0: par1_name = parameter_provider.getParameters().getArgumentNames()[0] object_arg = ExpressionVariableRef( variable = provider.getVariableForReference( variable_name = par1_name ), source_ref = source_ref ) if not object_arg.getVariable().isParameterVariable(): return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "SystemError" if python_version < 330 else "RuntimeError", exception_value = "super(): __class__ cell not found", ) else: return makeRaiseExceptionReplacementExpression( expression = node, exception_type = "RuntimeError", exception_value = "super(): no arguments" ) return ExpressionBuiltinSuper( super_type = type_arg, super_object = object_arg, source_ref = source_ref ) provider = node.getParentVariableProvider() if not provider.isCompiledPythonModule(): provider.discardFlag("has_super") return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapSuperBuiltin, builtin_spec = BuiltinOptimization.builtin_super_spec ) def hasattr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinHasattr, builtin_spec = BuiltinOptimization.builtin_hasattr_spec ) def getattr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinGetattr, builtin_spec = BuiltinOptimization.builtin_getattr_spec ) def setattr_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinSetattr, builtin_spec = BuiltinOptimization.builtin_setattr_spec ) def isinstance_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinIsinstance, builtin_spec = BuiltinOptimization.builtin_isinstance_spec ) def bytearray_extractor(node): def makeBytearray0(source_ref): return makeConstantRefNode( constant = bytearray(), source_ref = source_ref, ) def selectNextBuiltinClass(string, encoding, errors, source_ref): if encoding is None: return ExpressionBuiltinBytearray1( value = string, source_ref = source_ref ) else: return ExpressionBuiltinBytearray3( string = string, encoding = encoding, errors = errors, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = selectNextBuiltinClass, builtin_spec = BuiltinOptimization.builtin_bytearray_spec, empty_special_class = makeBytearray0 ) def slice_extractor(node): def wrapSlice(start, stop, step, source_ref): if start is not None and stop is None: # Default rules are strange. If one argument is given, it's the # second one then. stop = start start = None return ExpressionBuiltinSlice( start = start, stop = stop, step = step, source_ref = source_ref ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = wrapSlice, builtin_spec = BuiltinOptimization.builtin_slice_spec ) def hash_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinHash, builtin_spec = BuiltinOptimization.builtin_hash_spec ) def format_extractor(node): def makeFormat0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("format() takes at least 1 argument (0 given)") ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinFormat, builtin_spec = BuiltinOptimization.builtin_format_spec, empty_special_class = makeFormat0 ) def staticmethod_extractor(node): def makeStaticmethod0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("staticmethod expected 1 arguments, got 0") ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinStaticmethod, builtin_spec = BuiltinOptimization.builtin_staticmethod_spec, empty_special_class = makeStaticmethod0 ) def classmethod_extractor(node): def makeStaticmethod0(source_ref): # pylint: disable=unused-argument return makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = TypeError("classmethod expected 1 arguments, got 0") ) return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionBuiltinClassmethod, builtin_spec = BuiltinOptimization.builtin_classmethod_spec, empty_special_class = makeStaticmethod0 ) def divmod_extractor(node): return BuiltinOptimization.extractBuiltinArgs( node = node, builtin_class = ExpressionOperationBinaryDivmod, builtin_spec = BuiltinOptimization.builtin_divmod_spec, ) _dispatch_dict = { "compile" : compile_extractor, "globals" : globals_extractor, "locals" : locals_extractor, "eval" : eval_extractor, "dir" : dir_extractor, "vars" : vars_extractor, "__import__" : import_extractor, "chr" : chr_extractor, "ord" : ord_extractor, "bin" : bin_extractor, "oct" : oct_extractor, "hex" : hex_extractor, "id" : id_extractor, "type" : type_extractor, "iter" : iter_extractor, "next" : next_extractor, "sum" : sum_extractor, "tuple" : tuple_extractor, "list" : list_extractor, "dict" : dict_extractor, "set" : set_extractor, "frozenset" : frozenset_extractor, "float" : float_extractor, "complex" : complex_extractor, "str" : str_extractor, "bool" : bool_extractor, "int" : int_extractor, "repr" : repr_extractor, "len" : len_extractor, "super" : super_extractor, "hasattr" : hasattr_extractor, "getattr" : getattr_extractor, "setattr" : setattr_extractor, "isinstance" : isinstance_extractor, "bytearray" : bytearray_extractor, "slice" : slice_extractor, "hash" : hash_extractor, "format" : format_extractor, "open" : open_extractor, "staticmethod" : staticmethod_extractor, "classmethod" : classmethod_extractor, "divmod" : divmod_extractor } if python_version < 300: # These are not in Python3 _dispatch_dict["long"] = long_extractor _dispatch_dict["unicode"] = unicode_extractor _dispatch_dict["execfile"] = execfile_extractor _dispatch_dict["xrange"] = xrange_extractor _dispatch_dict["range"] = range_extractor else: # This one is not in Python2: _dispatch_dict["bytes"] = bytes_extractor _dispatch_dict["ascii"] = ascii_extractor _dispatch_dict["exec"] = exec_extractor # The Python3 range is really an xrange, use that. _dispatch_dict["range"] = xrange_extractor def check(): from nuitka.Builtins import builtin_names for builtin_name in _dispatch_dict: assert builtin_name in builtin_names, builtin_name check() _builtin_white_list = ( # Not supporting 'print', because it could be replaced, and is not # worth the effort yet. "print", # TODO: This could, and should be supported, as we could e.g. lower # types easily for it. "sorted", # TODO: This would be very worthwhile, as it could easily optimize # its iteration away. "zip", # TODO: This would be most precious due to the type hint it gives "enumerate", # TODO: Also worthwhile for known values. "reversed", # TODO: Not sure what this really is about. "memoryview", ) def _describeNewNode(builtin_name, inspect_node): """ Describe the change for better understanding. """ if inspect_node.isExpressionSideEffects(): inspect_node = inspect_node.getExpression() if inspect_node.isExpressionBuiltinImport(): tags = "new_import" message = """\ Replaced dynamic "__import__" call with static built-in call.""" elif inspect_node.isExpressionBuiltin() or \ inspect_node.isStatementExec(): tags = "new_builtin" message = "Replaced call to built-in '%s' with built-in call '%s'." % ( builtin_name, inspect_node.kind, ) elif inspect_node.isExpressionRaiseException(): tags = "new_raise" message = """\ Replaced call to built-in '%s' with exception raise.""" % ( builtin_name, ) elif inspect_node.isExpressionOperationBinary(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with binary operation '%s'.""" % ( builtin_name, inspect_node.getOperator() ) elif inspect_node.isExpressionOperationUnary(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with unary operation '%s'.""" % ( builtin_name, inspect_node.getOperator() ) elif inspect_node.isExpressionCall(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with call.""" % ( builtin_name, ) elif inspect_node.isExpressionOutlineBody(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with outlined call.""" % builtin_name elif inspect_node.isExpressionConstantRef(): tags = "new_expression" message = """\ Replaced call to built-in '%s' with constant value.""" % builtin_name else: assert False, (builtin_name, "->", inspect_node) return tags, message def computeBuiltinCall(builtin_name, call_node): # There is some dispatching for how to output various types of changes, # with lots of cases. if builtin_name in _dispatch_dict: new_node = _dispatch_dict[builtin_name](call_node) assert new_node is not call_node, builtin_name assert new_node is not None, builtin_name # For traces, we are going to ignore side effects, and output traces # only based on the basis of it. tags, message = _describeNewNode(builtin_name, new_node) return new_node, tags, message else: if False and builtin_name not in _builtin_white_list: warning( "Not handling built-in '%s', consider support." % builtin_name ) return call_node, None, None Nuitka-0.5.28.2/nuitka/optimizations/BuiltinOptimization.py0000644000372000001440000004263413207537242024274 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Optimizations of built-ins to built-in calls. """ from __future__ import print_function import math import sys from nuitka.nodes.ParameterSpecs import ( ParameterSpec, TooManyArguments, matchCall ) from nuitka.PythonVersions import python_version class BuiltinParameterSpec(ParameterSpec): __slots__ = ("builtin",) def __init__(self, name, arg_names, default_count, list_star_arg = None, dict_star_arg = None): ParameterSpec.__init__( self, ps_name = name, ps_normal_args = arg_names, ps_list_star_arg = list_star_arg, ps_dict_star_arg = dict_star_arg, ps_default_count = default_count, ps_kw_only_args = () ) self.builtin = __builtins__[name] assert default_count <= len(arg_names) def __repr__(self): return "" % self.name def getName(self): return self.name def isCompileTimeComputable(self, values): # By default, we make this dependent on the ability to compute the # arguments, which is of course a good start for most cases, so this # is for overloads, pylint: disable=no-self-use for value in values: if value is not None and not value.isCompileTimeConstant(): return False return True def simulateCall(self, given_values): # Using star dict call for simulation and catch any exception as really # fatal, pylint: disable=broad-except try: given_normal_args = given_values[:len(self.normal_args)] if self.list_star_arg: given_list_star_args = given_values[len(self.normal_args)] else: given_list_star_args = None if self.dict_star_arg: given_dict_star_args = given_values[ -1 ] else: given_dict_star_args = None arg_dict = {} for arg_name, given_value in zip(self.normal_args, given_normal_args): assert type(given_value) not in (tuple, list), \ ("do not like a tuple %s" % (given_value,)) if given_value is not None: arg_dict[ arg_name ] = given_value.getCompileTimeConstant() if given_dict_star_args: for given_dict_star_arg in reversed(given_dict_star_args): arg_name = given_dict_star_arg.getKey().getCompileTimeConstant() arg_value = given_dict_star_arg.getValue().getCompileTimeConstant() arg_dict[arg_name] = arg_value except Exception as e: sys.exit("Fatal problem: %r" % e) if given_list_star_args: return self.builtin( *(value.getCompileTimeConstant() for value in given_list_star_args), **arg_dict ) else: return self.builtin(**arg_dict) class BuiltinParameterSpecNoKeywords(BuiltinParameterSpec): __slots__ = () def allowsKeywords(self): return False def simulateCall(self, given_values): # Using star dict call for simulation and catch any exception as really fatal, # pylint: disable=broad-except try: if self.list_star_arg: given_list_star_arg = given_values[ len(self.normal_args) ] else: given_list_star_arg = None arg_list = [] refuse_more = False for _arg_name, given_value in zip(self.normal_args, given_values): assert type(given_value) not in (tuple, list), ("do not like tuple %s" % (given_value,)) if given_value is not None: if not refuse_more: arg_list.append(given_value.getCompileTimeConstant()) else: assert False else: refuse_more = True if given_list_star_arg is not None: arg_list += [ value.getCompileTimeConstant() for value in given_list_star_arg ] except Exception as e: print("Fatal error: ", end = ' ', file = sys.stderr) import traceback traceback.print_exc() sys.exit(repr(e)) return self.builtin(*arg_list) class BuiltinParameterSpecExceptions(BuiltinParameterSpec): def __init__(self, exception_name): # TODO: Parameter default_count makes no sense for exceptions probably. BuiltinParameterSpec.__init__( self, name = exception_name, arg_names = (), default_count = 0, list_star_arg = "args" ) def allowsKeywords(self): return False def getKeywordRefusalText(self): return "exceptions.%s does not take keyword arguments" % self.name def getCallableName(self): return "exceptions." + self.getName() def makeBuiltinExceptionParameterSpec(exception_name): if exception_name == "ImportError" and python_version >= 330: # TODO: Create this beast, needs keyword only arguments to be supported, # currently user of this function must take care to not have them. pass return BuiltinParameterSpecExceptions( exception_name = exception_name ) builtin_int_spec = BuiltinParameterSpec("int", ('x', "base"), 2) # These builtins are only available for Python2 if python_version < 300: builtin_long_spec = BuiltinParameterSpec( "long", ('x', "base"), 2 ) builtin_execfile_spec = BuiltinParameterSpecNoKeywords( "execfile", ("filename", "globals", "locals"), 2 ) builtin_unicode_spec = BuiltinParameterSpec( "unicode", ("string", "encoding", "errors"), 3 ) builtin_xrange_spec = BuiltinParameterSpecNoKeywords( "xrange" if python_version < 300 else "range", ("start", "stop", "step"), 2 ) builtin_bool_spec = BuiltinParameterSpec("bool", ('x',), 1) builtin_float_spec = BuiltinParameterSpec("float", ('x',), 1) builtin_complex_spec = BuiltinParameterSpec("complex", ("real", "imag"), 2) # This built-in have variable parameters for Python2/3 if python_version < 300: builtin_str_spec = BuiltinParameterSpec("str", ("object",), 1) else: builtin_str_spec = BuiltinParameterSpec("str", ("object", "encoding", "errors"), 3) builtin_len_spec = BuiltinParameterSpecNoKeywords("len", ("object",), 0) builtin_dict_spec = BuiltinParameterSpec("dict", (), 0, "list_args", "dict_args") builtin_len_spec = BuiltinParameterSpecNoKeywords("len", ("object",), 0) builtin_tuple_spec = BuiltinParameterSpec("tuple", ("sequence",), 1) builtin_list_spec = BuiltinParameterSpec("list", ("sequence",), 1) builtin_set_spec = BuiltinParameterSpecNoKeywords("set", ("iterable",), 1) builtin_frozenset_spec = BuiltinParameterSpecNoKeywords("frozenset", ("iterable",), 1) builtin_import_spec = BuiltinParameterSpec("__import__", ("name", "globals", "locals", "fromlist", "level"), 4) if python_version < 300: builtin_open_spec = BuiltinParameterSpec("open", ("name", "mode", "buffering"), 3) else: builtin_open_spec = BuiltinParameterSpec("open", ("name", "mode", "buffering", "encoding", "errors", "newline", "closefd", "opener"), 7) builtin_chr_spec = BuiltinParameterSpecNoKeywords("chr", ('i',), 0) builtin_ord_spec = BuiltinParameterSpecNoKeywords("ord", ('c',), 0) builtin_bin_spec = BuiltinParameterSpecNoKeywords("bin", ("number",), 0) builtin_oct_spec = BuiltinParameterSpecNoKeywords("oct", ("number",), 0) builtin_hex_spec = BuiltinParameterSpecNoKeywords("hex", ("number",), 0) builtin_id_spec = BuiltinParameterSpecNoKeywords("id", ("object",), 0) builtin_repr_spec = BuiltinParameterSpecNoKeywords("repr", ("object",), 0) builtin_dir_spec = BuiltinParameterSpecNoKeywords("dir", ("object",), 1) builtin_vars_spec = BuiltinParameterSpecNoKeywords("vars", ("object",), 1) builtin_locals_spec = BuiltinParameterSpecNoKeywords("locals", (), 0) builtin_globals_spec = BuiltinParameterSpecNoKeywords("globals", (), 0) builtin_eval_spec = BuiltinParameterSpecNoKeywords("eval", ("source", "globals", "locals"), 2) if python_version < 300: builtin_compile_spec = BuiltinParameterSpec( "compile", ("source", "filename", "mode", "flags", "dont_inherit"), 2 ) else: builtin_compile_spec = BuiltinParameterSpec( "compile", ("source", "filename", "mode", "flags", "dont_inherit", "optimize"), 3 ) if python_version >= 300: builtin_exec_spec = BuiltinParameterSpecNoKeywords( "exec", ("source", "globals", "locals"), 2 ) # Note: Iter in fact names its first argument if the default applies # "collection", fixed up in a wrapper. builtin_iter_spec = BuiltinParameterSpecNoKeywords("iter", ("callable", "sentinel"), 1) builtin_next_spec = BuiltinParameterSpecNoKeywords("next", ("iterator", "default"), 1) # Note: type with 1 and type with 3 arguments are too different. builtin_type1_spec = BuiltinParameterSpecNoKeywords("type", ("object",), 0) builtin_type3_spec = BuiltinParameterSpecNoKeywords("type", ("name", "bases", "dict"), 0) builtin_super_spec = BuiltinParameterSpecNoKeywords("super", ("type", "object"), 1 if python_version < 300 else 2) builtin_hasattr_spec = BuiltinParameterSpecNoKeywords("hasattr", ("object", "name"), 0) builtin_getattr_spec = BuiltinParameterSpecNoKeywords("getattr", ("object", "name", "default"), 1) builtin_setattr_spec = BuiltinParameterSpecNoKeywords("setattr", ("object", "name", "value"), 0) builtin_isinstance_spec = BuiltinParameterSpecNoKeywords("isinstance", ("instance", "classes"), 0) class BuiltinBytearraySpec(BuiltinParameterSpecNoKeywords): def isCompileTimeComputable(self, values): # For bytearrays, we need to avoid the case of large bytearray # construction from an integer at compile time. result = BuiltinParameterSpecNoKeywords.isCompileTimeComputable( self, values = values ) if result and len(values) == 1: index_value = values[0].getIndexValue() if index_value is None: return result return index_value < 256 else: return result builtin_bytearray_spec = BuiltinBytearraySpec("bytearray", ("string", "encoding", "errors"), 2) if python_version >= 300: builtin_bytes_spec = BuiltinBytearraySpec("bytes", ("string", "encoding", "errors"), 3) # Beware: One argument version defines "stop", not "start". builtin_slice_spec = BuiltinParameterSpecNoKeywords("slice", ("start", "stop", "step"), 2) builtin_hash_spec = BuiltinParameterSpecNoKeywords("hash", ("object",), 0) builtin_format_spec = BuiltinParameterSpecNoKeywords("format", ("value", "format_spec"), 1) builtin_sum_spec = BuiltinParameterSpecNoKeywords("sum", ("sequence", "start"), 1) builtin_staticmethod_spec = BuiltinParameterSpecNoKeywords("staticmethod", ("function",), 0) builtin_classmethod_spec = BuiltinParameterSpecNoKeywords("classmethod", ("function",), 0) if python_version < 300: builtin_sorted_spec = BuiltinParameterSpecNoKeywords("sorted", ("iterable", "cmp", "key", "reverse"), 2) else: builtin_sorted_spec = BuiltinParameterSpecNoKeywords("sorted", ("iterable", "key", "reverse"), 2) builtin_reversed_spec = BuiltinParameterSpecNoKeywords("reversed", ("object",), 0) builtin_reversed_spec = BuiltinParameterSpecNoKeywords("reversed", ("object",), 0) if python_version < 300: builtin_enumerate_spec = BuiltinParameterSpec("enumerate", ("sequence",), 0) else: builtin_enumerate_spec = BuiltinParameterSpec("enumerate", ("iterable",), 0) class BuiltinRangeSpec(BuiltinParameterSpecNoKeywords): def isCompileTimeComputable(self, values): # For ranges, we need have many cases that can prevent the ability # to pre-compute, pylint: disable=too-many-branches,too-many-return-statements result = BuiltinParameterSpecNoKeywords.isCompileTimeComputable( self, values = values ) if result: arg_count = len(values) if arg_count == 1: low = values[0] # If it's not a number constant, we can compute the exception # that will be raised. if not low.isNumberConstant(): return True return low.getConstant() < 256 elif arg_count == 2: low, high = values # If it's not a number constant, we can compute the exception # that will be raised. if not low.isNumberConstant() or not high.isNumberConstant(): return True return high.getConstant() - low.getConstant() < 256 elif arg_count == 3: low, high, step = values if not low.isNumberConstant() or \ not high.isNumberConstant() or \ not step.isNumberConstant(): return True low = low.getConstant() high = high.getConstant() step = step.getConstant() # It's going to give a ZeroDivisionError in this case. if step == 0: return True if low < high: if step < 0: return True else: return math.ceil(float(high - low) / step) < 256 else: if step > 0: return True else: return math.ceil(float(high - low) / step) < 256 else: assert False else: return False builtin_range_spec = BuiltinRangeSpec("range", ("start", "stop", "step"), 2) if python_version >= 300: builtin_ascii_spec = BuiltinParameterSpecNoKeywords("ascii", ("object",), 0) builtin_divmod_spec = BuiltinParameterSpecNoKeywords("divmod", ("left", "right"), 0) def extractBuiltinArgs(node, builtin_spec, builtin_class, empty_special_class = None): try: kw = node.getCallKw() # TODO: Could check for too many / too few, even if they are unknown, we # might raise that error, but that need not be optimized immediately. if kw is not None: if not kw.isMappingWithConstantStringKeys(): return None pairs = kw.getMappingStringKeyPairs() if pairs and not builtin_spec.allowsKeywords(): raise TooManyArguments( TypeError(builtin_spec.getKeywordRefusalText()) ) else: pairs = () args = node.getCallArgs() if args: if not args.canPredictIterationValues(): return None positional = args.getIterationValues() else: positional = () if not positional and not pairs and empty_special_class is not None: return empty_special_class(source_ref = node.getSourceReference()) args_dict = matchCall( func_name = builtin_spec.getName(), args = builtin_spec.getArgumentNames(), star_list_arg = builtin_spec.getStarListArgumentName(), star_dict_arg = builtin_spec.getStarDictArgumentName(), num_defaults = builtin_spec.getDefaultCount(), positional = positional, pairs = pairs ) except TooManyArguments as e: from nuitka.nodes.NodeMakingHelpers import ( makeRaiseExceptionReplacementExpressionFromInstance, wrapExpressionWithSideEffects ) return wrapExpressionWithSideEffects( new_node = makeRaiseExceptionReplacementExpressionFromInstance( expression = node, exception = e.getRealException() ), old_node = node, side_effects = node.extractSideEffectsPreCall() ) args_list = [] for argument_name in builtin_spec.getArgumentNames(): args_list.append(args_dict[argument_name]) if builtin_spec.getStarListArgumentName() is not None: args_list.append(args_dict[builtin_spec.getStarListArgumentName()]) if builtin_spec.getStarDictArgumentName() is not None: args_list.append(args_dict[builtin_spec.getStarDictArgumentName()]) # Using list reference for passing the arguments without names, result = builtin_class( *args_list, source_ref = node.getSourceReference() ) result.setCompatibleSourceReference(node.getCompatibleSourceReference()) return result Nuitka-0.5.28.2/nuitka/optimizations/VariableTraces.py0000644000372000001440000003400513207537242023137 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Variable trace objects. Variable traces indicate the flow of variables and merges their versions for the SSA (Single State Assignment) form being used in Nuitka. Variable version can start as: * Unknown (maybe initialized, maybe not, we cannot know) * Uninit (definitely not initialized, first version, or after "del" statement) * Init (definitely initialized, e.g. parameter variables) * Merge (result of diverged code paths) """ from logging import debug from nuitka.codegen.c_types.CTypePyObjectPtrs import ( CTypeCellObject, CTypePyObjectPtr, CTypePyObjectPtrPtr ) from nuitka.Options import isExperimental from nuitka.utils import InstanceCounters enable_bool_ctype = isExperimental("enable_bool_ctype") class VariableTraceBase(object): # We are going to have many instance attributes, pylint: disable=too-many-instance-attributes __slots__ = ( "owner", "variable", "version", "usage_count", "has_potential_usages", "name_usages", "closure_usages", "is_escaped", "previous" ) @InstanceCounters.counted_init def __init__(self, owner, variable, version, previous): self.owner = owner self.variable = variable self.version = version # Definite usage indicator. self.usage_count = 0 # Potential usages indicator that an assignment value may be used. self.has_potential_usages = False # If 0, this indicates, the variable name needs to be assigned as name. self.name_usages = 0 self.closure_usages = False # If False, this indicates that the value is not yet escaped. self.is_escaped = False # Previous trace this is replacing. self.previous = previous __del__ = InstanceCounters.counted_del() def getVariable(self): return self.variable def getOwner(self): return self.owner def getVersion(self): return self.version def addClosureUsage(self): self.addUsage() self.closure_usages = True def addUsage(self): self.usage_count += 1 def addPotentialUsage(self): self.has_potential_usages = True def addNameUsage(self): self.usage_count += 1 self.name_usages += 1 def onValueEscape(self): self.is_escaped = True def isEscaped(self): return self.is_escaped def hasDefiniteUsages(self): return self.usage_count > 0 def getDefiniteUsages(self): return self.usage_count def hasPotentialUsages(self): return self.has_potential_usages def getNameUsageCount(self): return self.name_usages def getPrevious(self): return self.previous def getPickedCType(self, context): """ Return type to use for specific context. """ user = context.getEntryPoint() owner = self.variable.getEntryPoint() if owner is user: if self.variable.isSharedTechnically(): result = CTypeCellObject else: if enable_bool_ctype: shapes = self.variable.getTypeShapes() if len(shapes) > 1: return CTypePyObjectPtr else: assert shapes, self return shapes.pop().getCType() else: return CTypePyObjectPtr elif context.isForDirectCall(): if self.variable.isSharedTechnically(): result = CTypeCellObject else: result = CTypePyObjectPtrPtr else: result = CTypeCellObject return result @staticmethod def isAssignTrace(): return False @staticmethod def isUninitTrace(): return False @staticmethod def isInitTrace(): return False @staticmethod def isUnknownTrace(): return False @staticmethod def isMergeTrace(): return False def mustHaveValue(self): # TODO: Temporarily disable far reaching of assumptions, until value # escaping can be trusted. if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically() is not False: return False # Merge traces have this overloaded. return self.isInitTrace() or self.isAssignTrace() def mustNotHaveValue(self): if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically() is not False: return False return self.isUninitTrace() def getReplacementNode(self, usage): # Virtual method, pylint: disable=no-self-use,unused-argument return None def hasShapeDictionaryExact(self): # Virtual method, pylint: disable=no-self-use return False class VariableTraceUninit(VariableTraceBase): __slots__ = () def __init__(self, owner, variable, version, previous): VariableTraceBase.__init__( self, owner = owner, variable = variable, version = version, previous = previous ) def __repr__(self): return "".format( variable = self.variable, version = self.version ) @staticmethod def isUninitTrace(): return True def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug(" Starts out uninitialized") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") class VariableTraceInit(VariableTraceBase): __slots__ = () def __init__(self, owner, variable, version): VariableTraceBase.__init__( self, owner = owner, variable = variable, version = version, previous = None ) def __repr__(self): return "".format( variable = self.variable, version = self.version ) def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug(" Starts initialized") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") @staticmethod def isInitTrace(): return True class VariableTraceUnknown(VariableTraceBase): def __init__(self, owner, variable, version, previous): VariableTraceBase.__init__( self, owner = owner, variable = variable, version = version, previous = previous ) def __repr__(self): return "".format( variable = self.variable, version = self.version ) def dump(self): debug( "Trace of %s %d:", self.variable, self.version ) debug(" Starts unknown") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") @staticmethod def isUnknownTrace(): return True def addUsage(self): self.usage_count += 1 if self.previous is not None: self.previous.addPotentialUsage() def addNameUsage(self): self.addUsage() if self.previous is not None: self.previous.addNameUsage() def addPotentialUsage(self): old = self.has_potential_usages if not old: self.has_potential_usages = True if self.previous is not None: self.previous.addPotentialUsage() class VariableTraceAssign(VariableTraceBase): __slots__ = ("assign_node", "replace_it") def __init__(self, owner, assign_node, variable, version, previous): VariableTraceBase.__init__( self, owner = owner, variable = variable, version = version, previous = previous ) self.assign_node = assign_node self.replace_it = None def __repr__(self): return """\ """.format( variable = self.variable, version = self.version, source_ref = self.assign_node.getSourceReference().getAsString() ) def dump(self): debug("Trace of %s %d:", self.variable, self.version) debug(" Starts assigned") if self.usage_count: debug(" -> has %s usages" % self.usage_count) if self.is_escaped: debug(" -> value escapes") @staticmethod def isAssignTrace(): return True def getAssignNode(self): return self.assign_node def setReplacementNode(self, replacement): self.replace_it = replacement def getReplacementNode(self, usage): if self.replace_it is not None: return self.replace_it(usage) else: return None def hasShapeDictionaryExact(self): return self.assign_node.getAssignSource().hasShapeDictionaryExact() class VariableTraceMerge(VariableTraceBase): """ Merge of two or more traces. Happens at the end of conditional blocks. This is "phi" in SSA theory. Also used for merging multiple "return", "break" or "continue" exits. """ __slots__ = () def __init__(self, variable, version, traces): VariableTraceBase.__init__( self, owner = traces[0].owner, variable = variable, version = version, previous = tuple(traces) ) def __repr__(self): return """\ """.format( variable = self.variable, version = self.version, previous = tuple(previous.getVersion() for previous in self.previous) ) @staticmethod def isMergeTrace(): return True def dump(self): debug("Trace of %s %d:", self.variable, self.version) debug( " Merge of %s", " <-> ".join(self.previous), ) def mustHaveValue(self): # TODO: Temporarily disable far reaching of assumptions, until value # escaping can be trusted. if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically() is not False: return False for previous in self.previous: if not previous.isInitTrace() and not previous.isAssignTrace(): return False return True def mustNotHaveValue(self): if self.variable.isModuleVariable() or \ self.variable.isSharedTechnically() is not False: return False for previous in self.previous: if not previous.isUninitTrace(): return False return True def addUsage(self): self.usage_count += 1 for previous in self.previous: previous.addPotentialUsage() def addNameUsage(self): self.usage_count += 1 for previous in self.previous: previous.addPotentialUsage() previous.addNameUsage() def addPotentialUsage(self): old = self.has_potential_usages if not old: self.has_potential_usages = True for previous in self.previous: previous.addPotentialUsage() def hasShapeDictionaryExact(self): for previous in self.previous: if not previous.hasShapeDictionaryExact(): return False return True class VariableTraceLoopMerge(VariableTraceBase): """ Merge of loop wrap around with loop start value. Happens at the start of loop blocks. This is for loop closed SSA, to make it clear, that the entered value, cannot be trusted inside the loop. They will start out with just one previous, and later be updated with all of the variable versions at loop continue times. """ __slots__ = ("loop_finished",) def __init__(self, variable, version, previous): VariableTraceBase.__init__( self, owner = previous.owner, variable = variable, version = version, previous = previous ) self.loop_finished = False previous.addPotentialUsage() def hasDefiniteUsages(self): if not self.loop_finished: return True return self.usage_count > 0 def hasPotentialUsages(self): if not self.loop_finished: return True return self.has_potential_usages def getNameUsageCount(self): if not self.loop_finished: return 10000 return self.name_usages def getPrevious(self): assert self.loop_finished return self.previous @staticmethod def isMergeTrace(): return True def addLoopContinueTraces(self, continue_traces): self.previous.addPotentialUsage() for continue_trace in continue_traces: continue_trace.addPotentialUsage() self.previous = (self.previous,) + tuple(continue_traces) Nuitka-0.5.28.2/nuitka/optimizations/TraceCollections.py0000644000372000001440000006412313207537242023511 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Trace collection (also often still referred to as constraint collection). At the core of value propagation there is the collection of constraints that allow to propagate knowledge forward or not. This is about collecting these constraints and to manage them. """ import contextlib from logging import debug from nuitka import Tracing, Variables from nuitka.__past__ import iterItems # Python3 compatibility. from nuitka.containers.oset import OrderedSet from nuitka.importing.ImportCache import ( getImportedModuleByName, isImportedModuleByName ) from nuitka.ModuleRegistry import addUsedModule from nuitka.nodes.NodeMakingHelpers import getComputationResult from nuitka.PythonVersions import python_version from nuitka.utils.InstanceCounters import counted_del, counted_init from .VariableTraces import ( VariableTraceAssign, VariableTraceInit, VariableTraceLoopMerge, VariableTraceMerge, VariableTraceUninit, VariableTraceUnknown ) signalChange = None class CollectionTracingMixin(object): def __init__(self): # For functions, when we are in here, the currently active one, self.variable_actives = {} def getVariableCurrentTrace(self, variable): return self.getVariableTrace( variable = variable, version = self.getCurrentVariableVersion(variable) ) def markCurrentVariableTrace(self, variable, version): self.variable_actives[variable] = version def getCurrentVariableVersion(self, variable): try: return self.variable_actives[variable] except KeyError: # Initialize variables on the fly. if not self.hasVariableTrace(variable, 0): self.initVariable(variable) self.markCurrentVariableTrace(variable, 0) return self.variable_actives[variable] def getActiveVariables(self): return self.variable_actives.keys() def markActiveVariableAsUnknown(self, variable): current = self.getVariableCurrentTrace( variable = variable, ) if not current.isUnknownTrace(): version = variable.allocateTargetNumber() self.addVariableTrace( variable = variable, version = version, trace = VariableTraceUnknown( owner = self.owner, variable = variable, version = version, previous = current ) ) self.markCurrentVariableTrace(variable, version) def markActiveVariableAsLoopMerge(self, variable): current = self.getVariableCurrentTrace( variable = variable, ) version = variable.allocateTargetNumber() result = VariableTraceLoopMerge( variable = variable, version = version, previous = current ) self.addVariableTrace( variable = variable, version = version, trace = result ) self.markCurrentVariableTrace(variable, version) return result def markActiveVariablesAsUnknown(self): for variable in self.getActiveVariables(): self.markActiveVariableAsUnknown(variable) class CollectionStartpointMixin(object): def __init__(self): # Variable assignments performed in here, last issued number, only used # to determine the next number that should be used for a new assignment. self.variable_versions = {} # The full trace of a variable with a version for the function or module # this is. self.variable_traces = {} self.break_collections = None self.continue_collections = None self.return_collections = None self.exception_collections = None self.outline_functions = None def getLoopBreakCollections(self): return self.break_collections def onLoopBreak(self, collection = None): if collection is None: collection = self self.break_collections.append( TraceCollectionBranch( parent = collection, name = "loop break" ) ) def getLoopContinueCollections(self): return self.continue_collections def onLoopContinue(self, collection = None): if collection is None: collection = self self.continue_collections.append( TraceCollectionBranch( parent = collection, name = "loop continue" ) ) def onFunctionReturn(self, collection = None): if collection is None: collection = self if self.return_collections is not None: self.return_collections.append( TraceCollectionBranch( parent = collection, name = "return" ) ) def onExceptionRaiseExit(self, raisable_exceptions, collection = None): # TODO: We might want to track per exception, pylint: disable=unused-argument if collection is None: collection = self if self.exception_collections is not None: self.exception_collections.append( TraceCollectionBranch( parent = collection, name = "exception" ) ) def getFunctionReturnCollections(self): return self.return_collections def getExceptionRaiseCollections(self): return self.exception_collections def hasVariableTrace(self, variable, version): return (variable, version) in self.variable_traces def getVariableTrace(self, variable, version): return self.variable_traces[(variable, version)] def getVariableTraces(self, variable): result = [] for key, variable_trace in iterItems(self.variable_traces): candidate = key[0] if variable is candidate: result.append(variable_trace) return result def getVariableTracesAll(self): return self.variable_traces def addVariableTrace(self, variable, version, trace): key = variable, version assert key not in self.variable_traces, (key, self) self.variable_traces[key] = trace def addVariableMergeMultipleTrace(self, variable, traces): version = variable.allocateTargetNumber() trace_merge = VariableTraceMerge( variable = variable, version = version, traces = traces ) self.addVariableTrace(variable, version, trace_merge) return version # return version, trace_merge def dumpTraces(self): debug("Constraint collection state: %s", self) for _variable_desc, variable_trace in sorted(iterItems(self.variable_traces)): # debug( "%r: %r", variable_trace ) variable_trace.dump() def dumpActiveTraces(self): Tracing.printSeparator() Tracing.printLine("Active are:") for variable, _version in sorted(self.variable_actives.iteritems()): self.getVariableCurrentTrace(variable).dump() Tracing.printSeparator() def _initVariableUnknown(self, variable): trace = VariableTraceUnknown( owner = self.owner, variable = variable, version = 0, previous = None ) self.addVariableTrace( variable = variable, version = 0, trace = trace ) return trace def _initVariableInit(self, variable): trace = VariableTraceInit( owner = self.owner, variable = variable, version = 0 ) self.addVariableTrace( variable = variable, version = 0, trace = trace ) return trace def _initVariableUninit(self, variable): trace = VariableTraceUninit( owner = self.owner, variable = variable, version = 0, previous = None ) self.addVariableTrace( variable = variable, version = 0, trace = trace ) return trace def updateVariablesFromCollection(self, old_collection): Variables.updateVariablesFromCollection(old_collection, self) @contextlib.contextmanager def makeAbortStackContext(self, catch_breaks, catch_continues, catch_returns, catch_exceptions): if catch_breaks: old_break_collections = self.break_collections self.break_collections = [] if catch_continues: old_continue_collections = self.continue_collections self.continue_collections = [] if catch_returns: old_return_collections = self.return_collections self.return_collections = [] if catch_exceptions: old_exception_collections = self.exception_collections self.exception_collections = [] yield if catch_breaks: self.break_collections = old_break_collections if catch_continues: self.continue_collections = old_continue_collections if catch_returns: self.return_collections = old_return_collections if catch_exceptions: self.exception_collections = old_exception_collections def initVariable(self, variable): if variable.isParameterVariable(): result = self._initVariableInit(variable) elif variable.isLocalVariable(): result = self._initVariableUninit(variable) elif variable.isModuleVariable(): result = self._initVariableUnknown(variable) elif variable.isTempVariable(): result = self._initVariableUninit(variable) else: assert False, variable assert result.getVariable() is variable return result def addOutlineFunction(self, outline): if self.outline_functions is None: self.outline_functions = [outline] else: self.outline_functions.append(outline) def getOutlineFunctions(self): return self.outline_functions class TraceCollectionBase(CollectionTracingMixin): __del__ = counted_del() @counted_init def __init__(self, owner, name, parent): CollectionTracingMixin.__init__(self) self.owner = owner self.parent = parent self.name = name # Value state extra information per node. self.value_states = {} def __repr__(self): return "<%s for %s %d>" % ( self.__class__.__name__, self.name, id(self) ) @staticmethod def signalChange(tags, source_ref, message): # This is monkey patched from another module. signalChange(tags, source_ref, message) def onUsedModule(self, module_name): return self.parent.onUsedModule(module_name) @staticmethod def mustAlias(a, b): if a.isExpressionVariableRef() and b.isExpressionVariableRef(): return a.getVariable() is b.getVariable() return False @staticmethod def mustNotAlias(a, b): # TODO: not yet really implemented if a.isExpressionConstantRef() and b.isExpressionConstantRef(): if a.isMutable() or b.isMutable(): return True return False def removeKnowledge(self, node): pass def onControlFlowEscape(self, node): # TODO: One day, we should trace which nodes exactly cause a variable # to be considered escaped, pylint: disable=unused-argument for variable in self.getActiveVariables(): if variable.isModuleVariable(): # print variable self.markActiveVariableAsUnknown(variable) elif python_version >= 300 or variable.isSharedTechnically() is not False: # print variable # TODO: Could be limited to shared variables that are actually # written to. Most of the time, that won't be the case. self.markActiveVariableAsUnknown(variable) def removeAllKnowledge(self): self.markActiveVariablesAsUnknown() def getVariableTrace(self, variable, version): return self.parent.getVariableTrace(variable, version) def hasVariableTrace(self, variable, version): return self.parent.hasVariableTrace(variable, version) def addVariableTrace(self, variable, version, trace): self.parent.addVariableTrace(variable, version, trace) def addVariableMergeMultipleTrace(self, variable, traces): return self.parent.addVariableMergeMultipleTrace(variable, traces) def onVariableSet(self, assign_node): version = assign_node.getVariableVersion() variable = assign_node.getVariable() # TODO: The variable, version and assign_node are redundant to pass. variable_trace = VariableTraceAssign( owner = self.owner, assign_node = assign_node, variable = variable, version = version, previous = self.getVariableCurrentTrace( variable = variable ) ) self.addVariableTrace( variable = variable, version = version, trace = variable_trace ) # Make references point to it. self.markCurrentVariableTrace(variable, version) return variable_trace def onVariableDel(self, variable, version): # Add a new trace, allocating a new version for the variable, and # remember the delete of the current old_trace = self.getVariableCurrentTrace(variable) variable_trace = VariableTraceUninit( owner = self.owner, variable = variable, version = version, previous = old_trace ) # Assign to not initialized again. self.addVariableTrace( variable = variable, version = version, trace = variable_trace ) # Make references point to it. self.markCurrentVariableTrace(variable, version) def onLocalsUsage(self, locals_owner): result = [] include_closure = locals_owner.isExpressionFunctionBody() and \ not locals_owner.isUnoptimized() for variable in self.getActiveVariables(): if variable.isLocalVariable() and \ (variable.getOwner() is locals_owner or include_closure and locals_owner.hasClosureVariable(variable)) and \ variable.getName() != ".0": variable_trace = self.getVariableCurrentTrace( variable ) variable_trace.addNameUsage() result.append( ( variable, variable_trace.getVersion() ) ) return result def onVariableContentEscapes(self, variable): self.getVariableCurrentTrace(variable).onValueEscape() def onExpression(self, expression, allow_none = False): if expression is None and allow_none: return None assert expression.isExpression(), expression assert expression.parent, expression # Now compute this expression, allowing it to replace itself with # something else as part of a local peep hole optimization. r = expression.computeExpressionRaw( trace_collection = self ) assert type(r) is tuple, expression new_node, change_tags, change_desc = r if change_tags is not None: # This is mostly for tracing and indication that a change occurred # and it may be interesting to look again. self.signalChange( change_tags, expression.getSourceReference(), change_desc ) if new_node is not expression: expression.replaceWith(new_node) if new_node.isExpressionVariableRef() or \ new_node.isExpressionTempVariableRef(): # Remember the reference for constraint collection. assert new_node.variable_trace.hasDefiniteUsages() return new_node def onStatement(self, statement): try: assert statement.isStatement(), statement new_statement, change_tags, change_desc = \ statement.computeStatement(self) # print new_statement, change_tags, change_desc if new_statement is not statement: self.signalChange( change_tags, statement.getSourceReference(), change_desc ) return new_statement except Exception: Tracing.printError( "Problem with statement at %s:" % statement.getSourceReference().getAsString() ) raise def mergeBranches(self, collection_yes, collection_no): """ Merge two alternative branches into this trace. This is mostly for merging conditional branches, or other ways of having alternative control flow. This deals with up to two alternative branches to both change this collection. """ # Refuse to do stupid work if collection_yes is None and collection_no is None: return None elif collection_yes is None or collection_no is None: # Handle one branch case, we need to merge versions backwards as # they may make themselves obsolete. return self.mergeMultipleBranches( collections = (self, collection_yes or collection_no) ) else: return self.mergeMultipleBranches( collections = (collection_yes, collection_no) ) def mergeMultipleBranches(self, collections): assert collections # Optimize for length 1, which is trivial merge and needs not a # lot of work. if len(collections) == 1: self.replaceBranch(collections[0]) return None variable_versions = {} for collection in collections: for variable, version in iterItems(collection.variable_actives): if variable not in variable_versions: variable_versions[variable] = set([version]) else: variable_versions[variable].add(version) for collection in collections: for variable, versions in iterItems(variable_versions): if variable not in collection.variable_actives: versions.add(0) self.variable_actives = {} # merge_traces = None for variable, versions in iterItems(variable_versions): if len(versions) == 1: version, = versions else: version = self.addVariableMergeMultipleTrace( variable = variable, traces = [ self.getVariableTrace(variable, version) for version in versions ] ) # if merge_traces is None: # merge_traces = [trace_merge] # else: # merge_traces.append(trace_merge) self.markCurrentVariableTrace(variable, version) # Return "None", or turn the list into a tuple for memory savings. # return merge_traces and tuple(merge_traces) def replaceBranch(self, collection_replace): self.variable_actives.update(collection_replace.variable_actives) collection_replace.variable_actives = None def onLoopBreak(self, collection = None): if collection is None: collection = self return self.parent.onLoopBreak(collection) def onLoopContinue(self, collection = None): if collection is None: collection = self return self.parent.onLoopContinue(collection) def onFunctionReturn(self, collection = None): if collection is None: collection = self return self.parent.onFunctionReturn(collection) def onExceptionRaiseExit(self, raisable_exceptions, collection = None): if collection is None: collection = self return self.parent.onExceptionRaiseExit(raisable_exceptions, collection) def getLoopBreakCollections(self): return self.parent.getLoopBreakCollections() def getLoopContinueCollections(self): return self.parent.getLoopContinueCollections() def getFunctionReturnCollections(self): return self.parent.getFunctionReturnCollections() def getExceptionRaiseCollections(self): return self.parent.getExceptionRaiseCollections() def makeAbortStackContext(self, catch_breaks, catch_continues, catch_returns, catch_exceptions): return self.parent.makeAbortStackContext( catch_breaks = catch_breaks, catch_continues = catch_continues, catch_returns = catch_returns, catch_exceptions = catch_exceptions ) def getCompileTimeComputationResult(self, node, computation, description): new_node, change_tags, message = getComputationResult( node = node, computation = computation, description = description ) if change_tags == "new_raise": self.onExceptionRaiseExit(BaseException) return new_node, change_tags, message def getIteratorNextCount(self, iter_node): return self.value_states.get(iter_node, None) def initIteratorValue(self, iter_node): # TODO: More complex state information will be needed eventually. self.value_states[iter_node] = 0 def onIteratorNext(self, iter_node): if iter_node in self.value_states: self.value_states[iter_node] += 1 def resetValueStates(self): for key in self.value_states: self.value_states[key] = None def addOutlineFunction(self, outline): self.parent.addOutlineFunction(outline) class TraceCollectionBranch(TraceCollectionBase): def __init__(self, name, parent): TraceCollectionBase.__init__( self, owner = parent.owner, name = name, parent = parent ) self.variable_actives = dict(parent.variable_actives) def computeBranch(self, branch): if branch.isStatementsSequence(): result = branch.computeStatementsSequence( trace_collection = self ) if result is not branch: branch.replaceWith(result) else: self.onExpression( expression = branch ) def initVariable(self, variable): variable_trace = self.parent.initVariable(variable) assert variable_trace.getVersion() == 0 self.variable_actives[variable] = 0 return variable_trace def dumpTraces(self): Tracing.printSeparator() self.parent.dumpTraces() Tracing.printSeparator() def dumpActiveTraces(self): Tracing.printSeparator() Tracing.printLine("Active are:") for variable, _version in sorted(self.variable_actives.iteritems()): self.getVariableCurrentTrace(variable).dump() Tracing.printSeparator() class TraceCollectionFunction(CollectionStartpointMixin, TraceCollectionBase): def __init__(self, parent, function_body): assert function_body.isExpressionFunctionBody() or \ function_body.isExpressionClassBody() or \ function_body.isExpressionGeneratorObjectBody() or \ function_body.isExpressionCoroutineObjectBody() or \ function_body.isExpressionAsyncgenObjectBody(), function_body CollectionStartpointMixin.__init__(self) TraceCollectionBase.__init__( self, owner = function_body, name = "collection_" + function_body.getCodeName(), parent = parent ) if function_body.isExpressionFunctionBody(): for parameter_variable in function_body.getParameters().getAllVariables(): self._initVariableInit(parameter_variable) self.variable_actives[parameter_variable] = 0 for closure_variable in function_body.getClosureVariables(): self._initVariableUnknown(closure_variable) self.variable_actives[closure_variable] = 0 class TraceCollectionModule(CollectionStartpointMixin, TraceCollectionBase): def __init__(self, module): CollectionStartpointMixin.__init__(self) TraceCollectionBase.__init__( self, owner = module, name = "module", parent = None ) self.used_modules = OrderedSet() def onUsedModule(self, module_name): assert type(module_name) is str, module_name self.used_modules.add(module_name) if isImportedModuleByName(module_name): module = getImportedModuleByName(module_name) addUsedModule(module) def getUsedModules(self): return self.used_modules Nuitka-0.5.28.2/nuitka/optimizations/__init__.py0000644000372000001440000000150113112214770021773 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/optimizations/BytecodeDemotion.py0000644000372000001440000000472513207537242023513 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Demotion of compiled modules to bytecode modules. """ import marshal from logging import debug from nuitka.importing.ImportCache import replaceImportedModule from nuitka.ModuleRegistry import replaceRootModule from nuitka.nodes.ModuleNodes import makeUncompiledPythonModule from nuitka.plugins.Plugins import Plugins from nuitka.tree.SourceReading import readSourceCodeFromFilename def demoteCompiledModuleToBytecode(module): """ Demote a compiled module to uncompiled (bytecode). """ full_name = module.getFullName() filename = module.getCompileTimeFilename() debug( "Demoting module '%s' to bytecode from '%s'.", full_name, filename ) source_code = readSourceCodeFromFilename(full_name, filename) source_code = Plugins.onFrozenModuleSourceCode( module_name = full_name, is_package = False, source_code = source_code ) bytecode = compile(source_code, filename, "exec", dont_inherit = True) bytecode = Plugins.onFrozenModuleBytecode( module_name = full_name, is_package = False, bytecode = bytecode ) uncompiled_module = makeUncompiledPythonModule( module_name = full_name, filename = filename, bytecode = marshal.dumps(bytecode), is_package = module.isCompiledPythonPackage(), user_provided = True, technical = False ) replaceImportedModule( old = module, new = uncompiled_module ) replaceRootModule( old = module, new = uncompiled_module ) assert module.trace_collection is not None uncompiled_module.setUsedModules(module.trace_collection.getUsedModules()) Nuitka-0.5.28.2/nuitka/PythonVersions.py0000644000372000001440000001334513122472300020342 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Python version specifics. This abstracts the Python version decisions. This makes decisions based on the numbers, and attempts to give them meaningful names. Where possible it should attempt to make run time detections. """ import os import re import sys def getSupportedPythonVersions(): return ("2.6", "2.7", "3.2", "3.3", "3.4", "3.5", "3.6") def getSupportedPythonVersionStr(): supported_python_versions = getSupportedPythonVersions() supported_python_versions_str = repr(supported_python_versions)[1:-1] supported_python_versions_str = re.sub( r"(.*),(.*)$", r"\1, or\2", supported_python_versions_str ) return supported_python_versions_str def _getPythonVersion(): big, major, minor = sys.version_info[0:3] return big * 100 + major * 10 + minor python_version = _getPythonVersion() python_version_full_str = '.'.join(str(s) for s in sys.version_info[0:3]) python_version_str = '.'.join(str(s) for s in sys.version_info[0:2]) def isAtLeastSubVersion(version): if version < 280 and \ python_version >= 280 and \ python_version < 300: return True if version // 10 != python_version // 10: return False return python_version >= version def getErrorMessageExecWithNestedFunction(): """ Error message of the concrete Python in case an exec occurs in a function that takes a closure variable. """ assert python_version < 300 # Need to use "exec" to detect the syntax error, pylint: disable=W0122 try: exec(""" def f(): exec "" def nested(): return closure""") except SyntaxError as e: return e.message.replace("'f'", "'%s'") def getComplexCallSequenceErrorTemplate(): if not hasattr(getComplexCallSequenceErrorTemplate, "result"): try: # We are doing this on purpose, to get the exception. # pylint: disable= not-an-iterable,not-callable f = None f(*None) except TypeError as e: result = e.args[0].replace("NoneType object", "%s").replace("NoneType", "%s") getComplexCallSequenceErrorTemplate.result = result else: sys.exit("Error, cannot detect expected error message.") return getComplexCallSequenceErrorTemplate.result def needsSetLiteralReverseInsertion(): try: value = eval("{1,1.0}.pop()") # pylint: disable=eval-used except SyntaxError: return False else: return type(value) is float def needsDuplicateArgumentColOffset(): if python_version < 353: return False else: return True def isUninstalledPython(): if os.name == "nt": import ctypes.wintypes GetSystemDirectory = ctypes.windll.kernel32.GetSystemDirectoryW # @UndefinedVariable GetSystemDirectory.argtypes = ( ctypes.wintypes.LPWSTR, ctypes.wintypes.DWORD ) GetSystemDirectory.restype = ctypes.wintypes.DWORD MAX_PATH = 4096 buf = ctypes.create_unicode_buffer(MAX_PATH) res = GetSystemDirectory(buf, MAX_PATH) assert res != 0 system_path = os.path.normcase(buf.value) return not getRunningPythonDLLPath().startswith(system_path) return "Anaconda" in sys.version or "WinPython" in sys.version def getRunningPythonDLLPath(): import ctypes.wintypes GetModuleHandle = ctypes.windll.kernel32.GetModuleHandleW # @UndefinedVariable GetModuleHandle.argtypes = ( ctypes.wintypes.LPWSTR, ) GetModuleHandle.restype = ctypes.wintypes.DWORD big, major = sys.version_info[0:2] dll_module_name = "python%d%d" % (big, major) module_handle = GetModuleHandle(dll_module_name) if module_handle == 0: dll_module_name += "_d" module_handle = GetModuleHandle(dll_module_name) assert module_handle, (sys.executable, dll_module_name, sys.flags.debug) MAX_PATH = 4096 buf = ctypes.create_unicode_buffer(MAX_PATH) GetModuleFileName = ctypes.windll.kernel32.GetModuleFileNameW # @UndefinedVariable GetModuleFileName.argtypes = ( ctypes.wintypes.HANDLE, ctypes.wintypes.LPWSTR, ctypes.wintypes.DWORD ) GetModuleFileName.restype = ctypes.wintypes.DWORD res = GetModuleFileName(module_handle, buf, MAX_PATH) assert res != 0 dll_path = os.path.normcase(buf.value) assert os.path.exists(dll_path), dll_path return dll_path def getTargetPythonDLLPath(): dll_path = getRunningPythonDLLPath() from nuitka.Options import isPythonDebug if dll_path.endswith("_d.dll"): if not isPythonDebug(): dll_path = dll_path[:-6] + ".dll" if not os.path.exists(dll_path): sys.exit("Error, cannot switch to non-debug Python, not installed.") else: if isPythonDebug(): dll_path = dll_path[:-4] + "_d.dll" if not os.path.exists(dll_path): sys.exit("Error, cannot switch to debug Python, not installed.") return dll_path Nuitka-0.5.28.2/nuitka/__init__.py0000644000372000001440000000150113112214770017062 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/build/0000755000372000001440000000000013207540420016052 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/SingleExe.scons0000644000372000001440000015033213207540035021012 0ustar hayenusers00000000000000# -*- python -*- # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # The Nuitka scons file. If you have Scons or platform knowledge, please be # especially invited and contribute improvements. # # This file is used to build an executable or shared library. Nuitka needs no # build process for itself, although it can be compiled using the same method. from __future__ import print_function import hashlib import os import platform import re import shutil import signal import subprocess import sys import SCons def getArguments(): arg_encoding = ARGUMENTS.get("argument_encoding") result = {} for key, value in ARGUMENTS.items(): # @UndefinedVariable if arg_encoding is not None: value = value.decode(arg_encoding) result[key] = value return result ARGUMENTS = getArguments() # The directory containing the C files generated by Nuitka to be built using # scons. They are referred to as sources from here on. The ARGUMENTS is what # Scons provides to us. source_name = ARGUMENTS["name"] source_dir = ARGUMENTS["source_dir"] # The directory containing Nuitka provided C files to be built and where it # should be used. nuitka_src = ARGUMENTS["nuitka_src"] static_src = os.path.join(source_dir, "static") # The directory containing Nuitka cache. nuitka_cache = ARGUMENTS["nuitka_cache"] # The name of executable or extension module that we produce. result_basepath = ARGUMENTS["result_name"] def getBoolOption(option_name, default = None): """ Small helper for boolean mode flags.""" if default is None: value = ARGUMENTS[option_name] else: value = ARGUMENTS.get(option_name, "True" if default else "False") return value.lower() in ("yes", "true", '1') # The directory to use for cache directory. cache_mode = getBoolOption("cache_mode", False) # Module mode: Create a Python extension module, create an executable otherwise. module_mode = getBoolOption("module_mode", False) # Debug mode: Less optimizations, debug information in the resulting binary. debug_mode = getBoolOption("debug_mode", False) # Profiling mode: Outputs vmprof based information from program run. profile_mode = getBoolOption("profile_mode", False) # Python version to target. python_version = ARGUMENTS["python_version"] # The ABI flags to target. abiflags = ARGUMENTS.get("abiflags", "") python_abi_version = python_version + abiflags # Python debug mode: reference count checking, assertions in CPython core. python_debug = getBoolOption("python_debug", False) # Full compatibility, even where it's stupid, i.e. do not provide information, # even if available, in order to assert maximum compatibility. Intended to # control level of compatibility to absurd. full_compat_mode = getBoolOption("full_compat", False) # Experimental indications. Do things that are not yet safe to do. experimental = ARGUMENTS.get("experimental", "").split(",") # Tracing mode. Output program progress. trace_mode = getBoolOption("trace_mode", False) # LTO mode: Use link time optimizations of g++ compiler if available and known # good with the compiler in question. The 4.5 one didn't have good enough # support, the compiled result would not run correctly. lto_mode = getBoolOption("lto_mode", False) # Windows target mode: Compile for Windows. Used to be an option, but we # no longer cross compile this way. win_target = os.name == "nt" # Windows subsystem mode: Disable console for windows builds. win_disable_console = getBoolOption("win_disable_console", False) # Windows might be running a Python whose DLL we have to use. uninstalled_python = getBoolOption("uninstalled_python", False) # Unstriped mode: Do not remove debug symbols. unstripped_mode = getBoolOption("unstripped_mode", False) # Clang compiler mode, forced on MacOS X and FreeBSD, optional on Linux. clang_mode = getBoolOption("clang_mode", False) if sys.platform == "darwin" or "freebsd" in sys.platform: clang_mode = True # MinGW compiler mode, optional and interesting to Windows only. mingw_mode = getBoolOption("mingw_mode", False) # Shared library and compiled modules count, determines the need for the # compiled module loader. module_count = int(ARGUMENTS["module_count"]) # Frozen modules count, determines the need for the bytecode frozen # modules loader. frozen_modules = int(ARGUMENTS.get("frozen_modules", 0)) # Standalone mode standalone_mode = getBoolOption("standalone_mode", False) # Show scons mode, output information about Scons operation show_scons_mode = getBoolOption("show_scons", False) # Home of Python to be compiled against, used to find include files and # libraries to link against. python_prefix = ARGUMENTS["python_prefix"] # Target arch, uses for compiler choice and quick linking of constants binary # data. target_arch = ARGUMENTS["target_arch"] # Icon for executable (windows-only) icon_path = ARGUMENTS.get("icon_path", None) # Forced MSVC version (windows-only) msvc_version = ARGUMENTS.get("msvc_version", None) no_python_warnings = getBoolOption("no_python_warnings", False) # sys.flags values to pass along # python_sysflag_py3k_warning python_sysflag_py3k_warning = getBoolOption("python_sysflag_py3k_warning", False) # python_sysflag_division_warning python_sysflag_division_warning = getBoolOption("python_sysflag_division_warning", False) # python_sysflag_division_warning python_sysflag_bytes_warning = getBoolOption("python_sysflag_bytes_warning", False) # python_sysflag_no_site python_sysflag_no_site = getBoolOption("python_sysflag_no_site", False) # python_sysflag_verbose python_sysflag_verbose = getBoolOption("python_sysflag_verbose", False) # python_sysflag_unicode python_sysflag_unicode = getBoolOption("python_sysflag_unicode", False) # Amount of jobs to use. job_count = GetOption("num_jobs") # @UndefinedVariable # Add environment specified compilers to the PATH variable. if "CC" in os.environ: cc_dirname = os.path.dirname(os.environ["CC"]) if os.path.isdir(cc_dirname): os.environ["PATH"] = os.pathsep.join([ cc_dirname ] + os.environ["PATH"].split(os.pathsep)) if "CXX" in os.environ: cxx_dirname = os.path.dirname(os.environ["CXX"]) if os.path.isdir(cxx_dirname): os.environ["PATH"] = os.pathsep.join([ cxx_dirname ] + os.environ["PATH"].split(os.pathsep)) def getExecutablePath(filename, initial): """ Find an execute in either normal PATH, or Scons detected PATH. """ if os.path.exists(filename): return True # Variable substitution from environment is needed, because this can contain # "$CC" which should be looked up too. while filename.startswith('$'): filename = env[filename[1:]] # Append ".exe" suffix on Windows if not already present. if win_target and not filename.lower().endswith(".exe"): filename += ".exe" # Either look at the initial "PATH" as given from the outside or look at the # current environment. if initial: search_path = os.environ["PATH"] else: search_path = env._dict["ENV"]["PATH"] # Now check in each path element, much like the shell will. path_elements = search_path.split(os.pathsep) for path_element in path_elements: path_element = path_element.strip('"') full = os.path.join(path_element, filename) if os.path.exists(full): return full return None # Patch the compiler detection. orig_detect = Environment.Detect found_gcc = False gcc_version = None blacklisted_tools = ( "f95", "f90", "f77", "gfortran", "ifort", "javah", "tar", "dmd", "gdc", "flex", "bison", "ranlib", "ar", "ldc2", "pdflatex", "pdftex", "latex", "tex", "dvipdf", "dvips", "gs", "swig", "ifl", "rpcgen", "rpmbuild", "bk", "p4", "m4", "sccs", "rcs", "cvs" ) if win_target: blacklisted_tools += ( "as", "gas", "nasm", ) # From gcc.py of Scons def detectVersion(env, cc): """Return the version of the GNU compiler, or None if it is not a GNU compiler.""" cc = env.subst(cc) if not cc: return None version = None #pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) + ['-dumpversion'], pipe = SCons.Action._subproc( env, SCons.Util.CLVar(cc) + ['--version'], stdin = 'devnull', stderr = 'devnull', stdout = subprocess.PIPE ) # -dumpversion was added in GCC 3.0. As long as we're supporting # GCC versions older than that, we should use --version and a # regular expression. #line = pipe.stdout.read().strip() #if line: # version = line line = pipe.stdout.readline() if str is not bytes: line = line.decode("utf8") match = re.search(r'[0-9]+(\.[0-9]+)+', line) if match: version = match.group(0) # Non-GNU compiler's output (like AIX xlc's) may exceed the stdout buffer: # So continue with reading to let the child process actually terminate. while pipe.stdout.readline(): pass ret = pipe.wait() if ret != 0: return None return version def myDetect(self, progs): global found_gcc, gcc_version # Don't even search a C++ compiler if we found a suitable C11 compiler. if found_gcc and ("g++" in progs or "c++" in progs): return None # Don't consider Fortran, tar, D, we don't need it. for blacklisted_tool in blacklisted_tools: if blacklisted_tool in progs: return None # For RHEL and EPEL to work, smuggle these names in. if "g++" in progs and not win_target: progs += ["g++44", "eg++"] result = orig_detect(self, progs) # Special considerations for gcc. if result in ("gcc", "cc"): gcc_version = detectVersion(self, result) # Ignore gcc before gcc version 5, no C11 support. We will find the # C++ compiler of it though. if gcc_version < "5": progs = [ "gcc-6.5", "gcc-6.3", "gcc-6.1", "gcc-5.5", "gcc-5.3", "gcc-5.1", ] result = orig_detect(self, progs) if result is not None: gcc_version = detectVersion(self, result) if result is not None: found_gcc = True # print progs, result return result Environment.Detect = myDetect def createEnvironment(compiler_tools): args = {} # If we are on Windows, and MinGW is not enforced, lets see if we can # find "cl.exe", and if we do, disable automatic scan. if win_target and \ not mingw_mode and \ (getExecutablePath("cl", initial = True) is not None or \ getExecutablePath("gcc", initial = True) is not None): args["MSVC_USE_SCRIPT"] = False return Environment( # @UndefinedVariable # We want the outside environment to be passed through. ENV = os.environ, # Extra tools configuration for scons. tools = compiler_tools, # The shared libraries should not be named "lib...", because CPython # requires the filename "module_name.so" to load it. SHLIBPREFIX = "", # Under windows, specify the target architecture is needed for Scons # to pick up MSVC. TARGET_ARCH = target_arch, MSVC_VERSION = msvc_version, **args ) if mingw_mode: # Force usage of MinGW. compiler_tools = ["mingw"] else: # Everything else should use default. compiler_tools = ["default"] # Create Scons environment, the main control tool. Don't include "mingw" on # Windows immediately, we will default to MSVC if available. env = createEnvironment( compiler_tools = compiler_tools ) # On Windows, in case MSVC was not found and not previously forced, retry with # it and use that instead then as a fallback. Using both tools in one call # seems to not work in terms of fallback when both exist. if win_target and \ compiler_tools == ["default"] and \ getExecutablePath(env["CC"], initial = False) is None: env = createEnvironment( compiler_tools = ["mingw"] ) if show_scons_mode: print("Initial CC:", env.get("CC", None)) print("Initial CCVERSION:", env.get("CCVERSION", None)) print("Initial CXX:", env.get("CXX", None)) print("Initial CXXVERSION:", env.get("CXXVERSION", None)) if "CC" in os.environ: # If the environment variable CXX is set, use that. env["CC"] = os.environ["CC"] env["CCVERSION"] = None elif clang_mode: # If requested by the user, use the clang compiler, overriding what was # said in environment. env["CC"] = "clang" env["CCVERSION"] = None # Requested or user provided, detect if it's clang. if "clang" in env["CC"]: clang_mode = True env["CCVERSION"] = None # To work around Windows not supporting command lines of greater than 10K by # default: def setupSpawn(env): def spawn(sh, escape, cmd, args, env): # The "del" appears to not work, but is used with large amounts of # files to link. So, lets do this ourselves, plus it avoids a process # spawn. if cmd == "del": assert len(args) == 2 os.unlink(args[1]) return 0 # For quoted arguments that end in a backslash, things don't work well # this is a workaround for it. def removeTrailingSlashQuote(arg): if arg.endswith(r'\"'): return arg[:-1] + '\\"' else: return arg newargs = ' '.join( removeTrailingSlashQuote(arg) for arg in args[1:] ) cmdline = cmd + ' ' + newargs # Remove useless quoted include directories to "windres", which does not # handle them properly in its command line parsing, while they are not # used at all. if cmd == "windres": cmdline = re.sub('--include-dir ".*?"', "", cmdline) proc = subprocess.Popen( cmdline, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False, env = env ) data, err = proc.communicate() rv = proc.wait() if cmd == "cl": data = data[data.find(b"\r\n") + 2 :] source_basenames = [ os.path.basename(source_file) for source_file in source_files ] def check(line): return line in (b"", b"Generating Code...") or \ line in source_basenames data = b"\r\n".join( line for line in data.split(b"\r\n") if not check(line) ) elif cmd == "rc": data = data[data.find(b"reserved.\r") + 13:] data = b'\n'.join( line for line in data.split(b'\n') if b"identifier truncated to" not in line ) elif cmd == "link" and module_mode: data = b"\r\n".join( line for line in data.split(b"\r\n") if b" Creating library" not in line # On localized compilers, the message to ignore is not as clear. if not (module_mode and b".exp" in line) ) if data.rstrip(): if not show_scons_mode: print(cmdline) print(data, end='') if err: print(err, end='') return rv env["SPAWN"] = spawn the_compiler = env["CC"] or env["CXX"] if the_compiler is None or \ getExecutablePath(the_compiler, initial = False) is None: if win_target: sys.exit("""\ Error, cannot locate suitable C compiler. You have the following options: a) If a suitable Visual Studio version is installed, it will not be located automatically, unless you install pywin32 for the Python installation below "%s". b) To make it find Visual Studio execute from Start Menu the 'Visual Studio Command Prompt' or "vcvarsall.bat". That will add Visual Studio to the PATH. And it will be detected. c) Install MinGW64 to "C:\\MinGW64" or "\\MinGW", where then it is automatically detected or add it to PATH before executing Nuitka. But be sure to pick the proper variant (32/64 bits, your Python arch is %r). d) Set the environment variable 'CC' to the *full* path of either "gcc.exe". Also be sure to head proper architecture or else errors will occur. """ % (sys.exec_prefix, target_arch)) else: sys.exit("Error, cannot locate suitable C compiler.") gcc_mode = "gcc" in the_compiler or \ "g++" in the_compiler or \ "clang" in the_compiler msvc_mode = win_target and not gcc_mode mingw_mode = win_target and gcc_mode # Do we use C11 for source files, or C++. if msvc_mode: c11_mode = False elif mingw_mode and found_gcc: c11_mode = True elif clang_mode: c11_mode = True elif gcc_mode and found_gcc: c11_mode = True else: c11_mode = False if msvc_mode: def getMsvcVersionString(): import SCons.Tool.MSCommon.vc # @UnresolvedImport return SCons.Tool.MSCommon.vc.get_default_version(env) def getMsvcVersion(): value = getMsvcVersionString() value = value.replace("exp","") return float(value) if show_scons_mode: print("Scons compiler: Using", end=' ') print(getExecutablePath(the_compiler, initial = False), end=' ') if win_target and msvc_mode: print("(MSVC %s)" % getMsvcVersionString()) print() if win_target: setupSpawn(env) env["BUILD_DIR"] = source_dir # Store the file signatures database with the rest of the source files sconsign_dir = os.path.abspath(os.path.join(source_dir, ".sconsign")) if not os.path.exists(sconsign_dir): os.makedirs(sconsign_dir) env.SConsignFile(sconsign_dir) if gcc_mode: # Support for gcc and clang, restricting visibility as much as possible. if not win_target: env.Append( CCFLAGS = ["-fvisibility=hidden"] ) env.Append( CXXFLAGS = ["-fvisibility-inlines-hidden"] ) # Make it clear that we want C11. if c11_mode: env.Append( CCFLAGS = ["-std=c11"] ) # Support for clang. if "clang" in the_compiler: env.Append( CCFLAGS = ["-w"] ) env.Append( CPPDEFINES = ["_XOPEN_SOURCE"] ) env.Append( LINKFLAGS = ["-lstdc++" ] ) # Don't export anything by default, this should create smaller executables. env.Append( CCFLAGS = [ "-fvisibility=hidden", "-fvisibility-inlines-hidden" ] ) if debug_mode: env.Append( CCFLAGS = ["-Wunused-but-set-variable"] ) # Support details for real g++, not clang++. if gcc_mode and "clang" not in the_compiler: # Don't export anything by default, this should create smaller executables. # Enforce the minimum version, selecting a potentially existing g++-4.5 # binary if it's not high enough. This is esp. useful under Debian which # allows all compiler to exist next to each other and where g++ might not be # good enough, but g++-4.5 would be. if gcc_version < "4.4": sys.exit( """\ The gcc compiler %s (version %s) doesn't have the sufficient \ version (>= 4.4).""" % (env["CXX"], gcc_version) ) # Older g++ complains about aliasing with Py_True and Py_False, but we don't # care. if gcc_version < "4.5": env.Append( CCFLAGS = ["-fno-strict-aliasing"] ) # For LTO mode, the version requirement is even higher, so try that too. if lto_mode and gcc_version < "4.6": sys.exit("""\ The gcc compiler %s (version %s) doesn't have the sufficient \ version for lto mode (>= 4.6).""" % (env[ "CXX" ], env[ "CXXVERSION" ])) # For g++ 4.6 there are some new interesting functions. if gcc_version >= "4.6": env.Append( CCFLAGS = ["-fpartial-inlining"] ) if debug_mode: env.Append( CCFLAGS = ["-Wunused-but-set-variable"] ) # Use link time optimizations so that gcc can help with optimization across # files, but unfortunately at this time it seriously slows down the compiled # code. This may be because it needs -O3 option to be effective. if lto_mode and gcc_version >= "4.6": env.Append(CCFLAGS = ["-flto"]) env.Append( LINKFLAGS = ["-flto=%d" % job_count] ) # env.Append( LINKFLAGS = [ "-Wsuggest-attribute=noreturn" ] ) # env.Append( LINKFLAGS = [ "-Wsuggest-attribute=pure" ] ) # env.Append( LINKFLAGS = [ "-Wsuggest-attribute=const" ] ) # env.Append( CCFLAGS = [ "-Wnoexcept" ] ) if debug_mode: env.Append(LINKFLAGS = ["-O2"]) env.Append( LINKFLAGS = [ "-O3", "-fpartial-inlining", "-freorder-functions", ] ) # Give a warning if LTO mode was specified, but won't be used. if lto_mode and gcc_version < "4.6": print("Warning, LTO mode specified, but not available.", file=sys.stderr) # The var-tracking does not scale, disable it. Should we really need it, we # can enable it. TODO: Does this cause a performance loss? env.Append(CCFLAGS = ["-fno-var-tracking"]) if msvc_mode: env.Append(CCFLAGS = ["/EHsc", "/J", "/Gd"]) env.Append(LINKFLAGS = ["/INCREMENTAL:NO"]) # Stack size 4MB or 8MB, we need more than the default 1MB. if target_arch == "x86_64": env.Append(CCFLAGS = ["/F8388608"]) else: env.Append(CCFLAGS = ["/F4194304"]) # The 32 bits MinGW does not default for API level properly, so help it. if mingw_mode: # Windows XP env.Append( CPPDEFINES = ["_WIN32_WINNT=0x0501"] ) if debug_mode: if gcc_mode: # Allow gcc/clang to point out all kinds of inconsistency to us by # raising an error. env.Append( CCFLAGS = [ "-Wall", "-Werror", # Unfortunately Py_INCREF(Py_False) triggers aliasing warnings, # which are unfounded, so disable them. "-Wno-error=strict-aliasing", "-Wno-strict-aliasing", # At least for self-compiled Python3.2, and MinGW this happens # and has little use anyway. "-Wno-error=format", "-Wno-format" ] ) elif msvc_mode: # Disable warnings that system headers already show. env.Append( CCFLAGS = [ "/W4", "/wd4505", "/wd4127", "/wd4100", "/wd4702", "/wd4189", "/wd4211", "/WX" ] ) # Disable warnings, that CPython headers already show. if python_version >= "3.4": env.Append( CCFLAGS = [ "/wd4512", "/wd4510", "/wd4610" ] ) if full_compat_mode: env.Append( CPPDEFINES = ["_NUITKA_FULL_COMPAT"] ) if experimental: env.Append( CPPDEFINES = [ "_NUITKA_EXPERIMENTAL_" + experiment.upper() for experiment in experimental if experiment ] ) if profile_mode: env.Append( CPPDEFINES = ["_NUITKA_PROFILE"] ) if trace_mode: env.Append( CPPDEFINES = ["_NUITKA_TRACE"] ) if standalone_mode: env.Append( CPPDEFINES = ["_NUITKA_STANDALONE"] ) if "linux" in sys.platform: env.Append( LIBS = ["dl"] ) if no_python_warnings: env.Append( CPPDEFINES = ["_NUITKA_NO_PYTHON_WARNINGS"] ) if python_version < "3": env.Append( CPPDEFINES = ["_NUITKA_SYSFLAG_PY3K_WARNING=%d" % (1 if python_sysflag_py3k_warning else 0)] ) env.Append( CPPDEFINES = ["_NUITKA_SYSFLAG_DIVISION_WARNING=%d" % (1 if python_sysflag_division_warning else 0)] ) env.Append( CPPDEFINES = ["_NUITKA_SYSFLAG_UNICODE=%d" % (1 if python_sysflag_unicode else 0)] ) env.Append( CPPDEFINES = ["_NUITKA_SYSFLAG_BYTES_WARNING=%d" % (1 if python_sysflag_bytes_warning else 0)] ) env.Append( CPPDEFINES = ["_NUITKA_SYSFLAG_NO_SITE=%d" % (1 if python_sysflag_no_site else 0)] ) env.Append( CPPDEFINES = ["_NUITKA_SYSFLAG_VERBOSE=%d" % (1 if python_sysflag_verbose else 0)] ) if win_target: # For MinGW and cross compilation, we need to tell the subsystem # to target as well as to automatically import everything used. if gcc_mode: env.Append( LINKFLAGS = ["-Wl,--enable-auto-import"] ) if win_disable_console: env.Append( LINKFLAGS = ["-Wl,--subsystem,windows"] ) if win_disable_console: env.Append( CPPDEFINES = ["_NUITKA_WINMAIN_ENTRY_POINT"] ) # This is used for "PathRemoveFileSpec" used in Windows code. env.Append( LIBS = ["Shlwapi"] ) # This is used for "GetCommandLineW" used in Windows code for Python3. if python_version >= "3": env.Append( LIBS = ["Shell32"] ) if python_debug: env.Append( CPPDEFINES = ["Py_DEBUG"] ) def detectHostMultiarch(): pipe = SCons.Action._subproc( env, "dpkg-architecture", stdin = 'devnull', stderr = 'devnull', stdout = subprocess.PIPE ) for line in pipe.stdout: line = line.strip() if line.startswith("DEB_HOST_MULTIARCH="): return line.split('=', 1)[1] else: return None if gcc_mode and "linux" in sys.platform: if python_version.startswith("3.3"): host_multiarch = detectHostMultiarch() if host_multiarch is not None: env.Append( CCFLAGS = [ "-I" + os.path.join( "/usr/include/", host_multiarch, "python" + python_abi_version ) ] ) # For Nuitka, it generally is OK to break out of the virtualenv, and use the # original install. Mind you, this is not about executing anything, this is # about building, and finding the headers to compile against that Python, we # do not care about any site packages, and so on. # Some virtualenv, at least on MacOS, have such handy links, pointing to the # Python installation. if not win_target and os.path.islink(os.path.join(python_prefix, ".Python")): python_prefix = os.path.normpath( os.path.join( os.readlink(os.path.join(python_prefix, ".Python")), # @UndefinedVariable ".." ) ) # Some virtualenv created by "venv" seem to have a different structure, where # library and include files are outside of it. if not win_target and \ python_version >= "3.3" and \ os.path.exists(os.path.join(python_prefix, "bin/activate")): python_binary = os.path.join(python_prefix, "bin", "python") python_binary = os.path.realpath(python_binary) python_prefix = os.path.normpath( os.path.join( python_binary, "..", ".." ) ) # Some virtualenv contain the "orig-prefix.txt" as a textual link to # the target, this is Windows and standard virtualenv. There are two # places to look for. for candidate in ("Lib/orig-prefix.txt", "lib/python%s/orig-prefix.txt" % python_version): candidate = os.path.join(python_prefix, candidate) if os.path.exists(candidate): python_prefix = open(candidate).read() # Trailing spaces in the python prefix, please not. assert python_prefix == python_prefix.strip() break if win_target: # On Windows, the installation layout is relatively fixed. python_header_path = os.path.join(python_prefix, "include") else: # The python header path is a combination of python version and debug # indication, make sure the headers are found by adding it to the C # include path. python_header_path = os.path.join( python_prefix, "include", "python" + python_abi_version ) if not os.path.exists(os.path.join(python_header_path, "Python.h")): # Not, for --python-debug other headers are used than for normal # compilation. sys.exit( """\ Error, no 'Python.h' %s headers can be found at '%s', dependency \ not satisfied!""" % ( "debug" if python_debug else "development", python_header_path ) ) env.Append(CPPPATH = [python_header_path]) # To support self built Python on Windows, need to also add the "PC" directory, # that a normal install won't have. env.Append(CPPPATH = [os.path.join(python_prefix, "PC")]) def getWindowsPythonDLLPath(): return os.environ["NUITKA_PYTHON_DLL_PATH"] def makeWindowsMingGW64Lib(win_lib_path): python_dll_filename = getWindowsPythonDLLPath() checksum = hashlib.md5(open(python_dll_filename, "rb").read()).hexdigest() # Create the LIB in the build directory. new_win_lib_path = os.path.join( nuitka_cache, "win_libs", python_version, checksum ) if not os.path.exists(new_win_lib_path): os.makedirs(new_win_lib_path) lib_filename = os.path.join( new_win_lib_path, "libpython%s.a" % python_abi_version.replace('.', "") ) if os.path.exists(lib_filename): return new_win_lib_path gendef_tool = os.path.join( os.path.dirname( getExecutablePath(env["CXX"], initial = False) ), "gendef" ) command = [gendef_tool, "-", python_dll_filename] if show_scons_mode: print("Scons: Creating Python library defs with:", command) try: proc = subprocess.Popen( command, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) except WindowsError: sys.exit('Error, cannot find "gendef.exe" which is required though.') defs, stderr = proc.communicate() rv = proc.wait() if rv != 0: sys.exit('Error, unexpected error from "gendef.exe" %s.' % stderr) if str is not bytes: defs = defs.decode("utf8") defs_filename = os.path.join( new_win_lib_path, "python%s.defs" % python_abi_version.replace('.', "") ) defs_file = open( defs_filename, 'w' ) defs_file.write(defs) defs_file.close() shutil.copy( getWindowsPythonDLLPath(), new_win_lib_path ) dll_tool = os.path.join( os.path.dirname( getExecutablePath(env["CXX"], initial = False) ), "dlltool" ) result = subprocess.call( ( dll_tool, "-d", defs_filename, "-l", lib_filename ), stdout = subprocess.PIPE ) if result != 0: sys.exit("Error, call to 'dlltool.exe' failed.") return new_win_lib_path if win_target: # MinGW for 64 bits needs this due to CPython being compiled with MSVC. if python_debug: win_lib_name ="python" + python_abi_version.replace('.', "") + "_d" else: win_lib_name = "python" + python_abi_version.replace('.', "") for candidate in ("libs", "PCBuild"): win_lib_path = os.path.join(python_prefix, candidate) if os.path.exists(os.path.join(win_lib_path, win_lib_name + ".lib")): break else: sys.exit("Error, cannot find '%s.lib' file." % win_lib_name) if target_arch == "x86_64" and gcc_mode: win_lib_path = makeWindowsMingGW64Lib(win_lib_path) elif target_arch == "x86" and python_debug and gcc_mode: win_lib_path = makeWindowsMingGW64Lib(win_lib_path) env.Append(LIBPATH = [win_lib_path]) env.Append(LIBS = [win_lib_name]) else: # Debian and Ubuntu distinguish the system libraries like this. if python_debug and \ python_prefix == "/usr" and \ not python_version.startswith('3') and \ platform.dist()[0].lower() in ("debian", "ubuntu"): env.Append(LIBS = ["python" + python_abi_version + "_d"]) else: env.Append(LIBS = ["python" + python_abi_version]) if python_prefix != "/usr" and "linux" in sys.platform: env.Append( LIBS = ["dl", "pthread", "util", 'm'] ) if gcc_mode and not clang_mode: env.Append( LINKFLAGS = ["-export-dynamic"] ) # Add the python library path to the library path python_lib_path = os.path.join(python_prefix, "lib") env.Append(LIBPATH = [python_lib_path]) # For NetBSD the rpath is required, on FreeBSD it's warned as unused. if "netbsd" in sys.platform: env.Append( LINKFLAGS = ["-rpath=" + python_lib_path] ) # The static include files reside in Nuitka installation, which may be where # the "nuitka.build" package lives. nuitka_include = os.path.join( nuitka_src, "include" ) # We have include files in the build directory and the static include directory # that is located inside Nuitka installation. env.Append( CPPPATH = [ source_dir, nuitka_include, os.path.join(nuitka_src, "static_src") ] ) if debug_mode or unstripped_mode: # Use debug format, so we get good tracebacks from it. if gcc_mode: env.Append( CCFLAGS = ["-g"] ) env.Append( ASFLAGS = ["-g"] ) if "gcc" in the_compiler or "g++" in the_compiler: env.Append( CCFLAGS = ["-feliminate-unused-debug-types"] ) elif msvc_mode: env.Append( CCFLAGS = ["/Z7"] ) # Higher MSVC versions need this for parallel compilation if job_count > 1 and getMsvcVersion() >= 11: env.Append( CCFLAGS = ["/FS"] ) env.Append( LINKFLAGS = ["/DEBUG"] ) else: if gcc_mode: if sys.platform == "darwin": env.Append( LINKFLAGS = ["-Wno-deprecated-declarations"] ) else: env.Append( LINKFLAGS = ["-s"] ) # When debugging, optimize less than when optimizing, when not remove # assertions. if debug_mode: if gcc_mode or msvc_mode: env.Append( CCFLAGS = ["-O2"] ) else: if gcc_mode: env.Append( CCFLAGS = ["-O3"] ) # Check inlining of calls, except in debug mode, where it will all be # wrong due to additional code. # if not debug_mode: # env.Append( CCFLAGS = [ "-Winline" ] ) elif msvc_mode: env.Append( CCFLAGS = ["/Ox"] ) env.Append( CPPDEFINES = ["__NUITKA_NO_ASSERT__"] ) # MinGW for 64 bits needs this due to CPython bugs. if win_target and target_arch == "x86_64" and gcc_mode: env.Append( CPPDEFINES = ["MS_WIN64"] ) # Set load libpython from binary directory default if gcc_mode and sys.platform != "darwin" and not win_target: if standalone_mode: rpath = "$$ORIGIN" else: rpath = python_lib_path env.Append( LINKFLAGS = ["-Wl,-R,'%s'" % rpath] ) # The rpath is no longer used unless we do this on modern Linux. The # option name is not very revealing, but basically without this, the # rpath in the binary will be ignored by the loader. if "linux" in sys.platform: env.Append( LINKFLAGS = ["-Wl,--disable-new-dtags"] ) def getLinkerArch(): if "linux" in sys.platform: python_exe = os.environ["NUITKA_PYTHON_EXE_PATH"] assert os.path.exists(python_exe), python_exe command = ["objdump", "-f", python_exe] if show_scons_mode: print("Scons: Detecting Linux target arch with :", command) try: proc = subprocess.Popen( command, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) except OSError: return None data, _err = proc.communicate() rv = proc.wait() if rv != 0: return None if str is not bytes: data = data.decode("utf8") for line in data.splitlines(): if " file format " in line: return line.split(" file format ")[-1] else: # TODO: Missing for MacOS, FreeBSD, other Linux return None constants_bin_filename = os.path.join(source_dir,"__constants.bin") if win_target and not module_mode: # On Windows constants are accesses as a resource, except in shared # libraries, where that option is not available. constants_generated_filename = None env.Append( CPPDEFINES = ["_NUITKA_CONSTANTS_FROM_RESOURCE"] ) elif gcc_mode and getLinkerArch() is not None: env.Append( LINKFLAGS = [ "-Wl,-b", "-Wl,binary", "-Wl,%s" % constants_bin_filename, "-Wl,-b", "-Wl,%s" % getLinkerArch(), "-Wl,-defsym", "-Wl,%sconstant_bin=_binary_%s___constants_bin_start" % ( '_' if mingw_mode else "", "".join(re.sub("[^a-zA-Z0-9_]",'_',c) for c in source_dir) ) ] ) constants_generated_filename = None else: constants_generated_filename = os.path.join( source_dir, "__constants_data.c" ) with open(constants_generated_filename, 'w') as output: if not c11_mode: output.write('extern "C" ') output.write("const unsigned char constant_bin[] =\n{\n") for count, stream_byte in enumerate( open(constants_bin_filename, "rb").read() ): if count % 16 == 0: if count > 0: output.write('\n') output.write(" ") if str is bytes: stream_byte = ord(stream_byte) output.write(" 0x%02x," % stream_byte) output.write("\n};\n"); env.Append( CPPDEFINES = [ "_NUITKA_FROZEN=%d" % frozen_modules, "_NUITKA_MODULE_COUNT=%d" % module_count ] ) # Tell compiler to create a shared library or program. if module_mode: if "gcc" in the_compiler or "g++" in the_compiler: env.Append( CCFLAGS = ["-shared"] ) elif clang_mode: pass elif msvc_mode: env.Append( CCFLAGS = ["/LD"] ) else: assert False else: if msvc_mode: env.Append( CCFLAGS = ["/MT"] ) if module_mode: env.Append( CPPDEFINES = ["_NUITKA_MODULE"] ) else: env.Append( CPPDEFINES = ["_NUITKA_EXE"] ) def discoverSourceFiles(): result = [] # If we use C11 capable compiler, all good. Otherwise use C++, which Scons # needs to derive from filenames, so make copies (or links) with a different # name. def cheap_copy(src, dst): if win_target: # Windows has symlinks these days, but they do not integrate well # with Python2 at least.So make a copy in any case. if os.path.exists(dst): os.unlink(dst) shutil.copy(src, dst) else: # Relative paths work badly for links. Creating them relative is # not worth the effort. src = os.path.abspath(src) try: link_target = os.readlink(dst) # If it's already a proper link, do nothing then. if link_target == src: return os.unlink(dst) except OSError as e: # Broken links work like that, remove them, so we can replace # them. try: os.unlink(dst) except OSError: pass try: os.symlink(src, dst) except OSError: shutil.copy(src, dst) def provideStatic(sub_path): source_file = os.path.join(nuitka_src, "static_src", sub_path) target_file = os.path.join(source_dir, os.path.basename(sub_path)) if target_file.endswith(".c") and not c11_mode: target_file += "pp" cheap_copy(source_file, target_file) return target_file # Scan for Nuitka created source files, and add them too. for filename in os.listdir(source_dir): # Only C files are of interest here. if not filename.endswith((".c", "cpp")) or \ not filename.startswith(("module.", "__")): continue filename = os.path.join(source_dir, filename) target_file = filename # We pretend to use C++ if no C11 compiler is present. if c11_mode: result.append(filename) else: if filename.endswith(".c"): target_file += "pp" # .cpp" suffix then os.rename(filename, target_file) result.append(target_file) # Main program, unless of course it's a Python module/package we build. if not module_mode: result.append(provideStatic("MainProgram.c")) # Compiled types. result.append(provideStatic("CompiledCellType.c")) result.append(provideStatic("CompiledFunctionType.c")) result.append(provideStatic("CompiledMethodType.c")) result.append(provideStatic("CompiledGeneratorType.c")) if python_version >= "3.5": result.append(provideStatic("CompiledCoroutineType.c")) if python_version >= "3.6": result.append(provideStatic("CompiledAsyncgenType.c")) result.append(provideStatic("CompiledFrameType.c")) # Helper codes. result.append(provideStatic("CompiledCodeHelpers.c")) result.append(provideStatic("InspectPatcher.c")) result.append(provideStatic("MetaPathBasedLoader.c")) # Platform dependent fiber implementations for generators to use. if win_target: result.append(provideStatic("win32_ucontext_src/fibers_win32.c")) elif target_arch == "x86_64" and "linux" in sys.platform: result.append(provideStatic("x64_ucontext_src/fibers_x64.c")) result.append(provideStatic("x64_ucontext_src/swapfiber.S")) elif target_arch == "armv5tel": result.append(provideStatic("arm_ucontext_src/fibers_arm.c")) result.append(provideStatic("arm_ucontext_src/ucontext.c")) result.append(provideStatic("arm_ucontext_src/getcontext.asm")) elif "openbsd" in sys.platform: result.append(provideStatic("libcoro_ucontext_src/fibers_coro.c")) result.append(provideStatic("libcoro_ucontext_src/coro.c")) env.Append( CPPDEFINES = ["CORO_SJLJ"] ) elif os.path.isfile("/etc/alpine-release"): result.append(provideStatic("libcoro_ucontext_src/fibers_coro.c")) result.append(provideStatic("libcoro_ucontext_src/coro.c")) env.Append( CPPDEFINES = ["CORO_SJLJ", "__OpenBSD__"] ) else: # Variant based on deprecated, but still present versions of # getcontext/setcontext/swapcontext/makecontext result.append(provideStatic("gen_ucontext_src/fibers_gen.c")) return result source_targets = [] # Prepare the use of a custom specs file for windows targets. We change the used # specs for linking to avoid the use of the wrong (for CPython) run time # library. if win_target: rc_content = [] # Which files to depend on. rc_file_dependencies = [] if not module_mode: rc_content.append( '3 RCDATA "%s"' % constants_bin_filename.replace('\\', '/') ) rc_file_dependencies.append(constants_bin_filename) if python_version < "3.3": manifest_filename = os.path.join( source_dir, "resources.manifest" ) python_exe = os.environ["NUITKA_PYTHON_EXE_PATH"] assert os.path.exists(python_exe), python_exe if msvc_mode: try: result = subprocess.call( ( env["MT"], "-inputresource:%s;#1" % python_exe, "-out:%s" % manifest_filename ), stdout = subprocess.PIPE ) except OSError: result = -1 if result != 0: sys.exit( """\ Error, call to 'mt.exe' (%s) to extract manifest failed (error code %d.)""" % ( env["MT"], result ) ) else: command = ( "windres.exe", "--input", python_exe, "--output", manifest_filename, "--output-format", "rc" ) try: proc = subprocess.Popen( command, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = False ) except WindowsError: sys.exit("Error, cannot find 'windres.exe' but it is required.") _stdout, stderr = proc.communicate() result = proc.wait() if result != 0: sys.exit("""\ Error, call to 'windres.exe' to extract manifest failed with the following error: %s In case of "File format not recognized" you are most likely using the 64 bits version of Python and the 32 bits version of MinGW64. There is also a 64 bits version, download and use that one instead. """ % stderr ) # Need to reduce to what we use, and remove the quoting of the # actual manifest. We created a full "rc" file, but that we do # not have use for. manifest_file = open(manifest_filename, 'r') inside = False contents = [] for line in manifest_file: if "RT_MANIFEST" in line: inside = True continue if not inside: continue line = line.replace(r"\r\n", "\r\n") line = line.replace('""', '"') contents.append( line[line.find('"')+1:line.rfind('"')] ) manifest_file = open(manifest_filename, 'w') manifest_file.write("".join(contents[1:-1])) manifest_file.close() rc_content.append( "1 RT_MANIFEST resources.manifest" ) rc_file_dependencies.append(manifest_filename) if icon_path: rc_content.append( '2 ICON MOVEABLE PURE LOADONCALL DISCARDABLE "%s"' % ( icon_path.replace('\\', '/') ) ) rc_file_dependencies.append(icon_path) if rc_content: rc_filename = os.path.join(source_dir, "resources.rc") rc_file = open(rc_filename, 'w') rc_content.insert( 0, '#include "winuser.h"' ) rc_file.write('\n'.join(rc_content)) rc_file.close() res_target = env.RES(rc_filename) for rc_file_dependency in rc_file_dependencies: Depends(res_target, rc_file_dependency) # @UndefinedVariable source_targets.append( res_target ) source_files = discoverSourceFiles() if module_mode: # For Python modules, the standard shared library extension is not what # gets used. def getSharedLibrarySuffix(): import imp result = None for suffix, _mode, module_type in imp.get_suffixes(): if module_type != imp.C_EXTENSION: continue if result is None or len(suffix) < len(result): result = suffix return result env["SHLIBSUFFIX"] = getSharedLibrarySuffix() target = env.SharedLibrary( result_basepath, source_files + source_targets ) else: target = env.Program( result_basepath + ".exe", source_files + source_targets ) # Avoid dependency on MinGW libraries. if win_target and gcc_mode: env.Append( LINKFLAGS = [ "-static-libgcc", "-static-libstdc++" ] ) # On some architectures, makecontext cannot pass pointers reliably. if target_arch == "x86_64" and "linux" in sys.platform: env.Append(CPPDEFINES = ["_NUITKA_MAKECONTEXT_INTS"]) # Avoid IO for compilation as much as possible, this should make the # compilation more memory hungry, but also faster. if gcc_mode: env.Append(CCFLAGS = "-pipe") if "CPPFLAGS" in os.environ: env.Append(CCFLAGS = os.environ["CPPFLAGS"].split()) if "CCFLAGS" in os.environ: env.Append(CCFLAGS = os.environ["CCFLAGS"].split()) if "CXXFLAGS" in os.environ: env.Append(CCFLAGS = os.environ["CXXFLAGS"].split()) if "LDFLAGS" in os.environ: env.Append(LINKFLAGS = os.environ["LDFLAGS"].split()) # Remove the target file to avoid cases where it falsely doesn't get rebuild # and then lingers from previous builds, if os.path.exists(target[0].abspath): os.unlink(target[0].abspath) if show_scons_mode: print("scons: Told to run compilation on %d CPUs." % job_count) # Cached, when done, by the fastest possible algorithm and right inside the # build directory. Makes no sense of course, if that is removed later on by # Nuitka. if cache_mode: CacheDir(os.path.join(source_dir, "cache-" + target_arch + '-' + python_abi_version)) # @UndefinedVariable Decider("MD5-timestamp") # @UndefinedVariable # Before we go, also lets turn KeyboardInterrupt into a mere error exit. def signalHandler(signal, frame): sys.exit(2) signal.signal(signal.SIGINT, signalHandler) build_definitions = {} if uninstalled_python: if win_target: build_definitions["DLL_EXTRA_PATH"] = os.path.dirname(getWindowsPythonDLLPath()) else: build_definitions["PYTHON_HOME_PATH"] = python_prefix def makeCLiteral(value): value = value.replace("\\", r"\\") value = value.replace('"', r'\"') return '"' + value + '"' def createBuildDefinitionsFile(): build_definitions_filename = os.path.join(source_dir, "build_definitions.h") build_definitions_file = open(build_definitions_filename, 'w') for key, value in sorted(build_definitions.items()): build_definitions_file.write( "#define %s %s\n" % ( key, makeCLiteral(value) ) ) build_definitions_file.close() createBuildDefinitionsFile() # env["CFLAGS"] = env["CCFLAGS"] Default(target) # @UndefinedVariable # Copy the Python DLL to the target directory, so the executable can be run. if uninstalled_python: if win_target: # TODO: This should have become unnecessary by DLL_EXTRA_PATH shutil.copy( getWindowsPythonDLLPath(), os.path.dirname(result_basepath) or "." ) elif sys.platform == "darwin": # TODO: Move this to where we do it for Linux already. python_dll_filename = "libpython" + python_abi_version + ".dylib" result = subprocess.call( ( "install_name_tool", "--change", python_dll_filename, os.path.join(python_lib_path, python_dll_filename), result_basepath + ".exe" ), stdout = subprocess.PIPE ) if result != 0: sys.exit("Error, call to 'install_name_tool' to fix Python library path failed.") Nuitka-0.5.28.2/nuitka/build/static_src/0000755000372000001440000000000013207540420020210 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/static_src/CompiledFrameType.c0000644000372000001440000005161313207537242023743 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/freelists.h" #include "structmember.h" #define OFF( x ) offsetof( PyFrameObject, x ) static PyMemberDef Nuitka_Frame_memberlist[] = { { (char *)"f_back", T_OBJECT, OFF( f_back ), READONLY | RESTRICTED }, { (char *)"f_code", T_OBJECT, OFF( f_code ), READONLY | RESTRICTED }, { (char *)"f_builtins", T_OBJECT, OFF( f_builtins ), READONLY | RESTRICTED }, { (char *)"f_globals", T_OBJECT, OFF( f_globals ), READONLY | RESTRICTED }, { (char *)"f_lasti", T_INT, OFF( f_lasti ), READONLY | RESTRICTED }, { NULL } }; #if PYTHON_VERSION < 300 static PyObject *Nuitka_Frame_get_exc_traceback( struct Nuitka_FrameObject *frame ) { PyObject *result = frame->m_frame.f_exc_traceback; if ( result == NULL ) { result = Py_None; } Py_INCREF( result ); return result; } static int Nuitka_Frame_set_exc_traceback( struct Nuitka_FrameObject *frame, PyObject *traceback ) { Py_XDECREF( frame->m_frame.f_exc_traceback ); if ( traceback == Py_None ) { traceback = NULL; } frame->m_frame.f_exc_traceback = traceback; Py_XINCREF( traceback ); return 0; } static PyObject *Nuitka_Frame_get_exc_type( struct Nuitka_FrameObject *frame ) { PyObject *result; if ( frame->m_frame.f_exc_type != NULL ) { result = frame->m_frame.f_exc_type; } else { result = Py_None; } Py_INCREF( result ); return result; } static int Nuitka_Frame_set_exc_type( struct Nuitka_FrameObject *frame, PyObject *exception_type ) { PyObject *old = frame->m_frame.f_exc_type; if ( exception_type == Py_None ) { exception_type = NULL; } frame->m_frame.f_exc_type = exception_type; Py_XINCREF( frame->m_frame.f_exc_type ); Py_XDECREF( old ); return 0; } static PyObject *Nuitka_Frame_get_exc_value( struct Nuitka_FrameObject *frame ) { PyObject *result; if ( frame->m_frame.f_exc_value != NULL ) { result = frame->m_frame.f_exc_value; } else { result = Py_None; } Py_INCREF( result ); return result; } static int Nuitka_Frame_set_exc_value( struct Nuitka_FrameObject *frame, PyObject *exception_value ) { PyObject *old = frame->m_frame.f_exc_value; if ( exception_value == Py_None ) { exception_value = NULL; } frame->m_frame.f_exc_value = exception_value; Py_XINCREF( exception_value ); Py_XDECREF( old ); return 0; } static PyObject *Nuitka_Frame_get_restricted( struct Nuitka_FrameObject *frame, void *closure ) { Py_INCREF( Py_False ); return Py_False; } #endif static PyObject *Nuitka_Frame_getlocals( struct Nuitka_FrameObject *frame, void *closure ) { if ( frame->m_type_description == NULL ) { if ( frame->m_frame.f_locals == NULL ) { frame->m_frame.f_locals = PyDict_New(); } Py_INCREF( frame->m_frame.f_locals ); return frame->m_frame.f_locals; } else { PyObject *result = PyDict_New(); PyObject **varnames = &PyTuple_GET_ITEM( frame->m_frame.f_code->co_varnames, 0 ); char const *w = frame->m_type_description; char const *t = frame->m_locals_storage; while ( *w != 0 ) { switch( *w ) { case NUITKA_TYPE_DESCRIPTION_OBJECT: case NUITKA_TYPE_DESCRIPTION_OBJECT_PTR: { PyObject *value = *(PyObject **)t; PyDict_SetItem( result, *varnames, value ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_CELL: { struct Nuitka_CellObject *value = *(struct Nuitka_CellObject **)t; PyDict_SetItem( result, *varnames, value->ob_ref ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_NULL: { t += sizeof(void *); break; } case NUITKA_TYPE_DESCRIPTION_BOOL: { int value = *(int *)t; t += sizeof(int); switch ((nuitka_bool)value) { case NUITKA_BOOL_TRUE: { PyDict_SetItem( result, *varnames, Py_True); break; } case NUITKA_BOOL_FALSE: { PyDict_SetItem( result, *varnames, Py_False); break; } default: break; } break; } default: assert(false); } w += 1; varnames += 1; } return result; } } static PyObject *Nuitka_Frame_getlineno( PyFrameObject *frame, void *closure ) { return PyInt_FromLong( frame->f_lineno ); } static PyObject *Nuitka_Frame_gettrace( PyFrameObject *frame, void *closure ) { PyObject *result = frame->f_trace; Py_INCREF( result ); return result; } static int Nuitka_Frame_settrace( PyFrameObject *frame, PyObject* v, void *closure ) { PyErr_Format( PyExc_RuntimeError, "f_trace is not writable in Nuitka" ); return -1; } static PyGetSetDef Nuitka_Frame_getsetlist[] = { { (char *)"f_locals", (getter)Nuitka_Frame_getlocals, NULL, NULL }, { (char *)"f_lineno", (getter)Nuitka_Frame_getlineno, NULL, NULL }, { (char *)"f_trace", (getter)Nuitka_Frame_gettrace, (setter)Nuitka_Frame_settrace, NULL }, #if PYTHON_VERSION < 300 { (char *)"f_restricted", (getter)Nuitka_Frame_get_restricted, NULL, NULL }, { (char *)"f_exc_traceback", (getter)Nuitka_Frame_get_exc_traceback, (setter)Nuitka_Frame_set_exc_traceback, NULL }, { (char *)"f_exc_type", (getter)Nuitka_Frame_get_exc_type, (setter)Nuitka_Frame_set_exc_type, NULL }, { (char *)"f_exc_value", (getter)Nuitka_Frame_get_exc_value, (setter)Nuitka_Frame_set_exc_value, NULL }, #endif { NULL } }; // tp_repr slot, decide how a function shall be output static PyObject *Nuitka_Frame_tp_repr( struct Nuitka_FrameObject *nuitka_frame ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( #else return PyUnicode_FromFormat( #endif #if _DEBUG_FRAME || _DEBUG_REFRAME || _DEBUG_EXCEPTIONS "", Nuitka_String_AsString( nuitka_frame->m_frame.f_code->co_name ), nuitka_frame #else "", nuitka_frame #endif ); } static void Nuitka_Frame_tp_clear( struct Nuitka_FrameObject *frame ) { if ( frame->m_type_description ) { char const *w = frame->m_type_description; char const *t = frame->m_locals_storage; while ( *w != 0 ) { switch( *w ) { case NUITKA_TYPE_DESCRIPTION_OBJECT: case NUITKA_TYPE_DESCRIPTION_OBJECT_PTR: { PyObject *value = *(PyObject **)t; Py_XDECREF( value ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_CELL: { struct Nuitka_CellObject *value = *(struct Nuitka_CellObject **)t; Py_DECREF( value ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_NULL: { t += sizeof(void *); break; } case NUITKA_TYPE_DESCRIPTION_BOOL: { t += sizeof(int); break; } default: assert(false); } w += 1; } frame->m_type_description = NULL; } } void Nuitka_Frame_ReleaseLocals( struct Nuitka_FrameObject *frame ) { Nuitka_Frame_tp_clear( frame ); } #define MAX_FRAME_FREE_LIST_COUNT 100 static struct Nuitka_FrameObject *free_list_frames = NULL; static int free_list_frames_count = 0; static void Nuitka_Frame_tp_dealloc( struct Nuitka_FrameObject *nuitka_frame ) { #ifndef __NUITKA_NO_ASSERT__ // Save the current exception, if any, we must to not corrupt it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); #endif Nuitka_GC_UnTrack( nuitka_frame ); PyFrameObject *frame = &nuitka_frame->m_frame; Py_XDECREF( frame->f_back ); Py_DECREF( frame->f_builtins ); Py_DECREF( frame->f_globals ); Py_XDECREF( frame->f_locals ); Py_XDECREF( frame->f_exc_type ); Py_XDECREF( frame->f_exc_value ); Py_XDECREF( frame->f_exc_traceback ); Nuitka_Frame_tp_clear( nuitka_frame ); releaseToFreeList( free_list_frames, nuitka_frame, MAX_FRAME_FREE_LIST_COUNT ); #ifndef __NUITKA_NO_ASSERT__ PyThreadState *tstate = PyThreadState_GET(); assert( tstate->curexc_type == save_exception_type ); assert( tstate->curexc_value == save_exception_value ); assert( (PyTracebackObject *)tstate->curexc_traceback == save_exception_tb ); #endif } static int Nuitka_Frame_tp_traverse( struct Nuitka_FrameObject *frame, visitproc visit, void *arg ) { Py_VISIT( frame->m_frame.f_back ); // Py_VISIT( frame->f_code ); Py_VISIT( frame->m_frame.f_builtins ); Py_VISIT( frame->m_frame.f_globals ); // Py_VISIT( frame->f_locals ); // TODO: Traverse attached locals too. Py_VISIT( frame->m_frame.f_exc_type ); Py_VISIT( frame->m_frame.f_exc_value ); Py_VISIT( frame->m_frame.f_exc_traceback ); return 0; } #if PYTHON_VERSION >= 340 extern PyObject *Nuitka_Generator_close( struct Nuitka_GeneratorObject *generator, PyObject *args ); static PyObject *Nuitka_Frame_clear( struct Nuitka_FrameObject *frame ) { if ( frame->m_frame.f_executing ) { PyErr_Format( PyExc_RuntimeError, "cannot clear an executing frame" ); return NULL; } #if PYTHON_VERSION >= 340 // For frames that are closed, we also need to close the generator. if ( frame->m_frame.f_gen != NULL ) { Py_INCREF( frame ); assert( Nuitka_Generator_Check( frame->m_frame.f_gen ) ); struct Nuitka_GeneratorObject *generator = (struct Nuitka_GeneratorObject *)frame->m_frame.f_gen; frame->m_frame.f_gen = NULL; PyObject *close_result = Nuitka_Generator_close( generator, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)frame->m_frame.f_gen ); } else { Py_DECREF( close_result ); } Py_DECREF( frame ); } #endif Nuitka_Frame_tp_clear( frame ); Py_RETURN_NONE; } #endif static PyObject *Nuitka_Frame_sizeof( struct Nuitka_FrameObject *frame ) { return PyInt_FromSsize_t( sizeof( struct Nuitka_FrameObject ) + Py_SIZE( frame ) ); } static PyMethodDef Nuitka_Frame_methods[] = { #if PYTHON_VERSION >= 340 { "clear", (PyCFunction)Nuitka_Frame_clear, METH_NOARGS, "F.clear(): clear most references held by the frame" }, #endif { "__sizeof__", (PyCFunction)Nuitka_Frame_sizeof, METH_NOARGS, "F.__sizeof__() -> size of F in memory, in bytes" }, { NULL, NULL } }; PyTypeObject Nuitka_Frame_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_frame", sizeof(struct Nuitka_FrameObject), 1, (destructor)Nuitka_Frame_tp_dealloc, // tp_dealloc 0, // tp_print 0, // tp_getattr 0, // tp_setattr 0, // tp_compare (reprfunc)Nuitka_Frame_tp_repr, // tp_repr 0, // tp_as_number 0, // tp_as_sequence 0, // tp_as_mapping 0, // tp_hash 0, // tp_call 0, // tp_str PyObject_GenericGetAttr, // tp_getattro PyObject_GenericSetAttr, // tp_setattro 0, // tp_as_buffer Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, // tp_flags 0, // tp_doc (traverseproc)Nuitka_Frame_tp_traverse, // tp_traverse (inquiry)Nuitka_Frame_tp_clear, // tp_clear 0, // tp_richcompare 0, // tp_weaklistoffset 0, // tp_iter 0, // tp_iternext Nuitka_Frame_methods, // tp_methods Nuitka_Frame_memberlist, // tp_members Nuitka_Frame_getsetlist, // tp_getset 0, // tp_base 0, // tp_dict }; void _initCompiledFrameType( void ) { PyType_Ready( &Nuitka_Frame_Type ); // These are to be used interchangably. Make sure that's true. assert( offsetof(struct Nuitka_FrameObject, m_frame.f_exc_type) == offsetof(PyFrameObject, f_exc_type) ); } extern PyObject *const_str_plain___module__; static struct Nuitka_FrameObject *MAKE_FRAME( PyCodeObject *code, PyObject *module, bool is_module, Py_ssize_t locals_size ) { assertCodeObject( code ); PyObject *globals = ((PyModuleObject *)module)->md_dict; assert( PyDict_Check( globals ) ); struct Nuitka_FrameObject *result; // Macro to assign result memory from GC or free list. allocateFromFreeList( free_list_frames, struct Nuitka_FrameObject, Nuitka_Frame_Type, locals_size ); result->m_type_description = NULL; PyFrameObject *frame = &result->m_frame; frame->f_code = code; frame->f_trace = Py_None; frame->f_exc_type = NULL; frame->f_exc_value = NULL; frame->f_exc_traceback = NULL; Py_INCREF( dict_builtin ); frame->f_builtins = (PyObject *)dict_builtin; frame->f_back = NULL; Py_INCREF( globals ); frame->f_globals = globals; if (likely( (code->co_flags & CO_OPTIMIZED ) == CO_OPTIMIZED )) { frame->f_locals = NULL; } else if ( is_module ) { Py_INCREF( globals ); frame->f_locals = globals; } else { frame->f_locals = PyDict_New(); if (unlikely( frame->f_locals == NULL )) { Py_DECREF( result ); return NULL; } PyDict_SetItem( frame->f_locals, const_str_plain___module__, MODULE_NAME( module ) ); } #if PYTHON_VERSION < 340 frame->f_tstate = PyThreadState_GET(); #endif frame->f_lasti = -1; frame->f_lineno = code->co_firstlineno; frame->f_iblock = 0; #if PYTHON_VERSION >= 340 frame->f_gen = NULL; frame->f_executing = 0; #endif Nuitka_GC_Track( result ); return result; } struct Nuitka_FrameObject *MAKE_MODULE_FRAME( PyCodeObject *code, PyObject *module ) { return MAKE_FRAME( code, module, true, 0 ); } struct Nuitka_FrameObject *MAKE_FUNCTION_FRAME( PyCodeObject *code, PyObject *module, Py_ssize_t locals_size ) { return MAKE_FRAME( code, module, false, locals_size ); } extern PyObject *const_str_empty; extern PyObject *const_bytes_empty; #if PYTHON_VERSION < 300 PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int flags ) #else PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int kw_only_count, int flags ) #endif { CHECK_OBJECT( filename ); assert( Nuitka_String_Check( filename ) ); CHECK_OBJECT( function_name ); assert( Nuitka_String_Check( function_name ) ); CHECK_OBJECT( argnames ); assert( PyTuple_Check( argnames ) ); // The PyCode_New has funny code that interns, mutating the tuple that owns // it. Really serious non-immutable shit. We have triggered that changes // behind our back in the past. #ifndef __NUITKA_NO_ASSERT__ Py_hash_t hash = DEEP_HASH( argnames ); #endif // TODO: Consider using PyCode_NewEmpty PyCodeObject *result = PyCode_New( arg_count, // argcount #if PYTHON_VERSION >= 300 kw_only_count, // kw-only count #endif 0, // nlocals 0, // stacksize flags, // flags #if PYTHON_VERSION < 300 const_str_empty, // code (bytecode) #else const_bytes_empty, // code (bytecode) #endif const_tuple_empty, // consts (we are not going to be compatible) const_tuple_empty, // names (we are not going to be compatible) argnames, // varnames (we are not going to be compatible) const_tuple_empty, // freevars (we are not going to be compatible) const_tuple_empty, // cellvars (we are not going to be compatible) filename, // filename function_name, // name line, // firstlineno (offset of the code object) #if PYTHON_VERSION < 300 const_str_empty // lnotab (table to translate code object) #else const_bytes_empty // lnotab (table to translate code object) #endif ); assert( DEEP_HASH( argnames ) == hash ); if (unlikely( result == NULL )) { return NULL; } return result; } void Nuitka_Frame_AttachLocals( struct Nuitka_FrameObject *frame, char const *type_description, ... ) { assert( frame->m_type_description == NULL ); // TODO: Do not call this if there is nothing to do. Instead make all the // places handle NULL pointer. if ( type_description == NULL ) type_description = ""; frame->m_type_description = type_description; char const *w = type_description; char *t = frame->m_locals_storage; va_list( ap ); va_start( ap, type_description ); while ( *w != 0 ) { switch( *w ) { case NUITKA_TYPE_DESCRIPTION_OBJECT: { PyObject *value = va_arg( ap, PyObject * ); memcpy( t, &value, sizeof(value)); Py_XINCREF( value ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_OBJECT_PTR: { /* We store the pointed object only. */ PyObject **value = va_arg( ap, PyObject ** ); memcpy( t, value, sizeof(PyObject *)); Py_XINCREF( *value ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_CELL: { struct Nuitka_CellObject *value = va_arg( ap, struct Nuitka_CellObject * ); CHECK_OBJECT( value ); memcpy( t, &value, sizeof(value) ); Py_INCREF( value ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_NULL: { void *value = va_arg( ap, struct Nuitka_CellObject * ); t += sizeof(value); break; } case NUITKA_TYPE_DESCRIPTION_BOOL: { int value = va_arg( ap, int ); memcpy( t, &value, sizeof(int) ); t += sizeof(value); break; } default: assert(false); } w += 1; } va_end( ap ); assert( t - frame->m_locals_storage <= Py_SIZE( frame )); } Nuitka-0.5.28.2/nuitka/build/static_src/CompiledCodeHelpers.c0000644000372000001440000024334613207540035024244 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /* Implementations of compiled code helpers. * The definition of a compiled code helper is that it's being used in * generated C code and provides part of the operations implementation. * * Currently we also have standalone mode related code here, patches to CPython * runtime that we do, and e.g. the built-in module. TODO: Move these to their * own files for clarity. */ #include "nuitka/prelude.h" #include "HelpersPathTools.c" #include "HelpersBuiltin.c" #include "HelpersStrings.c" #if PYTHON_VERSION < 300 static Py_ssize_t ESTIMATE_RANGE( long low, long high, long step ) { if ( low >= high ) { return 0; } else { return ( high - low - 1 ) / step + 1; } } static PyObject *_BUILTIN_RANGE_INT3( long low, long high, long step ) { assert( step != 0 ); Py_ssize_t size; if ( step > 0 ) { size = ESTIMATE_RANGE( low, high, step ); } else { size = ESTIMATE_RANGE( high, low, -step ); } PyObject *result = PyList_New( size ); long current = low; for ( int i = 0; i < size; i++ ) { PyList_SET_ITEM( result, i, PyInt_FromLong( current ) ); current += step; } return result; } static PyObject *_BUILTIN_RANGE_INT2( long low, long high ) { return _BUILTIN_RANGE_INT3( low, high, 1 ); } static PyObject *_BUILTIN_RANGE_INT( long boundary ) { return _BUILTIN_RANGE_INT2( 0, boundary ); } static PyObject *TO_RANGE_ARG( PyObject *value, char const *name ) { if (likely( PyInt_Check( value ) || PyLong_Check( value ) )) { Py_INCREF( value ); return value; } PyTypeObject *type = Py_TYPE( value ); PyNumberMethods *tp_as_number = type->tp_as_number; // Everything that casts to int is allowed. if ( #if PYTHON_VERSION >= 270 PyFloat_Check( value ) || #endif tp_as_number == NULL || tp_as_number->nb_int == NULL ) { PyErr_Format( PyExc_TypeError, "range() integer %s argument expected, got %s.", name, type->tp_name ); return NULL; } PyObject *result = tp_as_number->nb_int( value ); if (unlikely( result == NULL )) { return NULL; } return result; } #endif #if PYTHON_VERSION < 300 NUITKA_DEFINE_BUILTIN( range ); PyObject *BUILTIN_RANGE( PyObject *boundary ) { PyObject *boundary_temp = TO_RANGE_ARG( boundary, "end" ); if (unlikely( boundary_temp == NULL )) { return NULL; } long start = PyInt_AsLong( boundary_temp ); if ( start == -1 && ERROR_OCCURRED() ) { CLEAR_ERROR_OCCURRED(); PyObject *args[] = { boundary_temp }; NUITKA_ASSIGN_BUILTIN( range ); PyObject *result = CALL_FUNCTION_WITH_ARGS1( NUITKA_ACCESS_BUILTIN( range ), args ); Py_DECREF( boundary_temp ); return result; } Py_DECREF( boundary_temp ); return _BUILTIN_RANGE_INT( start ); } PyObject *BUILTIN_RANGE2( PyObject *low, PyObject *high ) { PyObject *low_temp = TO_RANGE_ARG( low, "start" ); if (unlikely( low_temp == NULL )) { return NULL; } PyObject *high_temp = TO_RANGE_ARG( high, "end" ); if (unlikely( high_temp == NULL )) { Py_DECREF( low_temp ); return NULL; } bool fallback = false; long start = PyInt_AsLong( low_temp ); if (unlikely( start == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } long end = PyInt_AsLong( high_temp ); if (unlikely( end == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } if ( fallback ) { PyObject *pos_args = PyTuple_New( 2 ); PyTuple_SET_ITEM( pos_args, 0, low_temp ); PyTuple_SET_ITEM( pos_args, 1, high_temp ); NUITKA_ASSIGN_BUILTIN( range ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( NUITKA_ACCESS_BUILTIN( range ), pos_args ); Py_DECREF( pos_args ); return result; } else { Py_DECREF( low_temp ); Py_DECREF( high_temp ); return _BUILTIN_RANGE_INT2( start, end ); } } PyObject *BUILTIN_RANGE3( PyObject *low, PyObject *high, PyObject *step ) { PyObject *low_temp = TO_RANGE_ARG( low, "start" ); if (unlikely( low_temp == NULL )) { return NULL; } PyObject *high_temp = TO_RANGE_ARG( high, "end" ); if (unlikely( high_temp == NULL )) { Py_DECREF( low_temp ); return NULL; } PyObject *step_temp = TO_RANGE_ARG( step, "step" ); if (unlikely( high_temp == NULL )) { Py_DECREF( low_temp ); Py_DECREF( high_temp ); return NULL; } bool fallback = false; long start = PyInt_AsLong( low_temp ); if (unlikely( start == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } long end = PyInt_AsLong( high_temp ); if (unlikely( end == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } long step_long = PyInt_AsLong( step_temp ); if (unlikely( step_long == -1 && ERROR_OCCURRED() )) { CLEAR_ERROR_OCCURRED(); fallback = true; } if ( fallback ) { PyObject *pos_args = PyTuple_New( 3 ); PyTuple_SET_ITEM( pos_args, 0, low_temp ); PyTuple_SET_ITEM( pos_args, 1, high_temp ); PyTuple_SET_ITEM( pos_args, 2, step_temp ); NUITKA_ASSIGN_BUILTIN( range ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( NUITKA_ACCESS_BUILTIN( range ), pos_args ); Py_DECREF( pos_args ); return result; } else { Py_DECREF( low_temp ); Py_DECREF( high_temp ); Py_DECREF( step_temp ); if (unlikely( step_long == 0 )) { PyErr_Format( PyExc_ValueError, "range() step argument must not be zero" ); return NULL; } return _BUILTIN_RANGE_INT3( start, end, step_long ); } } #endif #if PYTHON_VERSION < 300 /* Same as CPython2: */ static unsigned long getLengthOfRange( long lo, long hi, long step ) { assert( step != 0 ); if ( step > 0 && lo < hi ) { return 1UL + (hi - 1UL - lo) / step; } else if (step < 0 && lo > hi) { return 1UL + (lo - 1UL - hi) / (0UL - step); } else { return 0UL; } } /* Create a "xrange" object from C long values. Used for constant ranges. */ PyObject *MAKE_XRANGE( long start, long stop, long step ) { /* TODO: It would be sweet to calculate that on user side already. */ unsigned long n = getLengthOfRange( start, stop, step ); if ( n > (unsigned long)LONG_MAX || (long)n > PY_SSIZE_T_MAX ) { PyErr_SetString( PyExc_OverflowError, "xrange() result has too many items" ); return NULL; } struct _rangeobject2 *result = (struct _rangeobject2 *)PyObject_New( struct _rangeobject2, &PyRange_Type ); assert( result != NULL ); result->start = start; result->len = (long)n; result->step = step; return (PyObject *)result; } #else /* Same as CPython3: */ static PyObject *getLengthOfRange( PyObject *start, PyObject *stop, PyObject *step ) { int res = PyObject_RichCompareBool( step, const_int_0, Py_GT ); if (unlikely( res == -1 )) { return NULL; } PyObject *lo, *hi; // Make sure we use step as a positive number. if ( res == 1 ) { lo = start; hi = stop; Py_INCREF( step ); } else { lo = stop; hi = start; step = PyNumber_Negative( step ); if (unlikely( step == NULL )) { return NULL; } res = PyObject_RichCompareBool( step, const_int_0, Py_EQ ); if (unlikely( res == -1 )) { return NULL; } if ( res == 1 ) { PyErr_Format( PyExc_ValueError, "range() arg 3 must not be zero" ); return NULL; } } // Negative difference, we got zero length. res = PyObject_RichCompareBool( lo, hi, Py_GE ); if ( res != 0 ) { Py_XDECREF( step ); if ( res < 0 ) { return NULL; } Py_INCREF( const_int_0 ); return const_int_0; } PyObject *tmp1 = PyNumber_Subtract( hi, lo ); if (unlikely( tmp1 == NULL )) { Py_DECREF( step ); return NULL; } PyObject *diff = PyNumber_Subtract( tmp1, const_int_pos_1 ); Py_DECREF( tmp1 ); if (unlikely( diff == NULL )) { Py_DECREF( step ); Py_DECREF( tmp1 ); return NULL; } tmp1 = PyNumber_FloorDivide( diff, step ); Py_DECREF( diff ); Py_DECREF( step ); if (unlikely( tmp1 == NULL )) { return NULL; } PyObject *result = PyNumber_Add( tmp1, const_int_pos_1 ); Py_DECREF( tmp1 ); return result; } static PyObject *MAKE_XRANGE( PyObject *start, PyObject *stop, PyObject *step ) { start = PyNumber_Index( start ); if (unlikely( start == NULL )) { return NULL; } stop = PyNumber_Index( stop ); if (unlikely( stop == NULL )) { return NULL; } step = PyNumber_Index( step ); if (unlikely( step == NULL )) { return NULL; } PyObject *length = getLengthOfRange( start, stop, step ); if (unlikely( length == NULL )) { return NULL; } struct _rangeobject3 *result = (struct _rangeobject3 *)PyObject_New( struct _rangeobject3, &PyRange_Type ); assert( result != NULL ); result->start = start; result->stop = stop; result->step = step; result->length = length; return (PyObject *)result; } #endif /* Built-in xrange (Python2) or xrange (Python3) with one argument. */ PyObject *BUILTIN_XRANGE1( PyObject *high ) { #if PYTHON_VERSION < 300 if (unlikely( PyFloat_Check( high ) )) { PyErr_SetString( PyExc_TypeError, "integer argument expected, got float" ); return NULL; } long int_high = PyInt_AsLong( high ); if (unlikely( int_high == -1 && ERROR_OCCURRED() )) { return NULL; } return MAKE_XRANGE( 0, int_high, 1 ); #else PyObject *stop = PyNumber_Index( high ); if (unlikely( stop == NULL )) { return NULL; } struct _rangeobject3 *result = (struct _rangeobject3 *)PyObject_New( struct _rangeobject3, &PyRange_Type ); assert( result != NULL ); result->start = const_int_0; Py_INCREF( const_int_0 ); result->stop = stop; result->step = const_int_pos_1; Py_INCREF( const_int_pos_1 ); result->length = stop; Py_INCREF( stop ); return (PyObject *)result; #endif } /* Built-in xrange (Python2) or xrange (Python3) with two arguments. */ PyObject *BUILTIN_XRANGE2( PyObject *low, PyObject *high ) { #if PYTHON_VERSION < 300 if (unlikely( PyFloat_Check( low ) )) { PyErr_SetString( PyExc_TypeError, "integer argument expected, got float" ); return NULL; } long int_low = PyInt_AsLong( low ); if (unlikely( int_low == -1 && ERROR_OCCURRED() )) { return NULL; } if (unlikely( PyFloat_Check( high ) )) { PyErr_SetString( PyExc_TypeError, "integer argument expected, got float" ); return NULL; } long int_high = PyInt_AsLong( high ); if (unlikely( int_high == -1 && ERROR_OCCURRED() )) { return NULL; } return MAKE_XRANGE( int_low, int_high, 1 ); #else return MAKE_XRANGE( low, high, const_int_pos_1 ); #endif } /* Built-in xrange (Python2) or xrange (Python3) with three arguments. */ PyObject *BUILTIN_XRANGE3( PyObject *low, PyObject *high, PyObject *step ) { #if PYTHON_VERSION < 300 if (unlikely( PyFloat_Check( low ) )) { PyErr_SetString( PyExc_TypeError, "integer argument expected, got float" ); return NULL; } long int_low = PyInt_AsLong( low ); if (unlikely( int_low == -1 && ERROR_OCCURRED() )) { return NULL; } if (unlikely( PyFloat_Check( high ) )) { PyErr_SetString( PyExc_TypeError, "integer argument expected, got float" ); return NULL; } long int_high = PyInt_AsLong( high ); if (unlikely( int_high == -1 && ERROR_OCCURRED() )) { return NULL; } if (unlikely( PyFloat_Check( step ) )) { PyErr_SetString( PyExc_TypeError, "integer argument expected, got float" ); return NULL; } long int_step = PyInt_AsLong( step ); if (unlikely( int_step == -1 && ERROR_OCCURRED() )) { return NULL; } if (unlikely( int_step == 0 )) { PyErr_Format( PyExc_ValueError, "range() arg 3 must not be zero" ); return NULL; } return MAKE_XRANGE( int_low, int_high, int_step ); #else return MAKE_XRANGE( low, high, step ); #endif } PyObject *BUILTIN_LEN( PyObject *value ) { CHECK_OBJECT( value ); Py_ssize_t res = PyObject_Size( value ); if (unlikely( res < 0 && ERROR_OCCURRED() )) { return NULL; } return PyInt_FromSsize_t( res ); } NUITKA_DEFINE_BUILTIN( format ); PyObject *BUILTIN_FORMAT( PyObject *value, PyObject *format_spec ) { CHECK_OBJECT( value ); CHECK_OBJECT( format_spec ); NUITKA_ASSIGN_BUILTIN( format ); PyObject *args[2] = { value, format_spec }; return CALL_FUNCTION_WITH_ARGS2( NUITKA_ACCESS_BUILTIN( format ), args ); } NUITKA_DEFINE_BUILTIN( __import__ ); extern PyObject *const_str_plain_name; extern PyObject *const_str_plain_globals; extern PyObject *const_str_plain_locals; extern PyObject *const_str_plain_fromlist; extern PyObject *const_str_plain_level; PyObject *IMPORT_MODULE_KW( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items, PyObject *level ) { if (module_name) CHECK_OBJECT( module_name ); if (globals) CHECK_OBJECT( globals ); if (locals) CHECK_OBJECT( locals ); if (import_items) CHECK_OBJECT( import_items ); if (level) CHECK_OBJECT( level ); PyObject *kw_args = PyDict_New(); if ( module_name ) { CHECK_OBJECT( module_name ); PyDict_SetItem( kw_args, const_str_plain_name, module_name ); } if ( globals ) { CHECK_OBJECT( globals ); PyDict_SetItem( kw_args, const_str_plain_globals, globals ); } if ( locals ) { CHECK_OBJECT( locals ); PyDict_SetItem( kw_args, const_str_plain_locals, locals ); } if ( import_items ) { CHECK_OBJECT( import_items ); PyDict_SetItem( kw_args, const_str_plain_fromlist, import_items ); } if ( level ) { CHECK_OBJECT( level ); PyDict_SetItem( kw_args, const_str_plain_level, level ); } NUITKA_ASSIGN_BUILTIN( __import__ ); PyObject *import_result = CALL_FUNCTION_WITH_KEYARGS( NUITKA_ACCESS_BUILTIN( __import__ ), kw_args ); Py_DECREF( kw_args ); return import_result; } PyObject *IMPORT_MODULE1( PyObject *module_name ) { CHECK_OBJECT( module_name ); PyObject *pos_args[] = { module_name }; NUITKA_ASSIGN_BUILTIN( __import__ ); PyObject *import_result = CALL_FUNCTION_WITH_ARGS1( NUITKA_ACCESS_BUILTIN( __import__ ), pos_args ); return import_result; } PyObject *IMPORT_MODULE2( PyObject *module_name, PyObject *globals ) { CHECK_OBJECT( module_name ); CHECK_OBJECT( globals ); PyObject *pos_args[] = { module_name, globals }; NUITKA_ASSIGN_BUILTIN( __import__ ); PyObject *import_result = CALL_FUNCTION_WITH_ARGS2( NUITKA_ACCESS_BUILTIN( __import__ ), pos_args ); return import_result; } PyObject *IMPORT_MODULE3( PyObject *module_name, PyObject *globals, PyObject *locals ) { CHECK_OBJECT( module_name ); CHECK_OBJECT( globals ); CHECK_OBJECT( locals ); PyObject *pos_args[] = { module_name, globals, locals }; NUITKA_ASSIGN_BUILTIN( __import__ ); PyObject *import_result = CALL_FUNCTION_WITH_ARGS3( NUITKA_ACCESS_BUILTIN( __import__ ), pos_args ); return import_result; } PyObject *IMPORT_MODULE4( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items ) { CHECK_OBJECT( module_name ); CHECK_OBJECT( globals ); CHECK_OBJECT( locals ); CHECK_OBJECT( import_items ); PyObject *pos_args[] = { module_name, globals, locals, import_items }; NUITKA_ASSIGN_BUILTIN( __import__ ); PyObject *import_result = CALL_FUNCTION_WITH_ARGS4( NUITKA_ACCESS_BUILTIN( __import__ ), pos_args ); return import_result; } PyObject *IMPORT_MODULE5( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items, PyObject *level ) { CHECK_OBJECT( module_name ); CHECK_OBJECT( globals ); CHECK_OBJECT( locals ); CHECK_OBJECT( import_items ); CHECK_OBJECT( level ); PyObject *pos_args[] = { module_name, globals, locals, import_items, level }; NUITKA_ASSIGN_BUILTIN( __import__ ); PyObject *import_result = CALL_FUNCTION_WITH_ARGS5( NUITKA_ACCESS_BUILTIN( __import__ ), pos_args ); return import_result; } extern PyObject *const_str_plain___all__; bool IMPORT_MODULE_STAR( PyObject *target, bool is_module, PyObject *module ) { // Check parameters. CHECK_OBJECT( module ); CHECK_OBJECT( target ); PyObject *iter; bool all_case; PyObject *all = PyObject_GetAttr( module, const_str_plain___all__ ); if ( all != NULL ) { iter = MAKE_ITERATOR( all ); Py_DECREF( all ); if (unlikely( iter == NULL )) { return false; } all_case = true; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); iter = MAKE_ITERATOR( PyModule_GetDict( module ) ); CHECK_OBJECT( iter ); all_case = false; } else { return false; } for(;;) { PyObject *item = ITERATOR_NEXT( iter ); if ( item == NULL ) break; #if PYTHON_VERSION < 300 if (unlikely( PyString_Check( item ) == false && PyUnicode_Check( item ) == false )) #else if (unlikely( PyUnicode_Check( item ) == false )) #endif { PyErr_Format( PyExc_TypeError, "attribute name must be string, not '%s'", Py_TYPE( item )->tp_name ); break; } // When we are not using the "__all__", we should skip private variables. if ( all_case == false ) { if ( Nuitka_String_AsString_Unchecked( item )[0] == '_' ) { continue; } } PyObject *value = LOOKUP_ATTRIBUTE( module, item ); // Might not exist, because of e.g. wrong "__all__" value. if (unlikely( value == NULL )) { Py_DECREF( item ); break; } // TODO: Check if the reference is handled correctly if ( is_module ) { SET_ATTRIBUTE( target, item, value ); } else { SET_SUBSCRIPT( target, item, value ); } Py_DECREF( value ); Py_DECREF( item ); } Py_DECREF( iter ); return !ERROR_OCCURRED(); } // Helper functions for print. Need to play nice with Python softspace // behaviour. #if PYTHON_VERSION >= 300 extern PyObject *const_str_plain_end; extern PyObject *const_str_plain_file; extern PyObject *const_str_empty; NUITKA_DEFINE_BUILTIN( print ); #endif bool PRINT_NEW_LINE_TO( PyObject *file ) { #if PYTHON_VERSION < 300 if ( file == NULL || file == Py_None ) { file = GET_STDOUT(); } // need to hold a reference to the file or else __getattr__ may release // "file" in the mean time. Py_INCREF( file ); if (unlikely( PyFile_WriteString( "\n", file ) == -1)) { Py_DECREF( file ); return false; } PyFile_SoftSpace( file, 0 ); CHECK_OBJECT( file ); Py_DECREF( file ); return true; #else NUITKA_ASSIGN_BUILTIN( print ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); PyObject *result; if (likely( file == NULL )) { result = CALL_FUNCTION_NO_ARGS( NUITKA_ACCESS_BUILTIN( print ) ); } else { PyObject *kw_args = PyDict_New(); PyDict_SetItem( kw_args, const_str_plain_file, GET_STDOUT() ); result = CALL_FUNCTION_WITH_KEYARGS( NUITKA_ACCESS_BUILTIN( print ), kw_args ); Py_DECREF( kw_args ); } Py_XDECREF( result ); RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return result != NULL; #endif } bool PRINT_ITEM_TO( PyObject *file, PyObject *object ) { // The print built-in function cannot replace "softspace" behavior of CPython // print statement, so this code is really necessary. #if PYTHON_VERSION < 300 if ( file == NULL || file == Py_None ) { file = GET_STDOUT(); } CHECK_OBJECT( file ); CHECK_OBJECT( object ); // need to hold a reference to the file or else "__getattr__" code may // release "file" in the mean time. Py_INCREF( file ); // Check for soft space indicator if ( PyFile_SoftSpace( file, 0 ) ) { if (unlikely( PyFile_WriteString( " ", file ) == -1 )) { Py_DECREF( file ); return false; } } if ( unlikely( PyFile_WriteObject( object, file, Py_PRINT_RAW ) == -1 )) { Py_DECREF( file ); return false; } if ( PyString_Check( object ) ) { char *buffer; Py_ssize_t length; #ifndef __NUITKA_NO_ASSERT__ int status = #endif PyString_AsStringAndSize( object, &buffer, &length ); assert( status != -1 ); if ( length == 0 || !isspace( Py_CHARMASK(buffer[length-1])) || buffer[ length - 1 ] == ' ' ) { PyFile_SoftSpace( file, 1 ); } } else if ( PyUnicode_Check( object ) ) { Py_UNICODE *buffer = PyUnicode_AS_UNICODE( object ); Py_ssize_t length = PyUnicode_GET_SIZE( object ); if ( length == 0 || !Py_UNICODE_ISSPACE( buffer[length-1]) || buffer[length-1] == ' ' ) { PyFile_SoftSpace( file, 1 ); } } else { PyFile_SoftSpace( file, 1 ); } CHECK_OBJECT( file ); Py_DECREF( file ); return true; #else NUITKA_ASSIGN_BUILTIN( print ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); PyObject *result; if (likely( file == NULL )) { PyObject *args[] = { object }; result = CALL_FUNCTION_WITH_ARGS1( NUITKA_ACCESS_BUILTIN( print ), args ); } else { PyObject *print_kw = PyDict_New(); PyDict_SetItem( print_kw, const_str_plain_end, const_str_empty ); PyDict_SetItem( print_kw, const_str_plain_file, GET_STDOUT() ); PyObject *print_args = PyTuple_New( 1 ); PyTuple_SET_ITEM( print_args, 0, object ); Py_INCREF( object ); result = CALL_FUNCTION( NUITKA_ACCESS_BUILTIN( print ), print_args, print_kw ); Py_DECREF( print_args ); Py_DECREF( print_kw ); } Py_XDECREF( result ); RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return result != NULL; #endif } void PRINT_REFCOUNT( PyObject *object ) { char buffer[ 1024 ]; snprintf( buffer, sizeof( buffer ) - 1, " refcnt %" PY_FORMAT_SIZE_T "d ", Py_REFCNT( object ) ); PRINT_STRING( buffer ); } bool PRINT_STRING( char const *str ) { int res = PyFile_WriteString( str, GET_STDOUT() ); return res != -1; } bool PRINT_REPR( PyObject *object ) { PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); bool res; if ( object != NULL ) { CHECK_OBJECT( object ); // Cannot have error set for this function, it asserts against that // in debug builds. PyObject *repr = PyObject_Repr( object ); res = PRINT_ITEM( repr ); Py_DECREF( repr ); } else { res = PRINT_NULL(); } RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return res; } bool PRINT_NULL( void ) { return PRINT_STRING(""); } void PRINT_EXCEPTION( PyObject *exception_type, PyObject *exception_value, PyObject *exception_tb ) { PRINT_REPR( exception_type ); PRINT_STRING("|"); PRINT_REPR( exception_value ); #if PYTHON_VERSION >= 300 if ( exception_value != NULL ) { PRINT_STRING(" <- "); PRINT_REPR( PyException_GetContext( exception_value ) ); } #endif PRINT_STRING("|"); PRINT_REPR( exception_tb ); PRINT_NEW_LINE(); } void PRINT_CURRENT_EXCEPTION( void ) { PyThreadState *tstate = PyThreadState_GET(); PRINT_EXCEPTION( tstate->curexc_type, tstate->curexc_value, tstate->curexc_traceback ); } // TODO: Could be ported, the "printf" stuff would need to be split. On Python3 // the normal C print output gets lost. #if PYTHON_VERSION < 300 void PRINT_TRACEBACK( PyTracebackObject *traceback ) { PRINT_STRING("Dumping traceback:\n"); if ( traceback == NULL ) PRINT_STRING( "\n" ); while( traceback ) { printf( " line %d (frame object chain):\n", traceback->tb_lineno ); PyFrameObject *frame = traceback->tb_frame; while ( frame ) { printf( " Frame at %s\n", PyString_AsString( PyObject_Str( (PyObject *)frame->f_code ))); frame = frame->f_back; } assert( traceback->tb_next != traceback ); traceback = traceback->tb_next; } PRINT_STRING("End of Dump.\n"); } #endif PyObject *GET_STDOUT() { PyObject *result = PySys_GetObject( (char *)"stdout" ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_RuntimeError, "lost sys.stdout" ); return NULL; } return result; } PyObject *GET_STDERR() { PyObject *result = PySys_GetObject( (char *)"stderr" ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_RuntimeError, "lost sys.stderr" ); return NULL; } return result; } bool PRINT_NEW_LINE( void ) { PyObject *target = GET_STDOUT(); return target != NULL && PRINT_NEW_LINE_TO( target ); } bool PRINT_ITEM( PyObject *object ) { if ( object == NULL ) { return PRINT_NULL(); } else { PyObject *target = GET_STDOUT(); return target != NULL && PRINT_ITEM_TO( target, object ); } } #if PYTHON_VERSION < 300 PyObject *UNSTREAM_UNICODE( unsigned char const *buffer, Py_ssize_t size ) { PyObject *result = PyUnicode_FromStringAndSize( (char const *)buffer, size ); assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); return result; } #endif PyObject *UNSTREAM_STRING( unsigned char const *buffer, Py_ssize_t size, bool intern ) { #if PYTHON_VERSION < 300 PyObject *result = PyString_FromStringAndSize( (char const *)buffer, size ); #else PyObject *result = PyUnicode_FromStringAndSize( (char const *)buffer, size ); #endif assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == size ); #endif if ( intern ) { Nuitka_StringIntern( &result ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == size ); #else assert( PyUnicode_GET_SIZE( result ) == size ); #endif } return result; } PyObject *UNSTREAM_CHAR( unsigned char value, bool intern ) { #if PYTHON_VERSION < 300 PyObject *result = PyString_FromStringAndSize( (char const *)&value, 1 ); #else PyObject *result = PyUnicode_FromStringAndSize( (char const *)&value, 1 ); #endif assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == 1 ); #else assert( PyUnicode_GET_SIZE( result ) == 1 ); #endif if ( intern ) { Nuitka_StringIntern( &result ); CHECK_OBJECT( result ); assert( Nuitka_String_Check( result ) ); #if PYTHON_VERSION < 300 assert( PyString_Size( result ) == 1 ); #else assert( PyUnicode_GET_SIZE( result ) == 1 ); #endif } return result; } PyObject *UNSTREAM_FLOAT( unsigned char const *buffer ) { double x = _PyFloat_Unpack8( buffer, 1 ); assert( x != -1.0 || !PyErr_Occurred() ); PyObject *result = PyFloat_FromDouble(x); assert( result != NULL ); return result; } #if PYTHON_VERSION >= 300 PyObject *UNSTREAM_BYTES( unsigned char const *buffer, Py_ssize_t size ) { PyObject *result = PyBytes_FromStringAndSize( (char const *)buffer, size ); assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( PyBytes_GET_SIZE( result ) == size ); return result; } #endif PyObject *UNSTREAM_BYTEARRAY( unsigned char const *buffer, Py_ssize_t size ) { PyObject *result = PyByteArray_FromStringAndSize( (char const *)buffer, size ); assert( !ERROR_OCCURRED() ); CHECK_OBJECT( result ); assert( PyByteArray_GET_SIZE( result ) == size ); return result; } #if PYTHON_VERSION < 300 static void set_slot( PyObject **slot, PyObject *value ) { PyObject *temp = *slot; Py_XINCREF( value ); *slot = value; Py_XDECREF( temp ); } extern PyObject *const_str_plain___getattr__; extern PyObject *const_str_plain___setattr__; extern PyObject *const_str_plain___delattr__; static void set_attr_slots( PyClassObject *klass ) { set_slot( &klass->cl_getattr, FIND_ATTRIBUTE_IN_CLASS( klass, const_str_plain___getattr__ ) ); set_slot( &klass->cl_setattr, FIND_ATTRIBUTE_IN_CLASS( klass, const_str_plain___setattr__ ) ); set_slot( &klass->cl_delattr, FIND_ATTRIBUTE_IN_CLASS( klass, const_str_plain___delattr__ ) ); } static bool set_dict( PyClassObject *klass, PyObject *value ) { if ( value == NULL || !PyDict_Check( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__dict__ must be a dictionary object" ); return false; } else { set_slot( &klass->cl_dict, value ); set_attr_slots( klass ); return true; } } static bool set_bases( PyClassObject *klass, PyObject *value ) { if ( value == NULL || !PyTuple_Check( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__bases__ must be a tuple object" ); return false; } else { Py_ssize_t n = PyTuple_Size( value ); for ( Py_ssize_t i = 0; i < n; i++ ) { PyObject *base = PyTuple_GET_ITEM( value, i ); if (unlikely( !PyClass_Check( base ) )) { PyErr_SetString( PyExc_TypeError, (char *)"__bases__ items must be classes" ); return false; } if (unlikely( PyClass_IsSubclass( base, (PyObject *)klass) )) { PyErr_SetString( PyExc_TypeError, (char *)"a __bases__ item causes an inheritance cycle" ); return false; } } set_slot( &klass->cl_bases, value ); set_attr_slots( klass ); return true; } } static bool set_name( PyClassObject *klass, PyObject *value ) { if ( value == NULL || !PyDict_Check( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__name__ must be a string object" ); return false; } if ( strlen( PyString_AS_STRING( value )) != (size_t)PyString_GET_SIZE( value ) ) { PyErr_SetString( PyExc_TypeError, (char *)"__name__ must not contain null bytes" ); return false; } set_slot( &klass->cl_name, value ); return true; } static int nuitka_class_setattr( PyClassObject *klass, PyObject *attr_name, PyObject *value ) { char *sattr_name = PyString_AsString( attr_name ); if ( sattr_name[0] == '_' && sattr_name[1] == '_' ) { Py_ssize_t n = PyString_Size( attr_name ); if ( sattr_name[ n-2 ] == '_' && sattr_name[ n-1 ] == '_' ) { if ( strcmp( sattr_name, "__dict__" ) == 0 ) { if ( set_dict( klass, value ) == false ) { return -1; } else { return 0; } } else if ( strcmp( sattr_name, "__bases__" ) == 0 ) { if ( set_bases( klass, value ) == false ) { return -1; } else { return 0; } } else if ( strcmp( sattr_name, "__name__" ) == 0 ) { if ( set_name( klass, value ) == false ) { return -1; } else { return 0; } } else if ( strcmp( sattr_name, "__getattr__" ) == 0 ) { set_slot( &klass->cl_getattr, value ); } else if ( strcmp(sattr_name, "__setattr__" ) == 0 ) { set_slot( &klass->cl_setattr, value ); } else if ( strcmp(sattr_name, "__delattr__" ) == 0 ) { set_slot( &klass->cl_delattr, value ); } } } if ( value == NULL ) { int status = PyDict_DelItem( klass->cl_dict, attr_name ); if ( status < 0 ) { PyErr_Format( PyExc_AttributeError, "class %s has no attribute '%s'", PyString_AS_STRING( klass->cl_name ), sattr_name ); } return status; } else { return PyDict_SetItem( klass->cl_dict, attr_name, value ); } } static PyObject *nuitka_class_getattr( PyClassObject *klass, PyObject *attr_name ) { char *sattr_name = PyString_AsString( attr_name ); if ( sattr_name[0] == '_' && sattr_name[1] == '_' ) { if ( strcmp( sattr_name, "__dict__" ) == 0 ) { Py_INCREF( klass->cl_dict ); return klass->cl_dict; } else if ( strcmp(sattr_name, "__bases__" ) == 0 ) { Py_INCREF( klass->cl_bases ); return klass->cl_bases; } else if ( strcmp(sattr_name, "__name__" ) == 0 ) { if ( klass->cl_name == NULL ) { Py_INCREF( Py_None ); return Py_None; } else { Py_INCREF( klass->cl_name ); return klass->cl_name; } } } PyObject *value = FIND_ATTRIBUTE_IN_CLASS( klass, attr_name ); if (unlikely( value == NULL )) { PyErr_Format( PyExc_AttributeError, "class %s has no attribute '%s'", PyString_AS_STRING( klass->cl_name ), sattr_name ); return NULL; } PyTypeObject *type = Py_TYPE( value ); descrgetfunc tp_descr_get = PyType_HasFeature( type, Py_TPFLAGS_HAVE_CLASS ) ? type->tp_descr_get : NULL; if ( tp_descr_get == NULL ) { Py_INCREF( value ); return value; } else { return tp_descr_get( value, (PyObject *)NULL, (PyObject *)klass ); } } #endif void enhancePythonTypes( void ) { #if PYTHON_VERSION < 300 // Our own variant won't call PyEval_GetRestricted, saving quite some cycles // not doing that. PyClass_Type.tp_setattro = (setattrofunc)nuitka_class_setattr; PyClass_Type.tp_getattro = (getattrofunc)nuitka_class_getattr; #endif } #ifdef __APPLE__ #ifdef __cplusplus extern "C" #endif wchar_t* _Py_DecodeUTF8_surrogateescape( const char *s, Py_ssize_t size ); #endif #ifdef __FreeBSD__ #include #endif #include #if PYTHON_VERSION >= 300 argv_type_t convertCommandLineParameters( int argc, char **argv ) { #if _WIN32 int new_argc; argv_type_t result = CommandLineToArgvW( GetCommandLineW(), &new_argc ); assert( new_argc == argc ); return result; #else // Originally taken from CPython3: There seems to be no sane way to use static wchar_t **argv_copy; argv_copy = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*) * argc); // Temporarily disable locale for conversions to not use it. char *oldloc = strdup( setlocale( LC_ALL, NULL ) ); setlocale( LC_ALL, "" ); for ( int i = 0; i < argc; i++ ) { #ifdef __APPLE__ argv_copy[i] = _Py_DecodeUTF8_surrogateescape( argv[ i ], strlen( argv[ i ] ) ); #elif PYTHON_VERSION < 350 argv_copy[i] = _Py_char2wchar( argv[ i ], NULL ); #else argv_copy[i] = Py_DecodeLocale( argv[ i ], NULL ); #endif assert ( argv_copy[ i ] ); } setlocale( LC_ALL, oldloc ); free( oldloc ); return argv_copy; #endif } #endif bool setCommandLineParameters( int argc, argv_type_t argv, bool initial ) { bool is_multiprocessing_fork = false; if ( initial ) { /* We might need to skip what multiprocessing has told us. */ for ( int i = 1; i < argc; i++ ) { #if PYTHON_VERSION < 300 if ((strcmp(argv[i], "--multiprocessing-fork")) == 0 && (i+1 < argc)) #else wchar_t constant_buffer[100]; mbstowcs( constant_buffer, "--multiprocessing-fork", 100 ); if ((wcscmp(argv[i], constant_buffer )) == 0 && (i+1 < argc)) #endif { is_multiprocessing_fork = true; break; } } } if ( initial ) { Py_SetProgramName( argv[0] ); } else { PySys_SetArgv( argc, argv ); } return is_multiprocessing_fork; } PyObject *original_isinstance = NULL; // Note: Installed and used by "InspectPatcher" as "instance" too. int Nuitka_IsInstance( PyObject *inst, PyObject *cls ) { CHECK_OBJECT( original_isinstance ); CHECK_OBJECT( inst ); CHECK_OBJECT( cls ); // Quick paths if ( Py_TYPE( inst ) == (PyTypeObject *)cls ) { return true; } // Our paths for the types we need to hook. if ( cls == (PyObject *)&PyFunction_Type && Nuitka_Function_Check( inst ) ) { return true; } if ( cls == (PyObject *)&PyGen_Type && Nuitka_Generator_Check( inst ) ) { return true; } if ( cls == (PyObject *)&PyMethod_Type && Nuitka_Method_Check( inst ) ) { return true; } if ( cls == (PyObject *)&PyFrame_Type && Nuitka_Frame_Check( inst ) ) { return true; } #if PYTHON_VERSION >= 350 if ( cls == (PyObject *)&PyCoro_Type && Nuitka_Coroutine_Check( inst ) ) { return true; } #endif #if PYTHON_VERSION >= 360 if ( cls == (PyObject *)&PyAsyncGen_Type && Nuitka_Asyncgen_Check( inst ) ) { return true; } #endif // May need to be recursive for tuple arguments. if ( PyTuple_Check( cls ) ) { for ( Py_ssize_t i = 0, size = PyTuple_GET_SIZE( cls ); i < size; i++ ) { PyObject *element = PyTuple_GET_ITEM( cls, i ); if (unlikely( Py_EnterRecursiveCall( (char *)" in __instancecheck__" ) )) { return -1; } int res = Nuitka_IsInstance( inst, element ); Py_LeaveRecursiveCall(); if ( res != 0 ) { return res; } } return 0; } else { PyObject *args[] = { inst, cls }; PyObject *result = CALL_FUNCTION_WITH_ARGS2( original_isinstance, args ); if ( result == NULL ) { return -1; } int res = CHECK_IF_TRUE( result ); Py_DECREF( result ); if ( res == 0 ) { if ( cls == (PyObject *)&PyFunction_Type ) { args[1] = (PyObject *)&Nuitka_Function_Type; } else if ( cls == (PyObject *)&PyMethod_Type ) { args[1] = (PyObject *)&Nuitka_Method_Type; } else if ( cls == (PyObject *)&PyFrame_Type ) { args[1] = (PyObject *)&Nuitka_Frame_Type; } #if PYTHON_VERSION >= 350 else if ( cls == (PyObject *)&PyCoro_Type ) { args[1] = (PyObject *)&Nuitka_Coroutine_Type; } #endif #if PYTHON_VERSION >= 360 else if ( cls == (PyObject *)&PyAsyncGen_Type ) { args[1] = (PyObject *)&Nuitka_Asyncgen_Type; } #endif else { return 0; } result = CALL_FUNCTION_WITH_ARGS2( original_isinstance, args ); if ( result == NULL ) { return -1; } res = CHECK_IF_TRUE( result ); Py_DECREF( result ); } return res; } } PyObject *BUILTIN_ISINSTANCE( PyObject *inst, PyObject *cls ) { int res = Nuitka_IsInstance( inst, cls ); if (unlikely( res < 0 )) { return NULL; } return BOOL_FROM( res != 0 ); } #define ITERATOR_GENERIC 0 #define ITERATOR_COMPILED_GENERATOR 1 #define ITERATOR_TUPLE 2 #define ITERATOR_LIST 3 struct Nuitka_QuickIterator { int iterator_mode; union { // ITERATOR_GENERIC PyObject *iter; // ITERATOR_COMPILED_GENERATOR struct Nuitka_GeneratorObject *generator; // ITERATOR_TUPLE struct { PyTupleObject *tuple; Py_ssize_t tuple_index; } tuple_data; // ITERATOR_LIST struct { PyListObject *list; Py_ssize_t list_index; } list_data; } iterator_data; }; static bool MAKE_QUICK_ITERATOR( PyObject *sequence, struct Nuitka_QuickIterator *qiter ) { if ( Nuitka_Generator_Check( sequence ) ) { qiter->iterator_mode = ITERATOR_COMPILED_GENERATOR; qiter->iterator_data.generator = (struct Nuitka_GeneratorObject *)sequence; } else if ( PyTuple_CheckExact( sequence ) ) { qiter->iterator_mode = ITERATOR_TUPLE; qiter->iterator_data.tuple_data.tuple = (PyTupleObject *)sequence; qiter->iterator_data.tuple_data.tuple_index = 0; } else if ( PyList_CheckExact( sequence ) ) { qiter->iterator_mode = ITERATOR_LIST; qiter->iterator_data.list_data.list = (PyListObject *)sequence; qiter->iterator_data.list_data.list_index = 0; } else { qiter->iterator_mode = ITERATOR_GENERIC; qiter->iterator_data.iter = MAKE_ITERATOR( sequence ); if (unlikely( qiter->iterator_data.iter == NULL )) { return false; } } return true; } static PyObject *QUICK_ITERATOR_NEXT( struct Nuitka_QuickIterator *qiter, bool *finished ) { PyObject *result; switch ( qiter->iterator_mode ) { case ITERATOR_GENERIC: result = ITERATOR_NEXT( qiter->iterator_data.iter ); if ( result == NULL ) { Py_DECREF( qiter->iterator_data.iter ); if (unlikely( !CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED() )) { *finished = false; return NULL; } *finished = true; return NULL; } *finished = false; return result; case ITERATOR_COMPILED_GENERATOR: result = Nuitka_Generator_qiter( qiter->iterator_data.generator, finished ); return result; case ITERATOR_TUPLE: if ( qiter->iterator_data.tuple_data.tuple_index < PyTuple_GET_SIZE( qiter->iterator_data.tuple_data.tuple ) ) { result = PyTuple_GET_ITEM( qiter->iterator_data.tuple_data.tuple, qiter->iterator_data.tuple_data.tuple_index ); qiter->iterator_data.tuple_data.tuple_index += 1; *finished = false; Py_INCREF( result ); return result; } else { *finished = true; return NULL; } case ITERATOR_LIST: if ( qiter->iterator_data.list_data.list_index < PyList_GET_SIZE( qiter->iterator_data.list_data.list ) ) { result = PyList_GET_ITEM( qiter->iterator_data.list_data.list, qiter->iterator_data.list_data.list_index ); qiter->iterator_data.list_data.list_index += 1; *finished = false; Py_INCREF( result ); return result; } else { *finished = true; return NULL; } } assert(false); return NULL; } PyObject *BUILTIN_SUM1( PyObject *sequence ) { struct Nuitka_QuickIterator qiter; if (unlikely( MAKE_QUICK_ITERATOR( sequence, &qiter ) == false )) { return NULL; } PyObject *result; long int_result = 0; PyObject *item; for(;;) { bool finished; item = QUICK_ITERATOR_NEXT( &qiter, &finished ); if ( finished ) { #if PYTHON_VERSION < 300 return PyInt_FromLong( int_result ); #else return PyLong_FromLong( int_result ); #endif } else if ( item == NULL ) { return NULL; } CHECK_OBJECT( item ); // For Python2 int objects: #if PYTHON_VERSION < 300 if ( PyInt_CheckExact( item ) ) { long b = PyInt_AS_LONG( item ); long x = int_result + b; if ( (x ^ int_result) >= 0 || (x ^ b) >= 0 ) { int_result = x; Py_DECREF( item ); continue; } } #endif // For Python2 long, Python3 int objects #if PYTHON_VERSION >= 270 if ( PyLong_CheckExact( item ) ) { int overflow; long b = PyLong_AsLongAndOverflow( item, &overflow ); if ( overflow ) { break; } long x = int_result + b; if ( (x ^ int_result) >= 0 || (x ^ b) >= 0 ) { int_result = x; Py_DECREF( item ); continue; } } #endif if ( item == Py_False ) { Py_DECREF( item ); continue; } if ( item == Py_True ) { long b = 1; long x = int_result + b; if ( (x ^ int_result) >= 0 || (x ^ b) >= 0 ) { int_result = x; Py_DECREF( item ); continue; } } /* Either overflowed or not one of the supported int alike types. */ break; } /* Switch over to objects, and redo last step. */ #if PYTHON_VERSION < 300 result = PyInt_FromLong( int_result ); #else result = PyLong_FromLong( int_result ); #endif CHECK_OBJECT( result ); PyObject *temp = PyNumber_Add( result, item ); Py_DECREF( result ); Py_DECREF( item ); result = temp; if (unlikely( result == NULL )) { return NULL; } for(;;) { CHECK_OBJECT( result ); bool finished; item = QUICK_ITERATOR_NEXT( &qiter, &finished ); if ( finished ) { break; } else if ( item == NULL ) { Py_DECREF( result ); return NULL; } CHECK_OBJECT( item ); PyObject *temp2 = PyNumber_Add( result, item ); Py_DECREF( item ); Py_DECREF( result ); if (unlikely( temp2 == NULL )) { return NULL; } result = temp2; } CHECK_OBJECT( result ); return result; } NUITKA_DEFINE_BUILTIN( sum ); PyObject *BUILTIN_SUM2( PyObject *sequence, PyObject *start ) { NUITKA_ASSIGN_BUILTIN( sum ); CHECK_OBJECT( sequence ); CHECK_OBJECT( start ); PyObject *pos_args = PyTuple_New( 2 ); PyTuple_SET_ITEM( pos_args, 0, sequence ); Py_INCREF( sequence ); PyTuple_SET_ITEM( pos_args, 1, start ); Py_INCREF( start ); PyObject *result = CALL_FUNCTION_WITH_POSARGS( NUITKA_ACCESS_BUILTIN( sum ), pos_args ); Py_DECREF( pos_args ); return result; } PyDictObject *dict_builtin = NULL; PyModuleObject *builtin_module = NULL; static PyTypeObject Nuitka_BuiltinModule_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_module", // tp_name sizeof(PyModuleObject), // tp_size }; extern PyObject *const_str_plain_open; int Nuitka_BuiltinModule_SetAttr( PyModuleObject *module, PyObject *name, PyObject *value ) { CHECK_OBJECT( (PyObject *)module ); CHECK_OBJECT( name ); // This is used for "del" as well. assert( value == NULL || Py_REFCNT( value ) > 0 ); // only checks the builtins that we can refresh at this time, if we have // many value to check maybe need create a dict first. bool found = false; int res = PyObject_RichCompareBool( name, const_str_plain_open, Py_EQ ); if (unlikely( res == -1 )) { return -1; } if ( res == 1 ) { NUITKA_UPDATE_BUILTIN( open, value ); found = true; } if ( found == false ) { res = PyObject_RichCompareBool( name, const_str_plain___import__, Py_EQ ); if (unlikely( res == -1 )) { return -1; } if ( res == 1 ) { NUITKA_UPDATE_BUILTIN( __import__, value ); found = true; } } #if PYTHON_VERSION >= 300 if ( found == false ) { res = PyObject_RichCompareBool( name, const_str_plain_print, Py_EQ ); if (unlikely( res == -1 )) { return -1; } if ( res == 1 ) { NUITKA_UPDATE_BUILTIN( print, value ); found = true; } } #endif return PyObject_GenericSetAttr( (PyObject *)module, name, value ); } #include #if defined(_WIN32) #include #elif defined(__APPLE__) #include #include #include #else #include #include #endif #if defined( __FreeBSD__ ) #include #endif #if defined(_NUITKA_EXE) char *getBinaryDirectoryUTF8Encoded() { static char binary_directory[ MAXPATHLEN + 1 ]; static bool init_done = false; if ( init_done ) { return binary_directory; } #if defined(_WIN32) #if PYTHON_VERSION >= 300 WCHAR binary_directory2[ MAXPATHLEN + 1 ]; binary_directory2[0] = 0; DWORD res = GetModuleFileNameW( NULL, binary_directory2, MAXPATHLEN ); assert( res != 0 ); int res2 = WideCharToMultiByte( CP_UTF8, 0, binary_directory2, -1, binary_directory, MAXPATHLEN, NULL, NULL ); assert( res2 != 0 ); #else DWORD res = GetModuleFileName( NULL, binary_directory, MAXPATHLEN ); assert( res != 0 ); #endif PathRemoveFileSpec( binary_directory ); #elif defined(__APPLE__) uint32_t bufsize = MAXPATHLEN; int res =_NSGetExecutablePath( binary_directory, &bufsize ); if (unlikely( res != 0 )) { abort(); } // On MacOS, the "dirname" call creates a separate internal string, we can // safely copy back. strncpy( binary_directory, dirname(binary_directory), MAXPATHLEN ); #elif defined( __FreeBSD__ ) /* Not all of FreeBSD has /proc file system, so use the appropriate * "sysctl" instead. */ int mib[4]; mib[0] = CTL_KERN; mib[1] = KERN_PROC; mib[2] = KERN_PROC_PATHNAME; mib[3] = -1; size_t cb = sizeof( binary_directory ); sysctl( mib, 4, binary_directory, &cb, NULL, 0 ); /* We want the directory name, the above gives the full executable name. */ strcpy( binary_directory, dirname( binary_directory ) ); #else /* The remaining platforms, mostly Linux or compatible. */ /* The "readlink" call does not terminate result, so fill zeros there, then * it is a proper C string right away. */ memset( binary_directory, 0, MAXPATHLEN + 1 ); ssize_t res = readlink( "/proc/self/exe", binary_directory, MAXPATHLEN ); if (unlikely( res == -1 )) { abort(); } strncpy( binary_directory, dirname( binary_directory ), MAXPATHLEN ); #endif init_done = true; return binary_directory; } char *getBinaryDirectoryHostEncoded() { #if defined(_WIN32) static char binary_directory[ MAXPATHLEN + 1 ]; static bool init_done = false; if ( init_done ) { return binary_directory; } #if PYTHON_VERSION >= 300 WCHAR binary_directory2[ MAXPATHLEN + 1 ]; binary_directory2[0] = 0; DWORD res = GetModuleFileNameW( NULL, binary_directory2, MAXPATHLEN ); assert( res != 0 ); int res2 = WideCharToMultiByte( CP_ACP, 0, binary_directory2, -1, binary_directory, MAXPATHLEN, NULL, NULL ); assert( res2 != 0 ); #else DWORD res = GetModuleFileName( NULL, binary_directory, MAXPATHLEN ); assert( res != 0 ); #endif PathRemoveFileSpec( binary_directory ); init_done = true; return binary_directory; #else return getBinaryDirectoryUTF8Encoded(); #endif } static PyObject *getBinaryDirectoryObject() { static PyObject *binary_directory = NULL; if ( binary_directory != NULL ) { return binary_directory; } #if PYTHON_VERSION >= 300 binary_directory = PyUnicode_FromString( getBinaryDirectoryUTF8Encoded() ); #else binary_directory = PyString_FromString( getBinaryDirectoryUTF8Encoded() ); #endif if (unlikely( binary_directory == NULL )) { PyErr_Print(); abort(); } return binary_directory; } #else static char *getDllDirectory() { #if defined(_WIN32) static char path[ MAXPATHLEN + 1 ]; HMODULE hm = NULL; path[0] = '\0'; #if PYTHON_VERSION >= 300 WCHAR path2[ MAXPATHLEN + 1 ]; path2[0] = 0; int res = GetModuleHandleExA( GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR) &getDllDirectory, &hm ); assert( res != 0 ); res = GetModuleFileNameW( hm, path2, MAXPATHLEN + 1 ); assert( res != 0 ); int res2 = WideCharToMultiByte(CP_UTF8, 0, path2, -1, path, MAXPATHLEN + 1, NULL, NULL); assert( res2 != 0 ); #else int res = GetModuleHandleExA( GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR) &getDllDirectory, &hm ); assert( res != 0 ); res = GetModuleFileNameA( hm, path, MAXPATHLEN + 1 ); assert( res != 0 ); #endif PathRemoveFileSpec( path ); return path; #else Dl_info where; int res = dladdr( (void *)getDllDirectory, &where ); assert( res != 0 ); return dirname( (char *)where.dli_fname ); #endif } #endif void _initBuiltinModule() { #if _NUITKA_MODULE if ( builtin_module ) return; #else assert( builtin_module == NULL ); #endif #if PYTHON_VERSION < 300 builtin_module = (PyModuleObject *)PyImport_ImportModule( "__builtin__" ); #else builtin_module = (PyModuleObject *)PyImport_ImportModule( "builtins" ); #endif assert( builtin_module ); dict_builtin = (PyDictObject *)builtin_module->md_dict; assert( PyDict_Check( dict_builtin ) ); #ifdef _NUITKA_STANDALONE int res = PyDict_SetItemString( (PyObject *)dict_builtin, "__nuitka_binary_dir", getBinaryDirectoryObject() ); assert( res == 0 ); #endif // init Nuitka_BuiltinModule_Type, PyType_Ready wont copy all member from // base type, so we need copy all members from PyModule_Type manual for // safety. PyType_Ready will change tp_flags, we need define it again. Set // tp_setattro to Nuitka_BuiltinModule_SetAttr and we can detect value // change. Set tp_base to PyModule_Type and PyModule_Check will pass. Nuitka_BuiltinModule_Type.tp_dealloc = PyModule_Type.tp_dealloc; Nuitka_BuiltinModule_Type.tp_repr = PyModule_Type.tp_repr; Nuitka_BuiltinModule_Type.tp_setattro = (setattrofunc)Nuitka_BuiltinModule_SetAttr; Nuitka_BuiltinModule_Type.tp_getattro = PyModule_Type.tp_getattro; Nuitka_BuiltinModule_Type.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE; Nuitka_BuiltinModule_Type.tp_doc = PyModule_Type.tp_doc; Nuitka_BuiltinModule_Type.tp_traverse = PyModule_Type.tp_traverse; Nuitka_BuiltinModule_Type.tp_members = PyModule_Type.tp_members; Nuitka_BuiltinModule_Type.tp_base = &PyModule_Type; Nuitka_BuiltinModule_Type.tp_dictoffset = PyModule_Type.tp_dictoffset; Nuitka_BuiltinModule_Type.tp_init = PyModule_Type.tp_init; Nuitka_BuiltinModule_Type.tp_alloc = PyModule_Type.tp_alloc; Nuitka_BuiltinModule_Type.tp_new = PyModule_Type.tp_new; Nuitka_BuiltinModule_Type.tp_free = PyModule_Type.tp_free; int ret = PyType_Ready( &Nuitka_BuiltinModule_Type ); assert( ret == 0 ); // Replace type of builtin module to take over. ((PyObject *)builtin_module)->ob_type = &Nuitka_BuiltinModule_Type; assert( PyModule_Check( builtin_module ) == 1 ); } #include "HelpersCalling.c" #if defined(_NUITKA_STANDALONE) || _NUITKA_FROZEN > 0 #ifdef _NUITKA_STANDALONE extern PyObject *const_str_plain___file__; // This is called for each module imported early on. void setEarlyFrozenModulesFileAttribute( void ) { Py_ssize_t ppos = 0; PyObject *key, *value; #if PYTHON_VERSION < 300 PyObject *file_value = MAKE_RELATIVE_PATH( PyString_FromString( "not_present.py" ) ); #else PyObject *file_value = MAKE_RELATIVE_PATH( PyUnicode_FromString( "not_present.py" ) ); #endif assert( file_value ); while( PyDict_Next( PyImport_GetModuleDict(), &ppos, &key, &value ) ) { if ( key != NULL && value != NULL && PyModule_Check( value ) ) { if ( !PyObject_HasAttr( value, const_str_plain___file__ ) ) { PyObject_SetAttr( value, const_str_plain___file__, file_value ); } } } Py_DECREF( file_value ); assert( !ERROR_OCCURRED() ); } #endif #endif PyObject *MAKE_RELATIVE_PATH( PyObject *relative ) { static PyObject *our_path_object = NULL; if ( our_path_object == NULL ) { #if defined( _NUITKA_EXE ) our_path_object = getBinaryDirectoryObject(); #else #if PYTHON_VERSION >= 300 our_path_object = PyUnicode_FromString( getDllDirectory() ); #else our_path_object = PyString_FromString( getDllDirectory() ); #endif #endif } char sep[2] = { SEP, 0 }; #if PYTHON_VERSION < 300 PyObject *result = PyNumber_Add( our_path_object, PyString_FromString( sep ) ); #else PyObject *result = PyNumber_Add( our_path_object, PyUnicode_FromString( sep ) ); #endif assert( result ); #if PYTHON_VERSION < 300 result = PyNumber_InPlaceAdd( result, relative ); #else result = PyNumber_InPlaceAdd( result, relative ); #endif assert( result ); return result; } #ifdef _NUITKA_EXE NUITKA_DEFINE_BUILTIN( type ) NUITKA_DEFINE_BUILTIN( len ) NUITKA_DEFINE_BUILTIN( repr ) NUITKA_DEFINE_BUILTIN( int ) NUITKA_DEFINE_BUILTIN( iter ) #if PYTHON_VERSION < 300 NUITKA_DEFINE_BUILTIN( long ) #else NUITKA_DEFINE_BUILTIN( range ); #endif void _initBuiltinOriginalValues() { NUITKA_ASSIGN_BUILTIN( type ); NUITKA_ASSIGN_BUILTIN( len ); NUITKA_ASSIGN_BUILTIN( range ); NUITKA_ASSIGN_BUILTIN( repr ); NUITKA_ASSIGN_BUILTIN( int ); NUITKA_ASSIGN_BUILTIN( iter ); #if PYTHON_VERSION < 300 NUITKA_ASSIGN_BUILTIN( long ); #endif CHECK_OBJECT( _python_original_builtin_value_range ); } #endif // Used for threading. #if PYTHON_VERSION >= 300 volatile int _Py_Ticker = _Py_CheckInterval; #endif // Reverse operation mapping. static int const swapped_op[] = { Py_GT, Py_GE, Py_EQ, Py_NE, Py_LT, Py_LE }; #if PYTHON_VERSION >= 270 iternextfunc default_iternext; extern PyObject *const_str_plain___iter__; void _initSlotIternext() { PyObject *pos_args = PyTuple_New(1); PyTuple_SET_ITEM( pos_args, 0, (PyObject *)&PyBaseObject_Type ); Py_INCREF( &PyBaseObject_Type ); PyObject *kw_args = PyDict_New(); PyDict_SetItem( kw_args, const_str_plain___iter__, Py_True ); PyObject *c = PyObject_CallFunctionObjArgs( (PyObject *)&PyType_Type, const_str_plain___iter__, pos_args, kw_args, NULL ); Py_DECREF( pos_args ); Py_DECREF( kw_args ); PyObject *r = PyObject_CallFunctionObjArgs( c, NULL ); Py_DECREF( c ); CHECK_OBJECT( r ); assert( Py_TYPE( r )->tp_iternext ); default_iternext = Py_TYPE(r)->tp_iternext; Py_DECREF( r ); } #endif #if PYTHON_VERSION < 300 extern PyObject *const_str_plain___cmp__; static cmpfunc default_tp_compare; void _initSlotCompare() { // Create a class with "__cmp__" attribute, to get a hand at the default // implementation of tp_compare. It's not part of the API and with shared // libraries it's not accessible. The name does not matter, nor does the // actual value used for "__cmp__". // Use "int" as the base class. PyObject *pos_args = PyTuple_New(1); PyTuple_SET_ITEM( pos_args, 0, (PyObject *)&PyInt_Type ); Py_INCREF( &PyInt_Type ); // Use "__cmp__" with true value, won't matter. PyObject *kw_args = PyDict_New(); PyDict_SetItem( kw_args, const_str_plain___cmp__, Py_True ); // Create the type. PyObject *c = PyObject_CallFunctionObjArgs( (PyObject *)&PyType_Type, const_str_plain___cmp__, pos_args, kw_args, NULL ); Py_DECREF( pos_args ); Py_DECREF( kw_args ); PyObject *r = PyObject_CallFunctionObjArgs( c, NULL ); Py_DECREF( c ); CHECK_OBJECT( r ); assert( Py_TYPE( r )->tp_compare ); default_tp_compare = Py_TYPE( r )->tp_compare; Py_DECREF( r ); } #define RICHCOMPARE( t ) ( PyType_HasFeature((t), Py_TPFLAGS_HAVE_RICHCOMPARE) ? (t)->tp_richcompare : NULL ) static inline int adjust_tp_compare( int c ) { if ( PyErr_Occurred() ) { return -2; } else if (c < -1 || c > 1) { return c < -1 ? -1 : 1; } else { return c; } } static inline int coerce_objects( PyObject **pa, PyObject **pb ) { PyObject *a = *pa; PyObject *b = *pb; // Shortcut only for old-style types if ( a->ob_type == b->ob_type && !PyType_HasFeature( a->ob_type, Py_TPFLAGS_CHECKTYPES )) { Py_INCREF( a ); Py_INCREF( b ); return 0; } if ( a->ob_type->tp_as_number && a->ob_type->tp_as_number->nb_coerce ) { int res = (*a->ob_type->tp_as_number->nb_coerce)( pa, pb ); if ( res <= 0 ) { return res; } } if ( b->ob_type->tp_as_number && b->ob_type->tp_as_number->nb_coerce ) { int res = (*b->ob_type->tp_as_number->nb_coerce)( pb, pa ); if ( res <= 0 ) { return res; } } return 1; } static int try_3way_compare( PyObject *a, PyObject *b ) { cmpfunc f1 = a->ob_type->tp_compare; cmpfunc f2 = b->ob_type->tp_compare; int c; // Same compares, just use it. if ( f1 != NULL && f1 == f2 ) { c = (*f1)( a, b ); return adjust_tp_compare( c ); } // If one slot is _PyObject_SlotCompare (which we got our hands on under a // different name in case it's a shared library), prefer it. if ( f1 == default_tp_compare || f2 == default_tp_compare ) { return default_tp_compare( a, b ); } // Try coercion. c = coerce_objects( &a, &b ); if (c < 0) { return -2; } if (c > 0) { return 2; } f1 = a->ob_type->tp_compare; if ( f1 != NULL && f1 == b->ob_type->tp_compare ) { c = (*f1)( a, b ); Py_DECREF( a ); Py_DECREF( b ); return adjust_tp_compare(c); } // No comparison defined. Py_DECREF( a ); Py_DECREF( b ); return 2; } PyObject *MY_RICHCOMPARE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); PyObject *result; // TODO: Type a-ware rich comparison would be really nice, but this is what // CPython does, and should be even in "richcomparisons.h" as the first // thing, so it's even cheaper. if ( PyInt_CheckExact( a ) && PyInt_CheckExact( b )) { long aa, bb; #ifdef __NUITKA_NO_ASSERT__ bool res; #else bool res = false; #endif aa = PyInt_AS_LONG( a ); bb = PyInt_AS_LONG( b ); switch( op ) { case Py_LT: res = aa < bb; break; case Py_LE: res = aa <= bb; break; case Py_EQ: res = aa == bb; break; case Py_NE: res = aa != bb; break; case Py_GT: res = aa > bb; break; case Py_GE: res = aa >= bb; break; default: assert( false ); } result = BOOL_FROM( res ); Py_INCREF( result ); return result; } // TODO: Get hint from recursion control if that's needed. if (unlikely( Py_EnterRecursiveCall((char *)" in cmp") )) { return NULL; } // If the types are equal, we may get away immediately. if ( a->ob_type == b->ob_type && !PyInstance_Check( a ) ) { richcmpfunc frich = RICHCOMPARE( a->ob_type ); if ( frich != NULL ) { result = (*frich)( a, b, op ); if (result != Py_NotImplemented) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } // No rich comparison, but maybe compare works. cmpfunc fcmp = a->ob_type->tp_compare; if ( fcmp != NULL ) { int c = (*fcmp)( a, b ); c = adjust_tp_compare( c ); Py_LeaveRecursiveCall(); if ( c == -2 ) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } result = BOOL_FROM( c != 0 ); Py_INCREF( result ); return result; } } // Fast path was not successful or not taken richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } } f = RICHCOMPARE( a->ob_type ); if ( f != NULL ) { result = (*f)( a, b, op ); if ( result != Py_NotImplemented ) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { Py_LeaveRecursiveCall(); return result; } Py_DECREF( result ); } int c; if ( PyInstance_Check( a ) ) { c = (*a->ob_type->tp_compare)( a, b ); } else if ( PyInstance_Check( b ) ) { c = (*b->ob_type->tp_compare)( a, b ); } else { c = try_3way_compare( a, b ); } if ( c >= 2 ) { if ( a->ob_type == b->ob_type ) { Py_uintptr_t aa = (Py_uintptr_t)a; Py_uintptr_t bb = (Py_uintptr_t)b; c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else if ( a == Py_None ) { // None is smaller than everything else c = -1; } else if ( b == Py_None ) { // None is smaller than everything else c = 1; } else if ( PyNumber_Check( a ) ) { // different type: compare type names but numbers are smaller than // others. if ( PyNumber_Check( b ) ) { // Both numbers, need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else { c = -1; } } else if ( PyNumber_Check( b ) ) { c = 1; } else { int s = strcmp( a->ob_type->tp_name, b->ob_type->tp_name ); if ( s < 0 ) { c = -1; } else if ( s > 0 ) { c = 1; } else { // Same type name need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } } } Py_LeaveRecursiveCall(); if (unlikely( c <= -2 )) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } result = BOOL_FROM( c != 0 ); Py_INCREF( result ); return result; } PyObject *MY_RICHCOMPARE_NORECURSE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); // TODO: Type a-ware rich comparison would be really nice, but this is what // CPython does, and should be even in "richcomparisons.h" as the first // thing, so it's even cheaper. if ( PyInt_CheckExact( a ) && PyInt_CheckExact( b )) { long aa, bb; #ifdef __NUITKA_NO_ASSERT__ bool res; #else bool res = false; #endif aa = PyInt_AS_LONG( a ); bb = PyInt_AS_LONG( b ); switch( op ) { case Py_LT: res = aa < bb; break; case Py_LE: res = aa <= bb; break; case Py_EQ: res = aa == bb; break; case Py_NE: res = aa != bb; break; case Py_GT: res = aa > bb; break; case Py_GE: res = aa >= bb; break; default: assert( false ); } PyObject *result = BOOL_FROM( res ); Py_INCREF( result ); return result; } PyObject *result; // If the types are equal, we may get away immediately. if ( a->ob_type == b->ob_type && !PyInstance_Check( a ) ) { richcmpfunc frich = RICHCOMPARE( a->ob_type ); if ( frich != NULL ) { result = (*frich)( a, b, op ); if (result != Py_NotImplemented) { return result; } Py_DECREF( result ); } // No rich comparison, but maybe compare works. cmpfunc fcmp = a->ob_type->tp_compare; if ( fcmp != NULL ) { int c = (*fcmp)( a, b ); c = adjust_tp_compare( c ); if ( c == -2 ) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } result = BOOL_FROM( c != 0 ); Py_INCREF( result ); return result; } } // Fast path was not successful or not taken richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { return result; } Py_DECREF( result ); } } f = RICHCOMPARE( a->ob_type ); if ( f != NULL ) { result = (*f)( a, b, op ); if ( result != Py_NotImplemented ) { return result; } Py_DECREF( result ); } f = RICHCOMPARE( b->ob_type ); if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if ( result != Py_NotImplemented ) { return result; } Py_DECREF( result ); } int c; if ( PyInstance_Check( a ) ) { c = (*a->ob_type->tp_compare)( a, b ); } else if ( PyInstance_Check( b ) ) { c = (*b->ob_type->tp_compare)( a, b ); } else { c = try_3way_compare( a, b ); } if ( c >= 2 ) { if ( a->ob_type == b->ob_type ) { Py_uintptr_t aa = (Py_uintptr_t)a; Py_uintptr_t bb = (Py_uintptr_t)b; c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else if ( a == Py_None ) { // None is smaller than everything else c = -1; } else if ( b == Py_None ) { // None is smaller than everything else c = 1; } else if ( PyNumber_Check( a ) ) { // different type: compare type names but numbers are smaller than // others. if ( PyNumber_Check( b ) ) { // Both numbers, need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } else { c = -1; } } else if ( PyNumber_Check( b ) ) { c = 1; } else { int s = strcmp( a->ob_type->tp_name, b->ob_type->tp_name ); if ( s < 0 ) { c = -1; } else if ( s > 0 ) { c = 1; } else { // Same type name need to make a decision based on types. Py_uintptr_t aa = (Py_uintptr_t)Py_TYPE( a ); Py_uintptr_t bb = (Py_uintptr_t)Py_TYPE( b ); c = ( aa < bb ) ? -1 : ( aa > bb ) ? 1 : 0; } } } if (unlikely( c <= -2 )) { return NULL; } switch( op ) { case Py_LT: c = c < 0; break; case Py_LE: c = c <= 0; break; case Py_EQ: c = c == 0; break; case Py_NE: c = c != 0; break; case Py_GT: c = c > 0; break; case Py_GE: c = c >= 0; break; } result = BOOL_FROM( c != 0 ); Py_INCREF( result ); return result; } #else // Table for operation names as strings. static char const *op_strings[] = { "<", "<=", "==", "!=", ">", ">=" }; PyObject *MY_RICHCOMPARE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); if (unlikely( Py_EnterRecursiveCall( (char *)" in comparison" ) )) { return NULL; } bool checked_reverse_op = false; PyObject *result = NULL; richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { checked_reverse_op = true; result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { Py_LeaveRecursiveCall(); return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL ) { f = a->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( a, b, op ); if (unlikely( result == NULL )) { Py_LeaveRecursiveCall(); return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL && checked_reverse_op == false ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { Py_LeaveRecursiveCall(); return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } Py_LeaveRecursiveCall(); if ( result != NULL ) { return result; } // If it is not implemented, do identify checks as "==" and "!=" and // otherwise give an error if ( op == Py_EQ ) { result = BOOL_FROM( a == b ); Py_INCREF( result ); return result; } else if ( op == Py_NE ) { result = BOOL_FROM( a != b ); Py_INCREF( result ); return result; } else { #if PYTHON_VERSION < 360 PyErr_Format( PyExc_TypeError, "unorderable types: %s() %s %s()", a->ob_type->tp_name, op_strings[ op ], b->ob_type->tp_name ); #else PyErr_Format( PyExc_TypeError, "'%s' not supported between instances of '%s' and '%s'", op_strings[ op ], a->ob_type->tp_name, b->ob_type->tp_name ); #endif return NULL; } } PyObject *MY_RICHCOMPARE_NORECURSE( PyObject *a, PyObject *b, int op ) { CHECK_OBJECT( a ); CHECK_OBJECT( b ); bool checked_reverse_op = false; PyObject *result = NULL; richcmpfunc f; if ( a->ob_type != b->ob_type && PyType_IsSubtype( b->ob_type, a->ob_type ) ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { checked_reverse_op = true; result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL ) { f = a->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( a, b, op ); if (unlikely( result == NULL )) { return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result == NULL && checked_reverse_op == false ) { f = b->ob_type->tp_richcompare; if ( f != NULL ) { result = (*f)( b, a, swapped_op[ op ] ); if (unlikely( result == NULL )) { return NULL; } if ( result == Py_NotImplemented ) { Py_DECREF( result ); result = NULL; } } } if ( result != NULL ) { return result; } // If it is not implemented, do identify checks as "==" and "!=" and // otherwise give an error if ( op == Py_EQ ) { result = BOOL_FROM( a == b ); Py_INCREF( result ); return result; } else if ( op == Py_NE ) { result = BOOL_FROM( a != b ); Py_INCREF( result ); return result; } else { #if PYTHON_VERSION < 360 PyErr_Format( PyExc_TypeError, "unorderable types: %s() %s %s()", a->ob_type->tp_name, op_strings[ op ], b->ob_type->tp_name ); #else PyErr_Format( PyExc_TypeError, "'%s' not supported between instances of '%s' and '%s'", op_strings[ op ], a->ob_type->tp_name, b->ob_type->tp_name ); #endif return NULL; } } #endif #include "HelpersDeepcopy.c" #if _NUITKA_PROFILE #include "HelpersProfiling.c" #endif Nuitka-0.5.28.2/nuitka/build/static_src/HelpersStrings.c0000644000372000001440000000763413134660221023343 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /* This helpers is used to quickly create a string object from C char. Currently this is used for string subscript code, but may also be used for the "char" C type in the future. */ PyObject *STRING_FROM_CHAR( unsigned char c ) { // TODO: A switch statement might be faster, because no object needs to be // created at all, this here is how CPython does it. char s[1]; s[0] = (char)c; #if PYTHON_VERSION < 300 return PyString_FromStringAndSize( s, 1 ); #else return PyUnicode_FromStringAndSize( s, 1 ); #endif } /* The "chr" built-in. This could also use a table for the interned single char strings, to be faster on Python2. For Python3 no such table is reasonable. */ PyObject *BUILTIN_CHR( PyObject *value ) { long x = PyInt_AsLong( value ); #if PYTHON_VERSION < 300 if ( x < 0 || x >= 256 ) { PyErr_Format( PyExc_ValueError, "chr() arg not in range(256)" ); return NULL; } // TODO: A switch statement might be faster, because no object needs to be // created at all, this is how CPython does it. char s[1]; s[0] = (char)x; return PyString_FromStringAndSize( s, 1 ); #else PyObject *result = PyUnicode_FromOrdinal( x ); if (unlikely( result == NULL )) { return NULL; } assert( PyUnicode_Check( result )); return result; #endif } /* The "ord" built-in. */ PyObject *BUILTIN_ORD( PyObject *value ) { long result; if (likely( PyBytes_Check( value ) )) { Py_ssize_t size = PyBytes_GET_SIZE( value ); if (likely( size == 1 )) { result = (long)( ((unsigned char *)PyBytes_AS_STRING( value ))[0] ); } else { PyErr_Format( PyExc_TypeError, "ord() expected a character, but string of length %zd found", size ); return NULL; } } else if ( PyByteArray_Check( value ) ) { Py_ssize_t size = PyByteArray_GET_SIZE( value ); if (likely( size == 1 )) { result = (long)( ((unsigned char *)PyByteArray_AS_STRING( value ))[0] ); } else { PyErr_Format( PyExc_TypeError, "ord() expected a character, but byte array of length %zd found", size ); return NULL; } } else if ( PyUnicode_Check( value ) ) { #if PYTHON_VERSION >= 330 if (unlikely( PyUnicode_READY( value ) == -1 )) { return NULL; } Py_ssize_t size = PyUnicode_GET_LENGTH( value ); #else Py_ssize_t size = PyUnicode_GET_SIZE( value ); #endif if (likely( size == 1 )) { #if PYTHON_VERSION >= 330 result = (long)( PyUnicode_READ_CHAR( value, 0 ) ); #else result = (long)( *PyUnicode_AS_UNICODE( value ) ); #endif } else { PyErr_Format( PyExc_TypeError, "ord() expected a character, but unicode string of length %zd found", size ); return NULL; } } else { PyErr_Format( PyExc_TypeError, "ord() expected string of length 1, but %s found", Py_TYPE( value )->tp_name ); return NULL; } return PyInt_FromLong( result ); } Nuitka-0.5.28.2/nuitka/build/static_src/MainProgram.c0000644000372000001440000004026713207537242022611 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /* The main program for a compiled program. * * It needs to prepare the interpreter and then loads and executes * the "__main__" module. * * This is currently still C++ code, but should become C code eventually. Maybe * this finished sooner than others. * */ #include "nuitka/prelude.h" #include "structseq.h" #include "osdefs.h" #if _NUITKA_NO_WARNINGSSYSFLAGS extern PyObject *const_str_plain_ignore; #endif extern PyCodeObject *codeobj_main; extern PyObject *const_str_plain___main__; #if _NUITKA_NO_PYTHON_WARNINGS extern PyObject *const_str_plain_ignore; #endif /* For later use in "Py_GetArgcArgv" */ static char **orig_argv; static int orig_argc; #if PYTHON_VERSION >= 300 static wchar_t **argv_unicode; #endif #ifdef _NUITKA_STANDALONE static char *original_home; static char *original_path; #if defined( _WIN32 ) static void setenv( char const *name, char const *value, int overwrite) { assert( overwrite ); char buffer[MAXPATHLEN + 100]; memset( buffer, 0, sizeof(buffer) ); snprintf( buffer, sizeof(buffer) - 1, "%s=%s", name, value ? value : "" ); NUITKA_MAY_BE_UNUSED int res = _putenv( buffer ); assert( res == 0 ); } static void unsetenv( char const *name ) { setenv( name, NULL, 1 ); } #endif #if _NUITKA_FROZEN > 0 extern void copyFrozenModulesTo( struct _frozen *destination ); #endif static void prepareStandaloneEnvironment() { // Tell the CPython library to use our pre-compiled modules as frozen // modules. This for those modules/packages like "encoding" that will be // loaded during "Py_Initialize" already, for the others they may be // compiled. #if _NUITKA_FROZEN > 0 // The CPython library has some pre-existing frozen modules, we only append // to that. struct _frozen const *search = PyImport_FrozenModules; while( search->name ) { search++; } int pre_existing_count = (int)( search - PyImport_FrozenModules ); /* Allocate new memory and merge the tables. Keeping the old ones has * the advantage that e.g. "import this" is going to work well. */ struct _frozen *merged = (struct _frozen *)malloc( sizeof(struct _frozen) * (_NUITKA_FROZEN + pre_existing_count + 1) ); memcpy( merged, PyImport_FrozenModules, pre_existing_count * sizeof( struct _frozen ) ); copyFrozenModulesTo(merged + pre_existing_count); PyImport_FrozenModules = merged; #endif /* Setup environment variables to tell CPython that we would like it to use * the provided binary directory as the place to look for DLLs and for * extension modules. */ char *binary_directory = getBinaryDirectoryHostEncoded(); #if defined( _WIN32 ) && defined( _MSC_VER ) SetDllDirectory( binary_directory ); #endif /* get original environment variable values */ original_home = getenv( "PYTHONHOME" ); original_path = getenv( "PYTHONPATH" ); assert( binary_directory != NULL ); assert( strlen( binary_directory ) > 0 ); NUITKA_PRINTF_TRACE("Binary dir is %s\n", binary_directory ); setenv( "PYTHONHOME", binary_directory, 1 ); // This has really failed before, on Windows. assert( getenv( "PYTHONHOME") != NULL ); assert( strcmp( binary_directory, getenv("PYTHONHOME") ) == 0 ); unsetenv( "PYTHONPATH" ); #if PYTHON_VERSION >= 300 wchar_t binary_directory2[MAXPATHLEN+1]; mbstowcs( binary_directory2, binary_directory, MAXPATHLEN ); Py_SetPath( binary_directory2 ); #endif } #if PYTHON_VERSION < 300 #define PY_FORMAT_GETPATH_RESULT "%s" #else #define PY_FORMAT_GETPATH_RESULT "%ls" #endif static void restoreStandaloneEnvironment() { /* Make use the PYTHONHOME set previously. */ NUITKA_PRINTF_TRACE("Path is '" PY_FORMAT_GETPATH_RESULT "' (PYTHONHOME %s)\n", Py_GetPath(), getenv( "PYTHONHOME" ) ); Py_GetPath(); // Restore PYTHONHOME and PYTHONPATH, so spawning executables of Python // will still work as expected. if ( original_home == NULL ) { unsetenv( "PYTHONHOME" ); } else { setenv( "PYTHONHOME", original_home, 1 ); } if ( original_path != NULL ) { setenv( "PYTHONPATH", original_path, 1 ); } // Emulate the effect of Py_SetPath for Python2, and cleanup the duplicate // produced for Python3. We do not want to have outside locations in the // "sys.path", this removes them reliably. For Python2 it's relatively late // but ought to be good enough still. char *binary_directory = getBinaryDirectoryHostEncoded(); #if PYTHON_VERSION < 300 PySys_SetPath( binary_directory ); #else wchar_t binary_directory2[MAXPATHLEN+1]; mbstowcs( binary_directory2, binary_directory, MAXPATHLEN ); PySys_SetPath( binary_directory2 ); #endif NUITKA_PRINTF_TRACE("Path is '" PY_FORMAT_GETPATH_RESULT "'.\n", Py_GetPath() ); } #endif extern void _initCompiledCellType(); extern void _initCompiledGeneratorType(); extern void _initCompiledFunctionType(); extern void _initCompiledMethodType(); extern void _initCompiledFrameType(); #if PYTHON_VERSION >= 350 extern void _initCompiledCoroutineTypes(); #endif #if PYTHON_VERSION >= 360 extern void _initCompiledAsyncgenTypes(); #endif #if defined(_NUITKA_CONSTANTS_FROM_RESOURCE) unsigned char const* constant_bin = NULL; #endif #ifdef _NUITKA_WINMAIN_ENTRY_POINT int __stdcall WinMain( HINSTANCE hInstance, HINSTANCE hPrevInstance, char* lpCmdLine, int nCmdShow ) { #if defined(__MINGW32__) && !defined(_W64) /* MINGW32 */ int argc = _argc; char** argv = _argv; #else /* MSVC, MINGW64 */ int argc = __argc; char** argv = __argv; #endif #else int main( int argc, char **argv ) { #endif NUITKA_PRINT_TRACE("main(): Entered."); orig_argv = argv; orig_argc = argc; #ifdef __FreeBSD__ /* 754 requires that FP exceptions run in "no stop" mode by default, and * * until C vendors implement C99's ways to control FP exceptions, Python * requires non-stop mode. Alas, some platforms enable FP exceptions by * default. Here we disable them. */ fp_except_t m; m = fpgetmask(); fpsetmask( m & ~FP_X_OFL ); #endif /* On Windows we support loading the constants blob from an embedded * resource. On Linux, where possible this is done automatically by * the linker already. */ #if defined(_NUITKA_CONSTANTS_FROM_RESOURCE) NUITKA_PRINT_TRACE("main(): Loading constants blob from Windows resource."); constant_bin = (const unsigned char*)LockResource( LoadResource( NULL, FindResource(NULL, MAKEINTRESOURCE(3), RT_RCDATA) ) ); assert( constant_bin ); #endif #ifdef _NUITKA_STANDALONE NUITKA_PRINT_TRACE("main(): Prepare standalone environment."); prepareStandaloneEnvironment(); #else /* For Python installations that need the PYTHONHOME set, we inject it back here. */ #if defined(PYTHON_HOME_PATH) NUITKA_PRINT_TRACE("main(): Prepare run environment PYTHONHOME."); { char buffer[MAXPATHLEN+10]; strcpy(buffer, "PYTHONHOME="); strcat(buffer, PYTHON_HOME_PATH); int res = putenv(buffer); assert( res == 0 ); } #endif #endif /* Initialize CPython library environment. */ Py_DebugFlag = 0; #if PYTHON_VERSION < 300 Py_Py3kWarningFlag = _NUITKA_SYSFLAG_PY3K_WARNING; Py_DivisionWarningFlag = _NUITKA_SYSFLAG_DIVISION_WARNING; Py_UnicodeFlag = _NUITKA_SYSFLAG_UNICODE; Py_TabcheckFlag = 0; #endif Py_InspectFlag = 0; Py_InteractiveFlag = 0; Py_OptimizeFlag = 0; Py_DontWriteBytecodeFlag = 0; Py_NoUserSiteDirectory = _NUITKA_SYSFLAG_NO_SITE; Py_IgnoreEnvironmentFlag = 0; Py_VerboseFlag = _NUITKA_SYSFLAG_VERBOSE; Py_BytesWarningFlag = _NUITKA_SYSFLAG_BYTES_WARNING; #if _NUITKA_SYSFLAG_BYTES_WARNING Py_HashRandomizationFlag = 1; #endif /* This suppresses warnings from getpath.c */ Py_FrozenFlag = 1; /* We want to import the site module, but only after we finished our own * setup. The site module import will be the first thing, the main module * does. */ Py_NoSiteFlag = 1; /* Initial command line handling only. */ NUITKA_PRINT_TRACE("main(): Calling convert/setCommandLineParameters."); #if PYTHON_VERSION >= 300 argv_unicode = convertCommandLineParameters( argc, argv ); #endif #if PYTHON_VERSION < 300 bool is_multiprocess_forking = setCommandLineParameters( argc, argv, true ); #else bool is_multiprocess_forking = setCommandLineParameters( argc, argv_unicode, true ); #endif /* Initialize the embedded CPython interpreter. */ NUITKA_PRINT_TRACE("main(): Calling Py_Initialize to initialize interpreter."); Py_Initialize(); /* Lie about it, believe it or not, there are "site" files, that check * against later imports, see below. */ Py_NoSiteFlag = _NUITKA_SYSFLAG_NO_SITE; /* Set the command line parameters for run time usage. */ NUITKA_PRINT_TRACE("main(): Calling setCommandLineParameters."); #if PYTHON_VERSION < 300 setCommandLineParameters( argc, argv, false ); #else setCommandLineParameters( argc, argv_unicode, false ); #endif #ifdef _NUITKA_STANDALONE NUITKA_PRINT_TRACE("main(): Restore standalone environment."); restoreStandaloneEnvironment(); #endif /* Initialize the built-in module tricks used. */ NUITKA_PRINT_TRACE("main(): Calling _initBuiltinModule()."); _initBuiltinModule(); /* Initialize the Python constant values used. This also sets * "sys.executable" while at it. */ NUITKA_PRINT_TRACE("main(): Calling createGlobalConstants()."); createGlobalConstants(); NUITKA_PRINT_TRACE("main(): Calling _initBuiltinOriginalValues()."); _initBuiltinOriginalValues(); /* Revert the wrong "sys.flags" value, it's used by "site" on at least * Debian for Python 3.3, more uses may exist. */ #if _NUITKA_SYSFLAG_NO_SITE == 0 #if PYTHON_VERSION >= 330 PyStructSequence_SetItem( PySys_GetObject( "flags" ), 6, const_int_0 ); #elif PYTHON_VERSION >= 320 PyStructSequence_SetItem( PySys_GetObject( "flags" ), 7, const_int_0 ); #elif PYTHON_VERSION >= 260 PyStructSequence_SET_ITEM( PySys_GetObject( (char *)"flags" ), 9, const_int_0 ); #endif #endif /* Initialize the compiled types of Nuitka. */ _initCompiledCellType(); _initCompiledGeneratorType(); _initCompiledFunctionType(); _initCompiledMethodType(); _initCompiledFrameType(); #if PYTHON_VERSION >= 350 _initCompiledCoroutineTypes(); #endif #if PYTHON_VERSION >= 360 _initCompiledAsyncgenTypes(); #endif #if PYTHON_VERSION < 300 _initSlotCompare(); #endif #if PYTHON_VERSION >= 270 _initSlotIternext(); #endif NUITKA_PRINT_TRACE("main(): Calling enhancePythonTypes()."); enhancePythonTypes(); NUITKA_PRINT_TRACE("main(): Calling patchBuiltinModule()."); patchBuiltinModule(); NUITKA_PRINT_TRACE("main(): Calling patchTypeComparison()."); patchTypeComparison(); NUITKA_PRINT_TRACE("main(): Calling patchTracebackDealloc()."); patchTracebackDealloc(); /* Allow to override the ticker value, to remove checks for threads in * CPython core from impact on benchmarks. */ char const *ticker_value = getenv( "NUITKA_TICKER" ); if ( ticker_value != NULL ) { _Py_Ticker = atoi( ticker_value ); assert ( _Py_Ticker >= 20 ); } #ifdef _NUITKA_STANDALONE NUITKA_PRINT_TRACE("main(): Calling setEarlyFrozenModulesFileAttribute()."); #if PYTHON_VERSION >= 300 PyObject *os_module = PyImport_ImportModule("os"); CHECK_OBJECT( os_module ); #endif setEarlyFrozenModulesFileAttribute(); #endif NUITKA_PRINT_TRACE("main(): Calling setupMetaPathBasedLoader()."); /* Enable meta path based loader. */ setupMetaPathBasedLoader(); _PyWarnings_Init(); /* Disable CPython warnings if requested to. */ #if _NUITKA_NO_PYTHON_WARNINGS /* Should be same as: * * warnings.simplefilter("ignore", UserWarning) * warnings.simplefilter("ignore", DeprecationWarning) * There is no C-API to control warnings. We don't care if it actually * works, i.e. return code of "simplefilter" function is not checked. */ { PyObject *warnings = PyImport_ImportModule( "warnings" ); if ( warnings != NULL ) { PyObject *simplefilter = PyObject_GetAttrString( warnings, "simplefilter" ); if ( simplefilter != NULL ) { PyObject *result1 = PyObject_CallFunctionObjArgs( simplefilter, const_str_plain_ignore, PyExc_UserWarning, NULL ); assert( result1 ); Py_XDECREF( result1 ); PyObject *result2 = PyObject_CallFunctionObjArgs( simplefilter, const_str_plain_ignore, PyExc_DeprecationWarning, NULL ); assert( result2 ); Py_XDECREF( result2 ); } } } #endif #if PYTHON_VERSION >= 300 NUITKA_PRINT_TRACE("main(): Calling patchInspectModule()."); patchInspectModule(); #endif #if _NUITKA_PROFILE startProfiling(); #endif /* Execute the main module. In case of multiprocessing making a fork on * Windows, we should execute something else instead. */ #if _NUITKA_MODULE_COUNT > 1 if (unlikely( is_multiprocess_forking )) { NUITKA_PRINT_TRACE("main(): Calling __parents_main__."); IMPORT_EMBEDDED_MODULE(PyUnicode_FromString("__parents_main__"), "__parents_main__"); } else #endif { assert( !is_multiprocess_forking ); NUITKA_PRINT_TRACE("main(): Calling __main__."); /* Execute the "__main__" module. */ PyDict_DelItem(PySys_GetObject((char *)"modules"), const_str_plain___main__); IMPORT_EMBEDDED_MODULE(const_str_plain___main__, "__main__"); } #if _NUITKA_PROFILE stopProfiling(); #endif #ifndef __NUITKA_NO_ASSERT__ checkGlobalConstants(); /* TODO: Walk over all loaded compiled modules, and make this kind of checks. */ #if 0 checkModuleConstants___main__(); #endif #endif if ( ERROR_OCCURRED() ) { #if PYTHON_VERSION >= 330 /* Remove the frozen importlib traceback part, which would not be compatible. */ PyThreadState *thread_state = PyThreadState_GET(); while( thread_state->curexc_traceback ) { PyTracebackObject *tb = (PyTracebackObject *)thread_state->curexc_traceback; PyFrameObject *frame = tb->tb_frame; if ( 0 == strcmp( PyUnicode_AsUTF8( frame->f_code->co_filename ), "" ) ) { thread_state->curexc_traceback = (PyObject *)tb->tb_next; Py_INCREF( tb->tb_next ); continue; } break; } #endif PyErr_PrintEx( 0 ); Py_Exit( 1 ); } else { Py_Exit( 0 ); } /* The above branches both do "Py_Exit()" calls which are not supposed to * return. */ NUITKA_CANNOT_GET_HERE( main ); } /* This is an inofficial API, not available on Windows, but on Linux and others * it is exported, and has been used by some code. */ #ifndef _WIN32 #ifdef __cplusplus extern "C" { #endif #if PYTHON_VERSION >= 300 #if defined( __GNUC__ ) __attribute__(( visibility( "default" ))) #endif void Py_GetArgcArgv( int *argc, wchar_t ***argv ) { *argc = orig_argc; *argv = argv_unicode; } #else #if defined( __GNUC__ ) __attribute__(( visibility( "default" ))) #endif void Py_GetArgcArgv( int *argc, char ***argv ) { *argc = orig_argc; *argv = orig_argv; } #endif #ifdef __cplusplus } #endif #endif Nuitka-0.5.28.2/nuitka/build/static_src/InspectPatcher.c0000644000372000001440000003077013134660221023300 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /** * This is responsible for updating parts of CPython to better work with Nuitka * by replacing CPython implementations with enhanced versions. */ #include "nuitka/prelude.h" #if PYTHON_VERSION >= 300 extern PyObject *const_str_plain_inspect; extern PyObject *const_str_plain_site; extern PyObject *const_int_0; static PyObject *module_inspect; #if PYTHON_VERSION >= 350 static PyObject *module_types; extern PyObject *const_str_plain_types; #endif static char *kwlist[] = { (char *)"object", NULL }; static PyObject *old_getgeneratorstate = NULL; static PyObject *_inspect_getgeneratorstate_replacement( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *object; if ( !PyArg_ParseTupleAndKeywords( args, kwds, "O:getgeneratorstate", kwlist, &object, NULL )) { return NULL; } if ( Nuitka_Generator_Check( object ) ) { struct Nuitka_GeneratorObject *generator = (struct Nuitka_GeneratorObject *)object; if ( generator->m_running ) { return PyObject_GetAttrString( module_inspect, "GEN_RUNNING" ); } else if ( generator->m_status == status_Finished ) { return PyObject_GetAttrString( module_inspect, "GEN_CLOSED" ); } else if ( generator->m_status == status_Unused ) { return PyObject_GetAttrString( module_inspect, "GEN_CREATED" ); } else { return PyObject_GetAttrString( module_inspect, "GEN_SUSPENDED" ); } } else { return old_getgeneratorstate->ob_type->tp_call( old_getgeneratorstate, args, kwds ); } } #if PYTHON_VERSION >= 350 static PyObject *old_getcoroutinestate = NULL; static PyObject *_inspect_getcoroutinestate_replacement( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *object; if ( !PyArg_ParseTupleAndKeywords( args, kwds, "O:getcoroutinestate", kwlist, &object, NULL )) { return NULL; } if ( Nuitka_Coroutine_Check( object ) ) { struct Nuitka_CoroutineObject *coroutine = (struct Nuitka_CoroutineObject *)object; if ( coroutine->m_running ) { return PyObject_GetAttrString( module_inspect, "CORO_RUNNING" ); } else if ( coroutine->m_status == status_Finished ) { return PyObject_GetAttrString( module_inspect, "CORO_CLOSED" ); } else if ( coroutine->m_status == status_Unused ) { return PyObject_GetAttrString( module_inspect, "CORO_CREATED" ); } else { return PyObject_GetAttrString( module_inspect, "CORO_SUSPENDED" ); } } else { return old_getcoroutinestate->ob_type->tp_call( old_getcoroutinestate, args, kwds ); } } static PyObject *old_types_coroutine = NULL; static char *kwlist_func[] = { (char *)"func", NULL }; static PyObject *_types_coroutine_replacement( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *func; if ( !PyArg_ParseTupleAndKeywords( args, kwds, "O:coroutine", kwlist_func, &func, NULL )) { return NULL; } if ( Nuitka_Function_Check( func ) ) { struct Nuitka_FunctionObject *function = (struct Nuitka_FunctionObject *)func; if ( function->m_code_object->co_flags & CO_GENERATOR ) { function->m_code_object->co_flags |= 0x100; } } return old_types_coroutine->ob_type->tp_call( old_types_coroutine, args, kwds ); } #endif #endif #if PYTHON_VERSION >= 300 static PyMethodDef _method_def_inspect_getgeneratorstate_replacement = { "getgeneratorstate", (PyCFunction)_inspect_getgeneratorstate_replacement, METH_VARARGS | METH_KEYWORDS, NULL }; #if PYTHON_VERSION >= 350 static PyMethodDef _method_def_inspect_getcoroutinestate_replacement = { "getcoroutinestate", (PyCFunction)_inspect_getcoroutinestate_replacement, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_types_coroutine_replacement = { "coroutine", (PyCFunction)_types_coroutine_replacement, METH_VARARGS | METH_KEYWORDS, NULL }; #endif /* Replace inspect functions with ones that handle compiles types too. */ void patchInspectModule( void ) { #if PYTHON_VERSION >= 300 #ifdef _NUITKA_EXE // May need to import the "site" module, because otherwise the patching can // fail with it being unable to load it. if ( Py_NoSiteFlag == 0 ) { PyObject *site_module = IMPORT_MODULE5( const_str_plain_site, Py_None, Py_None, const_tuple_empty, const_int_0 ); if ( site_module == NULL ) { // Ignore "ImportError", having a "site" module is not a must. CLEAR_ERROR_OCCURRED(); } } #endif module_inspect = IMPORT_MODULE5( const_str_plain_inspect, Py_None, Py_None, const_tuple_empty, const_int_0 ); if ( module_inspect == NULL ) { PyErr_PrintEx( 0 ); Py_Exit( 1 ); } CHECK_OBJECT( module_inspect ); // Patch "inspect.getgeneratorstate" unless it is already patched. old_getgeneratorstate = PyObject_GetAttrString( module_inspect, "getgeneratorstate" ); CHECK_OBJECT( old_getgeneratorstate ); if ( PyFunction_Check( old_getgeneratorstate ) ) { PyObject *inspect_getgeneratorstate_replacement = PyCFunction_New( &_method_def_inspect_getgeneratorstate_replacement, NULL ); CHECK_OBJECT( inspect_getgeneratorstate_replacement ); PyObject_SetAttrString( module_inspect, "getgeneratorstate", inspect_getgeneratorstate_replacement ); } #if PYTHON_VERSION >= 350 // Patch "inspect.getcoroutinestate" unless it is already patched. old_getcoroutinestate = PyObject_GetAttrString( module_inspect, "getcoroutinestate" ); CHECK_OBJECT( old_getcoroutinestate ); if ( PyFunction_Check( old_getcoroutinestate ) ) { PyObject *inspect_getcoroutinestate_replacement = PyCFunction_New( &_method_def_inspect_getcoroutinestate_replacement, NULL ); CHECK_OBJECT( inspect_getcoroutinestate_replacement ); PyObject_SetAttrString( module_inspect, "getcoroutinestate", inspect_getcoroutinestate_replacement ); } module_types = IMPORT_MODULE5( const_str_plain_types, Py_None, Py_None, const_tuple_empty, const_int_0 ); if ( module_types == NULL ) { PyErr_PrintEx( 0 ); Py_Exit( 1 ); } CHECK_OBJECT( module_types ); // Patch "types.coroutine" unless it is already patched. old_types_coroutine = PyObject_GetAttrString( module_types, "coroutine" ); CHECK_OBJECT( old_types_coroutine ); if ( PyFunction_Check( old_types_coroutine ) ) { PyObject *types_coroutine_replacement = PyCFunction_New( &_method_def_types_coroutine_replacement, NULL ); CHECK_OBJECT( types_coroutine_replacement ); PyObject_SetAttrString( module_types, "coroutine", types_coroutine_replacement ); } static char const *wrapper_enhancement_code = "\n\ import types\n\ _old_GeneratorWrapper = types._GeneratorWrapper\n\ class GeneratorWrapperEnhanced(_old_GeneratorWrapper):\n\ def __init__(self, gen):\n\ _old_GeneratorWrapper.__init__(self, gen)\n\ \n\ if hasattr(gen, 'gi_code'):\n\ if gen.gi_code.co_flags & 0x0020:\n\ self._GeneratorWrapper__isgen = True\n\ \n\ types._GeneratorWrapper = GeneratorWrapperEnhanced\ "; PyObject *wrapper_enhencement_codeobject = Py_CompileString( wrapper_enhancement_code, "", Py_file_input ); CHECK_OBJECT( wrapper_enhencement_codeobject ); PyImport_ExecCodeModuleEx( "_types_patch", wrapper_enhencement_codeobject, "" ); #endif #endif } #endif extern int Nuitka_IsInstance( PyObject *inst, PyObject *cls ); extern PyObject *original_isinstance; static PyObject *_builtin_isinstance_replacement( PyObject *self, PyObject *args ) { PyObject *inst, *cls; if (unlikely( PyArg_UnpackTuple(args, "isinstance", 2, 2, &inst, &cls) == 0 )) { return NULL; } int res = Nuitka_IsInstance( inst, cls ); if (unlikely( res < 0 )) { return NULL; } PyObject *result = BOOL_FROM( res != 0 ); Py_INCREF( result ); return result; } static PyMethodDef _method_def_builtin_isinstance_replacement = { "isinstance", (PyCFunction)_builtin_isinstance_replacement, METH_VARARGS, NULL }; extern PyModuleObject *builtin_module; void patchBuiltinModule() { #if defined(_NUITKA_MODULE) static bool init_done = false; if (init_done == true) return; init_done = true; #endif CHECK_OBJECT( (PyObject *)builtin_module ); // Patch "inspect.isinstance" unless it is already patched. original_isinstance = PyObject_GetAttrString( (PyObject *)builtin_module, "isinstance" ); CHECK_OBJECT( original_isinstance ); // Copy the doc attribute over, needed for "inspect.signature" at least. if ( PyCFunction_Check( original_isinstance )) { _method_def_builtin_isinstance_replacement.ml_doc = ((PyCFunctionObject *)original_isinstance)->m_ml->ml_doc; } PyObject *builtin_isinstance_replacement = PyCFunction_New( &_method_def_builtin_isinstance_replacement, NULL ); CHECK_OBJECT( builtin_isinstance_replacement ); PyObject_SetAttrString( (PyObject *)builtin_module, "isinstance", builtin_isinstance_replacement ); } static richcmpfunc original_PyType_tp_richcompare = NULL; static PyObject *Nuitka_type_tp_richcompare( PyObject *a, PyObject *b, int op ) { if (likely( op == Py_EQ || op == Py_NE )) { if ( a == (PyObject *)&Nuitka_Function_Type ) { a = (PyObject *)&PyFunction_Type; } else if ( a == (PyObject *)&Nuitka_Method_Type ) { a = (PyObject *)&PyMethod_Type; } else if ( a == (PyObject *)&Nuitka_Generator_Type ) { a = (PyObject *)&PyGen_Type; } if ( b == (PyObject *)&Nuitka_Function_Type ) { b = (PyObject *)&PyFunction_Type; } else if ( b == (PyObject *)&Nuitka_Method_Type ) { b = (PyObject *)&PyMethod_Type; } else if ( b == (PyObject *)&Nuitka_Generator_Type ) { b = (PyObject *)&PyGen_Type; } } CHECK_OBJECT( a ); CHECK_OBJECT( b ); assert( original_PyType_tp_richcompare ); return original_PyType_tp_richcompare( a, b, op ); } void patchTypeComparison() { if ( original_PyType_tp_richcompare == NULL ) { original_PyType_tp_richcompare = PyType_Type.tp_richcompare; PyType_Type.tp_richcompare = Nuitka_type_tp_richcompare; } } #include "nuitka/freelists.h" #define MAX_TRACEBACK_FREE_LIST_COUNT 1000 static PyTracebackObject *free_list_tracebacks = NULL; static int free_list_tracebacks_count = 0; // Create a traceback for a given frame, using a freelist hacked into the // existing type. PyTracebackObject *MAKE_TRACEBACK( struct Nuitka_FrameObject *frame, int lineno ) { CHECK_OBJECT( frame ); assert( lineno != 0 ); PyTracebackObject *result; allocateFromFreeListFixed( free_list_tracebacks, PyTracebackObject, PyTraceBack_Type ); result->tb_next = NULL; result->tb_frame = (PyFrameObject *)frame; Py_INCREF( frame ); result->tb_lasti = 0; result->tb_lineno = lineno; Nuitka_GC_Track( result ); return result; } static void Nuitka_tb_dealloc( PyTracebackObject *tb ) { Nuitka_GC_UnTrack( tb ); // TODO: This seems to clash with our free list implementation. // Py_TRASHCAN_SAFE_BEGIN( tb ) Py_XDECREF( tb->tb_next ); Py_XDECREF( tb->tb_frame ); releaseToFreeList( free_list_tracebacks, tb, MAX_TRACEBACK_FREE_LIST_COUNT ); // Py_TRASHCAN_SAFE_END( tb ) } void patchTracebackDealloc( void ) { PyTraceBack_Type.tp_dealloc = (destructor)Nuitka_tb_dealloc; } Nuitka-0.5.28.2/nuitka/build/static_src/CompiledGeneratorType.c0000644000372000001440000013702513207537242024641 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/freelists.h" static PyObject *Nuitka_Generator_tp_repr( struct Nuitka_GeneratorObject *generator ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( "", Nuitka_String_AsString( generator->m_name ), generator ); #else return PyUnicode_FromFormat( "", #if PYTHON_VERSION < 350 Nuitka_String_AsString( generator->m_name ), #else Nuitka_String_AsString( generator->m_qualname ), #endif generator ); #endif } static long Nuitka_Generator_tp_traverse( struct Nuitka_GeneratorObject *generator, visitproc visit, void *arg ) { // Not needed. // Py_VISIT( (PyObject *)generator->m_frame ); return 0; } static void Nuitka_Generator_release_closure( struct Nuitka_GeneratorObject *generator ) { for( Py_ssize_t i = 0; i < generator->m_closure_given; i++ ) { CHECK_OBJECT( generator->m_closure[ i ] ); Py_DECREF( generator->m_closure[ i ] ); } generator->m_closure_given = 0; } #ifndef _NUITKA_EXPERIMENTAL_GENERATOR_GOTO // For the generator object fiber entry point, we may need to follow what // "makecontext" will support and that is only a list of integers, but we will need // to push a pointer through it, and so it's two of them, which might be fully // sufficient. #ifdef _NUITKA_MAKECONTEXT_INTS static void Nuitka_Generator_entry_point( int address_1, int address_2 ) { // Restore the pointer from integers should it be necessary, depending on // the platform. This requires pointers to be no larger that to "int" value. int addresses[2] = { address_1, address_2 }; struct Nuitka_GeneratorObject *generator = (struct Nuitka_GeneratorObject *)*(uintptr_t *)&addresses[0]; #else static void Nuitka_Generator_entry_point( struct Nuitka_GeneratorObject *generator ) { #endif ((generator_code)generator->m_code)( generator ); swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); } #endif static PyObject *Nuitka_Generator_send2( struct Nuitka_GeneratorObject *generator, PyObject *value ) { if ( generator->m_status != status_Finished ) { PyThreadState *thread_state = PyThreadState_GET(); #if PYTHON_VERSION < 300 PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = NULL; PyTracebackObject *saved_exception_traceback = NULL; if ( saved_exception_type != Py_None && saved_exception_type != NULL ) { Py_INCREF( saved_exception_type ); saved_exception_value = thread_state->exc_value; Py_XINCREF( saved_exception_value ); saved_exception_traceback = (PyTracebackObject *)thread_state->exc_traceback; Py_XINCREF( saved_exception_traceback ); } #endif if ( generator->m_running ) { PyErr_Format( PyExc_ValueError, "generator already executing" ); return NULL; } if ( generator->m_status == status_Unused ) { #ifndef _NUITKA_EXPERIMENTAL_GENERATOR_GOTO // Prepare the generator context to run. int res = prepareFiber( &generator->m_yielder_context, (void *)Nuitka_Generator_entry_point, (uintptr_t)generator ); if ( res != 0 ) { PyErr_Format( PyExc_MemoryError, "generator cannot be allocated" ); return NULL; } #endif generator->m_status = status_Running; } // Put the generator back on the frame stack. PyFrameObject *return_frame = thread_state->frame; #ifndef __NUITKA_NO_ASSERT__ if ( return_frame ) { assertFrameObject( (struct Nuitka_FrameObject *)return_frame ); } #endif if ( generator->m_frame ) { // It would be nice if our frame were still alive. Nobody had the // right to release it. assertFrameObject( generator->m_frame ); // It's not supposed to be on the top right now. assert( return_frame != &generator->m_frame->m_frame ); Py_XINCREF( return_frame ); generator->m_frame->m_frame.f_back = return_frame; thread_state->frame = &generator->m_frame->m_frame; } // Continue the yielder function while preventing recursion. generator->m_running = true; #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO PyObject *yielded = ((generator_code)generator->m_code)( generator, value ); #else generator->m_yielded = value; swapFiber( &generator->m_caller_context, &generator->m_yielder_context ); PyObject *yielded = generator->m_yielded; #endif generator->m_running = false; thread_state = PyThreadState_GET(); // Remove the generator from the frame stack. if ( generator->m_frame ) { assert( thread_state->frame == &generator->m_frame->m_frame ); assertFrameObject( generator->m_frame ); Py_CLEAR( generator->m_frame->m_frame.f_back ); } thread_state->frame = return_frame; if ( yielded == NULL ) { generator->m_status = status_Finished; Py_XDECREF( generator->m_frame ); generator->m_frame = NULL; Nuitka_Generator_release_closure( generator ); #if PYTHON_VERSION < 300 if ( saved_exception_type != NULL && saved_exception_type != Py_None ) { Py_DECREF( saved_exception_type ); Py_XDECREF( saved_exception_value ); Py_XDECREF( saved_exception_traceback ); } #endif #if PYTHON_VERSION >= 350 if ( generator->m_code_object->co_flags & CO_FUTURE_GENERATOR_STOP && GET_ERROR_OCCURRED() == PyExc_StopIteration ) { PyObject *saved_exception_type, *saved_exception_value; PyTracebackObject *saved_exception_tb; FETCH_ERROR_OCCURRED( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); NORMALIZE_EXCEPTION( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); PyErr_Format( PyExc_RuntimeError, "generator raised StopIteration" ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, saved_exception_value ); CHECK_OBJECT( exception_value ); CHECK_OBJECT( saved_exception_value ); Py_INCREF( saved_exception_value ); PyException_SetContext( exception_value, saved_exception_value ); Py_DECREF( saved_exception_type ); Py_XDECREF( saved_exception_tb ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } #endif return NULL; } else { #if PYTHON_VERSION < 300 PyObject *old_type = thread_state->exc_type; PyObject *old_value = thread_state->exc_value; PyTracebackObject *old_tb = (PyTracebackObject *)thread_state->exc_traceback; // Set sys attributes in the fastest possible way. PyObject *sys_dict = thread_state->interp->sysdict; CHECK_OBJECT( sys_dict ); if ( saved_exception_type != NULL && saved_exception_type != Py_None ) { thread_state->exc_type = saved_exception_type; thread_state->exc_value = saved_exception_value; thread_state->exc_traceback = (PyObject *)saved_exception_traceback; Py_XDECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); if ( old_type != saved_exception_type ) { PyDict_SetItem( sys_dict, const_str_plain_exc_type, saved_exception_type ); } if ( saved_exception_value != old_value ) { PyDict_SetItem( sys_dict, const_str_plain_exc_value, saved_exception_value ? saved_exception_value : Py_None ); } if ( saved_exception_traceback != old_tb ) { PyDict_SetItem( sys_dict, const_str_plain_exc_traceback, saved_exception_traceback ? (PyObject *)saved_exception_traceback : Py_None ); } } else { thread_state->exc_type = Py_None; thread_state->exc_value = Py_None; thread_state->exc_traceback = (PyObject *)Py_None; Py_INCREF( Py_None ); Py_INCREF( Py_None ); Py_INCREF( Py_None ); Py_XDECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); if ( old_type != Py_None ) { PyDict_SetItem( sys_dict, const_str_plain_exc_type, Py_None ); } if ( old_value != Py_None ) { PyDict_SetItem( sys_dict, const_str_plain_exc_value, Py_None ); } if ( old_tb != (PyTracebackObject *)Py_None ) { PyDict_SetItem( sys_dict, const_str_plain_exc_traceback, Py_None ); } } #endif return yielded; } } else { return NULL; } } static PyObject *Nuitka_Generator_send( struct Nuitka_GeneratorObject *generator, PyObject *value ) { if ( generator->m_status == status_Unused && value != NULL && value != Py_None ) { PyErr_Format( PyExc_TypeError, "can't send non-None value to a just-started generator" ); return NULL; } PyObject *result = Nuitka_Generator_send2( generator, value ); if ( result == NULL ) { if ( GET_ERROR_OCCURRED() == NULL ) { RESTORE_ERROR_OCCURRED( PyExc_StopIteration, NULL, NULL ); Py_INCREF( PyExc_StopIteration ); } } return result; } static PyObject *Nuitka_Generator_tp_iternext( struct Nuitka_GeneratorObject *generator ) { return Nuitka_Generator_send2( generator, Py_None ); } /* Our own qiter interface, which is for quicker simple loop style iteration, that does not send anything in. */ PyObject *Nuitka_Generator_qiter( struct Nuitka_GeneratorObject *generator, bool *finished ) { PyObject *result = Nuitka_Generator_send2( generator, Py_None ); if ( result == NULL ) { if (unlikely( !CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED() )) { *finished = false; return NULL; } *finished = true; return NULL; } *finished = false; return result; } #if PYTHON_VERSION < 340 static #endif PyObject *Nuitka_Generator_close( struct Nuitka_GeneratorObject *generator, PyObject *args ) { if ( generator->m_status == status_Running ) { Py_INCREF( PyExc_GeneratorExit ); generator->m_exception_type = PyExc_GeneratorExit; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; PyObject *result = Nuitka_Generator_send2( generator, Py_None ); if (unlikely( result )) { Py_DECREF( result ); PyErr_Format( PyExc_RuntimeError, "generator ignored GeneratorExit" ); return NULL; } else { PyObject *error = GET_ERROR_OCCURRED(); // StopIteration as exception. if ( error == NULL ) { Py_INCREF( Py_None ); return Py_None; } // Maybe another acceptable exception for generator exit. if ( EXCEPTION_MATCH_GENERATOR( error ) ) { CLEAR_ERROR_OCCURRED(); Py_INCREF( Py_None ); return Py_None; } return NULL; } } Py_INCREF( Py_None ); return Py_None; } static PyObject *Nuitka_Generator_throw( struct Nuitka_GeneratorObject *generator, PyObject *args ) { assert( generator->m_exception_type == NULL ); assert( generator->m_exception_value == NULL ); assert( generator->m_exception_tb == NULL ); int res = PyArg_UnpackTuple( args, "throw", 1, 3, &generator->m_exception_type, &generator->m_exception_value, (PyObject **)&generator->m_exception_tb ); if (unlikely( res == 0 )) { generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } if ( (PyObject *)generator->m_exception_tb == Py_None ) { generator->m_exception_tb = NULL; } else if ( generator->m_exception_tb != NULL && !PyTraceBack_Check( generator->m_exception_tb ) ) { generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( PyExceptionClass_Check( generator->m_exception_type )) { Py_INCREF( generator->m_exception_type ); Py_XINCREF( generator->m_exception_value ); Py_XINCREF( generator->m_exception_tb ); NORMALIZE_EXCEPTION( &generator->m_exception_type, &generator->m_exception_value, &generator->m_exception_tb ); } else if ( PyExceptionInstance_Check( generator->m_exception_type ) ) { if ( generator->m_exception_value && generator->m_exception_value != Py_None ) { generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); return NULL; } generator->m_exception_value = generator->m_exception_type; Py_INCREF( generator->m_exception_value ); generator->m_exception_type = PyExceptionInstance_Class( generator->m_exception_type ); Py_INCREF( generator->m_exception_type ); Py_XINCREF( generator->m_exception_tb ); } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 300 "exceptions must be classes, or instances, not %s", #else "exceptions must be classes or instances deriving from BaseException, not %s", #endif Py_TYPE( generator->m_exception_type )->tp_name ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } if ( ( generator->m_exception_tb != NULL ) && ( (PyObject *)generator->m_exception_tb != Py_None ) && ( !PyTraceBack_Check( generator->m_exception_tb ) ) ) { PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( generator->m_status == status_Running ) { PyObject *result = Nuitka_Generator_send2( generator, Py_None ); if ( result == NULL ) { if ( GET_ERROR_OCCURRED() == NULL ) { RESTORE_ERROR_OCCURRED( PyExc_StopIteration, NULL, NULL ); Py_INCREF( PyExc_StopIteration ); } } return result; } else if ( generator->m_status == status_Finished ) { RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } else { if ( generator->m_exception_tb == NULL ) { // TODO: Our compiled objects really need a way to store common // stuff in a "shared" part across all instances, and outside of // run time, so we could reuse this. struct Nuitka_FrameObject *frame = MAKE_FUNCTION_FRAME( generator->m_code_object, generator->m_module, 0 ); generator->m_exception_tb = MAKE_TRACEBACK( frame, generator->m_code_object->co_firstlineno ); Py_DECREF( frame ); } RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; generator->m_status = status_Finished; return NULL; } } #if PYTHON_VERSION >= 340 static void Nuitka_Generator_tp_del( struct Nuitka_GeneratorObject *generator ) { if ( generator->m_status != status_Running ) { return; } PyObject *error_type, *error_value; PyTracebackObject *error_traceback; FETCH_ERROR_OCCURRED( &error_type, &error_value, &error_traceback ); PyObject *close_result = Nuitka_Generator_close( generator, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)generator ); } else { Py_DECREF( close_result ); } /* Restore the saved exception if any. */ RESTORE_ERROR_OCCURRED( error_type, error_value, error_traceback ); } #endif #define MAX_GENERATOR_FREE_LIST_COUNT 100 static struct Nuitka_GeneratorObject *free_list_generators = NULL; static int free_list_generators_count = 0; static void Nuitka_Generator_tp_dealloc( struct Nuitka_GeneratorObject *generator ) { // Revive temporarily. assert( Py_REFCNT( generator ) == 0 ); Py_REFCNT( generator ) = 1; // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); if ( generator->m_status == status_Running ) { PyObject *close_result = Nuitka_Generator_close( generator, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)generator ); } else { Py_DECREF( close_result ); } } Nuitka_Generator_release_closure( generator ); Py_XDECREF( generator->m_frame ); assert( Py_REFCNT( generator ) == 1 ); Py_REFCNT( generator ) = 0; #ifndef _NUITKA_EXPERIMENTAL_GENERATOR_GOTO releaseFiber( &generator->m_yielder_context ); #endif // Now it is safe to release references and memory for it. Nuitka_GC_UnTrack( generator ); if ( generator->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)generator ); assert( !ERROR_OCCURRED() ); } Py_DECREF( generator->m_name ); #if PYTHON_VERSION >= 350 Py_DECREF( generator->m_qualname ); #endif /* Put the object into freelist or release to GC */ releaseToFreeList( free_list_generators, generator, MAX_GENERATOR_FREE_LIST_COUNT ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } static PyObject *Nuitka_Generator_get_name( struct Nuitka_GeneratorObject *generator ) { PyObject *result = generator->m_name; Py_INCREF( result ); return result; } #if PYTHON_VERSION >= 350 static int Nuitka_Generator_set_name( struct Nuitka_GeneratorObject *generator, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *tmp = generator->m_name; Py_INCREF( value ); generator->m_name = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Generator_get_qualname( struct Nuitka_GeneratorObject *generator ) { PyObject *result = generator->m_qualname; Py_INCREF( result ); return result; } static int Nuitka_Generator_set_qualname( struct Nuitka_GeneratorObject *generator, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *tmp = generator->m_qualname; Py_INCREF( value ); generator->m_qualname = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Generator_get_yieldfrom( struct Nuitka_GeneratorObject *generator ) { if ( generator->m_yieldfrom ) { Py_INCREF( generator->m_yieldfrom ); return generator->m_yieldfrom; } else { Py_INCREF( Py_None ); return Py_None; } } #endif static PyObject *Nuitka_Generator_get_code( struct Nuitka_GeneratorObject *generator ) { PyObject *result = (PyObject *)generator->m_code_object; Py_INCREF( result ); return result; } static int Nuitka_Generator_set_code( struct Nuitka_GeneratorObject *generator, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "gi_code is not writable in Nuitka" ); return -1; } static PyObject *Nuitka_Generator_get_frame( struct Nuitka_GeneratorObject *generator ) { PyObject *result; if ( generator->m_frame ) { result = (PyObject *)generator->m_frame; } else { result = Py_None; } Py_INCREF( result ); return result; } static int Nuitka_Generator_set_frame( struct Nuitka_GeneratorObject *generator, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "gi_frame is not writable in Nuitka" ); return -1; } static PyGetSetDef Nuitka_Generator_getsetlist[] = { #if PYTHON_VERSION < 350 { (char *)"__name__", (getter)Nuitka_Generator_get_name, NULL, NULL }, #else { (char *)"__name__", (getter)Nuitka_Generator_get_name, (setter)Nuitka_Generator_set_name, NULL }, { (char *)"__qualname__", (getter)Nuitka_Generator_get_qualname, (setter)Nuitka_Generator_set_qualname, NULL }, { (char *)"gi_yieldfrom", (getter)Nuitka_Generator_get_yieldfrom, NULL, NULL }, #endif { (char *)"gi_code", (getter)Nuitka_Generator_get_code, (setter)Nuitka_Generator_set_code, NULL }, { (char *)"gi_frame", (getter)Nuitka_Generator_get_frame, (setter)Nuitka_Generator_set_frame, NULL }, { NULL } }; static PyMethodDef Nuitka_Generator_methods[] = { { "send", (PyCFunction)Nuitka_Generator_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_Generator_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_Generator_close, METH_NOARGS, NULL }, { NULL } }; #include static PyMemberDef Nuitka_Generator_members[] = { /* The type of "gi_running" changed in Python3. */ #if PYTHON_VERSION < 330 { (char *)"gi_running", T_INT, offsetof(struct Nuitka_GeneratorObject, m_running), READONLY }, #else { (char *)"gi_running", T_BOOL, offsetof(struct Nuitka_GeneratorObject, m_running), READONLY }, #endif { NULL } }; PyTypeObject Nuitka_Generator_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_generator", /* tp_name */ sizeof(struct Nuitka_GeneratorObject), /* tp_basicsize */ sizeof(struct Nuitka_CellObject *), /* tp_itemsize */ (destructor)Nuitka_Generator_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)Nuitka_Generator_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ #if PYTHON_VERSION < 340 Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, #else Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE, #endif /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Generator_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof(struct Nuitka_GeneratorObject, m_weakrefs), /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_Generator_tp_iternext, /* tp_iternext */ Nuitka_Generator_methods, /* tp_methods */ Nuitka_Generator_members, /* tp_members */ Nuitka_Generator_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0 /* tp_version_tag */ #if PYTHON_VERSION >= 340 ,(destructor)Nuitka_Generator_tp_del /* tp_finalizer */ #endif }; void _initCompiledGeneratorType( void ) { PyType_Ready( &Nuitka_Generator_Type ); } #if PYTHON_VERSION < 350 PyObject *Nuitka_Generator_New( generator_code code, PyObject *module, PyObject *name, PyCodeObject *code_object, Py_ssize_t closure_given ) #else PyObject *Nuitka_Generator_New( generator_code code, PyObject *module, PyObject *name, PyObject *qualname, PyCodeObject *code_object, Py_ssize_t closure_given ) #endif { struct Nuitka_GeneratorObject *result; // Macro to assign result memory from GC or free list. allocateFromFreeList( free_list_generators, struct Nuitka_GeneratorObject, Nuitka_Generator_Type, closure_given ); assert( result != NULL ); CHECK_OBJECT( result ); assert( Py_SIZE( result ) >= closure_given ); result->m_code = (void *)code; CHECK_OBJECT( module ); result->m_module = module; CHECK_OBJECT( name ); result->m_name = name; Py_INCREF( name ); #if PYTHON_VERSION >= 350 CHECK_OBJECT( qualname ); result->m_qualname = qualname; Py_INCREF( qualname ); result->m_yieldfrom = NULL; #endif /* Note: The closure is set externally. */ result->m_closure_given = closure_given; result->m_weakrefs = NULL; result->m_status = status_Unused; result->m_running = false; result->m_exception_type = NULL; result->m_exception_value = NULL; result->m_exception_tb = NULL; #ifndef _NUITKA_EXPERIMENTAL_GENERATOR_GOTO result->m_yielded = NULL; #else result->m_yield_return_index = 0; #endif result->m_frame = NULL; result->m_code_object = code_object; #ifndef _NUITKA_EXPERIMENTAL_GENERATOR_GOTO initFiber( &result->m_yielder_context ); #endif Nuitka_GC_Track( result ); return (PyObject *)result; } #if PYTHON_VERSION >= 330 // This is for CPython iterator objects, the respective code is not exported as // API, so we need to redo it. This is an re-implementation that closely follows // what it does. It's unrelated to compiled generators. PyObject *PyGen_Send( PyGenObject *generator, PyObject *arg ) { if (unlikely( generator->gi_running )) { PyErr_SetString( PyExc_ValueError, "generator already executing" ); return NULL; } PyFrameObject *frame = generator->gi_frame; assert( frame == NULL || PyFrame_Check( frame )); if ( frame == NULL || frame->f_stacktop == NULL ) { // Set exception if called from send() if ( arg != NULL ) { PyErr_SetNone( PyExc_StopIteration ); } return NULL; } if ( frame->f_lasti == -1 ) { if (unlikely( arg && arg != Py_None )) { PyErr_SetString( PyExc_TypeError, "can't send non-None value to a just-started generator" ); return NULL; } } else { // Put arg on top of the value stack PyObject *tmp = arg ? arg : Py_None; Py_INCREF( tmp ); *(frame->f_stacktop++) = tmp; } // Generators always return to their most recent caller, not necessarily // their creator. PyThreadState *tstate = PyThreadState_GET(); Py_XINCREF( tstate->frame ); assert( frame->f_back == NULL ); frame->f_back = tstate->frame; generator->gi_running = 1; PyObject *result = PyEval_EvalFrameEx( frame, 0 ); generator->gi_running = 0; // Don't keep the reference to f_back any longer than necessary. It // may keep a chain of frames alive or it could create a reference // cycle. assert( frame->f_back == tstate->frame ); Py_CLEAR( frame->f_back ); // If the generator just returned (as opposed to yielding), signal that the // generator is exhausted. if ( result && frame->f_stacktop == NULL ) { if ( result == Py_None ) { PyErr_SetNone( PyExc_StopIteration ); } else { PyObject *e = PyObject_CallFunctionObjArgs( PyExc_StopIteration, result, NULL ); if ( e != NULL ) { PyErr_SetObject( PyExc_StopIteration, e ); Py_DECREF( e ); } } Py_CLEAR( result ); } if ( result == NULL || frame->f_stacktop == NULL ) { // Generator is finished, remove exception from frame before releasing // it. PyObject *type = frame->f_exc_type; PyObject *value = frame->f_exc_value; PyObject *traceback = frame->f_exc_traceback; frame->f_exc_type = NULL; frame->f_exc_value = NULL; frame->f_exc_traceback = NULL; Py_XDECREF( type ); Py_XDECREF( value ); Py_XDECREF( traceback ); // Now release frame. generator->gi_frame = NULL; Py_DECREF( frame ); } return result; } #if PYTHON_VERSION >= 300 PyObject *ERROR_GET_STOP_ITERATION_VALUE() { assert( PyErr_ExceptionMatches( PyExc_StopIteration ) ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); Py_DECREF( exception_type ); Py_XDECREF( exception_tb ); PyObject *value = NULL; if ( exception_value ) { if ( EXCEPTION_MATCH_BOOL_SINGLE( exception_value, PyExc_StopIteration ) ) { value = ((PyStopIterationObject *)exception_value)->value; Py_XINCREF( value ); Py_DECREF( exception_value ); } else { value = exception_value; } } if ( value == NULL ) { Py_INCREF( Py_None ); value = Py_None; } return value; } static void RAISE_GENERATOR_EXCEPTION( struct Nuitka_GeneratorObject *generator ) { CHECK_OBJECT( generator->m_exception_type ); RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } extern PyObject *ERROR_GET_STOP_ITERATION_VALUE(); extern PyObject *const_str_plain_send, *const_str_plain_throw, *const_str_plain_close; static PyObject *_YIELD_FROM( struct Nuitka_GeneratorObject *generator, PyObject *value ) { // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( generator->m_exception_type ) { // The yielding generator is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( generator->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL && !EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } } RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( generator->m_exception_type ); Py_CLEAR( generator->m_exception_value ); Py_CLEAR( generator->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } #if PYTHON_VERSION >= 350 else if ( PyCoro_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } #endif else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { // Bug compatibility here, before 3.3 tuples were unrolled in calls, which is what // PyObject_CallMethod does. #if PYTHON_VERSION >= 340 retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); #else retval = PyObject_CallMethod( value, (char *)"send", (char *)"O", send_value ); #endif } // Check the sub-generator result if ( retval == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { Py_INCREF( Py_None ); return Py_None; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { generator->m_yielded = retval; #if PYTHON_VERSION >= 350 generator->m_yieldfrom = value; #endif // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); #if PYTHON_VERSION >= 350 generator->m_yieldfrom = NULL; #endif send_value = generator->m_yielded; CHECK_OBJECT( send_value ); } } } PyObject *GENERATOR_YIELD_FROM( struct Nuitka_GeneratorObject *generator, PyObject *target ) { PyObject *value; #if PYTHON_VERSION >= 350 if ( PyCoro_CheckExact( target ) || Nuitka_Coroutine_Check( target )) { if (unlikely( (generator->m_code_object->co_flags & CO_ITERABLE_COROUTINE) == 0 )) { PyErr_SetString( PyExc_TypeError, "cannot 'yield from' a coroutine object in a non-coroutine generator" ); return NULL; } return _YIELD_FROM( generator, target ); } else #endif { value = MAKE_ITERATOR( target ); if (unlikely( value == NULL )) { return NULL; } PyObject *result = _YIELD_FROM( generator, value ); Py_DECREF( value ); return result; } } // Note: This is copy if YIELD_FROM with changes only at the end. As it's not // easy to split up, we are going to tolerate the copy. static PyObject *_YIELD_FROM_IN_HANDLER( struct Nuitka_GeneratorObject *generator, PyObject *value ) { // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( generator->m_exception_type ) { // The yielding generator is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( generator->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else if ( !EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_GENERATOR_EXCEPTION( generator ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( generator->m_exception_type ); Py_CLEAR( generator->m_exception_value ); Py_CLEAR( generator->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } #if PYTHON_VERSION >= 350 else if ( PyCoro_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } #endif else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { // Bug compatibility here, before 3.3 tuples were unrolled in calls, which is what // PyObject_CallMethod does. #if PYTHON_VERSION >= 340 retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); #else retval = PyObject_CallMethod( value, (char *)"send", (char *)"O", send_value ); #endif } // Check the sub-generator result if ( retval == NULL ) { if ( !ERROR_OCCURRED() ) { Py_INCREF( Py_None ); return Py_None; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { generator->m_yielded = retval; // When yielding from an exception handler in Python3, the exception // preserved to the frame is restore, while the current one is put there. PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; #if PYTHON_VERSION >= 350 generator->m_yieldfrom = value; #endif // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); #if PYTHON_VERSION >= 350 generator->m_yieldfrom = NULL; #endif // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; send_value = generator->m_yielded; CHECK_OBJECT( send_value ); } } } PyObject *GENERATOR_YIELD_FROM_IN_HANDLER( struct Nuitka_GeneratorObject *generator, PyObject *target ) { PyObject *value; #if PYTHON_VERSION >= 350 if ( PyCoro_CheckExact( target ) || Nuitka_Coroutine_Check( target )) { if (unlikely( (generator->m_code_object->co_flags & CO_ITERABLE_COROUTINE) == 0 )) { PyErr_SetString( PyExc_TypeError, "cannot 'yield from' a coroutine object in a non-coroutine generator" ); return NULL; } return _YIELD_FROM_IN_HANDLER( generator, target ); } else #endif { value = MAKE_ITERATOR( target ); if (unlikely( value == NULL )) { return NULL; } PyObject *result = _YIELD_FROM_IN_HANDLER( generator, value ); Py_DECREF( value ); return result; } } #endif #endif Nuitka-0.5.28.2/nuitka/build/static_src/HelpersCalling.c0000644000372000001440000005102513134660221023254 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // PyObject *callPythonFunction( PyObject *func, PyObject **args, int count ) { PyCodeObject *co = (PyCodeObject *)PyFunction_GET_CODE( func ); PyObject *globals = PyFunction_GET_GLOBALS( func ); PyObject *argdefs = PyFunction_GET_DEFAULTS( func ); #if PYTHON_VERSION >= 300 PyObject *kwdefs = PyFunction_GET_KW_DEFAULTS( func ); if ( kwdefs == NULL && argdefs == NULL && co->co_argcount == count && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #else if ( argdefs == NULL && co->co_argcount == count && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #endif { PyThreadState *tstate = PyThreadState_GET(); CHECK_OBJECT( globals ); PyFrameObject *frame = PyFrame_New( tstate, co, globals, NULL ); if (unlikely( frame == NULL )) { return NULL; }; for ( int i = 0; i < count; i++ ) { frame->f_localsplus[i] = args[i]; Py_INCREF( frame->f_localsplus[i] ); } PyObject *result = PyEval_EvalFrameEx( frame, 0 ); // Frame release protects against recursion as it may lead to variable // destruction. ++tstate->recursion_depth; Py_DECREF( frame ); --tstate->recursion_depth; return result; } PyObject **defaults = NULL; int nd = 0; if ( argdefs != NULL ) { defaults = &PyTuple_GET_ITEM( argdefs, 0 ); nd = (int)( Py_SIZE( argdefs ) ); } PyObject *result = PyEval_EvalCodeEx( #if PYTHON_VERSION >= 300 (PyObject *)co, #else co, // code object #endif globals, // globals NULL, // no locals args, // args count, // argcount NULL, // kwds 0, // kwcount defaults, // defaults nd, // defcount #if PYTHON_VERSION >= 300 kwdefs, #endif PyFunction_GET_CLOSURE( func ) ); return result; } static PyObject *_fast_function_noargs( PyObject *func ) { PyCodeObject *co = (PyCodeObject *)PyFunction_GET_CODE( func ); PyObject *globals = PyFunction_GET_GLOBALS( func ); PyObject *argdefs = PyFunction_GET_DEFAULTS( func ); #if PYTHON_VERSION >= 300 PyObject *kwdefs = PyFunction_GET_KW_DEFAULTS( func ); if ( kwdefs == NULL && argdefs == NULL && co->co_argcount == 0 && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #else if ( argdefs == NULL && co->co_argcount == 0 && co->co_flags == ( CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE )) #endif { PyThreadState *tstate = PyThreadState_GET(); CHECK_OBJECT( globals ); PyFrameObject *frame = PyFrame_New( tstate, co, globals, NULL ); if (unlikely( frame == NULL )) { return NULL; }; PyObject *result = PyEval_EvalFrameEx( frame, 0 ); // Frame release protects against recursion as it may lead to variable // destruction. ++tstate->recursion_depth; Py_DECREF( frame ); --tstate->recursion_depth; return result; } PyObject **defaults = NULL; int nd = 0; if ( argdefs != NULL ) { defaults = &PyTuple_GET_ITEM( argdefs, 0 ); nd = (int)( Py_SIZE( argdefs ) ); } PyObject *result = PyEval_EvalCodeEx( #if PYTHON_VERSION >= 300 (PyObject *)co, #else co, // code object #endif globals, // globals NULL, // no locals NULL, // args 0, // argcount NULL, // kwds 0, // kwcount defaults, // defaults nd, // defcount #if PYTHON_VERSION >= 300 kwdefs, #endif PyFunction_GET_CLOSURE( func ) ); return result; } PyObject *CALL_FUNCTION_NO_ARGS( PyObject *called ) { CHECK_OBJECT( called ); if ( Nuitka_Function_Check( called ) ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } struct Nuitka_FunctionObject *function = (struct Nuitka_FunctionObject *)called; PyObject *result; if ( function->m_args_simple && 0 == function->m_args_positional_count ) { result = function->m_c_code( function, NULL ); } else if ( function->m_args_simple && function->m_defaults_given == function->m_args_positional_count ) { PyObject **python_pars = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = 0; i < function->m_defaults_given; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsPos( function, python_pars, NULL, 0 ) ) { result = function->m_c_code( function, python_pars ); } else { result = NULL; } } Py_LeaveRecursiveCall(); return result; } else if ( Nuitka_Method_Check( called ) ) { struct Nuitka_MethodObject *method = (struct Nuitka_MethodObject *)called; // Unbound method without arguments, let the error path be slow. if ( method->m_object != NULL ) { if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object" ) )) { return NULL; } struct Nuitka_FunctionObject *function = method->m_function; PyObject *result; if ( function->m_args_simple && 1 == function->m_args_positional_count ) { Py_INCREF( method->m_object ); result = function->m_c_code( function, &method->m_object ); } else if ( function->m_args_simple && function->m_defaults_given == function->m_args_positional_count - 1 ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif python_pars[0] = method->m_object; memcpy( python_pars+1, &PyTuple_GET_ITEM( function->m_defaults, 0 ), sizeof(PyObject *) * function->m_defaults_given ); for( Py_ssize_t i = 0; i < function->m_args_positional_count; i++ ) { Py_INCREF( python_pars[ i ] ); } result = function->m_c_code( function, python_pars ); } else { result = Nuitka_CallMethodFunctionNoArgs( function, method->m_object ); } Py_LeaveRecursiveCall(); return result; } } else if ( PyFunction_Check( called ) ) { return _fast_function_noargs( called ); } return CALL_FUNCTION( called, const_tuple_empty, NULL ); } PyObject *CALL_METHOD_WITH_POSARGS( PyObject *source, PyObject *attribute, PyObject *positional_args ) { CHECK_OBJECT( source ); CHECK_OBJECT( attribute ); CHECK_OBJECT( positional_args ); #if PYTHON_VERSION < 300 if ( PyInstance_Check( source ) ) { PyInstanceObject *source_instance = (PyInstanceObject *)source; // The special cases have their own variant on the code generation level // as we are called with constants only. assert( attribute != const_str_plain___dict__ ); assert( attribute != const_str_plain___class__ ); // Try the instance dict first. PyObject *called_object = GET_STRING_DICT_VALUE( (PyDictObject *)source_instance->in_dict, (PyStringObject *)attribute ); // Note: The "called_object" was found without taking a reference, // so we need not release it in this branch. if ( called_object != NULL ) { return CALL_FUNCTION_WITH_POSARGS( called_object, positional_args ); } // Then check the class dictionaries. called_object = FIND_ATTRIBUTE_IN_CLASS( source_instance->in_class, attribute ); // Note: The "called_object" was found without taking a reference, // so we need not release it in this branch. if ( called_object != NULL ) { descrgetfunc descr_get = Py_TYPE( called_object )->tp_descr_get; if ( descr_get == Nuitka_Function_Type.tp_descr_get ) { return Nuitka_CallMethodFunctionPosArgs( (struct Nuitka_FunctionObject const *)called_object, source, &PyTuple_GET_ITEM(positional_args, 0), PyTuple_GET_SIZE(positional_args) ); } else if ( descr_get != NULL ) { PyObject *method = descr_get( called_object, source, (PyObject *)source_instance->in_class ); if (unlikely( method == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_POSARGS( method, positional_args ); Py_DECREF( method ); return result; } else { return CALL_FUNCTION_WITH_POSARGS( called_object, positional_args ); } } else if (unlikely( source_instance->in_class->cl_getattr == NULL )) { PyErr_Format( PyExc_AttributeError, "%s instance has no attribute '%s'", PyString_AS_STRING( source_instance->in_class->cl_name ), PyString_AS_STRING( attribute ) ); return NULL; } else { // Finally allow the "__getattr__" override to provide it or else // it's an error. PyObject *args[] = { source, attribute }; called_object = CALL_FUNCTION_WITH_ARGS2( source_instance->in_class->cl_getattr, args ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_POSARGS( called_object, positional_args ); Py_DECREF( called_object ); return result; } } else #endif { PyObject *called_object; PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro != NULL ) { called_object = (*type->tp_getattro)( source, attribute ); } else if ( type->tp_getattr != NULL ) { called_object = (*type->tp_getattr)( source, Nuitka_String_AsString_Unchecked( attribute ) ); } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%s'", type->tp_name, Nuitka_String_AsString_Unchecked( attribute ) ); return NULL; } if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_WITH_POSARGS( called_object, positional_args ); Py_DECREF( called_object ); return result; } } PyObject *CALL_METHOD_NO_ARGS( PyObject *source, PyObject *attr_name ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro == PyObject_GenericGetAttr ) { // Unfortunately this is required, although of cause rarely necessary. if (unlikely( type->tp_dict == NULL )) { if (unlikely( PyType_Ready( type ) < 0 )) { return NULL; } } PyObject *descr = _PyType_Lookup( type, attr_name ); descrgetfunc func = NULL; if ( descr != NULL ) { Py_INCREF( descr ); #if PYTHON_VERSION < 300 if ( PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) ) { #endif func = Py_TYPE( descr )->tp_descr_get; if ( func != NULL && PyDescr_IsData( descr ) ) { PyObject *called_object = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); PyObject *result = CALL_FUNCTION_NO_ARGS( called_object ); Py_DECREF( called_object ); return result; } #if PYTHON_VERSION < 300 } #endif } Py_ssize_t dictoffset = type->tp_dictoffset; PyObject *dict = NULL; if ( dictoffset != 0 ) { // Negative dictionary offsets have special meaning. if ( dictoffset < 0 ) { Py_ssize_t tsize; size_t size; tsize = ((PyVarObject *)source)->ob_size; if (tsize < 0) tsize = -tsize; size = _PyObject_VAR_SIZE( type, tsize ); dictoffset += (long)size; } PyObject **dictptr = (PyObject **) ((char *)source + dictoffset); dict = *dictptr; } if ( dict != NULL ) { CHECK_OBJECT( dict ); Py_INCREF( dict ); PyObject *called_object = PyDict_GetItem( dict, attr_name ); if ( called_object != NULL ) { Py_INCREF( called_object ); Py_XDECREF( descr ); Py_DECREF( dict ); PyObject *result = CALL_FUNCTION_NO_ARGS( called_object ); Py_DECREF( called_object ); return result; } Py_DECREF( dict ); } if ( func != NULL ) { if ( func == Nuitka_Function_Type.tp_descr_get ) { PyObject *result = Nuitka_CallMethodFunctionNoArgs( (struct Nuitka_FunctionObject const *)descr, source ); Py_DECREF( descr ); return result; } else { PyObject *called_object = func( descr, source, (PyObject *)type ); CHECK_OBJECT( called_object ); Py_DECREF( descr ); PyObject *result = CALL_FUNCTION_NO_ARGS( called_object ); Py_DECREF( called_object ); return result; } } if ( descr != NULL ) { CHECK_OBJECT( descr ); return CALL_FUNCTION_NO_ARGS( descr ); } #if PYTHON_VERSION < 300 PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%s'", type->tp_name, PyString_AS_STRING( attr_name ) ); #else PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%U'", type->tp_name, attr_name ); #endif return NULL; } #if PYTHON_VERSION < 300 else if ( type == &PyInstance_Type ) { PyInstanceObject *source_instance = (PyInstanceObject *)source; // The special cases have their own variant on the code generation level // as we are called with constants only. assert( attr_name != const_str_plain___dict__ ); assert( attr_name != const_str_plain___class__ ); // Try the instance dict first. PyObject *called_object = GET_STRING_DICT_VALUE( (PyDictObject *)source_instance->in_dict, (PyStringObject *)attr_name ); // Note: The "called_object" was found without taking a reference, // so we need not release it in this branch. if ( called_object != NULL ) { return CALL_FUNCTION_NO_ARGS( called_object ); } // Then check the class dictionaries. called_object = FIND_ATTRIBUTE_IN_CLASS( source_instance->in_class, attr_name ); // Note: The "called_object" was found without taking a reference, // so we need not release it in this branch. if ( called_object != NULL ) { descrgetfunc descr_get = Py_TYPE( called_object )->tp_descr_get; if ( descr_get == Nuitka_Function_Type.tp_descr_get ) { return Nuitka_CallMethodFunctionNoArgs( (struct Nuitka_FunctionObject const *)called_object, source ); } else if ( descr_get != NULL ) { PyObject *method = descr_get( called_object, source, (PyObject *)source_instance->in_class ); if (unlikely( method == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_NO_ARGS( method ); Py_DECREF( method ); return result; } else { return CALL_FUNCTION_NO_ARGS( called_object ); } } else if (unlikely( source_instance->in_class->cl_getattr == NULL )) { PyErr_Format( PyExc_AttributeError, "%s instance has no attribute '%s'", PyString_AS_STRING( source_instance->in_class->cl_name ), PyString_AS_STRING( attr_name ) ); return NULL; } else { // Finally allow the "__getattr__" override to provide it or else // it's an error. PyObject *args[] = { source, attr_name }; called_object = CALL_FUNCTION_WITH_ARGS2( source_instance->in_class->cl_getattr, args ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_NO_ARGS( called_object ); Py_DECREF( called_object ); return result; } } #endif else if ( type->tp_getattro != NULL ) { PyObject *called_object = (*type->tp_getattro)( source, attr_name ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_NO_ARGS( called_object ); Py_DECREF( called_object ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *called_object = (*type->tp_getattr)( source, Nuitka_String_AsString_Unchecked( attr_name ) ); if (unlikely( called_object == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_NO_ARGS( called_object ); Py_DECREF( called_object ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%s'", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return NULL; } } Nuitka-0.5.28.2/nuitka/build/static_src/gen_ucontext_src/0000755000372000001440000000000013207540420023561 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/static_src/gen_ucontext_src/fibers_gen.c0000644000372000001440000000420713112214770026034 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for generic targets. #include "nuitka/prelude.h" // TODO: Make stack size rational. #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { to->f_context.uc_stack.ss_sp = NULL; to->f_context.uc_link = NULL; to->start_stack = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { int res = getcontext( &to->f_context ); if (unlikely( res != 0 )) { return 1; } to->f_context.uc_stack.ss_size = STACK_SIZE; to->f_context.uc_stack.ss_sp = last_stack ? (char *)last_stack : (char *)malloc( STACK_SIZE ); to->start_stack = to->f_context.uc_stack.ss_sp; to->f_context.uc_link = NULL; last_stack = NULL; makecontext( &to->f_context, (void (*)())code, 1, (unsigned long)arg ); return 0; } void _releaseFiber( Fiber *to ) { if ( to->start_stack != NULL ) { if ( last_stack == NULL ) { last_stack = to->start_stack; } else { free( to->start_stack ); } to->start_stack = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { int res = swapcontext( &to->f_context, &from->f_context ); assert( res == 0 ); } Nuitka-0.5.28.2/nuitka/build/static_src/HelpersBuiltin.c0000644000372000001440000004727213134660221023322 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /** For calling built-ins, calls it and uses keyword dictionary if necessary. * * This helper simplifies calling built-ins with optional arguments that can * be given as keyword arguments. We basically re-construct the minimal call * using keywords here. This obviously is for inefficient calls to the original * built-in and should be avoided. * **/ static PyObject *CALL_BUILTIN_KW_ARGS( PyObject *callable, PyObject **args, char const **arg_names, int max_args ) { int i = 0; while( i < max_args ) { if ( args[i] == NULL ) break; CHECK_OBJECT( args[i] ); i++; } int usable_args = i; PyObject *kw_dict = NULL; while( i < max_args ) { if ( args[i] != NULL ) { CHECK_OBJECT( args[i] ); if ( kw_dict == NULL ) { kw_dict = PyDict_New(); } int res = PyDict_SetItemString( kw_dict, arg_names[i], args[i] ); assert( res == 0 ); } i++; } PyObject *args_tuple = PyTuple_New( usable_args ); for ( i = 0; i < usable_args; i++ ) { PyTuple_SET_ITEM( args_tuple, i, args[i] ); Py_INCREF( args[i] ); } PyObject *result = CALL_FUNCTION( callable, args_tuple, kw_dict ); Py_XDECREF( kw_dict ); Py_DECREF( args_tuple ); return result; } /** The "compile" built-in. * */ NUITKA_DEFINE_BUILTIN( compile ) #if PYTHON_VERSION < 300 PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit ) #else PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit, PyObject *optimize ) #endif { // May be a source, but also could already be a compiled object, in which // case this should just return it. if ( PyCode_Check( source_code ) ) { Py_INCREF( source_code ); return source_code; } PyObject *pos_args = PyTuple_New(3); PyTuple_SET_ITEM( pos_args, 0, source_code ); Py_INCREF( source_code ); PyTuple_SET_ITEM( pos_args, 1, file_name ); Py_INCREF( file_name ); PyTuple_SET_ITEM( pos_args, 2, mode ); Py_INCREF( mode ); PyObject *kw_args = NULL; if ( flags != NULL ) { if ( kw_args == NULL ) kw_args = PyDict_New(); PyDict_SetItemString( kw_args, "flags", flags ); } if ( dont_inherit != NULL ) { if ( kw_args == NULL ) kw_args = PyDict_New(); PyDict_SetItemString( kw_args, "dont_inherit", dont_inherit ); } #if PYTHON_VERSION >= 300 if ( optimize != NULL ) { if ( kw_args == NULL ) kw_args = PyDict_New(); PyDict_SetItemString( kw_args, "optimize", optimize ); } #endif NUITKA_ASSIGN_BUILTIN( compile ); PyObject *result = CALL_FUNCTION( NUITKA_ACCESS_BUILTIN( compile ), pos_args, kw_args ); Py_DECREF( pos_args ); Py_XDECREF( kw_args ); return result; } /** * The "eval" implementation, used for "exec" too. */ PyObject *EVAL_CODE( PyObject *code, PyObject *globals, PyObject *locals ) { CHECK_OBJECT( code ); CHECK_OBJECT( globals ); CHECK_OBJECT( locals ); if ( PyDict_Check( globals ) == 0 ) { PyErr_Format( PyExc_TypeError, "exec: arg 2 must be a dictionary or None" ); return NULL; } // TODO: Our re-formulation prevents this externally, doesn't it. if ( locals == Py_None ) { locals = globals; } if ( PyMapping_Check( locals ) == 0 ) { PyErr_Format( PyExc_TypeError, "exec: arg 3 must be a mapping or None" ); return NULL; } // Set the __builtins__ in globals, it is expected to be present. if ( PyDict_GetItem( globals, const_str_plain___builtins__ ) == NULL ) { if ( PyDict_SetItem( globals, const_str_plain___builtins__, (PyObject *)builtin_module ) != 0 ) { return NULL; } } #if PYTHON_VERSION < 300 PyObject *result = PyEval_EvalCode( (PyCodeObject *)code, globals, locals ); #else PyObject *result = PyEval_EvalCode( code, globals, locals ); #endif if (unlikely( result == NULL )) { return NULL; } return result; } /** The "open" built-in. * * Different for Python2 and Python3, the later has more arguments and * both accept keyword arguments. * **/ NUITKA_DEFINE_BUILTIN( open ); #if PYTHON_VERSION < 300 PyObject *BUILTIN_OPEN( PyObject *file_name, PyObject *mode, PyObject *buffering ) { NUITKA_ASSIGN_BUILTIN( open ); PyObject *args[] = { file_name, mode, buffering }; char const *arg_names[] = { "file_name", "mode", "buffering" }; return CALL_BUILTIN_KW_ARGS( NUITKA_ACCESS_BUILTIN( open ), args, arg_names, 3 ); } #else PyObject *BUILTIN_OPEN( PyObject *file_name, PyObject *mode, PyObject *buffering, PyObject *encoding, PyObject *errors, PyObject *newline, PyObject *closefd, PyObject *opener) { NUITKA_ASSIGN_BUILTIN( open ); PyObject *args[] = { file_name, mode, buffering, encoding, errors, newline, closefd, opener }; char const *arg_names[] = { "file_name", "mode", "buffering", "encoding", "errors", "newline", "closefd", "opener" }; return CALL_BUILTIN_KW_ARGS( NUITKA_ACCESS_BUILTIN( open ), args, arg_names, 8 ); } #endif /** The "staticmethod" built-in. * **/ NUITKA_DEFINE_BUILTIN(staticmethod) PyObject *BUILTIN_STATICMETHOD( PyObject *value ) { NUITKA_ASSIGN_BUILTIN( staticmethod ); PyObject *args[] = { value }; return CALL_FUNCTION_WITH_ARGS1( NUITKA_ACCESS_BUILTIN( staticmethod ), args ); } /** The "classmethod" built-in. * **/ NUITKA_DEFINE_BUILTIN(classmethod) PyObject *BUILTIN_CLASSMETHOD( PyObject *value ) { NUITKA_ASSIGN_BUILTIN( classmethod ); PyObject *args[] = { value }; return CALL_FUNCTION_WITH_ARGS1( NUITKA_ACCESS_BUILTIN( classmethod ), args ); } #if PYTHON_VERSION >= 300 /** The "bytes" built-in. * * Only for Python3. There is not BYTES_BUILTIN1 yet, this only delegates to * the actual built-in which is wasteful. TODO: Have dedicated implementation * for this. * **/ NUITKA_DEFINE_BUILTIN( bytes ); PyObject *BUILTIN_BYTES3( PyObject *value, PyObject *encoding, PyObject *errors ) { NUITKA_ASSIGN_BUILTIN( bytes ); PyObject *args[] = { value, encoding, errors }; char const *arg_names[] = { "value", "encoding", "errors" }; return CALL_BUILTIN_KW_ARGS( NUITKA_ACCESS_BUILTIN( bytes ), args, arg_names, 3 ); } #endif /** The "bin" built-in. * **/ PyObject *BUILTIN_BIN( PyObject *value ) { // Note: I don't really know why "oct" and "hex" don't use this as well. PyObject *result = PyNumber_ToBase( value, 2 ); if ( unlikely( result == NULL )) { return NULL; } return result; } /** The "oct" built-in. * **/ PyObject *BUILTIN_OCT( PyObject *value ) { #if PYTHON_VERSION >= 300 PyObject *result = PyNumber_ToBase( value, 8 ); if ( unlikely( result == NULL )) { return NULL; } return result; #else if (unlikely( value == NULL )) { PyErr_Format( PyExc_TypeError, "oct() argument can't be converted to oct" ); return NULL; } PyNumberMethods *nb = Py_TYPE( value )->tp_as_number; if (unlikely( nb == NULL || nb->nb_oct == NULL )) { PyErr_Format( PyExc_TypeError, "oct() argument can't be converted to oct" ); return NULL; } PyObject *result = (*nb->nb_oct)( value ); if ( result ) { if (unlikely( !PyString_Check( result ) )) { PyErr_Format( PyExc_TypeError, "__oct__ returned non-string (type %s)", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } } return result; #endif } /** The "hex" built-in. * **/ PyObject *BUILTIN_HEX( PyObject *value ) { #if PYTHON_VERSION >= 300 PyObject *result = PyNumber_ToBase( value, 16 ); if ( unlikely( result == NULL )) { return NULL; } return result; #else if (unlikely( value == NULL )) { PyErr_Format( PyExc_TypeError, "hex() argument can't be converted to hex" ); return NULL; } PyNumberMethods *nb = Py_TYPE( value )->tp_as_number; if (unlikely( nb == NULL || nb->nb_hex == NULL )) { PyErr_Format( PyExc_TypeError, "hex() argument can't be converted to hex" ); return NULL; } PyObject *result = (*nb->nb_hex)( value ); if (likely( result )) { if (unlikely( !PyString_Check( result ) )) { PyErr_Format( PyExc_TypeError, "__hex__ returned non-string (type %s)", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } } return result; #endif } /** The "hash" built-in. * **/ PyObject *BUILTIN_HASH( PyObject *value ) { Py_hash_t hash = PyObject_Hash( value ); if (unlikely( hash == -1 )) { return NULL; } #if PYTHON_VERSION < 300 return PyInt_FromLong( hash ); #else return PyLong_FromSsize_t( hash ); #endif } /** The "bytearray" built-in. * * These should be more in-lined maybe, as a lot of checks are not necessary * and the error checking for the 3 arguments variant may even not be enough, * as it could be keyword arguments. * **/ PyObject *BUILTIN_BYTEARRAY1( PyObject *value ) { PyObject *result = PyByteArray_FromObject( value ); if ( unlikely( result == NULL )) { return NULL; } return result; } NUITKA_DEFINE_BUILTIN( bytearray ) PyObject *BUILTIN_BYTEARRAY3( PyObject *string, PyObject *encoding, PyObject *errors ) { CHECK_OBJECT( string ); CHECK_OBJECT( encoding ); NUITKA_ASSIGN_BUILTIN( bytearray ); if ( errors == NULL ) { PyObject *args[] = { string, encoding }; PyObject *result = CALL_FUNCTION_WITH_ARGS2( NUITKA_ACCESS_BUILTIN( bytearray ), args ); return result; } else { PyObject *args[] = { string, encoding, errors }; PyObject *result = CALL_FUNCTION_WITH_ARGS3( NUITKA_ACCESS_BUILTIN( bytearray ), args ); return result; } } /** The "iter" built-in. * * This comes in two flavors, with one or two arguments. The second one * creates a "calliterobject" that is private to CPython. We define it here * for ourselves. The one argument version is in headers for in-lining of * the code. * **/ // From CPython: typedef struct { PyObject_HEAD PyObject *it_callable; PyObject *it_sentinel; } calliterobject; PyObject *BUILTIN_ITER2( PyObject *callable, PyObject *sentinel ) { calliterobject *result = PyObject_GC_New( calliterobject, &PyCallIter_Type ); if (unlikely( result == NULL )) { return NULL; } // Note: References were taken at call site already. result->it_callable = callable; Py_INCREF( callable ); result->it_sentinel = sentinel; Py_INCREF( sentinel ); Nuitka_GC_Track( result ); return (PyObject *)result; } /** The "type" built-in. * * This comes in two flavors, one being the detection of a values type, * and 3 argument variant creates a new type. * **/ PyObject *BUILTIN_TYPE1( PyObject *arg ) { PyObject *result = (PyObject *)Py_TYPE( arg ); Py_INCREF( result ); return result; } extern PyObject *const_str_plain___module__; PyObject *BUILTIN_TYPE3( PyObject *module_name, PyObject *name, PyObject *bases, PyObject *dict ) { PyObject *pos_args = PyTuple_New(3); PyTuple_SET_ITEM( pos_args, 0, name ); Py_INCREF( name ); PyTuple_SET_ITEM( pos_args, 1, bases ); Py_INCREF( bases ); PyTuple_SET_ITEM( pos_args, 2, dict ); Py_INCREF( dict ); PyObject *result = PyType_Type.tp_new( &PyType_Type, pos_args, NULL ); if (unlikely( result == NULL )) { Py_DECREF( pos_args ); return NULL; } PyTypeObject *type = Py_TYPE( result ); if (likely( PyType_IsSubtype( type, &PyType_Type ) )) { if ( #if PYTHON_VERSION < 300 PyType_HasFeature( type, Py_TPFLAGS_HAVE_CLASS ) && #endif type->tp_init != NULL ) { int res = type->tp_init( result, pos_args, NULL ); if (unlikely( res < 0 )) { Py_DECREF( pos_args ); Py_DECREF( result ); return NULL; } } } Py_DECREF( pos_args ); int res = PyObject_SetAttr( result, const_str_plain___module__, module_name ); if ( res < 0 ) { return NULL; } return result; } /** The "super" built-in. * * This uses a private structure "superobject" that we declare here too. * **/ typedef struct { PyObject_HEAD PyTypeObject *type; PyObject *obj; PyTypeObject *obj_type; } superobject; extern PyObject *const_str_plain___class__; PyObject *BUILTIN_SUPER( PyObject *type, PyObject *object ) { CHECK_OBJECT( type ); superobject *result = PyObject_GC_New( superobject, &PySuper_Type ); assert( result ); if ( object == Py_None ) { object = NULL; } if (unlikely( PyType_Check( type ) == false )) { PyErr_Format( PyExc_RuntimeError, "super(): __class__ is not a type (%s)", Py_TYPE( type )->tp_name ); return NULL; } result->type = (PyTypeObject *)type; Py_INCREF( type ); if ( object ) { result->obj = object; Py_INCREF( object ); if ( PyType_Check( object ) && PyType_IsSubtype( (PyTypeObject *)object, (PyTypeObject *)type )) { result->obj_type = (PyTypeObject *)object; Py_INCREF( object ); } else if ( PyType_IsSubtype( Py_TYPE(object ), (PyTypeObject *)type) ) { result->obj_type = Py_TYPE( object ); Py_INCREF( result->obj_type ); } else { PyObject *class_attr = PyObject_GetAttr( object, const_str_plain___class__); if (likely( class_attr != NULL && PyType_Check( class_attr ) && (PyTypeObject *)class_attr != Py_TYPE( object ) )) { result->obj_type = (PyTypeObject *)class_attr; } else { if ( class_attr == NULL ) { CLEAR_ERROR_OCCURRED(); } else { Py_DECREF( class_attr ); } PyErr_Format( PyExc_TypeError, "super(type, obj): obj must be an instance or subtype of type" ); return NULL; } } } else { result->obj = NULL; result->obj_type = NULL; } Nuitka_GC_Track( result ); CHECK_OBJECT( (PyObject *)result ); assert( Py_TYPE( result ) == &PySuper_Type ); return (PyObject *)result; } /** The "callable" built-in. * **/ PyObject *BUILTIN_CALLABLE( PyObject *value ) { int res = PyCallable_Check( value ); PyObject *result = BOOL_FROM( res != 0 ); Py_INCREF( result ); return result; } /* The "getattr" built-in with default value. * * We might want to split it off for a variant without default value. * **/ PyObject *BUILTIN_GETATTR( PyObject *object, PyObject *attribute, PyObject *default_value ) { #if PYTHON_VERSION < 300 if ( PyUnicode_Check( attribute ) ) { attribute = _PyUnicode_AsDefaultEncodedString( attribute, NULL ); if (unlikely( attribute == NULL )) { return NULL; } } if (unlikely( !PyString_Check( attribute ) )) { PyErr_Format( PyExc_TypeError, "getattr(): attribute name must be string" ); return NULL; } #else if (!PyUnicode_Check( attribute )) { PyErr_Format( PyExc_TypeError, "getattr(): attribute name must be string" ); return NULL; } #endif PyObject *result = PyObject_GetAttr( object, attribute ); if ( result == NULL ) { if ( default_value != NULL && EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); Py_INCREF( default_value ); return default_value; } else { return NULL; } } else { return result; } } /** The "setattr" built-in. * **/ PyObject *BUILTIN_SETATTR( PyObject *object, PyObject *attribute, PyObject *value ) { int res = PyObject_SetAttr( object, attribute, value ); if ( res < 0 ) { return NULL; } Py_INCREF( Py_None ); return Py_None; } /** The "divmod" built-in. * * This is really a binary number operation and probably should live * where the others are. * **/ PyObject *BUILTIN_DIVMOD( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_divmod; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_divmod; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_divmod; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for divmod(): '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } Nuitka-0.5.28.2/nuitka/build/static_src/CompiledCellType.c0000644000372000001440000001531613122472300023555 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/freelists.h" #define MAX_CELL_FREE_LIST_COUNT 1000 static struct Nuitka_CellObject *free_list_cells = NULL; static int free_list_cells_count = 0; static void Nuitka_Cell_tp_dealloc( struct Nuitka_CellObject *cell ) { Nuitka_GC_UnTrack( cell ); Py_XDECREF( cell->ob_ref ); releaseToFreeList( free_list_cells, cell, MAX_CELL_FREE_LIST_COUNT ); } #if PYTHON_VERSION < 300 static int Nuitka_Cell_tp_compare( struct Nuitka_CellObject *cell_a, struct Nuitka_CellObject *cell_b ) { /* Empty cells compare specifically different. */ if ( cell_a->ob_ref == NULL ) { if ( cell_b->ob_ref == NULL ) { return 0; } return -1; } if ( cell_b->ob_ref == NULL ) { return 1; } return PyObject_Compare( cell_a->ob_ref, cell_b->ob_ref ); } #else static PyObject *Nuitka_Cell_tp_richcompare( PyObject *a, PyObject *b, int op) { PyObject *result; CHECK_OBJECT( a ); CHECK_OBJECT( b ); if (unlikely( !Nuitka_Cell_Check(a) || !Nuitka_Cell_Check( b ) )) { result = Py_NotImplemented; Py_INCREF( result ); return result; } /* Now just dereference, and compare from there by contents. */ a = ((struct Nuitka_CellObject *)a)->ob_ref; b = ((struct Nuitka_CellObject *)b)->ob_ref; if (a != NULL && b != NULL) { return PyObject_RichCompare( a, b, op ); } int res = (b == NULL) - (a == NULL); switch (op) { case Py_EQ: result = BOOL_FROM( res == 0 ); break; case Py_NE: result = BOOL_FROM( res != 0 ); break; case Py_LE: result = BOOL_FROM( res <= 0 ); break; case Py_GE: result = BOOL_FROM( res >= 0 ); break; case Py_LT: result = BOOL_FROM( res < 0 ); break; case Py_GT: result = BOOL_FROM( res > 0 ); break; default: PyErr_BadArgument(); return NULL; } Py_INCREF( result ); return result; } #endif static PyObject *Nuitka_Cell_tp_repr( struct Nuitka_CellObject *cell ) { if ( cell->ob_ref == NULL ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( #else return PyUnicode_FromFormat( #endif "", cell ); } else { #if PYTHON_VERSION < 300 return PyString_FromFormat( #else return PyUnicode_FromFormat( #endif "", cell, cell->ob_ref->ob_type->tp_name, cell->ob_ref ); } } static int Nuitka_Cell_tp_traverse( struct Nuitka_CellObject *cell, visitproc visit, void *arg ) { Py_VISIT( cell->ob_ref ); return 0; } static int Nuitka_Cell_tp_clear( struct Nuitka_CellObject *cell ) { Py_CLEAR( cell->ob_ref ); return 0; } static PyObject *Nuitka_Cell_get_contents( struct Nuitka_CellObject *cell, void *closure ) { if ( cell->ob_ref == NULL ) { PyErr_SetString( PyExc_ValueError, "Cell is empty" ); return NULL; } Py_INCREF( cell->ob_ref ); return cell->ob_ref; } static PyGetSetDef Nuitka_Cell_getsetlist[] = { { (char *)"cell_contents", (getter)Nuitka_Cell_get_contents, NULL, NULL }, { NULL } }; PyTypeObject Nuitka_Cell_Type = { PyVarObject_HEAD_INIT(NULL , 0) "compiled_cell", sizeof(struct Nuitka_CellObject), 0, (destructor)Nuitka_Cell_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ #if PYTHON_VERSION < 300 (cmpfunc)Nuitka_Cell_tp_compare, /* tp_compare */ #else 0, /* tp_reserved */ #endif (reprfunc)Nuitka_Cell_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Cell_tp_traverse, /* tp_traverse */ (inquiry)Nuitka_Cell_tp_clear, /* tp_clear */ #if PYTHON_VERSION < 300 0, /* tp_richcompare */ #else Nuitka_Cell_tp_richcompare, /* tp_richcompare */ #endif 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ Nuitka_Cell_getsetlist, /* tp_getset */ }; void _initCompiledCellType( void ) { PyType_Ready( &Nuitka_Cell_Type ); } struct Nuitka_CellObject *Nuitka_Cell_New( void ) { struct Nuitka_CellObject *result; allocateFromFreeListFixed( free_list_cells, struct Nuitka_CellObject, Nuitka_Cell_Type ); Nuitka_GC_Track( result ); return result; } void Nuitka_Cells_New( struct Nuitka_CellObject **closure, int count ) { assert( count > 0 ); while( count > 0 ) { *closure = Nuitka_Cell_New(); closure += 1; count -= 1; } } Nuitka-0.5.28.2/nuitka/build/static_src/HelpersProfiling.c0000644000372000001440000000611613134660221023635 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /** * This is responsible for profiling Nuitka using "vmprof". */ #if _NUITKA_PROFILE timespec diff(timespec start, timespec end); static timespec getTimespecDiff( timespec start, timespec end ) { timespec temp; if ( ( end.tv_nsec - start.tv_nsec ) < 0 ) { temp.tv_sec = end.tv_sec-start.tv_sec-1; temp.tv_nsec = 1000000000+end.tv_nsec-start.tv_nsec; } else { temp.tv_sec = end.tv_sec-start.tv_sec; temp.tv_nsec = end.tv_nsec-start.tv_nsec; } return temp; } static FILE *tempfile_profile; static PyObject *vmprof_module; static timespec time1, time2; void startProfiling( void ) { tempfile_profile = fopen("nuitka-performance.dat", "wb"); // Might be necessary to import "site" module to find "vmprof", lets just // hope we don't suffer too much from that. If we do, what might be done // is to try and just have the "PYTHONPATH" from it from out user. PyImport_ImportModule( "site" ); vmprof_module = PyImport_ImportModule( "vmprof" ); // Abort if it's not there. if ( vmprof_module == NULL ) { PyErr_Print(); abort(); } PyObject *result = CALL_FUNCTION_WITH_ARGS1( PyObject_GetAttrString( vmprof_module, "enable"), PyInt_FromLong( fileno( tempfile_profile ) ) ); if ( result == NULL ) { PyErr_Print(); abort(); } clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time1); } void stopProfiling( void ) { clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &time2); // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); PyObject *result = CALL_FUNCTION_NO_ARGS( PyObject_GetAttrString( vmprof_module, "disable") ); if ( result == NULL ) CLEAR_ERROR_OCCURRED(); fclose( tempfile_profile ); FILE *tempfile_times = fopen( "nuitka-times.dat", "wb" ); timespec diff = getTimespecDiff( time1, time2 ); long delta_ns = diff.tv_sec * 1000000000 + diff.tv_nsec; fprintf( tempfile_times, "%ld\n", delta_ns); fclose( tempfile_times ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } #endif Nuitka-0.5.28.2/nuitka/build/static_src/HelpersPathTools.c0000644000372000001440000000221513207537242023624 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /** * This is responsible for small helper routines that do path * manipulations. */ #if defined(_WIN32) #include #else #include #endif /* This abstracts "dirname" for use on Windows and Unix alike. */ char *getDirname( char *path ) { #if defined(_WIN32) PathRemoveFileSpec( path ); return path; #else return dirname( path ); #endif }Nuitka-0.5.28.2/nuitka/build/static_src/CompiledFunctionType.c0000644000372000001440000021331413134660221024465 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/compiled_method.h" #include "nuitka/freelists.h" // Needed for offsetof #include // tp_descr_get slot, bind a function to an object. static PyObject *Nuitka_Function_descr_get( PyObject *function, PyObject *object, PyObject *klass ) { assert( Nuitka_Function_Check( function ) ); #if PYTHON_VERSION >= 300 if ( object == NULL || object == Py_None ) { Py_INCREF( function ); return function; } #endif return Nuitka_Method_New( (struct Nuitka_FunctionObject *)function, object == Py_None ? NULL : object, klass ); } // tp_repr slot, decide how compiled function shall be output to "repr" built-in static PyObject *Nuitka_Function_tp_repr( struct Nuitka_FunctionObject *function ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( #else return PyUnicode_FromFormat( #endif "", #if PYTHON_VERSION < 330 Nuitka_String_AsString( function->m_name ), #else Nuitka_String_AsString( function->m_qualname ), #endif function ); } static PyObject *Nuitka_Function_tp_call( struct Nuitka_FunctionObject *function, PyObject *tuple_args, PyObject *kw ) { CHECK_OBJECT( tuple_args ); assert( PyTuple_CheckExact( tuple_args ) ); if ( kw == NULL ) { PyObject **args = &PyTuple_GET_ITEM( tuple_args, 0 ); Py_ssize_t args_size = PyTuple_GET_SIZE( tuple_args ); if ( function->m_args_simple && args_size == function->m_args_positional_count ) { for( Py_ssize_t i = 0; i < args_size; i++ ) { Py_INCREF( args[ i ] ); } return function->m_c_code( function, args ); } else if ( function->m_args_simple && args_size + function->m_defaults_given == function->m_args_positional_count ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memcpy( python_pars, args, args_size * sizeof(PyObject *) ); memcpy( python_pars + args_size, &PyTuple_GET_ITEM( function->m_defaults, 0 ), function->m_defaults_given * sizeof(PyObject *) ); for( Py_ssize_t i = 0; i < function->m_args_overall_count; i++ ) { Py_INCREF( python_pars[ i ] ); } return function->m_c_code( function, python_pars ); } else { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if ( parseArgumentsPos( function, python_pars, args, args_size )) { return function->m_c_code( function, python_pars ); } else { return NULL; } } } else { return Nuitka_CallFunctionPosArgsKwArgs( function, &PyTuple_GET_ITEM( tuple_args, 0 ), PyTuple_GET_SIZE( tuple_args ), kw ); } } static long Nuitka_Function_tp_traverse( struct Nuitka_FunctionObject *function, visitproc visit, void *arg ) { // TODO: Identify the impact of not visiting other owned objects. It appears // to be mostly harmless, as these are strings. Py_VISIT( function->m_dict ); for( Py_ssize_t i = 0; i < function->m_closure_given; i++ ) { Py_VISIT( function->m_closure[i] ); } return 0; } static long Nuitka_Function_tp_hash( struct Nuitka_FunctionObject *function ) { return function->m_counter; } static PyObject *Nuitka_Function_get_name( struct Nuitka_FunctionObject *object ) { PyObject *result = object->m_name; Py_INCREF( result ); return result; } static int Nuitka_Function_set_name( struct Nuitka_FunctionObject *object, PyObject *value ) { #if PYTHON_VERSION < 300 if (unlikely( value == NULL || PyString_Check( value ) == 0 )) #else if (unlikely( value == NULL || PyUnicode_Check( value ) == 0 )) #endif { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *old = object->m_name; Py_INCREF( value ); object->m_name = value; Py_DECREF( old ); return 0; } #if PYTHON_VERSION >= 330 static PyObject *Nuitka_Function_get_qualname( struct Nuitka_FunctionObject *object ) { PyObject *result = object->m_qualname; Py_INCREF( result ); return result; } static int Nuitka_Function_set_qualname( struct Nuitka_FunctionObject *object, PyObject *value ) { if (unlikely( value == NULL || PyUnicode_Check( value ) == 0 )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *old = object->m_qualname; Py_INCREF( value ); object->m_qualname = value; Py_DECREF( old ); return 0; } #endif static PyObject *Nuitka_Function_get_doc( struct Nuitka_FunctionObject *object ) { PyObject *result = object->m_doc; Py_INCREF( result ); return result; } static int Nuitka_Function_set_doc( struct Nuitka_FunctionObject *object, PyObject *value ) { PyObject *old = object->m_doc; if ( value == NULL ) { value = Py_None; } object->m_doc = value; Py_INCREF( value ); Py_DECREF( old ); return 0; } static PyObject *Nuitka_Function_get_dict( struct Nuitka_FunctionObject *object ) { if ( object->m_dict == NULL ) { object->m_dict = PyDict_New(); } Py_INCREF( object->m_dict ); return object->m_dict; } static int Nuitka_Function_set_dict( struct Nuitka_FunctionObject *object, PyObject *value ) { if (unlikely( value == NULL )) { PyErr_Format( PyExc_TypeError, "function's dictionary may not be deleted"); return -1; } if (likely( PyDict_Check( value ) )) { PyObject *old = object->m_dict; Py_INCREF( value ); object->m_dict = value; Py_XDECREF( old ); return 0; } else { PyErr_SetString( PyExc_TypeError, "setting function's dictionary to a non-dict" ); return -1; } } static PyObject *Nuitka_Function_get_code( struct Nuitka_FunctionObject *object ) { PyObject *result = (PyObject *)object->m_code_object; Py_INCREF( result ); return result; } static int Nuitka_Function_set_code( struct Nuitka_FunctionObject *object, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "__code__ is not writable in Nuitka" ); return -1; } static PyObject *Nuitka_Function_get_closure( struct Nuitka_FunctionObject *object ) { if ( object->m_closure_given > 0 ) { PyObject *result = PyTuple_New( object->m_closure_given ); for( Py_ssize_t i = 0; i < object->m_closure_given; i++ ) { PyTuple_SET_ITEM( result, i, (PyObject *)object->m_closure[i] ); Py_INCREF( (PyObject *)object->m_closure[i] ); } return result; } else { Py_INCREF( Py_None ); return Py_None; } } static int Nuitka_Function_set_closure( struct Nuitka_FunctionObject *object, PyObject *value ) { PyErr_Format( #if PYTHON_VERSION < 300 PyExc_TypeError, #else PyExc_AttributeError, #endif "readonly attribute" ); return -1; } static PyObject *Nuitka_Function_get_defaults( struct Nuitka_FunctionObject *object ) { PyObject *result = (PyObject *)object->m_defaults; Py_INCREF( result ); return result; } static void onUpdatedDefaultsValue( struct Nuitka_FunctionObject *function ) { if ( function->m_defaults == Py_None ) { function->m_defaults_given = 0; } else { function->m_defaults_given = PyTuple_GET_SIZE( function->m_defaults ); } } static int Nuitka_Function_set_defaults( struct Nuitka_FunctionObject *object, PyObject *value ) { if ( value == NULL ) { value = Py_None; } if (unlikely( value != Py_None && PyTuple_Check( value ) == false )) { PyErr_Format( PyExc_TypeError, "__defaults__ must be set to a tuple object" ); return -1; } if ( object->m_defaults == Py_None && value != Py_None ) { PyErr_Format( PyExc_TypeError, "Nuitka doesn't support __defaults__ size changes" ); return -1; } if ( object->m_defaults != Py_None && ( value == Py_None || PyTuple_Size( object->m_defaults ) != PyTuple_Size( value ) ) ) { PyErr_Format( PyExc_TypeError, "Nuitka doesn't support __defaults__ size changes" ); return -1; } PyObject *old = object->m_defaults; Py_INCREF( value ); object->m_defaults = value; Py_DECREF( old ); onUpdatedDefaultsValue( object ); return 0; } #if PYTHON_VERSION >= 300 static PyObject *Nuitka_Function_get_kwdefaults( struct Nuitka_FunctionObject *object ) { PyObject *result = (PyObject *)object->m_kwdefaults; Py_INCREF( result ); return result; } static int Nuitka_Function_set_kwdefaults( struct Nuitka_FunctionObject *object, PyObject *value ) { if ( value == NULL ) { value = Py_None; } if (unlikely( value != Py_None && PyDict_Check( value ) == false )) { PyErr_Format( PyExc_TypeError, "__kwdefaults__ must be set to a dict object" ); return -1; } PyObject *old = object->m_kwdefaults; Py_INCREF( value ); object->m_kwdefaults = value; Py_DECREF( old ); return 0; } static PyObject *Nuitka_Function_get_annotations( struct Nuitka_FunctionObject *object ) { PyObject *result = (PyObject *)object->m_annotations; Py_INCREF( result ); return result; } static int Nuitka_Function_set_annotations( struct Nuitka_FunctionObject *object, PyObject *value ) { // CPython silently converts None to empty dictionary. if ( value == Py_None || value == NULL ) { value = PyDict_New(); } if (unlikely( PyDict_Check( value ) == false )) { PyErr_Format( PyExc_TypeError, "__annotations__ must be set to a dict object" ); return -1; } PyObject *old = object->m_annotations; Py_INCREF( value ); object->m_annotations = value; Py_XDECREF( old ); return 0; } #endif static int Nuitka_Function_set_globals( struct Nuitka_FunctionObject *function, PyObject *value ) { PyErr_Format( PyExc_TypeError, "readonly attribute" ); return -1; } static PyObject *Nuitka_Function_get_globals( struct Nuitka_FunctionObject *function ) { PyObject *result = PyModule_GetDict( function->m_module ); Py_INCREF( result ); return result; } extern PyObject *const_str_plain___module__; static int Nuitka_Function_set_module( struct Nuitka_FunctionObject *object, PyObject *value ) { if ( object->m_dict == NULL ) { object->m_dict = PyDict_New(); } if ( value == NULL ) { value = Py_None; } return PyDict_SetItem( object->m_dict, const_str_plain___module__, value ); } static PyObject *Nuitka_Function_get_module( struct Nuitka_FunctionObject *object ) { PyObject *result; // The __dict__ might overrule this. if ( object->m_dict ) { result = PyDict_GetItem( object->m_dict, const_str_plain___module__ ); if ( result != NULL ) { Py_INCREF( result ); return result; } } result = MODULE_NAME( object->m_module ); Py_INCREF( result ); return result; } static PyGetSetDef Nuitka_Function_getset[] = { #if PYTHON_VERSION >= 330 { (char *)"__qualname__", (getter)Nuitka_Function_get_qualname, (setter)Nuitka_Function_set_qualname, NULL }, #endif #if PYTHON_VERSION < 300 { (char *)"func_name", (getter)Nuitka_Function_get_name, (setter)Nuitka_Function_set_name, NULL }, #endif { (char *)"__name__" , (getter)Nuitka_Function_get_name, (setter)Nuitka_Function_set_name, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_doc", (getter)Nuitka_Function_get_doc, (setter)Nuitka_Function_set_doc, NULL }, #endif { (char *)"__doc__" , (getter)Nuitka_Function_get_doc, (setter)Nuitka_Function_set_doc, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_dict", (getter)Nuitka_Function_get_dict, (setter)Nuitka_Function_set_dict, NULL }, #endif { (char *)"__dict__", (getter)Nuitka_Function_get_dict, (setter)Nuitka_Function_set_dict, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_code", (getter)Nuitka_Function_get_code, (setter)Nuitka_Function_set_code, NULL }, #endif { (char *)"__code__", (getter)Nuitka_Function_get_code, (setter)Nuitka_Function_set_code, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_defaults", (getter)Nuitka_Function_get_defaults, (setter)Nuitka_Function_set_defaults, NULL }, #endif { (char *)"__defaults__", (getter)Nuitka_Function_get_defaults, (setter)Nuitka_Function_set_defaults, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_globals", (getter)Nuitka_Function_get_globals, (setter)Nuitka_Function_set_globals, NULL }, #endif { (char *)"__closure__", (getter)Nuitka_Function_get_closure, (setter)Nuitka_Function_set_closure, NULL }, #if PYTHON_VERSION < 300 { (char *)"func_closure", (getter)Nuitka_Function_get_closure, (setter)Nuitka_Function_set_closure, NULL }, #endif { (char *)"__globals__", (getter)Nuitka_Function_get_globals, (setter)Nuitka_Function_set_globals, NULL }, { (char *)"__module__", (getter)Nuitka_Function_get_module, (setter)Nuitka_Function_set_module, NULL }, #if PYTHON_VERSION >= 300 { (char *)"__kwdefaults__", (getter)Nuitka_Function_get_kwdefaults, (setter)Nuitka_Function_set_kwdefaults, NULL }, { (char *)"__annotations__", (getter)Nuitka_Function_get_annotations, (setter)Nuitka_Function_set_annotations, NULL }, #endif { NULL } }; static PyObject *Nuitka_Function_reduce( struct Nuitka_FunctionObject *function ) { PyObject *result; #if PYTHON_VERSION < 330 result = function->m_name; #else result = function->m_qualname; #endif Py_INCREF( result ); return result; } #define MAX_FUNCTION_FREE_LIST_COUNT 100 static struct Nuitka_FunctionObject *free_list_functions = NULL; static int free_list_functions_count = 0; static void Nuitka_Function_tp_dealloc( struct Nuitka_FunctionObject *function ) { #ifndef __NUITKA_NO_ASSERT__ // Save the current exception, if any, we must to not corrupt it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); #endif Nuitka_GC_UnTrack( function ); if ( function->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)function ); } Py_DECREF( function->m_name ); #if PYTHON_VERSION >= 330 Py_DECREF( function->m_qualname ); #endif // These may actually re-surrect the object, not? Py_XDECREF( function->m_dict ); Py_DECREF( function->m_defaults ); Py_DECREF( function->m_doc ); #if PYTHON_VERSION >= 300 Py_DECREF( function->m_kwdefaults ); Py_DECREF( function->m_annotations ); #endif for( Py_ssize_t i = 0; i < function->m_closure_given; i++ ) { assert( function->m_closure[i] ); Py_DECREF( function->m_closure[i] ); } /* Put the object into freelist or release to GC */ releaseToFreeList( free_list_functions, function, MAX_FUNCTION_FREE_LIST_COUNT ); #ifndef __NUITKA_NO_ASSERT__ PyThreadState *tstate = PyThreadState_GET(); assert( tstate->curexc_type == save_exception_type ); assert( tstate->curexc_value == save_exception_value ); assert( (PyTracebackObject *)tstate->curexc_traceback == save_exception_tb ); #endif } static PyMethodDef Nuitka_Function_methods[] = { { "__reduce__", (PyCFunction)Nuitka_Function_reduce, METH_NOARGS, NULL }, { NULL } }; PyTypeObject Nuitka_Function_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_function", /* tp_name */ sizeof(struct Nuitka_FunctionObject), /* tp_basicsize */ sizeof(struct Nuitka_CellObject *), /* tp_itemsize */ (destructor)Nuitka_Function_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)Nuitka_Function_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ (hashfunc)Nuitka_Function_tp_hash, /* tp_hash */ (ternaryfunc)Nuitka_Function_tp_call, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | #if PYTHON_VERSION < 300 Py_TPFLAGS_HAVE_WEAKREFS | #endif Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Function_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof( struct Nuitka_FunctionObject, m_weakrefs ), /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Nuitka_Function_methods, /* tp_methods */ 0, /* tp_members */ Nuitka_Function_getset, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ Nuitka_Function_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */ offsetof( struct Nuitka_FunctionObject, m_dict ), /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0 /* tp_version_tag */ #if PYTHON_VERSION >= 340 ,0 /* tp_finalizer */ #endif }; void _initCompiledFunctionType( void ) { PyType_Ready( &Nuitka_Function_Type ); } // Make a function with closure. #if PYTHON_VERSION < 300 struct Nuitka_FunctionObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc, Py_ssize_t closure_given ) #elif PYTHON_VERSION < 330 struct Nuitka_FunctionObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, Py_ssize_t closure_given ) #else struct Nuitka_FunctionObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, Py_ssize_t closure_given ) #endif { struct Nuitka_FunctionObject *result; // Macro to assign result memory from GC or free list. allocateFromFreeList( free_list_functions, struct Nuitka_FunctionObject, Nuitka_Function_Type, closure_given ); /* Closure is set externally after we return */ result->m_closure_given = closure_given; result->m_c_code = c_code; Py_INCREF( name ); result->m_name = name; #if PYTHON_VERSION >= 330 // The "qualname" defaults to NULL for most compact C code. if ( qualname == NULL ) { qualname = name; } Py_INCREF( qualname ); result->m_qualname = qualname; #endif if ( defaults == NULL ) { Py_INCREF( Py_None ); defaults = Py_None; } CHECK_OBJECT( defaults ); assert( defaults == Py_None || ( PyTuple_Check( defaults ) && PyTuple_Size( defaults ) > 0 ) ); result->m_defaults = defaults; onUpdatedDefaultsValue( result ); #if PYTHON_VERSION >= 300 if ( kwdefaults == NULL ) { Py_INCREF( Py_None ); kwdefaults = Py_None; } assert( kwdefaults == Py_None || ( PyDict_Check( kwdefaults ) && DICT_SIZE( kwdefaults ) > 0 ) ); result->m_kwdefaults = kwdefaults; assert( annotations == Py_None || PyDict_Check( annotations ) ); Py_INCREF( annotations ); result->m_annotations = annotations; #endif result->m_code_object = code_object; result->m_args_positional_count = code_object->co_argcount; result->m_args_keywords_count = result->m_args_positional_count; #if PYTHON_VERSION >= 300 result->m_args_keywords_count += code_object->co_kwonlyargcount; #endif result->m_args_overall_count = result->m_args_keywords_count + (( code_object->co_flags & CO_VARARGS ) ? 1 : 0) + (( code_object->co_flags & CO_VARKEYWORDS ) ? 1 : 0); result->m_args_simple = ( code_object->co_flags & (CO_VARARGS|CO_VARKEYWORDS) ) == 0; #if PYTHON_VERSION >= 300 if ( code_object->co_kwonlyargcount > 0 ) result->m_args_simple = false; #endif if ( ( code_object->co_flags & CO_VARARGS ) != 0 ) { result->m_args_star_list_index = result->m_args_keywords_count; } else { result->m_args_star_list_index = -1; } if ( ( code_object->co_flags & CO_VARKEYWORDS ) != 0 ) { result->m_args_star_dict_index = result->m_args_keywords_count; if ( code_object->co_flags & CO_VARARGS ) { result->m_args_star_dict_index += 1; } } else { result->m_args_star_dict_index = -1; } result->m_varnames = &PyTuple_GET_ITEM( code_object->co_varnames, 0 ); result->m_module = module; Py_INCREF( doc ); result->m_doc = doc; result->m_dict = NULL; result->m_weakrefs = NULL; static long Nuitka_Function_counter = 0; result->m_counter = Nuitka_Function_counter++; Nuitka_GC_Track( result ); assert( Py_REFCNT( result ) == 1 ); return result; } static void formatErrorNoArgumentAllowed( struct Nuitka_FunctionObject const *function, #if PYTHON_VERSION >= 330 PyObject *kw, #endif Py_ssize_t given ) { char const *function_name = Nuitka_String_AsString( function->m_name ); #if PYTHON_VERSION < 330 PyErr_Format( PyExc_TypeError, "%s() takes no arguments (%zd given)", function_name, given ); #else if ( kw == NULL ) { PyErr_Format( PyExc_TypeError, "%s() takes 0 positional arguments but %zd was given", function_name, given ); } else { PyObject *tmp_iter = PyObject_GetIter( kw ); PyObject *tmp_arg_name = PyIter_Next( tmp_iter ); Py_DECREF( tmp_iter ); PyErr_Format( PyExc_TypeError, "%s() got an unexpected keyword argument '%s'", function_name, Nuitka_String_AsString( tmp_arg_name ) ); Py_DECREF( tmp_arg_name ); } #endif } static void formatErrorMultipleValuesGiven( struct Nuitka_FunctionObject const *function, Py_ssize_t index ) { char const *function_name = Nuitka_String_AsString( function->m_name ); PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 330 "%s() got multiple values for keyword argument '%s'", #else "%s() got multiple values for argument '%s'", #endif function_name, Nuitka_String_AsString( function->m_varnames[ index ] ) ); } #if PYTHON_VERSION < 330 static void formatErrorTooFewArguments( struct Nuitka_FunctionObject const *function, #if PYTHON_VERSION < 270 Py_ssize_t kw_size, #endif Py_ssize_t given ) { Py_ssize_t required_parameter_count = function->m_args_positional_count - function->m_defaults_given; char const *function_name = Nuitka_String_AsString( function->m_name ); char const *violation = ( function->m_defaults != Py_None || function->m_args_star_list_index != -1 ) ? "at least" : "exactly"; char const *plural = required_parameter_count == 1 ? "" : "s"; #if PYTHON_VERSION < 270 if ( kw_size > 0 ) { PyErr_Format( PyExc_TypeError, "%s() takes %s %zd non-keyword argument%s (%zd given)", function_name, violation, required_parameter_count, plural, given - function->m_defaults_given ); } else { PyErr_Format( PyExc_TypeError, "%s() takes %s %zd argument%s (%zd given)", function_name, violation, required_parameter_count, plural, given ); } #else PyErr_Format( PyExc_TypeError, "%s() takes %s %zd argument%s (%zd given)", function_name, violation, required_parameter_count, plural, given ); #endif } #else static void formatErrorTooFewArguments( struct Nuitka_FunctionObject const *function, PyObject **values ) { char const *function_name = Nuitka_String_AsString( function->m_name ); Py_ssize_t max_missing = 0; for( Py_ssize_t i = function->m_args_positional_count - 1 - function->m_defaults_given; i >= 0; --i ) { if ( values[ i ] == NULL ) { max_missing += 1; } } PyObject *list_str = PyUnicode_FromString( "" ); PyObject *comma_str = PyUnicode_FromString( ", " ); PyObject *and_str = PyUnicode_FromString( max_missing == 2 ? " and " : ", and " ); Py_ssize_t missing = 0; for( Py_ssize_t i = function->m_args_positional_count - 1 - function->m_defaults_given; i >= 0; --i ) { if ( values[ i ] == NULL ) { PyObject *current_str = function->m_varnames[ i ]; PyObject *current = PyObject_Repr( current_str ); if ( missing == 0 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( list_str, current ); Py_DECREF( old ); } else if ( missing == 1 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( and_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } else { PyObject *old = list_str; list_str = PyUnicode_Concat( comma_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } Py_DECREF( current ); missing += 1; } } Py_DECREF( comma_str ); Py_DECREF( and_str ); PyErr_Format( PyExc_TypeError, "%s() missing %zd required positional argument%s: %s", function_name, max_missing, max_missing > 1 ? "s" : "", Nuitka_String_AsString( list_str ) ); Py_DECREF( list_str ); } #endif static void formatErrorTooManyArguments( struct Nuitka_FunctionObject const *function, Py_ssize_t given #if PYTHON_VERSION < 270 , Py_ssize_t kw_size #endif #if PYTHON_VERSION >= 330 , Py_ssize_t kw_only #endif ) { Py_ssize_t top_level_parameter_count = function->m_args_positional_count; char const *function_name = Nuitka_String_AsString( function->m_name ); #if PYTHON_VERSION < 330 char const *violation = function->m_defaults != Py_None ? "at most" : "exactly"; #endif char const *plural = top_level_parameter_count == 1 ? "" : "s"; #if PYTHON_VERSION < 270 PyErr_Format( PyExc_TypeError, "%s() takes %s %zd %sargument%s (%zd given)", function_name, violation, top_level_parameter_count, kw_size > 0 ? "non-keyword " : "", plural, given ); #elif PYTHON_VERSION < 300 PyErr_Format( PyExc_TypeError, "%s() takes %s %zd argument%s (%zd given)", function_name, violation, top_level_parameter_count, plural, given ); #elif PYTHON_VERSION < 330 PyErr_Format( PyExc_TypeError, "%s() takes %s %zd positional argument%s (%zd given)", function_name, violation, top_level_parameter_count, plural, given ); #else char keyword_only_part[100]; if ( kw_only > 0 ) { snprintf( keyword_only_part, sizeof(keyword_only_part)-1, " positional argument%s (and %" PY_FORMAT_SIZE_T "d keyword-only argument%s)", given != 1 ? "s" : "", kw_only, kw_only != 1 ? "s" : "" ); } else { keyword_only_part[0] = 0; } if ( function->m_defaults_given == 0 ) { PyErr_Format( PyExc_TypeError, "%s() takes %zd positional argument%s but %zd%s were given", function_name, top_level_parameter_count, plural, given, keyword_only_part ); } else { PyErr_Format( PyExc_TypeError, "%s() takes from %zd to %zd positional argument%s but %zd%s were given", function_name, top_level_parameter_count - function->m_defaults_given, top_level_parameter_count, plural, given, keyword_only_part ); } #endif } #if PYTHON_VERSION >= 330 static void formatErrorTooFewKwOnlyArguments( struct Nuitka_FunctionObject const *function, PyObject **kw_vars ) { char const *function_name = Nuitka_String_AsString( function->m_name ); Py_ssize_t kwonlyargcount = function->m_code_object->co_kwonlyargcount; Py_ssize_t max_missing = 0; for( Py_ssize_t i = kwonlyargcount-1; i >= 0; --i ) { if ( kw_vars[ i ] == NULL ) { max_missing += 1; } } PyObject *list_str = PyUnicode_FromString( "" ); PyObject *comma_str = PyUnicode_FromString( ", " ); PyObject *and_str = PyUnicode_FromString( max_missing == 2 ? " and " : ", and " ); Py_ssize_t missing = 0; for( Py_ssize_t i = kwonlyargcount-1; i >= 0; --i ) { if ( kw_vars[ i ] == NULL ) { PyObject *current_str = function->m_varnames[ function->m_args_positional_count + i ]; PyObject *current = PyObject_Repr( current_str ); if ( missing == 0 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( list_str, current ); Py_DECREF( old ); } else if ( missing == 1 ) { PyObject *old = list_str; list_str = PyUnicode_Concat( and_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } else { PyObject *old = list_str; list_str = PyUnicode_Concat( comma_str, list_str ); Py_DECREF( old ); old = list_str; list_str = PyUnicode_Concat( current, list_str ); Py_DECREF( old ); } Py_DECREF( current ); missing += 1; } } Py_DECREF( comma_str ); Py_DECREF( and_str ); PyErr_Format( PyExc_TypeError, "%s() missing %zd required keyword-only argument%s: %s", function_name, max_missing, max_missing > 1 ? "s" : "", Nuitka_String_AsString( list_str ) ); Py_DECREF( list_str ); } #endif #if PYTHON_VERSION < 300 static Py_ssize_t handleKeywordArgs( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw ) #else static Py_ssize_t handleKeywordArgs( struct Nuitka_FunctionObject const *function, PyObject **python_pars, Py_ssize_t *kw_only_found, PyObject *kw ) #endif { Py_ssize_t keywords_count = function->m_args_keywords_count; #if PYTHON_VERSION >= 300 Py_ssize_t keyword_after_index = function->m_args_positional_count; #endif assert( function->m_args_star_dict_index == -1 ); Py_ssize_t kw_found = 0; Py_ssize_t ppos = 0; PyObject *key, *value; while( PyDict_Next( kw, &ppos, &key, &value ) ) { #if PYTHON_VERSION < 300 if (unlikely( !PyString_Check( key ) && !PyUnicode_Check( key ) )) #else if (unlikely( !PyUnicode_Check( key ) )) #endif { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return -1; } NUITKA_MAY_BE_UNUSED bool found = false; Py_INCREF( key ); Py_INCREF( value ); for( Py_ssize_t i = 0; i < keywords_count; i++ ) { if ( function->m_varnames[ i ] == key ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = value; #if PYTHON_VERSION >= 300 if ( i >= keyword_after_index ) { *kw_only_found += 1; } #endif found = true; break; } } if ( found == false ) { PyObject **varnames = function->m_varnames; for( Py_ssize_t i = 0; i < keywords_count; i++ ) { if ( RICH_COMPARE_BOOL_EQ_NORECURSE( varnames[ i ], key ) ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = value; #if PYTHON_VERSION >= 300 if ( i >= keyword_after_index ) { *kw_only_found += 1; } #endif found = true; break; } } } if (unlikely( found == false )) { PyErr_Format( PyExc_TypeError, "%s() got an unexpected keyword argument '%s'", Nuitka_String_AsString( function->m_name ), Nuitka_String_Check( key ) ? Nuitka_String_AsString( key ) : "" ); Py_DECREF( key ); Py_DECREF( value ); return -1; } Py_DECREF( key ); kw_found += 1; } return kw_found; } static bool MAKE_STAR_DICT_DICTIONARY_COPY( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw ) { Py_ssize_t star_dict_index = function->m_args_star_dict_index; assert( star_dict_index != -1 ); if ( kw == NULL ) { python_pars[ star_dict_index ] = PyDict_New(); } else if ( ((PyDictObject *)kw)->ma_used > 0 ) { #if PYTHON_VERSION < 330 python_pars[ star_dict_index ] = _PyDict_NewPresized( ((PyDictObject *)kw)->ma_used ); for ( int i = 0; i <= ((PyDictObject *)kw)->ma_mask; i++ ) { PyDictEntry *entry = &((PyDictObject *)kw)->ma_table[ i ]; if ( entry->me_value != NULL ) { #if PYTHON_VERSION < 300 if (unlikely( !PyString_Check( entry->me_key ) && !PyUnicode_Check( entry->me_key ) )) #else if (unlikely( !PyUnicode_Check( entry->me_key ) )) #endif { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return false; } int res = PyDict_SetItem( python_pars[ star_dict_index ], entry->me_key, entry->me_value ); if (unlikely( res != 0 )) { return false; } } } #else if ( _PyDict_HasSplitTable( (PyDictObject *)kw) ) { PyDictObject *mp = (PyDictObject *)kw; PyObject **newvalues = PyMem_NEW( PyObject *, mp->ma_keys->dk_size ); assert( newvalues != NULL ); PyDictObject *split_copy = PyObject_GC_New( PyDictObject, &PyDict_Type ); assert( split_copy != NULL ); split_copy->ma_values = newvalues; split_copy->ma_keys = mp->ma_keys; split_copy->ma_used = mp->ma_used; mp->ma_keys->dk_refcnt += 1; Nuitka_GC_Track( split_copy ); #if PYTHON_VERSION < 360 Py_ssize_t size = mp->ma_keys->dk_size; #else Py_ssize_t size = DK_USABLE_FRACTION(DK_SIZE(mp->ma_keys)); #endif for ( Py_ssize_t i = 0; i < size; i++ ) { #if PYTHON_VERSION < 360 PyDictKeyEntry *entry = &split_copy->ma_keys->dk_entries[ i ]; #else PyDictKeyEntry *entry = &DK_ENTRIES( split_copy->ma_keys )[ i ]; #endif if ( ( entry->me_key != NULL ) && unlikely( !PyUnicode_Check( entry->me_key ) )) { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return false; } split_copy->ma_values[ i ] = mp->ma_values[ i ]; Py_XINCREF( split_copy->ma_values[ i ] ); } python_pars[ star_dict_index ] = (PyObject *)split_copy; } else { python_pars[ star_dict_index ] = PyDict_New(); PyDictObject *mp = (PyDictObject *)kw; #if PYTHON_VERSION < 360 Py_ssize_t size = mp->ma_keys->dk_size; #else Py_ssize_t size = mp->ma_keys->dk_nentries; #endif for ( Py_ssize_t i = 0; i < size; i++ ) { #if PYTHON_VERSION < 360 PyDictKeyEntry *entry = &mp->ma_keys->dk_entries[ i ]; #else PyDictKeyEntry *entry = &DK_ENTRIES( mp->ma_keys )[ i ]; #endif PyObject *value = entry->me_value; if ( value != NULL ) { if (unlikely( !PyUnicode_Check( entry->me_key ) )) { PyErr_Format( PyExc_TypeError, "%s() keywords must be strings", Nuitka_String_AsString( function->m_name ) ); return false; } int res = PyDict_SetItem( python_pars[ star_dict_index ], entry->me_key, value ); if (unlikely( res != 0 )) { return false; } } } } #endif } else { python_pars[ star_dict_index ] = PyDict_New(); } return true; } #if PYTHON_VERSION < 300 static Py_ssize_t handleKeywordArgsWithStarDict( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw ) #else static Py_ssize_t handleKeywordArgsWithStarDict( struct Nuitka_FunctionObject const *function, PyObject **python_pars, Py_ssize_t *kw_only_found, PyObject *kw ) #endif { assert( function->m_args_star_dict_index != -1 ); if (unlikely( MAKE_STAR_DICT_DICTIONARY_COPY( function, python_pars, kw ) == false )) { return -1; } Py_ssize_t kw_found = 0; Py_ssize_t keywords_count = function->m_args_keywords_count; #if PYTHON_VERSION >= 300 Py_ssize_t keyword_after_index = function->m_args_positional_count; #endif Py_ssize_t dict_star_index = function->m_args_star_dict_index; PyObject **varnames = function->m_varnames; for( Py_ssize_t i = 0; i < keywords_count; i++ ) { PyObject *arg_name = varnames[ i ]; PyObject *kw_arg_value = PyDict_GetItem( python_pars[ dict_star_index ], arg_name ); if ( kw_arg_value != NULL ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = kw_arg_value; Py_INCREF( kw_arg_value ); PyDict_DelItem( python_pars[ dict_star_index ], arg_name ); kw_found += 1; #if PYTHON_VERSION >= 300 if ( i >= keyword_after_index ) { *kw_only_found += 1; } #endif } } return kw_found; } static void makeStarListTupleCopy( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { assert( function->m_args_star_list_index != -1 ); Py_ssize_t list_star_index = function->m_args_star_list_index; // Copy left-over argument values to the star list parameter given. if ( args_size > function->m_args_positional_count ) { python_pars[ list_star_index ] = PyTuple_New( args_size - function->m_args_positional_count ); for( Py_ssize_t i = 0; i < args_size - function->m_args_positional_count; i++ ) { PyObject *value = args[ function->m_args_positional_count + i ]; PyTuple_SET_ITEM( python_pars[ list_star_index ], i, value ); Py_INCREF( value ); } } else { python_pars[ list_star_index ] = const_tuple_empty; Py_INCREF( const_tuple_empty ); } } static void makeStarListTupleCopyMethod( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { assert( function->m_args_star_list_index != -1 ); Py_ssize_t list_star_index = function->m_args_star_list_index; // Copy left-over argument values to the star list parameter given. if ( args_size + 1 > function->m_args_positional_count ) { python_pars[ list_star_index ] = PyTuple_New( args_size + 1 - function->m_args_positional_count ); for( Py_ssize_t i = 0; i < args_size + 1 - function->m_args_positional_count; i++ ) { PyObject *value = args[ function->m_args_positional_count + i - 1 ]; PyTuple_SET_ITEM( python_pars[ list_star_index ], i, value ); Py_INCREF( value ); } } else { python_pars[ list_star_index ] = const_tuple_empty; Py_INCREF( const_tuple_empty ); } } static bool _handleArgumentsPlainOnly( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { Py_ssize_t arg_count = function->m_args_positional_count; // Check if too many arguments were given in case of non list star arg. // For Python3.3 it's done only later, when more knowledge has // been gained. TODO: Could be done this way for improved mode // on all versions. #if PYTHON_VERSION < 330 if ( function->m_args_star_list_index == -1 ) { if (unlikely( args_size > arg_count )) { #if PYTHON_VERSION < 270 formatErrorTooManyArguments( function, args_size, 0 ); #else formatErrorTooManyArguments( function, args_size ); #endif return false; } } #endif #if PYTHON_VERSION >= 330 bool parameter_error = false; #endif Py_ssize_t defaults_given = function->m_defaults_given; if ( args_size + defaults_given < arg_count ) { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, 0, args_size ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size ); return false; #else parameter_error = true; #endif } for( Py_ssize_t i = 0; i < args_size; i++ ) { if ( i >= arg_count ) break; assert( python_pars[ i ] == NULL ); python_pars[ i ] = args[ i ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 if ( parameter_error == false ) { #endif PyObject **source = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = args_size; i < arg_count; i++ ) { assert( python_pars[ i ] == NULL ); assert( i + defaults_given >= arg_count ); python_pars[ i ] = source[ defaults_given + i - arg_count ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 } #endif #if PYTHON_VERSION >= 330 if (unlikely( parameter_error )) { formatErrorTooFewArguments( function, python_pars ); return false; } if ( function->m_args_star_list_index == -1 ) { // Check if too many arguments were given in case of non list star arg if (unlikely( args_size > arg_count )) { formatErrorTooManyArguments( function, args_size, 0 ); return false; } } #endif if ( function->m_args_star_list_index != -1 ) { makeStarListTupleCopy( function, python_pars, args, args_size ); } return true; } static bool handleMethodArgumentsPlainOnly( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *object, PyObject **args, Py_ssize_t args_size ) { Py_ssize_t arg_count = function->m_args_positional_count; // There may be no self, otherwise we can directly assign it. if ( arg_count >= 1 ) { python_pars[ 0 ] = object; Py_INCREF( object ); } else { // Without self, there can only be star list to get the object as its // first element. Or we complain about illegal arguments. if ( function->m_args_star_list_index == 0 ) { python_pars[ 0 ] = PyTuple_New( args_size+1 ); PyTuple_SET_ITEM( python_pars[ 0 ], 0, object ); Py_INCREF( object ); for( Py_ssize_t i = 0; i < args_size; i++ ) { PyTuple_SET_ITEM( python_pars[ 0 ], i + 1, args[ i ] ); Py_INCREF( args[ i ]); } return true; } } // Check if too many arguments were given in case of non list star arg. // For Python3.3 it's done only later, when more knowledge has // been gained. TODO: Could be done this way for improved mode // on all versions. #if PYTHON_VERSION < 330 if ( function->m_args_star_list_index == -1 ) { if (unlikely( args_size + 1 > arg_count )) { #if PYTHON_VERSION < 270 formatErrorTooManyArguments( function, args_size + 1, 0 ); #else formatErrorTooManyArguments( function, args_size + 1 ); #endif return false; } } #endif #if PYTHON_VERSION >= 330 bool parameter_error = false; #endif Py_ssize_t defaults_given = function->m_defaults_given; if ( args_size + 1 + defaults_given < arg_count ) { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, 0, args_size + 1 ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size + 1 ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size + 1 ); return false; #else parameter_error = true; #endif } for( Py_ssize_t i = 0; i < args_size; i++ ) { if ( i+1 >= arg_count ) break; assert( python_pars[ i+1 ] == NULL ); python_pars[ i+1 ] = args[ i ]; Py_INCREF( python_pars[ i+1 ] ); } #if PYTHON_VERSION >= 330 if ( parameter_error == false ) { #endif PyObject **source = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = args_size + 1; i < arg_count; i++ ) { assert( python_pars[ i ] == NULL ); assert( i + defaults_given >= arg_count ); python_pars[ i ] = source[ defaults_given + i - arg_count ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 } #endif #if PYTHON_VERSION >= 330 if (unlikely( parameter_error )) { formatErrorTooFewArguments( function, python_pars ); return false; } if ( function->m_args_star_list_index == -1 ) { // Check if too many arguments were given in case of non list star arg if (unlikely( args_size + 1 > arg_count )) { formatErrorTooManyArguments( function, args_size + 1, 0 ); return false; } } #endif if ( function->m_args_star_list_index != -1 ) { makeStarListTupleCopyMethod( function, python_pars, args, args_size ); } return true; } #if PYTHON_VERSION < 270 static bool handleArgumentsPlain( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw, PyObject **args, Py_ssize_t args_size, Py_ssize_t kw_found, Py_ssize_t kw_size ) #elif PYTHON_VERSION < 300 static bool handleArgumentsPlain( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw, PyObject **args, Py_ssize_t args_size, Py_ssize_t kw_found ) #else static bool handleArgumentsPlain( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *kw, PyObject **args, Py_ssize_t args_size, Py_ssize_t kw_found, Py_ssize_t kw_only_found ) #endif { Py_ssize_t arg_count = function->m_args_positional_count; // Check if too many arguments were given in case of non list star arg. // For Python3.3 it's done only later, when more knowledge has // been gained. TODO: Could be done this way for improved mode // on all versions. #if PYTHON_VERSION < 330 if ( function->m_args_star_list_index == -1 ) { if (unlikely( args_size > arg_count )) { #if PYTHON_VERSION < 270 formatErrorTooManyArguments( function, args_size, kw_size ); #else formatErrorTooManyArguments( function, args_size + kw_found ); #endif return false; } } #endif #if PYTHON_VERSION >= 330 bool parameter_error = false; #endif if ( kw_found > 0 ) { Py_ssize_t i; for( i = 0; i < (args_size < arg_count ? args_size : arg_count); i++ ) { if (unlikely( python_pars[ i ] != NULL )) { formatErrorMultipleValuesGiven( function, i ); return false; } python_pars[ i ] = args[ i ]; Py_INCREF( python_pars[ i ] ); } Py_ssize_t defaults_given = function->m_defaults_given; for( ; i < arg_count; i++ ) { if ( python_pars[ i ] == NULL ) { if ( i + defaults_given >= arg_count ) { python_pars[ i ] = PyTuple_GET_ITEM( function->m_defaults, defaults_given + i - arg_count ); Py_INCREF( python_pars[ i ] ); } else { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, kw_size, args_size + kw_found ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size + kw_found ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size + kw_found - kw_only_found ); return false; #else parameter_error = true; #endif } } } } else { Py_ssize_t usable = (args_size < arg_count ? args_size : arg_count); Py_ssize_t defaults_given = function->m_defaults_given; if ( defaults_given < arg_count - usable ) { #if PYTHON_VERSION < 270 formatErrorTooFewArguments( function, kw_size, args_size + kw_found ); return false; #elif PYTHON_VERSION < 300 formatErrorTooFewArguments( function, args_size + kw_found ); return false; #elif PYTHON_VERSION < 330 formatErrorTooFewArguments( function, args_size + kw_found - kw_only_found ); return false; #else parameter_error = true; #endif } for( Py_ssize_t i = 0; i < usable; i++ ) { assert( python_pars[ i ] == NULL ); python_pars[ i ] = args[ i ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 if ( parameter_error == false ) { #endif PyObject **source = &PyTuple_GET_ITEM( function->m_defaults, 0 ); for( Py_ssize_t i = usable; i < arg_count; i++ ) { assert( python_pars[ i ] == NULL ); assert( i + defaults_given >= arg_count ); python_pars[ i ] = source[ defaults_given + i - arg_count ]; Py_INCREF( python_pars[ i ] ); } #if PYTHON_VERSION >= 330 } #endif } #if PYTHON_VERSION >= 330 if (unlikely( parameter_error )) { formatErrorTooFewArguments( function, python_pars ); return false; } if ( function->m_args_star_list_index == -1 ) { // Check if too many arguments were given in case of non list star arg if (unlikely( args_size > arg_count )) { formatErrorTooManyArguments( function, args_size, kw_only_found ); return false; } } #endif if ( function->m_args_star_list_index != -1 ) { makeStarListTupleCopy( function, python_pars, args, args_size ); } return true; } // Release them all in case of an error. static void releaseParameters( struct Nuitka_FunctionObject const *function, PyObject **python_pars ) { Py_ssize_t arg_count = function->m_args_overall_count; for ( Py_ssize_t i = 0; i < arg_count; i++ ) { Py_XDECREF( python_pars[ i ] ); } } bool parseArgumentsPos( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ) { bool result; Py_ssize_t arg_count = function->m_args_positional_count; #if PYTHON_VERSION >= 330 bool kw_only_error; #endif if (unlikely( arg_count == 0 && function->m_args_simple && args_size != 0 )) { #if PYTHON_VERSION < 330 formatErrorNoArgumentAllowed( function, args_size ); #else formatErrorNoArgumentAllowed( function, NULL, args_size ); #endif goto error_exit; } result = _handleArgumentsPlainOnly( function, python_pars, args, args_size ); if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif if ( function->m_args_star_dict_index != -1 ) { python_pars[ function->m_args_star_dict_index ] = PyDict_New(); } return true; error_exit: releaseParameters( function, python_pars ); return false; } bool parseArgumentsMethodPos( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *object, PyObject **args, Py_ssize_t args_size ) { bool result; #if PYTHON_VERSION >= 330 bool kw_only_error; #endif result = handleMethodArgumentsPlainOnly( function, python_pars, object, args, args_size ); if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif if ( function->m_args_star_dict_index != -1 ) { python_pars[ function->m_args_star_dict_index ] = PyDict_New(); } return true; error_exit: releaseParameters( function, python_pars ); return false; } static bool parseArgumentsFull( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size, PyObject *kw ) { Py_ssize_t kw_size = kw ? DICT_SIZE( kw ) : 0; Py_ssize_t kw_found; bool result; #if PYTHON_VERSION >= 300 Py_ssize_t kw_only_found; #endif #if PYTHON_VERSION >= 330 bool kw_only_error; #endif Py_ssize_t arg_count = function->m_args_keywords_count; assert( kw == NULL || PyDict_CheckExact( kw ) ); if (unlikely( arg_count == 0 && function->m_args_simple && args_size + kw_size > 0 )) { #if PYTHON_VERSION < 330 formatErrorNoArgumentAllowed( function, args_size + kw_size ); #else formatErrorNoArgumentAllowed( function, kw_size > 0 ? kw : NULL, args_size ); #endif goto error_exit; } #if PYTHON_VERSION >= 300 kw_only_found = 0; #endif if ( function->m_args_star_dict_index != -1 ) { #if PYTHON_VERSION < 300 kw_found = handleKeywordArgsWithStarDict( function, python_pars, kw ); #else kw_found = handleKeywordArgsWithStarDict( function, python_pars, &kw_only_found, kw ); #endif if ( kw_found == -1 ) goto error_exit; } else if ( kw == NULL || DICT_SIZE( kw ) == 0 ) { kw_found = 0; } else { #if PYTHON_VERSION < 300 kw_found = handleKeywordArgs( function, python_pars, kw ); #else kw_found = handleKeywordArgs( function, python_pars, &kw_only_found, kw ); #endif if ( kw_found == -1 ) goto error_exit; } #if PYTHON_VERSION < 270 result = handleArgumentsPlain( function, python_pars, kw, args, args_size, kw_found, kw_size ); #elif PYTHON_VERSION < 300 result = handleArgumentsPlain( function, python_pars, kw, args, args_size, kw_found ); #else result = handleArgumentsPlain( function, python_pars, kw, args, args_size, kw_found, kw_only_found ); #endif if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif return true; error_exit: releaseParameters( function, python_pars ); return false; } PyObject *Nuitka_CallFunctionPosArgsKwArgs( struct Nuitka_FunctionObject const *function, PyObject **args, Py_ssize_t args_size, PyObject *kw ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); if (!parseArgumentsFull( function, python_pars, args, args_size, kw )) return NULL; return function->m_c_code( function, python_pars ); } PyObject *Nuitka_CallMethodFunctionNoArgs( struct Nuitka_FunctionObject const *function, PyObject *object ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); bool result = handleMethodArgumentsPlainOnly( function, python_pars, object, NULL, 0 ); // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 bool kw_only_error; #endif if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif if ( function->m_args_star_dict_index != -1 ) { python_pars[ function->m_args_star_dict_index ] = PyDict_New(); } return function->m_c_code( function, python_pars ); error_exit: releaseParameters( function, python_pars ); return NULL; } PyObject *Nuitka_CallMethodFunctionPosArgs( struct Nuitka_FunctionObject const *function, PyObject *object, PyObject **args, Py_ssize_t args_size ) { #ifdef _MSC_VER PyObject **python_pars = (PyObject **)_alloca( sizeof( PyObject * ) * function->m_args_overall_count ); #else PyObject *python_pars[ function->m_args_overall_count ]; #endif memset( python_pars, 0, function->m_args_overall_count * sizeof(PyObject *) ); bool result = handleMethodArgumentsPlainOnly( function, python_pars, object, args, args_size ); // For Python3.3 the keyword only errors are all reported at once. #if PYTHON_VERSION >= 330 bool kw_only_error; #endif if ( result == false ) goto error_exit; #if PYTHON_VERSION >= 300 #if PYTHON_VERSION >= 330 kw_only_error = false; #endif for( Py_ssize_t i = function->m_args_positional_count; i < function->m_args_keywords_count; i++ ) { if ( python_pars[ i ] == NULL ) { PyObject *arg_name = function->m_varnames[ i ]; python_pars[ i ] = PyDict_GetItem( function->m_kwdefaults, arg_name ); #if PYTHON_VERSION < 330 if (unlikely( python_pars[ i ] == NULL )) { PyErr_Format( PyExc_TypeError, "%s() needs keyword-only argument %s", Nuitka_String_AsString( function->m_name ), Nuitka_String_AsString( arg_name ) ); goto error_exit; } Py_INCREF( python_pars[ i ] ); #else if ( python_pars[ i ] == NULL ) { kw_only_error = true; } else { Py_INCREF( python_pars[ i ] ); } #endif } } #if PYTHON_VERSION >= 330 if (unlikely( kw_only_error )) { formatErrorTooFewKwOnlyArguments( function, &python_pars[ function->m_args_positional_count ] ); goto error_exit; } #endif #endif if ( function->m_args_star_dict_index != -1 ) { python_pars[ function->m_args_star_dict_index ] = PyDict_New(); } return function->m_c_code( function, python_pars ); error_exit: releaseParameters( function, python_pars ); return NULL; } PyObject *Nuitka_CallMethodFunctionPosArgsKwArgs( struct Nuitka_FunctionObject const *function, PyObject *object, PyObject **args, Py_ssize_t args_size, PyObject *kw ) { #ifdef _MSC_VER PyObject **new_args = (PyObject **)_alloca( sizeof( PyObject * ) *( args_size + 1 ) ); #else PyObject *new_args[ args_size + 1 ]; #endif new_args[ 0 ] = object; memcpy( new_args + 1, args, args_size * sizeof( PyObject *) ); // TODO: Specialize implementation for massive gains. return Nuitka_CallFunctionPosArgsKwArgs( function, new_args, args_size + 1, kw ); } Nuitka-0.5.28.2/nuitka/build/static_src/HelpersDeepcopy.c0000644000372000001440000002763413134660221023464 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // /** * This is responsible for deep copy and hashing of constants. */ PyObject *DEEP_COPY( PyObject *value ) { if ( PyDict_Check( value ) ) { // For Python3.3, this can be done much faster in the same way as it is // done in parameter parsing. #if PYTHON_VERSION < 330 PyObject *result = _PyDict_NewPresized( ((PyDictObject *)value)->ma_used ); for ( Py_ssize_t i = 0; i <= ((PyDictObject *)value)->ma_mask; i++ ) { PyDictEntry *entry = &((PyDictObject *)value)->ma_table[ i ]; if ( entry->me_value != NULL ) { PyObject *deep_copy = DEEP_COPY( entry->me_value ); int res = PyDict_SetItem( result, entry->me_key, deep_copy ); Py_DECREF( deep_copy ); if (unlikely( res != 0 )) { return NULL; } } } return result; #else /* Python 3.3 or higher */ if ( _PyDict_HasSplitTable( (PyDictObject *)value) ) { PyDictObject *mp = (PyDictObject *)value; PyObject **newvalues = PyMem_NEW( PyObject *, mp->ma_keys->dk_size ); assert( newvalues != NULL ); PyDictObject *result = PyObject_GC_New( PyDictObject, &PyDict_Type ); assert( result != NULL ); result->ma_values = newvalues; result->ma_keys = mp->ma_keys; result->ma_used = mp->ma_used; mp->ma_keys->dk_refcnt += 1; Nuitka_GC_Track( result ); #if PYTHON_VERSION < 360 Py_ssize_t size = mp->ma_keys->dk_size; #else Py_ssize_t size = DK_USABLE_FRACTION(DK_SIZE(mp->ma_keys)); #endif for ( Py_ssize_t i = 0; i < size; i++ ) { if ( mp->ma_values[ i ] ) { result->ma_values[ i ] = DEEP_COPY( mp->ma_values[ i ] ); } else { result->ma_values[ i ] = NULL; } } return (PyObject *)result; } else { PyObject *result = _PyDict_NewPresized( ((PyDictObject *)value)->ma_used ); PyDictObject *mp = (PyDictObject *)value; #if PYTHON_VERSION < 360 Py_ssize_t size = mp->ma_keys->dk_size; #else Py_ssize_t size = mp->ma_keys->dk_nentries; #endif for ( Py_ssize_t i = 0; i < size; i++ ) { #if PYTHON_VERSION < 360 PyDictKeyEntry *entry = &mp->ma_keys->dk_entries[i]; #else PyDictKeyEntry *entry = &DK_ENTRIES( mp->ma_keys )[ i ]; #endif if ( entry->me_value != NULL ) { PyObject *deep_copy = DEEP_COPY( entry->me_value ); PyDict_SetItem( result, entry->me_key, deep_copy ); Py_DECREF( deep_copy ); } } return result; } #endif } else if ( PyTuple_Check( value ) ) { Py_ssize_t n = PyTuple_Size( value ); PyObject *result = PyTuple_New( n ); for( Py_ssize_t i = 0; i < n; i++ ) { PyTuple_SET_ITEM( result, i, DEEP_COPY( PyTuple_GET_ITEM( value, i ) ) ); } return result; } else if ( PyList_Check( value ) ) { Py_ssize_t n = PyList_GET_SIZE( value ); PyObject *result = PyList_New( n ); for( Py_ssize_t i = 0; i < n; i++ ) { PyList_SET_ITEM( result, i, DEEP_COPY( PyList_GET_ITEM( value, i ) ) ); } return result; } else if ( PySet_Check( value ) ) { // Sets cannot contain unhashable types, so they must be immutable. return PySet_New( value ); } else if ( PyFrozenSet_Check( value ) ) { // Sets cannot contain unhashable types, so they must be immutable. return PyFrozenSet_New( value ); } else if ( #if PYTHON_VERSION < 300 PyString_Check( value ) || #endif PyUnicode_Check( value ) || #if PYTHON_VERSION < 300 PyInt_Check( value ) || #endif PyLong_Check( value ) || value == Py_None || PyBool_Check( value ) || PyFloat_Check( value ) || PyBytes_Check( value ) || PyRange_Check( value ) || PyType_Check( value ) || PySlice_Check( value ) || PyComplex_Check( value ) || value == Py_Ellipsis ) { Py_INCREF( value ); return value; } else if ( PyByteArray_Check( value ) ) { // TODO: Could make an exception for zero size. return PyByteArray_FromObject( value ); } else { PyErr_Format( PyExc_TypeError, "DEEP_COPY does not implement: %s", value->ob_type->tp_name ); return NULL; } } #ifndef __NUITKA_NO_ASSERT__ static Py_hash_t DEEP_HASH_INIT( PyObject *value ) { // To avoid warnings about reduced sizes, we put an intermediate value // that is size_t. size_t value2 = (size_t)value; Py_hash_t result = (Py_hash_t)( value2 ); if ( Py_TYPE( value ) != &PyType_Type ) { result ^= DEEP_HASH( (PyObject *)Py_TYPE( value ) ); } return result; } static void DEEP_HASH_BLOB( Py_hash_t *hash, char const *s, Py_ssize_t size ) { while( size > 0 ) { *hash = ( 1000003 * (*hash) ) ^ (Py_hash_t)( *s++ ); size--; } } static void DEEP_HASH_CSTR( Py_hash_t *hash, char const *s ) { DEEP_HASH_BLOB( hash, s, strlen( s ) ); } // Hash function that actually verifies things done to the bit level. Can be // used to detect corruption. Py_hash_t DEEP_HASH( PyObject *value ) { assert( value != NULL ); if ( PyType_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); DEEP_HASH_CSTR( &result, ((PyTypeObject *)value)->tp_name ); return result; } else if ( PyDict_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t ppos = 0; PyObject *key, *dict_value; while( PyDict_Next( value, &ppos, &key, &dict_value ) ) { if ( key != NULL && value != NULL ) { result ^= DEEP_HASH( key ); result ^= DEEP_HASH( dict_value ); } } return result; } else if ( PyTuple_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t n = PyTuple_Size( value ); for( Py_ssize_t i = 0; i < n; i++ ) { result ^= DEEP_HASH( PyTuple_GET_ITEM( value, i ) ); } return result; } else if ( PyList_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t n = PyList_GET_SIZE( value ); for( Py_ssize_t i = 0; i < n; i++ ) { result ^= DEEP_HASH( PyList_GET_ITEM( value, i ) ); } return result; } else if ( PySet_Check( value ) || PyFrozenSet_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); PyObject *iterator = PyObject_GetIter( value ); CHECK_OBJECT( iterator ); while( true ) { PyObject *item = PyIter_Next( iterator ); if (!item) break; CHECK_OBJECT( item ); result ^= DEEP_HASH( item ); Py_DECREF( item ); } Py_DECREF( iterator ); return result; } else if ( PyLong_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); // Use string to hash the long value, which relies on that to not // use the object address. PyObject *str = PyObject_Str( value ); result ^= DEEP_HASH( str ); Py_DECREF( str ); RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return result; } else if ( PyUnicode_Check( value ) ) { Py_hash_t result = DEEP_HASH( (PyObject *)Py_TYPE( value ) ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED_UNTRACED( &exception_type, &exception_value, &exception_tb ); #if PYTHON_PYTHON >= 330 Py_ssize_t size; char *s = PyUnicode_AsUTF8AndSize( value, &size ); if ( s != NULL ) { DEEP_HASH_BLOB( &result, s, size ); } #elif PYTHON_VERSION >= 300 // Not done for Python3.2 really yet. #else PyObject *str = PyUnicode_AsUTF8String( value ); if ( str ) { result ^= DEEP_HASH( str ); } Py_DECREF( str ); #endif RESTORE_ERROR_OCCURRED_UNTRACED( exception_type, exception_value, exception_tb ); return result; } #if PYTHON_VERSION < 300 else if ( PyString_Check( value ) ) { Py_hash_t result = DEEP_HASH( (PyObject *)Py_TYPE( value ) ); Py_ssize_t size; char *s; int res = PyString_AsStringAndSize( value, &s, &size ); assert( res != -1 ); DEEP_HASH_BLOB( &result, s, size ); return result; } #else else if ( PyBytes_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size; char *s; int res = PyBytes_AsStringAndSize( value, &s, &size ); assert( res != -1 ); DEEP_HASH_BLOB( &result, s, size ); return result; } #endif else if ( PyByteArray_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size = PyByteArray_Size( value ); assert( size >= 0 ); char *s = PyByteArray_AsString( value ); DEEP_HASH_BLOB( &result, s, size ); return result; } else if ( value == Py_None || value == Py_Ellipsis ) { return DEEP_HASH_INIT( value ); } else if ( PyComplex_Check( value ) ) { Py_complex c = PyComplex_AsCComplex( value ); Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size = sizeof(c); char *s = (char *)&c; DEEP_HASH_BLOB( &result, s, size ); return result; } else if ( PyFloat_Check( value ) ) { double f = PyFloat_AsDouble( value ); Py_hash_t result = DEEP_HASH_INIT( value ); Py_ssize_t size = sizeof(f); char *s = (char *)&f; DEEP_HASH_BLOB( &result, s, size ); return result; } else if ( #if PYTHON_VERSION < 300 PyInt_Check( value ) || #endif PyBool_Check( value ) || PyRange_Check( value ) || PySlice_Check( value ) ) { Py_hash_t result = DEEP_HASH_INIT( value ); #if 0 printf("Too simple deep hash: %s\n", Py_TYPE( value )->tp_name ); #endif return result; } else { assert( false ); return -1; } } #endif Nuitka-0.5.28.2/nuitka/build/static_src/win32_ucontext_src/0000755000372000001440000000000013207540420023752 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/static_src/win32_ucontext_src/fibers_win32.c0000644000372000001440000000317113112214770026415 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for Win32 #include "nuitka/prelude.h" // Less than 1MB is ignored on Win32 apparently. #ifdef MS_WIN64 #define STACK_SIZE 2*(1024*1024) #else #define STACK_SIZE (1024*1024) #endif void _initFiber( Fiber *to ) { // Need to call this at least once per thread, so we have a main FIBER. ConvertThreadToFiber( NULL ); to->fiber = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { to->fiber = CreateFiber( STACK_SIZE, (LPFIBER_START_ROUTINE)code, (LPVOID)arg ); return to->fiber != NULL ? 0 : 1; } void _releaseFiber( Fiber *to ) { if ( to->fiber ) { DeleteFiber( to->fiber ); to->fiber = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { to->fiber = GetCurrentFiber(); assert( from->fiber != NULL ); SwitchToFiber( from->fiber ); } Nuitka-0.5.28.2/nuitka/build/static_src/CompiledAsyncgenType.c0000644000372000001440000017372113207537242024465 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/freelists.h" static PyObject *Nuitka_Asyncgen_get_name( struct Nuitka_AsyncgenObject *asyncgen ) { Py_INCREF( asyncgen->m_name ); return asyncgen->m_name; } static int Nuitka_Asyncgen_set_name( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *tmp = asyncgen->m_name; Py_INCREF( value ); asyncgen->m_name = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Asyncgen_get_qualname( struct Nuitka_AsyncgenObject *asyncgen ) { Py_INCREF( asyncgen->m_qualname ); return asyncgen->m_qualname; } static int Nuitka_Asyncgen_set_qualname( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *tmp = asyncgen->m_qualname; Py_INCREF( value ); asyncgen->m_qualname = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Asyncgen_get_ag_await( struct Nuitka_AsyncgenObject *asyncgen ) { if ( asyncgen->m_yieldfrom ) { Py_INCREF( asyncgen->m_yieldfrom ); return asyncgen->m_yieldfrom; } else { Py_INCREF( Py_None ); return Py_None; } } static PyObject *Nuitka_Asyncgen_get_code( struct Nuitka_AsyncgenObject *asyncgen ) { Py_INCREF( asyncgen->m_code_object ); return (PyObject *)asyncgen->m_code_object; } static int Nuitka_Asyncgen_set_code( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "ag_code is not writable in Nuitka" ); return -1; } static void Nuitka_Asyncgen_release_closure( struct Nuitka_AsyncgenObject *asyncgen ) { for( Py_ssize_t i = 0; i < asyncgen->m_closure_given; i++ ) { CHECK_OBJECT( asyncgen->m_closure[ i ] ); Py_DECREF( asyncgen->m_closure[ i ] ); } asyncgen->m_closure_given = 0; } #ifdef _NUITKA_MAKECONTEXT_INTS static void Nuitka_Asyncgen_entry_point( int address_1, int address_2 ) { // Restore the pointer from integers should it be necessary, depending on // the platform. This requires pointers to be no larger that to "int" value. int addresses[2] = { address_1, address_2 }; struct Nuitka_AsyncgenObject *asyncgen = (struct Nuitka_AsyncgenObject *)*(uintptr_t *)&addresses[0]; #else static void Nuitka_Asyncgen_entry_point( struct Nuitka_AsyncgenObject *asyncgen ) { #endif ((asyncgen_code)asyncgen->m_code)( asyncgen ); swapFiber( &asyncgen->m_yielder_context, &asyncgen->m_caller_context ); } static PyObject *_Nuitka_Asyncgen_send( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value, bool closing ) { if ( value == NULL ) value = Py_None; if ( asyncgen->m_status == status_Unused && value != NULL && value != Py_None ) { PyErr_Format( PyExc_TypeError, "can't send non-None value to a just-started async generator" ); return NULL; } if ( asyncgen->m_status != status_Finished ) { PyThreadState *thread_state = PyThreadState_GET(); if ( asyncgen->m_running ) { PyErr_Format( PyExc_ValueError, "async generator already executing" ); return NULL; } if ( asyncgen->m_status == status_Unused ) { // Prepare the asyncgen context to run. int res = prepareFiber( &asyncgen->m_yielder_context, (void *)Nuitka_Asyncgen_entry_point, (uintptr_t)asyncgen ); if ( res != 0 ) { PyErr_Format( PyExc_MemoryError, "async generator cannot be allocated" ); return NULL; } asyncgen->m_status = status_Running; } asyncgen->m_yielded = value; // Put the generator back on the frame stack. PyFrameObject *return_frame = thread_state->frame; #ifndef __NUITKA_NO_ASSERT__ if ( return_frame ) { assertFrameObject( (struct Nuitka_FrameObject *)return_frame ); } #endif if ( asyncgen->m_frame ) { // It would be nice if our frame were still alive. Nobody had the // right to release it. assertFrameObject( asyncgen->m_frame ); // It's not supposed to be on the top right now. assert( return_frame != &asyncgen->m_frame->m_frame ); Py_XINCREF( return_frame ); asyncgen->m_frame->m_frame.f_back = return_frame; thread_state->frame = &asyncgen->m_frame->m_frame; } // Continue the yielder function while preventing recursion. asyncgen->m_running = true; swapFiber( &asyncgen->m_caller_context, &asyncgen->m_yielder_context ); asyncgen->m_running = false; thread_state = PyThreadState_GET(); // Remove the asyncgen from the frame stack. if ( asyncgen->m_frame ) { assert( thread_state->frame == &asyncgen->m_frame->m_frame ); assertFrameObject( asyncgen->m_frame ); Py_CLEAR( asyncgen->m_frame->m_frame.f_back ); } thread_state->frame = return_frame; // Generator return does set this. if ( asyncgen->m_status == status_Finished ) { Py_XDECREF( asyncgen->m_frame ); asyncgen->m_frame = NULL; Nuitka_Asyncgen_release_closure( asyncgen ); return NULL; } else if ( asyncgen->m_yielded == NULL ) { assert( ERROR_OCCURRED() ); asyncgen->m_status = status_Finished; Py_XDECREF( asyncgen->m_frame ); asyncgen->m_frame = NULL; Nuitka_Asyncgen_release_closure( asyncgen ); assert( ERROR_OCCURRED() ); PyObject *error_occurred = GET_ERROR_OCCURRED(); if ( error_occurred == PyExc_StopIteration || error_occurred == PyExc_StopAsyncIteration ) { PyObject *saved_exception_type, *saved_exception_value; PyTracebackObject *saved_exception_tb; FETCH_ERROR_OCCURRED( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); NORMALIZE_EXCEPTION( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); if ( error_occurred == PyExc_StopIteration ) { PyErr_Format( PyExc_RuntimeError, "async generator raised StopIteration" ); } else { PyErr_Format( PyExc_RuntimeError, "async generator raised StopAsyncIteration" ); } PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, saved_exception_value ); CHECK_OBJECT( exception_value ); CHECK_OBJECT( saved_exception_value ); Py_INCREF( saved_exception_value ); PyException_SetContext( exception_value, saved_exception_value ); Py_DECREF( saved_exception_type ); Py_XDECREF( saved_exception_tb ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } return NULL; } else { return asyncgen->m_yielded; } } else { PyErr_SetObject( PyExc_StopAsyncIteration, (PyObject *)NULL ); return NULL; } } PyObject *Nuitka_Asyncgen_close( struct Nuitka_AsyncgenObject *asyncgen, PyObject *args ) { if ( asyncgen->m_status == status_Running ) { asyncgen->m_exception_type = PyExc_GeneratorExit; Py_INCREF( PyExc_GeneratorExit ); asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; PyObject *result = _Nuitka_Asyncgen_send( asyncgen, Py_None, true ); if (unlikely( result )) { Py_DECREF( result ); PyErr_Format( PyExc_RuntimeError, "async generator ignored GeneratorExit" ); return NULL; } else { PyObject *error = GET_ERROR_OCCURRED(); assert( error != NULL ); if ( EXCEPTION_MATCH_GENERATOR( error ) ) { CLEAR_ERROR_OCCURRED(); Py_INCREF( Py_None ); return Py_None; } return NULL; } } Py_INCREF( Py_None ); return Py_None; } /* Shared from coroutines. */ extern bool Nuitka_gen_close_iter( PyObject *yieldfrom ); extern PyObject *const_str_plain_throw; static PyObject *_Nuitka_Asyncgen_throw2( struct Nuitka_AsyncgenObject *asyncgen, bool close_on_genexit ) { if ( asyncgen->m_yieldfrom != NULL ) { if ( close_on_genexit ) { if ( PyErr_GivenExceptionMatches( asyncgen->m_exception_type, PyExc_GeneratorExit ) ) { // Asynchronous generators need to close the yield_from. asyncgen->m_running = 1; bool res = Nuitka_gen_close_iter( asyncgen->m_yieldfrom ); asyncgen->m_running = 0; if ( res == true ) { return _Nuitka_Asyncgen_send( asyncgen, Py_None, false ); } goto throw_here; } } PyObject *meth = PyObject_GetAttr( asyncgen->m_yieldfrom, const_str_plain_throw ); if (unlikely( meth == NULL )) { if ( !PyErr_ExceptionMatches( PyExc_AttributeError ) ) { return NULL; } CLEAR_ERROR_OCCURRED(); goto throw_here; } asyncgen->m_running = 1; PyObject *ret = PyObject_CallFunctionObjArgs( meth, asyncgen->m_exception_type, asyncgen->m_exception_value, asyncgen->m_exception_tb, NULL ); asyncgen->m_running = 0; Py_DECREF( meth ); if (unlikely( ret == NULL )) { PyObject *val; if ( _PyGen_FetchStopIterationValue( &val ) == 0 ) { ret = _Nuitka_Asyncgen_send( asyncgen, val, false ); Py_DECREF( val ); } else { ret = _Nuitka_Asyncgen_send( asyncgen, Py_None, false ); } } return ret; } throw_here: if ( (PyObject *)asyncgen->m_exception_tb == Py_None ) { asyncgen->m_exception_tb = NULL; } else if ( asyncgen->m_exception_tb != NULL && !PyTraceBack_Check( asyncgen->m_exception_tb ) ) { asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( PyExceptionClass_Check( asyncgen->m_exception_type )) { Py_INCREF( asyncgen->m_exception_type ); Py_XINCREF( asyncgen->m_exception_value ); Py_XINCREF( asyncgen->m_exception_tb ); NORMALIZE_EXCEPTION( &asyncgen->m_exception_type, &asyncgen->m_exception_value, &asyncgen->m_exception_tb ); } else if ( PyExceptionInstance_Check( asyncgen->m_exception_type ) ) { if ( asyncgen->m_exception_value && asyncgen->m_exception_value != Py_None ) { asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); return NULL; } asyncgen->m_exception_value = asyncgen->m_exception_type; Py_INCREF( asyncgen->m_exception_value ); asyncgen->m_exception_type = PyExceptionInstance_Class( asyncgen->m_exception_type ); Py_INCREF( asyncgen->m_exception_type ); Py_XINCREF( asyncgen->m_exception_tb ); } else { PyErr_Format( PyExc_TypeError, "exceptions must be classes or instances deriving from BaseException, not %s", Py_TYPE( asyncgen->m_exception_type )->tp_name ); asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; return NULL; } if ( ( asyncgen->m_exception_tb != NULL ) && ( (PyObject *)asyncgen->m_exception_tb != Py_None ) && ( !PyTraceBack_Check( asyncgen->m_exception_tb ) ) ) { PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( asyncgen->m_status != status_Finished ) { PyObject *result = _Nuitka_Asyncgen_send( asyncgen, Py_None, false ); return result; } else { PyErr_SetNone( PyExc_StopAsyncIteration ); return NULL; } } static PyObject *_Nuitka_Asyncgen_throw( struct Nuitka_AsyncgenObject *asyncgen, bool close_for_exit, PyObject *exc_type, PyObject *exc_value, PyObject *exc_tb ) { asyncgen->m_exception_type = exc_type; asyncgen->m_exception_value = exc_value; asyncgen->m_exception_tb = (PyTracebackObject *)exc_tb; return _Nuitka_Asyncgen_throw2( asyncgen, close_for_exit ); } static PyObject *Nuitka_Asyncgen_throw( struct Nuitka_AsyncgenObject *asyncgen, PyObject *args ) { assert( asyncgen->m_exception_type == NULL ); assert( asyncgen->m_exception_value == NULL ); assert( asyncgen->m_exception_tb == NULL ); int res = PyArg_UnpackTuple( args, "throw", 1, 3, &asyncgen->m_exception_type, &asyncgen->m_exception_value, (PyObject **)&asyncgen->m_exception_tb ); if (unlikely( res == 0 )) { asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; return NULL; } return _Nuitka_Asyncgen_throw2( asyncgen, true ); } static int Nuitka_Asyncgen_init_hooks( struct Nuitka_AsyncgenObject *asyncgen ) { /* Just do this once per async generator object. */ if ( asyncgen->m_hooks_init_done ) { return 0; } asyncgen->m_hooks_init_done = 1; PyThreadState *tstate = PyThreadState_GET(); /* Attach the finalizer if any. */ PyObject *finalizer = tstate->async_gen_finalizer; if ( finalizer ) { Py_INCREF( finalizer ); asyncgen->m_finalizer = finalizer; } /* Call the "firstiter" hook for async generator. */ PyObject *firstiter = tstate->async_gen_firstiter; if ( firstiter ) { Py_INCREF( firstiter ); PyObject *args[] = { (PyObject *)asyncgen }; PyObject *res = CALL_FUNCTION_WITH_ARGS1( firstiter, args ); Py_DECREF( firstiter ); if (unlikely( res == NULL )) { return 1; } Py_DECREF( res ); } return 0; } static PyObject *Nuitka_AsyncgenAsend_New( struct Nuitka_AsyncgenObject *asyncgen, PyObject *sendval ); static PyObject *Nuitka_AsyncgenAthrow_New( struct Nuitka_AsyncgenObject *asyncgen, PyObject *args ); static PyObject *Nuitka_Asyncgen_anext( struct Nuitka_AsyncgenObject *asyncgen ) { if ( Nuitka_Asyncgen_init_hooks( asyncgen ) ) { return NULL; } #if _DEBUG_ASYNCGEN PRINT_STRING("Nuitka_Asyncgen_anext: Making Nuitka_AsyncgenAsend object."); PRINT_NEW_LINE(); #endif return Nuitka_AsyncgenAsend_New( asyncgen, NULL ); } static PyObject *Nuitka_Asyncgen_asend( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { if ( Nuitka_Asyncgen_init_hooks( asyncgen ) ) { return NULL; } return Nuitka_AsyncgenAsend_New( asyncgen, value ); } static PyObject *Nuitka_Asyncgen_aclose( struct Nuitka_AsyncgenObject *asyncgen, PyObject *arg ) { if ( Nuitka_Asyncgen_init_hooks( asyncgen ) ) { return NULL; } return Nuitka_AsyncgenAthrow_New( asyncgen, NULL ); } static PyObject *Nuitka_Asyncgen_athrow( struct Nuitka_AsyncgenObject *asyncgen, PyObject *args ) { if ( Nuitka_Asyncgen_init_hooks( asyncgen ) ) { return NULL; } return Nuitka_AsyncgenAthrow_New( asyncgen, args ); } #define MAX_ASYNCGEN_FREE_LIST_COUNT 100 static struct Nuitka_AsyncgenObject *free_list_asyncgens = NULL; static int free_list_asyncgens_count = 0; // TODO: This might have to be finalize actually. static void Nuitka_Asyncgen_tp_dealloc( struct Nuitka_AsyncgenObject *asyncgen ) { // Revive temporarily. assert( Py_REFCNT( asyncgen ) == 0 ); Py_REFCNT( asyncgen ) = 1; // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; PyObject *finalizer = asyncgen->m_finalizer; if ( finalizer && asyncgen->m_closed == false ) { /* Save the current exception, if any. */ FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); // TODO: Call with args 1 function. PyObject *res = PyObject_CallFunctionObjArgs( finalizer, asyncgen, NULL ); if (unlikely( res == NULL )) { PyErr_WriteUnraisable( (PyObject *)asyncgen ); } else { Py_DECREF( res ); } RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); return; } FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); PyObject *close_result = Nuitka_Asyncgen_close( asyncgen, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)asyncgen ); } else { Py_DECREF( close_result ); } Nuitka_Asyncgen_release_closure( asyncgen ); Py_XDECREF( asyncgen->m_frame ); assert( Py_REFCNT( asyncgen ) == 1 ); Py_REFCNT( asyncgen ) = 0; releaseFiber( &asyncgen->m_yielder_context ); // Now it is safe to release references and memory for it. Nuitka_GC_UnTrack( asyncgen ); Py_CLEAR( asyncgen->m_finalizer ); if ( asyncgen->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)asyncgen ); assert( !ERROR_OCCURRED() ); } Py_DECREF( asyncgen->m_name ); Py_DECREF( asyncgen->m_qualname ); /* Put the object into freelist or release to GC */ releaseToFreeList( free_list_asyncgens, asyncgen, MAX_ASYNCGEN_FREE_LIST_COUNT ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } static PyObject *Nuitka_Asyncgen_tp_repr( struct Nuitka_AsyncgenObject *asyncgen ) { return PyUnicode_FromFormat( "", Nuitka_String_AsString( asyncgen->m_qualname ), asyncgen ); } static int Nuitka_Asyncgen_tp_traverse( struct Nuitka_AsyncgenObject *asyncgen, visitproc visit, void *arg ) { Py_VISIT( asyncgen->m_finalizer ); // TODO: Identify the impact of not visiting owned objects and/or if it // could be NULL instead. The "methodobject" visits its self and module. I // understand this is probably so that back references of this function to // its upper do not make it stay in the memory. A specific test if that // works might be needed. return 0; } #include // TODO: Set "__doc__" automatically for method clones of compiled types from // the documentation of built-in original type. static PyMethodDef Nuitka_Asyncgen_methods[] = { { "asend", (PyCFunction)Nuitka_Asyncgen_asend, METH_O, NULL }, { "athrow", (PyCFunction)Nuitka_Asyncgen_athrow, METH_VARARGS, NULL }, { "aclose", (PyCFunction)Nuitka_Asyncgen_aclose, METH_NOARGS, NULL }, { NULL } }; static PyAsyncMethods Nuitka_Asyncgen_as_async = { 0, /* am_await */ PyObject_SelfIter, /* am_aiter */ (unaryfunc)Nuitka_Asyncgen_anext /* am_anext */ }; // TODO: Set "__doc__" automatically for method clones of compiled types from // the documentation of built-in original type. static PyGetSetDef Nuitka_Asyncgen_getsetlist[] = { { (char *)"__name__", (getter)Nuitka_Asyncgen_get_name, (setter)Nuitka_Asyncgen_set_name, NULL }, { (char *)"__qualname__", (getter)Nuitka_Asyncgen_get_qualname, (setter)Nuitka_Asyncgen_set_qualname, NULL }, { (char *)"ag_await", (getter)Nuitka_Asyncgen_get_ag_await, (setter)NULL, NULL }, { (char *)"ag_code", (getter)Nuitka_Asyncgen_get_code, (setter)Nuitka_Asyncgen_set_code, NULL }, { NULL } }; static PyMemberDef Nuitka_Asyncgen_members[] = { { (char *)"ag_frame", T_OBJECT, offsetof(struct Nuitka_AsyncgenObject, m_frame), READONLY }, { (char *)"ag_running", T_BOOL, offsetof(struct Nuitka_AsyncgenObject, m_running), READONLY }, { NULL } }; PyTypeObject Nuitka_Asyncgen_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_async_generator", /* tp_name */ sizeof(struct Nuitka_AsyncgenObject), /* tp_basicsize */ sizeof(struct Nuitka_CellObject *), /* tp_itemsize */ (destructor)Nuitka_Asyncgen_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ &Nuitka_Asyncgen_as_async, /* tp_as_async */ (reprfunc)Nuitka_Asyncgen_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Asyncgen_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof(struct Nuitka_AsyncgenObject, m_weakrefs), /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Nuitka_Asyncgen_methods, /* tp_methods */ Nuitka_Asyncgen_members, /* tp_members */ Nuitka_Asyncgen_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ }; PyObject *Nuitka_Asyncgen_New( asyncgen_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, Py_ssize_t closure_given ) { struct Nuitka_AsyncgenObject *result; // Macro to assign result memory from GC or free list. allocateFromFreeList( free_list_asyncgens, struct Nuitka_AsyncgenObject, Nuitka_Asyncgen_Type, closure_given ); result->m_code = (void *)code; CHECK_OBJECT( name ); result->m_name = name; Py_INCREF( name ); CHECK_OBJECT( qualname ); result->m_qualname = qualname; Py_INCREF( qualname ); // TODO: Makes no sense with asyncgens maybe? result->m_yieldfrom = NULL; // The m_closure is set from the outside. result->m_closure_given = closure_given; result->m_weakrefs = NULL; result->m_status = status_Unused; result->m_running = false; result->m_awaiting = false; result->m_exception_type = NULL; result->m_exception_value = NULL; result->m_exception_tb = NULL; result->m_yielded = NULL; result->m_frame = NULL; result->m_code_object = code_object; result->m_finalizer = NULL; result->m_hooks_init_done = false; result->m_closed = false; initFiber( &result->m_yielder_context ); Nuitka_GC_Track( result ); return (PyObject *)result; } struct Nuitka_AsyncgenWrappedValueObject { PyObject_HEAD PyObject *m_value; }; static struct Nuitka_AsyncgenWrappedValueObject *free_list_asyncgen_value_wrappers = NULL; static int free_list_asyncgen_value_wrappers_count = 0; static void asyncgen_value_wrapper_tp_dealloc( struct Nuitka_AsyncgenWrappedValueObject *asyncgen_value_wrapper ) { Nuitka_GC_UnTrack( (PyObject *)asyncgen_value_wrapper ); Py_DECREF( asyncgen_value_wrapper->m_value ); releaseToFreeList( free_list_asyncgen_value_wrappers, asyncgen_value_wrapper, MAX_ASYNCGEN_FREE_LIST_COUNT ); } static int asyncgen_value_wrapper_tp_traverse( struct Nuitka_AsyncgenWrappedValueObject *asyncgen_value_wrapper, visitproc visit, void *arg ) { Py_VISIT( asyncgen_value_wrapper->m_value ); return 0; } static PyTypeObject Nuitka_AsyncgenValueWrapper_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_async_generator_wrapped_value", /* tp_name */ sizeof(struct Nuitka_AsyncgenWrappedValueObject), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor)asyncgen_value_wrapper_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)asyncgen_value_wrapper_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ }; PyObject *Nuitka_AsyncGenValueWrapperNew( PyObject *value ) { CHECK_OBJECT( value ); struct Nuitka_AsyncgenWrappedValueObject *result; allocateFromFreeListFixed( free_list_asyncgen_value_wrappers, struct Nuitka_AsyncgenWrappedValueObject, Nuitka_AsyncgenValueWrapper_Type ); result->m_value = value; Py_INCREF( value ); Nuitka_GC_Track( result ); return (PyObject *)result; } #define Nuitka_AsyncgenWrappedValue_CheckExact(o) (Py_TYPE(o) == &Nuitka_AsyncgenValueWrapper_Type) // TODO: We could change all generators to use that kind of enum, if // it's properly supported. typedef enum { AWAITABLE_STATE_INIT = 0, /* Has not yet been iterated. */ AWAITABLE_STATE_ITER = 1, /* Being iterated currently. */ AWAITABLE_STATE_CLOSED = 2, /* Closed, no more. */ } AwaitableState; struct Nuitka_AsyncgenAsendObject { PyObject_HEAD struct Nuitka_AsyncgenObject *m_gen; PyObject *m_sendval; AwaitableState m_state; }; /** * These can be created by byte code loop, and we don't now its internals, * yet we have to unwrap ourselves too. These could break in future updates, * and ideally we would have checks to cover those. */ typedef struct { PyObject_HEAD PyObject *agw_val; } _PyAsyncGenWrappedValue; #define _PyAsyncGenWrappedValue_CheckExact(o) (Py_TYPE(o) == &_PyAsyncGenWrappedValue_Type) static PyObject *Nuitka_Asyncgen_unwrap_value( struct Nuitka_AsyncgenObject *asyncgen, PyObject *result ) { if ( result == NULL ) { if ( !ERROR_OCCURRED() ) { PyErr_SetNone( PyExc_StopAsyncIteration ); asyncgen->m_closed = true; } else if ( PyErr_ExceptionMatches( PyExc_StopAsyncIteration ) || PyErr_ExceptionMatches( PyExc_GeneratorExit ) ) { asyncgen->m_closed = true; } return NULL; } if ( _PyAsyncGenWrappedValue_CheckExact( result ) ) { /* async yield */ _PyGen_SetStopIterationValue( ((_PyAsyncGenWrappedValue*)result)->agw_val ); Py_DECREF( result ); return NULL; } else if ( Nuitka_AsyncgenWrappedValue_CheckExact( result ) ) { /* async yield */ _PyGen_SetStopIterationValue( ((struct Nuitka_AsyncgenWrappedValueObject *)result)->m_value ); Py_DECREF( result ); return NULL; } return result; } static struct Nuitka_AsyncgenAsendObject *free_list_asyncgen_asends = NULL; static int free_list_asyncgen_asends_count = 0; static void Nuitka_AsyncgenAsend_tp_dealloc( struct Nuitka_AsyncgenAsendObject *asyncgen_asend ) { Nuitka_GC_UnTrack( asyncgen_asend ); Py_DECREF( asyncgen_asend->m_gen ); Py_XDECREF( asyncgen_asend->m_sendval ); releaseToFreeList( free_list_asyncgen_asends, asyncgen_asend, MAX_ASYNCGEN_FREE_LIST_COUNT ); } static int Nuitka_AsyncgenAsend_tp_traverse( struct Nuitka_AsyncgenAsendObject *asyncgen_asend, visitproc visit, void *arg ) { Py_VISIT( asyncgen_asend->m_gen ); Py_VISIT( asyncgen_asend->m_sendval ); return 0; } static PyObject *Nuitka_AsyncgenAsend_send( struct Nuitka_AsyncgenAsendObject *asyncgen_asend, PyObject *arg ) { #if _DEBUG_ASYNCGEN PRINT_STRING("Nuitka_AsyncgenAsend_send: Enter with state" ); printf("State on entry is asyncgen_send->m_state = %d\n", asyncgen_asend->m_state ); fflush(stdout); PRINT_NEW_LINE(); #endif PyObject *result; if ( asyncgen_asend->m_state == AWAITABLE_STATE_CLOSED ) { PyErr_SetNone( PyExc_StopIteration ); return NULL; } if ( asyncgen_asend->m_state == AWAITABLE_STATE_INIT ) { if ( arg == NULL || arg == Py_None ) { arg = asyncgen_asend->m_sendval; } asyncgen_asend->m_state = AWAITABLE_STATE_ITER; } result = _Nuitka_Asyncgen_send( asyncgen_asend->m_gen, arg, false ); result = Nuitka_Asyncgen_unwrap_value( asyncgen_asend->m_gen, result ); if ( result == NULL ) { asyncgen_asend->m_state = AWAITABLE_STATE_CLOSED; } return result; } static PyObject *Nuitka_AsyncgenAsend_tp_iternext( struct Nuitka_AsyncgenAsendObject *asyncgen_asend ) { #if _DEBUG_ASYNCGEN PRINT_STRING("Nuitka_AsyncgenAsend_tp_iternext: refer to Nuitka_AsyncgenAsend_send"); PRINT_NEW_LINE(); #endif return Nuitka_AsyncgenAsend_send( asyncgen_asend, NULL ); } static PyObject *Nuitka_AsyncgenAsend_throw( struct Nuitka_AsyncgenAsendObject *asyncgen_asend, PyObject *args ) { PyObject *result; if ( asyncgen_asend->m_state == AWAITABLE_STATE_CLOSED ) { PyErr_SetNone( PyExc_StopIteration ); return NULL; } result = Nuitka_Asyncgen_throw( asyncgen_asend->m_gen, args ); result = Nuitka_Asyncgen_unwrap_value( asyncgen_asend->m_gen, result ); if ( asyncgen_asend == NULL ) { asyncgen_asend->m_state = AWAITABLE_STATE_CLOSED; } return result; } static PyObject *Nuitka_AsyncgenAsend_close( struct Nuitka_AsyncgenAsendObject *asyncgen_asend, PyObject *args ) { asyncgen_asend->m_state = AWAITABLE_STATE_CLOSED; Py_INCREF( Py_None ); return Py_None; } static PyMethodDef Nuitka_AsyncgenAsend_methods[] = { { "send", (PyCFunction)Nuitka_AsyncgenAsend_send, METH_O, NULL}, { "throw", (PyCFunction)Nuitka_AsyncgenAsend_throw, METH_VARARGS, NULL}, { "close", (PyCFunction)Nuitka_AsyncgenAsend_close, METH_NOARGS, NULL}, { NULL } }; static PyAsyncMethods Nuitka_AsyncgenAsend_as_async = { PyObject_SelfIter, /* am_await */ 0, /* am_aiter */ 0 /* am_anext */ }; static PyTypeObject Nuitka_AsyncgenAsend_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_async_generator_asend", /* tp_name */ sizeof(struct Nuitka_AsyncgenAsendObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_AsyncgenAsend_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ &Nuitka_AsyncgenAsend_as_async, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_AsyncgenAsend_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_AsyncgenAsend_tp_iternext, /* tp_iternext */ Nuitka_AsyncgenAsend_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ }; static PyObject *Nuitka_AsyncgenAsend_New( struct Nuitka_AsyncgenObject *asyncgen, PyObject *sendval ) { struct Nuitka_AsyncgenAsendObject *result; allocateFromFreeListFixed( free_list_asyncgen_asends, struct Nuitka_AsyncgenAsendObject, Nuitka_AsyncgenAsend_Type ) Py_INCREF( asyncgen ); result->m_gen = asyncgen; // TODO: We could make the user do that, so NULL is handled faster. Py_XINCREF( sendval ); result->m_sendval = sendval; result->m_state = AWAITABLE_STATE_INIT; Nuitka_GC_Track( result ); return (PyObject*)result; } struct Nuitka_AsyncgenAthrowObject { PyObject_HEAD // The asyncgen we are working for. struct Nuitka_AsyncgenObject *m_gen; // Arguments, NULL in case of close, otherwise throw arguments. PyObject *m_args; AwaitableState m_state; }; static struct Nuitka_AsyncgenAthrowObject *free_list_asyncgen_athrows = NULL; static int free_list_asyncgen_athrows_count = 0; static void Nuitka_AsyncgenAthrow_dealloc( struct Nuitka_AsyncgenAthrowObject *asyncgen_athrow ) { Nuitka_GC_UnTrack( asyncgen_athrow ); Py_DECREF( asyncgen_athrow->m_gen ); Py_XDECREF( asyncgen_athrow->m_args ); releaseToFreeList( free_list_asyncgen_athrows, asyncgen_athrow, MAX_ASYNCGEN_FREE_LIST_COUNT ); } static int Nuitka_AsyncgenAthrow_traverse( struct Nuitka_AsyncgenAthrowObject *asyncgen_athrow, visitproc visit, void *arg ) { Py_VISIT( asyncgen_athrow->m_gen ); Py_VISIT( asyncgen_athrow->m_args ); return 0; } static PyObject *Nuitka_AsyncgenAthrow_send( struct Nuitka_AsyncgenAthrowObject *asyncgen_athrow, PyObject *arg ) { struct Nuitka_AsyncgenObject *asyncgen = asyncgen_athrow->m_gen; // If finished, just report StopIteration. if ( asyncgen->m_status == status_Finished || asyncgen_athrow->m_state == AWAITABLE_STATE_CLOSED ) { PyErr_SetNone( PyExc_StopIteration ); return NULL; } PyObject *retval; if ( asyncgen_athrow->m_state == AWAITABLE_STATE_INIT ) { // Can also close only once. if ( asyncgen->m_closed ) { PyErr_SetNone( PyExc_StopIteration ); return NULL; } // Starting accepts only "None" as input value. if ( arg != Py_None ) { PyErr_Format( PyExc_RuntimeError, "can't send non-None value to a just-started coroutine" ); return NULL; } asyncgen_athrow->m_state = AWAITABLE_STATE_ITER; if ( asyncgen_athrow->m_args == NULL ) { asyncgen->m_closed = true; retval = _Nuitka_Asyncgen_throw( asyncgen, 0, /* Do not close generator when PyExc_GeneratorExit is passed */ PyExc_GeneratorExit, NULL, NULL ); if ( retval ) { if ( _PyAsyncGenWrappedValue_CheckExact(retval) || Nuitka_AsyncgenWrappedValue_CheckExact(retval) ) { Py_DECREF( retval ); PyErr_Format( PyExc_RuntimeError, "async generator ignored GeneratorExit" ); return NULL; } } } else { PyObject *typ; PyObject *tb = NULL; PyObject *val = NULL; if (unlikely( !PyArg_UnpackTuple( asyncgen_athrow->m_args, "athrow", 1, 3, &typ, &val, &tb))) { return NULL; } retval = _Nuitka_Asyncgen_throw( asyncgen, 0, /* Do not close generator when PyExc_GeneratorExit is passed */ typ, val, tb ); retval = Nuitka_Asyncgen_unwrap_value( asyncgen, retval ); } if ( retval == NULL ) { goto check_error; } return retval; } assert( asyncgen_athrow->m_state == AWAITABLE_STATE_ITER ); retval = _Nuitka_Asyncgen_send( asyncgen, arg, false ); if ( asyncgen_athrow->m_args ) { return Nuitka_Asyncgen_unwrap_value( asyncgen, retval ); } else { /* We are here to close if no args. */ if ( retval ) { if ( _PyAsyncGenWrappedValue_CheckExact(retval) || Nuitka_AsyncgenWrappedValue_CheckExact(retval) ) { Py_DECREF( retval ); PyErr_Format( PyExc_RuntimeError, "async generator ignored GeneratorExit" ); return NULL; } return retval; } } check_error: if ( PyErr_ExceptionMatches( PyExc_StopAsyncIteration ) ) { asyncgen_athrow->m_state = AWAITABLE_STATE_CLOSED; if ( asyncgen_athrow->m_args == NULL ) { CLEAR_ERROR_OCCURRED(); PyErr_SetNone( PyExc_StopIteration ); } } else if ( PyErr_ExceptionMatches( PyExc_GeneratorExit ) ) { asyncgen_athrow->m_state = AWAITABLE_STATE_CLOSED; CLEAR_ERROR_OCCURRED(); PyErr_SetNone( PyExc_StopIteration ); } return NULL; } static PyObject *Nuitka_AsyncgenAthrow_throw( struct Nuitka_AsyncgenAthrowObject *asyncgen_athrow, PyObject *args ) { PyObject *retval; if ( asyncgen_athrow->m_state == AWAITABLE_STATE_INIT ) { PyErr_Format( PyExc_RuntimeError, "can't send non-None value to a just-started coroutine" ); return NULL; } if ( asyncgen_athrow->m_state == AWAITABLE_STATE_CLOSED ) { PyErr_SetNone( PyExc_StopIteration ); return NULL; } retval = Nuitka_Asyncgen_throw( asyncgen_athrow->m_gen, args ); if ( asyncgen_athrow->m_args ) { return Nuitka_Asyncgen_unwrap_value( asyncgen_athrow->m_gen, retval ); } else { if ( retval ) { if ( _PyAsyncGenWrappedValue_CheckExact(retval) || Nuitka_AsyncgenWrappedValue_CheckExact(retval) ) { Py_DECREF( retval ); PyErr_Format( PyExc_RuntimeError, "async generator ignored GeneratorExit" ); return NULL; } } return retval; } } static PyObject *Nuitka_AsyncgenAthrow_iternext( struct Nuitka_AsyncgenAthrowObject *asyncgen_athrow ) { return Nuitka_AsyncgenAthrow_send( asyncgen_athrow, Py_None ); } static PyObject *Nuitka_AsyncgenAthrow_close( struct Nuitka_AsyncgenAthrowObject *asyncgen_athrow, PyObject *args ) { asyncgen_athrow->m_state = AWAITABLE_STATE_CLOSED; Py_INCREF( Py_None ); return Py_None; } static PyMethodDef Nuitka_AsyncgenAthrow_methods[] = { { "send", (PyCFunction)Nuitka_AsyncgenAthrow_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_AsyncgenAthrow_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_AsyncgenAthrow_close, METH_NOARGS, NULL }, { NULL } }; static PyAsyncMethods Nuitka_AsyncgenAthrow_as_async = { PyObject_SelfIter, /* am_await */ 0, /* am_aiter */ 0 /* am_anext */ }; static PyTypeObject Nuitka_AsyncgenAthrow_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_async_generator_athrow", /* tp_name */ sizeof(struct Nuitka_AsyncgenAthrowObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_AsyncgenAthrow_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ &Nuitka_AsyncgenAthrow_as_async, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_AsyncgenAthrow_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_AsyncgenAthrow_iternext, /* tp_iternext */ Nuitka_AsyncgenAthrow_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ }; static PyObject *Nuitka_AsyncgenAthrow_New( struct Nuitka_AsyncgenObject *asyncgen, PyObject *args ) { struct Nuitka_AsyncgenAthrowObject *result; allocateFromFreeListFixed( free_list_asyncgen_athrows, struct Nuitka_AsyncgenAthrowObject, Nuitka_AsyncgenAthrow_Type ) Py_INCREF( asyncgen ); result->m_gen = asyncgen; Py_XINCREF( args ); result->m_args = args; result->m_state = AWAITABLE_STATE_INIT; Nuitka_GC_Track( result ); return (PyObject*)result; } static void RAISE_ASYNCGEN_EXCEPTION( struct Nuitka_AsyncgenObject *asyncgen ) { CHECK_OBJECT( asyncgen->m_exception_type ); RESTORE_ERROR_OCCURRED( asyncgen->m_exception_type, asyncgen->m_exception_value, asyncgen->m_exception_tb ); asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; assert( ERROR_OCCURRED() ); } extern PyObject *ERROR_GET_STOP_ITERATION_VALUE(); extern PyObject *PyGen_Send( PyGenObject *gen, PyObject *arg ); extern PyObject *const_str_plain_send, *const_str_plain_throw, *const_str_plain_close; static PyObject *yieldFromAsyncgen( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { CHECK_OBJECT( value ); // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( asyncgen->m_exception_type ) { // The yielding generator is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( asyncgen->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL && !EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } } RAISE_ASYNCGEN_EXCEPTION( asyncgen ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, asyncgen->m_exception_type, asyncgen->m_exception_value, asyncgen->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_ASYNCGEN_EXCEPTION( asyncgen ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( asyncgen->m_exception_type ); Py_CLEAR( asyncgen->m_exception_value ); Py_CLEAR( asyncgen->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) || PyCoro_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); } // Check the sub-generator result if ( retval == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); // No exception we take it as stop iteration. if ( error == NULL ) { Py_INCREF( Py_None ); return Py_None; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { asyncgen->m_yielded = retval; asyncgen->m_yieldfrom = value; // Return to the calling context. swapFiber( &asyncgen->m_yielder_context, &asyncgen->m_caller_context ); asyncgen->m_yieldfrom = NULL; send_value = asyncgen->m_yielded; if ( ERROR_OCCURRED() ) { assert( asyncgen->m_exception_type != NULL ); CLEAR_ERROR_OCCURRED(); } } } } PyObject *ASYNCGEN_AWAIT( struct Nuitka_AsyncgenObject *asyncgen, PyObject *awaitable ) { #if _DEBUG_ASYNCGEN PRINT_STRING("AWAIT entry:"); PRINT_ITEM( awaitable ); PRINT_NEW_LINE(); #endif PyObject *awaitable_iter = PyCoro_GetAwaitableIter( awaitable ); if (unlikely( awaitable_iter == NULL )) { return NULL; } if ( Nuitka_Coroutine_Check( awaitable ) ) { struct Nuitka_CoroutineObject *awaited_coroutine = (struct Nuitka_CoroutineObject *)awaitable; if ( awaited_coroutine->m_awaiting ) { Py_DECREF( awaitable_iter ); PyErr_Format( PyExc_RuntimeError, "coroutine is being awaited already" ); return NULL; } } asyncgen->m_awaiting = true; PyObject *retval = yieldFromAsyncgen( asyncgen, awaitable_iter ); asyncgen->m_awaiting = false; Py_DECREF( awaitable_iter ); #if _DEBUG_ASYNCGEN PRINT_STRING("AWAIT exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } PyObject *ASYNCGEN_AWAIT_IN_HANDLER( struct Nuitka_AsyncgenObject *asyncgen, PyObject *awaitable ) { #if _DEBUG_ASYNCGEN PRINT_STRING("AWAIT entry:"); PRINT_ITEM( awaitable ); PRINT_NEW_LINE(); #endif PyObject *awaitable_iter = PyCoro_GetAwaitableIter( awaitable ); if (unlikely( awaitable_iter == NULL )) { return NULL; } if ( Nuitka_Coroutine_Check( awaitable ) ) { struct Nuitka_CoroutineObject *awaited_coroutine = (struct Nuitka_CoroutineObject *)awaitable; if ( awaited_coroutine->m_awaiting ) { Py_DECREF( awaitable_iter ); PyErr_Format( PyExc_RuntimeError, "coroutine is being awaited already" ); return NULL; } } /* When yielding from an exception handler in Python3, the exception * preserved to the frame is restore, while the current one is put there. */ PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; if ( saved_exception_type ) CHECK_OBJECT( saved_exception_type ); if ( saved_exception_value ) CHECK_OBJECT( saved_exception_value ); if ( saved_exception_traceback ) CHECK_OBJECT( saved_exception_traceback ); thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD exit:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; if ( saved_exception_type ) CHECK_OBJECT( saved_exception_type ); if ( saved_exception_value ) CHECK_OBJECT( saved_exception_value ); if ( saved_exception_traceback ) CHECK_OBJECT( saved_exception_traceback ); asyncgen->m_awaiting = true; PyObject *retval = yieldFromAsyncgen( asyncgen, awaitable_iter ); asyncgen->m_awaiting = false; Py_DECREF( awaitable_iter ); // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; if ( saved_exception_type ) CHECK_OBJECT( saved_exception_type ); if ( saved_exception_value ) CHECK_OBJECT( saved_exception_value ); if ( saved_exception_traceback ) CHECK_OBJECT( saved_exception_traceback ); #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD return:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; if ( thread_state->exc_type ) CHECK_OBJECT( thread_state->exc_type ); if ( thread_state->exc_value ) CHECK_OBJECT( thread_state->exc_value ); if ( thread_state->exc_traceback ) CHECK_OBJECT( thread_state->exc_traceback ); thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; #if _DEBUG_ASYNCGEN PRINT_STRING("AWAIT exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } extern PyObject *Nuitka_AIterWrapper_New( PyObject *aiter ); PyObject *ASYNCGEN_ASYNC_MAKE_ITERATOR( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { #if _DEBUG_COROUTINE PRINT_STRING("AITER entry:"); PRINT_ITEM( value ); PRINT_NEW_LINE(); #endif unaryfunc getter = NULL; if ( Py_TYPE( value )->tp_as_async ) { getter = Py_TYPE( value )->tp_as_async->am_aiter; } if (unlikely( getter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' requires an object with __aiter__ method, got %s", Py_TYPE( value )->tp_name ); return NULL; } PyObject *iter = (*getter)( value ); if (unlikely( iter == NULL )) { return NULL; } /* Starting with Python 3.5.2 it is acceptable to return an async iterator * directly, instead of an awaitable. */ if ( Py_TYPE( iter )->tp_as_async != NULL && Py_TYPE( iter )->tp_as_async->am_anext != NULL) { PyObject *wrapper = Nuitka_AIterWrapper_New( iter ); Py_DECREF( iter ); iter = wrapper; } PyObject *awaitable_iter = PyCoro_GetAwaitableIter( iter ); if (unlikely( awaitable_iter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' received an invalid object from __aiter__: %s", Py_TYPE( iter )->tp_name ); Py_DECREF( iter ); return NULL; } Py_DECREF( iter ); PyObject *retval = yieldFromAsyncgen( asyncgen, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("AITER exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } PyObject *ASYNCGEN_ASYNC_ITERATOR_NEXT( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { #if _DEBUG_COROUTINE PRINT_STRING("ANEXT entry:"); PRINT_ITEM( value ); PRINT_NEW_LINE(); #endif unaryfunc getter = NULL; if ( Py_TYPE( value )->tp_as_async ) { getter = Py_TYPE( value )->tp_as_async->am_anext; } if (unlikely( getter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' requires an iterator with __anext__ method, got %s", Py_TYPE( value )->tp_name ); return NULL; } PyObject *next_value = (*getter)( value ); if (unlikely( next_value == NULL )) { return NULL; } PyObject *awaitable_iter = PyCoro_GetAwaitableIter( next_value ); if (unlikely( awaitable_iter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' received an invalid object from __anext__: %s", Py_TYPE( next_value )->tp_name ); Py_DECREF( next_value ); return NULL; } Py_DECREF( next_value ); PyObject *retval = yieldFromAsyncgen( asyncgen, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("ANEXT exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } void _initCompiledAsyncgenTypes( void ) { PyType_Ready( &Nuitka_Asyncgen_Type ); PyType_Ready( &Nuitka_AsyncgenAsend_Type ); PyType_Ready( &Nuitka_AsyncgenAthrow_Type ); PyType_Ready( &Nuitka_AsyncgenValueWrapper_Type ); } Nuitka-0.5.28.2/nuitka/build/static_src/x64_ucontext_src/0000755000372000001440000000000013207540420023431 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/static_src/x64_ucontext_src/fibers_x64.c0000644000372000001440000000422313112214770025552 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for x64. #include "nuitka/prelude.h" #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { to->f_context.uc_stack.ss_sp = NULL; to->f_context.uc_link = NULL; to->start_stack = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { #ifdef _NUITKA_MAKECONTEXT_INTS int ar[2]; memcpy( &ar[0], &arg, sizeof(arg) ); #endif int res = getcontext( &to->f_context ); if (unlikely( res != 0 )) { return 1; } to->f_context.uc_stack.ss_size = STACK_SIZE; to->f_context.uc_stack.ss_sp = last_stack ? last_stack : malloc( STACK_SIZE ); to->start_stack = to->f_context.uc_stack.ss_sp; to->f_context.uc_link = NULL; last_stack = NULL; #ifdef _NUITKA_MAKECONTEXT_INTS makecontext( &to->f_context, (void (*)())code, 2, ar[0], ar[1] ); #else makecontext( &to->f_context, (void (*)())code, 1, (unsigned long)arg ); #endif return 0; } void _releaseFiber( Fiber *to ) { if ( to->start_stack != NULL ) { if ( last_stack == NULL ) { last_stack = to->start_stack; } else { free( to->start_stack ); } to->start_stack = NULL; } } Nuitka-0.5.28.2/nuitka/build/static_src/x64_ucontext_src/swapfiber.S0000644000372000001440000000452313112214770025544 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch "swapFiber" for x64 #define oRBP 120 #define oRSP 160 #define oRBX 128 #define oR8 40 #define oR9 48 #define oR12 72 #define oR13 80 #define oR14 88 #define oR15 96 #define oRDI 104 #define oRSI 112 #define oRDX 136 #define oRCX 152 #define oRIP 168 #define oFPREGS 224 #define oFPREGSMEM 424 .globl _swapFiber _swapFiber: // Save the preserved used for passing args and the return address. movq %rbx, oRBX(%rdi) movq %rbp, oRBP(%rdi) movq %r12, oR12(%rdi) movq %r13, oR13(%rdi) movq %r14, oR14(%rdi) movq %r15, oR15(%rdi) movq %rdi, oRDI(%rdi) movq %rsi, oRSI(%rdi) movq %rdx, oRDX(%rdi) movq %rcx, oRCX(%rdi) movq %r8, oR8(%rdi) movq %r9, oR9(%rdi) movq (%rsp), %rcx movq %rcx, oRIP(%rdi) // Make room for return address. leaq 8(%rsp), %rcx movq %rcx, oRSP(%rdi) // Separate floating-point register content memory leaq oFPREGSMEM(%rdi), %rcx movq %rcx, oFPREGS(%rdi) // Load the new stack pointer and the preserved registers. movq oRSP(%rsi), %rsp movq oRBX(%rsi), %rbx movq oRBP(%rsi), %rbp movq oR12(%rsi), %r12 movq oR13(%rsi), %r13 movq oR14(%rsi), %r14 movq oR15(%rsi), %r15 // We should return to the address provided, put it on top of stack. movq oRIP(%rsi), %rcx pushq %rcx // Setup registers for arguments. movq oRDI(%rsi), %rdi movq oRDX(%rsi), %rdx movq oRCX(%rsi), %rcx movq oR8(%rsi), %r8 movq oR9(%rsi), %r9 // Setup %rsi. movq oRSI(%rsi), %rsi ret Nuitka-0.5.28.2/nuitka/build/static_src/libcoro_ucontext_src/0000755000372000001440000000000013207540420024441 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/static_src/libcoro_ucontext_src/coro.c0000644000372000037200000004523312702504125025525 0ustar hayenhayen00000000000000/* * Copyright (c) 2001-2011 Marc Alexander Lehmann * * Redistribution and use in source and binary forms, with or without modifica- * tion, are permitted provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER- * CHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO * EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPE- * CIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTH- * ERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE. * * Alternatively, the contents of this file may be used under the terms of * the GNU General Public License ("GPL") version 2 or any later version, * in which case the provisions of the GPL are applicable instead of * the above. If you wish to allow the use of your version of this file * only under the terms of the GPL and not to allow others to use your * version of this file under the BSD license, indicate your decision * by deleting the provisions above and replace them with the notice * and other provisions required by the GPL. If you do not delete the * provisions above, a recipient may use your version of this file under * either the BSD or the GPL. * * This library is modelled strictly after Ralf S. Engelschalls article at * http://www.gnu.org/software/pth/rse-pmt.ps. So most of the credit must * go to Ralf S. Engelschall . */ #include "coro.h" #include #include /*****************************************************************************/ /* ucontext/setjmp/asm backends */ /*****************************************************************************/ #if CORO_UCONTEXT || CORO_SJLJ || CORO_LOSER || CORO_LINUX || CORO_IRIX || CORO_ASM # if CORO_UCONTEXT # include # endif # if !defined(STACK_ADJUST_PTR) # if __sgi /* IRIX is decidedly NON-unix */ # define STACK_ADJUST_PTR(sp,ss) ((char *)(sp) + (ss) - 8) # define STACK_ADJUST_SIZE(sp,ss) ((ss) - 8) # elif (__i386__ && CORO_LINUX) || (_M_IX86 && CORO_LOSER) # define STACK_ADJUST_PTR(sp,ss) ((char *)(sp) + (ss)) # define STACK_ADJUST_SIZE(sp,ss) (ss) # elif (__amd64__ && CORO_LINUX) || ((_M_AMD64 || _M_IA64) && CORO_LOSER) # define STACK_ADJUST_PTR(sp,ss) ((char *)(sp) + (ss) - 8) # define STACK_ADJUST_SIZE(sp,ss) (ss) # else # define STACK_ADJUST_PTR(sp,ss) (sp) # define STACK_ADJUST_SIZE(sp,ss) (ss) # endif # endif # include # if CORO_SJLJ # include # include # include # endif static coro_func coro_init_func; static void *coro_init_arg; static coro_context *new_coro, *create_coro; static void coro_init (void) { volatile coro_func func = coro_init_func; volatile void *arg = coro_init_arg; coro_transfer (new_coro, create_coro); #if __GCC_HAVE_DWARF2_CFI_ASM && __amd64 asm (".cfi_undefined rip"); #endif func ((void *)arg); /* the new coro returned. bad. just abort() for now */ abort (); } # if CORO_SJLJ static volatile int trampoline_done; /* trampoline signal handler */ static void trampoline (int sig) { if (coro_setjmp (new_coro->env)) coro_init (); /* start it */ else trampoline_done = 1; } # endif # if CORO_ASM #if _WIN32 || __CYGWIN__ #define CORO_WIN_TIB 1 #endif asm ( "\t.text\n" #if _WIN32 || __CYGWIN__ "\t.globl _coro_transfer\n" "_coro_transfer:\n" #else "\t.globl coro_transfer\n" "coro_transfer:\n" #endif /* windows, of course, gives a shit on the amd64 ABI and uses different registers */ /* http://blogs.msdn.com/freik/archive/2005/03/17/398200.aspx */ #if __amd64 #if _WIN32 || __CYGWIN__ #define NUM_SAVED 29 "\tsubq $168, %rsp\t" /* one dummy qword to improve alignment */ "\tmovaps %xmm6, (%rsp)\n" "\tmovaps %xmm7, 16(%rsp)\n" "\tmovaps %xmm8, 32(%rsp)\n" "\tmovaps %xmm9, 48(%rsp)\n" "\tmovaps %xmm10, 64(%rsp)\n" "\tmovaps %xmm11, 80(%rsp)\n" "\tmovaps %xmm12, 96(%rsp)\n" "\tmovaps %xmm13, 112(%rsp)\n" "\tmovaps %xmm14, 128(%rsp)\n" "\tmovaps %xmm15, 144(%rsp)\n" "\tpushq %rsi\n" "\tpushq %rdi\n" "\tpushq %rbp\n" "\tpushq %rbx\n" "\tpushq %r12\n" "\tpushq %r13\n" "\tpushq %r14\n" "\tpushq %r15\n" #if CORO_WIN_TIB "\tpushq %fs:0x0\n" "\tpushq %fs:0x8\n" "\tpushq %fs:0xc\n" #endif "\tmovq %rsp, (%rcx)\n" "\tmovq (%rdx), %rsp\n" #if CORO_WIN_TIB "\tpopq %fs:0xc\n" "\tpopq %fs:0x8\n" "\tpopq %fs:0x0\n" #endif "\tpopq %r15\n" "\tpopq %r14\n" "\tpopq %r13\n" "\tpopq %r12\n" "\tpopq %rbx\n" "\tpopq %rbp\n" "\tpopq %rdi\n" "\tpopq %rsi\n" "\tmovaps (%rsp), %xmm6\n" "\tmovaps 16(%rsp), %xmm7\n" "\tmovaps 32(%rsp), %xmm8\n" "\tmovaps 48(%rsp), %xmm9\n" "\tmovaps 64(%rsp), %xmm10\n" "\tmovaps 80(%rsp), %xmm11\n" "\tmovaps 96(%rsp), %xmm12\n" "\tmovaps 112(%rsp), %xmm13\n" "\tmovaps 128(%rsp), %xmm14\n" "\tmovaps 144(%rsp), %xmm15\n" "\taddq $168, %rsp\n" #else #define NUM_SAVED 6 "\tpushq %rbp\n" "\tpushq %rbx\n" "\tpushq %r12\n" "\tpushq %r13\n" "\tpushq %r14\n" "\tpushq %r15\n" "\tmovq %rsp, (%rdi)\n" "\tmovq (%rsi), %rsp\n" "\tpopq %r15\n" "\tpopq %r14\n" "\tpopq %r13\n" "\tpopq %r12\n" "\tpopq %rbx\n" "\tpopq %rbp\n" #endif "\tpopq %rcx\n" "\tjmpq *%rcx\n" #elif __i386 #define NUM_SAVED 4 "\tpushl %ebp\n" "\tpushl %ebx\n" "\tpushl %esi\n" "\tpushl %edi\n" #if CORO_WIN_TIB #undef NUM_SAVED #define NUM_SAVED 7 "\tpushl %fs:0\n" "\tpushl %fs:4\n" "\tpushl %fs:8\n" #endif "\tmovl %esp, (%eax)\n" "\tmovl (%edx), %esp\n" #if CORO_WIN_TIB "\tpopl %fs:8\n" "\tpopl %fs:4\n" "\tpopl %fs:0\n" #endif "\tpopl %edi\n" "\tpopl %esi\n" "\tpopl %ebx\n" "\tpopl %ebp\n" "\tpopl %ecx\n" "\tjmpl *%ecx\n" #else #error unsupported architecture #endif ); # endif void coro_create (coro_context *ctx, coro_func coro, void *arg, void *sptr, size_t ssize) { coro_context nctx; # if CORO_SJLJ stack_t ostk, nstk; struct sigaction osa, nsa; sigset_t nsig, osig; # endif if (!coro) return; coro_init_func = coro; coro_init_arg = arg; new_coro = ctx; create_coro = &nctx; # if CORO_SJLJ /* we use SIGUSR2. first block it, then fiddle with it. */ sigemptyset (&nsig); sigaddset (&nsig, SIGUSR2); sigprocmask (SIG_BLOCK, &nsig, &osig); nsa.sa_handler = trampoline; sigemptyset (&nsa.sa_mask); nsa.sa_flags = SA_ONSTACK; if (sigaction (SIGUSR2, &nsa, &osa)) { perror ("sigaction"); abort (); } /* set the new stack */ nstk.ss_sp = STACK_ADJUST_PTR (sptr, ssize); /* yes, some platforms (IRIX) get this wrong. */ nstk.ss_size = STACK_ADJUST_SIZE (sptr, ssize); nstk.ss_flags = 0; if (sigaltstack (&nstk, &ostk) < 0) { perror ("sigaltstack"); abort (); } trampoline_done = 0; kill (getpid (), SIGUSR2); sigfillset (&nsig); sigdelset (&nsig, SIGUSR2); while (!trampoline_done) sigsuspend (&nsig); sigaltstack (0, &nstk); nstk.ss_flags = SS_DISABLE; if (sigaltstack (&nstk, 0) < 0) perror ("sigaltstack"); sigaltstack (0, &nstk); if (~nstk.ss_flags & SS_DISABLE) abort (); if (~ostk.ss_flags & SS_DISABLE) sigaltstack (&ostk, 0); sigaction (SIGUSR2, &osa, 0); sigprocmask (SIG_SETMASK, &osig, 0); # elif CORO_LOSER coro_setjmp (ctx->env); #if __CYGWIN__ && __i386 ctx->env[8] = (long) coro_init; ctx->env[7] = (long) ((char *)sptr + ssize) - sizeof (long); #elif __CYGWIN__ && __x86_64 ctx->env[7] = (long) coro_init; ctx->env[6] = (long) ((char *)sptr + ssize) - sizeof (long); #elif defined __MINGW32__ ctx->env[5] = (long) coro_init; ctx->env[4] = (long) ((char *)sptr + ssize) - sizeof (long); #elif defined _M_IX86 ((_JUMP_BUFFER *)&ctx->env)->Eip = (long) coro_init; ((_JUMP_BUFFER *)&ctx->env)->Esp = (long) STACK_ADJUST_PTR (sptr, ssize) - sizeof (long); #elif defined _M_AMD64 ((_JUMP_BUFFER *)&ctx->env)->Rip = (__int64) coro_init; ((_JUMP_BUFFER *)&ctx->env)->Rsp = (__int64) STACK_ADJUST_PTR (sptr, ssize) - sizeof (__int64); #elif defined _M_IA64 ((_JUMP_BUFFER *)&ctx->env)->StIIP = (__int64) coro_init; ((_JUMP_BUFFER *)&ctx->env)->IntSp = (__int64) STACK_ADJUST_PTR (sptr, ssize) - sizeof (__int64); #else #error "microsoft libc or architecture not supported" #endif # elif CORO_LINUX coro_setjmp (ctx->env); #if __GLIBC__ >= 2 && __GLIBC_MINOR__ >= 0 && defined (JB_PC) && defined (JB_SP) ctx->env[0].__jmpbuf[JB_PC] = (long) coro_init; ctx->env[0].__jmpbuf[JB_SP] = (long) STACK_ADJUST_PTR (sptr, ssize) - sizeof (long); #elif __GLIBC__ >= 2 && __GLIBC_MINOR__ >= 0 && defined (__mc68000__) ctx->env[0].__jmpbuf[0].__aregs[0] = (long int)coro_init; ctx->env[0].__jmpbuf[0].__sp = (int *) ((char *)sptr + ssize) - sizeof (long); #elif defined (__GNU_LIBRARY__) && defined (__i386__) ctx->env[0].__jmpbuf[0].__pc = (char *) coro_init; ctx->env[0].__jmpbuf[0].__sp = (void *) ((char *)sptr + ssize) - sizeof (long); #elif defined (__GNU_LIBRARY__) && defined (__amd64__) ctx->env[0].__jmpbuf[JB_PC] = (long) coro_init; ctx->env[0].__jmpbuf[0].__sp = (void *) ((char *)sptr + ssize) - sizeof (long); #else #error "linux libc or architecture not supported" #endif # elif CORO_IRIX coro_setjmp (ctx->env, 0); ctx->env[JB_PC] = (__uint64_t)coro_init; ctx->env[JB_SP] = (__uint64_t)STACK_ADJUST_PTR (sptr, ssize) - sizeof (long); # elif CORO_ASM ctx->sp = (void **)(ssize + (char *)sptr); *--ctx->sp = (void *)abort; /* needed for alignment only */ *--ctx->sp = (void *)coro_init; #if CORO_WIN_TIB *--ctx->sp = 0; /* ExceptionList */ *--ctx->sp = (char *)sptr + ssize; /* StackBase */ *--ctx->sp = sptr; /* StackLimit */ #endif ctx->sp -= NUM_SAVED; memset (ctx->sp, 0, sizeof (*ctx->sp) * NUM_SAVED); # elif CORO_UCONTEXT getcontext (&(ctx->uc)); ctx->uc.uc_link = 0; ctx->uc.uc_stack.ss_sp = sptr; ctx->uc.uc_stack.ss_size = (size_t)ssize; ctx->uc.uc_stack.ss_flags = 0; makecontext (&(ctx->uc), (void (*)())coro_init, 0); # endif coro_transfer (create_coro, new_coro); } /*****************************************************************************/ /* pthread backend */ /*****************************************************************************/ #elif CORO_PTHREAD /* this mutex will be locked by the running coroutine */ pthread_mutex_t coro_mutex = PTHREAD_MUTEX_INITIALIZER; struct coro_init_args { coro_func func; void *arg; coro_context *self, *main; }; static pthread_t null_tid; /* I'd so love to cast pthread_mutex_unlock to void (*)(void *)... */ static void mutex_unlock_wrapper (void *arg) { pthread_mutex_unlock ((pthread_mutex_t *)arg); } static void * coro_init (void *args_) { struct coro_init_args *args = (struct coro_init_args *)args_; coro_func func = args->func; void *arg = args->arg; pthread_mutex_lock (&coro_mutex); /* we try to be good citizens and use deferred cancellation and cleanup handlers */ pthread_cleanup_push (mutex_unlock_wrapper, &coro_mutex); coro_transfer (args->self, args->main); func (arg); pthread_cleanup_pop (1); return 0; } void coro_transfer (coro_context *prev, coro_context *next) { pthread_cond_signal (&next->cv); pthread_cond_wait (&prev->cv, &coro_mutex); #if __FreeBSD__ /* freebsd is of course broken and needs manual testcancel calls... yay... */ pthread_testcancel (); #endif } void coro_create (coro_context *ctx, coro_func coro, void *arg, void *sptr, size_t ssize) { static coro_context nctx; static int once; if (!once) { once = 1; pthread_mutex_lock (&coro_mutex); pthread_cond_init (&nctx.cv, 0); null_tid = pthread_self (); } pthread_cond_init (&ctx->cv, 0); if (coro) { pthread_attr_t attr; struct coro_init_args args; args.func = coro; args.arg = arg; args.self = ctx; args.main = &nctx; pthread_attr_init (&attr); #if __UCLIBC__ /* exists, but is borked */ /*pthread_attr_setstacksize (&attr, (size_t)ssize);*/ #elif __CYGWIN__ /* POSIX, not here */ pthread_attr_setstacksize (&attr, (size_t)ssize); #else pthread_attr_setstack (&attr, sptr, (size_t)ssize); #endif pthread_attr_setscope (&attr, PTHREAD_SCOPE_PROCESS); pthread_create (&ctx->id, &attr, coro_init, &args); coro_transfer (args.main, args.self); } else ctx->id = null_tid; } void coro_destroy (coro_context *ctx) { if (!pthread_equal (ctx->id, null_tid)) { pthread_cancel (ctx->id); pthread_mutex_unlock (&coro_mutex); pthread_join (ctx->id, 0); pthread_mutex_lock (&coro_mutex); } pthread_cond_destroy (&ctx->cv); } /*****************************************************************************/ /* fiber backend */ /*****************************************************************************/ #elif CORO_FIBER #define WIN32_LEAN_AND_MEAN #if _WIN32_WINNT < 0x0400 #undef _WIN32_WINNT #define _WIN32_WINNT 0x0400 #endif #include VOID CALLBACK coro_init (PVOID arg) { coro_context *ctx = (coro_context *)arg; ctx->coro (ctx->arg); } void coro_transfer (coro_context *prev, coro_context *next) { if (!prev->fiber) { prev->fiber = GetCurrentFiber (); if (prev->fiber == 0 || prev->fiber == (void *)0x1e00) prev->fiber = ConvertThreadToFiber (0); } SwitchToFiber (next->fiber); } void coro_create (coro_context *ctx, coro_func coro, void *arg, void *sptr, size_t ssize) { ctx->fiber = 0; ctx->coro = coro; ctx->arg = arg; if (!coro) return; ctx->fiber = CreateFiber (ssize, coro_init, ctx); } void coro_destroy (coro_context *ctx) { DeleteFiber (ctx->fiber); } #else #error unsupported backend #endif /*****************************************************************************/ /* stack management */ /*****************************************************************************/ #if CORO_STACKALLOC #include #ifndef _WIN32 # include #endif #if CORO_USE_VALGRIND # include #endif #if _POSIX_MAPPED_FILES # include # define CORO_MMAP 1 # ifndef MAP_ANONYMOUS # ifdef MAP_ANON # define MAP_ANONYMOUS MAP_ANON # else # undef CORO_MMAP # endif # endif # include #else # undef CORO_MMAP #endif #if _POSIX_MEMORY_PROTECTION # ifndef CORO_GUARDPAGES # define CORO_GUARDPAGES 4 # endif #else # undef CORO_GUARDPAGES #endif #if !CORO_MMAP # undef CORO_GUARDPAGES #endif #if !__i386 && !__x86_64 && !__powerpc && !__m68k && !__alpha && !__mips && !__sparc64 # undef CORO_GUARDPAGES #endif #ifndef CORO_GUARDPAGES # define CORO_GUARDPAGES 0 #endif #if !PAGESIZE #if !CORO_MMAP #define PAGESIZE 4096 #else static size_t coro_pagesize (void) { static size_t pagesize; if (!pagesize) pagesize = sysconf (_SC_PAGESIZE); return pagesize; } #define PAGESIZE coro_pagesize () #endif #endif int coro_stack_alloc (struct coro_stack *stack, unsigned int size) { if (!size) size = 256 * 1024; stack->sptr = 0; stack->ssze = ((size_t)size * sizeof (void *) + PAGESIZE - 1) / PAGESIZE * PAGESIZE; #if CORO_FIBER stack->sptr = (void *)stack; return 1; #else size_t ssze = stack->ssze + CORO_GUARDPAGES * PAGESIZE; void *base; #if CORO_MMAP /* mmap supposedly does allocate-on-write for us */ base = mmap (0, ssze, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (base == (void *)-1) { /* some systems don't let us have executable heap */ /* we assume they won't need executable stack in that case */ base = mmap (0, ssze, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (base == (void *)-1) return 0; } #if CORO_GUARDPAGES mprotect (base, CORO_GUARDPAGES * PAGESIZE, PROT_NONE); #endif base = (void*)((char *)base + CORO_GUARDPAGES * PAGESIZE); #else base = malloc (ssze); if (!base) return 0; #endif #if CORO_USE_VALGRIND stack->valgrind_id = VALGRIND_STACK_REGISTER ((char *)base, ((char *)base) + ssze - CORO_GUARDPAGES * PAGESIZE); #endif stack->sptr = base; return 1; #endif } void coro_stack_free (struct coro_stack *stack) { #if CORO_FIBER /* nop */ #else #if CORO_USE_VALGRIND VALGRIND_STACK_DEREGISTER (stack->valgrind_id); #endif #if CORO_MMAP if (stack->sptr) munmap ((void*)((char *)stack->sptr - CORO_GUARDPAGES * PAGESIZE), stack->ssze + CORO_GUARDPAGES * PAGESIZE); #else free (stack->sptr); #endif #endif } #endif Nuitka-0.5.28.2/nuitka/build/static_src/libcoro_ucontext_src/fibers_coro.c0000644000372000001440000000367413112214770027114 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // Implementation of process context switch for generic targets. #include "nuitka/prelude.h" #ifdef __cplusplus extern "C" { #endif #include "coro.h" #ifdef __cplusplus } #endif // TODO: Make stack size rational. #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { /* Not much to do. */ to->sptr = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { /* Need to allocate stack manually. */ to->sptr = last_stack ? (char *)last_stack : (char *)malloc( STACK_SIZE ); coro_create( &to->coro_ctx, (coro_func)code, (void *)arg, to->sptr, STACK_SIZE ); return 0; } void _releaseFiber( Fiber *to ) { if ( to->sptr != NULL ) { if ( last_stack == NULL && false ) { last_stack = to->sptr; } else { free( to->sptr ); } to->sptr = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { assert( to->sptr ); assert( from->sptr ); coro_transfer( &to->coro_ctx, &from->coro_ctx ); } Nuitka-0.5.28.2/nuitka/build/static_src/CompiledMethodType.c0000644000372000001440000005030413134660221024116 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/freelists.h" #include "structmember.h" static PyObject *Nuitka_Method_get__doc__( struct Nuitka_MethodObject *method, void *closure ) { PyObject *result = method->m_function->m_doc; Py_INCREF( result ); return result; } static PyGetSetDef Nuitka_Method_getsets[] = { { (char *)"__doc__", (getter)Nuitka_Method_get__doc__, NULL, NULL }, { NULL } }; #define OFF( x ) offsetof( struct Nuitka_MethodObject, x ) static PyMemberDef Nuitka_Method_members[] = { { (char *)"im_class", T_OBJECT, OFF( m_class ), READONLY | RESTRICTED, (char *)"the class associated with a method"}, { (char *)"im_func", T_OBJECT, OFF( m_function ), READONLY | RESTRICTED, (char *)"the function (or other callable) implementing a method" }, { (char *)"__func__", T_OBJECT, OFF( m_function ), READONLY | RESTRICTED, (char *)"the function (or other callable) implementing a method" }, { (char *)"im_self", T_OBJECT, OFF( m_object ), READONLY | RESTRICTED, (char *)"the instance to which a method is bound; None for unbound method" }, { (char *)"__self__", T_OBJECT, OFF( m_object ), READONLY | RESTRICTED, (char *)"the instance to which a method is bound; None for unbound method" }, { NULL } }; static PyObject *Nuitka_Method_reduce( struct Nuitka_MethodObject *method ) { PyErr_Format( PyExc_TypeError, "Can't pickle instancemethod objects" ); return NULL; } static PyObject *Nuitka_Method_reduce_ex( struct Nuitka_MethodObject *method, PyObject *args ) { int proto; if ( !PyArg_ParseTuple(args, "|i:__reduce_ex__", &proto) ) { return NULL; } PyErr_Format( PyExc_TypeError, "Can't pickle instancemethod objects" ); return NULL; } static PyObject *Nuitka_Method_deepcopy( struct Nuitka_MethodObject *method, PyObject *memo ) { assert( Nuitka_Method_Check( (PyObject *)method )); static PyObject *module_copy = NULL; static PyObject *deepcopy_function = NULL; if ( module_copy == NULL ) { module_copy = PyImport_ImportModule( "copy" ); CHECK_OBJECT( module_copy ); deepcopy_function = PyObject_GetAttrString( module_copy, "deepcopy" ); CHECK_OBJECT( deepcopy_function ); } PyObject *object = PyObject_CallFunctionObjArgs( deepcopy_function, method->m_object, memo, NULL ); if (unlikely( object == NULL )) { return NULL; } return Nuitka_Method_New( method->m_function, object, method->m_class ); } static PyMethodDef Nuitka_Method_methods[] = { { "__reduce__", (PyCFunction)Nuitka_Method_reduce, METH_NOARGS, NULL }, { "__reduce_ex__", (PyCFunction)Nuitka_Method_reduce_ex, METH_VARARGS, NULL }, { "__deepcopy__", (PyCFunction)Nuitka_Method_deepcopy, METH_O, NULL }, { NULL } }; extern PyObject *const_str_plain___name__; static char const *GET_CLASS_NAME( PyObject *klass ) { if ( klass == NULL ) { return "?"; } else { #if PYTHON_VERSION < 300 if ( PyClass_Check( klass ) ) { return Nuitka_String_AsString( ((PyClassObject *)klass)->cl_name ); } #endif if ( !PyType_Check( klass ) ) { klass = (PyObject *)Py_TYPE( klass ); } return ((PyTypeObject *)klass)->tp_name; } } extern PyObject *const_str_plain___class__; static char const *GET_INSTANCE_CLASS_NAME( PyObject *instance ) { PyObject *klass = PyObject_GetAttr( instance, const_str_plain___class__ ); // Fallback to type as this cannot fail. if ( klass == NULL ) { CLEAR_ERROR_OCCURRED(); klass = (PyObject *)Py_TYPE( instance ); Py_INCREF( klass ); } char const *result = GET_CLASS_NAME( klass ); Py_DECREF( klass ); return result; } static char const *GET_CALLABLE_DESC( PyObject *object ) { if ( Nuitka_Function_Check( object ) || Nuitka_Generator_Check( object ) || PyMethod_Check( object ) || PyFunction_Check( object ) || PyCFunction_Check( object ) ) { return "()"; } #if PYTHON_VERSION < 300 else if ( PyClass_Check( object ) ) { return " constructor"; } else if ( PyInstance_Check( object )) { return " instance"; } #endif else { return " object"; } } static char const *GET_CALLABLE_NAME( PyObject *object ) { if ( Nuitka_Function_Check( object ) ) { return Nuitka_String_AsString( Nuitka_Function_GetName( object ) ); } else if ( Nuitka_Generator_Check( object ) ) { return Nuitka_String_AsString( Nuitka_Generator_GetName( object ) ); } else if ( PyMethod_Check( object ) ) { return PyEval_GetFuncName( PyMethod_GET_FUNCTION( object ) ); } else if ( PyFunction_Check( object ) ) { return Nuitka_String_AsString( ((PyFunctionObject*)object)->func_name ); } #if PYTHON_VERSION < 300 else if ( PyInstance_Check( object ) ) { return Nuitka_String_AsString( ((PyInstanceObject*)object)->in_class->cl_name ); } else if ( PyClass_Check( object ) ) { return Nuitka_String_AsString( ((PyClassObject*)object)->cl_name ); } #endif else if ( PyCFunction_Check( object ) ) { return ((PyCFunctionObject*)object)->m_ml->ml_name; } else { return Py_TYPE( object )->tp_name; } } static PyObject *Nuitka_Method_tp_call( struct Nuitka_MethodObject *method, PyObject *args, PyObject *kw ) { Py_ssize_t arg_count = PyTuple_Size( args ); if ( method->m_object == NULL ) { if (unlikely( arg_count < 1 )) { PyErr_Format( PyExc_TypeError, "unbound compiled_method %s%s must be called with %s instance as first argument (got nothing instead)", GET_CALLABLE_NAME( (PyObject *)method->m_function ), GET_CALLABLE_DESC( (PyObject *)method->m_function ), GET_CLASS_NAME( method->m_class ) ); return NULL; } else { PyObject *self = PyTuple_GET_ITEM( args, 0 ); CHECK_OBJECT( self ); int result = PyObject_IsInstance( self, method->m_class ); if (unlikely( result < 0 )) { return NULL; } else if (unlikely( result == 0 )) { PyErr_Format( PyExc_TypeError, "unbound compiled_method %s%s must be called with %s instance as first argument (got %s instance instead)", GET_CALLABLE_NAME( (PyObject *)method->m_function ), GET_CALLABLE_DESC( (PyObject *)method->m_function ), GET_CLASS_NAME( method->m_class ), GET_INSTANCE_CLASS_NAME( (PyObject *)self ) ); return NULL; } } return Py_TYPE( method->m_function )->tp_call( (PyObject *)method->m_function, args, kw ); } else { return Nuitka_CallMethodFunctionPosArgsKwArgs( method->m_function, method->m_object, &PyTuple_GET_ITEM( args, 0 ), arg_count, kw ); } } static PyObject *Nuitka_Method_tp_descr_get( struct Nuitka_MethodObject *method, PyObject *object, PyObject *klass ) { // Don't rebind already bound methods. if ( method->m_object != NULL ) { Py_INCREF( method ); return (PyObject *)method; } if ( method->m_class != NULL && klass != NULL ) { // Quick subclass test, bound methods remain the same if the class is a sub class int result = PyObject_IsSubclass( klass, method->m_class ); if (unlikely( result < 0 )) { return NULL; } else if ( result == 0 ) { Py_INCREF( method ); return (PyObject *)method; } } return Nuitka_Method_New( method->m_function, object, klass ); } static PyObject *Nuitka_Method_tp_getattro( struct Nuitka_MethodObject *method, PyObject *name ) { PyObject *descr = _PyType_Lookup( &Nuitka_Method_Type, name ); if ( descr != NULL ) { if ( #if PYTHON_VERSION < 300 PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) && #endif ( Py_TYPE( descr )->tp_descr_get != NULL ) ) { return Py_TYPE( descr )->tp_descr_get( descr, (PyObject *)method, (PyObject *)Py_TYPE( method ) ); } else { Py_INCREF( descr ); return descr; } } return PyObject_GetAttr( (PyObject *)method->m_function, name ); } static long Nuitka_Method_tp_traverse( struct Nuitka_MethodObject *method, visitproc visit, void *arg ) { Py_VISIT( method->m_function ); Py_VISIT( method->m_object ); Py_VISIT( method->m_class ); return 0; } // tp_repr slot, decide how a function shall be output static PyObject *Nuitka_Method_tp_repr( struct Nuitka_MethodObject *method ) { if ( method->m_object == NULL ) { #if PYTHON_VERSION < 300 return PyString_FromFormat( "", GET_CLASS_NAME( method->m_class ), Nuitka_String_AsString( method->m_function->m_name ) ); #else return PyUnicode_FromFormat( "", Nuitka_String_AsString( method->m_function->m_name ), method->m_function ); #endif } else { // Note: CPython uses repr ob the object, although a comment despises // it, we do it for compatibility. PyObject *object_repr = PyObject_Repr( method->m_object ); if ( object_repr == NULL ) { return NULL; } #if PYTHON_VERSION < 300 else if ( !PyString_Check( object_repr ) ) { Py_DECREF( object_repr ); return NULL; } #else else if ( !PyUnicode_Check( object_repr ) ) { Py_DECREF( object_repr ); return NULL; } #endif #if PYTHON_VERSION < 300 PyObject *result = PyString_FromFormat( "", GET_CLASS_NAME( method->m_class ), Nuitka_String_AsString( method->m_function->m_name ), Nuitka_String_AsString_Unchecked( object_repr ) ); #elif PYTHON_VERSION < 350 PyObject *result = PyUnicode_FromFormat( "", GET_CLASS_NAME( method->m_class ), Nuitka_String_AsString( method->m_function->m_name ), Nuitka_String_AsString_Unchecked( object_repr ) ); #else PyObject *result = PyUnicode_FromFormat( "", Nuitka_String_AsString( method->m_function->m_qualname ), Nuitka_String_AsString_Unchecked( object_repr ) ); #endif Py_DECREF( object_repr ); return result; } } #if PYTHON_VERSION < 300 static int Nuitka_Method_tp_compare( struct Nuitka_MethodObject *a, struct Nuitka_MethodObject *b ) { if ( a->m_function->m_counter < b->m_function->m_counter ) { return -1; } else if ( a->m_function->m_counter > b->m_function->m_counter ) { return 1; } else if ( a->m_object == b->m_object ) { return 0; } else if ( a->m_object == NULL ) { return -1; } else if ( b->m_object == NULL ) { return 1; } else { return PyObject_Compare( a->m_object, b->m_object ); } } #endif static PyObject *Nuitka_Method_tp_richcompare( struct Nuitka_MethodObject *a, struct Nuitka_MethodObject *b, int op ) { if ( op != Py_EQ && op != Py_NE ) { Py_INCREF( Py_NotImplemented ); return Py_NotImplemented; } if ( Nuitka_Method_Check( (PyObject *)a ) == false || Nuitka_Method_Check( (PyObject *)b ) == false ) { Py_INCREF( Py_NotImplemented ); return Py_NotImplemented; } bool b_res = a->m_function->m_counter == b->m_function->m_counter; // If the underlying function objects are the same, check the objects, which // may be NULL in case of unbound methods, which would be the same again. if ( b_res ) { if ( a->m_object == NULL ) { b_res = b->m_object == NULL; } else if ( b->m_object == NULL ) { b_res = false; } else { int res = PyObject_RichCompareBool( a->m_object, b->m_object, Py_EQ ); b_res = res != 0; } } PyObject *result; if ( op == Py_EQ ) { result = BOOL_FROM( b_res ); } else { result = BOOL_FROM( !b_res ); } Py_INCREF( result ); return result; } static long Nuitka_Method_tp_hash( struct Nuitka_MethodObject *method ) { // Just give the hash of the method function, that ought to be good enough. return method->m_function->m_counter; } #define MAX_METHOD_FREE_LIST_COUNT 100 static struct Nuitka_MethodObject *free_list_methods = NULL; static int free_list_methods_count = 0; static void Nuitka_Method_tp_dealloc( struct Nuitka_MethodObject *method ) { #ifndef __NUITKA_NO_ASSERT__ // Save the current exception, if any, we must to not corrupt it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); #endif Nuitka_GC_UnTrack( method ); if ( method->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)method ); } Py_XDECREF( method->m_object ); Py_XDECREF( method->m_class ); Py_DECREF( (PyObject *)method->m_function ); /* Put the object into freelist or release to GC */ releaseToFreeList( free_list_methods, method, MAX_METHOD_FREE_LIST_COUNT ); #ifndef __NUITKA_NO_ASSERT__ PyThreadState *tstate = PyThreadState_GET(); assert( tstate->curexc_type == save_exception_type ); assert( tstate->curexc_value == save_exception_value ); assert( (PyTracebackObject *)tstate->curexc_traceback == save_exception_tb ); #endif } static PyObject *Nuitka_Method_tp_new( PyTypeObject* type, PyObject* args, PyObject *kw ) { PyObject *func; PyObject *self; PyObject *klass = NULL; if ( !_PyArg_NoKeywords( "instancemethod", kw ) ) { return NULL; } else if ( !PyArg_UnpackTuple( args, "compiled_method", 2, 3, &func, &self, &klass ) ) { return NULL; } else if ( !PyCallable_Check( func ) ) { PyErr_Format( PyExc_TypeError, "first argument must be callable" ); return NULL; } else { if ( self == Py_None ) { self = NULL; } if ( self == NULL && klass == NULL ) { PyErr_Format( PyExc_TypeError, "unbound methods must have non-NULL im_class" ); return NULL; } } assert( Nuitka_Function_Check( func ) ); return Nuitka_Method_New( (struct Nuitka_FunctionObject *)func, self, klass ); } PyTypeObject Nuitka_Method_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_method", sizeof(struct Nuitka_MethodObject), 0, (destructor)Nuitka_Method_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ #if PYTHON_VERSION < 300 (cmpfunc)Nuitka_Method_tp_compare, /* tp_compare */ #else 0, #endif (reprfunc)Nuitka_Method_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ (hashfunc)Nuitka_Method_tp_hash, /* tp_hash */ (ternaryfunc)Nuitka_Method_tp_call, /* tp_call */ 0, /* tp_str */ (getattrofunc)Nuitka_Method_tp_getattro, /* tp_getattro */ PyObject_GenericSetAttr, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | /* tp_flags */ #if PYTHON_VERSION < 300 Py_TPFLAGS_HAVE_WEAKREFS | #endif Py_TPFLAGS_HAVE_GC, 0, /* tp_doc */ (traverseproc)Nuitka_Method_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ (richcmpfunc)Nuitka_Method_tp_richcompare, /* tp_richcompare */ offsetof(struct Nuitka_MethodObject, m_weakrefs), /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Nuitka_Method_methods, /* tp_methods */ Nuitka_Method_members, /* tp_members */ Nuitka_Method_getsets, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ (descrgetfunc)Nuitka_Method_tp_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ Nuitka_Method_tp_new, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0 /* tp_version_tag */ #if PYTHON_VERSION >= 340 ,0 /* tp_finalizer */ #endif }; void _initCompiledMethodType( void ) { PyType_Ready( &Nuitka_Method_Type ); } PyObject *Nuitka_Method_New( struct Nuitka_FunctionObject *function, PyObject *object, PyObject *klass ) { struct Nuitka_MethodObject *result; allocateFromFreeListFixed( free_list_methods, struct Nuitka_MethodObject, Nuitka_Method_Type ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_RuntimeError, "cannot create method %s", Nuitka_String_AsString( function->m_name ) ); return NULL; } Py_INCREF( function ); result->m_function = function; result->m_object = object; Py_XINCREF( object ); result->m_class = klass; Py_XINCREF( klass ); result->m_weakrefs = NULL; Nuitka_GC_Track( result ); return (PyObject *)result; } Nuitka-0.5.28.2/nuitka/build/static_src/CompiledCoroutineType.c0000644000372000001440000014651513207537242024666 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #include "nuitka/prelude.h" #include "nuitka/freelists.h" static PyObject *Nuitka_Coroutine_get_name( struct Nuitka_CoroutineObject *coroutine ) { Py_INCREF( coroutine->m_name ); return coroutine->m_name; } static int Nuitka_Coroutine_set_name( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__name__ must be set to a string object" ); return -1; } PyObject *tmp = coroutine->m_name; Py_INCREF( value ); coroutine->m_name = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Coroutine_get_qualname( struct Nuitka_CoroutineObject *coroutine ) { Py_INCREF( coroutine->m_qualname ); return coroutine->m_qualname; } static int Nuitka_Coroutine_set_qualname( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { // Cannot be deleted, not be non-unicode value. if (unlikely( ( value == NULL ) || !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "__qualname__ must be set to a string object" ); return -1; } PyObject *tmp = coroutine->m_qualname; Py_INCREF( value ); coroutine->m_qualname = value; Py_DECREF( tmp ); return 0; } static PyObject *Nuitka_Coroutine_get_cr_await( struct Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_yieldfrom ) { Py_INCREF( coroutine->m_yieldfrom ); return coroutine->m_yieldfrom; } else { Py_INCREF( Py_None ); return Py_None; } } static PyObject *Nuitka_Coroutine_get_code( struct Nuitka_CoroutineObject *coroutine ) { Py_INCREF( coroutine->m_code_object ); return (PyObject *)coroutine->m_code_object; } static int Nuitka_Coroutine_set_code( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "cr_code is not writable in Nuitka" ); return -1; } static PyObject *Nuitka_Coroutine_get_frame( struct Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_frame ) { Py_INCREF( coroutine->m_frame ); return (PyObject *)coroutine->m_frame; } else { Py_INCREF( Py_None ); return Py_None; } } static int Nuitka_Coroutine_set_frame( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { PyErr_Format( PyExc_RuntimeError, "gi_frame is not writable in Nuitka" ); return -1; } static void Nuitka_Coroutine_release_closure( struct Nuitka_CoroutineObject *coroutine ) { for( Py_ssize_t i = 0; i < coroutine->m_closure_given; i++ ) { CHECK_OBJECT( coroutine->m_closure[ i ] ); Py_DECREF( coroutine->m_closure[ i ] ); } coroutine->m_closure_given = 0; } // For the coroutine object fiber entry point, we may need to follow what // "makecontext" will support and that is only a list of integers, but we will need // to push a pointer through it, and so it's two of them, which might be fully // sufficient. #ifdef _NUITKA_MAKECONTEXT_INTS static void Nuitka_Coroutine_entry_point( int address_1, int address_2 ) { // Restore the pointer from integers should it be necessary, depending on // the platform. This requires pointers to be no larger that to "int" value. int addresses[2] = { address_1, address_2 }; struct Nuitka_CoroutineObject *coroutine = (struct Nuitka_CoroutineObject *)*(uintptr_t *)&addresses[0]; #else static void Nuitka_Coroutine_entry_point( struct Nuitka_CoroutineObject *coroutine ) { #endif ((coroutine_code)coroutine->m_code)( coroutine ); swapFiber( &coroutine->m_yielder_context, &coroutine->m_caller_context ); } static PyObject *_Nuitka_Coroutine_send( struct Nuitka_CoroutineObject *coroutine, PyObject *value, bool closing ) { if ( coroutine->m_status == status_Unused && value != NULL && value != Py_None ) { PyErr_Format( PyExc_TypeError, "can't send non-None value to a just-started coroutine" ); return NULL; } if ( coroutine->m_status != status_Finished ) { PyThreadState *thread_state = PyThreadState_GET(); if ( coroutine->m_running ) { PyErr_Format( PyExc_ValueError, "coroutine already executing" ); return NULL; } if ( coroutine->m_status == status_Unused ) { // Prepare the coroutine context to run. int res = prepareFiber( &coroutine->m_yielder_context, (void *)Nuitka_Coroutine_entry_point, (uintptr_t)coroutine ); if ( res != 0 ) { PyErr_Format( PyExc_MemoryError, "coroutine cannot be allocated" ); return NULL; } coroutine->m_status = status_Running; } coroutine->m_yielded = value; // Put the coroutine back on the frame stack. PyFrameObject *return_frame = thread_state->frame; #ifndef __NUITKA_NO_ASSERT__ if ( return_frame ) { assertFrameObject( (struct Nuitka_FrameObject *)return_frame ); } #endif if ( coroutine->m_frame ) { // It would be nice if our frame were still alive. Nobody had the // right to release it. assertFrameObject( coroutine->m_frame ); // It's not supposed to be on the top right now. assert( return_frame != &coroutine->m_frame->m_frame ); Py_XINCREF( return_frame ); coroutine->m_frame->m_frame.f_back = return_frame; thread_state->frame = &coroutine->m_frame->m_frame; } // Continue the yielder function while preventing recursion. coroutine->m_running = true; swapFiber( &coroutine->m_caller_context, &coroutine->m_yielder_context ); coroutine->m_running = false; thread_state = PyThreadState_GET(); // Remove the coroutine from the frame stack. if ( coroutine->m_frame ) { assert( thread_state->frame == &coroutine->m_frame->m_frame ); assertFrameObject( coroutine->m_frame ); Py_CLEAR( coroutine->m_frame->m_frame.f_back ); } thread_state->frame = return_frame; #ifndef __NUITKA_NO_ASSERT__ if ( return_frame ) { assertFrameObject( (struct Nuitka_FrameObject *)return_frame ); } #endif if ( coroutine->m_returned != NULL ) { coroutine->m_status = status_Finished; Py_XDECREF( coroutine->m_frame ); coroutine->m_frame = NULL; Nuitka_Coroutine_release_closure( coroutine ); PyObject *result = coroutine->m_returned; if ( result == Py_None ) { PyErr_SetObject( PyExc_StopIteration, Py_None ); Py_DECREF( Py_None ); } else { PyObject *exc_result = PyObject_CallFunctionObjArgs( PyExc_StopIteration, result, NULL ); Py_DECREF( result ); if (likely( exc_result != NULL )) { PyErr_SetObject( PyExc_StopIteration, exc_result ); Py_DECREF( exc_result ); } } return NULL; } else if ( coroutine->m_yielded == NULL ) { assert( ERROR_OCCURRED() ); coroutine->m_status = status_Finished; Py_XDECREF( coroutine->m_frame ); coroutine->m_frame = NULL; Nuitka_Coroutine_release_closure( coroutine ); assert( ERROR_OCCURRED() ); if ( GET_ERROR_OCCURRED() == PyExc_StopIteration ) { PyObject *saved_exception_type, *saved_exception_value; PyTracebackObject *saved_exception_tb; FETCH_ERROR_OCCURRED( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); NORMALIZE_EXCEPTION( &saved_exception_type, &saved_exception_value, &saved_exception_tb ); PyErr_Format( PyExc_RuntimeError, "coroutine raised StopIteration" ); PyObject *exception_type, *exception_value; PyTracebackObject *exception_tb; FETCH_ERROR_OCCURRED( &exception_type, &exception_value, &exception_tb ); RAISE_EXCEPTION_WITH_CAUSE( &exception_type, &exception_value, &exception_tb, saved_exception_value ); CHECK_OBJECT( exception_value ); CHECK_OBJECT( saved_exception_value ); Py_INCREF( saved_exception_value ); PyException_SetContext( exception_value, saved_exception_value ); Py_DECREF( saved_exception_type ); Py_XDECREF( saved_exception_tb ); RESTORE_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } return NULL; } else { return coroutine->m_yielded; } } else { #if PYTHON_VERSION >= 352 || !defined(_NUITKA_FULL_COMPAT) /* This check got added in Python 3.5.2 only. It's good to do it, but * not fully compatible, therefore guard it. */ if ( closing == false ) { PyErr_Format( PyExc_RuntimeError, "cannot reuse already awaited coroutine" ); } else #endif { PyErr_SetObject( PyExc_StopIteration, (PyObject *)NULL ); } return NULL; } } static PyObject *Nuitka_Coroutine_send( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { return _Nuitka_Coroutine_send( coroutine, value, false ); } static PyObject *Nuitka_Coroutine_tp_iternext( struct Nuitka_CoroutineObject *coroutine ) { return Nuitka_Coroutine_send( coroutine, Py_None ); } PyObject *Nuitka_Coroutine_close( struct Nuitka_CoroutineObject *coroutine, PyObject *args ) { if ( coroutine->m_status == status_Running ) { coroutine->m_exception_type = PyExc_GeneratorExit; Py_INCREF( PyExc_GeneratorExit ); CHECK_OBJECT( coroutine->m_exception_type ); coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; PyObject *result = _Nuitka_Coroutine_send( coroutine, Py_None, true ); if (unlikely( result )) { Py_DECREF( result ); PyErr_Format( PyExc_RuntimeError, "coroutine ignored GeneratorExit" ); return NULL; } else { PyObject *error = GET_ERROR_OCCURRED(); assert( error != NULL ); if ( EXCEPTION_MATCH_GENERATOR( error ) ) { CLEAR_ERROR_OCCURRED(); Py_INCREF( Py_None ); return Py_None; } return NULL; } } Py_INCREF( Py_None ); return Py_None; } extern PyObject *const_str_plain_close; /* Also used for asyncgen. */ bool Nuitka_gen_close_iter( PyObject *yieldfrom ) { PyObject *meth = PyObject_GetAttr( yieldfrom, const_str_plain_close ); if (unlikely( meth == NULL )) { if (unlikely( !PyErr_ExceptionMatches( PyExc_AttributeError ))) { PyErr_WriteUnraisable( yieldfrom ); } CLEAR_ERROR_OCCURRED(); return true; } else { PyObject *retval = CALL_FUNCTION_NO_ARGS( meth ); Py_DECREF( meth ); if (unlikely (retval == NULL)) { return false; } Py_DECREF( retval ); return true; } } extern PyObject *const_str_plain_throw; static PyObject *_Nuitka_Coroutine_throw2( struct Nuitka_CoroutineObject *coroutine, bool close_on_genexit ) { if ( coroutine->m_yieldfrom != NULL ) { if ( close_on_genexit ) { if ( PyErr_GivenExceptionMatches( coroutine->m_exception_type, PyExc_GeneratorExit ) ) { // Coroutines need to close the yield_from. coroutine->m_running = 1; bool res = Nuitka_gen_close_iter( coroutine->m_yieldfrom ); coroutine->m_running = 0; if ( res == true ) { return _Nuitka_Coroutine_send( coroutine, Py_None, false ); } goto throw_here; } } PyObject *meth = PyObject_GetAttr( coroutine->m_yieldfrom, const_str_plain_throw ); if (unlikely( meth == NULL )) { if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { return NULL; } CLEAR_ERROR_OCCURRED(); goto throw_here; } coroutine->m_running = 1; CHECK_OBJECT( coroutine->m_exception_type ); PyObject *ret = PyObject_CallFunctionObjArgs( meth, coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb, NULL ); coroutine->m_running = 0; Py_DECREF( meth ); if (unlikely( ret == NULL )) { PyObject *val; if ( _PyGen_FetchStopIterationValue( &val ) == 0 ) { ret = _Nuitka_Coroutine_send( coroutine, val, false ); Py_DECREF( val ); } else { ret = _Nuitka_Coroutine_send( coroutine, Py_None, false ); } } return ret; } throw_here: CHECK_OBJECT( coroutine->m_exception_type ); if ( (PyObject *)coroutine->m_exception_tb == Py_None ) { coroutine->m_exception_tb = NULL; } else if ( coroutine->m_exception_tb != NULL && !PyTraceBack_Check( coroutine->m_exception_tb ) ) { coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( PyExceptionClass_Check( coroutine->m_exception_type )) { Py_INCREF( coroutine->m_exception_type ); Py_XINCREF( coroutine->m_exception_value ); Py_XINCREF( coroutine->m_exception_tb ); NORMALIZE_EXCEPTION( &coroutine->m_exception_type, &coroutine->m_exception_value, &coroutine->m_exception_tb ); } else if ( PyExceptionInstance_Check( coroutine->m_exception_type ) ) { if ( coroutine->m_exception_value && coroutine->m_exception_value != Py_None ) { coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); return NULL; } coroutine->m_exception_value = coroutine->m_exception_type; Py_INCREF( coroutine->m_exception_value ); coroutine->m_exception_type = PyExceptionInstance_Class( coroutine->m_exception_type ); Py_INCREF( coroutine->m_exception_type ); Py_XINCREF( coroutine->m_exception_tb ); } else { PyErr_Format( PyExc_TypeError, "exceptions must be classes or instances deriving from BaseException, not %s", Py_TYPE( coroutine->m_exception_type )->tp_name ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } if ( ( coroutine->m_exception_tb != NULL ) && ( (PyObject *)coroutine->m_exception_tb != Py_None ) && ( !PyTraceBack_Check( coroutine->m_exception_tb ) ) ) { PyErr_Format( PyExc_TypeError, "throw() third argument must be a traceback object" ); return NULL; } if ( coroutine->m_status != status_Finished ) { PyObject *result = _Nuitka_Coroutine_send( coroutine, Py_None, false ); return result; } else { /* This seems wasteful to do it like this, but it's a corner case. */ RESTORE_ERROR_OCCURRED( coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; #if PYTHON_VERSION >= 352 || !defined(_NUITKA_FULL_COMPAT) /* This check got added in Python 3.5.2 only. It's good to do it, but * not fully compatible, therefore guard it. */ PyErr_Format( PyExc_RuntimeError, "cannot reuse already awaited coroutine" ); #endif return NULL; } } static PyObject *Nuitka_Coroutine_throw( struct Nuitka_CoroutineObject *coroutine, PyObject *args ) { assert( coroutine->m_exception_type == NULL ); assert( coroutine->m_exception_value == NULL ); assert( coroutine->m_exception_tb == NULL ); int res = PyArg_UnpackTuple( args, "throw", 1, 3, &coroutine->m_exception_type, &coroutine->m_exception_value, (PyObject **)&coroutine->m_exception_tb ); if (unlikely( res == 0 )) { coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } return _Nuitka_Coroutine_throw2( coroutine, true ); } static void Nuitka_Coroutine_tp_del( struct Nuitka_CoroutineObject *coroutine ) { if ( coroutine->m_status != status_Running ) { return; } PyObject *error_type, *error_value; PyTracebackObject *error_traceback; FETCH_ERROR_OCCURRED( &error_type, &error_value, &error_traceback ); PyObject *close_result = Nuitka_Coroutine_close( coroutine, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)coroutine ); } else { Py_DECREF( close_result ); } /* Restore the saved exception if any. */ RESTORE_ERROR_OCCURRED( error_type, error_value, error_traceback ); } static PyObject *Nuitka_Coroutine_tp_repr( struct Nuitka_CoroutineObject *coroutine ) { return PyUnicode_FromFormat( "", Nuitka_String_AsString( coroutine->m_qualname ), coroutine ); } static long Nuitka_Coroutine_tp_traverse( PyObject *coroutine, visitproc visit, void *arg ) { // TODO: Identify the impact of not visiting owned objects and/or if it // could be NULL instead. The "methodobject" visits its self and module. I // understand this is probably so that back references of this function to // its upper do not make it stay in the memory. A specific test if that // works might be needed. return 0; } static struct Nuitka_CoroutineWrapperObject *free_list_coro_wrappers = NULL; static int free_list_coro_wrappers_count = 0; static PyObject *Nuitka_Coroutine_await( struct Nuitka_CoroutineObject *coroutine ) { #if _DEBUG_COROUTINE PRINT_STRING("Nuitka_Coroutine_await enter"); PRINT_NEW_LINE(); #endif struct Nuitka_CoroutineWrapperObject *result; allocateFromFreeListFixed( free_list_coro_wrappers, struct Nuitka_CoroutineWrapperObject, Nuitka_CoroutineWrapper_Type ); if (unlikely(result == NULL)) { return NULL; } result->m_coroutine = coroutine; Py_INCREF( result->m_coroutine ); Nuitka_GC_Track( result ); return (PyObject *)result; } #define MAX_COROUTINE_FREE_LIST_COUNT 100 static struct Nuitka_CoroutineObject *free_list_coros = NULL; static int free_list_coros_count = 0; static void Nuitka_Coroutine_tp_dealloc( struct Nuitka_CoroutineObject *coroutine ) { // Revive temporarily. assert( Py_REFCNT( coroutine ) == 0 ); Py_REFCNT( coroutine ) = 1; // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); PyObject *close_result = Nuitka_Coroutine_close( coroutine, NULL ); if (unlikely( close_result == NULL )) { PyErr_WriteUnraisable( (PyObject *)coroutine ); } else { Py_DECREF( close_result ); } Nuitka_Coroutine_release_closure( coroutine ); Py_XDECREF( coroutine->m_frame ); assert( Py_REFCNT( coroutine ) == 1 ); Py_REFCNT( coroutine ) = 0; releaseFiber( &coroutine->m_yielder_context ); // Now it is safe to release references and memory for it. Nuitka_GC_UnTrack( coroutine ); if ( coroutine->m_weakrefs != NULL ) { PyObject_ClearWeakRefs( (PyObject *)coroutine ); assert( !ERROR_OCCURRED() ); } Py_DECREF( coroutine->m_name ); Py_DECREF( coroutine->m_qualname ); /* Put the object into freelist or release to GC */ releaseToFreeList( free_list_coros, coroutine, MAX_COROUTINE_FREE_LIST_COUNT ); RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); } #include // TODO: Set "__doc__" automatically for method clones of compiled types from // the documentation of built-in original type. static PyMethodDef Nuitka_Coroutine_methods[] = { { "send", (PyCFunction)Nuitka_Coroutine_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_Coroutine_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_Coroutine_close, METH_NOARGS, NULL }, { NULL } }; // TODO: Set "__doc__" automatically for method clones of compiled types from // the documentation of built-in original type. static PyGetSetDef Nuitka_Coroutine_getsetlist[] = { { (char *)"__name__", (getter)Nuitka_Coroutine_get_name, (setter)Nuitka_Coroutine_set_name, NULL }, { (char *)"__qualname__", (getter)Nuitka_Coroutine_get_qualname, (setter)Nuitka_Coroutine_set_qualname, NULL }, { (char *)"cr_await", (getter)Nuitka_Coroutine_get_cr_await, (setter)NULL, NULL }, { (char *)"cr_code", (getter)Nuitka_Coroutine_get_code, (setter)Nuitka_Coroutine_set_code, NULL }, { (char *)"cr_frame", (getter)Nuitka_Coroutine_get_frame, (setter)Nuitka_Coroutine_set_frame, NULL }, { NULL } }; static PyMemberDef Nuitka_Coroutine_members[] = { { (char *)"cr_running", T_BOOL, offsetof(struct Nuitka_CoroutineObject, m_running), READONLY }, { NULL } }; static PyAsyncMethods Nuitka_Coroutine_as_async = { (unaryfunc)Nuitka_Coroutine_await, /* am_await */ 0, /* am_aiter */ 0 /* am_anext */ }; PyTypeObject Nuitka_Coroutine_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_coroutine", /* tp_name */ sizeof(struct Nuitka_CoroutineObject), /* tp_basicsize */ sizeof(struct Nuitka_CellObject *), /* tp_itemsize */ (destructor)Nuitka_Coroutine_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ &Nuitka_Coroutine_as_async, /* tp_as_async */ (reprfunc)Nuitka_Coroutine_tp_repr, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_FINALIZE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_Coroutine_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ offsetof(struct Nuitka_CoroutineObject, m_weakrefs), /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_Coroutine_tp_iternext, /* tp_iternext */ Nuitka_Coroutine_methods, /* tp_methods */ Nuitka_Coroutine_members, /* tp_members */ Nuitka_Coroutine_getsetlist, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ 0, /* tp_is_gc */ 0, /* tp_bases */ 0, /* tp_mro */ 0, /* tp_cache */ 0, /* tp_subclasses */ 0, /* tp_weaklist */ 0, /* tp_del */ 0, /* tp_version_tag */ (destructor)Nuitka_Coroutine_tp_del, /* tp_finalize */ }; static void Nuitka_CoroutineWrapper_tp_dealloc( struct Nuitka_CoroutineWrapperObject *cw ) { Nuitka_GC_UnTrack( (PyObject *)cw ); Py_DECREF( cw->m_coroutine ); cw->m_coroutine = NULL; releaseToFreeList( free_list_coro_wrappers, cw, MAX_COROUTINE_FREE_LIST_COUNT ); } static PyObject *Nuitka_CoroutineWrapper_tp_iternext( struct Nuitka_CoroutineWrapperObject *cw ) { return Nuitka_Coroutine_send( cw->m_coroutine, Py_None ); } static int Nuitka_CoroutineWrapper_tp_traverse( struct Nuitka_CoroutineWrapperObject *cw, visitproc visit, void *arg ) { Py_VISIT( (PyObject *)cw->m_coroutine ); return 0; } static PyObject *Nuitka_CoroutineWrapper_send( struct Nuitka_CoroutineWrapperObject *cw, PyObject *arg ) { return Nuitka_Coroutine_send( cw->m_coroutine, arg ); } static PyObject *Nuitka_CoroutineWrapper_throw( struct Nuitka_CoroutineWrapperObject *cw, PyObject *args ) { return Nuitka_Coroutine_throw( cw->m_coroutine, args ); } static PyObject *Nuitka_CoroutineWrapper_close( struct Nuitka_CoroutineWrapperObject *cw, PyObject *args ) { return Nuitka_Coroutine_close( cw->m_coroutine, args ); } static PyMethodDef Nuitka_CoroutineWrapper_methods[] = { { "send", (PyCFunction)Nuitka_CoroutineWrapper_send, METH_O, NULL }, { "throw", (PyCFunction)Nuitka_CoroutineWrapper_throw, METH_VARARGS, NULL }, { "close", (PyCFunction)Nuitka_CoroutineWrapper_close, METH_NOARGS, NULL }, { NULL } }; PyTypeObject Nuitka_CoroutineWrapper_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_coroutine_wrapper", sizeof(struct Nuitka_CoroutineWrapperObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_CoroutineWrapper_tp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Nuitka_CoroutineWrapper_tp_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_CoroutineWrapper_tp_iternext, /* tp_iternext */ Nuitka_CoroutineWrapper_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ }; PyObject *Nuitka_Coroutine_New( coroutine_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, Py_ssize_t closure_given ) { struct Nuitka_CoroutineObject *result; // Macro to assign result memory from GC or free list. allocateFromFreeList( free_list_coros, struct Nuitka_CoroutineObject, Nuitka_Coroutine_Type, closure_given ); result->m_code = (void *)code; CHECK_OBJECT( name ); result->m_name = name; Py_INCREF( name ); CHECK_OBJECT( qualname ); result->m_qualname = qualname; Py_INCREF( qualname ); // TODO: Makes no sense with coroutines maybe? result->m_yieldfrom = NULL; // The m_closure is set from the outside. result->m_closure_given = closure_given; result->m_weakrefs = NULL; result->m_status = status_Unused; result->m_running = false; result->m_awaiting = false; result->m_exception_type = NULL; result->m_exception_value = NULL; result->m_exception_tb = NULL; result->m_yielded = NULL; result->m_returned = NULL; result->m_frame = NULL; result->m_code_object = code_object; initFiber( &result->m_yielder_context ); Nuitka_GC_Track( result ); return (PyObject *)result; } extern PyObject *PyGen_Send( PyGenObject *gen, PyObject *arg ); extern PyObject *const_str_plain_send; static int gen_is_coroutine( PyObject *object ) { if ( PyGen_CheckExact( object ) ) { PyCodeObject *code = (PyCodeObject *)((PyGenObject *)object)->gi_code; if ( code->co_flags & CO_ITERABLE_COROUTINE ) { return 1; } } return 0; } #if PYTHON_VERSION < 360 static #endif PyObject *PyCoro_GetAwaitableIter( PyObject *value ) { unaryfunc getter = NULL; if ( PyCoro_CheckExact( value ) || gen_is_coroutine( value ) ) { Py_INCREF( value ); return value; } if ( Py_TYPE( value )->tp_as_async != NULL ) { getter = Py_TYPE( value )->tp_as_async->am_await; } if ( getter != NULL ) { PyObject *result = (*getter)( value ); if ( result != NULL ) { if (unlikely( PyCoro_CheckExact( result ) || gen_is_coroutine( result ) || Nuitka_Coroutine_Check( result ) )) { Py_DECREF( result ); PyErr_Format( PyExc_TypeError, "__await__() returned a coroutine" ); return NULL; } if (unlikely( !HAS_ITERNEXT( result ) )) { PyErr_Format( PyExc_TypeError, "__await__() returned non-iterator of type '%s'", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } } return result; } PyErr_Format( PyExc_TypeError, "object %s can't be used in 'await' expression", Py_TYPE( value )->tp_name ); return NULL; } static void RAISE_COROUTINE_EXCEPTION( struct Nuitka_CoroutineObject *coroutine ) { CHECK_OBJECT( coroutine->m_exception_type ); RESTORE_ERROR_OCCURRED( coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; } extern PyObject *ERROR_GET_STOP_ITERATION_VALUE(); extern PyObject *const_str_plain_send, *const_str_plain_throw, *const_str_plain_close; static PyObject *yieldFromCoroutine( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { CHECK_OBJECT( value ); // This is the value, propagated back and forth the sub-generator and the // yield from consumer. PyObject *send_value = Py_None; while( 1 ) { // Send iteration value to the sub-generator, which may be a CPython // generator object, something with an iterator next, or a send method, // where the later is only required if values other than "None" need to // be passed in. PyObject *retval; // Exception, was thrown into us, need to send that to sub-generator. if ( coroutine->m_exception_type ) { // The yielding coroutine is being closed, but we also are tasked to // immediately close the currently running sub-generator. if ( EXCEPTION_MATCH_BOOL_SINGLE( coroutine->m_exception_type, PyExc_GeneratorExit ) ) { PyObject *close_method = PyObject_GetAttr( value, const_str_plain_close ); if ( close_method ) { PyObject *close_value = PyObject_Call( close_method, const_tuple_empty, NULL ); Py_DECREF( close_method ); if (unlikely( close_value == NULL )) { return NULL; } Py_DECREF( close_value ); } else { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL && !EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_AttributeError ) ) { PyErr_WriteUnraisable( (PyObject *)value ); } } RAISE_COROUTINE_EXCEPTION( coroutine ); return NULL; } PyObject *throw_method = PyObject_GetAttr( value, const_str_plain_throw ); if ( throw_method ) { retval = PyObject_CallFunctionObjArgs( throw_method, coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb, NULL ); Py_DECREF( throw_method ); if (unlikely( send_value == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) ) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { CLEAR_ERROR_OCCURRED(); RAISE_COROUTINE_EXCEPTION( coroutine ); return NULL; } else { assert( ERROR_OCCURRED() ); Py_CLEAR( coroutine->m_exception_type ); Py_CLEAR( coroutine->m_exception_value ); Py_CLEAR( coroutine->m_exception_tb ); return NULL; } } else if ( PyGen_CheckExact( value ) || PyCoro_CheckExact( value ) ) { retval = PyGen_Send( (PyGenObject *)value, Py_None ); } else if ( send_value == Py_None && Py_TYPE( value )->tp_iternext != NULL ) { retval = Py_TYPE( value )->tp_iternext( value ); } else { retval = PyObject_CallMethodObjArgs( value, const_str_plain_send, send_value, NULL ); } // Check the sub-generator result if ( retval == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { Py_INCREF( Py_None ); return Py_None; } // The sub-generator has given an exception. In case of // StopIteration, we need to check the value, as it is going to be // the expression value of this "yield from", and we are done. All // other errors, we need to raise. if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { return ERROR_GET_STOP_ITERATION_VALUE(); } return NULL; } else { coroutine->m_yielded = retval; coroutine->m_yieldfrom = value; // Return to the calling context. swapFiber( &coroutine->m_yielder_context, &coroutine->m_caller_context ); coroutine->m_yieldfrom = NULL; send_value = coroutine->m_yielded; if ( ERROR_OCCURRED() ) { assert( coroutine->m_exception_type != NULL ); CLEAR_ERROR_OCCURRED(); CHECK_OBJECT( coroutine->m_exception_type ); } CHECK_OBJECT( send_value ); } } } PyObject *COROUTINE_AWAIT( struct Nuitka_CoroutineObject *coroutine, PyObject *awaitable ) { #if _DEBUG_COROUTINE PRINT_STRING("COROUTINE_AWAIT entry:"); PRINT_NEW_LINE(); PRINT_ITEM( awaitable ); PRINT_NEW_LINE(); #endif PyObject *awaitable_iter = PyCoro_GetAwaitableIter( awaitable ); if (unlikely( awaitable_iter == NULL )) { return NULL; } #if PYTHON_VERSION >= 352 || !defined(_NUITKA_FULL_COMPAT) /* This check got added in Python 3.5.2 only. It's good to do it, but * not fully compatible, therefore guard it. */ if ( Nuitka_Coroutine_Check( awaitable ) ) { struct Nuitka_CoroutineObject *awaited_coroutine = (struct Nuitka_CoroutineObject *)awaitable; if ( awaited_coroutine->m_awaiting ) { Py_DECREF( awaitable_iter ); PyErr_Format( PyExc_RuntimeError, "coroutine is being awaited already" ); return NULL; } } #endif coroutine->m_awaiting = true; PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); coroutine->m_awaiting = false; Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("COROUTINE_AWAIT exit: "); if ( retval ) { PRINT_ITEM( retval ); } else { PRINT_CURRENT_EXCEPTION(); } PRINT_REFCOUNT( (PyObject *)coroutine ); PRINT_NEW_LINE(); #endif return retval; } PyObject *COROUTINE_AWAIT_IN_HANDLER( struct Nuitka_CoroutineObject *coroutine, PyObject *awaitable ) { #if _DEBUG_COROUTINE PRINT_STRING("AWAIT entry:"); PRINT_ITEM( awaitable ); PRINT_NEW_LINE(); #endif PyObject *awaitable_iter = PyCoro_GetAwaitableIter( awaitable ); if (unlikely( awaitable_iter == NULL )) { return NULL; } #if PYTHON_VERSION >= 352 || !defined(_NUITKA_FULL_COMPAT) /* This check got added in Python 3.5.2 only. It's good to do it, but * not fully compatible, therefore guard it. */ if ( Nuitka_Coroutine_Check( awaitable ) ) { struct Nuitka_CoroutineObject *awaited_coroutine = (struct Nuitka_CoroutineObject *)awaitable; if ( awaited_coroutine->m_awaiting ) { Py_DECREF( awaitable_iter ); PyErr_Format( PyExc_RuntimeError, "coroutine is being awaited already" ); return NULL; } } #endif /* When yielding from an exception handler in Python3, the exception * preserved to the frame is restore, while the current one is put there. */ PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD exit:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; coroutine->m_awaiting = true; PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); coroutine->m_awaiting = false; Py_DECREF( awaitable_iter ); // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD return:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; #if _DEBUG_COROUTINE PRINT_STRING("AWAIT exit: "); if ( retval ) { PRINT_ITEM( retval ); } else { PRINT_CURRENT_EXCEPTION(); } PRINT_NEW_LINE(); #endif return retval; } #if PYTHON_VERSION >= 352 /* Our "aiter" wrapper clone */ struct Nuitka_AIterWrapper { PyObject_HEAD PyObject *aw_aiter; }; static PyObject *Nuitka_AIterWrapper_iternext( struct Nuitka_AIterWrapper *aw ) { PyErr_SetObject( PyExc_StopIteration, aw->aw_aiter ); return NULL; } static int Nuitka_AIterWrapper_traverse( struct Nuitka_AIterWrapper *aw, visitproc visit, void *arg ) { Py_VISIT( (PyObject *)aw->aw_aiter ); return 0; } static struct Nuitka_AIterWrapper *free_list_coroutine_aiter_wrappers = NULL; static int free_list_coroutine_aiter_wrappers_count = 0; static void Nuitka_AIterWrapper_dealloc( struct Nuitka_AIterWrapper *aw ) { Nuitka_GC_UnTrack( (PyObject *)aw ); Py_CLEAR( aw->aw_aiter ); /* Put the object into freelist or release to GC */ releaseToFreeList( free_list_coroutine_aiter_wrappers, aw, MAX_COROUTINE_FREE_LIST_COUNT ); } static PyAsyncMethods Nuitka_AIterWrapper_as_async = { PyObject_SelfIter, /* am_await */ 0, /* am_aiter */ 0 /* am_anext */ }; PyTypeObject Nuitka_AIterWrapper_Type = { PyVarObject_HEAD_INIT(NULL, 0) "compiled_aiter_wrapper", sizeof(struct Nuitka_AIterWrapper), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Nuitka_AIterWrapper_dealloc, /* destructor tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ &Nuitka_AIterWrapper_as_async, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ PyObject_GenericGetAttr, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ "A wrapper object for '__aiter__' backwards compatibility.", (traverseproc)Nuitka_AIterWrapper_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Nuitka_AIterWrapper_iternext, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ 0, /* tp_free */ }; PyObject *Nuitka_AIterWrapper_New( PyObject *aiter ) { struct Nuitka_AIterWrapper *result; allocateFromFreeListFixed( free_list_coroutine_aiter_wrappers, struct Nuitka_AIterWrapper, Nuitka_AIterWrapper_Type ); Py_INCREF( aiter ); result->aw_aiter = aiter; Nuitka_GC_Track( result ); return (PyObject *)result; } #endif PyObject *COROUTINE_ASYNC_MAKE_ITERATOR( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { #if _DEBUG_COROUTINE PRINT_STRING("AITER entry:"); PRINT_ITEM( value ); PRINT_NEW_LINE(); #endif unaryfunc getter = NULL; if ( Py_TYPE( value )->tp_as_async ) { getter = Py_TYPE( value )->tp_as_async->am_aiter; } if (unlikely( getter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' requires an object with __aiter__ method, got %s", Py_TYPE( value )->tp_name ); return NULL; } PyObject *iter = (*getter)( value ); if (unlikely( iter == NULL )) { return NULL; } #if PYTHON_VERSION >= 352 /* Starting with Python 3.5.2 it is acceptable to return an async iterator * directly, instead of an awaitable. */ if ( Py_TYPE( iter )->tp_as_async != NULL && Py_TYPE( iter )->tp_as_async->am_anext != NULL) { PyObject *wrapper = Nuitka_AIterWrapper_New( iter ); Py_DECREF( iter ); iter = wrapper; } #endif PyObject *awaitable_iter = PyCoro_GetAwaitableIter( iter ); if (unlikely( awaitable_iter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' received an invalid object from __aiter__: %s", Py_TYPE( iter )->tp_name ); Py_DECREF( iter ); return NULL; } Py_DECREF( iter ); PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("AITER exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } PyObject *COROUTINE_ASYNC_ITERATOR_NEXT( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { #if _DEBUG_COROUTINE PRINT_STRING("ANEXT entry:"); PRINT_ITEM( value ); PRINT_NEW_LINE(); #endif unaryfunc getter = NULL; if ( Py_TYPE( value )->tp_as_async ) { getter = Py_TYPE( value )->tp_as_async->am_anext; } if (unlikely( getter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' requires an iterator with __anext__ method, got %s", Py_TYPE( value )->tp_name ); return NULL; } PyObject *next_value = (*getter)( value ); if (unlikely( next_value == NULL )) { return NULL; } PyObject *awaitable_iter = PyCoro_GetAwaitableIter( next_value ); if (unlikely( awaitable_iter == NULL )) { PyErr_Format( PyExc_TypeError, "'async for' received an invalid object from __anext__: %s", Py_TYPE( next_value )->tp_name ); Py_DECREF( next_value ); return NULL; } Py_DECREF( next_value ); PyObject *retval = yieldFromCoroutine( coroutine, awaitable_iter ); Py_DECREF( awaitable_iter ); #if _DEBUG_COROUTINE PRINT_STRING("ANEXT exit"); PRINT_ITEM( retval ); PRINT_NEW_LINE(); #endif return retval; } void _initCompiledCoroutineTypes( void ) { PyType_Ready( &Nuitka_Coroutine_Type ); PyType_Ready( &Nuitka_CoroutineWrapper_Type ); #if PYTHON_VERSION >= 352 PyType_Ready( &Nuitka_AIterWrapper_Type ); #endif } Nuitka-0.5.28.2/nuitka/build/static_src/arm_ucontext_src/0000755000372000001440000000000013207540420023567 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/static_src/arm_ucontext_src/ucontext.h0000644000372000037200000000472313022760405025567 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ #ifndef ARM_UCONTEXT_H #define ARM_UCONTEXT_H #include #include #ifdef __cplusplus extern "C" #endif int getmcontext(mcontext_t*); #ifdef __cplusplus extern "C" #endif void setmcontext(const mcontext_t*); int swapcontext(ucontext_t *oucp, const ucontext_t *ucp); void makecontext(ucontext_t *uc, void (*fn)(void), int argc, ...); #define setcontext(u) setmcontext(&(u)->uc_mcontext) #define getcontext(u) getmcontext(&(u)->uc_mcontext) #endif Nuitka-0.5.28.2/nuitka/build/static_src/arm_ucontext_src/getcontext.asm0000644000372000037200000000607312711355343026440 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ .globl getmcontext getmcontext: str r1, [r0,#4] str r2, [r0,#8] str r3, [r0,#12] str r4, [r0,#16] str r5, [r0,#20] str r6, [r0,#24] str r7, [r0,#28] str r8, [r0,#32] str r9, [r0,#36] str r10, [r0,#40] str r11, [r0,#44] str r12, [r0,#48] str r13, [r0,#52] str r14, [r0,#56] /* store 1 as r0-to-restore */ mov r1, #1 str r1, [r0] /* return 0 */ mov r0, #0 mov pc, lr .globl setmcontext setmcontext: ldr r1, [r0,#4] ldr r2, [r0,#8] ldr r3, [r0,#12] ldr r4, [r0,#16] ldr r5, [r0,#20] ldr r6, [r0,#24] ldr r7, [r0,#28] ldr r8, [r0,#32] ldr r9, [r0,#36] ldr r10, [r0,#40] ldr r11, [r0,#44] ldr r12, [r0,#48] ldr r13, [r0,#52] ldr r14, [r0,#56] ldr r0, [r0] mov pc, lr Nuitka-0.5.28.2/nuitka/build/static_src/arm_ucontext_src/fibers_arm.c0000644000372000037200000000720313022760374026023 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ // Implementation of process context switch for ARM #include "nuitka/prelude.h" #ifdef __cplusplus extern "C" { #endif int getmcontext( mcontext_t *mcontext ); void setmcontext( const mcontext_t *mcontext ); #ifdef __cplusplus } #endif #define setcontext(u) setmcontext( &(u)->uc_mcontext ) #define getcontext(u) getmcontext( &(u)->uc_mcontext ) void makecontext( ucontext_t *uc, void (*fn)(void), int argc, ... ); #define STACK_SIZE (1024*1024) // Keep one stack around to avoid the overhead of repeated malloc/free in // case of frequent instantiations in a loop. static void *last_stack = NULL; void _initFiber( Fiber *to ) { to->f_context.uc_stack.ss_sp = NULL; to->f_context.uc_link = NULL; to->start_stack = NULL; } int _prepareFiber( Fiber *to, void *code, uintptr_t arg ) { int res = getcontext( &to->f_context ); if (unlikely( res != 0 )) { return 1; } to->f_context.uc_stack.ss_size = STACK_SIZE; to->f_context.uc_stack.ss_sp = last_stack ? last_stack : malloc( STACK_SIZE ); to->start_stack = to->f_context.uc_stack.ss_sp; to->f_context.uc_link = NULL; last_stack = NULL; makecontext( &to->f_context, (void (*)())code, 1, (unsigned long)arg ); return 0; } void _releaseFiber( Fiber *to ) { if ( to->start_stack != NULL ) { if ( last_stack == NULL ) { last_stack = to->start_stack; } else { free( to->start_stack ); } to->start_stack = NULL; } } void _swapFiber( Fiber *to, Fiber *from ) { if ( getcontext( &to->f_context ) == 0 ) { setcontext( &from->f_context ); } } Nuitka-0.5.28.2/nuitka/build/static_src/arm_ucontext_src/ucontext.c0000644000372000037200000000473513022760374025572 0ustar hayenhayen00000000000000/* Extract of libtask, just the ARM specific portions. */ /* This software was developed as part of a project at MIT. Copyright (c) 2005-2007 Russ Cox, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. === Contains parts of an earlier library that has: * The authors of this software are Rob Pike, Sape Mullender, and Russ Cox * Copyright (c) 2003 by Lucent Technologies. * Permission to use, copy, modify, and distribute this software for any * purpose without fee is hereby granted, provided that this entire notice * is included in all copies of any software which is or includes a copy * or modification of this software and in all copies of the supporting * documentation for such software. * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED * WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR LUCENT TECHNOLOGIES MAKE ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. */ #include "ucontext.h" #include #include void makecontext(ucontext_t *uc, void (*fn)(void), int argc, ...) { int *sp; va_list arg; sp = (int*)uc->uc_stack.ss_sp+uc->uc_stack.ss_size/4; va_start(arg, argc); assert( argc == 1 ); // That allows me to hard code r0 uc->uc_mcontext.trap_no = va_arg(arg, unsigned long); va_end(arg); uc->uc_mcontext.arm_r10 = (unsigned long)sp; uc->uc_mcontext.arm_fp = (unsigned long)fn; } Nuitka-0.5.28.2/nuitka/build/static_src/MetaPathBasedLoader.c0000644000372000001440000006225013207540035024154 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // // This implements the loading of C compiled modules and shared library // extension modules bundled for standalone mode. // This is achieved mainly by registered a "sys.meta_path" loader, that then // gets asked for module names, and responds if knows about one. It's fed by // a table created at compile time. // The nature and use of these 2 loaded module kinds is very different, but // having them as distinct loaders would only require to duplicate the search // and registering of stuff. #include #ifdef _WIN32 #undef SEP #define SEP '\\' #endif #include "nuitka/prelude.h" #include "nuitka/unfreezing.h" extern char *getDirname( char *path ); // For Python3.3, the loader is a module attribute, so we need to make it // accessible from this variable. #if PYTHON_VERSION < 330 static #endif PyObject *metapath_based_loader = NULL; #ifdef _NUITKA_EXE static inline bool isVerbose( void ) { return Py_VerboseFlag != 0; } #elif _NUITKA_SYSFLAG_VERBOSE static inline bool isVerbose( void ) { return true; } #else static inline bool isVerbose( void ) { return false; } #endif static struct Nuitka_MetaPathBasedLoaderEntry *loader_entries = NULL; static bool hasFrozenModule( char const *name ) { for ( struct _frozen const *p = PyImport_FrozenModules; ; p++ ) { if ( p->name == NULL ) { return false; } if ( strcmp( p->name, name ) == 0 ) { break; } } return true; } static char *copyModulenameAsPath( char *buffer, char const *module_name ) { while( *module_name ) { if ( *module_name == '.' ) { *buffer++ = SEP; module_name++; } else { *buffer++ = *module_name++; } } *buffer = 0; return buffer; } extern PyObject *const_str_plain___path__; extern PyObject *const_str_plain___file__; extern PyObject *const_str_plain___loader__; // TODO: This updates the wrong absolute path. We ought to change it to // the "module_path_name" at the time of writing it, then we save a few // bytes in the blob, and don't have to create that string here. #ifdef _NUITKA_STANDALONE static void patchCodeObjectPaths( PyCodeObject *code_object, PyObject *module_path ) { code_object->co_filename = module_path; Py_INCREF( module_path ); Py_ssize_t nconsts = PyTuple_GET_SIZE( code_object->co_consts ); for ( int i = 0; i < nconsts; i++ ) { PyObject *constant = PyTuple_GET_ITEM( code_object->co_consts, i ); if ( PyCode_Check( constant ) ) { patchCodeObjectPaths( (PyCodeObject *)constant, module_path ); } } } #endif static PyObject *loadModuleFromCodeObject( PyCodeObject *code_object, char const *name, bool is_package ) { assert( code_object != NULL ); PyObject *modules = PyImport_GetModuleDict(); PyObject *module; assert( PyDict_GetItemString( modules, name ) == NULL ); module = PyModule_New( name ); assert( module != NULL ); int res = PyDict_SetItemString( modules, name, module ); assert( res == 0 ); char buffer[ MAXPATHLEN+1 ]; PyObject *module_path_entry = NULL; if ( is_package ) { copyModulenameAsPath( buffer, name ); #if PYTHON_VERSION < 300 PyObject *module_path_entry_base = PyString_FromString( buffer ); #else PyObject *module_path_entry_base = PyUnicode_FromString( buffer ); #endif module_path_entry = MAKE_RELATIVE_PATH( module_path_entry_base ); Py_DECREF( module_path_entry_base ); char sep_str[2] = { SEP, 0 }; strncat( buffer, sep_str, sizeof(buffer)-1 ); strncat( buffer, "__init__.py", sizeof(buffer)-1 ); } else { copyModulenameAsPath( buffer, name ); strncat( buffer, ".py", sizeof(buffer)-1 ); } #if PYTHON_VERSION < 300 PyObject *module_path_name = PyString_FromString( buffer ); #else PyObject *module_path_name = PyUnicode_FromString( buffer ); #endif PyObject *module_path = MAKE_RELATIVE_PATH( module_path_name ); Py_DECREF( module_path_name ); if ( is_package ) { /* Set __path__ properly, unlike frozen module importer does. */ PyObject *path_list = PyList_New(1); if (unlikely( path_list == NULL )) return NULL; res = PyList_SetItem( path_list, 0, module_path_entry ); if (unlikely( res != 0 )) return NULL; Py_INCREF( module_path_entry ); res = PyObject_SetAttr( module, const_str_plain___path__, path_list ); if (unlikely( res != 0 )) return NULL; Py_DECREF( path_list ); } #ifdef _NUITKA_STANDALONE patchCodeObjectPaths( code_object, module_path ); #endif module = PyImport_ExecCodeModuleEx( (char *)name, (PyObject *)code_object, Nuitka_String_AsString( module_path ) ); Py_DECREF( module_path ); return module; } static struct Nuitka_MetaPathBasedLoaderEntry *findEntry( char const *name ) { struct Nuitka_MetaPathBasedLoaderEntry *current = loader_entries; assert( current ); while ( current->name != NULL ) { if ( strcmp( name, current->name ) == 0 ) { return current; } current++; } return NULL; } static char *_kwlist[] = { (char *)"fullname", (char *)"unused", NULL }; static PyObject *_path_unfreezer_find_module( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; PyObject *unused; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O|O:find_module", _kwlist, &module_name, &unused ); if (unlikely( res == 0 )) { return NULL; } char const *name = Nuitka_String_AsString( module_name ); if ( isVerbose() ) { PySys_WriteStderr( "import %s # considering responsibility\n", name ); } struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); if ( entry ) { if ( isVerbose() ) { PySys_WriteStderr( "import %s # claimed responsibility (compiled)\n", name ); } Py_INCREF( metapath_based_loader ); return metapath_based_loader; } if ( hasFrozenModule( name ) ) { if ( isVerbose() ) { PySys_WriteStderr( "import %s # claimed responsibility (frozen)\n", name ); } Py_INCREF( metapath_based_loader ); return metapath_based_loader; } if ( isVerbose() ) { PySys_WriteStderr( "import %s # denied responsibility\n", name ); } Py_INCREF( Py_None ); return Py_None; } static char *_kwlist_get_data[] = { (char *)"filename", NULL }; extern PyObject *const_str_plain_read, *const_str_plain_rb; static PyObject *_path_unfreezer_get_data( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *filename; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O:get_data", _kwlist_get_data, &filename ); if (unlikely( res == 0 )) { return NULL; } #if PYTHON_VERSION < 300 PyObject *data_file = BUILTIN_OPEN( filename, const_str_plain_rb, NULL ); #else PyObject *data_file = BUILTIN_OPEN( filename, const_str_plain_rb, NULL, NULL, NULL, NULL, NULL, NULL ); #endif if (unlikely( data_file == NULL )) { // TODO: Issue a runtime warning maybe. return NULL; } PyObject *read_method = PyObject_GetAttr( data_file, const_str_plain_read ); Py_DECREF( data_file ); if (unlikely( read_method == NULL )) { return NULL; } PyObject *result = CALL_FUNCTION_NO_ARGS( read_method ); Py_DECREF( read_method ); return result; } #ifdef _NUITKA_STANDALONE #if PYTHON_VERSION < 300 typedef void (*entrypoint_t)( void ); #else typedef PyObject * (*entrypoint_t)( void ); #endif #ifndef _WIN32 // Shared libraries loading. #include #endif #if PYTHON_VERSION >= 350 static PyObject *createModuleSpec( PyObject *module_name ); #endif PyObject *callIntoShlibModule( const char *full_name, const char *filename ) { // Determine the package name and basename of the module to load. char const *dot = strrchr( full_name, '.' ); char const *name; char const *package; if ( dot == NULL ) { package = NULL; name = full_name; } else { package = (char *)full_name; name = dot+1; } char entry_function_name[1024]; snprintf( entry_function_name, sizeof( entry_function_name ), #if PYTHON_VERSION < 300 "init%s", #else "PyInit_%s", #endif name ); #ifdef _WIN32 unsigned int old_mode = SetErrorMode( SEM_FAILCRITICALERRORS ); if ( isVerbose() ) { PySys_WriteStderr( "import %s # LoadLibraryEx(\"%s\");\n", full_name, filename ); } HINSTANCE hDLL = LoadLibraryEx( filename, NULL, LOAD_WITH_ALTERED_SEARCH_PATH ); if (unlikely( hDLL == NULL )) { PyErr_Format( PyExc_ImportError, "LoadLibraryEx '%s' failed", filename ); return NULL; } entrypoint_t entrypoint = (entrypoint_t)GetProcAddress( hDLL, entry_function_name ); SetErrorMode( old_mode ); #else int dlopenflags = PyThreadState_GET()->interp->dlopenflags; if ( isVerbose() ) { PySys_WriteStderr( "import %s # dlopen(\"%s\", %x);\n", full_name, filename, dlopenflags ); } void *handle = dlopen( filename, dlopenflags ); if (unlikely( handle == NULL )) { const char *error = dlerror(); if (unlikely( error == NULL )) { error = "unknown dlopen() error"; } PyErr_SetString( PyExc_ImportError, error ); return NULL; } entrypoint_t entrypoint = (entrypoint_t)dlsym( handle, entry_function_name ); #endif assert( entrypoint ); char *old_context = _Py_PackageContext; _Py_PackageContext = (char *)package; // Finally call into the DLL. #if PYTHON_VERSION < 300 (*entrypoint)(); #else PyObject *module = (*entrypoint)(); #endif _Py_PackageContext = old_context; #if PYTHON_VERSION < 300 PyObject *module = PyDict_GetItemString( PyImport_GetModuleDict(), full_name ); #endif if (unlikely( module == NULL )) { if ( !ERROR_OCCURRED() ) { PyErr_Format( PyExc_SystemError, "dynamic module '%s' not initialized properly", full_name ); } return NULL; } #if PYTHON_VERSION >= 300 #if PYTHON_VERSION >= 350 PyModuleDef *def; if ( Py_TYPE( module ) == &PyModuleDef_Type ) { def = (PyModuleDef *)module; PyObject *spec = createModuleSpec( PyUnicode_FromString( full_name ) ); module = PyModule_FromDefAndSpec( def, spec ); Py_DECREF( spec ); if (unlikely( module == NULL )) { PyErr_Format( PyExc_SystemError, "dynamic module '%s' not initialized properly from def", full_name ); return NULL; } assert( PyModule_Check( module )); int res = PyModule_ExecDef( module, def ); if (unlikely( res == -1 )) { return NULL; } PyDict_SetItemString( PyImport_GetModuleDict(), full_name, module ); return module; } else { def = PyModule_GetDef( module ); } if (likely( def != NULL )) { def->m_base.m_init = entrypoint; } #else PyModuleDef *def = PyModule_GetDef( module ); if (unlikely( def == NULL )) { PyErr_Format( PyExc_SystemError, "initialization of %s did not return an extension module", filename ); return NULL; } def->m_base.m_init = entrypoint; #endif #endif // Set filename attribute int res = PyModule_AddStringConstant( module, "__file__", filename ); if (unlikely( res < 0 )) { // Might be refuted, which wouldn't be harmful. CLEAR_ERROR_OCCURRED(); } // Call the standard import fix-ups for extension modules. Their interface // changed over releases. #if PYTHON_VERSION < 300 PyObject *res2 = _PyImport_FixupExtension( (char *)full_name, (char *)filename ); if (unlikely( res2 == NULL )) { return NULL; } #elif PYTHON_VERSION < 330 PyObject *filename_obj = PyUnicode_DecodeFSDefault( filename ); CHECK_OBJECT( filename_obj ); res = _PyImport_FixupExtensionUnicode( module, (char *)full_name, filename_obj ); Py_DECREF( filename_obj ); if (unlikely( res == -1 )) { return NULL; } #else PyObject *full_name_obj = PyUnicode_FromString( full_name ); CHECK_OBJECT( full_name_obj ); PyObject *filename_obj = PyUnicode_DecodeFSDefault( filename ); CHECK_OBJECT( filename_obj ); res = _PyImport_FixupExtensionObject( module, full_name_obj, filename_obj ); Py_DECREF( full_name_obj ); Py_DECREF( filename_obj ); if (unlikely( res == -1 )) { return NULL; } #endif return module; } #endif static void loadTriggeredModule( char const *name, char const *trigger_name ) { char trigger_module_name[2048]; strncpy( trigger_module_name, name, sizeof(trigger_module_name)-1 ); strncat( trigger_module_name, trigger_name, sizeof(trigger_module_name)-1 ); struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( trigger_module_name ); if ( entry != NULL ) { if ( isVerbose() ) { PySys_WriteStderr( "Loading %s\n", trigger_module_name ); } entry->python_initfunc(); if (unlikely( ERROR_OCCURRED() )) { PyErr_Print(); abort(); } } } static PyObject *loadModule( PyObject *module_name, struct Nuitka_MetaPathBasedLoaderEntry *entry ) { #ifdef _NUITKA_STANDALONE if ( ( entry->flags & NUITKA_SHLIB_FLAG ) != 0 ) { // Append the the entry name from full path module name with dots, // and translate these into directory separators. char filename[ MAXPATHLEN + 1 ]; strcpy( filename, getBinaryDirectoryHostEncoded() ); char *d = filename; d += strlen( filename ); assert( *d == 0 ); *d++ = SEP; d = copyModulenameAsPath( d, entry->name ); #ifdef _WIN32 strcat( d, ".pyd" ); #else strcat( d, ".so" ); #endif callIntoShlibModule( entry->name, filename ); } else #endif if ( ( entry->flags & NUITKA_BYTECODE_FLAG ) != 0 ) { PyCodeObject *code_object = (PyCodeObject *)PyMarshal_ReadObjectFromString( (char *)&constant_bin[ entry->bytecode_start ], entry->bytecode_size ); // TODO: Probably a bit harsh reaction. if (unlikely( code_object == NULL )) { PyErr_Print(); abort(); } return loadModuleFromCodeObject( code_object, entry->name, ( entry->flags & NUITKA_PACKAGE_FLAG ) != 0 ); } else { assert( ( entry->flags & NUITKA_SHLIB_FLAG ) == 0 ); assert( entry->python_initfunc ); entry->python_initfunc(); } if (unlikely( ERROR_OCCURRED() )) { return NULL; } if ( isVerbose() ) { PySys_WriteStderr( "Loaded %s\n", entry->name ); } return LOOKUP_SUBSCRIPT( PyImport_GetModuleDict(), module_name ); } // Note: This may become an entry point for hard coded imports of compiled // stuff. PyObject *IMPORT_EMBEDDED_MODULE( PyObject *module_name, char const *name ) { struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); bool frozen_import = entry == NULL && hasFrozenModule( name ); if ( entry != NULL || frozen_import ) { // Execute the "preLoad" code produced for the module potentially. This // is from plug-ins typically, that want to modify things for the the // module before loading, to e.g. set a plug-in path, or do some monkey // patching in order to make things compatible. loadTriggeredModule( name, "-preLoad" ); } PyObject *result = NULL; if ( entry != NULL ) { result = loadModule( module_name, entry ); if ( result == NULL ) { return NULL; } } if ( frozen_import ) { int res = PyImport_ImportFrozenModule( (char *)name ); if (unlikely( res == -1 )) { return NULL; } if ( res == 1 ) { result = LOOKUP_SUBSCRIPT( PyImport_GetModuleDict(), module_name ); } } if ( result != NULL ) { // Execute the "postLoad" code produced for the module potentially. This // is from plug-ins typically, that want to modify the module immediately // after loading, to e.g. set a plug-in path, or do some monkey patching // in order to make things compatible. loadTriggeredModule( name, "-postLoad" ); return result; } Py_INCREF( Py_None ); return Py_None; } static PyObject *_path_unfreezer_load_module( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; PyObject *unused; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O|O:load_module", _kwlist, &module_name, &unused ); if (unlikely( res == 0 )) { return NULL; } assert( module_name ); assert( Nuitka_String_Check( module_name ) ); char *const name = Nuitka_String_AsString( module_name ); if ( isVerbose() ) { PySys_WriteStderr( "Loading %s\n", name ); } return IMPORT_EMBEDDED_MODULE( module_name, name ); } static PyObject *_path_unfreezer_is_package( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O:is_package", _kwlist, &module_name ); if (unlikely( res == 0 )) { return NULL; } assert( module_name ); assert( Nuitka_String_Check( module_name ) ); char *name = Nuitka_String_AsString( module_name ); struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); PyObject *result; if ( entry ) { result = BOOL_FROM( ( entry->flags & NUITKA_PACKAGE_FLAG ) != 0 ); } else { // TODO: Maybe needs to be an exception. result = Py_None; } Py_INCREF( result ); return result; } #if PYTHON_VERSION >= 340 static PyObject *_path_unfreezer_repr_module( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module; PyObject *unused; int res = PyArg_ParseTupleAndKeywords( args, kwds, "O|O:module_repr", _kwlist, &module, &unused ); if (unlikely( res == 0 )) { return NULL; } return PyUnicode_FromFormat("", PyModule_GetName( module ), PyModule_GetFilename( module )); } static char *_kwlist2[] = { (char *)"fullname", (char *)"is_package", (char *)"path", NULL }; static PyObject *createModuleSpec( PyObject *module_name ) { assert( module_name ); assert( Nuitka_String_Check( module_name ) ); char *name = Nuitka_String_AsString( module_name ); struct Nuitka_MetaPathBasedLoaderEntry *entry = findEntry( name ); if ( entry == NULL ) { Py_INCREF( Py_None ); return Py_None; } static PyObject *importlib = NULL; if ( importlib == NULL ) { importlib = PyImport_ImportModule( "importlib._bootstrap" ); } if (unlikely( importlib == NULL )) { return NULL; } static PyObject *module_spec_class = NULL; if ( module_spec_class == NULL ) { module_spec_class = PyObject_GetAttrString( importlib, "ModuleSpec" ); } if (unlikely( module_spec_class == NULL )) { return NULL; } PyObject *result = PyObject_CallFunctionObjArgs( module_spec_class, module_name, metapath_based_loader, NULL ); return result; } static PyObject *_path_unfreezer_find_spec( PyObject *self, PyObject *args, PyObject *kwds ) { PyObject *module_name; PyObject *unused1; PyObject *unused2; int res = PyArg_ParseTupleAndKeywords( args, kwds, "OO|O:find_spec", _kwlist2, &module_name, &unused1, &unused2 ); if (unlikely( res == 0 )) { return NULL; } return createModuleSpec( module_name ); } #endif static PyMethodDef _method_def_loader_get_data = { "get_data", (PyCFunction)_path_unfreezer_get_data, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_find_module = { "find_module", (PyCFunction)_path_unfreezer_find_module, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_load_module = { "load_module", (PyCFunction)_path_unfreezer_load_module, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_is_package = { "is_package", (PyCFunction)_path_unfreezer_is_package, METH_VARARGS | METH_KEYWORDS, NULL }; #if PYTHON_VERSION >= 340 static PyMethodDef _method_def_loader_repr_module = { "module_repr", (PyCFunction)_path_unfreezer_repr_module, METH_VARARGS | METH_KEYWORDS, NULL }; static PyMethodDef _method_def_loader_find_spec = { "module_repr", (PyCFunction)_path_unfreezer_find_spec, METH_VARARGS | METH_KEYWORDS, NULL }; #endif void registerMetaPathBasedUnfreezer( struct Nuitka_MetaPathBasedLoaderEntry *_loader_entries ) { // Do it only once. if ( loader_entries ) { assert( _loader_entries == loader_entries ); return; } loader_entries = _loader_entries; // Build the dictionary of the "loader" object, which needs to have two // methods "find_module" where we acknowledge that we are capable of loading // the module, and "load_module" that does the actual thing. PyObject *method_dict = PyDict_New(); CHECK_OBJECT( method_dict ); PyObject *loader_get_data = PyCFunction_New( &_method_def_loader_get_data, NULL ); CHECK_OBJECT( loader_get_data ); PyDict_SetItemString( method_dict, "get_data", loader_get_data ); PyObject *loader_find_module = PyCFunction_New( &_method_def_loader_find_module, NULL ); CHECK_OBJECT( loader_find_module ); PyDict_SetItemString( method_dict, "find_module", loader_find_module ); PyObject *loader_load_module = PyCFunction_New( &_method_def_loader_load_module, NULL ); CHECK_OBJECT( loader_load_module ); PyDict_SetItemString( method_dict, "load_module", loader_load_module ); PyObject *loader_is_package = PyCFunction_New( &_method_def_loader_is_package, NULL ); CHECK_OBJECT( loader_is_package ); PyDict_SetItemString( method_dict, "is_package", loader_is_package ); #if PYTHON_VERSION >= 330 PyDict_SetItemString( method_dict, "__module__", Py_None ); #endif #if PYTHON_VERSION >= 340 PyObject *loader_repr_module = PyCFunction_New( &_method_def_loader_repr_module, NULL ); CHECK_OBJECT( loader_repr_module ); PyDict_SetItemString( method_dict, "module_repr", loader_repr_module ); PyObject *loader_find_spec = PyCFunction_New( &_method_def_loader_find_spec, NULL ); CHECK_OBJECT( loader_find_spec ); PyDict_SetItemString( method_dict, "find_spec", loader_find_spec ); #endif // Build the actual class. metapath_based_loader = PyObject_CallFunctionObjArgs( (PyObject *)&PyType_Type, #if PYTHON_VERSION < 300 PyString_FromString( "_nuitka_compiled_modules_loader" ), #else PyUnicode_FromString( "_nuitka_compiled_modules_loader" ), #endif const_tuple_empty, method_dict, NULL ); CHECK_OBJECT( metapath_based_loader ); if ( isVerbose() ) { PySys_WriteStderr( "Setup nuitka compiled module/bytecode/shlib importer.\n" ); } // Register it as a meta path loader. int res = PyList_Insert( PySys_GetObject( ( char *)"meta_path" ), #if PYTHON_VERSION < 330 0, #else 2, #endif metapath_based_loader ); assert( res == 0 ); } Nuitka-0.5.28.2/nuitka/build/__init__.py0000644000372000001440000000150113112214770020161 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/build/include/0000755000372000001440000000000013207540420017475 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/include/coro.h0000644000372000037200000003437512702504125020573 0ustar hayenhayen00000000000000/* * Copyright (c) 2001-2012 Marc Alexander Lehmann * * Redistribution and use in source and binary forms, with or without modifica- * tion, are permitted provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER- * CHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO * EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPE- * CIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTH- * ERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE. * * Alternatively, the contents of this file may be used under the terms of * the GNU General Public License ("GPL") version 2 or any later version, * in which case the provisions of the GPL are applicable instead of * the above. If you wish to allow the use of your version of this file * only under the terms of the GPL and not to allow others to use your * version of this file under the BSD license, indicate your decision * by deleting the provisions above and replace them with the notice * and other provisions required by the GPL. If you do not delete the * provisions above, a recipient may use your version of this file under * either the BSD or the GPL. * * This library is modelled strictly after Ralf S. Engelschalls article at * http://www.gnu.org/software/pth/rse-pmt.ps. So most of the credit must * go to Ralf S. Engelschall . * * This coroutine library is very much stripped down. You should either * build your own process abstraction using it or - better - just use GNU * Portable Threads, http://www.gnu.org/software/pth/. * */ /* * 2006-10-26 Include stddef.h on OS X to work around one of its bugs. * Reported by Michael_G_Schwern. * 2006-11-26 Use _setjmp instead of setjmp on GNU/Linux. * 2007-04-27 Set unwind frame info if gcc 3+ and ELF is detected. * Use _setjmp instead of setjmp on _XOPEN_SOURCE >= 600. * 2007-05-02 Add assembly versions for x86 and amd64 (to avoid reliance * on SIGUSR2 and sigaltstack in Crossfire). * 2008-01-21 Disable CFI usage on anything but GNU/Linux. * 2008-03-02 Switched to 2-clause BSD license with GPL exception. * 2008-04-04 New (but highly unrecommended) pthreads backend. * 2008-04-24 Reinstate CORO_LOSER (had wrong stack adjustments). * 2008-10-30 Support assembly method on x86 with and without frame pointer. * 2008-11-03 Use a global asm statement for CORO_ASM, idea by pippijn. * 2008-11-05 Hopefully fix misaligned stacks with CORO_ASM/SETJMP. * 2008-11-07 rbp wasn't saved in CORO_ASM on x86_64. * introduce coro_destroy, which is a nop except for pthreads. * speed up CORO_PTHREAD. Do no longer leak threads either. * coro_create now allows one to create source coro_contexts. * do not rely on makecontext passing a void * correctly. * try harder to get _setjmp/_longjmp. * major code cleanup/restructuring. * 2008-11-10 the .cfi hacks are no longer needed. * 2008-11-16 work around a freebsd pthread bug. * 2008-11-19 define coro_*jmp symbols for easier porting. * 2009-06-23 tentative win32-backend support for mingw32 (Yasuhiro Matsumoto). * 2010-12-03 tentative support for uclibc (which lacks all sorts of things). * 2011-05-30 set initial callee-saved-registers to zero with CORO_ASM. * use .cfi_undefined rip on linux-amd64 for better backtraces. * 2011-06-08 maybe properly implement weird windows amd64 calling conventions. * 2011-07-03 rely on __GCC_HAVE_DWARF2_CFI_ASM for cfi detection. * 2011-08-08 cygwin trashes stacks, use pthreads with double stack on cygwin. * 2012-12-04 reduce misprediction penalty for x86/amd64 assembly switcher. * 2012-12-05 experimental fiber backend (allocates stack twice). * 2012-12-07 API version 3 - add coro_stack_alloc/coro_stack_free. * 2012-12-21 valgrind stack registering was broken. */ #ifndef CORO_H #define CORO_H #if __cplusplus extern "C" { #endif /* * This library consists of only three files * coro.h, coro.c and LICENSE (and optionally README) * * It implements what is known as coroutines, in a hopefully * portable way. * * All compiletime symbols must be defined both when including coro.h * (using libcoro) as well as when compiling coro.c (the implementation). * * You can manually specify which flavour you want. If you don't define * any of these, libcoro tries to choose a safe and fast default: * * -DCORO_UCONTEXT * * This flavour uses SUSv2's get/set/swap/makecontext functions that * unfortunately only some unices support, and is quite slow. * * -DCORO_SJLJ * * This flavour uses SUSv2's setjmp/longjmp and sigaltstack functions to * do it's job. Coroutine creation is much slower than UCONTEXT, but * context switching is a bit cheaper. It should work on almost all unices. * * -DCORO_LINUX * * CORO_SJLJ variant. * Old GNU/Linux systems (<= glibc-2.1) only work with this implementation * (it is very fast and therefore recommended over other methods, but * doesn't work with anything newer). * * -DCORO_LOSER * * CORO_SJLJ variant. * Microsoft's highly proprietary platform doesn't support sigaltstack, and * this selects a suitable workaround for this platform. It might not work * with your compiler though - it has only been tested with MSVC 6. * * -DCORO_FIBER * * Slower, but probably more portable variant for the Microsoft operating * system, using fibers. Ignores the passed stack and allocates it internally. * Also, due to bugs in cygwin, this does not work with cygwin. * * -DCORO_IRIX * * CORO_SJLJ variant. * For SGI's version of Microsoft's NT ;) * * -DCORO_ASM * * Hand coded assembly, known to work only on a few architectures/ABI: * GCC + x86/IA32 and amd64/x86_64 + GNU/Linux and a few BSDs. Fastest choice, * if it works. * * -DCORO_PTHREAD * * Use the pthread API. You have to provide and -lpthread. * This is likely the slowest backend, and it also does not support fork(), * so avoid it at all costs. * * If you define neither of these symbols, coro.h will try to autodetect * the best/safest model. To help with the autodetection, you should check * (e.g. using autoconf) and define the following symbols: HAVE_UCONTEXT_H * / HAVE_SETJMP_H / HAVE_SIGALTSTACK. */ /* * Changes when the API changes incompatibly. * This is ONLY the API version - there is no ABI compatibility between releases. * * Changes in API version 2: * replaced bogus -DCORO_LOOSE with grammatically more correct -DCORO_LOSER * Changes in API version 3: * introduced stack management (CORO_STACKALLOC) */ #define CORO_VERSION 3 #include /* * This is the type for the initialization function of a new coroutine. */ typedef void (*coro_func)(void *); /* * A coroutine state is saved in the following structure. Treat it as an * opaque type. errno and sigmask might be saved, but don't rely on it, * implement your own switching primitive if you need that. */ typedef struct coro_context coro_context; /* * This function creates a new coroutine. Apart from a pointer to an * uninitialised coro_context, it expects a pointer to the entry function * and the single pointer value that is given to it as argument. * * Allocating/deallocating the stack is your own responsibility. * * As a special case, if coro, arg, sptr and ssze are all zero, * then an "empty" coro_context will be created that is suitable * as an initial source for coro_transfer. * * This function is not reentrant, but putting a mutex around it * will work. */ void coro_create (coro_context *ctx, /* an uninitialised coro_context */ coro_func coro, /* the coroutine code to be executed */ void *arg, /* a single pointer passed to the coro */ void *sptr, /* start of stack area */ size_t ssze); /* size of stack area in bytes */ /* * The following prototype defines the coroutine switching function. It is * sometimes implemented as a macro, so watch out. * * This function is thread-safe and reentrant. */ #if 0 void coro_transfer (coro_context *prev, coro_context *next); #endif /* * The following prototype defines the coroutine destroy function. It * is sometimes implemented as a macro, so watch out. It also serves no * purpose unless you want to use the CORO_PTHREAD backend, where it is * used to clean up the thread. You are responsible for freeing the stack * and the context itself. * * This function is thread-safe and reentrant. */ #if 0 void coro_destroy (coro_context *ctx); #endif /*****************************************************************************/ /* optional stack management */ /*****************************************************************************/ /* * You can disable all of the stack management functions by * defining CORO_STACKALLOC to 0. Otherwise, they are enabled by default. * * If stack management is enabled, you can influence the implementation via these * symbols: * * -DCORO_USE_VALGRIND * * If defined, then libcoro will include valgrind/valgrind.h and register * and unregister stacks with valgrind. * * -DCORO_GUARDPAGES=n * * libcoro will try to use the specified number of guard pages to protect against * stack overflow. If n is 0, then the feature will be disabled. If it isn't * defined, then libcoro will choose a suitable default. If guardpages are not * supported on the platform, then the feature will be silently disabled. */ #ifndef CORO_STACKALLOC # define CORO_STACKALLOC 1 #endif #if CORO_STACKALLOC /* * The only allowed operations on these struct members is to read the * "sptr" and "ssze" members to pass it to coro_create, to read the "sptr" * member to see if it is false, in which case the stack isn't allocated, * and to set the "sptr" member to 0, to indicate to coro_stack_free to * not actually do anything. */ struct coro_stack { void *sptr; size_t ssze; #if CORO_USE_VALGRIND int valgrind_id; #endif }; /* * Try to allocate a stack of at least the given size and return true if * successful, or false otherwise. * * The size is *NOT* specified in bytes, but in units of sizeof (void *), * i.e. the stack is typically 4(8) times larger on 32 bit(64 bit) platforms * then the size passed in. * * If size is 0, then a "suitable" stack size is chosen (usually 1-2MB). */ int coro_stack_alloc (struct coro_stack *stack, unsigned int size); /* * Free the stack allocated by coro_stack_alloc again. It is safe to * call this function on the coro_stack structure even if coro_stack_alloc * failed. */ void coro_stack_free (struct coro_stack *stack); #endif /* * That was it. No other user-serviceable parts below here. */ /*****************************************************************************/ #if !defined CORO_LOSER && !defined CORO_UCONTEXT \ && !defined CORO_SJLJ && !defined CORO_LINUX \ && !defined CORO_IRIX && !defined CORO_ASM \ && !defined CORO_PTHREAD && !defined CORO_FIBER # if defined WINDOWS && (defined __i386 || defined __x86_64 || defined _M_IX86 || defined _M_AMD64) # define CORO_ASM 1 # elif defined WINDOWS || defined _WIN32 # define CORO_LOSER 1 /* you don't win with windoze */ # elif __linux && (__i386 || (__x86_64 && !__ILP32)) # define CORO_ASM 1 # elif defined HAVE_UCONTEXT_H # define CORO_UCONTEXT 1 # elif defined HAVE_SETJMP_H && defined HAVE_SIGALTSTACK # define CORO_SJLJ 1 # else error unknown or unsupported architecture # endif #endif /*****************************************************************************/ #if CORO_UCONTEXT # include struct coro_context { ucontext_t uc; }; # define coro_transfer(p,n) swapcontext (&((p)->uc), &((n)->uc)) # define coro_destroy(ctx) (void *)(ctx) #elif CORO_SJLJ || CORO_LOSER || CORO_LINUX || CORO_IRIX # if defined(CORO_LINUX) && !defined(_GNU_SOURCE) # define _GNU_SOURCE /* for glibc */ # endif # if !CORO_LOSER # include # endif /* solaris is hopelessly borked, it expands _XOPEN_UNIX to nothing */ # if __sun # undef _XOPEN_UNIX # define _XOPEN_UNIX 1 # endif # include # if _XOPEN_UNIX > 0 || defined (_setjmp) # define coro_jmp_buf jmp_buf # define coro_setjmp(env) _setjmp (env) # define coro_longjmp(env) _longjmp ((env), 1) # elif CORO_LOSER # define coro_jmp_buf jmp_buf # define coro_setjmp(env) setjmp (env) # define coro_longjmp(env) longjmp ((env), 1) # else # define coro_jmp_buf sigjmp_buf # define coro_setjmp(env) sigsetjmp (env, 0) # define coro_longjmp(env) siglongjmp ((env), 1) # endif struct coro_context { coro_jmp_buf env; }; # define coro_transfer(p,n) do { if (!coro_setjmp ((p)->env)) coro_longjmp ((n)->env); } while (0) # define coro_destroy(ctx) (void *)(ctx) #elif CORO_ASM struct coro_context { void **sp; /* must be at offset 0 */ }; void __attribute__ ((__noinline__, __regparm__(2))) coro_transfer (coro_context *prev, coro_context *next); # define coro_destroy(ctx) (void *)(ctx) #elif CORO_PTHREAD # include extern pthread_mutex_t coro_mutex; struct coro_context { pthread_cond_t cv; pthread_t id; }; void coro_transfer (coro_context *prev, coro_context *next); void coro_destroy (coro_context *ctx); #elif CORO_FIBER struct coro_context { void *fiber; /* only used for initialisation */ coro_func coro; void *arg; }; void coro_transfer (coro_context *prev, coro_context *next); void coro_destroy (coro_context *ctx); #endif #if __cplusplus } #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/0000755000372000001440000000000013207540420020770 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/include/nuitka/threading.h0000644000372000001440000000411313122472300023102 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_THREADING_H__ #define __NUITKA_THREADING_H__ #if PYTHON_VERSION < 300 // We share this with CPython bytecode main loop. PyAPI_DATA(volatile int) _Py_Ticker; #else extern volatile int _Py_Ticker; #define _Py_CheckInterval 20 #endif NUITKA_MAY_BE_UNUSED static inline bool CONSIDER_THREADING( void ) { // Decrease ticker if ( --_Py_Ticker < 0 ) { _Py_Ticker = _Py_CheckInterval; int res = Py_MakePendingCalls(); if (unlikely( res < 0 && ERROR_OCCURRED() )) { return false; } PyThreadState *tstate = PyThreadState_GET(); assert( tstate ); if ( PyEval_ThreadsInitialized() ) { // Release and acquire the GIL, it's very inefficient, because we // don't even know if it makes sense to do it. A controlling thread // should be used to determine if it's needed at all. PyEval_SaveThread(); PyEval_AcquireThread( tstate ); } if (unlikely( tstate->async_exc != NULL )) { PyObject *async_exc = tstate->async_exc; tstate->async_exc = NULL; Py_INCREF( async_exc ); RESTORE_ERROR_OCCURRED( async_exc, NULL, NULL ); return false; } } return true; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helpers.h0000644000372000001440000004700713134660221022614 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPERS_H__ #define __NUITKA_HELPERS_H__ #define _DEBUG_FRAME 0 #define _DEBUG_REFRAME 0 #define _DEBUG_EXCEPTIONS 0 #define _DEBUG_COROUTINE 0 #define _DEBUG_ASYNCGEN 0 extern PyObject *const_tuple_empty; extern PyObject *const_str_plain___dict__; extern PyObject *const_str_plain___class__; extern PyObject *const_str_plain___enter__; extern PyObject *const_str_plain___exit__; extern PyObject *const_int_0; extern PyObject *const_int_pos_1; // From CPython, to allow us quick access to the dictionary of an module, the // structure is normally private, but we need it for quick access to the module // dictionary. typedef struct { PyObject_HEAD PyObject *md_dict; } PyModuleObject; // Most fundamental, because we use it for debugging in everything else. #include "nuitka/helper/printing.h" // Helper to check that an object is valid and has positive reference count. #define CHECK_OBJECT( value ) ( assert( value != NULL ), assert( Py_REFCNT( value ) > 0 ) ); #include "nuitka/exceptions.h" // For use with "--trace-execution", code can make outputs. Otherwise they // are just like comments. #include "nuitka/tracing.h" // For checking values if they changed or not. #ifndef __NUITKA_NO_ASSERT__ extern Py_hash_t DEEP_HASH( PyObject *value ); #endif // For profiling of Nuitka compiled binaries #if _NUITKA_PROFILE extern void startProfiling( void ); extern void stopProfiling( void ); #endif #include "nuitka/helper/boolean.h" #include "nuitka/helper/dictionaries.h" #if PYTHON_VERSION >= 300 static char *_PyUnicode_AS_STRING( PyObject *unicode ) { #if PYTHON_VERSION < 330 PyObject *bytes = _PyUnicode_AsDefaultEncodedString( unicode, NULL ); if (unlikely( bytes == NULL )) { return NULL; } return PyBytes_AS_STRING( bytes ); #else return PyUnicode_AsUTF8( unicode ); #endif } #endif #include "nuitka/helper/raising.h" #include "helper/operations.h" #include "nuitka/helper/richcomparisons.h" #include "nuitka/helper/sequences.h" static inline bool Nuitka_Function_Check( PyObject *object ); static inline PyObject *Nuitka_Function_GetName( PyObject *object ); static inline bool Nuitka_Generator_Check( PyObject *object ); static inline PyObject *Nuitka_Generator_GetName( PyObject *object ); #include "nuitka/calling.h" NUITKA_MAY_BE_UNUSED static PyObject *TO_FLOAT( PyObject *value ) { PyObject *result; #if PYTHON_VERSION < 300 if ( PyString_CheckExact( value ) ) { result = PyFloat_FromString( value, NULL ); } #else if ( PyUnicode_CheckExact( value ) ) { result = PyFloat_FromString( value ); } #endif else { result = PyNumber_Float( value ); } if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *TO_COMPLEX( PyObject *real, PyObject *imag ) { // TODO: Very lazy here, we should create the values ourselves, surely a // a lot of optimization can be had that way. if ( real == NULL ) { assert( imag != NULL ); real = const_int_0; } if ( imag == NULL) { PyObject *args[] = { real }; return CALL_FUNCTION_WITH_ARGS1( (PyObject *)&PyComplex_Type, args ); } else { PyObject *args[] = { real, imag }; return CALL_FUNCTION_WITH_ARGS2( (PyObject *)&PyComplex_Type, args ); } } NUITKA_MAY_BE_UNUSED static PyObject *TO_INT2( PyObject *value, PyObject *base ) { // TODO: Need to check if 3.4 is really the first version to do this. #if PYTHON_VERSION < 340 long base_int = PyInt_AsLong( base ); #else Py_ssize_t base_int = PyNumber_AsSsize_t( base, NULL ); #endif if (unlikely( base_int == -1 )) { PyObject *error = GET_ERROR_OCCURRED(); if (likely( error )) { #if PYTHON_VERSION >= 300 if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_OverflowError ) ) { PyErr_Format( PyExc_ValueError, #if PYTHON_VERSION < 324 "int() arg 2 must be >= 2 and <= 36" #else "int() base must be >= 2 and <= 36" #endif ); } #endif return NULL; } } #if PYTHON_VERSION >= 300 if (unlikely( ( base_int != 0 && base_int < 2 ) || base_int > 36 )) { PyErr_Format( PyExc_ValueError, #if PYTHON_VERSION < 324 "int() arg 2 must be >= 2 and <= 36" #else "int() base must be >= 2 and <= 36" #endif ); return NULL; } #endif #if PYTHON_VERSION < 300 if (unlikely( !Nuitka_String_Check( value ) && !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "int() can't convert non-string with explicit base" ); return NULL; } char *value_str = Nuitka_String_AsString( value ); if (unlikely( value_str == NULL )) { return NULL; } PyObject *result = PyInt_FromString( value_str, NULL, base_int ); if (unlikely( result == NULL )) { return NULL; } return result; #else if ( PyUnicode_Check( value ) ) { #if PYTHON_VERSION < 330 char *value_str = Nuitka_String_AsString( value ); if (unlikely( value_str == NULL )) { return NULL; } PyObject *result = PyInt_FromString( value_str, NULL, base_int ); if (unlikely( result == NULL )) { return NULL; } return result; #else return PyLong_FromUnicodeObject( value, (int)base_int ); #endif } else if ( PyBytes_Check( value ) || PyByteArray_Check( value ) ) { // Check for "NUL" as PyLong_FromString has no length parameter, Py_ssize_t size = Py_SIZE( value ); char *value_str; if ( PyByteArray_Check( value ) ) { value_str = PyByteArray_AS_STRING( value ); } else { value_str = PyBytes_AS_STRING( value ); } PyObject *result = NULL; if ( size != 0 && strlen( value_str ) == (size_t)size ) { result = PyInt_FromString( value_str, NULL, (int)base_int ); } if (unlikely( result == NULL )) { PyErr_Format( PyExc_ValueError, "invalid literal for int() with base %d: %R", base_int, value ); return NULL; } return result; } else { PyErr_Format( PyExc_TypeError, "int() can't convert non-string with explicit base" ); return NULL; } #endif } #if PYTHON_VERSION < 300 // Note: Python3 uses TO_INT2 function. NUITKA_MAY_BE_UNUSED static PyObject *TO_LONG2( PyObject *value, PyObject *base ) { long base_int = PyInt_AsLong( base ); if (unlikely( base_int == -1 )) { if (likely( ERROR_OCCURRED() )) { return NULL; } } if (unlikely( !Nuitka_String_Check( value ) && !PyUnicode_Check( value ) )) { PyErr_Format( PyExc_TypeError, "long() can't convert non-string with explicit base" ); return NULL; } char *value_str = Nuitka_String_AsString( value ); if (unlikely( value_str == NULL )) { return NULL; } PyObject *result = PyLong_FromString( value_str, NULL, base_int ); if (unlikely( result == NULL )) { return NULL; } return result; } #endif NUITKA_MAY_BE_UNUSED static PyObject *TO_BOOL( PyObject *value ) { int res = CHECK_IF_TRUE( value ); if (unlikely( res == -1 )) return NULL; return BOOL_FROM( res != 0 ); } NUITKA_MAY_BE_UNUSED static PyObject *TO_UNICODE3( PyObject *value, PyObject *encoding, PyObject *errors ) { CHECK_OBJECT( value ); if ( encoding ) CHECK_OBJECT( encoding ); if ( errors ) CHECK_OBJECT( errors ); char *encoding_str; if ( encoding == NULL ) { encoding_str = NULL; } else if ( Nuitka_String_Check( encoding ) ) { encoding_str = Nuitka_String_AsString_Unchecked( encoding ); } #if PYTHON_VERSION < 300 else if ( PyUnicode_Check( encoding ) ) { PyObject *uarg2 = _PyUnicode_AsDefaultEncodedString( encoding, NULL ); CHECK_OBJECT( uarg2 ); encoding_str = Nuitka_String_AsString_Unchecked( uarg2 ); } #endif else { PyErr_Format( PyExc_TypeError, "unicode() argument 2 must be string, not %s", Py_TYPE( encoding )->tp_name ); return NULL; } char *errors_str; if ( errors == NULL ) { errors_str = NULL; } else if ( Nuitka_String_Check( errors ) ) { errors_str = Nuitka_String_AsString_Unchecked( errors ); } #if PYTHON_VERSION < 300 else if ( PyUnicode_Check( errors ) ) { PyObject *uarg3 = _PyUnicode_AsDefaultEncodedString( errors, NULL ); CHECK_OBJECT( uarg3 ); errors_str = Nuitka_String_AsString_Unchecked( uarg3 ); } #endif else { PyErr_Format( PyExc_TypeError, "unicode() argument 3 must be string, not %s", Py_TYPE( errors )->tp_name ); return NULL; } PyObject *result = PyUnicode_FromEncodedObject( value, encoding_str, errors_str ); if (unlikely( result == NULL )) { return NULL; } assert( PyUnicode_Check( result ) ); return result; } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_VARS( PyObject *source ) { CHECK_OBJECT( source ); PyObject *result = PyObject_GetAttr( source, const_str_plain___dict__ ); if (unlikely( result == NULL )) { PyErr_Format( PyExc_TypeError, "vars() argument must have __dict__ attribute" ); return NULL; } return result; } #include "nuitka/helper/subscripts.h" #include "nuitka/helper/attributes.h" #include "nuitka/helper/iterators.h" #include "nuitka/helper/slices.h" #include "nuitka/helper/rangeobjects.h" #include "nuitka/helper/lists.h" #include "nuitka/helper/bytearrays.h" #include "nuitka/builtins.h" #include "nuitka/allocator.h" // Compile source code given, pretending the file name was given. #if PYTHON_VERSION < 300 extern PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit ); #else extern PyObject *COMPILE_CODE( PyObject *source_code, PyObject *file_name, PyObject *mode, PyObject *flags, PyObject *dont_inherit, PyObject *optimize ); #endif // For quicker built-in open() functionality. #if PYTHON_VERSION < 300 extern PyObject *BUILTIN_OPEN( PyObject *file_name, PyObject *mode, PyObject *buffering ); #else extern PyObject *BUILTIN_OPEN( PyObject *file_name, PyObject *mode, PyObject *buffering, PyObject *encoding, PyObject *errors, PyObject *newline, PyObject *closefd, PyObject *opener ); #endif // For quicker built-in chr() functionality. extern PyObject *BUILTIN_CHR( PyObject *value ); // For quicker built-in ord() functionality. extern PyObject *BUILTIN_ORD( PyObject *value ); // For quicker built-in bin() functionality. extern PyObject *BUILTIN_BIN( PyObject *value ); // For quicker built-in oct() functionality. extern PyObject *BUILTIN_OCT( PyObject *value ); // For quicker built-in hex() functionality. extern PyObject *BUILTIN_HEX( PyObject *value ); // For quicker callable() functionality. extern PyObject *BUILTIN_CALLABLE( PyObject *value ); // For quicker iter() functionality if 2 arguments arg given. extern PyObject *BUILTIN_ITER2( PyObject *callable, PyObject *sentinel ); // For quicker type() functionality if 1 argument is given. extern PyObject *BUILTIN_TYPE1( PyObject *arg ); // For quicker type() functionality if 3 arguments are given (to build a new // type). extern PyObject *BUILTIN_TYPE3( PyObject *module_name, PyObject *name, PyObject *bases, PyObject *dict ); // For built-in built-in len() functionality. extern PyObject *BUILTIN_LEN( PyObject *boundary ); // For built-in built-in super() functionality. extern PyObject *BUILTIN_SUPER( PyObject *type, PyObject *object ); // For built-in isinstance() functionality. extern PyObject *BUILTIN_ISINSTANCE( PyObject *inst, PyObject *cls ); // The patched isinstance() functionality used for the built-in. extern int Nuitka_IsInstance( PyObject *inst, PyObject *cls ); // For built-in getattr() functionality. extern PyObject *BUILTIN_GETATTR( PyObject *object, PyObject *attribute, PyObject *default_value ); // For built-in setattr() functionality. extern PyObject *BUILTIN_SETATTR( PyObject *object, PyObject *attribute, PyObject *value ); // For built-in bytearray() functionality. extern PyObject *BUILTIN_BYTEARRAY1( PyObject *value ); extern PyObject *BUILTIN_BYTEARRAY3( PyObject *string, PyObject *encoding, PyObject *errors ); // For built-in hash() functionality. extern PyObject *BUILTIN_HASH( PyObject *value ); // For built-in sum() functionality. extern PyObject *BUILTIN_SUM1( PyObject *sequence ); extern PyObject *BUILTIN_SUM2( PyObject *sequence, PyObject *start ); // For built-in bytes() functionality. #if PYTHON_VERSION >= 300 extern PyObject *BUILTIN_BYTES3( PyObject *value, PyObject *encoding, PyObject *errors ); #endif extern PyObject *const_str_plain___builtins__; // For built-in eval() functionality, works on byte compiled code already. extern PyObject *EVAL_CODE( PyObject *code, PyObject *globals, PyObject *locals ); // For built-in format() functionality. extern PyObject *BUILTIN_FORMAT( PyObject *value, PyObject *format_spec ); // For built-in staticmethod() functionality. extern PyObject *BUILTIN_STATICMETHOD( PyObject *function ); // For built-in classmethod() functionality. extern PyObject *BUILTIN_CLASSMETHOD( PyObject *function ); // For built-in divmod() functionality. extern PyObject *BUILTIN_DIVMOD( PyObject *operand1, PyObject *operand2 ); #include "nuitka/importing.h" // For the constant loading: // Call this to initialize all common constants pre-main. extern void createGlobalConstants( void ); // Call this to check of common constants are still intact. #ifndef __NUITKA_NO_ASSERT__ extern void checkGlobalConstants( void ); #endif // Unstreaming constants from a blob. #include "nuitka/constants_blob.h" extern void UNSTREAM_INIT( void ); extern PyObject *UNSTREAM_STRING( unsigned char const *buffer, Py_ssize_t size, bool intern ); extern PyObject *UNSTREAM_CHAR( unsigned char value, bool intern ); #if PYTHON_VERSION < 300 extern PyObject *UNSTREAM_UNICODE( unsigned char const *buffer, Py_ssize_t size ); #else extern PyObject *UNSTREAM_BYTES( unsigned char const *buffer, Py_ssize_t size ); #endif extern PyObject *UNSTREAM_FLOAT( unsigned char const *buffer ); extern PyObject *UNSTREAM_BYTEARRAY( unsigned char const *buffer, Py_ssize_t size ); // Performance enhancements to Python types. extern void enhancePythonTypes( void ); // Setup meta path based loader if any. extern void setupMetaPathBasedLoader( void ); // Parse the command line parameters and provide it to "sys" built-in module. #if PYTHON_VERSION >= 300 typedef wchar_t ** argv_type_t; extern argv_type_t convertCommandLineParameters( int argc, char **argv ); #else typedef char ** argv_type_t; #endif extern bool setCommandLineParameters( int argc, argv_type_t argv, bool initial ); // Replace built-in functions with ones that accept compiled types too. extern void patchBuiltinModule( void ); /* Replace inspect functions with ones that handle compiles types too. */ #if PYTHON_VERSION >= 300 extern void patchInspectModule( void ); #endif // Replace type comparison with one that accepts compiled types too, will work // for "==" and "!=", but not for "is" checks. extern void patchTypeComparison( void ); // Patch the CPython type for tracebacks and make it use a freelist mechanism // to be slightly faster for exception control flows. extern void patchTracebackDealloc( void ); #if PYTHON_VERSION < 300 // Initialize value for "tp_compare" default. extern void _initSlotCompare( void ); #endif #if PYTHON_VERSION >= 300 NUITKA_MAY_BE_UNUSED static PyObject *SELECT_METACLASS( PyObject *metaclass, PyObject *bases ) { CHECK_OBJECT( metaclass ); CHECK_OBJECT( bases ); if (likely( PyType_Check( metaclass ) )) { // Determine the proper metatype Py_ssize_t nbases = PyTuple_GET_SIZE( bases ); PyTypeObject *winner = (PyTypeObject *)metaclass; for ( int i = 0; i < nbases; i++ ) { PyObject *base = PyTuple_GET_ITEM( bases, i ); PyTypeObject *base_type = Py_TYPE( base ); if ( PyType_IsSubtype( winner, base_type ) ) { // Ignore if current winner is already a subtype. continue; } else if ( PyType_IsSubtype( base_type, winner ) ) { // Use if, if it's a subtype of the current winner. winner = base_type; continue; } else { PyErr_Format( PyExc_TypeError, "metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases" ); return NULL; } } if (unlikely( winner == NULL )) { return NULL; } Py_INCREF( winner ); return (PyObject *)winner; } else { Py_INCREF( metaclass ); return metaclass; } } #endif extern PyObject *const_str_plain___name__; NUITKA_MAY_BE_UNUSED static PyObject *MODULE_NAME( PyObject *module ) { assert( PyModule_Check( module ) ); PyObject *module_dict = ((PyModuleObject *)module)->md_dict; return PyDict_GetItem( module_dict, const_str_plain___name__ ); } #if defined(_NUITKA_STANDALONE) || _NUITKA_FROZEN > 0 // Get the binary directory, translated to UTF8 or usable as a native path, // e.g. ANSI on Windows. extern char *getBinaryDirectoryUTF8Encoded(); extern char *getBinaryDirectoryHostEncoded(); #endif #if _NUITKA_STANDALONE extern void setEarlyFrozenModulesFileAttribute( void ); #endif /* For making paths relative to where we got loaded from. Do not provide any * absolute paths as relative value, this is not as capable as "os.path.join", * instead just works on strings. */ extern PyObject *MAKE_RELATIVE_PATH( PyObject *relative ); #include NUITKA_MAY_BE_UNUSED static PyObject *MAKE_TUPLE( PyObject **elements, Py_ssize_t size ) { PyObject *result = PyTuple_New( size ); for( Py_ssize_t i = 0; i < size; i++ ) { PyObject *item = elements[i]; Py_INCREF( item ); PyTuple_SET_ITEM( result, i, item ); } return result; } // Make a deep copy of an object. extern PyObject *DEEP_COPY( PyObject *value ); // Force a garbage collection, for debugging purposes. NUITKA_MAY_BE_UNUSED static void forceGC() { PyObject_CallObject(PyObject_GetAttrString(PyImport_ImportModule("gc"), "collect"), NULL ); } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_coroutine.h0000644000372000001440000001512413207537242025037 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_COROUTINE_H__ #define __NUITKA_COMPILED_COROUTINE_H__ // Compiled coroutine type. // Another cornerstone of the integration into CPython. Try to behave as well as // normal coroutine objects do or even better. #if PYTHON_VERSION >= 350 // The Nuitka_CoroutineObject is the storage associated with a compiled // coroutine object instance of which there can be many for each code. struct Nuitka_CoroutineObject { PyObject_VAR_HEAD PyObject *m_name; PyObject *m_qualname; PyObject *m_yieldfrom; Fiber m_yielder_context; Fiber m_caller_context; // Weak references are supported for coroutine objects in CPython. PyObject *m_weakrefs; int m_running; int m_awaiting; void *m_code; PyObject *m_yielded; PyObject *m_returned; PyObject *m_exception_type, *m_exception_value; PyTracebackObject *m_exception_tb; struct Nuitka_FrameObject *m_frame; PyCodeObject *m_code_object; // Was it ever used, is it still running, or already finished. Generator_Status m_status; // Closure variables given, if any, we reference cells here. Py_ssize_t m_closure_given; struct Nuitka_CellObject *m_closure[1]; }; extern PyTypeObject Nuitka_Coroutine_Type; typedef void (*coroutine_code)( struct Nuitka_CoroutineObject * ); extern PyObject *Nuitka_Coroutine_New( coroutine_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, Py_ssize_t closure_given ); static inline bool Nuitka_Coroutine_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Coroutine_Type; } struct Nuitka_CoroutineWrapperObject { PyObject_HEAD struct Nuitka_CoroutineObject *m_coroutine; }; extern PyTypeObject Nuitka_CoroutineWrapper_Type; extern PyObject *COROUTINE_AWAIT( struct Nuitka_CoroutineObject *coroutine, PyObject *awaitable ); extern PyObject *COROUTINE_AWAIT_IN_HANDLER( struct Nuitka_CoroutineObject *coroutine, PyObject *awaitable ); extern PyObject *COROUTINE_ASYNC_MAKE_ITERATOR( struct Nuitka_CoroutineObject *coroutine, PyObject *value ); extern PyObject *COROUTINE_ASYNC_ITERATOR_NEXT( struct Nuitka_CoroutineObject *coroutine, PyObject *value ); static inline PyObject *COROUTINE_YIELD( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { CHECK_OBJECT( value ); coroutine->m_yielded = value; Nuitka_Frame_MarkAsNotExecuting( coroutine->m_frame ); // Return to the calling context. swapFiber( &coroutine->m_yielder_context, &coroutine->m_caller_context ); Nuitka_Frame_MarkAsExecuting( coroutine->m_frame ); // Check for thrown exception. if (unlikely( coroutine->m_exception_type )) { RESTORE_ERROR_OCCURRED( coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } CHECK_OBJECT( coroutine->m_yielded ); return coroutine->m_yielded; } static inline PyObject *COROUTINE_YIELD_IN_HANDLER( struct Nuitka_CoroutineObject *coroutine, PyObject *value ) { CHECK_OBJECT( value ); coroutine->m_yielded = value; /* When yielding from an exception handler in Python3, the exception * preserved to the frame is restore, while the current one is put there. */ PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD exit:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; Nuitka_Frame_MarkAsNotExecuting( coroutine->m_frame ); // Return to the calling context. swapFiber( &coroutine->m_yielder_context, &coroutine->m_caller_context ); Nuitka_Frame_MarkAsExecuting( coroutine->m_frame ); // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD return:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; // Check for thrown exception. if (unlikely( coroutine->m_exception_type )) { RESTORE_ERROR_OCCURRED( coroutine->m_exception_type, coroutine->m_exception_value, coroutine->m_exception_tb ); coroutine->m_exception_type = NULL; coroutine->m_exception_value = NULL; coroutine->m_exception_tb = NULL; return NULL; } return coroutine->m_yielded; } #if PYTHON_VERSION >= 360 extern PyObject *PyCoro_GetAwaitableIter( PyObject *value ); #endif #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_generator.h0000644000372000001440000001675713207537242025033 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_GENERATOR_H__ #define __NUITKA_COMPILED_GENERATOR_H__ #include "Python.h" #include "methodobject.h" #include "frameobject.h" // Compiled generator function type. // Another cornerstone of the integration into CPython. Try to behave as well as // normal generator objects do or even better. #include "fibers.h" // Status of the generator object. #ifdef __cplusplus enum Generator_Status { status_Unused, // Not used so far status_Running, // Running, used but didn't stop yet status_Finished // Stopped, no more values to come }; #else typedef int Generator_Status; static const int status_Unused = 0; static const int status_Running = 1; static const int status_Finished = 2; #endif // The Nuitka_GeneratorObject is the storage associated with a compiled // generator object instance of which there can be many for each code. struct Nuitka_GeneratorObject { PyObject_VAR_HEAD PyObject *m_name; PyObject *m_module; #if PYTHON_VERSION >= 350 PyObject *m_qualname; PyObject *m_yieldfrom; #endif // Weak references are supported for generator objects in CPython. PyObject *m_weakrefs; int m_running; void *m_code; PyObject *m_exception_type, *m_exception_value; PyTracebackObject *m_exception_tb; struct Nuitka_FrameObject *m_frame; PyCodeObject *m_code_object; // Was it ever used, is it still running, or already finished. Generator_Status m_status; #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO int m_yield_return_index; #else Fiber m_yielder_context; Fiber m_caller_context; // The yielded value, NULL is case of exception. PyObject *m_yielded; #endif /* Closure variables given, if any, we reference cells here. The last * part is dynamically allocated, the array size differs per generator. */ Py_ssize_t m_closure_given; struct Nuitka_CellObject *m_closure[1]; }; extern PyTypeObject Nuitka_Generator_Type; #if _NUITKA_EXPERIMENTAL_GENERATOR_GOTO typedef PyObject *(*generator_code)( struct Nuitka_GeneratorObject *, PyObject * ); #else typedef void (*generator_code)( struct Nuitka_GeneratorObject * ); #endif #if PYTHON_VERSION < 350 extern PyObject *Nuitka_Generator_New( generator_code code, PyObject *module, PyObject *name, PyCodeObject *code_object, Py_ssize_t closure_given ); #else extern PyObject *Nuitka_Generator_New( generator_code code, PyObject *module, PyObject *name, PyObject *qualname, PyCodeObject *code_object, Py_ssize_t closure_given ); #endif extern PyObject *Nuitka_Generator_qiter( struct Nuitka_GeneratorObject *generator, bool *finished ); static inline bool Nuitka_Generator_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Generator_Type; } static inline PyObject *Nuitka_Generator_GetName( PyObject *object ) { return ((struct Nuitka_GeneratorObject *)object)->m_name; } #ifndef _NUITKA_EXPERIMENTAL_GENERATOR_GOTO static inline PyObject *GENERATOR_YIELD( struct Nuitka_GeneratorObject *generator, PyObject *value ) { CHECK_OBJECT( value ); generator->m_yielded = value; Nuitka_Frame_MarkAsNotExecuting( generator->m_frame ); // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); Nuitka_Frame_MarkAsExecuting( generator->m_frame ); // Check for thrown exception. if (unlikely( generator->m_exception_type )) { RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } CHECK_OBJECT( generator->m_yielded ); return generator->m_yielded; } #if PYTHON_VERSION >= 300 static inline PyObject *GENERATOR_YIELD_IN_HANDLER( struct Nuitka_GeneratorObject *generator, PyObject *value ) { CHECK_OBJECT( value ); generator->m_yielded = value; /* When yielding from an exception handler in Python3, the exception * preserved to the frame is restore, while the current one is put there. */ PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD exit:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; Nuitka_Frame_MarkAsNotExecuting( generator->m_frame ); // Return to the calling context. swapFiber( &generator->m_yielder_context, &generator->m_caller_context ); Nuitka_Frame_MarkAsExecuting( generator->m_frame ); // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD return:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; // Check for thrown exception. if (unlikely( generator->m_exception_type )) { RESTORE_ERROR_OCCURRED( generator->m_exception_type, generator->m_exception_value, generator->m_exception_tb ); generator->m_exception_type = NULL; generator->m_exception_value = NULL; generator->m_exception_tb = NULL; return NULL; } return generator->m_yielded; } #endif #if PYTHON_VERSION >= 330 extern PyObject *GENERATOR_YIELD_FROM( struct Nuitka_GeneratorObject *generator, PyObject *target ); extern PyObject *GENERATOR_YIELD_FROM_IN_HANDLER( struct Nuitka_GeneratorObject *generator, PyObject *target ); #endif #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/freelists.h0000644000372000001440000001304013122472300023134 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_FREELISTS_H__ #define __NUITKA_FREELISTS_H__ #define allocateFromFreeList( free_list, object_type, type_type, size ) \ if ( free_list != NULL ) \ { \ result = free_list; \ free_list = *((object_type **)free_list); \ free_list ## _count -= 1; \ assert( free_list ## _count >= 0 ); \ \ if ( Py_SIZE( result ) < size ) \ { \ result = PyObject_GC_Resize( object_type, result, size ); \ assert( result != NULL ); \ } \ \ _Py_NewReference( (PyObject *)result ); \ } \ else \ { \ result = (object_type *)Nuitka_GC_NewVar( \ &type_type, \ size \ ); \ } \ CHECK_OBJECT( result ); #define allocateFromFreeListFixed( free_list, object_type, type_type ) \ if ( free_list != NULL ) \ { \ result = free_list; \ free_list = *((object_type **)free_list); \ free_list ## _count -= 1; \ assert( free_list ## _count >= 0 ); \ \ _Py_NewReference( (PyObject *)result ); \ } \ else \ { \ result = (object_type *)PyObject_GC_New( \ object_type, \ &type_type \ ); \ } \ CHECK_OBJECT( result ); #define releaseToFreeList( free_list, object, max_free_list_count ) \ if ( free_list != NULL ) \ { \ if ( free_list ## _count > max_free_list_count ) \ { \ PyObject_GC_Del( object ); \ } \ else \ { \ *((void **)object) = (void *)free_list; \ free_list = object; \ \ free_list ## _count += 1; \ } \ } \ else \ { \ free_list = object; \ *((void **)object) = NULL; \ \ assert( free_list ## _count == 0 ); \ \ free_list ## _count += 1; \ } \ #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/calling.h0000644000372000001440000000607513207540035022564 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_CALLING_H__ #define __NUITKA_CALLING_H__ #include "__helpers.h" extern PyObject *const_tuple_empty; NUITKA_MAY_BE_UNUSED static PyObject *CALL_FUNCTION( PyObject *function_object, PyObject *positional_args, PyObject *named_args ) { assert( !ERROR_OCCURRED() ); CHECK_OBJECT( function_object ); CHECK_OBJECT( positional_args ); assert( named_args == NULL || Py_REFCNT( named_args ) > 0 ); ternaryfunc call_slot = Py_TYPE( function_object )->tp_call; if (unlikely( call_slot == NULL )) { PyErr_Format( PyExc_TypeError, "'%s' object is not callable", function_object->ob_type->tp_name ); return NULL; } if (unlikely( Py_EnterRecursiveCall( (char *)" while calling a Python object") )) { return NULL; } PyObject *result = (*call_slot)( function_object, positional_args, named_args ); Py_LeaveRecursiveCall(); if ( result == NULL ) { if (unlikely( !ERROR_OCCURRED() )) { PyErr_Format( PyExc_SystemError, "NULL result without error in CALL_FUNCTION" ); } return NULL; } else { // Some buggy C functions do this, and Nuitka inner workings can get // upset from it. DROP_ERROR_OCCURRED(); return result; } } // Function call variant with no arguments provided at all. extern PyObject *CALL_FUNCTION_NO_ARGS( PyObject *called ); // Function call variants with positional arguments tuple. NUITKA_MAY_BE_UNUSED static PyObject *CALL_FUNCTION_WITH_POSARGS( PyObject *function_object, PyObject *positional_args ) { return CALL_FUNCTION( function_object, positional_args, NULL ); } // Method call variants with positional arguments tuple. extern PyObject *CALL_METHOD_WITH_POSARGS( PyObject *source, PyObject *attr_name, PyObject *positional_args ); NUITKA_MAY_BE_UNUSED static PyObject *CALL_FUNCTION_WITH_KEYARGS( PyObject *function_object, PyObject *named_args ) { return CALL_FUNCTION( function_object, const_tuple_empty, named_args ); } // Method call variant with no arguments provided at all. extern PyObject *CALL_METHOD_NO_ARGS( PyObject *source, PyObject *attribute ); #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/constants_blob.h0000644000372000001440000000256313112214770024162 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_CONSTANTS_BLOB_H__ #define __NUITKA_CONSTANTS_BLOB_H__ /* There are multiple ways, the constants binary is accessed, and its * definition depends on how that is done. * * It could be a Windows resource, then it must be a pointer. If it's defined * externally in a C file, or at link time with "ld", it must be an array. This * hides these facts. */ #if defined(_NUITKA_CONSTANTS_FROM_RESOURCE) extern const unsigned char* constant_bin; #else #ifdef __cplusplus extern "C" const unsigned char constant_bin[]; #else const unsigned char constant_bin[0]; #endif #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/importing.h0000644000372000001440000000477613122472300023164 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_IMPORTING_H__ #define __NUITKA_IMPORTING_H__ /* These are for the built-in import. * * They call the real thing with varying amount of arguments. For keyword * calls using default values, the _KW helper is used. * */ extern PyObject *IMPORT_MODULE1( PyObject *module_name ); extern PyObject *IMPORT_MODULE2( PyObject *module_name, PyObject *globals ); extern PyObject *IMPORT_MODULE3( PyObject *module_name, PyObject *globals, PyObject *locals ); extern PyObject *IMPORT_MODULE4( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items ); extern PyObject *IMPORT_MODULE5( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items, PyObject *level ); extern PyObject *IMPORT_MODULE_KW( PyObject *module_name, PyObject *globals, PyObject *locals, PyObject *import_items, PyObject *level ); extern bool IMPORT_MODULE_STAR( PyObject *target, bool is_module, PyObject *module ); extern PyObject *IMPORT_EMBEDDED_MODULE( PyObject *module_name, char const *name ); NUITKA_MAY_BE_UNUSED static PyObject *IMPORT_NAME( PyObject *module, PyObject *import_name ) { CHECK_OBJECT( module ); CHECK_OBJECT( import_name ); PyObject *result = PyObject_GetAttr( module, import_name ); if (unlikely( result == NULL )) { if ( EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_AttributeError ) ) { #if PYTHON_VERSION >= 340 || !defined(_NUITKA_FULL_COMPAT) PyErr_Format( PyExc_ImportError, "cannot import name '%s'", Nuitka_String_AsString( import_name )); #else PyErr_Format( PyExc_ImportError, "cannot import name %s", Nuitka_String_AsString( import_name )); #endif } return NULL; } return result; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/prelude.h0000644000372000001440000001644013207537242022616 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_PRELUDE_H__ #define __NUITKA_PRELUDE_H__ #include "build_definitions.h" #ifdef __NUITKA_NO_ASSERT__ #define NDEBUG #endif /* Include the CPython version numbers, and define our own take of what version * numbers should be. */ #include "patchlevel.h" #define PYTHON_VERSION (PY_MAJOR_VERSION*100+PY_MINOR_VERSION*10+PY_MICRO_VERSION) /* This is needed or else we can't create modules name "proc" or "func". For * Python3, the name collision can't happen, so we can limit it to Python2. */ #if PYTHON_VERSION < 300 #define initproc python_initproc #define initfunc python_initfunc #define initstate system_initstate #endif /* Include the relevant Python C-API header files. */ #include "Python.h" #include "methodobject.h" #include "frameobject.h" #include "pydebug.h" #include "marshal.h" /* The bool type. From Python2 header or self defined for Python3. */ #if PYTHON_VERSION < 300 #include "asdl.h" #else #ifndef __cplusplus typedef enum {false, true} bool; #endif #endif /* See above. */ #if PYTHON_VERSION < 300 #undef initproc #undef initfunc #undef initstate #endif /* Include the C header files most often used. */ #include /* Using "_alloca" extension due to MSVC restrictions for array variables * on the local stack. */ #ifdef _MSC_VER #include #endif /* An idea I first saw used with Cython, hint the compiler about branches * that are more or less likely to be taken. And hint the compiler about * things that we assume to be normally true. If other compilers can do * similar, I would be grateful for instructions. */ #ifdef __GNUC__ #define likely(x) __builtin_expect(!!(x), 1) #define unlikely(x) __builtin_expect(!!(x), 0) #else #define likely(x) (x) #define unlikely(x) (x) #endif /* A way to not give warnings about things that are declared, but might not * be used like in-line helper functions in headers or static per module * variables from headers. */ #ifdef __GNUC__ #define NUITKA_MAY_BE_UNUSED __attribute__((__unused__)) #else #define NUITKA_MAY_BE_UNUSED #endif /* A way to indicate that a specific function won't return, so the C compiler * can create better code. */ #ifdef __GNUC__ #define NUITKA_NO_RETURN __attribute__((__noreturn__)) #elif defined(_MSC_VER) #define NUITKA_NO_RETURN __declspec(noreturn) #else #define NUITKA_NO_RETURN #endif /* This is used to indicate code control flows we know cannot happen. */ #define NUITKA_CANNOT_GET_HERE(NAME) assert(false && #NAME); abort(); #ifdef __GNUC__ #define NUITKA_FORCE_INLINE __attribute__((always_inline)) #else #define NUITKA_FORCE_INLINE #endif /* Python3 removed PyInt instead of renaming PyLong, and PyObject_Str instead * of renaming PyObject_Unicode. Define this to be easily portable. */ #if PYTHON_VERSION >= 300 #define PyInt_FromString PyLong_FromString #define PyInt_FromLong PyLong_FromLong #define PyInt_AsLong PyLong_AsLong #define PyInt_FromSsize_t PyLong_FromSsize_t #define PyNumber_Int PyNumber_Long #define PyObject_Unicode PyObject_Str #endif /* String handling that uses the proper version of strings for Python3 or not, * which makes it easier to write portable code. */ #if PYTHON_VERSION < 300 #define Nuitka_String_AsString PyString_AsString #define Nuitka_String_AsString_Unchecked PyString_AS_STRING #define Nuitka_String_Check PyString_Check #define Nuitka_String_CheckExact PyString_CheckExact #define Nuitka_StringObject PyStringObject #define Nuitka_StringIntern PyString_InternInPlace #else #define Nuitka_String_AsString _PyUnicode_AsString /* Note: There seems to be no variant that does it without checks, so we rolled * our own. */ #define Nuitka_String_AsString_Unchecked _PyUnicode_AS_STRING #define Nuitka_String_Check PyUnicode_Check #define Nuitka_String_CheckExact PyUnicode_CheckExact #define Nuitka_StringObject PyUnicodeObject #define Nuitka_StringIntern PyUnicode_InternInPlace #endif /* With the idea to reduce the amount of exported symbols in the DLLs, make it * clear that the module "init" function should of course be exported, but not * for executable, where we call it ourselves from the main code. */ #if defined( _NUITKA_EXE ) #if PYTHON_VERSION < 300 #define NUITKA_MODULE_INIT_FUNCTION void #else #define NUITKA_MODULE_INIT_FUNCTION PyObject * #endif #elif defined( __GNUC__ ) #if PYTHON_VERSION < 300 #define NUITKA_MODULE_INIT_FUNCTION PyMODINIT_FUNC __attribute__(( visibility( "default" ))) #else #ifdef __cplusplus #define NUITKA_MODULE_INIT_FUNCTION extern "C" __attribute__(( visibility( "default" ))) PyObject * #else #define NUITKA_MODULE_INIT_FUNCTION __attribute__(( visibility( "default" ))) PyObject * #endif #endif #else #define NUITKA_MODULE_INIT_FUNCTION PyMODINIT_FUNC #endif /* The name of the entry point for DLLs changed between CPython versions, and * this is here to hide that. */ #if PYTHON_VERSION < 300 #define MOD_INIT_NAME( name ) init##name #define MOD_INIT_DECL( name ) NUITKA_MODULE_INIT_FUNCTION init##name( void ) #define MOD_RETURN_VALUE( value ) #else #define MOD_INIT_NAME( name ) PyInit_##name #define MOD_INIT_DECL( name ) NUITKA_MODULE_INIT_FUNCTION PyInit_##name( void ) #define MOD_RETURN_VALUE( value ) value #endif #if PYTHON_VERSION < 300 typedef long Py_hash_t; #endif /* These two express if a directly called function should be exported (C level) * or if it can be local to the file. */ #define NUITKA_CROSS_MODULE #define NUITKA_LOCAL_MODULE static /* Due to ABI issues, it seems that on Windows the symbols used by * "_PyObject_GC_TRACK" are not exported and we need to use a function that does * it instead. */ #if defined( _WIN32 ) #define Nuitka_GC_Track PyObject_GC_Track #define Nuitka_GC_UnTrack PyObject_GC_UnTrack #else #define Nuitka_GC_Track _PyObject_GC_TRACK #define Nuitka_GC_UnTrack _PyObject_GC_UNTRACK #endif #if _NUITKA_EXPERIMENTAL_FAST_THREAD_GET && PYTHON_VERSION >= 300 // We are careful, access without locking under the assumption that we hold // the GIL over uses of this or the same thread continues to execute code of // ours. #undef PyThreadState_GET extern PyThreadState *_PyThreadState_Current; # define PyThreadState_GET() (_PyThreadState_Current) #endif #include "nuitka/helpers.h" #include "nuitka/compiled_frame.h" #include "nuitka/compiled_cell.h" #include "nuitka/compiled_function.h" /* Sentinel PyObject to be used for all our call iterator endings. */ extern PyObject *_sentinel_value; #include "nuitka/compiled_generator.h" #include "nuitka/compiled_method.h" #if PYTHON_VERSION >= 350 #include "nuitka/compiled_coroutine.h" #endif #if PYTHON_VERSION >= 360 #include "nuitka/compiled_asyncgen.h" #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/unfreezing.h0000644000372000001440000000374313207537242023334 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_UNFREEZING_H__ #define __NUITKA_UNFREEZING_H__ /* Modes for loading modules, can be compiled, external shared library, or * bytecode. */ #define NUITKA_COMPILED_MODULE 0 #define NUITKA_SHLIB_FLAG 1 #define NUITKA_PACKAGE_FLAG 2 #define NUITKA_BYTECODE_FLAG 4 #if PYTHON_VERSION < 300 typedef void (*module_initfunc)( void ); #else typedef PyObject * (*module_initfunc)( void ); #endif struct Nuitka_MetaPathBasedLoaderEntry { /* Full module name, including package name. */ char const *name; /* Entry function if compiled module, otherwise NULL. */ module_initfunc python_initfunc; /* For bytecode modules, start and size inside the constants blob. */ int bytecode_start; int bytecode_size; /* Flags: Indicators if this is compiled, bytecode or shared library. */ int flags; }; /* For embedded modules, register the meta path based loader. Used by main * program/package only. */ extern void registerMetaPathBasedUnfreezer( struct Nuitka_MetaPathBasedLoaderEntry *loader_entries ); /* For use as the "__loader__" attribute of compiled modules in newer Python * versions. */ #if PYTHON_VERSION >= 330 extern PyObject *metapath_based_loader; #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_asyncgen.h0000644000372000001440000001537713207537242024651 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_ASYNCGEN_H__ #define __NUITKA_COMPILED_ASYNCGEN_H__ // Compiled async generator type. // Another cornerstone of the integration into CPython. Try to behave as well as // normal asyncgen objects do or even better. #if PYTHON_VERSION >= 360 // The Nuitka_AsyncgenObject is the storage associated with a compiled // async generator object instance of which there can be many for each code. struct Nuitka_AsyncgenObject { PyObject_VAR_HEAD PyObject *m_name; PyObject *m_qualname; PyObject *m_yieldfrom; Fiber m_yielder_context; Fiber m_caller_context; // Weak references are supported for async generator objects in CPython. PyObject *m_weakrefs; int m_running; int m_awaiting; void *m_code; PyObject *m_yielded; PyObject *m_exception_type, *m_exception_value; PyTracebackObject *m_exception_tb; struct Nuitka_FrameObject *m_frame; PyCodeObject *m_code_object; // Was it ever used, is it still running, or already finished. Generator_Status m_status; // The finalizer associated PyObject *m_finalizer; // The hooks were initialized bool m_hooks_init_done; // It is closed. bool m_closed; // Closure variables given, if any, we reference cells here. Py_ssize_t m_closure_given; struct Nuitka_CellObject *m_closure[1]; }; extern PyTypeObject Nuitka_Asyncgen_Type; typedef void (*asyncgen_code)( struct Nuitka_AsyncgenObject * ); extern PyObject *Nuitka_Asyncgen_New( asyncgen_code code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, Py_ssize_t closure_given ); static inline bool Nuitka_Asyncgen_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Asyncgen_Type; } extern PyObject *ASYNCGEN_AWAIT( struct Nuitka_AsyncgenObject *asyngen, PyObject *awaitable ); extern PyObject *ASYNCGEN_AWAIT_IN_HANDLER( struct Nuitka_AsyncgenObject *asyngen, PyObject *awaitable ); extern PyObject *ASYNCGEN_ASYNC_MAKE_ITERATOR( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ); extern PyObject *ASYNCGEN_ASYNC_ITERATOR_NEXT( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ); extern PyObject *Nuitka_AsyncGenValueWrapperNew( PyObject *value ); static inline PyObject *ASYNCGEN_YIELD( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { CHECK_OBJECT( value ); asyncgen->m_yielded = Nuitka_AsyncGenValueWrapperNew( value ); Py_DECREF( value ); Nuitka_Frame_MarkAsNotExecuting( asyncgen->m_frame ); // Return to the calling context. swapFiber( &asyncgen->m_yielder_context, &asyncgen->m_caller_context ); Nuitka_Frame_MarkAsExecuting( asyncgen->m_frame ); // Check for thrown exception. if (unlikely( asyncgen->m_exception_type )) { RESTORE_ERROR_OCCURRED( asyncgen->m_exception_type, asyncgen->m_exception_value, asyncgen->m_exception_tb ); asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; return NULL; } CHECK_OBJECT( asyncgen->m_yielded ); return asyncgen->m_yielded; } static inline PyObject *ASYNCGEN_YIELD_IN_HANDLER( struct Nuitka_AsyncgenObject *asyncgen, PyObject *value ) { CHECK_OBJECT( value ); asyncgen->m_yielded = Nuitka_AsyncGenValueWrapperNew( value ); Py_DECREF( value ); /* When yielding from an exception handler in Python3, the exception * preserved to the frame is restore, while the current one is put there. */ PyThreadState *thread_state = PyThreadState_GET(); PyObject *saved_exception_type = thread_state->exc_type; PyObject *saved_exception_value = thread_state->exc_value; PyObject *saved_exception_traceback = thread_state->exc_traceback; thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD exit:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; #if _DEBUG_ASYNCGEN PRINT_STRING("ASYNCGEN_YIELD_FROM_HANDLER:"); PRINT_NEW_LINE(); #endif Nuitka_Frame_MarkAsNotExecuting( asyncgen->m_frame ); // Return to the calling context. swapFiber( &asyncgen->m_yielder_context, &asyncgen->m_caller_context ); Nuitka_Frame_MarkAsExecuting( asyncgen->m_frame ); // When returning from yield, the exception of the frame is preserved, and // the one that enters should be there. thread_state = PyThreadState_GET(); saved_exception_type = thread_state->exc_type; saved_exception_value = thread_state->exc_value; saved_exception_traceback = thread_state->exc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("YIELD return:\n"); PRINT_EXCEPTION( thread_state->exc_type, thread_state->exc_value, (PyObject *)thread_state->exc_traceback ); #endif thread_state->exc_type = thread_state->frame->f_exc_type; thread_state->exc_value = thread_state->frame->f_exc_value; thread_state->exc_traceback = thread_state->frame->f_exc_traceback; thread_state->frame->f_exc_type = saved_exception_type; thread_state->frame->f_exc_value = saved_exception_value; thread_state->frame->f_exc_traceback = saved_exception_traceback; // Check for thrown exception. if (unlikely( asyncgen->m_exception_type )) { RESTORE_ERROR_OCCURRED( asyncgen->m_exception_type, asyncgen->m_exception_value, asyncgen->m_exception_tb ); asyncgen->m_exception_type = NULL; asyncgen->m_exception_value = NULL; asyncgen->m_exception_tb = NULL; return NULL; } assert( asyncgen->m_yielded != NULL ); return asyncgen->m_yielded; } #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/builtins.h0000644000372000001440000000465213122472300022776 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_BUILTINS_H__ #define __NUITKA_BUILTINS_H__ #include "__helpers.h" extern PyModuleObject *builtin_module; extern PyDictObject *dict_builtin; #include "nuitka/calling.h" NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_BUILTIN( PyObject *name ) { CHECK_OBJECT( (PyObject *)dict_builtin ); CHECK_OBJECT( name ); assert( Nuitka_String_CheckExact( name ) ); PyObject *result = GET_STRING_DICT_VALUE( dict_builtin, (Nuitka_StringObject *)name ); // This is assumed to not fail, abort if it does. if (unlikely( result == NULL )) { PyErr_PrintEx( 0 ); Py_Exit( 1 ); } return result; } extern void _initBuiltinModule(); #define NUITKA_DECLARE_BUILTIN( name ) extern PyObject *_python_original_builtin_value_##name; #define NUITKA_DEFINE_BUILTIN( name ) extern PyObject *const_str_plain_##name; PyObject *_python_original_builtin_value_##name = NULL; #define NUITKA_ASSIGN_BUILTIN( name ) if ( _python_original_builtin_value_##name == NULL ) _python_original_builtin_value_##name = LOOKUP_BUILTIN( const_str_plain_##name ); #define NUITKA_UPDATE_BUILTIN( name, value ) _python_original_builtin_value_##name = value; #define NUITKA_ACCESS_BUILTIN( name ) ( _python_original_builtin_value_##name ) #ifdef _NUITKA_EXE // Original builtin values, currently only used for assertions. NUITKA_DECLARE_BUILTIN( type ); NUITKA_DECLARE_BUILTIN( len ); NUITKA_DECLARE_BUILTIN( range ); NUITKA_DECLARE_BUILTIN( repr ); NUITKA_DECLARE_BUILTIN( int ); NUITKA_DECLARE_BUILTIN( iter ); #if PYTHON_VERSION < 300 NUITKA_DECLARE_BUILTIN( long ); #endif extern void _initBuiltinOriginalValues(); #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/allocator.h0000644000372000001440000000237113112214770023125 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_ALLOCATOR_H__ #define __NUITKA_ALLOCATOR_H__ NUITKA_MAY_BE_UNUSED static void *Nuitka_GC_NewVar( PyTypeObject *tp, Py_ssize_t nitems ) { assert( nitems >= 0 ); size_t size = _PyObject_VAR_SIZE( tp, nitems ); PyVarObject *op = (PyVarObject *)_PyObject_GC_Malloc( size ); assert( op != NULL ); Py_TYPE( op ) = tp; Py_SIZE( op ) = size; Py_REFCNT( op ) = 1; op = PyObject_INIT_VAR( op, tp, nitems ); return op; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_function.h0000644000372000001440000001144513112214770024650 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_FUNCTION_H__ #define __NUITKA_COMPILED_FUNCTION_H__ #include "Python.h" #include "frameobject.h" // Compiled function type. // The backbone of the integration into CPython. Try to behave as well as normal // functions and built-in functions, or even better. struct Nuitka_FunctionObject; // The actual function code with arguments as an array. typedef PyObject *(*function_impl_code)( struct Nuitka_FunctionObject const *, PyObject ** ); // The Nuitka_FunctionObject is the storage associated with a compiled function // instance of which there can be many for each code. struct Nuitka_FunctionObject { PyObject_VAR_HEAD PyObject *m_name; PyObject *m_module; PyObject *m_doc; PyCodeObject *m_code_object; Py_ssize_t m_args_overall_count; Py_ssize_t m_args_positional_count; Py_ssize_t m_args_keywords_count; bool m_args_simple; Py_ssize_t m_args_star_list_index; Py_ssize_t m_args_star_dict_index; // Same as code_object->co_varnames PyObject **m_varnames; function_impl_code m_c_code; PyObject *m_dict; PyObject *m_weakrefs; // List of defaults, for use in __defaults__ and parameter parsing. PyObject *m_defaults; Py_ssize_t m_defaults_given; #if PYTHON_VERSION >= 300 // List of keyword only defaults, for use in __kwdefaults__ and parameter // parsing. PyObject *m_kwdefaults; // Annotations to the function arguments and return value. PyObject *m_annotations; #endif #if PYTHON_VERSION >= 330 PyObject *m_qualname; #endif // A kind of uuid for the function object, used in comparisons. long m_counter; // Closure taken objects, for use in __closure__ and for accessing it. Py_ssize_t m_closure_given; struct Nuitka_CellObject *m_closure[1]; }; extern PyTypeObject Nuitka_Function_Type; // Make a function with context. #if PYTHON_VERSION < 300 extern struct Nuitka_FunctionObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *module, PyObject *doc, Py_ssize_t closure_given ); #elif PYTHON_VERSION < 330 extern struct Nuitka_FunctionObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, Py_ssize_t closure_given ); #else extern struct Nuitka_FunctionObject *Nuitka_Function_New( function_impl_code c_code, PyObject *name, PyObject *qualname, PyCodeObject *code_object, PyObject *defaults, PyObject *kwdefaults, PyObject *annotations, PyObject *module, PyObject *doc, Py_ssize_t closure_given ); #endif static inline bool Nuitka_Function_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Function_Type; } static inline PyObject *Nuitka_Function_GetName( PyObject *object ) { return ((struct Nuitka_FunctionObject *)object)->m_name; } extern bool parseArgumentsPos( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject **args, Py_ssize_t args_size ); extern bool parseArgumentsMethodPos( struct Nuitka_FunctionObject const *function, PyObject **python_pars, PyObject *object, PyObject **args, Py_ssize_t args_size ); extern PyObject *Nuitka_CallFunctionPosArgsKwArgs( struct Nuitka_FunctionObject const *function, PyObject **args, Py_ssize_t args_size, PyObject *kw ); // These are fast calls of known compiled methods, without an actual object // of that kind. The object is that first argument, "self" or whatever, to // which the function would be bound. extern PyObject *Nuitka_CallMethodFunctionNoArgs( struct Nuitka_FunctionObject const *function, PyObject *object ); extern PyObject *Nuitka_CallMethodFunctionPosArgs( struct Nuitka_FunctionObject const *function, PyObject *object, PyObject **args, Py_ssize_t args_size ); // This is also used by bound compiled methods extern PyObject *Nuitka_CallMethodFunctionPosArgsKwArgs( struct Nuitka_FunctionObject const *function, PyObject *object, PyObject **args, Py_ssize_t args_size, PyObject *kw ); #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_frame.h0000644000372000001440000001721213134660221024113 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_FRAME_H__ #define __NUITKA_COMPILED_FRAME_H__ // Create a frame object for the given code object, frame or module. extern struct Nuitka_FrameObject *MAKE_MODULE_FRAME( PyCodeObject *code, PyObject *module ); extern struct Nuitka_FrameObject *MAKE_FUNCTION_FRAME( PyCodeObject *code, PyObject *module, Py_ssize_t locals_size ); // Create a code object for the given filename and function name #if PYTHON_VERSION < 300 extern PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int flags ); #else extern PyCodeObject *MAKE_CODEOBJ( PyObject *filename, PyObject *function_name, int line, PyObject *argnames, int arg_count, int kw_only_count, int flags ); #endif extern PyTypeObject Nuitka_Frame_Type; static inline bool Nuitka_Frame_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Frame_Type; } struct Nuitka_FrameObject { PyFrameObject m_frame; char const *m_type_description; char m_locals_storage[1]; }; #define MAKE_OR_REUSE_FRAME( cache_identifier, code_identifier, module_identifier, locals_size ) \ if ( isFrameUnusable( cache_identifier ) ) \ { \ Py_XDECREF( cache_identifier ); \ cache_identifier = MAKE_FUNCTION_FRAME( code_identifier, module_identifier, locals_size ); \ } \ assert( ((struct Nuitka_FrameObject *)cache_identifier)->m_type_description == NULL ); \ inline static void assertCodeObject( PyCodeObject *code_object ) { CHECK_OBJECT( (PyObject *)code_object ); } NUITKA_MAY_BE_UNUSED static bool isFrameUnusable( struct Nuitka_FrameObject *frame_object ) { if ( frame_object ) CHECK_OBJECT( (PyObject *)frame_object ); bool result = // Never used. frame_object == NULL || // Still in use Py_REFCNT( frame_object ) > 1 || #if PYTHON_VERSION < 340 // Last used by another thread (TODO: Could just set it when re-using) frame_object->m_frame.f_tstate != PyThreadState_GET() || #endif // Not currently linked. frame_object->m_frame.f_back != NULL; #if _DEBUG_REFRAME if (result && frame_object) { PRINT_STRING("NOT REUSING FRAME:"); PRINT_ITEM((PyObject *)frame_object); PRINT_REFCOUNT( (PyObject *)frame_object ); if ( frame_object->f_back ) PRINT_ITEM( (PyObject *)frame_object->f_back ); PRINT_NEW_LINE(); } #endif return result; } #if _DEBUG_REFRAME static inline void dumpFrameStack( void ) { PyFrameObject *current = PyThreadState_GET()->frame; int total = 0; while( current ) { total++; current = current->f_back; } current = PyThreadState_GET()->frame; puts( ">--------->" ); while( current ) { printf( "Frame stack %d: %s %s\n", total--, PyString_AsString( PyObject_Str( (PyObject *)current ) ), PyString_AsString( PyObject_Str( (PyObject *)current->f_code ) ) ); current = current->f_back; } puts( ">---------<" ); } #endif inline static void assertFrameObject( struct Nuitka_FrameObject *frame_object ) { CHECK_OBJECT( (PyObject *)frame_object ); assertCodeObject( frame_object->m_frame.f_code ); } // Mark frame as currently executed. Starting with Python 3.4 that means it // can or cannot be cleared, or should lead to a generator close. For Python2 // this is a no-op. Using a define to spare the compile from inlining an empty // function. #if PYTHON_VERSION >= 340 static inline void Nuitka_Frame_MarkAsExecuting( struct Nuitka_FrameObject *frame ) { frame->m_frame.f_executing = 1; } #else #define Nuitka_Frame_MarkAsExecuting( frame ) ; #endif #if PYTHON_VERSION >= 340 static inline void Nuitka_Frame_MarkAsNotExecuting( struct Nuitka_FrameObject *frame ) { frame->m_frame.f_executing = 0; } #else #define Nuitka_Frame_MarkAsNotExecuting( frame ) ; #endif // Put frame at the top of the frame stack and mark as executing. NUITKA_MAY_BE_UNUSED inline static void pushFrameStack( struct Nuitka_FrameObject *frame_object ) { // Make sure it's healthy. assertFrameObject( frame_object ); // We don't allow frame objects where this is not true. assert( frame_object->m_frame.f_back == NULL ); // Look at current frame, "old" is the one previously active. PyThreadState *tstate = PyThreadState_GET(); PyFrameObject *old = tstate->frame; #if _DEBUG_FRAME if ( old ) { assertCodeObject( old->f_code ); printf( "Upstacking to frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)old ) ), PyString_AsString( PyObject_Repr( (PyObject *)old->f_code ) ) ); } #endif // No recursion with identical frames allowed, assert against it. assert( old != &frame_object->m_frame ); // Push the new frame as the currently active one. tstate->frame = (PyFrameObject *)frame_object; // Transfer ownership of old frame. if ( old != NULL ) { assertFrameObject( (struct Nuitka_FrameObject *)old ); frame_object->m_frame.f_back = old; } Nuitka_Frame_MarkAsExecuting( frame_object ); Py_INCREF( frame_object ); #if _DEBUG_FRAME printf( "Now at top frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)tstate->frame ) ), PyString_AsString( PyObject_Repr( (PyObject *)tstate->frame->f_code ) ) ); #endif } NUITKA_MAY_BE_UNUSED inline static void popFrameStack( void ) { PyThreadState *tstate = PyThreadState_GET(); PyFrameObject *old = tstate->frame; #if _DEBUG_FRAME printf( "Taking off frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)old ) ), PyString_AsString( PyObject_Repr( (PyObject *)old->f_code ) ) ); #endif // Put previous frame on top. tstate->frame = old->f_back; old->f_back = NULL; Nuitka_Frame_MarkAsNotExecuting( (struct Nuitka_FrameObject *)old ); Py_DECREF( old ); #if _DEBUG_FRAME if ( tstate->frame ) { printf( "Now at top frame %s %s\n", PyString_AsString( PyObject_Str( (PyObject *)tstate->frame ) ), PyString_AsString( PyObject_Repr( (PyObject *)tstate->frame->f_code ) ) ); } else { printf( "Now at top no frame\n"); } #endif } // Attach locals to a frame object. extern void Nuitka_Frame_AttachLocals( struct Nuitka_FrameObject *frame, char const *type_description, ... ); extern void Nuitka_Frame_ReleaseLocals( struct Nuitka_FrameObject *frame ); // Codes used for type_description. #define NUITKA_TYPE_DESCRIPTION_NULL 'N' #define NUITKA_TYPE_DESCRIPTION_CELL 'c' #define NUITKA_TYPE_DESCRIPTION_OBJECT 'o' #define NUITKA_TYPE_DESCRIPTION_OBJECT_PTR 'O' #define NUITKA_TYPE_DESCRIPTION_BOOL 'b' #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_cell.h0000644000372000001440000000413613112214770023741 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_CELL_H__ #define __NUITKA_COMPILED_CELL_H__ /* This is a clone of the normal PyCell structure. We should keep it binary * compatible, just in case somebody crazy insists on it. */ extern PyTypeObject Nuitka_Cell_Type; static inline bool Nuitka_Cell_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Cell_Type; } struct Nuitka_CellObject { PyObject_HEAD PyObject *ob_ref; /* Content of the cell or NULL when empty */ }; extern struct Nuitka_CellObject *Nuitka_Cell_New( void ); extern void Nuitka_Cells_New( struct Nuitka_CellObject **closure, int count ); NUITKA_MAY_BE_UNUSED static struct Nuitka_CellObject *PyCell_NEW0( PyObject *value ) { CHECK_OBJECT( value ); struct Nuitka_CellObject *result = Nuitka_Cell_New(); assert( result != NULL ); result->ob_ref = value; Py_INCREF( value ); return result; } NUITKA_MAY_BE_UNUSED static struct Nuitka_CellObject *PyCell_NEW1( PyObject *value ) { CHECK_OBJECT( value ); struct Nuitka_CellObject *result = Nuitka_Cell_New(); assert( result != NULL ); result->ob_ref = value; return result; } NUITKA_MAY_BE_UNUSED static struct Nuitka_CellObject *PyCell_EMPTY( void ) { struct Nuitka_CellObject *result = Nuitka_Cell_New(); result->ob_ref = NULL; return result; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/compiled_method.h0000644000372000001440000000331513112214770024300 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_COMPILED_METHOD_H__ #define __NUITKA_COMPILED_METHOD_H__ // Compiled function and compile generator types may be referenced. #include "compiled_function.h" #include "compiled_generator.h" // The backbone of the integration into CPython. Try to behave as well as normal // method objects, or even better. // The Nuitka_MethodObject is the storage associated with a compiled method // instance of which there can be many for each code. struct Nuitka_MethodObject { PyObject_HEAD struct Nuitka_FunctionObject *m_function; PyObject *m_weakrefs; PyObject *m_object; PyObject *m_class; }; extern PyTypeObject Nuitka_Method_Type; // Make a method out of a function. extern PyObject *Nuitka_Method_New( struct Nuitka_FunctionObject *function, PyObject *object, PyObject *klass ); static inline bool Nuitka_Method_Check( PyObject *object ) { return Py_TYPE( object ) == &Nuitka_Method_Type; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/tracing.h0000644000372000001440000000235413207537242022604 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_TRACING_H__ #define __NUITKA_TRACING_H__ /* Stupid tracing, intended to help where debugging is not an option * and to give kind of progress record of startup and the running of * the program. */ #ifdef _NUITKA_TRACE #define NUITKA_PRINT_TRACE(value) {puts(value);fflush(stdout);} #define NUITKA_PRINTF_TRACE(...) {printf (__VA_ARGS__);fflush(stdout);} #else #define NUITKA_PRINT_TRACE(value) #define NUITKA_PRINTF_TRACE(...) #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/0000755000372000001440000000000013207540420022247 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/iterators.h0000644000372000001440000001656213134660221024447 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_ITERATORS_H__ #define __NUITKA_HELPER_ITERATORS_H__ #if PYTHON_VERSION >= 270 // Initialize value for "tp_iternext" to compare with, needed by HAS_ITERNEXT // which emulates "PyCheck_Iter" but is bug free. extern iternextfunc default_iternext; extern void _initSlotIternext( void ); #endif // This is like "PyIter_Check" but without bugs due to shared library pointers. NUITKA_MAY_BE_UNUSED static inline bool HAS_ITERNEXT( PyObject *value ) { #if PYTHON_VERSION < 300 if ( !PyType_HasFeature( Py_TYPE( value ), Py_TPFLAGS_HAVE_ITER ) ) { return false; } #endif iternextfunc tp_iternext = Py_TYPE( value )->tp_iternext; if ( tp_iternext == NULL ) { return false; } #if PYTHON_VERSION >= 270 return tp_iternext != default_iternext; #else return true; #endif } // Stolen from CPython implementation, so we can access it. typedef struct { PyObject_HEAD #if PYTHON_VERSION < 340 long it_index; #else Py_ssize_t it_index; #endif PyObject *it_seq; } seqiterobject; NUITKA_MAY_BE_UNUSED static PyObject *MAKE_ITERATOR( PyObject *iterated ) { #if PYTHON_VERSION < 300 getiterfunc tp_iter = NULL; if ( PyType_HasFeature( Py_TYPE( iterated ), Py_TPFLAGS_HAVE_ITER )) { tp_iter = Py_TYPE( iterated )->tp_iter; } #else getiterfunc tp_iter = Py_TYPE( iterated )->tp_iter; #endif if ( tp_iter ) { PyObject *result = (*tp_iter)( iterated ); if (unlikely( result == NULL )) { return NULL; } else if (unlikely( !HAS_ITERNEXT( result )) ) { PyErr_Format( PyExc_TypeError, "iter() returned non-iterator of type '%s'", Py_TYPE( result )->tp_name ); Py_DECREF( result ); return NULL; } else { return result; } } else if ( PySequence_Check( iterated ) ) { seqiterobject *result = PyObject_GC_New( seqiterobject, &PySeqIter_Type ); assert( result ); result->it_index = 0; Py_INCREF( iterated ); result->it_seq = iterated; Nuitka_GC_Track( result ); return (PyObject *)result; } else { PyErr_Format( PyExc_TypeError, "'%s' object is not iterable", Py_TYPE( iterated )->tp_name ); return NULL; } } NUITKA_MAY_BE_UNUSED static PyObject *ITERATOR_NEXT( PyObject *iterator ) { CHECK_OBJECT( iterator ); iternextfunc iternext = Py_TYPE( iterator )->tp_iternext; if (unlikely( iternext == NULL )) { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 330 "%s object is not an iterator", #else "'%s' object is not an iterator", #endif Py_TYPE( iterator )->tp_name ); return NULL; } PyObject *result = (*iternext)( iterator ); #if PYTHON_VERSION < 330 if ( result == NULL ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error ) { if (likely( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) )) { CLEAR_ERROR_OCCURRED(); } } } #endif return result; } NUITKA_MAY_BE_UNUSED static PyObject *BUILTIN_NEXT1( PyObject *iterator ) { CHECK_OBJECT( iterator ); iternextfunc iternext = Py_TYPE( iterator )->tp_iternext; if (unlikely( iternext == NULL )) { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 330 "%s object is not an iterator", #else "'%s' object is not an iterator", #endif Py_TYPE( iterator )->tp_name ); return NULL; } PyObject *result = (*iternext)( iterator ); if (unlikely( result == NULL )) { // The iteration can return NULL with no error, which means // StopIteration. if ( !ERROR_OCCURRED() ) { PyErr_SetNone( PyExc_StopIteration ); } return NULL; } else { CHECK_OBJECT( result ); } return result; } NUITKA_MAY_BE_UNUSED static PyObject *BUILTIN_NEXT2( PyObject *iterator, PyObject *default_value ) { CHECK_OBJECT( iterator ); CHECK_OBJECT( default_value ); PyObject *result = (*Py_TYPE( iterator )->tp_iternext)( iterator ); if (unlikely( result == NULL )) { PyObject *error = GET_ERROR_OCCURRED(); if ( error != NULL ) { if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration )) { DROP_ERROR_OCCURRED(); Py_INCREF( default_value ); return default_value; } else { return NULL; } } else { Py_INCREF( default_value ); return default_value; } } else { CHECK_OBJECT( result ); } return result; } #if PYTHON_VERSION < 350 NUITKA_MAY_BE_UNUSED static PyObject *UNPACK_NEXT( PyObject *iterator, int seq_size_so_far ) #else NUITKA_MAY_BE_UNUSED static PyObject *UNPACK_NEXT( PyObject *iterator, int seq_size_so_far, int expected ) #endif { CHECK_OBJECT( iterator ); assert( HAS_ITERNEXT( iterator ) ); PyObject *result = (*Py_TYPE( iterator )->tp_iternext)( iterator ); if (unlikely( result == NULL )) { #if PYTHON_VERSION < 300 if (unlikely( !ERROR_OCCURRED() )) #else if (unlikely( !ERROR_OCCURRED() || EXCEPTION_MATCH_BOOL_SINGLE( GET_ERROR_OCCURRED(), PyExc_StopIteration ) )) #endif { #if PYTHON_VERSION < 350 if ( seq_size_so_far == 1 ) { PyErr_Format( PyExc_ValueError, "need more than 1 value to unpack" ); } else { PyErr_Format( PyExc_ValueError, "need more than %d values to unpack", seq_size_so_far ); } #else PyErr_Format( PyExc_ValueError, "not enough values to unpack (expected %d, got %d)", expected, seq_size_so_far ); #endif } return NULL; } CHECK_OBJECT( result ); return result; } NUITKA_MAY_BE_UNUSED static bool UNPACK_ITERATOR_CHECK( PyObject *iterator ) { CHECK_OBJECT( iterator ); assert( HAS_ITERNEXT( iterator ) ); PyObject *attempt = (*Py_TYPE( iterator )->tp_iternext)( iterator ); if (likely( attempt == NULL )) { return CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED(); } else { Py_DECREF( attempt ); PyErr_Format( PyExc_ValueError, "too many values to unpack" ); return false; } } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/subscripts.h0000644000372000001440000002463413122472300024627 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_SUBSCRIPTS_H__ #define __NUITKA_HELPER_SUBSCRIPTS_H__ extern PyObject *STRING_FROM_CHAR( unsigned char c ); NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SUBSCRIPT_CONST( PyObject *source, PyObject *const_subscript, Py_ssize_t int_subscript ) { CHECK_OBJECT( source ); CHECK_OBJECT( const_subscript ); PyTypeObject *type = Py_TYPE( source ); PyMappingMethods *mapping_methods = type->tp_as_mapping; PyObject *result; if ( mapping_methods && mapping_methods->mp_subscript ) { if ( PyList_CheckExact( source ) ) { Py_ssize_t list_size = PyList_GET_SIZE( source ); if ( int_subscript < 0 ) { if ( -int_subscript > list_size ) { PyErr_Format( PyExc_IndexError, "list index out of range" ); return NULL; } int_subscript += list_size; } else { if ( int_subscript >= list_size ) { PyErr_Format( PyExc_IndexError, "list index out of range" ); return NULL; } } result = ((PyListObject *)source)->ob_item[ int_subscript ]; Py_INCREF( result ); return result; } #if PYTHON_VERSION < 300 else if ( PyString_CheckExact( source ) ) { Py_ssize_t string_size = PyString_GET_SIZE( source ); if ( int_subscript < 0 ) { if ( -int_subscript > string_size ) { PyErr_Format( PyExc_IndexError, "string index out of range" ); return NULL; } int_subscript += string_size; } else { if ( int_subscript >= string_size ) { PyErr_Format( PyExc_IndexError, "string index out of range" ); return NULL; } } unsigned char c = ((PyStringObject *)source)->ob_sval[ int_subscript ]; return STRING_FROM_CHAR( c ); } #else else if ( PyUnicode_CheckExact( source ) ) { if ( int_subscript < 0 ) { #if PYTHON_VERSION < 330 int_subscript += PyUnicode_GET_SIZE( source ); #else int_subscript += PyUnicode_GET_LENGTH( source ); #endif } result = type->tp_as_sequence->sq_item( source, int_subscript ); } #endif else { result = mapping_methods->mp_subscript( source, const_subscript ); } } else if ( type->tp_as_sequence ) { result = PySequence_GetItem( source, int_subscript ); } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 270 "'%s' object is unsubscriptable", #elif PYTHON_VERSION >= 300 || PYTHON_VERSION <= 272 "'%s' object is not subscriptable", #else "'%s' object has no attribute '__getitem__'", #endif Py_TYPE( source )->tp_name ); return NULL; } if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SUBSCRIPT( PyObject *source, PyObject *subscript ) { CHECK_OBJECT( source ); CHECK_OBJECT( subscript ); PyTypeObject *type = Py_TYPE( source ); PyMappingMethods *mapping = type->tp_as_mapping; PyObject *result; if ( mapping != NULL && mapping->mp_subscript != NULL ) { result = mapping->mp_subscript( source, subscript ); } else if ( type->tp_as_sequence != NULL ) { if ( PyIndex_Check( subscript ) ) { Py_ssize_t index = PyNumber_AsSsize_t( subscript, NULL ); if ( index == -1 && ERROR_OCCURRED() ) { return NULL; } result = PySequence_GetItem( source, index ); } else if ( type->tp_as_sequence->sq_item ) { PyErr_Format( PyExc_TypeError, "sequence index must be integer, not '%s'", Py_TYPE( subscript )->tp_name ); return NULL; } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 270 "'%s' object is unsubscriptable", #elif PYTHON_VERSION >= 300 || PYTHON_VERSION <= 272 "'%s' object is not subscriptable", #else "'%s' object has no attribute '__getitem__'", #endif Py_TYPE( source )->tp_name ); return NULL; } } else { PyErr_Format( PyExc_TypeError, #if PYTHON_VERSION < 270 "'%s' object is unsubscriptable", #elif PYTHON_VERSION >= 300 || PYTHON_VERSION <= 272 "'%s' object is not subscriptable", #else "'%s' object has no attribute '__getitem__'", #endif Py_TYPE( source )->tp_name ); return NULL; } if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool SET_SUBSCRIPT_CONST( PyObject *target, PyObject *subscript, Py_ssize_t int_subscript, PyObject *value ) { CHECK_OBJECT( value ); CHECK_OBJECT( target ); CHECK_OBJECT( subscript ); PyMappingMethods *mapping_methods = Py_TYPE( target )->tp_as_mapping; if ( mapping_methods != NULL && mapping_methods->mp_ass_subscript ) { if ( PyList_CheckExact( target ) ) { Py_ssize_t list_size = PyList_GET_SIZE( target ); if ( int_subscript < 0 ) { if ( -int_subscript > list_size ) { PyErr_Format( PyExc_IndexError, "list assignment index out of range" ); return false; } int_subscript += list_size; } PyListObject *target_list = (PyListObject *)target; PyObject *old_value = target_list->ob_item[ int_subscript ]; Py_INCREF( value ); target_list->ob_item[ int_subscript ] = value; Py_DECREF( old_value ); return true; } else { int res = mapping_methods->mp_ass_subscript( target, subscript, value ); if (unlikely( res == -1 )) { return false; } return true; } } else if ( Py_TYPE( target )->tp_as_sequence ) { if ( PyIndex_Check( subscript ) ) { Py_ssize_t key_value = PyNumber_AsSsize_t( subscript, PyExc_IndexError ); if ( key_value == -1 ) { if ( ERROR_OCCURRED() ) { return false; } } return SEQUENCE_SETITEM( target, key_value, value ); } else if ( Py_TYPE( target )->tp_as_sequence->sq_ass_item ) { PyErr_Format( PyExc_TypeError, "sequence index must be integer, not '%s'", Py_TYPE( subscript )->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } } NUITKA_MAY_BE_UNUSED static bool SET_SUBSCRIPT( PyObject *target, PyObject *subscript, PyObject *value ) { CHECK_OBJECT( value ); CHECK_OBJECT( target ); CHECK_OBJECT( subscript ); PyMappingMethods *mapping_methods = Py_TYPE( target )->tp_as_mapping; if ( mapping_methods != NULL && mapping_methods->mp_ass_subscript ) { int res = mapping_methods->mp_ass_subscript( target, subscript, value ); if (unlikely( res == -1 )) { return false; } } else if ( Py_TYPE( target )->tp_as_sequence ) { if ( PyIndex_Check( subscript ) ) { Py_ssize_t key_value = PyNumber_AsSsize_t( subscript, PyExc_IndexError ); if ( key_value == -1 ) { if ( ERROR_OCCURRED() ) { return false; } } return SEQUENCE_SETITEM( target, key_value, value ); } else if ( Py_TYPE( target )->tp_as_sequence->sq_ass_item ) { PyErr_Format( PyExc_TypeError, "sequence index must be integer, not '%s'", Py_TYPE( subscript )->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( target )->tp_name ); return false; } return true; } NUITKA_MAY_BE_UNUSED static bool DEL_SUBSCRIPT( PyObject *target, PyObject *subscript ) { CHECK_OBJECT( target ); CHECK_OBJECT( subscript ); int status = PyObject_DelItem( target, subscript ); if (unlikely( status == -1 )) { return false; } return true; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/printing.h0000644000372000001440000000431413122472300024251 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_PRINTING_H__ #define __NUITKA_PRINTING_H__ // Helper functions for print. Need to play nice with Python softspace // behavior. extern bool PRINT_NEW_LINE( void ); extern bool PRINT_ITEM( PyObject *object ); extern bool PRINT_STRING( char const *str ); extern bool PRINT_ITEM_TO( PyObject *file, PyObject *object ); extern bool PRINT_NEW_LINE_TO( PyObject *file ); extern PyObject *GET_STDOUT(); extern PyObject *GET_STDERR(); // ----------------------------------------------------------------------- // Helper functions to debug the run time operation of the compiled binary // manually or in debug modes. // Print the reference count of the object. extern void PRINT_REFCOUNT( PyObject *object ); // Print the full traceback stack. // TODO: Could be ported, the "printf" stuff would need to be split. On Python3 // the normal C print output gets lost. #if PYTHON_VERSION < 300 extern void PRINT_TRACEBACK( PyTracebackObject *traceback ); #endif // Print the exception state, including NULL values. extern void PRINT_EXCEPTION( PyObject *exception_type, PyObject *exception_value, PyObject *exception_tb ); // Print the current exception state, including NULL values. extern void PRINT_CURRENT_EXCEPTION( void ); // Print the representation of the object, or "" if it's not set. extern bool PRINT_REPR( PyObject *object ); // Print the word , as an alternative to pointers. extern bool PRINT_NULL( void ); #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/bytearrays.h0000644000372000001440000000225613134660221024613 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_BYTEARRAYS_H__ #define __NUITKA_HELPER_BYTEARRAYS_H__ NUITKA_MAY_BE_UNUSED static PyObject *BYTEARRAY_COPY( PyObject *bytearray ) { CHECK_OBJECT( bytearray ); assert( PyByteArray_CheckExact( bytearray ) ); PyObject *result = PyByteArray_FromObject( bytearray ); if (unlikely( result == NULL )) { return NULL; } return result; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/lists.h0000644000372000001440000000253613122472300023561 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_LISTS_H__ #define __NUITKA_HELPER_LISTS_H__ NUITKA_MAY_BE_UNUSED static PyObject *LIST_COPY( PyObject *list ) { CHECK_OBJECT( list ); assert( PyList_CheckExact( list ) ); Py_ssize_t size = PyList_GET_SIZE( list ); PyObject *result = PyList_New( size ); if (unlikely( result == NULL )) { return NULL; } for ( Py_ssize_t i = 0; i < size; i++ ) { PyObject *item = PyList_GET_ITEM( list, i ); Py_INCREF( item ); PyList_SET_ITEM( result, i, item ); } return result; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/richcomparisons.h0000644000372000001440000002201313122472300025616 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_RICHCOMPARISONS_H__ #define __NUITKA_HELPER_RICHCOMPARISONS_H__ static inline bool IS_SANE_TYPE( PyTypeObject *type ) { return #if PYTHON_VERSION < 300 type == &PyString_Type || type == &PyInt_Type || #endif type == &PyLong_Type || type == &PyList_Type || type == &PyTuple_Type; } extern PyObject *MY_RICHCOMPARE( PyObject *v, PyObject *w, int op ); extern PyObject *MY_RICHCOMPARE_NORECURSE( PyObject *v, PyObject *w, int op ); NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_LT( PyObject *operand1, PyObject *operand2 ) { PyObject *result = MY_RICHCOMPARE( operand1, operand2, Py_LT ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_LE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { Py_INCREF( Py_True ); return Py_True; } PyObject *result = MY_RICHCOMPARE( operand1, operand2, Py_LE ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_EQ( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { Py_INCREF( Py_True ); return Py_True; } return MY_RICHCOMPARE( operand1, operand2, Py_EQ ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_EQ_NORECURSE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { Py_INCREF( Py_True ); return Py_True; } return MY_RICHCOMPARE_NORECURSE( operand1, operand2, Py_EQ ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_NE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { Py_INCREF( Py_False ); return Py_False; } return MY_RICHCOMPARE( operand1, operand2, Py_NE ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_GT( PyObject *operand1, PyObject *operand2 ) { return MY_RICHCOMPARE( operand1, operand2, Py_GT ); } NUITKA_MAY_BE_UNUSED static PyObject *RICH_COMPARE_GE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { Py_INCREF( Py_True ); return Py_True; } return MY_RICHCOMPARE( operand1, operand2, Py_GE ); } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_LT( PyObject *operand1, PyObject *operand2 ) { PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_LT ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_LE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_LE ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_EQ( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_EQ ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_EQ_NORECURSE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE_NORECURSE( operand1, operand2, Py_EQ ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_NE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 0; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_NE ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_GT( PyObject *operand1, PyObject *operand2 ) { PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_GT ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } NUITKA_MAY_BE_UNUSED static int RICH_COMPARE_BOOL_GE( PyObject *operand1, PyObject *operand2 ) { // Quick path for avoidable checks, compatible with CPython. if ( operand1 == operand2 && IS_SANE_TYPE( Py_TYPE( operand1 ) ) ) { return 1; } PyObject *rich_result = MY_RICHCOMPARE( operand1, operand2, Py_GE ); if (unlikely( rich_result == NULL )) { return -1; } int result; // Doing the quick tests on the outside spares the function call, with // "partial inline" this should become unneeded. if ( rich_result == Py_True ) { result = 1; } else if ( rich_result == Py_False || rich_result == Py_None ) { result = 0; } else { result = CHECK_IF_TRUE( rich_result ); } Py_DECREF( rich_result ); return result; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/boolean.h0000644000372000001440000000551113134660221024042 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_BOOLEAN_H__ #define __NUITKA_HELPER_BOOLEAN_H__ // The slot in Python3 got renamed, compensate it like this. #if PYTHON_VERSION >= 300 #define nb_nonzero nb_bool #endif NUITKA_MAY_BE_UNUSED static int CHECK_IF_TRUE( PyObject *object ) { CHECK_OBJECT( object ); if ( object == Py_True ) { return 1; } else if ( object == Py_False || object == Py_None ) { return 0; } else { Py_ssize_t result; if ( Py_TYPE( object )->tp_as_number != NULL && Py_TYPE( object )->tp_as_number->nb_nonzero != NULL ) { result = (*Py_TYPE( object )->tp_as_number->nb_nonzero)( object ); } else if ( Py_TYPE( object )->tp_as_mapping != NULL && Py_TYPE( object )->tp_as_mapping->mp_length != NULL ) { result = (*Py_TYPE( object )->tp_as_mapping->mp_length)( object ); } else if ( Py_TYPE( object )->tp_as_sequence != NULL && Py_TYPE( object )->tp_as_sequence->sq_length != NULL ) { result = (*Py_TYPE( object )->tp_as_sequence->sq_length)( object ); } else { return 1; } if ( result > 0 ) { return 1; } else if ( result == 0 ) { return 0; } else { return -1; } } } NUITKA_MAY_BE_UNUSED static int CHECK_IF_FALSE( PyObject *object ) { int result = CHECK_IF_TRUE( object ); if ( result == 0 ) return 1; if ( result == 1 ) return 0; return -1; } NUITKA_MAY_BE_UNUSED static PyObject *BOOL_FROM( bool value ) { CHECK_OBJECT( Py_True ); CHECK_OBJECT( Py_False ); return value ? Py_True : Py_False; } NUITKA_MAY_BE_UNUSED static PyObject *UNARY_NOT( PyObject *object ) { int res = CHECK_IF_TRUE( object ); if ( res == 0 ) return Py_True; if ( res == 1 ) return Py_False; return NULL; } #undef nb_nonzero typedef enum { NUITKA_BOOL_FALSE = 0, NUITKA_BOOL_TRUE = 1, NUITKA_BOOL_UNASSIGNED = 2, } nuitka_bool; #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/attributes.h0000644000372000001440000005512113122472300024607 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_ATTRIBUTES_H__ #define __NUITKA_HELPER_ATTRIBUTES_H__ #if PYTHON_VERSION < 300 NUITKA_MAY_BE_UNUSED static PyObject *FIND_ATTRIBUTE_IN_CLASS( PyClassObject *klass, PyObject *attr_name ) { CHECK_OBJECT( klass ); CHECK_OBJECT( attr_name ); assert( PyClass_Check( klass )); assert( PyString_CheckExact( attr_name ) ); PyObject *result = GET_STRING_DICT_VALUE( (PyDictObject *)klass->cl_dict, (PyStringObject *)attr_name ); if ( result == NULL ) { Py_ssize_t base_count = PyTuple_Size( klass->cl_bases ); for ( Py_ssize_t i = 0; i < base_count; i++ ) { result = FIND_ATTRIBUTE_IN_CLASS( (PyClassObject *)PyTuple_GetItem( klass->cl_bases, i ), attr_name ); if ( result != NULL ) { break; } } } return result; } #endif #if PYTHON_VERSION < 300 extern PyObject *CALL_FUNCTION_WITH_ARGS2( PyObject *called, PyObject **args ); static PyObject *LOOKUP_INSTANCE( PyObject *source, PyObject *attr_name ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); assert( PyInstance_Check( source ) ); assert( PyString_CheckExact( attr_name ) ); PyInstanceObject *source_instance = (PyInstanceObject *)source; // The special cases have their own variant on the code generation level // as we are called with constants only. assert( attr_name != const_str_plain___dict__ ); assert( attr_name != const_str_plain___class__ ); // Try the instance dict first. PyObject *result = GET_STRING_DICT_VALUE( (PyDictObject *)source_instance->in_dict, (PyStringObject *)attr_name ); if ( result ) { Py_INCREF( result ); return result; } // Next see if a class has it result = FIND_ATTRIBUTE_IN_CLASS( source_instance->in_class, attr_name ); if ( result != NULL ) { descrgetfunc func = Py_TYPE( result )->tp_descr_get; if ( func ) { result = func( result, source, (PyObject *)source_instance->in_class ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else { Py_INCREF( result ); return result; } } // Finally allow a __getattr__ to handle it or else it's an error. if (unlikely( source_instance->in_class->cl_getattr == NULL )) { PyErr_Format( PyExc_AttributeError, "%s instance has no attribute '%s'", PyString_AS_STRING( source_instance->in_class->cl_name ), PyString_AS_STRING( attr_name ) ); return NULL; } else { PyObject *args[] = { source, attr_name }; return CALL_FUNCTION_WITH_ARGS2( source_instance->in_class->cl_getattr, args ); } } #endif NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_ATTRIBUTE( PyObject *source, PyObject *attr_name ) { /* Note: There are 2 specializations of this function, that need to be * updated in line with this: LOOKUP_ATTRIBUTE_[DICT|CLASS]_SLOT */ CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro == PyObject_GenericGetAttr ) { // Unfortunately this is required, although of cause rarely necessary. if (unlikely( type->tp_dict == NULL )) { if (unlikely( PyType_Ready( type ) < 0 )) { return NULL; } } PyObject *descr = _PyType_Lookup( type, attr_name ); descrgetfunc func = NULL; if ( descr != NULL ) { Py_INCREF( descr ); #if PYTHON_VERSION < 300 if ( PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) ) { #endif func = Py_TYPE( descr )->tp_descr_get; if ( func != NULL && PyDescr_IsData( descr ) ) { PyObject *result = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); return result; } #if PYTHON_VERSION < 300 } #endif } Py_ssize_t dictoffset = type->tp_dictoffset; PyObject *dict = NULL; if ( dictoffset != 0 ) { // Negative dictionary offsets have special meaning. if ( dictoffset < 0 ) { Py_ssize_t tsize; size_t size; tsize = ((PyVarObject *)source)->ob_size; if (tsize < 0) tsize = -tsize; size = _PyObject_VAR_SIZE( type, tsize ); dictoffset += (long)size; } PyObject **dictptr = (PyObject **) ((char *)source + dictoffset); dict = *dictptr; } if ( dict != NULL ) { CHECK_OBJECT( dict ); Py_INCREF( dict ); PyObject *result = PyDict_GetItem( dict, attr_name ); if ( result != NULL ) { Py_INCREF( result ); Py_XDECREF( descr ); Py_DECREF( dict ); CHECK_OBJECT( result ); return result; } Py_DECREF( dict ); } if ( func != NULL ) { PyObject *result = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); CHECK_OBJECT( result ); return result; } if ( descr != NULL ) { CHECK_OBJECT( descr ); return descr; } #if PYTHON_VERSION < 300 PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%s'", type->tp_name, PyString_AS_STRING( attr_name ) ); #else PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%U'", type->tp_name, attr_name ); #endif return NULL; } #if PYTHON_VERSION < 300 else if ( type->tp_getattro == PyInstance_Type.tp_getattro ) { PyObject *result = LOOKUP_INSTANCE( source, attr_name ); return result; } #endif else if ( type->tp_getattro != NULL ) { PyObject *result = (*type->tp_getattro)( source, attr_name ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *result = (*type->tp_getattr)( source, Nuitka_String_AsString_Unchecked( attr_name ) ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '%s'", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return NULL; } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_ATTRIBUTE_DICT_SLOT( PyObject *source ) { CHECK_OBJECT( source ); PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro == PyObject_GenericGetAttr ) { if (unlikely( type->tp_dict == NULL )) { if (unlikely( PyType_Ready( type ) < 0 )) { return NULL; } } PyObject *descr = _PyType_Lookup( type, const_str_plain___dict__ ); descrgetfunc func = NULL; if ( descr != NULL ) { Py_INCREF( descr ); #if PYTHON_VERSION < 300 if ( PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) ) { #endif func = Py_TYPE( descr )->tp_descr_get; if ( func != NULL && PyDescr_IsData( descr ) ) { PyObject *result = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); return result; } #if PYTHON_VERSION < 300 } #endif } Py_ssize_t dictoffset = type->tp_dictoffset; PyObject *dict = NULL; if ( dictoffset != 0 ) { // Negative dictionary offsets have special meaning. if ( dictoffset < 0 ) { Py_ssize_t tsize; size_t size; tsize = ((PyVarObject *)source)->ob_size; if (tsize < 0) tsize = -tsize; size = _PyObject_VAR_SIZE( type, tsize ); dictoffset += (long)size; } PyObject **dictptr = (PyObject **) ((char *)source + dictoffset); dict = *dictptr; } if ( dict != NULL ) { CHECK_OBJECT( dict ); Py_INCREF( dict ); PyObject *result = PyDict_GetItem( dict, const_str_plain___dict__ ); if ( result != NULL ) { Py_INCREF( result ); Py_XDECREF( descr ); Py_DECREF( dict ); CHECK_OBJECT( result ); return result; } Py_DECREF( dict ); } if ( func != NULL ) { PyObject *result = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); CHECK_OBJECT( result ); return result; } if ( descr != NULL ) { CHECK_OBJECT( descr ); return descr; } PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '__dict__'", type->tp_name ); return NULL; } #if PYTHON_VERSION < 300 else if ( type->tp_getattro == PyInstance_Type.tp_getattro ) { PyInstanceObject *source_instance = (PyInstanceObject *)source; PyObject *result = source_instance->in_dict; Py_INCREF( result ); return result; } #endif else if ( type->tp_getattro != NULL ) { PyObject *result = (*type->tp_getattro)( source, const_str_plain___dict__ ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *result = (*type->tp_getattr)( source, (char *)"__dict__" ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '__dict__'", type->tp_name ); return NULL; } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_ATTRIBUTE_CLASS_SLOT( PyObject *source ) { CHECK_OBJECT( source ); PyTypeObject *type = Py_TYPE( source ); if ( type->tp_getattro == PyObject_GenericGetAttr ) { if (unlikely( type->tp_dict == NULL )) { if (unlikely( PyType_Ready( type ) < 0 )) { return NULL; } } PyObject *descr = _PyType_Lookup( type, const_str_plain___class__ ); descrgetfunc func = NULL; if ( descr != NULL ) { Py_INCREF( descr ); #if PYTHON_VERSION < 300 if ( PyType_HasFeature( Py_TYPE( descr ), Py_TPFLAGS_HAVE_CLASS ) ) { #endif func = Py_TYPE( descr )->tp_descr_get; if ( func != NULL && PyDescr_IsData( descr ) ) { PyObject *result = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); return result; } #if PYTHON_VERSION < 300 } #endif } Py_ssize_t dictoffset = type->tp_dictoffset; PyObject *dict = NULL; if ( dictoffset != 0 ) { // Negative dictionary offsets have special meaning. if ( dictoffset < 0 ) { Py_ssize_t tsize; size_t size; tsize = ((PyVarObject *)source)->ob_size; if (tsize < 0) tsize = -tsize; size = _PyObject_VAR_SIZE( type, tsize ); dictoffset += (long)size; } PyObject **dictptr = (PyObject **) ((char *)source + dictoffset); dict = *dictptr; } if ( dict != NULL ) { CHECK_OBJECT( dict ); Py_INCREF( dict ); PyObject *result = PyDict_GetItem( dict, const_str_plain___class__ ); if ( result != NULL ) { Py_INCREF( result ); Py_XDECREF( descr ); Py_DECREF( dict ); CHECK_OBJECT( result ); return result; } Py_DECREF( dict ); } if ( func != NULL ) { PyObject *result = func( descr, source, (PyObject *)type ); Py_DECREF( descr ); CHECK_OBJECT( result ); return result; } if ( descr != NULL ) { CHECK_OBJECT( descr ); return descr; } PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '__class__'", type->tp_name ); return NULL; } #if PYTHON_VERSION < 300 else if ( type->tp_getattro == PyInstance_Type.tp_getattro ) { PyInstanceObject *source_instance = (PyInstanceObject *)source; PyObject *result = (PyObject *)source_instance->in_class; Py_INCREF( result ); return result; } #endif else if ( type->tp_getattro != NULL ) { PyObject *result = (*type->tp_getattro)( source, const_str_plain___class__ ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } else if ( type->tp_getattr != NULL ) { PyObject *result = (*type->tp_getattr)( source, (char *)"__class__" ); return result; } else { PyErr_Format( PyExc_AttributeError, "'%s' object has no attribute '__class__'", type->tp_name ); return NULL; } } NUITKA_MAY_BE_UNUSED static PyObject *BUILTIN_HASATTR( PyObject *source, PyObject *attr_name ) { CHECK_OBJECT( source ); CHECK_OBJECT( attr_name ); int res = PyObject_HasAttr( source, attr_name ); if (unlikely( res == -1 )) { return NULL; } return BOOL_FROM(res == 1); } #if PYTHON_VERSION < 300 extern PyObject *CALL_FUNCTION_WITH_ARGS3( PyObject *called, PyObject **args ); static bool SET_INSTANCE( PyObject *target, PyObject *attr_name, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( attr_name ); CHECK_OBJECT( value ); assert( PyInstance_Check( target ) ); assert( PyString_Check( attr_name ) ); PyInstanceObject *target_instance = (PyInstanceObject *)target; // The special cases should get their own SET_ATTRIBUTE_xxxx_SLOT variants // on the code generation level as SET_ATTRIBUTE is called with constants // only. assert( attr_name != const_str_plain___dict__ ); assert( attr_name != const_str_plain___class__ ); if ( target_instance->in_class->cl_setattr != NULL ) { PyObject *args[] = { target, attr_name, value }; PyObject *result = CALL_FUNCTION_WITH_ARGS3( target_instance->in_class->cl_setattr, args ); if (unlikely( result == NULL )) { return false; } Py_DECREF( result ); return true; } else { int status = PyDict_SetItem( target_instance->in_dict, attr_name, value ); if (unlikely( status != 0 )) { return false; } return true; } } #endif NUITKA_MAY_BE_UNUSED static bool SET_ATTRIBUTE( PyObject *target, PyObject *attr_name, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( attr_name ); CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if ( PyInstance_Check( target ) ) { return SET_INSTANCE( target, attr_name, value ); } else #endif { PyTypeObject *type = Py_TYPE( target ); if ( type->tp_setattro != NULL ) { int status = (*type->tp_setattro)( target, attr_name, value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_setattr != NULL ) { int status = (*type->tp_setattr)( target, Nuitka_String_AsString_Unchecked( attr_name ), value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_getattr == NULL && type->tp_getattro == NULL ) { PyErr_Format( PyExc_TypeError, "'%s' object has no attributes (assign to %s)", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object has only read-only attributes (assign to %s)", type->tp_name, Nuitka_String_AsString_Unchecked( attr_name ) ); return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool SET_ATTRIBUTE_DICT_SLOT( PyObject *target, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if ( likely( PyInstance_Check( target ) )) { PyInstanceObject *target_instance = (PyInstanceObject *)target; /* Note seems this doesn't have to be an exact dictionary. */ if (unlikely( !PyDict_Check( value ) )) { PyErr_SetString( PyExc_TypeError, "__dict__ must be set to a dictionary" ); return false; } PyObject *old = target_instance->in_dict; Py_INCREF( value ); target_instance->in_dict = value; Py_DECREF( old ); } else #endif { PyTypeObject *type = Py_TYPE( target ); if ( type->tp_setattro != NULL ) { int status = (*type->tp_setattro)( target, const_str_plain___dict__, value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_setattr != NULL ) { int status = (*type->tp_setattr)( target, (char *)"__dict__", value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_getattr == NULL && type->tp_getattro == NULL ) { PyErr_Format( PyExc_TypeError, "'%s' object has no attributes (assign to __dict__)", type->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object has only read-only attributes (assign to __dict__)", type->tp_name ); return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool SET_ATTRIBUTE_CLASS_SLOT( PyObject *target, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( value ); #if PYTHON_VERSION < 300 if (likely( PyInstance_Check( target ) )) { PyInstanceObject *target_instance = (PyInstanceObject *)target; if (unlikely( !PyClass_Check( value ) )) { PyErr_SetString( PyExc_TypeError, "__class__ must be set to a class" ); return false; } PyObject *old = (PyObject *)( target_instance->in_class ); Py_INCREF( value ); target_instance->in_class = (PyClassObject *)value; Py_DECREF( old ); } else #endif { PyTypeObject *type = Py_TYPE( target ); if ( type->tp_setattro != NULL ) { int status = (*type->tp_setattro)( target, const_str_plain___class__, value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_setattr != NULL ) { int status = (*type->tp_setattr)( target, (char *)"__class__", value ); if (unlikely( status == -1 )) { return false; } } else if ( type->tp_getattr == NULL && type->tp_getattro == NULL ) { PyErr_Format( PyExc_TypeError, "'%s' object has no attributes (assign to __class__)", type->tp_name ); return false; } else { PyErr_Format( PyExc_TypeError, "'%s' object has only read-only attributes (assign to __class__)", type->tp_name ); return false; } } return true; } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SPECIAL( PyObject *source, PyObject *attr_name ) { #if PYTHON_VERSION < 300 if ( PyInstance_Check( source ) ) { return LOOKUP_INSTANCE( source, attr_name ); } #endif // TODO: There is heavy optimization in CPython to avoid it. Potentially // that's worth it to imitate that. PyObject *result = _PyType_Lookup( Py_TYPE( source ), attr_name ); if (likely( result )) { descrgetfunc func = Py_TYPE( result )->tp_descr_get; if ( func == NULL ) { Py_INCREF( result ); return result; } else { PyObject *func_result = func( result, source, (PyObject *)( Py_TYPE( source ) ) ); if (unlikely( func_result == NULL )) { return NULL; } CHECK_OBJECT( func_result ); return func_result; } } PyErr_SetObject( PyExc_AttributeError, attr_name ); return NULL; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/raising.h0000644000372000001440000003671413207537242024077 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_RAISING_H__ #define __NUITKA_HELPER_RAISING_H__ #if PYTHON_VERSION < 266 #define WRONG_EXCEPTION_TYPE_ERROR_MESSAGE "exceptions must be classes or instances, not %s" #elif PYTHON_VERSION < 300 #define WRONG_EXCEPTION_TYPE_ERROR_MESSAGE "exceptions must be old-style classes or derived from BaseException, not %s" #else #define WRONG_EXCEPTION_TYPE_ERROR_MESSAGE "exceptions must derive from BaseException" #endif #if PYTHON_VERSION >= 300 static void CHAIN_EXCEPTION( PyObject *exception_value ) { // Implicit chain of exception already existing. PyThreadState *thread_state = PyThreadState_GET(); // Normalize existing exception first. TODO: Will normally be done already. NORMALIZE_EXCEPTION( &thread_state->exc_type, &thread_state->exc_value, (PyTracebackObject **)&thread_state->exc_traceback ); PyObject *old_exc_value = thread_state->exc_value; if ( old_exc_value != NULL && old_exc_value != Py_None && old_exc_value != exception_value ) { PyObject *current = old_exc_value; while( true ) { PyObject *context = PyException_GetContext( current ); if (!context) break; CHECK_OBJECT( context ); Py_DECREF( context ); CHECK_OBJECT( context ); if ( context == exception_value ) { PyException_SetContext( current, NULL ); break; } current = context; } CHECK_OBJECT( old_exc_value ); Py_INCREF( old_exc_value ); PyException_SetContext( exception_value, old_exc_value ); CHECK_OBJECT( thread_state->exc_traceback ); PyException_SetTraceback( old_exc_value, thread_state->exc_traceback ); } } #endif NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_WITH_TYPE( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { *exception_value = NULL; *exception_tb = NULL; #if PYTHON_VERSION < 300 // Next, repeatedly, replace a tuple exception with its first item while( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 ) { PyObject *tmp = *exception_type; *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); Py_INCREF( *exception_type ); Py_DECREF( tmp ); } #endif if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 270 if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)*exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); Py_DECREF( *exception_type ); Py_DECREF( *exception_value ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } #endif #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); // TODO: Ever true? if ( *exception_tb ) { PyTracebackObject *prev = (PyTracebackObject *)PyException_GetTraceback( *exception_value ); if ( prev != NULL ) { assert( (*exception_tb)->tb_next == NULL ); (*exception_tb)->tb_next = prev; } PyException_SetTraceback( *exception_value, (PyObject *)(*exception_tb ? *exception_tb : (PyTracebackObject *)Py_None ) ); } *exception_tb = (PyTracebackObject *)PyException_GetTraceback( *exception_value ); #endif return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( *exception_type )->tp_name ); Py_DECREF( *exception_type ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } #if PYTHON_VERSION >= 300 NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_WITH_CAUSE( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb, PyObject *exception_cause ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( exception_cause ); *exception_tb = NULL; #if PYTHON_VERSION >= 330 // None is not a cause. if ( exception_cause == Py_None ) { Py_DECREF( exception_cause ); exception_cause = NULL; } else #endif if ( PyExceptionClass_Check( exception_cause ) ) { PyObject *old_exception_cause = exception_cause; exception_cause = PyObject_CallObject( exception_cause, NULL ); Py_DECREF( old_exception_cause ); if (unlikely( exception_cause == NULL )) { Py_DECREF( *exception_type ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } #if PYTHON_VERSION >= 330 if (unlikely( exception_cause != NULL && !PyExceptionInstance_Check( exception_cause ) )) #else if (unlikely( !PyExceptionInstance_Check( exception_cause ) )) #endif { Py_DECREF( *exception_type ); Py_XDECREF( *exception_tb ); Py_XDECREF( exception_cause ); #ifdef _NUITKA_FULL_COMPAT PyErr_Format( PyExc_TypeError, "exception causes must derive from BaseException" ); #else PyErr_Format( PyExc_TypeError, "exception causes must derive from BaseException (%s does not)", Py_TYPE( exception_cause )->tp_name ); #endif FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_DECREF( *exception_tb ); Py_XDECREF( exception_cause ); PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } PyException_SetCause( *exception_value, exception_cause ); #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); PyException_SetCause( *exception_value, exception_cause ); #if PYTHON_VERSION >= 300 CHAIN_EXCEPTION( *exception_value ); #endif return; } else { Py_DECREF( *exception_type ); Py_XDECREF( exception_cause ); PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } #endif NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_WITH_VALUE( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( *exception_value ); *exception_tb = NULL; // Non-empty tuple exceptions are the first element. while (unlikely( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 )) { *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); } if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 270 if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)*exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { if (unlikely( *exception_value != NULL && *exception_value != Py_None )) { PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } // The type is rather a value, so we are overriding it here. *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } NUITKA_MAY_BE_UNUSED static void RAISE_EXCEPTION_IMPLICIT( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( *exception_value ); *exception_tb = NULL; // Non-empty tuple exceptions are the first element. while (unlikely( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 )) { *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); } if ( PyExceptionClass_Check( *exception_type ) ) { #if PYTHON_VERSION >= 340 NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); CHAIN_EXCEPTION( *exception_value ); #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { #if PYTHON_VERSION >= 340 CHAIN_EXCEPTION( *exception_value ); #endif // The type is rather a value, so we are overriding it here. *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 340 CHAIN_EXCEPTION( *exception_value ); #endif return; } } NUITKA_MAY_BE_UNUSED static inline void RAISE_EXCEPTION_WITH_TRACEBACK( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { CHECK_OBJECT( *exception_type ); CHECK_OBJECT( *exception_value ); if ( *exception_tb == (PyTracebackObject *)Py_None ) { Py_DECREF( *exception_tb ); *exception_tb = NULL; } // Non-empty tuple exceptions are the first element. while (unlikely( PyTuple_Check( *exception_type ) && PyTuple_Size( *exception_type ) > 0 )) { *exception_type = PyTuple_GET_ITEM( *exception_type, 0 ); } if ( PyExceptionClass_Check( *exception_type ) ) { NORMALIZE_EXCEPTION( exception_type, exception_value, exception_tb ); #if PYTHON_VERSION >= 270 if (unlikely( !PyExceptionInstance_Check( *exception_value ) )) { PyErr_Format( PyExc_TypeError, "calling %s() should have returned an instance of BaseException, not '%s'", ((PyTypeObject *)*exception_type)->tp_name, Py_TYPE( *exception_value )->tp_name ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); } #endif return; } else if ( PyExceptionInstance_Check( *exception_type ) ) { if (unlikely( *exception_value != NULL && *exception_value != Py_None )) { PyErr_Format( PyExc_TypeError, "instance exception may not have a separate value" ); Py_DECREF( *exception_type ); Py_XDECREF( *exception_value ); Py_XDECREF( *exception_tb ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } // The type is rather a value, so we are overriding it here. *exception_value = *exception_type; *exception_type = PyExceptionInstance_Class( *exception_type ); Py_INCREF( *exception_type ); return; } else { PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); return; } } NUITKA_MAY_BE_UNUSED static bool RERAISE_EXCEPTION( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { PyThreadState *tstate = PyThreadState_GET(); assert( tstate ); *exception_type = tstate->exc_type != NULL ? tstate->exc_type : Py_None; Py_INCREF( *exception_type ); *exception_value = tstate->exc_value; Py_XINCREF( *exception_value ); *exception_tb = (PyTracebackObject *)tstate->exc_traceback; Py_XINCREF( *exception_tb ); CHECK_OBJECT( *exception_type ); if ( *exception_type == Py_None ) { #if PYTHON_VERSION >= 300 Py_DECREF( *exception_type ); Py_INCREF( PyExc_RuntimeError ); *exception_type = PyExc_RuntimeError; *exception_value = PyUnicode_FromString( "No active exception to reraise" ); *exception_tb = NULL; #else PyErr_Format( PyExc_TypeError, WRONG_EXCEPTION_TYPE_ERROR_MESSAGE, Py_TYPE( *exception_type )->tp_name ); FETCH_ERROR_OCCURRED( exception_type, exception_value, exception_tb ); #endif return false; } return true; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/rangeobjects.h0000644000372000001440000000415013112214770025067 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_RANGEOBJECTS_H__ #define __NUITKA_HELPER_RANGEOBJECTS_H__ /* For built-in built-in range() functionality. */ extern PyObject *BUILTIN_RANGE3( PyObject *low, PyObject *high, PyObject *step ); extern PyObject *BUILTIN_RANGE2( PyObject *low, PyObject *high ); extern PyObject *BUILTIN_RANGE( PyObject *boundary ); /* For built-in built-in xrange() functionality. */ extern PyObject *BUILTIN_XRANGE1( PyObject *high ); extern PyObject *BUILTIN_XRANGE2( PyObject *low, PyObject *high ); extern PyObject *BUILTIN_XRANGE3( PyObject *low, PyObject *high, PyObject *step ); #if PYTHON_VERSION >= 300 /* Python3 range objects */ struct _rangeobject3 { PyObject_HEAD PyObject *start; PyObject *stop; PyObject *step; PyObject *length; } ; NUITKA_MAY_BE_UNUSED static PyObject *PyRange_Start( PyObject *range ) { return ((struct _rangeobject3 *)range)->start; } NUITKA_MAY_BE_UNUSED static PyObject *PyRange_Stop( PyObject *range ) { return ((struct _rangeobject3 *)range)->stop; } NUITKA_MAY_BE_UNUSED static PyObject *PyRange_Step( PyObject *range ) { return ((struct _rangeobject3 *)range)->step; } #else struct _rangeobject2 { PyObject_HEAD long start; long step; long len; }; extern PyObject *MAKE_XRANGE( long start, long stop, long step ); #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/dictionaries.h0000644000372000001440000002341113134660221025077 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_DICTIONARIES_H__ #define __NUITKA_DICTIONARIES_H__ static inline Py_ssize_t DICT_SIZE( PyObject *dict ) { CHECK_OBJECT( dict ); return ((PyDictObject *)dict)->ma_used; } static inline PyDictObject *MODULE_DICT( PyObject *module ) { PyDictObject *dict = (PyDictObject *)(((PyModuleObject *)module)->md_dict); return dict; } #if PYTHON_VERSION < 330 // Quick dictionary lookup for a string value. typedef PyDictEntry *Nuitka_DictEntryHandle; static PyDictEntry *GET_STRING_DICT_ENTRY( PyDictObject *dict, Nuitka_StringObject *key ) { assert( PyDict_CheckExact( dict ) ); assert( Nuitka_String_CheckExact( key ) ); #if PYTHON_VERSION < 300 Py_hash_t hash = key->ob_shash; #else Py_hash_t hash = key->hash; #endif // Only improvement would be to identify how to ensure that the hash is // computed already. Calling hash early on could do that potentially. if ( hash == -1 ) { #if PYTHON_VERSION < 300 hash = PyString_Type.tp_hash( (PyObject *)key ); key->ob_shash = hash; #else hash = PyUnicode_Type.tp_hash( (PyObject *)key ); key->hash = hash; #endif } PyDictEntry *entry = dict->ma_lookup( dict, (PyObject *)key, hash ); // The "entry" cannot be NULL, it can only be empty for a string dict // lookup, but at least assert it. assert( entry != NULL ); return entry; } NUITKA_MAY_BE_UNUSED static PyObject *GET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle ) { return handle->me_value; } NUITKA_MAY_BE_UNUSED static void SET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle, PyObject *value ) { handle->me_value = value; } static PyObject *GET_STRING_DICT_VALUE( PyDictObject *dict, Nuitka_StringObject *key ) { return GET_STRING_DICT_ENTRY( dict, key )->me_value; } #else // Python 3.3 or higher. // Quick dictionary lookup for a string value. typedef struct { /* Cached hash code of me_key. */ Py_hash_t me_hash; PyObject *me_key; PyObject *me_value; /* This field is only meaningful for combined tables */ } PyDictKeyEntry; #if PYTHON_VERSION < 360 typedef PyDictKeyEntry *(*dict_lookup_func)( PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject ***value_addr ); #else typedef Py_ssize_t (*dict_lookup_func)( PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject ***value_addr, Py_ssize_t *hashpos ); #endif // Taken from CPython3.3 "Objects/dictobject.c", lives in "Objects/dict-common.h" later struct _dictkeysobject { Py_ssize_t dk_refcnt; Py_ssize_t dk_size; dict_lookup_func dk_lookup; Py_ssize_t dk_usable; #if PYTHON_VERSION < 360 PyDictKeyEntry dk_entries[1]; #else Py_ssize_t dk_nentries; union { int8_t as_1[8]; int16_t as_2[4]; int32_t as_4[2]; #if SIZEOF_VOID_P > 4 int64_t as_8[1]; #endif } dk_indices; #endif }; // Taken from Objects/dictobject.c of CPython 3.6 #if PYTHON_VERSION >= 360 #define DK_SIZE(dk) ((dk)->dk_size) #if SIZEOF_VOID_P > 4 #define DK_IXSIZE(dk) \ (DK_SIZE(dk) <= 0xff ? \ 1 : DK_SIZE(dk) <= 0xffff ? \ 2 : DK_SIZE(dk) <= 0xffffffff ? \ 4 : sizeof(int64_t)) #else #define DK_IXSIZE(dk) \ (DK_SIZE(dk) <= 0xff ? \ 1 : DK_SIZE(dk) <= 0xffff ? \ 2 : sizeof(int32_t)) #endif #define DK_ENTRIES(dk) \ ((PyDictKeyEntry *)(&(dk)->dk_indices.as_1[DK_SIZE(dk) * DK_IXSIZE(dk)])) #define DK_USABLE_FRACTION(n) (((n) << 1)/3) #endif typedef PyObject **Nuitka_DictEntryHandle; static Nuitka_DictEntryHandle GET_STRING_DICT_ENTRY( PyDictObject *dict, Nuitka_StringObject *key ) { assert( PyDict_CheckExact( dict ) ); assert( Nuitka_String_CheckExact( key ) ); Py_hash_t hash = key->_base._base.hash; // Only improvement would be to identify how to ensure that the hash is computed // already. Calling hash early on could do that potentially. if ( hash == -1 ) { hash = PyUnicode_Type.tp_hash( (PyObject *)key ); key->_base._base.hash = hash; } PyObject **value_addr; #if PYTHON_VERSION < 360 PyDictKeyEntry *entry = dict->ma_keys->dk_lookup( dict, (PyObject *)key, hash, &value_addr ); // The "entry" cannot be NULL, it can only be empty for a string dict lookup, but at // least assert it. assert( entry != NULL ); #else // TODO: Find out what the returned Py_ssize_t "ix" might be good for. dict->ma_keys->dk_lookup( dict, (PyObject *)key, hash, &value_addr, NULL // hashpos, TODO: Find out what we could use it for. ); #endif return value_addr; } NUITKA_MAY_BE_UNUSED static PyObject *GET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle ) { return *handle; } NUITKA_MAY_BE_UNUSED static void SET_DICT_ENTRY_VALUE( Nuitka_DictEntryHandle handle, PyObject *value ) { *handle = value; } NUITKA_MAY_BE_UNUSED static PyObject *GET_STRING_DICT_VALUE( PyDictObject *dict, Nuitka_StringObject *key ) { Nuitka_DictEntryHandle handle = GET_STRING_DICT_ENTRY( dict, key ); #if PYTHON_VERSION >= 360 if ( handle == NULL ) { return NULL; } #endif return GET_DICT_ENTRY_VALUE( handle ); } #endif NUITKA_MAY_BE_UNUSED static bool DICT_SET_ITEM( PyObject *dict, PyObject *key, PyObject *value ) { CHECK_OBJECT( dict ); CHECK_OBJECT( key ); CHECK_OBJECT( value ); int status = PyDict_SetItem( dict, key, value ); if (unlikely( status != 0 )) { return false; } return true; } NUITKA_MAY_BE_UNUSED static bool DICT_REMOVE_ITEM( PyObject *dict, PyObject *key ) { int status = PyDict_DelItem( dict, key ); if (unlikely( status == -1 )) { return false; } return true; } NUITKA_MAY_BE_UNUSED static PyObject *DICT_GET_ITEM( PyObject *dict, PyObject *key ) { CHECK_OBJECT( dict ); assert( PyDict_CheckExact( dict ) ); CHECK_OBJECT( key ); PyObject *result = PyDict_GetItem( dict, key ); if ( result == NULL ) { if (unlikely( PyErr_Occurred() )) { return NULL; } /* Wrap all kinds of tuples, because normalization will later unwrap * it, but then that changes the key for the KeyError, which is not * welcome. The check is inexact, as the unwrapping one is too. */ if ( PyTuple_Check( key ) ) { PyObject *tuple = PyTuple_Pack( 1, key ); PyErr_SetObject( PyExc_KeyError, tuple ); Py_DECREF( tuple ); } else { PyErr_SetObject( PyExc_KeyError, key ); } return NULL; } else { Py_INCREF( result ); return result; } } // Convert to dictionary, helper for built-in "dict" mainly. NUITKA_MAY_BE_UNUSED static PyObject *TO_DICT( PyObject *seq_obj, PyObject *dict_obj ) { PyObject *result = PyDict_New(); if ( seq_obj != NULL ) { int res; if ( PyObject_HasAttrString( seq_obj, "keys" ) ) { res = PyDict_Merge( result, seq_obj, 1 ); } else { res = PyDict_MergeFromSeq2( result, seq_obj, 1 ); } if ( res == -1 ) { return NULL; } } if ( dict_obj != NULL ) { int res = PyDict_Merge( result, dict_obj, 1 ); if ( res == -1 ) { return NULL; } } return result; } NUITKA_MAY_BE_UNUSED static void UPDATE_STRING_DICT0( PyDictObject *dict, Nuitka_StringObject *key, PyObject *value ) { Nuitka_DictEntryHandle entry = GET_STRING_DICT_ENTRY( dict, key ); #if PYTHON_VERSION >= 360 if ( entry == NULL ) { DICT_SET_ITEM( (PyObject *)dict, (PyObject *)key, value ); return; } #endif PyObject *old = GET_DICT_ENTRY_VALUE( entry ); // Values are more likely (more often) set than not set, in that case // speculatively try the quickest access method. if (likely( old != NULL )) { Py_INCREF( value ); SET_DICT_ENTRY_VALUE( entry, value ); CHECK_OBJECT( old ); Py_DECREF( old ); } else { DICT_SET_ITEM( (PyObject *)dict, (PyObject *)key, value ); } } NUITKA_MAY_BE_UNUSED static void UPDATE_STRING_DICT1( PyDictObject *dict, Nuitka_StringObject *key, PyObject *value ) { Nuitka_DictEntryHandle entry = GET_STRING_DICT_ENTRY( dict, key ); #if PYTHON_VERSION >= 360 if ( entry == NULL ) { DICT_SET_ITEM( (PyObject *)dict, (PyObject *)key, value ); Py_DECREF( value ); return; } #endif PyObject *old = GET_DICT_ENTRY_VALUE( entry ); // Values are more likely (more often) set than not set, in that case // speculatively try the quickest access method. if (likely( old != NULL )) { SET_DICT_ENTRY_VALUE( entry, value ); Py_DECREF( old ); } else { DICT_SET_ITEM( (PyObject *)dict, (PyObject *)key, value ); Py_DECREF( value ); } } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/operations.h0000644000372000001440000010040513122472300024600 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_OPERATIONS_H__ #define __NUITKA_OPERATIONS_H__ #if PYTHON_VERSION < 300 #define NEW_STYLE_NUMBER( o ) PyType_HasFeature( Py_TYPE( o ), Py_TPFLAGS_CHECKTYPES ) #else #define NEW_STYLE_NUMBER( o ) ( true ) #endif typedef PyObject *(unary_api)( PyObject * ); NUITKA_MAY_BE_UNUSED static PyObject *UNARY_OPERATION( unary_api api, PyObject *operand ) { CHECK_OBJECT( operand ); PyObject *result = api( operand ); if (unlikely( result == NULL )) { return NULL; } CHECK_OBJECT( result ); return result; } typedef PyObject *(binary_api)( PyObject *, PyObject * ); NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION( binary_api api, PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); PyObject *result = api( operand1, operand2 ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool BINARY_OPERATION_INPLACE( binary_api api, PyObject **operand1, PyObject *operand2 ) { assert( operand1 ); CHECK_OBJECT( *operand1 ); CHECK_OBJECT( operand2 ); // TODO: There is not really much point in these things. PyObject *result = BINARY_OPERATION( api, *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } // We got an object handed, that we have to release. Py_DECREF( *operand1 ); // That's our return value then. As we use a dedicated variable, it's // OK that way. *operand1 = result; return true; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_ADD( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); #if PYTHON_VERSION < 300 // Something similar for Python3 should exist too. if ( PyInt_CheckExact( operand1 ) && PyInt_CheckExact( operand2 ) ) { long a, b, i; a = PyInt_AS_LONG( operand1 ); b = PyInt_AS_LONG( operand2 ); i = a + b; // Detect overflow, in which case, a "long" object would have to be // created, which we won't handle here. if (likely(!( (i^a) < 0 && (i^b) < 0 ) )) { return PyInt_FromLong( i ); } } #endif binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_add; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_add; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_add; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif // Special case for "+", also works as sequence concat. PySequenceMethods *seq_methods = Py_TYPE( operand1 )->tp_as_sequence; if ( seq_methods && seq_methods->sq_concat ) { PyObject *result = (*seq_methods->sq_concat)( operand1, operand2 ); if (unlikely( result == NULL )) { return NULL; } return result; } PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for +: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } #if PYTHON_VERSION < 300 #include #define PyStringObject_SIZE ( offsetof( PyStringObject, ob_sval ) + 1 ) NUITKA_MAY_BE_UNUSED static bool STRING_RESIZE( PyObject **value, Py_ssize_t newsize ) { PyStringObject *sv; _Py_DEC_REFTOTAL; _Py_ForgetReference( *value ); *value = (PyObject *)PyObject_REALLOC( (char *)*value, PyStringObject_SIZE + newsize ); if (unlikely( *value == NULL )) { PyErr_NoMemory(); return false; } _Py_NewReference( *value ); sv = (PyStringObject *)*value; Py_SIZE( sv ) = newsize; sv->ob_sval[newsize] = '\0'; sv->ob_shash = -1; return true; } NUITKA_MAY_BE_UNUSED static bool STRING_ADD_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { assert( PyString_CheckExact( *operand1 ) ); assert( !PyString_CHECK_INTERNED( *operand1 ) ); assert( PyString_CheckExact( operand2 ) ); Py_ssize_t operand1_size = PyString_GET_SIZE( *operand1 ); Py_ssize_t operand2_size = PyString_GET_SIZE( operand2 ); Py_ssize_t new_size = operand1_size + operand2_size; if (unlikely( new_size < 0 )) { PyErr_Format( PyExc_OverflowError, "strings are too large to concat" ); return false; } if (unlikely( STRING_RESIZE( operand1, new_size ) == false )) { return false; } memcpy( PyString_AS_STRING( *operand1 ) + operand1_size, PyString_AS_STRING( operand2 ), operand2_size ); return true; } #else NUITKA_MAY_BE_UNUSED static bool UNICODE_ADD_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { #if PYTHON_VERSION < 330 Py_ssize_t operand1_size = PyUnicode_GET_SIZE( *operand1 ); Py_ssize_t operand2_size = PyUnicode_GET_SIZE( operand2 ); Py_ssize_t new_size = operand1_size + operand2_size; if (unlikely( new_size < 0 )) { PyErr_Format( PyExc_OverflowError, "strings are too large to concat" ); return false; } if (unlikely( PyUnicode_Resize( operand1, new_size ) != 0 )) { return false; } /* copy 'w' into the newly allocated area of 'v' */ memcpy( PyUnicode_AS_UNICODE( *operand1 ) + operand1_size, PyUnicode_AS_UNICODE( operand2 ), operand2_size * sizeof( Py_UNICODE ) ); return true; #else PyUnicode_Append( operand1, operand2 ); return !ERROR_OCCURRED(); #endif } #endif static bool FLOAT_ADD_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { assert( PyFloat_CheckExact( *operand1 ) ); assert( PyFloat_CheckExact( operand2 ) ); PyFPE_START_PROTECT("add", return false); PyFloat_AS_DOUBLE( *operand1 ) += PyFloat_AS_DOUBLE( operand2 ); PyFPE_END_PROTECT( *operand1 ); return true; } static bool FLOAT_MUL_INCREMENTAL( PyObject **operand1, PyObject *operand2 ) { assert( PyFloat_CheckExact( *operand1 ) ); assert( PyFloat_CheckExact( operand2 ) ); PyFPE_START_PROTECT("mul", return false); PyFloat_AS_DOUBLE( *operand1 ) *= PyFloat_AS_DOUBLE( operand2 ); PyFPE_END_PROTECT( *operand1 ); return true; } NUITKA_MAY_BE_UNUSED static bool BINARY_OPERATION_ADD_INPLACE( PyObject **operand1, PyObject *operand2 ) { assert( operand1 ); CHECK_OBJECT( *operand1 ); CHECK_OBJECT( operand2 ); #if PYTHON_VERSION < 300 // Something similar for Python3 should exist too. if ( PyInt_CheckExact( *operand1 ) && PyInt_CheckExact( operand2 ) ) { long a, b, i; a = PyInt_AS_LONG( *operand1 ); b = PyInt_AS_LONG( operand2 ); i = a + b; // Detect overflow, in which case, a "long" object would have to be // created, which we won't handle here. TODO: Add an else for that // case. if (likely(!( (i^a) < 0 && (i^b) < 0 ) )) { PyObject *result = PyInt_FromLong( i ); Py_DECREF( *operand1 ); *operand1 = result; return true; } } #endif #if PYTHON_VERSION < 300 if ( Py_REFCNT( *operand1 ) == 1 ) { // We more or less own the operand, so we might re-use its storage and // execute stuff in-place. if ( PyString_CheckExact( *operand1 ) && !PyString_CHECK_INTERNED( *operand1 ) && PyString_CheckExact( operand2 ) ) { return STRING_ADD_INCREMENTAL( operand1, operand2 ); } else if ( PyFloat_CheckExact( *operand1 ) && PyFloat_CheckExact( operand2 ) ) { return FLOAT_ADD_INCREMENTAL( operand1, operand2 ); } } // Strings are to be treated differently. if ( PyString_CheckExact( *operand1 ) && PyString_CheckExact( operand2 ) ) { PyString_Concat( operand1, operand2 ); return !ERROR_OCCURRED(); } #else if ( Py_REFCNT( *operand1 ) == 1 ) { // We more or less own the operand, so we might re-use its storage and // execute stuff in-place. if ( PyUnicode_CheckExact( *operand1 ) && !PyUnicode_CHECK_INTERNED( *operand1 ) && PyUnicode_CheckExact( operand2 ) ) { return UNICODE_ADD_INCREMENTAL( operand1, operand2 ); } else if ( PyFloat_CheckExact( *operand1 ) && PyFloat_CheckExact( operand2 ) ) { return FLOAT_ADD_INCREMENTAL( operand1, operand2 ); } } // Strings are to be treated differently. if ( PyUnicode_CheckExact( *operand1 ) && PyUnicode_CheckExact( operand2 ) ) { PyObject *result = PyUnicode_Concat( *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } Py_DECREF( *operand1 ); *operand1 = result; return true; } #endif PyObject *result = PyNumber_InPlaceAdd( *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } // We got an object handed, that we have to release. Py_DECREF( *operand1 ); // That's our return value then. As we use a dedicated variable, it's // OK that way. *operand1 = result; return true; } NUITKA_MAY_BE_UNUSED static bool BINARY_OPERATION_MUL_INPLACE( PyObject **operand1, PyObject *operand2 ) { assert( operand1 ); CHECK_OBJECT( *operand1 ); CHECK_OBJECT( operand2 ); if ( Py_REFCNT( *operand1 ) == 1 ) { if ( PyFloat_CheckExact( *operand1 ) && PyFloat_CheckExact( operand2 ) ) { return FLOAT_MUL_INCREMENTAL( operand1, operand2 ); } } PyObject *result = PyNumber_InPlaceMultiply( *operand1, operand2 ); if (unlikely( result == NULL )) { return false; } // We got an object handed, that we have to release. Py_DECREF( *operand1 ); // That's our return value then. As we use a dedicated variable, it's // OK that way. *operand1 = result; return true; } static PyObject *SEQUENCE_REPEAT( ssizeargfunc repeatfunc, PyObject *seq, PyObject *n ) { if (unlikely( !PyIndex_Check( n ) )) { PyErr_Format( PyExc_TypeError, "can't multiply sequence by non-int of type '%s'", Py_TYPE( n )->tp_name ); return NULL; } PyObject *index_value = PyNumber_Index( n ); if (unlikely( index_value == NULL )) { return NULL; } /* We're done if PyInt_AsSsize_t() returns without error. */ #if PYTHON_VERSION < 300 Py_ssize_t count = PyInt_AsSsize_t( index_value ); #else Py_ssize_t count = PyLong_AsSsize_t( index_value ); #endif Py_DECREF( index_value ); if (unlikely( count == -1 )) // Note: -1 is an unlikely repetition count { PyObject *exception = GET_ERROR_OCCURRED(); if (unlikely( exception )) { if ( !EXCEPTION_MATCH_BOOL_SINGLE( exception, PyExc_OverflowError ) ) { return NULL; } PyErr_Format( PyExc_OverflowError, "cannot fit '%s' into an index-sized integer", Py_TYPE( n )->tp_name ); return NULL; } } PyObject *result = (*repeatfunc)( seq, count ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_MUL( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_multiply; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_multiply; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_multiply; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif // Special case for "+", also works as sequence concat. PySequenceMethods *seq_methods1 = Py_TYPE( operand1 )->tp_as_sequence; PySequenceMethods *seq_methods2 = Py_TYPE( operand2 )->tp_as_sequence; if ( seq_methods1 != NULL && seq_methods1->sq_repeat ) { return SEQUENCE_REPEAT( seq_methods1->sq_repeat, operand1, operand2 ); } if ( seq_methods2 != NULL && seq_methods2->sq_repeat ) { return SEQUENCE_REPEAT( seq_methods2->sq_repeat, operand2, operand1 ); } PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for *: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_SUB( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_subtract; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_subtract; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_subtract; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for -: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } #if PYTHON_VERSION < 300 NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_DIV( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_divide; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_divide; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_divide; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for /: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } #endif NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_FLOORDIV( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_floor_divide; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_floor_divide; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_floor_divide; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for //: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_TRUEDIV( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_true_divide; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_true_divide; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_true_divide; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for /: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } NUITKA_MAY_BE_UNUSED static PyObject *BINARY_OPERATION_REMAINDER( PyObject *operand1, PyObject *operand2 ) { CHECK_OBJECT( operand1 ); CHECK_OBJECT( operand2 ); binaryfunc slot1 = NULL; binaryfunc slot2 = NULL; PyTypeObject *type1 = Py_TYPE( operand1 ); PyTypeObject *type2 = Py_TYPE( operand2 ); if ( type1->tp_as_number != NULL && NEW_STYLE_NUMBER( operand1 ) ) { slot1 = type1->tp_as_number->nb_remainder; } if ( type1 != type2 ) { if ( type2->tp_as_number != NULL && NEW_STYLE_NUMBER( operand2 ) ) { slot2 = type2->tp_as_number->nb_remainder; if ( slot1 == slot2 ) { slot2 = NULL; } } } if ( slot1 != NULL ) { if ( slot2 && PyType_IsSubtype( type2, type1 ) ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); slot2 = NULL; } PyObject *x = slot1( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } if ( slot2 != NULL ) { PyObject *x = slot2( operand1, operand2 ); if ( x != Py_NotImplemented ) { if (unlikely( x == NULL )) { return NULL; } return x; } Py_DECREF( x ); } #if PYTHON_VERSION < 300 if ( !NEW_STYLE_NUMBER( operand1 ) || !NEW_STYLE_NUMBER( operand2 ) ) { int err = PyNumber_CoerceEx( &operand1, &operand2 ); if ( err < 0 ) { return NULL; } if ( err == 0 ) { PyNumberMethods *mv = Py_TYPE( operand1 )->tp_as_number; if ( mv ) { binaryfunc slot = mv->nb_remainder; if ( slot != NULL ) { PyObject *x = slot( operand1, operand2 ); Py_DECREF( operand1 ); Py_DECREF( operand2 ); if (unlikely( x == NULL )) { return NULL; } return x; } } // CoerceEx did that Py_DECREF( operand1 ); Py_DECREF( operand2 ); } } #endif PyErr_Format( PyExc_TypeError, "unsupported operand type(s) for %%: '%s' and '%s'", type1->tp_name, type2->tp_name ); return NULL; } NUITKA_MAY_BE_UNUSED static PyObject *POWER_OPERATION( PyObject *operand1, PyObject *operand2 ) { PyObject *result = PyNumber_Power( operand1, operand2, Py_None ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static PyObject *POWER_OPERATION2( PyObject *operand1, PyObject *operand2 ) { PyObject *result = PyNumber_InPlacePower( operand1, operand2, Py_None ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool POWER_OPERATION_INPLACE( PyObject **operand1, PyObject *operand2 ) { PyObject *result = PyNumber_InPlacePower( *operand1, operand2, Py_None ); if (unlikely( result == NULL )) { return false; } if ( result != *operand1 ) { Py_DECREF( *operand1 ); *operand1 = result; } return true; } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/slices.h0000644000372000001440000002012613112214770023704 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_SLICES_H__ #define __NUITKA_HELPER_SLICES_H__ #if PYTHON_VERSION < 300 // Note: It appears that Python3 has no index slicing operations anymore, but // uses slice objects all the time. That's fine and make sure we adhere to it by // guarding the presence of the helpers. static inline bool IS_INDEXABLE( PyObject *value ) { return value == Py_None || #if PYTHON_VERSION < 300 PyInt_Check( value ) || #endif PyLong_Check( value ) || PyIndex_Check( value ); } static Py_ssize_t CONVERT_TO_INDEX( PyObject *value ) { CHECK_OBJECT( value ); if ( PyInt_Check( value ) ) { return PyInt_AS_LONG( value ); } else if ( PyIndex_Check( value ) ) { return PyNumber_AsSsize_t( value, NULL ); } else { PyErr_Format( PyExc_TypeError, "slice indices must be integers or None or have an __index__ method" ); return -1; } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_SLICE( PyObject *source, PyObject *lower, PyObject *upper ) { CHECK_OBJECT( source ); CHECK_OBJECT( lower ); CHECK_OBJECT( upper ); PySequenceMethods *tp_as_sequence = Py_TYPE( source )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_slice && IS_INDEXABLE( lower ) && IS_INDEXABLE( upper ) ) { Py_ssize_t ilow = 0; if ( lower != Py_None ) { ilow = CONVERT_TO_INDEX( lower ); if ( ilow == -1 && ERROR_OCCURRED() ) { return NULL; } } Py_ssize_t ihigh = PY_SSIZE_T_MAX; if ( upper != Py_None ) { ihigh = CONVERT_TO_INDEX( upper ); if ( ihigh == -1 && ERROR_OCCURRED() ) { return NULL; } } PyObject *result = PySequence_GetSlice( source, ilow, ihigh ); if (unlikely( result == NULL )) { return NULL; } return result; } else { PyObject *slice = PySlice_New( lower, upper, NULL ); if (unlikely( slice == NULL )) { return NULL; } PyObject *result = PyObject_GetItem( source, slice ); Py_DECREF( slice ); if (unlikely( result == NULL )) { return NULL; } return result; } } NUITKA_MAY_BE_UNUSED static PyObject *LOOKUP_INDEX_SLICE( PyObject *source, Py_ssize_t lower, Py_ssize_t upper ) { CHECK_OBJECT( source ); PyObject *result = PySequence_GetSlice( source, lower, upper ); if (unlikely( result == NULL )) { return NULL; } return result; } NUITKA_MAY_BE_UNUSED static bool SET_SLICE( PyObject *target, PyObject *lower, PyObject *upper, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( lower ); CHECK_OBJECT( upper ); CHECK_OBJECT( value ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice && IS_INDEXABLE( lower ) && IS_INDEXABLE( upper ) ) { Py_ssize_t lower_int = 0; if ( lower != Py_None ) { lower_int = CONVERT_TO_INDEX( lower ); if ( lower_int == -1 && ERROR_OCCURRED() ) { return false; } } Py_ssize_t upper_int = PY_SSIZE_T_MAX; if ( upper != Py_None ) { upper_int = CONVERT_TO_INDEX( upper ); if ( upper_int == -1 && ERROR_OCCURRED() ) { return false; } } int status = PySequence_SetSlice( target, lower_int, upper_int, value ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = PySlice_New( lower, upper, NULL ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_SetItem( target, slice, value ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool SET_INDEX_SLICE( PyObject *target, Py_ssize_t lower, Py_ssize_t upper, PyObject *value ) { CHECK_OBJECT( target ); CHECK_OBJECT( value ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice ) { int status = PySequence_SetSlice( target, lower, upper, value ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = _PySlice_FromIndices( lower, upper ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_SetItem( target, slice, value ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool DEL_SLICE( PyObject *target, PyObject *lower, PyObject *upper ) { CHECK_OBJECT( target ); CHECK_OBJECT( lower ); CHECK_OBJECT( upper ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice && IS_INDEXABLE( lower ) && IS_INDEXABLE( upper ) ) { Py_ssize_t lower_int = 0; if ( lower != Py_None ) { lower_int = CONVERT_TO_INDEX( lower ); if ( lower_int == -1 && ERROR_OCCURRED() ) { return false; } } Py_ssize_t upper_int = PY_SSIZE_T_MAX; if ( upper != Py_None ) { upper_int = CONVERT_TO_INDEX( upper ); if ( upper_int == -1 && ERROR_OCCURRED() ) { return false; } } int status = PySequence_DelSlice( target, lower_int, upper_int ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = PySlice_New( lower, upper, NULL ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_DelItem( target, slice ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } NUITKA_MAY_BE_UNUSED static bool DEL_INDEX_SLICE( PyObject *target, Py_ssize_t lower, Py_ssize_t upper ) { CHECK_OBJECT( target ); PySequenceMethods *tp_as_sequence = Py_TYPE( target )->tp_as_sequence; if ( tp_as_sequence && tp_as_sequence->sq_ass_slice ) { int status = PySequence_DelSlice( target, lower, upper ); if (unlikely( status == -1 )) { return false; } } else { PyObject *slice = _PySlice_FromIndices( lower, upper ); if (unlikely( slice == NULL )) { return false; } int status = PyObject_DelItem( target, slice ); Py_DECREF( slice ); if (unlikely( status == -1 )) { return false; } } return true; } #endif // Note: Cannot fail NUITKA_MAY_BE_UNUSED static PyObject *MAKE_SLICEOBJ3( PyObject *start, PyObject *stop, PyObject *step ) { CHECK_OBJECT( start ); CHECK_OBJECT( stop ); CHECK_OBJECT( step ); return PySlice_New( start, stop, step ); } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/helper/sequences.h0000644000372000001440000000565613112214770024430 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_HELPER_SEQUENCES_H__ #define __NUITKA_HELPER_SEQUENCES_H__ NUITKA_MAY_BE_UNUSED static PyObject *SEQUENCE_CONTAINS( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return NULL; } return BOOL_FROM( result == 1 ); } NUITKA_MAY_BE_UNUSED static PyObject *SEQUENCE_CONTAINS_NOT( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return NULL; } return BOOL_FROM( result == 0 ); } NUITKA_MAY_BE_UNUSED static bool SEQUENCE_CONTAINS_BOOL( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return false; } return result == 1; } NUITKA_MAY_BE_UNUSED static bool SEQUENCE_CONTAINS_NOT_BOOL( PyObject *element, PyObject *sequence ) { int result = PySequence_Contains( sequence, element ); if (unlikely( result == -1 )) { return false; } return result == 0; } NUITKA_MAY_BE_UNUSED static bool SEQUENCE_SETITEM( PyObject *sequence, Py_ssize_t index, PyObject *value ) { CHECK_OBJECT( sequence ); CHECK_OBJECT( value ); PySequenceMethods *sequence_methods = Py_TYPE( sequence )->tp_as_sequence; if ( sequence_methods != NULL && sequence_methods->sq_ass_item ) { if ( index < 0 ) { if ( sequence_methods->sq_length ) { Py_ssize_t length = (*sequence_methods->sq_length)( sequence ); if ( length < 0 ) { return false; } index += length; } } int res = sequence_methods->sq_ass_item( sequence, index, value ); if (unlikely( res == -1 )) { return false; } return true; } else { PyErr_Format( PyExc_TypeError, "'%s' object does not support item assignment", Py_TYPE( sequence )->tp_name ); return false; } } #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/fibers.h0000644000372000001440000000472213112214770022421 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_FIBERS_H__ #define __NUITKA_FIBERS_H__ #if defined( _WIN32 ) #include #elif defined( __OpenBSD__ ) #ifdef __cplusplus extern "C" { #endif #include "coro.h" #ifdef __cplusplus } #endif #else #include #endif typedef struct _Fiber { #if defined( _WIN32 ) LPVOID fiber; #elif defined( __OpenBSD__ ) struct coro_context coro_ctx; void *sptr; #else ucontext_t f_context; void *start_stack; #endif } Fiber; #ifdef __cplusplus extern "C" void _initFiber( Fiber *to ); extern "C" void _swapFiber( Fiber *to, Fiber *from ); extern "C" int _prepareFiber( Fiber *to, void *code, uintptr_t arg ); extern "C" void _releaseFiber( Fiber *to ); #else void _initFiber( Fiber *to ); void _swapFiber( Fiber *to, Fiber *from ); int _prepareFiber( Fiber *to, void *code, uintptr_t arg ); void _releaseFiber( Fiber *to ); #endif // Have centralized assertions as wrappers in debug mode, or directly access // the fiber implementions of a given platform. #ifdef __NUITKA_NO_ASSERT__ #define initFiber _initFiber #define swapFiber _swapFiber #define prepareFiber _prepareFiber #define releaseFiber _releaseFiber #else static inline void initFiber( Fiber *to ) { assert( to ); _initFiber( to ); } static inline void swapFiber( Fiber *to, Fiber *from ) { assert( to != NULL ); assert( from != NULL ); _swapFiber( to, from ); } static inline int prepareFiber( Fiber *to, void *code, uintptr_t arg ) { assert( to != NULL ); assert( code != NULL ); CHECK_OBJECT( (PyObject *)arg ); return _prepareFiber( to, code, arg ); } static inline void releaseFiber( Fiber *to ) { assert( to != NULL ); _releaseFiber( to ); } #endif #endif Nuitka-0.5.28.2/nuitka/build/include/nuitka/exceptions.h0000644000372000001440000003740713134660221023336 0ustar hayenusers00000000000000// Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com // // Part of "Nuitka", an optimizing Python compiler that is compatible and // integrates with CPython, but also works on its own. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // #ifndef __NUITKA_EXCEPTIONS_H__ #define __NUITKA_EXCEPTIONS_H__ // Exception helpers for generated code and compiled code helpers. // Did an error occur. NUITKA_MAY_BE_UNUSED static inline bool ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); return tstate->curexc_type != NULL; } // Get the error type occurred. NUITKA_MAY_BE_UNUSED static inline PyObject *GET_ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); return tstate->curexc_type; } // Clear error, which likely set. NUITKA_MAY_BE_UNUSED static inline void CLEAR_ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *old_type = tstate->curexc_type; PyObject *old_value = tstate->curexc_value; PyObject *old_tb = tstate->curexc_traceback; tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; Py_XDECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); } // Clear error, which is not likely set. This is about bugs from CPython, // use CLEAR_ERROR_OCCURRED is not sure. NUITKA_MAY_BE_UNUSED static inline void DROP_ERROR_OCCURRED( void ) { PyThreadState *tstate = PyThreadState_GET(); if (unlikely( tstate->curexc_type != NULL )) { PyObject *old_type = tstate->curexc_type; PyObject *old_value = tstate->curexc_value; PyObject *old_tb = tstate->curexc_traceback; tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; Py_DECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); } } // Fetch the current error into object variables. NUITKA_MAY_BE_UNUSED static void FETCH_ERROR_OCCURRED( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_traceback) { PyThreadState *tstate = PyThreadState_GET(); *exception_type = tstate->curexc_type; *exception_value = tstate->curexc_value; *exception_traceback = (PyTracebackObject *)tstate->curexc_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("FETCH_ERROR_OCCURRED:\n"); PRINT_EXCEPTION( tstate->curexc_type, tstate->curexc_value, tstate->curexc_traceback ); #endif tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; } // Fetch the current error into object variables. NUITKA_MAY_BE_UNUSED static void FETCH_ERROR_OCCURRED_UNTRACED( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_traceback) { PyThreadState *tstate = PyThreadState_GET(); *exception_type = tstate->curexc_type; *exception_value = tstate->curexc_value; *exception_traceback = (PyTracebackObject *)tstate->curexc_traceback; tstate->curexc_type = NULL; tstate->curexc_value = NULL; tstate->curexc_traceback = NULL; } NUITKA_MAY_BE_UNUSED static void RESTORE_ERROR_OCCURRED( PyObject *exception_type, PyObject *exception_value, PyTracebackObject *exception_traceback ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *old_exception_type = tstate->curexc_type; PyObject *old_exception_value = tstate->curexc_value; PyObject *old_exception_traceback = tstate->curexc_traceback; tstate->curexc_type = exception_type; tstate->curexc_value = exception_value; tstate->curexc_traceback = (PyObject *)exception_traceback; #if _DEBUG_EXCEPTIONS PRINT_STRING("RESTORE_ERROR_OCCURRED:\n"); PRINT_EXCEPTION( tstate->curexc_type, tstate->curexc_value, tstate->curexc_traceback ); #endif Py_XDECREF( old_exception_type ); Py_XDECREF( old_exception_value ); Py_XDECREF( old_exception_traceback ); } NUITKA_MAY_BE_UNUSED static void RESTORE_ERROR_OCCURRED_UNTRACED( PyObject *exception_type, PyObject *exception_value, PyTracebackObject *exception_traceback ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *old_exception_type = tstate->curexc_type; PyObject *old_exception_value = tstate->curexc_value; PyObject *old_exception_traceback = tstate->curexc_traceback; tstate->curexc_type = exception_type; tstate->curexc_value = exception_value; tstate->curexc_traceback = (PyObject *)exception_traceback; Py_XDECREF( old_exception_type ); Py_XDECREF( old_exception_value ); Py_XDECREF( old_exception_traceback ); } struct Nuitka_FrameObject; extern PyTracebackObject *MAKE_TRACEBACK( struct Nuitka_FrameObject *frame, int lineno ); // Add a frame to an existing exception trace-back. NUITKA_MAY_BE_UNUSED static PyTracebackObject *ADD_TRACEBACK( PyTracebackObject *exception_tb, struct Nuitka_FrameObject *frame, int lineno ) { PyTracebackObject *traceback_new = MAKE_TRACEBACK( frame, lineno ); traceback_new->tb_next = exception_tb; return traceback_new; } #if PYTHON_VERSION < 300 extern PyObject *const_str_plain_exc_type, *const_str_plain_exc_value, *const_str_plain_exc_traceback; #endif // Helper that sets the current thread exception, releasing the current one, for // use in this file only. NUITKA_MAY_BE_UNUSED inline static void SET_CURRENT_EXCEPTION( PyObject *exception_type, PyObject *exception_value, PyTracebackObject *exception_tb ) { PyThreadState *thread_state = PyThreadState_GET(); PyObject *old_type = thread_state->exc_type; PyObject *old_value = thread_state->exc_value; PyObject *old_tb = thread_state->exc_traceback; thread_state->exc_type = exception_type; thread_state->exc_value = exception_value; thread_state->exc_traceback = (PyObject *)exception_tb; #if _DEBUG_EXCEPTIONS PRINT_STRING("SET_CURRENT_EXCEPTION:\n"); PRINT_EXCEPTION( exception_type, exception_value, (PyObject *)exception_tb ); #endif Py_XDECREF( old_type ); Py_XDECREF( old_value ); Py_XDECREF( old_tb ); #if PYTHON_VERSION < 300 // Set sys attributes in the fastest possible way. PyObject *sys_dict = thread_state->interp->sysdict; CHECK_OBJECT( sys_dict ); PyDict_SetItem( sys_dict, const_str_plain_exc_type, exception_type ? exception_type : Py_None ); PyDict_SetItem( sys_dict, const_str_plain_exc_value, exception_value ? exception_value : Py_None ); PyDict_SetItem( sys_dict, const_str_plain_exc_traceback, exception_tb ? (PyObject *)exception_tb : Py_None ); if ( exception_type ) assert( Py_REFCNT( exception_type ) >= 2 ); if ( exception_value ) assert( Py_REFCNT( exception_value ) >= 2 ); if ( exception_tb ) assert( Py_REFCNT( exception_tb ) >= 2 ); #endif } // Preserve the current exception as the frame to restore. NUITKA_MAY_BE_UNUSED static inline void PRESERVE_FRAME_EXCEPTION( struct Nuitka_FrameObject *frame_object ) { PyFrameObject *frame = (PyFrameObject *)frame_object; // Setting exception for frame if not already done. if ( frame->f_exc_type == NULL ) { PyThreadState *thread_state = PyThreadState_GET(); if ( thread_state->exc_type != NULL && thread_state->exc_type != Py_None ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("PRESERVE_FRAME_EXCEPTION: preserve thread exception\n"); #endif frame->f_exc_type = thread_state->exc_type; Py_INCREF( frame->f_exc_type ); frame->f_exc_value = thread_state->exc_value; Py_XINCREF( frame->f_exc_value ); frame->f_exc_traceback = thread_state->exc_traceback; Py_XINCREF( frame->f_exc_traceback ); } else { #if _DEBUG_EXCEPTIONS PRINT_STRING("PRESERVE_FRAME_EXCEPTION: no exception to preserve\n"); #endif frame->f_exc_type = Py_None; Py_INCREF( frame->f_exc_type ); frame->f_exc_value = NULL; frame->f_exc_traceback = NULL; } } #if _DEBUG_EXCEPTIONS else { PRINT_STRING("PRESERVE_FRAME_EXCEPTION: already preserving\n"); } PRINT_ITEM( (PyObject *)frame_object ); PRINT_NEW_LINE(); PRINT_EXCEPTION( frame->f_exc_type, frame->f_exc_value, frame->f_exc_traceback ); #endif } // Restore a previously preserved exception to the frame. NUITKA_MAY_BE_UNUSED static inline void RESTORE_FRAME_EXCEPTION( struct Nuitka_FrameObject *frame_object ) { PyFrameObject *frame = (PyFrameObject *)frame_object; if ( frame->f_exc_type ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("RESTORE_FRAME_EXCEPTION: restoring preserved\n"); PRINT_ITEM( (PyObject *)frame_object ); PRINT_NEW_LINE(); #endif SET_CURRENT_EXCEPTION( frame->f_exc_type, frame->f_exc_value, (PyTracebackObject *)frame->f_exc_traceback ); frame->f_exc_type = NULL; frame->f_exc_value = NULL; frame->f_exc_traceback = NULL; } #if _DEBUG_EXCEPTIONS else { PRINT_STRING("RESTORE_FRAME_EXCEPTION: nothing to restore\n"); PRINT_ITEM( (PyObject *)frame_object ); PRINT_NEW_LINE(); } #endif } // Publish an exception, erasing the values of the variables. NUITKA_MAY_BE_UNUSED static inline void PUBLISH_EXCEPTION( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("PUBLISH_EXCEPTION:\n"); #endif SET_CURRENT_EXCEPTION( *exception_type, *exception_value, *exception_tb ); *exception_type = NULL; *exception_value = NULL; *exception_tb = NULL; } // Normalize an exception. NUITKA_MAY_BE_UNUSED static inline void NORMALIZE_EXCEPTION( PyObject **exception_type, PyObject **exception_value, PyTracebackObject **exception_tb ) { #if _DEBUG_EXCEPTIONS PRINT_STRING("NORMALIZE_EXCEPTION:\n"); PRINT_EXCEPTION( *exception_type, *exception_value, (PyObject *)*exception_tb ); #endif if ( *exception_type != Py_None && *exception_type != NULL ) { PyErr_NormalizeException( exception_type, exception_value, (PyObject **)exception_tb ); } #if _DEBUG_EXCEPTIONS PRINT_STRING("normalized:\n"); PRINT_EXCEPTION( *exception_type, *exception_value, (PyObject *)*exception_tb ); #endif } NUITKA_MAY_BE_UNUSED static bool EXCEPTION_MATCH_GENERATOR( PyObject *exception_value ) { CHECK_OBJECT( exception_value ); // We need to check the class. if ( PyExceptionInstance_Check( exception_value ) ) { exception_value = PyExceptionInstance_Class( exception_value ); } // Lets be optimistic. If it matches, we would be wasting our time. if ( exception_value == PyExc_GeneratorExit || exception_value == PyExc_StopIteration ) { return true; } if ( PyExceptionClass_Check( exception_value ) ) { // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); int res = PyObject_IsSubclass( exception_value, PyExc_GeneratorExit ); // This function must not fail, so print the error here */ if (unlikely( res == -1 )) { PyErr_WriteUnraisable( exception_value ); } if (res == 1) return true; res = PyObject_IsSubclass( exception_value, PyExc_StopIteration ); // This function must not fail, so print the error here */ if (unlikely( res == -1 )) { PyErr_WriteUnraisable( exception_value ); } RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); return res == 1; } return false; } NUITKA_MAY_BE_UNUSED static bool EXCEPTION_MATCH_BOOL_SINGLE( PyObject *exception_value, PyObject *exception_checked ) { CHECK_OBJECT( exception_value ); CHECK_OBJECT( exception_checked ); // We need to check the class. if ( PyExceptionInstance_Check( exception_value ) ) { exception_value = PyExceptionInstance_Class( exception_value ); } // Lets be optimistic. If it matches, we would be wasting our time. if ( exception_value == exception_checked ) { return true; } if ( PyExceptionClass_Check( exception_value ) ) { // Save the current exception, if any, we must preserve it. PyObject *save_exception_type, *save_exception_value; PyTracebackObject *save_exception_tb; FETCH_ERROR_OCCURRED( &save_exception_type, &save_exception_value, &save_exception_tb ); int res = PyObject_IsSubclass( exception_value, exception_checked ); // This function must not fail, so print the error here */ if (unlikely( res == -1 )) { PyErr_WriteUnraisable( exception_value ); } RESTORE_ERROR_OCCURRED( save_exception_type, save_exception_value, save_exception_tb ); return res == 1; } return false; } // This is for the actual comparison operation that is being done in the // node tree, no other code should use it. TODO: Then it's probably not // properly located here, and it could still in-line the code of // "PyErr_GivenExceptionMatches" to save on Python3 doing two tuple checks // and iterations. NUITKA_MAY_BE_UNUSED static inline int EXCEPTION_MATCH_BOOL( PyObject *exception_value, PyObject *exception_checked ) { CHECK_OBJECT( exception_value ); CHECK_OBJECT( exception_checked ); #if PYTHON_VERSION >= 300 /* Note: Exact matching tuples seems to needed, despite using GET_ITEM later on, this probably cannot be overloaded that deep. */ if ( PyTuple_Check( exception_checked )) { Py_ssize_t length = PyTuple_Size( exception_checked ); for ( Py_ssize_t i = 0; i < length; i += 1 ) { PyObject *element = PyTuple_GET_ITEM( exception_checked, i ); if (unlikely( !PyExceptionClass_Check( element ) )) { PyErr_Format( PyExc_TypeError, "catching classes that do not inherit from BaseException is not allowed" ); return -1; } } } else if (unlikely( !PyExceptionClass_Check( exception_checked ) )) { PyErr_Format( PyExc_TypeError, "catching classes that do not inherit from BaseException is not allowed" ); return -1; } #endif return PyErr_GivenExceptionMatches( exception_value, exception_checked ); } #if PYTHON_VERSION >= 300 // Attach the exception context if necessary. NUITKA_MAY_BE_UNUSED static inline void ADD_EXCEPTION_CONTEXT( PyObject **exception_type, PyObject **exception_value ) { PyThreadState *tstate = PyThreadState_GET(); PyObject *context = tstate->exc_value; if ( context != NULL ) { NORMALIZE_EXCEPTION( exception_type, exception_value, NULL ); Py_INCREF( context ); PyException_SetContext( *exception_value, context ); } } #endif // Special helper that checks for StopIteration and if so clears it, only // indicating if it was set. NUITKA_MAY_BE_UNUSED static bool CHECK_AND_CLEAR_STOP_ITERATION_OCCURRED( void ) { PyObject *error = GET_ERROR_OCCURRED(); if ( error == NULL ) { return true; } else if ( EXCEPTION_MATCH_BOOL_SINGLE( error, PyExc_StopIteration ) ) { CLEAR_ERROR_OCCURRED(); return true; } else { return false; } } #endif Nuitka-0.5.28.2/nuitka/build/SconsInterface.py0000644000372000001440000002042413207537242021344 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Scons interface. Interaction with scons. Find the binary, and run it with a set of given options. """ import contextlib import os import subprocess import sys from nuitka import Options, Tracing from nuitka.PythonVersions import getTargetPythonDLLPath, python_version from nuitka.utils import Execution, Utils def getSconsDataPath(): """ Return path to where data for scons lives, e.g. static C source files. """ return os.path.dirname(__file__) def _getSconsInlinePath(): """ Return path to inline copy of scons. """ return os.path.join( getSconsDataPath(), "inline_copy" ) def _getSconsBinaryCall(): """ Return a way to execute Scons. Using potentially in-line copy if no system Scons is available or if we are on Windows, there it is mandatory. """ if Utils.getOS() != "Windows": scons_path = Execution.getExecutablePath("scons") if scons_path is not None: return [ scons_path ] return [ _getPythonForSconsExePath(), os.path.join( _getSconsInlinePath(), "bin", "scons.py" ) ] def _getPythonSconsExePathWindows(): """ Find Python2 on Windows. First try a few guesses, the look into registry for user or system wide installations of Python2. Both Python 2.6 and 2.7 will do. """ # Shortcuts for the default installation directories, to avoid going to # registry at all unless necessary. Any Python2 will do for Scons, so it # might be avoided entirely. if os.path.isfile(r"c:\Python27\python.exe"): return r"c:\Python27\python.exe" elif os.path.isfile(r"c:\Python26\python.exe"): return r"c:\Python26\python.exe" elif os.path.isfile(r"c:\Python35\python.exe"): return r"c:\Python35\python.exe" elif os.path.isfile(r"c:\Python36\python.exe"): return r"c:\Python36\python.exe" # Windows only code, pylint: disable=I0021,import-error,undefined-variable try: import _winreg as winreg except ImportError: import winreg # lint:ok for search in ("2.7", "2.6", "3.5", "3.6"): for hkey_branch in (winreg.HKEY_LOCAL_MACHINE, winreg.HKEY_CURRENT_USER): for arch_key in 0, winreg.KEY_WOW64_32KEY, winreg.KEY_WOW64_64KEY: try: key = winreg.OpenKey( hkey_branch, r"SOFTWARE\Python\PythonCore\%s\InstallPath" % search, 0, winreg.KEY_READ | arch_key ) return os.path.join( winreg.QueryValue(key, ""), "python.exe" ) except WindowsError: # @UndefinedVariable pass def _getPythonForSconsExePath(): """ Find a way to call any Python2. Scons needs it as it doesn't support Python3. """ python_exe = Options.getPythonPathForScons() if python_exe is not None: return python_exe if python_version < 300 or python_version >= 350: return sys.executable elif Utils.getOS() == "Windows": python_exe = _getPythonSconsExePathWindows() if python_exe is not None: return python_exe else: sys.exit("""\ Error, while Nuitka works with Python 3.2 to 3.4, scons does not, and Nuitka needs to find a Python executable 2.6/2.7 or 3.5 or higher. Simply under the C:\\PythonXY, e.g. C:\\Python27 to execute the scons utility which is used to build the C files to binary. You may provide it using option "--python-for-scons=path_to_python.exe" in case it is not visible in registry, e.g. due to using uninstalled AnaConda Python. """) for version_candidate in ("2.7", "2.6", "3.5", "3.6"): candidate = Execution.getExecutablePath("python" + version_candidate) if candidate is not None: return candidate # Lets be optimistic, this is most often going to be new enough or a # Python2 variant. return "python" @contextlib.contextmanager def _setupSconsEnvironment(): """ Setup the scons execution environment. For the scons inline copy on Windows needs to find the library, using the "SCONS_LIB_DIR" environment variable "NUITKA_SCONS". And for the target Python we provide "NUITKA_PYTHON_DLL_PATH" to see where the Python DLL lives, in case it needs to be copied, and then the "NUITKA_PYTHON_EXE_PATH" to find the Python binary itself. """ if Utils.getOS() == "Windows": # On Windows this Scons variable must be set by us. os.environ["SCONS_LIB_DIR"] = os.path.join( _getSconsInlinePath(), "lib", "scons-2.3.2" ) # On Windows, we use the Python.DLL path for some things. We pass it # via environment variable os.environ["NUITKA_PYTHON_DLL_PATH"] = getTargetPythonDLLPath() os.environ["NUITKA_PYTHON_EXE_PATH"] = sys.executable # Remove environment variables that can only harm if we have to switch # major Python versions, these cannot help Python2 to execute scons, this # is a bit of noise, but helpful. if python_version >= 300: if "PYTHONPATH" in os.environ: old_pythonpath = os.environ["PYTHONPATH"] del os.environ["PYTHONPATH"] else: old_pythonpath = None if "PYTHONHOME" in os.environ: old_pythonhome = os.environ["PYTHONHOME"] del os.environ["PYTHONHOME"] else: old_pythonhome = None yield if python_version >= 300: if old_pythonpath is not None: os.environ["PYTHONPATH"] = old_pythonpath if old_pythonhome is not None: os.environ["PYTHONHOME"] = old_pythonhome if Utils.getOS() == "Windows": del os.environ["NUITKA_PYTHON_DLL_PATH"] del os.environ["NUITKA_PYTHON_EXE_PATH"] def _buildSconsCommand(quiet, options): """ Build the scons command to run. The options are a dictionary to be passed to scons as a command line, and other scons stuff is set. """ scons_command = _getSconsBinaryCall() if quiet: scons_command.append("--quiet") scons_command += [ # The scons file "-f", os.path.join( getSconsDataPath(), "SingleExe.scons" ), # Parallel compilation. "--jobs", str(Options.getJobLimit()), # Do not warn about deprecation from Scons "--warn=no-deprecated", # Don't load "site_scons" at all. "--no-site-dir", ] if Options.isShowScons(): scons_command.append("--debug=explain") # Python2, encoding unicode values def encode(value): if str is bytes and type(value) is unicode: return value.encode("utf8") else: return value # Option values to provide to scons. Find these in the caller. for key, value in options.items(): scons_command.append( key + '=' + encode(value) ) # Python2, make argument encoding recognizable. if str is bytes: scons_command.append( "arg_encoding=utf8" ) return scons_command def runScons(options, quiet): with _setupSconsEnvironment(): scons_command = _buildSconsCommand(quiet, options) if Options.isShowScons(): Tracing.printLine("Scons command:", ' '.join(scons_command)) Tracing.flushStdout() return subprocess.call(scons_command, shell = False) == 0 Nuitka-0.5.28.2/nuitka/Constants.py0000644000372000001440000002231613134660221017306 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Module for constants in Nuitka. This contains tools to compare, classify and test constants. """ import math from nuitka.Builtins import builtin_type_names from nuitka.PythonVersions import python_version from .__past__ import ( # pylint: disable=I0021,redefined-builtin iterItems, long, unicode, xrange ) from .Builtins import builtin_anon_names NoneType = type(None) def compareConstants(a, b): # Many many cases to deal with, pylint: disable=too-many-branches,too-many-return-statements # Supposed fast path for comparison. if type(a) is not type(b): return False # Now it's either not the same, or it is a container that contains NaN or it # is a complex or float that is NaN, the other cases can use == at the end. if type(a) is complex: return compareConstants(a.imag, b.imag) and \ compareConstants(a.real, b.real) if type(a) is float: # Check sign first, -0.0 is not 0.0, or -nan is not nan, it has a # different sign for a start. if math.copysign(1.0, a) != math.copysign(1.0, b): return False if math.isnan(a) and math.isnan(b): return True return a == b if type(a) in (tuple, list): if len(a) != len(b): return False for ea, eb in zip(a, b): if not compareConstants(ea, eb): return False return True if type(a) is dict: if len(a) != len(b): return False for ea1, ea2 in iterItems(a): for eb1, eb2 in iterItems(b): if compareConstants(ea1, eb1) and \ compareConstants(ea2, eb2): break else: return False return True if type(a) in (frozenset, set): if len(a) != len(b): return False for ea in a: if ea not in b: # Due to NaN values, we need to compare each set element with # all the other set to be really sure. for eb in b: if compareConstants(ea, eb): break else: return False return True if type(a) is xrange: return str(a) == str(b) # The NaN values of float and complex may let this fail, even if the # constants are built in the same way, therefore above checks. return a == b # These built-in type references are kind of constant too. The list should be # complete. constant_builtin_types = ( int, str, float, list, tuple, set, dict, slice, complex, xrange, NoneType, ) if python_version >= 300: constant_builtin_types += ( bytes, ) else: constant_builtin_types += ( unicode, long, # This has no name in Python, but the natural one in C-API. builtin_anon_names["instance"] ) def isConstant(constant): # Too many cases and all return, that is how we do it here, # pylint: disable=too-many-branches,too-many-return-statements constant_type = type(constant) if constant_type is dict: for key, value in iterItems(constant): if not isConstant(key): return False if not isConstant(value): return False return True elif constant_type in (tuple, list): for element_value in constant: if not isConstant(element_value): return False return True elif constant_type is slice: if not isConstant(constant.start) or \ not isConstant(constant.stop) or \ not isConstant(constant.step): return False return True elif constant_type in (str, unicode, complex, int, long, bool, float, NoneType, range, bytes, set, frozenset, xrange, bytearray): return True elif constant in (Ellipsis, NoneType): return True elif constant_type is type: return constant.__name__ in builtin_type_names else: return False def isMutable(constant): """ Is a constant mutable That means a user of a reference to it, can modify it. Strings are a prime example of mutable, dictionaries are mutable. """ # Many cases and all return, that is how we do it here, # pylint: disable=too-many-return-statements constant_type = type(constant) if constant_type in (str, unicode, complex, int, long, bool, float, NoneType, range, bytes, slice, xrange): return False elif constant_type in (dict, list, set, bytearray): return True elif constant_type is tuple: for value in constant: if isMutable(value): return True return False elif constant_type is frozenset: for value in constant: if isMutable(value): return True return False elif constant_type is type: return False elif constant is Ellipsis: return False elif constant is None: return False else: assert False, repr(constant) def isHashable(constant): """ Is a constant hashable That means a user of a reference to it, can use it for dicts and set keys. This is distinct from mutable, there is one types that is not mutable, and still not hashable: slices. """ # Many cases and all return, that is how we do it here, # pylint: disable=too-many-return-statements constant_type = type(constant) if constant_type in (str, unicode, complex, int, long, bool, float, NoneType, xrange, bytes): return True elif constant_type in (dict, list, set, slice, bytearray): return False elif constant_type is tuple: for value in constant: if not isHashable(value): return False return True elif constant_type is frozenset: for value in constant: if not isHashable(value): return False return True elif constant is Ellipsis: return True elif constant_type is type: return True else: assert False, constant_type def getUnhashableConstant(constant): # Too many cases and all return, that is how we do it here, # pylint: disable=too-many-return-statements constant_type = type(constant) if constant_type in (str, unicode, complex, int, long, bool, float, NoneType, xrange, bytes): return None elif constant_type in (dict, list, set): return constant elif constant_type is tuple: for value in constant: res = getUnhashableConstant(value) if res is not None: return res return None elif constant is Ellipsis: return None elif constant in constant_builtin_types: return None elif constant_type is slice: return None else: assert False, constant_type def isIterableConstant(constant): return type(constant) in ( str, unicode, list, tuple, set, frozenset, dict, xrange, bytes ) def getConstantIterationLength(constant): assert isIterableConstant(constant) return len(constant) def isNumberConstant(constant): return type(constant) in (int, long, float, bool) def isIndexConstant(constant): return type(constant) in (int, long, bool) def createConstantDict(keys, values): # Create it proper size immediately. constant_value = dict.fromkeys( keys, None ) for key, value in zip(keys, values): constant_value[key] = value return constant_value def getConstantWeight(constant): constant_type = type(constant) if constant_type is dict: result = 0 for key, value in iterItems(constant): result += getConstantWeight(key) result += getConstantWeight(value) return result elif constant_type in (tuple, list, set, frozenset): result = 0 for element_value in constant: result += getConstantWeight(element_value) return result else: return 1 def isCompileTimeConstantValue(value): """ Determine if a value will be usable at compile time. """ # This needs to match code in makeCompileTimeConstantReplacementNode if isConstant(value): return True elif type(value) is type: return True else: return False Nuitka-0.5.28.2/nuitka/tree/0000755000372000001440000000000013207540420015712 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/tree/Extractions.py0000644000372000001440000000426613134660221020600 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Extracting visitors. This is used for look-aheads supporting abstract execution. We need to e.g. know the variables written by a piece of code ahead of abstractly executing a loop. """ from .Operations import VisitorNoopMixin, visitTree class VariableWriteExtractor(VisitorNoopMixin): """ Extract variables written to. """ def __init__(self): self.written_to = set() def onEnterNode(self, node): if node.isStatementAssignmentVariable() or \ node.isStatementDelVariable(): self.written_to.add(node.getVariable()) def getResult(self): return self.written_to def getVariablesWritten(node): visitor = VariableWriteExtractor() visitTree(node, visitor) return visitor.getResult() class VariableUsageUpdater(VisitorNoopMixin): def __init__(self, old_variable, new_variable): self.old_variable = old_variable self.new_variable = new_variable def onEnterNode(self, node): if node.isStatementAssignmentVariable() or \ node.isStatementDelVariable() or \ node.isStatementReleaseVariable(): if node.getVariable() is self.old_variable: node.setVariable(self.new_variable) def updateVariableUsage(provider, old_variable, new_variable): visitor = VariableUsageUpdater( old_variable = old_variable, new_variable = new_variable ) visitTree(provider, visitor) Nuitka-0.5.28.2/nuitka/tree/ReformulationBooleanExpressions.py0000644000372000001440000000620013207537242024663 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of boolean and/or expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.ConditionalNodes import ( ExpressionConditionalAND, ExpressionConditionalOR ) from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from .TreeHelpers import buildNode, buildNodeList, getKind def buildBoolOpNode(provider, node, source_ref): bool_op = getKind(node.op) if bool_op == "Or": # The "or" may be short circuit and is therefore not a plain operation. values = buildNodeList(provider, node.values, source_ref) for value in values[:-1]: value.setCompatibleSourceReference(values[-1].getSourceReference()) source_ref = values[-1].getSourceReference() return buildOrNode( values = values, source_ref = source_ref ) elif bool_op == "And": # The "and" may be short circuit and is therefore not a plain operation. values = buildNodeList(provider, node.values, source_ref) for value in values[:-1]: value.setCompatibleSourceReference(values[-1].getSourceReference()) source_ref = values[-1].getSourceReference() return buildAndNode( values = values, source_ref = source_ref ) elif bool_op == "Not": # The "not" is really only a unary operation and no special. return ExpressionOperationNOT( operand = buildNode(provider, node.operand, source_ref), source_ref = source_ref ) else: assert False, bool_op def buildOrNode(values, source_ref): values = list(values) result = values.pop() # When we encounter, "or", we expect it to be at least two values. assert values while values: result = ExpressionConditionalOR( left = values.pop(), right = result, source_ref = source_ref ) return result def buildAndNode(values, source_ref): values = list(values) result = values.pop() # Unlike "or", for "and", this is used with only one value. while values: result = ExpressionConditionalAND( left = values.pop(), right = result, source_ref = source_ref ) return result Nuitka-0.5.28.2/nuitka/tree/ReformulationClasses.py0000644000372000001440000007724613207537242022460 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of class statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementAssignmentVariableName, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionBuiltinHasattr ) from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, makeExpressionBuiltinRef ) from nuitka.nodes.CallNodes import makeExpressionCall from nuitka.nodes.ClassNodes import ( ExpressionClassBody, ExpressionSelectMetaclass ) from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ComparisonNodes import ExpressionComparisonIn from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.DictionaryNodes import ( ExpressionDictOperationGet, StatementDictOperationRemove ) from nuitka.nodes.FunctionNodes import ExpressionFunctionQualnameRef from nuitka.nodes.GlobalsLocalsNodes import ( ExpressionBuiltinLocalsCopy, ExpressionBuiltinLocalsUpdated, StatementSetLocals ) from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.SubscriptNodes import ExpressionSubscriptLookup from nuitka.nodes.TryNodes import StatementTry from nuitka.nodes.TypeNodes import ExpressionBuiltinType1 from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableNameRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildFrameNode, buildNode, buildNodeList, extractDocFromBody, getKind, makeDictCreationOrConstant2, makeSequenceCreationOrConstant, makeStatementsSequence, makeStatementsSequenceFromStatement, mangleName ) def _buildClassNode3(provider, node, source_ref): # Many variables, due to the huge re-formulation that is going on here, # which just has the complexity, pylint: disable=too-many-locals # This function is the Python3 special case with special re-formulation as # according to developer manual. class_statement_nodes, class_doc = extractDocFromBody(node) # We need a scope for the temporary variables, and they might be closured. temp_scope = provider.allocateTempScope( name = "class_creation" ) tmp_bases = provider.allocateTempVariable( temp_scope = temp_scope, name = "bases" ) tmp_class_decl_dict = provider.allocateTempVariable( temp_scope = temp_scope, name = "class_decl_dict" ) tmp_metaclass = provider.allocateTempVariable( temp_scope = temp_scope, name = "metaclass" ) tmp_prepared = provider.allocateTempVariable( temp_scope = temp_scope, name = "prepared" ) class_creation_function = ExpressionClassBody( provider = provider, name = node.name, doc = class_doc, source_ref = source_ref ) class_variable = class_creation_function.getVariableForAssignment( "__class__" ) class_variable_ref = ExpressionVariableRef( variable = class_variable, source_ref = source_ref ) parent_module = provider.getParentModule() code_object = CodeObjectSpec( co_name = node.name, co_kind = "Class", co_varnames = (), co_argcount = 0, co_kwonlyargcount = 0, co_has_starlist = False, co_has_stardict = False, co_filename = parent_module.getRunTimeFilename(), co_lineno = source_ref.getLineNumber(), future_spec = parent_module.getFutureSpec() ) body = buildFrameNode( provider = class_creation_function, nodes = class_statement_nodes, code_object = code_object, source_ref = source_ref ) source_ref_orig = source_ref if body is not None: # The frame guard has nothing to tell its line number to. body.source_ref = source_ref statements = [ StatementSetLocals( new_locals = ExpressionTempVariableRef( variable = tmp_prepared, source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariableName( variable_name = "__module__", source = makeConstantRefNode( constant = provider.getParentModule().getFullName(), source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ] if class_doc is not None: statements.append( StatementAssignmentVariableName( variable_name = "__doc__", source = makeConstantRefNode( constant = class_doc, source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ) # The "__qualname__" attribute is new in Python 3.3. if python_version >= 330: qualname = class_creation_function.getFunctionQualname() qualname_variable = class_creation_function.getVariableForAssignment( "__qualname__" ) if python_version < 340: qualname_ref = makeConstantRefNode( constant = qualname, source_ref = source_ref, user_provided = True ) else: qualname_ref = ExpressionFunctionQualnameRef( function_body = class_creation_function, source_ref = source_ref, ) statements.append( StatementAssignmentVariableName( variable_name = qualname_variable.getName(), source = qualname_ref, source_ref = source_ref ) ) if python_version >= 340: qualname_assign = statements[-1] if python_version >= 360 and \ class_creation_function.needsAnnotationsDictionary(): statements.append( StatementAssignmentVariableName( variable_name = "__annotations__", source = makeConstantRefNode( constant = {}, source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ) statements.append(body) statements += [ StatementAssignmentVariableName( variable_name = "__class__", source = makeExpressionCall( called = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), args = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = ( makeConstantRefNode( constant = node.name, source_ref = source_ref, user_provided = True ), ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), ExpressionBuiltinLocalsUpdated( source_ref = source_ref ) ), source_ref = source_ref ), kw = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementReturn( expression = class_variable_ref, source_ref = source_ref ) ] body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) # The class body is basically a function that implicitly, at the end # returns its locals and cannot have other return statements contained. class_creation_function.setBody(body) class_creation_function.registerProvidedVariable(tmp_bases) class_creation_function.registerProvidedVariable(tmp_class_decl_dict) class_creation_function.registerProvidedVariable(tmp_metaclass) class_creation_function.registerProvidedVariable(tmp_prepared) # The class body is basically a function that implicitly, at the end # returns its created class and cannot have other return statements # contained. decorated_body = class_creation_function for decorator in buildNodeList( provider, reversed(node.decorator_list), source_ref ): decorated_body = makeExpressionCall( called = decorator, args = ExpressionMakeTuple( elements = ( decorated_body, ), source_ref = source_ref ), kw = None, source_ref = decorator.getSourceReference() ) statements = ( StatementAssignmentVariable( variable = tmp_bases, source = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = buildNodeList( provider, node.bases, source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_class_decl_dict, source = makeDictCreationOrConstant2( keys = [ keyword.arg for keyword in node.keywords ], values = [ buildNode(provider, keyword.value, source_ref) for keyword in node.keywords ], source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_metaclass, source = ExpressionSelectMetaclass( metaclass = ExpressionConditional( condition = ExpressionComparisonIn( left = makeConstantRefNode( constant = "metaclass", source_ref = source_ref, user_provided = True ), right = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), expression_yes = ExpressionDictOperationGet( dict_arg = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), key = makeConstantRefNode( constant = "metaclass", source_ref = source_ref, user_provided = True ), source_ref = source_ref ), expression_no = ExpressionConditional( condition = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), expression_no = makeExpressionBuiltinRef( builtin_name = "type", source_ref = source_ref ), expression_yes = ExpressionBuiltinType1( value = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), subscript = makeConstantRefNode( constant = 0, source_ref = source_ref, user_provided = True ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), bases = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref_orig ), StatementConditional( condition = ExpressionComparisonIn( left = makeConstantRefNode( constant = "metaclass", source_ref = source_ref, user_provided = True ), right = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), no_branch = None, yes_branch = makeStatementsSequenceFromStatement( statement = StatementDictOperationRemove( dict_arg = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), key = makeConstantRefNode( constant = "metaclass", source_ref = source_ref, user_provided = True ), source_ref = source_ref ) ), source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_prepared, source = ExpressionConditional( condition = ExpressionBuiltinHasattr( object_arg = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), name = makeConstantRefNode( constant = "__prepare__", source_ref = source_ref, user_provided = True ), source_ref = source_ref ), expression_no = makeConstantRefNode( constant = {}, source_ref = source_ref, user_provided = True ), expression_yes = makeExpressionCall( called = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), attribute_name = "__prepare__", source_ref = source_ref ), args = ExpressionMakeTuple( elements = ( makeConstantRefNode( constant = node.name, source_ref = source_ref, user_provided = True ), ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ) ), source_ref = source_ref ), kw = ExpressionTempVariableRef( variable = tmp_class_decl_dict, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable = provider.getVariableForAssignment( mangleName(node.name, provider) ), source = decorated_body, source_ref = source_ref ), ) if python_version >= 340: class_creation_function.qualname_setup = node.name, qualname_assign final = ( StatementReleaseVariable( variable = tmp_bases, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_class_decl_dict, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_metaclass, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_prepared, source_ref = source_ref ) ) return makeTryFinallyStatement( provider = provider, tried = statements, final = final, source_ref = source_ref ) def _buildClassNode2(provider, node, source_ref): # This function is the Python2 special case with special re-formulation as # according to developer manual, and it's very detailed, pylint: disable=too-many-locals class_statement_nodes, class_doc = extractDocFromBody(node) function_body = ExpressionClassBody( provider = provider, name = node.name, doc = class_doc, source_ref = source_ref ) parent_module = provider.getParentModule() code_object = CodeObjectSpec( co_name = node.name, co_kind = "Class", co_varnames = (), co_argcount = 0, co_kwonlyargcount = 0, co_has_starlist = False, co_has_stardict = False, co_filename = parent_module.getRunTimeFilename(), co_lineno = source_ref.getLineNumber(), future_spec = parent_module.getFutureSpec() ) body = buildFrameNode( provider = function_body, nodes = class_statement_nodes, code_object = code_object, source_ref = source_ref ) if body is not None: # The frame guard has nothing to tell its line number to. body.source_ref = source_ref.atInternal() # The class body is basically a function that implicitly, at the end # returns its locals and cannot have other return statements contained, and # starts out with a variables "__module__" and potentially "__doc__" set. statements = [ StatementAssignmentVariableName( variable_name = "__module__", source = makeConstantRefNode( constant = provider.getParentModule().getFullName(), source_ref = source_ref, user_provided = True ), source_ref = source_ref.atInternal() ) ] if class_doc is not None: statements.append( StatementAssignmentVariableName( variable_name = "__doc__", source = makeConstantRefNode( constant = class_doc, source_ref = source_ref, user_provided = True ), source_ref = source_ref.atInternal() ) ) statements += [ body, StatementReturn( expression = ExpressionBuiltinLocalsCopy( source_ref = source_ref ), source_ref = source_ref.atInternal() ) ] body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) # The class body is basically a function that implicitly, at the end # returns its locals and cannot have other return statements contained. function_body.setBody(body) temp_scope = provider.allocateTempScope("class_creation") tmp_bases = provider.allocateTempVariable(temp_scope, "bases") tmp_class_dict = provider.allocateTempVariable(temp_scope, "class_dict") tmp_metaclass = provider.allocateTempVariable(temp_scope, "metaclass") tmp_class = provider.allocateTempVariable(temp_scope, "class") select_metaclass = ExpressionOutlineBody( provider = provider, name = "select_metaclass", body = None, source_ref = source_ref ) if node.bases: tmp_base = select_metaclass.allocateTempVariable( temp_scope = None, name = "base" ) statements = ( StatementAssignmentVariable( variable = tmp_base, source = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), subscript = makeConstantRefNode( constant = 0, source_ref = source_ref, user_provided = True ), source_ref = source_ref, ), source_ref = source_ref ), makeTryFinallyStatement( provider, tried = StatementTry( tried = makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = tmp_base, source_ref = source_ref ), attribute_name = "__class__", source_ref = source_ref ), source_ref = source_ref ) ), except_handler = makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = tmp_base, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ), break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ), final = StatementReleaseVariable( variable = tmp_base, source_ref = source_ref ), source_ref = source_ref, public_exc = False ), ) else: statements = ( StatementTry( tried = makeStatementsSequenceFromStatement( statement = StatementReturn( # TODO: Should avoid checking __builtins__ for this. expression = ExpressionVariableNameRef( variable_name = "__metaclass__", source_ref = source_ref ), source_ref = source_ref ) ), except_handler = makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionBuiltinAnonymousRef( builtin_name = "classobj", source_ref = source_ref ), source_ref = source_ref ) ), break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ), ) select_metaclass.setBody( makeStatementsSequence( statements = statements, allow_none = False, source_ref = source_ref ) ) statements = [ StatementAssignmentVariable( variable = tmp_bases, source = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = buildNodeList( provider = provider, nodes = node.bases, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_class_dict, source = function_body, source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_metaclass, source = ExpressionConditional( condition = ExpressionComparisonIn( left = makeConstantRefNode( constant = "__metaclass__", source_ref = source_ref, user_provided = True ), right = ExpressionTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ), source_ref = source_ref ), expression_yes = ExpressionDictOperationGet( dict_arg = ExpressionTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ), key = makeConstantRefNode( constant = "__metaclass__", source_ref = source_ref, user_provided = True ), source_ref = source_ref ), expression_no = select_metaclass, source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_class, source = makeExpressionCall( called = ExpressionTempVariableRef( variable = tmp_metaclass, source_ref = source_ref ), args = ExpressionMakeTuple( elements = ( makeConstantRefNode( constant = node.name, source_ref = source_ref, user_provided = True ), ExpressionTempVariableRef( variable = tmp_bases, source_ref = source_ref ), ExpressionTempVariableRef( variable = tmp_class_dict, source_ref = source_ref ) ), source_ref = source_ref ), kw = None, source_ref = source_ref ), source_ref = source_ref ), ] for decorator in buildNodeList( provider, reversed(node.decorator_list), source_ref ): statements.append( StatementAssignmentVariable( variable = tmp_class, source = makeExpressionCall( called = decorator, args = ExpressionMakeTuple( elements = ( ExpressionTempVariableRef( variable = tmp_class, source_ref = source_ref ), ), source_ref = source_ref ), kw = None, source_ref = decorator.getSourceReference() ), source_ref = decorator.getSourceReference() ) ) statements.append( StatementAssignmentVariableName( variable_name = mangleName(node.name, provider), source = ExpressionTempVariableRef( variable = tmp_class, source_ref = source_ref ), source_ref = source_ref ) ) final = ( StatementReleaseVariable( variable = tmp_class, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_bases, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_class_dict, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_metaclass, source_ref = source_ref ) ) return makeTryFinallyStatement( provider = function_body, tried = statements, final = final, source_ref = source_ref ) def buildClassNode(provider, node, source_ref): assert getKind(node) == "ClassDef" # There appears to be a inconsistency with the top level line number # not being the one really the class has, if there are bases, and a # decorator. if node.bases: source_ref = source_ref.atLineNumber(node.bases[-1].lineno) # Python2 and Python3 are similar, but fundamentally different, so handle # them in dedicated code. if python_version < 300: return _buildClassNode2(provider, node, source_ref) else: return _buildClassNode3(provider, node, source_ref) Nuitka-0.5.28.2/nuitka/tree/ReformulationImportStatements.py0000644000372000001440000003164713207537242024400 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of import statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementAssignmentVariableName, StatementReleaseVariable ) from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.FutureSpecs import FutureSpec from nuitka.nodes.GlobalsLocalsNodes import ExpressionBuiltinGlobals from nuitka.nodes.ImportNodes import ( ExpressionBuiltinImport, ExpressionImportModuleHard, ExpressionImportModuleNameHard, ExpressionImportName, StatementImportStar ) from nuitka.nodes.NodeMakingHelpers import mergeStatements from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from nuitka.PythonVersions import python_version from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .SyntaxErrors import raiseSyntaxError from .TreeHelpers import makeStatementsSequenceOrStatement, mangleName # For checking afterwards, if __future__ imports really were at the beginning # of the file. _future_import_nodes = [] def checkFutureImportsOnlyAtStart(body): # Check if a __future__ imports really were at the beginning of the file. for node in body: if node in _future_import_nodes: _future_import_nodes.remove(node) else: if _future_import_nodes: raiseSyntaxError( """\ from __future__ imports must occur at the beginning of the file""", _future_import_nodes[0].source_ref.atColumnNumber( _future_import_nodes[0].col_offset ) ) def _handleFutureImport(provider, node, source_ref): # Don't allow future imports in functions or classes. if not provider.isCompiledPythonModule(): raiseSyntaxError( """\ from __future__ imports must occur at the beginning of the file""", source_ref.atColumnNumber(node.col_offset) ) for import_desc in node.names: object_name, _local_name = import_desc.name, import_desc.asname _enableFutureFeature( node = node, object_name = object_name, source_ref = source_ref ) # Remember it for checks to be applied once module is complete, e.g. if # they are all at module start. node.source_ref = source_ref _future_import_nodes.append(node) _future_specs = [] def pushFutureSpec(): _future_specs.append(FutureSpec()) def getFutureSpec(): return _future_specs[-1] def popFutureSpec(): del _future_specs[-1] def _enableFutureFeature(node, object_name, source_ref): future_spec = _future_specs[-1] if object_name == "unicode_literals": future_spec.enableUnicodeLiterals() elif object_name == "absolute_import": future_spec.enableAbsoluteImport() elif object_name == "division": future_spec.enableFutureDivision() elif object_name == "print_function": future_spec.enableFuturePrint() elif object_name == "barry_as_FLUFL" and python_version >= 300: future_spec.enableBarry() elif object_name == "generator_stop": future_spec.enableGeneratorStop() elif object_name == "braces": raiseSyntaxError( "not a chance", source_ref.atColumnNumber(node.col_offset) ) elif object_name in ("nested_scopes", "generators", "with_statement"): # These are enabled in all cases already. pass else: raiseSyntaxError( "future feature %s is not defined" % object_name, source_ref.atColumnNumber(node.col_offset) ) def buildImportFromNode(provider, node, source_ref): # "from .. import .." statements. This may trigger a star import, or # multiple names being looked up from the given module variable name. # This is pretty complex, pylint: disable=too-many-branches,too-many-locals module_name = node.module if node.module is not None else "" level = node.level # Use default level under some circumstances. if level == -1: level = None elif level == 0 and not _future_specs[-1].isAbsoluteImport(): level = None if level is not None: level = makeConstantRefNode(level, source_ref, True) # Importing from "__future__" module may enable flags to the parser, # that we need to know about, handle that. if module_name == "__future__": _handleFutureImport(provider, node, source_ref) target_names = [] import_names = [] # Mapping imported "fromlist" to assigned "fromlist" if any, handling the # star case as well. for import_desc in node.names: object_name, local_name = import_desc.name, import_desc.asname if object_name == '*': target_names.append(None) assert local_name is None else: target_names.append( local_name if local_name is not None else object_name ) import_names.append(object_name) # Star imports get special treatment. if None in target_names: # More than "*" is a syntax error in Python, need not care about this at # all, it's only allowed value for import list in this case. assert target_names == [None] # Python3 made it so that these can only occur on the module level, # so this a syntax error if not there. For Python2 it is OK to # occur everywhere though. if not provider.isCompiledPythonModule() and python_version >= 300: raiseSyntaxError( "import * only allowed at module level", source_ref.atColumnNumber(node.col_offset) ) if provider.isCompiledPythonModule(): import_globals = ExpressionBuiltinGlobals(source_ref) import_locals = ExpressionBuiltinGlobals(source_ref) else: import_globals = ExpressionBuiltinGlobals(source_ref) import_locals = makeConstantRefNode({}, source_ref, True) return StatementImportStar( module_import = ExpressionBuiltinImport( name = makeConstantRefNode(module_name, source_ref, True), globals_arg = import_globals, locals_arg = import_locals, fromlist = makeConstantRefNode(('*',), source_ref, True), level = level, source_ref = source_ref ), source_ref = source_ref ) else: if module_name == "__future__": imported_from_module = ExpressionImportModuleHard( module_name = "__future__", source_ref = source_ref ) else: imported_from_module = ExpressionBuiltinImport( name = makeConstantRefNode(module_name, source_ref, True), globals_arg = ExpressionBuiltinGlobals(source_ref), locals_arg = makeConstantRefNode(None, source_ref, True), fromlist = makeConstantRefNode(tuple(import_names), source_ref, True), level = level, source_ref = source_ref ) # If we have multiple names to import, consider each. multi_names = len(target_names) > 1 statements = [] if multi_names: tmp_import_from = provider.allocateTempVariable( temp_scope = provider.allocateTempScope("import_from"), name = "module" ) statements.append( StatementAssignmentVariable( variable = tmp_import_from, source = imported_from_module, source_ref = source_ref ) ) imported_from_module = ExpressionTempVariableRef( variable = tmp_import_from, source_ref = source_ref ) import_statements = [] first = True for target_name, import_name in zip(target_names, import_names): # Make a clone of the variable reference, if we are going to use # another one. if not first: imported_from_module = imported_from_module.makeClone() first = False import_statements.append( StatementAssignmentVariableName( variable_name = mangleName(target_name, provider), source = ExpressionImportName( module = imported_from_module, import_name = import_name, source_ref = source_ref ), source_ref = source_ref ) ) # Release the temporary module value as well. if multi_names: statements.append( makeTryFinallyStatement( provider = provider, tried = import_statements, final = ( StatementReleaseVariable( variable = tmp_import_from, source_ref = source_ref ), ), source_ref = source_ref ) ) else: statements.extend(import_statements) # Note: Each import is sequential. It can succeed, and the failure of a # later one is not undoing previous ones. We can therefore have a # sequence of imports that each only import one thing therefore. return StatementsSequence( statements = mergeStatements(statements), source_ref = source_ref ) def buildImportModulesNode(provider, node, source_ref): # Import modules statement. As described in the developer manual, these # statements can be treated as several ones. import_names = [ ( import_desc.name, import_desc.asname ) for import_desc in node.names ] import_nodes = [] for import_desc in import_names: module_name, local_name = import_desc module_topname = module_name.split('.')[0] # Note: The "level" of import is influenced by the future absolute # imports. level = makeConstantRefNode(0, source_ref, True) if _future_specs[-1].isAbsoluteImport() else None import_node = ExpressionBuiltinImport( name = makeConstantRefNode(module_name, source_ref, True), globals_arg = ExpressionBuiltinGlobals(source_ref), locals_arg = makeConstantRefNode(None, source_ref, True), fromlist = makeConstantRefNode(None, source_ref, True), level = level, source_ref = source_ref ) if local_name: # If is gets a local name, the real name must be used as a # temporary value only, being looked up recursively. for import_name in module_name.split('.')[1:]: import_node = ExpressionImportName( module = import_node, import_name = import_name, source_ref = source_ref ) # If a name was given, use the one provided, otherwise the import gives # the top level package name given for assignment of the imported # module. import_nodes.append( StatementAssignmentVariableName( variable_name = mangleName( local_name if local_name is not None else module_topname, provider ), source = import_node, source_ref = source_ref ) ) # Note: Each import is sequential. It will potentially succeed, and the # failure of a later one is not changing that one bit . We can therefore # have a sequence of imports that only import one thing therefore. return makeStatementsSequenceOrStatement( statements = import_nodes, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/ReformulationTryFinallyStatements.py0000644000372000001440000001670613207537242025222 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of try/finally statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.LoopNodes import StatementLoopBreak, StatementLoopContinue from nuitka.nodes.ReturnNodes import ( ExpressionReturnedValueRef, StatementReturn ) from nuitka.nodes.StatementNodes import ( StatementPreserveFrameException, StatementPublishException, StatementRestoreFrameException, StatementsSequence ) from nuitka.nodes.TryNodes import StatementTry from nuitka.Options import isDebug from nuitka.PythonVersions import python_version from .TreeHelpers import ( buildStatementsNode, getStatementsAppended, getStatementsPrepended, makeReraiseExceptionStatement, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, mergeStatements, popBuildContext, pushBuildContext ) def makeTryFinallyStatement(provider, tried, final, source_ref, public_exc = False): # Complex handling, due to the many variants, pylint: disable=too-many-branches,too-many-locals if type(tried) in (tuple, list): tried = makeStatementsSequenceFromStatements( *tried ) if type(final) in (tuple, list): final = StatementsSequence( statements = mergeStatements(final, False), source_ref = source_ref ) if tried is not None and not tried.isStatementsSequence(): tried = makeStatementsSequenceFromStatement(tried) if final is not None and not final.isStatementsSequence(): final = makeStatementsSequenceFromStatement(final) if tried is None: return final if final is None: return tried if provider is not None: tried.parent = provider final.parent = provider assert tried is not None, source_ref assert final is not None, source_ref # TODO: Currently it's not possible anymore to get at XML for all codes # during the building phase. So this error catcher cannot work currently. if False and isDebug(): final2 = final.makeClone() final2.parent = provider import nuitka.TreeXML if nuitka.TreeXML.Element is not None: f1 = final.asXml() f2 = final2.asXml() def compare(a, b): for c1, c2 in zip(a, b): compare(c1, c2) assert a.attrib == b.attrib, (a.attrib, b.attrib) compare(f1, f2) if tried.mayRaiseException(BaseException): except_handler = final.makeClone() except_handler = getStatementsAppended( statement_sequence = except_handler, statements = makeReraiseExceptionStatement( source_ref = source_ref ) ) if public_exc: preserver_id = provider.allocatePreserverId() except_handler = getStatementsPrepended( statement_sequence = except_handler, statements = ( StatementPreserveFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref ) ) ) except_handler = makeTryFinallyStatement( provider = provider, tried = except_handler, final = StatementRestoreFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), public_exc = False, source_ref = source_ref, ) except_handler = makeStatementsSequenceFromStatement( statement = except_handler ) except_handler.parent = provider else: except_handler = None if tried.mayBreak(): break_handler = getStatementsAppended( statement_sequence = final.makeClone(), statements = StatementLoopBreak( source_ref = source_ref ) ) break_handler.parent = provider else: break_handler = None if tried.mayContinue(): continue_handler = getStatementsAppended( statement_sequence = final.makeClone(), statements = StatementLoopContinue( source_ref = source_ref ) ) continue_handler.parent = provider else: continue_handler = None if tried.mayReturn(): return_handler = getStatementsAppended( statement_sequence = final.makeClone(), statements = StatementReturn( expression = ExpressionReturnedValueRef( source_ref = source_ref ), source_ref = source_ref ) ) else: return_handler = None result = StatementTry( tried = tried, except_handler = except_handler, break_handler = break_handler, continue_handler = continue_handler, return_handler = return_handler, source_ref = source_ref ) if result.isStatementAborting(): return result else: return makeStatementsSequence( statements = ( result, final ), allow_none = False, source_ref = source_ref ) def buildTryFinallyNode(provider, build_tried, node, source_ref): if python_version < 300: # Prevent "continue" statements in the final blocks pushBuildContext("finally") final = buildStatementsNode( provider = provider, nodes = node.finalbody, source_ref = source_ref ) popBuildContext() return makeTryFinallyStatement( provider = provider, tried = build_tried(), final = final, source_ref = source_ref ) else: tried = build_tried() # Prevent "continue" statements in the final blocks, these have to # become "SyntaxError". pushBuildContext("finally") final = buildStatementsNode( provider = provider, nodes = node.finalbody, source_ref = source_ref ) popBuildContext() return makeTryFinallyStatement( provider = provider, tried = tried, final = final, public_exc = True, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/ComplexCallHelperFunctions.py0000644000372000001440000026227613207537242023547 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module is providing helper functions for complex call re-formulations. One for each type of call. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionBuiltinHasattr ) from nuitka.nodes.BuiltinDictNodes import ExpressionBuiltinDict from nuitka.nodes.BuiltinIteratorNodes import ExpressionBuiltinIter1 from nuitka.nodes.BuiltinNextNodes import ExpressionBuiltinNext1 from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, makeExpressionBuiltinRef ) from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinTuple from nuitka.nodes.CallNodes import makeExpressionCall from nuitka.nodes.ComparisonNodes import ( ExpressionComparisonIn, ExpressionComparisonIsNOT ) from nuitka.nodes.ConditionalNodes import ( ExpressionConditionalOR, StatementConditional ) from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.DictionaryNodes import StatementDictOperationSet from nuitka.nodes.ExceptionNodes import ( ExpressionBuiltinMakeException, StatementRaiseException ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.OperatorNodes import makeBinaryOperationNode from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.SubscriptNodes import ( ExpressionSubscriptLookup, StatementAssignmentSubscript ) from nuitka.nodes.TypeNodes import ( ExpressionBuiltinIsinstance, ExpressionBuiltinType1 ) from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import ( getComplexCallSequenceErrorTemplate, python_version ) from .InternalModule import ( getInternalModule, internal_source_ref, once_decorator ) from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( makeCallNode, makeConditionalStatement, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) # TODO: Consider using ExpressionOutlineNodes for at least some of these # or their own helpers. def orderArgs(*args): if python_version >= 350: def weight(arg): result = args.index(arg) if arg == "kw": result += 1.5 elif arg == "star_arg_list": result -= 1.5 return result return tuple( sorted(args, key = weight) ) return args def _makeNameAttributeLookup(node, attribute_name = "__name__"): return ExpressionAttributeLookup( source = node, attribute_name = attribute_name, source_ref = internal_source_ref ) @once_decorator def getCallableNameDescBody(): helper_name = "get_callable_name_desc" # Equivalent of: # # Note: The "called_type" is a temporary variable. # # called_type = type( BuiltinFunctionType ) # # if ininstance( called, (FunctionType, MethodType, BuiltinFunctionType) ): # return called.__name__ # elif python_version < 3 and isinstance( called, ClassType ): # return called_type.__name__ + " constructor" # elif python_version < 3 and isinstance( called, InstanceType ): # return called_type.__name__ + " instance" # else: # return called_type.__name__ + " object" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) functions_case = makeStatementsSequenceFromStatement( statement = ( StatementReturn( expression = makeBinaryOperationNode( operator = "Add", right = makeConstantRefNode( constant = "()", source_ref = internal_source_ref, user_provided = True ), left = _makeNameAttributeLookup( ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) ) no_branch = StatementReturn( expression = makeBinaryOperationNode( operator = "Add", right = makeConstantRefNode( constant = " object", source_ref = internal_source_ref, user_provided = True ), left = _makeNameAttributeLookup( ExpressionBuiltinType1( value = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) if python_version < 300: instance_case = StatementReturn( expression = makeBinaryOperationNode( operator = "Add", right = makeConstantRefNode( constant = " instance", source_ref = internal_source_ref, user_provided = True ), left = _makeNameAttributeLookup( _makeNameAttributeLookup( ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), attribute_name = "__class__", ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) no_branch = makeConditionalStatement( condition = ExpressionBuiltinIsinstance( instance = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "instance", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = instance_case, no_branch = no_branch, source_ref = internal_source_ref ) class_case = StatementReturn( expression = makeBinaryOperationNode( operator = "Add", right = makeConstantRefNode( constant = " constructor", source_ref = internal_source_ref, user_provided = True ), left = _makeNameAttributeLookup( ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) no_branch = makeConditionalStatement( condition = ExpressionBuiltinIsinstance( instance = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "classobj", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = class_case, no_branch = no_branch, source_ref = internal_source_ref ) if python_version < 300: normal_cases = ( "function", "builtin_function_or_method", "instancemethod" ) else: normal_cases = ( "function", "builtin_function_or_method" ) result.setBody( makeStatementsSequenceFromStatement( statement = makeConditionalStatement( condition = ExpressionBuiltinIsinstance( instance = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), classes = ExpressionMakeTuple( elements = tuple( ExpressionBuiltinAnonymousRef( builtin_name = builtin_name, source_ref = internal_source_ref ) for builtin_name in normal_cases ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = functions_case, no_branch = no_branch, source_ref = internal_source_ref ) ) ) return result def makeStarListArgumentErrorRaise(called_variable, star_list_variable): return StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( makeBinaryOperationNode( operator = "Mod", left = makeConstantRefNode( constant = getComplexCallSequenceErrorTemplate(), source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getCallableNameDescBody(), source_ref = internal_source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = internal_source_ref ), values = ( ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), _makeNameAttributeLookup( ExpressionBuiltinType1( value = ExpressionVariableRef( variable = star_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ) def _makeStarListArgumentToTupleStatement(called_variable, star_list_variable): if python_version >= 350: non_tuple_code = makeConditionalStatement( condition = ExpressionConditionalOR( left = ExpressionBuiltinHasattr( object_arg = ExpressionVariableRef( variable = star_list_variable, source_ref = internal_source_ref ), name = makeConstantRefNode("__iter__", internal_source_ref), source_ref = internal_source_ref ), right = ExpressionBuiltinHasattr( object_arg = ExpressionVariableRef( variable = star_list_variable, source_ref = internal_source_ref ), name = makeConstantRefNode("__getitem__", internal_source_ref), source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = StatementAssignmentVariable( variable = star_list_variable, source = ExpressionBuiltinTuple( value = ExpressionVariableRef( variable = star_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), no_branch = makeStarListArgumentErrorRaise( called_variable = called_variable, star_list_variable = star_list_variable, ), source_ref = internal_source_ref ) else: non_tuple_code = makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = star_list_variable, source = ExpressionBuiltinTuple( value = ExpressionVariableRef( variable = star_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "TypeError", handler_body = makeStarListArgumentErrorRaise( called_variable = called_variable, star_list_variable = star_list_variable, ), source_ref = internal_source_ref ) return makeConditionalStatement( condition = ExpressionComparisonIsNOT( left = ExpressionBuiltinType1( value = ExpressionVariableRef( variable = star_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), right = makeExpressionBuiltinRef( builtin_name = "tuple", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = non_tuple_code, no_branch = None, source_ref = internal_source_ref ) def _makeRaiseExceptionMustBeMapping(called_variable, star_dict_variable): return StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( makeBinaryOperationNode( operator = "Mod", left = makeConstantRefNode( constant = """\ %s argument after ** must be a mapping, not %s""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getCallableNameDescBody(), source_ref = internal_source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = internal_source_ref ), values = ( ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), _makeNameAttributeLookup( ExpressionBuiltinType1( value = ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ) def _makeIteratingLoopStatement(tmp_iter_variable, tmp_item_variable, statements): loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_item_variable, source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), *statements ) return StatementLoop( body = loop_body, source_ref = internal_source_ref ) def _makeStarDictArgumentToDictStatement(result, called_variable, star_dict_variable): temp_scope = result.allocateTempScope("mapping") tmp_dict_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_keys_variable = result.allocateTempVariable(temp_scope, "keys") tmp_key_variable = result.allocateTempVariable(temp_scope, "key") loop_body = ( StatementAssignmentSubscript( expression = ExpressionTempVariableRef( variable = tmp_dict_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) mapping_case = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_keys_variable, source = makeCallNode( _makeNameAttributeLookup( ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), attribute_name = "keys" ), internal_source_ref ), source_ref = internal_source_ref ), exception_name = "AttributeError", handler_body = _makeRaiseExceptionMustBeMapping( called_variable = called_variable, star_dict_variable = star_dict_variable ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = ExpressionTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_dict_variable, source = makeConstantRefNode( constant = {}, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_key_variable, statements = loop_body ), StatementAssignmentVariable( variable = star_dict_variable, source = ExpressionTempVariableRef( variable = tmp_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) tried = StatementConditional( condition = ExpressionComparisonIsNOT( left = ExpressionBuiltinType1( value = ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), right = makeExpressionBuiltinRef( builtin_name = "dict", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = mapping_case, no_branch = None, source_ref = internal_source_ref ) final = ( StatementReleaseVariable( variable = tmp_dict_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_keys_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref ) ) return makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) def _makeRaiseDuplicationItem(called_variable, tmp_key_variable): return StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( makeBinaryOperationNode( operator = "Mod", left = makeConstantRefNode( constant = """\ %s got multiple values for keyword argument '%s'""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getCallableNameDescBody( ), source_ref = internal_source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = internal_source_ref ), values = ( ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ) ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ) def _makeStarDictArgumentMergeToKwStatement(result, called_variable, kw_variable, star_dict_variable): # This is plain terribly complex, pylint: disable=too-many-locals temp_scope = result.allocateTempScope("dict") tmp_dict_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_keys_variable = result.allocateTempVariable(temp_scope, "keys") tmp_key_variable = result.allocateTempVariable(temp_scope, "key_xxx") final = [ StatementReleaseVariable( variable = tmp_dict_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_keys_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref, ) ] mapping_loop_body = ( StatementConditional( condition = ExpressionComparisonIn( left = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = _makeRaiseDuplicationItem( called_variable = called_variable, tmp_key_variable = tmp_key_variable ) ), no_branch = None, source_ref = internal_source_ref ), StatementAssignmentSubscript( expression = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) mapping_case = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_keys_variable, source = makeCallNode( _makeNameAttributeLookup( ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), attribute_name = "keys" ), internal_source_ref ), source_ref = internal_source_ref ), exception_name = "AttributeError", handler_body = _makeRaiseExceptionMustBeMapping( called_variable = called_variable, star_dict_variable = star_dict_variable ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = ExpressionTempVariableRef( variable = tmp_keys_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_dict_variable, source = makeConstantRefNode( constant = {}, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_key_variable, statements = mapping_loop_body ) ) temp_scope = result.allocateTempScope("dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "item") tmp_key_variable = result.allocateTempVariable(temp_scope, "key") final += [ StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref, ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref, ) ] dict_loop_body = ( StatementAssignmentVariable( variable = tmp_key_variable, source = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), subscript = makeConstantRefNode( constant = 0, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementConditional( condition = ExpressionComparisonIn( left = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = _makeRaiseDuplicationItem( called_variable = called_variable, tmp_key_variable = tmp_key_variable ) ), no_branch = None, source_ref = internal_source_ref ), StatementAssignmentSubscript( expression = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), subscript = makeConstantRefNode( constant = 1, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) dict_case = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = kw_variable, source = ExpressionBuiltinDict( pos_arg = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), pairs = (), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = makeCallNode( _makeNameAttributeLookup( ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), attribute_name = "iteritems" if python_version < 300 else "items" ), internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_item_variable, statements = dict_loop_body, ), ) dict_case = makeStatementsSequenceFromStatement( statement = StatementConditional( condition = ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), yes_branch = dict_case, no_branch = None, source_ref = internal_source_ref ) ) tried = StatementConditional( condition = ExpressionComparisonIsNOT( left = ExpressionBuiltinType1( value = ExpressionVariableRef( variable = star_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), right = makeExpressionBuiltinRef( builtin_name = "dict", source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = mapping_case, no_branch = dict_case, source_ref = internal_source_ref ) return makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) @once_decorator def getFunctionCallHelperStarList(): helper_name = "complex_call_helper_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "star_arg_list" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable = called_variable, star_list_variable = star_arg_list_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = None, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperKeywordsStarList(): helper_name = "complex_call_helper_keywords_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = orderArgs( "called", "kw", "star_arg_list" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable = called_variable, star_list_variable = star_arg_list_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosStarList(): helper_name = "complex_call_helper_pos_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "args", "star_arg_list" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable = called_variable, star_list_variable = star_arg_list_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = makeBinaryOperationNode( operator = "Add", left = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = None, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosKeywordsStarList(): helper_name = "complex_call_helper_pos_keywords_star_list" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_list, tuple ): # try: # star_arg_list = tuple( star_arg_list ) # except TypeError: # raise TypeError, "%s argument after * must be a sequence, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_list ).__name__ # ) # # return called( *star_arg_list ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = orderArgs( "called", "args", "kw", "star_arg_list" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) statements = ( _makeStarListArgumentToTupleStatement( called_variable = called_variable, star_list_variable = star_arg_list_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = makeBinaryOperationNode( operator = "Add", left = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperStarDict(): helper_name = "complex_call_helper_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # tmp_dict[ tmp_key ] = star_dict_arg[ tmp_key ) # # star_arg_dict = new # # return called( **star_arg_dict ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentToDictStatement( result = result, called_variable = called_variable, star_dict_variable = star_arg_dict_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = None, kw = ExpressionVariableRef( variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosStarDict(): helper_name = "complex_call_helper_pos_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # tmp_dict[ tmp_key ] = star_dict_arg[ tmp_key ) # # star_arg_dict = new # # return called( args, **star_arg_dict ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "args", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentToDictStatement( result = result, called_variable = called_variable, star_dict_variable = star_arg_dict_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperKeywordsStarDict(): helper_name = "complex_call_helper_keywords_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. One goal is to avoid copying "kw" unless really # necessary, and to take the slow route only for non-dictionaries. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # if keys: # kw = dict( kw ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = star_dict_arg[ tmp_key ) # # elif star_arg_dict: # tmp_iter = star_arg_dict.iteritems() # # kw = dict( kw ) # while 1: # try: # tmp_key, tmp_value = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = tmp_value # # return called( **kw ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "kw", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable = called_variable, kw_variable = kw_variable, star_dict_variable = star_arg_dict_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = None, kw = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosKeywordsStarDict(): helper_name = "complex_call_helper_pos_keywords_star_dict" # Equivalent of: # # Note: Call in here is not the same, as it can go without checks directly # to PyObject_Call. One goal is to avoid copying "kw" unless really # necessary, and to take the slow route only for non-dictionaries. # # if not isinstance( star_arg_dict, dict ): # try: # tmp_keys = star_arg_dict.keys() # except AttributeError: # raise TypeError, ""%s argument after ** must be a mapping, not %s" % ( # get_callable_name_desc( function ), # type( star_arg_dict ).__name__ # ) # # if keys: # kw = dict( kw ) # # tmp_iter = iter( keys ) # tmp_dict = {} # # while 1: # try: # tmp_key = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = star_dict_arg[ tmp_key ) # # elif star_arg_dict: # tmp_iter = star_arg_dict.iteritems() # # kw = dict( kw ) # while 1: # try: # tmp_key, tmp_value = tmp_iter.next() # except StopIteration: # break # # if tmp_key in kw: # raise TypeError, "%s got multiple values for keyword argument '%s'" % ( # get_callable_name_desc( function ), # tmp_key # ) # # kw[ tmp_key ] = tmp_value # # return called( **kw ) # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "args", "kw", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = ( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable = called_variable, kw_variable = kw_variable, star_dict_variable = star_arg_dict_variable, ), StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result def getDoubleStarArgsConversion(result, called_variable, kw_variable, star_arg_list_variable, star_arg_dict_variable): statements = [] if kw_variable is not None: statements.append( _makeStarDictArgumentMergeToKwStatement( result = result, called_variable = called_variable, kw_variable = kw_variable, star_dict_variable = star_arg_dict_variable, ) ) else: statements.append( _makeStarDictArgumentToDictStatement( result = result, called_variable = called_variable, star_dict_variable = star_arg_dict_variable, ) ) statements.append( _makeStarListArgumentToTupleStatement( called_variable = called_variable, star_list_variable = star_arg_list_variable, ) ) return statements @once_decorator def getFunctionCallHelperStarListStarDict(): helper_name = "complex_call_helper_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "star_arg_list", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = getDoubleStarArgsConversion( result = result, called_variable = called_variable, star_arg_list_variable = star_arg_list_variable, kw_variable = None, star_arg_dict_variable = star_arg_dict_variable ) statements.append( StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosStarListStarDict(): helper_name = "complex_call_helper_pos_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", "args", "star_arg_list", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = getDoubleStarArgsConversion( result = result, called_variable = called_variable, star_arg_list_variable = star_arg_list_variable, kw_variable = None, star_arg_dict_variable = star_arg_dict_variable ) if python_version >= 360: statements.reverse() statements.append( StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = makeBinaryOperationNode( operator = "Add", left = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = star_arg_dict_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperKeywordsStarListStarDict(): helper_name = "complex_call_helper_keywords_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = orderArgs( "called", "kw", "star_arg_list", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = getDoubleStarArgsConversion( result = result, called_variable = called_variable, star_arg_list_variable = star_arg_list_variable, kw_variable = kw_variable, star_arg_dict_variable = star_arg_dict_variable ) statements.append( StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperPosKeywordsStarListStarDict(): helper_name = "complex_call_helper_pos_keywords_star_list_star_dict" # Only need to check if the star argument value is a sequence and then # convert to tuple. result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = orderArgs( "called", "args", "kw", "star_arg_list", "star_arg_dict" ), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) called_variable = result.getVariableForAssignment( variable_name = "called" ) args_variable = result.getVariableForAssignment( variable_name = "args" ) kw_variable = result.getVariableForAssignment( variable_name = "kw" ) star_arg_list_variable = result.getVariableForAssignment( variable_name = "star_arg_list" ) star_arg_dict_variable = result.getVariableForAssignment( variable_name = "star_arg_dict" ) statements = getDoubleStarArgsConversion( result = result, called_variable = called_variable, star_arg_list_variable = star_arg_list_variable, kw_variable = kw_variable, star_arg_dict_variable = star_arg_dict_variable ) if python_version >= 360: statements.reverse() statements.append( StatementReturn( expression = makeExpressionCall( called = ExpressionVariableRef( variable = called_variable, source_ref = internal_source_ref ), args = makeBinaryOperationNode( operator = "Add", left = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), right = ExpressionVariableRef( variable = star_arg_list_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), kw = ExpressionVariableRef( variable = kw_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) final = ( StatementReleaseVariable( variable = called_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = args_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = kw_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_list_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = star_arg_dict_variable, source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = statements, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getFunctionCallHelperDictionaryUnpacking(): helper_name = "complex_call_helper_dict_unpacking_checks" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = ( "called", ), ps_list_star_arg = "args", ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) args_variable = result.getVariableForAssignment( variable_name = "args" ) called_variable = result.getVariableForAssignment( variable_name = "called" ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "dicts_iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "args_item") tmp_iter2_variable = result.allocateTempVariable(temp_scope, "dict_iter") tmp_key_variable = result.allocateTempVariable(temp_scope, "dict_key") update_body = ( StatementConditional( condition = ExpressionComparisonIn( left = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), right = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = _makeRaiseDuplicationItem( called_variable = called_variable, tmp_key_variable = tmp_key_variable ) ), no_branch = None, source_ref = internal_source_ref ), StatementDictOperationSet( dict_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), key = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), value = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_key_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) loop_body = ( makeTryExceptSingleHandlerNode( tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_iter2_variable, source = ExpressionBuiltinIter1( value = makeCallNode( _makeNameAttributeLookup( ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), attribute_name = "keys" ), internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter2_variable, tmp_item_variable = tmp_key_variable, statements = update_body ) ), exception_name = "AttributeError", handler_body = StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( makeBinaryOperationNode( operator = "Mod", left = makeConstantRefNode( constant = """\ '%s' object is not a mapping""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( _makeNameAttributeLookup( ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ), ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ), source_ref = internal_source_ref ), ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter2_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_key_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_result_variable, source = makeConstantRefNode( constant = {}, source_ref = internal_source_ref ), source_ref = internal_source_ref ), _makeIteratingLoopStatement( tmp_iter_variable = tmp_iter_variable, tmp_item_variable = tmp_item_variable, statements = loop_body ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result Nuitka-0.5.28.2/nuitka/tree/ReformulationAssignmentStatements.py0000644000372000001440000011217313207537242025230 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of assignment statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementAssignmentVariableName, StatementDelVariable, StatementDelVariableName, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, StatementAssignmentAttribute, StatementDelAttribute ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, StatementSpecialUnpackCheck ) from nuitka.nodes.BuiltinNextNodes import ExpressionSpecialUnpack from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinList from nuitka.nodes.ComparisonNodes import ExpressionComparisonIsNOT from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantEllipsisRef from nuitka.nodes.ContainerOperationNodes import ExpressionListOperationPop from nuitka.nodes.OperatorNodes import makeExpressionOperationBinaryInplace from nuitka.nodes.SliceNodes import ( ExpressionBuiltinSlice, ExpressionSliceLookup, StatementAssignmentSlice, StatementDelSlice ) from nuitka.nodes.SubscriptNodes import ( ExpressionSubscriptLookup, StatementAssignmentSubscript, StatementDelSubscript ) from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableNameRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .ReformulationImportStatements import getFutureSpec from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .SyntaxErrors import raiseSyntaxError from .TreeHelpers import ( buildNode, getKind, makeConstantRefNode, makeSequenceCreationOrConstant, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, makeStatementsSequenceOrStatement, mangleName ) def buildExtSliceNode(provider, node, source_ref): elements = [] for dim in node.slice.dims: dim_kind = getKind(dim) if dim_kind == "Slice": lower = buildNode(provider, dim.lower, source_ref, True) upper = buildNode(provider, dim.upper, source_ref, True) step = buildNode(provider, dim.step, source_ref, True) element = ExpressionBuiltinSlice( start = lower, stop = upper, step = step, source_ref = source_ref ) elif dim_kind == "Ellipsis": element = ExpressionConstantEllipsisRef( source_ref = source_ref, user_provided = True ) elif dim_kind == "Index": element = buildNode( provider = provider, node = dim.value, source_ref = source_ref ) else: assert False, dim elements.append(element) return makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = elements, source_ref = source_ref ) def buildAssignmentStatementsFromDecoded(provider, kind, detail, source, source_ref): # This is using many variable names on purpose, so as to give names to the # unpacked detail values, and has many branches due to the many cases # dealt with, pylint: disable=too-many-branches,too-many-locals if kind == "Name": return StatementAssignmentVariableName( variable_name = detail, source = source, source_ref = source_ref ) elif kind == "Attribute": lookup_source, attribute_name = detail return StatementAssignmentAttribute( expression = lookup_source, attribute_name = attribute_name, source = source, source_ref = source_ref ) elif kind == "Subscript": subscribed, subscript = detail return StatementAssignmentSubscript( expression = subscribed, subscript = subscript, source = source, source_ref = source_ref ) elif kind == "Slice": lookup_source, lower, upper = detail # For Python3 there is no slicing operation, this is always done # with subscript using a slice object. For Python2, it is only done # if no "step" is provided. use_sliceobj = python_version >= 300 if use_sliceobj: return StatementAssignmentSubscript( expression = lookup_source, source = source, subscript = ExpressionBuiltinSlice( start = lower, stop = upper, step = None, source_ref = source_ref ), source_ref = source_ref ) else: return StatementAssignmentSlice( expression = lookup_source, lower = lower, upper = upper, source = source, source_ref = source_ref ) elif kind == "Tuple": temp_scope = provider.allocateTempScope("tuple_unpack") source_iter_var = provider.allocateTempVariable( temp_scope = temp_scope, name = "source_iter" ) element_vars = [ provider.allocateTempVariable( temp_scope = temp_scope, name = "element_%d" % ( element_index + 1 ) ) for element_index in range(len(detail)) ] starred_list_var = None starred_index = None statements = [] for element_index, element in enumerate(detail): element_var = element_vars[element_index] if starred_list_var is not None: if element[0] == "Starred": raiseSyntaxError( "two starred expressions in assignment", source_ref.atColumnNumber(0) ) statements.insert( starred_index+1, StatementAssignmentVariable( variable = element_var, source = ExpressionListOperationPop( list_arg = ExpressionTempVariableRef( variable = starred_list_var, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ) elif element[0] != "Starred": statements.append( StatementAssignmentVariable( variable = element_var, source = ExpressionSpecialUnpack( value = ExpressionTempVariableRef( variable = source_iter_var, source_ref = source_ref ), count = element_index + 1, expected = len(detail), source_ref = source_ref ), source_ref = source_ref ) ) else: starred_index = element_index starred_list_var = element_var statements.append( StatementAssignmentVariable( variable = element_var, source = ExpressionBuiltinList( value = ExpressionTempVariableRef( variable = source_iter_var, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ) if starred_list_var is None: statements.append( StatementSpecialUnpackCheck( iterator = ExpressionTempVariableRef( variable = source_iter_var, source_ref = source_ref ), count = len(detail), source_ref = source_ref ) ) statements = [ StatementAssignmentVariable( variable = source_iter_var, source = ExpressionBuiltinIter1( value = source, source_ref = source_ref ), source_ref = source_ref ), makeTryFinallyStatement( provider = provider, tried = statements, final = ( StatementReleaseVariable( variable = source_iter_var, source_ref = source_ref ), ), source_ref = source_ref ) ] # When all is done, copy over to the actual assignment targets, starred # or not makes no difference here anymore. for element_index, element in enumerate(detail): if element[0] == "Starred": element = element[1] element_var = element_vars[element_index] statements.append( buildAssignmentStatementsFromDecoded( provider = provider, kind = element[0], detail = element[1], source = ExpressionTempVariableRef( variable = element_var, source_ref = source_ref ), source_ref = source_ref ) ) # Need to release temporary variables right after successful # usage. statements.append( StatementDelVariable( variable = element_var, tolerant = True, source_ref = source_ref, ) ) final_statements = [] for element_var in element_vars: final_statements.append( StatementReleaseVariable( variable = element_var, source_ref = source_ref, ) ) return makeTryFinallyStatement( provider = provider, tried = statements, final = final_statements, source_ref = source_ref ) elif kind == "Starred": raiseSyntaxError( "starred assignment target must be in a list or tuple", source_ref.atColumnNumber(0) ) else: assert False, (kind, source_ref, detail) def buildAssignmentStatements(provider, node, source, source_ref, allow_none = False, temp_provider = None): if node is None and allow_none: return None if temp_provider is None: temp_provider = provider kind, detail = decodeAssignTarget( provider = provider, node = node, source_ref = source_ref ) return buildAssignmentStatementsFromDecoded( provider = temp_provider, kind = kind, detail = detail, source = source, source_ref = source_ref ) def decodeAssignTarget(provider, node, source_ref, allow_none = False): # Many cases to deal with, because of the different assign targets, # pylint: disable=too-many-branches,too-many-return-statements if node is None and allow_none: return None if type(node) is str: return "Name", mangleName(node, provider) kind = getKind(node) if hasattr(node, "ctx"): assert getKind(node.ctx) in ("Store", "Del") if kind == "Name": return kind, mangleName(node.id, provider) elif kind == "Attribute": return kind, ( buildNode(provider, node.value, source_ref), node.attr ) elif kind == "Subscript": slice_kind = getKind(node.slice) if slice_kind == "Index": return "Subscript", ( buildNode(provider, node.value, source_ref), buildNode(provider, node.slice.value, source_ref) ) elif slice_kind == "Slice": lower = buildNode(provider, node.slice.lower, source_ref, True) upper = buildNode(provider, node.slice.upper, source_ref, True) if node.slice.step is not None: step = buildNode(provider, node.slice.step, source_ref) return "Subscript", ( buildNode(provider, node.value, source_ref), ExpressionBuiltinSlice( start = lower, stop = upper, step = step, source_ref = source_ref ) ) else: return "Slice", ( buildNode(provider, node.value, source_ref), lower, upper ) elif slice_kind == "ExtSlice": return "Subscript", ( buildNode(provider, node.value, source_ref), buildExtSliceNode(provider, node, source_ref) ) elif slice_kind == "Ellipsis": return "Subscript", ( buildNode(provider, node.value, source_ref), ExpressionConstantEllipsisRef( source_ref = source_ref ) ) else: assert False, slice_kind elif kind in ("Tuple", "List"): return "Tuple", tuple( decodeAssignTarget( provider = provider, node = sub_node, source_ref = source_ref, allow_none = False ) for sub_node in node.elts ) elif kind == "Starred": return "Starred", decodeAssignTarget( provider = provider, node = node.value, source_ref = source_ref, allow_none = False ) else: assert False, (source_ref, kind) def buildAssignNode(provider, node, source_ref): assert len(node.targets) >= 1, source_ref # Evaluate the right hand side first, so it can get names provided # before the left hand side exists. source = buildNode(provider, node.value, source_ref) if len(node.targets) == 1: # Simple assignment case, one source, one target. return buildAssignmentStatements( provider = provider, node = node.targets[0], source = source, source_ref = source_ref ) else: # Complex assignment case, one source, but multiple targets. We keep the # source in a temporary variable, and then assign from it multiple # times. temp_scope = provider.allocateTempScope("assign_unpack") tmp_source = provider.allocateTempVariable( temp_scope = temp_scope, name = "assign_source" ) statements = [ StatementAssignmentVariable( variable = tmp_source, source = source, source_ref = source_ref ) ] for target in node.targets: statements.append( buildAssignmentStatements( provider = provider, node = target, source = ExpressionTempVariableRef( variable = tmp_source, source_ref = source_ref ), source_ref = source_ref ) ) return makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_source, source_ref = source_ref ), source_ref = source_ref ) def buildAnnAssignNode(provider, node, source_ref): """ Python3.6 annotation assignment. """ if provider.isCompiledPythonModule() or provider.isExpressionClassBody(): provider.markAsNeedsAnnotationsDictionary() # Evaluate the right hand side first, so it can get names provided # before the left hand side exists. statements = [] if node.value is not None: source = buildNode(provider, node.value, source_ref) statements.append( buildAssignmentStatements( provider = provider, node = node.target, source = source, source_ref = source_ref ) ) # Only name referencing annotations are effective right now. if statements[-1].isStatementAssignmentVariable(): variable_name = statements[-1].getVariableName() else: variable_name = None else: # Only name referencing annotations are effective right now. kind, detail = decodeAssignTarget( provider = provider, node = node.target, source_ref = source_ref ) if kind == "Name": variable_name = detail else: variable_name = None # Only annotations for modules and classes are really made, for functions # they are ignored like comments. if variable_name is not None: if provider.isExpressionFunctionBody(): provider.getVariableForAssignment(variable_name) else: annotation = buildNode(provider, node.annotation, source_ref) statements.append( StatementAssignmentSubscript( expression = ExpressionVariableRef( variable = provider.getVariableForAssignment("__annotations__"), source_ref = source_ref ), subscript = makeConstantRefNode( constant = variable_name, source_ref = source_ref ), source = annotation, source_ref = source_ref ) ) return makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) def buildDeleteStatementFromDecoded(kind, detail, source_ref): if kind in ("Name", "Name_Exception"): # Note: Name_Exception is a "del" for exception handlers that doesn't # insist on the variable being defined, user code may do it too, and # that will be fine, so make that tolerant. return StatementDelVariableName( variable_name = detail, tolerant = kind == "Name_Exception", source_ref = source_ref ) elif kind == "Attribute": lookup_source, attribute_name = detail return StatementDelAttribute( expression = lookup_source, attribute_name = attribute_name, source_ref = source_ref ) elif kind == "Subscript": subscribed, subscript = detail return StatementDelSubscript( expression = subscribed, subscript = subscript, source_ref = source_ref ) elif kind == "Slice": lookup_source, lower, upper = detail use_sliceobj = python_version >= 300 if use_sliceobj: return StatementDelSubscript( expression = lookup_source, subscript = ExpressionBuiltinSlice( start = lower, stop = upper, step = None, source_ref = source_ref ), source_ref = source_ref ) else: return StatementDelSlice( expression = lookup_source, lower = lower, upper = upper, source_ref = source_ref ) elif kind == "Tuple": result = [] for sub_node in detail: result.append( buildDeleteStatementFromDecoded( kind = sub_node[0], detail = sub_node[1], source_ref = source_ref ) ) return makeStatementsSequenceOrStatement( statements = result, source_ref = source_ref ) else: assert False, (kind, detail, source_ref) def buildDeleteNode(provider, node, source_ref): # Build "del" statements. # Note: Each delete is sequential. It can succeed, and the failure of a # later one does not prevent the former to succeed. We can therefore have a # simple sequence of "del" statements that each only delete one thing # therefore. In output tree "del" therefore only ever has single arguments. statements = [] for target in node.targets: kind, detail = decodeAssignTarget( provider = provider, node = target, source_ref = source_ref ) statements.append( buildDeleteStatementFromDecoded( kind = kind, detail = detail, source_ref = source_ref ) ) return makeStatementsSequenceOrStatement( statements = statements, source_ref = source_ref ) def _buildInplaceAssignVariableNode(variable_name, operator, expression, source_ref): inplace_node = makeExpressionOperationBinaryInplace( operator = operator, left = ExpressionVariableNameRef( variable_name = variable_name, source_ref = source_ref ), right = expression, source_ref = source_ref ) inplace_node.markAsInplaceSuspect() return ( StatementAssignmentVariableName( variable_name = variable_name, source = inplace_node, source_ref = source_ref ), ) def _buildInplaceAssignAttributeNode(provider, lookup_source, attribute_name, tmp_variable1, tmp_variable2, operator, expression, source_ref): # First assign the target value to a temporary variable. preserve_to_tmp = StatementAssignmentVariable( variable = tmp_variable1, source = ExpressionAttributeLookup( source = lookup_source.makeClone(), attribute_name = attribute_name, source_ref = source_ref ), source_ref = source_ref ) # Second assign the in-place result to a temporary variable inplace_to_tmp = StatementAssignmentVariable( variable = tmp_variable2, source = makeExpressionOperationBinaryInplace( operator = operator, left = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) # Third, copy it over, if the reference values change, i.e. IsNot is true. copy_back_from_tmp = StatementConditional( condition = ExpressionComparisonIsNOT( left = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), right = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementAssignmentAttribute( expression = lookup_source.makeClone(), attribute_name = attribute_name, source = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ) copy_back_from_tmp = makeTryFinallyStatement( provider = provider, tried = copy_back_from_tmp, final = StatementReleaseVariable( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ) return ( preserve_to_tmp, # making sure the above temporary variable is deleted in any case. makeTryFinallyStatement( provider = provider, tried = ( inplace_to_tmp, copy_back_from_tmp, ), final = StatementReleaseVariable( variable = tmp_variable1, source_ref = source_ref ), source_ref = source_ref ) ) def _buildInplaceAssignSubscriptNode(provider, subscribed, subscript, tmp_variable1, tmp_variable2, operator, expression, source_ref): # First assign the subscribed value to a temporary variable. preserve_to_tmp1 = StatementAssignmentVariable( variable = tmp_variable1, source = subscribed, source_ref = source_ref ) # Second assign the subscript value to a temporary variable preserve_to_tmp2 = StatementAssignmentVariable( variable = tmp_variable2, source = subscript, source_ref = source_ref ) execute_in_place = StatementAssignmentSubscript( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source = makeExpressionOperationBinaryInplace( operator = operator, left = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ), source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) # Note: No copy back is happening, for subscripts that is implied. return ( preserve_to_tmp1, makeTryFinallyStatement( provider = provider, tried = ( preserve_to_tmp2, execute_in_place, ), final = ( StatementReleaseVariable( variable = tmp_variable1, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_variable2, source_ref = source_ref ) ), source_ref = source_ref ) ) def _buildInplaceAssignSliceNode(provider, lookup_source, lower, upper, tmp_variable1, tmp_variable2, tmp_variable3, operator, expression, source_ref): # Due to the 3 inputs, which we need to also put into temporary variables, # there are too many variables here, but they are needed. # pylint: disable=too-many-locals # First assign the target value, lower and upper to temporary variables. copy_to_tmp = StatementAssignmentVariable( variable = tmp_variable1, source = lookup_source, source_ref = source_ref ) final_statements = [ StatementReleaseVariable( variable = tmp_variable1, source_ref = source_ref ) ] statements = [] if lower is not None: statements.append( StatementAssignmentVariable( variable = tmp_variable2, source = lower, source_ref = source_ref ) ) final_statements.append( StatementReleaseVariable( variable = tmp_variable2, source_ref = source_ref ) ) lower_ref1 = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ) lower_ref2 = ExpressionTempVariableRef( variable = tmp_variable2, source_ref = source_ref ) else: assert tmp_variable2 is None lower_ref1 = lower_ref2 = None if upper is not None: statements.append( StatementAssignmentVariable( variable = tmp_variable3, source = upper, source_ref = source_ref ) ) final_statements.append( StatementReleaseVariable( variable = tmp_variable3, source_ref = source_ref ) ) upper_ref1 = ExpressionTempVariableRef( variable = tmp_variable3, source_ref = source_ref ) upper_ref2 = ExpressionTempVariableRef( variable = tmp_variable3, source_ref = source_ref ) else: assert tmp_variable3 is None upper_ref1 = upper_ref2 = None use_sliceobj = python_version >= 300 # Second assign the in-place result over the original value. if use_sliceobj: statements.append( StatementAssignmentSubscript( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionBuiltinSlice( start = lower_ref1, stop = upper_ref1, step = None, source_ref = source_ref ), source = makeExpressionOperationBinaryInplace( operator = operator, left = ExpressionSubscriptLookup( subscribed = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), subscript = ExpressionBuiltinSlice( start = lower_ref2, stop = upper_ref2, step = None, source_ref = source_ref ), source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) ) else: statements.append( StatementAssignmentSlice( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), lower = lower_ref1, upper = upper_ref1, source = makeExpressionOperationBinaryInplace( operator = operator, left = ExpressionSliceLookup( expression = ExpressionTempVariableRef( variable = tmp_variable1, source_ref = source_ref ), lower = lower_ref2, upper = upper_ref2, source_ref = source_ref ), right = expression, source_ref = source_ref ), source_ref = source_ref ) ) return ( copy_to_tmp, makeTryFinallyStatement( provider = provider, tried = statements, final = final_statements, source_ref = source_ref ) ) def buildInplaceAssignNode(provider, node, source_ref): # There are many inplace assignment variables, and the detail is unpacked # into names, so we end up with a lot of variables, which is on purpose, # pylint: disable=too-many-locals operator = getKind(node.op) if operator == "Div" and getFutureSpec().isFutureDivision(): operator = "TrueDiv" operator = 'I' + operator expression = buildNode(provider, node.value, source_ref) kind, detail = decodeAssignTarget( provider = provider, node = node.target, source_ref = source_ref ) if kind == "Name": statements = _buildInplaceAssignVariableNode( variable_name = detail, operator = operator, expression = expression, source_ref = source_ref ) elif kind == "Attribute": lookup_source, attribute_name = detail temp_scope = provider.allocateTempScope("inplace_assign_attr") tmp_variable1 = provider.allocateTempVariable( temp_scope = temp_scope, name = "start" ) tmp_variable2 = provider.allocateTempVariable( temp_scope = temp_scope, name = "end" ) statements = _buildInplaceAssignAttributeNode( provider = provider, lookup_source = lookup_source, attribute_name = attribute_name, tmp_variable1 = tmp_variable1, tmp_variable2 = tmp_variable2, operator = operator, expression = expression, source_ref = source_ref ) elif kind == "Subscript": subscribed, subscript = detail temp_scope = provider.allocateTempScope("inplace_assign_subscr") tmp_variable1 = provider.allocateTempVariable( temp_scope = temp_scope, name = "target" ) tmp_variable2 = provider.allocateTempVariable( temp_scope = temp_scope, name = "subscript" ) statements = _buildInplaceAssignSubscriptNode( provider = provider, subscribed = subscribed, subscript = subscript, tmp_variable1 = tmp_variable1, tmp_variable2 = tmp_variable2, operator = operator, expression = expression, source_ref = source_ref ) elif kind == "Slice": lookup_source, lower, upper = detail temp_scope = provider.allocateTempScope("inplace_assign_slice") tmp_variable1 = provider.allocateTempVariable( temp_scope = temp_scope, name = "target" ) if lower is not None: tmp_variable2 = provider.allocateTempVariable( temp_scope = temp_scope, name = "lower" ) else: tmp_variable2 = None if upper is not None: tmp_variable3 = provider.allocateTempVariable( temp_scope = temp_scope, name = "upper" ) else: tmp_variable3 = None statements = _buildInplaceAssignSliceNode( provider = provider, lookup_source = lookup_source, lower = lower, upper = upper, tmp_variable1 = tmp_variable1, tmp_variable2 = tmp_variable2, tmp_variable3 = tmp_variable3, operator = operator, expression = expression, source_ref = source_ref ) else: assert False, kind return makeStatementsSequenceFromStatements( *statements ) Nuitka-0.5.28.2/nuitka/tree/ReformulationComparisonExpressions.py0000644000372000001440000001533513207537242025427 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of comparison chain expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.NodeMakingHelpers import makeComparisonNode from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, getKind, makeStatementsSequenceFromStatement ) def buildComparisonNode(provider, node, source_ref): assert len(node.comparators) == len(node.ops) # Comparisons are re-formulated as described in the developer manual. When # having multiple comparators, things require assignment expressions and # references of them to work properly. Then they can become normal "and" # code. # The operands are split out in two parts strangely. left = buildNode(provider, node.left, source_ref) rights = [ buildNode(provider, comparator, source_ref) for comparator in node.comparators ] comparators = [ getKind(comparator) for comparator in node.ops ] # Normal, and simple case, we only have one comparison, which is what our # node handles only. Then we can handle it if len(rights) == 1: return makeComparisonNode( left = left, right = rights[0], # TODO: The terminology of Nuitka might be messed up here. comparator = comparators[0], source_ref = source_ref ) return buildComplexComparisonNode(provider, left, rights, comparators, source_ref) def buildComplexComparisonNode(provider, left, rights, comparators, source_ref): # This is a bit complex, due to the many details, pylint: disable=too-many-locals outline_body = ExpressionOutlineBody( provider = provider, name = "comparison_chain", source_ref = source_ref ) variables = [ outline_body.allocateTempVariable( temp_scope = None, name = "operand_%d" % count ) for count in range(2, len(rights)+2) ] tmp_variable = outline_body.allocateTempVariable( temp_scope = None, name = "comparison_result" ) def makeTempAssignment(count, value): return StatementAssignmentVariable( variable = variables[count], source = value, source_ref = source_ref, ) def makeReleaseStatement(count): return StatementReleaseVariable( variable = variables[count], source_ref = source_ref ) def makeValueComparisonReturn(left, right, comparator): yield StatementAssignmentVariable( variable = tmp_variable, source = makeComparisonNode( left = left, right = right, comparator = comparator, source_ref = source_ref ), source_ref = source_ref, ) yield StatementConditional( condition = ExpressionOperationNOT( operand = ExpressionTempVariableRef( variable = tmp_variable, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_variable, source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ) statements = [] final = [] for count, value in enumerate(rights): if value is not rights[-1]: statements.append( makeTempAssignment(count, value) ) final.append( makeReleaseStatement(count) ) right = ExpressionTempVariableRef( variable = variables[count], source_ref = source_ref ) else: right = value if count != 0: left = ExpressionTempVariableRef( variable = variables[count-1], source_ref = source_ref ) comparator = comparators[count] if value is not rights[-1]: statements.extend( makeValueComparisonReturn( left, right, comparator ) ) else: statements.append( StatementReturn( expression = makeComparisonNode( left = left, right = right, comparator = comparator, source_ref = source_ref ), source_ref = source_ref ) ) final.append( StatementReleaseVariable( variable = tmp_variable, source_ref = source_ref ) ) outline_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outline_body, tried = statements, final = final, source_ref = source_ref ) ) ) return outline_body Nuitka-0.5.28.2/nuitka/tree/ReformulationExecStatements.py0000644000372000001440000004424513207537242024010 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of "exec" statements Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.BuiltinRefNodes import ( ExpressionBuiltinAnonymousRef, ExpressionBuiltinExceptionRef ) from nuitka.nodes.CallNodes import ExpressionCallEmpty from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import ( ExpressionConstantNoneRef, makeConstantRefNode ) from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.ExecEvalNodes import StatementExec, StatementLocalsDictSync from nuitka.nodes.GlobalsLocalsNodes import ExpressionBuiltinGlobals from nuitka.nodes.NodeMakingHelpers import makeExpressionBuiltinLocals from nuitka.nodes.TypeNodes import ExpressionBuiltinIsinstance from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, getKind, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) def wrapEvalGlobalsAndLocals(provider, globals_node, locals_node, temp_scope, source_ref): """ Wrap the locals and globals arguments for "eval". This is called from the outside, and when the node tree already exists. """ globals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "globals" ) locals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "locals" ) if locals_node is None: locals_node = ExpressionConstantNoneRef( source_ref = source_ref ) if globals_node is None: globals_node = ExpressionConstantNoneRef( source_ref = source_ref ) post_statements = [] if provider.isExpressionClassBody(): post_statements.append( StatementLocalsDictSync( locals_arg = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref, ), source_ref = source_ref.atInternal() ) ) post_statements += [ StatementReleaseVariable( variable = globals_keeper_variable, source_ref = source_ref ), StatementReleaseVariable( variable = locals_keeper_variable, source_ref = source_ref ) ] # The locals default is dependent on exec_mode, globals or locals. locals_default = ExpressionConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), expression_no = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), expression_yes = makeExpressionBuiltinLocals( provider = provider, source_ref = source_ref ), source_ref = source_ref ) pre_statements = [ # First assign globals and locals temporary the values given. StatementAssignmentVariable( variable = globals_keeper_variable, source = globals_node, source_ref = source_ref, ), StatementAssignmentVariable( variable = locals_keeper_variable, source = locals_node, source_ref = source_ref, ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( StatementAssignmentVariable( variable = locals_keeper_variable, source = locals_default, source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( StatementAssignmentVariable( variable = globals_keeper_variable, source = ExpressionBuiltinGlobals( source_ref = source_ref ), source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ) ] return ( ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref if globals_node is None else globals_node.getSourceReference() ), ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref if locals_node is None else locals_node.getSourceReference() ), makeStatementsSequence(pre_statements, False, source_ref), makeStatementsSequence(post_statements, False, source_ref) ) def buildExecNode(provider, node, source_ref): # "exec" statements, should only occur with Python2. # This is using many variables, due to the many details this is # dealing with. The locals and globals need to be dealt with in # temporary variables, and we need handling of indicators, so # that is just the complexity, pylint: disable=too-many-locals exec_globals = node.globals exec_locals = node.locals body = node.body orig_globals = exec_globals # Handle exec(a,b,c) to be same as exec a, b, c if exec_locals is None and exec_globals is None and \ getKind(body) == "Tuple": parts = body.elts body = parts[0] if len(parts) > 1: exec_globals = parts[1] if len(parts) > 2: exec_locals = parts[2] else: return StatementRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = "TypeError", source_ref = source_ref ), exception_value = makeConstantRefNode( constant = """\ exec: arg 1 must be a string, file, or code object""", source_ref = source_ref ), exception_trace = None, exception_cause = None, source_ref = source_ref ) if not provider.isCompiledPythonModule(): if orig_globals is None: provider.markAsUnqualifiedExecContaining(source_ref) temp_scope = provider.allocateTempScope("exec") locals_value = buildNode(provider, exec_locals, source_ref, True) if locals_value is None: locals_value = ExpressionConstantNoneRef( source_ref = source_ref ) globals_value = buildNode(provider, exec_globals, source_ref, True) if globals_value is None: globals_value = ExpressionConstantNoneRef( source_ref = source_ref ) source_code = buildNode(provider, body, source_ref) source_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "exec_source" ) globals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "globals" ) locals_keeper_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "locals" ) plain_indicator_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "plain" ) tried = ( # First evaluate the source code expressions. StatementAssignmentVariable( variable = source_variable, source = source_code, source_ref = source_ref ), # Assign globals and locals temporary the values given, then fix it # up, taking note in the "plain" temporary variable, if it was an # "exec" statement with None arguments, in which case the copy back # will be necessary. StatementAssignmentVariable( variable = globals_keeper_variable, source = globals_value, source_ref = source_ref ), StatementAssignmentVariable( variable = locals_keeper_variable, source = locals_value, source_ref = source_ref ), StatementAssignmentVariable( variable = plain_indicator_variable, source = makeConstantRefNode( constant = False, source_ref = source_ref ), source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = globals_keeper_variable, source = ExpressionBuiltinGlobals( source_ref = source_ref ), source_ref = source_ref, ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = locals_keeper_variable, source = makeExpressionBuiltinLocals( provider = provider, source_ref = source_ref ), source_ref = source_ref, ), StatementAssignmentVariable( variable = plain_indicator_variable, source = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ), ), no_branch = makeStatementsSequenceFromStatements( StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementAssignmentVariable( variable = locals_keeper_variable, source = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), source_ref = source_ref, ) ), no_branch = None, source_ref = source_ref ) ), source_ref = source_ref ), # Source needs some special treatment for not done for "eval", if it's a # file object, then must be read. StatementConditional( condition = ExpressionBuiltinIsinstance( instance = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), classes = ExpressionBuiltinAnonymousRef( builtin_name = "file", source_ref = source_ref, ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementAssignmentVariable( variable = source_variable, source = ExpressionCallEmpty( called = ExpressionAttributeLookup( source = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), attribute_name = "read", source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ), makeTryFinallyStatement( provider = provider, tried = StatementExec( source_code = ExpressionTempVariableRef( variable = source_variable, source_ref = source_ref ), globals_arg = ExpressionTempVariableRef( variable = globals_keeper_variable, source_ref = source_ref ), locals_arg = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref ), source_ref = source_ref ), final = StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = plain_indicator_variable, source_ref = source_ref ), right = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementLocalsDictSync( locals_arg = ExpressionTempVariableRef( variable = locals_keeper_variable, source_ref = source_ref, ), source_ref = source_ref.atInternal() ) ), no_branch = None, source_ref = source_ref ), source_ref = source_ref ) ) final = ( StatementReleaseVariable( variable = source_variable, source_ref = source_ref ), StatementReleaseVariable( variable = globals_keeper_variable, source_ref = source_ref ), StatementReleaseVariable( variable = locals_keeper_variable, source_ref = source_ref ), StatementReleaseVariable( variable = plain_indicator_variable, source_ref = source_ref ), ) return makeTryFinallyStatement( provider = provider, tried = tried, final = final, source_ref = source_ref ) # This is here, to make sure it can register, pylint: disable=W0611 import nuitka.optimizations.OptimizeBuiltinCalls # isort:skip @UnusedImport Nuitka-0.5.28.2/nuitka/tree/ReformulationWithStatements.py0000644000372000001440000003423613207537242024036 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of with statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka import Options from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ( ExpressionAttributeLookup, ExpressionAttributeLookupSpecial ) from nuitka.nodes.CallNodes import ( ExpressionCallEmpty, ExpressionCallNoKeywords ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.CoroutineNodes import ExpressionAsyncWait from nuitka.nodes.ExceptionNodes import ( ExpressionCaughtExceptionTracebackRef, ExpressionCaughtExceptionTypeRef, ExpressionCaughtExceptionValueRef ) from nuitka.nodes.StatementNodes import ( StatementExpressionOnly, StatementsSequence ) from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from nuitka.PythonVersions import python_version from .ReformulationAssignmentStatements import buildAssignmentStatements from .ReformulationTryExceptStatements import ( makeTryExceptSingleHandlerNodeWithPublish ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildStatementsNode, getKind, makeConditionalStatement, makeReraiseExceptionStatement, makeStatementsSequence, makeStatementsSequenceFromStatement ) def _buildWithNode(provider, context_expr, assign_target, body, body_lineno, sync, source_ref): # Many details, pylint: disable=too-many-locals with_source = buildNode(provider, context_expr, source_ref) if Options.isFullCompat(): source_ref = with_source.getCompatibleSourceReference() temp_scope = provider.allocateTempScope("with") tmp_source_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "source" ) tmp_exit_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "exit" ) tmp_enter_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "enter" ) tmp_indicator_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "indicator" ) statements = ( buildAssignmentStatements( provider = provider, node = assign_target, allow_none = True, source = ExpressionTempVariableRef( variable = tmp_enter_variable, source_ref = source_ref ), source_ref = source_ref ), body ) with_body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) if Options.isFullCompat(): if body: deepest = body while deepest.getVisitableNodes(): deepest = deepest.getVisitableNodes()[-1] body_lineno = deepest.getCompatibleSourceReference().getLineNumber() with_exit_source_ref = source_ref.atLineNumber(body_lineno) else: with_exit_source_ref = source_ref # The "__enter__" and "__exit__" were normal attribute lookups under # CPython2.6, but that changed with CPython2.7. if python_version < 270: attribute_lookup_class = ExpressionAttributeLookup else: attribute_lookup_class = ExpressionAttributeLookupSpecial enter_value = ExpressionCallEmpty( called = attribute_lookup_class( source = ExpressionTempVariableRef( variable = tmp_source_variable, source_ref = source_ref ), attribute_name = "__enter__" if sync else "__aenter__", source_ref = source_ref ), source_ref = source_ref ) exit_value_exception = ExpressionCallNoKeywords( called = ExpressionTempVariableRef( variable = tmp_exit_variable, source_ref = with_exit_source_ref ), args = ExpressionMakeTuple( elements = ( ExpressionCaughtExceptionTypeRef( source_ref = with_exit_source_ref ), ExpressionCaughtExceptionValueRef( source_ref = with_exit_source_ref ), ExpressionCaughtExceptionTracebackRef( source_ref = source_ref ), ), source_ref = source_ref ), source_ref = with_exit_source_ref ) exit_value_no_exception = ExpressionCallNoKeywords( called = ExpressionTempVariableRef( variable = tmp_exit_variable, source_ref = source_ref ), args = makeConstantRefNode( constant = (None, None, None), source_ref = source_ref ), source_ref = with_exit_source_ref ) # For "async with", await the entered value and exit value must be awaited. if not sync: enter_value = ExpressionAsyncWait( expression = enter_value, source_ref = source_ref ) exit_value_exception = ExpressionAsyncWait( expression = exit_value_exception, source_ref = source_ref ) exit_value_no_exception = ExpressionAsyncWait( expression = exit_value_no_exception, source_ref = source_ref ) statements = [ # First assign the with context to a temporary variable. StatementAssignmentVariable( variable = tmp_source_variable, source = with_source, source_ref = source_ref ) ] attribute_assignments = [ # Next, assign "__enter__" and "__exit__" attributes to temporary # variables. StatementAssignmentVariable( variable = tmp_exit_variable, source = attribute_lookup_class( source = ExpressionTempVariableRef( variable = tmp_source_variable, source_ref = source_ref ), attribute_name = "__exit__" if sync else "__aexit__", source_ref = source_ref ), source_ref = source_ref ), StatementAssignmentVariable( variable = tmp_enter_variable, source = enter_value, source_ref = source_ref ) ] if python_version >= 360 and sync: attribute_assignments.reverse() statements += attribute_assignments statements.append( StatementAssignmentVariable( variable = tmp_indicator_variable, source = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ) ) statements += [ makeTryFinallyStatement( provider = provider, tried = makeTryExceptSingleHandlerNodeWithPublish( provider = provider, tried = with_body, exception_name = "BaseException", handler_body = StatementsSequence( statements = ( # Prevents final block from calling __exit__ as # well. StatementAssignmentVariable( variable = tmp_indicator_variable, source = makeConstantRefNode( constant = False, source_ref = source_ref ), source_ref = source_ref ), makeConditionalStatement( condition = exit_value_exception, no_branch = makeReraiseExceptionStatement( source_ref = with_exit_source_ref ), yes_branch = None, source_ref = with_exit_source_ref ), ), source_ref = source_ref ), public_exc = python_version >= 270, source_ref = source_ref ), final = StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_indicator_variable, source_ref = source_ref ), right = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementExpressionOnly( expression = exit_value_no_exception, source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ), source_ref = source_ref ) ] return makeTryFinallyStatement( provider = provider, tried = statements, final = ( StatementReleaseVariable( variable = tmp_source_variable, source_ref = with_exit_source_ref ), StatementReleaseVariable( variable = tmp_enter_variable, source_ref = with_exit_source_ref ), StatementReleaseVariable( variable = tmp_exit_variable, source_ref = with_exit_source_ref ), StatementReleaseVariable( variable = tmp_indicator_variable, source_ref = with_exit_source_ref ), ), source_ref = source_ref ) def buildWithNode(provider, node, source_ref): # "with" statements are re-formulated as described in the developer # manual. Catches exceptions, and provides them to "__exit__", while making # the "__enter__" value available under a given name. # Before Python3.3, multiple context managers are not visible in the parse # tree, now we need to handle it ourselves. if hasattr(node, "items"): context_exprs = [item.context_expr for item in node.items] assign_targets = [item.optional_vars for item in node.items] else: # Make it a list for before Python3.3 context_exprs = [node.context_expr] assign_targets = [node.optional_vars] # The body for the first context manager is the other things. body = buildStatementsNode(provider, node.body, source_ref) assert context_exprs and len(context_exprs) == len(assign_targets) context_exprs.reverse() assign_targets.reverse() # For compatibility, we need to gather a line number for the body here # already, but only the full compatibility mode will use it. terminal_statement = node.body[-1] while getKind(terminal_statement) in ("With", "AsyncWith"): terminal_statement = terminal_statement.body[-1] body_lineno = terminal_statement.lineno for context_expr, assign_target in zip(context_exprs, assign_targets): body = _buildWithNode( provider = provider, body = body, body_lineno = body_lineno, context_expr = context_expr, assign_target = assign_target, sync = True, source_ref = source_ref ) return body def buildAsyncWithNode(provider, node, source_ref): # "with" statements are re-formulated as described in the developer # manual. Catches exceptions, and provides them to "__exit__", while making # the "__enter__" value available under a given name. # Before Python3.3, multiple context managers are not visible in the parse # tree, now we need to handle it ourselves. context_exprs = [item.context_expr for item in node.items] assign_targets = [item.optional_vars for item in node.items] # The body for the first context manager is the other things. body = buildStatementsNode(provider, node.body, source_ref) assert context_exprs and len(context_exprs) == len(assign_targets) context_exprs.reverse() assign_targets.reverse() # For compatibility, we need to gather a line number for the body here # already, but only the full compatibility mode will use it. terminal_statement = node.body[-1] while getKind(terminal_statement) in ("With", "AsyncWith"): terminal_statement = terminal_statement.body[-1] body_lineno = terminal_statement.lineno for context_expr, assign_target in zip(context_exprs, assign_targets): body = _buildWithNode( provider = provider, body = body, body_lineno = body_lineno, context_expr = context_expr, assign_target = assign_target, sync = False, source_ref = source_ref ) return body Nuitka-0.5.28.2/nuitka/tree/ReformulationWhileLoopStatements.py0000644000372000001440000001472213207537242025023 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of while loop statements. Loops in Nuitka have no condition attached anymore, so while loops are re-formulated like this: .. code-block:: python while condition: something() .. code-block:: python while 1: if not condition: break something() This is to totally remove the specialization of loops, with the condition moved to the loop body in an initial conditional statement, which contains a ``break`` statement. That achieves, that only ``break`` statements exit the loop, and allow for optimization to remove always true loop conditions, without concerning code generation about it, and to detect such a situation, consider e.g. endless loops. .. note:: Loop analysis (not yet done) can then work on a reduced problem (which ``break`` statements are executed under what conditions) and is then automatically very general. The fact that the loop body may not be entered at all, is still optimized, but also in the general sense. Explicit breaks at the loop start and loop conditions are the same. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildStatementsNode, makeStatementsSequence, mergeStatements, popBuildContext, pushBuildContext ) def buildWhileLoopNode(provider, node, source_ref): # The while loop is re-formulated according to developer manual. The # condition becomes an early condition to break the loop. The else block is # taken if a while loop exits normally, i.e. because of condition not being # true. We do this by introducing an indicator variable. else_block = buildStatementsNode( provider = provider, nodes = node.orelse if node.orelse else None, source_ref = source_ref ) if else_block is not None: temp_scope = provider.allocateTempScope("while_loop") tmp_break_indicator = provider.allocateTempVariable( temp_scope = temp_scope, name = "break_indicator" ) statements = ( StatementAssignmentVariable( variable = tmp_break_indicator, source = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ), StatementLoopBreak( source_ref = source_ref ) ) else: statements = ( StatementLoopBreak( source_ref = source_ref ), ) pushBuildContext("loop_body") loop_statements = buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ) popBuildContext() # The loop body contains a conditional statement at the start that breaks # the loop if it fails. loop_body = makeStatementsSequence( statements = ( StatementConditional( condition = ExpressionOperationNOT( operand = buildNode(provider, node.test, source_ref), source_ref = source_ref, ), yes_branch = StatementsSequence( statements = statements, source_ref = source_ref ), no_branch = None, source_ref = source_ref ), loop_statements ), allow_none = True, source_ref = source_ref ) loop_statement = StatementLoop( body = loop_body, source_ref = source_ref ) if else_block is None: return loop_statement else: statements = ( StatementAssignmentVariable( variable = tmp_break_indicator, source = makeConstantRefNode( constant = False, source_ref = source_ref ), source_ref = source_ref ), loop_statement, StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), right = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = else_block, no_branch = None, source_ref = source_ref ) ) statements = ( makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_break_indicator, source_ref = source_ref ), source_ref = source_ref ), ) return StatementsSequence( statements = mergeStatements(statements, False), source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/ReformulationCallExpressions.py0000644000372000001440000002752113207537242024170 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of call expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import StatementAssignmentVariable from nuitka.nodes.CallNodes import makeExpressionCall from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.FunctionNodes import ( ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.OutlineNodes import ExpressionOutlineBody from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from nuitka.PythonVersions import python_version from .ComplexCallHelperFunctions import ( getFunctionCallHelperDictionaryUnpacking, getFunctionCallHelperKeywordsStarDict, getFunctionCallHelperKeywordsStarList, getFunctionCallHelperKeywordsStarListStarDict, getFunctionCallHelperPosKeywordsStarDict, getFunctionCallHelperPosKeywordsStarList, getFunctionCallHelperPosKeywordsStarListStarDict, getFunctionCallHelperPosStarDict, getFunctionCallHelperPosStarList, getFunctionCallHelperPosStarListStarDict, getFunctionCallHelperStarDict, getFunctionCallHelperStarList, getFunctionCallHelperStarListStarDict ) from .ReformulationDictionaryCreation import buildDictionaryUnpackingArgs from .ReformulationSequenceCreation import buildListUnpacking from .TreeHelpers import ( buildNode, buildNodeList, getKind, makeDictCreationOrConstant, makeSequenceCreationOrConstant, makeStatementsSequenceFromStatements ) def buildCallNode(provider, node, source_ref): called = buildNode(provider, node.func, source_ref) if python_version >= 350: list_star_arg = None dict_star_arg = None positional_args = [] # For Python3.5 compatibility, the error handling with star argument last # is the old one, only with a starred argument before that, things use the # new unpacking code. for node_arg in node.args[:-1]: if getKind(node_arg) == "Starred": assert python_version >= 350 list_star_arg = buildListUnpacking(provider, node.args, source_ref) positional_args = [] break else: if node.args and getKind(node.args[-1]) == "Starred": assert python_version >= 350 list_star_arg = buildNode(provider, node.args[-1].value, source_ref) positional_args = buildNodeList(provider, node.args[:-1], source_ref) else: positional_args = buildNodeList(provider, node.args, source_ref) # Only the values of keyword pairs have a real source ref, and those only # really matter, so that makes sense. keys = [] values = [] for keyword in node.keywords[:-1]: if keyword.arg is None: assert python_version >= 350 outline_body = ExpressionOutlineBody( provider = provider, name = "dict_unpacking_call", source_ref = source_ref ) tmp_called = outline_body.allocateTempVariable( temp_scope = None, name = "called" ) helper_args = [ ExpressionTempVariableRef( variable = tmp_called, source_ref = source_ref ), ExpressionMakeTuple( elements = buildDictionaryUnpackingArgs( provider = provider, keys = (keyword.arg for keyword in node.keywords), values = (keyword.value for keyword in node.keywords), source_ref = source_ref ), source_ref = source_ref ) ] dict_star_arg = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getFunctionCallHelperDictionaryUnpacking(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = helper_args, source_ref = source_ref, ) outline_body.setBody( makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_called, source = called, source_ref = source_ref ), StatementReturn( expression = _makeCallNode( called = ExpressionTempVariableRef( variable = tmp_called, source_ref = source_ref ), positional_args = positional_args, keys = keys, values = values, list_star_arg = list_star_arg, dict_star_arg = dict_star_arg, source_ref = source_ref, ), source_ref = source_ref ) ) ) return outline_body # For Python3.5 compatibility, the error handling with star argument last # is the old one, only with a starred argument before that, things use the # new unpacking code. if node.keywords and node.keywords[-1].arg is None: assert python_version >= 350 dict_star_arg = buildNode(provider, node.keywords[-1].value, source_ref) keywords = node.keywords[:-1] else: keywords = node.keywords for keyword in keywords: keys.append( makeConstantRefNode( constant = keyword.arg, source_ref = source_ref, user_provided = True ) ) values.append( buildNode(provider, keyword.value, source_ref) ) if python_version < 350: list_star_arg = buildNode(provider, node.starargs, source_ref, True) dict_star_arg = buildNode(provider, node.kwargs, source_ref, True) return _makeCallNode( called = called, positional_args = positional_args, keys = keys, values = values, list_star_arg = list_star_arg, dict_star_arg = dict_star_arg, source_ref = source_ref, ) def _makeCallNode(called, positional_args, keys, values, list_star_arg, dict_star_arg, source_ref): # Many variables, but only to cover the many complex call cases. if list_star_arg is None and dict_star_arg is None: result = makeExpressionCall( called = called, args = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = positional_args, source_ref = source_ref ), kw = makeDictCreationOrConstant( keys = keys, values = values, source_ref = source_ref ), source_ref = source_ref, ) if values: result.setCompatibleSourceReference( source_ref = values[-1].getCompatibleSourceReference() ) elif positional_args: result.setCompatibleSourceReference( source_ref = positional_args[-1].getCompatibleSourceReference() ) return result else: # Dispatch to complex helper function for each case. These do # re-formulation of complex calls according to developer manual. key = ( bool(positional_args), bool(keys), list_star_arg is not None, dict_star_arg is not None ) table = { (True, True, True, False) : getFunctionCallHelperPosKeywordsStarList, (True, False, True, False) : getFunctionCallHelperPosStarList, (False, True, True, False) : getFunctionCallHelperKeywordsStarList, (False, False, True, False) : getFunctionCallHelperStarList, (True, True, False, True) : getFunctionCallHelperPosKeywordsStarDict, (True, False, False, True) : getFunctionCallHelperPosStarDict, (False, True, False, True) : getFunctionCallHelperKeywordsStarDict, (False, False, False, True) : getFunctionCallHelperStarDict, (True, True, True, True) : getFunctionCallHelperPosKeywordsStarListStarDict, (True, False, True, True) : getFunctionCallHelperPosStarListStarDict, (False, True, True, True) : getFunctionCallHelperKeywordsStarListStarDict, (False, False, True, True) : getFunctionCallHelperStarListStarDict, } get_helper = table[key] helper_args = [called] if positional_args: helper_args.append( makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = positional_args, source_ref = source_ref ) ) # Order of evaluation changed in Python3.5. if python_version >= 350 and list_star_arg is not None: helper_args.append(list_star_arg) if keys: helper_args.append( makeDictCreationOrConstant( keys = keys, values = values, source_ref = source_ref ) ) # Order of evaluation changed in Python3.5. if python_version < 350 and list_star_arg is not None: helper_args.append(list_star_arg) if dict_star_arg is not None: helper_args.append(dict_star_arg) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = get_helper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = helper_args, source_ref = source_ref, ) result.setCompatibleSourceReference( source_ref = helper_args[-1].getCompatibleSourceReference() ) return result Nuitka-0.5.28.2/nuitka/tree/ReformulationContractionExpressions.py0000644000372000001440000005053513207537242025601 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of contraction expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.__past__ import intern # pylint: disable=I0021,redefined-builtin from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ExpressionBuiltinIter1 from nuitka.nodes.BuiltinNextNodes import ExpressionBuiltinNext1 from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerOperationNodes import ( StatementListOperationAppend, StatementSetOperationAdd ) from nuitka.nodes.DictionaryNodes import StatementDictOperationSet from nuitka.nodes.FrameNodes import ( StatementsFrameFunction, StatementsFrameGenerator ) from nuitka.nodes.FunctionNodes import ExpressionFunctionRef from nuitka.nodes.GeneratorNodes import ( ExpressionGeneratorObjectBody, ExpressionMakeGeneratorObject ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.NodeMakingHelpers import makeVariableRefNode from nuitka.nodes.OutlineNodes import ( ExpressionOutlineBody, ExpressionOutlineFunction ) from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import ( StatementExpressionOnly, StatementsSequence ) from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from nuitka.nodes.YieldNodes import ExpressionYield from nuitka.PythonVersions import python_version from .ReformulationAssignmentStatements import buildAssignmentStatements from .ReformulationBooleanExpressions import buildAndNode from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildNodeList, getKind, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, mergeStatements ) def _buildPython2ListContraction(provider, node, source_ref): # The contraction nodes are reformulated to function bodies, with loops as # described in the developer manual. They use a lot of temporary names, # nested blocks, etc. and so a lot of variable names. # Note: The assign_provider is only to cover Python2 list contractions, # assigning one of the loop variables to the outside scope. function_body = ExpressionOutlineBody( provider = provider, name = "list_contraction", source_ref = source_ref ) iter_tmp = function_body.allocateTempVariable( temp_scope = None, name = ".0" ) container_tmp = function_body.allocateTempVariable( temp_scope = None, name = "contraction_result" ) statements, release_statements = _buildContractionBodyNode( function_body = function_body, assign_provider = True, provider = provider, node = node, emit_class = StatementListOperationAppend, iter_tmp = iter_tmp, temp_scope = None, start_value = [], container_tmp = container_tmp, source_ref = source_ref, ) statements.append( StatementReturn( expression = ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), source_ref = source_ref ) ) statement = makeTryFinallyStatement( provider = function_body, tried = statements, final = release_statements, source_ref = source_ref.atInternal() ) function_body.setBody( makeStatementsSequenceFromStatement( statement = statement ) ) return function_body def buildListContractionNode(provider, node, source_ref): # List contractions are dealt with by general code. if python_version < 300: return _buildPython2ListContraction( provider = provider, node = node, source_ref = source_ref ) return _buildContractionNode( provider = provider, node = node, name = "", emit_class = StatementListOperationAppend, start_value = [], source_ref = source_ref ) def buildSetContractionNode(provider, node, source_ref): # Set contractions are dealt with by general code. return _buildContractionNode( provider = provider, node = node, name = "", emit_class = StatementSetOperationAdd, start_value = set(), source_ref = source_ref ) def buildDictContractionNode(provider, node, source_ref): # Dict contractions are dealt with by general code. return _buildContractionNode( provider = provider, node = node, name = "", emit_class = StatementDictOperationSet, start_value = {}, source_ref = source_ref ) def buildGeneratorExpressionNode(provider, node, source_ref): # Generator expressions are dealt with by general code. assert getKind(node) == "GeneratorExp" function_body = ExpressionOutlineBody( provider = provider, name = "genexpr", source_ref = source_ref ) iter_tmp = function_body.allocateTempVariable( temp_scope = None, name = ".0" ) parent_module = provider.getParentModule() code_object = CodeObjectSpec( co_name = "", co_kind = "Generator", co_varnames = (".0",), co_argcount = 1, co_kwonlyargcount = 0, co_has_starlist = False, co_has_stardict = False, co_filename = parent_module.getRunTimeFilename(), co_lineno = source_ref.getLineNumber(), future_spec = parent_module.getFutureSpec() ) code_body = ExpressionGeneratorObjectBody( provider = provider, name = "", flags = set(), source_ref = source_ref ) function_body.setBody( makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = iter_tmp, source = ExpressionBuiltinIter1( value = buildNode( provider = provider, node = node.generators[0].iter, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ), makeTryFinallyStatement( provider = function_body, tried = StatementReturn( expression = ExpressionMakeGeneratorObject( generator_ref = ExpressionFunctionRef( function_body = code_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ), final = StatementReleaseVariable( variable = iter_tmp, source_ref = source_ref ), source_ref = source_ref ) ) ) statements, release_statements = _buildContractionBodyNode( function_body = code_body, provider = provider, node = node, emit_class = ExpressionYield, iter_tmp = iter_tmp, temp_scope = None, start_value = None, container_tmp = None, assign_provider = False, source_ref = source_ref, ) statements = ( makeTryFinallyStatement( provider = function_body, tried = statements, final = release_statements, source_ref = source_ref.atInternal() ), ) code_body.setBody( makeStatementsSequenceFromStatement( statement = StatementsFrameGenerator( statements = mergeStatements(statements, False), code_object = code_object, source_ref = source_ref ) ) ) return function_body def _buildContractionBodyNode(provider, node, emit_class, start_value, container_tmp, iter_tmp, temp_scope, assign_provider, function_body, source_ref): # This uses lots of variables and branches. There is no good way # around that, and we deal with many cases, due to having generator # expressions sharing this code, pylint: disable=too-many-branches,too-many-locals # Note: The assign_provider is only to cover Python2 list contractions, # assigning one of the loop variables to the outside scope. tmp_variables = [] if emit_class is not ExpressionYield: tmp_variables.append(iter_tmp) if container_tmp is not None: tmp_variables.append(container_tmp) # First assign the iterator if we are an outline. if assign_provider: statements = [ StatementAssignmentVariable( variable = iter_tmp, source = ExpressionBuiltinIter1( value = buildNode( provider = provider, node = node.generators[0].iter, source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref.atInternal() ) ] else: statements = [] if start_value is not None: statements.append( StatementAssignmentVariable( variable = container_tmp, source = makeConstantRefNode( constant = start_value, source_ref = source_ref ), source_ref = source_ref.atInternal() ) ) if hasattr(node, "elt"): if start_value is not None: current_body = emit_class( ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), buildNode( provider = function_body, node = node.elt, source_ref = source_ref ), source_ref = source_ref ) else: assert emit_class is ExpressionYield current_body = emit_class( buildNode( provider = function_body, node = node.elt, source_ref = source_ref ), source_ref = source_ref ) else: assert emit_class is StatementDictOperationSet current_body = StatementDictOperationSet( dict_arg = ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), key = buildNode( provider = function_body, node = node.key, source_ref = source_ref, ), value = buildNode( provider = function_body, node = node.value, source_ref = source_ref, ), source_ref = source_ref ) if current_body.isExpression(): current_body = StatementExpressionOnly( expression = current_body, source_ref = source_ref ) for count, qual in enumerate(reversed(node.generators)): tmp_value_variable = function_body.allocateTempVariable( temp_scope = temp_scope, name = "iter_value_%d" % count ) tmp_variables.append(tmp_value_variable) # The first iterated value is to be calculated outside of the function # and will be given as a parameter "_iterated", the others are built # inside the function. if qual is node.generators[0]: iterator_ref = makeVariableRefNode( variable = iter_tmp, source_ref = source_ref ) tmp_iter_variable = None nested_statements = [] else: # First create the iterator and store it, next should be loop body value_iterator = ExpressionBuiltinIter1( value = buildNode( provider = function_body, node = qual.iter, source_ref = source_ref ), source_ref = source_ref ) tmp_iter_variable = function_body.allocateTempVariable( temp_scope = temp_scope, name = "contraction_iter_%d" % count ) tmp_variables.append(tmp_iter_variable) nested_statements = [ StatementAssignmentVariable( variable = tmp_iter_variable, source = value_iterator, source_ref = source_ref ) ] iterator_ref = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ) loop_statements = [ makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_value_variable, source = ExpressionBuiltinNext1( value = iterator_ref, source_ref = source_ref ), source_ref = source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = source_ref.atInternal() ), source_ref = source_ref ), buildAssignmentStatements( provider = provider if assign_provider else function_body, temp_provider = function_body, node = qual.target, source = ExpressionTempVariableRef( variable = tmp_value_variable, source_ref = source_ref ), source_ref = source_ref ) ] conditions = buildNodeList( provider = function_body, nodes = qual.ifs, source_ref = source_ref ) if len(conditions) >= 1: loop_statements.append( StatementConditional( condition = buildAndNode( values = conditions, source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = current_body ), no_branch = None, source_ref = source_ref ) ) else: loop_statements.append(current_body) nested_statements.append( StatementLoop( body = StatementsSequence( statements = mergeStatements(loop_statements), source_ref = source_ref ), source_ref = source_ref ) ) if tmp_iter_variable is not None: nested_statements.append( StatementReleaseVariable( variable = tmp_iter_variable, source_ref = source_ref ) ) current_body = StatementsSequence( statements = mergeStatements(nested_statements, False), source_ref = source_ref ) statements.append(current_body) statements = mergeStatements(statements) release_statements = [ StatementReleaseVariable( variable = tmp_variable, source_ref = source_ref ) for tmp_variable in tmp_variables ] return statements, release_statements def _buildContractionNode(provider, node, name, emit_class, start_value, source_ref): # The contraction nodes are reformulated to function bodies, with loops as # described in the developer manual. They use a lot of temporary names, # nested blocks, etc. and so a lot of variable names. function_body = ExpressionOutlineFunction( provider = provider, name = intern(name[1:-1]), source_ref = source_ref ) iter_tmp = function_body.allocateTempVariable( temp_scope = None, name = ".0" ) container_tmp = function_body.allocateTempVariable( temp_scope = None, name = "contraction" ) statements, release_statements = _buildContractionBodyNode( function_body = function_body, provider = provider, node = node, emit_class = emit_class, iter_tmp = iter_tmp, temp_scope = None, start_value = start_value, container_tmp = container_tmp, assign_provider = False, source_ref = source_ref, ) assign_iter_statement = StatementAssignmentVariable( source = ExpressionBuiltinIter1( value = buildNode( provider = provider, node = node.generators[0].iter, source_ref = source_ref ), source_ref = source_ref ), variable = iter_tmp, source_ref = source_ref ) statements.append( StatementReturn( expression = ExpressionTempVariableRef( variable = container_tmp, source_ref = source_ref ), source_ref = source_ref ) ) statements = ( makeTryFinallyStatement( provider = function_body, tried = statements, final = release_statements, source_ref = source_ref.atInternal() ), ) if python_version < 300: body = makeStatementsSequenceFromStatements( assign_iter_statement, statements ) else: parent_module = provider.getParentModule() code_object = CodeObjectSpec( co_name = name, co_kind = "Function", co_varnames = (), co_argcount = 1, co_kwonlyargcount = 0, co_has_starlist = False, co_has_stardict = False, co_filename = parent_module.getRunTimeFilename(), co_lineno = source_ref.getLineNumber(), future_spec = parent_module.getFutureSpec() ) body = makeStatementsSequenceFromStatements( assign_iter_statement, StatementsFrameFunction( statements = mergeStatements(statements, False), code_object = code_object, source_ref = source_ref ) ) function_body.setBody(body) return function_body Nuitka-0.5.28.2/nuitka/tree/ReformulationNamespacePackages.py0000644000372000001440000001570313207537242024404 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Namespace packages of Python3.3 """ import os from nuitka import Options from nuitka.nodes.AssignNodes import StatementAssignmentVariableName from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.CallNodes import ExpressionCallNoKeywords from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ( ExpressionMakeList, ExpressionMakeTuple ) from nuitka.nodes.FutureSpecs import FutureSpec from nuitka.nodes.ImportNodes import ( ExpressionImportModuleNameHard, ExpressionImportName ) from nuitka.nodes.ModuleNodes import ( CompiledPythonPackage, ExpressionModuleFileAttributeRef ) from nuitka.PythonVersions import python_version from nuitka.SourceCodeReferences import SourceCodeReference from .TreeHelpers import ( makeAbsoluteImportNode, makeStatementsSequenceFromStatement ) from .VariableClosure import completeVariableClosures def createPathAssignment(package, source_ref): if Options.getFileReferenceMode() == "original": path_value = makeConstantRefNode( constant = [ os.path.dirname( source_ref.getFilename() ) ], source_ref = source_ref, user_provided = True ) else: elements = [ ExpressionCallNoKeywords( called = ExpressionAttributeLookup( source = ExpressionImportModuleNameHard( module_name = "os", import_name = "path", source_ref = source_ref ), attribute_name = "dirname", source_ref = source_ref ), args = ExpressionMakeTuple( elements = ( ExpressionModuleFileAttributeRef( source_ref = source_ref, ), ), source_ref = source_ref, ), source_ref = source_ref, ) ] def makeCall(module_name, import_name, attribute_name, *args): return ExpressionCallNoKeywords( called = ExpressionAttributeLookup( source = ExpressionImportModuleNameHard( module_name = module_name, import_name = import_name, source_ref = source_ref ), attribute_name = attribute_name, source_ref = source_ref ), args = ExpressionMakeTuple( elements = args, source_ref = source_ref ), source_ref = source_ref ) if package.canHaveExternalImports(): parts = package.getFullName().split('.') for count in range(len(parts)): path_part = makeCall( "os", "environ", "get", makeConstantRefNode( constant = "NUITKA_PACKAGE_%s" % '_'.join( parts[:count+1] ), source_ref = source_ref, ), makeConstantRefNode( constant = "/notexist", source_ref = source_ref, ) ) if parts[count+1:]: path_part = makeCall( "os", "path", "join", path_part, makeConstantRefNode( constant = os.path.join(*parts[count+1:]), source_ref = source_ref, ) ) elements.append(path_part) path_value = ExpressionMakeList( elements = elements, source_ref = source_ref ) return StatementAssignmentVariableName( variable_name = "__path__", source = path_value, source_ref = source_ref ) def createPython3NamespacePath(package_name, module_relpath, source_ref): return StatementAssignmentVariableName( variable_name = "__path__", source = ExpressionCallNoKeywords( called = ExpressionImportName( module = makeAbsoluteImportNode( module_name = "_frozen_importlib" if python_version < 350 else "_frozen_importlib_external", source_ref = source_ref ), import_name = "_NamespacePath", source_ref = source_ref ), args = makeConstantRefNode( constant = ( package_name, [module_relpath], None ), source_ref = source_ref ), source_ref = source_ref ), source_ref = source_ref ) def createNamespacePackage(package_name, module_relpath): parts = package_name.split('.') source_ref = SourceCodeReference.fromFilenameAndLine( filename = module_relpath, line = 1 ) source_ref = source_ref.atInternal() package_package_name = '.'.join(parts[:-1]) or None package = CompiledPythonPackage( name = parts[-1], mode = "compiled", package_name = package_package_name, future_spec = FutureSpec(), source_ref = source_ref, ) if python_version >= 300: statement = createPython3NamespacePath( package_name = package_name, module_relpath = module_relpath, source_ref = source_ref ) else: statement = createPathAssignment(package, source_ref) package.setBody( makeStatementsSequenceFromStatement( statement = statement ) ) completeVariableClosures(package) return source_ref, package Nuitka-0.5.28.2/nuitka/tree/SyntaxErrors.py0000644000372000001440000000707613122472300020756 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Handling of syntax errors. Format SyntaxError/IndentationError exception for output, as well as raise it for the given source code reference. """ def formatOutput(e): if len(e.args) > 1: reason, (filename, lineno, colno, message) = e.args if message is None and colno is not None: colno = None if lineno is not None and lineno == 0: lineno = 1 else: reason, = e.args filename = None lineno = None colno = None message = None # On CPython3.4 at least, this attribute appears to override reason for # SyntaxErrors at least. if hasattr(e, "msg"): reason = e.msg if colno is not None: colno = colno - len(message) + len(message.lstrip()) return """\ File "%s", line %d %s %s^ %s: %s""" % ( filename, lineno, message.strip(), ' ' * (colno-1) if colno is not None else "", e.__class__.__name__, reason ) elif message is not None: return """\ File "%s", line %d %s %s: %s""" % ( filename, lineno, message.strip(), e.__class__.__name__, reason ) elif filename is not None: return """\ File "%s", line %s %s: %s""" % ( filename, lineno, e.__class__.__name__, reason ) else: return """\ %s: %s""" % ( e.__class__.__name__, reason ) def raiseSyntaxError(reason, source_ref, display_file = True, display_line = True): col_offset = source_ref.getColumnNumber() def readSource(): import linecache return linecache.getline( filename = source_ref.getFilename(), lineno = source_ref.getLineNumber() ) if display_file and display_line: source_line = readSource() raise SyntaxError( reason, ( source_ref.getFilename(), source_ref.getLineNumber(), col_offset, source_line ) ) else: if source_ref is not None: if display_line: source_line = readSource() else: source_line = None raise SyntaxError( reason, ( source_ref.getFilename(), source_ref.getLineNumber(), col_offset, source_line ) ) else: raise SyntaxError( reason, ( None, None, None, None ) ) Nuitka-0.5.28.2/nuitka/tree/ReformulationFunctionStatements.py0000644000372000001440000006425213207537242024711 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of function statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementAssignmentVariableName, StatementReleaseVariable ) from nuitka.nodes.AsyncgenNodes import ( ExpressionAsyncgenObjectBody, ExpressionMakeAsyncgenObject ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionBuiltinIter1, StatementSpecialUnpackCheck ) from nuitka.nodes.BuiltinNextNodes import ExpressionSpecialUnpack from nuitka.nodes.BuiltinRefNodes import makeExpressionBuiltinRef from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.CoroutineNodes import ( ExpressionCoroutineObjectBody, ExpressionMakeCoroutineObject ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.GeneratorNodes import ( ExpressionGeneratorObjectBody, ExpressionMakeGeneratorObject, StatementGeneratorReturnNone ) from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn, StatementReturnNone from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableNameRef ) from nuitka.PythonVersions import python_version from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .SyntaxErrors import raiseSyntaxError from .TreeHelpers import ( buildFrameNode, buildNode, buildNodeList, detectFunctionBodyKind, extractDocFromBody, getKind, makeCallNode, makeDictCreationOrConstant2, makeStatementsSequenceFromStatement, mangleName ) def _insertFinalReturnStatement(function_statements_body, return_statement): if function_statements_body is None: function_statements_body = makeStatementsSequenceFromStatement( statement = return_statement ) elif not function_statements_body.isStatementAborting(): function_statements_body.setStatements( function_statements_body.getStatements() + ( return_statement, ) ) return function_statements_body def buildFunctionNode(provider, node, source_ref): # Functions have way too many details, pylint: disable=too-many-branches,too-many-locals assert getKind(node) == "FunctionDef" function_statement_nodes, function_doc = extractDocFromBody(node) function_kind, flags = detectFunctionBodyKind( nodes = function_statement_nodes ) outer_body, function_body, code_object = buildFunctionWithParsing( provider = provider, function_kind = function_kind, name = node.name, function_doc = function_doc, flags = flags, node = node, source_ref = source_ref ) if function_kind == "Function": code_body = function_body elif function_kind == "Generator": code_body = ExpressionGeneratorObjectBody( provider = function_body, name = node.name, flags = flags, source_ref = source_ref ) code_body.qualname_provider = provider for variable in function_body.getVariables(): code_body.getVariableForReference(variable.getName()) else: assert False, function_kind if function_kind == "Generator": function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionMakeGeneratorObject( generator_ref = ExpressionFunctionRef( function_body = code_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ) ) ) decorators = buildNodeList( provider = provider, nodes = reversed(node.decorator_list), source_ref = source_ref ) defaults = buildNodeList( provider = provider, nodes = node.args.defaults, source_ref = source_ref ) kw_defaults = buildParameterKwDefaults( provider = provider, node = node, function_body = function_body, source_ref = source_ref ) function_statements_body = buildFrameNode( provider = code_body, nodes = function_statement_nodes, code_object = code_object, source_ref = source_ref ) if function_kind == "Function": # TODO: Generators might have to raise GeneratorExit instead. function_statements_body = _insertFinalReturnStatement( function_statements_body = function_statements_body, return_statement = StatementReturnNone( source_ref = source_ref ) ) if function_statements_body.isStatementsFrame(): function_statements_body = makeStatementsSequenceFromStatement( statement = function_statements_body ) code_body.setBody( function_statements_body ) annotations = buildParameterAnnotations(provider, node, source_ref) function_creation = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = outer_body, source_ref = source_ref ), code_object = code_object, defaults = defaults, kw_defaults = kw_defaults, annotations = annotations, source_ref = source_ref ) # Add the "staticmethod" decorator to __new__ methods if not provided. # CPython made these optional, but secretly applies them when it does # "class __new__". We add them earlier, so our optimization will see it. if node.name == "__new__" and \ provider.isExpressionClassBody(): for decorator in decorators: if decorator.isExpressionVariableNameRef() and \ decorator.getVariableName() == "staticmethod": break else: decorators.append( makeExpressionBuiltinRef( builtin_name = "staticmethod", source_ref = source_ref ) ) if python_version >= 360 and \ node.name == "__init_subclass__" and \ provider.isExpressionClassBody(): for decorator in decorators: if decorator.isExpressionVariableNameRef() and \ decorator.getVariableName() == "classmethod": break else: decorators.append( makeExpressionBuiltinRef( builtin_name = "classmethod", source_ref = source_ref ) ) decorated_function = function_creation for decorator in decorators: decorated_function = makeCallNode( decorator, decorated_function, decorator.getSourceReference() ) result = StatementAssignmentVariableName( variable_name = mangleName(node.name, provider), source = decorated_function, source_ref = source_ref ) if python_version >= 340: function_body.qualname_setup = result.getVariableName() return result def buildAsyncFunctionNode(provider, node, source_ref): # We are creating a function here that creates coroutine objects, with # many details each, pylint: disable=too-many-locals assert getKind(node) == "AsyncFunctionDef" function_statement_nodes, function_doc = extractDocFromBody(node) function_kind, flags = detectFunctionBodyKind( nodes = function_statement_nodes, start_value = "Coroutine" ) creator_function_body, _, code_object = buildFunctionWithParsing( provider = provider, function_kind = function_kind, name = node.name, flags = (), function_doc = function_doc, node = node, source_ref = source_ref ) if function_kind == "Coroutine": function_body = ExpressionCoroutineObjectBody( provider = creator_function_body, name = node.name, flags = flags, source_ref = source_ref ) else: function_body = ExpressionAsyncgenObjectBody( provider = creator_function_body, name = node.name, flags = flags, source_ref = source_ref ) for variable in creator_function_body.getVariables(): function_body.getVariableForReference(variable.getName()) decorators = buildNodeList( provider = provider, nodes = reversed(node.decorator_list), source_ref = source_ref ) defaults = buildNodeList( provider = provider, nodes = node.args.defaults, source_ref = source_ref ) function_statements_body = buildFrameNode( provider = function_body, nodes = function_statement_nodes, code_object = code_object, source_ref = source_ref ) function_statements_body = _insertFinalReturnStatement( function_statements_body = function_statements_body, return_statement = StatementGeneratorReturnNone( source_ref = source_ref ) ) if function_statements_body.isStatementsFrame(): function_statements_body = makeStatementsSequenceFromStatement( statement = function_statements_body ) function_body.setBody( function_statements_body ) annotations = buildParameterAnnotations(provider, node, source_ref) kw_defaults = buildParameterKwDefaults( provider = provider, node = node, function_body = creator_function_body, source_ref = source_ref ) if function_kind == "Coroutine": creation_node = ExpressionMakeCoroutineObject( coroutine_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ) else: creation_node = ExpressionMakeAsyncgenObject( asyncgen_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ) creator_function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = creation_node, source_ref = source_ref ) ) ) function_creation = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = creator_function_body, source_ref = source_ref ), code_object = code_object, defaults = defaults, kw_defaults = kw_defaults, annotations = annotations, source_ref = source_ref ) decorated_function = function_creation for decorator in decorators: decorated_function = makeCallNode( decorator, decorated_function, decorator.getSourceReference() ) result = StatementAssignmentVariableName( variable_name = mangleName(node.name, provider), source = decorated_function, source_ref = source_ref ) function_body.qualname_setup = result.getVariableName() # Share the non-local declarations. TODO: This may also apply to generators # and async generators. creator_function_body.non_local_declarations = function_body.non_local_declarations return result def buildParameterKwDefaults(provider, node, function_body, source_ref): # Build keyword only arguments default values. We are hiding here, that it # is a Python3 only feature. if python_version >= 300: kw_only_names = function_body.getParameters().getKwOnlyParameterNames() if kw_only_names: keys = [] values = [] for kw_only_name, kw_default in \ zip(kw_only_names, node.args.kw_defaults): if kw_default is not None: keys.append(kw_only_name) values.append( buildNode(provider, kw_default, source_ref) ) kw_defaults = makeDictCreationOrConstant2( keys = keys, values = values, source_ref = source_ref ) else: kw_defaults = None else: kw_defaults = None return kw_defaults def buildParameterAnnotations(provider, node, source_ref): # Too many branches, because there is too many cases, pylint: disable=too-many-branches # Build annotations. We are hiding here, that it is a Python3 only feature. if python_version < 300: return None # Starting with Python 3.4, the names of parameters are mangled in # annotations as well. if python_version < 340: mangle = lambda variable_name: variable_name else: mangle = lambda variable_name: mangleName(variable_name, provider) keys = [] values = [] def addAnnotation(key, value): keys.append(mangle(key)) values.append(value) def extractArg(arg): if getKind(arg) == "Name": assert arg.annotation is None elif getKind(arg) == "arg": if arg.annotation is not None: addAnnotation( key = arg.arg, value = buildNode(provider, arg.annotation, source_ref) ) elif getKind(arg) == "Tuple": for sub_arg in arg.elts: extractArg(sub_arg) else: assert False, getKind(arg) for arg in node.args.args: extractArg(arg) for arg in node.args.kwonlyargs: extractArg(arg) if python_version < 340: if node.args.varargannotation is not None: addAnnotation( key = node.args.vararg, value = buildNode( provider, node.args.varargannotation, source_ref ) ) if node.args.kwargannotation is not None: addAnnotation( key = node.args.kwarg, value = buildNode( provider, node.args.kwargannotation, source_ref ) ) else: if node.args.vararg is not None: extractArg(node.args.vararg) if node.args.kwarg is not None: extractArg(node.args.kwarg) # Return value annotation (not there for lambdas) if hasattr(node, "returns") and node.returns is not None: addAnnotation( key = "return", value = buildNode( provider, node.returns, source_ref ) ) if keys: return makeDictCreationOrConstant2( keys = keys, values = values, source_ref = source_ref ) else: return None def buildFunctionWithParsing(provider, function_kind, name, function_doc, flags, node, source_ref): # This contains a complex re-formulation for nested parameter functions. # pylint: disable=too-many-locals kind = getKind(node) assert kind in ("FunctionDef", "Lambda", "AsyncFunctionDef"), "unsupported for kind " + kind def extractArg(arg): if arg is None: return None elif type(arg) is str: return mangleName(arg, provider) elif getKind(arg) == "Name": return mangleName(arg.id, provider) elif getKind(arg) == "arg": return mangleName(arg.arg, provider) elif getKind(arg) == "Tuple": # These are to be re-formulated on the outside. assert False else: assert False, getKind(arg) special_args = {} def extractNormalArgs(args): normal_args = [] for arg in args: if type(arg) is not str and getKind(arg) == "Tuple": special_arg_name = ".%d" % (len(special_args) + 1) special_args[special_arg_name] = arg.elts normal_args.append(special_arg_name) else: normal_args.append(extractArg(arg)) return normal_args normal_args = extractNormalArgs(node.args.args) parameters = ParameterSpec( ps_name = name, ps_normal_args = normal_args, ps_kw_only_args = [ extractArg(arg) for arg in node.args.kwonlyargs ] if python_version >= 300 else [], ps_list_star_arg = extractArg(node.args.vararg), ps_dict_star_arg = extractArg(node.args.kwarg), ps_default_count = len(node.args.defaults) ) message = parameters.checkParametersValid() if message is not None: raiseSyntaxError( message, source_ref.atColumnNumber(node.col_offset), ) parent_module = provider.getParentModule() code_object = CodeObjectSpec( co_name = name, co_kind = function_kind, co_varnames = parameters.getParameterNames(), co_argcount = parameters.getArgumentCount(), co_kwonlyargcount = parameters.getKwOnlyParameterCount(), co_has_starlist = parameters.getStarListArgumentName() is not None, co_has_stardict = parameters.getStarDictArgumentName() is not None, co_filename = parent_module.getRunTimeFilename(), co_lineno = source_ref.getLineNumber(), future_spec = parent_module.getFutureSpec() ) outer_body = ExpressionFunctionBody( provider = provider, name = name, flags = flags, doc = function_doc, parameters = parameters, source_ref = source_ref ) if special_args: inner_name = name.strip("<>") + "$inner" inner_arg_names = [] iter_vars = [] values = [] statements = [] def unpackFrom(source, arg_names): accesses = [] sub_special_index = 0 iter_var = outer_body.allocateTempVariable(None, "arg_iter_%d" % len(iter_vars)) iter_vars.append(iter_var) statements.append( StatementAssignmentVariable( variable = iter_var, source = ExpressionBuiltinIter1( value = source, source_ref = source_ref ), source_ref = source_ref ) ) for element_index, arg_name in enumerate(arg_names): if getKind(arg_name) == "Name": inner_arg_names.append(arg_name.id) arg_var = outer_body.allocateTempVariable(None, "tmp_" + arg_name.id) statements.append( StatementAssignmentVariable( variable = arg_var, source = ExpressionSpecialUnpack( value = ExpressionTempVariableRef( variable = iter_var, source_ref = source_ref ), count = element_index + 1, expected = len(arg_names), source_ref = source_ref ), source_ref = source_ref ) ) accesses.append( ExpressionTempVariableRef( variable = arg_var, source_ref = source_ref ) ) elif getKind(arg_name) == "Tuple": accesses.extend( unpackFrom( source = ExpressionSpecialUnpack( value = ExpressionTempVariableRef( variable = iter_var, source_ref = source_ref ), count = element_index + 1, expected = len(arg_names), source_ref = source_ref ), arg_names = arg_name.elts ) ) sub_special_index += 1 else: assert False, arg_name statements.append( StatementSpecialUnpackCheck( iterator = ExpressionTempVariableRef( variable = iter_var, source_ref = source_ref ), count = len(arg_names), source_ref = source_ref ) ) return accesses for arg_name in parameters.getParameterNames(): if arg_name.startswith('.'): source = ExpressionVariableNameRef( variable_name = arg_name, source_ref = source_ref ) values.extend( unpackFrom(source, special_args[arg_name]) ) else: values.append( ExpressionVariableNameRef( variable_name = arg_name, source_ref = source_ref ) ) inner_arg_names.append(arg_name) inner_parameters = ParameterSpec( ps_name = inner_name, ps_normal_args = inner_arg_names, ps_kw_only_args = (), ps_list_star_arg = None, ps_dict_star_arg = None, ps_default_count = None ) function_body = ExpressionFunctionBody( provider = outer_body, name = inner_name, flags = flags, doc = function_doc, parameters = inner_parameters, source_ref = source_ref ) statements.append( StatementReturn( ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = function_body, source_ref = source_ref ), code_object = code_object, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = values, source_ref = source_ref ), source_ref = source_ref ) ) outer_body.setBody( makeStatementsSequenceFromStatement( statement = makeTryFinallyStatement( provider = outer_body, tried = statements, final = [ StatementReleaseVariable( variable = variable, source_ref = source_ref ) for variable in sorted( outer_body.getTempVariables(), key = lambda variable: variable.getName() ) ], source_ref = source_ref, public_exc = False ) ) ) else: function_body = outer_body return outer_body, function_body, code_object def addFunctionVariableReleases(function): assert function.isExpressionFunctionBodyBase() releases = [] # We attach everything to the function definition source location. source_ref = function.getSourceReference() for variable in function.getLocalVariables(): # Shared variables are freed by function object attachment. if variable.getOwner() is not function: continue releases.append( StatementReleaseVariable( variable = variable, source_ref = source_ref ) ) if releases: body = function.getBody() if body.isStatementsFrame(): body = makeStatementsSequenceFromStatement( statement = body ) body = makeTryFinallyStatement( provider = function, tried = body, final = releases, source_ref = source_ref ) function.setBody( makeStatementsSequenceFromStatement( statement = body ) ) # assert body.isStatementAborting(), body.asXmlText() Nuitka-0.5.28.2/nuitka/tree/TreeHelpers.py0000644000372000001440000006020213207537242020516 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Helper functions for parsing the AST nodes and building the Nuitka node tree. """ import ast from logging import warning from nuitka import Constants, Options, Tracing from nuitka.nodes.CallNodes import makeExpressionCall from nuitka.nodes.CodeObjectSpecs import CodeObjectSpec from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ( ExpressionMakeList, ExpressionMakeSetLiteral, ExpressionMakeTuple ) from nuitka.nodes.DictionaryNodes import ( ExpressionKeyValuePair, ExpressionMakeDict ) from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.FrameNodes import ( StatementsFrameAsyncgen, StatementsFrameCoroutine, StatementsFrameFunction, StatementsFrameGenerator, StatementsFrameModule ) from nuitka.nodes.ImportNodes import ExpressionBuiltinImport from nuitka.nodes.NodeBases import NodeBase from nuitka.nodes.NodeMakingHelpers import mergeStatements from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.PythonVersions import ( needsSetLiteralReverseInsertion, python_version ) def dump(node): Tracing.printLine(ast.dump(node)) def getKind(node): return node.__class__.__name__.split('.')[-1] def extractDocFromBody(node): body = node.body doc = None # Work around ast.get_docstring breakage. if node.body and \ getKind(node.body[0]) == "Expr" and \ getKind(node.body[0].value) == "Str": if "no_asserts" not in Options.getPythonFlags(): doc = body[0].value.s body = body[1:] return body, doc def parseSourceCodeToAst(source_code, filename, line_offset): # Workaround: ast.parse cannot cope with some situations where a file is not # terminated by a new line. if not source_code.endswith('\n'): source_code = source_code + '\n' body = ast.parse(source_code, filename) assert getKind(body) == "Module" if line_offset > 0: ast.increment_lineno(body, line_offset) return body def detectFunctionBodyKind(nodes, start_value = None): # This is a complex mess, following the scope means a lot of checks need # to be done. pylint: disable=too-many-branches,too-many-statements indications = set() if start_value is not None: indications.add(start_value) flags = set() def _check(node): node_class = node.__class__ if node_class is ast.Yield: indications.add("Generator") elif python_version >= 330 and node_class is ast.YieldFrom: # @UndefinedVariable indications.add("Generator") elif python_version >= 350 and node_class in (ast.Await, ast.AsyncWith): # @UndefinedVariable indications.add("Coroutine") # Recurse to children, but do not cross scope boundary doing so. if node_class is ast.ClassDef: for name, field in ast.iter_fields(node): if name in ("name", "body"): pass elif name in ("bases", "decorator_list", "keywords"): for child in field: _check(child) elif name == "starargs": if field is not None: _check(field) elif name == "kwargs": if field is not None: _check(field) else: assert False, (name, field, ast.dump(node)) elif node_class in (ast.FunctionDef, ast.Lambda) or \ (python_version >= 350 and node_class is ast.AsyncFunctionDef): # @UndefinedVariable for name, field in ast.iter_fields(node): if name in ("name", "body"): pass elif name in ("bases", "decorator_list"): for child in field: _check(child) elif name == "args": for child in field.defaults: _check(child) if python_version >= 300: for child in node.args.kw_defaults: if child is not None: _check(child) for child in node.args.args: if child.annotation is not None: _check(child.annotation) elif name == "returns": if field is not None: _check(field) else: assert False, (name, field, ast.dump(node)) elif node_class is ast.GeneratorExp: for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "elt"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif node_class is ast.ListComp and python_version >= 300: for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators"): pass elif name == "generators": _check(field[0].iter) elif name in("body", "elt"): _check(field) else: assert False, (name, field, ast.dump(node)) elif python_version >= 270 and node_class is ast.SetComp: # @UndefinedVariable for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "elt"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif python_version >= 270 and node_class is ast.DictComp: # @UndefinedVariable for name, field in ast.iter_fields(node): if name in ("name", "body", "comparators", "key", "value"): pass elif name == "generators": _check(field[0].iter) else: assert False, (name, field, ast.dump(node)) elif node_class is ast.Name: if python_version >= 300 and node.id == "super": flags.add("has_super") elif python_version < 300 and node_class is ast.Exec: flags.add("has_exec") if node.globals is None: flags.add("has_unqualified_exec") for child in ast.iter_child_nodes(node): _check(child) elif python_version < 300 and node_class is ast.ImportFrom: for import_desc in node.names: if import_desc.name[0] == '*': flags.add("has_exec") for child in ast.iter_child_nodes(node): _check(child) else: for child in ast.iter_child_nodes(node): _check(child) for node in nodes: _check(node) if indications: if "Coroutine" in indications and "Generator" in indications: function_kind = "Asyncgen" else: # If we found something, make sure we agree on all clues. assert len(indications) == 1, indications function_kind = indications.pop() else: function_kind = "Function" return function_kind, flags build_nodes_args3 = None build_nodes_args2 = None build_nodes_args1 = None def setBuildingDispatchers(path_args3, path_args2, path_args1): # Using global here, as this is really a singleton, in the form of a module, # and this is to break the cyclic dependency it has, pylint: disable=global-statement global build_nodes_args3, build_nodes_args2, build_nodes_args1 build_nodes_args3 = path_args3 build_nodes_args2 = path_args2 build_nodes_args1 = path_args1 def buildNode(provider, node, source_ref, allow_none = False): if node is None and allow_none: return None try: kind = getKind(node) if hasattr(node, "lineno"): source_ref = source_ref.atLineNumber(node.lineno) else: source_ref = source_ref if kind in build_nodes_args3: result = build_nodes_args3[kind]( provider = provider, node = node, source_ref = source_ref ) elif kind in build_nodes_args2: result = build_nodes_args2[kind]( node = node, source_ref = source_ref ) elif kind in build_nodes_args1: result = build_nodes_args1[kind]( source_ref = source_ref ) elif kind == "Pass": result = None else: assert False, ast.dump(node) if result is None and allow_none: return None assert isinstance(result, NodeBase), result return result except SyntaxError: raise except RuntimeError: # Very likely the stack overflow, which we will turn into too complex # code exception, don't warn about it with a code dump then. raise except: warning("Problem at '%s' with %s." % (source_ref, ast.dump(node))) raise def buildNodeList(provider, nodes, source_ref, allow_none = False): if nodes is not None: result = [] for node in nodes: if hasattr(node, "lineno"): node_source_ref = source_ref.atLineNumber(node.lineno) else: node_source_ref = source_ref entry = buildNode(provider, node, node_source_ref, allow_none) if entry is not None: result.append(entry) return result else: return [] def makeModuleFrame(module, statements, source_ref): assert module.isCompiledPythonModule() if Options.isFullCompat(): code_name = "" else: if module.isMainModule(): code_name = "" else: code_name = "" % module.getFullName() return StatementsFrameModule( statements = statements, code_object = CodeObjectSpec( co_name = code_name, co_kind = "Module", co_varnames = (), co_argcount = 0, co_kwonlyargcount = 0, co_has_starlist = False, co_has_stardict = False, co_filename = module.getRunTimeFilename(), co_lineno = source_ref.getLineNumber(), future_spec = module.getFutureSpec() ), source_ref = source_ref ) def buildStatementsNode(provider, nodes, source_ref): # We are not creating empty statement sequences. if nodes is None: return None # Build as list of statements, throw away empty ones, and remove useless # nesting. statements = buildNodeList(provider, nodes, source_ref, allow_none = True) statements = mergeStatements(statements) # We are not creating empty statement sequences. Might be empty, because # e.g. a global node generates not really a statement, or pass statements. if not statements: return None else: return StatementsSequence( statements = statements, source_ref = source_ref ) def buildFrameNode(provider, nodes, code_object, source_ref): # We are not creating empty statement sequences. if nodes is None: return None # Build as list of statements, throw away empty ones, and remove useless # nesting. statements = buildNodeList(provider, nodes, source_ref, allow_none = True) statements = mergeStatements(statements) # We are not creating empty statement sequences. Might be empty, because # e.g. a global node generates not really a statement, or pass statements. if not statements: return None if provider.isExpressionFunctionBody() or \ provider.isExpressionClassBody(): result = StatementsFrameFunction( statements = statements, code_object = code_object, source_ref = source_ref ) elif provider.isExpressionGeneratorObjectBody(): result = StatementsFrameGenerator( statements = statements, code_object = code_object, source_ref = source_ref ) elif provider.isExpressionCoroutineObjectBody(): result = StatementsFrameCoroutine( statements = statements, code_object = code_object, source_ref = source_ref ) elif provider.isExpressionAsyncgenObjectBody(): result = StatementsFrameAsyncgen( statements = statements, code_object = code_object, source_ref = source_ref ) else: assert False, provider return result def makeStatementsSequenceOrStatement(statements, source_ref): """ Make a statement sequence, but only if more than one statement Useful for when we can unroll constructs already here, but are not sure if we actually did that. This avoids the branch or the pollution of doing it always. """ if len(statements) > 1: return StatementsSequence( statements = mergeStatements(statements), source_ref = source_ref ) else: return statements[0] def makeStatementsSequence(statements, allow_none, source_ref): if allow_none: statements = tuple( statement for statement in statements if statement is not None ) if statements: return StatementsSequence( statements = mergeStatements(statements), source_ref = source_ref ) else: return None def makeStatementsSequenceFromStatement(statement): return StatementsSequence( statements = mergeStatements( (statement,) ), source_ref = statement.getSourceReference() ) def makeStatementsSequenceFromStatements(*statements): assert statements assert None not in statements statements = mergeStatements(statements, allow_none = False) return StatementsSequence( statements = statements, source_ref = statements[0].getSourceReference() ) def makeSequenceCreationOrConstant(sequence_kind, elements, source_ref): # Sequence creation. Tries to avoid creations with only constant # elements. Would be caught by optimization, but would be useless churn. For # mutable constants we cannot do it though. # Due to the many sequence types, there is a lot of cases here # pylint: disable=too-many-branches for element in elements: if not element.isExpressionConstantRef(): constant = False break else: constant = True sequence_kind = sequence_kind.lower() # Note: This would happen in optimization instead, but lets just do it # immediately to save some time. if constant: if sequence_kind == "tuple": const_type = tuple elif sequence_kind == "list": const_type = list elif sequence_kind == "set": const_type = set if needsSetLiteralReverseInsertion(): elements = tuple(reversed(elements)) else: assert False, sequence_kind result = makeConstantRefNode( constant = const_type( element.getConstant() for element in elements ), source_ref = source_ref, user_provided = True ) else: if sequence_kind == "tuple": result = ExpressionMakeTuple( elements = elements, source_ref = source_ref ) elif sequence_kind == "list": result = ExpressionMakeList( elements = elements, source_ref = source_ref ) elif sequence_kind == "set": result = ExpressionMakeSetLiteral( elements = elements, source_ref = source_ref ) else: assert False, sequence_kind if elements: result.setCompatibleSourceReference( source_ref = elements[-1].getCompatibleSourceReference() ) return result def makeDictCreationOrConstant(keys, values, source_ref): # Create dictionary node. Tries to avoid it for constant values that are not # mutable. assert len(keys) == len(values) for key, value in zip(keys, values): if not key.isExpressionConstantRef() or not key.isKnownToBeHashable(): constant = False break if not value.isExpressionConstantRef(): constant = False break else: constant = True # Note: This would happen in optimization instead, but lets just do it # immediately to save some time. if constant: # Unless told otherwise, create the dictionary in its full size, so # that no growing occurs and the constant becomes as similar as possible # before being marshaled. result = makeConstantRefNode( constant = Constants.createConstantDict( keys = [ key.getConstant() for key in keys ], values = [ value.getConstant() for value in values ] ), user_provided = True, source_ref = source_ref ) else: result = ExpressionMakeDict( pairs = [ ExpressionKeyValuePair( key = key, value = value, source_ref = key.getSourceReference() ) for key, value in zip(keys, values) ], source_ref = source_ref ) if values: result.setCompatibleSourceReference( source_ref = values[-1].getCompatibleSourceReference() ) return result def makeDictCreationOrConstant2(keys, values, source_ref): # Create dictionary node. Tries to avoid it for constant values that are not # mutable. Keys are strings. assert len(keys) == len(values) for value in values: if not value.isExpressionConstantRef(): constant = False break else: constant = True # Note: This would happen in optimization instead, but lets just do it # immediately to save some time. if constant: # Unless told otherwise, create the dictionary in its full size, so # that no growing occurs and the constant becomes as similar as possible # before being marshaled. result = makeConstantRefNode( constant = Constants.createConstantDict( keys = keys, values = [ value.getConstant() for value in values ] ), user_provided = True, source_ref = source_ref ) else: result = ExpressionMakeDict( pairs = [ ExpressionKeyValuePair( key = makeConstantRefNode( constant = key, source_ref = value.getSourceReference(), user_provided = True ), value = value, source_ref = value.getSourceReference() ) for key, value in zip(keys, values) ], source_ref = source_ref ) if values: result.setCompatibleSourceReference( source_ref = values[-1].getCompatibleSourceReference() ) return result def getStatementsAppended(statement_sequence, statements): return makeStatementsSequence( statements = (statement_sequence, statements), allow_none = False, source_ref = statement_sequence.getSourceReference() ) def getStatementsPrepended(statement_sequence, statements): return makeStatementsSequence( statements = (statements, statement_sequence), allow_none = False, source_ref = statement_sequence.getSourceReference() ) def makeReraiseExceptionStatement(source_ref): return StatementsSequence( statements = ( StatementRaiseException( exception_type = None, exception_value = None, exception_trace = None, exception_cause = None, source_ref = source_ref ), ), source_ref = source_ref ) def makeAbsoluteImportNode(module_name, source_ref): return ExpressionBuiltinImport( name = makeConstantRefNode(module_name, source_ref, True), globals_arg = None, locals_arg = None, fromlist = None, level = makeConstantRefNode(0, source_ref, True), source_ref = source_ref ) def mangleName(variable_name, owner): if not variable_name.startswith("__") or variable_name.endswith("__"): return variable_name else: # The mangling of function variable names depends on being inside a # class. class_container = owner.getContainingClassDictCreation() if class_container is None: return variable_name else: return "_%s%s" % ( class_container.getName().lstrip('_'), variable_name ) def makeConditionalStatement(condition, yes_branch, no_branch, source_ref): """ Create conditional statement, with yes_branch not being empty. May have to invert condition to achieve that. """ if yes_branch is None: condition = ExpressionOperationNOT( operand = condition, source_ref = condition.getSourceReference() ) yes_branch, no_branch = no_branch, yes_branch if not yes_branch.isStatementsSequence(): yes_branch = makeStatementsSequenceFromStatement(yes_branch) if no_branch is not None and not no_branch.isStatementsSequence(): no_branch = makeStatementsSequenceFromStatement(no_branch) return StatementConditional( condition = condition, yes_branch = yes_branch, no_branch = no_branch, source_ref = source_ref ) def makeCallNode(called, *args, **kwargs): source_ref = args[-1] if len(args) > 1: args = makeSequenceCreationOrConstant( sequence_kind = "tuple", elements = args[:-1], source_ref = source_ref ) else: args = None if kwargs: kwargs = makeDictCreationOrConstant2( keys = tuple(kwargs.keys()), values = tuple(kwargs.values()), source_ref = source_ref ) else: kwargs = None return makeExpressionCall( called = called, args = args, kw = kwargs, source_ref = source_ref ) build_contexts = [None] def pushBuildContext(value): build_contexts.append(value) def popBuildContext(): del build_contexts[-1] def getBuildContext(): return build_contexts[-1] Nuitka-0.5.28.2/nuitka/tree/ReformulationForLoopStatements.py0000644000372000001440000002056113207537242024477 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of for loop statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ( ExpressionAsyncIter, ExpressionAsyncNext, ExpressionBuiltinIter1 ) from nuitka.nodes.BuiltinNextNodes import ExpressionBuiltinNext1 from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from .ReformulationAssignmentStatements import buildAssignmentStatements from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildStatementsNode, makeStatementsSequence, makeStatementsSequenceFromStatements, popBuildContext, pushBuildContext ) def _buildForLoopNode(provider, node, sync, source_ref): # The for loop is re-formulated according to developer manual. An iterator # is created, and looped until it gives StopIteration. The else block is # taken if a for loop exits normally, i.e. because of iterator # exhaustion. We do this by introducing an indicator variable. # We handle async and sync both here, leading to cases, pylint: disable=too-many-locals source = buildNode(provider, node.iter, source_ref) # Temporary variables, we need one for the iterator, and one for the current # value. temp_scope = provider.allocateTempScope("for_loop") tmp_iter_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "for_iterator" ) tmp_value_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "iter_value" ) else_block = buildStatementsNode( provider = provider, nodes = node.orelse if node.orelse else None, source_ref = source_ref ) if else_block is not None: tmp_break_indicator = provider.allocateTempVariable( temp_scope = temp_scope, name = "break_indicator" ) statements = [ StatementAssignmentVariable( variable = tmp_break_indicator, source = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ) ] else: statements = [] statements.append( StatementLoopBreak( source_ref = source_ref ) ) handler_body = makeStatementsSequence( statements = statements, allow_none = False, source_ref = source_ref ) if sync: next_node = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ), source_ref = source_ref ) else: next_node = ExpressionAsyncNext( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = source_ref ), source_ref = source_ref ) statements = ( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_value_variable, source = next_node, source_ref = source_ref ), exception_name = "StopIteration" if sync else "StopAsyncIteration", handler_body = handler_body, source_ref = source_ref ), buildAssignmentStatements( provider = provider, node = node.target, source = ExpressionTempVariableRef( variable = tmp_value_variable, source_ref = source_ref ), source_ref = source_ref ) ) pushBuildContext("loop_body") statements += ( buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ), ) popBuildContext() loop_body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) cleanup_statements = [ StatementReleaseVariable( variable = tmp_value_variable, source_ref = source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = source_ref ) ] if else_block is not None: statements = [ StatementAssignmentVariable( variable = tmp_break_indicator, source = makeConstantRefNode( constant = False, source_ref = source_ref ), source_ref = source_ref ) ] else: statements = [] if sync: iter_source = ExpressionBuiltinIter1( value = source, source_ref = source.getSourceReference() ) else: iter_source = ExpressionAsyncIter( value = source, source_ref = source.getSourceReference() ) statements += [ # First create the iterator and store it. StatementAssignmentVariable( variable = tmp_iter_variable, source = iter_source, source_ref = source_ref ), makeTryFinallyStatement( provider = provider, tried = StatementLoop( body = loop_body, source_ref = source_ref ), final = StatementsSequence( statements = cleanup_statements, source_ref = source_ref ), source_ref = source_ref ) ] if else_block is not None: statements += [ StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_break_indicator, source_ref = source_ref ), right = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = else_block, no_branch = None, source_ref = source_ref ) ] statements = ( makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_break_indicator, source_ref = source_ref ), source_ref = source_ref ), ) return makeStatementsSequenceFromStatements( *statements ) def buildForLoopNode(provider, node, source_ref): return _buildForLoopNode(provider, node, True, source_ref) def buildAsyncForLoopNode(provider, node, source_ref): return _buildForLoopNode(provider, node, False, source_ref) Nuitka-0.5.28.2/nuitka/tree/ReformulationDictionaryCreation.py0000644000372000001440000002736313207537242024650 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of dictionary creations. Dictionary creations might be directly translated to constants, or they might become nodes that build dictionaries. For Python3.5, unpacking can happen while creating dictionaries, these are being re-formulated to an internal function. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.BuiltinIteratorNodes import ExpressionBuiltinIter1 from nuitka.nodes.BuiltinNextNodes import ExpressionBuiltinNext1 from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.DictionaryNodes import ( ExpressionKeyValuePair, ExpressionMakeDict, StatementDictOperationUpdate ) from nuitka.nodes.ExceptionNodes import ( ExpressionBuiltinMakeException, StatementRaiseException ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.OperatorNodes import makeBinaryOperationNode from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.TypeNodes import ExpressionBuiltinType1 from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from .InternalModule import ( getInternalModule, internal_source_ref, once_decorator ) from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildNodeList, makeDictCreationOrConstant, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) def buildDictionaryNode(provider, node, source_ref): if python_version >= 350: for key in node.keys: if key is None: return buildDictionaryUnpacking( provider = provider, node = node, source_ref = source_ref ) return makeDictCreationOrConstant( keys = buildNodeList(provider, node.keys, source_ref), values = buildNodeList(provider, node.values, source_ref), source_ref = source_ref ) @once_decorator def getDictUnpackingHelper(): helper_name = "_unpack_dict" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = (), ps_list_star_arg = "args", ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "dict") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "keys") loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_item_variable, source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), makeTryExceptSingleHandlerNode( tried = StatementDictOperationUpdate( dict_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "AttributeError", handler_body = StatementRaiseException( exception_type = ExpressionBuiltinMakeException( exception_name = "TypeError", args = ( makeBinaryOperationNode( operator = "Mod", left = makeConstantRefNode( constant = """\ '%s' object is not a mapping""", source_ref = internal_source_ref, user_provided = True ), right = ExpressionMakeTuple( elements = ( ExpressionAttributeLookup( source = ExpressionBuiltinType1( value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), attribute_name = "__name__", source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), ), source_ref = internal_source_ref ), exception_value = None, exception_trace = None, exception_cause = None, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) args_variable = result.getVariableForAssignment( variable_name = "args" ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_result_variable, source = makeConstantRefNode( constant = {}, source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementLoop( body = loop_body, source_ref = internal_source_ref ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result def buildDictionaryUnpackingArgs(provider, keys, values, source_ref): result = [] for key, value in zip(keys, values): # TODO: We could be a lot cleverer about the dictionaries for non-starred # arguments, but lets get this to work first. if key is None: result.append( buildNode(provider, value, source_ref), ) elif type(key) is str: result.append( ExpressionMakeDict( pairs = ( ExpressionKeyValuePair( key = makeConstantRefNode( constant = key, source_ref = source_ref ), value = buildNode(provider, value, source_ref), source_ref = source_ref ), ), source_ref = source_ref ) ) else: result.append( ExpressionMakeDict( pairs = ( ExpressionKeyValuePair( key = buildNode(provider, key, source_ref), value = buildNode(provider, value, source_ref), source_ref = source_ref ), ), source_ref = source_ref ) ) return result def buildDictionaryUnpacking(provider, node, source_ref): helper_args = buildDictionaryUnpackingArgs(provider, node.keys, node.values, source_ref) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getDictUnpackingHelper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionMakeTuple( helper_args, source_ref ), ), source_ref = source_ref, ) result.setCompatibleSourceReference(helper_args[-1].getCompatibleSourceReference()) return result Nuitka-0.5.28.2/nuitka/tree/__init__.py0000644000372000001440000000150113112214770020021 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/tree/InternalModule.py0000644000372000001440000000336413112214770021215 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Internal module This is a container for helper functions that are shared across modules. It may not exist, and is treated specially in code generation. This avoids to own these functions to a random module. """ from nuitka.nodes.ModuleNodes import PythonInternalModule from nuitka.SourceCodeReferences import fromFilename internal_source_ref = fromFilename("internal").atInternal() def once_decorator(func): """ Cache result of a function call without arguments. Used for all internal function accesses to become a singleton. Note: This doesn't much specific anymore, but we are not having this often enough to warrent re-use or generalization. """ func.cached_value = None def replacement(): if func.cached_value is None: func.cached_value = func() return func.cached_value return replacement @once_decorator def getInternalModule(): """ Get the singleton internal module. """ return PythonInternalModule() Nuitka-0.5.28.2/nuitka/tree/ReformulationSubscriptExpressions.py0000644000372000001440000001122113207537242025261 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of subscript into slicing. For Python2, there is a difference between x[a], x[a:b], x[a:b:c] whereas Python3 treats the later by making a slice object, Python2 tries to have special slice access, if available, or building a slice object only at the end. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.ConstantRefNodes import ExpressionConstantEllipsisRef from nuitka.nodes.SliceNodes import ( ExpressionBuiltinSlice, ExpressionSliceLookup ) from nuitka.nodes.SubscriptNodes import ExpressionSubscriptLookup from nuitka.PythonVersions import python_version from .ReformulationAssignmentStatements import buildExtSliceNode from .TreeHelpers import buildNode, getKind def buildSubscriptNode(provider, node, source_ref): # Subscript expression nodes, various types are dispatched here. assert getKind(node.ctx) == "Load", source_ref # The subscript "[]" operator is one of many different things. This is # expressed by this kind, there are "slice" lookups (two values, even if one # is using default), and then "index" lookups. The form with three argument # is really an "index" lookup, with a slice object. And the "..." lookup is # also an index loop-up, with it as the argument. So this splits things into # two different operations, "subscript" with a single "subscript" object. Or # a slice lookup with a lower and higher boundary. These things should # behave similar, but they are different slots. kind = getKind(node.slice) if kind == "Index": return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = buildNode(provider, node.slice.value, source_ref), source_ref = source_ref ) elif kind == "Slice": lower = buildNode( provider = provider, node = node.slice.lower, source_ref = source_ref, allow_none = True ) upper = buildNode( provider = provider, node = node.slice.upper, source_ref = source_ref, allow_none = True ) step = buildNode( provider = provider, node = node.slice.step, source_ref = source_ref, allow_none = True ) # For Python3 there is no slicing operation, this is always done # with subscript using a slice object. For Python2, it is only done # if no "step" is provided. use_sliceobj = step is not None or python_version >= 300 if use_sliceobj: return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = ExpressionBuiltinSlice( start = lower, stop = upper, step = step, source_ref = source_ref ), source_ref = source_ref ) else: return ExpressionSliceLookup( expression = buildNode(provider, node.value, source_ref), lower = lower, upper = upper, source_ref = source_ref ) elif kind == "ExtSlice": return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = buildExtSliceNode(provider, node, source_ref), source_ref = source_ref ) elif kind == "Ellipsis": return ExpressionSubscriptLookup( subscribed = buildNode(provider, node.value, source_ref), subscript = ExpressionConstantEllipsisRef( source_ref = source_ref ), source_ref = source_ref ) else: assert False, kind Nuitka-0.5.28.2/nuitka/tree/ReformulationPrintStatements.py0000644000372000001440000001216013207537242024207 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of print statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIs from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantNoneRef from nuitka.nodes.ImportNodes import ExpressionImportModuleNameHard from nuitka.nodes.PrintNodes import StatementPrintNewline, StatementPrintValue from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildNodeList, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) def buildPrintNode(provider, node, source_ref): # "print" statements, should only occur with Python2. if node.dest is not None: temp_scope = provider.allocateTempScope("print") tmp_target_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "target" ) target_default_statement = StatementAssignmentVariable( variable = tmp_target_variable, source = ExpressionImportModuleNameHard( module_name = "sys", import_name = "stdout", source_ref = source_ref ), source_ref = source_ref ) statements = [ StatementAssignmentVariable( variable = tmp_target_variable, source = buildNode( provider = provider, node = node.dest, source_ref = source_ref ), source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = target_default_statement ), no_branch = None, source_ref = source_ref ) ] values = buildNodeList( provider = provider, nodes = node.values, source_ref = source_ref ) if node.dest is not None: print_statements = [ StatementPrintValue( dest = ExpressionTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), value = value, source_ref = source_ref ) for value in values ] if node.nl: print_statements.append( StatementPrintNewline( dest = ExpressionTempVariableRef( variable = tmp_target_variable, source_ref = source_ref ), source_ref = source_ref ) ) statements.append( makeTryFinallyStatement( provider = provider, tried = print_statements, final = StatementReleaseVariable( variable = tmp_target_variable, source_ref = source_ref ), source_ref = source_ref ) ) else: statements = [ StatementPrintValue( dest = None, value = value, source_ref = source_ref ) for value in values ] if node.nl: statements.append( StatementPrintNewline( dest = None, source_ref = source_ref ) ) return makeStatementsSequenceFromStatements( *statements ) Nuitka-0.5.28.2/nuitka/tree/VariableClosure.py0000644000372000001440000004167313207537242021371 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Variable closure taking. This is the completion of variable object completion. The variables were not immediately resolved to be bound to actual scopes, but are only now. Only after this is executed, variable reference nodes can be considered complete. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementDelVariable ) from nuitka.nodes.FunctionNodes import MaybeLocalVariableUsage from nuitka.nodes.NodeMakingHelpers import makeConstantReplacementNode from nuitka.nodes.VariableRefNodes import ( ExpressionLocalsVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import ( getErrorMessageExecWithNestedFunction, python_version ) from .Operations import VisitorNoopMixin, visitTree from .ReformulationFunctionStatements import addFunctionVariableReleases from .SyntaxErrors import raiseSyntaxError # Note: We do the variable scope assignment, as an extra step from tree # building, because tree building creates the tree without any consideration of # evaluation order. And the ordered way these visitors are entered, will ensure # this order. # The main complexity is that there are two ways of visiting. One where variable # lookups are to be done immediately, and one where it is delayed. This is # basically class vs. function scope handling. class VariableClosureLookupVisitorPhase1(VisitorNoopMixin): """ Variable closure phase 1: Find assignments and early closure references. In class context, a reference to a variable must be obeyed immediately, so that "variable = variable" takes first "variable" as a closure and then adds a new local "variable" to override it from there on. For the not early closure case of a function, this will not be done and only assignments shall add local variables, and references will be ignored until phase 2. """ @staticmethod def _handleNonLocal(node): # Take closure variables for non-local declarations. for non_local_names, source_ref in node.consumeNonlocalDeclarations(): for non_local_name in non_local_names: variable = node.takeVariableForClosure( variable_name = non_local_name ) node.registerProvidedVariable(variable) if variable.isModuleVariable(): raiseSyntaxError( "no binding for nonlocal '%s' found" % ( non_local_name ), source_ref ) variable.addVariableUser(node) @staticmethod def _handleQualnameSetup(node): if node.qualname_setup is not None: if node.isExpressionClassBody(): class_variable_name, qualname_assign = node.qualname_setup class_variable = node.getParentVariableProvider().getVariableForAssignment( class_variable_name ) if class_variable.isModuleVariable(): qualname_node = qualname_assign.getAssignSource() qualname_node.replaceWith( makeConstantReplacementNode( constant = class_variable.getName(), node = qualname_node ) ) node.qualname_provider = node.getParentModule() else: function_variable = node.getParentVariableProvider().getVariableForAssignment( node.qualname_setup ) if function_variable.isModuleVariable(): node.qualname_provider = node.getParentModule() # TODO: Actually for nested global classes, this approach # may not work, as their "qualname" will be wrong. In that # case a dedicated node for "qualname" references might be # needed. node.qualname_setup = None def onLeaveNode(self, node): if node.isStatementAssignmentVariableName(): provider = node.getParentVariableProvider() variable = provider.getVariableForAssignment( variable_name = node.getVariableName() ) node.replaceWith( StatementAssignmentVariable( variable = variable, source = node.subnode_source, source_ref = node.source_ref ) ) variable.addVariableUser(provider) elif node.isStatementDelVariableName(): provider = node.getParentVariableProvider() variable = provider.getVariableForAssignment( variable_name = node.getVariableName() ) node.replaceWith( StatementDelVariable( variable = variable, tolerant = node.tolerant, source_ref = node.source_ref ) ) variable.addVariableUser(provider) def onEnterNode(self, node): # Mighty complex code with lots of branches and statements, but it # couldn't be less without making it more difficult. # pylint: disable=too-many-branches,too-many-statements if node.isExpressionVariableNameRef(): provider = node.getParentVariableProvider() if provider.isEarlyClosure(): variable = provider.getVariableForReference( variable_name = node.getVariableName() ) # Python3.4 version respects closure variables taken can be # overridden by writes to locals. It should be done for # globals too, on all versions, but for Python2 the locals # dictionary is avoided unless "exec" appears, so it's not # done. owner = variable.getOwner() user = provider while user is not owner: if user.isExpressionFunctionBody() or \ user.isExpressionClassBody(): break user = user.getParentVariableProvider() if owner is not user: if python_version >= 340 or \ (python_version >= 300 and \ variable.isModuleVariable()): node.replaceWith( ExpressionLocalsVariableRef( variable_name = node.getVariableName(), fallback_variable = variable, source_ref = node.getSourceReference() ) ) else: node.replaceWith( ExpressionVariableRef( variable = variable, source_ref = node.source_ref ) ) else: node.replaceWith( ExpressionVariableRef( variable = variable, source_ref = node.source_ref ) ) variable.addVariableUser(provider) elif node.isExpressionTempVariableRef(): if node.getVariable().getOwner() != node.getParentVariableProvider(): node.getParentVariableProvider().addClosureVariable( node.getVariable() ) elif node.isExpressionGeneratorObjectBody(): self._handleNonLocal(node) # Only Python3.4 or later allows for generators to have qualname. if python_version >= 340: self._handleQualnameSetup(node) elif node.isExpressionCoroutineObjectBody(): self._handleNonLocal(node) self._handleQualnameSetup(node) elif node.isExpressionAsyncgenObjectBody(): self._handleNonLocal(node) self._handleQualnameSetup(node) elif node.isExpressionClassBody(): self._handleNonLocal(node) # Python3.4 allows for class declarations to be made global, even # after they were declared, so we need to fix this up. if python_version >= 340: self._handleQualnameSetup(node) elif node.isExpressionFunctionBody(): self._handleNonLocal(node) # Python 3.4 allows for class declarations to be made global, even # after they were declared, so we need to fix this up. if python_version >= 340: self._handleQualnameSetup(node) # Attribute access of names of class functions should be mangled, if # they start with "__", but do not end in "__" as well. elif node.isExpressionAttributeLookup() or \ node.isStatementAssignmentAttribute() or \ node.isStatementDelAttribute(): attribute_name = node.getAttributeName() if attribute_name.startswith("__") and \ not attribute_name.endswith("__"): seen_function = False current = node while True: current = current.getParentVariableProvider() if current.isCompiledPythonModule(): break if current.isExpressionClassBody(): if seen_function: node.setAttributeName( "_%s%s" % ( current.getName().lstrip('_'), attribute_name ) ) break else: seen_function = True # Check if continue and break are properly in loops. If not, raise a # syntax error. elif node.isStatementLoopBreak() or node.isStatementLoopContinue(): current = node while True: current = current.getParent() if current.isStatementLoop(): break if current.isParentVariableProvider(): if node.isStatementLoopContinue(): message = "'continue' not properly in loop" else: message = "'break' outside loop" raiseSyntaxError( message, node.getSourceReference(), ) class VariableClosureLookupVisitorPhase2(VisitorNoopMixin): """ Variable closure phase 2: Find assignments and references. In class context, a reference to a variable must be obeyed immediately, so that "variable = variable" takes first "variable" as a closure and then adds a new local "variable" to override it from there on. So, assignments for early closure, accesses will already have a variable set now, the others, only in this phase. """ @staticmethod def _attachVariable(node, provider): # print "Late reference", node.getVariableName(), "for", provider, "caused at", node, "of", node.getParent() variable_name = node.getVariableName() was_taken = provider.hasTakenVariable(variable_name) variable = provider.getVariableForReference( variable_name = variable_name ) # Need to catch functions with "exec" and closure variables not allowed. if python_version < 300 and \ not was_taken and \ provider.isExpressionFunctionBodyBase() and \ variable.getOwner() is not provider: parent_provider = provider.getParentVariableProvider() while parent_provider.isExpressionClassBody(): parent_provider = parent_provider.getParentVariableProvider() if parent_provider.isExpressionFunctionBody() and \ parent_provider.isUnqualifiedExec(): raiseSyntaxError( getErrorMessageExecWithNestedFunction() % parent_provider.getName(), node.getSourceReference(), display_line = False # Wrong line anyway ) return variable def onEnterNode(self, node): if node.isExpressionVariableNameRef(): provider = node.getParentVariableProvider() try: variable = self._attachVariable(node, provider) except MaybeLocalVariableUsage: variable_name = node.getVariableName() node.replaceWith( ExpressionLocalsVariableRef( variable_name = variable_name, fallback_variable = node.getParentModule().getVariableForReference(variable_name), source_ref = node.getSourceReference() ) ) else: node.replaceWith( ExpressionVariableRef( variable = variable, source_ref = node.source_ref ) ) variable.addVariableUser(provider) elif node.isExpressionVariableRef(): provider = node.getParentVariableProvider() variable = node.getVariable() if variable is None: try: variable = self._attachVariable(node, provider) except MaybeLocalVariableUsage: variable_name = node.getVariableName() node.replaceWith( ExpressionLocalsVariableRef( variable_name = variable_name, fallback_variable = node.getParentModule().getVariableForReference(variable_name), source_ref = node.getSourceReference() ) ) else: node.setVariable( variable ) variable.addVariableUser(provider) else: variable.addVariableUser(provider) class VariableClosureLookupVisitorPhase3(VisitorNoopMixin): """ Variable closure phase 3: Find errors and complete frame variables. In this phase, we can do some fix-ups and find errors. We might e.g. detect that a "del" was executed on a shared variable, which is not allowed for Python 2.x, so it must be caught. The parsing wouldn't do that. Also, frame objects for functions should learn their variable names. """ def onEnterNode(self, node): if python_version < 300 and node.isStatementDelVariable(): variable = node.getVariable() if not variable.isModuleVariable() and \ variable.isSharedAmongScopes(): raiseSyntaxError( """\ can not delete variable '%s' referenced in nested scope""" % ( variable.getName() ), node.getSourceReference() ) elif node.isStatementsFrame(): node.updateLocalNames() elif node.isExpressionFunctionBodyBase(): addFunctionVariableReleases(node) # Python3 is influenced by the mere use of a variable named as # "super". So we need to prepare ability to take closure. if node.hasFlag("has_super"): if not node.hasVariableName("__class__"): class_var = node.takeVariableForClosure("__class__") class_var.addVariableUser(node) node.registerProvidedVariable(class_var) while node != class_var.getOwner(): node = node.getParentVariableProvider() node.registerProvidedVariable(class_var) def completeVariableClosures(tree): visitors = ( VariableClosureLookupVisitorPhase1(), VariableClosureLookupVisitorPhase2(), VariableClosureLookupVisitorPhase3() ) for visitor in visitors: visitTree(tree, visitor) Nuitka-0.5.28.2/nuitka/tree/ReformulationSequenceCreation.py0000644000372000001440000003626013207537242024307 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of sequence creations. Sequences might be directly translated to constants, or they might become nodes that build tuples, lists, or sets. For Python3.5, unpacking can happen while creating sequences, these are being re-formulated to an internal function. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinIteratorNodes import ExpressionBuiltinIter1 from nuitka.nodes.BuiltinNextNodes import ExpressionBuiltinNext1 from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinTuple from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ContainerOperationNodes import ( ExpressionListOperationExtend, ExpressionSetOperationUpdate ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionBody, ExpressionFunctionCall, ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.LoopNodes import StatementLoop, StatementLoopBreak from nuitka.nodes.ParameterSpecs import ParameterSpec from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.VariableRefNodes import ( ExpressionTempVariableRef, ExpressionVariableRef ) from nuitka.PythonVersions import python_version from . import SyntaxErrors from .InternalModule import ( getInternalModule, internal_source_ref, once_decorator ) from .ReformulationTryExceptStatements import makeTryExceptSingleHandlerNode from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildNodeList, getKind, makeSequenceCreationOrConstant, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements ) def buildSequenceCreationNode(provider, node, source_ref): if python_version >= 300: for element in node.elts: if getKind(element) == "Starred": if python_version < 350: SyntaxErrors.raiseSyntaxError( "can use starred expression only as assignment target", source_ref.atColumnNumber(element.col_offset), ) else: return _buildSequenceUnpacking( provider = provider, node = node, source_ref = source_ref ) return makeSequenceCreationOrConstant( sequence_kind = getKind(node).upper(), elements = buildNodeList(provider, node.elts, source_ref), source_ref = source_ref ) @once_decorator def getListUnpackingHelper(): helper_name = "_unpack_list" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = (), ps_list_star_arg = "args", ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "list") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "keys") loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_item_variable, source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementExpressionOnly( expression = ExpressionListOperationExtend( list_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) args_variable = result.getVariableForAssignment( variable_name = "args" ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_result_variable, source = makeConstantRefNode( constant = [], source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementLoop( body = loop_body, source_ref = internal_source_ref ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result @once_decorator def getSetUnpackingHelper(): helper_name = "_unpack_set" result = ExpressionFunctionBody( provider = getInternalModule(), name = helper_name, doc = None, parameters = ParameterSpec( ps_name = helper_name, ps_normal_args = (), ps_list_star_arg = "args", ps_dict_star_arg = None, ps_default_count = 0, ps_kw_only_args = () ), flags = set(), source_ref = internal_source_ref ) temp_scope = None tmp_result_variable = result.allocateTempVariable(temp_scope, "set") tmp_iter_variable = result.allocateTempVariable(temp_scope, "iter") tmp_item_variable = result.allocateTempVariable(temp_scope, "keys") loop_body = makeStatementsSequenceFromStatements( makeTryExceptSingleHandlerNode( tried = StatementAssignmentVariable( variable = tmp_item_variable, source = ExpressionBuiltinNext1( value = ExpressionTempVariableRef( variable = tmp_iter_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), exception_name = "StopIteration", handler_body = StatementLoopBreak( source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementExpressionOnly( expression = ExpressionSetOperationUpdate( set_arg = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), value = ExpressionTempVariableRef( variable = tmp_item_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) args_variable = result.getVariableForAssignment( variable_name = "args" ) final = ( StatementReleaseVariable( variable = tmp_result_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_iter_variable, source_ref = internal_source_ref ), StatementReleaseVariable( variable = tmp_item_variable, source_ref = internal_source_ref ), ) tried = makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_iter_variable, source = ExpressionBuiltinIter1( value = ExpressionVariableRef( variable = args_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementAssignmentVariable( variable = tmp_result_variable, source = makeConstantRefNode( constant = set(), source_ref = internal_source_ref ), source_ref = internal_source_ref ), StatementLoop( body = loop_body, source_ref = internal_source_ref ), StatementReturn( expression = ExpressionTempVariableRef( variable = tmp_result_variable, source_ref = internal_source_ref ), source_ref = internal_source_ref ) ) result.setBody( makeStatementsSequenceFromStatement( makeTryFinallyStatement( provider = result, tried = tried, final = final, source_ref = internal_source_ref ) ) ) return result def buildListUnpacking(provider, elements, source_ref): helper_args = [] for element in elements: # We could be a lot cleverer about the tuples for non-starred # arguments, but lets get this to work first. And then rely on # future optimization to inline the list unpacking helper in a # way that has the same effect. if getKind(element) == "Starred": helper_args.append( buildNode(provider, element.value, source_ref), ) else: helper_args.append( ExpressionMakeTuple( elements = ( buildNode(provider, element, source_ref), ), source_ref = source_ref ) ) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getListUnpackingHelper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionMakeTuple( helper_args, source_ref ), ), source_ref = source_ref, ) result.setCompatibleSourceReference(helper_args[-1].getCompatibleSourceReference()) return result def _buildTupleUnpacking(provider, elements, source_ref): return ExpressionBuiltinTuple( value = buildListUnpacking(provider, elements, source_ref), source_ref = source_ref ) def _buildSetUnpacking(provider, elements, source_ref): helper_args = [] for element in elements: # We could be a lot cleverer about the tuples for non-starred # arguments, but lets get this to work first. And then rely on # future optimization to inline the list unpacking helper in a # way that has the same effect. if getKind(element) == "Starred": helper_args.append( buildNode(provider, element.value, source_ref), ) else: helper_args.append( ExpressionMakeTuple( elements = ( buildNode(provider, element, source_ref), ), source_ref = source_ref ) ) result = ExpressionFunctionCall( function = ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = getSetUnpackingHelper(), source_ref = source_ref ), code_object = None, defaults = (), kw_defaults = None, annotations = None, source_ref = source_ref ), values = ( ExpressionMakeTuple( helper_args, source_ref ), ), source_ref = source_ref, ) result.setCompatibleSourceReference(helper_args[-1].getCompatibleSourceReference()) return result def _buildSequenceUnpacking(provider, node, source_ref): kind = getKind(node) if kind == "List": return buildListUnpacking(provider, node.elts, source_ref) elif kind == "Tuple": return _buildTupleUnpacking(provider, node.elts, source_ref) elif kind == "Set": return _buildSetUnpacking(provider, node.elts, source_ref) else: assert False, kind Nuitka-0.5.28.2/nuitka/tree/Operations.py0000644000372000001440000000342213122472300020405 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Operations on the tree. This is mostly for the different kinds of visits that the node tree can have. You can visit a scope, a tree (module), or every scope of a tree (module). """ def visitTree(tree, visitor): visitor.onEnterNode(tree) for visitable in tree.getVisitableNodes(): if visitable is None: raise AssertionError( "'None' child encountered", tree, tree.source_ref ) visitTree(visitable, visitor) visitor.onLeaveNode(tree) def visitFunction(function, visitor): visitor.onEnterNode(function) visitor.onLeaveNode(function) def visitModule(module, visitor): visitor.onEnterNode(module) visitor.onLeaveNode(module) class VisitorNoopMixin(object): def onEnterNode(self, node): """ Overloaded for operation before the node children were done. """ pass def onLeaveNode(self, node): """ Overloaded for operation after the node children were done. """ pass Nuitka-0.5.28.2/nuitka/tree/Building.py0000644000372000001440000011410413207537242020032 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Build the internal node tree from source code. Does all the Python parsing and puts it into a tree structure for use in later stages of the compilation process. In the "nuitka.tree.TreeHelpers" module, the dispatching is happening. One function deals with every node kind as found in the AST. The parsing is centered around the module "ast" output. Many higher level language features and translated into lower level ones. In-place assignments, for loops, while loops, classes, complex calls, with statements, and even or/and etc. are all translated to simpler constructs. The output of this module is a node tree, which contains only relatively low level operations. A property of the output is also an overlaid tree of provider structure that indicates variable provision. Classes are handled in a separate module. They are re-formulated into functions producing dictionaries used to call the metaclass with. Try/except/else statements are handled in a separate module. They are re-formulated into using a temporary variable to track if the else branch should execute. Try/finally statements are handled in a separate module. They are re-formulated to use a nested try/finally for (un)publishing the exception for Python3. With statements are handled in a separate module. They are re-formulated into special attribute lookups for "__enter__" and "__exit__", calls of them, catching and passing in exceptions raised. """ import os import sys from logging import info, warning from nuitka import Options, SourceCodeReferences from nuitka.__past__ import ( # pylint: disable=I0021,redefined-builtin long, unicode ) from nuitka.importing import Importing from nuitka.importing.ImportCache import addImportedModule from nuitka.importing.PreloadedPackages import getPthImportedPackages from nuitka.nodes.AssignNodes import StatementAssignmentVariableName from nuitka.nodes.AttributeNodes import ExpressionAttributeLookup from nuitka.nodes.BuiltinFormatNodes import ( ExpressionBuiltinAscii, ExpressionBuiltinFormat ) from nuitka.nodes.BuiltinTypeNodes import ExpressionBuiltinStr from nuitka.nodes.ConditionalNodes import ( ExpressionConditional, StatementConditional ) from nuitka.nodes.ConstantRefNodes import ( ExpressionConstantEllipsisRef, ExpressionConstantNoneRef, makeConstantRefNode ) from nuitka.nodes.CoroutineNodes import ExpressionAsyncWait from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.GeneratorNodes import StatementGeneratorReturn from nuitka.nodes.ImportNodes import ExpressionImportModuleNameHard from nuitka.nodes.LoopNodes import StatementLoopBreak, StatementLoopContinue from nuitka.nodes.ModuleNodes import ( CompiledPythonModule, CompiledPythonPackage, ExpressionModuleFileAttributeRef, ExpressionModuleLoaderRef, PythonMainModule, PythonShlibModule ) from nuitka.nodes.OperatorNodes import ( ExpressionOperationUnary, makeBinaryOperationNode ) from nuitka.nodes.ReturnNodes import ( StatementReturn, StatementReturnNone, makeStatementReturnConstant ) from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.StringConcatenationNodes import ExpressionStringConcatenation from nuitka.nodes.VariableRefNodes import ExpressionVariableNameRef from nuitka.Options import shallWarnUnusualCode from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import python_version from nuitka.utils import MemoryUsage from nuitka.utils.FileOperations import splitPath from . import SyntaxErrors from .ReformulationAssertStatements import buildAssertNode from .ReformulationAssignmentStatements import ( buildAnnAssignNode, buildAssignNode, buildDeleteNode, buildInplaceAssignNode ) from .ReformulationBooleanExpressions import buildBoolOpNode from .ReformulationCallExpressions import buildCallNode from .ReformulationClasses import buildClassNode from .ReformulationComparisonExpressions import buildComparisonNode from .ReformulationContractionExpressions import ( buildDictContractionNode, buildGeneratorExpressionNode, buildListContractionNode, buildSetContractionNode ) from .ReformulationDictionaryCreation import buildDictionaryNode from .ReformulationExecStatements import buildExecNode from .ReformulationForLoopStatements import ( buildAsyncForLoopNode, buildForLoopNode ) from .ReformulationFunctionStatements import ( buildAsyncFunctionNode, buildFunctionNode ) from .ReformulationImportStatements import ( buildImportFromNode, buildImportModulesNode, checkFutureImportsOnlyAtStart, getFutureSpec, popFutureSpec, pushFutureSpec ) from .ReformulationLambdaExpressions import buildLambdaNode from .ReformulationNamespacePackages import ( createNamespacePackage, createPathAssignment ) from .ReformulationPrintStatements import buildPrintNode from .ReformulationSequenceCreation import buildSequenceCreationNode from .ReformulationSubscriptExpressions import buildSubscriptNode from .ReformulationTryExceptStatements import buildTryExceptionNode from .ReformulationTryFinallyStatements import buildTryFinallyNode from .ReformulationWhileLoopStatements import buildWhileLoopNode from .ReformulationWithStatements import buildAsyncWithNode, buildWithNode from .ReformulationYieldExpressions import buildYieldFromNode, buildYieldNode from .SourceReading import ( checkPythonVersionFromCode, readSourceCodeFromFilename ) from .TreeHelpers import ( buildNode, buildNodeList, buildStatementsNode, extractDocFromBody, getBuildContext, getKind, makeAbsoluteImportNode, makeCallNode, makeModuleFrame, makeStatementsSequence, makeStatementsSequenceFromStatement, mangleName, mergeStatements, parseSourceCodeToAst, setBuildingDispatchers ) from .VariableClosure import completeVariableClosures def buildVariableReferenceNode(provider, node, source_ref): return ExpressionVariableNameRef( variable_name = mangleName(node.id, provider), source_ref = source_ref ) # Python3.4 or higher, True and False, are not given as variables anymore. def buildNamedConstantNode(node, source_ref): return makeConstantRefNode( constant = node.value, source_ref = source_ref ) def buildConditionNode(provider, node, source_ref): # Conditional statements may have one or two branches. We will never see an # "elif", because that's already dealt with by module "ast", which turns it # into nested conditional statements. return StatementConditional( condition = buildNode(provider, node.test, source_ref), yes_branch = buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ), no_branch = buildStatementsNode( provider = provider, nodes = node.orelse if node.orelse else None, source_ref = source_ref ), source_ref = source_ref ) def buildTryFinallyNode2(provider, node, source_ref): # Try/finally node statements of old style. return buildTryFinallyNode( provider = provider, build_tried = lambda : buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ), node = node, source_ref = source_ref ) def buildTryNode(provider, node, source_ref): # Note: This variant is used for Python3.3 or higher only, older stuff uses # the above ones, this one merges try/except with try/finally in the # "ast". We split it up again, as it's logically separated of course. # Shortcut missing try/finally. if not node.handlers: return buildTryFinallyNode2(provider, node, source_ref) if not node.finalbody: return buildTryExceptionNode( provider = provider, node = node, source_ref = source_ref ) return buildTryFinallyNode( provider = provider, build_tried = lambda : makeStatementsSequence( statements = mergeStatements( ( buildTryExceptionNode( provider = provider, node = node, source_ref = source_ref ), ), allow_none = True ), allow_none = True, source_ref = source_ref ), node = node, source_ref = source_ref ) def buildRaiseNode(provider, node, source_ref): # Raise statements. Under Python2 they may have type, value and traceback # attached, for Python3, you can only give type (actually value) and cause. if python_version < 300: exception_type = buildNode(provider, node.type, source_ref, allow_none = True) exception_value = buildNode(provider, node.inst, source_ref, allow_none = True) exception_trace = buildNode(provider, node.tback, source_ref, allow_none = True) exception_cause = None else: exception_type = buildNode(provider, node.exc, source_ref, allow_none = True) exception_value = None exception_trace = None exception_cause = buildNode(provider, node.cause, source_ref, allow_none = True) result = StatementRaiseException( exception_type = exception_type, exception_value = exception_value, exception_trace = exception_trace, exception_cause = exception_cause, source_ref = source_ref ) if exception_cause is not None: result.setCompatibleSourceReference( source_ref = exception_cause.getCompatibleSourceReference() ) elif exception_trace is not None: result.setCompatibleSourceReference( source_ref = exception_trace.getCompatibleSourceReference() ) elif exception_value is not None: result.setCompatibleSourceReference( source_ref = exception_value.getCompatibleSourceReference() ) elif exception_type is not None: result.setCompatibleSourceReference( source_ref = exception_type.getCompatibleSourceReference() ) return result def handleGlobalDeclarationNode(provider, node, source_ref): # On the module level, there is nothing to do. if provider.isCompiledPythonModule(): if shallWarnUnusualCode(): warning( "%s: Using 'global' statement on module level has no effect.", source_ref.getAsString(), ) return None # Need to catch the error of declaring a parameter variable as global # ourselves here. The AST parsing doesn't catch it, so we check here. if provider.isExpressionFunctionBody(): parameters = provider.getParameters() for variable_name in node.names: if variable_name in parameters.getParameterNames(): SyntaxErrors.raiseSyntaxError( "name '%s' is %s and global" % ( variable_name, "local" if python_version < 300 else "parameter" ), source_ref.atColumnNumber(node.col_offset) ) # The module the "global" statement refers to. module = provider.getParentModule() # Can give multiple names. for variable_name in node.names: closure_variable = None # Re-use already taken global variables, in order to avoid creating yet # another instance, esp. as the indications could then potentially not # be shared. if provider.hasTakenVariable(variable_name): closure_variable = provider.getTakenVariable(variable_name) # Only global variables count. Could have a closure reference to # a location of a parent function here. if not closure_variable.isModuleVariable(): closure_variable = None if closure_variable is None: module_variable = module.getVariableForAssignment( variable_name = variable_name ) closure_variable = provider.addClosureVariable( variable = module_variable ) assert closure_variable.isModuleVariable() if python_version < 340 and \ provider.isExpressionClassBody() and \ closure_variable.getName() == "__class__": SyntaxErrors.raiseSyntaxError( "cannot make __class__ global", source_ref ) provider.registerProvidedVariable( variable = closure_variable ) # Drop this, not really part of our tree. return None def handleNonlocalDeclarationNode(provider, node, source_ref): # Need to catch the error of declaring a parameter variable as global # ourselves here. The AST parsing doesn't catch it, but we can do it here. parameter_provider = provider while parameter_provider.isExpressionGeneratorObjectBody() or \ parameter_provider.isExpressionCoroutineObjectBody() or \ parameter_provider.isExpressionAsyncgenObjectBody(): parameter_provider = parameter_provider.getParentVariableProvider() if parameter_provider.isExpressionClassBody(): parameter_names = () else: parameter_names = parameter_provider.getParameters().getParameterNames() for variable_name in node.names: if variable_name in parameter_names: SyntaxErrors.raiseSyntaxError( "name '%s' is parameter and nonlocal" % ( variable_name ), source_ref.atColumnNumber(node.col_offset) ) provider.addNonlocalsDeclaration( names = node.names, source_ref = source_ref.atColumnNumber(node.col_offset) ) # Drop this, not really part of our tree. return None def buildStringNode(node, source_ref): assert type(node.s) in (str, unicode) return makeConstantRefNode( constant = node.s, source_ref = source_ref, user_provided = True ) def buildNumberNode(node, source_ref): assert type(node.n) in (int, long, float, complex), type(node.n) return makeConstantRefNode( constant = node.n, source_ref = source_ref, user_provided = True ) def buildBytesNode(node, source_ref): return makeConstantRefNode( constant = node.s, source_ref = source_ref, user_provided = True ) def buildEllipsisNode(source_ref): return ExpressionConstantEllipsisRef( source_ref = source_ref, user_provided = True ) def buildStatementLoopContinue(node, source_ref): source_ref = source_ref.atColumnNumber(node.col_offset) # Python forbids this, although technically it's probably not much of # an issue. if getBuildContext() == "finally": SyntaxErrors.raiseSyntaxError( "'continue' not supported inside 'finally' clause", source_ref ) return StatementLoopContinue( source_ref = source_ref ) def buildStatementLoopBreak(provider, node, source_ref): # A bit unusual, we need the provider, but not the node, # pylint: disable=unused-argument return StatementLoopBreak( source_ref = source_ref.atColumnNumber(node.col_offset) ) def buildAttributeNode(provider, node, source_ref): return ExpressionAttributeLookup( source = buildNode(provider, node.value, source_ref), attribute_name = node.attr, source_ref = source_ref ) def buildReturnNode(provider, node, source_ref): if provider.isExpressionClassBody() or provider.isCompiledPythonModule(): SyntaxErrors.raiseSyntaxError( "'return' outside function", source_ref.atColumnNumber(node.col_offset) ) expression = buildNode(provider, node.value, source_ref, allow_none = True) if provider.isExpressionGeneratorObjectBody(): if expression is not None and python_version < 330: SyntaxErrors.raiseSyntaxError( "'return' with argument inside generator", source_ref.atColumnNumber(node.col_offset) ) if provider.isExpressionAsyncgenObjectBody(): if expression is not None: SyntaxErrors.raiseSyntaxError( "'return' with value in async generator", source_ref.atColumnNumber(node.col_offset) ) if provider.isExpressionGeneratorObjectBody() or \ provider.isExpressionAsyncgenObjectBody(): if expression is None: expression = ExpressionConstantNoneRef( source_ref = source_ref, user_provided = True ) return StatementGeneratorReturn( expression = expression, source_ref = source_ref ) else: if expression is None: return StatementReturnNone( source_ref = source_ref ) elif expression.isExpressionConstantRef(): return makeStatementReturnConstant( constant = expression.getCompileTimeConstant(), source_ref = source_ref ) else: return StatementReturn( expression = expression, source_ref = source_ref ) def buildExprOnlyNode(provider, node, source_ref): result = StatementExpressionOnly( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) result.setCompatibleSourceReference( result.getExpression().getCompatibleSourceReference() ) return result def buildUnaryOpNode(provider, node, source_ref): if getKind(node.op) == "Not": return buildBoolOpNode( provider = provider, node = node, source_ref = source_ref ) else: return ExpressionOperationUnary( operator = getKind(node.op), operand = buildNode(provider, node.operand, source_ref), source_ref = source_ref ) def buildBinaryOpNode(provider, node, source_ref): operator = getKind(node.op) if operator == "Div": if getFutureSpec().isFutureDivision(): operator = "TrueDiv" left = buildNode(provider, node.left, source_ref) right = buildNode(provider, node.right, source_ref) result = makeBinaryOperationNode( operator = operator, left = left, right = right, source_ref = source_ref ) result.setCompatibleSourceReference( source_ref = right.getCompatibleSourceReference() ) return result def buildReprNode(provider, node, source_ref): return ExpressionOperationUnary( operator = "Repr", operand = buildNode(provider, node.value, source_ref), source_ref = source_ref ) def buildConditionalExpressionNode(provider, node, source_ref): return ExpressionConditional( condition = buildNode(provider, node.test, source_ref), expression_yes = buildNode(provider, node.body, source_ref), expression_no = buildNode(provider, node.orelse, source_ref), source_ref = source_ref ) def buildAwaitNode(provider, node, source_ref): return ExpressionAsyncWait( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) def buildFormattedValueNode(provider, node, source_ref): value = buildNode(provider, node.value, source_ref) conversion = node.conversion % 4 if node.conversion > 0 else 0 if conversion == 0: pass elif conversion == 3: value = ExpressionBuiltinStr( value = value, encoding = None, errors = None, source_ref = source_ref ) elif conversion == 2: value = ExpressionOperationUnary( operator = "Repr", operand = value, source_ref = source_ref ) elif conversion == 1: value = ExpressionBuiltinAscii( value = value, source_ref = source_ref ) else: assert False, conversion return ExpressionBuiltinFormat( value = value, format_spec = buildNode(provider, node.format_spec, source_ref, allow_none = True), source_ref = source_ref ) def buildJoinedStrNode(provider, node, source_ref): return ExpressionStringConcatenation( values = buildNodeList(provider, node.values, source_ref), source_ref = source_ref ) setBuildingDispatchers( path_args3 = { "Name" : buildVariableReferenceNode, "Assign" : buildAssignNode, "AnnAssign" : buildAnnAssignNode, "Delete" : buildDeleteNode, "Lambda" : buildLambdaNode, "GeneratorExp" : buildGeneratorExpressionNode, "If" : buildConditionNode, "While" : buildWhileLoopNode, "For" : buildForLoopNode, "AsyncFor" : buildAsyncForLoopNode, "Compare" : buildComparisonNode, "ListComp" : buildListContractionNode, "DictComp" : buildDictContractionNode, "SetComp" : buildSetContractionNode, "Dict" : buildDictionaryNode, "Set" : buildSequenceCreationNode, "Tuple" : buildSequenceCreationNode, "List" : buildSequenceCreationNode, "Global" : handleGlobalDeclarationNode, "Nonlocal" : handleNonlocalDeclarationNode, "TryExcept" : buildTryExceptionNode, "TryFinally" : buildTryFinallyNode2, "Try" : buildTryNode, "Raise" : buildRaiseNode, "Import" : buildImportModulesNode, "ImportFrom" : buildImportFromNode, "Assert" : buildAssertNode, "Exec" : buildExecNode, "With" : buildWithNode, "AsyncWith" : buildAsyncWithNode, "FunctionDef" : buildFunctionNode, "AsyncFunctionDef" : buildAsyncFunctionNode, "Await" : buildAwaitNode, "ClassDef" : buildClassNode, "Print" : buildPrintNode, "Call" : buildCallNode, "Subscript" : buildSubscriptNode, "BoolOp" : buildBoolOpNode, "Attribute" : buildAttributeNode, "Return" : buildReturnNode, "Yield" : buildYieldNode, "YieldFrom" : buildYieldFromNode, "Expr" : buildExprOnlyNode, "UnaryOp" : buildUnaryOpNode, "BinOp" : buildBinaryOpNode, "Repr" : buildReprNode, "AugAssign" : buildInplaceAssignNode, "IfExp" : buildConditionalExpressionNode, "Break" : buildStatementLoopBreak, "JoinedStr" : buildJoinedStrNode, "FormattedValue" : buildFormattedValueNode, }, path_args2 = { "NameConstant" : buildNamedConstantNode, "Str" : buildStringNode, "Num" : buildNumberNode, "Bytes" : buildBytesNode, "Continue" : buildStatementLoopContinue, }, path_args1 = { "Ellipsis" : buildEllipsisNode, } ) def buildParseTree(provider, source_code, source_ref, is_module, is_main): # There are a bunch of branches here, mostly to deal with version # differences for module default variables. pylint: disable=too-many-branches pushFutureSpec() if is_module: provider.future_spec = getFutureSpec() body = parseSourceCodeToAst( source_code = source_code, filename = source_ref.getFilename(), line_offset = source_ref.getLineNumber() - 1 ) body, doc = extractDocFromBody(body) if is_module and is_main and python_version >= 360: provider.markAsNeedsAnnotationsDictionary() result = buildStatementsNode( provider = provider, nodes = body, source_ref = source_ref ) checkFutureImportsOnlyAtStart(body) internal_source_ref = source_ref.atInternal() statements = [] if is_module: # Add import of "site" module of main programs visibly in the node tree, # so recursion and optimization can pick it up, checking its effects. if is_main and "no_site" not in Options.getPythonFlags(): for path_imported_name in getPthImportedPackages(): statements.append( StatementExpressionOnly( expression = makeAbsoluteImportNode( module_name = path_imported_name, source_ref = source_ref, ), source_ref = source_ref ) ) statements.append( StatementExpressionOnly( expression = makeAbsoluteImportNode( module_name = "site", source_ref = source_ref, ), source_ref = source_ref ) ) statements.append( StatementAssignmentVariableName( variable_name = "__doc__", source = makeConstantRefNode( constant = doc, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) statements.append( StatementAssignmentVariableName( variable_name = "__file__", source = ExpressionModuleFileAttributeRef( source_ref = internal_source_ref, ), source_ref = internal_source_ref ) ) if provider.isCompiledPythonPackage(): # This assigns "__path__" value. statements.append( createPathAssignment(provider, internal_source_ref) ) if python_version >= 330: statements.append( StatementAssignmentVariableName( variable_name = "__loader__", source = ExpressionModuleLoaderRef( source_ref = internal_source_ref, ), source_ref = internal_source_ref ) ) if python_version >= 340: if provider.isMainModule(): spec_value = makeConstantRefNode( constant = None, source_ref = internal_source_ref ) else: spec_value = makeCallNode( ExpressionImportModuleNameHard( module_name = "importlib._bootstrap", import_name = "ModuleSpec", source_ref = internal_source_ref, ), makeConstantRefNode( constant = provider.getFullName(), source_ref = internal_source_ref, user_provided = True ), ExpressionModuleLoaderRef( source_ref = internal_source_ref, ), internal_source_ref ) statements.append( StatementAssignmentVariableName( variable_name = "__spec__", source = spec_value, source_ref = internal_source_ref ) ) if python_version >= 300: statements.append( StatementAssignmentVariableName( variable_name = "__cached__", source = ExpressionConstantNoneRef( source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) # The "__package__" attribute is set to proper value for 3.3 or higher even # for normal modules, previously for packages only. statements.append( StatementAssignmentVariableName( variable_name = "__package__", source = makeConstantRefNode( constant = ( ( None if '.' not in provider.getFullName() or python_version >= 320 else provider.getFullName() ) if python_version < 330 else provider.getFullName() ) if provider.isCompiledPythonPackage() else ( provider.getPackage() if python_version >= 330 else None ), source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) needs__initializing__ = not provider.isMainModule() and \ (python_version >= 330 and python_version < 340) if needs__initializing__: # Set "__initializing__" at the beginning to True statements.append( StatementAssignmentVariableName( variable_name = "__initializing__", source = makeConstantRefNode( constant = True, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) if provider.needsAnnotationsDictionary(): # Set "__annotations__" on module level to {} statements.append( StatementAssignmentVariableName( variable_name = "__annotations__", source = makeConstantRefNode( constant = {}, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) # Now the module body if there is any at all. if result is not None: statements.extend( result.getStatements() ) if needs__initializing__: # Set "__initializing__" at the end to False statements.append( StatementAssignmentVariableName( variable_name = "__initializing__", source = makeConstantRefNode( constant = False, source_ref = internal_source_ref, user_provided = True ), source_ref = internal_source_ref ) ) if is_module: result = makeModuleFrame( module = provider, statements = statements, source_ref = source_ref ) popFutureSpec() return result else: assert False def decideModuleTree(filename, package, is_shlib, is_top, is_main): # Many variables, branches, due to the many cases, pylint: disable=too-many-branches,too-many-statements assert package is None or type(package) is str assert filename is not None if is_main and os.path.isdir(filename): source_filename = os.path.join(filename, "__main__.py") if not os.path.isfile(source_filename): sys.stderr.write( "%s: can't find '__main__' module in '%s'\n" % ( os.path.basename(sys.argv[0]), filename ) ) sys.exit(2) filename = source_filename main_added = True else: main_added = False if os.path.isfile(filename): source_filename = filename source_ref = SourceCodeReferences.fromFilename( filename = filename, ) if is_main: module_name = "__main__" else: module_name = os.path.basename(filename) if module_name.endswith(".py"): module_name = module_name[:-3] if is_shlib: module_name = module_name.split('.')[0] if '.' in module_name: sys.stderr.write( "Error, '%s' is not a proper python module name.\n" % ( module_name ) ) sys.exit(2) if is_shlib: result = PythonShlibModule( name = module_name, package_name = package, source_ref = source_ref ) elif is_main: result = PythonMainModule( main_added = main_added, mode = Plugins.decideCompilation("__main__", source_ref), future_spec = None, source_ref = source_ref ) else: if package is not None: full_name = package + '.' + module_name else: full_name = module_name result = CompiledPythonModule( name = module_name, package_name = package, mode = Plugins.decideCompilation(full_name, source_ref), future_spec = None, source_ref = source_ref ) elif Importing.isPackageDir(filename): if is_top: package_name = splitPath(filename)[-1] else: package_name = os.path.basename(filename) source_filename = os.path.join(filename, "__init__.py") if not os.path.isfile(source_filename): source_ref, result = createNamespacePackage( package_name = package_name, module_relpath = filename ) source_filename = None else: source_ref = SourceCodeReferences.fromFilename( filename = os.path.abspath(source_filename), ) if package is not None: full_name = package + '.' + package_name else: full_name = package_name result = CompiledPythonPackage( name = package_name, package_name = package, mode = Plugins.decideCompilation(full_name, source_ref), future_spec = None, source_ref = source_ref ) assert result.getFullName() == full_name, result else: sys.stderr.write( "%s: can't open file '%s'.\n" % ( os.path.basename(sys.argv[0]), filename ) ) sys.exit(2) if not Options.shallHaveStatementLines(): source_ref = source_ref.atInternal() return result, source_ref, source_filename class CodeTooComplexCode(Exception): """ The code of the module is too complex. It cannot be compiled, with recursive code, and therefore the bytecode should be used instead. Example of this is "idnadata". """ pass def createModuleTree(module, source_ref, source_code, is_main): if Options.isShowMemory(): memory_watch = MemoryUsage.MemoryWatch() try: module_body = buildParseTree( provider = module, source_code = source_code, source_ref = source_ref, is_module = True, is_main = is_main ) except RuntimeError as e: if "maximum recursion depth" in e.args[0]: raise CodeTooComplexCode( module.getFullName(), module.getCompileTimeFilename() ) raise if module_body.isStatementsFrame(): module_body = makeStatementsSequenceFromStatement( statement = module_body, ) module.setBody(module_body) completeVariableClosures(module) if Options.isShowMemory(): memory_watch.finish() info( "Memory usage changed loading module '%s': %s" % ( module.getFullName(), memory_watch.asStr() ) ) def buildModuleTree(filename, package, is_top, is_main): module, source_ref, source_filename = decideModuleTree( filename = filename, package = package, is_top = is_top, is_main = is_main, is_shlib = False ) # If there is source code associated (not the case for namespace packages of # Python3.3 or higher, then read it. if source_filename is not None: source_code = readSourceCodeFromFilename( module_name = module.getFullName(), source_filename = source_filename ) if is_main: checkPythonVersionFromCode(source_code) # Read source code. createModuleTree( module = module, source_ref = source_ref, source_code = source_code, is_main = is_main ) if not module.isMainModule(): addImportedModule( imported_module = module ) return module Nuitka-0.5.28.2/nuitka/tree/ReformulationYieldExpressions.py0000644000372000001440000000543213207537242024360 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of "yield" and "yield from" expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ import ast from nuitka.nodes.ConstantRefNodes import ExpressionConstantNoneRef from nuitka.nodes.YieldNodes import ExpressionYield, ExpressionYieldFrom from nuitka.PythonVersions import python_version from .SyntaxErrors import raiseSyntaxError from .TreeHelpers import buildNode def _checkInsideGenerator(provider, node, source_ref): if provider.isCompiledPythonModule(): raiseSyntaxError( "'yield' outside function", source_ref.atColumnNumber(node.col_offset) ) # This yield is forbidden in 3.5, but allowed in 3.6, but yield_from # is neither. if provider.isExpressionAsyncgenObjectBody() and \ (node.__class__ is not ast.Yield or python_version < 360): raiseSyntaxError( "'%s' inside async function" % ( "yield" if node.__class__ is ast.Yield else "yield from", ), source_ref.atColumnNumber(node.col_offset) ) assert provider.isExpressionGeneratorObjectBody() or \ provider.isExpressionAsyncgenObjectBody(), provider def buildYieldNode(provider, node, source_ref): _checkInsideGenerator(provider, node, source_ref) if node.value is not None: return ExpressionYield( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) else: return ExpressionYield( expression = ExpressionConstantNoneRef( source_ref = source_ref, user_provided = True ), source_ref = source_ref ) def buildYieldFromNode(provider, node, source_ref): assert python_version >= 330 _checkInsideGenerator(provider, node, source_ref) return ExpressionYieldFrom( expression = buildNode(provider, node.value, source_ref), source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/ReformulationAssertStatements.py0000644000372000001440000000575013207537242024363 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of assert statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.BuiltinRefNodes import ExpressionBuiltinExceptionRef from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ContainerMakingNodes import ExpressionMakeTuple from nuitka.nodes.ExceptionNodes import StatementRaiseException from nuitka.nodes.OperatorNodes import ExpressionOperationNOT from nuitka.nodes.StatementNodes import StatementsSequence from nuitka.Options import getPythonFlags from nuitka.PythonVersions import python_version from .TreeHelpers import buildNode def buildAssertNode(provider, node, source_ref): # Build assert statements. These are re-formulated as described in the # developer manual too. They end up as conditional statement with raises of # AssertionError exceptions. # Underlying assumption: # # Assert x, y is the same as: # if not x: # raise AssertionError, y # Therefore assert statements are really just conditional statements with a # static raise contained. # exception_value = buildNode(provider, node.msg, source_ref, True) if "no_asserts" in getPythonFlags(): return None if exception_value is not None and python_version > 272: exception_value = ExpressionMakeTuple( elements = (exception_value,), source_ref = source_ref ) raise_statement = StatementRaiseException( exception_type = ExpressionBuiltinExceptionRef( exception_name = "AssertionError", source_ref = source_ref ), exception_value = exception_value, exception_trace = None, exception_cause = None, source_ref = source_ref ) return StatementConditional( condition = ExpressionOperationNOT( operand = buildNode(provider, node.test, source_ref), source_ref = source_ref ), yes_branch = StatementsSequence( statements = ( raise_statement, ), source_ref = source_ref ), no_branch = None, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/ReformulationLambdaExpressions.py0000644000372000001440000001622413207537242024473 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of lambda expressions. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.ComparisonNodes import ExpressionComparisonIsNOT from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import ExpressionConstantNoneRef from nuitka.nodes.FrameNodes import ( StatementsFrameFunction, StatementsFrameGenerator ) from nuitka.nodes.FunctionNodes import ( ExpressionFunctionCreation, ExpressionFunctionRef ) from nuitka.nodes.GeneratorNodes import ( ExpressionGeneratorObjectBody, ExpressionMakeGeneratorObject ) from nuitka.nodes.ReturnNodes import StatementReturn from nuitka.nodes.StatementNodes import StatementExpressionOnly from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from nuitka.nodes.YieldNodes import ExpressionYield from nuitka.PythonVersions import python_version from .ReformulationFunctionStatements import ( buildFunctionWithParsing, buildParameterAnnotations, buildParameterKwDefaults ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .TreeHelpers import ( buildNode, buildNodeList, detectFunctionBodyKind, getKind, makeStatementsSequenceFromStatement, mergeStatements ) def buildLambdaNode(provider, node, source_ref): # Many details to deal with, pylint: disable=too-many-locals assert getKind(node) == "Lambda" function_kind, flags = detectFunctionBodyKind( nodes = (node.body,) ) outer_body, function_body, code_object = buildFunctionWithParsing( provider = provider, function_kind = function_kind, name = "", function_doc = None, flags = flags, node = node, source_ref = source_ref ) if function_kind == "Function": code_body = function_body else: code_body = ExpressionGeneratorObjectBody( provider = function_body, name = "", flags = set(), source_ref = source_ref ) code_body.qualname_provider = provider if function_kind == "Generator": function_body.setBody( makeStatementsSequenceFromStatement( statement = StatementReturn( expression = ExpressionMakeGeneratorObject( generator_ref = ExpressionFunctionRef( function_body = code_body, source_ref = source_ref ), code_object = code_object, source_ref = source_ref ), source_ref = source_ref ) ) ) defaults = buildNodeList(provider, node.args.defaults, source_ref) kw_defaults = buildParameterKwDefaults( provider = provider, node = node, function_body = function_body, source_ref = source_ref ) body = buildNode( provider = code_body, node = node.body, source_ref = source_ref, ) if function_kind == "Generator": if python_version < 270: tmp_return_value = code_body.allocateTempVariable( temp_scope = None, name = "yield_return" ) statements = ( StatementAssignmentVariable( variable = tmp_return_value, source = body, source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIsNOT( left = ExpressionTempVariableRef( variable = tmp_return_value, source_ref = source_ref, ), right = ExpressionConstantNoneRef( source_ref = source_ref ), source_ref = source_ref ), yes_branch = makeStatementsSequenceFromStatement( statement = StatementExpressionOnly( expression = ExpressionYield( expression = ExpressionTempVariableRef( variable = tmp_return_value, source_ref = source_ref, ), source_ref = source_ref ), source_ref = source_ref ) ), no_branch = None, source_ref = source_ref ) ) body = makeTryFinallyStatement( provider = provider, tried = statements, final = StatementReleaseVariable( variable = tmp_return_value, source_ref = source_ref ), source_ref = source_ref ) else: body = StatementExpressionOnly( expression = body, source_ref = source_ref ) else: body = StatementReturn( expression = body, source_ref = source_ref ) if function_kind == "Generator": frame_class = StatementsFrameGenerator else: frame_class = StatementsFrameFunction body = frame_class( statements = mergeStatements( (body,) ), code_object = code_object, source_ref = body.getSourceReference() ) body = makeStatementsSequenceFromStatement( statement = body, ) code_body.setBody(body) annotations = buildParameterAnnotations(provider, node, source_ref) return ExpressionFunctionCreation( function_ref = ExpressionFunctionRef( function_body = outer_body, source_ref = source_ref ), code_object = code_object, defaults = defaults, kw_defaults = kw_defaults, annotations = annotations, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/ReformulationTryExceptStatements.py0000644000372000001440000004014513207537242025046 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Reformulation of try/except statements. Consult the developer manual for information. TODO: Add ability to sync source code comments with developer manual sections. """ from nuitka.nodes.AssignNodes import ( StatementAssignmentVariable, StatementReleaseVariable ) from nuitka.nodes.BuiltinRefNodes import ExpressionBuiltinExceptionRef from nuitka.nodes.ComparisonNodes import ( ExpressionComparisonExceptionMatch, ExpressionComparisonIs ) from nuitka.nodes.ConditionalNodes import StatementConditional from nuitka.nodes.ConstantRefNodes import makeConstantRefNode from nuitka.nodes.ExceptionNodes import ( ExpressionCaughtExceptionTypeRef, ExpressionCaughtExceptionValueRef ) from nuitka.nodes.StatementNodes import ( StatementPreserveFrameException, StatementPublishException, StatementRestoreFrameException, StatementsSequence ) from nuitka.nodes.TryNodes import StatementTry from nuitka.nodes.VariableRefNodes import ExpressionTempVariableRef from nuitka.PythonVersions import python_version from .ReformulationAssignmentStatements import ( buildAssignmentStatements, buildDeleteStatementFromDecoded, decodeAssignTarget ) from .ReformulationTryFinallyStatements import makeTryFinallyStatement from .SyntaxErrors import raiseSyntaxError from .TreeHelpers import ( buildNode, buildStatementsNode, makeReraiseExceptionStatement, makeStatementsSequence, makeStatementsSequenceFromStatement, makeStatementsSequenceFromStatements, mergeStatements ) def makeTryExceptNoRaise(provider, temp_scope, tried, handling, no_raise, source_ref): # This helper executes the core re-formulation of "no_raise" blocks, which # are the "else" blocks of "try"/"except" statements. In order to limit the # execution, we use an indicator variable instead, which will signal that # the tried block executed up to the end. And then we make the else block be # a conditional statement checking that. tmp_handler_indicator_variable = provider.allocateTempVariable( temp_scope = temp_scope, name = "unhandled_indicator" ) statements = mergeStatements( ( StatementAssignmentVariable( variable = tmp_handler_indicator_variable, source = makeConstantRefNode( constant = False, source_ref = source_ref ), source_ref = no_raise.getSourceReference().atInternal() ), handling ), allow_none = True ) handling = StatementsSequence( statements = statements, source_ref = source_ref ) tried = ( StatementTry( tried = tried, except_handler = handling, break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ), StatementConditional( condition = ExpressionComparisonIs( left = ExpressionTempVariableRef( variable = tmp_handler_indicator_variable, source_ref = source_ref ), right = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref ), yes_branch = no_raise, no_branch = None, source_ref = source_ref ) ) final = StatementReleaseVariable( variable = tmp_handler_indicator_variable, source_ref = source_ref.atInternal() ) return makeStatementsSequenceFromStatements( StatementAssignmentVariable( variable = tmp_handler_indicator_variable, source = makeConstantRefNode( constant = True, source_ref = source_ref ), source_ref = source_ref.atInternal() ), makeTryFinallyStatement( provider = provider, tried = tried, final = final, source_ref = source_ref ) ) def _makeTryExceptSingleHandlerNode(provider, public_exc, tried, exception_name, handler_body, source_ref): # No need to create this in the first place if nothing is tried. if tried is None: return None if public_exc: preserver_id = provider.allocatePreserverId() handling = [ StatementPreserveFrameException( preserver_id = preserver_id, source_ref = source_ref ), StatementPublishException( source_ref = source_ref ) ] else: handling = [] if not handler_body.isStatementsSequence(): handler_body = makeStatementsSequenceFromStatement( statement = handler_body ) if not tried.isStatementsSequence(): tried = makeStatementsSequenceFromStatement( statement = tried ) handling.append( StatementConditional( condition = ExpressionComparisonExceptionMatch( left = ExpressionCaughtExceptionTypeRef( source_ref = source_ref ), right = ExpressionBuiltinExceptionRef( exception_name = exception_name, source_ref = source_ref ), source_ref = source_ref ), yes_branch = handler_body, no_branch = makeReraiseExceptionStatement( source_ref = source_ref ), source_ref = source_ref ) ) if python_version >= 300 and public_exc: handling = ( makeTryFinallyStatement( provider = provider, tried = handling, final = StatementRestoreFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), source_ref = source_ref.atInternal() ), ) handling = makeStatementsSequenceFromStatements( *handling ) return StatementTry( tried = tried, except_handler = handling, break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ) def makeTryExceptSingleHandlerNode(tried, exception_name, handler_body, source_ref): return _makeTryExceptSingleHandlerNode( provider = None, public_exc = False, tried = tried, exception_name = exception_name, handler_body = handler_body, source_ref = source_ref ) def makeTryExceptSingleHandlerNodeWithPublish(provider, public_exc, tried, exception_name, handler_body, source_ref): return _makeTryExceptSingleHandlerNode( provider = provider, public_exc = public_exc, tried = tried, exception_name = exception_name, handler_body = handler_body, source_ref = source_ref ) def buildTryExceptionNode(provider, node, source_ref): # Try/except nodes. Re-formulated as described in the developer # manual. Exception handlers made the assignment to variables explicit. Same # for the "del" as done for Python3. Also catches always work a tuple of # exception types and hides away that they may be built or not. # Many variables and branches, due to the re-formulation that is going on # here, which just has the complexity, pylint: disable=too-many-branches,too-many-locals tried = buildStatementsNode( provider = provider, nodes = node.body, source_ref = source_ref ) handlers = [] for handler in node.handlers: exception_expression, exception_assign, exception_block = ( handler.type, handler.name, handler.body ) if exception_assign is None: statements = [ buildStatementsNode( provider = provider, nodes = exception_block, source_ref = source_ref ) ] elif python_version < 300: statements = [ buildAssignmentStatements( provider = provider, node = exception_assign, source = ExpressionCaughtExceptionValueRef( source_ref = source_ref.atInternal() ), source_ref = source_ref.atInternal() ), buildStatementsNode( provider = provider, nodes = exception_block, source_ref = source_ref ) ] else: # Python3 requires temporary assignment of exception assignment. target_info = decodeAssignTarget( provider = provider, node = exception_assign, source_ref = source_ref, ) kind, detail = target_info assert kind == "Name", kind kind = "Name_Exception" statements = [ buildAssignmentStatements( provider = provider, node = exception_assign, source = ExpressionCaughtExceptionValueRef( source_ref = source_ref.atInternal() ), source_ref = source_ref.atInternal() ), makeTryFinallyStatement( provider = provider, tried = buildStatementsNode( provider = provider, nodes = exception_block, source_ref = source_ref ), final = buildDeleteStatementFromDecoded( kind = kind, detail = detail, source_ref = source_ref ), source_ref = source_ref ) ] handler_body = makeStatementsSequence( statements = statements, allow_none = True, source_ref = source_ref ) exception_types = buildNode( provider = provider, node = exception_expression, source_ref = source_ref, allow_none = True ) # The exception types should be a tuple, so as to be most general. if exception_types is None: if handler is not node.handlers[-1]: raiseSyntaxError( "default 'except:' must be last", source_ref.atLineNumber(handler.lineno).atColumnNumber( handler.col_offset ) ) handlers.append( ( exception_types, handler_body, ) ) # Re-raise by default exception_handling = makeReraiseExceptionStatement( source_ref = source_ref ) for exception_type, handler in reversed(handlers): if exception_type is None: # A default handler was given, so use that indeed. exception_handling = handler else: exception_handling = StatementsSequence( statements = ( StatementConditional( condition = ExpressionComparisonExceptionMatch( left = ExpressionCaughtExceptionTypeRef( source_ref = exception_type.source_ref ), right = exception_type, source_ref = exception_type.source_ref ), yes_branch = handler, no_branch = exception_handling, source_ref = exception_type.source_ref ), ), source_ref = exception_type.source_ref ) if exception_handling is None: # For Python3, we need not publish at all, if all we do is to revert # that immediately. For Python2, the publish may release previously # published exception, which has side effects potentially. if python_version < 300: exception_handling = StatementsSequence( statements = ( StatementPreserveFrameException( preserver_id = 0, # unused with Python2 source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref.atInternal() ), ), source_ref = source_ref.atInternal() ) else: if python_version < 300: exception_handling.setStatements( ( StatementPreserveFrameException( preserver_id = 0, # unused with Python2 source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref.atInternal() ), ) + exception_handling.getStatements() ) else: preserver_id = provider.allocatePreserverId() exception_handling = makeStatementsSequenceFromStatements( StatementPreserveFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), StatementPublishException( source_ref = source_ref.atInternal() ), makeTryFinallyStatement( provider = provider, tried = exception_handling, final = StatementRestoreFrameException( preserver_id = preserver_id, source_ref = source_ref.atInternal() ), source_ref = source_ref ), ) no_raise = buildStatementsNode( provider = provider, nodes = node.orelse, source_ref = source_ref ) if no_raise is None: if tried is None: return None return StatementTry( tried = tried, except_handler = exception_handling, break_handler = None, continue_handler = None, return_handler = None, source_ref = source_ref ) else: if tried is None: return no_raise return makeTryExceptNoRaise( provider = provider, temp_scope = provider.allocateTempScope("try_except"), handling = exception_handling, tried = tried, no_raise = no_raise, source_ref = source_ref ) Nuitka-0.5.28.2/nuitka/tree/SourceReading.py0000644000372000001440000001367613134660221021034 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Read source code from files. This is tremendously more complex than one might think, due to encoding issues and version differences of Python versions. """ import os import re import sys from nuitka import Options, SourceCodeReferences from nuitka.__past__ import unicode # pylint: disable=I0021,redefined-builtin from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import python_version, python_version_str from nuitka.utils.Shebang import getShebangFromSource, parseShebang from nuitka.utils.Utils import getOS from .SyntaxErrors import raiseSyntaxError def _readSourceCodeFromFilename3(source_filename): # Only using this for Python3, for Python2 it's too buggy. import tokenize with tokenize.open(source_filename) as source_file: # @UndefinedVariable return source_file.read() def _detectEncoding2(source_file): # Detect the encoding. encoding = "ascii" line1 = source_file.readline() if line1.startswith(b'\xef\xbb\xbf'): encoding = "utf-8" else: line1_match = re.search(b"coding[:=]\\s*([-\\w.]+)", line1) if line1_match: encoding = line1_match.group(1) else: line2 = source_file.readline() line2_match = re.search(b"coding[:=]\\s*([-\\w.]+)", line2) if line2_match: encoding = line2_match.group(1) source_file.seek(0) return encoding def _readSourceCodeFromFilename2(source_filename): # Detect the encoding. with open(source_filename, "rU") as source_file: encoding = _detectEncoding2(source_file) source_code = source_file.read() # Try and detect SyntaxError from missing or wrong encodings. if type(source_code) is not unicode and encoding == "ascii": try: _source_code = source_code.decode(encoding) except UnicodeDecodeError as e: lines = source_code.split('\n') so_far = 0 for count, line in enumerate(lines): so_far += len(line) + 1 if so_far > e.args[2]: break else: # Cannot happen, decode error implies non-empty. count = -1 wrong_byte = re.search( "byte 0x([a-f0-9]{2}) in position", str(e) ).group(1) raiseSyntaxError( """\ Non-ASCII character '\\x%s' in file %s on line %d, but no encoding declared; \ see http://python.org/dev/peps/pep-0263/ for details""" % ( wrong_byte, source_filename, count+1, ), SourceCodeReferences.fromFilename(source_filename).atLineNumber( count+1 ), display_line = False ) return source_code def readSourceCodeFromFilename(module_name, source_filename): if python_version < 300: source_code = _readSourceCodeFromFilename2(source_filename) else: source_code = _readSourceCodeFromFilename3(source_filename) # Allow plug-ins to mess with source code. source_code = Plugins.onModuleSourceCode(module_name, source_code) return source_code def checkPythonVersionFromCode(source_code): # There is a lot of cases to consider, pylint: disable=too-many-branches shebang = getShebangFromSource(source_code) if shebang is not None: binary, _args = parseShebang(shebang) if getOS() != "Windows": try: if os.path.samefile(sys.executable, binary): return True except OSError: # Might not exist pass basename = os.path.basename(binary) # Not sure if we should do that. if basename == "python": result = python_version < 300 elif basename == "python3": result = python_version > 300 elif basename == "python2": result = python_version < 300 elif basename == "python2.7": result = python_version < 300 elif basename == "python2.6": result = python_version < 270 elif basename == "python3.2": result = python_version < 330 and python_version >= 300 elif basename == "python3.3": result = python_version < 340 and python_version >= 330 elif basename == "python3.4": result = python_version < 350 and python_version >= 340 elif basename == "python3.5": result = python_version < 360 and python_version >= 350 elif basename == "python3.6": result = python_version < 370 and python_version >= 360 else: result = None if result is False and Options.getIntendedPythonVersion() is None: sys.exit("""\ The program you compiled wants to be run with: %s. Nuitka is currently running with Python version '%s', which seems to not match that. Nuitka cannot guess the Python version of your source code. You therefore might want to specify '--python-version=' to make it clear. """ % (shebang, python_version_str) ) Nuitka-0.5.28.2/nuitka/importing/0000755000372000001440000000000013207540420016763 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/importing/Importing.py0000644000372000001440000004224513207537706021331 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Locating modules and package source on disk. The actual import of a module would already execute code that changes things. Imagine a module that does ``os.system()``, it would be done during compilation. People often connect to databases, and these kind of things, at import time. Therefore CPython exhibits the interfaces in an ``imp`` module in standard library, which one can use those to know ahead of time, what file import would load. For us unfortunately there is nothing in CPython that is easily accessible and gives us this functionality for packages and search paths exactly like CPython does, so we implement here a multi step search process that is compatible. This approach is much safer of course and there is no loss. To determine if it's from the standard library, one can abuse the attribute ``__file__`` of the ``os`` module like it's done in ``isStandardLibraryPath`` of this module. """ from __future__ import print_function import imp import os import sys from logging import warning from nuitka import Options from nuitka.containers.oset import OrderedSet from nuitka.importing import StandardLibrary from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import python_version from nuitka.utils.FileOperations import listDir from .PreloadedPackages import getPreloadedPackagePath, isPreloadedPackagePath from .Whitelisting import isWhiteListedNotExistingModule _debug_module_finding = Options.shallExplainImports() warned_about = set() # Directory where the main script lives. Should attempt to import from there. main_path = None def setMainScriptDirectory(main_dir): """ Initialize the main script directory. We use this as part of the search path for modules. """ # We need to set this from the outside, pylint: disable=global-statement global main_path main_path = main_dir def isPackageDir(dirname): """ Decide if a directory is a package. Before Python3.3 it's required to have a "__init__.py" file, but then it became impossible to decide, and for extra fun, there is also the extra packages provided via "*.pth" file tricks by "site.py" loading. """ return os.path.isdir(dirname) and \ ( python_version >= 330 or os.path.isfile(os.path.join(dirname, "__init__.py")) or isPreloadedPackagePath(dirname) ) def getPackageNameFromFullName(full_name): if '.' in full_name: return full_name[:full_name.rfind('.')] else: return None def getModuleFullNameFromPackageAndName(package, name): if not package: return name else: return package + '.' + name def getExtensionModuleSuffixes(): for suffix, _mode, kind in imp.get_suffixes(): if kind == imp.C_EXTENSION: yield suffix def getModuleNameAndKindFromFilename(module_filename): if os.path.isdir(module_filename): module_name = os.path.basename(module_filename) module_kind = "py" elif module_filename.endswith(".py"): module_name = os.path.basename(module_filename)[:-3] module_kind = "py" else: for suffix, _mode, kind in imp.get_suffixes(): if kind != imp.C_EXTENSION: continue if module_filename.endswith(suffix): module_name = os.path.basename(module_filename)[:-len(suffix)] module_kind = "shlib" break else: module_kind = None module_name = None return module_name, module_kind def isWhiteListedImport(node): module = node.getParentModule() return StandardLibrary.isStandardLibraryPath(module.getFilename()) def warnAbout(importing, module_name, parent_package, level): # This probably should not be dealt with here. if module_name == "": return if not isWhiteListedImport(importing) and \ not isWhiteListedNotExistingModule(module_name): key = module_name, parent_package, level if key not in warned_about: warned_about.add(key) if parent_package is None: full_name = module_name else: full_name = module_name if Plugins.suppressUnknownImportWarning(importing, full_name): return if level == 0: level_desc = "as absolute import" elif level == -1: level_desc = "as relative or absolute import" elif level == 1: level_desc = "%d package level up" % level else: level_desc = "%d package levels up" % level if parent_package is not None: warning( "%s: Cannot find '%s' in package '%s' %s.", importing.getSourceReference().getAsString(), module_name, parent_package, level_desc ) else: warning( "%s: Cannot find '%s' %s.", importing.getSourceReference().getAsString(), module_name, level_desc ) def normalizePackageName(module_name): # The "os.path" is strangely hacked into the "os" module, dispatching per # platform, we either cannot look into it, or we require that we resolve it # here correctly. if module_name == "os.path": module_name = os.path.basename(os.path.__name__) return module_name def findModule(importing, module_name, parent_package, level, warn): """ Find a module with given package name as parent. The package name can be None of course. Level is the same as with "__import__" built-in. Warnings are optional. Returns a triple of package name the module is in, filename of it, which can be a directory for packages, and the location method. """ # We have many branches here, because there are a lot of cases to try. # pylint: disable=too-many-branches if _debug_module_finding: print( "findModule: Enter to search %r in package %r level %s." % ( module_name, parent_package, level ) ) # Do not allow star imports to get here. We just won't find modules with # that name, but it would be wasteful. assert module_name != '*' tried_names = [] if level > 1: # TODO: Should give a warning and return not found if the levels # exceed the package name. if parent_package is not None: parent_package = '.'.join(parent_package.split('.')[:-level+1]) if parent_package == "": parent_package = None else: return None, None, "not-found" # Try relative imports first if we have a parent package. if level != 0 and parent_package is not None: full_name = normalizePackageName(parent_package + '.' + module_name) if full_name.endswith('.'): full_name = full_name[:-1] tried_names.append(full_name) try: module_filename = _findModule( module_name = full_name, ) except ImportError: # For relative import, that is OK, we will still try absolute. pass else: if _debug_module_finding: print( "findModule: Relative imported module '%s' as '%s' in filename '%s':" % ( module_name, full_name, module_filename ) ) return getPackageNameFromFullName(full_name), module_filename, "relative" if level <= 1 and module_name != "": module_name = normalizePackageName(module_name) tried_names.append(module_name) package_name = getPackageNameFromFullName(module_name) # Built-in module names must not be searched any further. if module_name in sys.builtin_module_names: if _debug_module_finding: print( "findModule: Absolute imported module '%s' in as built-in':" % ( module_name, ) ) return package_name, None, "built-in" try: module_filename = _findModule( module_name = module_name, ) except ImportError: # For relative import, that is OK, we will still try absolute. pass else: if _debug_module_finding: print( "findModule: Found absolute imported module '%s' in filename '%s':" % ( module_name, module_filename ) ) return package_name, module_filename, "absolute" if warn: warnAbout( importing = importing, module_name = module_name, parent_package = parent_package, level = level ) return None, None, "not-found" # Some platforms are case insensitive. case_sensitive = not sys.platform.startswith(("win", "cygwin", "darwin")) def _findModuleInPath2(module_name, search_path): """ This is out own module finding low level implementation. Just the full module name and search path are given. This is then tasked to raise "ImportError" or return a path if it finds it, or None, if it is a built-in. """ # We have many branches here, because there are a lot of cases to try. # pylint: disable=too-many-branches # We may have to decide between package and module, therefore build # a list of candidates. candidates = OrderedSet() considered = set() for entry in search_path: # Don't try again, just with an entry of different casing or complete # duplicate. if os.path.normcase(entry) in considered: continue considered.add(os.path.normcase(entry)) package_directory = os.path.join(entry, module_name) # First, check for a package with an init file, that would be the # first choice. if os.path.isdir(package_directory): for suffix, _mode, mtype in imp.get_suffixes(): if mtype == imp.C_EXTENSION: continue package_file_name = "__init__" + suffix file_path = os.path.join(package_directory, package_file_name) if os.path.isfile(file_path): candidates.add( (entry, 1, package_directory) ) break else: if python_version >= 330: candidates.add( (entry, 2, package_directory) ) # Then, check out suffixes of all kinds. for suffix, _mode, _type in imp.get_suffixes(): file_path = os.path.join(entry, module_name + suffix) if os.path.isfile(file_path): candidates.add( (entry, 1, file_path) ) break if _debug_module_finding: print("Candidates", candidates) if candidates: # Ignore lower priority matches from package directories without # "__init__.py" file. min_prio = min(candidate[1] for candidate in candidates) candidates = [ candidate for candidate in candidates if candidate[1] == min_prio ] # On case sensitive systems, no resolution needed. if case_sensitive: return candidates[0][2] else: for candidate in candidates: for fullname, _filename in listDir(candidate[0]): if fullname == candidate[2]: return candidate[2] # Only exact case matches matter, all candidates were ignored, # lets just fall through to raising the import error. # Nothing found. raise ImportError def getPackageSearchPath(package_name): assert main_path is not None if package_name is None: return [os.getcwd(), main_path] + sys.path elif '.' in package_name: parent_package_name, child_package_name = package_name.rsplit('.', 1) result = [] for element in getPackageSearchPath(parent_package_name): package_dir = os.path.join( element, child_package_name ) if isPackageDir(package_dir): result.append(package_dir) # Hack for "uniconverter". TODO: Move this to plug-in decision. This # fails the above test, but at run time should be a package. elif package_name == "uniconvertor.app.modules": result.append(package_dir) return result else: preloaded_path = getPreloadedPackagePath(package_name) if preloaded_path is not None: return preloaded_path def getPackageDirCandidates(element): yield os.path.join(element, package_name), False # Hack for PyWin32. TODO: Move this "__path__" extensions to be # plug-in decisions. if package_name == "win32com": yield os.path.join(element, "win32comext"), True result = [] for element in getPackageSearchPath(None): for package_dir, force_package in getPackageDirCandidates(element): if isPackageDir(package_dir) or force_package: result.append(package_dir) return result def _findModuleInPath(module_name, package_name): if _debug_module_finding: print("_findModuleInPath: Enter", module_name, "in", package_name) # The "site" module must be located based on PYTHONPATH before it was # executed, while we normally search in PYTHONPATH after it was executed, # and on some systems, that fails. if package_name is None and module_name == "site": candidate = os.environ.get("NUITKA_SITE_FILENAME", "") if candidate: return candidate # Free pass for built-in modules, the need not exist. if package_name is None and imp.is_builtin(module_name): return None, module_name search_path = getPackageSearchPath(package_name) if _debug_module_finding: print("_findModuleInPath: Using search path", search_path, "for", package_name) try: module_filename = _findModuleInPath2( module_name = module_name, search_path = search_path ) except SyntaxError: # Warn user, as this is kind of unusual. warning( "%s: Module cannot be imported due to syntax errors.", module_name if package_name is None else package_name + '.' + module_name, ) return None if _debug_module_finding: print("_findModuleInPath: _findModuleInPath2 gave", module_filename) return module_filename module_search_cache = {} def _findModule(module_name): if _debug_module_finding: print( "_findModule: Enter to search '%s'." % ( module_name, ) ) assert not module_name.endswith('.'), module_name key = module_name if key in module_search_cache: result = module_search_cache[key] if _debug_module_finding: print("_findModule: Cached result (see previous call).") if result is ImportError: raise ImportError else: return result try: module_search_cache[key] = _findModule2(module_name) except ImportError: new_module_name = Plugins.considerFailedImportReferrals(module_name) if new_module_name is None: module_search_cache[key] = ImportError raise else: module_search_cache[key] = _findModule(new_module_name) return module_search_cache[key] def _findModule2(module_name): # Need a real module name. assert module_name != "" if '.' in module_name: package_part = module_name[ : module_name.rfind('.') ] module_name = module_name[ module_name.rfind('.') + 1 : ] else: package_part = None preloaded_path = getPreloadedPackagePath(module_name) if preloaded_path is not None: return preloaded_path[0] return _findModuleInPath( module_name = module_name, package_name = package_part ) Nuitka-0.5.28.2/nuitka/importing/PreloadedPackages.py0000644000372000001440000000743213122472300022676 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This module abstracts what site.py is normally doing in .pth files. This tries to extract "namespaces" packages that were manually created and point to package directories, which need no "__init__.py" to count as a package. Nuitka will pretend for those that there be one, but without content. """ import os import sys from logging import warning from nuitka.utils.FileOperations import listDir def getLoadedPackages(): """ Extract packages with no __file__, i.e. they got added manually. They are frequently created with "*.pth" files that then check for the "__init__.py" to exist, and when it doesn't, then they create during the loading of "site.py" an package with "__path__" set. """ for module_name, module in sys.modules.items(): if not hasattr(module, "__path__"): continue if hasattr(module, "__file__"): continue yield module_name, module def detectPreLoadedPackagePaths(): result = {} for package_name, module in getLoadedPackages(): result[package_name] = list(module.__path__) return result preloaded_packages = None def getPreloadedPackagePaths(): # We need to set this from the outside, pylint: disable=global-statement global preloaded_packages if preloaded_packages is None: preloaded_packages = detectPreLoadedPackagePaths() return preloaded_packages def setPreloadedPackagePaths(value): # We need to set this from the outside, pylint: disable=global-statement global preloaded_packages preloaded_packages = value def getPreloadedPackagePath(package_name): return getPreloadedPackagePaths().get(package_name, None) def isPreloadedPackagePath(path): path = os.path.normcase(path) for paths in getPreloadedPackagePaths().values(): for element in paths: if os.path.normcase(element) == path: return True return False def detectPthImportedPackages(): if not hasattr(sys.modules["site"], "getsitepackages"): return () pth_imports = set() for prefix in sys.modules["site"].getsitepackages(): if not os.path.isdir(prefix): continue for path, filename in listDir(prefix): if filename.endswith(".pth"): try: for line in open(path, "rU"): if line.startswith("import "): if ';' in line: line = line[:line.find(';')] for part in line[7:].split(','): pth_imports.add(part.strip()) except OSError: warning("Python installation problem, cannot read file '%s'.") return tuple(sorted(pth_imports)) pth_imported_packages = () def setPthImportedPackages(value): # We need to set this from the outside, pylint: disable=global-statement global pth_imported_packages pth_imported_packages = value def getPthImportedPackages(): return pth_imported_packages Nuitka-0.5.28.2/nuitka/importing/Whitelisting.py0000644000372000001440000002165313207537706022033 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Whitelist modules that are not found, but probably that's acceptable. """ import sys from logging import warning def getModuleWhiteList(): return ( "mac", "nt", "os2", "posix", "_emx_link", "riscos", "ce", "riscospath", "riscosenviron", "Carbon.File", "org.python.core", "_sha", "_sha256", "array", "_sha512", "_md5", "_subprocess", "msvcrt", "cPickle", "marshal", "imp", "sys", "itertools", "cStringIO", "time", "zlib", "thread", "math", "errno", "operator", "signal", "gc", "exceptions", "win32process", "unicodedata", "__builtin__", "fcntl", "_socket", "_ssl", "pwd", "spwd", "_random", "grp", "_io", "_string", "select", "__main__", "_winreg", "_warnings", "_sre", "_functools", "_hashlib", "_collections", "_locale", "_codecs", "_weakref", "_struct", "_dummy_threading", "binascii", "datetime", "_ast", "xxsubtype", "_bytesio", "cmath", "_fileio", "aetypes", "aepack", "MacOS", "cd", "cl", "gdbm", "gl", "GL", "aetools", "_bisect", "_heapq", "_symtable", "syslog", "_datetime", "_elementtree", "_pickle", "_posixsubprocess", "_thread", "atexit", "pyexpat", "_imp", "_sha1", "faulthandler", "_osx_support", "sysconfig", "copyreg", "ipaddress", "reprlib", "win32event", "win32file", # Python-Qt4 does these if missing python3 parts: "PyQt4.uic.port_v3.string_io", "PyQt4.uic.port_v3.load_plugin", "PyQt4.uic.port_v3.ascii_upper", "PyQt4.uic.port_v3.proxy_base", "PyQt4.uic.port_v3.as_string", # CPython3 does these: "builtins", "UserDict", "os.path", "StringIO", # "test_array", "_testcapi", # test_applesingle.py "applesingle", # test_buffer.py "_testbuffer", # test_bsddb.py "bsddb.test", # test_collections.py "collections.abc", # test_compile.py "__package__.module", "__mangled_mod", "__package__", # test_ctypes "ctypes.test", # test_dbm.py "dbm.dumb", # test_dbm_ndbm.py "dbm.ndbm", # test_distutils.py "distutils.tests", "distutils.mwerkscompiler", # test_docxmlrpc.py "xmlrpc.server", # test_emails.py "email.test.test_email", "email.test.test_email_renamed", "email.test.test_email_codecs", # test_email_codecs.py "email.test", # test_enum.py "enum", # test_file.py "_pyio", # test_frozen.py "__hello__", "__phello__", "__phello__.spam", "__phello__.foo", # test_fork1.py "fake test module", # test_html.py "html", "html.entities", # test_http_cookiejar.py "urllib.request", "http", # test_imp.py "importlib.test.import_", "pep3147.foo", "pep3147", # test_import.py "RAnDoM", "infinite_reload", "test_trailing_slash", "nonexistent_xyzzy", "_parent_foo.bar", "_parent_foo", "test_unc_path", # test_importhooks.py "hooktestmodule", "hooktestpackage", "hooktestpackage.sub", "reloadmodule", "hooktestpackage.sub.subber", "hooktestpackage.oldabs", "hooktestpackage.newrel", "hooktestpackage.sub.subber.subest", "hooktestpackage.futrel", "sub", "hooktestpackage.newabs", # test_imporlib.py" "importlib.test.__main__", "importlib", # test_inspect.py "inspect_fodder3", "test.test_import", # test_imageop.py "imgfile", # test_json.py "json.tests", # test_lib2to3.py "lib2to3.tests", # test_logging.py "win32evtlog", "win32evtlogutil", "pywintypes", # test_lzma.py "lzma", # test_macostools.py "macostools", # test_namespace_pkgs.py "foo.one", "foo.two", "parent.child.one", "parent.child.two", "parent.child.three", "bar.two", "a_test", "parent.child", "parent", "bar", # test_new.py "Spam", # test_ossaudiodev.py "ossaudiodev", # test_pathlib.py "pathlib", # test_platform.py "gestalt", # test_pkg.py "t1", "t2", "t2.sub", "t2.sub.subsub", "t3.sub.subsub", "t5", "t6", "t7", "t7.sub", "t7.sub.subsub", "t8", "t3.sub", "t3", # test_pkgutil.py "foo", "foo.bar", "foo.baz", "zipimport", "pkg", "pkg.subpkg", "pkg.subpkg.c", "pkg.subpkg.d", # test_urllib.py "urllib.parse", # test_urllib_response.py "urllib.response", # test_repr.py """areallylongpackageandmodulenametotestreprtruncation.\ areallylongpackageandmodulenametotestreprtruncation""", "areallylongpackageandmodulenametotestreprtruncation", # test_robotparser.py "urllib.error", "urllib.robotparser", # test_runpy.py "test.script_helper", # test_secrets.py "secrets", # test_selectors.py "selectors", # test_statistics.py "statistics", # test_shelve.py "test.test_dbm", # test_strftime.py "java", # test_strop.py "strop", # test_sqlite3.py "sqlite3.test", # test_sundry.py "distutils.emxccompiler", "os2emxpath", # test_tcl.py "tkinter", # test_tk.py "runtktests", "tkinter.test", "tkinter.test.support", # test_tools.py "analyze_dxp", "test_unparse", "importlib.machinery", # test_traceback.py "test_bug737473", # test_tracemalloc "tracemalloc", # test_typing.py "mock", "typing.io", "typing.re", # test_unittest.py "unittest.test", # test_wsgiref.py "test.test_httpservers", # test_xml_etree.py "xml.parsers.expat.errors", # test_xmlrpc.py "xmlrpc.client", # test_zipimport_support.py "test_zipped_doctest", "zip_pkg", # test/test_zipimport_support.py "test.test_cmd_line_script", # Python3: modules that no longer exist "commands", "dummy_thread", "_dummy_thread", "httplib", "Queue", "sets", # Python2: modules that don't yet exit "http.client", "queue", "winreg", # Very old modules with older names "simplejson", "sets", # Standalone mode "site" import flexibilities "sitecustomize", "usercustomize", "apport_python_hook", "_frozen_importlib", # Standard library stuff that is optional "comtypes.server.inprocserver", "_tkinter", "_scproxy", "EasyDialogs", "SOCKS", "rourl2path", "_winapi", "win32api", "win32con", "_gestalt", "java.lang", "vms_lib", "ic", "readline", "termios", "_sysconfigdata", "al", "AL", "sunaudiodev", "SUNAUDIODEV", "Audio_mac", "nis", "test.test_MimeWriter", "dos", "win32pipe", "Carbon", "Carbon.Files", "sgi", "ctypes.macholib.dyld", "bsddb3", "_pybsddb", "_xmlrpclib", "netbios", "win32wnet", "email.Parser", "elementree.cElementTree", "elementree.ElementTree", "_gbdm", "resource", "crypt", "bz2", "dbm", "mmap", "Mailman", # Mercurial test "statprof", "email.Generator", "email.Utils", # setuptools does a lot of speculative stuff "wincertstore", "setuptools_svn", # reportlab does use this if present only and warns about itself. "pyfribidi2", "macfs", # psutils "_psutil_windows", # nose "unittest2", "IronPython", "clr", "compiler.consts", "new", # pkg_resources "pkg_resources.extern", # appdirs "com.sun", "win32com", # six "six.moves", ) def isWhiteListedNotExistingModule(module_name): result = False for white_listed in getModuleWhiteList(): if module_name == white_listed or module_name.startswith(white_listed + '.'): result = True break if not result and module_name in sys.builtin_module_names: warning("""\ Your CPython version has a built-in module '%s', that is not whitelisted please report this to http://bugs.nuitka.net.""", module_name) return result Nuitka-0.5.28.2/nuitka/importing/Recursion.py0000644000372000001440000003375013207537706021333 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Recursion into other modules. """ import fnmatch import glob import marshal import os import sys from logging import debug, info, warning from nuitka import ModuleRegistry, Options from nuitka.importing import ImportCache, Importing, StandardLibrary from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import python_version from nuitka.tree.SourceReading import readSourceCodeFromFilename from nuitka.utils.FileOperations import listDir, relpath def logRecursion(*args): if Options.isShowInclusion(): info(*args) else: debug(*args) def matchesModuleNameToPatterns(module_name, patterns): for pattern in patterns: if module_name == pattern: return True, "is exact match of %r" % pattern elif module_name.startswith(pattern): return True, "is package content of %r" % pattern elif fnmatch.fnmatch(module_name, pattern): return True, "matches pattern %r" % pattern elif fnmatch.fnmatch(module_name, pattern + ".*"): return True, "is package content of match to pattern %r" % pattern return False, None def _recurseTo(module_package, module_filename, module_relpath, module_kind, reason): from nuitka.tree import Building from nuitka.nodes.ModuleNodes import makeUncompiledPythonModule module, source_ref, source_filename = Building.decideModuleTree( filename = module_filename, package = module_package, is_top = False, is_main = False, is_shlib = module_kind == "shlib" ) # Check if the module name is known. In order to avoid duplicates, # learn the new filename, and continue build if its not. if not ImportCache.isImportedModuleByName(module.getFullName()): logRecursion( "Recurse to import '%s' from '%s'. (%s)", module.getFullName(), module_relpath, reason ) if module_kind == "py" and source_filename is not None: try: source_code = readSourceCodeFromFilename( module_name = module.getFullName(), source_filename = source_filename ) Building.createModuleTree( module = module, source_ref = source_ref, source_code = source_code, is_main = False ) except (SyntaxError, IndentationError) as e: if module_filename not in Importing.warned_about: Importing.warned_about.add(module_filename) warning( """\ Cannot recurse to import module '%s' (%s) because of '%s'""", module_relpath, module_filename, e.__class__.__name__ ) return None, False except Building.CodeTooComplexCode: if module_filename not in Importing.warned_about: Importing.warned_about.add(module_filename) warning( """\ Cannot recurse to import module '%s' (%s) because code is too complex.""", module_relpath, module_filename, ) if Options.isStandaloneMode(): module = makeUncompiledPythonModule( module_name = module.getFullName(), filename = module_filename, bytecode = marshal.dumps( compile( source_code, module_filename, "exec", dont_inherit = True ) ), is_package = module.isCompiledPythonPackage(), user_provided = True, technical = False ) ModuleRegistry.addUncompiledModule(module) return None, False ImportCache.addImportedModule(module) is_added = True else: module = ImportCache.getImportedModuleByName( module.getFullName() ) is_added = False return module, is_added def recurseTo(module_package, module_filename, module_relpath, module_kind, reason): if ImportCache.isImportedModuleByPath(module_relpath): module = ImportCache.getImportedModuleByPath(module_relpath) if module.getCompileTimeFilename().endswith("__init__.py") and \ module.getPackage() != module_package: module = None else: module = None if module is None: return _recurseTo( module_package = module_package, module_filename = module_filename, module_relpath = module_relpath, module_kind = module_kind, reason = reason ) else: return module, False def decideRecursion(module_filename, module_name, module_package, module_kind, extra_recursion = False): # Many branches, which make decisions immediately, by returning # pylint: disable=too-many-return-statements Plugins.onModuleEncounter( module_filename, module_name, module_package, module_kind ) if module_kind == "shlib": if Options.isStandaloneMode(): return True, "Shared library for inclusion." else: return False, "Shared library cannot be inspected." if module_package is None: full_name = module_name else: full_name = module_package + '.' + module_name no_case, reason = matchesModuleNameToPatterns( module_name = full_name, patterns = Options.getShallFollowInNoCase() ) if no_case: return ( False, "Module %s instructed by user to not recurse to." % reason ) any_case, reason = matchesModuleNameToPatterns( module_name = full_name, patterns = Options.getShallFollowModules() ) if any_case: return ( True, "Module %s instructed by user to recurse to." % reason ) if Options.shallFollowNoImports(): return ( False, "Requested to not recurse at all." ) if StandardLibrary.isStandardLibraryPath(module_filename): return ( Options.shallFollowStandardLibrary(), "Requested to %srecurse to standard library." % ( "" if Options.shallFollowStandardLibrary() else "not " ) ) if Options.shallFollowAllImports(): return ( True, "Requested to recurse to all non-standard library modules." ) # Means, we were not given instructions how to handle things. if extra_recursion: return ( True, "Lives in plug-in directory." ) else: return ( None, "Default behavior, not recursing without request." ) def considerFilename(module_filename): module_filename = os.path.normpath(module_filename) if os.path.isdir(module_filename): module_filename = os.path.abspath(module_filename) module_name = os.path.basename(module_filename) module_relpath = relpath(module_filename) return module_filename, module_relpath, module_name elif module_filename.endswith(".py"): module_name = os.path.basename(module_filename)[:-3] module_relpath = relpath(module_filename) return module_filename, module_relpath, module_name elif module_filename.endswith(".pyw"): module_name = os.path.basename(module_filename)[:-4] module_relpath = relpath(module_filename) return module_filename, module_relpath, module_name else: return None def isSameModulePath(path1, path2): if os.path.basename(path1) == "__init__.py": path1 = os.path.dirname(path1) if os.path.basename(path2) == "__init__.py": path2 = os.path.dirname(path2) return os.path.abspath(path1) == os.path.abspath(path2) def _checkPluginPath(plugin_filename, module_package): # Many branches, for the decision is very complex, pylint: disable=too-many-branches debug( "Checking detail plug-in path '%s' '%s':", plugin_filename, module_package ) module_name, module_kind = Importing.getModuleNameAndKindFromFilename(plugin_filename) if module_kind is not None: decision, reason = decideRecursion( module_filename = plugin_filename, module_name = module_name, module_package = module_package, module_kind = module_kind, extra_recursion = True ) if decision: module_relpath = relpath(plugin_filename) module, is_added = recurseTo( module_filename = plugin_filename, module_relpath = module_relpath, module_package = module_package, module_kind = "py", reason = reason ) if module: if not is_added: warning( "Recursed to %s '%s' at '%s' twice.", "package" if module.isCompiledPythonPackage() else "module", module.getName(), plugin_filename ) if not isSameModulePath(module.getFilename(), plugin_filename): warning( "Duplicate '%s' of '%s' ignored .", plugin_filename, module.getFilename() ) return debug( "Recursed to %s %s %s", module.getName(), module.getPackage(), module ) ImportCache.addImportedModule(module) if module.isCompiledPythonPackage(): package_filename = module.getFilename() if os.path.isdir(package_filename): # Must be a namespace package. assert python_version >= 330 package_dir = package_filename # Only include it, if it contains actual modules, which will # recurse to this one and find it again. else: package_dir = os.path.dirname(package_filename) # Real packages will always be included. ModuleRegistry.addRootModule(module) debug( "Package directory %s", package_dir ) for sub_path, sub_filename in listDir(package_dir): if sub_filename in ("__init__.py", "__pycache__"): continue assert sub_path != plugin_filename if Importing.isPackageDir(sub_path) or \ sub_path.endswith(".py"): _checkPluginPath(sub_path, module.getFullName()) elif module.isCompiledPythonModule(): ModuleRegistry.addRootModule(module) else: warning("Failed to include module from '%s'.", plugin_filename) def checkPluginPath(plugin_filename, module_package): plugin_filename = os.path.normpath(plugin_filename) debug( "Checking top level plug-in path %s %s", plugin_filename, module_package ) plugin_info = considerFilename( module_filename = plugin_filename ) if plugin_info is not None: # File or package makes a difference, handle that if os.path.isfile(plugin_info[0]) or \ Importing.isPackageDir(plugin_info[0]): _checkPluginPath(plugin_filename, module_package) elif os.path.isdir(plugin_info[0]): for sub_path, sub_filename in listDir(plugin_info[0]): assert sub_filename != "__init__.py" if Importing.isPackageDir(sub_path) or \ sub_path.endswith(".py"): _checkPluginPath(sub_path, None) else: warning("Failed to include module from '%s'.", plugin_info[0]) else: warning("Failed to recurse to directory '%s'.", plugin_filename) def checkPluginFilenamePattern(pattern): debug( "Checking plug-in pattern '%s':", pattern, ) if os.path.isdir(pattern): sys.exit("Error, pattern cannot be a directory name.") found = False for filename in glob.iglob(pattern): if filename.endswith(".pyc"): continue if not os.path.isfile(filename): continue found = True _checkPluginPath(filename, None) if not found: warning("Didn't match any files against pattern '%s'." % pattern) Nuitka-0.5.28.2/nuitka/importing/ImportCache.py0000644000372000001440000000541613122472300021536 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Import cache. This is not about caching the search of modules in the file system, but about maintaining a cache of module trees built. It can happen that modules become unused, and then dropped from active modules, and then later active again, via another import, and in this case, we should not start anew, but reuse what we already found out about it. """ import os from nuitka.plugins.Plugins import Plugins from nuitka.utils.FileOperations import relpath imported_modules = {} imported_by_name = {} def addImportedModule(imported_module): module_filename = relpath(imported_module.getFilename()) if os.path.basename(module_filename) == "__init__.py": module_filename = os.path.dirname(module_filename) key = ( module_filename, imported_module.getFullName() ) if key in imported_modules: assert imported_module is imported_modules[key], key else: Plugins.onModuleDiscovered(imported_module) imported_modules[key] = imported_module imported_by_name[imported_module.getFullName()] = imported_module # We don't expect that to happen. assert not imported_module.isMainModule() def isImportedModuleByPath(module_relpath): for key in imported_modules: if key[0] == module_relpath: return True return False def isImportedModuleByName(full_name): return full_name in imported_by_name def getImportedModuleByName(full_name): return imported_by_name[full_name] def getImportedModuleByPath(module_relpath): for key in imported_modules: if key[0] == module_relpath: return imported_modules[key] raise KeyError(module_relpath) def replaceImportedModule(old, new): for key, value in imported_by_name.items(): if value == old: imported_by_name[key] = new break else: assert False for key, value in imported_modules.items(): if value == old: imported_modules[key] = new break else: assert False Nuitka-0.5.28.2/nuitka/importing/__init__.py0000644000372000001440000000150113112214770021072 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/nuitka/importing/StandardLibrary.py0000644000372000001440000001147013207537242022435 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Access to standard library distinction. For code to be in the standard library means that it's not written by the user for sure. We treat code differently based on that information, by e.g. including as byte code. To determine if a module from the standard library, we can abuse the attribute "__file__" of the "os" module like it's done in "isStandardLibraryPath" of this module. """ import os from nuitka.utils.Utils import getOS def getStandardLibraryPaths(): """ Get the standard library paths. """ # Using the function object to cache its result, avoiding global variable # usage. if not hasattr(getStandardLibraryPaths, "result"): os_filename = os.__file__ if os_filename.endswith(".pyc"): os_filename = os_filename[:-1] os_path = os.path.normcase( os.path.dirname(os_filename) ) stdlib_paths = set([os_path]) # Happens for virtualenv situation, some modules will come from the link # this points to. if os.path.islink(os_filename): os_filename = os.readlink(os_filename) # @UndefinedVariable stdlib_paths.add( os.path.normcase( os.path.dirname(os_filename) ) ) # Another possibility is "orig-prefix.txt" file near the os.py, which # points to the original install. orig_prefix_filename = os.path.join(os_path, "orig-prefix.txt") if os.path.isfile(orig_prefix_filename): # Scan upwards, until we find a "bin" folder, with "activate" to # locate the structural path to be added. We do not know for sure # if there is a sub-directory under "lib" to use or not. So we try # to detect it. search = os_path lib_part = "" while os.path.splitdrive(search)[1] not in (os.path.sep, ""): if os.path.isfile(os.path.join(search,"bin/activate")) or \ os.path.isfile(os.path.join(search,"scripts/activate")): break lib_part = os.path.join(os.path.basename(search), lib_part) search = os.path.dirname(search) assert search and lib_part stdlib_paths.add( os.path.normcase( os.path.join( open(orig_prefix_filename).read(), lib_part, ) ) ) # And yet another possibility, for MacOS Homebrew created virtualenv # at least is a link ".Python", which points to the original install. python_link_filename = os.path.join(os_path, "..", ".Python") if os.path.islink(python_link_filename): stdlib_paths.add( os.path.normcase( os.path.join( os.readlink(python_link_filename), "lib" ) ) ) for stdlib_path in set(stdlib_paths): candidate = os.path.join(stdlib_path, "lib-tk") if os.path.isdir(candidate): stdlib_paths.add(candidate) if getOS() == "Windows": import _ctypes stdlib_paths.add( os.path.dirname(_ctypes.__file__) ) getStandardLibraryPaths.result = [ os.path.normcase(stdlib_path) for stdlib_path in stdlib_paths ] return getStandardLibraryPaths.result def isStandardLibraryPath(path): """ Check if a path is in the standard library. """ path = os.path.normcase(path) # In virtualenv, the "site.py" lives in a place that suggests it is not in # standard library, although it is. if os.path.basename(path) == "site.py": return True # These never are in standard library paths. if "dist-packages" in path or "site-packages" in path: return False for candidate in getStandardLibraryPaths(): if path.startswith(candidate): return True return False Nuitka-0.5.28.2/nuitka/MainControl.py0000644000372000001440000007033213207540035017561 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This is the main actions of Nuitka. This can do all the steps to translate one module to a target language using the Python C/API, to compile it to either an executable or an extension module, potentially with bytecode included and used library copied into a distribution folder. """ import os import shutil import subprocess import sys from logging import info, warning from nuitka.finalizations.FinalizeMarkups import getImportedNames from nuitka.importing import Importing, Recursion from nuitka.Options import getPythonFlags from nuitka.plugins.Plugins import Plugins from nuitka.PythonVersions import ( getSupportedPythonVersions, isUninstalledPython, python_version, python_version_str ) from nuitka.tree import SyntaxErrors from nuitka.utils import Execution, InstanceCounters, MemoryUsage, Utils from nuitka.utils.AppDirs import getCacheDir from nuitka.utils.FileOperations import ( deleteFile, hasFilenameExtension, listDir, makePath, removeDirectory ) from . import ModuleRegistry, Options, TreeXML from .build import SconsInterface from .codegen import CodeGeneration, ConstantCodes from .finalizations import Finalization from .freezer.BytecodeModuleFreezer import generateBytecodeFrozenCode from .freezer.Standalone import copyUsedDLLs, detectEarlyImports from .optimizations import Optimization from .tree import Building def createNodeTree(filename): """ Create a node tree. Turn that source code into a node tree structure. If recursion into imported modules is available, more trees will be available during optimization, or immediately through recursed directory paths. """ # First, build the raw node tree from the source code. main_module = Building.buildModuleTree( filename = filename, package = None, is_top = True, is_main = not Options.shallMakeModule() ) ModuleRegistry.addRootModule(main_module) # First remove old object files and old generated files, old binary or # module, and standalone mode program directory if any, they can only do # harm. source_dir = getSourceDirectoryPath(main_module) if not Options.shallOnlyExecCCompilerCall(): cleanSourceDirectory(source_dir) # Prepare the ".dist" directory, throwing away what was there before. if Options.isStandaloneMode(): standalone_dir = getStandaloneDirectoryPath(main_module) removeDirectory( path = standalone_dir, ignore_errors = True ) makePath(standalone_dir) deleteFile( path = getResultFullpath(main_module), must_exist = False ) # Second, do it for the directories given. for plugin_filename in Options.getShallFollowExtra(): Recursion.checkPluginPath( plugin_filename = plugin_filename, module_package = None ) for pattern in Options.getShallFollowExtraFilePatterns(): Recursion.checkPluginFilenamePattern( pattern = pattern ) # Then optimize the tree and potentially recursed modules. Optimization.optimize() if Options.isExperimental("check_xml_persistence"): for module in ModuleRegistry.getRootModules(): if module.isMainModule(): return module assert False else: # Main module might change behind our back, look it up again. return main_module def dumpTreeXML(tree): xml_root = tree.asXml() TreeXML.dump(xml_root) def displayTree(tree): # pragma: no cover # Import only locally so the Qt4 dependency doesn't normally come into play # when it's not strictly needed. from .gui import TreeDisplay TreeDisplay.displayTreeInspector(tree) def getTreeFilenameWithSuffix(tree, suffix): return tree.getOutputFilename() + suffix def getSourceDirectoryPath(main_module): assert main_module.isCompiledPythonModule() return Options.getOutputPath( path = os.path.basename( getTreeFilenameWithSuffix(main_module, ".build") ) ) def getStandaloneDirectoryPath(main_module): return Options.getOutputPath( path = os.path.basename( getTreeFilenameWithSuffix(main_module, ".dist") ) ) def getResultBasepath(main_module): assert main_module.isCompiledPythonModule() if Options.isStandaloneMode(): return os.path.join( getStandaloneDirectoryPath(main_module), os.path.basename( getTreeFilenameWithSuffix(main_module, "") ) ) else: return Options.getOutputPath( path = os.path.basename( getTreeFilenameWithSuffix(main_module, "") ) ) def getResultFullpath(main_module): result = getResultBasepath(main_module) if Options.shallMakeModule(): result += Utils.getSharedLibrarySuffix() else: result += ".exe" return result def cleanSourceDirectory(source_dir): if os.path.isdir(source_dir): for path, _filename in listDir(source_dir): if hasFilenameExtension( path = path, extensions = (".c", ".h", ".o", ".os", ".obj", ".bin", ".res", ".rc", ".S", ".cpp", ".manifest") ): deleteFile(path, must_exist = True) else: makePath(source_dir) def pickSourceFilenames(source_dir, modules): collision_filenames = set() seen_filenames = set() # Our output. module_filenames = {} def getFilenames(module): base_filename = os.path.join( source_dir, "module." + module.getFullName() if not module.isInternalModule() else module.getFullName() ) # Note: Could detect if the file system is cases sensitive in source_dir # or not, but that's probably not worth the effort. False positives do # no harm at all. collision_filename = os.path.normcase(base_filename) return base_filename, collision_filename # First pass, check for collisions. for module in modules: if module.isPythonShlibModule(): continue base_filename = base_filename, collision_filename = getFilenames(module) if collision_filename in seen_filenames: collision_filenames.add(collision_filename) seen_filenames.add(collision_filename) # Count up for colliding filenames. collision_counts = {} # Second pass, this time sorted, so we get deterministic results. We will # apply an @1/@2 to disambiguate the filenames. for module in sorted(modules, key = lambda x : x.getFullName()): if module.isPythonShlibModule(): continue base_filename = base_filename, collision_filename = getFilenames(module) if collision_filename in collision_filenames: collision_counts[ collision_filename ] = \ collision_counts.get(collision_filename, 0) + 1 hash_suffix = "@%d" % collision_counts[ collision_filename ] else: hash_suffix = "" base_filename += hash_suffix module_filenames[module] = base_filename + ".c" return module_filenames standalone_entry_points = [] def makeSourceDirectory(main_module): """ Get the full list of modules imported, create code for all of them. """ # We deal with a lot of details here, but rather one by one, and split makes # no sense, pylint: disable=too-many-branches,too-many-locals,too-many-statements assert main_module.isCompiledPythonModule() # The global context used to generate code. global_context = CodeGeneration.makeGlobalContext() # assert main_module in ModuleRegistry.getDoneModules() # We might have chosen to include it as bytecode, and only compiled it for # fun, and to find its imports. In this case, now we just can drop it. Or # a module may shadow a frozen module, but be a different one, then we can # drop the frozen one. # TODO: This really should be done when the compiled module comes into # existence. for module in ModuleRegistry.getDoneUserModules(): if module.isCompiledPythonModule(): uncompiled_module = ModuleRegistry.getUncompiledModule( module_name = module.getFullName(), module_filename = module.getCompileTimeFilename() ) if uncompiled_module is not None: # We now need to decide which one to keep, compiled or uncompiled # module. Some uncompiled modules may have been asked by the user # or technically required. By default, frozen code if it exists # is preferred, as it will be from standalone mode adding it. if uncompiled_module.isUserProvided() or \ uncompiled_module.isTechnical(): ModuleRegistry.removeDoneModule(module) else: ModuleRegistry.removeUncompiledModule(uncompiled_module) # Lets check if the recurse-to modules are actually present, and warn the # user if one of those was not found. for any_case_module in Options.getShallFollowModules(): if '*' in any_case_module or '{' in any_case_module: continue for module in ModuleRegistry.getDoneUserModules(): if module.getFullName() == any_case_module: break else: warning( "Didn't recurse to '%s', apparently not used." % \ any_case_module ) # Prepare code generation, i.e. execute finalization for it. for module in ModuleRegistry.getDoneModules(): if module.isCompiledPythonModule(): Finalization.prepareCodeGeneration(module) # Pick filenames. source_dir = getSourceDirectoryPath(main_module) module_filenames = pickSourceFilenames( source_dir = source_dir, modules = ModuleRegistry.getDoneModules() ) # First pass, generate code and use constants doing so, but prepare the # final code generation only, because constants code will be added at the # end only. prepared_modules = {} for module in ModuleRegistry.getDoneModules(): if module.isCompiledPythonModule(): c_filename = module_filenames[module] prepared_modules[c_filename] = CodeGeneration.prepareModuleCode( global_context = global_context, module = module, module_name = module.getFullName(), ) # Main code constants need to be allocated already too. if module is main_module and not Options.shallMakeModule(): prepared_modules[c_filename][1].getConstantCode(0) # Second pass, generate the actual module code into the files. for module in ModuleRegistry.getDoneModules(): if module.isCompiledPythonModule(): c_filename = module_filenames[module] template_values, module_context = prepared_modules[c_filename] source_code = CodeGeneration.generateModuleCode( module_context = module_context, template_values = template_values ) writeSourceCode( filename = c_filename, source_code = source_code ) if Options.isShowInclusion(): info("Included compiled module '%s'." % module.getFullName()) elif module.isPythonShlibModule(): target_filename = os.path.join( getStandaloneDirectoryPath(main_module), *module.getFullName().split('.') ) if Utils.getOS() == "Windows": target_filename += ".pyd" else: target_filename += ".so" target_dir = os.path.dirname(target_filename) if not os.path.isdir(target_dir): makePath(target_dir) shutil.copy( module.getFilename(), target_filename ) standalone_entry_points.append( ( module.getFilename(), target_filename, module.getPackage() ) ) elif module.isUncompiledPythonModule(): pass else: assert False, module writeSourceCode( filename = os.path.join( source_dir, "__constants.c" ), source_code = ConstantCodes.getConstantsDefinitionCode( context = global_context ) ) helper_decl_code, helper_impl_code = CodeGeneration.generateHelpersCode( ModuleRegistry.getDoneUserModules() ) writeSourceCode( filename = os.path.join( source_dir, "__helpers.h" ), source_code = helper_decl_code ) writeSourceCode( filename = os.path.join( source_dir, "__helpers.c" ), source_code = helper_impl_code ) def runScons(main_module, quiet): # Scons gets transported many details, that we express as variables, and # have checks for them, leading to many branches, pylint: disable=too-many-branches if hasattr(sys, "abiflags"): abiflags = sys.abiflags if Options.isPythonDebug() or \ hasattr(sys, "getobjects"): if not abiflags.startswith('d'): abiflags = 'd' + abiflags else: abiflags = None def asBoolStr(value): return "true" if value else "false" options = { "name" : os.path.basename( getTreeFilenameWithSuffix(main_module, "") ), "result_name" : getResultBasepath(main_module), "source_dir" : getSourceDirectoryPath(main_module), "debug_mode" : asBoolStr(Options.isDebug()), "python_debug" : asBoolStr(Options.isPythonDebug()), "unstripped_mode" : asBoolStr(Options.isUnstripped()), "module_mode" : asBoolStr(Options.shallMakeModule()), "full_compat" : asBoolStr(Options.isFullCompat()), "experimental" : ','.join(Options.getExperimentalIndications()), "trace_mode" : asBoolStr(Options.shallTraceExecution()), "python_version" : python_version_str, "target_arch" : Utils.getArchitecture(), "python_prefix" : sys.prefix, "nuitka_src" : SconsInterface.getSconsDataPath(), "nuitka_cache" : getCacheDir(), "module_count" : "%d" % ( 1 + \ len(ModuleRegistry.getDoneUserModules()) + \ len(ModuleRegistry.getUncompiledNonTechnicalModules()) ) } # Ask Scons to cache on Windows, except where the directory is thrown # away. On non-Windows you can should use ccache instead. if not Options.isRemoveBuildDir() and Utils.getOS() == "Windows": options["cache_mode"] = "true" if Options.isLto(): options["lto_mode"] = "true" if Options.shallDisableConsoleWindow(): options["win_disable_console"] = "true" if Options.isStandaloneMode(): options["standalone_mode"] = "true" if not Options.isStandaloneMode() and \ not Options.shallMakeModule() and \ isUninstalledPython(): options["uninstalled_python"] = "true" if ModuleRegistry.getUncompiledTechnicalModules(): options["frozen_modules"] = str( len(ModuleRegistry.getUncompiledTechnicalModules()) ) if Options.isShowScons(): options["show_scons"] = "true" if Options.isMingw(): options["mingw_mode"] = "true" if Options.getMsvcVersion(): msvc_version = Options.getMsvcVersion() msvc_version = msvc_version.replace("exp", "Exp") if '.' not in msvc_version: msvc_version += ".0" options["msvc_version"] = msvc_version if Options.isClang(): options["clang_mode"] = "true" if Options.getIconPath(): options["icon_path"] = Options.getIconPath() if Options.isProfile(): options["profile_mode"] = "true" if "no_warnings" in getPythonFlags(): options["no_python_warnings"] = "true" if python_version < 300 and sys.flags.py3k_warning: options["python_sysflag_py3k_warning"] = "true" if python_version < 300 and (sys.flags.division_warning or sys.flags.py3k_warning): options["python_sysflag_division_warning"] = "true" if sys.flags.bytes_warning: options["python_sysflag_bytes_warning"] = "true" if int(os.environ.get("NUITKA_SITE_FLAG", "no_site" in Options.getPythonFlags())): options["python_sysflag_no_site"] = "true" if "trace_imports" in Options.getPythonFlags(): options["python_sysflag_verbose"] = "true" if python_version < 300 and sys.flags.unicode: options["python_sysflag_unicode"] = "true" if abiflags: options["abiflags"] = abiflags return SconsInterface.runScons(options, quiet), options def writeSourceCode(filename, source_code): # Prevent accidental overwriting. When this happens the collision detection # or something else has failed. assert not os.path.isfile(filename), filename if python_version >= 300: with open(filename, "wb") as output_file: output_file.write(source_code.encode("latin1")) else: with open(filename, 'w') as output_file: output_file.write(source_code) def writeBinaryData(filename, binary_data): # Prevent accidental overwriting. When this happens the collision detection # or something else has failed. assert not os.path.isfile(filename), filename assert type(binary_data) is bytes with open(filename, "wb") as output_file: output_file.write(binary_data) def callExecPython(args, clean_path, add_path): old_python_path = os.environ.get("PYTHONPATH", None) if clean_path and old_python_path is not None: os.environ["PYTHONPATH"] = "" if add_path: if "PYTHONPATH" in os.environ: os.environ["PYTHONPATH"] += ':' + Options.getOutputDir() else: os.environ["PYTHONPATH"] = Options.getOutputDir() # We better flush these, "os.execl" won't do it anymore. sys.stdout.flush() sys.stderr.flush() # Add the main arguments, previous separated. args += Options.getPositionalArgs()[1:] + Options.getMainArgs() Execution.callExec(args) def executeMain(binary_filename, clean_path): args = (binary_filename, binary_filename) if Options.shallRunInDebugger(): gdb_path = Execution.getExecutablePath("gdb") if gdb_path is None: sys.exit("Error, no 'gdb' binary found in path.") args = (gdb_path, "gdb", "-ex=run", "-ex=where", "--args", binary_filename) callExecPython( clean_path = clean_path, add_path = False, args = args ) def executeModule(tree, clean_path): python_command = "__import__('%s')" % tree.getName() args = ( sys.executable, "python", "-c", python_command, ) callExecPython( clean_path = clean_path, add_path = True, args = args ) def compileTree(main_module): source_dir = getSourceDirectoryPath(main_module) if not Options.shallOnlyExecCCompilerCall(): # Now build the target language code for the whole tree. makeSourceDirectory( main_module = main_module ) frozen_code = generateBytecodeFrozenCode() if frozen_code is not None: writeSourceCode( filename = os.path.join( source_dir, "__frozen.c" ), source_code = frozen_code ) writeBinaryData( filename = os.path.join( source_dir, "__constants.bin" ), binary_data = ConstantCodes.stream_data.getBytes() ) else: source_dir = getSourceDirectoryPath(main_module) if not os.path.isfile(os.path.join(source_dir, "__helpers.h")): sys.exit("Error, no previous build directory exists.") if Options.isShowProgress() or Options.isShowMemory(): info( "Total memory usage before running scons: {memory}:".format( memory = MemoryUsage.getHumanReadableProcessMemoryUsage() ) ) if Options.isShowMemory(): InstanceCounters.printStats() if Options.shallNotDoExecCCompilerCall(): return True, {} # Run the Scons to build things. result, options = runScons( main_module = main_module, quiet = not Options.isShowScons() ) return result, options def handleSyntaxError(e): # Syntax or indentation errors, output them to the user and abort. If # we are not in full compat, and user has not specified the Python # versions he wants, tell him about the potential version problem. error_message = SyntaxErrors.formatOutput(e) if not Options.isFullCompat() and \ Options.getIntendedPythonVersion() is None: if python_version < 300: suggested_python_version_str = getSupportedPythonVersions()[-1] else: suggested_python_version_str = "2.7" error_message += """ Nuitka is very syntax compatible with standard Python. It is currently running with Python version '%s', you might want to specify more clearly with the use of e.g. '--python-version=%s' option, if that's not the one expected. """ % (python_version_str, suggested_python_version_str) sys.exit(error_message) data_files = [] def main(): """ Main program flow of Nuitka At this point, options will be parsed already, Nuitka will be executing in the desired version of Python with desired flags, and we just get to execute the task assigned. We might be asked to only re-compile generated C, dump only an XML representation of the internal node tree after optimization, etc. """ # Main has to fulfill many options, leading to many branches and statements # to deal with them. pylint: disable=too-many-branches filename = Options.getPositionalArgs()[0] # Inform the importing layer about the main script directory, so it can use # it when attempting to follow imports. Importing.setMainScriptDirectory( main_dir = os.path.dirname(os.path.abspath(filename)) ) # Detect to be frozen modules if any, so we can consider to not recurse # to them. if Options.isStandaloneMode(): for module in detectEarlyImports(): ModuleRegistry.addUncompiledModule(module) if module.getName() == "site": origin_prefix_filename = os.path.join( os.path.dirname(module.getCompileTimeFilename()), "orig-prefix.txt" ) if os.path.isfile(origin_prefix_filename): data_files.append( (filename, "orig-prefix.txt") ) # Turn that source code into a node tree structure. try: main_module = createNodeTree( filename = filename ) except (SyntaxError, IndentationError) as e: handleSyntaxError(e) if Options.shallDumpBuiltTreeXML(): # XML only. for module in ModuleRegistry.getDoneModules(): dumpTreeXML(module) elif Options.shallDisplayBuiltTree(): # GUI only. displayTree(main_module) else: # Make the actual compilation. result, options = compileTree( main_module = main_module ) # Exit if compilation failed. if not result: sys.exit(1) if Options.shallNotDoExecCCompilerCall(): if Options.isShowMemory(): MemoryUsage.showMemoryTrace() sys.exit(0) # Remove the source directory (now build directory too) if asked to. if Options.isRemoveBuildDir(): removeDirectory( path = getSourceDirectoryPath(main_module), ignore_errors = False ) if Options.isStandaloneMode(): binary_filename = options["result_name"] + ".exe" standalone_entry_points.insert( 0, (binary_filename, binary_filename, None) ) dist_dir = getStandaloneDirectoryPath(main_module) for module in ModuleRegistry.getDoneUserModules(): standalone_entry_points.extend( Plugins.considerExtraDlls(dist_dir, module) ) for module in ModuleRegistry.getUncompiledModules(): standalone_entry_points.extend( Plugins.considerExtraDlls(dist_dir, module) ) copyUsedDLLs( dist_dir = dist_dir, standalone_entry_points = standalone_entry_points ) for module in ModuleRegistry.getDoneModules(): data_files.extend( Plugins.considerDataFiles(module) ) for source_filename, target_filename in data_files: target_filename = os.path.join( getStandaloneDirectoryPath(main_module), target_filename ) makePath(os.path.dirname(target_filename)) shutil.copy2( source_filename, target_filename ) # Modules should not be executable, but Scons creates them like it, fix # it up here. if Utils.getOS() != "Windows" and Options.shallMakeModule(): subprocess.call( ( "chmod", "-x", getResultFullpath(main_module) ) ) if Options.shallMakeModule() and Options.shallCreatePyiFile(): pyi_filename = getResultBasepath(main_module) + ".pyi" with open(pyi_filename, 'w') as pyi_file: pyi_file.write( """\ # This file was generated by Nuitka and describes the types of the # created shared library. # At this time it lists only the imports made and can be used by the # tools that bundle libraries, including Nuitka itself. For instance # standalone mode usage of the created library will need it. # In the future, this will also contain type information for values # in the module, so IDEs will use this. Therfore please include it # when you make software releases of the extension module that it # describes. %(imports)s # This is not Python source even if it looks so. Make it clear for # now. This was decided by PEP 484 designers. __name__ = ... """ % { "imports" : '\n'.join( "import %s" % module_name for module_name in getImportedNames() ) } ) # Execute the module immediately if option was given. if Options.shallExecuteImmediately(): if Options.shallMakeModule(): executeModule( tree = main_module, clean_path = Options.shallClearPythonPathEnvironment() ) else: executeMain( binary_filename = getResultFullpath(main_module), clean_path = Options.shallClearPythonPathEnvironment() ) Nuitka-0.5.28.2/nuitka/Variables.py0000644000372000001440000002225213207537242017250 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Variables link the storage and use of a Python variable together. Different kinds of variables represent different scopes and owners types, and their links between each other, i.e. references as in closure or module variable references. """ from nuitka.nodes.shapes.StandardShapes import ShapeUnknown from nuitka.utils import InstanceCounters, Utils complete = False class Variable(object): # We will need all of these attributes, since we track the global # state and cache some decisions as attributes, pylint: disable=too-many-instance-attributes __slots__ = ( "variable_name", "owner", "version_number", "shared_users", "shared_scopes", "traces", "users", "writers" ) @InstanceCounters.counted_init def __init__(self, owner, variable_name): assert type(variable_name) is str, variable_name assert type(owner) not in (tuple, list), owner self.variable_name = variable_name self.owner = owner self.version_number = 0 self.shared_users = False self.shared_scopes = False self.traces = set() # Derived from all traces. self.users = None self.writers = None __del__ = InstanceCounters.counted_del() def getDescription(self): return "variable '%s'" % self.variable_name def getName(self): return self.variable_name def getOwner(self): return self.owner def getEntryPoint(self): return self.owner.getEntryPoint() def getCodeName(self): var_name = self.variable_name var_name = var_name.replace('.', '$') var_name = Utils.encodeNonAscii(var_name) return var_name def allocateTargetNumber(self): self.version_number += 1 return self.version_number # pylint: disable=no-self-use def isLocalVariable(self): return False def isParameterVariable(self): return False def isModuleVariable(self): return False def isTempVariable(self): return False # pylint: enable=R0201 def addVariableUser(self, user): # Update the shared scopes flag. if user is not self.owner: # These are not really scopes. self.shared_users = True if user.isExpressionGeneratorObjectBody() or \ user.isExpressionCoroutineObjectBody() or \ user.isExpressionAsyncgenObjectBody(): if self.owner is user.getParentVariableProvider(): return self.shared_scopes = True def isSharedAmongScopes(self): return self.shared_scopes def isSharedTechnically(self): if not self.shared_users: return False if not complete: return None if not self.users: return False owner = self.owner.getEntryPoint() for user in self.users: user = user.getEntryPoint() while user is not owner and \ ( (user.isExpressionFunctionBody() and not user.needsCreation()) or \ user.isExpressionClassBody() ): user = user.getParentVariableProvider() if user is not owner: return True return False def addTrace(self, variable_trace): self.traces.add(variable_trace) def removeTrace(self, variable_trace): # Make it unusable, and break GC cycles while at it. variable_trace.variable = None variable_trace.previous = None self.traces.remove(variable_trace) def updateUsageState(self): writers = set() users = set() for trace in self.traces: owner = trace.owner users.add(owner) if trace.isAssignTrace(): writers.add(owner) self.writers = writers self.users = users def hasWritesOutsideOf(self, user): if not complete: return None elif user in self.writers: return len(self.writers) > 1 else: return bool(self.writers) def hasAccessesOutsideOf(self, provider): if not complete: return None elif self.users is None: return False elif provider in self.users: return len(self.users) > 1 else: return bool(self.users) def hasDefiniteWrites(self): if not complete: return None else: return bool(self.writers) def getMatchingAssignTrace(self, assign_node): for trace in self.traces: if trace.isAssignTrace() and trace.getAssignNode() is assign_node: return trace return None def getTypeShapes(self): result = set() for trace in self.traces: if trace.isAssignTrace(): result.add( trace.getAssignNode().getAssignSource().getTypeShape() ) elif trace.isUnknownTrace(): result.add(ShapeUnknown) elif trace.isUninitTrace(): if trace.hasDefiniteUsages() or trace.hasPotentialUsages(): result.add(ShapeUnknown) elif trace.isInitTrace(): result.add(ShapeUnknown) elif trace.isMergeTrace(): pass else: assert False, trace return result class LocalVariable(Variable): __slots__ = () def __init__(self, owner, variable_name): Variable.__init__( self, owner = owner, variable_name = variable_name ) def __repr__(self): return "<%s '%s' of '%s'>" % ( self.__class__.__name__, self.variable_name, self.owner.getName() ) def isLocalVariable(self): return True class ParameterVariable(LocalVariable): __slots__ = () def __init__(self, owner, parameter_name): LocalVariable.__init__( self, owner = owner, variable_name = parameter_name ) def getDescription(self): return "parameter variable '%s'" % self.variable_name def isParameterVariable(self): return True class ModuleVariable(Variable): __slots__ = ("module",) def __init__(self, module, variable_name): assert type(variable_name) is str, repr(variable_name) assert module.isCompiledPythonModule() Variable.__init__( self, owner = module, variable_name = variable_name ) self.module = module def __repr__(self): return "" % ( self.variable_name, self.getModule().getFullName() ) def getDescription(self): return "global variable '%s'" % self.variable_name def isModuleVariable(self): return True def getModule(self): return self.module class TempVariable(Variable): __slots__ = () def __init__(self, owner, variable_name): Variable.__init__( self, owner = owner, variable_name = variable_name ) def __repr__(self): return "" % ( self.getName(), self.getOwner().getName() ) def getDescription(self): return "temp variable '%s'" % self.variable_name def isTempVariable(self): return True def updateVariablesFromCollection(old_collection, new_collection): # After removing/adding traces, we need to pre-compute the users state # information. touched_variables = set() if old_collection is not None: for variable_trace in old_collection.getVariableTracesAll().values(): variable = variable_trace.getVariable() variable.removeTrace(variable_trace) touched_variables.add(variable) if new_collection is not None: for variable_trace in new_collection.getVariableTracesAll().values(): variable = variable_trace.getVariable() variable.addTrace(variable_trace) touched_variables.add(variable) # Release the memory, and prevent the "active" state from being ever # inspected, it's useless now. new_collection.variable_actives.clear() del new_collection.variable_actives for variable in touched_variables: variable.updateUsageState() Nuitka-0.5.28.2/nuitka/Options.py0000644000372000001440000006634513207537242017006 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Options module """ import logging import os import sys from optparse import SUPPRESS_HELP, OptionGroup, OptionParser from nuitka.PythonVersions import ( getSupportedPythonVersions, getSupportedPythonVersionStr, python_version_str ) from nuitka.utils import Utils from nuitka.Version import getNuitkaVersion # Indicator if we were called as "nuitka-run" in which case we assume some # other defaults and work a bit different with parameters. is_nuitka_run = os.path.basename(sys.argv[0]).lower().startswith("nuitka-run") if not is_nuitka_run: usage = "usage: %prog [--module] [--execute] [options] main_module.py" else: usage = "usage: %prog [options] main_module.py" parser = OptionParser( usage = usage, version = getNuitkaVersion() ) # This option is obsolete, and module should be used. parser.add_option( "--exe", action = "store_true", dest = "obsolete_executable", default = False, help = SUPPRESS_HELP ) parser.add_option( "--module", action = "store_false", dest = "executable", default = True, help = """\ Create an extension module executable instead of a program. Defaults to off.""" ) parser.add_option( "--standalone", "--portable", action = "store_true", dest = "is_standalone", default = False, help = """\ Enable standalone mode in build. This allows you to transfer the created binary to other machines without it relying on an existing Python installation. It implies these option: "--recurse-all". You may also want to use "--python-flag=no_site" to avoid the "site.py" module, which can save a lot of code dependencies. Defaults to off.""", ) parser.add_option( "--nofreeze-stdlib", action = "store_false", dest = "freeze_stdlib", default = True, help = """\ In standalone mode by default all modules of standard library will be frozen as bytecode. This compiles them all and as a result compilation time will increase very much. """, ) parser.add_option( "--python-version", action = "store", dest = "python_version", choices = getSupportedPythonVersions(), default = None, help = """\ Major version of Python to be used, one of %s. Defaults to what you run Nuitka with (currently %s)""" % ( getSupportedPythonVersionStr(), python_version_str ) ) if os.name == "nt": parser.add_option( "--python-arch", action = "store", dest = "python_arch", choices = ("x86", "x86_64"), default = None, help = """\ Architecture of Python to use. One of "x86" or "x86_64". Defaults to what you run Nuitka with (currently "%s").""" % ( Utils.getArchitecture() ) ) parser.add_option( "--python-debug", "--python-dbg", action = "store_true", dest = "python_debug", default = None, help = """\ Use debug version or not. Default uses what you are using to run Nuitka, most likely a non-debug version.""" ) parser.add_option( "--python-flag", action = "append", dest = "python_flags", default = [], help = """\ Python flags to use. Default uses what you are using to run Nuitka, this enforces a specific mode. These are options that also exist to standard Python executable. Currently supported: "-S" (alias "nosite"), "static_hashes" (not use hash randomization), "no_warnings" (do not give Python runtime warnings), "-O" (alias "noasserts"). Default empty.""" ) parser.add_option( "--python-for-scons", "--python2-for-scons", action = "store", dest = "python_scons", default = None, help = """\ If using Python3.2 to Python3.4, provide the path of a Python binary to use for Scons. Otherwise Nuitka can use what you run Nuitka with or a "scons" binary that is found in PATH, or a Python installation from Windows registry.""" ) parser.add_option( "--warn-implicit-exceptions", action = "store_true", dest = "warn_implicit_exceptions", default = False, help = """\ Enable warnings for implicit exceptions detected at compile time.""", ) parser.add_option( "--warn-unusual-code", action = "store_true", dest = "warn_unusual_code", default = False, help = """\ Enable warnings for unusual code detected at compile time.""", ) recurse_group = OptionGroup( parser, "Control the recursion into imported modules" ) recurse_group.add_option( "--recurse-stdlib", action = "store_true", dest = "recurse_stdlib", default = False, help = """\ Also descend into imported modules from standard library. Defaults to off.""" ) recurse_group.add_option( "--recurse-none", action = "store_true", dest = "recurse_none", default = False, help = """\ When --recurse-none is used, do not descend into any imported modules at all, overrides all other recursion options. Defaults to off.""" ) recurse_group.add_option( "--recurse-all", "--recurse-on", action = "store_true", dest = "recurse_all", default = False, help = """\ When --recurse-all is used, attempt to descend into all imported modules. Defaults to off.""" ) recurse_group.add_option( "--recurse-to", action = "append", dest = "recurse_modules", metavar = "MODULE/PACKAGE", default = [], help = """\ Recurse to that module, or if a package, to the whole package. Can be given multiple times. Default empty.""" ) recurse_group.add_option( "--recurse-not-to", action = "append", dest = "recurse_not_modules", metavar = "MODULE/PACKAGE", default = [], help = """\ Do not recurse to that module, or if a package, to the whole package in any case, overrides all other options. Can be given multiple times. Default empty.""" ) recurse_group.add_option( "--recurse-plugins", "--recurse-directory", action = "append", dest = "recurse_extra", metavar = "MODULE/PACKAGE", default = [], help = """\ Recurse into that directory, no matter if it's used by the given main program in a visible form. Overrides all other recursion options. Can be given multiple times. Default empty.""" ) recurse_group.add_option( "--recurse-files", "--recurse-pattern", action = "append", dest = "recurse_extra_files", metavar = "PATTERN", default = [], help = """\ Recurse into files matching the PATTERN. Overrides all recursion other options. Can be given multiple times. Default empty.""" ) parser.add_option_group(recurse_group) execute_group = OptionGroup( parser, "Immediate execution after compilation" ) execute_group.add_option( "--run", "--execute", action = "store_true", dest = "immediate_execution", default = is_nuitka_run, help = """\ Execute immediately the created binary (or import the compiled module). Defaults to %s.""" % ("on" if is_nuitka_run else "off") ) execute_group.add_option( "--debugger", "--gdb", action = "store_true", dest = "debugger", default = False, help = """\ Execute inside "gdb" to automatically get a stack trace. Defaults to off.""" ) execute_group.add_option( "--execute-with-pythonpath", "--keep-pythonpath", action = "store_true", dest = "keep_pythonpath", default = False, help = """\ When immediately executing the created binary (--execute), don't reset PYTHONPATH. When all modules are successfully included, you ought to not need PYTHONPATH anymore.""" ) parser.add_option_group(execute_group) dump_group = OptionGroup( parser, "Dump options for internal tree" ) dump_group.add_option( "--dump-xml", "--xml", action = "store_true", dest = "dump_xml", default = False, help = "Dump the final result of optimization as XML, then exit." ) dump_group.add_option( "--display-tree", action = "store_true", dest = "display_tree", default = False, help = """\ Display the final result of optimization in a GUI, then exit.""" ) parser.add_option_group(dump_group) codegen_group = OptionGroup( parser, "Code generation choices" ) codegen_group.add_option( "--improved", "--enhanced", action = "store_true", dest = "disabled", default = False, help = SUPPRESS_HELP, ) codegen_group.add_option( "--full-compat", action = "store_false", dest = "improved", default = True, help = """\ Enforce absolute compatibility with CPython. Do not even allow minor deviations from CPython behavior, e.g. better tracebacks, which are not really incompatible, but different. This is intended for tests only and should not be necessary for normal use.""", ) codegen_group.add_option( "--code-gen-no-statement-lines", action = "store_false", dest = "statement_lines", default = True, help = SUPPRESS_HELP, # help = """\ # Statements shall have their line numbers set. Disable this for less precise # exceptions and slightly faster code. Not recommended. Defaults to off.""" ) codegen_group.add_option( "--file-reference-choice", action = "store", dest = "file_reference_mode", choices = ("original", "runtime", "frozen"), default = None, help = """\ Select what value "__file__" is going to be. With "runtime" (default for standalone binary mode and module mode), the created binaries and modules, use the location of themselves to deduct the value of "__file__". Included packages pretend to be in directories below that location. This allows you to include data files in deployments. If you merely seek acceleration, it's better for you to use the "original" value, where the source files location will be used. With "frozen" a notation "" is used. For compatibility reasons, the "__file__" value will always have ".py" suffix independent of what it really is.""" ) parser.add_option_group(codegen_group) outputdir_group = OptionGroup( parser, "Output choices" ) outputdir_group.add_option( "--output-dir", action = "store", dest = "output_dir", metavar = "DIRECTORY", default = "", help = """\ Specify where intermediate and final output files should be put. The DIRECTORY will be populated with C files, object files, etc. Defaults to current directory. """ ) outputdir_group.add_option( "--remove-output", action = "store_true", dest = "remove_build", default = False, help = """\ Removes the build directory after producing the module or exe file. Defaults to off.""" ) outputdir_group.add_option( "--no-pyi-file", action = "store_false", dest = "pyi_file", default = True, help = """\ Do not create a ".pyi" file for extension modules created by Nuitka. Defaults to off.""" ) parser.add_option_group(outputdir_group) windows_group = OptionGroup( parser, "Windows specific output control" ) debug_group = OptionGroup( parser, "Debug features" ) debug_group.add_option( "--debug", action = "store_true", dest = "debug", default = False, help = """\ Executing all self checks possible to find errors in Nuitka, do not use for production. Defaults to off.""" ) debug_group.add_option( "--unstripped", "--no-strip", "--unstriped", action = "store_true", dest = "unstripped", default = False, help = """\ Keep debug info in the resulting object file for better debugger interaction. Defaults to off.""" ) debug_group.add_option( "--profile", action = "store_true", dest = "profile", default = False, help = """\ Enable vmprof based profiling of time spent. Defaults to off.""" ) debug_group.add_option( "--graph", action = "store_true", dest = "graph", default = False, help = """\ Create graph of optimization process. Defaults to off.""" ) debug_group.add_option( "--trace-execution", action = "store_true", dest = "trace_execution", default = False, help = """\ Traced execution output, output the line of code before executing it. Defaults to off.""" ) debug_group.add_option( "--recompile-c-only", "--recompile-c++-only", action = "store_true", dest = "recompile_c_only", default = False, help = """\ Take existing files and compile them again. Allows compiling edited C files with the C compiler for quick debugging changes to the generated source. Defaults to off. Depends on compiling Python source to determine which files it should look at.""" ) debug_group.add_option( "--generate-c-only", action = "store_true", dest = "generate_c_only", default = False, help = """\ Generate only C source code, and do not compile it to binary or module. This is for debugging and code coverage analysis that doesn't waste CPU. Defaults to off.""" ) debug_group.add_option( "--experimental", action = "append", dest = "experimental", default = [], help = """\ Use features declared as 'experimental'. May have no effect if no experimental features are present in the code. Uses secret tags (check source) per experimented feature.""" ) debug_group.add_option( "--explain-imports", action = "store_true", dest = "explain_imports", default = False, help = SUPPRESS_HELP ) # This is for testing framework, "coverage.py" hates to loose the process. And # we can use it to make sure it's not done unknowingly. parser.add_option( "--must-not-re-execute", action = "store_false", dest = "allow_reexecute", default = True, help = SUPPRESS_HELP ) parser.add_option_group(debug_group) c_compiler_group = OptionGroup( parser, "Backend C compiler choice" ) c_compiler_group.add_option( "--clang", action = "store_true", dest = "clang", default = False, help = """\ Enforce the use of clang (needs clang 3.2 or higher). Defaults to off.""" ) c_compiler_group.add_option( "--mingw", action = "store_true", dest = "mingw", default = False, help = """\ Enforce the use of MinGW on Windows. Defaults to off.""" ) c_compiler_group.add_option( "--msvc", action = "store", dest = "msvc", default = None, help = """\ Enforce the use of specific MSVC version on Windows. Allowed values are e.g. 9.0, 9.0exp, specify an illegal value for a list of installed compilers. Defaults to the most recent version.""" ) c_compiler_group.add_option( "-j", "--jobs", action = "store", dest = "jobs", metavar = 'N', default = Utils.getCoreCount(), help = """\ Specify the allowed number of parallel C compiler jobs. Defaults to the system CPU count.""", ) c_compiler_group.add_option( "--lto", action = "store_true", dest = "lto", default = False, help = """\ Use link time optimizations if available and usable (g++ 4.6 and higher). Defaults to off.""" ) parser.add_option_group(c_compiler_group) tracing_group = OptionGroup( parser, "Tracing features" ) tracing_group.add_option( "--show-scons", action = "store_true", dest = "show_scons", default = False, help = """\ Operate Scons in non-quiet mode, showing the executed commands. Defaults to off.""" ) tracing_group.add_option( "--show-progress", action = "store_true", dest = "show_progress", default = False, help = """Provide progress information and statistics. Defaults to off.""" ) tracing_group.add_option( "--show-memory", action = "store_true", dest = "show_memory", default = False, help = """Provide memory information and statistics. Defaults to off.""" ) tracing_group.add_option( "--show-modules", action = "store_true", dest = "show_inclusion", default = False, help = """Provide a final summary on included modules. Defaults to off.""" ) tracing_group.add_option( "--verbose", action = "store_true", dest = "verbose", default = False, help = """\ Output details of actions taken, esp. in optimizations. Can become a lot. Defaults to off.""" ) parser.add_option_group(tracing_group) windows_group.add_option( "--windows-disable-console", action = "store_true", dest = "win_disable_console", default = False, help = """\ When compiling for Windows, disable the console window. Defaults to off.""" ) windows_group.add_option( "--windows-icon", "--icon", action = "store", dest = "icon_path", metavar = "ICON_PATH", default = None, help = "Add executable icon (Windows only).", ) parser.add_option_group(windows_group) plugin_group = OptionGroup( parser, "Plugin control" ) plugin_group.add_option( "--plugin-enable", "--enable-plugin", action = "append", dest = "plugins_enabled", default = [], help = """\ Enabled plugins. Must be plug-in names. Use --plugin-list to query the full list and exit. Default empty.""" ) plugin_group.add_option( "--plugin-disable", "--disable-plugin", action = "append", dest = "plugins_disabled", default = [], help = """\ Disabled plugins. Must be plug-in names. Use --plugin-list to query the full list and exit. Default empty.""" ) plugin_group.add_option( "--plugin-no-detection", action = "store_false", dest = "detect_missing_plugins", default = True, help = """\ Plugins can detect if they might be used, and the you can disable the warning via --plugin-disable=plugin-that-warned, or you can use this option to disable the mechanism entirely, which also speeds up compilation slightly of course as this detection code is run in vain once you are certain of which plug-ins to use. Defaults to off.""" ) plugin_group.add_option( "--plugin-list", action = "store_true", dest = "list_plugins", default = False, help = """\ Show list of all available plugins and exit. Defaults to off.""" ) parser.add_option_group(plugin_group) options = None positional_args = None extra_args = [] def parseArgs(): # singleton with many cases, pylint: disable=global-statement,too-many-branches,too-many-statements global options, positional_args, extra_args # First, isolate the first non-option arguments. if is_nuitka_run: count = 0 for count, arg in enumerate(sys.argv): if count == 0: continue if arg[0] != '-': break # Treat "--" as a terminator. if arg == "--": count += 1 break if count > 0: extra_args = sys.argv[count+1:] sys.argv = sys.argv[0:count+1] options, positional_args = parser.parse_args() if shallListPlugins(): from nuitka.plugins.Plugins import listPlugins listPlugins() if not positional_args: parser.print_help() sys.exit(""" Error, need positional argument with python module or main program.""") if not options.immediate_execution and len(positional_args) > 1: parser.print_help() sys.exit(""" Error, need only one positional argument unless "--run" is specified to pass them to the compiled program execution.""") if options.verbose: logging.getLogger().setLevel(logging.DEBUG) else: logging.getLogger().setLevel(logging.INFO) # Standalone mode implies an executable, not importing "site" module, which is # only for this machine, recursing to all modules, and even including the # standard library. if options.is_standalone: if not options.executable: sys.exit("""\ Error, conflicting options, cannot make standalone module, only executable.""") options.recurse_all = True if Utils.getOS() == "NetBSD": logging.warning("Standalone mode on NetBSD is not functional, due to $ORIGIN linkage not being supported.") for any_case_module in getShallFollowModules(): if any_case_module.startswith('.'): bad = True else: for char in "/\\:": if char in any_case_module: bad = True break else: bad = False if bad: sys.exit( """\ Error, '--recurse-to' takes only module names, not directory path '%s'.""" % \ any_case_module ) for no_case_module in getShallFollowInNoCase(): if no_case_module.startswith('.'): bad = True else: for char in "/\\:": if char in no_case_module: bad = True break else: bad = False if bad: sys.exit( """\ Error, '--recurse-not-to' takes only module names, not directory path '%s'.""" % \ no_case_module ) scons_python = getPythonPathForScons() if scons_python is not None and not os.path.exists(scons_python): sys.exit("Error, no such Python2 binary '%s'." % scons_python) def shallTraceExecution(): return options.trace_execution def shallExecuteImmediately(): return options.immediate_execution def shallRunInDebugger(): return options.debugger def shallDumpBuiltTreeXML(): return options.dump_xml def shallDisplayBuiltTree(): return options.display_tree def shallOnlyExecCCompilerCall(): return options.recompile_c_only def shallNotDoExecCCompilerCall(): return options.generate_c_only def shallHaveStatementLines(): return options.statement_lines def getFileReferenceMode(): if options.file_reference_mode is None: value = ("runtime" if shallMakeModule() or isStandaloneMode() else "original") else: value = options.file_reference_mode return value def shallMakeModule(): return not options.executable def shallCreatePyiFile(): return options.pyi_file def isAllowedToReexecute(): return options.allow_reexecute def shallFollowStandardLibrary(): return options.recurse_stdlib def shallFollowNoImports(): return options.recurse_none def shallFollowAllImports(): return options.recurse_all def _splitShellPattern(value): return value.split(',') if '{' not in value else [value] def getShallFollowInNoCase(): return sum([ _splitShellPattern(x) for x in options.recurse_not_modules ], []) def getShallFollowModules(): return sum([ _splitShellPattern(x) for x in options.recurse_modules ], []) def getShallFollowExtra(): return sum([ x.split(',') for x in options.recurse_extra ], []) def getShallFollowExtraFilePatterns(): return sum([ x.split(',') for x in options.recurse_extra_files ], []) def shallWarnImplicitRaises(): return options.warn_implicit_exceptions def shallWarnUnusualCode(): return options.warn_unusual_code def isDebug(): return options is not None and (options.debug or options.debugger) def isPythonDebug(): return options.python_debug or sys.flags.debug def isUnstripped(): return options.unstripped or options.profile def isProfile(): return options.profile def shouldCreateGraph(): return options.graph def getOutputPath(path): if options.output_dir: return os.path.normpath(os.path.join(options.output_dir, path)) else: return path def getOutputDir(): return options.output_dir if options.output_dir else '.' def getPositionalArgs(): return tuple(positional_args) def getMainArgs(): return tuple(extra_args) def shallOptimizeStringExec(): return False def shallClearPythonPathEnvironment(): return not options.keep_pythonpath def isShowScons(): return options.show_scons def getJobLimit(): return int(options.jobs) def isLto(): return options.lto def isClang(): return options.clang def isMingw(): return options.mingw def getMsvcVersion(): return options.msvc def shallDisableConsoleWindow(): return options.win_disable_console def isFullCompat(): return not options.improved def isShowProgress(): return options.show_progress def isShowMemory(): return options is not None and options.show_memory def isShowInclusion(): return options.show_inclusion def isRemoveBuildDir(): return options.remove_build and not options.generate_c_only def getIntendedPythonVersion(): return options.python_version def getIntendedPythonArch(): return options.python_arch def isExperimental(indication): """ Are experimental features to be enabled.""" return hasattr(options, "experimental") and indication in options.experimental def getExperimentalIndications(): if hasattr(options, "experimental"): return options.experimental else: return () def shallExplainImports(): return options is not None and options.explain_imports def isStandaloneMode(): return options.is_standalone def getIconPath(): return options.icon_path _python_flags = None def getPythonFlags(): # singleton, pylint: disable=global-statement global _python_flags if _python_flags is None: _python_flags = set() for parts in options.python_flags: for part in parts.split(','): if part in ("-S", "nosite", "no_site"): _python_flags.add("no_site") elif part in ("static_hashes", "norandomization", "no_randomization"): _python_flags.add("no_randomization") elif part in ("-v", "trace_imports", "trace_import"): _python_flags.add("trace_imports") elif part in ("no_warnings", "nowarnings"): _python_flags.add("no_warnings") elif part in ("-O", "no_asserts", "noasserts"): _python_flags.add("no_asserts") else: # Do not warn before executing in final context. if "PYTHONHASHSEED" in os.environ: logging.warning("Unsupported flag '%s'.", part) return _python_flags def shallFreezeAllStdlib(): return options.freeze_stdlib def shallListPlugins(): return options is not None and options.list_plugins def getPluginsEnabled(): if not options: return () return options.plugins_enabled def getPluginsDisabled(): if not options: return () return options.plugins_disabled def shallDetectMissingPlugins(): return options is not None and options.detect_missing_plugins def getPluginOptions(plugin_name): # TODO: This should come from command line, pylint: disable=unused-argument return {} def getPythonPathForScons(): return options.python_scons Nuitka-0.5.28.2/nuitka/SourceCodeReferences.py0000644000372000001440000001117113134660221021364 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Source code reference record. All the information to lookup line and file of a code location, together with the future flags in use there. """ from nuitka.__past__ import total_ordering from nuitka.utils.InstanceCounters import counted_del, counted_init @total_ordering class SourceCodeReference(object): __slots__ = ["filename", "line", "column"] @classmethod def fromFilenameAndLine(cls, filename, line): result = cls() result.filename = filename result.line = line return result __del__ = counted_del() @counted_init def __init__(self): self.filename = None self.line = None self.column = None def __repr__(self): return "<%s to %s:%s>" % (self.__class__.__name__, self.filename, self.line) def __lt__(self, other): # Many cases decide early, pylint: disable=too-many-return-statements if other is None: return True if other is self: return False assert isinstance(other, SourceCodeReference), other if self.filename < other.filename: return True elif self.filename > other.filename: return False else: if self.line < other.line: return True elif self.line > other.line: return False else: if self.column < other.column: return True elif self.column > other.column: return False else: return self.isInternal() < other.isInternal() def __eq__(self, other): if other is None: return False if other is self: return True assert isinstance(other, SourceCodeReference), other if self.filename != other.filename: return False if self.line != other.line: return False if self.column != other.column: return False return self.isInternal() is other.isInternal() def _clone(self, line): """ Make a copy it itself. """ return self.fromFilenameAndLine( filename = self.filename, line = line ) def atInternal(self): """ Make a copy it itself but mark as internal code. Avoids useless copies, by returning an internal object again if it is already internal. """ if not self.isInternal(): result = self._clone(self.line) return result else: return self def atLineNumber(self, line): """ Make a reference to the same file, but different line. Avoids useless copies, by returning same object if the line is the same. """ assert type(line) is int, line if self.line != line: return self._clone(line) else: return self def atColumnNumber(self, column): assert type(column) is int, column if self.column != column: result = self._clone(self.line) result.column = column return result else: return self def getLineNumber(self): return self.line def getColumnNumber(self): return self.column def getFilename(self): return self.filename def getAsString(self): return "%s:%s" % (self.filename, self.line) @staticmethod def isInternal(): return False class SourceCodeReferenceInternal(SourceCodeReference): __slots__ = () __del__ = counted_del() @counted_init def __init__(self): SourceCodeReference.__init__(self) @staticmethod def isInternal(): return True def fromFilename(filename): return SourceCodeReference.fromFilenameAndLine( filename = filename, line = 1 ) Nuitka-0.5.28.2/nuitka/ModuleRegistry.py0000644000372000001440000001337213207537242020321 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ This to keep track of used modules. There is a set of root modules, which are user specified, and must be processed. As they go, they add more modules to active modules list and move done modules out of it. That process can be restarted and modules will be fetched back from the existing set of modules. """ import os from nuitka.containers.oset import OrderedSet from nuitka.PythonVersions import python_version from nuitka.utils.FileOperations import areSamePaths # One or more root modules, i.e. entry points that must be there. root_modules = OrderedSet() # To be traversed modules active_modules = OrderedSet() # Already traversed modules done_modules = set() # Uncompiled modules uncompiled_modules = set() def addRootModule(module): root_modules.add(module) def getRootModules(): return root_modules def hasRootModule(module_name): for module in root_modules: if module.getFullName() == module_name: return True return False def replaceRootModule(old, new): # Using global here, as this is really a singleton, in the form of a module, # pylint: disable=global-statement global root_modules new_root_modules = OrderedSet() for module in root_modules: new_root_modules.add(module if module is not old else new) root_modules = new_root_modules def addUncompiledModule(module): uncompiled_modules.add(module) def getUncompiledModules(): return sorted( uncompiled_modules, key = lambda module : module.getFullName() ) def getUncompiledTechnicalModules(): result = [ module for module in uncompiled_modules if module.isTechnical() ] return sorted( result, key = lambda module : module.getFullName() ) def getUncompiledNonTechnicalModules(): result = [ module for module in uncompiled_modules if not module.isTechnical() ] return sorted( result, key = lambda module : module.getFullName() ) def _normalizeModuleFilename(filename): if python_version >= 300: filename = filename.replace("__pycache__", "") suffix = ".cpython-%d.pyc" % (python_version // 10) if filename.endswith(suffix): filename = filename[:-len(suffix)] + ".py" else: if filename.endswith(".pyc"): filename = filename[:-3] + ".py" if os.path.basename(filename) == "__init__.py": filename = os.path.dirname(filename) return filename def getUncompiledModule(module_name, module_filename): for uncompiled_module in uncompiled_modules: if module_name == uncompiled_module.getFullName(): if areSamePaths( _normalizeModuleFilename(module_filename), _normalizeModuleFilename(uncompiled_module.filename) ): return uncompiled_module return None def removeUncompiledModule(module): uncompiled_modules.remove(module) def startTraversal(): # Using global here, as this is really a singleton, in the form of a module, # pylint: disable=global-statement global active_modules, done_modules active_modules = OrderedSet(root_modules) done_modules = set() for active_module in active_modules: active_module.startTraversal() def addUsedModule(module): if module not in done_modules and module not in active_modules: active_modules.add(module) module.startTraversal() def nextModule(): if active_modules: result = active_modules.pop() done_modules.add(result) return result else: return None def remainingCount(): return len(active_modules) def getDoneModules(): return sorted( done_modules, key = lambda module : module.getFullName() ) def getDoneUserModules(): return sorted( [ module for module in done_modules if not module.isInternalModule() ], key = lambda module : module.getFullName() ) def removeDoneModule(module): done_modules.remove(module) def getModuleFromCodeName(code_name): # TODO: We need something to just load modules. for module in root_modules: if module.getCodeName() == code_name: return module assert False, code_name def getOwnerFromCodeName(code_name): if "$$$" in code_name: module_code_name, _function_code_name = code_name.split("$$$", 1) module = getModuleFromCodeName(module_code_name) return module.getFunctionFromCodeName(code_name) else: return getModuleFromCodeName(code_name) def getModuleByName(module_name): for module in active_modules: if module.getFullName() == module_name: return module for module in done_modules: if module.getFullName() == module_name: return module for module in uncompiled_modules: if module.getFullName() == module_name: return module return None Nuitka-0.5.28.2/nuitka/containers/0000755000372000001440000000000013207540420017120 5ustar hayenusers00000000000000Nuitka-0.5.28.2/nuitka/containers/odict.py0000644000372000001440000001445313122472300020600 0ustar hayenusers00000000000000""" This module is only an abstraction of OrderedDict as present in 2.7 and 3.x. It is not in 2.6, for this version we are using the odict.py as mentioned in the PEP-0372. This can be removed safely after Python2.6 support is dropped (if ever), note that the documentation was removed, as it's not interesting really, being redundant to the Python 2.7 documentation. Starting with Python 3.6, we can safely use the built-in dictionary. """ # :copyright: (c) 2008 by Armin Ronacher and PEP 273 authors. # :license: modified BSD license. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # * Neither the name of the nor the # names of its contributors may be used to endorse or promote products # derived from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE # DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY # DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES # (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; # LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # Kay Hayen did some changes for Nuitka, and put everything he added under the same # modified BSD license. from nuitka.PythonVersions import python_version # pylint: disable=E0611,W0141 try: from collections import OrderedDict if python_version >= 360: OrderedDict = dict except ImportError: from itertools import izip, imap from copy import deepcopy missing = object() class OrderedDict(dict): def __init__(self, *args, **kwargs): dict.__init__(self) self._keys = [] self.update(*args, **kwargs) def __delitem__(self, key): dict.__delitem__(self, key) self._keys.remove(key) def __setitem__(self, key, item): if key not in self: self._keys.append(key) dict.__setitem__(self, key, item) def __deepcopy__(self, memo = None): if memo is None: memo = {} d = memo.get(id(self), missing) if d is not missing: return d memo[id(self)] = d = self.__class__() dict.__init__(d, deepcopy(self.items(), memo)) d._keys = self._keys[:] return d def __getstate__(self): return {"items": dict(self), "keys": self._keys} def __setstate__(self, d): self._keys = d["keys"] dict.update(d["items"]) def __reversed__(self): return reversed(self._keys) def __eq__(self, other): if isinstance(other, OrderedDict): if not dict.__eq__(self, other): return False return self.items() == other.items() return dict.__eq__(self, other) def __ne__(self, other): return not self.__eq__(other) def __cmp__(self, other): if isinstance(other, OrderedDict): return cmp(self.items(), other.items()) elif isinstance(other, dict): return dict.__cmp__(self, other) return NotImplemented @classmethod def fromkeys(cls, iterable, default = None): return cls((key, default) for key in iterable) def clear(self): del self._keys[:] dict.clear(self) def copy(self): return self.__class__(self) def items(self): return zip(self._keys, self.values()) def iteritems(self): return izip(self._keys, self.itervalues()) def keys(self): return self._keys[:] def iterkeys(self): return iter(self._keys) def pop(self, key, default = missing): if default is missing: return dict.pop(self, key) elif key not in self: return default self._keys.remove(key) return dict.pop(self, key, default) def popitem(self, key): self._keys.remove(key) return dict.popitem(key) def setdefault(self, key, default = None): if key not in self: self._keys.append(key) dict.setdefault(self, key, default) def update(self, *args, **kwargs): sources = [] if len(args) == 1: if hasattr(args[0], "iteritems"): sources.append(args[0].iteritems()) else: sources.append(iter(args[0])) elif args: raise TypeError("expected at most one positional argument") if kwargs: sources.append(kwargs.iteritems()) for iterable in sources: for key, val in iterable: self[key] = val def values(self): return map(self.get, self._keys) def itervalues(self): return imap(self.get, self._keys) def index(self, item): return self._keys.index(item) def byindex(self, item): key = self._keys[item] return (key, dict.__getitem__(self, key)) def reverse(self): self._keys.reverse() def sort(self, *args, **kwargs): self._keys.sort(*args, **kwargs) def __repr__(self): return "OrderedDict(%r)" % self.items() __copy__ = copy __iter__ = iterkeys Nuitka-0.5.28.2/nuitka/containers/oset.py0000644000372000001440000000611413122472300020443 0ustar hayenusers00000000000000# Copyright 2009 Raymond Hettinger # # Permission is hereby granted, free of charge, to any person obtaining # a copy of this software and associated documentation files (the # "Software"), to deal in the Software without restriction, including # without limitation the rights to use, copy, modify, merge, publish, # distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so, subject to # the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. # # Note: Kay Hayen did some changes for Nuitka keeping this license. These # changes are not improvements, use the original source instead, not this # file. """ This module is only an abstraction of OrderedSet which is not present in Python at all. It was originally downloaded from http://code.activestate.com/recipes/576694/ """ # pylint: disable=W0221,redefined-builtin import collections class OrderedSet(collections.MutableSet): def __init__(self, iterable = None): self.end = end = [] end += [None, end, end] # sentinel node for doubly linked list self.map = {} # key --> [key, prev, next] if iterable is not None: self |= iterable def __len__(self): return len(self.map) def __contains__(self, key): return key in self.map def add(self, key): if key not in self.map: end = self.end curr = end[1] curr[2] = end[1] = self.map[key] = [key, curr, end] def discard(self, key): if key in self.map: key, prev, next = self.map.pop(key) # @ReservedAssignment prev[2] = next next[1] = prev def __iter__(self): end = self.end curr = end[2] while curr is not end: yield curr[0] curr = curr[2] def __reversed__(self): end = self.end curr = end[1] while curr is not end: yield curr[0] curr = curr[1] def pop(self, last = True): if not self: raise KeyError("set is empty") key = self.end[1][0] if last else self.end[2][0] self.discard(key) return key def __repr__(self): if not self: return "%s()" % (self.__class__.__name__,) return "%s(%r)" % (self.__class__.__name__, list(self)) def __eq__(self, other): if isinstance(other, OrderedSet): return len(self) == len(other) and list(self) == list(other) return set(self) == set(other) Nuitka-0.5.28.2/nuitka/containers/__init__.py0000644000372000001440000000150113112214770021227 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Dummy file to make this directory a package. """ Nuitka-0.5.28.2/Changelog.rst0000644000372000001440000141716613207540035016123 0ustar hayenusers00000000000000Nuitka Release 0.5.28 ===================== This release has a focus on compatibility work and contains bug fixes and work to enhance the usability of Nuitka by integrating with distutils. The major improvement is that contractions no longer use pseudo functions to achieve their own local scope, but that there is now a dedicated structure for that representing an in-lined function. Bug Fixes --------- - Python3.6: Fix, ``async for`` was not yet implemented for async generators. - Fix, functions with keyword arguments where the value was determined to be a static raise could crash the compiler. - Detect using MinGW64 32 bits C compiler being used with 64 bits Python with better error message. - Fix, when extracting side effects of a static raise, extract them more recursively to catch expressions that themselves have no code generation being used. This fixes at least static raises in keyword arguments of a function call. - Compatibility: Added support for proper operation of ```pkgutil.get_data`` by implementing ``get_data`` in our meta path based loader. - Compatibility: Added ``__spec__`` module attribute was previously missing, present on Python3.4 and higher. - Compatibility: Made ``__loader__`` module attribute set when the module is loading already. - Standalone: Resolve the ``@rpath`` and ``@loader_path`` from ``otool`` on MacOS manually to actual paths, which adds support for libraries compiled with that. - Fix, nested functions calling ``super`` could crash the compiler. - Fix, could not use ``--recurse-directory`` with arguments that had a trailing slash. - Fix, using ``--recurse-directory`` on packages that are not in the search crashed the compiler. - Compatibility: Python2 ``set`` and ``dict`` contractions were using extra frames like Python3 does, but those are not needed. - Standalone: Fix, the way ``PYTHONHOME`` was set on Windows had no effect, which allowed the compiled binary to access the original installation still. - Standalone: Added some newly discovered missing hidden dependencies of extension modules. - Compatiblity: The name mangling of private names (e.g. ``__var``) in classes was applied to variable names, and function declarations, but not to classes yet. - Python3.6: Fix, added support for list contractions with ``await`` expressions in async generators. - Python3.6: Fix, ``async for`` was not working in async generators yet. - Fix, for module tracebacks, we output the module name `` instead of merely ````, but if the module was in a package, that was not indicated. Now it is ````. - Windows: The cache directory could be unicode which then failed to pass as an argument to scons. We now encode such names as UTF-8 and decode in Scons afterwards, solving the problem in a generic way. - Standalone: Need to recursively resolve shared libraries with ``ldd``, otherwise not all could be included. - Standalone: Make sure ``sys.path`` has no references to CPython compile time paths, or else things may work on the compiling machine, but not on another. - Standalone: Added various missing dependencies. - Standalone: Wasn't considering the DLLs directory for standard library extensions for freezing, which would leave out these. - Compatibility: For ``__future__`` imports the ``__import__`` function was called more than once. Optimization ------------ - Contractions are now all properly inlined and allow for optimization as if they were fully local. This should give better code in some cases. - Classes are now all building their locals dictionary inline to the using scope, allowing for more compact code. - The dictionary API was not used in module template code, although it helps to generate more compact code. New Features ------------ - Experimental support for building platform dependent wheel distribution. .. code-block:: sh python setup.py --command-packages=nuitka.distutils clean -a bdist_nuitka Use with caution, this is incomplete work. - Experimental support for running tests against compiled installation with ``nose`` and ``py.test``. - When specifiying what to recurse to, now patterns can be used, e.g. like this ``--recurse-not-to=*.tests`` which will skip all tests in submodules from compilation. - By setting ``NUITKA_PACKAGE_packagename=/some/path`` the ``__path__`` of packages can be extended automatically in order to allow and load uncompiled sources from another location. This can be e.g. a ``tests`` sub-package or other plug-ins. - By default when creating a module, now also a ``module.pyi`` file is created that contains all imported modules. This should be deployed alongside the extension module, so that standalone mode creation can benefit from knowing the dependencies of compiled code. - Added option ``--plugin-list`` that was mentioned in the help output, but still missing so far. - The import tracing of the ``hints`` module has achieved experimental status and can be used to test compatibility with regards to import behavior. Cleanups -------- - Rename tree and codegen ``Helper`` modules to unique names, making them easier to work with. - Share the code that decides to not warn for standard library paths with more warnings. - Use the ``bool`` enum definition of Python2 which is more elegant than ours. - Move quality tools, autoformat, isort, etc. to the ``nuitka.tools.quality`` namespace. - Move output comparison tool to the ``nuitka.tools.testing`` namespace. - Made frame code generation capable of using nested frames, allowing the real inline of classes and contraction bodies, instead of "direct" calls to pseudo functions being used. - Proper base classes for functions that are entry points, and functions that are merely a local expression using return statements. Tests ----- - The search mode with pattern, was not working anymore. - Resume hash values now consider the Python version too. - Added test that covers using test runners like ``nose`` and ``py.test`` with Nuitka compiled extension modules. Organizational -------------- - Added support for Scons 3.0 and running Scons with Python3.5 or higher. The option to specifiy the Python to use for scons has been renamed to reflect that it may also be a Python3 now. Only for Python3.2 to Python3.4 we now need another Python installation. - Made recursion the default for ``--recurse-directory`` with packages. Before you also had to tell it to recurse into that package or else it would only include the top level package, but nothing below. - Updated the man pages, correct mentions of its C++ to C and don't use now deprecated options. - Updated the help output which still said that standalone mode implies recursion into standard library, which is no longer true and even not recommended. - Added option to disable the output of ``.pyi`` file when creating an extension module. - Removed Ubuntu Wily package download, no longer supported by Ubuntu. Summary ------- This release was done to get the fixes and new features out for testing. There is work started that should make generators use an explicit extra stack via pointer, and restore instruction state via goto dispatchers at function entry, but that is not complete. This feature, dubbed "goto generators" will remove the need for fibers (which is itself a lot of code), reduce the memory footprint at run time for anything that uses a lot of generators, or coroutines. Integrating with ``distutils`` is also a new thing, and once completed will make use of Nuitka for existing projects automatic and trivial to do. There is a lot missing for that goal, but we will get there. Also, documenting how to run tests against compiled code, if that test code lives inside of that package, will make a huge difference, as that will make it easier for people to torture Nuitka with their own test cases. And then of course, nested frames now mean that every function could be inlined, which was previously not possible due to collisions of frames. This will pave the route for better optimization in those cases in future releases. The experimental features will require more work, but should make it easier to use Nuitka for existing projects. Future releases will make integrating Nuitka dead simple, or that is the hope. And last but not least, now that Scons works with Python3, chances are that Nuitka will more often work out the of the box. The older Python3 versions that still retain the issue are not very widespread. Nuitka Release 0.5.27 ===================== This release comes a lot of bug fixes and improvements. Bug Fixes --------- - Fix, need to add recursed modules immediately to the working set, or else they might first be processed in second pass, where global names that are locally assigned, are optimized to the built-in names although that should not happen. Fixed in 0.5.26.1 already. - Fix, the accelerated call of methods could crash for some special types. This had been a regress of 0.5.25, but only happens with custom extension types. Fixed in 0.5.26.1 already. - Python3.5: For ``async def`` functions parameter variables could fail to properly work with in-place assignments to them. Fixed in 0.5.26.4 already. - Compatability: Decorators that overload type checks didn't pass the checks for compiled types. Now ``isinstance`` and as a result ``inspect`` module work fine for them. - Compatiblity: Fix, imports from ``__init__`` were crashing the compiler. You are not supposed to do them, because they duplicate the package code, but they work. - Compatiblity: Fix, the ``super`` built-in on module level was crashing the compiler. - Standalone: For Linux, BSD and MacOS extension modules and shared libraries using their own ``$ORIGIN`` to find loaded DLLs resulted in those not being included in the distribution. - Standalone: Added more missing implicit dependencies. - Standalone: Fix, implicit imports now also can be optional, as e.g. ``_tkinter`` if not installed. Only include those if available. - The ``--recompile-c-only`` was only working with C compiler as a backend, but not in the C++ compatibility fallback, where files get renamed. This prevented that edit and test debug approach with at least MSVC. - Plugins: The PyLint plug-in didn't consider the symbolic name ``import-error`` but only the code ``F0401``. - Implicit exception raises in conditional expressions would crash the compiler. New Features ------------ - Added support for Visual Studio 2017. `Issue#368 `__. - Added option ``--python2-for-scons`` to specify the Python2 execute to use for calling Scons. This should allow using AnaConda Python for that task. Optimization ------------ - References to known unassigned variables are now statically optimized to exception raises and warned about if the according option is enabled. - Unhashable keys in dictionaries are now statically optimized to exception raises and warned about if the according option is enabled. - Enable forward propagation for classes too, resulting in some classes to create only static dictionaries. Currently this never happens for Python3, but it will, once we can statically optimize ``__prepare__`` too. - Enable inlining of class dictionary creations if they are mere return statements of the created dictionary. Currently this never happens for Python3, see above for why. - Python2: Selecting the metaclass is now visible in the tree and can be statically optimized. - For executables, we now also use a freelist for traceback objects, which also makes exception cases slightly faster. - Generator expressions no longer require the use of a function call with a ``.0`` argument value to carry the iterator value, instead their creation is directly inlined. - Remove "pass through" frames for Python2 list contractions, they are no longer needed. Minimal gain for generated code, but more lightweight at compile time. - When compiling Windows x64 with MinGW64 a link library needs to be created for linking against the Python DLL. This one is now cached and re-used if already done. - Use common code for ``NameError`` and ``UnboundLocalError`` exception code raises. In some cases it was creating the full string at compile time, in others at run time. Since the later is more efficient in terms of code size, we now use that everywhere, saving a bit of binary size. - Make sure to release unused functions from a module. This saves memory and can be decided after a full pass. - Avoid using ``OrderedDict`` in a couple of places, where they are not needed, but can be replaced with a later sorting, e.g. temporary variables by name, to achieve deterministic output. This saves memory at compile time. - Add specialized return nodes for the most frequent constant values, which are ``None``, ``True``, and ``False``. Also a general one, for constant value return, which avoids the constant references. This saves quite a bit of memory and makes traversal of the tree a lot faster, due to not having any child nodes for the new forms of return statements. - Previously the empty dictionary constant reference was specialized to save memory. Now we also specialize empty set, list, and tuple constants to the same end. Also the hack to make ``is`` not say that ``{} is {}`` was made more general, mutable constant references and now known to never alias. - The source references can be marked internal, which means that they should never be visible to the user, but that was tracked as a flag to each of the many source references attached to each node in the tree. Making a special class for internal references avoids storing this in the object, but instead it's now a class property. - The nodes for named variable reference, assignment, and deletion got split into separate nodes, one to be used before the actual variable can be determined during tree building, and one for use later on. This makes their API clearer and saves a tiny bit of memory at compile time. - Also eliminated target variable references, which were pseudo children of assignments and deletion nodes for variable names, that didn't really do much, but consume processing time and memory. - Added optimization for calls to ``staticmethod`` and ``classmethod`` built-in methods along with type shapes. - Added optimization for ``open`` built-in on Python3, also adding the type shape ``file`` for the result. - Added optimization for ``bytearray`` built-in and constant values. These mutable constants can now be compile time computed as well. - Added optimization for ``frozenset`` built-in and constant values. These mutable constants can now be compile time computed as well. - Added optimization for ``divmod`` built-in. - Treat all built-in constant types, e.g. ``type`` itself as a constant. So far we did this only for constant values types, but of course this applies to all types, giving slightly more compact code for their uses. - Detect static raises if iterating over non-iterables and warn about them if the option is enabled. - Split of ``locals`` node into different types, one which needs the updated value, and one which just makes a copy. Properly track if a functions needs an updated locals dict, and if it doesn't, don't use that. This gives more efficient code for Python2 classes, and ``exec`` using functions in Python2. - Build all constant values without use of the ``pickle`` module which has a lot more overhead than ``marshal``, instead use that for too large ``long`` values, non-UTF8 ``unicode`` values, ``nan`` float, etc. - Detect the linker arch for all Linux platforms using ``objdump`` instead of only a hand few hard coded ones. Cleanups -------- - The use of ``INCREASE_REFCOUNT`` got fully eliminated. - Use functions not vulenerable for buffer overflow. This is generally good and avoids warnings given on OpenBSD during linking. - Variable closure for classes is different from all functions, don't handle the difference in the base class, but for class nodes only. - Make sure ``mayBeNone`` doesn't return ``None`` which means normally "unclear", but ``False`` instead, since it's always clear for those cases. - Comparison nodes were using the general comparison node as a base class, but now a proper base class was added instead, allowing for cleaner code. - Valgrind test runners got changed to using proper tool namespace for their code and share it. - Made construct case generation code common testing code for re-use in the speedcenter web site. The code also has minor beauty bugs which will then become fixable. - Use ``appdirs`` package to determine place to store the downloaded copy of ``depends.exe``. - The code still mentioned C++ in a lot of places, in comments or identifiers, which might be confusing readers of the code. - Code objects now carry all information necessary for their creation, and no longer need to access their parent to determine flag values. That parent is subject to change in the future. - Our import sorting wrapper automatically detects imports that could be local and makes them so, removing a few existing ones and preventing further ones on the future. - Cleanups and annotations to become Python3 PyLint clean as well. This found e.g. that source code references only had ``__cmp__`` and need rich comparison to be fully portable. Tests ----- - The test runner for construct tests got cleaned up and the constructs now avoid using ``xrange`` so as to not need conversion for Python3 execution as much. - The main test runner got cleaned up and uses common code making it more versatile and robust. - Do not run test in debugger if CPython also segfaulted executing the test, then it's not a Nuitka issue, so we can ignore that. - Improve the way the Python to test with is found in the main test runner, prefer the running interpreter, then ``PATH`` and registry on Windows, this will find the interesting version more often. - Added support for "Landscape.io" to ignore the inline copies of code, they are not under our control. - The test runner for Valgrind got merged with the usage for constructs and uses common code now. - Construct generation is now common code, intended for sharing it with the Speedcenter web site generation. - Rebased Python 3.6 test suite to 3.6.1 as that is the Python generally used now. Organizational -------------- - Added inline copy of ``appdirs`` package from PyPI. - Added credits for RedBaron and isort. - The ``--experimental`` flag is now creating a list of indications and more than one can be used that way. - The PyLint runner can also work with Python3 pylint. - The Nuitka Speedcenter got more fine tuning and produces more tags to more easily identify trends in results. This needs to become more visible though. - The MSI files are also built on AppVeyor, where their building will not depend on me booting Windows. Getting these artifacts as downloads will be the next step. Summary ------- This release improves many areas. The variable closure taking is now fully transparent due to different node types, the memory usage dropped again, a few obvious missing static optimizations were added, and many built-ins were completed. This release again improves the scalability of Nuitka, which again uses less memory than before, although not an as big jump as before. This does not extend or use special C code generation for ``bool`` or any type yet, which still needs design decisions to proceed and will come in a later release. Nuitka Release 0.5.26 ===================== This release comes after a long time and contains large amounts of changes in all areas. The driving goal was to prepare generating C specific code, which is still not the case, but this is very likely going to change soon. However this release improves all aspects. Bug Fixes --------- - Compatibility: Fix, for star imports didn't check the values from the ``__all__`` iterable, if they were string values which could cause problems at run time. .. code-block:: python # Module level __all__ = (1,) # ... # other module: from module import * - Fix, for star imports, also didn't check for values from ``__all__`` if they actually exist in the original values. - Corner cases of imports should work a lot more precise, as the level of compatibility for calls to ``__import__`` went from absurd to insane. - Windows: Fixed detection of uninstalled Python versions (not for all users and DLL is not in system directory). This of course only affected the accelerated mode, not standalone mode. - Windows: Scan directories for ``.pyd`` files for used DLLs as well. This should make the PyQt5 wheel work. - Python3.5: Fix, coroutines could have different code objects for the object and the frame using by it. - Fix, slices with built-in names crashed the compiler. .. code-block:: python something[id:len:range] - Fix, the C11 via C++ compatibility uses symlinks tp C++ filenames where possible instead of making a copy from the C source. However, even on Linux that may not be allowed, e.g. on a DOS file system. Added fallback to using full copy in that case. `Issue#353 `__. - Python3.5: Fix coroutines to close the "yield from" where an exception is thrown into them. - Python3: Fix, list contractions should have their own frame too. - Linux: Copy the "rpath" of compiling Python binary to the created binary. This will make compiled binaries using uninstalled Python versions transparently find the Python shared library. - Standalone: Add the "rpath" of the compiling Python binary to the search path when checking for DLL dependencies on Linux. This fixes standalone support for Travis and Anaconda on Linux. - Scons: When calling scons, also try to locate a Python2 binary to overcome a potential Python3 virtualenv in which Nuitka is running. - Standalone: Ignore more Windows only encodings on non-Windows. New Features ------------ - Support for Python 3.6 with only few corner cases not supported yet. - Added options ``--python-arch`` to pick 32 or 64 bits Python target of the ``--python-version`` argument. - Added support for more kinds of virtualenv configurations. - Uninstalled Python versions such as Anaconda will work fine in accelerated mode, except on Windows. Optimization ------------ - The node tree children are no longer stored in a separate dictionary, but in the instance dictionary as attributes, making the tree more lightweight and in principle faster to access. This also saved about 6% of the memory usage. - The memory usage of Nuitka for the Python part has fallen by roughly 40% due to the use of new style classes, and slots where that is possible (some classes use multiple inheritance, where they don't work), and generally by reducing useless members e.g. in source code references. This of course also will make things compiled faster (the C compilation of course is not affected by this.) - The code generation for frames was creating the dictionary for the raised exception by making a dictionary and then adding all variables, each tested to be set. This was a lot of code for each frame specific, and has been replaced by a generic "attach" mechanism which merely stores the values, and only takes a reference. When asked for frame locals, it only then builds the dictionary. So this is now only done, when that is absolutely necessary, which it normally never is. This of course makes the C code much less verbose, and actual handling of exceptions much more efficient. - For imports, we now detect for built-in modules, that their import cannot fail, and if name lookups can fail. This leads to less code generated for error handling of these. The following code now e.g. fully detects that no ``ImportError`` or ``AttributeError`` will occur. .. code-block:: python try: from __builtin__ import len except ImportError: from builtins import len - Added more type shapes for built-in type calls. These will improve type tracing. - Compiled frames now have a free list mechanism that should speed up frames that recurse and frames that exit with exceptions. In case of an exception, the frame ownership is immediately transferred to the exception making it easier to deal with. - The free list implementations have been merged into a new common one that can be used via macro expansion. It is now type agnostic and be slightly more efficient too. - Also optimize "true" division and "floor division", not only the default division of Python2. - Removed the need for statement context during code generation making it less memory intensive and faster. Cleanups -------- - Now always uses the ``__import__`` built-in node for all kinds of imports and directly optimizes and recursion into other modules based on that kind of node, instead of a static variant. This removes duplication and some incompatability regarding defaults usage when doing the actual imports at run time. - Split the expression node bases and mixin classes to a dedicated module, moving methods that only belong to expressions outside of the node base, making for a cleaner class hierachy. - Cleaned up the class structure of nodes, added base classes for typical compositions, e.g. expression with and without children, computation based on built-in, etc. while also checking proper ordering of base classes in the metaclass. - Moved directory and file operations to dedicated module, making also sure it is more generally used. This makes it easier to make more error resilient deletions of directories on e.g. Windows, where locks tend to live for short times beyond program ends, requiring second attempts. - Code generation for existing supported types, ``PyObject *``, ``PyObject **``, and ``struct Nuitka_CellObject *`` is now done via a C type class hierachy instead of ``elif`` sequences. - Closure taking is now always done immediately correctly and references are take for closure variables still needed, making sure the tree is correct and needs no finalization. - When doing variable traces, initialize more traces immediately so it can be more reliable. - Code to setup a function for local variables and clean it up has been made common code instead of many similar copies. - The code was treating the ``f_executing`` frame member as if it were a counter with increases and decreases. Turn it into a mere boolean value and hide its usage behind helper functions. - The "maybe local variables" are no more. They were replaced by a new locals dict access node with a fallback to a module or closure variable should the dictionary not contain the name. This avoids many ugly checks to not do certain things for that kind of variable. - We now detect "exec" and "unqualified exec" as well as "star import" ahead of time as flags of the function to be created. We no longer need to mark functions as we go. - Handle "true", "floor" and normal division properly by applying future flags to decide which one to use. - We now use symbolic identifiers in all PyLint annotations. - The release scripts started to move into ``nuitka.tools.release`` so they get PyLint checks, autoformat and proper code re-use. - The use of ``INCREASE_REFCOUNT_X`` was removed, it got replaced with proper ``Py_XINCREF`` usages. - The use of ``INCREASE_REFCOUNT`` got reduced further, e.g. no generated code uses it anymore, and only a few compiled types do. The function was once required before "C-ish" lifted the need to do everything in one single function call. Tests ----- - More robust deletion of directories, temporary stages used by CPython test suites, and standalone directories during test execution. - Moved tests common code into ``nuitka.tools.testing`` namespace and use it from there. The code now is allowed to use ``nuitka.utils`` and therefore often better implementations. - Made standalone binaries robust against GTK theme access, checking the Python binary (some site.py files do that), Organizational -------------- - Added repository for Ubuntu Zesty (17.04) for download. - Added support for testing with Travis to complement the internal Buildbot based infrastructure and have pull requests on Github automatically tested before merge. - The ``factory`` branch is now also on Github. - Removed MSI for Python3.4 32 bits. It seems impossible to co-install this one with the 64 bits variant. All other versions are provided for both bit sizes still. Summary ------- This release marks huge progress. The node tree is now absolutely clean, the variable closure taking is fully represented, and code generation is prepared to add another type, e.g. for ``bool`` for which work has already started. On a practical level, the scalability of the release will have increased very much, as this uses so much less memory, generates simpler C code, while at the same time getting faster for the exception cases. Coming releases will expand on the work of this release. Frame objects should be allowed to be nested inside a function for better re-formulations of classes and contractions of all kinds, as well as real inline of functions, even if they could raise. The memory savings could be even larger, if we stopped doing multiple inheritance for more node types. The ``__slots__`` were and the child API change could potentially make things not only more compact, but faster to use too. And also once special C code generation for ``bool`` is done, it will set the stage for more types to follow (``int``, ``float``, etc). Only this will finally start to give the C type speed we are looking for. Until then, this release marks a huge cleanup and progress to what we already had, as well as preparing the big jump in speed. Nuitka Release 0.5.25 ===================== This release contains a huge amount of bug fixes, lots of optimization gains, and many new features. It also presents many organizational improvements, and many cleanups. Bug Fixes --------- - Python3.5: Coroutine methods using ``super`` were crashing the compiler. `Issue#340 `__. Fixed in 0.5.24.2 already. - Python3.3: Generator return values were not properly transmitted in case of ``tuple`` or ``StopIteration`` values. - Python3.5: Better interoperability between compiled coroutines and uncompiled generator coroutines. - Python3.5: Added support to compile in Python debug mode under Windows too. - Generators with arguments were using two code objects, one with, and one without the ``CO_NOFREE`` flag, one for the generator object creating function, and one for the generator object. - Python3.5: The duplicate code objects for generators with arguments lead to interoperability issues with between such compiled generator coroutines and compiled coroutines. `Issue#341 `__. Fixed in 0.5.24.2 already. - Standalone: On some Linux variants, e.g. Debian Stretch and Gentoo, the linker needs more flags to really compile to a binary with ``RPATH``. - Compatibility: For set literal values, insertion order is wrong on some versions of Python, we now detect the bug and emulate it if necessary, previous Nuitka was always correct, but incompatible. .. code-block:: python {1, 1.0}.pop() # the only element of the set should be 1 - Windows: Make the batch files detect where they live at run time, instead of during ``setup.py``, making it possible to use them for all cases. - Standalone: Added package paths to DLL scan for ``depends.exe``, as with wheels there now sometimes live important DLLs too. - Fix, the clang mode was regressed and didn't work anymore, breaking the MacOS support entirely. - Compatibility: For imports, we were passing for ``locals`` argument a real dictionary with actual values. That is not what CPython does, so stopped doing it. - Fix, for raised exceptions not passing the validity tests, they could be used after free, causing crashes. - Fix, the environment ``CC`` wasn't working unless also specifying ``CXX``. - Windows: The value of ``__file__`` in module mode was wrong, and didn't point to the compiled module. - Windows: Better support for ``--python-debug`` for installations that have both variants, it is now possible to switch to the right variant. New Features ------------ - Added parsing for shebang to Nuitka. When compiling an executable, now Nuitka will check of the ``#!`` portion indicates a different Python version and ask the user to clarify with ``--python-version`` in case of a mismatch. - Added support for Python flag ``-O``, which allows to disable assertions and remove doc strings. Optimization ------------ - Faster method calls, combining attribute lookup and method call into one, where order of evaluation with arguments doesn't matter. This gives really huge relative speedups for method calls with no arguments. - Faster attribute lookup in general for ``object`` descendants, which is all new style classes, and all built-in types. - Added dedicated ``xrange`` built-in implementation for Python2 and ``range`` for Python3. This makes those faster while also solving ordering problems when creating constants of these types. - Faster ``sum`` again, using quick iteration interface and specialized quick iteration code for typical standard type containers, ``tuple`` and ``list``. - Compiled generators were making sure ``StopIteration`` was set after their iteration, although most users were only going to clear it. Now only the ``send`` method, which really needs that does it. This speed up the closing of generators quite a bit. - Compiled generators were preparing a ``throw`` into non-started compilers, to be checked for immediately after their start. This is now handled in a generic way for all generators, saving code and execution time in the normal case. - Compiled generators were applying checks only useful for manual ``send`` calls even during iteration, slowing them down. - Compiled generators could duplicate code objects due to handling a flag for closure variables differently. - For compiled frames, the ``f_trace`` is not writable, but was taking and releasing references to what must be ``None``, which is not useful. - Not passing ``locals`` to import calls make it less code and faster too. Organizational -------------- - This release also prepares Python 3.6 support, it includes full language support on the level of CPython 3.6.0 with the sole exception of the new generator coroutines. - The improved mode is now the default, and full compatibility is now the option, used by test suites. For syntax errors, improved mode is always used, and for test suites, now only the error message is compared, but not call stack or caret positioning anymore. - Removed long deprecated option "--no-optimization". Code generation too frequently depends on not seeing unoptimized code. This has been hidden and broken long enough to finally remove it. - Added support for Python3.5 numbers to Speedcenter. There are now also tags for speedcenter, indicating how well "develop" branch fares in comparison to master. - With a new tool, source code and developer manual contents can be kept in sync, so that descriptions can be quoted there. Eventually a full Sphinx documentation might become available, but for now this makes it workable. - Added repository for Ubuntu Yakkety (16.10) for download. - Added repository for Fedora 25 for download. Cleanups -------- - Moved the tools to compare CPython output, to sort import statements (isort) to autoformat the source code (Redbaron usage), and to check with PyLint to a common new ``nuitka.tools`` package, runnable with ``__main__`` modules and dedicated runners in ``bin`` directory. - The tools now share code to find source files, or have it for the first time, and other things, e.g. finding needed binaries on Windows installations. - No longer patch traceback objects dealloc function. Should not be needed anymore, and most probably was only bug hiding. - Moved handling of ast nodes related to import handling to the proper reformulation module. - Moved statement generation code to helpers module, making it accessible without cyclic dependencies that require local imports. - Removed deprecated method for getting constant code objects in favor of the new way of doing it. Both methods were still used, making it harder to analyse. - Removed useless temporary variable initializations from complex call helper internal functions. They worked around code generation issues that have long been solved. - The ABI flags are no longer passed to Scons together with the version. Tests ----- - Windows: Added support to detect and to switch debug Python where available to also be able to execute reference counting tests. - Added the CPython 3.3 test suite, after cleaning up the worst bits of it, and added the brandnew 3.6 test suite with a minimal set of changes. - Use the original 3.4 test suite instead of the one that comes from Debian as it has patched quite a few issues that never made it upstream, and might cause crashes. - More construct tests, making a difference between old style classes, which have instances and new style classes, with their objects. - It is now possible to run a test program with Python3 and Valgrind. Summary ------- The quick iteration is a precursor to generally faster iteration over unknown object iterables. Expanding this to general code generation, and not just the ``sum`` built-in, might yield significant gains for normal code in the future, once we do code generation based on type inference. The faster method calls complete work that was already prepared in this domain and also will be expanded to more types than compiled functions. More work will be needed to round this up. Adding support for 3.6.0 in the early stages of its release, made sure we pretty much have support for it ready right after release. This is always a huge amount of work, and it's good to catch up. This release is again a significant improvement in performance, and is very important to clean up open ends. Now the focus of coming releases will now be on both structural optimization, e.g. taking advantage of the iterator tracing, and specialized code generation, e.g. for those iterations really necessary to use quick iteration code. Nuitka Release 0.5.24 ===================== This release is again focusing on optimization, this time very heavily on the generator performance, which was found to be much slower than CPython for some cases. Also there is the usual compatibility work and improvements for Pure C support. Bug Fixes --------- - Windows: The 3.5.2 coroutine new protocol implementation was using the wrapper from CPython, but it's not part of the ABI on Windows. Have our own instead. Fixed in 0.5.23.1 already. - Windows: Fixed second compilation with MSVC failing. The files renamed to be C++ files already existed, crashing the compilation. Fixed in 0.5.23.1 already. - Mac OS: Fixed creating extension modules with ``.so`` suffix. This is now properly determined by looking at the importer details, leading to correct suffix on all platforms. Fixed in 0.5.23.1 already. - Debian: Don't depend on a C++ compiler primarily anymore, the C compiler from GNU or clang will do too. Fixed in 0.5.23.1 already. - Pure C: Adapted scons compiler detecting to properly consider C11 compilers from the environment, and more gracefully report things. Optimization ------------ - Python2: Generators were saving and restoring exceptions, updating the variables ``sys.exc_type`` for every context switch, making it really slow, as these are 3 dictionary updates, normally not needed. Now it's only doing it if it means a change. - Sped up creating generators, coroutines and coroutines by attaching the closure variable storage directly to the object, using one variable size allocation, instead of two, once of which was a standard ``malloc``. This makes creating them easier and avoids maintaining the closure pointer entirely. - Using dedicated compiled cell implementation similar to ``PyCellObject`` but fully under our control. This allowed for smaller code generated, while still giving a slight performance improvement. - Added free list implementation to cache generator, coroutines, and function objects, avoiding the need to create and delete this kind of objects in a loop. - Added support for the built-in ``sum``, making slight optimizations to be much faster when iterating over lists and tuples, as well as fast ``long`` sum for Python2, and much faster ``bool`` sums too. This is using a prototype version of a "qiter" concept. - Provide type shape for ``xrange`` calls that are not constant too, allowing for better optimization related to those. Tests ----- - Added workarounds for locks being held by Virus Scanners on Windows to our test runner. - Enhanced constructs that test generator expressions to more clearly show the actual construct cost. - Added construct tests for the ``sum`` built-in on verious types of ``int`` containers, making sure we can do all of those really fast. Summary ------- This release improves very heavily on generators in Nuitka. The memory allocator is used more cleverly, and free lists all around save a lot of interactions with it. More work lies ahead in this field, as these are not yet as fast as they should be. However, at least Nuitka should be faster than CPython for these kind of usages now. Also, proper pure C in the Scons is relatively important to cover more of the rarer use cases, where the C compiler is too old. The most important part is actually how ``sum`` optimization is staging a new kind of approach for code generation. This could become the standard code for iterators in loops eventually, making ``for`` loops even faster. This will be for future releases to expand. Nuitka Release 0.5.23 ===================== This release is focusing on optimization, the most significant part for the users being enhanced scalability due to memory usage, but also break through structural improvements for static analysis of iterators and the debut of type shapes and value shapes, giving way to "shape tracing". Bug Fixes --------- - Fix support Python 3.5.2 coroutine changes. The checks got added for improved mode for older 3.5.x, the new protocol is only supported when run with that version or higher. - Fix, was falsely optimizing away unused iterations for non-iterable compile time constants. .. code-block:: python iter(1) # needs to raise. - Python3: Fix, ``eval`` must not attempt to ``strip`` memoryviews. The was preventing it from working with that type. - Fix, calling ``type`` without any arguments was crashing the compiler. Also the exception raised for anything but 1 or 3 arguments was claiming that only 3 arguments were allowed, which is not the compatible thing. - Python3.5: Fix, follow enhanced error checking for complex call handling of star arguments. - Compatibility: The ``from x import x, y`` re-formulation was doing two ``__import__`` calls instead of re-using the module value. Optimization ------------ - Uses only about 66% of the memory compared to last release, which is very important step for scalability independent of re-loading. This was achieved by making sure to break loop traces and their reference cycle when they become unused. - Properly detect the ``len`` of multiplications at compile time from newly introduces value shapes, so that this is e.g. statically optimized. .. code-block:: python print(len("*" * 10000000000)) - Due to newly introduced type shapes, ``len`` and ``iter`` now properly detect more often if values will raise or not, and warn about detected raises. .. code-block:: python iter(len((something)) # Will always raise - Due to newly introduced "iterator tracing", we can now properly detect if the length of an unpacking matches its source or not. This allows to remove the check of the generic re-formulations of unpackings at compile time. .. code-block:: python a, b = b, a # Will never raise due to unpacking a, b = b, a, c # Will always raise, 3 items cannot unpack to 2 - Added support for optimization of the ``xrange`` built-in for Python2. - Python2: Added support for ``xrange`` iterable constant values, pre-building those constants ahead of time. - Python3: Added support and ``range`` iterable constant values, pre-building those constants ahead of time. This brings optimization support for Python3 ranges to what was available for Python2 already. - Avoid having a special node variange for ``range`` with no arguments, but create the exception raising node directly. - Specialized constant value nodes are using less generic implementations to query e.g. their length or iteration capabilities, which should speed up many checks on them. - Added support for the ``format`` built-in. - Python3: Added support for the ``ascii`` built-in. Organizational -------------- - The movement to pure C got the final big push. All C++ only idoms of C++ were removed, and everything works with C11 compilers. A C++03 compiler can be used as a fallback, in case of MSVC or too old gcc for instance. - Using pure C, MinGW64 6x is now working properly. The latest version had problems with ``hypot`` related changes in the C++ standard library. Using C11 solves that. - This release also prepares Python 3.6 support, it includes full language support on the level of CPython 3.6.0b1. - The CPython 3.6 test suite was run with Python 3.5 to ensure bug level compatibility, and had a few findings of incompatibilities. Cleanups -------- - The last holdouts of classes in Nuitka were removed, and many idioms of C++ were stopped using. - Moved range related helper functions to a dedicated include file. - Using ``str is not bytes`` to detect Python3 ``str`` handling or actual ``bytes`` type existence. - Trace collections were using a mix-in that was merged with the base class that every user of it was having. Tests ----- - Added more static optimization tests, a lot more has become feasible to decide at run time, and is now done. These are to detect regressions in that domain. - The CPython 3.6 test suite is now also run with CPython 3.5 which found some incompatibilities. Summary ------- This release marks a huge step forward. We are having the structure for type inference now. This will expand in coming releases to cover more cases, and there are many low hanging fruits for optimization. Specialized codes for variable versions of certain known shapes seems feasible now. Then there is also the move towards pure C. This will make the backend compilation lighter, but due to using C11, we will not suffer any loss of convinience compared to "C-ish". The plan is to use continue to use C++ for compilation for compilers not capable of supporting C11. The amount of static analysis done in Nuitka is now going to quickly expand, with more and more constructs predicted to raise errors or simplified. This will be an ongoing activity, as many types of expressions need to be enhanced, and only one missing will not let it optimize as well. Also, it seems about time to add dedicated code for specific types to be as fast as C code. This opens up vast possibilities for acceleration and will lead us to zero overhead C bindings eventually. But initially the drive is towards enhanced ``import`` analysis, to become able to know the precide module expected to be imported, and derive type information from this. The coming work will attack to start whole program optimization, as well as enhanced local value shape analysis, as well specialized type code generation, which will make Nuitka improve speed. Nuitka Release 0.5.22 ===================== This release is mostly an intermediate release on the way to the large goal of having per module compilation that is cachable and requires far less memory for large programs. This is currently in progress, but required many changes that are in this release, more will be needed. It also contains a bunch of bug fixes and enhancements that are worth to be released, and the next changes are going to be more invasive. Bug Fixes --------- - Compatibility: Classes with decorated ``__new__`` functions could miss out on the ``staticmethod`` decorator that is implicit. It's now applied always, unless of course it's already done manually. This corrects an issue found with Pandas. Fixed in 0.5.22.1 already. - Standalone: For at least Python 3.4 or higher, it could happen that the locale needed was not importable. Fixed in 0.5.22.1 already. - Compatibility: Do not falsely assume that ``not`` expressions cannot raise on boolean expressions, since those arguments might raise during creation. This could lead to wrong optimization. Fixed in 0.5.22.2 already. - Standalone: Do not include system specific C libraries in the distribution created. This would lead to problems for some configurations on Linux in cases the glibc is no longer compatible with newer oder older kernels. Fixed in 0.5.22.2 already. - The ``--recurse-directory`` option didn't check with decision mechanisms for module inclusion, making it impossible to avoid some things. Optimization ------------ - Introduced specialized constant classes for empty dictionaries and other special constants, e.g. "True" and "False", so that they can have more hard coded properties and save memory by sharing constant values. - The "technical" sharing of a variable is only consider for variables that had some sharing going in the first place, speeing things up quite a bit for that still critical check. - Memory savings coming from enhanced trace storage are already visible at about 1%. That is not as much as the reloading will mean, but still helpful to use less overall. Cleanups -------- - The global variable registry was removed. It was in the way of unloading and reloading modules easily. Instead variables are now attached to their owner and referenced by other users. When they are released, these variables are released. - Global variable traces were removed. Instead each variable has a list of the traces attached to it. For non-shared variables, this allows to sooner tell attributes of those variables, allowing for sooner optimization of them. - No longer trace all initial users of a variable, just merely if there were such and if it constitutes sharing syntactically too. Not only does this save memory, it avoids useless references of the variable to functions that stop using it due to optimization. - Create constant nodes via a factory function to avoid non-special instances where variants exist that would be faster to use. - Moved the C string functions to a proper ``nuitka.utils.CStrings`` package as we use it for better code names of functions and modules. - Made ``functions`` and explicit child node of modules, which makes their use more generic, esp. for re-loading modules. - Have a dedicated function for building frame nodes, making it easier to see where they are created. Summary ------- This release is the result of a couple of months work, and somwhat means that proper re-loading of cached results is becoming in sight. The reloading of modules still fails for some things, and more changes will be needed, but with that out of the way, Nuitka's footprint is about to drop and making it then absolutely scalable. Something considered very important before starting to trace more information about values. This next thing big ought to be one thing that structurally holds Nuitka back from generating C level performance code with say integer operations. Nuitka Release 0.5.21 ===================== This release focused on scalability work. Making Nuitka more usable in the common case, and covering more standalone use cases. Bug Fixes --------- - Windows: Support for newer MinGW64 was broken by a workaround for older MinGW64 versions. - Compatibility: Added support for the (inofficial) C-Python API ``Py_GetArgcArgv`` that was causing ``prctl`` module to fail loading on ARM platforms. - Compatibility: The proper error message template for complex call arguments is now detected as compile time. There are changes comming, that are already in some pre-releases of CPython. - Standalone: Wasn't properly ignoring ``Tools`` and other directories in the standard library. New Features ------------ - Windows: Detect the MinGW compiler arch and compare it to the Python arch. In case of a mismatch, the compiler is not used. Otherwise compilation or linking gives hard to understand errors. This also rules out MinGW32 as a compiler that can be used, as its arch doesn't match MinGW64 32 bits variant. - Compile modules in two passes with the option to specify which modules will be considered for a second pass at all (compiled without program optimization) or even become bytecode. - The developer mode installation of Nuitka in ``develop`` mode with the command ``pip install -e nuitka_git_checkout_dir`` is now supported too. Optimization ------------ - Popular modules known to not be performance relevant are no longer C compiled, e.g. ``numpy.distutils`` and many others frequently imported (from some other module), but mostly not used and definitely not performance relevant. Cleanups -------- - The progress tracing and the memory tracing and now more clearly separate and therefore more readable. - Moved RPM related files to new ``rpm`` directory. - Moved documentation related files to ``doc`` directory. - Converted import sorting helper script to Python and made it run fast. Organizational -------------- - The Buildbot infrastructure for Nuitka was updated to Buildbot 0.8.12 and is now maintained up to date with Ansible. - Upgraded the Nuitka bug tracker to Roundup 1.5.1 to which I had previously contributed security fixes already active. - Added SSL certificates from Let's Encrypt for the web server. Summary ------- This release advances the scalability of Nuitka somewhat. The two pass approach does not yet carry all possible fruits. Caching of single pass compiled modules should follow for it to become consistently fast. More work will be needed to achieve fast and scalable compilation, and that is going to remain the focus for some time. Nuitka Release 0.5.20 ===================== This release is mostly about catching up with issues. Most address standalone problems with special modules, but there are also some general compatibility corrections, as well as important fixes for Python3.5 and coroutines and to improve compatibility with special Python variants like AnaConda under the Windows system. Bug Fixes --------- - Standalone Python3.5: The ``_decimal`` module at least is using a ``__name__`` that doesn't match the name at load time, causing programs that use it to crash. - Compatibility: For Python3.3 the ``__loader__`` attribute is now set in all cases, and it needs to have a ``__module__`` attribute. This makes inspection as done by e.g. ``flask`` working. - Standalone: Added missing hidden dependencies for ``Tkinter`` module, adding support for this to work properly. - Windows: Detecting the Python DLL and EXE used at compile time and preserving this information use during backend compilation. This should make sure we use the proper ones, and avoids hacks for specific Python variants, enhancing the support for AnaConda, WinPython, and CPython installations. - Windows: The ``--python-debug`` flag now properly detects if the run time is supporting things and error exits if it's not available. For a CPython3.5 installation, it will switch between debug and non-debug Python binaries and DLLs. - Standalone: Added plug-in for the ``Pwm`` package to properly combine it into a single file, suitable for distribution. - Standalone: Packages from standard library, e.g. ``xml`` now have proper ``__path__`` as a list and not as a string value, which breaks code of e.g. PyXML. `Issue#183 `__. - Standalone: Added missing dependency of ``twisted.protocols.tls``. `Issue#288 `__. - Python3.5: When finalizing coroutines that were not finished, a corruption of its reference count could happen under some circumstances. - Standalone: Added missing DLL dependency of the ``uuid`` module at run time, which uses ctypes to load it. New Features ------------ - Added support for AnaConda Python on this Linux. Both accelerated and standalone mode work now. `Issue#295 `__. - Added support for standalone mode on FreeBSD. `Issue#294 `__. - The plug-in framework was expanded with new features to allow addressing some specific issues. Cleanups -------- - Moved memory related stuff to dedicated utils package ``nuitka.utils.MemoryUsage`` as part of an effort to have more topical modules. - Plug-ins how have a dedicated module through which the core accesses the API, which was partially cleaned up. - No more "early" and "late" import detections for standalone mode. We now scan everything at the start. Summary ------- This release focused on expanding plugins. These were then used to enhance the success of standalone compatibility. Eventually this should lead to a finished and documented plug-in API, which will open up the Nuitka core to easier hacks and more user contribution for these topics. Nuitka Release 0.5.19 ===================== This release brings optimization improvements for dictionary using code. This is now lowering subscripts to dictionary accesses where possible and adds new code generation for known dictionary values. Besides this there is the usual range of bug fixes. Bug Fixes --------- - Fix, attribute assignments or deletions where the assigned value or the attribute source was statically raising crashed the compiler. - Fix, the order of evaluation during optimization was considered in the wrong order for attribute assignments source and value. - Windows: Fix, when ``g++`` is the path, it was not used automatically, but now it is. - Windows: Detect the 32 bits variant of MinGW64 too. - Python3.4: The finalize of compiled generators could corrupt reference counts for shared generator objects. Fixed in 0.5.18.1 already. - Python3.5: The finalize of compiled coroutines could corrupt reference counts for shared generator objects. Optimization ------------ - When a variable is known to have dictionary shape (assigned from a constant value, result of ``dict`` built-in, or a general dictionary creation), or the branch merge thereof, we lower subscripts from expecting mapping nodes to dictionary specific nodes. These generate more efficient code, and some are then known to not raise an exception. .. code-block:: python def someFunction(a,b): value = {a : b} value["c"] = 1 return value The above function is not yet fully optimized (dictionary key/value tracing is not yet finished), however it at least knows that no exception can raise from assigning ``value["c"]`` anymore and creates more efficient code for the typical ``result = {}`` functions. - The use of "logical" sharing during optimization has been replaced with checks for actual sharing. So closure variables that were written to in dead code no longer inhibit optimization of the then no more shared local variable. - Global variable traces are now faster to decide definite writes without need to check traces for this each time. Cleanups -------- - No more using "logical sharing" allowed to remove that function entirely. - Using "technical sharing" less often for decisions during optimization and instead rely more often on proper variable registry. - Connected variables with their global variable trace statically avoid the need to check in variable registry for it. - Removed old and mostly unused "assume unclear locals" indications, we use global variable traces for this now. Summary ------- This release aimed at dictionary tracing. As a first step, the value assign is now traced to have a dictionary shape, and this this then used to lower the operations which used to be normal subscript operations to mapping, but now can be more specific. Making use of the dictionary values knowledge, tracing keys and values is not yet inside the scope, but expected to follow. We got the first signs of type inference here, but to really take advantage, more specific shape tracing will be needed. Nuitka Release 0.5.18 ===================== This release mainly has a scalability focus. While there are few compatibility improvements, the larger goal has been to make Nuitka compilation and the final C compilation faster. Bug Fixes --------- - Compatibility: The nested arguments functions can now be called using their keyword arguments. .. code-block:: python def someFunction(a,(b,c)): return a, b, c someFunction(a = 1, **{".1" : (2,3)}) - Compatibility: Generators with Python3.4 or higher now also have a ``__del__`` attribute, and therefore properly participate in finalization. This should improve their interactions with garbage collection reference cycles, although no issues had been observed so far. - Windows: Was outputting command line arguments debug information at program start. `Issue#284 `__. Fixed in 0.5.17.1 already. Optimization ------------ - Code generated for parameter parsing is now a *lot* less verbose. Python level loops and conditionals to generate code for each variable has been replaced with C level generic code. This will speed up the backend compilation by a lot. - Function calls with constant arguments were speed up specifically, as their call is now fully prepared, and yet using less code. Variable arguments are also faster, and all defaulted arguments are also much faster. Method calls are not affected by these improvements though. - Nested argument functions now have a quick call entry point as well, making them faster to call too. - The ``slice`` built-in, and internal creation of slices (e.g. in re-formulations of Python3 slices as subscripts) cannot raise. `Issue#262 `__. - Standalone: Avoid inclusion of bytecode of ``unittest.test``, ``sqlite3.test``, ``distutils.test``, and ``ensurepip``. These are not needed, but simply bloat the amount of bytecode used on e.g. MacOS. `Issue#272 `__. - Speed up compilation with Nuitka itself by avoid to copying and constructing variable lists as much as possible using an always accurate variable registry. Cleanups -------- - Nested argument functions of Python2 are now re-formulated into a wrapping function that directly calls the actual function body with the unpacking of nested arguments done in nodes explicitly. This allows for better optimization and checks of these steps and potential in-lining of these functions too. - Unified slice object creation and built-in ``slice`` nodes, these were two distinct nodes before. - The code generation for all statement kinds is now done via dispatching from a dictionary instead of long ``elif`` chains. - Named nodes more often consistently, e.g. all loop related nodes start with ``Loop`` now, making them easier to group. - Parameter specifications got simplified to work without variables where it is possible. Organizational -------------- - Nuitka is now available on the social code platforms gitlab as well. Summary ------- Long standing weaknesses have been addressed in this release, also quite a few structural cleanups have been performed, e.g. strengthening the role of the variable registry to always be accurate, is groundlaying to further improvement of optimization. However, this release cycle was mostly dedicated to performance of the actual compilation, and more accurate information was needed to e.g. not search for information that should be instant. Upcoming releases will focus on usability issues and further optimization, it was nice however to see speedups of created code even from these scalability improvements. Nuitka Release 0.5.17 ===================== This release is a major feature release, as it adds full support for Python3.5 and its coroutines. In addition, in order to properly support coroutines, the generator implementation got enhanced. On top of that, there is the usual range of corrections. Bug Fixes --------- - Windows: Command line arguments that are unicode strings were not properly working. - Compatibility: Fix, only the code object attached to exceptions contained all variable names, but not the one of the function object. - Python3: Support for virtualenv on Windows was using non-portable code and therefore failing. `Issue#266 `__. - The tree displayed with ``--display-tree`` duplicated all functions and did not resolve source lines for functions. It also displayed unused functions, which is not helpful. - Generators with parameters leaked C level memory for each instance of them leading to memory bloat for long running programs that use a lot of generators. Fixed in 0.5.16.1 already. - Don't drop positional arguments when called with ``--run``, also make it an error if they are present without that option. New Features ------------ - Added full support for Python3.5, coroutines work now too. Optimization ------------ - Optimized frame access of generators to not use both a local frame variable and the frame object stored in the generator object itself. This gave about 1% speed up to setting them up. - Avoid having multiple code objects for functions that can raise and have local variables. Previously one code object would be used to create the function (with parameter variable names only) and when raising an exception, another one would be used (with all local variable names). Creating them both at start-up was wasteful and also needed two tuples to be created, thus more constants setup code. - The entry point for generators is now shared code instead of being generated for each one over and over. This should make things more cache local and also results in less generated C code. - When creating frame codes, avoid working with strings, but use proper emission for less memory churn during code generation. Organizational -------------- - Updated the key for the Debian/Ubuntu repositories to remain valid for 2 more years. - Added support for Fedora 23. - MinGW32 is no more supported, use MinGW64 in the 32 bits variant, which has less issues. Cleanups -------- - Detecting function type ahead of times, allows to handle generators different from normal functions immediately. - Massive removal of code duplication between normal functions and generator functions. The later are now normal functions creating generator objects, which makes them much more lightweight. - The ``return`` statement in generators is now immediately set to the proper node as opposed to doing this in variable closure phase only. We can now use the ahead knowledge of the function type. - The ``nonlocal`` statement is now immediately checked for syntax errors as opposed to doing that only in variable closure phase. - The name of contraction making functions is no longer skewed to empty, but the real thing instead. The code name is solved differently now. - The ``local_locals`` mode for function node was removed, it was always true ever since Python2 list contractions stop using pseudo functions. - The outline nodes allowed to provide a body when creating them, although creating that body required using the outline node already to create temporary variables. Removed that argument. - Removed PyLint false positive annotations no more needed for PyLint 1.5 and solved some TODOs. - Code objects are now mostly created from specs (not yet complete) which are attached and shared between statement frames and function creations nodes, in order to have less guess work to do. Tests ----- - Added the CPython3.5 test suite. - Updated generated doctests to fix typos and use common code in all CPython test suites. Summary ------- This release continues to address technical debt. Adding support for Python3.5 was the major driving force, while at the same time removing obstacles to the changes that were needed for coroutine support. With Python3.5 sorted out, it will be time to focus on general optimization again, but there is more technical debt related to classes, so the cleanup has to continue. Nuitka Release 0.5.16 ===================== This is a maintenance release, largely intended to put out improved support for new platforms and minor corrections. It should improve the speed for standalone mode, and compilation in general for some use cases, but this is mostly to clean up open ends. Bug Fixes --------- - Fix, the ``len`` built-in could give false values for dictionary and set creations with the same element. .. code-block:: python # This was falsely optimized to 2 even if "a is b and a == b" was true. len({a, b}) - Python: Fix, the ``gi_running`` attribute of generators is no longer an ``int``, but ``bool`` instead. - Python3: Fix, the ``int`` built-in with two arguments, value and base, raised ``UnicodeDecodeError`` instead of ``ValueError`` for illegal bytes given as value. - Python3: Using ``tokenize.open`` to read source code, instead of reading manually and decoding from ``tokenize.detect_encoding``, this handles corner cases more compatible. - Fix, the PyLint warnings plug-in could crash in some cases, make sure it's more robust. - Windows: Fix, the combination of AnaConda Python, MinGW 64 bits and mere acceleration was not working. `Issue#254 `__. - Standalone: Preserve not only namespace packages created by ``.pth`` files, but also make the imports done by them. This makes it more compatible with uses of it in Fedora 22. - Standalone: The extension modules could be duplicated, turned this into an error and cache finding them during compile time and during early import resolution to avoid duplication. - Standalone: Handle "not found" from ``ldd`` output, on some systems not all the libraries wanted are accessible for every library. - Python3.5: Fixed support for namespace packages, these were not yet working for that version yet. - Python3.5: Fixes lack of support for unpacking in normal ``tuple``, ``list``, and ``set`` creations. .. code-block:: python [*a] # this has become legal in 3.5 and now works too. Now also gives compatible ``SyntaxError`` for earlier versions. Python2 was good already. - Python3.5: Fix, need to reduce compiled functions to ``__qualname__`` value, rather than just ``__name__`` or else pickling methods doesn't work. - Python3.5: Fix, added ``gi_yieldfrom`` attribute to generator objects. - Windows: Fixed harmless warnings for Visual Studio 2015 in ``--debug`` mode. Optimization ------------ - Re-formulate ``exec`` and ``eval`` to default to ``globals()`` as the default for the locals dictionary in modules. - The ``try`` node was making a description of nodes moved to the outside when shrinking its scope, which was using a lot of time, just to not be output, now these can be postponed. - Refactored how freezing of bytecode works. Uncompiled modules are now explicit nodes too, and in the registry. We only have one or the other of it, avoiding to compile both. Tests ----- - When ``strace`` or ``dtruss`` are not found, given proper error message, so people know what to do. - The doc tests extracted and then generated for CPython3 test suites were not printing the expressions of the doc test, leading to largely decreased test coverage here. - The CPython 3.4 test suite is now also using common runner code, and avoids ignoring all Nuitka warnings, instead more white listing was added. - Started to run CPython 3.5 test suite almost completely, but coroutines are blocking some parts of that, so these tests that use this feature are currently skipped. - Removed more CPython tests that access the network and are generally useless to testing Nuitka. - When comparing outputs, normalize typical temporary file names used on posix systems. - Coverage tests have made some progress, and some changes were made due to its results. - Added test to cover too complex code module of ``idna`` module. - Added Python3.5 only test for unpacking variants. Cleanups -------- - Prepare plug-in interface to allow suppression of import warnings to access the node doing it, making the import node is accessible. - Have dedicated class function body object, which is a specialization of the function body node base class. This allowed removing class specific code from that class. - The use of "win_target" as a scons parameter was useless. Make more consistent use of it as a flag indicator in the scons file. - Compiled types were mixing uses of ``compiled_`` prefixes, something with a space, sometimes with an underscore. Organizational -------------- - Improved support for Python3.5 missing compatibility with new language features. - Updated the Developer Manual with changes that SSA is now a fact. - Added Python3.5 Windows MSI downloads. - Added repository for Ubuntu Wily (15.10) for download. Removed Ubuntu Utopic package download, no longer supported by Ubuntu. - Added repository with RPM packages for Fedora 22. Summary ------- So this release is mostly to lower the technical debt incurred that holds it back from supporting making more interesting changes. Upcoming releases may have continue that trend for some time. This release is mostly about catching up with Python3.5, to make sure we did not miss anything important. The new function body variants will make it easier to implement coroutines, and help with optimization and compatibility problems that remain for Python3 classes. Ultimately it will be nice to require a lot less checks for when function in-line is going to be acceptable. Also code generation will need a continued push to use the new structure in preparation for making type specific code generation a reality. Nuitka Release 0.5.15 ===================== This release enables SSA based optimization, the huge leap, not so much in terms of actual performance increase, but for now making the things possible that will allow it. This has been in the making literally for years. Over and over, there was just "one more thing" needed. But now it's there. The release includes much stuff, and there is a perspective on the open tasks in the summary, but first out to the many details. Bug Fixes --------- - Standalone: Added implicit import for ``reportlab`` package configuration dynamic import. Fixed in 0.5.14.1 already. - Standalone: Fix, compilation of the ``ctypes`` module could happen for some import patterns, and then prevented the distribution to contain all necessary libraries. Now it is made sure to not include compiled and frozen form both. `Issue#241 `__. Fixed in 0.5.14.1 already. - Fix, compilation for conditional statements where the boolean check on the condition cannot raise, could fail compilation. `Issue#240 `__. Fixed in 0.5.14.2 already. - Fix, the ``__import__`` built-in was making static optimization assuming compile time constants to be strings, which in the error case they are not, which was crashing the compiler. `Issue#240 `__. .. code-block:: python __import__(("some.module",)) # tuples don't work This error became only apparent, because now in some cases, Nuitka forward propagates values. - Windows: Fix, when installing Python2 only for the user, the detection of it via registry failed as it was only searching system key. This was `a github pull request `__. Fixed in 0.5.14.3 already. - Some modules have extremely complex expressions requiring too deep recursion to work on all platforms. These modules are now included entirely as bytecode fallback. `Issue#240 `__. - The standard library may contain broken code due to installation mistakes. We have to ignore their ``SyntaxError``. `Issue#244 `__. - Fix, pickling compiled methods was failing with the wrong kind of error, because they should not implement ``__reduce__``, but only ``__deepcopy__``. `Issue#219 `__. - Fix, when running under ``wine``, the check for scons binary was fooled by existance of ``/usr/bin/scons``. `Issue#251 `__. New Features ------------ - Added experimental support for Python3.5, coroutines don't work yet, but it works perfectly as a 3.4 replacement. - Added experimental Nuitka plug-in framework, and use it for the packaging of Qt plugins in standalone mode. The API is not yet stable nor polished. - New option ``--debugger`` that makes ``--run`` execute directly in ``gdb`` and gives a stack trace on crash. - New option ``--profile`` executes compiled binary and outputs measured performance with ``vmprof``. This is work in progress and not functional yet. - Started work on ``--graph`` to render the SSA state into diagrams. This is work in progress and not functional yet. - Plug-in framework added. Not yet ready for users. Working ``PyQt4`` and ``PyQt5`` plug-in support. Experimental Windows ``multiprocessing`` support. Experimental PyLint warnings disable support. More to come. - Added support for AnaConda accelerated mode on MacOS by modifying the rpath to the Python DLL. - Added experimental support for ``multiprocessing`` on Windows, which needs money patching of the module to support compiled methods. Optimization ------------ - The SSA analysis is now enabled by default, eliminating variables that are not shared, and can be forward propagated. This is currently limited mostly to compile time constants, but things won't remain that way. - Code generation for many constructs now takes into account if a specific operation can raise or not. If e.g. an attribute look-up is known to not raise, then that is now decided by the node the looked is done to, and then more often can determine this, or even directly the value. - Calls to C-API that we know cannot raise, no longer check, but merely assert the result. - For attribute look-up and other operations that might be known to not raise, we now only assert that it succeeds. - Built-in loop-ups cannot fail, merely assert that. - Creation of built-in exceptions never raises, merely assert that too. - More Python operation slots now have their own computations and some of these gained overloads for more compile time constant optimization. - When taking an iterator cannot raise, this is now detected more often. - The ``try``/``finally`` construct is now represented by duplicating the final block into all kinds of handlers (``break``, ``continue``, ``return``, or ``except``) and optimized separately. This allows for SSA to trace values more correctly. - The ``hash`` built-in now has dedicated node and code generation too. This is mostly intended to represent the side effects of dictionary look-up, but gives more compact and faster code too. - Type ``type`` built-in cannot raise and has no side effect. - Speed improvement for in-place float operations for ``+=`` and ``*=``, as these will be common cases. Tests ----- - Made the construct based testing executable with Python3. - Removed warnings using the new PyLint warnings plug-in for the reflected test. Nuitka now uses the PyLint annotations to not warn. Also do not go into PyQt for reflected test, not needed. Many Python3 improvements for cases where there are differences to report. - The optimization tests no longer use 2to3 anymore, made the tests portable to all versions. - Checked more in-place operations for speed. Organizational -------------- - Many improvements to the coverage taking. We can hope to see public data from this, some improvements were triggered from this already, but full runs of the test suite with coverage data collection are yet to be done. Summary ------- The release includes many important new directions. Coverage analysis will be important to remain certain of test coverage of Nuitka itself. This is mostly done, but needs more work to complete. Then the graphing surely will help us to debug and understand code examples. So instead of tracing, and reading stuff, we should visualize things, to more clearly see, how things evolve under optimization iteration, and where exactly one thing goes wrong. This will be improved as it proves necessary to do just that. So far, this has been rare. Expect this to become end user capable with time. If only to allow you to understand why Nuitka won't optimize code of yours, and what change of Nuitka it will need to improve. The comparative performance benchmarking is clearly the most important thing to have for users. It deserves to be the top priority. Thanks to the PyPy tool ``vmprof``, we may already be there on the data taking side, but the presenting and correlation part, is still open and a fair bit of work. It will be most important to empower users to make competent performance bug reports, now that Nuitka enters the phase, where these things matter. As this is a lot of ground to cover. More than ever. We can make this compiler, but only if you help, it will arrive in your life time. Nuitka Release 0.5.14 ===================== This release is an intermediate step towards value propagation, which is not considered ready for stable release yet. The major point is the elimination of the ``try``/``finally`` expressions, as they are problems to SSA. The ``try``/``finally`` statement change is delayed. There are also a lot of bug fixes, and enhancements to code generation, as well as major cleanups of code base. Bug Fixes --------- - Python3: Added support assignments trailing star assignment. .. code-block:: python *a, b = 1, 2 This raised ``ValueError`` before. - Python3: Properly detect illegal double star assignments. .. code-block:: python *a, *b = c - Python3: Properly detect the syntax error to star assign from non-tuple/list. .. code-block:: python *a = 1 - Python3.4: Fixed a crash of the binary when copying dictionaries with split tables received as star arguments. - Python3: Fixed reference loss, when using ``raise a from b`` where ``b`` was an exception instance. Fixed in 0.5.13.8 already. - Windows: Fix, the flag ``--disable-windows-console`` was not properly handled for MinGW32 run time resulting in a crash. - Python2.7.10: Was not recognizing this as a 2.7.x variant and therefore not applying minor version compatibility levels properly. - Fix, when choosing to have frozen source references, code objects were not use the same value as ``__file__`` did for its filename. - Fix, when re-executing itself to drop the ``site`` module, make sure we find the same file again, and not according to the ``PYTHONPATH`` changes coming from it. `Issue#223 `__. Fixed in 0.5.13.4 already. - Enhanced code generation for ``del variable`` statements, where it's clear that the value must be assigned. - When pressing CTRL-C, the stack traces from both Nuitka and Scons were given, we now avoid the one from Scons. - Fix, the dump from ``--xml`` no longer contains functions that have become unused during analysis. - Standalone: Creating or running programs from inside unicode paths was not working on Windows. `Issue#231 `__ `Issue#229 `__ and. Fixed in 0.5.13.7 already. - Namespace package support was not yet complete, importing the parent of a package was still failing. `Issue#230 `__. Fixed in 0.5.13.7 already. - Python2.6: Compatibility for exception check messages enhanced with newest minor releases. - Compatibility: The ``NameError`` in classes needs to say ``global name`` and not just ``name`` too. - Python3: Fixed creation of XML representation, now done without ``lxml`` as it doesn't support needed features on that version. Fixed in 0.5.13.5 already. - Python2: Fix, when creating code for the largest negative constant to still fit into ``int``, that was only working in the main module. `Issue#228 `__. Fixed in 0.5.13.5 already. - Compatibility: The ``print`` statement raised an assertion on unicode objects that could not be encoded with ``ascii`` codec. New Features ------------ - Added support for Windows 10. - Followed changes for Python 3.5 beta 2. Still only usable as a Python 3.4 replacement, no new features. - Using a self compiled Python running from the source tree is now supported. - Added support for ``AnaConda`` Python distribution. As it doesn't install the Python DLL, we copy it along for acceleration mode. - Added support for Visual Studio 2015. `Issue#222 `__. Fixed in 0.5.13.3 already. - Added support for self compiled Python versions running from build tree, this is intended to help debug things on Windows. Optimization ------------ - Function in-lining is now present in the code, but still disabled, because it needs more changes in other areas, before we can generally do it. - Trivial outlines, result of re-formulations or function in-lining, are now in-lined, in case they just return an expression. - The re-formulation for ``or`` and ``and`` has been giving up, eliminating the use of a ``try``/``finally`` expression, at the cost of dedicated boolean nodes and code generation for these. This saves around 8% of compile time memory for Nuitka, and allows for faster and more complete optimization, and gets rid of a complicated structure for analysis. - When a frame is used in an exception, its locals are detached. This was done more often than necessary and even for frames that are not necessary our own ones. This will speed up some exception cases. - When the default arguments, or the keyword default arguments (Python3) or the annotations (Python3) were raising an exception, the function definition is now replaced with the exception, saving a code generation. This happens frequently with Python2/Python3 compatible code guarded by version checks. - The SSA analysis for loops now properly traces "break" statement situations and merges the post-loop situation from all of them. This significantly allows for and improves optimization of code following the loop. - The SSA analysis of ``try``/``finally`` statements has been greatly enhanced. The handler for ``finally`` is now optimized for exception raise and no exception raise individually, as well as for ``break``, ``continue`` and ``return`` in the tried code. The SSA analysis for after the statement is now the result of merging these different cases, should they not abort. - The code generation for `del` statements is now taking advantage should there be definite knowledge of previous value. This speed them up slightly. - The SSA analysis of `del` statements now properly decided if the statement can raise or not, allowing for more optimization. - For list contractions, the re-formulation was enhanced using the new outline construct instead of a pseudo function, leading to better analysis and code generation. - Comparison chains are now re-formulated into outlines too, allowing for better analysis of them. - Exceptions raised in function creations, e.g. in default values, are now propagated, eliminating the function's code. This happens most often with Python2/Python3 in branches. On the other hand, function creations that cannot are also annotated now. - Closure variables that become unreferenced outside of the function become normal variables leading to better tracing and code generation for them. - Function creations cannot raise except their defaults, keyword defaults or annotations do. - Built-in references can now be converted to strings at compile time, e.g. when printed. Organizational -------------- - Removed gitorious mirror of the git repository, they shut down. - Make it more clear in the documentation that Python2 is needed at compile time to create Python3 executables. Cleanups -------- - Moved more parts of code generation to their own modules, and used registry for code generation for more expression kinds. - Unified ``try``/``except`` and ``try``/``finally`` into a single construct that handles both through ``try``/``except``/``break``/``continue``/``return`` semantics. Finally is now solved via duplicating the handler into cases necessary. No longer are nodes annotated with information if they need to publish the exception or not, this is now all done with the dedicated nodes. - The ``try``/``finally`` expressions have been replaced with outline function bodies, that instead of side effect statements, are more like functions with return values, allowing for easier analysis and dedicated code generation of much lower complexity. - No more "tolerant" flag for release nodes, we now decide this fully based on SSA information. - Added helper for assertions that code flow does not reach certain positions, e.g. a function must return or raise, aborting statements do not continue and so on. - To keep cloning of code parts as simple as possible, the limited use of ``makeCloneAt`` has been changed to a new ``makeClone`` which produces identical copies, which is what we always do. And a generic cloning based on "details" has been added, requiring to make constructor arguments and details complete and consistent. - The re-formulation code helpers have been improved to be more convenient at creating nodes. - The old ``nuitka.codegen`` module ``Generator`` was still used for many things. These now all got moved to appropriate code generation modules, and their users got updated, also moving some code generator functions in the process. - The module ``nuitka.codegen.CodeTemplates`` got replaces with direct uses of the proper topic module from ``nuitka.codegen.templates``, with some more added, and their names harmonized to be more easily recognizable. - Added more assertions to the generated code, to aid bug finding. - The autoformat now sorts pylint markups for increased consistency. - Releases no longer have a ``tolerant`` flag, this was not needed anymore as we use SSA. - Handle CTRL-C in scons code preventing per job messages that are not helpful and avoid tracebacks from scons, also remove more unused tools like ``rpm`` from out in-line copy. Tests ----- - Added the CPython3.4 test suite. - The CPython3.2, CPython3.3, and CPython3.4 test suite now run with Python2 giving the same errors. Previously there were a few specific errors, some with line numbers, some with different ``SyntaxError`` be raised, due to different order of checks. This increases the coverage of the exception raising tests somewhat. - Also the CPython3.x test suites now all pass with debug Python, as does the CPython 2.6 test suite with 2.6 now. - Added tests to cover all forms of unpacking assignments supported in Python3, to be sure there are no other errors unknown to us. - Started to document the reference count tests, and to make it more robust against SSA optimization. This will take some time and is work in progress. - Made the compile library test robust against modules that raise a syntax error, checking that Nuitka does the same. - Refined more tests to be directly execuable with Python3, this is an ongoing effort. Summary ------- This release is clearly major. It represents a huge step forward for Nuitka as it improves nearly every aspect of code generation and analysis. Removing the ``try``/``finally`` expression nodes proved to be necessary in order to even have the correct SSA in their cases. Very important optimization was blocked by it. Going forward, the ``try``/``finally`` statements will be removed and dead variable elimination will happen, which then will give function inlining. This is expected to happen in one of the next releases. This release is a consolidation of 8 hotfix releases, and many refactorings needed towards the next big step, which might also break things, and for that reason is going to get its own release cycle. Nuitka Release 0.5.13 ===================== This release contains the first use of SSA for value propagation and massive amounts of bug fixes and optimization. Some of the bugs that were delivered as hotfixes, were only revealed when doing the value propagation as they still could apply to real code. Bug Fixes --------- - Fix, relative imports in packages were not working with absolute imports enabled via future flags. Fixed in 0.5.12.1 already. - Loops were not properly degrading knowledge from inside the loop at loop exit, and therefore this could have lead missing checks and releases in code generation for cases, for ``del`` statements in the loop body. Fixed in 0.5.12.1 already. - The ``or`` and ``and`` re-formulation could trigger false assertions, due to early releases for compatibility. Fixed in 0.5.12.1 already. - Fix, optimizion of calls of constant objects (always an exception), crashed the compiler. This corrects `Issue#202 `__. Fixed in 0.5.12.2 already. - Standalone: Added support for ``site.py`` installations with a leading ``def`` or ``class`` statement, which is defeating our attempt to patch ``__file__`` for it. This corrects `Issue#189 `__. - Compatibility: In full compatibility mode, the tracebacks of ``or`` and ``and`` expressions are now as wrong as they are in CPython. Does not apply to ``--improved`` mode. - Standalone: Added missing dependency on ``QtGui`` by ``QtWidgets`` for PyQt5. - MacOS: Improved parsing of ``otool`` output to avoid duplicate entries, which can also be entirely wrong in the case of Qt plugins at least. - Avoid relative paths for main program with file reference mode ``original``, as it otherwise changes as the file moves. - MinGW: The created modules depended on MinGW to be in ``PATH`` for their usage. This is no longer necessary, as we now link these libraries statically for modules too. - Windows: For modules, the option ``--run`` to immediately load the modules had been broken for a while. - Standalone: Ignore Windows DLLs that were attempted to be loaded, but then failed to load. This happens e.g. when both PySide and PyQt are installed, and could cause the dreaded conflicting DLLs message. The DLL loaded in error is now ignored, which avoids this. - MinGW: The resource file used might be empty, in which case it doesn't get created, avoiding an error due to that. - MinGW: Modules can now be created again. The run time relative code uses an API that is WinXP only, and MinGW failed to find it without guidance. Optimization ------------ - Make direct calls out of called function creations. Initially this applies to lambda functions only, but it's expected to become common place in coming releases. This is now 20x faster than CPython. .. code-block:: python # Nuitka avoids creating a function object, parsing function arguments: (lambda x:x)(something) - Propagate assignments from non-mutable constants forward based on SSA information. This is the first step of using SSA for real compile time optimization. - Specialized the creation of call nodes at creation, avoiding to have all kinds be the most flexible form (keyword and plain arguments), but instead only what kind of call they really are. This saves lots of memory, and makes the tree faster to visit. - Added support for optimizing the ``slice`` built-in with compile time constant arguments to constants. The re-formulation for slices in Python3 uses these a lot. And the lack of this optimization prevented a bunch of optimization in this area. For Python2 the built-in is optimized too, but not as important probably. - Added support for optimizing ``isinstance`` calls with compile time constant arguments. This avoids static exception raises in the ``exec`` re-formulation which tests for ``file`` type, and then optimization couldn't tell that a ``str`` is not a ``file`` instance. Now it can. - Lower in-place operations on immutable types to normal operations. This will allow to compile time compute these more accurately. - The re-formulation of loops puts the loop condition as a conditional statement with break. The ``not`` that needs to apply was only added in later optimization, leading to unnecessary compile time efforts. - Removed per variable trace visit from optimization, removing useless code and compile time overhead. We are going to optimize things by making decision in assignment and reference nodes based on forward looking statements using the last trace collection. New Features ------------ - Added experimental support for Python 3.5, which seems to be passing the test suites just fine. The new ``@`` matrix multiplicator operators are not yet supported though. - Added support for patching source on the fly. This is used to work around a (now fixed) issue with ``numexpr.cpuinfo`` making type checks with the ``is`` operation, about the only thing we cannot detect. Organizational -------------- - Added repository for Ubuntu Vivid (15.04) for download. Removed Ubuntu Saucy and Ubuntu Raring package downloads, these are no longer supported by Ubuntu. - Added repository for Debian Stretch, after Jessie release. - Make it more clear in the documentation that in order to compile Python3, a Python2 is needed to execute Scons, but that the end result is a Python3 binary. - The PyLint checker tool now can operate on directories given on the command line, and whitelists an error that is Windows only. Cleanups -------- - Split up standalone code further, moving ``depends.exe`` handling to a separate module. - Reduced code complexity of scons interface. - Cleaned up where trace collection is being done. It was partially still done inside the collection itself instead in the owner. - In case of conflicting DLLs for standalone mode, these are now output with nicer formatting, that makes it easy to recognize what is going on. - Moved code to fetch ``depends.exe`` to dedicated module, so it's not as much in the way of standalone code. Tests ----- - Made ``BuiltinsTest`` directly executable with Python3. - Added construct test to demonstrate the speed up of direct lambda calls. - The deletion of ``@test`` for the CPython test suite is more robust now, esp. on Windows, the symbolic links are now handled. - Added test to cover ``or`` usage with in-place assignment. - Cover local relative ``import from .`` with ``absolute_import`` future flag enabled. - Again, more basic tests are now directly executable with Python3. Summary ------- This release is major due to amount of ground covered. The reduction in memory usage of Nuitka itself (the C++ compiler will still use much memory) is very massive and an important aspect of scalability too. Then the SSA changes are truly the first sign of major improvements to come. In their current form, without eliminating dead assignments, the full advantage is not taken yet, but the next releases will do this, and that's a major milestone to Nuitka. The other optimization mostly stem from looking at things closer, and trying to work towards function in-lining, for which we are making a lot of progress now. Nuitka Release 0.5.12 ===================== This release contains massive amounts of corrections for long standing issues in the import recursion mechanism, as well as for standalone issues now visible after the ``__file__`` and ``__path__`` values have changed to become runtime dependent values. Bug Fixes --------- - Fix, the ``__path__`` attribute for packages was still the original filename's directory, even in file reference mode was ``runtime``. - The use of ``runtime`` as default file reference mode for executables, even if not in standalone mode, was making acceleration harder than necessary. Changed to ``original`` for that case. Fixed in 0.5.11.1 already. - The constant value for the smallest ``int`` that is not yet a ``long`` is created using ``1`` due to C compiler limitations, but ``1`` was not yet initialized properly, if this was a global constant, i.e. used in multiple modules. Fixed in 0.5.11.2 already. - Standalone: Recent fixes around ``__path__`` revealed issues with PyWin32, where modules from ``win32com.shell`` were not properly recursed to. Fixed in 0.5.11.2 already. - The importing of modules with the same name as a built-in module inside a package falsely assumed these were the built-ins which need not exist, and then didn't recurse into them. This affected standalone mode the most, as the module was then missing entirely. This corrects `Issue#178 `__. .. code-block:: python # Inside "x.y" module: import x.y.exceptions - Similarily, the importing of modules with the same name as standard library modules could go wrong. This corrects `Issue#184 `__. .. code-block:: python # Inside "x.y" module: import x.y.types - Importing modules on Windows and MacOS was not properly checking the checking the case, making it associate wrong modules from files with mismatching case. This corrects `Issue#188 `__. - Standalone: Importing with ``from __future__ import absolute_import`` would prefer relative imports still. This corrects `Issue#187 `__. - Python3: Code generation for ``try``/``return expr``/``finally`` could loose exceptions when ``expr`` raised an exception, leading to a ``RuntimeError`` for ``NULL`` return value. The real exception was lost. - Lambda expressions that were directly called with star arguments caused the compiler to crash. .. code-block:: python (lambda *args:args)(*args) # was crashing Nuitka New Optimization ---------------- - Focusing on compile time memory usage, cyclic dependencies of trace merges that prevented them from being released, even when replaced were removed. - More memory efficient updating of global SSA traces, reducing memory usage during optimization by ca. 50%. - Code paths that cannot and therefore must not happen are now more clearly indicated to the backend compiler, allowing for slightly better code to be generated by it, as it can tell that certain code flows need not be merged. New Features ------------ - Standalone: On systems, where ``.pth`` files inject Python packages at launch, these are now detected, and taking into account. Previously Nuitka did not recognize them, due to lack of ``__init__.py`` files. These are mostly pip installations of e.g. ``zope.interface``. - Added option ``--explain-imports`` to debug the import resolution code of Nuitka. - Added options ``--show-memory`` to display the amount of memory used in total and how it's spread across the different node types during compilation. - The option ``--trace-execution`` now also covers early program initialisation before any Python code runs, to ease finding bugs in this domain as well. Organizational -------------- - Changed default for file reference mode to ``original`` unless standalone or module mode are used. For mere acceleration, breaking the reading of data files from ``__file__`` is useless. - Added check that the in-line copy of scons is not run with Python3, which is not supported. Nuitka works fine with Python3, but a Python2 is required to execute scons. - Discover more kinds of Python2 installations on Linux/MacOS installations. - Added instructions for MacOS to the download page. Cleanups -------- - Moved ``oset`` and ``odict`` modules which provide ordered sets and dictionaries into a new package ``nuitka.container`` to clean up the top level scope. - Moved ``SyntaxErrors`` to ``nuitka.tree`` package, where it is used to format error messages. - Moved ``nuitka.Utils`` package to ``nuitka.utils.Utils`` creating a whole package for utils, so as to better structure them for their purpose. Summary ------- This release is a major maintenance release. Support for namespace modules injected by ``*.pth`` is a major step for new compatibility. The import logic improvements expand the ability of standalone mode widely. Many more use cases will now work out of the box, and less errors will be found on case insensitive systems. There is aside of memory issues, no new optimization though as many of these improvements could not be delivered as hotfixes (too invasive code changes), and should be out to the users as a stable release. Real optimization changes have been postponed to be next release. Nuitka Release 0.5.11 ===================== The last release represented a significant change and introduced a few regressions, which got addressed with hot fix releases. But it also had a focus on cleaning up open optimization issues that were postponed in the last release. New Features ------------ - The filenames of source files as found in the ``__file__`` attribute are now made relative for all modes, not just standalone mode. This makes it possible to put data files along side compiled modules in a deployment. This solves `Issue#170 `__. Bug Fixes --------- - Local functions that reference themselves were not released. They now are. .. code-block:: python def someFunction(): def f(): f() # referencing 'f' in 'f' caused the garbage collection to fail. Recent changes to code generation attached closure variable values to the function object, so now they can be properly visited. This corrects `Issue#45 `__. Fixed in 0.5.10.1 already. - Python2.6: The complex constants with real or imaginary parts ``-0.0`` were collapsed with constants of value ``0.0``. This became more evident after we started to optimize the ``complex`` built-in. Fixed in 0.5.10.1 already. .. code-block:: python complex(0.0, 0.0) complex(-0.0, -0.0) # Could be confused with the above. - Complex call helpers could leak references to their arguments. This was a regression. Fixed in 0.5.10.1 already. - Parameter variables offered as closure variables were not properly released, only the cell object was, but not the value. This was a regression. Fixed in 0.5.10.1 already. - Compatibility: The exception type given when accessing local variable values not initialized in a closure taking function, needs to be ``NameError`` and ``UnboundLocalError`` for accesses in the providing function. Fixed in 0.5.10.1 already. - Fix support for "venv" on systems, where the system Python uses symbolic links too. This is the case on at least on Mageia Linux. Fixed in 0.5.10.2 already. - Python3.4: On systems where ``long`` and ``Py_ssize_t`` are different (e.g. Win64) iterators could be corrupted if used by uncompiled Python code. Fixed in 0.5.10.2 already. - Fix, generator objects didn't release weak references to them properly. Fixed in 0.5.10.2 already. - Compatiblity: The ``__closure__`` attributes of functions was so far not supported, and rarely missing. Recent changes made it easy to expose, so now it was added. This corrects `Issue#45 `__. - MacOS: A linker warning about deprecated linker option ``-s`` was solved by removing the option. - Compatibility: Nuitka was enforcing that the ``__doc__`` attribute to be a string object, and gave a misleading error message. This check must not be done though, ``__doc__`` can be any type in Python. This corrects `Issue#177 `__. New Optimization ---------------- - Variables that need not be shared, because the uses in closure taking functions were eliminated, no longer use cell objects. - The ``try``/``except`` and ``try``/``finally`` statements now both have actual merging for SSA, allowing for better optimization of code behind it. .. code-block:: python def f(): try: a = something() except: return 2 # Since the above exception handling cannot continue the code flow, # we do not have to invalidate the trace of "a", and e.g. do not have # to generate code to check if it's assigned. return a Since ``try``/``finally`` is used in almost all re-formulations of complex Python constructs this is improving SSA application widely. The uses of ``try``/``except`` in user code will no longer degrade optimization and code generation efficiency as much as they did. - The ``try``/``except`` statement now reduces the scope of tried block if possible. When no statement raised, already the handling was removed, but leading and trailing statements that cannot raise, were not considered. .. code-block:: python def f(): try: b = 1 a = something() c = 1 except: return 2 This is now optimized to. .. code-block:: python def f(): b = 1 try: a = something() except: return 2 c = 1 The impact may on execution speed may be marginal, but it is definitely going to improve the branch merging to be added later. Note that ``c`` can only be optimized, because the exception handler is aborting, otherwise it would change behaviour. - The creation of code objects for standalone mode and now all code objects was creating a distinct filename object for every function in a module, despite them being same content. This was wasteful for module loading. Now it's done only once. Also, when having multiple modules, the code to build the run time filename used for code objects, was calling import logic, and doing lookups to find ``os.path.join`` again and again. These are now cached, speeding up the use of many modules as well. Cleanups -------- - Nuitka used to have "variable usage profiles" and still used them to decide if a global variable is written to, in which case, it stays away from doing optimization of it to built-in lookups, and later calls. The have been replaced by "global variable traces", which collect the traces to a variable across all modules and functions. While this is now only a replacement, and getting rid of old code, and basing on SSA, later it will also allow to become more correct and more optimized. - The standalone now queries its hidden dependencies from a plugin framework, which will become an interface to Nuitka internals in the future. Testing ------- - The use of deep hashing of constants allows us to check if constants become mutated during the run-time of a program. This allows to discover corruption should we encounter it. - The tests of CPython are now also run with Python in debug mode, but only on Linux, enhancing reference leak coverage. - The CPython test parts which had been disabled due to reference cycles involving compiled functions, or usage of ``__closure__`` attribute, were reactivated. Organizational -------------- - Since Google Code has shutdown, it has been removed from the Nuitka git mirrors. Summary ------- This release brings exciting new optimization with the focus on the ``try`` constructs, now being done more optimal. It is also a maintenance release, bringing out compatibility improvements, and important bug fixes, and important usability features for the deployment of modules and packages, that further expand the use cases of Nuitka. The git flow had to be applied this time to get out fixes for regression bug fixes, that the big change of the last release brought, so this is also to consolidate these and the other corrections into a full release before making more invasive changes. The cleanups are leading the way to expanded SSA applied to global variable and shared variable values as well. Already the built-in detect is now based on global SSA information, which was an important step ahead. Nuitka Release 0.5.10 ===================== This release has a focus on code generation optimization. Doing major changes away from "C++-ish" code to "C-ish" code, many constructs are now faster or got looked at and optimized. Bug Fixes --------- - Compatibility: The variable name in locals for the iterator provided to the generator expression should be ``.0``, now it is. - Generators could leak frames until program exit, these are now properly freed immediately. New Optimization ---------------- - Faster exception save and restore functions that might be in-lined by the backend C compiler. - Faster error checks for many operations, where these errors are expected, e.g. instance attribute lookups. - Do not create traceback and locals dictionary for frame when ``StopIteration`` or ``GeneratorExit`` are raised. These tracebacks were wasted, as they were immediately released afterwards. - Closure variables to functions and parameters of generator functions are now attached to the function and generator objects. - The creation of functions with closure taking was accelerated. - The creation and destruction of generator objects was accelerated. - The re-formulation for in-place assignments got simplified and got faster doing so. - In-place operations of ``str`` were always copying the string, even if was not necessary. This corrects `Issue#124 `__. .. code-block:: python a += b # Was not re-using the storage of "a" in case of strings - Python2: Additions of ``int`` for Python2 are now even faster. - Access to local variable values got slightly accelerated at the expense of closure variables. - Added support for optimizing the ``complex`` built-in. - Removing unused temporary and local variables as a result of optimization, these previously still allocated storage. Cleanup ------- - The use of C++ classes for variable objects was removed. Closure variables are now attached as ``PyCellObject`` to the function objects owning them. - The use of C++ context classes for closure taking and generator parameters has been replaced with attaching values directly to functions and generator objects. - The indentation of code template instantiations spanning multiple was not in all cases proper. We were using emission objects that handle it new lines in code and mere ``list`` objects, that don't handle them in mixed forms. Now only the emission objects are used. - Some templates with C++ helper functions that had no variables got changed to be properly formatted templates. - The internal API for handling of exceptions is now more consistent and used more efficiently. - The printing helpers got cleaned up and moved to static code, removing any need for forward declaration. - The use of ``INCREASE_REFCOUNT_X`` was removed, it got replaced with proper ``Py_XINCREF`` usages. The function was once required before "C-ish" lifted the need to do everything in one function call. - The use of ``INCREASE_REFCOUNT`` got reduced. See above for why that is any good. The idea is that ``Py_INCREF`` must be good enough, and that we want to avoid the C function it was, even if in-lined. - The ``assertObject`` function that checks if an object is not ``NULL`` and has positive reference count, i.e. is sane, got turned into a preprocessor macro. - Deep hashes of constant values created in ``--debug`` mode, which cover also mutable values, and attempt to depend on actual content. These are checked at program exit for corruption. This may help uncover bugs. Organizational -------------- - Speedcenter has been enhanced with better graphing and has more benchmarks now. More work will be needed to make it useful. - Updates to the Developer Manual, reflecting the current near finished state of "C-ish" code generation. Tests ----- - New reference count tests to cover generator expressions and their usage got added. - Many new construct based tests got added, these will be used for performance graphing, and serve as micro benchmarks now. - Again, more basic tests are directly executable with Python3. Summary ------- This is the next evolution of "C-ish" coming to pass. The use of C++ has for all practical purposes vanished. It will remain an ongoing activity to clear that up and become real C. The C++ classes were a huge road block to many things, that now will become simpler. One example of these were in-place operations, which now can be dealt with easily. Also, lots of polishing and tweaking was done while adding construct benchmarks that were made to check the impact of these changes. Here, generators probably stand out the most, as some of the missed optimization got revealed and then addressed. Their speed increases will be visible to some programs that depend a lot on generators. This release is clearly major in that the most important issues got addressed, future releases will provide more tuning and completeness, but structurally the "C-ish" migration has succeeded, and now we can reap the benefits in the coming releases. More work will be needed for all in-place operations to be accelerated. More work will be needed to complete this, but it's good that this is coming to an end, so we can focus on SSA based optimization for the major gains to be had. Nuitka Release 0.5.9 ==================== This release is mostly a maintenance release, bringing out minor compatibility improvements, and some standalone improvements. Also new options to control the recursion into modules are added. Bug Fixes --------- - Compatibility: Checks for iterators were using ``PyIter_Check`` which is buggy when running outside of Python core, because it's comparing pointers we don't see. Replaced with ``HAS_ITERNEXT`` helper which compares against the pointer as extracting for a real non-iterator object. .. code-block:: python class Iterable: def __init__(self): self.consumed = 2 def __iter__(self): return Iterable() iter(Iterable()) # This is suppose to raise, but didn't with Nuitka - Python3: Errors when creating class dictionaries raised by the ``__prepare__`` dictionary (e.g. ``enum`` classes with wrong identifiers) were not immediately raised, but only by the ``type`` call. This was not observable, but might have caused issues potentially. - Standalone MacOS: Shared libraries and extension modules didn't have their DLL load paths updated, but only the main binary. This is not sufficient for more complex programs. - Standalone Linux: Shared libraries copied into the ``.dist`` folder were read-only and executing ``chrpath`` could potentially then fail. This has not been observed, but is a conclusion of MacOS fix. - Standalone: When freezing standard library, the path of Nuitka and the current directory remained in the search path, which could lead to looking at the wrong files. Organizational -------------- - The ``getattr`` built-in is now optimized for compile time constants if possible, even in the presence of a ``default`` argument. This is more a cleanup than actually useful yet. - The calling of ``PyCFunction`` from normal Python extension modules got accelerated, especially for the no or single argument cases where Nuitka now avoids building the tuple. New Features ------------ - Added the option ``--recurse-pattern`` to include modules per filename, which for Python3 is the only way to not have them in a package automatically. - Added the option ``--generate-c++-only`` to only generate the C++ source code without starting the compiler. Mostly used for debugging and testing coverage. In the later case we do not want the C++ compiler to create any binary, but only to measure what would have been used. Organizational -------------- - Renamed the debug option ``--c++-only`` to ``--recompile-c++-only`` to make its purpose more clear and there now is ``--generate-c++-only`` too. Tests ----- - Added support for taking coverage of Nuitka in a test run on a given input file. - Added support for taking coverage for all Nuitka test runners, migrating them all to common code for searching. - Added uniform way of reporting skipped tests, not generally used yet. Summary ------- This release marks progress towards having coverage testing. Recent releases had made it clear that not all code of Nuitka is actually used at least once in our release tests. We aim at identifying these. Another direction was to catch cases, where Nuitka leaks exceptions or is subject to leaked exceptions, which revealed previously unnoticed errors. Important changes have been delayed, e.g. the closure variables will not yet use C++ objects to share storage, but proper ``PyCellObject`` for improved compatibility, and to approach a more "C-ish" status. These is unfinished code that does this. And the forward propagation of values is not enabled yet again either. So this is an interim step to get the bug fixes and improvements accumulated out. Expect more actual changes in the next releases. Nuitka Release 0.5.8 ==================== This release has mainly a focus on cleanups and compatibility improvements. It also advances standalone support, and a few optimization improvements, but it mostly is a maintenance release, attacking long standing issues. Bug Fixes --------- - Compatibility Windows MacOS: Fix importing on case insensitive systems. It was not always working properly, if there was both a package ``Something`` and ``something``, by merit of having files ``Something/__init__.py`` and ``something.py``. - Standalone: The search path was preferring system directories and therefore could have conflicting DLLs. `Issue#144 `__. - Fix, the optimization of ``getattr`` with predictable result was crashing the compilation. This was a regression, fixed in 0.5.7.1 already. - Compatibility: The name mangling inside classes also needs to be applied to global variables. - Fix, proving ``clang++`` for ``CXX`` was mistakingly thinking of it as a ``g++`` and making version checks on it. - Python3: Declaring ``__class__`` global is now a ``SyntaxError`` before Python3.4. - Standalone Python3: Making use of module state in extension modules was not working properly. New Features ------------ - The filenames of source files as found in the ``__file__`` attribute are now made relative in standalone mode. This should make it more apparent if things outside of the distribution folder are used, at the cost of tracebacks. Expect the default ability to copy the source code along in an upcoming release. - Added experimental standalone mode support for PyQt5. At least headless mode should be working, plug-ins (needed for anything graphical) are not yet copied and will need more work. Cleanup ------- - No longer using ``imp.find_module`` anymore. To solve the casing issues we needed to make our own module finding implementation finally. - The name mangling was handled during code generation only. Moved to tree building instead. - More code generation cleanups. The compatible line numbers are now attached during tree building and therefore better preserved, as well as that code no longer polluting code generation as much. Organizational -------------- - No more packages for openSUSE 12.1/12.2/12.3 and Fedora 17/18/19 as requested by the openSUSE Build Service. - Added RPM packages for Fedora 21 and CentOS 7 on openSUSE Build Service. Tests ----- - Lots of test refinements for the CPython test suites to be run continuously in Buildbot for both Windows and Linux. Summary ------- This release brings about two major changes, each with the risk to break things. One is that we finally started to have our own import logic, which has the risk to cause breakage, but apparently currently rather improved compatibility. The case issues were not fixable with standard library code. The second one is that the ``__file__`` attributes for standalone mode is now no longer pointing to the original install and therefore will expose missing stuff sooner. This will have to be followed up with code to scan for missing "data" files later on. For SSA based optimization, there are cleanups in here, esp. the one removing the name mangling, allowing to remove special code for class variables. This makes the SSA tree more reliable. Hope is that the big step (forward propagation through variables) can be made in one of the next releases. Nuitka Release 0.5.7 ==================== This release is brings a newly supported platform, bug fixes, and again lots of cleanups. Bug Fixes --------- - Fix, creation of dictionary and set literals with non-hashable indexes did not raise an exception. .. code-block:: python {[]: None} # This is now a TypeError New Optimization ---------------- - Calls to the ``dict`` built-in with only keyword arguments are now optimized to mere dictionary creations. This is new for the case of non-constant arguments only of course. .. code-block:: python dict(a = b, c = d) # equivalent to {"a" : b, "c" : d} - Slice ``del`` with indexable arguments are now using optimized code that avoids Python objects too. This was already done for slice look-ups. - Added support for ``bytearray`` built-in. Organizational -------------- - Added support for OpenBSD with fiber implementation from library, as it has no context support. Cleanups -------- - Moved slicing solutions for Python3 to the re-formulation stage. So far the slice nodes were used, but only at code generation time, there was made a distinction between Python2 and Python3 for them. Now these nodes are purely Python2 and slice objects are used universally for Python3. Tests ----- - The test runners now have common code to scan for the first file to compile, an implementation of the ``search`` mode. This will allow to introduce the ability to search for pattern matches, etc. - More tests are directly executable with Python3. - Added ``recurse_none`` mode to test comparison, making using extra options for that purpose unnecessary. Summary ------- This solves long standing issues with slicing and subscript not being properly distinguished in the Nuitka code. It also contains major bug fixes that really problematic. Due to the involved nature of these fixes they are made in this new release. Nuitka Release 0.5.6 ==================== This release brings bug fixes, important new optimization, newly supported platforms, and important compatibility improvements. Progress on all fronts. Bug Fixes --------- - Closure taking of global variables in member functions of classes that had a class variable of the same name was binding to the class variable as opposed to the module variable. - Overwriting compiled function's ``__doc__`` attribute more than once could corrupt the old value, leading to crashes. `Issue#156 `__. Fixed in 0.5.5.2 already. - Compatibility Python2: The ``exec`` statement ``execfile`` were changing ``locals()`` was given as an argument. .. code-block:: python def function(): a = 1 exec code in locals() # Cannot change local "a". exec code in None # Can change local "a" exec code Previously Nuitka treated all 3 variants the same. - Compatibility: Empty branches with a condition were reduced to only the condition, but they need in fact to also check the truth value: .. code-block:: python if condition: pass # must be treated as bool(condition) # and not (bug) condition - Detection of Windows virtualenv was not working properly. Fixed in 0.5.5.2 already. - Large enough constants structures are now unstreamed via ``marshal`` module, avoiding large codes being generated with no point. Fixed in 0.5.5.2 already. - Windows: Pressing CTRL-C gave two stack traces, one from the re-execution of Nuitka which was rather pointless. Fixed in 0.5.5.1 already. - Windows: Searching for virtualenv environments didn't terminate in all cases. Fixed in 0.5.5.1 already. - During installation from PyPI with Python3 versions, there were errors given for the Python2 only scons files. `Issue#153 `__. Fixed in 0.5.5.3 already. - Fix, the arguments of ``yield from`` expressions could be leaked. - Fix, closure taking of a class variable could have in a sub class where the module variable was meant. .. code-block:: python var = 1 class C: var = 2 class D: def f(): # was C.var, now correctly addressed top level var return var - Fix, setting ``CXX`` environment variable because the installed gcc has too low version, wasn't affecting the version check at all. - Fix, on Debian/Ubuntu with ``hardening-wrapper`` installed the version check was always failing, because these report a shortened version number to Scons. New Optimization ---------------- - Local variables that must be assigned also have no side effects, making use of SSA. This allows for a host of optimization to be applied to them as well, often yielding simpler access/assign code, and discovering in more cases that frames are not necessary. - Micro optimization to ``dict`` built-in for simpler code generation. Organizational -------------- - Added support for ARM "hard float" architecture. - Added package for Ubuntu 14.10 for download. - Added package for openSUSE 13.2 for download. - Donations were used to buy a Cubox-i4 Pro. It got Debian Jessie installed on it, and will be used to run an even larger amount of tests. - Made it more clear in the user documentation that the ``.exe`` suffix is used for all platforms, and why. - Generally updated information in user manual and developer manual about the optimization status. - Using Nikola 7.1 with external filters instead of our own, outdated branch for the web site. Cleanups -------- - PyLint clean for the first time ever. We now have a Buildbot driven test that this stays that way. - Massive indentation cleanup of keyword argument calls. We have a rule to align the keywords, but as this was done manually, it could easily get out of touch. Now with a "autoformat" tool based on RedBaron, it's correct. Also, spacing around arguments is now automatically corrected. More to come. - For ``exec`` statements, the coping back to local variables is now an explicit node in the tree, leader to cleaner code generation, as it now uses normal variable assignment code generation. - The ``MaybeLocalVariables`` became explicit about which variable they might be, and contribute to its SSA trace as well, which was incomplete before. - Removed some cases of code duplication that were marked as TODO items. This often resulted in cleanups. - Do not use ``replaceWith`` on child nodes, that potentially were re-used during their computation. Summary ------- The release is mainly the result of consolidation work. While the previous release contained many important enhancements, this is another important step towards full SSA, closing one loop whole (class variables and ``exec`` functions), as well as applying it to local variables, largely extending its use. The amount of cleanups is tremendous, in huge part due to infrastructure problems that prevented release repeatedly. This reduces the technological debt very much. More importantly, it would appear that now eliminating local and temporary variables that are not necessary is only a small step away. But as usual, while this may be easy to implement now, it will uncover more bugs in existing code, that we need to address before we continue. Nuitka Release 0.5.5 ==================== This release is finally making full use of SSA analysis knowledge for code generation, leading to many enhancements over previous releases. It also adds support for Python3.4, which has been longer in the making, due to many rather subtle issues. In fact, even more work will be needed to fully solve remaining minor issues, but these should affect no real code. And then there is much improved support for using standalone mode together with virtualenv. This combination was not previously supported, but should work now. New Features ------------ - Added support for Python3.4 This means support for ``clear`` method of frames to close generators, dynamic ``__qualname__``, affected by ``global`` statements, tuples as ``yield from`` arguments, improved error messages, additional checks, and many more detail changes. New Optimization ---------------- - Using SSA knowledge, local variable assignments now no longer need to check if they need to release previous values, they know definitely for the most cases. .. code-block:: python def f(): a = 1 # This used to check if old value of "a" needs a release ... - Using SSA knowledge, local variable references now no longer need to check for raising exceptions, let alone produce exceptions for cases, where that cannot be. .. code-block:: python def f(): a = 1 return a # This used to check if "a" is assigned - Using SSA knowledge, local variable references now are known if they can raise the ``UnboundLocalError`` exception or not. This allows to eliminate frame usages for many cases. Including the above example. - Using less memory for keeping variable information. - Also using less memory for constant nodes. Bug Fixes --------- - The standalone freezing code was reading Python source as UTF-8 and not using the code that handles the Python encoding properly. On some platforms there are files in standard library that are not encoded like that. - The fiber implementation for Linux amd64 was not working with glibc from RHEL 5. Fixed to use now multiple ``int`` to pass pointers as necessary. Also use ``uintptr_t`` instead of ``intprt_t`` to transport pointers, which may be more optimal. - Line numbers for exceptions were corrupted by ``with`` statements due to setting line numbers even for statements marked as internal. - Partial support for ``win32com`` by adding support for its hidden ``__path__`` change. - Python3: Finally figured out proper chaining of exceptions, given proper context messages for exception raised during the handling of exceptions. - Corrected C++ memory leak for each closure variable taken, each time a function object was created. - Python3: Raising exceptions with tracebacks already attached, wasn't using always them, but producing new ones instead. - Some constants could cause errors, as they cannot be handled with the ``marshal`` module as expected, e.g. ``(int,)``. - Standalone: Make sure to propagate ``sys.path`` to the Python instance used to check for standard library import dependencies. This is important for virtualenv environments, which need ``site.py`` to set the path, which is not executed in that mode. - Windows: Added support for different path layout there, so using virtualenv should work there too. - The code object flag "optimized" (fast locals as opposed to locals dictionary) for functions was set wrongly to value for the parent, but for frames inside it, one with the correct value. This lead to more code objects than necessary and false ``co_flags`` values attached to the function. - Options passed to ``nuitka-python`` could get lost. .. code-block:: sh nuitka-python program.py argument1 argument2 ... The above is supposed to compile program.py, execute it immediately and pass the arguments to it. But when Nuitka decides to restart itself, it would forget these options. It does so to e.g. disable hash randomization as it would affect code generation. - Raising tuples exception as exceptions was not compatible (Python2) or reference leaking (Python3). Tests ----- - Running ``2to3`` is now avoided for tests that are already running on both Python2 and Python3. - Made XML based optimization tests work with Python3 too. Previously these were only working on Python2. - Added support for ignoring messages that come from linking against self compiled Pythons. - Added test case for threaded generators that tortures the fiber layer a bit and exposed issues on RHEL 5. - Made reference count test of compiled functions generic. No more code duplication, and automatic detection of shared stuff. Also a more clear interface for disabling test cases. - Added Python2 specific reference counting tests, so the other cases can be executed with Python3 directly, making debugging them less tedious. Cleanups -------- - Really important removal of "variable references". They didn't solve any problem anymore, but their complexity was not helpful either. This allowed to make SSA usable finally, and removed a lot of code. - Removed special code generation for parameter variables, and their dedicated classes, no more needed, as every variable access code is now optimized like this. - Stop using C++ class methods at all. Now only the destructor of local variables is actually supposed to do anything, and their are no methods anymore. The unused ``var_name`` got removed, ``setVariableValue`` is now done manually. - Moved assertions for the fiber layer to a common place in the header, so they are executed on all platforms in debug mode. - As usual, also a bunch of cleanups for PyLint were applied. - The ``locals`` built-in code now uses code generation for accessing local variable values instead having its own stuff. Organizational -------------- - The Python version 3.4 is now officially supported. There are a few problems open, that will be addressed in future releases, none of which will affect normal people though. - Major cleanup of Nuitka options. - Windows specific stuff is now in a dedicated option group. This includes options for icon, disabling console, etc. - There is now a dedicated group for controlling backend compiler choices and options. - Also pickup ``g++44`` automatically, which makes using Nuitka on CentOS5 more automatic. Summary ------- This release represents a very important step ahead. Using SSA for real stuff will allow us to build the trust necessary to take the next steps. Using the SSA information, we could start implementing more optimizations. Nuitka Release 0.5.4 ==================== This release is aiming at preparatory changes to enable optimization based on SSA analysis, introducing a variable registry, so that variables no longer trace their references to themselves. Otherwise, MinGW64 support has been added, and lots of bug fixes were made to improve the compatibility. New Optimization ---------------- - Using new variable registry, now properly detecting actual need for sharing variables. Optimization may discover that it is unnecessary to share a variable, and then it no longer is. This also allows ``--debug`` without it reporting unused variable warnings on Python3. - Scons startup has been accelerated, removing scans for unused tools, and avoiding making more than one gcc version check. Bug Fixes --------- - Compatibility: In case of unknown encodings, Nuitka was not giving the name of the problematic encoding in the error message. Fixed in 0.5.3.3 already. - Submodules with the same name as built-in modules were wrongly shadowed. Fixed in 0.5.3.2 already. - Python3: Added implementations of ``is_package`` to the meta path based loader. - Python3.4: Added ``find_spec`` implementation to the meta path based loader for increased compatibility. - Python3: Corrections for ``--debug`` to work with Python3 and MSVC compiler more often. - Fixed crash with ``--show-scons`` when no compiler was found. Fixed in 0.5.3.5 already. - Standalone: Need to blacklist ``lib2to3`` from standard library as well. Fixed in 0.5.3.4 already. - Python3: Adapted to changes in ``SyntaxError`` on newer Python releases, there is now a ``msg`` that can override ``reason``. - Standalone Windows: Preserve ``sys.executable`` as it might be used to fork binaries. - Windows: The caching of Scons was not arch specific, and files could be used again, even if changing the arch from ```x86`` to ``x86_64`` or back. - Windows: On 32 bit Python it can happen that with large number of generators running concurrently (>1500), one cannot be started anymore. Raising an ``MemoryError`` now. Organizational -------------- - Added support for MinGW64. Currently needs to be run with ``PATH`` environment properly set up. - Updated internal version of Scons to 2.3.2, which breaks support for VS 2008, but adds support for VS 2013 and VS 2012. The VS 2013 is now the recommended compiler. - Added RPM package and repository for RHEL 7. - The output of ``--show-scons`` now includes the used compiler, including the MSVC version. - Added option ``--msvc`` to select the MSVC compiler version to use, which overrides automatic selection of the latest. - Added option ``-python-flag=no_warnings`` to disable user and deprecation warnings at run time. - Repository for Ubuntu Raring was removed, no more supported by Ubuntu. Cleanups -------- - Made technical and logical sharing decisions separate functions and implement them in a dedicated variable registry. - The Scons file has seen a major cleanup. Summary ------- This release is mostly a maintenance release. The Scons integrations has been heavily visited, as has been Python3 and esp. Python3.4 compatibility, and results from the now possible debug test runs. Standalone should be even more practical now, and MinGW64 is an option for those cases, where MSVC is too slow. Nuitka Release 0.5.3 ==================== This release is mostly a follow up, resolving points that have become possible to resolve after completing the C-ish evolution of Nuitka. So this is more of a service release. New Features ------------ - Improved mode ``--improved`` now sets error lines more properly than CPython does in many cases. - The ``-python-flag=-S`` mode now preserves ``PYTHONPATH`` and therefore became usable with virtualenv. New Optimization ---------------- - Line numbers of frames no longer get set unless an exception occurs, speeding up the normal path of execution. - For standalone mode, using ``--python-flag-S`` is now always possible and yields less module usage, resulting in smaller binaries and faster compilation. Bug Fixes --------- - Corrected an issue for frames being optimized away where in fact they are still necessary. `Issue#140 `__. Fixed in 0.5.2.1 already. - Fixed handling of exception tests as side effects. These could be remainders of optimization, but didn't have code generation. Fixed in 0.5.2.1 already. - Previously Nuitka only ever used the statement line as the line number for all the expression, even if it spawned multiple lines. Usually nothing important, and often even more correct, but sometimes not. Now the line number is most often the same as CPython in full compatibility mode, or better, see above. `Issue#9 `__. - Python3.4: Standalone mode for Windows is working now. - Standalone: Undo changes to ``PYTHONPATH`` or ``PYTHONHOME`` allowing potentially forked CPython programs to run properly. - Standalone: Fixed import error when using PyQt and Python3. New Tests --------- - For our testing approach, the improved line number handling means we can undo lots of changes that are no more necessary. - The compile library test has been extended to cover a third potential location where modules may live, covering the ``matplotlib`` module as a result. Cleanups -------- - In Python2, the list contractions used to be re-formulated to be function calls that have no frame stack entry of their own right. This required some special handling, in e.g. closure taking, and determining variable sharing across functions. This now got cleaned up to be properly in-lined in a ``try``/``finally`` expression. - The line number handling got simplified by pushing it into error exits only, removing the need to micro manage a line number stack which got removed. - Use ``intptr_t`` over ``unsigned long`` to store fiber code pointers, increasing portability. Organizational -------------- - Providing own Debian/Ubuntu repositories for all relevant distributions. - Windows MSI files for Python 3.4 were added. - Hosting of the web site was moved to metal server with more RAM and performance. Summary ------- This release brings about structural simplification that is both a follow-up to C-ish, as well as results from a failed attempt to remove static "variable references" and be fully SSA based. It incorporates changes aimed at making this next step in Nuitka evolution smaller. Nuitka Release 0.5.2 ==================== This is a major release, with huge changes to code generation that improve performance in a significant way. It is a the result of a long development period, and therefore contains a huge jump ahead. New Features ------------ - Added experimental support for Python 3.4, which is still work in progress. - Added support for virtualenv on MacOS. - Added support for virtualenv on Windows. - Added support for MacOS X standalone mode. - The code generation uses no header files anymore, therefore adding a module doesn't invalidate all compiled object files from caches anymore. - Constants code creation is now distributed, and constants referenced in a module are declared locally. This means that changing a module doesn't affect the validity of other modules object files from caches anymore. New Optimization ---------------- - C-ish code generation uses less C++ classes and generates more C-like code. Explicit temporary objects are now used for statement temporary variables. - The constants creation code is no more in a single file, but distributed across all modules, with only shared values created in a single file. This means improved scalability. There are remaining bad modules, but more often, standalone mode is now fast. - Exception handling no longer uses C++ exception, therefore has become much faster. - Loops that only break are eliminated. - Dead code after loops that do not break is now removed. - The ``try``/``finally`` and ``try``/``except`` constructs are now eliminated, where that is possible. - The ``try``/``finally`` part of the re-formulation for ``print`` statements is now only done when printing to a file, avoiding useless node tree bloat. - Tuples and lists are now generated with faster code. - Locals and global variables are now access with more direct code. - Added support for the anonymous ``code`` type built-in. - Added support for ``compile`` built-in. - Generators that statically return immediately, e.g. due to optimization results, are no longer using frame objects. - The complex call helpers use no pseudo frames anymore. Previous code generation required to have them, but with C-ish code generation that is no more necessary, speeding up those kind of calls. - Modules with only code that cannot raise, need not have a frame created for them. This avoids useless code size bloat because of them. Previously the frame stack entry was mandatory. Bug Fixes --------- - Windows: The resource files were cached by Scons and re-used, even if the input changed. The could lead to corrupted incremental builds. `Issue#129 `__. Fixed in 0.5.1.1 already. - Windows: For functions with too many local variables, the MSVC failed with an error "C1026: parser stack overflow, program too complex". The rewritten code generation doesn't burden the compiler as much. `Issue#127 `__. - Compatibility: The timing deletion of nested call arguments was different from C++. This shortcoming has been addressed in the rewritten code generation. `Issue#62 `__. - Compatibility: The ``__future__`` flags and ``CO_FREECELL`` were not present in frame flags. These were then not always properly inherited to ``eval`` and ``exec`` in all cases. - Compatibility: Compiled frames for Python3 had ``f_restricted`` attribute, which is Python2 only. Removed it. - Compatibility: The ``SyntaxError`` of having a ``continue`` in a finally clause is now properly raised. - Python2: The ``exec`` statement with no locals argument provided, was preventing list contractions to take closure variables. - Python2: Having the ASCII encoding declared in a module wasn't working. - Standalone: Included the ``idna`` encoding as well. `Issue#135 `__. - Standalone: For virtualenv, the file ``orig-prefix.txt`` needs to be present, now it's copied into the "dist" directory as well. `Issue#126 `__. Fixed in 0.5.1.1 already. - Windows: Handle cases, where Python and user program are installed on different volumes. - Compatibility: Can now finally use ``execfile`` as an expression. `Issue#5 `__ is finally fixed after all this time thanks to C-ish code generation. - Compatibility: The order or call arguments deletion is now finally compatible. `Issue#62 `__ also is finally fixed. This too is thanks to C-ish code generation. - Compatibility: Code object flags are now more compatible for Python3. - Standalone: Removing "rpath" settings of shared libraries and extension modules included. This makes standalone binaries more robust on Fedora 20. - Python2: Wasn't falsely rejecting ``unicode`` strings as values for ``int`` and ``long`` variants with base argument provided. - Windows: For Python3.2 and 64 bits, global variable accesses could give false ``NameError`` exceptions. Fixed in 0.5.1.6 already. - Compatibility: Many ``exec`` and ``eval`` details have become more correctly, the argument handling is more compatible, and e.g. future flags are now passed along properly. - Compatibility: Using ``open`` with no arguments is now giving the same error. Organizational -------------- - Replying to email from the `issue tracker `__ works now. - Added option name alias ``--xml`` for ``--dump-xml``. - Added option name alias ``--python-dbg`` for ``--python-debug``, which actually might make it a bit more clear that it is about using the CPython debug run time. - Remove option ``--dump-tree``, it had been broken for a long time and unused in favor of XML dumps. - New digital art folder with 3D version of Nuitka logo. Thanks to Juan Carlos for creating it. - Using "README.rst" instead of "README.txt" to make it look better on web pages. - More complete whitelisting of missing imports in standard library. These should give no warnings anymore. - Updated the Nuitka GUI to the latest version, with enhanced features. - The builds of releases and update of the `downloads page `__ is now driven by Buildbot. Page will be automatically updated as updated binaries arrive. Cleanups -------- - Temporary keeper variables and the nodes to handle them are now unified with normal temporary variables, greatly simplifying variable handling on that level. - Less code is coming from templates, more is actually derived from the node tree instead. - Releasing the references to temporary variables is now always explicit in the node tree. - The publishing and preservation of exceptions in frames was turned into explicit nodes. - Exception handling is now done with a single handle that checks with branches on the exception. This eliminates exception handler nodes. - The ``dir`` built-in with no arguments is now re-formulated to ``locals`` or ``globals`` with their ``.keys()`` attribute taken. - Dramatic amounts of cleanups to code generation specialties, that got done right for the new C-ish code generation. New Tests --------- - Warnings from MSVC are now error exits for ``--debug`` mode too, expanding the coverage of these tests. - The outputs with ``python-dbg`` can now also be compared, allowing to expand test coverage for reference counts. - Many of the basic tests are now executable with Python3 directly. This allows for easier debug. - The library compilation test is now also executed with Python3. Summary ------- This release would deserve more than a minor number increase. The C-ish code generation, is a huge body of work. In many ways, it lays ground to taking benefit of SSA results, that previously would not have been possible. In other ways, it's incomplete in not yet taking full advantage yet. The release contains so many improvements, that are not yet fully realized, but as a compiler, it also reflects a stable and improved state. The important changes are about making SSA even more viable. Many of the problematic cases, e.g. exception handlers, have been stream lined. A whole class of variables, temporary keepers, has been eliminated. This is big news in this domain. For the standalone users, there are lots of refinements. There is esp. a lot of work to create code that doesn't show scalability issues. While some remain, the most important problems have been dealt with. Others are still in the pipeline. More work will be needed to take full advantage. This has been explained in a `separate post `__ in greater detail. Nuitka Release 0.5.1 ==================== This release brings corrections and major improvements to how standalone mode performs. Much of it was contributed via patches and bug reports. Bug Fixes --------- - There was a crash when using ``next`` on a non-iterable. Fixed in 0.5.0.1 already. - Module names with special characters not allowed in C identifiers were not fully supported. `Issue#118 `__. Fixed in 0.5.0.1 already. - Name mangling for classes with leading underscores was not removing them from resulting attribute names. This broke at ``__slots__`` with private attributes for such classes. `Issue#119 `__. Fixed in 0.5.0.1 already. - Standalone on Windows might need "cp430" encoding. `Issue#120 `__. Fixed in 0.5.0.2 already. - Standalone mode didn't work with ``lxml.etree`` due to lack of hard coded dependencies. When a shared library imports things, Nuitka cannot detect it easily. - Wasn't working on MacOS 64 bits due to using Linux 64 bits specific code. `Issue#123 `__. Fixed in 0.5.0.2 already. - On MinGW the constants blob was not properly linked on some installations, this is now done differently (see below). New Features ------------ - Memory usages are now traced with ``--show-progress`` allowing us to trace where things go wrong. New Optimization ---------------- - Standalone mode now includes standard library as bytecode by default. This is workaround scalability issues with many constants from many modules. Future releases are going to undo it. - On Windows the constants blob is now stored as a resource, avoiding compilation via C code for MSVC as well. MinGW was changed to use the same code. New Tests --------- - Expanded test coverage for "standalone mode" demonstrating usage of "hex" encoding, PySide, and PyGtk packages. Summary ------- This release is mostly an interim maintenance release for standalone. Major changes that provide optimization beyond that, termed "C-ish code generation" are delayed for future releases. This release makes standalone practical which is an important point. Instead of hour long compilation, even for small programs, we are down to less than a minute. The solution of the scalability issues with many constants from many modules will be top priority going forward. Since they are about how even single use constants are created all in one place, this will be easy, but as large changes are happening in "C-ish code generation", we are waiting for these to complete. Nuitka Release 0.5.0 ==================== This release breaks interface compatibility, therefore the major version number change. Also "standalone mode" has seen significant improvements on both Windows, and Linux. Should work much better now. But consider that this part of Nuitka is still in its infancy. As it is not the top priority of mine for Nuitka, which primarily is intended as an super compatible accelerator of Python, it will continue to evolve nearby. There is also many new optimization based on structural improvements in the direction of actual SSA. Bug Fixes --------- - The "standalone mode" was not working on all Redhat, Fedora, and openSUSE platforms and gave warnings with older compilers. Fixed in 0.4.7.1 already. - The "standalone mode" was not including all useful encodings. `Issue#116 `__. Fixed in 0.4.7.2 already. - The "standalone mode" was defaulting to ``--python-flag=-S`` which disables the parsing of "site" module. That unfortunately made it necessary to reach some modules without modifying ``PYTHONPATH`` which conflicts with the "out-of-the-box" experience. - The "standalone mode" is now handling packages properly and generally working on Windows as well. - The syntax error of having an all catching except clause and then a more specific one wasn't causing a ``SyntaxError`` with Nuitka. .. code-block:: python try: something() except: somehandling(): except TypeError: notallowed() - A corruption bug was identified, when re-raising exceptions, the top entry of the traceback was modified after usage. Depending on ``malloc`` this was potentially causing an endless loop when using it for output. New Features ------------ - Windows: The "standalone" mode now properly detects used DLLs using `Dependency Walker `__ which it offers to download and extra for you. It is used as a replacement to ``ldd`` on Linux when building the binary, and as a replacement of ``strace`` on Linux when running the tests to check that nothing is loaded from the outside. New Optimization ---------------- - When iterating over ``list``, ``set``, this is now automatically lowered to ``tuples`` avoiding the mutable container types. So the following code is now equivalent: .. code-block:: python for x in [ a, b, c ]: ... # same as for x in (a, b, c): ... For constants, this is even more effective, because for mutable constants, no more is it necessary to make a copy. - Python2: The iteration of large ``range`` is now automatically lowered to ``xrange`` which is faster to loop over, and more memory efficient. - Added support for the ``xrange`` built-in. - The statement only expression optimization got generalized and now is capable of removing useless parts of operations, not only the whole thing when it has not side effects. .. code-block:: python [a,b] # same as a b This works for all container types. Another example is ``type`` built-in operation with single argument. When the result is not used, it need not be called. .. code-block:: python type(a) # same as a And another example ``is`` and ``is not`` have no effect of their own as well, therefore: .. code-block:: python a is b # same as a b - Added proper handling of conditional expression branches in SSA based optimization. So far these branches were ignored, which only acceptable for temporary variables as created by tree building, but not other variable types. This is preparatory for introducing SSA for local variables. Organizational -------------- - The option ``--exe`` is now ignored and creating an executable is the default behavior of ``nuitka``, a new option ``--module`` allows to produce extension modules. - The binary ``nuitka-python`` was removed, and is replaced by ``nuitka-run`` with now only implies ``--execute`` on top of what ``nuitka`` is. - Using dedicated `Buildbot `__ for continuous integration testing and release creation as well. - The `Downloads `__ now offers MSI files for Win64 as well. - Discontinued the support for cross compilation to Win32. That was too limited and the design choice is to have a running CPython instance of matching architecture at Nuitka compile time. New Tests --------- - Expanded test coverage for "standalone mode" demonstrating usage of "hex" encoding, and PySide package. Summary ------- The "executable by default" interface change improves on the already high ease of use. The new optimization do not give all that much in terms of numbers, but are all signs of structural improvements, and it is steadily approaching the point, where the really interesting stuff will happen. The progress for standalone mode is of course significant. It is still not quite there yet, but it is making quick progress now. This will attract a lot of attention hopefully. As for optimization, the focus for it has shifted to making exception handlers work optimal by default (publish the exception to sys.exc_info() and create traceback only when necessary) and be based on standard branches. Removing special handling of exception handlers, will be the next big step. This release includes some correctness fixes stemming from that work already. Nuitka Release 0.4.7 ==================== This release includes important new features, lots of polishing cleanups, and some important performance improvements as well. Bug Fixes --------- - The RPM packages didn't build due to missing in-line copy of Scons. Fixed in 0.4.6.1 already. - The recursion into modules and unfreezing them was not working for packages and modules anymore. Fixed in 0.4.6.2 already. - The Windows installer was not including Scons. Fixed in 0.4.6.3 already. - Windows: The immediate execution as performed by ``nuitka --execute`` was not preserving the exit code. `Issue#26 `__. - Python3.3: Packages without ``__init.py__`` were not properly embedding the name-space package as well. - Python3: Fix, modules and packages didn't add themselves to ``sys.modules`` which they should, happened only for programs. - Python3.3: Packages should set ``__package`` to their own name, not the one of their parents. - Python3.3: The ``__qualname__`` of nested classes was corrected. - For modules that recursed to other modules, an infinite loop could be triggered when comparing types with rich comparisons. `Issue#115 `__. New Features ------------ - The "standalone" mode allows to compile standalone binaries for programs and run them without Python installation. The DLLs loaded by extension modules on Windows need to be added manually, on Linux these are determined automatically already. To achieve running without Python installation, Nuitka learned to freeze bytecode as an alternative to compiling modules, as some modules need to be present when the CPython library is initialized. - New option ``--python-flag`` allows to specify flags to the compiler that the "python" binary normally would. So far ``-S`` and ``-v`` are supported, with sane aliases ``no_site`` and ``trace_imports``. The recommended use of ``--python-flag=-S`` is to avoid dependency creep in standalone mode compilations, because the ``site`` module often imports many useless things that often don't apply to target systems. New Optimization ---------------- - Faster frame stack handling for functions without ``try``/``except`` (or ``try``/``finally`` in Python3). This gives a speed boost to "PyStone" of ca. 2.5% overall. - Python2: Faster attribute getting and setting, handling special cases at compile time. This gives a minor speed boost to "PyStone" of ca. 0.5% overall. - Python2: Much quicker calls of ``__getattr__`` and ``__setattr__`` as this is now using the quicker call method avoiding temporary tuples. - Don't treat variables usages used in functions called directly by their owner as shared. This leads to more efficient code generation for contractions and class bodies. - Create ``unicode`` constants directly from their UTF-8 string representation for Python2 as well instead of un-streaming. So far this was only done for Python3. Affects only program start-up. - Directly create ``int`` and ``long`` constants outside of ``2**31`` and ``2**32-1``, but only limited according to actual platform values. Affects only program start-up. - When creating ``set`` values, no longer use a temporary ``tuple`` value, but use a properly generated helper functions instead. This makes creating sets much faster. - Directly create ``set`` constants instead of un-streaming them. Affects only program start-up. - For correct line numbers in traceback, the current frame line number must be updated during execution. This was done more often than necessary, e.g. loops set the line number before loop entry, and at first statement. - Module variables are now accessed even faster, the gain for "PyStone" is only 0.1% and mostly the result of leaner code. Organizational -------------- - The "standalone mode" code (formerly known as "portable mode" has been redone and activated. This is a feature that a lot of people expect from a compiler naturally. And although the overall goal for Nuitka is of course acceleration, this kind of packaging is one of the areas where CPython needs improvement. - Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04 and 11.10, no more supported. - Added package for openSUSE 13.1 for download. - Nuitka is now part of Arch and can be installed with ``pacman -S nuitka``. - Using dedicated `Buildbot `__ for continuous integration testing. Not yet public. - Windows: In order to speed up repeated compilation on a platform without ``ccache``, added Scons level caching in the build directory. - Disabled hash randomization for inside Nuitka (but not in ultimately created binaries) for a more stable output, because dictionary constants will not change around. This makes the build results possible to cache for ``ccache`` and Scons as well. Tests ----- - The ``programs`` tests cases now fail if module or directory recursion is not working, being executed in another directory. - Added test runner for packages, with initial test case for package with recursion and sub-packages. - Made some test cases more strict by reducing ``PYTHONPATH`` provision. - Detect use of extra flags in tests that don't get consumed avoiding ineffective flags. - Use ``--execute`` on Windows as well, the issue that prevented it has been solved after all. Cleanups -------- - The generated code uses ``const_``, ``var_``, ``par_`` prefixes in the generated code and centralized the decision about these into single place. - Module variables no longer use C++ classes for their access, but instead accessor functions, leading to much less code generated per module variable and removing the need to trace their usage during code generation. - The test runners now share common code in a dedicated module, previously they replicated it all, but that turned out to be too tedious. - Massive general cleanups, many of which came from new contributor Juan Carlos Paco. - Moved standalone and freezer related codes to dedicated package ``nuitka.freezer`` to not pollute the ``nuitka`` package name space. - The code generation use variable identifiers and their accesses was cleaned up. - Removed several not-so-special case identifier classes because they now behave more identical and all work the same way, so a parameters can be used to distinguish them. - Moved main program, function object, set related code generation to dedicated modules. Summary ------- This release marks major technological progress with the introduction of the much sought standalone mode and performance improvements from improved code generation. The major break through for SSA optimization was not yet achieved, but this is again making progress in the direction of it. Harmonizing variables of different kinds was an important step ahead. Also very nice is the packaging progress, Nuitka was accepted into Arch after being in Debian Testing for a while already. Hope is to see more of this kind of integration in the future. Nuitka Release 0.4.6 ==================== This release includes progress on all fronts. The primary focus was to advance SSA optimization over older optimization code that was already in place. In this domain, there are mostly cleanups. Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds an installed MSVC compiler automatically, properly handles architecture of Python and Windows. This improves usability a lot. Then this is also very much about bug fixes. There have been several hot fixes for the last release, but a complicated and major issue forced a new release, and many other small issues. And then there is performance. As can be seen in the `performance graph `__, this release is the fastest so far. This came mainly from examining the need for comparison slots for compiled types. And last, but not least, this also expands the base of supported platforms, adding Gentoo, and self compiled Python to the mix. Bug Fixes --------- - Support Nuitka being installed to a path that contains spaces and handle main programs with spaces in their paths. `Issue#106 `__. Fixed in 0.4.5.1 already. - Support Python being installed to a path that contains spaces. `Issue#106 `__. Fixed in 0.4.5.2 already. - Windows: User provided constants larger than 65k didn't work with MSVC. `Issue#108 `__. Fixed in 0.4.5.3 already. - Windows: The option ``--windows-disable-console`` was not effective with MSVC. `Issue#107 `__. Fixed in 0.4.5.3 already. - Windows: For some users, Scons was detecting their MSVC installation properly already from registry, but it didn't honor the target architecture. `Issue#99 `__. Fixed in 0.4.5.3 already. - When creating Python modules, these were marked as executable ("x" bit), which they are of course not. Fixed in 0.4.5.3 already. - Python3.3: On architectures where ``Py_ssize_t`` is not the same as ``long`` this could lead to errors. Fixed in 0.4.5.3 already. - Code that was using nested mutable constants and changed the nested ones was not executing correctly. `Issue#112 `__. - Python2: Due to list contractions being re-formulated as functions, ``del`` was rejected for the variables assigned in the contraction. `Issue#111 `__. .. code-block:: python [ expr(x) for x in iterable() ] del x # Should work, was gave an unjustified SyntaxError. New Features ------------ - Compiled types when used in Python comparison now work. Code like this will work: .. code-block:: python def f(): pass assert type(f) == types.FunctionType This of course also works for ``in`` operator, and is another step ahead in compatibility, and surprising too. And best of all, this works even if the checking code is not compiled with Nuitka. - Windows: Detecting MSVC installation from registry, if no compiler is already present in PATH. - Windows: Now options ``--mingw`` to force compilation with MinGW. New Optimization ---------------- - Rich comparisons (``==``, ``<``, and the like) are now faster than ever before due to a full implementation of its own in Nuitka that eliminates a bit of the overhead. In the future, we will aim at giving it type hints to make it even faster. This gives a minor speed boost to PyStone of ca. 0.7% overall. - Integer comparisons are now treated preferably, as they are in CPython, which gives 1.3% speed boost to CPython. - The SSA based analysis is now used to provide variable scopes for temporary variables as well as reference count needs. Cleanups -------- - Replaced "value friend" based optimization code with SSA based optimization, which allowed to remove complicated and old code that was still used mainly in optimization of ``or`` and ``and`` expressions. - Delayed declaration of temp variables and their reference type is now performed based on information from SSA, which may given more accurate results. Not using "variable usage" profiles for this anymore. - The Scons interface and related code got a massive overhaul, making it more consistent and better documented. Also updated the internal copy to 2.3.0 for the platforms that use it, mostly Windows. - Stop using ``os.system`` and ``subprocess.call(..., shell = True)`` as it is not really portable at all, use ``subprocess.call(..., shell = False)`` instead. - As usual lots of cleanups related to line length issues and PyLint. Organizational -------------- - Added support for Gentoo Linux. - Added support for self compiled Python versions with and without debug enabled. `Issue#110 `__ - Added use of Nuitka fonts for headers in manuals. - Does not install in-line copy of Scons only on systems where it is not going to be used, that is mostly non-Windows, and Linux where it is not already present. This makes for cleaner RPM packages. Summary ------- While the SSA stuff is not yet bearing performance fruits, it starts to carry weight. Taking over the temporary variable handling now also means we can apply the same stuff to local variables later. To make up for the delay in SSA driven performance improvements, there is more traditional code acceleration for rich comparisons, making it significant, and the bug fixes make Nuitka more compatible than ever. So give this a roll, it's worth it. And feel free to `join the mailing list `_ or `make a donation `__ to support Nuitka. Nuitka Release 0.4.5 ==================== This release incorporates very many bug fixes, most of which were already part of hot fixes, usability improvements, documentation improvements, new logo, simpler Python3 on Windows, warnings for recursion options, and so on. So it's mostly a consolidation release. Bug Fixes --------- - When targeting Python 3.x, Nuitka was using "python" to run Scons to run it under Python 2.x, which is not good enough on systems, where that is already Python3. Improved to only do the guessing where necessary (i.e. when using the in-line copy of Scons) and then to prefer "python2". `Issue#95 `__. Fixed in 0.4.4.1 already. - When using Nuitka created binaries inside a "virtualenv", created programs would instantly crash. The attempt to load and patch ``inspect`` module was not making sure that ``site`` module was already imported, but inside the "virtualenv", it cannot be found unless. `Issue#96 `__. Fixed in 0.4.4.1 already. - The option ``--recurse-directory`` to include plugin directories was broken. `Issue#97 `__. Fixed in 0.4.4.2 already. - Python3: Files with "BOM" marker causes the compiler to crash. `Issue#98 `__. Fixed in 0.4.4.2 already. - Windows: The generated code for ``try``/``return``/``finally`` was working with gcc (and therefore MinGW), but not with MSVC, causing crashes. `Issue#102 `__. Fixed in 0.4.4.2 already. - The option ``--recurse-all`` did not recurse to package ``__init__.py`` files in case ``from x.y import z`` syntax was used. `Issue#100 `__. Fixed in 0.4.4.2 already. - Python3 on MacOS: Corrected link time error. Fixed in 0.4.4.2 already. - Python3.3 on Windows: Fixed crash with too many arguments to a kwonly argument using function. Fixed in 0.4.4.2 already. - Python3.3 on Windows: Using "yield from" resulted in a link time error. Fixed in 0.4.4.2 already. - Windows: Added back XML manifest, found a case where it is needed to prevent clashes with binary modules. - Windows: Generators only worked in the main Python threads. Some unusual threading modules therefore failed. - Using ``sys.prefix`` to find the Python installation instead of hard coded paths. `Issue#103 `__. New Features ------------ - Windows: Python3 finds Python2 installation to run Scons automatically now. Nuitka itself runs under Python3 just fine, but in order to build the generated C++ code into binaries, it uses Scons which still needs Python2. Nuitka will now find the Python2 installation searching Windows registry instead of requiring hard coded paths. - Windows: Python2 and Python3 find their headers now even if Python is not installed to specific paths. The installation path now is passed on to Scons which then uses it. - Better error checking for ``--recurse-to`` and ``--recurse-not-to`` arguments, tell the user not to use directory paths. - Added a warning for ``--recurse-to`` arguments that end up having no effect to the final result. Cleanups -------- - Import mechanism got cleaned up, stopped using "PyImport_ExtendInittab". It does not handle packages, and the ``sys.meta_path`` based importer is now well proven. - Moved some of the constraint collection code mess into proper places. It still remains a mess. Organizational -------------- - Added ``LICENSE.txt`` file with Apache License 2.0 text to make it more immediately obvious which license Nuitka is under. - Added section about Nuitka license to the "`User Manual `__". - Added `Nuitka Logo `__ to the distribution. - Use Nuitka Logo as the bitmap in the Windows installer. - Use Nuitka Logo in the documentation ("`User Manual `__" and "`Developer Manual `__"). - Enhanced documentation to number page numbers starting after table of contents, removed header/footer from cover pages. Summary ------- This release is mostly the result of improvements made based on the surge of users after Europython 2013. Some people went to extents and reported their experience very detailed, and so I could aim at making e.g. their misconceptions about how recursion options work, more obvious through warnings and errors. This release is not addressing performance improvements. The next release will be able to focus on that. I am taking my claim of full compatibility very serious, so any time it's broken, it's the highest priority to restore it. Nuitka Release 0.4.4 ==================== This release marks the point, where Nuitka for the first time supports all major current Python versions and all major features. It adds Python 3.3 support and it adds support for threading. And then there is a massive amount of fixes that improve compatibility even further. Aside of that, there is major performance work. One side is the optimization of call performance (to CPython non-compiled functions) and to compiled functions, both. This gave a serious improvement to performance. Then of course, we are making other, long term performance progress, as in "--experimental" mode, the SSA code starts to optimize unused code away. That code is not yet ready for prime time yet, but the trace structure will hold. New Features ------------ - Python3.3 support. The test suite of CPython3.3 passes now too. The ``yield from`` is now supported, but the improved argument parsing error messages are not implemented yet. - Tracing user provided constants, now Nuitka warns about too large constants produced during optimization. - Line numbers of expressions are now updates as evaluation progresses. This almost corrects. Finally improves `Issue#9 `__. Now only expression parts that cannot raise, do not update, which can still cause difference, but much less often, and then definitely useless. - Experimental support for threads. Threading appears to work just fine in the most cases. It's not as optimal as I wanted it to be, but that's going to change with time. New Optimization ---------------- - Previous corrections for ``==``, ``!=``, and ``<=``, caused a performance regression for these operations in case of handling identical objects. For built-in objects of sane types (not ``float``), these operations are now accelerated again. The overreaching acceleration of ``>=`` was still there (bug, see below) and has been adapted too. - Calling non-compiled Python functions from compiled functions was slower than in CPython. It is now just as fast. - Calling compiled functions without keyword arguments has been accelerated with a dedicated entry point that may call the implementation directly and avoid parameter parsing almost entirely. - Making calls to compiled and non-compiled Python functions no longer requires to build a temporary tuple and therefore is much faster. - Parameter parsing code is now more compact, and re-uses error raises, or creates them on the fly, instead of hard coding it. Saves binary size and should be more cache friendly. Bug Fixes --------- - Corrected false optimization of ``a >= a`` on C++ level. When it's not done during Nuitka compile time optimization, the rich comparison helper still contained short cuts for ``>=``. This is now the same for all the comparison operators. - Calling a function with default values, not providing it, and not providing a value for a value without default, was not properly detecting the error, and instead causing a run time crash. .. code-block:: python def f(a, b = 2): pass f(b = 2) This now properly raises the ``TypeError`` exception. - Constants created with ``+`` could become larger than the normally enforced limits. Not as likely to become huge, but still potentially an issue. - The ``vars`` built-in, when used on something without ``__dict__`` attribute, was giving ``AttributeError`` instead of ``TypeError``. - When re-cursing to modules at compile time, script directory and current directory were used last, while at run time, it was the other way around, which caused overloaded standard library modules to not be embedded. Corrects `Issue#94 `__. Thanks for the patch to James Michael DuPont. - Super without arguments was not raising the correct ``RuntimeError`` exception in functions that cannot be methods, but ``UnboundLocalError`` instead. .. code-block:: python def f(): super() # Error, cannot refer to first argument of f - Generators no longer use ``raise StopIteration`` for return statements, because that one is not properly handled in ``try``/``except`` clauses, where it's not supposed to trigger, while ``try``/``finally`` should be honored. - Exception error message when throwing non-exceptions into generators was not compatible. - The use of ``return`` with value in generators is a ``SyntaxError`` before Python3.3, but that was not raised. - Variable names of the "__var" style need to be mangled. This was only done for classes, but not for functions contained in classes, there they are now mangled too. - Python3: Exceptions raised with causes were not properly chaining. - Python3: Specifying the file encoding corrupted line numbers, making them all of by one. Cleanups -------- - For containers (``tuple``, ``list``, ``set``, ``dict``) defined on the source code level, Nuitka immediately created constant references from them. For function calls, class creations, slice objects, this code is now re-used, and its dictionaries and tuples, may now become constants immediately, reducing noise in optimization steps. - The parameter parsing code got cleaned up. There were a lot of relics from previously explored paths. And error raises were part of the templates, but now are external code. - Global variable management moved to module objects and out of "Variables" module. - Make sure, nodes in the tree are not shared by accident. This helped to find a case of duplicate use in the complex call helpers functions. Code generation will now notice this kind of duplication in debug mode. - The complex call helper functions were manually taking variable closure, which made these functions inconsistent to other functions, e.g. no variable version was allocated to assignments. Removing the manual setting of variables allowed a huge reduction of code volume, as it became more generic code. - Converting user provided constants to create containers into constants immediately, to avoid noise from doing this in optimization. - The ``site`` module is now imported explicitly in the ``__main__`` module, so it can be handled by the recursion code as well. This will help portable mode. - Many line length 80 changes, improved comments. New Tests --------- - The CPython3.3 test suite was added, and run with both Python3.2 and Python3.3, finding new bugs. - The ``doctest`` to code generation didn't successfully handle all tests, most notably, "test_generators.py" was giving a ``SyntaxError`` and therefore not actually active. Correcting that improved the coverage of generator testing. Organizational -------------- - The portable code is still delayed. Support for Python3.3 was a higher priority, but the intention is to get it into shape for Europython still. Added notes about it being disabled it in the "`User Manual `__" documentation. Summary ------- This release is in preparation for Europython 2013. Wanted to get this much out, as it changes the status slides quite a bit, and all of that was mostly done in my Cyprus holiday a while ago. The portable code has not seen progress. The idea here is to get this into a development version later. Nuitka Release 0.4.3 ==================== This release expands the reach of Nuitka substantially, as new platforms and compilers are now supported. A lot of polish has been applied. Under the hood there is the continued and in-progress effort to implement SSA form in Nuitka. New Features ------------ - Support for new compiler: Microsoft Visual C++. You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows. - Support for NetBSD. Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic "fibers" implementation. - Support for Python3 under Windows too. Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to ``C:\Python26`` or ``C:\Python27`` for Nuitka to be able to find it. - Enhanced Python 3.3 compatibility. The support the newest version of Python has been extended, improving compatibility for many minor corner cases. - Added warning when a user compiles a module and executes it immediately when that references ``__name__``. Because very likely the intention was to create an executable. And esp. if there is code like this: .. code-block:: python if __name__ == "__main__": main() In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command .. code-block:: sh nuitka --execute module is behavioral more like python -c "import module" and that was a trap for new users. - All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don't have to implement for specific architectures anymore. Bug Fixes --------- - Dictionary creation was not fully compatible. As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn't observe this before, and it's likely the new dictionary implementation that finds it. Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing. - Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was. - Using a ``__future__`` import inside a function was giving an assertion, instead of the proper syntax error. - Python3: Do not set the attributes ``sys.exc_type``, ``sys.exc_value``, ``sys.exc_traceback``. - Python3: Annotations of function worked only as long as their definition was not referring to local variables. New Optimization ---------------- - Calls with no positional arguments are now using the faster call methods. The generated C++ code was using the ``()`` constant at call site, when doing calls that use no positional arguments, which is of course useless. - For Windows now uses OS "Fibers" for Nuitka "Fibers". Using threads for fibers was causing only overhead and with this API, MSVC had less issues too. Organizational -------------- - Accepting `Donations `__ via Paypal, please support funding travels, website, etc. - The "`User Manual `__" has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated. - The Changelog has been improved for consistency, wordings, and styles. - Nuitka is now available on the social code platforms as well * `Bitbucket `__ * `Github `__ * `Gitorious `__ * `Google Code `__ - Removed "clean-up.sh", which is practically useless, as tests now clean up after themselves reasonably, and with ``git clean -dfx`` working better. - Removed "create-environment.sh" script, which was only setting the ``PATH`` variable, which is not necessary. - Added ``check-with-pylint --emacs`` option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint. - Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated. Cleanups -------- - Removed useless manifest and resource file creation under Windows. Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it. - PyLint massive cleanups and annotations bringing down the number of warnings by a lot. - Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes. - Do not track needed tuple, list, and dict creation code variants in context, but e.g. in ``nuitka.codegen.TupleCodes`` module instead. - Introduced an "internal" module to host the complex call helper functions, instead of just adding it to any module that first uses it. New Tests --------- - Added basic tests for order evaluation, where there currently were None. - Added support for "2to3" execution under Windows too, so we can run tests for Python3 installations too. Summary ------- The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now. Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization. Nuitka Release 0.4.2 ==================== This release comes with many bug fixes, some of which are severe. It also contains new features, like basic Python 3.3 support. And the `performance diagrams `__ got expanded. New Features ------------ - Support for FreeBSD. Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some "Linuxisms" in the build process. - New option for warning about compile time detected exception raises. Nuitka can now warn about exceptions that will be raised at run time. - Basic Python3.3 support. The test suite of CPython3.2 passes and fails in a compatible way. New feature ``yield from`` is not yet supported, and the improved argument parsing error messages are not implemented yet. Bug Fixes --------- - Nuitka already supported compilation of "main directories", i.e. directories with a "__main__.py" file inside. The resulting binary name was "__main__.exe" though, but now it is "directory.exe" .. code-block:: sh # ls directory __main__.py # nuitka --exe directory # ls directory directory.exe This makes this usage more obvious, and fixes the older issue `Issue#49 `__ for this feature. - Evaluation order of binary operators was not enforced. Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems. Corrects `Issue#61 `__. - Providing an ``# coding: no-exist`` was crashing under Python2, and ignored under Python3, now it does the compatible thing for both. - Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are. .. code-block:: python global a # Not in a function, but on module level. Pointless but legal! a = 1 Effectively these statements can be ignored. Corrects part of `Issue#65 `__. - Future imports are only legal when they are at the start of the file. This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error. Corrects part of `Issue#65 `__. - Raising exceptions from context was leaking references. .. code-block:: python raise ValueError() from None Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments. Corrects `Issue#76 `__. - Importing the module that became ``__main__`` through the module name, didn't recurse to it. This also gives a warning. PyBench does it, and then stumbles over the non-found "pybench" module. Of course, programmers should use ``sys.modules[ "__main__" ]`` to access main module code. Not only because the duplicated modules don't share data. Corrects `Issue#68 `__. - Compiled method ``repr`` leaked references when printed. When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects ``__del__`` to run at some point. Corrects `Issue#81 `__. - The ``super`` built-in leaked references to given object. This was added, because Python3 needs it. It supplies the arguments to ``super`` automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing ``__del__`` to run. Corrects `Issue#81 `__. - The ``raise`` statement didn't enforce type of third argument. This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it's None (which is also legal), no reference to None is leaked. - Python3 built-in exceptions were strings instead of exceptions. A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn't help their usage of course, fixed. Corrects `Issue#82 `__. - The ``-nan`` and ``nan`` both exist and make a difference. A older story continued. There is a sign to ``nan``, which can be copied away and should be present. This is now also supported by Nuitka. Corrects `Issue#75 `__. - Wrong optimization of ``a == a``, ``a != a``, ``a <= a`` on C++ level. While it's not done during Nuitka optimization, the rich comparison helpers still contained short cuts for ``==``, ``!=``, and ``<=``. - The ``sys.executable`` for ``nuitka-python --python-version 3.2`` was still ``python``. When determining the value for ``sys.executable`` the CPython library code looks at the name ``exec`` had received. It was ``python`` in all cases, but now it depends on the running version, so it propagates. - Keyword only functions with default values were loosing references to defaults. .. code-block:: python def f(*, a = X()) pass f() f() # Can crash, X() should already be released. This is now corrected. Of course, a Python3 only issue. - Pressing CTRL-C didn't generate ``KeyboardInterrupt`` in compiled code. Nuitka never executes "pending calls". It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this. - For ``with`` statements with ``return``, ``break``, or ``continue`` to leave their body, the ``__exit__`` was not called. .. code-block:: python with a: # This called a.__enter__(). return 2 # This didn't call a.__exit__(None, None, None). This is of course quite huge, and unfortunately wasn't covered by any test suite so far. Turns out, the re-formulation of ``with`` statements, was wrongly using ``try/except/else``, but these ignore the problematic statements. Only ``try/finally`` does. The enhanced re-formulation now does the correct thing. Corrects `Issue#59 `__. - Starting with Python3, absolute imports are now the default. This was already present for Python3.3, and it turns out that all of Python3 does it. New Optimization ---------------- - Constants are now much less often created with ``pickle`` module, but created directly. This esp. applies for nested constants, now more values become ``is`` identical instead of only ``==`` identical, which indicates a reduced memory usage. .. code-block:: python a = ("something_special",) b = "something_special" assert a[0] is b # Now true This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the "==" operator will be able to shortcut dramatically in cases of identical objects. Constants now created without ``pickle`` usage, cover ``float``, ``list``, and ``dict``, which is enough for PyStone to not use it at all, which has been added support for as well. - Continue statements might be optimized away. A terminal ``continue`` in a loop, was not optimized away: .. code-block:: python while 1: something continue # Now optimized away The trailing ``continue`` has no effect and can therefore be removed. .. code-block:: python while 1: something - Loops with only break statements are optimized away. .. code-block:: python while 1: break A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate "if ... : break" checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed. New Tests --------- - Added tests for the found issues. - Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered. - Running the "CPython3.2" test suite with Python3.3 based Nuitka works and found a few minor issues. Organizational -------------- - The `Downloads `__ page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and "ownssh" for contributing an RPM spec file. The page got improved with logos for the distributions. - Added "ownssh" as contributor. - Revamped the "`User Manual `__" in terms of layout, structure, and content. Summary ------- This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around. Nuitka Release 0.4.1 ==================== This release is the first follow-up with a focus on optimization. The major highlight is progress towards SSA form in the node tree. Also a lot of cleanups have been performed, for both the tree building, which is now considered mostly finished, and will be only reviewed. And for the optimization part there have been large amounts of changes. New Features ------------ - Python 3.3 experimental support * Now compiles many basic tests. Ported the dictionary quick access and update code to a more generic and useful interface. * Added support for ``__qualname__`` to classes and functions. * Small compatibility changes. Some exceptions changed, absolute imports are now default, etc. * For comparison tests, the hash randomization is disabled. - Python 3.2 support has been expanded. The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code. Bug fixes --------- - Default values were not "is" identical. .. code-block:: python def defaultKeepsIdentity(arg = "str_value"): print arg is "str_value" defaultKeepsIdentity() This now prints "True" as it does with CPython. The solution is actually a general code optimization, see below. `Issue#55 `__ - Usage of ``unicode`` built-in with more than one argument could corrupt the encoding argument string. An implementation error of the ``unicode`` was releasing references to arguments converted to default encoding, which could corrupt it. - Assigning Python3 function annotations could cause a segmentation fault. New Optimization ---------------- - Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization. .. code-block:: python # Now optimizing: raise TypeError, 1/0 # into (minus normalization): raise ZeroDivisionError, "integer division or modulo by zero" # Now optimizing: (1/0).something # into (minus normalization): raise ZeroDivisionError, "integer division or modulo by zero" # Now optimizing: function(a, 1/0).something # into (minus normalization), notice the side effects of first checking # function and a as names to be defined, these may be removed only if # they can be demonstrated to have no effect. function a raise ZeroDivisionError, "integer division or modulo by zero" There is more examples, where the raise propagation is new, but you get the idea. - Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things. .. code-block:: python # This was already optimized, as it's a compile time constant. a if ("a",) else b a if True else b # These are now optimized, as their truth value is known. a if (c,) else b a if not (c,) else b This is simply taking advantage of infrastructure that now exists. Each node kind can overload "getTruthValue" and benefit from it. Help would be welcome to review which ones can be added. - Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused. - Code generation for constants now shares element values used in tuples. The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up. Cleanups -------- - Optimization was initially designed around visitors that each did one thing, and did it well. It turns out though, that this approach is unnecessary, and constraint collection, allows for the most consistent results. All remaining optimization has been merged into constraint collection. - The names of modules containing node classes were harmonized to always be plural. In the beginning, this was used to convey the information that only a single node kind would be contained, but that has long changed, and is unimportant information. - The class names of nodes were stripped from the "CPython" prefix. Originally the intent was to express strict correlation to CPython, but with increasing amounts of re-formulations, this was not used at all, and it's also not important enough to dominate the class name. - The re-formulations performed in tree building have moved out of the "Building" module, into names "ReformulationClasses" e.g., so they are easier to locate and review. Helpers for node building are now in a separate module, and generally it's much easier to find the content of interest now. - Added new re-formulation of ``print`` statements. The conversion to strings is now made explicit in the node tree. New Tests --------- - Added test to cover default value identity. Organizational -------------- - The upload of `Nuitka to PyPI `__ has been repaired and now properly displays project information again. Summary ------- The quicker release is mostly a consolidation effort, without actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization. Nuitka Release 0.4.0 ==================== This release brings massive progress on all fronts. The big highlight is of course: Full Python3.2 support. With this release, the test suite of CPython3.2 is considered passing when compiled with Nuitka. Then lots of work on optimization and infrastructure. The major goal of this release was to get in shape for actual optimization. This is also why for the first time, it is tested that some things are indeed compile time optimized to spot regressions easier. And we are having performance diagrams, `even if weak ones `__: New Features ------------ - Python3.2 is now fully supported. - Fully correct ``metaclass =`` semantics now correctly supported. It had been working somewhat previously, but now all the corner cases are covered too. - Keyword only parameters. - Annotations of functions return value and their arguments. - Exception causes, chaining, automatic deletion of exception handlers ``as`` values. - Added support for starred assigns. - Unicode variable names are also supported, although it's of course ugly, to find a way to translate these to C++ ones. Bug fixes --------- - Checking compiled code with ``instance(some_function, types.FunctionType)`` as "zope.interfaces" does, was causing compatibility problems. Now this kind of check passes for compiled functions too. `Issue#53 `__ - The frame of modules had an empty locals dictionary, which is not compatible to CPython which puts the globals dictionary there too. Also discussed in `Issue#53 `__ - For nested exceptions and interactions with generator objects, the exceptions in "sys.exc_info()" were not always fully compatible. They now are. - The ``range`` builtin was not raising exceptions if given arguments appeared to not have side effects, but were still illegal, e.g. ``range([], 1, -1)`` was optimized away if the value was not used. - Don't crash on imported modules with syntax errors. Instead, the attempted recursion is simply not done. - Doing a ``del`` on ``__defaults`` and ``__module__`` of compiled functions was crashing. This was noticed by a Python3 test for ``__kwdefaults__`` that exposed this compiled functions weakness. - Wasn't detecting duplicate arguments, if one of them was not a plain arguments. Star arguments could collide with normal ones. - The ``__doc__`` of classes is now only set, where it was in fact specified. Otherwise it only polluted the name space of ``locals()``. - When ``return`` from the tried statements of a ``try/finally`` block, was overridden, by the final block, a reference was leaked. Example code: .. code-block:: python try: return 1 finally: return 2 - Raising exception instances with value, was leaking references, and not raising the ``TypeError`` error it is supposed to do. - When raising with multiple arguments, the evaluation order of them was not enforced, it now is. This fixes a reference leak when raising exceptions, where building the exception was raising an exception. New Optimization ---------------- - Optimizing attribute access to compile time constants for the first time. The old registry had no actual user yet. - Optimizing subscript and slices for all compile time constants beyond constant values, made easy by using inheritance. - Built-in references now convert to strings directly, e.g. when used in a print statement. Needed for the testing approach "compiled file contains only prints with constant value". - Optimizing calls to constant nodes directly into exceptions. - Optimizing built-in ``bool`` for arguments with known truth value. This would be creations of tuples, lists, and dictionaries. - Optimizing ``a is b`` and ``a is not b`` based on aliasing interface, which at this time effectively is limited to telling that ``a is a`` is true and ``a is not a`` is false, but this will expand. - Added support for optimizing ``hasattr``, ``getattr``, and ``setattr`` built-ins as well. The ``hasattr`` was needed for the ``class`` re-formulation of Python3 anyway. - Optimizing ``getattr`` with string argument and no default to simple attribute access. - Added support for optimizing ``isinstance`` built-in. - Was handling "BreakException" and "ContinueException" in all loops that used ``break`` or ``continue`` instead of only where necessary. - When catching "ReturnValueException", was raising an exception where a normal return was sufficient. Raising them now only where needed, which also means, function need not catch them ever. Cleanups -------- - The handling of classes for Python2 and Python3 have been re-formulated in Python more completely. * The calling of the determined "metaclass" is now in the node tree, so this call may possible to in-line in the future. This eliminated some static C++ code. * Passing of values into dictionary creation function is no longer using hard coded special parameters, but temporary variables can now have closure references, making this normal and visible to the optimization. * Class dictionary creation functions are therefore no longer as special as they used to be. * There is no class creation node anymore, it's merely a call to ``type`` or the metaclass detected. - Re-formulated complex calls through helper functions that process the star list and dict arguments and do merges, checks, etc. * Moves much C++ code into the node tree visibility. * Will allow optimization to eliminate checks and to compile time merge, once in-line functions and loop unrolling are supported. - Added "return None" to function bodies without a an aborting statement at the end, and removed the hard coded fallback from function templates. Makes it explicit in the node tree and available for optimization. - Merged C++ classes for frame exception keeper with frame guards. * The exception is now saved in the compiled frame object, making it potentially more compatible to start with. * Aligned module and function frame guard usage, now using the same class. * There is now a clear difference in the frame guard classes. One is for generators and one is for functions, allowing to implement their different exception behavior there. - The optimization registries for calls, subscripts, slices, and attributes have been replaced with attaching them to nodes. * The ensuing circular dependency has been resolved by more local imports for created nodes. * The package "nuitka.transform.optimization.registries" is no more. * New per node methods "computeNodeCall", "computeNodeSubscript", etc. dispatch the optimization process to the nodes directly. - Use the standard frame guard code generation for modules too. * Added a variant "once", that avoids caching of frames entirely. - The variable closure taking has been cleaned up. * Stages are now properly numbered. * Python3 only stage is not executed for Python2 anymore. * Added comments explaining things a bit better. * Now an early step done directly after building a tree. - The special code generation used for unpacking from iterators and catching "StopIteration" was cleaned up. * Now uses template, Generator functions, and proper identifiers. - The ``return`` statements in generators are now re-formulated into ``raise StopIteration`` for generators, because that's what they really are. Allowed to remove special handling of ``return`` nodes in generators. - The specialty of CPython2.6 yielding non-None values of lambda generators, was so far implemented in code generation. This was moved to tree building as a re-formulation, making it subject to normal optimization. - Mangling of attribute names in functions contained in classes, has been moved into the early tree building. So far it was done during code generation, making it invisible to the optimization stages. - Removed tags attribute from node classes. This was once intended to make up for non-inheritance of similar node kinds, but since we have function references, the structure got so clean, it's no more needed. - Introduced new package ``nuitka.tree``, where the building of node trees, and operations on them live, as well as recursion and variable closure. - Removed ``nuitka.transform`` and move its former children ``nuitka.optimization`` and ``nuitka.finalization`` one level up. The deeply nested structure turned out to have no advantage. - Checks for Python version was sometimes "> 300", where of course ">= 300" is the only thing that makes sense. - Split out helper code for exception raising from the handling of exception objects. New Tests --------- - The complete CPython3.2 test suite was adapted (no ``__code__``, no ``__closure__``, etc.) and is now passing, but only without "--debug", because otherwise some of the generated C++ triggers (harmless) warnings. - Added new test suite designed to prove that expressions that are known to be compile time constant are indeed so. This works using the XML output done with "--dump-xml" and then searching it to only have print statements with constant values. - Added new basic CPython3.2 test "Functions32" and "ParameterErrors32" to cover keyword only parameter handling. - Added tests to cover generator object and exception interactions. - Added tests to cover ``try/finally`` and ``return`` in one or both branches correctly handling the references. - Added tests to cover evaluation order of arguments when raising exceptions. Organizational -------------- - Changed my email from GMX over to Gmail, the old one will still continue to work. Updated the copyright notices accordingly. - Uploaded `Nuitka to PyPI `__ as well. Summary ------- This release marks a milestone. The support of Python3 is here. The re-formulation of complex calls, and the code generation improvements are quite huge. More re-formulation could be done for argument parsing, but generally this is now mostly complete. The 0.3.x series had a lot releases. Many of which brought progress with re-formulations that aimed at making optimization easier or possible. Sometimes small things like making "return None" explicit. Sometimes bigger things, like making class creations normal functions, or getting rid of ``or`` and ``and``. All of this was important ground work, to make sure, that optimization doesn't deal with complex stuff. So, the 0.4.x series begins with this. The focus from now on can be almost purely optimization. This release contains already some of it, with frames being optimized away, with the assignment keepers from the ``or`` and ``and`` re-formulation being optimized away. This will be about achieving goals from the "ctypes" plan as discussed in the developer manual. Also the performance page will be expanded with more benchmarks and diagrams as I go forward. I have finally given up on "codespeed", and do my own diagrams. Nuitka Release 0.3.25 ===================== This release brings about changes on all fronts, bug fixes, new features. Also very importantly Nuitka no longer uses C++11 for its code, but mere C++03. There is new re-formulation work, and re-factoring of functions. But the most important part is this: Mercurial unit tests are working. Nearly. With the usual disclaimer of me being wrong, all remaining errors are errors of the test, or minor things. Hope is that these unit tests can be added as release tests to Nuitka. And once that is done, the next big Python application can come. Bug fixes --------- - Local variables were released when an exception was raised that escaped the local function. They should only be released, after another exception was raised somewhere. `Issue#39 `__. - Identifiers of nested tuples and lists could collide. .. code-block:: python a = ((1, 2), 3) b = ((1,), 2, 3) Both tuples had the same name previously, not the end of the tuple is marked too. Fixed in 0.3.24.1 already. - The ``__name__`` when used read-only in modules in packages was optimized to a string value that didn't contain the package name. - Exceptions set when entering compiled functions were unset at function exit. New Features ------------ - Compiled frames support. Before, Nuitka was creating frames with the standard CPython C/API functions, and tried its best to cache them. This involved some difficulties, but as it turns out, it is actually possible to instead provide a compatible type of our own, that we have full control over. This will become the base of enhanced compatibility. Keeping references to local variables attached to exception tracebacks is something we may be able to solve now. - Enhanced Python3 support, added support for ``nonlocal`` declarations and many small corrections for it. - Writable ``__defaults__`` attribute for compiled functions, actually changes the default value used at call time. Not supported is changing the amount of default parameters. Cleanups -------- - Keep the functions along with the module and added "FunctionRef" node kind to point to them. - Reformulated ``or`` and ``and`` operators with the conditional expression construct which makes the "short-circuit" branch. - Access ``self`` in methods from the compiled function object instead of pointer to context object, making it possible to access the function object. - Removed "OverflowCheck" module and its usage, avoids one useless scan per function to determine the need for "locals dictionary". - Make "compileTree" of "MainControl" module to only do what the name says and moved the rest out, making the top level control clearer. - Don't export module entry points when building executable and not modules. These exports cause MinGW and MSVC compilers to create export libraries. New Optimization ---------------- - More efficient code for conditional expressions in conditions: .. code-block:: python if a if b else c See above, this code is now the typical pattern for each ``or`` and ``and``, so this was much needed now. Organizational -------------- - The remaining uses of C++11 have been removed. Code generated with Nuitka and complementary C++ code now compile with standard C++03 compilers. This lowers the Nuitka requirements and enables at least g++ 4.4 to work with Nuitka. - The usages of the GNU extension operation ``a ?: b`` have replaced with standard C++ constructs. This is needed to support MSVC which doesn't have this. - Added examples for the typical use cases to the "`User Manual `__". - The "compare_with_cpython" script has gained an option to immediately remove the Nuitka outputs (build directory and binary) if successful. Also the temporary files are now put under "/var/tmp" if available. - Debian package improvements, registering with "doc-base" the "`User Manual `__" so it is easier to discover. Also suggest "mingw32" package which provides the cross compiler to Windows. - Partial support for MSVC (Visual Studio 2008 to be exact, the version that works with CPython2.6 and CPython2.7). All basic tests that do not use generators are working now, but those will currently cause crashes. - Renamed the ``--g++-only`` option to ``--c++-only``. The old name is no longer correct after clang and MSVC have gained support, and it could be misunderstood to influence compiler selection, rather than causing the C++ source code to not be updated, so manual changes will the used. This solves `Issue#47 `__. - Catch exceptions for ``continue``, ``break``, and ``return`` only where needed for ``try``/``finally`` and loop constructs. New Tests --------- - Added CPython3.2 test suite as "tests/CPython32" from 3.2.3 and run it with CPython2.7 to check that Nuitka gives compatible error messages. It is not expected to pass yet on Python3.2, but work will be done towards this goal. - Make CPython2.7 test suite runner also execute the generated "doctest" modules. - Enabled tests for default parameters and their reference counts. Summary ------- This release marks an important point. The compiled frames are exciting new technology, that will allow even better integration with CPython, while improving speed. Lowering the requirements to C++03 means, we will become usable on Android and with MSVC, which will make adoption of Nuitka on Windows easier for many. Structurally the outstanding part is the function as references cleanup. This was a blocker for value propagation, because now functions references can be copied, whereas previously this was duplicating the whole function body, which didn't work, and wasn't acceptable. Now, work can resume in this domain. Also very exciting when it comes to optimization is the remove of special code for ``or`` and ``and`` operators, as these are now only mere conditional expressions. Again, this will make value propagation easier with two special cases less. And then of course, with Mercurial unit tests running compiled with Nuitka, an important milestone has been hit. For a while now, the focus will be on completing Python3 support, XML based optimization regression tests, benchmarks, and other open ends. Once that is done, and more certainty about Mercurial tests support, I may call it a 0.4 and start with local type inference for actual speed gains. Nuitka Release 0.3.24 ===================== This release contains progress on many fronts, except performance. The extended coverage from running the CPython 2.7 and CPython 3.2 (partially) test suites shows in a couple of bug fixes and general improvements in compatibility. Then there is a promised new feature that allows to compile whole packages. Also there is more Python3 compatibility, the CPython 3.2 test suite now succeeds up to "test_builtin.py", where it finds that ``str`` doesn't support the new parameters it has gained, future releases will improve on this. And then of course, more re-formulation work, in this case, class definitions are now mere simple functions. This and later function references, is the important and only progress towards type inference. Bug fixes --------- - The compiled method type can now be used with ``copy`` module. That means, instances with methods can now be copied too. `Issue#40 `__. Fixed in 0.3.23.1 already. - The ``assert`` statement as of Python2.7 creates the ``AssertionError`` object from a given value immediately, instead of delayed as it was with Python2.6. This makes a difference for the form with 2 arguments, and if the value is a tuple. `Issue#41 `__. Fixed in 0.3.23.1 already. - Sets written like this didn't work unless they were predicted at compile time: .. code-block:: python { value } This apparently rarely used Python2.7 syntax didn't have code generation yet and crashed the compiler. `Issue#42 `__. Fixed in 0.3.23.1 already. - For Python2, the default encoding for source files is ``ascii``, and it is now enforced by Nuitka as well, with the same ``SyntaxError``. - Corner cases of ``exec`` statements with nested functions now give proper ``SyntaxError`` exceptions under Python2. - The ``exec`` statement with a tuple of length 1 as argument, now also gives a ``TypeError`` exception under Python2. - For Python2, the ``del`` of a closure variable is a ``SyntaxError``. New Features ------------ - Added support creating compiled packages. If you give Nuitka a directory with an "__init__.py" file, it will compile that package into a ".so" file. Adding the package contents with ``--recurse-dir`` allows to compile complete packages now. Later there will be a cleaner interface likely, where the later is automatic. - Added support for providing directories as main programs. It's OK if they contain a "__main__.py" file, then it's used instead, otherwise give compatible error message. - Added support for optimizing the ``super`` built-in. It was already working correctly, but not optimized on CPython2. But for CPython3, the variant without any arguments required dedicated code. - Added support for optimizing the ``unicode`` built-in under Python2. It was already working, but will become the basis for the ``str`` built-in of Python3 in future releases. - For Python3, lots of compatibility work has been done. The Unicode issues appear to be ironed out now. The ``del`` of closure variables is allowed and supported now. Built-ins like ``ord`` and ``chr`` work more correctly and attributes are now interned strings, so that monkey patching classes works. Organizational -------------- - Migrated "bin/benchmark.sh" to Python as "misc/run-valgrind.py" and made it a bit more portable that way. Prefers "/var/tmp" if it exists and creates temporary files in a secure manner. Triggered by the Debian "insecure temp file" bug. - Migrated "bin/make-dependency-graph.sh" to Python as "misc/make-dependency-graph.py" and made a more portable and powerful that way. The filtering is done a more robust way. Also it creates temporary files in a secure manner, also triggered by the Debian "insecure temp file" bug. And it creates SVG files and no longer PostScript as the first one is more easily rendered these days. - Removed the "misc/gist" git sub-module, which was previously used by "misc/make-doc.py" to generate HTML from "`User Manual `__" and "`Developer Manual `__". These are now done with Nikola, which is much better at it and it integrates with the web site. - Lots of formatting improvements to the change log, and manuals: * Marking identifiers with better suited ReStructured Text markup. * Added links to the bug tracker all Issues. * Unified wordings, quotation, across the documents. Cleanups -------- - The creation of the class dictionaries is now done with normal function bodies, that only needed to learn how to throw an exception when directly called, instead of returning ``NULL``. Also the assignment of ``__module__`` and ``__doc__`` in these has become visible in the node tree, allowing their proper optimization. These re-formulation changes allowed to remove all sorts of special treatment of ``class`` code in the code generation phase, making things a lot simpler. - There was still a declaration of ``PRINT_ITEMS`` and uses of it, but no definition of it. - Code generation for "main" module and "other" modules are now merged, and no longer special. - The use of raw strings was found unnecessary and potentially still buggy and has been removed. The dependence on C++11 is getting less and less. New Tests --------- - Updated CPython2.6 test suite "tests/CPython26" to 2.6.8, adding tests for recent bug fixes in CPython. No changes to Nuitka were needed in order to pass, which is always good news. - Added CPython2.7 test suite as "tests/CPython27" from 2.7.3, making it public for the first time. Previously a private copy of some age, with many no longer needed changes had been used by me. Now it is up to par with what was done before for "tests/CPython26", so this pending action is finally done. - Added test to cover Python2 syntax error of having a function with closure variables nested inside a function that is an overflow function. - Added test "BuiltinSuper" to cover ``super`` usage details. - Added test to cover ``del`` on nested scope as syntax error. - Added test to cover ``exec`` with a tuple argument of length 1. - Added test to cover ``barry_as_FLUFL`` future import to work. - Removed "Unicode" from known error cases for CPython3.2, it's now working. Summary ------- This release brought forward the most important remaining re-formulation changes needed for Nuitka. Removing class bodies, makes optimization yet again simpler. Still, making function references, so they can be copied, is missing for value propagation to progress. Generally, as usual, a focus has been laid on correctness. This is also the first time, I am release with a known bug though: That is `Issue#39 `__ which I believe now, may be the root cause of the mercurial tests not yet passing. The solution will be involved and take a bit of time. It will be about "compiled frames" and be a (invasive) solution. It likely will make Nuitka faster too. But this release includes lots of tiny improvements, for Python3 and also for Python2. So I wanted to get this out now. As usual, please check it out, and let me know how you fare. Nuitka Release 0.3.23 ===================== This release is the one that completes the Nuitka "sun rise phase". All of Nuitka is now released under `Apache License 2.0 `__ which is a very liberal license, and compatible with basically all Free Software licenses there are. It's only asking to allow integration, of what you send back, and patent grants for the code. In the first phase of Nuitka development, I wanted to keep control over Nuitka, so it wouldn't repeat mistakes of other projects. This is no longer a concern for me, it's not going to happen anymore. I would like to thank Debian Legal team, for originally bringing to my attention, that this license will be better suited, than any copyright assignment could be. Bug fixes --------- - The compiled functions could not be used with ``multiprocessing`` or ``copy.copy``. `Issue#19 `__. Fixed in 0.3.22.1 already. - In-place operations for slices with not both bounds specified crashed the compiler. `Issue#36 `__. Fixed in 0.3.22.1 already. - Cyclic imports could trigger an endless loop, because module import expressions became the parent of the imported module object. `Issue#37 `__. Fixed in 0.3.22.2 already. - Modules named ``proc`` or ``func`` could not be compiled to modules or embedded due to a collision with identifiers of CPython2.7 includes. `Issue#38 `__. Fixed in 0.3.22.2 already. New Features ------------ - The fix for `Issue#19 `__ also makes pickling of compiled functions available. As it is the case for non-compiled functions in CPython, no code objects are stored, only names of module level variables. Organizational -------------- - Using the Apache License 2.0 for all of Nuitka now. - Speedcenter has been re-activated, but is not yet having a lot of benchmarks yet, subject to change. .. admonition:: Update We have given up on speedcenter meanwhile, and generate static pages with graphs instead. New Tests --------- - Changed the "CPython26" tests to no longer disable the parts that relied on copying of functions to work, as `Issue#19 `__ is now supported. - Extended in-place assignment tests to cover error cases of `Issue#36 `__. - Extended compile library test to also try and compile the path where ``numpy`` lives. This is apparently another path, where Debian installs some modules, and compiling this would have revealed `Issue#36 `__ sooner. Summary ------- The release contains bug fixes, and the huge step of changing `the license `__. It is made in preparation to `PyCON EU `__. Nuitka Release 0.3.22 ===================== This release is a continuation of the trend of previous releases, and added more re-formulations of Python that lower the burden on code generation and optimization. It also improves Python3 support substantially. In fact this is the first release to not only run itself under Python3, but for Nuitka to *compile itself* with Nuitka under Python3, which previously only worked for Python2. For the common language subset, it's quite fine now. Bug fixes --------- - List contractions produced extra entries on the call stack, after they became functions, these are no more existent. That was made possible my making frame stack entries an optional element in the node tree, left out for list contractions. - Calling a compiled function in an exception handler cleared the exception on return, it no longer does that. - Reference counter handling with generator ``throw`` method is now correct. - A module "builtins" conflicted with the handling of the Python ``builtins`` module. Those now use different identifiers. New Features ------------ - New ``metaclass`` syntax for the ``class`` statement works, and the old ``__metaclass__`` attribute is properly ignored. .. code-block:: python # Metaclass syntax in Python3, illegal in Python2 class X(metaclass = Y): pass .. code-block:: python # Metaclass syntax in Python2, no effect in Python3 class X: __metaclass__ = Y .. note:: The way to make a use of a metaclass in a portable way, is to create a based class that has it and then inherit from it. Sad, isn' it. Surely, the support for ``__metaclass__`` could still live. .. code-block:: python # For Python2/3 compatible source, we create a base class that has the # metaclass used and doesn't require making a choice. CPythonNodeMetaClassBase = NodeCheckMetaClass( "CPythonNodeMetaClassBase", (object,), {} ) - The ``--dump-xml`` option works with Nuitka running under Python3. This was not previously supported. - Python3 now also has compatible parameter errors and compatible exception error messages. - Python3 has changed scope rules for list contractions (assignments don't affect outside values) and this is now respected as well. - Python3 has gained support for recursive programs and stand alone extension modules, these are now both possible as well. New Optimization ---------------- - Avoid frame stack entries for functions that cannot raise exceptions, i.e. where they would not be used. This avoids overhead for the very simple functions. And example of this can be seen here: .. code-block:: python def simple(): return 7 - Optimize ``len`` built-in for non-constant, but known length values. An example can be seen here: .. code-block:: python # The range isn't constructed at compile time, but we still know its # length. len(range(10000000)) # The string isn't constructed at compile time, but we still know its # length. len("*" * 1000) # The tuple isn't constructed, instead it's known length is used, and # side effects are maintained. len((a(), b())) This new optimization applies to all kinds of container creations and the ``range`` built-in initially. - Optimize conditions for non-constant, but known truth values. At this time, known truth values of non-constants means ``range`` built-in calls with know size and container creations. An example can be seen here: .. code-block:: python if (a,): print "In Branch" It's clear, that the tuple will be true, we just need to maintain the side effect, which we do. - Optimize ``or`` and ``and`` operators for known truth values. See above for what has known truth values currently. This will be most useful to predict conditions that need not be evaluated at all due to short circuit nature, and to avoid checking against constant values. Previously this could not be optimized, but now it can: .. code-block:: python # The access and call to "something()" cannot possibly happen 0 and something() # Can be replaced with "something()", as "1" is true. If it had a side # effect, it would be maintained. 1 and something() # The access and call to "something()" cannot possibly happen, the value # is already decided, it's "1". 1 or something() # Can be replaced with "something()", as "0" is false. If it had a side # effect, it would be maintained. 0 or something() - Optimize print arguments to become strings. The arguments to ``print`` statements are now converted to strings at compile time if possible. .. code-block:: python print 1 becomes: .. code-block:: python print "1" - Combine print arguments to single ones. When multiple strings are printed, these are now combined. .. code-block:: python print "1+1=", 1+1 becomes: .. code-block:: python print "1+1= 2" Organizational -------------- - Enhanced Python3 support, enabling support for most basic tests. - Check files with PyLint in deterministic (alphabetical) order. Cleanups -------- - Frame stack entries are now part of the node tree instead of part of the template for every function, generator, class or module. - The ``try``/``except``/``else`` has been re-formulated to use an indicator variable visible in the node tree, that tells if a handler has been executed or not. - Side effects are now a dedicated node, used in several optimization to maintain the effect of an expression with known value. New Tests --------- - Expanded and adapted basic tests to work for Python3 as well. - Added reference count tests for generator functions ``throw``, ``send``, and ``close`` methods. - Cover calling a function with ``try``/``except`` in an exception handler twice. No test was previously doing that. Summary ------- This release offers enhanced compatibility with Python3, as well as the solution to many structural problems. Calculating lengths of large non-constant values at compile time, is technically a break through, as is avoiding lengthy calculations. The frame guards as nodes is a huge improvement, making that costly operational possible to be optimized away. There still is more work ahead, before value propagation will be safe enough to enable, but we are seeing the glimpse of it already. Not for long, and looking at numbers will make sense. Nuitka Release 0.3.21 ===================== This releases contains some really major enhancements, all heading towards enabling value propagation inside Nuitka. Assignments of all forms are now all simple and explicit, and as a result, now it will be easy to start tracking them. Contractions have become functions internally, with statements use temporary variables, complex unpacking statement were reduced to more simple ones, etc. Also there are the usual few small bug fixes, and a bunch of organizational improvements, that make the release complete. Bug fixes --------- - The built-in ``next`` could causes a program crash when iterating past the end of an iterator. `Issue#34 `__. Fixed in 0.3.20.1 already. - The ``set`` constants could cause a compiler error, as that type was not considered in the "mutable" check yet. Fixed in 0.3.20.2 already. - Performance regression. Optimize expression for exception types caught as well again, this was lost in last release. - Functions that contain ``exec``, are supposed to have a writable locals. But when removing that ``exec`` statement as part of optimization, this property of the function could get lost. - The so called "overflow functions" are once again correctly handled. These once were left behind in some refactoring and had not been repaired until now. An overflow function is a nested function with an ``exec`` or a star import. - The syntax error for ``return`` outside of a function, was not given, instead the code returned at run time. Fixed to raise a ``SyntaxError`` at compile time. New Optimization ---------------- - Avoid ``tuple`` objects to be created when catching multiple exception types, instead call exception match check function multiple times. - Removal of dead code following ``break``, ``continue``, ``return``, and ``raise``. Code that follows these statements, or conditional statements, where all branches end with it. .. note:: These may not actually occur often in actual code, but future optimization may produce them more frequently, and their removal may in turn make other possible optimization. - Detect module variables as "read only" after all writes have been detected to not be executed as removed. Previously the "read only indicator" was determined only once and then stayed the same. - Expanded conditional statement optimization to detect cases, where condition is a compile time constant, not just a constant value. - Optimize away assignments from a variable to the same variable, they have no effect. The potential side effect of accessing the variable is left intact though, so exceptions will be raised still. .. note:: An exception is where ``len = len`` actually does have an impact, because that variable becomes assignable. The "compile itself" test of Nuitka found that to happen with ``long`` from the ``nuitka.__past__`` module. - Created Python3 variant of quick ``unicode`` string access, there was no such thing in the CPython C/API, but we make the distinction in the source code, so it makes sense to have it. - Created an optimized implementation for the built-in ``iter`` with 2 parameters as well. This allows for slightly more efficient code to be created with regards to reference handling, rather than using the CPython C/API. - For all types of variable assigned in the generated code, there are now methods that accept already taken references or not, and the code generator picks the optimal variant. This avoids the drop of references, that e.g. the local variable will insist to take. - Don't use a "context" object for generator functions (and generator expressions) that don't need one. And even if it does to store e.g. the given parameter values, avoid to have a "common context" if there is no closure taken. This avoids useless ``malloc`` calls and speeds up repeated generator object creation. Organizational -------------- - Changed the Scons build file database to reside in the build directory as opposed to the current directory, not polluting it anymore. Thanks for the patch go to Michael H Kent, very much appreciated. - The ``--experimental`` option is no longer available outside of checkouts of git, and even there not on stable branches (``master``, ``hotfix/...``). It only pollutes ``--help`` output as stable releases have no experimental code options, not even development version will make a difference. - The binary "bin/Nuitka.py" has been removed from the git repository. It was deprecated a while ago, not part of the distribution and served no good use, as it was a symbolic link only anyway. - The ``--python-version`` option is applied at Nuitka start time to re-launch Nuitka with the given Python version, to make sure that the Python run time used for computations and link time Python versions are the same. The allowed values are now checked (2.6, 2.7 and 3.2) and the user gets a nice error with wrong values. - Added ``--keep-pythonpath`` alias for ``--execute-with-pythonpath`` option, probably easier to remember. - Support ``--debug`` with clang, so it can also be used to check the generated code for all warnings, and perform assertions. Didn't report anything new. - The contents environment variable ``CXX`` determines the default C++ compiler when set, so that checking with ``CXX=g++-4.7 nuitka-python ...`` has become supported. - The ``check-with-pylint`` script now has a real command line option to control the display of ``TODO`` items. Cleanups -------- - Changed complex assignments, i.e. assignments with multiple targets to such using a temporary variable and multiple simple assignments instead. .. code-block:: python a = b = c .. code-block:: python _tmp = c b = _tmp a = _tmp In CPython, when one assignment raises an exception, the whole thing is aborted, so the complexity of having multiple targets is no more needed, now that we have temporary variables in a block. All that was really needed, was to evaluate the complete source expression only once, but that made code generation contain ugly loops that are no more needed. - Changed unpacking assignments to use temporary variables. Code like this: .. code-block:: python a, b = c Is handled more like this: .. code-block:: python _tmp_iter = iter(c) _tmp1 = next(_tmp_iter) _tmp2 = next(_tmp_iter) if not finished(_tmp_iter): raise ValueError("too many values to unpack") a = _tmp1 b = _tmp2 In reality, not really ``next`` is used, as it wouldn't raise the correct exception for unpacking, and the ``finished`` check is more condensed into it. Generally this cleanup allowed that the ``AssignTargetTuple`` and associated code generation was removed, and in the future value propagation may optimize these ``next`` and ``iter`` calls away where possible. At this time, this is not done yet. - Exception handlers assign caught exception value through assignment statement. Previously the code generated for assigning from the caught exception was not considered part of the handler. It now is the first statement of an exception handler or not present, this way it may be optimized as well. - Exception handlers now explicitly catch more than one type. Catching multiple types worked by merits of the created tuple object working with the Python C/API function called, but that was not explicit at all. Now every handler has a tuple of exceptions it catches, which may only be one, or if None, it's all. - Contractions are now functions as well. Contractions (list, dict, and set) are now re-formulated as function bodies that contain for loops and conditional statements. This allowed to remove a lot of special code that dealt with them and will make these easier to understand for optimization and value propagation. - Global is handled during tree building. Previously the global statement was its own node, which got removed during the optimization phase in a dedicated early optimization that applied its effect, and then removed the node. It was determined, that there is no reason to not immediately apply the effect of the global variable and take closure variables and add them to the provider of that ``global`` statement, allowing to remove the node class. - Read only module variable detection integrated to constraint collection. The detection of read only module variables was so far done as a separate step, which is no more necessary as the constraint collection tracks the usages of module variables anyway, so this separate and slow step could be removed. New Tests --------- - Added test to cover order of calls for complex assignments that unpack, to see that they make a fresh iterator for each part of a complex assignment. - Added test that unpacks in an exception catch. It worked, due to the generic handling of assignment targets by Nuitka, and I didn't even know it can be done, example: .. code-block:: python try: raise ValueError(1,2) except ValueError as (a,b): print "Unpacking caught exception and unpacked", a, b Will assign ``a=1`` and ``b=2``. - Added test to cover return statements on module level and class level, they both must give syntax errors. - Cover exceptions from accessing unassigned global names. - Added syntax test to show that star imports do not allow other names to be imported at the same time as well. - Python3 is now also running the compile itself test successfully. Summary ------- The progress made towards value propagation and type inference is *very* significant, and makes those appears as if they are achievable. Nuitka Release 0.3.20 ===================== This time there are a few bug fixes and some really major cleanups, lots of new optimization and preparations for more. And then there is a new compiler clang and a new platform supported. MacOS X appears to work mostly, thanks for the patches from Pete Hunt. Bug fixes --------- - The use of a local variable name as an expression was not covered and lead to a compiler crash. Totally amazing, but true, nothing in the test suite of CPython covered this. `Issue#30 `__. Fixed in release 0.3.19.1 already. - The use of a closure variable name as an expression was not covered as well. And in this case corrupted the reference count. `Issue#31 `__. Fixed in release 0.3.19.1 already. - The ``from x import *`` attempted to respect ``__all__`` but failed to do so. `Issue#32 `__. Fixed in release 0.3.19.2 already. - The ``from x import *`` didn't give a ``SyntaxError`` when used on Python3. Fixed in release 0.3.19.2 already. - The syntax error messages for "global for function argument name" and "duplicate function argument name" are now identical as well. - Parameter values of generator function could cause compilation errors when used in the closure of list contractions. Fixed. New Features ------------ - Added support for disabling the console for Windows binaries. Thanks for the patch go to Michael H Kent. - Enhanced Python3 support for syntax errors, these are now also compatible. - Support for MacOS X was added. - Support for using the clang compiler was added, it can be enforced via ``--clang`` option. Currently this option is mainly intended to allow testing the "MacOS X" support as good as possible under Linux. New Optimization ---------------- - Enhanced all optimization that previously worked on "constants" to work on "compile time constants" instead. A "compile time constant" can currently also be any form of a built-in name or exception reference. It is intended to expand this in the future. - Added support for built-ins ``bin``, ``oct``, and ``hex``, which also can be computed at compile time, if their arguments are compile time constant. - Added support for the ``iter`` built-in in both forms, one and two arguments. These cannot be computed at compile time, but now will execute faster. - Added support for the ``next`` built-in, also in its both forms, one and two arguments. These also cannot be computed at compile time, but now will execute faster as well. - Added support for the ``open`` built-in in all its form. We intend for future releases to be able to track file opens for including them into the executable if data files. - Optimize the ``__debug__`` built-in constant as well. It cannot be assigned, yet code can determine a mode of operation from it, and apparently some code does. When compiling the mode is decided. - Optimize the ``Ellipsis`` built-in constant as well. It falls in the same category as ``True``, ``False``, ``None``, i.e. names of built-in constants that a singletons. - Added support for anonymous built-in references, i.e. built-ins which have names that are not normally accessible. An example is ``type(None)`` which is not accessible from anywhere. Other examples of such names are ``compiled_method_or_function``. Having these as represented internally, and flagged as "compile time constants", allows the compiler to make more compile time optimization and to generate more efficient C++ code for it that won't e.g. call the ``type`` built-in with ``None`` as an argument. - All built-in names used in the program are now converted to "built-in name references" in a first step. Unsupported built-ins like e.g. ``zip``, for which Nuitka has no own code or understanding yet, remained as "module variables", which made access to them slow, and difficult to recognize. - Added optimization for module attributes ``__file__``, ``__doc__`` and ``__package__`` if they are read only. It's the same as ``__name__``. - Added optimization for slices and subscripts of "compile time constant" values. These will play a more important role, once value propagation makes them more frequent. Organizational -------------- - Created a "change log" from the previous release announcements. It's as ReStructured Text and converted to PDF for the release as well, but I chose not to include that in Debian, because it's so easy to generate the PDF on that yourself. - The posting of release announcements is now prepared by a script that converts the ReStructured Text to HTML and adds it to Wordpress as a draft posting or updates it, until it's release time. Simple, sweet and elegant. Cleanups -------- - Split out the ``nuitka.nodes.Nodes`` module into many topic nodes, so that there are now ``nuitka.nodes.BoolNodes`` or ``nuitka.nodes.LoopNodes`` to host nodes of similar kinds, so that it is now cleaner. - Split ``del`` statements into their own node kind, and use much simpler node structures for them. The following blocks are absolutely the same: .. code-block:: python del a, b.c, d .. code-block:: python del a del b.c del d So that's now represented in the node tree. And even more complex looking cases, like this one, also the same: .. code-block:: python del a, (b.c, d) This one gives a different parse tree, but the same bytecode. And so Nuitka need no longer concern itself with this at all, and can remove the tuple from the parse tree immediately. That makes them easy to handle. As you may have noted already, it also means, there is no way to enforce that two things are deleted or none at all. - Turned the function and class builder statements into mere assignment statements, where defaults and base classes are handled by wrapping expressions. Previously they are also kind of assignment statements too, which is not needed. Now they were reduced to only handle the ``bases`` for classes and the ``defaults`` for functions and make optional. - Refactored the decorator handling to the tree building stage, presenting them as function calls on "function body expression" or class body expression". This allowed to remove the special code for decorators from code generation and C++ templates, making decorations easy subjects for future optimization, as they practically are now just function calls. .. code-block:: python @some_classdecorator class C: @staticmethod def f(): pass It's just a different form of writing things. Nothing requires the implementation of decorators, it's just functions calls with function bodies before the assignment. The following is only similar: .. code-block:: python class C: def f(): pass f = staticmethod(f) C = some_classdecorator(C) It's only similar, because the assignment to an intermediate value of ``C`` and ``f`` is not done, and if an exception was raised by the decoration, that name could persist. For Nuitka, the function and class body, before having a name, are an expression, and so can of course be passed to decorators already. - The in-place assignments statements are now handled using temporary variable blocks Adding support for scoped temporary variables and references to them, it was possible to re-formulate in-place assignments expressions as normal look-ups, in-place operation call and then assignment statement. This allowed to remove static templates and will yield even better generated code in the future. - The for loop used to have has a "source" expression as child, and the iterator over it was only taken at the code generation level, so that step was therefore invisible to optimization. Moved it to tree building stage instead, where optimization can work on it then. - Tree building now generally allows statement sequences to be ``None`` everywhere, and pass statements are immediately eliminated from them immediately. Empty statement sequences are now forbidden to exist. - Moved the optimization for ``__name__`` to compute node of variable references, where it doesn't need anything complex to replace with the constant value if it's only read. - Added new bases classes and mix-in classes dedicated to expressions, giving a place for some defaults. - Made the built-in code more reusable. New Tests --------- - Added some more diagnostic tests about complex assignment and ``del`` statements. - Added syntax test for star import on function level, that must fail on Python3. - Added syntax test for duplicate argument name. - Added syntax test for global on a function argument name. Summary ------- The decorator and building changes, the assignment changes, and the node cleanups are all very important progress for the type inference work, because they remove special casing the that previously would have been required. Lambdas and functions now really are the same thing right after tree building. The in-place assignments are now merely done using standard assignment code, the built functions and classes are now assigned to names in assignment statements, much *more* consistency there. Yet, even more work will be needed in the same direction. There may e.g. be work required to cover ``with`` statements as well. And assignments will become no more complex than unpacking from a temporary variable. For this release, there is only minimal progress on the Python3 front, despite the syntax support, which is only miniscule progress. The remaining tasks appear all more or less difficult work that I don't want to touch now. There are still remaining steps, but we can foresee that a release may be done that finally actually does type inference and becomes the effective Python compiler this project is all about. Nuitka Release 0.3.19 ===================== This time there are a few bug fixes, major cleanups, more Python3 support, and even new features. A lot things in this are justifying a new release. Bug fixes --------- - The man pages of ``nuitka`` and ``nuitka-python`` had no special layout for the option groups and broken whitespace for ``--recurse-to`` option. Also ``--g++-only`` was only partially bold. Released as 0.3.18.1 hot fix already. - The command line length improvement we made to Scons for Windows was not portable to Python2.6. Released as 0.3.18.2 hot fix already. - Code to detect already considered packages detection was not portable to Windows, for one case, there was still a use of ``/`` instead of using a ``joinpath`` call. Released as 0.3.18.3 already. - A call to the range built-in with no arguments would crash the compiler, see `Issue#29 `__. Released as 0.3.18.4 already. - Compatibility Fix: When rich comparison operators returned false value other ``False``, for comparison chains, these would not be used, but ``False`` instead, see . - The support for ``__import__`` didn't cover keyword arguments, these were simply ignored. See `Issue#28 `__. Fixed, but no warning is given yet. New Features ------------ - A new option has been added, one can now specify ``--recurse-directory`` and Nuitka will attempt to embed these modules even if not obviously imported. This is not yet working perfect yet, but will receive future improvements. - Added support for the ``exec`` built-in of Python3, this enables us to run one more basic test, ``GlobalStatement.py`` with Python3. The test ``ExecEval.py`` nearly works now. New Optimization ---------------- - The no arguments ``range()`` call now optimized into the static CPython exception it raises. - Parts of comparison chains with constant arguments are now optimized away. Cleanups -------- - Simplified the ``CPythonExpressionComparison`` node, it now always has only 2 operands. If there are more, the so called "comparison chain", it's done via ``and`` with assignments to temporary variables, which are expressed by a new node type ``CPythonExpressionTempVariableRef``. This allowed to remove ``expression_temps`` from C++ code templates and generation, reducing the overall complexity. - When executing a module (``--execute`` but not ``--exe``), no longer does Nuitka import it into itself, instead a new interpreter is launched with a fresh environment. - The calls to the variadic ``MAKE_TUPLE`` were replaced with calls the ``MAKE_TUPLExx`` (where ``xx`` is the number of arguments), that are generated on a as-needed basis. This gives more readable code, because no ``EVAL_ORDERED_xx`` is needed at call site anymore. - Many node classes have moved to new modules in ``nuitka.nodes`` and grouped by theme. That makes them more accessible. - The choosing of the debug python has moved from Scons to Nuitka itself. That way it can respect the ``sys.abiflags`` and works with Python3. - The replacing of ``.py`` in filenames was made more robust. No longer is ``str.replace`` used, but instead proper means to assure that having ``.py`` as other parts of the filenames won't be a trouble. - Module recursion was changed into its own module, instead of being hidden in the optimization that considers import statements. - As always, some PyLint work, and some minor ``TODO`` were solved. Organizational -------------- - Added more information to the "`Developer Manual `__", e.g. documenting the tree changes for ``assert`` to become a conditional statement with a raise statement, etc. - The Debian package is as of this version verified to be installable and functional on to Ubuntu Natty, Maverick, Oneiric, and Precise. - Added support to specify the binary under test with a ``NUITKA`` environment, so the test framework can run with installed version of Nuitka too. - Made sure the test runners work under Windows as well. Required making them more portable. And a workaround for ``os.execl`` not propagating exit codes under Windows. See `Issue#26 `__ for more information. - For windows target the MinGW library is now linked statically. That means there is no requirement for MinGW to be in the ``PATH`` or even installed to execute the binary. New Tests --------- - The ``basic``, ``programs``, ``syntax``, and ``reflected`` were made executable under Windows. Occasionally this meant to make the test runners more portable, or to work around limitations. - Added test to cover return values of rich comparisons in comparison chains, and order of argument evaluation for comparison chains. - The ``Referencing.py`` test was made portable to Python3. - Cover no arguments ``range()`` exception as well. - Added test to demonstrate that ``--recurse-directory`` actually works. This is using an ``__import__`` that cannot be predicted at run time (yet). - The created source package is now tested on pbuilder chroots to be pass installation and the basic tests, in addition to the full tests during package build time on these chroots. This will make sure, that Nuitka works fine on Ubuntu Natty and doesn't break without notice. Summary ------- This releases contains many changes. The "temporary variable ref" and "assignment expression" work is ground breaking. I foresee that it will lead to even more simplifications of code generation in the future, when e.g. in-place assignments can be reduced to assignments to temporary variables and conditional statements. While there were many improvements related to Windows support and fixing portability bugs, or the Debian package, the real focus is the optimization work, which will ultimately end with "value propagation" working. These are the real focus. The old comparison chain handling was a big wart. Working, but no way understood by any form of analysis in Nuitka. Now they have a structure which makes their code generation based on semantics and allows for future optimization to see through them. Going down this route is an important preparatory step. And there will be more work like this needed. Consider e.g. handling of in-place assignments. With an "assignment expression" to a "temporary variable ref", these become the same as user code using such a variable. There will be more of these to find. So, that is where the focus is. The release now was mostly aiming at getting involved fixes out. The bug fixed by comparison chain reworking, and the ``__import__`` related one, were not suitable for hot fix releases, so that is why the 0.3.19 release had to occur now. But with plugin support, with this comparison chain cleanup, with improved Python3 support, and so on, there was plenty of good stuff already, also worth to get out. Nuitka Release 0.3.18 ===================== This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, and the important step that triggered the release: Nuitka has entered Debian Unstable. So you if want, you will get stable Nuitka releases from now on via ``apt-get install nuitka``. The release cycle was too short to have much focus. It merely includes fixes, which were available as hot fixes, and some additional optimization and node tree cleanups, as well as source cleanups. But not much else. Bug fixes --------- - Conditional statements with both branches empty were not optimized away in all cases, triggering an assertion of code generation. `Issue#16 `__. Released as 0.3.17a hot fix already. - Nuitka was considering directories to contain packages that had no "__init__.py" which could lead to errors when it couldn't find the package later in the compilation process. Released as 0.3.17a hot fix already. - When providing ``locals()`` to ``exec`` statements, this was not making the ``locals()`` writable. The logic to detect the case that default value is used (None) and be pessimistic about it, didn't consider the actual value ``locals()``. Released as 0.3.17b hot fix already. - Compatibility Fix: When no defaults are given, CPython uses ``None`` for ``func.func_defaults``, but Nuitka had been using ``None``. New Optimization ---------------- - If the condition of assert statements can be predicted, these are now optimized in a static raise or removed. - For built-in name references, there is now dedicated code to look them up, that doesn't check the module level at all. Currently these are used in only a few cases though. - Cleaner code is generated for the simple case of ``print`` statements. This is not only faster code, it's also more readable. Cleanups -------- - Removed the ``CPythonStatementAssert`` node. It's not needed, instead at tree building, assert statements are converted to conditional statements with the asserted condition result inverted and a raise statement with ``AssertionError`` and the assertion argument. This allowed to remove code and complexity from the subsequent steps of Nuitka, and enabled existing optimization to work on assert statements as well. - Moved built-in exception names and built-in names to a new module ``nuitka.Builtins`` instead of having in other places. This was previously a bit spread-out and misplaced. - Added cumulative ``tags`` to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks. - New node for built-in name loopups, which allowed to remove tricks played with adding module variable lookups for ``staticmethod`` when adding them for ``__new__`` or module variable lookups for ``str`` when predicting the result of ``type('a')``, which was unlikely to cause a problem, but an important ``TODO`` item still. Organizational -------------- - The `"Download" <../pages/download.html>`__ page is now finally updated for releases automatically. This closes `Issue#7 `__ completely. Up to this release, I had to manually edit that page, but now mastered the art of upload via XMLRCP and a Python script, so that don't loose as much time with editing, checking it, etc. - The Debian package is backportable to Ubuntu Natty, Maverick, Oneiric, I expect to make a separate announcement with links to packages. - Made sure the test runners worth with bare ``python2.6`` as well. New Tests --------- - Added some tests intended for type inference development. Summary ------- This releases contains not as much changes as others, mostly because it's the intended base for a Debian upload. The ``exec`` fix was detected by continued work on the branch ``feature/minimize_CPython26_tests_diff`` branch, but that work is now complete. It is being made pretty (many git rebase iterations) with lots of Issues being added to the bug tracker and referenced for each change. The intention is to have a clean commits repository with the changed made. But of course, the real excitement is the "type inference" work. It will give a huge boost to Nuitka. With this in place, new benchmarks may make sense. I am working on getting it off the ground, but also to make us more efficient. So when I learn something. e.g. ``assert`` is not special, I apply it to the ``develop`` branch immediately, to keep the differences as small as possible, and to immediately benefit from such improvements. Nuitka Release 0.3.17 ===================== This is to inform you about the new stable release of Nuitka. This time there are a few bug fixes, lots of very important organisational work, and yet again improved compatibility and cleanups. Also huge is the advance in making ``--deep`` go away and making the recursion of Nuitka controllable, which means a lot for scalability of projects that use a lot of packages that use other packages, because now you can choose which ones to embed and which ones one. The release cycle had a focus on improving the quality of the test scripts, the packaging, and generally to prepare the work on "type inference" in a new feature branch. I have also continued to work towards CPython3.2 compatibility, and this version, while not there, supports Python3 with a large subset of the basic tests programs running fine (of course via ``2to3`` conversion) without trouble. There is still work to do, exceptions don't seem to work fully yet, parameter parsing seems to have changed, etc. but it seems that CPython3.2 is going to work one day. And there has been a lot of effort, to address the Debian packaging to be cleaner and more complete, addressing issues that prevented it from entering the Debian repository. Bug fixes --------- - Fixed the handling of modules and packages of the same name, but with different casing. Problem showed under Windows only. Released as 0.3.16a hot fix already. - Fixed an error where the command line length of Windows was exceeded when many modules were embedded, Christopher Tott provided a fix for it. Released as 0.3.16a hot fix already. - Fix, avoid to introduce new variables for where built-in exception references are sufficient. Released as 0.3.16b hot fix already. - Fix, add the missing ``staticmethod`` decorator to ``__new__`` methods before resolving the scopes of variables, this avoids the use of that variable before it was assigned a scope. Released as 0.3.16b hot fix already. New Features ------------ - Enhanced compatibility again, provide enough ``co_varnames`` in the code objects, so that slicing them up to ``code_object.co_argcount`` will work. They are needed by ``inspect`` module and might be used by some decorators as well. - New options to control the recursion: ``--recurse-none`` (do not warn about not-done recursions) ``--recurse-all`` (recurse to all otherwise warned modules) ``--recurse-to`` (confirm to recurse to those modules) ``--recurse-not-to`` (confirm to not recurse to those modules) New Optimization ---------------- - The optimization of constant conditional expressions was not done yet. Added this missing constant propagation case. - Eliminate near empty statement sequences (only contain a pass statement) in more places, giving a cleaner node structure for many constructs. - Use the pickle "protocol 2" on CPython2 except for ``unicode`` strings where it does not work well. It gives a more compressed and binary representation, that is generally more efficient to un-stream as well. Also use the cPickle protocol, the use of ``pickle`` was not really necessary anymore. Organizational -------------- - Added a "`Developer Manual `__" to the release. It's incomplete, but it details some of the existing stuff, coding rules, plans for "type inference", etc. - Improved the ``--help`` output to use ``metavar`` where applicable. This makes it more readable for some options. - Instead of error message, give help output when no module or program file name was given. This makes Nuitka help out more convenient. - Consistently use ``#!/usr/bin/env python`` for all scripts, this was previously only done for some of them. - Ported the PyLint check script to Python as well, enhancing it on the way to check the exit code, and to only output changes things, as well as making the output of warnings for ``TODO`` items optional. - All scripts used for testing, PyLint checking, etc. now work with Python3 as well. Most useful on Arch Linux, where it's also already the default for ``Python``. - The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together. - Make the tests run without any dependence on ``PATH`` to contain the executables of Nuitka. This makes it easier to use. - Add license texts to 3rd party file that were missing them, apply ``licensecheck`` results to cleanup Nuitka. Also removed own copyright statement from in-line copy of Scons, it had been added by accident only. - Release the tests that I own as well as the Debian packaging I created under "Apache License 2.0" which is very liberal, meaning every project will be able to use it. - Don't require copyright assignment for contributions anymore, instead only "Apache License 2.0", the future Nuitka license, so that the code won't be a problem when changing the license of all of Nuitka to that license. - Give contributors listed in the "`User Manual `__" an exception to the GPL terms until Nuitka is licensed under "Apache License 2.0" as well. - Added an ``--experimental`` option which can be used to control experimental features, like the one currently being added on ``feature/ctypes_annotation``, where "type inference" is currently only activated when that option is given. For this stable release, it does nothing. - Check the static C++ files of Nuitka with ``cppcheck`` as well. Didn't find anything. - Arch Linux packages have been contributed, these are linked for download, but the stable package may lag behind a bit. Cleanups -------- - Changed ``not`` boolean operation to become a normal operator. Changed ``and`` and ``or`` boolean operators to a new base class, and making their interface more similar to that of operations. - Added cumulative ``tags`` to node classes for use in checks. Use it annotate which node kinds to visit in e.g. per scope finalization steps. That avoids kinds and class checks. - Enhanced the "visitor" interface to provide more kinds of callbacks, enhanced the way "each scope" visiting is achieved by generalizing is as "child has not tag 'closure_taker'" and that for every "node that has tag 'closure_taker'". - Moved ``SyntaxHighlighting`` module to ``nuitka.gui`` package where it belongs. - More white listing work for imports. As recursion is now the default, and leads to warnings for non-existent modules, the CPython tests gave a lot of good candidates for import errors that were white listed. - Consistently use ``nuitka`` in test scripts, as there isn't a ``Nuitka.py`` on all platforms. The later is scheduled for removal. - Some more PyLint cleanups. New Tests --------- - Make sure the basic tests pass with CPython or else fail the test. This is to prevent false positives, where a test passes, but only because it fails in CPython early on and then does so with Nuitka too. For the syntax tests we make sure they fail. - The basic tests can now be run with ``PYTHON=python3.2`` and use ``2to3`` conversion in that case. Also the currently not passing tests are not run, so the passing tests continue to do so, with this run from the release test script ``check-release``. - Include the syntax tests in release tests as well. - Changed many existing tests so that they can run under CPython3 too. Of course this is via ``2to3`` conversion. - Don't fail if the CPython test suites are not there. Currently they remain largely unpublished, and as such are mostly only available to me (exception, ``feature/minimize_CPython26_tests_diff`` branch references the CPython2.6 tests repository, but that remains work in progress). - For the compile itself test: Make the presence of the Scons in-line copy optional, the Debian package doesn't contain it. - Also make it more portable, so it runs under Windows too, and allow to choose the Python version to test. Check this test with both CPython2.6 and CPython2.7 not only the default Python. - Before releasing, test that the created Debian package builds fine in a minimal Debian ``unstable`` chroot, and passes all the tests included in the package (``basics``, ``syntax``, ``programs``, ``reflected``). Also many other Debian packaging improvements. Summary ------- The "git flow" was used again in this release cycle and proved to be useful not only for hot fix, but also for creating the branch ``feature/ctypes_annotation`` and rebasing it often while things are still flowing. The few hot fixes didn't require a new release, but the many organizational improvements and the new features did warrant the new release, because of e.g. the much better test handling in this release and the improved recursion control. The work on Python3 support has slowed down a bit. I mostly only added some bits for compatibility, but generally it has slowed down. I wanted to make sure it doesn't regress by accident, so running with CPython3.2 is now part of the normal release tests. What's still missing is more "hg" completeness. Only the ``co_varnames`` work for ``inspect`` was going in that direction, and this has slowed down. It was more important to make Nuitka's recursion more accessible with the new options, so that was done first. And of course, the real excitement is the "type inference" work. It will give a huge boost to Nuitka, and I am happy that it seems to go well. With this in place, new benchmarks may make sense. I am working on getting it off the ground, so other people can work on it too. My idea of ``ctypes`` native calls may become true sooner than expected. To support that, I would like to add more tools to make sure we discover changes earlier on, checking the XML representations of tests to discover improvements and regressions more clearly. Nuitka Release 0.3.16 ===================== This time there are many bug fixes, some important scalability work, and again improved compatibility and cleanups. The release cycle had a focus on fixing the bug reports I received. I have also continued to look at CPython3 compatibility, and this is the first version to support Python3 somewhat, at least some of the basic tests programs run (of course via ``2to3`` conversion) without trouble. I don't know when, but it seems that it's going to work one day. Also there has an effort to make the Debian packaging cleaner, addressing all kinds of small issues that prevented it from entering the Debian repository. It's still not there, but it's making progress. Bug fixes --------- - Fixed a packaging problem for Linux and x64 platform, the new ``swapFiber.S`` file for the fiber management was not included. Released as 0.3.15a hot fix already. - Fixed an error where optimization was performed on removed unreachable code, which lead to an error. Released as 0.3.15b hot fix already. - Fixed an issue with ``__import__`` and recursion not happening in any case, because when it did, it failed due to not being ported to new internal APIs. Released as 0.3.15c hot fix already. - Fixed ``eval()`` and ``locals()`` to be supported in generator expressions and contractions too. Released as 0.3.15d hot fix already. - Fixed the Windows batch files ``nuitka.bat`` and ``nuitka-python.bat`` to not output the ``rem`` statements with the copyright header. Released as 0.3.15d hot fix already. - Fixed re-raise with ``raise``, but without a current exception set. Released as 0.3.15e hot fix already. - Fixed ``vars()`` call on the module level, needs to be treated as ``globals()``. Released as 0.3.15e hot fix already. - Fix handling of broken new lines in source files. Read the source code in "universal line ending mode". Released as 0.3.15f hot fix already. - Fixed handling of constant module attribute ``__name__`` being replaced. Don't replace local variables of the same name too. Released as 0.3.15g hot fix already. - Fixed assigning to ``True``, ``False`` or ``None``. There was this old ``TODO``, and some code has compatibility craft that does it. Released as 0.3.15g hot fix already. - Fix constant dictionaries not always being recognized as shared. Released as 0.3.15g hot fix already. - Fix generator function objects to not require a return frame to exist. In finalize cleanup it may not. - Fixed non-execution of cleanup codes that e.g. flush ``sys.stdout``, by adding ``Py_Finalize()``. - Fix ``throw()`` method of generator expression objects to not check arguments properly. - Fix missing fallback to subscript operations for slicing with non-indexable objects. - Fix, in-place subscript operations could fail to apply the update, if the intermediate object was e.g. a list and the handle just not changed by the operation, but e.g. the length did. - Fix, the future spec was not properly preserving the future division flag. New Optimization ---------------- - The optimization scales now much better, because per-module optimization only require the module to be reconsidered, but not all modules all the time. With many modules recursed into, this makes a huge difference in compilation time. - The creation of dictionaries from constants is now also optimized. New Features ------------ - As a new feature functions now have the ``func_defaults`` and ``__defaults__`` attribute. It works only well for non-nested parameters and is not yet fully integrated into the parameter parsing. This improves the compatibility somewhat already though. - The names ``True``, ``False`` and ``None`` are now converted to constants only when they are read-only module variables. - The ``PYTHONPATH`` variable is now cleared when immediately executing a compiled binary unless ``--execute-with-pythonpath`` is given, in which case it is preserved. This allows to make sure that a binary is in fact containing everything required. Organizational -------------- - The help output of Nuitka was polished a lot more. It is now more readable and uses option groups to combine related options together. - The in-line copy of Scons is not checked with PyLint anymore. We of course don't care. - Program tests are no longer executed in the program directory, so failed module inclusions become immediately obvious. - The basic tests can now be run with ``PYTHON=python3.2`` and use ``2to3`` conversion in that case. Cleanups -------- - Moved ``tags`` to a separate module, make optimization emit only documented tags, checked against the list of allowed ones. - The Debian package has seen lots of improvements, to make it "lintian clean", even in pedantic mode. The homepage of Nuitka is listed, a watch file can check for new releases, the git repository and the gitweb are referenced, etc. - Use ``os.path.join`` in more of the test code to achieve more Windows portability for them. - Some more PyLint cleanups. New Tests --------- - There is now a ``Crasher`` test, for tests that crashed Nuitka previously. - Added a program test where the imported module does a ``sys.exit()`` and make sure it really doesn't continue after the ``SystemExit`` exception that creates. - Cover the type of ``__builtins__`` in the main program and in imported modules in tests too. It's funny and differs between module and dict in CPython2. - Cover a final ``print`` statement without newline in the test. Must still receive a newline, which only happens when ``Py_Finalize()`` is called. - Added test with functions that makes a ``raise`` without an exception set. - Cover the calling of ``vars()`` on module level too. - Cover the use of eval in contractions and generator expressions too. - Cover ``func_defaults`` and ``__default__`` attributes for a function too. - Added test function with two ``raise`` in an exception handler, so that one becomes dead code and removed without the crash. Summary ------- The "git flow" was really great in this release cycle. There were many hot fix releases being made, so that the bugs could be addressed immediately without requiring the overhead of a full release. I believe that this makes Nuitka clearly one of the best supported projects. This quick turn-around also encourages people to report more bugs, which is only good. And the structure is there to hold it. Of course, the many bug fixes meant that there is not as much new development, but that is not the priority, correctness is. The work on Python3 is a bit strange. I don't need Python3 at all. I also believe it is that evil project to remove cruft from the Python core and make developers of all relevant Python software, add compatibility cruft to their software instead. Yet, I can't really stop to work on it. It has that appeal of small fixups here and there, and then something else works too. Python3 work is like when I was first struggling with Nuitka to pass the CPython2 unit tests for a first time. It's fun. And then it finds real actual bugs that apply to CPython2 too. Not doing ``Py_Finalize`` (but having to), the slice operations shortcomings, the bug of subscript in-place, and so on. There is likely more things hidden, and the earlier Python3 is supported, the more benefit from increased test covered. What's missing is more "hg" completeness. I think only the ``raise`` without exception set and the ``func_defaults`` issue were going into its direction, but it won't be enough yet. Nuitka Release 0.3.15 ===================== This is to inform you about the new stable release of Nuitka. This time again many organizational improvements, some bug fixes, much improved compatibility and cleanups. This release cycle had a focus on packaging Nuitka for easier consumption, i.e. automatic packaging, making automatic uploads, improvement documentation, and generally cleaning things up, so that Nuitka becomes more compatible and ultimately capable to run the "hg" test suite. It's not there yet, but this is a huge jump for usability of Nuitka and its compatibility, again. Then lots of changes that make Nuitka approach Python3 support, the generated C++ for at least one large example is compiling with this new release. It won't link, but there will be later releases. And there is a lot of cleanup going on, geared towards compatibility with line numbers in the frame object. Bug fixes --------- - The main module was using ``__main__`` in tracebacks, but it must be ````. Released as 0.3.14a hot fix already. - Workaround for "execfile cannot be used as an expression". It wasn't possible to use ``execfile`` in an expression, only as a statement. But then there is crazy enough code in e.g. mercurial that uses it in a lambda function, which made the issue more prominent. The fix now allows it to be an expression, except on the class level, which wasn't seen yet. - The in-line copy of Scons was not complete enough to work for "Windows" or with ``--windows-target`` for cross compile. Fixed. - Cached frames didn't release the "back" frame, therefore holding variables of these longer than CPython does, which could cause ordering problems. Fixed for increased compatibility. - Handle "yield outside of function" syntax error in compiled source correctly. This one was giving a Nuitka backtrace, now it gives a ``SyntaxError`` as it needs to. - Made syntax/indentation error output absolutely identical to CPython. - Using the frame objects ``f_lineno`` may fix endless amounts bugs related to traceback line numbers. New Features ------------ - Guesses the location of the MinGW compiler under Windows to default install location, so it need not be added to ``PATH`` environment variable. Removes the need to modify ``PATH`` environment just for Nuitka to find it. - Added support for "lambda generators". You don't want to know what it is. Lets just say, it was the last absurd language feature out there, plus that didn't work. It now works perfect. Organizational -------------- - You can now download a Windows installer and a Debian package that works on Debian Testing, current Ubuntu and Mint Linux. - New release scripts give us the ability to have hot fix releases as download packages immediately. That means the "git flow" makes even more beneficial to the users. - Including the generated "README.pdf" in the distribution archives, so it can be read instead of "README.txt". The text file is fairly readable, due to the use of ReStructured Text, but the PDF is even nicer to read, due to e.g. syntax highlighting of the examples. - Renamed the main binaries to ``nuitka`` and ``nuitka-python``, so that there is no dependency on case sensitive file systems. - For Windows there are batch files ``nuitka.bat`` and ``nuitka-python.bat`` to make Nuitka directly executable without finding the ``Python.exe``, which the batch files can tell from their own location. - There are now man pages of ``nuitka`` and ``nuitka-python`` with examples for the most common use cases. They are of course included in the Debian package. - Don't strip the binary when executing it to analyse compiled binary with ``valgrind``. It will give better information that way, without changing the code. New Optimization ---------------- - Implemented ``swapcontext`` alike (``swapFiber``) for x64 to achieve 8 times speedup for Generators. It doesn't do useless syscalls to preserve signal masks. Now Nuitka is faster at frame switching than CPython on x64, which is already good by design. Cleanups -------- - Using the frame objects to store current line of execution avoids the need to store it away in helper code at all. It ought to also help a lot with threading support, and makes Nuitka even more compatible, because now line numbers will be correct even outside tracebacks, but for mere stack frame dumps. - Moved the ``for_return`` detection from code generation to tree building where it belongs. Yield statements used as return statements need slightly different code for Python2.6 difference. That solved an old ``TODO``. - Much Python3 portability work. Sometimes even improving existing code, the Python compiler code had picked up a few points, where the latest Nuitka didn't work with Python3 anymore, when put to actual compile. The test covered only syntax, but e.g. meta classes need different code in CPython3, and that's now supported. Also helper code was made portable in more places, but not yet fully. This will need more work. - Cleaned up uses of debug defines, so they are now more consistent and in one place. - Some more PyLint cleanups. New Tests --------- - The tests are now executed by Python scripts and cover ``stderr`` output too. Before we only checked ``stdout``. This unveiled a bunch of issues Nuitka had, but went unnoticed so far, and triggered e.g. the frame line number improvements. - Separate syntax tests. - The scripts to run the tests now are all in pure Python. This means, no more MinGW shell is needed to execute the tests. Summary ------- The Debian package, Windows installer, etc. are now automatically updated and uploaded. From here on, there can be such packages for the hot fix releases too. The exception tracebacks are now correct by design, and better covered. The generator performance work showed that the approach taken by Nuitka is in fact fast. It was fast on ARM already, but it's nice to see that it's now also fast on x64. Programs using generators will be affected a lot by this. Overall, this release brings Nuitka closer to usability. Better binary names, man pages, improved documentation, issue tracker, etc. all there now. I am in fact now looking for a sponsor for the Debian package to upload it into Debian directly. .. admonition:: Update The upload to Debian happened for 0.3.18 and was done by Yaroslav Halchenko. What's missing is more "hg" completeness. The frame release issue helped it, but ``inspect.getargs()`` doesn't work yet, and is a topic for a future release. Won't be easy, as ``func_defaults`` will be an invasive change too. Nuitka Release 0.3.14 ===================== This is to inform you about the new stable release of Nuitka. This time it contains mostly organisational improvements, some bug fixes, improved compatibility and cleanups. It is again the result of working towards compilation of a real program (Mercurial). This time, I have added support for proper handling of compiled types by the ``inspect`` module. Bug fixes --------- - Fix for "Missing checks in parameter parsing with star list, star dict and positional arguments". There was whole in the checks for argument counts, now the correct error is given. Fixed in 0.3.13a already. - The simple slice operations with 2 values, not extended with 3 values, were not applying the correct order for evaluation. Fixed in 0.3.13a already. - The simple slice operations couldn't handle ``None`` as the value for lower or upper index. Fixed in 0.3.11a already. - The in-place simple slice operations evaluated the slice index expressions twice, which could cause problems if they had side effects. Fixed in 0.3.11a already. New Features ------------ - Run time patching the ``inspect`` module so it accepts compiled functions, compiled methods, and compiled generator objects. The ``test_inspect`` test of CPython is nearly working unchanged with this. - The generator functions didn't have ``CO_GENERATOR`` set in their code object, setting it made compatible with CPython in this regard too. The inspect module will therefore return correct value for ``inspect.isgeneratorfunction()`` too. New Optimization ---------------- - Slice indexes that are ``None`` are now constant propagated as well. - Slightly more efficient code generation for dual star arg functions, removing useless checks. Cleanups -------- - Moved the Scons, static C++ files, and assembler files to new package ``nuitka.build`` where also now ``SconsInterface`` module lives. - Moved the Qt dialog files to ``nuitka.gui`` - Moved the "unfreezer" code to its own static C++ file. - Some PyLint cleanups. New Tests --------- - New test ``Recursion`` to cover recursive functions. - New test ``Inspection`` to cover the patching of ``inspect`` module. - Cover ``execfile`` on the class level as well in ``ExecEval`` test. - Cover evaluation order of simple slices in ``OrderCheck`` too. Organizational -------------- - There is a new issue tracker available under http://bugs.nuitka.net Please register and report issues you encounter with Nuitka. I have put all the known issues there and started to use it recently. It's Roundup based like http://bugs.python.org is, so people will find it familiar. - The ``setup.py`` is now apparently functional. The source releases for download are made it with, and it appears the binary distributions work too. We may now build a windows installer. It's currently in testing, we will make it available when finished. Summary ------- The new source organisation makes packaging Nuitka really easy now. From here, we can likely provide "binary" package of Nuitka soon. A windows installer will be nice. The patching of ``inspect`` works wonders for compatibility for those programs that insist on checking types, instead of doing duck typing. The function call problem, was an issue found by the Mercurial test suite. For the "hg.exe" to pass all of its test suite, more work may be needed, this is the overall goal I am currently striving for. Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization. Nuitka Release 0.3.13 ===================== This release is mostly the result of working towards compilation of a real programs (Mercurial) and to merge and finalize the frame stack work. Now Nuitka has a correct frame stack at all times, and supports ``func_code`` and ``gi_code`` objects, something previously thought to be impossible. Actually now it's only the "bytecode" objects that won't be there. And not attributes of ``func_code`` are meaningful yet, but in theory can be supported. Due to the use of the "git flow" for Nuitka, most of the bugs listed here were already fixed in on the stable release before this release. This time there were 5 such hot fix releases, sometimes fixing multiple bugs. Bug fixes --------- - In case of syntax errors in the main program, an exception stack was giving that included Nuitka code. Changed to make the same output as CPython does. Fixed in 0.3.12a already. - The star import (``from x import *``) didn't work for submodules. Providing ``*`` as the import list to the respective code allowed to drop the complex lookups we were doing before, and to simply trust CPython C/API to do it correctly. Fixed in 0.3.12 already. - The absolute import is *not* the default of CPython 2.7 it seems. A local ``posix`` package shadows the standard library one. Fixed in 0.3.12 already. - In ``--deep`` mode, a module may contain a syntax error. This is e.g. true of "PyQt" with ``port_v3`` included. These files contain Python3 syntax and fail to be imported in Python2, but that is not to be considered an error. These modules are now skipped with a warning. Fixed in 0.3.12b already. - The code to import modules wasn't using the ``__import__`` built-in, which prevented ``__import__`` overriding code to work. Changed import to use the built-in. Fixed in 0.3.12c already. - The code generated for the ``__import__`` built-in with constant values was doing relative imports only. It needs to attempt relative and absolut imports. Fixed in 0.3.12c already. - The code of packages in "__init__.py" believed it was outside of the package, giving problems for package local imports. Fixed in 0.3.12d already. - It appears that "Scons", which Nuitka uses internally and transparent to you, to execute the compilation and linking tasks, was sometimes not building the binaries or shared libraries, due to a false caching. As a workaround, these are now erased before doing the build. Fixed in 0.3.12d already. - The use of ``in`` and ``not in`` in comparison chains (e.g. ``a < b < c`` is one), wasn't supported yet. The use of these in comparison chains ``a in b in c`` is very strange. Only in the ``test_grammar.py`` it was ever used I believe. Anyway, it's supported now, solving this ``TODO`` and reducing the difference. Fixed in 0.3.12e already. - The order of evaluation for ``in`` and ``not in`` operators wasn't enforced in a portable way. Now it is correct on "ARM" too. Fixed in 0.3.12e already. New Optimization ---------------- - The built-ins ``GeneratorExit`` and ``StopIteration`` are optimized to their Python C/API names where possible as well. Cleanups -------- - The ``__file__`` attribute of modules was the relative filename, but for absolute filenames these become a horrible mess at least on Linux. - Added assertion helpers for sane frame and code objects and use them. - Make use of ``assertObject`` in more places. - Instead of using ``os.path.sep`` all over, added a helper ``Utils.joinpath`` that hides this and using ``os.path.join``. This gives more readable code. - Added traces to the "unfreezer" guarded by a define. Helpful in analyzing import problems. - Some PyLint cleanups removing dead code, unused variables, useless pass statement, etc. New Tests --------- - New tests to cover ``SyntaxError`` and ``IndentationError`` from ``--deep`` imports and in main program. - New test to cover evaluation order of ``in`` and ``not in`` comparisons. - New test to cover package local imports made by the "__init__.py" of the package. Organizational -------------- - Drop "compile_itself.sh" in favor of the new "compile_itself.py", because the later is more portable. - The logging output is now nicer, and for failed recursions, outputs the line that is having the problem. Summary ------- The frame stack work and the ``func_code`` are big for compatibility. The ``func_code`` was also needed for "hg" to work. For Mercurial to pass all of its test suite, more work will be needed, esp. the ``inspect`` module needs to be run-time patched to accept compiled functions and generators too. Once real world programs like Mercurial work, we can use these as more meaningful benchmarks and resume work on optimization. Nuitka Release 0.3.12 ===================== This is to inform you about the new release of Nuitka many bug fixes, and substantial improvements especially in the organizational area. There is a new "`User Manual `__" (`PDF `__), with much improved content, a ``sys.meta_path`` based import mechanism for ``--deep`` mode, git flow goodness. This release is generally also the result of working towards compilation of a real programs (Mercurial) and to get things work more nicely on Windows by default. Thanks go to Liu Zhenhai for helping me with this goal. Due to the use of the "git flow", most of the bugs listed here were already fixed in on the stable release before this release. And there were many of these. Bug fixes --------- - The order of evaluation for base classes and class dictionaries was not enforced. Apparently nothing in the CPython test suite did that, I only noticed during debugging that Nuitka gave a different error than CPython did, for a class that had an undefined base class, because both class body and base classes were giving an error. Fixed in 0.3.11a already. - Method objects didn't hold a reference to the used class. The effect was only noticed when ``--python-debug`` was used, i.e. the debug version of Python linked, because then the garbage collector makes searches. Fixed in 0.3.11b already. - Set ``sys.executable`` on Linux as well. On Debian it is otherwise ``/usr/bin/python`` which might be a different version of Python entirely. Fixed in 0.3.11c already. - Embedded modules inside a package could hide package variables of the same name. Learned during PyCON DE about this corner case. Fixed in 0.3.11d already. - Packages could be duplicated internally. This had no effect on generated code other than appearing twice in the list if frozen modules. Fixed in 0.3.11d already. - When embedding modules from outside current directory, the look-up failed. The embedding only ever worked for the compile itself and programs test cases, because they are all in the current directory then. Fixed in 0.3.11e already. - The check for ARM target broke Windows support in the Scons file. Fixed in 0.3.11f already. - The star import from external modules failed with an error in ``--deep`` mode. Fixed in 0.3.11g already. - Modules with a parent package could cause a problem under some circumstances. Fixed in 0.3.11h already. - One call variant, with both list and dict star arguments and keyword arguments, but no positional parameters, didn't have the required C++ helper function implemented. Fixed in 0.3.11h already. - The detection of the CPU core count was broken on my hexacore at least. Gave 36 instead of 6, which is a problem for large programs. Fixed in 0.3.11h already. - The in-line copy of Scons didn't really work on Windows, which was sad, because we added it to simplify installation on Windows precisely because of this. - Cleaning up the build directory from old sources and object files wasn't portable to Windows and therefore wasn't effective there. - From imports where part of the imported were found modules and parts were not, didn't work. Solved by the feature branch ``meta_path_import`` that was merged for this release. - Newer MinGW gave warnings about the default visibility not being possible to apply to class members. Fixed by not setting this default visibility anymore on Windows. - The ``sys.executable`` gave warnings on Windows because of backslashes in the path. Using a raw string to prevent such problems. - The standard library path was hard coded. Changed to run time detection. Cleanups -------- - Version checks on Python runtime now use a new define ``PYTHON_VERSION`` that makes it easier. I don't like ``PY_VERSION_HEX``, because it is so unreadable. Makes some of the checks a lot more safe. - The ``sys.meta_path`` based import from the ``meta_path_import`` feature branch allowed the cleanup the way importing is done. It's a lot less code now. - Removed some unused code. We will aim at making Nuitka the tool to detect dead code really. - Moved ``nuitka.Nodes`` to ``nuitka.nodes.Nodes``, that is what the package is intended for, the split will come later. New Tests --------- - New tests for import variants that previously didn't work: Mixed imports. Imports from a package one level up. Modules hidden by a package variable, etc. - Added test of function call variant that had no test previously. Only found it when compiling "hg". Amazing how nothing in my tests, CPython tests, etc. used it. - Added test to cover the partial success of import statements. - Added test to cover evaluation order of class definitions. Organizational -------------- - Migrated the "README.txt" from org-mode to ReStructured Text, which allows for a more readable document, and to generate a nice "`User Manual `__" in PDF form. - The amount of information in "README.txt" was increased, with many more subjects are now covered, e.g. "git flow" and how to join Nuitka development. It's also impressive to see what code blocks and syntax highlighting can do for readability. - The Nuitka git repository has seen multiple hot fixes. These allowed to publish bug fixes immediately after they were made, and avoided the need for a new release just to get these out. This really saves me a lot of time too, because I can postpone releasing the new version until it makes sense because of other things. - Then there was a feature branch ``meta_path_import`` that lived until being merged to ``develop`` to improve the import code, which is now released on ``master`` as stable. Getting that feature right took a while. - And there is the feature branch ``minimize_CPython26_tests_diff`` which has some success already in documenting the required changes to the "CPython26" test suite and in reducing the amount of differences, while doing it. We have a frame stack working there, albeit in too ugly code form. - The release archives are now built using ``setuptools``. You can now also download a zip file, which is probably more Windows friendly. The intention is to work on that to make ``setup.py`` produce a Nuitka install that won't rely on any environment variables at all. Right now ``setup.py`` won't even allow any other options than ``sdist`` to be given. - Ported "compile_itself.sh" to "compile_itself.py", i.e. ported it to Python. This way, we can execute it easily on Windows too, where it currently still fails. Replacing ``diff``, ``rm -rf``, etc. is a challenge, but it reduces the dependency on MSYS tools on Windows. - The compilation of standard library is disabled by default, but ``site`` or ``dist`` packages are now embedded. To include even standard library, there is a ``--really-deep`` option that has to be given in addition to ``--deep``, which forces this. Summary ------- Again, huge progress. The improved import mechanism is very beautiful. It appears that little is missing to compile real world programs like "hg" with Nuitka. The next release cycle will focus on that and continue to improve the Windows support which appears to have some issues. Nuitka Release 0.3.11 ===================== This is to inform you about the new release of Nuitka with some bug fixes and portability work. This release is generally cleaning up things, and makes Nuitka portable to ARM Linux. I used to host the Nuitka homepage on that machine, but now that it's no longer so, I can run heavy compile jobs on it. To my surprise, it found many portability problems. So I chose to fix that first, the result being that Nuitka now works on ARM Linux too. Bug fixes --------- - The order of slice expressions was not correct on x86 as well, and I found that with new tests only. So the porting to ARM revealed a bug category, I previously didn't consider. - The use of ``linux2`` in the Scons file is potentially incompatible with Linux 3.0, although it seems that at least on Debian the ``sys.platform`` was changed back to ``linux2``. Anyway, it's probably best to allow just anything that starts with ``linux`` these days. - The ``print`` statement worked like a ``print`` function, i.e. it first evaluated all printed expressions, and did the output only then. That is incompatible in case of exceptions, where partial outputs need to be done, and so that got fixed. New Optimization ---------------- - Function calls now each have a dedicated helper function, avoiding in some cases unnecessary work. We will may build further on this and in-line ``PyObject_Call`` differently for the special cases. Cleanups -------- - Moved many C++ helper declarations and in-line implementations to dedicated header files for better organisation. - Some dependencies were removed and consolidated to make the dependency graph sane. - Multiple decorators were in reverse order in the node tree. The code generation reversed it back, so no bug, yet that was a distorted tree. Finding this came from the ARM work, because the "reversal" was in fact just the argument evaluation order of C++ under x86/x64, but on ARM that broke. Correcting it highlighted this issue. - The deletion of slices, was not using ``Py_ssize`` for indexes, disallowing some kinds of optimization, so that was harmonized. - The function call code generation got a general overhaul. It is now more consistent, has more helpers available, and creates more readable code. - PyLint is again happier than ever. New Tests --------- - There is a new basic test ``OrderChecks`` that covers the order of expression evaluation. These problems were otherwise very hard to detect, and in some cases not previously covered at all. - Executing Nuitka with Python3 (it won't produce correct Python3 C/API code) is now part of the release tests, so non-portable code of Nuitka gets caught. Organizational -------------- - Support for ARM Linux. I will make a separate posting on the challenges of this. Suffice to say now, that C++ leaves way too much things unspecified. - The Nuitka git repository now uses "git flow". The new git policy will be detailed in another `separate posting `__. - There is an unstable ``develop`` branch in which the development occurs. For this release ca. 40 commits were done to this branch, before merging it. I am also doing more fine grained commits now. - Unlike previously, there is ``master`` branch for the stable release. - There is a script "make-dependency-graph.sh" (Update: meanwhile it was renamed to "make-dependency-graph.py") to produce a dependency graphs of Nuitka. I detected a couple of strange things through this. - The Python3 ``__pycache__`` directories get removed too by the cleanup script. Numbers ------- We only have "PyStone" now, and on a new machine, so the numbers cannot be compared to previous releases: python 2.6:: Pystone(1.1) time for 50000 passes = 0.48 This machine benchmarks at 104167 pystones/second Nuitka 0.3.11 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.19 This machine benchmarks at 263158 pystones/second So this a speedup factor of 258%, last time on another machine it was 240%. Yet it only proves that the generated and compiled are more efficient than bytecode, but Nuitka doesn't yet do the relevant optimization. Only once it does, the factor will be significantly higher. Summary ------- Overall, there is quite some progress. Nuitka is a lot cleaner now, which will help us later only. I wanted to get this out, mostly because of the bug fixes, and of course just in case somebody attempts to use it on ARM. Nuitka Release 0.3.10 ===================== This new release is major milestone 2 work, enhancing practically all areas of Nuitka. The focus was roundup and breaking new grounds with structural optimization enhancements. Bug fixes --------- - Exceptions now correctly stack. When you catch an exception, there always was the exception set, but calling a new function, and it catching the exception, the values of ``sys.exc_info()`` didn't get reset after the function returned. This was a small difference (of which there are nearly none left now) but one that might effect existing code, which affects code that calls functions in exception handling to check something about it. So it's good this is resolved now too. Also because it is difficult to understand, and now it's just like CPython behaves, which means that we don't have to document anything at all about it. - Using ``exec`` in generator functions got fixed up. I realized that this wouldn't work while working on other things. It's obscure yes, but it ought to work. - Lambda generator functions can now be nested and in generator functions. There were some problems here with the allocation of closure variables that got resolved. - List contractions could not be returned by lambda functions. Also a closure issue. - When using a mapping for globals to ``exec`` or ``eval`` that had a side effect on lookup, it was evident that the lookup was made twice. Correcting this also improves the performance for the normal case. New Optimization ---------------- - Statically raised as well as predicted exceptions are propagated upwards, leading to code and block removal where possible, while maintaining the side effects. This is brand new and doesn't do everything possible yet. Most notable, the matching of raised exception to handlers is not yet performed. - Built-in exception name references and creation of instances of them are now optimized as well, which leads to faster exception raising/catching for these cases. - More kinds of calls to built-ins are handled, positional parameters are checked and more built-ins are covered. Notable is that now checks are performed if you didn't potentially overload e.g. the ``len`` with your own version in the module. Locally it was always detected already. So it's now also safe. - All operations and comparisons are now simulated if possible and replaced with their result. - In the case of predictable true or false conditions, not taken branches are removed. - Empty branches are now removed from most constructs, leading to sometimes cleaner code generated. Cleanups -------- - Removed the lambda body node and replaced it with function body. This is a great win for the split into body and builder. Regular functions and lambda functions now only differ in how the created body is used. - Large cleanup of the operation/comparison code. There is now only use of a simulator function, which exists for every operator and comparison. This one is then used in a prediction call, shared with the built-in predictions. - Added a ``Tracing`` module to avoid future imports of ``print_function``, which annoyed me many times by causing syntax failures for when I quickly added a print statement, not noting it must have the braces. - PyLint is happier than ever. New Tests --------- - Enhanced ``OverflowFunctions`` test to cover even deeper nesting of overflow functions taking closure from each level. While it's not yet working, this makes clearer what will be needed. Even if this code is obscure, I would like to be that correct here. - Made ``Operators`` test to cover the `` operator as well. - Added to ``ListContractions`` the case where a contraction is returned by a lambda function, but still needs to leak its loop variable. - Enhanced ``GeneratorExpressions`` test to cover lambda generators, which is really crazy code: .. code-block:: python def y(): yield((yield 1),(yield 2)) - Added to ``ExecEval`` a case where the ``exec`` is inside a generator, to cover that too. - Activated the testing of ``sys.exc_info()`` in ``ExceptionRaising`` test. This was previously commented out, and now I added stuff to illustrate all of the behavior of CPython there. - Enhanced ``ComparisonChains`` test to demonstrate that the order of evaluations is done right and that side effects are maintained. - Added ``BuiltinOverload`` test to show that overloaded built-ins are actually called and not the optimized version. So code like this has to print 2 lines: .. code-block:: python from __builtin__ import len as _len def len(x): print x return _len(x) print len(range(9)) Organizational -------------- - Changed "README.txt" to no longer say that "Scons" is a requirement. Now that it's included (patched up to work with ``ctypes`` on Windows), we don't have to say that anymore. - Documented the status of optimization and added some more ideas. - There is now an option to dump the node tree after optimization as XML. Not currently use, but is for regression testing, to identify where new optimization and changes have an impact. This make it more feasible to be sure that Nuitka is only becoming better. - Executable with Python3 again, although it won't do anything, the necessary code changes were done. Summary ------- It's nice to see, that I some long standing issues were resolved, and that structural optimization has become almost a reality. The difficult parts of exception propagation are all in place, now it's only details. With that we can eliminate and predict even more of the stupid code of "pybench" at compile time, achieving more infinite speedups. Nuitka Release 0.3.9 ==================== This is about the new release of Nuitka which some bug fixes and offers a good speed improvement. This new release is major milestone 2 work, enhancing practically all areas of Nuitka. The main focus was on faster function calls, faster class attributes (not instance), faster unpacking, and more built-ins detected and more thoroughly optimizing them. Bug fixes --------- - Exceptions raised inside with statements had references to the exception and traceback leaked. - On Windows the binaries ``sys.executable`` pointed to the binary itself instead of the Python interpreter. Changed, because some code uses ``sys.executable`` to know how to start Python scripts. - There is a bug (fixed in their repository) related to C++ raw strings and C++ "trigraphs" that affects Nuitka, added a workaround that makes Nuitka not emit "trigraphs" at all. - The check for mutable constants was erroneous for tuples, which could lead to assuming a tuple with only mutable elements to be not mutable, which is of course wrong. New Optimization ---------------- This time there are so many new optimization, it makes sense to group them by the subject. Exceptions ~~~~~~~~~~ - The code to add a traceback is now our own, which made it possible to use frames that do not contain line numbers and a code object capable of lookups. - Raising exceptions or adding to tracebacks has been made way faster by reusing a cached frame objects for the task. - The class used for saving exceptions temporarily (e.g. used in ``try``/``finally`` code, or with statement) has been improved so it doesn't make a copy of the exception with a C++ ``new`` call, but it simply stores the exception properties itself and creates the exception object only on demand, which is more efficient. - When catching exceptions, the addition of tracebacks is now done without exporting and re-importing the exception to Python, but directly on the exception objects traceback, this avoids a useless round trip. Function Calls ~~~~~~~~~~~~~~ - Uses of PyObject_Call provide ``NULL`` as the dictionary, instead of an empty dictionary, which is slightly faster for function calls. - There are now dedicated variants for complex function calls with ``*`` and ``**`` arguments in all forms. These can take advantage of easier cases. For example, a merge with star arguments is only needed if there actually were any of these. - The check for non-string values in the ``**`` arguments can now be completely short-cut for the case of a dictionary that has never had a string added. There is now code that detects this case and skips the check, eliminating it as a performance concern. Parameter Parsing ~~~~~~~~~~~~~~~~~ - Reversed the order in which parameters are checked. Now the keyword dictionary is iterated first and only then the positional arguments after that is done. This iteration is not only much faster (avoiding repeated lookups for each possible parameter), it also can be more correct, in case the keyword argument is derived from a dictionary and its keys mutate it when being compared. - Comparing parameter names is now done with a fast path, in which the pointer values are compare first. This can avoid a call to the comparison at all, which has become very likely due to the interning of parameter name strings, see below. - Added a dedicated call to check for parameter equality with rich equality comparison, which doesn't raise an exception. - Unpacking of tuples is now using dedicated variants of the normal unpacking code instead of rolling out everything themselves. Attribute Access ~~~~~~~~~~~~~~~~ - The class type (in executables, not yet for extension modules) is changed to a faster variant of our own making that doesn't consider the restricted mode a possibility. This avoids very expensive calls, and makes accessing class attributes in compiled code and in non-compiled code faster. - Access to attributes (but not of instances) got in-lined and therefore much faster. Due to other optimization, a specific step to intern the string used for attribute access is not necessary with Nuitka at all anymore. This made access to attributes about 50% faster which is big of course. Constants ~~~~~~~~~ - The bug for mutable tuples also caused non-mutable tuples to be considered as mutable, which lead to less efficient code. - The constant creation with the g++ bug worked around, can now use raw strings to create string constants, without resorting to un-pickling them as a work around. This allows us to use ``PyString_FromStringAndSize`` to create strings again, which is obviously faster, and had not been done, because of the confusion caused by the g++ bug. - For string constants that are usable as attributes (i.e. match the identifier regular expression), these are now interned, directly after creation. With this, the check for identical value of pointers for parameters has a bigger chance to succeed, and this saves some memory too. - For empty containers (set, dict, list, tuple) the constants created are now are not unstreamed, but created with the dedicated API calls, saving a bit of code and being less ugly. - For mutable empty constant access (set, dict, list) the values are no longer made by copying the constant, but instead with the API functions to create new ones. This makes code like ``a = []`` a tiny bit faster. - For slice indices the code generation now takes advantage of creating a C++ ``Py_ssize_t`` from constant value if possible. Before it was converting the integer constant at run time, which was of course wasteful even if not (very) slow. Iteration ~~~~~~~~~ - The creation of iterators got our own code. This avoids a function call and is otherwise only a small gain for anything but sequence iterators. These may be much faster to create now, as it avoids another call and repeated checks. - The next on iterator got our own code too, which has simpler code flow, because it avoids the double check in case of NULL returned. - The unpack check got simlar code to the next iterator, it also has simpler code flow now and avoids double checks. Built-ins ~~~~~~~~~ - Added support for the ``list``, ``tuple``, ``dict``, ``str``, ``float`` and ``bool`` built-ins along with optimizing their use with constant parameter. - Added support for the ``int`` and ``long`` built-ins, based on a new "call spec" object, that detects parameter errors at compile time and raises appropriate exceptions as required, plus it deals with keyword arguments just as well. So, to Nuitka it doesn't matter now it you write ``int(value) ``or ``int(x = value)`` anymore. The ``base`` parameter of these built-ins is also supported. The use of this call spec mechanism will the expanded, currently it is not applied to the built-ins that take only one parameter. This is a work in progress as is the whole built-ins business as not all the built-ins are covered yet. Cleanups ~~~~~~~~ - In 0.3.8 per module global classes were introduced, but the ``IMPORT_MODULE`` kept using the old universal class, this got resolved and the old class is now fully gone. - Using ``assertObject`` in more cases, and in more places at all, catches errors earlier on. - Moved the addition to tracebacks into the ``_PythonException`` class, where it works directly on the contained traceback. This is cleaner as it no longer requires to export exceptions to Python, just to add a traceback entry. - Some ``PyLint`` cleanups were done, reducing the number of reports a bit, but there is still a lot to do. - Added a ``DefaultValueIdentifier`` class that encapsulates the access to default values in the parameter parsing more cleanly. - The module ``CodeTemplatesListContractions`` was renamed to ``CodeTemplatesContractions`` to reflect the fact that it deals with all kinds of contractions (also set and dict contractions), not just list contractions. - Moved the with related template to its own module ``CodeTemplatesWith``, so its easier to find. - The options handling for g++ based compilers was cleaned up, so that g++ 4.6 and MinGW are better supported now. - Documented more aspects of the Scons build file. - Some more generated code white space fixes. - Moved some helpers to dedicated files. There is now ``calling.hpp`` for function calls, an ``importing.cpp`` for import related stuff. - Moved the manifest generation to the scons file, which now produces ready to use executables. New Tests --------- - Added a improved version of "pybench" that can cope with the "0 ms" execution time that Nuitka has for some if its sub-tests. - Reference counting test for with statement was added. - Micro benchmarks to demonstrate try finally performance when an exception travels through it. - Micro benchmark for with statement that eats up exceptions raised inside the block. - Micro benchmarks for the read and write access to class attributes. - Enhanced ``Printing`` test to cover the trigraphs constant bug case. Output is required to make the error detectable. - Enhanced ``Constants`` test to cover repeated mutation of mutable tuple constants, this covers the bug mentioned. Organizational -------------- - Added a credits section to the "README.txt" where I give credit to the people who contributed to Nuitka, and the projects it is using. I will make it a separate posting to cite these. - Documented the requirements on the compiler more clearly, document the fact that we require scons and which version of Python (2.6 or 2.7). - The is now a codespeed implementation up and running with historical data for up to Nuitka 0.3.8 runs of "PyStone" and with pybench. It will be updated for 0.3.9 once I have the infrastructure in place to do that automatically. - The cleanup script now also removes .so files. - The handling of options for g++ got improved, so it's the same for g++ and MinGW compilers, plus adequate errors messages are given, if the compiler version is too low. - There is now a ``--unstriped`` option that just keeps the debug information in the file, but doesn't keep the assertions. This will be helpful when looking at generated assembler code from Nuitka to not have the distortions that ``--debug`` causes (reduced optimization level, assertions, etc.) and instead a clear view. Nuitka Release 0.3.8 ==================== This is to inform you about the new release of Nuitka with some real news and a slight performance increase. The significant news is added "Windows Support". You can now hope to run Nuitka on Windows too and have it produce working executables against either the standard Python distribution or a MinGW compiled Python. There are still some small things to iron out, and clearly documentation needs to be created, and esp. the DLL hell problem of ``msvcr90.dll`` vs. ``msvcrt.dll``, is not yet fully resolved, but appears to be not as harmful, at least not on native Windows. I am thanking Khalid Abu Bakr for making this possible. I was surprised to see this happen. I clearly didn't make it easy. He found a good way around ``ucontext``, identifier clashes, and a very tricky symbol problems where the CPython library under Windows exports less than under Linux. Thanks a whole lot. Currently the Windows support is considered experimental and works with MinGW 4.5 or higher only. Otherwise there have been the usual round of performance improvements and more cleanups. This release is otherwise milestone 2 work only, which will have to continue for some time more. Bug fixes --------- - Lambda generators were not fully compatible, their simple form could yield an extra value. The behavior for Python 2.6 and 2.7 is also different and Nuitka now mimics both correctly, depending on the used Python version - The given parameter count cited in the error message in case of too many parameters, didn't include the given keyword parameters in the error message. - There was an ``assert False`` right after warning about not found modules in the ``--deep`` mode, which was of course unnecessary. New Optimization ---------------- - When unpacking variables in assignments, the temporary variables are now held in a new temporary class that is designed for the task specifically. This avoids the taking of a reference just because the ``PyObjectTemporary`` destructor insisted on releasing one. The new class ``PyObjectTempHolder`` hands the existing reference over and releases only in case of exceptions. - When unpacking variable in for loops, the value from the iterator may be directly assigned, if it's to a variable. In general this would be possible for every assignment target that cannot raise, but the infrastructure cannot tell yet, which these would be. This will improve with more milestone 3 work. - Branches with only ``pass`` inside are removed, ``pass`` statements are removed before the code generation stage. This makes it easier to achieve and decide empty branches. - There is now a global variable class per module. It appears that it is indeed faster to roll out a class per module accessing the ``module *`` rather than having one class and use a ``module **``, which is quite disappointing from the C++ compiler. - Also ``MAKE_LIST`` and ``MAKE_TUPLE`` have gained special cases for the 0 arguments case. Even when the size of the variadic template parameters should be known to the compiler, it seems, it wasn't eliminating the branch, so this was a speedup measured with valgrind. - Empty tried branches are now replaced when possible with ``try``/``except`` statements, ``try``/``finally`` is simplified in this case. This gives a cleaner tree structure and less verbose C++ code which the compiler threw away, but was strange to have in the first place. - In conditions the ``or`` and ``and`` were evaluated with Python objects instead of with C++ bool, which was unnecessary overhead. - List contractions got more clever in how they assign from the iterator value. It now uses a ``PyObjectTemporary`` if it's assigned to multiple values, a ``PyObjectTempHolder`` if it's only assigned once, to something that could raise, or a ``PyObject *`` if an exception cannot be raised. This avoids temporary references completely for the common case. Cleanups -------- - The ``if``, ``for``, and ``while`` statements had always empty ``else`` nodes which were then also in the generated C++ code as empty branches. No harm to performance, but this got cleaned up. - Some more generated code white space fixes. New Tests --------- - The CPython 2.7 test suite now also has the ``doctests`` extracted to static tests, which improves test coverage for Nuitka again. This was previously only done for CPython 2.6 test suite, but the test suites are different enough to make this useful, e.g. to discover newly changed behavior like with the lambda generators. - Added Shed Skin 0.7.1 examples as benchmarks, so we can start to compare Nuitka performance in these tests. These will be the focus of numbers for the 0.4.x release series. - Added a micro benchmark to check unpacking behavior. Some of these are needed to prove that a change is an actual improvement, when its effect can go under in noise of in-line vs. no in-line behavior of the C++ compiler. - Added "pybench" benchmark which reveals that Nuitka is for some things much faster, but there are still fields to work on. This version needed changes to stand the speed of Nuitka. These will be subject of a later posting. Organizational -------------- - There is now a "tests/benchmarks/micro" directory to contain tiny benchmarks that just look at a single aspect, but have no other meaning, e.g. the "PyStone" extracts fall into this category. - There is now a ``--windows-target`` option that attempts a cross-platform build on Linux to Windows executable. This is using "MingGW-cross-env" cross compilation tool chain. It's not yet working fully correctly due to the DLL hell problem with the C runtime. I hope to get this right in subsequent releases. - The ``--execute`` option uses wine to execute the binary if it's a cross-compile for windows. - Native windows build is recognized and handled with MinGW 4.5, the VC++ is not supported yet due to missing C++0x support. - The basic test suite ran with Windows so far only and some adaptations were necessary. Windows new lines are now ignored in difference check, and addresses under Windows are upper case, small things. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.8 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.27 This machine benchmarks at 185185 pystones/second This is a 140% speed increase of 0.3.8 compared to CPython, up from 132% compared to the previous release. Nuitka Release 0.3.7 ==================== This is about the new release with focus on performance and cleanups. It indicates significant progress with the milestone this release series really is about as it adds a ``compiled_method`` type. So far functions, generator function, generator expressions were compiled objects, but in the context of classes, functions were wrapped in CPython ``instancemethod`` objects. The new ``compiled_method`` is specifically designed for wrapping ``compiled_function`` and therefore more efficient at it. Bug fixes --------- - When using ``Python`` or ``Nuitka.py`` to execute some script, the exit code in case of "file not found" was not the same as CPython. It should be 2, not 1. - The exit code of the created programs (``--deep`` mode) in case of an uncaught exception was 0, now it an error exit with value 1, like CPython does it. - Exception tracebacks created inside ``with`` statements could contain duplicate lines, this was corrected. New Optimization ---------------- - Global variable assignments now also use ``assign0`` where no reference exists. The assignment code for module variables is actually faster if it needs not drop the reference, but clearly the code shouldn't bother to take it on the outside just for that. This variant existed, but wasn't used as much so far. - The instance method objects are now Nuitka's own compiled type too. This should make things slightly faster by itself. - Our new compiled method objects support dedicated method parsing code, where ``self`` is passed directly, allowing to make calls taking a fast path in parameter parsing. This avoids allocating/freeing a ``tuple`` object per method call, while reduced 3% ticks in "PyStone" benchmark, so that's significant. - Solved a ``TODO`` of ``BUILTIN_RANGE`` to change it to pre-allocating the list in the final size as we normally do everywhere else. This was a tick reduction of 0.4% in "PyStone" benchmark, but the measurement method normalizes on loop speed, so it's not visible in the numbers output. - Parameter variables cannot possibly be uninitialized at creation and most often they are never subject to a ``del`` statement. Adding dedicated C++ variable classes gave a big speedup, around 3% of "PyStone" benchmark ticks. - Some abstract object operations were re-implemented, which allows to avoid function calls e.g. in the ``ITERATOR_NEXT`` case, this gave a few percent on "PyStone" as well. Cleanups -------- - New package ``nuitka.codegen`` to contain all code generation related stuff, moved ``nuitka.templates`` to ``nuitka.codegen.templates`` as part of that. - Inside the ``nuitka.codegen`` package the ``MainControl`` module now longer reaches into ``Generator`` for simple things, but goes through ``CodeGeneration`` for everything now. - The ``Generator`` module uses almost no tree nodes anymore, but instead gets information passed in function calls. This allows for a cleanup of the interface towards ``CodeGeneration``. Gives a cleaner view on the C++ code generation, and generally furthers the goal of other than C++ language backends. - More "PyLint" work, many of the reported warnings have been addressed, but it's not yet happy. - Defaults for ``yield`` and ``return`` are ``None`` and these values are now already added (as constants) during tree building so that no such special cases need to be dealt with in ``CodeGeneration`` and future analysis steps. - Parameter parsing code has been unified even further, now the whole entry point is generated by one of the function in the new ``nuitka.codegen.ParameterParsing`` module. - Split variable, exception, built-in helper classes into separate header files. New Tests --------- - The exit codes of CPython execution and Nuitka compiled programs are now compared as well. - Errors messages of methods are now covered by the ``ParameterErrors`` test as well. Organizational -------------- - A new script "benchmark.sh" (now called "run-valgrind.py") script now starts "kcachegrind" to display the valgrind result directly. One can now use it to execute a test and inspect valgrind information right away, then improve it. Very useful to discover methods for improvements, test them, then refine some more. - The "check-release.sh" script needs to unset ``NUITKA_EXTRA_OPTIONS`` or else the reflection test will trip over the changed output paths. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.7 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.28 This machine benchmarks at 178571 pystones/second This is a 132% speed of 0.3.7 compared to CPython, up from 109% compare to the previous release. This is a another small increase, that can be fully attributed to milestone 2 measures, i.e. not analysis, but purely more efficient C++ code generation and the new compiled method type. One can now safely assume that it is at least twice as fast, but I will try and get the PyPy or Shedskin test suite to run as benchmarks to prove it. No milestone 3 work in this release. I believe it's best to finish with milestone 2 first, because these are quite universal gains that we should have covered. Nuitka Release 0.3.6 ==================== The major point this for this release is cleanup work, and generally bug fixes, esp. in the field of importing. This release cleans up many small open ends of Nuitka, closing quite a bunch of consistency ``TODO`` items, and then aims at cleaner structures internally, so optimization analysis shall become "easy". It is a correctness and framework release, not a performance improvement at all. Bug fixes --------- - Imports were not respecting the ``level`` yet. Code like this was not working, now it is: .. code-block:: python from .. import something - Absolute and relative imports were e.g. both tried all the time, now if you specify absolute or relative imports, it will be attempted in the same way than CPython does. This can make a difference with compatibility. - Functions with a "locals dict" (using ``locals`` built-in or ``exec`` statement) were not 100% compatible in the way the locals dictionary was updated, this got fixed. It seems that directly updating a dict is not what CPython does at all, instead it only pushes things to the dictionary, when it believes it has to. Nuitka now does the same thing, making it faster and more compatible at the same time with these kind of corner cases. - Nested packages didn't work, they do now. Nuitka itself is now successfully using nested packages (e.g. ``nuitka.transform.optimizations``) New Features ------------ - The ``--lto`` option becomes usable. It's not measurably faster immediately, and it requires g++ 4.6 to be available, but then it at least creates smaller binaries and may provide more optimization in the future. New Optimization ---------------- - Exceptions raised by pre-computed built-ins, unpacking, etc. are now transformed to raising the exception statically. Cleanups -------- - There is now a ``getVariableForClosure`` that a variable provider can use. Before that it guessed from ``getVariableForReference`` or ``getVariableForAssignment`` what might be the intention. This makes some corner cases easier. - Classes, functions and lambdas now also have separate builder and body nodes, which enabled to make getSameScopeNodes() really simple. Either something has children which are all in a new scope or it has them in the same scope. - Twisted workarounds like ``TransitiveProvider`` are no longer needed, because class builder and class body were separated. - New packages ``nuitka.transform.optimizations`` and ``nuitka.transform.finalizations``, where the first was ``nuitka.optimizations`` before. There is also code in ``nuitka.transform`` that was previously in a dedicated module. This allowed to move a lot of displaced code. - ``TreeBuilding`` now has fast paths for all 3 forms, things that need a "provider", "node", and "source_ref"; things that need "node" and "source_ref"; things that need nothing at all, e.g. pass. - Variables now avoid building duplicated instances, but instead share one. Better for analysis of them. New Tests --------- - The Python 2.7 test suite is no longer run with Python 2.6 as it will just crash with the same exception all the time, there is no ``importlib`` in 2.6, but every test is using that through test_support. - Nested packages are now covered with tests too. - Imports of upper level packages are covered now too. Organizational -------------- - Updated the "README.txt" with the current plan on optimization. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.6 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.31 This machine benchmarks at 161290 pystones/second This is 109% for 0.3.6, but no change from the previous release. No surprise, because no new effective new optimization means have been implemented. Stay tuned for future release for actual progress. Nuitka Release 0.3.5 ==================== This new release of Nuitka is an overall improvement on many fronts, there is no real focus this time, likely due to the long time it was in the making. The major points are more optimization work, largely enhanced import handling and another improvement on the performance side. But there are also many bug fixes, more test coverage, usability and compatibility. Something esp. noteworthy to me and valued is that many important changes were performed or at least triggered by Nicolas Dumazet, who contributed a lot of high quality commits as you can see from the gitweb history. He appears to try and compile Mercurial and Nuitka, and this resulted in important contributions. Bug fixes --------- - Nicolas found a reference counting bug with nested parameter calls. Where a function had parameters of the form ``a, (b,c)`` it could crash. This got fixed and covered with a reference count test. - Another reference count problem when accessing the locals dictionary was corrected. - Values ``0.0`` and ``-0.0`` were treated as the same. They are not though, they have a different sign that should not get lost. - Nested contractions didn't work correctly, when the contraction was to iterate over another contraction which needs a closure. The problem was addressing by splitting the building of a contraction from the body of the contraction, so that these are now 2 nodes, making it easy for the closure handling to get things right. - Global statements in function with local ``exec()`` would still use the value from the locals dictionary. Nuitka is now compatible to CPython with this too. - Nicolas fixed problems with modules of the same name inside different packages. We now use the full name including parent package names for code generation and look-ups. - The ``__module__`` attribute of classes was only set after the class was created. Now it is already available in the class body. - The ``__doc__`` attribute of classes was not set at all. Now it is. - The relative import inside nested packages now works correctly. With Nicolas moving all of Nuitka to a package, the compile itself exposed many weaknesses. - A local re-raise of an exception didn't have the original line attached but the re-raise statement line. New Features ------------ - Modules and packages have been unified. Packages can now also have code in "__init__.py" and then it will be executed when the package is imported. - Nicolas added the ability to create deep output directory structures without having to create them beforehand. This makes ``--output-dir=some/deep/path`` usable. - Parallel build by Scons was added as an option and enabled by default, which enhances scalability for ``--deep`` compilations a lot. - Nicolas enhanced the CPU count detection used for the parallel build. Turned out that ``multithreading.cpu_count()`` doesn't give us the number of available cores, so he contributed code to determine that. - Support for upcoming g++ 4.6 has been added. The use of the new option ``--lto`` has been been prepared, but right now it appears that the C++ compiler will need more fixes, before we can this feature with Nuitka. - The ``--display-tree`` feature got an overhaul and now displays the node tree along with the source code. It puts the cursor on the line of the node you selected. Unfortunately I cannot get it to work two-way yet. I will ask for help with this in a separate posting as we can really use a "python-qt" expert it seems. - Added meaningful error messages in the "file not found" case. Previously I just didn't care, but we sort of approach end user usability with this. New Optimization ---------------- - Added optimization for the built-in ``range()`` which otherwise requires a module and ``builtin`` module lookup, then parameter parsing. Now this is much faster with Nuitka and small ranges (less than 256 values) are converted to constants directly, avoiding run time overhead entirely. - Code for re-raise statements now use a simple re-throw of the exception where possible, and only do the hard work where the re-throw is not inside an exception handler. - Constant folding of operations and comparisons is now performed if the operands are constants. - Values of some built-ins are pre-computed if the operands are constants. - The value of module attribute ``__name__`` is replaced by a constant unless it is assigned to. This is the first sign of upcoming constant propagation, even if only a weak one. - Conditional statement and/or their branches are eliminated where constant conditions allow it. Cleanups -------- - Nicolas moved the Nuitka source code to its own ``nuitka`` package. That is going to make packaging it a lot easier and allows cleaner code. - Nicolas introduced a fast path in the tree building which often delegates (or should do that) to a function. This reduced a lot of the dispatching code and highlights more clearly where such is missing right now. - Together we worked on the line length issues of Nuitka. We agreed on a style and very long lines will vanish from Nuitka with time. Thanks for pushing me there. - Nicolas also did provide many style fixes and general improvements, e.g. using ``PyObjectTemporary`` in more places in the C++ code, or not using ``str.find`` where ``x in y`` is a better choice. - The node structure got cleaned up towards the direction that assigments always have an assignment as a child. A function definition, or a class definition, are effectively assignments, and in order to not have to treat this as special cases everywhere, they need to have assignment targets as child nodes. Without such changes, optimization will have to take too many things into account. This is not yet completed. - Nicolas merged some node tree building functions that previously handled deletion and assigning differently, giving us better code reuse. - The constants code generation was moved to a ``__constants.cpp`` where it doesn't make __main__.cpp so much harder to read anymore. - The module declarations have been moved to their own header files. - Nicolas cleaned up the scripts used to test Nuitka big time, removing repetitive code and improving the logic. Very much appreciated. - Nicolas also documented a things in the Nuitka source code or got me to document things that looked strange, but have reasons behind it. - Nicolas solved the ``TODO`` related to built-in module accesses. These will now be way faster than before. - Nicolas also solved the ``TODO`` related to the performance of "locals dict" variable accesses. - Generator.py no longer contains classes. The Contexts objects are supposed to contain the state, and as such the generator objects never made much sense. - Also with the help of Scons community, I figured out how to avoid having object files inside the ``src`` directory of Nuitka. That should also help packaging, now all build products go to the .build directory as they should. - The vertical white space of the generated C++ got a few cleanups, trailing/leading new line is more consistent now, and there were some assertions added that it doesn't happen. New Tests --------- - The CPython 2.6 tests are now also run by CPython 2.7 and the other way around and need to report the same test failure reports, which found a couple of issues. - Now the test suite is run with and without ``--debug`` mode. - Basic tests got extended to cover more topics and catch more issues. - Program tests got extended to cover code in packages. - Added more exec scope tests. Currently inlining of exec statements is disabled though, because it requires entirely different rules to be done right, it has been pushed back to the next release. Organizational -------------- - The ``g++-nuitka`` script is no more. With the help of the Scons community, this is now performed inside the scons and only once instead of each time for every C++ file. - When using ``--debug``, the generated C++ is compiled with ``-Wall`` and ``-Werror`` so that some form of bugs in the generated C++ code will be detected immediately. This found a few issues already. - There is a new git merge policy in place. Basically it says, that if you submit me a pull request, that I will deal with it before publishing anything new, so you can rely on the current git to provide you a good base to work on. I am doing more frequent pre-releases already and I would like to merge from your git. - The "README.txt" was updated to reflect current optimization status and plans. There is still a lot to do before constant propagation can work, but this explains things a bit better now. I hope to expand this more and more with time. - There is now a "misc/clean-up.sh" script that prints the commands to erase all the temporary files sticking around in the source tree. That is for you if you like me, have other directories inside, ignored, that you don't want to delete. - Then there is now a script that prints all source filenames, so you can more easily open them all in your editor. - And very important, there is now a "check-release.sh" script that performs all the tests I think should be done before making a release. - Pylint got more happy with the current Nuitka source. In some places, I added comments where rules should be granted exceptions. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.5 (driven by python 2.6):: Pystone(1.1) time for 50000 passes = 0.31 This machine benchmarks at 161290 pystones/second This is 109% for 0.3.5, up from 91% before. Overall this release is primarily an improvement in the domain of compatibility and contains important bug and feature fixes to the users. The optimization framework only makes a first showing of with the framework to organize them. There is still work to do to migrate optimization previously present It will take more time before we will see effect from these. I believe that even more cleanups of ``TreeBuilding``, ``Nodes`` and ``CodeGeneration`` will be required, before everything is in place for the big jump in performance numbers. But still, passing 100% feels good. Time to rejoice. Nuitka Release 0.3.4 ==================== This new release of Nuitka has a focus on re-organizing the Nuitka generated source code and a modest improvement on the performance side. For a long time now, Nuitka has generated a single C++ file and asked the C++ compiler to translate it to an executable or shared library for CPython to load. This was done even when embedding many modules into one (the "deep" compilation mode, option ``--deep``). This was simple to do and in theory ought to allow the compiler to do the most optimization. But for large programs, the resulting source code could have exponential compile time behavior in the C++ compiler. At least for the GNU g++ this was the case, others probably as well. This is of course at the end a scalability issue of Nuitka, which now has been addressed. So the major advancement of this release is to make the ``--deep`` option useful. But also there have been a performance improvements, which end up giving us another boost for the "PyStone" benchmark. Bug fixes --------- - Imports of modules local to packages now work correctly, closing the small compatibility gap that was there. - Modules with a "-" in their name are allowed in CPython through dynamic imports. This lead to wrong C++ code created. (Thanks to Li Xuan Ji for reporting and submitting a patch to fix it.) - There were warnings about wrong format used for ``Ssize_t`` type of CPython. (Again, thanks to Li Xuan Ji for reporting and submitting the patch to fix it.) - When a wrong exception type is raised, the traceback should still be the one of the original one. - Set and dict contractions (Python 2.7 features) declared local variables for global variables used. This went unnoticed, because list contractions don't generate code for local variables at all, as they cannot have such. - Using the ``type()`` built-in to create a new class could attribute it to the wrong module, this is now corrected. New Features ------------ - Uses Scons to execute the actual C++ build, giving some immediate improvements. - Now caches build results and Scons will only rebuild as needed. - The direct use of ``__import__()`` with a constant module name as parameter is also followed in "deep" mode. With time, non-constants may still become predictable, right now it must be a real CPython constant string. New Optimization ---------------- - Added optimization for the built-ins ``ord()`` and ``chr()``, these require a module and built-in module lookup, then parameter parsing. Now these are really quick with Nuitka. - Added optimization for the ``type()`` built-in with one parameter. As above, using from builtin module can be very slow. Now it is instantaneous. - Added optimization for the ``type()`` built-in with three parameters. It's rarely used, but providing our own variant, allowed to fix the bug mentioned above. Cleanups -------- - Using scons is a big cleanup for the way how C++ compiler related options are applied. It also makes it easier to re-build without Nuitka, e.g. if you were using Nuitka in your packages, you can easily build in the same way than Nuitka does. - Static helpers source code has been moved to ".hpp" and ".cpp" files, instead of being in ".py" files. This makes C++ compiler messages more readable and allows us to use C++ mode in Emacs etc., making it easier to write things. - Generated code for each module ends up in a separate file per module or package. - Constants etc. go to their own file (although not named sensible yet, likely going to change too) - Module variables are now created by the ``CPythonModule`` node only and are unique, this is to make optimization of these feasible. This is a pre-step to module variable optimization. New Tests --------- - Added "ExtremeClosure" from my Python quiz, it was not covered by existing tests. - Added test case for program that imports a module with a dash in its name. - Added test case for main program that starts with a dash. - Extended the built-in tests to cover ``type()`` as well. Organizational -------------- - There is now a new environment variable ``NUITKA_SCONS`` which should point to the directory with the ``SingleExe.scons`` file for Nuitka. The scons file could be named better, because it is actually one and the same who builds extension modules and executables. - There is now a new environment variable ``NUITKA_CPP`` which should point to the directory with the C++ helper code of Nuitka. - The script "create-environment.sh" can now be sourced (if you are in the top level directory of Nuitka) or be used with eval. In either case it also reports what it does. .. admonition:: Update The script has become obsolete now, as the environment variables are no longer necessary. - To cleanup the many "Program.build" directories, there is now a "clean-up.sh" script for your use. Can be handy, but if you use git, you may prefer its clean command. .. admonition:: Update The script has become obsolete now, as Nuitka test executions now by default delete the build results. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.4:: Pystone(1.1) time for 50000 passes = 0.34 This machine benchmarks at 147059 pystones/second This is 91% for 0.3.4, up from 80% before. Nuitka Release 0.3.3 ==================== This release of Nuitka continues the focus on performance. It also cleans up a few open topics. One is "doctests", these are now extracted from the CPython 2.6 test suite more completely. The other is that the CPython 2.7 test suite is now passed completely. There is some more work ahead though, to extract all of the "doctests" and to do that for both versions of the tests. This means an even higher level of compatibility has been achieved, then there is performance improvements, and ever cleaner structure. Bug fixes --------- Generators ~~~~~~~~~~ - Generator functions tracked references to the common and the instance context independently, now the common context is not released before the instance contexts are. - Generator functions didn't check the arguments to ``throw()`` the way they are in CPython, now they are. - Generator functions didn't trace exceptions to "stderr" if they occurred while closing unfinished ones in "del". - Generator functions used the slightly different wordings for some error messages. Function Calls ~~~~~~~~~~~~~~ - Extended call syntax with ``**`` allows that to use a mapping, and it is now checked if it really is a mapping and if the contents has string keys. - Similarly, extended call syntax with ``*`` allows a sequence, it is now checked if it really is a sequence. - Error message for duplicate keyword arguments or too little arguments now describe the duplicate parameter and the callable the same way CPython does. - Now checks to the keyword argument list first before considering the parameter counts. This is slower in the error case, but more compatible with CPython. Classes ~~~~~~~ - The "locals()" built-in when used in the class scope (not in a method) now is correctly writable and writes to it change the resulting class. - Name mangling for private identifiers was not always done entirely correct. Others ~~~~~~ - Exceptions didn't always have the correct stack reported. - The pickling of some tuples showed that "cPickle" can have non-reproducible results, using "pickle" to stream constants now New Optimization ---------------- - Access to instance attributes has become faster by writing specific code for the case. This is done in JIT way, attempting at run time to optimize attribute access for instances. - Assignments now often consider what's cheaper for the other side, instead of taking a reference to a global variable, just to have to release it. - The function call code built argument tuples and dictionaries as constants, now that is true for every tuple usage. Cleanups -------- - The static helper classes, and the prelude code needed have been moved to separate C++ files and are now accessed "#include". This makes the code inside C++ files as opposed to a Python string and therefore easier to read and or change. New Features ------------ - The generator functions and generator expressions have the attribute "gi_running" now. These indicate if they are currently running. New Tests --------- - The script to extract the "doctests" from the CPython test suite has been rewritten entirely and works with more doctests now. Running these tests created increased the test coverage a lot. - The Python 2.7 test suite has been added. Organizational -------------- - One can now run multiple "compare_with_cpython" instances in parallel, which enables background test runs. - There is now a new environment variable "NUITKA_INCLUDE" which needs to point to the directory Nuitka's C++ includes live in. Of course the "create-environment.sh" script generates that for you easily. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.3:: Pystone(1.1) time for 50000 passes = 0.36 This machine benchmarks at 138889 pystones/second This is 80% for 0.3.3, up from 66% before. Nuitka Release 0.3.2 ==================== This release of Nuitka continues the focus on performance. But this release also revisits the topic of feature parity. Before, feature parity had been reached "only" with Python 2.6. This is of course a big thing, but you know there is always more, e.g. Python 2.7. With the addition of set contractions and dict contractions in this very release, Nuitka is approaching Python support for 2.7, and then there are some bug fixes. Bug fixes --------- - Calling a function with ``**`` and using a non-dict for it was leading to wrong behavior. Now a mapping is good enough as input for the ``**`` parameter and it's checked. - Deeply nested packages "package.subpackage.module" were not found and gave a warning from Nuitka, with the consequence that they were not embedded in the executable. They now are. - Some error messages for wrong parameters didn't match literally. For example "function got multiple..." as opposed to "function() got multiple..." and alike. - Files that ended in line with a "#" but without a new line gave an error from "ast.parse". As a workaround, a new line is added to the end of the file if it's "missing". - More correct exception locations for complex code lines. I noted that the current line indication should not only be restored when the call at hand failed, but in any case. Otherwise sometimes the exception stack would not be correct. It now is - more often. Right now, this has no systematic test. - Re-raised exceptions didn't appear on the stack if caught inside the same function, these are now correct. - For ``exec`` the globals argument needs to have "__builtins__" added, but the check was performed with the mapping interface. That is not how CPython does it, and so e.g. the mapping could use a default value for "__builtins__" which could lead to incorrect behavior. Clearly a corner case, but one that works fully compatible now. New Optimization ---------------- - The local and shared local variable C++ classes have a flag "free_value" to indicate if an "PY_DECREF" needs to be done when releasing the object. But still the code used "Py_XDECREF" (which allows for "NULL" values to be ignored.) when the releasing of the object was done. Now the inconsistency of using "NULL" as "object" value with "free_value" set to true was removed. - Tuple constants were copied before using them without a point. They are immutable anyway. Cleanups -------- - Improved more of the indentation of the generated C++ which was not very good for contractions so far. Now it is. Also assignments should be better now. - The generation of code for contractions was made more general and templates split into multiple parts. This enabled reuse of the code for list contractions in dictionary and set contractions. - The with statement has its own template now and got cleaned up regarding indentation. New Tests --------- - There is now a script to extract the "doctests" from the CPython test suite and it generates Python source code from them. This can be compiled with Nuitka and output compared to CPython. Without this, the doctest parts of the CPython test suite is mostly useless. Solving this improved test coverage, leading to many small fixes. I will dedicate a later posting to the tool, maybe it is useful in other contexts as well. - Reference count tests have been expanded to cover assignment to multiple assignment targets, and to attributes. - The deep program test case, now also have a module in a sub-package to cover this case as well. Organizational -------------- - The `gitweb interface `__ might be considered an alternative to downloading the source if you want to provide a pointer, or want to take a quick glance at the source code. You can already download with git, follow the link below to the page explaining it. - The "README.txt" has documented more of the differences and I consequently updated the Differences page. There is now a distinction between generally missing functionality and things that don't work in ``--deep`` mode, where Nuitka is supposed to create one executable. I will make it a priority to remove the (minor) issues of ``--deep`` mode in the next release, as this is only relatively little work, and not a good difference to have. We want these to be empty, right? But for the time being, I document the known differences there. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.2:: Pystone(1.1) time for 50000 passes = 0.39 This machine benchmarks at 128205 pystones/second This is 66% for 0.3.2, slightly up from the 58% of 0.3.1 before. The optimization done were somewhat fruitful, but as you can see, they were also more cleanups, not the big things. Nuitka Release 0.3.1 ==================== This release of Nuitka continues the focus on performance and contains only cleanups and optimization. Most go into the direction of more readable code, some aim at making the basic things faster, with good results as to performance as you can see below. New Optimization ---------------- - Constants in conditions of conditional expressions (``a if cond else d``), ``if``/``elif`` or ``while`` are now evaluated to ``true`` or ``false`` directly. Before there would be temporary python object created from it which was then checked if it had a truth value. All of that is obviously overhead only. And it hurts the typically ``while 1:`` infinite loop case badly. - Do not generate code to catch ``BreakException`` or ``ContinueException`` unless a ``break`` or ``continue`` statement being in a ``try: finally:`` block inside that loop actually require this. Even while uncaught exceptions are cheap, it is still an improvement worthwhile and it clearly improves the readability for the normal case. - The compiler more aggressively prepares tuples, lists and dicts from the source code as constants if their contents is "immutable" instead of building at run time. An example of a "mutable" tuple would be ``({},)`` which is not safe to share, and therefore will still be built at run time. For dictionaries and lists, copies will be made, under the assumption that copying a dictionary will always be faster, than making it from scratch. - The parameter parsing code was dynamically building the tuple of argument names to check if an argument name was allowed by checking the equivalent of ``name in argument_names``. This was of course wasteful and now a pre-built constant is used for this, so it should be much faster to call functions with keyword arguments. - There are new templates files and also actual templates now for the ``while`` and ``for`` loop code generation. And I started work on having a template for assignments. Cleanups -------- - Do not generate code for the else of ``while`` and ``for`` loops if there is no such branch. This uncluttered the generated code somewhat. - The indentation of the generated C++ was not very good and whitespace was often trailing, or e.g. a real tab was used instead of "\t". Some things didn't play well together here. Now much of the generated C++ code is much more readable and white space cleaner. For optimization to be done, the humans need to be able to read the generated code too. Mind you, the aim is not to produce usable C++, but on the other hand, it must be possible to understand it. - To the same end of readability, the empty ``else {}`` branches are avoided for ``if``, ``while`` and ``for`` loops. While the C++ compiler can be expected to remove these, they seriously cluttered up things. - The constant management code in ``Context`` was largely simplified. Now the code is using the ``Constant`` class to find its way around the problem that dicts, sets, etc. are not hashable, or that ``complex`` is not being ordered; this was necessary to allow deeply nested constants, but it is also a simpler code now. - The C++ code generated for functions now has two entry points, one for Python calls (arguments as a list and dictionary for parsing) and one where this has happened successfully. In the future this should allow for faster function calls avoiding the building of argument tuples and dictionaries all-together. - For every function there was a "traceback adder" which was only used in the C++ exception handling before exit to CPython to add to the traceback object. This was now in-lined, as it won't be shared ever. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.1:: Pystone(1.1) time for 50000 passes = 0.41 This machine benchmarks at 121951 pystones/second This is 58% for 0.3.1, up from the 25% before. So it's getting somewhere. As always you will find its latest version here. Nuitka Release 0.3.0 ==================== This release 0.3.0 is the first release to focus on performance. In the 0.2.x series Nuitka achieved feature parity with CPython 2.6 and that was very important, but now it is time to make it really useful. Optimization has been one of the main points, although I was also a bit forward looking to Python 2.7 language constructs. This release is the first where I really started to measure things and removed the most important bottlenecks. New Features ------------ - Added option to control ``--debug``. With this option the C++ debug information is present in the file, otherwise it is not. This will give much smaller ".so" and ".exe" files than before. - Added option ``--no-optimization`` to disable all optimization. It enables C++ asserts and compiles with less aggressive C++ compiler optimization, so it can be used for debugging purposes. - Support for Python 2.7 set literals has been added. Performance Enhancements ------------------------ - Fast global variables: Reads of global variables were fast already. This was due to a trick that is now also used to check them and to do a much quicker update if they are already set. - Fast ``break``/``continue`` statements: To make sure these statements execute the finally handlers if inside a try, these used C++ exceptions that were caught by ``try``/``finally`` in ``while`` or ``for`` loops. This was very slow and had very bad performance. Now it is checked if this is at all necessary and then it's only done for the rare case where a ``break``/``continue`` really is inside the tried block. Otherwise it is now translated to a C++ ``break``/``continue`` which the C++ compiler handles more efficiently. - Added ``unlikely()`` compiler hints to all errors handling cases to allow the C++ compiler to generate more efficient branch code. - The for loop code was using an exception handler to make sure the iterated value was released, using ``PyObjectTemporary`` for that instead now, which should lead to better generated code. - Using constant dictionaries and copy from them instead of building them at run time even when contents was constant. New Tests --------- - Merged some bits from the CPython 2.7 test suite that do not harm 2.6, but generally it's a lot due to some ``unittest`` module interface changes. - Added CPython 2.7 tests ``test_dictcomps.py`` and ``test_dictviews.py`` which both pass when using Python 2.7. - Added another benchmark extract from "PyStone" which uses a while loop with break. Numbers ------- python 2.6:: Pystone(1.1) time for 50000 passes = 0.65 This machine benchmarks at 76923.1 pystones/second Nuitka 0.3.0:: Pystone(1.1) time for 50000 passes = 0.52 This machine benchmarks at 96153.8 pystones/second That's a 25% speedup now and a good start clearly. It's not yet in the range of where i want it to be, but there is always room for more. And the ``break``/``continue`` exception was an important performance regression fix. Nuitka Release 0.2.4 ==================== This release 0.2.4 is likely the last 0.2.x release, as it's the one that achieved feature parity with CPython 2.6, which was the whole point of the release series, so time to celebrate. I have stayed away (mostly) from any optimization, so as to not be premature. From now on speed optimization is going to be the focus though. Because right now, frankly, there is not much of a point to use Nuitka yet, with only a minor run time speed gain in trade for a long compile time. But hopefully we can change that quickly now. New Features ------------ - The use of exec in a local function now adds local variables to scope it is in. - The same applies to ``from module_name import *`` which is now compiled correctly and adds variables to the local variables. Bug Fixes --------- - Raises ``UnboundLocalError`` when deleting a local variable with ``del`` twice. - Raises ``NameError`` when deleting a global variable with ``del`` twice. - Read of to uninitialized closure variables gave ``NameError``, but ``UnboundLocalError`` is correct and raised now. Cleanups -------- - There is now a dedicated pass over the node tree right before code generation starts, so that some analysis can be done as late as that. Currently this is used for determining which functions should have a dictionary of locals. - Checking the exported symbols list, fixed all the cases where a ``static`` was missing. This reduces the "module.so" sizes. - With gcc the "visibility=hidden" is used to avoid exporting the helper classes. Also reduces the "module.so" sizes, because classes cannot be made static otherwise. New Tests --------- - Added "DoubleDeletions" to cover behaviour of ``del``. It seems that this is not part of the CPython test suite. - The "OverflowFunctions" (those with dynamic local variables) now has an interesting test, exec on a local scope, effectively adding a local variable while a closure variable is still accessible, and a module variable too. This is also not in the CPython test suite. - Restored the parts of the CPython test suite that did local star imports or exec to provide new variables. Previously these have been removed. - Also "test_with.py" which covers PEP 343 has been reactivated, the with statement works as expected. Nuitka Release 0.2.3 ==================== This new release is marking a closing in on feature parity to CPython 2.6 which is an important mile stone. Once this is reached, a "Nuitka 0.3.x" series will strive for performance. Bug Fixes --------- - Generator functions no longer leak references when started, but not finished. - Yield can in fact be used as an expression and returns values that the generator user ``send()`` to it. Reduced Differences / New Features ---------------------------------- - Generator functions already worked quite fine, but now they have the ``throw()``, ``send()`` and ``close()`` methods. - Yield is now an expression as is ought to be, it returns values put in by ``send()`` on the generator user. - Support for extended slices: .. code-block:: python x = d[:42, ..., :24:, 24, 100] d[:42, ..., :24:, 24, 100] = "Strange" del d[:42, ..., :24:, 24, 100] Tests Work ---------- - The "test_contextlib" is now working perfectly due to the generator functions having a correct ``throw()``. Added that test back, so context managers are now fully covered. - Added a basic test for "overflow functions" has been added, these are the ones which have an unknown number of locals due to the use of language constructs ``exec`` or ``from bla import *`` on the function level. This one currently only highlights the failure to support it. - Reverted removals of extended slice syntax from some parts of the CPython test suite. Cleanups -------- - The compiled generator types are using the new C++0x type safe enums feature. - Resolved a circular dependency between ``TreeBuilding`` and ``TreeTransforming`` modules. Nuitka Release 0.2.2 ==================== This is some significant progress, a lot of important things were addressed. Bug Fixes --------- - Scope analysis is now done during the tree building instead of sometimes during code generation, this fixed a few issues that didn't show up in tests previously. - Reference leaks of generator expressions that were not fishing, but then deleted are not more. - Inlining of exec is more correct now. - More accurate exception lines when iterator creation executes compiled code, e.g. in a for loop - The list of base classes of a class was evaluated in the context of the class, now it is done in the context of the containing scope. - The first iterated of a generator expression was evaluated in its own context, now it is done in the context of the containing scope. Reduced Differences ------------------- - With the enhanced scope analysis, ``UnboundLocalError`` is now correctly supported. - Generator expressions (but not yet functions) have a ``throw()``, ``send()`` and ``close()`` method. - Exec can now write to local function namespace even if ``None`` is provided at run time. - Relative imports inside packages are now correctly resolved at compile time when using ``--deep``. Cleanups -------- - The compiled function type got further enhanced and cleaned up. - The compiled generator expression function type lead to a massive cleanup of the code for generator expressions. - Cleaned up namespaces, was still using old names, or "Py*" which is reserved to core CPython. - Overhaul of the code responsible for ``eval`` and ``exec``, it has been split, and it pushed the detection defaults to the C++ compiler which means, we can do it at run time or compile time, depending on circumstances. - Made ``PyTemporaryObject`` safer to use, disabling copy constructor it should be also a relief to the C++ compiler if it doesn't have to eliminate all its uses. - The way delayed work is handled in ``TreeBuilding`` step has been changed to use closured functions, should be more readable. - Some more code templates have been created, making the code generation more readable in some parts. More to come. New Features ------------ - As I start to consider announcing Nuitka, I moved the version logic so that the version can now be queried with ``--version``. New Optimization ---------------- - Name lookups for ``None``, ``True`` and ``False`` and now always detected as constants, eliminating many useless module variable lookups. New Tests --------- - More complete test of generator expressions. - Added test program for packages with relative imports inside the package. - The built-in ``dir()`` in a function was not having fully deterministic output list, now it does. Summary ------- Overall, the amount of differences between CPython and Nuitka is heading towards zero. Also most of the improvements done in this release were very straightforward cleanups and not much work was required, mostly things are about cleanups and then it becomes easily right. The new type for the compiled generator expressions was simple to create, esp. as I could check what CPython does in its source code. For optimization purposes, I decided that generator expressions and generator functions will be separate compiled types, as most of their behavior will not be shared. I believe optimizing generator expressions to run well is an important enough goal to warrant that they have their own implementation. Now that this is done, I will repeat it with generator functions. Generator functions already work quite fine, but like generator expressions did before this release, they can leak references if not finished , and they don't have the ``throw()`` method, which seems very important to the correct operation of ``contextlib``. So I will introduce a decicated type for these too, possibly in the next release. Nuitka Release 0.2.1 ==================== The march goes on, this is another minor release with a bunch of substantial improvements: Bug Fixes --------- - Packages now also can be embedded with the ``--deep`` option too, before they could not be imported from the executable. - In-lined exec with their own future statements leaked these to the surrounding code. Reduced Differences ------------------- - The future print function import is now supported too. Cleanups -------- - Independence of the compiled function type. When I started it was merely ``PyCFunction`` and then a copy of it patched at run time, using increasingly less code from CPython. Now it's nothing at all anymore. - This lead to major cleanup of run time compiled function creation code, no more ``methoddefs``, ``PyCObject`` holding context, etc. - PyLint was used to find the more important style issues and potential bugs, also helping to identify some dead code. Summary ------- The major difference now is the lack of a throw method for generator functions. I will try to address that in a 0.2.2 release if possible. The plan is that the 0.2.x series will complete these tasks, and 0.3 could aim at some basic optimization finally. Nuitka Release 0.2 ================== Good day, this is a major step ahead, improvements everywhere. Bug fixes --------- - Migrated the Python parser from the deprecated and problematic ``compiler`` module to the ``ast`` module which fixes the ``d[a,] = b`` parser problem. A pity it was not available at the time I started, but the migration was relatively painless now. - I found and fixed wrong encoding of binary data into C++ literals. Now Nuitka uses C++0x raw strings, and these problems are gone. - The decoding of constants was done with the ``marshal`` module, but that appears to not deeply care enough about unicode encoding it seems. Using ``cPickle`` now, which seems less efficient, but is more correct. - Another difference is gone: The ``continue`` and ``break`` inside loops do no longer prevent the execution of finally blocks inside the loop. Organizational -------------- - I now maintain the "README.txt" in org-mode, and intend to use it as the issue tracker, but I am still a beginner at that. .. admonition:: Update Turned out I never master it, and used ReStructured Text instead. - There is a public git repository for you to track Nuitka releases. Make your changes and then ``git pull --rebase``. If you encounter conflicts in things you consider useful, please submit the patches and a pull offer. When you make your clones of Nuitka public, use ``nuitka-unofficial`` or not the name ``Nuitka`` at all. - There is a now a `mailing list `__ available too. Reduced Differences ------------------- - Did you know you could write ``lambda : (yield something)`` and it gives you a lambda that creates a generator that produces that one value? Well, now Nuitka has support for lambda generator functions. - The ``from __future__ import division`` statement works as expected now, leading to some newly passing CPython tests. - Same for ``from __future__ import unicode_literals`` statement, these work as expected now, removing many differences in the CPython tests that use this already. New Features ------------ - The ``Python`` binary provided and ``Nuitka.py`` are now capable of accepting parameters for the program executed, in order to make it even more of a drop-in replacement to ``python``. - Inlining of ``exec`` statements with constant expressions. These are now compiled at compile time, not at run time anymore. I observed that an increasing number of CPython tests use exec to do things in isolation or to avoid warnings, and many more these tests will now be more effective. I intend to do the same with eval expressions too, probably in a minor release. Summary ------- So give it a whirl. I consider it to be substantially better than before, and the list of differences to CPython is getting small enough, plus there is already a fair bit of polish to it. Just watch out that it needs gcc-4.5 or higher now. Nuitka Release 0.1.1 ==================== I just have just updated Nuitka to version 0.1.1 which is a bug fix release to 0.1, which corrects many of the small things: - Updated the CPython test suite to 2.6.6rc and minimized much of existing differences in the course. - Compiles standalone executable that includes modules (with --deep option), but packages are not yet included successfully. - Reference leaks with exceptions are no more. - sys.exc_info() works now mostly as expected (it's not a stack of exceptions). - More readable generated code, better organisation of C++ template code. - Restored debug option ``--g++-only``. The biggest thing probably is the progress with exception tracebacks objects in exception handlers, which were not there before (always ``None``). Having these in place will make it much more compatible. Also with manually raised exceptions and assertions, tracebacks will now be more correct to the line. On a bad news, I discovered that the ``compiler`` module that I use to create the AST from Python source code, is not only deprecated, but also broken. I created the `CPython bug `__ about it, basically it cannot distinguish some code of the form ``d[1,] = None`` from ``d[1] = None``. This will require a migration of the ``ast`` module, which should not be too challenging, but will take some time. I am aiming at it for a 0.2 release. Generating wrong code (Nuitka sees ``d[1] = None`` in both cases) is a show blocker and needs a solution. So, yeah. It's better, it's there, but still experimental. You will find its latest version here. Please try it out and let me know what you think in the comments section. Nuitka Release 0.1 (Releasing Nuitka to the World) ================================================== Obviously this is very exciting step for me. I am releasing Nuitka today. Finally. For a long time I knew I would, but actually doing it, is a different beast. Reaching my goals for release turned out to be less far away than I hope, so instead of end of August, I can already release it now. Currently it's not more than 4% faster than CPython. No surprise there, if all you did, is removing the bytecode interpretation so far. It's not impressive at all. It's not even a reason to use it. But it's also only a start. Clearly, once I get into optimizing the code generation of Nuitka, it will only get better, and then probably in sometimes dramatic steps. But I see this as a long term goal. I want to have infrastructure in the code place, before doing lots of possible optimization that just make Nuitka unmaintainable. And I will want to have a look at what others did so far in the domain of type inference and how to apply that for my project. I look forward to the reactions about getting this far. The supported language volume is amazing, and I have a set of nice tricks used. For example the way generator functions are done is a clever hack. Where to go from here? Well, I guess, I am going to judge it by the feedback I receive. I personally see "constant propagation" as a laudable first low hanging fruit, that could be solved. Consider this readable code on the module level: .. code-block:: python meters_per_nautical_mile = 1852 def convertMetersToNauticalMiles(meters): return meters / meters_per_nautical_mile def convertNauticalMilesToMeters(miles): return miles * meters_per_nautical_mile Now imagine you are using this very frequently in code. Quickly you determine that the following will be much faster: .. code-block:: python def convertMetersToNauticalMiles(meters): return meters / 1852 def convertNauticalMilesToMeters(miles): return miles * 1852 Still good? Well, probably next step you are going to in-line the function calls entirely. For optimization, you are making your code less readable. I do not all appreciate that. My first goal is there to make the more readable code perform as well or better as the less readable variant. Nuitka-0.5.28.2/lib/0000755000372000001440000000000013207540420014226 5ustar hayenusers00000000000000Nuitka-0.5.28.2/lib/hints.py0000644000372000001440000000616313207540035015735 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys, os original_import = __import__ _indentation = 0 def _normalizePath(path): path = os.path.abspath(path) best = None for path_entry in sys.path: if path.startswith(path_entry): if best is None or len(path_entry) > len(best): best = path_entry if best is not None: path = path.replace(best, "$PYTHONPATH") return path def _moduleRepr(module): try: module_file = module.__file__ module_file = module_file.replace(".pyc", ".py") if module_file.endswith(".so"): module_file = os.path.join( os.path.dirname(module_file), os.path.basename(module_file).split(".")[0] + ".so" ) file_desc = "file " + _normalizePath(module_file).replace(".pyc", ".py") if file_desc.endswith("not_present.py"): raise AttributeError except AttributeError: file_desc = "built-in" return "" % ( module.__name__, file_desc ) def enableImportTracing(normalize_paths = True, show_source = False): def _ourimport(name, globals = None, locals = None, fromlist = None, # @ReservedAssignment level = -1): global _indentation try: _indentation += 1 print(_indentation * " " + "called with: name=%r level=%d" % (name, level)) for entry in traceback.extract_stack()[:-1]: if entry[2] == "_ourimport": print(_indentation * " " + "by __import__") else: entry = list(entry) if not show_source: del entry[-1] del entry[-1] if normalize_paths: entry[0] = _normalizePath(entry[0]) print(_indentation * " " + "by " + "|".join(str(s) for s in entry)) print(_indentation * " " + "*" * 40) result = original_import(name, globals, locals, fromlist, level) print(_indentation * " " + "RESULT:", _moduleRepr(result)) return result finally: _indentation -= 1 try: import __builtin__ as builtins except ImportError: import builtins import traceback builtins.__import__ = _ourimport Nuitka-0.5.28.2/PKG-INFO0000644000372000001440000000272113207540420014557 0ustar hayenusers00000000000000Metadata-Version: 1.1 Name: Nuitka Version: 0.5.28.2 Summary: Python compiler with full language support and CPython compatibility Home-page: http://nuitka.net Author: Kay Hayen Author-email: Kay.Hayen@gmail.com License: Apache License, Version 2.0 Description: UNKNOWN Keywords: compiler,python,nuitka Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Classifier: Topic :: Software Development :: Compilers Classifier: Topic :: Software Development :: Build Tools Classifier: Topic :: Software Development :: Quality Assurance Classifier: Topic :: System :: Software Distribution Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: C Classifier: Operating System :: POSIX :: Linux Classifier: Operating System :: POSIX :: BSD :: FreeBSD Classifier: Operating System :: POSIX :: BSD :: NetBSD Classifier: Operating System :: POSIX :: BSD :: OpenBSD Classifier: Operating System :: Microsoft :: Windows Classifier: License :: OSI Approved :: Apache Software License Nuitka-0.5.28.2/doc/0000755000372000001440000000000013207540420014225 5ustar hayenusers00000000000000Nuitka-0.5.28.2/doc/nuitka.10000644000372000001440000002705613207540417015622 0ustar hayenusers00000000000000.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. .TH NUITKA "1" "November 2017" "nuitka 0.5.28.2" "User Commands" .SH NAME nuitka \- the Python compiler .SH SYNOPSIS .B nuitka [\fI\,--module\/\fR] [\fI\,--execute\/\fR] [\fI\,options\/\fR] \fI\,main_module.py\/\fR .SH OPTIONS .TP \fB\-\-version\fR show program's version number and exit .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP \fB\-\-module\fR Create an extension module executable instead of a program. Defaults to off. .TP \fB\-\-standalone\fR, \fB\-\-portable\fR Enable standalone mode in build. This allows you to transfer the created binary to other machines without it relying on an existing Python installation. It implies these option: "\-\-recurse\-all". You may also want to use "\-\-python\-flag=no_site" to avoid the "site.py" module, which can save a lot of code dependencies. Defaults to off. .TP \fB\-\-nofreeze\-stdlib\fR In standalone mode by default all modules of standard library will be frozen as bytecode. This compiles them all and as a result compilation time will increase very much. .TP \fB\-\-python\-version\fR=\fI\,PYTHON_VERSION\/\fR Major version of Python to be used, one of '2.6', \&'2.7', '3.2', '3.3', '3.4', '3.5', or '3.6'. Defaults to what you run Nuitka with (currently 2.7) .TP \fB\-\-python\-debug\fR, \fB\-\-python\-dbg\fR Use debug version or not. Default uses what you are using to run Nuitka, most likely a non\-debug version. .TP \fB\-\-python\-flag\fR=\fI\,PYTHON_FLAGS\/\fR Python flags to use. Default uses what you are using to run Nuitka, this enforces a specific mode. These are options that also exist to standard Python executable. Currently supported: "\-S" (alias "nosite"), "static_hashes" (not use hash randomization), "no_warnings" (do not give Python runtime warnings), "\-O" (alias "noasserts"). Default empty. .TP \fB\-\-python\-for\-scons\fR=\fI\,PYTHON_SCONS\/\fR, \fB\-\-python2\-for\-scons\fR=\fI\,PYTHON_SCONS\/\fR If using Python3.2 to Python3.4, provide the path of a Python binary to use for Scons. Otherwise Nuitka can use what you run Nuitka with or a "scons" binary that is found in PATH, or a Python installation from Windows registry. .TP \fB\-\-warn\-implicit\-exceptions\fR Enable warnings for implicit exceptions detected at compile time. .TP \fB\-\-warn\-unusual\-code\fR Enable warnings for unusual code detected at compile time. .SS Control the recursion into imported modules: .BR .TP \fB\-\-recurse\-stdlib\fR Also descend into imported modules from standard library. Defaults to off. .TP \fB\-\-recurse\-none\fR When \fB\-\-recurse\-none\fR is used, do not descend into any imported modules at all, overrides all other recursion options. Defaults to off. .TP \fB\-\-recurse\-all\fR, \fB\-\-recurse\-on\fR When \fB\-\-recurse\-all\fR is used, attempt to descend into all imported modules. Defaults to off. .TP \fB\-\-recurse\-to\fR=\fI\,MODULE\/\fR/PACKAGE Recurse to that module, or if a package, to the whole package. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-not\-to\fR=\fI\,MODULE\/\fR/PACKAGE Do not recurse to that module, or if a package, to the whole package in any case, overrides all other options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-plugins\fR=\fI\,MODULE\/\fR/PACKAGE, \fB\-\-recurse\-directory\fR=\fI\,MODULE\/\fR/PACKAGE Recurse into that directory, no matter if it's used by the given main program in a visible form. Overrides all other recursion options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-files\fR=\fI\,PATTERN\/\fR, \fB\-\-recurse\-pattern\fR=\fI\,PATTERN\/\fR Recurse into files matching the PATTERN. Overrides all recursion other options. Can be given multiple times. Default empty. .SS Immediate execution after compilation: .BR .TP \fB\-\-run\fR, \fB\-\-execute\fR Execute immediately the created binary (or import the compiled module). Defaults to off. .TP \fB\-\-debugger\fR, \fB\-\-gdb\fR Execute inside "gdb" to automatically get a stack trace. Defaults to off. .TP \fB\-\-execute\-with\-pythonpath\fR, \fB\-\-keep\-pythonpath\fR When immediately executing the created binary (\fB\-\-execute\fR), don't reset PYTHONPATH. When all modules are successfully included, you ought to not need PYTHONPATH anymore. .SS Dump options for internal tree: .BR .TP \fB\-\-dump\-xml\fR, \fB\-\-xml\fR Dump the final result of optimization as XML, then exit. .TP \fB\-\-display\-tree\fR Display the final result of optimization in a GUI, then exit. .SS Code generation choices: .BR .TP \fB\-\-full\-compat\fR Enforce absolute compatibility with CPython. Do not even allow minor deviations from CPython behavior, e.g. better tracebacks, which are not really incompatible, but different. This is intended for tests only and should not be necessary for normal use. .TP \fB\-\-file\-reference\-choice\fR=\fI\,FILE_REFERENCE_MODE\/\fR Select what value "__file__" is going to be. With "runtime" (default for standalone binary mode and module mode), the created binaries and modules, use the location of themselves to deduct the value of "__file__". Included packages pretend to be in directories below that location. This allows you to include data files in deployments. If you merely seek acceleration, it's better for you to use the "original" value, where the source files location will be used. With "frozen" a notation "" is used. For compatibility reasons, the "__file__" value will always have ".py" suffix independent of what it really is. .SS Output choices: .BR .TP \fB\-\-output\-dir\fR=\fI\,DIRECTORY\/\fR Specify where intermediate and final output files should be put. The DIRECTORY will be populated with C files, object files, etc. Defaults to current directory. .TP \fB\-\-remove\-output\fR Removes the build directory after producing the module or exe file. Defaults to off. .TP \fB\-\-no\-pyi\-file\fR Do not create a ".pyi" file for extension modules created by Nuitka. Defaults to off. .SS Debug features: .BR .TP \fB\-\-debug\fR Executing all self checks possible to find errors in Nuitka, do not use for production. Defaults to off. .TP \fB\-\-unstripped\fR, \fB\-\-no\-strip\fR, \fB\-\-unstriped\fR Keep debug info in the resulting object file for better debugger interaction. Defaults to off. .TP \fB\-\-profile\fR Enable vmprof based profiling of time spent. Defaults to off. .TP \fB\-\-graph\fR Create graph of optimization process. Defaults to off. .TP \fB\-\-trace\-execution\fR Traced execution output, output the line of code before executing it. Defaults to off. .TP \fB\-\-recompile\-c\-only\fR, \fB\-\-recompile\-c\fR++\-only Take existing files and compile them again. Allows compiling edited C files with the C compiler for quick debugging changes to the generated source. Defaults to off. Depends on compiling Python source to determine which files it should look at. .TP \fB\-\-generate\-c\-only\fR Generate only C source code, and do not compile it to binary or module. This is for debugging and code coverage analysis that doesn't waste CPU. Defaults to off. .TP \fB\-\-experimental\fR=\fI\,EXPERIMENTAL\/\fR Use features declared as 'experimental'. May have no effect if no experimental features are present in the code. Uses secret tags (check source) per experimented feature. .SS Backend C compiler choice: .BR .TP \fB\-\-clang\fR Enforce the use of clang (needs clang 3.2 or higher). Defaults to off. .TP \fB\-\-mingw\fR Enforce the use of MinGW on Windows. Defaults to off. .TP \fB\-\-msvc\fR=\fI\,MSVC\/\fR Enforce the use of specific MSVC version on Windows. Allowed values are e.g. 9.0, 9.0exp, specify an illegal value for a list of installed compilers. Defaults to the most recent version. .TP \fB\-j\fR N, \fB\-\-jobs\fR=\fI\,N\/\fR Specify the allowed number of parallel C compiler jobs. Defaults to the system CPU count. .TP \fB\-\-lto\fR Use link time optimizations if available and usable (g++ 4.6 and higher). Defaults to off. .SS Tracing features: .BR .TP \fB\-\-show\-scons\fR Operate Scons in non\-quiet mode, showing the executed commands. Defaults to off. .TP \fB\-\-show\-progress\fR Provide progress information and statistics. Defaults to off. .TP \fB\-\-show\-memory\fR Provide memory information and statistics. Defaults to off. .TP \fB\-\-show\-modules\fR Provide a final summary on included modules. Defaults to off. .TP \fB\-\-verbose\fR Output details of actions taken, esp. in optimizations. Can become a lot. Defaults to off. .SS Windows specific output control: .BR .TP \fB\-\-windows\-disable\-console\fR When compiling for Windows, disable the console window. Defaults to off. .TP \fB\-\-windows\-icon\fR=\fI\,ICON_PATH\/\fR, \fB\-\-icon\fR=\fI\,ICON_PATH\/\fR Add executable icon (Windows only). .SS Plugin control: .BR .TP \fB\-\-plugin\-enable\fR=\fI\,PLUGINS_ENABLED\/\fR, \fB\-\-enable\-plugin\fR=\fI\,PLUGINS_ENABLED\/\fR Enabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-disable\fR=\fI\,PLUGINS_DISABLED\/\fR, \fB\-\-disable\-plugin\fR=\fI\,PLUGINS_DISABLED\/\fR Disabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-no\-detection\fR Plugins can detect if they might be used, and the you can disable the warning via \fB\-\-plugin\-disable\fR=\fI\,pluginthat\-warned\/\fR, or you can use this option to disable the mechanism entirely, which also speeds up compilation slightly of course as this detection code is run in vain once you are certain of which plug\-ins to use. Defaults to off. .TP \fB\-\-plugin\-list\fR Show list of all available plugins and exit. Defaults to off. .SH EXAMPLES Compile a python file "some_module.py" to a module "some_module.so": .IP \f(CW$ nuitka \-\-module some_module.py\fR .PP Compile a python program "some_program.py" to an executable "some_program.exe": .IP \f(CW$ nuitka some_program.py\fR .PP Compile a python program "some_program.py" and the package "some_package" it uses to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-recurse\-to=some_package some_program.py\fR .PP Compile a python program "some_program.py" and all the modules it uses to an executable "some_program.exe". Then execute it immediately when ready: .IP \f(CW$ nuitka \-\-run \-\-recurse\-all some_program.py\fR .PP Compile a python program "some_program.py" and the modules it uses (even standard library) to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-recurse\-all \-\-recurse\-stdlib some_program.py\fR .PP Compile a python program "some_program.py" and the modules it uses to an executable "some_program.exe". Keep the debug information, so valrind, gdb, etc. work nice. Note: This will *not* degrade performance: .IP \f(CW$ nuitka \-\-unstriped \-\-recurse\-all some_program.py\fR .PP Compile a python program "some_program.py" and the modules it uses to an executable "some_program.exe". Perform all kinds of checks about correctness of the generated C and run\-time checks. Note: This will degrade performance and should only be used to debug Nuitka: .IP \f(CW$ nuitka \-\-debug \-\-recurse\-all some_program.py\fR .PP Compile a python program "some_program.py" and the modules it uses to an executable "some_program.exe". Perform all kinds of checks about correctness of the generated C and run\-time checks. Also use the debug Python library, which does its own checks. Note: This will degrade performance and should only be used to debug Nuitka: .IP \f(CW$ nuitka \-\-debug \-\-python-debug \-\-recurse\-all some_program.py\fR .PP Compile a python program "some_program.py" and the plugins modules it loads at run time to an executable "some_program.exe": .IP \f(CW$ nuitka \-\-recurse\-all \-\-recurse\-directory=plugins_dir some_program.py\fR .PP Nuitka-0.5.28.2/doc/Logo/0000755000372000001440000000000013207540420015125 5ustar hayenusers00000000000000Nuitka-0.5.28.2/doc/Logo/Nuitka-Logo-Symbol.svg0000644000372000037200000001635112711355343021225 0ustar hayenhayen00000000000000 image/svg+xml Nuitka-0.5.28.2/doc/Logo/Nuitka-Logo-Vertical.svg0000644000372000037200000004277512711355343021542 0ustar hayenhayen00000000000000 image/svg+xml Nuitka Nuitka-0.5.28.2/doc/Logo/Nuitka-Logo-Horizontal.svg0000644000372000037200000001623112711355343022106 0ustar hayenhayen00000000000000 image/svg+xml Nuitka Nuitka-0.5.28.2/doc/nuitka-run.10000644000372000001440000002271613207540417016422 0ustar hayenusers00000000000000.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4. .TH NUITKA-RUN "1" "November 2017" "nuitka-run 0.5.28.2" "User Commands" .SH NAME nuitka-run \- the Python compiler .SH SYNOPSIS .B nuitka-run [\fI\,options\/\fR] \fI\,main_module.py\/\fR .SH OPTIONS .TP \fB\-\-version\fR show program's version number and exit .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP \fB\-\-module\fR Create an extension module executable instead of a program. Defaults to off. .TP \fB\-\-standalone\fR, \fB\-\-portable\fR Enable standalone mode in build. This allows you to transfer the created binary to other machines without it relying on an existing Python installation. It implies these option: "\-\-recurse\-all". You may also want to use "\-\-python\-flag=no_site" to avoid the "site.py" module, which can save a lot of code dependencies. Defaults to off. .TP \fB\-\-nofreeze\-stdlib\fR In standalone mode by default all modules of standard library will be frozen as bytecode. This compiles them all and as a result compilation time will increase very much. .TP \fB\-\-python\-version\fR=\fI\,PYTHON_VERSION\/\fR Major version of Python to be used, one of '2.6', \&'2.7', '3.2', '3.3', '3.4', '3.5', or '3.6'. Defaults to what you run Nuitka with (currently 2.7) .TP \fB\-\-python\-debug\fR, \fB\-\-python\-dbg\fR Use debug version or not. Default uses what you are using to run Nuitka, most likely a non\-debug version. .TP \fB\-\-python\-flag\fR=\fI\,PYTHON_FLAGS\/\fR Python flags to use. Default uses what you are using to run Nuitka, this enforces a specific mode. These are options that also exist to standard Python executable. Currently supported: "\-S" (alias "nosite"), "static_hashes" (not use hash randomization), "no_warnings" (do not give Python runtime warnings), "\-O" (alias "noasserts"). Default empty. .TP \fB\-\-python\-for\-scons\fR=\fI\,PYTHON_SCONS\/\fR, \fB\-\-python2\-for\-scons\fR=\fI\,PYTHON_SCONS\/\fR If using Python3.2 to Python3.4, provide the path of a Python binary to use for Scons. Otherwise Nuitka can use what you run Nuitka with or a "scons" binary that is found in PATH, or a Python installation from Windows registry. .TP \fB\-\-warn\-implicit\-exceptions\fR Enable warnings for implicit exceptions detected at compile time. .TP \fB\-\-warn\-unusual\-code\fR Enable warnings for unusual code detected at compile time. .SS Control the recursion into imported modules: .BR .TP \fB\-\-recurse\-stdlib\fR Also descend into imported modules from standard library. Defaults to off. .TP \fB\-\-recurse\-none\fR When \fB\-\-recurse\-none\fR is used, do not descend into any imported modules at all, overrides all other recursion options. Defaults to off. .TP \fB\-\-recurse\-all\fR, \fB\-\-recurse\-on\fR When \fB\-\-recurse\-all\fR is used, attempt to descend into all imported modules. Defaults to off. .TP \fB\-\-recurse\-to\fR=\fI\,MODULE\/\fR/PACKAGE Recurse to that module, or if a package, to the whole package. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-not\-to\fR=\fI\,MODULE\/\fR/PACKAGE Do not recurse to that module, or if a package, to the whole package in any case, overrides all other options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-plugins\fR=\fI\,MODULE\/\fR/PACKAGE, \fB\-\-recurse\-directory\fR=\fI\,MODULE\/\fR/PACKAGE Recurse into that directory, no matter if it's used by the given main program in a visible form. Overrides all other recursion options. Can be given multiple times. Default empty. .TP \fB\-\-recurse\-files\fR=\fI\,PATTERN\/\fR, \fB\-\-recurse\-pattern\fR=\fI\,PATTERN\/\fR Recurse into files matching the PATTERN. Overrides all recursion other options. Can be given multiple times. Default empty. .SS Immediate execution after compilation: .BR .TP \fB\-\-run\fR, \fB\-\-execute\fR Execute immediately the created binary (or import the compiled module). Defaults to on. .TP \fB\-\-debugger\fR, \fB\-\-gdb\fR Execute inside "gdb" to automatically get a stack trace. Defaults to off. .TP \fB\-\-execute\-with\-pythonpath\fR, \fB\-\-keep\-pythonpath\fR When immediately executing the created binary (\fB\-\-execute\fR), don't reset PYTHONPATH. When all modules are successfully included, you ought to not need PYTHONPATH anymore. .SS Dump options for internal tree: .BR .TP \fB\-\-dump\-xml\fR, \fB\-\-xml\fR Dump the final result of optimization as XML, then exit. .TP \fB\-\-display\-tree\fR Display the final result of optimization in a GUI, then exit. .SS Code generation choices: .BR .TP \fB\-\-full\-compat\fR Enforce absolute compatibility with CPython. Do not even allow minor deviations from CPython behavior, e.g. better tracebacks, which are not really incompatible, but different. This is intended for tests only and should not be necessary for normal use. .TP \fB\-\-file\-reference\-choice\fR=\fI\,FILE_REFERENCE_MODE\/\fR Select what value "__file__" is going to be. With "runtime" (default for standalone binary mode and module mode), the created binaries and modules, use the location of themselves to deduct the value of "__file__". Included packages pretend to be in directories below that location. This allows you to include data files in deployments. If you merely seek acceleration, it's better for you to use the "original" value, where the source files location will be used. With "frozen" a notation "" is used. For compatibility reasons, the "__file__" value will always have ".py" suffix independent of what it really is. .SS Output choices: .BR .TP \fB\-\-output\-dir\fR=\fI\,DIRECTORY\/\fR Specify where intermediate and final output files should be put. The DIRECTORY will be populated with C files, object files, etc. Defaults to current directory. .TP \fB\-\-remove\-output\fR Removes the build directory after producing the module or exe file. Defaults to off. .TP \fB\-\-no\-pyi\-file\fR Do not create a ".pyi" file for extension modules created by Nuitka. Defaults to off. .SS Debug features: .BR .TP \fB\-\-debug\fR Executing all self checks possible to find errors in Nuitka, do not use for production. Defaults to off. .TP \fB\-\-unstripped\fR, \fB\-\-no\-strip\fR, \fB\-\-unstriped\fR Keep debug info in the resulting object file for better debugger interaction. Defaults to off. .TP \fB\-\-profile\fR Enable vmprof based profiling of time spent. Defaults to off. .TP \fB\-\-graph\fR Create graph of optimization process. Defaults to off. .TP \fB\-\-trace\-execution\fR Traced execution output, output the line of code before executing it. Defaults to off. .TP \fB\-\-recompile\-c\-only\fR, \fB\-\-recompile\-c\fR++\-only Take existing files and compile them again. Allows compiling edited C files with the C compiler for quick debugging changes to the generated source. Defaults to off. Depends on compiling Python source to determine which files it should look at. .TP \fB\-\-generate\-c\-only\fR Generate only C source code, and do not compile it to binary or module. This is for debugging and code coverage analysis that doesn't waste CPU. Defaults to off. .TP \fB\-\-experimental\fR=\fI\,EXPERIMENTAL\/\fR Use features declared as 'experimental'. May have no effect if no experimental features are present in the code. Uses secret tags (check source) per experimented feature. .SS Backend C compiler choice: .BR .TP \fB\-\-clang\fR Enforce the use of clang (needs clang 3.2 or higher). Defaults to off. .TP \fB\-\-mingw\fR Enforce the use of MinGW on Windows. Defaults to off. .TP \fB\-\-msvc\fR=\fI\,MSVC\/\fR Enforce the use of specific MSVC version on Windows. Allowed values are e.g. 9.0, 9.0exp, specify an illegal value for a list of installed compilers. Defaults to the most recent version. .TP \fB\-j\fR N, \fB\-\-jobs\fR=\fI\,N\/\fR Specify the allowed number of parallel C compiler jobs. Defaults to the system CPU count. .TP \fB\-\-lto\fR Use link time optimizations if available and usable (g++ 4.6 and higher). Defaults to off. .SS Tracing features: .BR .TP \fB\-\-show\-scons\fR Operate Scons in non\-quiet mode, showing the executed commands. Defaults to off. .TP \fB\-\-show\-progress\fR Provide progress information and statistics. Defaults to off. .TP \fB\-\-show\-memory\fR Provide memory information and statistics. Defaults to off. .TP \fB\-\-show\-modules\fR Provide a final summary on included modules. Defaults to off. .TP \fB\-\-verbose\fR Output details of actions taken, esp. in optimizations. Can become a lot. Defaults to off. .SS Windows specific output control: .BR .TP \fB\-\-windows\-disable\-console\fR When compiling for Windows, disable the console window. Defaults to off. .TP \fB\-\-windows\-icon\fR=\fI\,ICON_PATH\/\fR, \fB\-\-icon\fR=\fI\,ICON_PATH\/\fR Add executable icon (Windows only). .SS Plugin control: .BR .TP \fB\-\-plugin\-enable\fR=\fI\,PLUGINS_ENABLED\/\fR, \fB\-\-enable\-plugin\fR=\fI\,PLUGINS_ENABLED\/\fR Enabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-disable\fR=\fI\,PLUGINS_DISABLED\/\fR, \fB\-\-disable\-plugin\fR=\fI\,PLUGINS_DISABLED\/\fR Disabled plugins. Must be plug\-in names. Use \fB\-\-pluginlist\fR to query the full list and exit. Default empty. .TP \fB\-\-plugin\-no\-detection\fR Plugins can detect if they might be used, and the you can disable the warning via \fB\-\-plugin\-disable\fR=\fI\,pluginthat\-warned\/\fR, or you can use this option to disable the mechanism entirely, which also speeds up compilation slightly of course as this detection code is run in vain once you are certain of which plug\-ins to use. Defaults to off. .TP \fB\-\-plugin\-list\fR Show list of all available plugins and exit. Defaults to off. Nuitka-0.5.28.2/doc/images/0000755000372000001440000000000013207540420015472 5ustar hayenusers00000000000000Nuitka-0.5.28.2/doc/images/Nuitka-Logo-Horizontal.png0000644000372000037200000001664112711355343022445 0ustar hayenhayen00000000000000‰PNG  IHDRÞaxuÀ×gAMA± üasRGB®Îé cHRMz&€„ú€èu0ê`:˜pœºQ< pHYsHHFÉk>bKGDùC»–IDATxÚíyœUµÇ¿ÝÓ3™%ë$˜@A’‚ ¸E@QÙƒKØ…ˆd‘M6‚ ø@(ûN@>AÀ‡²ˆ„$1d'û$™™¾ïS=éîé®{k¦«»ªû|?¶ºnuWߪ[¿:÷žEQzÏ8`*° hf—ýµk8 X˜¼W'ðТ]¤(Š¢¸rIAɼæÛiñªO`?í"EQÅ…,‚’ßA5ÞO3-}tˆ^JµCR»@Q”Ò üÜ¡ÝÇó´»EHzÓ“œ3gÎîÀÀ( ±’9}‰¹b}Úg $ iylØkDÃì†:ÓÙ§nEKcránmsm×?ëÍæK7t¶®íéqc:‰Ä`Nccãs#GŽlÓKL©bö†8¶œ£]¦(yÂûî»ï6¶··Ÿ6{öì³€Eå §/êàébbƈ|ƒÁ°Û°:ê“  Ù¬ñ™LÃ,‘Í›å1òþ¢ û²ÞônFÌxŸÕÖÖÖ6kÖ¬)©Têâ­·Þz‘^jJ²C€¶›­Àrí6¥Öéšjž;wî–/WGIt­"—'ºÞÿòZE·À^½¥ ˜ØÑÑñÖÌ™3÷ÖKM©æû‡#uÚeŠâ œ3f´¶··OK§ÓcŒ1DíHtó÷uÝðŽ0õí·ßÞ=Äóx=°ØHqçõÀëÀ®1½V¿,-òÛÖçêp.;s´]î?EQáH¥R·#—?Ñín¹º‰î¦m¥'‘H4MŸ>=ŒX½p 0¨÷i×<ôáµz"0¸È¶fà:œËÎ4$6Õ…{C`Š7á}ë­·ö4ÆEK·Õk]“eõfíä&º,澆766žÂ9ì+[ÄðZµ9ø5ép.;+‘^ Ë´»ÅÞD"qlµXº¢§&'GÑ ÏâÍ:þãBøØTö™|B/¥\Üà³}p '¾Š¢IcÌ—â`íºŠnî¾é@¢[†ß²ÝŒ3†•ø&z°Oà:½ü•1 8x øؼ \‰,o¼¦]¤(¹ÖÒˆغ΢[À¥¹â–nN‡§R£"òôÿ $MÝÓ: ”0Í{)Šâ`ñö¼ÅHt³­×tÑ-Ïoéèèè¡óþYŠ'†S<5ãGÀAÚEJ$„7Dt»…9Šn¹-߈ð)ÔX©¾JñèŒH^iE©8©bq²‘Þt0Ñë2[GÝE7ýQb.þ,Ñá Äœ–^nWµx7Éf0Ñíf/;‰n͆~¦CAQE-Þ<ítÝ®õTd×Ut-™²JE"‘ˆbŸÜ¼¢CBQE-^±x]E·j;ˆî¦¶UÉ;t/Ë”üR‡ƒ¢(ŠZ¼›¤×Ut³,Þî"k]þoggg%:ðMà$íb1ö¾Ü©ÃBQ¥Æ-^‚ˆnŽÕDt«~÷|ìyuŽ: (Š¢„+¼QáÍXÑÍηœ&º•ª´T&cÏ—;ÜhEQ¥–-Þ ¢›g*;Zº5áÕ|#0ËÒæGÀÖ:4EQÂ!q¼Y©]tÅÒ5ÙeDW2]Uyo;p&ð¸O›LçCtx(Š¢Ô¨ÅDt»—ãu]j'qÆH2{?¾¢ÃCQ¥f-^œE×PÀ«9€èÖHï$`üó4ßìtè0QE©9‹×]té¶Æë*º5•¹j&ð+K›€Suˆ(Š¢Ô¤Åë.ºÆ•¹*€èšò¬ñ¦Óé¨të¥À`3Ÿ6— q½Ku¨(Š¢Ô’Å@t [®vÑ­Á\Í+ -m¢yœEQjÏâ%€èšœ\Í&€èZS¯Ñ’|“ç'ßÈ:–‚ïülrÚ­M¡^½8Ø٧͉HçWu¸(E¸8h.°m’í/ÚMŠâ oLT×Yt3­!Ht1†¾©ùôå‚„ ù×û…ü\ÑéÆÖul8ø»eVäFà :\”"œ^Dt†zo*¼Š’¹©Æ!sUÚ¸ˆnÆÍÊeè•H–Zt#:ý,p¿¥ÍçoÕ虈¤ÚÌ;ùtÿô‹Ð±Ž>,p¬˜‹8Ì…As/·ìéÞáq|]oùÌý|Vök)ðµ*»®[‘±³E~÷GÀ6e<¦`ààà·ÀŸ—‡Ð÷eÀzoü­–ïÿ®ŽC"2"»”‹×ÕÒí¶V1Ñî¬þYޥɧÍÕÀ£ÞÅ^KœRD\“ÀîÀç°ÇE—‹ À°"Û>ü'&ý~06"Ç2سÚ¯’kz4ð$0ÂÒn#â  ÞÃâAÞÃýØ€šÔä½[ؾøp·wî6¨Å8DZ›èCý¢"º‘UÞ÷½'E?FP›yœ{¹]µg4éñ„Â探»Î{_Â1ìÜì}ö4d©âÓ!‚­ÀáÀÀàbÄa´òÂkÅ>½ÜõßùÎb‘ÝH;²ýøÀÒæGžå¤(Jüh¦:ˆn'pð¯ÿþÀsÞçNú—ñ··"á‘ïǨÅëbíµtM7Ñ5kíFIt!nê \œë`1ýBï_Š;À€]ÚžJi§Õ[¿"K2Ÿ¯p?´w÷Vr(F¯ÿôrv¬o¡}£ º1ˆ¾xÞÒæP$ݤ¢„I›OI¹Yã·qâÔTJ~ |9bý1Þ{諯ï ü¦—sld[½‘ÝXÄLŸŽx’ú1™Ø„¢)1å)àˆ šeÀïbÜ—ÇcŸÍ˜‚=©NOÑ~Ùx°÷²XÜ<]-]S°ÊŠn@^nG¼8‹±#ð}$¾WQÂ`Á½šˆHÑ4Ä[º–çhÁþønXžë­˜Ìñ^³½×"`-°Æûw-â¡ÜˆDlŽ„=íäYÖ{жý§«‹Ê*¼ñÈ\å.º¹¹šÓ*º=ã|àHü.îBó8+JTÙÖ³èê-í¦#Þ¿•p@yYO~xX`ßuÞkð⽌'Ä'"µÇ9|ÎyH ðËåúѱXã "ºÝDNE·',.·´ˆ¬Ý(Š=Z‘ÚÛ¶Lyï_V—ñØæ?A"$ÆxÂ÷d@ÑõãCïÞ´÷àa£‰êP‹·°ø:ˆn–Å»jãf¯úz–PfÍh˜üŽBS0ßrÁÿ6Ÿt$úÍ‹ÉÀ œì=5ã$$ï5½Ï)Jd¨÷g[K»€‘×0Ɉú_k=‘-Ë‘,X×!5ÈýØØ x¡, ÑõþßIt³DuCg6töÔoI&“«b2xÛ½‹õqËŒÉÀõ^§(‘áfdm× À!”'“Ùw<Ë{N…úã,dýwoK»‰eÞ8X¼É@¢køpu;M©„ÝY<}£±lÏÞP¸ëŒÏŸ†úT"Npž@¼KýR¾|¸GïwŠRqÎNp°eŽEÖUËey.¯`Ÿ¤‘¤¯ã?õ~„'¾ëÕâöÛºž…«;YµÑ.ºÆîxuMîºpVIÁ\ËÙ䊹Å{zS8SÄ]Ÿ±©¤`~E%c`Üöƒ¶Ù@žäMÃø9h\ pô˜¦¼¾UýEw“`FOtÁàÔÕÑb¦wÁþЧÍHàLjÄ¢(ågWàH†*?&#kžµÈ͈#—ŸÁ¹OY„wûaÏÓ'U )KÁ”‹Ì.Ÿõ^ŽÀÉSl,Û³ÞkëÀŒ¥Óžn¦µ)ÉÑcû0ez«6ÄStãâÈV€K‘Ê7C|Úœ ܆”èR¥|Œ@fœlåDBlj•H,÷A>mv)Ç'Š€è‚¡)µ‚O%•hÃCkc‚cÆ6ѯxŠ®8š¼ˆ›¿-£æqV”òÓ‚>ØÂÒîyïá9]ãýesž*K)ÊdTE7³Osj; ~”TRÖ»[›D|[ˆèLœ/Ø[°‡ †ÝsPQ”ÒÝ¿ïv¶´› |ƒ28 Å€Z¶ú”Wx#&º™m-õKÙqð£Ôe,ߌøÖ/Ñ5±Þ4p†C»ÉH@º¢(ár'¨~,D¢–kwâÙlc‹°"uÑͼגZÂèÁtY¾Cš“»sK®øF\tMü/Úgû-mF§èøV”P9ûzí$+Õ<í®.>rh3¸<ÂqÑÍ_KýRvl}¸ËòÜ”`ÂN-4¥ˆ‡èS îÙØK¤]VŽ‹WQj”}H?:¸ÔWµ»rèÄž3ô\ ÉnQÑͼ߷a©gùn`hK’cvn¡©>¡¢[æ!ißü„æqV”0Ø)`ËÁp2âÁ«tg¥e{c,Þøˆnæ½¾õ‹Ù±õQ’lÀÃЖ$G{–¯ŠnY¸ ˜ois’¦MQ”Ò0Iá:ÐÒîb¤´§RÜêõ#t•dÜD7ów¿†=øaêؼ_ÇìÒ—¦TBE7|Öç8\¼“uŒ+JIh>aiw+²Ô£D˜TE7sÌý2zðü±ô:MÃúÖñ­±Í<5k«™M9—3®MÝE—¬r‚…„“¬ÄYëÈyûv‰zÞ¾ù4¥Õ’VñnàTàs>m¾ŒîÓ¡¦(½âVàó–6O¢Ž1Þ˜Šnæïþ 3øÞXv(¦žQë9y÷‘íðD"1§Š?Ï^¿®ó5H€›7EéG[Úü8qªªF† õ{G ±¶Ã€!Óî@d}¶É{õAfÞ’Þ¿‘ sL­Yß©úbHqÑÍÞ^Tt±ˆ.>3÷ó›ëWRÇ|ס ôoø/Ÿ4•7—ŒñÕ€ÊÓÙÙYMƒáeà÷øWC…äq¾X˜ñاŽß¾¬­‚ßÛØø40 OÜèWUï› öŠìÁÝl*ƒß÷ÝŒ(·6¾G*¹žöt³Õòr>¶àWø8“ÇYã Å=€;ð/|° I±(¦¿1ì‰äO>qȬú<‘®Ç›½¶ê4ý³R&—#SÊÅhBBŽÔîR'FâÚ²É\5+†¿¯8øžgÑÖ©—÷e}:™#v›Jîeyìv9e•ò”rXdö5tÿ¬MYœ²‹È{›5Á^#²K¿škÎJ…˜ŒÄ úÕ><£Ý¥(¾ôC††ZÚý{Òÿ¨1©vö]ʨ"²ÂûÞº>¬éHf‰î¦Š?›<€³Beº½çýÝ¥‘¦K+³=~sÛæ}GVèÍV’ì9< ¥›ÿ=J¹iGÒ×Mµ´»)»Õ©]¦(©îAÖ7mŒCŠÚ_ƒß•D|A®Ä¿¼hM̵t{*ºYåùºö.ºÝ…³ÞÕJ¥x{¦œ1ÞÔ’¢(…¹ɯìÊ•ø‡ôEÅ‚ ©pVÑíD¦á× ÑQˆ”1&R¢kLº€øº‰n÷}£G"‘¨æ›Æ$¤ú‡_:»Ë`­–¢(¹œ œôÜ‹Ì$-‰àoú82¶c >Ë Ü3·€÷ zó½ûÉ ïe«9ü°e$„7–®1“è‘è¢SÌQà-àWÀ}Ú´"ÎX§jw)Jû#iV{ÂpàNÄ+8J…î‡">£z¸ððwd-û`U5œìTeD—"¢krÓ3vé­‹èÆc·Êâx q)0Á2¥4¸xCï·%%©][z»óà"oüEFà‘ˆn;²lõdéj]uÔŠˆ®)(ºÆä­Õ]%2¬@2íøQ‡8Z)%~Ö.¨JÞsæŸûF䘯> ý:¤¢Ùà0$/õºj=¡ÉòO/û‹®É[ïuÝü}£ùªn¦[ÚŒCãzð2”êb2 ý[ìÞËIdÊy‹ óÎȬ–+ ±¼‹kaœ$+¿¦›+ºÝq]%R¤‘<Î6®¥†cùBàcÚUÅ:$d&AÆyÀ?®{*<û1·eNàtïüÃ2_Åï9IÑ°ˆˆn&éFδ³£èµx#Æßû-mFa//u¢ä¦>¥Z辉$ÉÈÐå`~øY…Ž{7à‹í ’Dã—м–Ê[¼DOt»ž@Õ‹”q6öªDçÒs¯Ç(БãhŽy?*¹|ŸÂ ißÆî½|’N²ÜïØî§H•r3Œä‚NVÊ{¹¸¥›ÎÙ?ˆèªÅ9æa_—j"Ú™wl'­ODŽóS¨Wsµpð?>Ûÿ\â0sG[.êo9´{ )¬#¢p‚Sㆬ¡£KË6Ýcº[ž&ïdòBg»—Ý<4+íhéšîe ƒÔéU¢ÈUÞS°ß$ð%dz:j´[¶ŠÈqî©—ZUð;ÜJh^d¬ÚߧÍ@à>¯ÝÆ2ûXÇñp‘ø ‹1Q8É©ÍÃ=]ù›í¢›ÉùœS•ÈUtcbU¦ÓéZ»‘¬C¦“ï´´»©Áµ@çõ–íQqhú²jVìywoà4ðàU`¤O»Ý4”åHX³»C›÷±çt“]£p¢Ë35å*ºÆtÏ@å(ºJ¤¹ xÞáiùä»m: ëª ÀÞz™Åš—¢÷A<—!ÎV6ëñûˆ£VØìæÐæÉ ?\!Â[ÞõM»èæì“N]õjŽ8g`w¹œèLÝf°å¿Ý6Çx 2­XëÄy€]IÏF¼ˆ[dÀ-Àö!ÿ—øáT°G£kÄâ &º…ÇŽ]t•Èó2v/ÆÁTÎ颶ø©¼—äÑzyµ[nòàAK›¾Hx_sˆÇáòÐünûé𨜰ð-ÞtK7Ï;9˜èÊw©ÅiÎÇžäü{DÄÂã‡ÚØ ßVÀ!ª¹€Ä¹Ö*'³-mF¿ ñ\f]*Y•ì¤¨œ¬²¬ñ:‹n±z¼¢«ë¼±`¿çG’ù&*üÇ¡Í~<¾³#`qG…¶þí«€#úàXàÄŽÁÅs´R!oû!!w5bñ]“NçZ‡DW×xcÁd‡'ó/GhZèM‡6•²8·#ši•b­e{s•ÿþ×8´û°Sß¿Ô¡Í° õM¤–°ÊdñIÐAžÅëj骨ńÀ™í®%IÿßZÚ|–ÒúBø5Z‘(ßêó£ ôÁmÀí–6Èzo©ûc‰C›‘è“cq uª.‹7PV¬<Ë5ˆèÆÁâíèèÐÛ£ÔÛœfi³ÑÉãüW‡6g–ù˜NöéXW3¶õÃá52¾Nõ¬_?¶E’u”—e™¯”¹/†#qÌ‘¢Lóíî¢ÛÝru´t5_sܘ„ÝæÜ =!çó¤ãSu¹¬Þ/×Ù–®aáµÍL ‰Èõ6mÈz¯màܪˆ¹òœC›Ë8KÓ²c5ó¡ÃÃܸ_³qs¢ºØ£Dßù"ö®AHÆ­°éƒL§ïÖƒÔð…·,ö®£èv'µt«K°;dEå³Í¬AŠuÛø â¸{ ù¬[‹l¿À²µ¯w"…9ü8®†Æ×Ø#ê‘|έ%ø¾ÕÀSí®"ÜT«ƒ€?Ùþ'Ëjèã$Y /€èfÇâ]Í\CV:´»‘ÊWÞ¹Úñ o"p]Ç{Š'ºÅ¦ÍB’”جòjǶθ7Á×ÆãÌÙäÖô-Ä(à”f)Â¥¾î0OüZCø½»{ã`oK÷ǖϽÔgÙ,^'K7+‰FPÑU«7¶ÜL·´Ù™ÊOÎð¦®\˜„8mQ‚ïÝYcþ ÅK®Fê¯Ú¨¯ëéßm~GÏËÃÅÍ3ºÉ½ÌÒî«Ày%ø¾§·Úí ¼æ}o)ì= ¿ˆ)ÄÛ7,Ÿz4EyÖx]E×kkõh.$ºFãxcJ7(ˆÆ$ì+f"åÛz»8Ö»I¼ `i{n©økàzr)-¹%’3ص”b ð]ïq¥7û'>&8X'—•à× ^Õ.7»‘HE¦§Øýž‚;z3Ló¼qà÷o§;|fèñÞ)d]$Ü)¨¢[¸:‘]tãbñc4ž¨ðÍòÄË2Ê,ðîïÛ÷CÒdž <ü ø0XŒ8lÕyVÔ0ï&òdmj;Çï¸)xîB-ïóžu7ØA|_ðnü{û-òfzVÓnHøˈ‡l†=€›bÖ/O!Yã.òiSÜ ì‚ÝCÜ¿x34®¥÷÷^ ¼™¢W¼‡œÞƒîjdú·/2-þI$v~oÜ‹¬Å­EkØ'#eŒY l¢æ]³©¯éª]ë&ºq°*“Éä"ÕÙ‚œ|-âp‡wc †Q‡¬+–zmñosªáíðŒ~äØþ Š;ácpLûæ`/Ëu8Ìß}é]щs¼‡È ‰+¶ œLl‹‘)íw¢r~“ÈtXØfž“èc ®®–n,,ÞummmóP 1h„¹p&pk,»¯,?qc\KWn2þ¸ oø¶gIú1™vîսΛ)x­Â¿ùïaãå(ߤ1æ±ò¬óº‰nîo0ÑÁïÓãǯåDî.7Ìù1¹ŒLÝUâ‰oŠwS[p¿ZÞ¥H•«°ÎMœk/¾‰=ÞùÕœloo¿{ÍÑÞ¼®¢ÛÍ«¹j,]€t2™¼ Åö”|no.³2bõ²ý‘|Îå`%’|àX$¶¸ë- µÂýÞÌD×Ř÷Ísˆï $ÄhT/¿k5RðäTdµ¼¬ÿžHñxÝ >û7….¼ãÇ_cŒ9Ãáf¬rÝÜêDÁD7âÖîM|𫪭VîBœ^‚²±BÇûg¤ÜØyž5+ê*[{ýãÇMž~^AésóFÊQ¥ô«˜ûúq”¹xÔÁò»ÞGÄÙj;é7½ ì€ø?øq³Ï8¹=ìÎOvØa÷'‰ ²œ-Ý¢ŽQ±¶tžX²dɤ>{½C›¸MoŸFð¢æK+x¼ë‘iò‘HØÆã%èó Hüî÷‚?ÁmÝò $"‘÷ÜâLEo¶‡Écˆì¥Ø×6‹±Ö³ ¶zòÝÁ1zvŸ=z1 •Ïb$$o’õ¿=çùÌE‰öòŽõÇ™Ë}ÆÉ]aw|Wj¬C=ôʇzè}cÌu”0Ww¯f»èš¬úºAD7‚^Í€:;;/˜8qbgHß±Y˘@áŒ+Kp‰ ¯xâ{öõ–vÏ¢¹-ǽ¸Ó{5{7‚]‘˜Ü-½Î d­µw³]í½–y7ÁHìîó!Z¥æFd격ÈͶÒçfâÑ{9ð9ÄyèÓÀ'ˆŽoì¬õú|>0Ë;Ï"!`½ˆß¿@B̲I{Ÿÿ|úe…×—"qç[P8]âÀÎÇÍÞ«?¶õ$œn”÷;À»ž½q²ÒÛo>;I–2#ŽÓ ÝR„M™2¥¥¥¥e’íä“ÞEÙc:Ó0ù…Õ½‰d«èn5¨¾ã¨¬ØsÄÔ¾Cšç§\-Ýis_¹±³¹ÒÊ»ø¯1æÑD"qÛá‡>EQEñø‹$!ÄÊk%tEXtdate:create2013-08-13T07:50:42+02:00Q6íù%tEXtdate:modify2013-08-13T07:50:42+02:00 kUEIEND®B`‚Nuitka-0.5.28.2/doc/images/Nuitka-Logo-Vertical.png0000644000372000037200000002326312711355343022063 0ustar hayenhayen00000000000000‰PNG  IHDRU?öcPgAMA± üasRGB®Îé cHRMz&€„ú€èu0ê`:˜pœºQ< pHYsHHFÉk>bKGDùC»%¨IDATxÚíyœU}ÿßϽ“™Ìd²M@ö … BD«ÕŠ‚((¶ S¢? H+¨XŠ²i«´?ÜpiíOÑŠ¿ÚbÕ €èÏV\jEIÌ"ûÈB $™ÌÌ=ýã<™äÞçœ{ç9Ïv?ï×k^YæÜ{Ï=÷yÞ÷{¶ï‰©qÇwôöööEÑkŒ1GDQ4Ï3˜ ô«…&²}Ôð½ßvüxc&ükâßwúç«æNaïÁÚ„ÇN©mfÆ”5IÏÞäy›¾pËrÆÔÙ8ò’"5ù“ÀðPEk€_FQôÃ:èv]é© &Ï]wÝuH£Ñ8x 0¤ñcËÿÚÑ¡PÍn’3M„j0¼}áTž]ŸðøžØ·ÿ‡­ÍîÒ4¦µxÍÄÇ4Lwo;§ ÅÀUccc_:ôÐCŸÒ•99zÔ³zõê=¢(ºblll)PS‹LBŒ„Š±åw}­–¯’PŸ©Q‡ï1c>Y«Õ.^½zõŇrÈ?DQ4¦+´3$‚Y»víqQÝ ¼]혓ˆ=„ÚÙxB:Bm¿¹3+Š¢Ï­]»ö¦Õ«W匿MRÍŒ5kÖœoŒ¹ØS­Qt¡zÊ5„PMi›ú¨(Šn]½zõKtÕ©ûŸE—ÿãƘ ÕÙvÿ;ªiÑýÏJ¨%éþ7c/cÌMk×®=aÞ¼y?ÕÕªH5”PßH¨%‰PMîª)u»GQ4`ŒY¾jÕªçë*”TCõÀ•j‰² 5ŸµìBÝ©ýg×jµëV­Z5]W£¤šk×®í®ÒpI•„JB­ŒXçEQt©®HI5Í‹ê|àµD …jâ2Éf•PÝœ{çwÎוéF‘—ƒ{î¹gÖððð_ª%Ê-ÔÝש62j‰'ªvsE­V»x³®PEª“bÇŽof¨%ª¡’M„j*­ž´jÕª}u•Jª“¢Ñh,U+H¨*ucÌ[u¥JªoCÕè2 Õ%· B­žX£(zƒ®VÇ8‰š ñF¹1FIg‚µox¡öþg TS½ˆõ¥7ÝtSÏ1Ç3ª+X‘j'RÕlg‰#Ô6ök)Bõg`ï½÷þ]¹’j§]穲 [Óê.Z3y µ²rÕ}!©vF£ÑЬé„jÚS•P;ì]ÌÒE+©v©NU+”9Bmg€„ÚÆ}¡S,ÐD•û[YÖ§) µùø¬1 Œ©5ù\ Õ•«¼÷„&o%UQØ4µ.‹èט|"T}«û/DnZ Ðå7>b .T‰U‘ª˜@£ÑP#„îþêòãu&Ux¡jøHR"§x5L—ß´3Y¥U¨û/$Ô" Ur•T…(¼P)PåTuÿ…ÈÁª#Tã®@0¡Êª’ª˜x»i’!pûf TV U2j¯¡(Ò2UuÿE±‚tù»®U(RUŽVÓŽPóªäªHUˆœ¢ÕpBM²«„*©fJ£ÑÐøQð(5d—?þwÓ1ÕF&B­â˜ª6ÅHª¢jØÕãˆêÔ…ªHUÝ!*(T¯üÿ¡„*¯JªBdïÔ0]þgžÛ¹L5¤PeUuÿ…È:L ¡J¨BR-àm¯ áÛ6¨PMÓ‰ª¦M]¨F׺ÿBd¬ªo¤@¨BR"§ºó¡NJ¨Î5ªªTE5Ç2* UríF4¦šxŸë¦È¤}ƒ µyb³Û±a„ª„*ŠT…¨P„jüâÅBÕ IUˆÜ¼4B•P…¤*$Ô…j$T‘)SM@ U·¯1™Õ´LRZ¨Õ\§ª¹Eª¢t7jʪïÂÿ”…*$U!ºH¨d T‰UR¢bBuk- PåTIUˆlŠ„**‡&ªÚŠ¤D˜¶ (TcZgþ.T£ÅÿŠT…È!\ (TÿP9}¡ IUˆ¼œšƒPÉ@¨«ºÿB]ÿŒÛ7´PM‹ƒÿ²ŠPuðŸ¤*D®ájÚBMŽC Õ1ÊÜþ=‡Z†ïѯiòÏV© ñ|žÖÏùtcîºf%U!¡&‹5@—?2cÌé]‘áƒcBÒñ1Æÿx˜FÔ·‡®ÛÖhLUäîÔàBuuýCLJ™,&Á²ªÆ‰%UQ ¯†ªÉT¨íJJBU÷¿ÛnvMTUF¨Mª„\å…j@óTŠTEñ»ÿÙE¨„\7Uq†¤*J¯j›4†*¡JªBBõÍÉš¶P»e U;ÅÚAcª®[^ãªÛ–L„šœ¤Z]þö…ªAUEª¢ÈjàÕ£û¯.?Úz+©Š š5ˆPMÚBíæ.¿„ªî¿P/­.¿„ªHUTM«…jüÆÔåïè E(Rmeã /ÓðBMHR­.GB5º/$UQüî(¡º{­êò·÷þ©ªû/J¶†ªG.Uuù%TIUH¨“*êòw,TuÿÕýïø^×7s`—êøÓ4ÿ,Õåo_¨º'©Š"k5¡&©êòK¨’ª¨hÈJ¨œ¨*¡J¨êþ u2BU—_B•T3¼çuAeѶ!…Úp&T‘PÛjC÷„ºÿ¢ Áj¡º¿ÕåW„*© 5¡ UB•TE%µšS„ª.¿„*©ŠJG¬„j$T 5[4Q•€ªd#ÒB5-ª<1<—_=~n×õÒy¶Ú¯ár]À’ªè¶Õ$Ê$ÂP×PK«û/D û4„P $U!¡J¨BÝÿêwME˜ö /Ôfcª¢s¢(R#Hª¢ˆL©ÁŒÞˆÍÃ&œP›æÎÇFZ™½ÅkîMïêå §µîv‚Á„?šþ®É³ìò%ƒ#éK³cMó—3-kC«ô‡Më`à€9SŸ««WR¤·ñö…}|íöažØf‚cX³~¤Åãš¿¦ÙML; µÉA…f£&þÛ{sCëaÝ3n59‚»ÅkúÔÁ˜ÝåÚ´ñß_UÚSWok4¦*reF_Ä ú˜5µD¨-‡s$ÔŽ„ª1jEª“¢Ñhhü(¦÷Âé z¹úöí<±MB-¼Pµ|[Rí”Þ-ý½=Û:x¤é°ˆ óZí–5“­SóÇídÛ謦¥göE,]0•«Wnç‰í µ Bµÿ–U%ÕyÎŒ{÷Ýwæš6]’´å/i©OÒdƒ™d¹Ýo––¯°þ†:«7Ǧí¶ë©óûøÚÊí<5l$ÔB UÝSM 5±^£2ûÌžz_Ë&꯱táT¦÷!¡J¨’ª„*¡ºêU‹2û‡Ìè}à™õ£»þ M8}ÁT¦÷FjÑ„*¯Jªjq„:þ÷Z4Ê †ndf߃-›tN¥ û™ÞI¨ª¤*¡J¨®rV¬ßOëPÄÒEý N‰$Ô‚ÕȬ’ª„Z<¡NŒX¿ïŽXõ3Ø[“P%TIUB•P]åjÑ/º™}µlê=jœ±¨ŸÁx(@BÍQ¨òª¤: #¡fP±^ÏŒÞ[N^Íé8ma?ýS" 5W¡­R•T¡Y¨;G¬‡9"ÖçL«±táS" 5'¡*P•T%Ôu¼þµh„̾.Q¬{ ÖXºhS" 5/¡*¢¤*¡–§þõÚh,Ö‡Åzú¢¦Ô$T UR•P%TWý­X¿ËŒÞ‡ZŽ±î5ÍŠµ¿ 5c¡ˆ)©J¨å«½6Êas®gzï£-?š½k,=b©S" Uª¤Z¡"¡æXÿ:ÃÌŸó]¦÷®Kë¢iô×# UB•TKjW 5Ãú×£>´<1bÝgzSM£· UB•T%T ÕUÿžÚº–Á)¶cÝoFÓ ÒW‹$T UR-`%ÔÌëßSÛÁü9ˇöŸQç´#¦ÑWGB•P%U UBuÕ¿§¶ƒùC×&Šõ€™=œºh¾žHB•P%U UBuÕ¿§¶ùCËœÒZ¬sgõpÚ¢Azë‘„*¡Jªª„êªOm˜s<ÄzÄ ½u$T UR-ŒN͘„ZÐú÷Ô¶³`Îr¦õ´ž¼š;³‡ÓŽ˜No-’PSjC‹ÿÑÁ^ª„ZÄú÷Ô¶³på¬\[Föjúñ8«‡sÿx&;ÆLËNHǽ—N:;i?wÇOÓyr”žzmåçdI5½«SB-RýmÄz-+7,æé‘=š~‚ƒ½ê¥Ûƒ3ÃjuÿVB-Jý§Ô¶±pη˜6e½.Q¡HUB•PÓ¨ÿ¸XW¬?…§Gæèr’jÙðô>oÛ1Ðæð@«¿“çØUH)œ¨jd#Ôä„*…j”P%4J¨¢HU´ìþ‡Ë›4†ª’ªÈ©û:ùv^B•Y…¤*rˆTà ÕSlª¨ SuÞ÷L¨Z¨¦yÛíú¼¡„ª1U¡HUdnÖÐÉ·¡ IUtWç?ôi¹ U‚’ªÈɪ¡‡ñ¨„„**‡ÆT¨âz<“Ùy[I‰M Ucª¡¯!µ¯"UÑ$PÍæÃì#TÝðBRù„«Õªœ*$U‘Cœþˆm:H¨BR T Õxf©’PEÑD•¯|$Ô¶„êL¨D¨¦²Ÿ›P¤*Šo×€j;‹ÿÓª’ª¨žPMžB•\…¤*²vj`¡úçT‘PEõИª3¨3|?¡…j‹ÿà UcªaÑiªŠTEŸ…ª"T!©ŠîëþªÿÁ!„*Á IUd?V¨^ÿªœ*rFcª T3¡J6BM^§V¨SÕ}¡HUdÜý,TŸî°UB’ªÈÞªa…êZS%¡ IµKÝcL…û9ªèPµÚ¨Z¡5SM橪®S -Ô¦cª “‰P5¦–F£ñ”ZA‘j§z¼šï+¼P=BÕ@ª„š(ŠS+Hª²ºª] UL"R]­VT;To©ªSƒ µ“}ÿjY'Ÿ|òµ„¤Ú+W®¼X_E«†ªq NB-­0jµ›Ô Éh¢*Ë.»¬±|ùòήP¤‘™P›„ª&ª‚^?ßT+(RFãê Þa…ê›TEjÙ¸ddä'jIuR,^¼øfàß«éÕ0BuçS•PK¥^¾dÉ’1µ„¤:iêõúyÀˆ„ê+T#¡V£££_R3¸Ñ˜ªo|ãïøÎw¾óÀ{+md"Ô– U2ªÆTS§EÑyŠR©¦ÊæÍ›/ˆ¢è7êÊe¡¶£"Ô¢EÑÇN:餟¨%$ÕT9óÌ3·ŒŒO6H¨¢kåªO<ñRµ„¤„%K–¹bÅŠ·-[¶lDw½¤œ7½éM¯_¿þ•Q} .iša„Š"Ôòp?pââÅ‹Ï¿ì²Ë”â_RÍŽeË–œ|òɇ_Ê13š«P‘P‹ËzcÌ…ccc‡.^¼ø{jŽÎÑ’ª†€¥Ë—/¿¤Ñhœœ ,*ô–„*,[ŸDQtÍèèèw–,Y²EM2y"5Aú|ó›ßÜ3Š¢…õz}0³HuÛ>Úèùñ]Ûÿp¢ÂÐh"5³ËÍþ§¯xÞàm³ûëÛvþ¿½ÖìsÀÌ»ñ}ŽNh4"sëº~¦«0©Õëõ5}}}+N8á„aµˆB!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!„B!:ãhà:`° ¸ø0CMÀ»€'±g‡ïü3ü0MM$„ç²&²ÿ¹x¾šˆß$´‘ŽS !ÞáŸXgwy;­r´Ñ‰º”ŠCMM rbø;rÏ.Ts IUˆd^ìáYö45—T…Hæ°6Êî ©É„¤*Dz×^]M&ÊraxØAëðíÀJàE%}Ÿ ¬oñÞž.Ð¥9¿o£ìÆøó¢lÇ=;þs/0XÂ÷x­ã}= Ë sf›=¯»¿ïò¶ÒìÉ"Õ¾6Ê|¸„ïsªã÷ýº2g3vª‹Gª¹D™0mþl*Ù{üã=©k™ŸNø\ŽP)R-[¤Ú.}À§Ôt"%Þ¼.þâÛ «ËÀmj"QõHµŒ[ã© Eª¢°‘ê8Ÿ¦¨ EEØ/A^›€7¨‰Dh©¾øs5¡¨'Ð:yË,lž!‚JàR`O5£¨Ó&ù{!R‘êLàãjF!„HGªÄÝ¢#Õ”Bá'Õ»€†ã9>§¦B?©þøª£ÌËSÕœBIÕaÏÈIâïÐ`¾BRõâ1Üû¯÷‹å+„’ªŸÖ8Ê|ø5«BRu3¼ßQFy„’jÜ€ÝGŸÄ›€×ªi…’ªï‹£Ö$>ô¨y…’ª›UÀçeÎQó !$U?><î(sþG !DWKu3p‘£Ì,”@!©zóeÜYÙß ¼PÍ,¸{¢m³<¦Ǫ‰D·HµœëñüŸU3‹Þ ´øÝ^ñ³]!U€Ÿÿæ(ó à­]ھ˰Û{wÀÆ€[€éªëÑÀ#-"Æ»±“!˜äï[ÉxEüÅï{<ЧÏy<;´ø“Š]×CØmék´>-áà ë4-îŸü%ðÀ€[±“ë÷°—Ž[À½ÀÍØcì¯Þ,š¬]ŵŽÇÏ+˜ôtxs¤E^gTÝæxÝ×èFù²£®zÝ-“¼þšqŸ½âgù$Û¨HgTÍïg×{^GØö½Øsò®ŒÅ9’òg¶ø°»©)³H•øà G™ýéμS'ù{Õµ3úUŸ ìÜßÏIl£óÇÔáÅÀãçþa<|t$鯋Š…ú-àaì)'³²’*qWàG™ÏÓˆ‹¥d¸ÎC¨cÀŸÿ?å×?øyü¼Ë€w\†Í-½4+©n.ðˆt>©kSˆÒ_^äQöàú”…öcìÞ+rn‡!àkÀ5I=·ZŠ/ø/À/eNBKdDx¶©>©r9p²G¹a'ˆÒäo€c ÖKbц–*Øñ†£Ì•(/€Ë€Û±y³¸ªÄmy¦G/àjÂLfÎ/h»¼ øv3—¥-·_c^IZ[x8ð´~U„c°°ÍÇœGò²ªR¬ÕYp´gäù#ઃoàg€û€ßÇ?kãŸuØÍ%[â?Ÿ†ãîûtìäÛÁØeTÇ/mËÇa'°.)U°³üo&y ù#À7·”I19æÅ‘ØG¹ØYò‘êø[ìøíÏ€_O´ñØ­ñÏ:ìÒÇoÅÿ¿O¾˜íñ<b—ýݪûöè•¿v”™…+B!lîä!G¹û€§2¬ÛïK°+‰ÄR»±M¡&ñH즃ã/uì꧶Cëv¹2½“xp„®_! Å”X&óå6¯Ç®á ɸ° |^´Ýøu7bwg}Ú£ì±Ø¥ƒJu›ÌÚ5V¢qU!ŠÅ±c©I cwqÝ™A}NEzl‘fÍù±Ð], -Uð;zå(à-ºŽ…(ïp”1ØmÀ?˨Nã.^4° þ7:ÊB¼vµ¸B>G¯üoòÍ „°kÈ/÷ŒÜ®é²¶yø‚£ÌðÊ,¤ºÊ£2¥kZˆÜxð±;§’¸’î=-ù‹À¨£Ì±YHìò)×Ò©ÏÕµ-Dæì|Ï£·ømÜGÔW™‡±k•“xaVR}÷N å"{¦a“¤ìë(÷ à4Ü»%«Î9~¿0+©| ÷Ñ+'¯Öu.D&Ô°p\ËWoÄ&wîvnvü~/ /+©ú½v̦®ÏNˆà\Ë2‰G±[s7ª¹XéQfßZ†ò9ze>p¶>;!‚ònÜã£[°‹íïSs=Ã&2sjWꃸӠ}˜£ÏOˆ ‹{EÎ(vÝåoÔ\ý%·?k©Þ|ÂQf6Ê DÅ&q%Rz7î™îne³ã÷Sk9TêoeÞ…MÅ%„H‡=°\ç,]ŠMß)ZG«IÔóêV첉ÃNZ !&O/öD׃後~“ –Óëú½ò*ì±BˆÉñeÜç;݈&‰S!ÏcMÎ~åûØÅÉÛôQ ѧ;Êü76±ühEÛ`lþÕý±kI÷žƒ ™ÌÄ&ÕŸŠ=N¼è‹{̵øO丹yJõVàŸHΊ3›àRÝB´Íîü=ÀŸ`);ƒØãPŽÄ&°žM6=½["U°G¯œBòÑ+¾‚ÖË Ñ/ŧœ”$evqÿº’¾Ç{ßâ÷±ˆlªåüúëp½Ò{–bbï»$œMÝvúF`M ßßð¸î¿ˆƒ³#)ÈnÌZêàsôÊ)¸³‘ !lW÷zìØa7ãNR4fcO y0´.b%‹ ÕüRŠ}å"‰:ð¯ØñDGcN—öì5À_Ľ×BW¶\{Çà,Ý7B´äÓØýú¾\üq "ïïa3Ýí‘Áëa‡F¶`Wbñ¦§@÷>l˜¤:}»ÆUYs„˜È9q×îý 6¹òã|OÏÃ.©<<…ç2Ø•«€ßaOc}0þÙˆÍûü÷–Eª¿>œ—Pf;±uŽî!!žáxìÖîNØøgììy‘’Pïü;éÖ Û€Ÿ?ÅŽÿx²›ºÿãø½² ¿1#QîkAøs“›ox-pqÞÏTàÚ„:‚ÝŽ{rîCCp¦g¹¿Á&cÊš½ñ˜,²T}Ž^)z^עᾂÔóhö¿*|ø? ¿ÿà2Ô×°ëD³b ðVr¿Å/$ûû†³EÆçè•7cZ‘Çïg¤ž/“‹*ÁUø¥Éü€ofØ›Zèy?\ìq_…Âk)gÑ¥º5îâ»(j^€íŽßerèù¨ôÜ€ÿ¬y8xÀQîÅØ­¯Yð2÷Ó|X#+^T©|÷Ñ+ ±'@ טpÆ1{WËI¥æWØ„Ôcm¼w¢øùÀ߬ƒOo)ÏìtÞ kÊ$ÕuØõiIÔ±;2ŠÂeŽË±~D‰¿Çéæ{ŸÄž®ájƒ3€wªƒO†¬¼|uvÙaå¤ ~G¯ÓN¨˜ßz”É+R|>ÅœÜË ×i¢ÿ+?÷(÷yì{i³Þ£ÌÞ9µM[Êe“êüŽ^ùÅHr/ð¨£Ì‘NÞvˆ€/ ÌT»FkIÌè‚6ø ðUG™©ØñÕ´ÛÃ'Iö9´Éø-÷*­TÁïè•çRœ¼?ö(óþŒëôÀ±ˆ¸Ê¸Æëöë’/—sâ¨5‰yØiâ3TöÚŒÛb?:X§[Ö­‰ïÃ=±pANßl»r£ç·aVÑê++Zü®ÑÅRuõ(ö(ÈõšmØñUWä~ ~Ùä|ù¹G™×gØ»êÇ&Ênw™f­¬Rý]Ü}Mb AYr-î5„õ¸Ûú‚Y€MàÛlëáMŽ:V™G<¾¨î’hu-~RW/Mé5‰{YÛlìN°ÐôÅC/î ø¨]ôœôsmA?øYØqWý~à(³>p]¿æQO|1`^wq[½ö‹âvhõûOª×–]¿wÔå?rºÖW9êj²ó3×ì}سãÒàz×{„°Û»gÇŸs«×¿ÁqŸœXf©‚Ýëìªÿo€ÿ—³T¿Ý|Äú©Ã2gcó´zÍñÅßIËgº@ªßóø|Ží"©N‰#HW›ÜÒðÐñž÷ȧ(òy pwÂëŽÅ½½¤ûdIÙ¥Znóx;r–*Ø£€çÏ÷IçX‹yØ1ݤ×z’gS¼%],_è©^âñÙÜ‹g ¸&Ì(™TÁŽ#¯÷h—¥ðZ‘Ç{ÿ¹8!¥÷8›”iÌñš_ö¸O––]ª`Óþ™Iþd!Õ}ÍmÔéIlŠ¶NÖæ-ÄŽÑ{¼Î;=.éb¹ª ¤ê{-Ý‹ºÄiØ,õ+âÇž]2©Â³ÇW'Õa”tÆœm£Wg⡽c:ìÝ÷ ·x¼ÎJž]«œtŸœU©‚T.ºT‰ÖnÝF>ŒÝÙqpñô`Ô÷Œ»$o‰Ç=WµñÜ×ìR¿¤‹åŸ»@ª=žQÙøÏõØ C㱸ì*?Œåùì$åÎù§JlòkŸñÎ4较ûä!ì‡Ë°k¿çbç]êØ™ü=±óoÅn"º½çÞäyŸ|¨*R=»¤èR¿Áÿ,~~Ì­÷w­²TÁn ÙîוTªµøËÝõþnbò+E€[ rŸ¬c÷<ª‰ºU9Bã>Š±|ʇ÷ï46“¿þ”öö»wËÖKØÄsJÚ. ìùV;ÊGµ“ak<äp[Îïù.àåØ,yÞŸo­b7Ã%¹8ßM“GÏ«ã öé6×-R]‹…úlÊœ»ö±x˜Éµž÷B&?‰´ »QeyNïõ+ñ0Î]m>n¨*ÝÿqÞÖaˆÿxNõ}-6Gdݘ'âöIâÁ„Çÿ¨KºÿãœG{&í »”±û¿3ôxŸHçd‹{úÀ¦Œî“U¸zÈuŸTIªã]ÛN¹ób*ðWñØMˆ‹dSÜóY×÷avŸXŽ·t™TÁÓühŠŸÅÀ + UâÏÅõ~oÆ®uMƒç`÷á?è>¹»õÖ§÷~qÂ}ò6©þKɤz$öÜvtEêÝ‹Ý‚w]‹¬Ÿíص®Ë€™%èR&½—oä\¿Ø#êð³Ø‚=•ô$:?’ü7Ž×8>‡v™…Ý.îzÿø<Ί#‘IÞ'waW̤~šðU´^ÏøXNØd9+î~øl x€tC¤Á@Ü9?ýi|lŒ…;߬k°»^¾—0½DŸÕE _"ë€×¤žõxŒï’8J»=ÝOÆâ½ø:6¡ÏËH'ŸÃ»ZDhcq48˜S›ì…ÝZ}_‚à>ø ïÕq[_ ü$¾OÖcç Æâ/€a×¥f™ÀH!„Bˆ‚ð?WÛ²6i@%tEXtdate:create2013-08-13T07:50:42+02:00Q6íù%tEXtdate:modify2013-08-13T07:50:42+02:00 kUEIEND®B`‚Nuitka-0.5.28.2/doc/images/Nuitka-Logo-Symbol.png0000644000372000037200000001054412711355343021555 0ustar hayenhayen00000000000000‰PNG  IHDRÆÇBÇ,øgAMA± üasRGB®Îé cHRMz&€„ú€èu0ê`:˜pœºQ< pHYsHHFÉk>bKGDùC»YIDATxÚíÝÛoõðïovv×ÆŽq(—%ÁÄ€Jqm!y€´i@ñ¥‘ÚªÞ<ö¡T¨U[©·'úTU«ÁVˆ–[…Z¨š‡¶`lǪJLl !NìÓo{wî;ç·;“÷2?ïþ>{Ι™ýA˜™™Ùè8Îv[ Ø  `Œ5ñö‚‡ýÔR?^dÍOkÿ]ÿÑ1À×®­6=¶Ç=Œ®Ò‘°g÷yÞÓ?JÈýÏÆÉÚ…ø_m S^nÀG–¼cŒ™ðr©TznÓ¦M$y"7Þ›#æàÁƒÛEäaÛ”9åãÅü¢`rÎK‰Bš„H  ìßǟ羋õåÉq~Ø$OÃíŸ`°“`8.¨ÿû3"r+€ï.//×fgg_ðØÀÀÀ3ƯeÓÓÓ7¿õÖ[¿‘›9ÍíE\¾÷†ZCqfDI¶Ýž(¸À½³³³oÌÌÌ<488øJ”° 7Æ™™™ùã8¯E£ÄOž)Šäh{ cþ2;;û³ýû÷—Á˜œœì}ÆóH]££QQ$ #"ßëëëûóáÇ/ˆUJMNNöº®û‚ˆÜÂišñDWB!b|·ã÷|™¢•mä ” Š»_™œœ¼khhèX`Æ‘’ëº{EN2E¬9©„¢ 1äºîÄääd%ÆðNÓ¼¡6¡(ŒŽ/¹®û _SSS7ø>§i‘PÀŠÂàxhjjjë"bÇyŒv>Q„—SD·!wç1)i¾§§§wcØWXl¾³EÔÛA‘ãæ»1®=pàÀ7üÆcÌCœÂyͱºo½L!…Êð<ïapÞ|óÍKÜÃéšcB™ÕSÆÜ:55µÅqçËì-òŽBˆ"[;\wHÿÈs… àŸ'Ê(rs®TÒ¸Ó‘ë9móŒ"ö(TP4®w\É©«ˆb­ÒN…r¥`=§¯•´¡ˆBˆ"Û¨:º8k‰"5Š‚ö§.oež§ŽB$¬ÖE‘ó³kÃáÔµÕg衈,i˜)£ciK)FP5£‡"²”RAQØãÌölEQ÷LFaQ€(ƒ(ˆ‚0rÛcè¡èh âaó­mÂuâ!d•}l¾é *­Lº`3at2‹¶ €l¾5:(œ/Eì1:#[ˆ>ŠÀs¥ôQˆxì1­–SJ(bd*L!…M„Qw (xä›Ñb¡†Bˆ‚0ˆ‚(Ø|Ɇ& 8ÀçYAÁæ›Ñ™(â6ß™£`)Åh½˜ÒC! F‘9 6ß ¢h~lA“{ m"PHŒÅô2{ FKyC Eð§¶2Š¦FP„ŸH¨T>ñ8£õrJ…ØEq4ßì1¬$ }á—3ÎzémŠ8‡\Î\˸F¦ˆõ¥ f –RçŠßùVCÁãŒE!š(pî¡ [-†6 QDq~ƒÍ·Õl¡ˆÂ„4ÀZåW ad’7Ô2…D¤+…ò‰«„0ò€BâŒ!Óò©ø}K) ¥”: I°°sFåÓêÛyƒ‘QæÈL(Åy¥÷pE÷³!Û–ˆ ÓøZž=â1 ¯-aä$öö5ʧª9†jåZ_Í0£‹dJ6{ßÞ‹ÎD‘²|Šœ¤9FÁæ» (Bk|…ò©è(Ø|ÛסƒÂD¬vžõÞ§‚£`)UQÛfù”êC¥”E z(âaù7SFPˆúiçç ö6Hˆ&Æjç,Ÿ}(0cèw2…Äîqˆ"Þc Ê eIÖ–"ŠX% ‹{¦¬£öiÏ0& {.ÔPÄ[»–(b?VØ|[¬¦4Q$\íœ("ÇÊŒa³ÑPBóÒ­D‘à¼1Â(4 E ÌVmh¢¢È ì1ÔMÔOäÓD!Qßà#ŠÄceÆ°—0ÔP0Sd‹‚0 ƒ"ê`Q$+aX•A9Að8†…ÃSGÞcEšÅ©™1ÚPNe"vÖ ŠX(£ å”] Š(î =BY_Ûƒ0¬öD‘l¾ „Âï}<ôÉVúø®s*#g\%$ç(VŽ©›€í%¾)‚¥”Õ3 *˜„Á—Àž=”‘y)ÅÃF¥‹B`À÷‘=F®bC·iZ![@MOO¦³éѽPÃ8ý²SS%'Á%dÃxü“ø–¢Í—Çÿ͉„T±þ{ÉšVq©ÿϼúê«ü¨QŽékØ;}5/{ð›"‘(¤Hð•eƒ¶¹êç$ ­A!ké"ÇàWBJ’× p›k×ÿba!®é/ad°Š’!Š< à•¬âp°k°‚½3'áyDÑÉ(Àí©ÌÃ--¥ÚË’î.¢³­¤÷•VÇäÿ¸ãK±ìuùÞûÚ%Œ¢Š½Ó'á Qt* À|tè'Ò×u4鮖ɶQZ¼_Àm‘ãkã_¬É¹ûpªÖøò½yt¹Žƒ(:EºUBˆ"ô~]¥y m|•ÒñÀ—ðº ]Œ\S…cˆ¢Q€“¨Ç ŠX÷ë*-àºþ}(;ÇWV;÷ùïº+ 9ˆ¢ãP$»p Q$º_·»€¡þ}¨”> Í÷VaŒ!ŠB!±K)¢Hu¿Ó8ÊN0ŽÏ_ä⾫+0ÆE‡ ˆwƒ(Z·;á+eUPÜpq;ë8ˆ¢ý(¢aE&ãïvahãPv_ê/\RÆ}[ª0kÎ !Šv €„•RD‘éøÏsçp]ÿ\s"°!¿á"_½ººòí ¢hŠàŒA*ãïqçê=Gpæ¸ñ’òYDÑâ ƒ(TÇßSžÃPÿÜ7]ZÁÎÁn¢h ŠæRŠ(¬Œ¿§<‡áþ ¸Î‰Ð̱} ‹(l£hj¾‰Âêø{ÊG1Ü¿¥žã–Ï–±} J6Q¬é1ˆ¢-ã_Á^VÝvYÛ¯î& åÓêçvˆ¢½ãï-ÅpÿS8*+eQ¨–Oð-¥ˆ¢mãï-ÏÕqœ ÄqûåUÜ;ÐMŠå“ ¢h÷øOgŽ’Y ì9n¿¼‚{¯ê& ¥ò) cE»Çß[>‚¡þ}(™S™ãŽ+ª¸çª.¢P(ŸB`E»ÇßWùÃý¡8îü\¶mî&ŠŒË§ðƒ(Ú>þ¾Êû‘8¶nêÂÖM]D‘aùÜcEÇŒ?Žm›»q÷¦n¢È¨|Zãô¥°ˆ¢óÆßWyÃýpp2°!߶¹kQ´\>­3ð‡r~÷Qtðø?>u ^ŸAMʾo¢°°è5®÷‡T•i(I‘Ñ’M}cÚ-¹Ì?þ¾Ê{ÞðÞ8æÃà‚n®—e8D‘ñ¯¯þ·~œc‰³ÖB¸D‘Ÿñ¯¯¬àx}nW`YÅÈ&Ìëÿø½TÜ) ¹$.‘x¿´6–4ã ž£—ö¼ÞÊœ*ê÷Žß€Ù…mœ½šãÃãóUÈ06tF/æT3]Ø ŒÌz Föa©ücèe ®vžO|ߘ1òDmGC-c|ZsP“ºÙLþ&Ik½z Ç[í?ùLs °®bˆ¢¨0^<Ú‡÷ƒwýIÙgÛËÙCïMWj8MÀç¿r ùâgŸ¿á´††ç\}¿ú0Cžßï«¡k·z¿³À†nÞX%Š¢Âý´% _§Ÿ?´ÎW>ΣM=Q„£ˆ®ß´>2¬Ã Š(‚p´ãˆ{ ñ<¢H…"¬Î×GÁC»”"ŠT(âíÒþæ C±”"Šä™"b‚E`EQˆzÑø&EÜ¿9¨Î·ƒ‚=†½RD‘E¼`¦È9 ¢H†Bâï®%Šü6ßD‘Eºƒ|¡`eaw-Qïù&ŠT($æI„ (x¡µÝµD‘E²™)òWJE*‘óÔÚZ¸ µæ›(Ò ¢(tA)Q˜ˆ:_{ +¥Q$ÍíDÁÌa!clé]ÄÅU§éE— %[üvW6^o- A•€óˆšç˜ø¬1&>CYý;sæ“üè§À켧ŒÂE¡a¬;™ãáߟߘó0;¯)bwßÙ£  }E¬U¥~-gM"&ú‹Jj(xÃÂ^©F¢c3iP´3Sad CEœoñat¤ ¢`°Çh˜RŠ(ê¥Zà¹RÊ(x®3Fút¡Š"äûV/yÌ ŒT6,£œÀDATL©¢6¢ ÂHËBEº¬Al¾Ûž'ê(D¼çKe‚Í73FëY#óL•1Qp×-at2 ! ÂÈkó­‹¢•ƒ|DÁ£¨($â‹JÊ(Øc0c¤¤¡‰"b f ÂèÐ|¡BÚ…‚H£eš(~Q‰(Øc´ÝDý8†. ²àš.Šèc( fŒ C9S$^ñ<³LA„‘A§¡‡BÚ‡‚6£%DÁ »(¤­(¨ƒÍwêæ[…„-ºf›ofŒpÙ] ’™‚0 „"Ñ:DAÁ i aÑ*M§Î×CÁƒ£z(’ä-f Âè¤.CEº Tf„‚Ù‚0Zm4ÔPH¢A{Œ0á‰o1\‰#ƒRJE¬ÅÔ2QFúÎ[1SŹ£Vdz(²:òMÚcÐW(¢BŽcˆ'VP°ÇÐB€#ͪ("K)5a!>tÌEúË)Ç Qä.fù{qû ¢`$cÌkŽˆ¼TLº(Òù&Šíwcà÷1ÆÆÆž/•J7øc®÷JYË`¦ÈO¼X«Õn{:ènØ£wíÚõ;Ÿxâ‰íƘGÜ’«¼-(@ÿ2Æ<2::º/êŽnœgÀs{÷îý¢çyß4ÆìpQEâ}c̳~;22òRýLÈ0i·¶gÏž×u‡lPé¤Wâíù¥ §, œýñ‚ñ4Ýäó›ÆèrK'îÙÒûÏÆß_³áÕáuÕ…þ8Ï‘6>Yꟛž»m’sÞ?<Ï[2Ƽ#"SãããÓižãÿ†ö`Âþz%tEXtdate:create2013-08-08T10:34:23+02:00íq‚#%tEXtdate:modify2013-08-08T10:34:23+02:00œ,:ŸIEND®B`‚Nuitka-0.5.28.2/Developer_Manual.rst0000644000372000001440000034766713207537242017473 0ustar hayenusers00000000000000Nuitka Developer Manual ~~~~~~~~~~~~~~~~~~~~~~~ .. image:: doc/images/Nuitka-Logo-Symbol.png .. contents:: .. raw:: pdf PageBreak oneColumn SetPageCounter 1 The purpose of this developer manual is to present the current design of Nuitka, the project rules, and the motivations for choices made. It is intended to be a guide to the source code, and to give explanations that don't fit into the source code in comments form. It should be used as a reference for the process of planning and documenting decisions we made. Therefore we are e.g. presenting here the type inference plans before implementing them. And we update them as we proceed. It grows out of discussions and presentations made at conferences as well as private conversations or discussions on the mailing list or bug tracker. Milestones ========== 1. Feature parity with CPython, understand all the language construct and behave absolutely compatible. Feature parity has been reached for CPython 2.6 and 2.7. We do not target any older CPython release. For CPython 3.2 up to 3.6 it also has been reached. We do not target the older and practically unused CPython 3.1 and 3.0 releases. This milestone was reached. Dropping support for Python 2.6 and 3.2 is an option, should this prove to be any benefit. Currently it is not, as it extends the test coverage only. 2. Create the most efficient native code from this. This means to be fast with the basic Python object handling. This milestone was reached, although of course, micro optimizations to this are happening all the time. 3. Then do constant propagation, determine as many values and useful constraints as possible at compile time and create more efficient code. This milestone is considered almost reached. We continue to discover new things, but the infrastructure is there, and these are easy to add. 4. Type inference, detect and special case the handling of strings, integers, lists in the program. This milestone is considered in progress. 5. Add interfacing to C code, so Nuitka can turn a ``ctypes`` binding into an efficient binding as written with C. This milestone is planned only. 6. Add hints module with a useful Python implementation that the compiler can use to learn about types from the programmer. This milestone is planned only. Version Numbers =============== For Nuitka we use a defensive version numbering system to indicate that it is not yet ready for everything. We have defined milestones and the version numbers should express which of these, we consider done. - So far: Before milestone 1, we used ``0.1.x`` version numbers. After reaching it, we used ``0.2.x`` version numbers. Before milestone 2 and 3, we used ``0.3.x`` version numbers. After almost reaching 3, and beginning with 4, we use "0.4.x" version numbers. Due to an interface change, "0.5.x" version numbers are being used. - Future: When we start to have sufficient amount of type inference in a stable release, that will be ``0.6.x`` version numbers. With ``ctypes`` bindings in a usable state it will be ``0.7.x``. - Final: We will then round it up and call it Nuitka ``1.0`` when this works as expected for a bunch of people. The plan is to reach this goal during 2018. This is based on positive assumptions that may not hold up though. Of course, all of this may be subject to change. Current State ============= Nuitka top level works like this: - ``nuitka.tree.Building`` outputs node tree - ``nuitka.optimization`` enhances it as best as it can - ``nuitka.finalization`` prepares the tree for code generation - ``nuitka.codegen.CodeGeneration`` orchestrates the creation of code snippets - ``nuitka.codegen.*Codes`` knows how specific code kinds are created - ``nuitka.MainControl`` keeps it all together This design is intended to last. Regarding Types, the state is: - Types are always ``PyObject *``, implicitly. - There are a few more specific use of types beyond "compile time constant", that are encoded in type and value shapes, which can be used to predict some operations, conditions, etc. - Every code generation, every operation is expected to have ``PyObject *`` as result, if it is not a constant, then we know nothing about it. For some interfaces, e.g. iteration, there are initial attempts at abstracting it. The limitation to only do ``PyObject *`` will soon go away. Coding Rules ============ These rules should generally be adhered when working on Nuitka code. It's not library code and it's optimized for readability, and avoids all performance optimization for itself. Tool to format -------------- There is a tool ``bin/autoformat-nuitka-source`` which is to apply automatic formatting to code as much as possible. Line Length ----------- No more than 120 characters. Screens are wider these days, but most of the code aims at keeping the lines below 80. Long lines are also a sign of writing incomprehensible code. Indentation ----------- No tabs, 4 spaces, no trailing white space. Files end in a new line. Identifiers ----------- Classes are camel case with leading upper case. Methods are with leading verb in lower case, but also camel case. Around braces there are no spaces, but after comma, there is spaces for better readability. Variables and arguments are lower case with ``_`` as a separator. .. code-block:: python class SomeClass: def doSomething(some_parameter): some_var = ("foo", "bar") Base classes that are abstract have their name end with ``Base``, so that a meta class can use that convention, and readers immediately know. Function calls use keyword argument preferably. These are slower in CPython, but more readable: .. code-block:: python return Generator.getSequenceCreationCode( sequence_kind = sequence_kind, element_identifiers = identifiers, context = context ) The ``=`` are all aligned to the longest parameter names with an extra space around it. When the names don't add much value, sequential calls can be done, but ideally with one value per line: .. code-block:: python context.setLoopContinueTarget( handler_start_target, continue_name ) Here, ``setLoopContinueTarget`` will be so well known that the reader is expected to know the argument names and their meaning, but it would be still better to add them. Contractions should span across multiple lines for increased readability: .. code-block:: python result = [ "PyObject *decorator_%d" % (d + 1) for d in range(decorator_count) ] Module/Package Names -------------------- Normal modules are named in camel case with leading upper case, because their of role as singleton classes. The difference between a module and a class is small enough and in the source code they are also used similarly. For the packages, no real code is allowed in their ``__init__.py`` and they must be lower case, like e.g. ``nuitka`` or ``codegen``. This is to distinguish them from the modules. Packages shall only be used to group things. In ``nuitka.codegen`` the code generation packages are located, while the main interface is ``nuitka.codegen.CodeGeneration`` and may then use most of the entries as local imports. There is no code in packages themselves. Names of modules should be plurals if they contain classes. Example is that a ``Nodes`` module that contain a ``Node`` class. Prefer list contractions over built-ins --------------------------------------- This concerns ``map``, ``filter``, and ``apply``. Usage of these built-ins is highly discouraged within Nuitka source code. Using them is considered worth a warning by "PyLint" e.g. "Used built-in function 'map'". We should use list contractions instead, because they are more readable. List contractions are a generalization for all of them. We love readability and with Nuitka as a compiler, there won't be any performance difference at all. There are cases where a list contraction is faster because you can avoid to make a function call. And there may be cases, where map is faster, if a function must be called. These calls can be very expensive in CPython, and if you introduce a function, just for ``map``, then it might be slower. But of course, Nuitka is the project to free us from what is faster and to allow us to use what is more readable, so whatever is faster, we don't care. We make all options equally fast and let people choose. For Nuitka the choice is list contractions as these are more easily changed and readable. Look at this code examples from Python: .. code-block:: python class A: def getX(self): return 1 x = property(getX) class B(A): def getX(self): return 2 A().x == 1 # True B().x == 1 # True (!) This pretty much is what makes properties bad. One would hope ``B().x`` to be ``2``, but instead it's not changed. Because of the way properties take the functions and not members, and because they then are not part of the class, they cannot be overloaded without re-declaring them. Overloading is then not at all obvious anymore. Now imagine having a setter and only overloading the getter. How to update the property easily? So, that's not likeable about them. And then we are also for clarity in these internal APIs too. Properties try and hide the fact that code needs to run and may do things. So lets not use them. For an external API you may exactly want to hide things, but internally that has no use, and in Nuitka, every API is internal API. One exception may be the ``hints`` module, which will gladly use such tricks for an easier write syntax. The "git flow" model ==================== * The flow is used for releases and occasionally subsequent hot fixes. A few feature branches were used so far. It allows for quick delivery of fixes to both the stable and the development version, supported by a git plug-in, that can be installed via "apt-get install git-flow". * Stable (master branch) The stable version, is expected to pass all the tests at all times and is fully supported. As soon as bugs are discovered, they are fixed as hot fixes, and then merged to develop by the "git flow" automatically. * Development (develop branch) The future release, supposedly in almost ready for release state at nearly all times, but this is as strict. It is not officially supported, and may have problems and at times inconsistencies. Normally this branch is supposed to not be rebased. For severe problems it may be done though. * Factory (default feature branch) Code under construction. We publish commits there, that may not hold up in testing, and before it enters develop branch. Factory may have severe regressions frequently, and commits become **rebased all the time**, so do not base your patches on it, please prefer the ``develop`` branch for that, unless of course, it's about factory code itself. * Feature Branches We are not currently using these. They could be used for long lived changes that extend for multiple release cycles and are not ready yet. Currently we perform all changes in steps that can be included in releases or delay making those changes. Checking the Source =================== The static checking for errors is currently done with "PyLint". In the future, Nuitka itself will gain the ability to present its findings in a similar way, but this is not a priority, and we are not there yet. So, we currently use "PyLint" with options defined in a script. .. code-block:: sh ./bin/check-nuitka-with-pylint Ideally the above command gives no warnings. This is currently the case. If you submit a patch, it would be good if you checked that it doesn't introduce new warnings, but that is not strictly required. it will happen before release, and that is considered enough. You probably are already aware of the beneficial effects. Running the Tests ================= This section describes how to run Nuitka tests. Running all Tests ----------------- The top level access to the tests is as simple as this: .. code-block:: sh ./tests/run-tests For fine grained control, it has the following options:: --skip-basic-tests The basic tests, execute these to check if Nuitka is healthy. Default is True. --skip-syntax-tests The syntax tests, execute these to check if Nuitka handles Syntax errors fine. Default is True. --skip-program-tests The programs tests, execute these to check if Nuitka handles programs, e.g. import recursions, etc. fine. Default is True. --skip-package-tests The packages tests, execute these to check if Nuitka handles packages, e.g. import recursions, etc. fine. Default is True. --skip-optimizations-tests The optimization tests, execute these to check if Nuitka does optimize certain constructs fully away. Default is True. --skip-standalone-tests The standalone tests, execute these to check if Nuitka standalone mode, e.g. not referring to outside, important 3rd library packages like PyQt fine. Default is True. --skip-reflection-test The reflection test compiles Nuitka with Nuitka, and then Nuitka with the compile Nuitka and compares the outputs. Default is True. --skip-cpython26-tests The standard CPython2.6 test suite. Execute this for all corner cases to be covered. With Python 2.7 this covers exception behavior quite well. Default is True. --skip-cpython27-tests The standard CPython2.7 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is True. --skip-cpython32-tests The standard CPython3.2 test suite. Execute this for all corner cases to be covered. With Python 2.6 these are not run. Default is True. --skip-cpython33-tests The standard CPython3.3 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is True. --skip-cpython34-tests The standard CPython3.4 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is True. --skip-cpython35-tests The standard CPython3.5 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is True. --skip-cpython36-tests The standard CPython3.6 test suite. Execute this for all corner cases to be covered. With Python 2.x these are not run. Default is True. --no-python2.6 Do not use Python 2.6 even if available on the system. Default is False. --no-python2.7 Do not use Python 2.7 even if available on the system. Default is False. --no-python3.2 Do not use Python 3.2 even if available on the system. Default is False. --no-python3.3 Do not use Python 3.3 even if available on the system. Default is False. --no-python3.4 Do not use Python 3.4 even if available on the system. Default is False. --no-python3.5 Do not use Python 3.5 even if available on the system. Default is False. --no-python3.6 Do not use Python 3.6 even if available on the system. Default is False. --coverage Make a coverage analysis, that does not really check. Default is False. You will only run the CPython test suites, if you have the submodules of the Nuitka git repository checked out. Otherwise, these will be skipped with a warning that they are not available. The policy is generally, that ``./test/run-tests`` running and passing all the tests on Linux and Windows shall be considered sufficient for a release, but of course, depending on changes going on, that might have to be expanded. Basic Tests ----------- You can run the "basic" tests like this: .. code-block:: sh ./tests/basics/run_all.py search These tests normally give sufficient coverage to assume that a change is correct, if these "basic" tests pass. The most important constructs and built-ins are excercised. To control the Python version used for testing, you can set the ``PYTHON`` environment variable to e.g. ``python3.2`` (can also be full path), or simply execute the ``run_all.py`` script directly with the intended version, as it is portable across all supported Python versions, and defaults testing with the Python version is run with. Syntax Tests ------------ Then there are "syntax" tests, i.e. language constructs that need to give a syntax error. It sometimes so happens that Nuitka must do this itself, because the ``ast.parse`` doesn't see the problem and raises no ``SyntaxError`` of its own. These cases are then covered by tests to make sure they work as expected. Using the ``global`` statement on a function argument is an example of this. These tests make sure that the errors of Nuitka and CPython are totally the same for this: .. code-block:: sh ./tests/syntax/run_all.py search Program Tests ------------- Then there are small "programs" tests, that e.g. exercise many kinds of import tricks and are designed to reveal problems with inter-module behavior. These can be run like this: .. code-block:: sh ./tests/programs/run_all.py search Compile Nuitka with Nuitka -------------------------- And there is the "compile itself" or "reflected" test. This test makes Nuitka compile itself and compare the resulting C++ when running compiled to non-compiled, which helps to find in-determinism. The test compiles every module of Nuitka into an extension module and all of Nuitka into a single binary. That test case also gives good coverage of the ``import`` mechanisms, because Nuitka uses a lot of packages and imports between them. .. code-block:: sh ./tests/reflected/compile_itself.py Design Descriptions =================== These should be a lot more and contain graphics from presentations given. It will be filled in, but not now. Nuitka Logo ----------- The logo was submitted by "dr. Equivalent". It's source is contained in ``doc/Logo`` where 3 variants of the logo in SVG are placed. * Symbol only (symbol) .. image:: doc/images/Nuitka-Logo-Symbol.png * Text next to symbol (horizontal) .. image:: doc/images/Nuitka-Logo-Horizontal.png * Text beneath symbol (vertical) .. image:: doc/images/Nuitka-Logo-Vertical.png From these logos, PNG images, and "favicons", and are derived. The exact ImageMagick commands are in ``nuitka/tools/release/Documentation``, but are not executed each time, the commands are also replicated here: .. code-block:: sh convert -background none doc/Logo/Nuitka-Logo-Symbol.svg doc/images/Nuitka-Logo-Symbol.png convert -background none doc/Logo/Nuitka-Logo-Vertical.svg doc/images/Nuitka-Logo-Vertical.png convert -background none doc/Logo/Nuitka-Logo-Horizontal.svg doc/images/Nuitka-Logo-Horizontal.png optipng -o2 doc/images/Nuitka-Logo-Symbol.png optipng -o2 doc/images/Nuitka-Logo-Vertical.png optipng -o2 doc/images/Nuitka-Logo-Horizontal.png Choice of the Target Language ----------------------------- * Choosing the target language, is an important decision * The portability of Nuitka is decided here * Other factors: * How difficult is it to generate the code? * Does the Python C-API have bindings? * Is that language known? * Does the language aid to find bugs? * These candidates were considered * C++03, C++11, C11, C89, Ada .. table:: Requirement to Language matrix: ===================== ===== ======== ====== ========= ========= Requirement\\Language C89 C11 C++03 C++11 Ada ===================== ===== ======== ====== ========= ========= Portable Yes Yes [5]_ Yes No [1]_ Yes --------------------- ----- -------- ------ --------- --------- Knowledge Yes Yes Yes No [2]_ Yes --------------------- ----- -------- ------ --------- --------- Python C-API Yes Yes Yes Yes No [3]_ --------------------- ----- -------- ------ --------- --------- Runtime checks No No No No Yes [4]_ --------------------- ----- -------- ------ --------- --------- Code Generation Tough Medium Hard Easy Harder ===================== ===== ======== ====== ========= ========= _`1`:: C++11 is not fully supported by all compilers. _`2`:: Not a whole lot of people have C++11 knowledge. My *only* C++11 code was that in Nuitka. _`3`:: The Python C-API for Ada would have to be created by us, possible just big project by itself. _`4`:: Run time checks exist only for Ada in that quality. I miss automatic ``CONSTRAINT_ERROR`` exceptions, for data structures with validity indicators, where in other languages, I need to check myself. _`5`:: One can use a C++03 compiler as a C11 compiler for the largest part, e.g. with MSVC. The *decision for C11* is ultimately: * for portability * for language knowledge * for control over created code. All of these are important advantages. For C++11 initially spoke easy code generation: * variadic templates * raw strings Yet, as it turns out, variadic templates do not help at all with evaluation order, so that code that used it, needed to be changed to generating instances of their code. And raw strings turned out to be not as perfect as one wants to be, and solving the problem with C++03 is feasible too, even if not pretty. For C++03 initially spoke less explicit code generation: * Destructors can ensure cleanups happen * Local objects could e.g. repair the stack frames For Ada would have spoken the time savings through run time checks, which would have shortened some debugging sessions quite some. But building the Python C-API bindings on our own, and potentially incorrectly, would have eaten that up. Later, it was found that using C++ for exceptions is tremendously inefficient, and must be avoided. In order to do this, a more C style code generation is needed, where even less things are done with C++, e.g. the cleanup of temporary variables inside a statement will be done manually instead. The current status is Pure C11. All code compiles as C11, and also in terms of workaround to missing compiler support as C++03. Naturally we are not using any C++ features, just the allowances of C++ features that made it into C11. Use of Scons internally ----------------------- Nuitka does not involve Scons in its user interface at all; Scons is purely used internally. Nuitka itself, being pure Python, will run without any build process just fine. Nuitka simply prepares ``.build`` folders with lots of files and tasks scons to execute the final build, after which Nuitka again will take control and do more work as necessary. .. note:: When we speak of "standalone" mode, this is handled outside of Scons, and after it, creating the ".dist" folder. This is done in ``nuitka.MainControl`` module. For interfacing to Scons, there is the module ``nuitka.build.SconsInterface`` that will support calling ``scons`` - potentially from one of two inline copies (one for before / one for Python 3.5 or later). These are mainly used on Windows or when using source releases - and passing arguments to it. These arguments are passed as ``key=value``, and decoded in the scons file of Nuitka. The scons file is named ``SingleExe.scons`` for lack of better name. It's really wrong now, but we have yet to find a better name. It once expressed the intention to be used to create executables, but the same works for modules too, as in terms of building, and to Scons, things really are the same. The scons file supports operation in multiple modes for many things, and modules is just one of them. It runs outside of Nuitka process scope, even with a different Python version potentially, so all the information must be passed on the command line. What follows is the (lengthy) list of arguments that the scons file processes: * ``source_dir`` Where is the generated C source code. Scons will just compile everything it finds there. No list of files is passed, but instead this directory is being scanned. * ``nuitka_src`` Where do the include files and static C parts of Nuitka live. These provide e.g. the implementation of compiled function, generators, and other helper codes, this will point to where ``nuitka.build`` package lives normally. * ``module_mode`` Build a module instead of a program. * ``result_base`` This is not a full name, merely the basename for the result to be produced, but with path included, and the suffix comes from module or executable mode. * ``debug_mode`` Enable debug mode, which is a mode, where Nuitka tries to help identify errors in itself, and will generate less optimal code. This also asks for warnings, and makes the build fail if there are any. Scons will pass different compiler options in this case. * ``python_debug`` Compile and link against Python debug mode, which does assertions and extra checks, to identify errors, mostly related to reference counting. May make the build fail, if no debug build library of CPython is available. On Windows it is possible to install it for CPython3.5 or higher. * ``full_compat_mode`` Full compatibility, even where it's stupid, i.e. do not provide information, even if available, in order to assert maximum compatibility. Intended to control level of compatibility to absurd. * ``experimental_mode`` Do things that are not yet accepted to be safe. * ``lto_mode`` Make use of link time optimization of gcc compiler if available and known good with the compiler in question. So far, this was not found to make major differences. * ``win_disable_console`` Windows subsystem mode: Disable console for windows builds. * ``unstriped_mode`` Unstriped mode: Do not remove debug symbols. * ``clang_mode`` Clang compiler mode, default on MacOS X and FreeBSD, optional on Linux. * ``mingw_mode`` MinGW compiler mode, optional and useful on Windows only. * ``standalone_mode`` Building a standalone distribution for the binary. * ``show_scons`` Show scons mode, output information about Scons operation. This will e.g. also output the actual compiler used, output from compilation process, and generally debug information relating to be build process. * ``python_prefix`` Home of Python to be compiled against, used to locate headers and libraries. * ``target_arch`` Target architecture to build. Only meaningful on Windows. * ``python_version`` The major version of Python built against. * ``abiflags`` The flags needed for the Python ABI chosen. Might be necessary to find the folders for Python installations on some systems. * ``icon_path`` The icon to use for Windows programs if given. * ``nuitka_cache`` The cache directory to use. We put e.g. creating linker libraries for MinGW64 there. Locating Modules and Packages ----------------------------- The search for of modules used is driven by ``nuitka.importing.Importing`` module. * Quoting the ``nuitka.importing.Importing`` documentation: Locating modules and package source on disk. The actual import of a module would already execute code that changes things. Imagine a module that does ``os.system()``, it would be done during compilation. People often connect to databases, and these kind of things, at import time. Therefore CPython exhibits the interfaces in an ``imp`` module in standard library, which one can use those to know ahead of time, what file import would load. For us unfortunately there is nothing in CPython that is easily accessible and gives us this functionality for packages and search paths exactly like CPython does, so we implement here a multi step search process that is compatible. This approach is much safer of course and there is no loss. To determine if it's from the standard library, one can abuse the attribute ``__file__`` of the ``os`` module like it's done in ``isStandardLibraryPath`` of this module. End quoting the ``nuitka.importing.Importing`` documentation. * Role This module serves the recursion into modules and analysis if a module is a known one. It will give warnings for modules attempted to be located, but not found. These warnings are controlled by a while list inside the module. The decision making and caching are located in the ``nuitka.tree`` package, in modules ``nuitka.tree.Recursion`` and ``nuitka.tree.ImportCache``. Each module is only considered once (then cached), and we need to obey lots of user choices, e.g. to compile standard library or not. Hooking for module ``import`` process ------------------------------------- Currently, in generated code, for every ``import`` a normal ``__import__()`` built-in call is executed. The ``nuitka/build/static_src/ModuleUnfreezer.c`` file provides the implementation of a ``sys.meta_path`` hook. This meta path based importer allows us to have the Nuitka provided module imported even when imported by non-compiled code. .. note:: Of course it would make sense to compile time detect which module it is that is being imported and then to make it directly. At this time, we don't have this inter-module optimization yet, mid-term it should become easy to add. Supporting ``__class__`` of Python3 ----------------------------------- In Python3 the handling of ``__class__`` and ``super`` is different from Python2. It used to be a normal variable, and now the following things have changed. * The use of the ``super`` variable name triggers the addition of a closure variable ``__class__``, as can be witnessed by the following code: .. code-block:: python class X: def f1(self): print( locals() ) def f2(self): print( locals() ) super # Just using the name, not even calling it. x = X() x.f1() x.f2() .. code-block:: python {'self': <__main__.X object at 0x7f1773762390>} {'self': <__main__.X object at 0x7f1773762390>, '__class__': } * This value of ``__class__`` is also available in the child functions. * The parser marks up code objects usage of "super". It doesn't have to be a call, it can also be a local variable. If the ``super`` built-in is assigned to another name and that is used without arguments, it won't work unless ``__class__`` is taken as a closure variable. * As can be seen in the CPython3.2 code, the closure value is added after the class creation is performed. * It appears, that only functions locally defined to the class are affected and take the closure. This left Nuitka with the strange problem, of how to emulate that. The solution is this: * Under Python3, usage of ``__class__`` as a reference in a child function body is mandatory. It remains that way until all variable names have been resolved. * When recognizing calls to ``super`` without arguments, make the arguments into variable reference to ``__class__`` and potentially ``self`` (actually first argument name). * After all variables have been known, and no suspicious unresolved calls to anything named ``super`` are down, then unused references are optimized away by the normal unused closure variable. * Class dictionary definitions are added. These are special direct function calls, ready to propagate also "bases" and "metaclass" values, which need to be calculated outside. The function bodies used for classes will automatically store ``__class__`` as a shared local variable, if anything uses it. And if it's not assigned by user code, it doesn't show up in the "locals()" used for dictionary creation. Existing ``__class__`` local variable values are in fact provided as closure, and overridden with the built class , but they should be used for the closure giving, before the class is finished. So ``__class__`` will be local variable of the class body, until the class is built, then it will be the ``__class__`` itself. Frame Stack ----------- In Python, every function, class, and module has a frame. It creates created when the scope it entered, and there is a stack of these at run time, which becomes visible in tracebacks in case of exceptions. The choice of Nuitka is to make this an explicit element of the node tree, that are as such subject to optimization. In cases, where they are not needed, they may be removed. Consider the following code. .. code-block:: python def f(): if someNotRaisingCall(): return somePotentiallyRaisingCall() else: return None In this example, the frame is not needed for all the code, because the condition checked wouldn't possibly raise at all. The idea is the make the frame guard explicit and then to reduce its scope whenever possible. So we start out with code like this one: .. code-block:: python def f(): with frame_guard( "f" ): if someNotRaisingCall(): return somePotentiallyRaisingCall() else: return None This is to be optimized into: .. code-block:: python def f(): if someNotRaisingCall(): with frame_guard( "f" ): return somePotentiallyRaisingCall() else: return None Notice how the frame guard taking is limited and may be avoided, or in best cases, it might be removed completely. Also this will play a role when in-lining function. The frame stack entry will then be automatically preserved without extra care. .. note:: In the actual code, ``nuitka.nodes.FrameNodes.StatementsFrame`` is represents this as a set of statements to be guarded by a frame presence. Parameter Parsing ----------------- The parsing of parameters is very convoluted in Python, and doing it in an compatible way is not that easy. This is a description of the required process, for easier overview. Input +++++ The input is an argument ``tuple`` (the type is fixed), which contains the positional arguments, and potentially an argument ``dict`` (type is fixed as well, but could also be ``NULL``, indicating that there are no keyword arguments. Keyword dictionary ++++++++++++++++++ The keyword argument dictionary is checked first. Anything in there, that cannot be associated, either raises an error, or is added to a potentially given star dict argument. So there are two major cases. * No star dict argument: Iterate over dictionary, and assign or raise errors. This check covers extra arguments given. * With star dict argument: Iterate over dictionary, and assign or raise errors. Interesting case for optimization are no positional arguments, then no check is needed, and the keyword argument dictionary could be used as the star argument. Should it change, a copy is needed though. What's noteworthy here, is that in comparison of the keywords, we can hope that they are the same value as we use. The interning of strings increases chances for non-compiled code to do that, esp. for short names. We then can do a simple ``is`` comparison and only fall back to real string `==` comparisons, after all of these failed. That means more code, but also a lot faster code in the positive case. Argument tuple ++++++++++++++ After this completed, the argument tuple is up for processing. The first thing it needs to do is to check if it's too many of them, and then to complain. For arguments in Python2, there is the possibility of them being nested, in which case they cannot be provided in the keyword dictionary, and merely should get picked from the argument tuple. Otherwise, the length of the argument tuple should be checked against its position and if possible, values should be taken from there. If it's already set (from the keyword dictionary), raise an error instead. SSA form for Nuitka ------------------- The SSA form is critical to how optimization works. The so called trace collections builds up traces. These are facts about how this works: * Assignments draw from a counter unqiue for the variable, which becomes the variable version. This happens during tree building phase. * References are associated to the version of the variable active. This can be a merge of branches. Trace collection does do that and provides nodes with the currently active trace for a variable. The data structures used for trace collection need to be relatively compact as the trace information can become easily much more data than the program itself. Every trace collection has these: * variable_actives Dictionary, where per "variable" the currently used version is. Used to track situations changes in branches. This is the main input for merge process. * variable_traces Dictionary, where "variable" and "version" form the key. The values are objects with or without an assignment, and a list of usages, which starts out empty. These objects have usages appended to them. In "onVariableSet", a new version is allocated, which gives a new object for the dictionary, with an empty usages list, because each write starts a new version. In "onVariableUsage" the version is detected from the current version. It may be not set yet, which means, it's a read of an undefined value (local variable, not a parameter name), or unknown in case of global variable. These objects may be told that their value has escaped. This should influence the value friend they attached to the initial assignment. Each usage may have a current value friend state that is different. When merging branches of conditional statements, the merge shall apply as follows: * Branches have their own collection, with deviating sets of "variable_actives". These are children of an outer collections * Case a) One branch only. For that branch a collection is performed. As usual new assignments generate a new version making it "active", references then related to these "active" versions. Then, when the branch is merged, for all "active" variables, it is considered, if that is a change related to before the branch. If it's not the same, a merge trace with the branch condition is created with the one active in the collection before that statement. * Case b) Two branches. When there are two branches, they both as are treated as above, except for the merge. When merging, a difference in active variables between the two branches creates the merge trace. .. note:: For conditional expressions, there are always only two branches. Even if you think you have more than one branch, you do not. It's always nested branches, already when it comes out of the parser. Trace structure, there are different kinds of traces. * Initial write of the version There may be a initial write for each version. It can only occur at the start of the scope, but not later, and there is only one. This might be known to be "initialized" (parameter variables of functions are like that) or "uninitialized", or "unknown". * Merge of other one or two other versions This combines two or more previous versions. In cases of loop exits or entries, there are multiple branches to combine potentially. These branches can have vastly different properties. * Becoming unknown. When control flow escapes, e.g. for a module variable, any write can occur to it, and it's value cannot be trusted to be unchanged. These are then traced as unknown. All traces have a base class ``VariableTraceBase`` which provides the interface to query facts about the state of a variable in that trace. It's e.g. of some interest, if a variable must have a value or must not. This allows to e.g. omit checks, know what exceptions might raise. Code Generation towards C ------------------------- Currently, Nuitka uses Pure C and no C++ patterns at all. The use of C11 requires on some platforms to compile the C11 using a C++ compiler, which works relatively well, but also limits the amount of C11 that can be used. Exceptions ++++++++++ To handle and work with exceptions, every construct that can raise has either a ``bool`` or ``int`` return code or ``PyObject *`` with ``NULL`` return value. This is very much in line with that the Python C-API does. Every helper function that contains code that might raise needs these variables. After a failed call, our variant of ``PyErr_Fetch`` called ``FETCH_ERROR_OCCURRED`` must be used to catch the defined error, unless some quick exception cases apply. The quick exception means, ``NULL`` return from C-API without a set exception means means e.g. ``StopIteration``. As an optimization, functions that raise exceptions, but are known not to do so, for whatever reason, could only be asserted to not do so. Statement Temporary Variables +++++++++++++++++++++++++++++ For statements and larger constructs the context object track temporary values, that represent references. For some, these should be released at the end of the statement, or they represent a leak. The larger scope temporary variables, are tracked in the function or module context, where they are supposed to have explicit ``del`` to release their references. Local Variables Storage +++++++++++++++++++++++ Closure variables taken are to be released when the function object is later destroyed. For in-lined calls, variables are just passed, and it does not become an issue to release anything. For function exit, owned variables, local or shared to other functions, must be released. This cannot be a ``del`` operation, as it also involves setting a value, which would be wrong for shared variables (and wasteful to local variables, as that would be its last usage). Therefore we need a special operation that simply releases the reference to the cell or object variable. Exit Targets ++++++++++++ Each error or other exit releases statement temporary values and then executes a ``goto`` to the exit target. These targets need to be setup. The ``try``/``except`` will e.g. catch error exits. Other exits are ``continue``, ``break``, and ``return`` exits. They all work alike. Generally, the exits stack of with constructs that need to register themselves for some exit types. A loop e.g. registers the ``continue`` exit, and a contained ``try``/``finally`` too, so it can execute the final code should it be needed. Frames ++++++ Frames are containers for variable declarations and cleanups. As such, frames provide error exits and success exits, which remove the frame from the frame stack, and then proceed to the parent exit. With the use of non ``PyObject **`` C types, but frame exception exits, the need to convert those types becomes apparent. Exceptions should still resolve the C version. When using different C types at frame exception exits, there is a need to trace the active type, so it can be used in the correct form. Abortive Statements +++++++++++++++++++ The way ``try``/``finally`` is handled, copies of the ``finally`` block are made, and optimized independently for each abort method. The ones there are of course, ``return``, ``continue``, and ``break``, but also implicit and explicit ``raise`` of an exception. Code trailing an abortive statement can be discarded, and the control flow will follow these "exits". Constant Preparation -------------------- Early versions of Nuitka, created all constants for the whole program for ready access to generated code, before the program launches. It did so in a single file, but that approach didn't scale well. Problems were * Even unused code contributed to start-up time, this can become a lot for large programs, especially in standalone mode. * The massive amount of constant creation codes gave backend C compilers a much harder time than necessary to analyse it all at once. The current approach is as follows. Code generation detects constants used in only one module, and declared ``static`` there, if the module is the only user, or ``extern`` if it is not. Some values or forced to be global, as they are used pre-main or in helpers. These ``extern`` values are globally created before anything is used. The ``static`` values are created when the module is loaded, i.e. something did import it. We trace used constants per module, and for nested ones, we also associate them. The global constants code is special in that it can only use ``static`` for nested values it exclusively uses, and has to export values that others use. Language Conversions to make things simpler ------------------------------------------- There are some cases, where the Python language has things that can in fact be expressed in a simpler or more general way, and where we choose to do that at either tree building or optimization time. The ``assert`` statement ++++++++++++++++++++++++ The ``assert`` statement is a special statement in Python, allowed by the syntax. It has two forms, with and without a second argument. The later is probably less known, as is the fact that raise statements can have multiple arguments too. The handling in Nuitka is: .. code-block:: python assert value # Absolutely the same as: if not value: raise AssertionError .. code-block:: python assert value, raise_arg # Absolutely the same as: if not value: raise AssertionError, raise_arg This makes assertions absolutely the same as a raise exception in a conditional statement. This transformation is performed at tree building already, so Nuitka never knows about ``assert`` as an element and standard optimizations apply. If e.g. the truth value of the assertion can be predicted, the conditional statement will have the branch statically executed or removed. The "comparison chain" expressions ++++++++++++++++++++++++++++++++++ In Nuitka we have the concept of an outline, and therefore we can make the following re-formulation instead: .. code-block:: python a < b() > c < d def _comparison_chain(): # So called "outline" function tmp_a = a tmp_b = b() tmp = tmp_a < tmp_b if not tmp: return tmp del tmp_a tmp_c = c tmp = tmp_b > tmp_c if not tmp: return tmp del tmp_b return tmp_c < d _comparison_chain() This transformation is performed at tree building already. The temporary variables keep the value for the use of the same expression. Only the last expression needs no temporary variable to keep it. What we got from this, is making the checks of the comparison chain explicit and comparisons in Nuitka to be internally always about two operands only. The ``execfile`` built-in +++++++++++++++++++++++++ Handling is: .. code-block:: python execfile(filename) # Basically the same as: exec compile(open(filename).read()), filename, "exec" .. note:: This allows optimizations to discover the file opening nature easily and apply file embedding or whatever we will have there one day. This transformation is performed when the ``execfile`` built-in is detected as such during optimization. Generator expressions with ``yield`` ++++++++++++++++++++++++++++++++++++ These are converted at tree building time into a generator function body that yields from the iterator given, which is the put into a for loop to iterate, created a lambda function of and then called with the first iterator. That eliminates the generator expression for this case. It's a bizarre construct and with this trick needs no special code generation. This is a complex example, demonstrating multiple cases of yield in unexpected cases: .. code-block:: python x = ((yield i) for i in (1,2) if not (yield)) # Basically the same as: def x(): for i in (1,2): if not (yield): yield(yield i) Function Decorators +++++++++++++++++++ When one learns about decorators, you see that: .. code-block:: python @decorator def function(): pass # Is basically the same as: def function(): pass function = decorator( function ) The only difference is the assignment to function. In the ``@decorator`` case, if the decorator fails with an exception, the name ``function`` is not assigned yet, but kept in a temporary variable. Therefore in Nuitka this assignment is more similar to that of a lambda expression, where the assignment to the name is only at the end, which also has the extra benefit of not treating real function and lambda functions any different. This removes the need for optimization and code generation to support decorators at all. And it should make the two variants optimize equally well. Functions nested arguments ++++++++++++++++++++++++++ Nested arguments are a Python2 only feature supported by Nuitka. Consider this example: .. code-block:: python def function(a,(b,c)): return a, b, c We solve this, by kind of wrapping the function with another function that does the unpacking and gives the errors that come from this: .. code-block:: python def function(a,".1"): def _tmp(a, b, c): return a, b, c a, b = ".1" return _tmp(a, b, c) The ``".1"`` is the variable name used by CPython internally, and actually works if you use keyword arguments via star dictionary. So this is very compatible and actually the right kind of re-formulation, but it removes the need from the code that does parameter parsing to deal with these. Obviously, there is no frame for ``_tmp``, just one for ``function`` and we do not use local variables, but temporary functions. In-place Assignments ++++++++++++++++++++ In-place assignments are re-formulated to an expression using temporary variables. These are not as much a reformulation of ``+=`` to ``+``, but instead one which makes it explicit that the assign target may change its value. .. code-block:: python a += b .. code-block:: python _tmp = a.__iadd__( b ) if a is not _tmp: a = _tmp Using ``__iadd__`` here to express that for the ``+``, the in-place variant ``iadd`` is used instead. The ``is`` check may be optimized away depending on type and value knowledge later on. Complex Assignments +++++++++++++++++++ Complex assignments are defined as those with multiple targets to assign from a single source and are re-formulated to such using a temporary variable and multiple simple assignments instead. .. code-block:: python a = b = c .. code-block:: python _tmp = c b = _tmp a = _tmp del _tmp This is possible, because in Python, if one assignment fails, it can just be interrupted, so in fact, they are sequential, and all that is required is to not calculate ``c`` twice, which the temporary variable takes care of. Were ``a`` a more complex expression, e.g. ``a.some_attribute`` that might raise an exception, ``b`` would still be assigned. Unpacking Assignments +++++++++++++++++++++ Unpacking assignments are re-formulated to use temporary variables as well. .. code-block:: python a, b.attr, c[ind] = d = e, f, g = h() Becomes this: .. code-block:: python _tmp = h() _iter1 = iter(_tmp) _tmp1 = unpack(_iter1, 3) _tmp2 = unpack(_iter1, 3) _tmp3 = unpack(_iter1, 3) unpack_check(_iter1) a = _tmp1 b.attr = _tmp2 c[ind] = _tmp3 d = _tmp _iter2 = iter(_tmp) _tmp4 = unpack(_iter2, 3) _tmp5 = unpack(_iter2, 3) _tmp6 = unpack(_iter2, 3) unpack_check(_iter1) e = _tmp4 f = _tmp5 g = _tmp6 That way, the unpacking is decomposed into multiple simple statements. It will be the job of optimizations to try and remove unnecessary unpacking, in case e.g. the source is a known tuple or list creation. .. note:: The ``unpack`` is a special node which is a form of ``next`` that will raise a ``ValueError`` when it cannot get the next value, rather than a ``StopIteration``. The message text contains the number of values to unpack, therefore the integer argument. .. note:: The ``unpack_check`` is a special node that raises a ``ValueError`` exception if the iterator is not finished, i.e. there are more values to unpack. Again the number of values to unpack is provided to construct the error message. With Statements +++++++++++++++ The ``with`` statements are re-formulated to use temporary variables as well. The taking and calling of ``__enter__`` and ``__exit__`` with arguments, is presented with standard operations instead. The promise to call ``__exit__`` is fulfilled by ``try``/``except`` clause instead. .. code-block:: python with some_context as x: something( x ) .. code-block:: python tmp_source = some_context # Actually it needs to be "special look-up" for Python2.7, so attribute # look-up won't be exactly what is there. tmp_exit = tmp_source.__exit__ # This one must be held for the whole with statement, it may be assigned # or not, in our example it is. If an exception occurs when calling # ``__enter__``, the ``__exit__`` should not be called. tmp_enter_result = tmp_source.__enter__() # Indicator variable to know if "tmp_exit" has been called. tmp_indicator = False try: # Now the assignment is to be done, if there is any name for the # manager given, this may become multiple assignment statements and # even unpacking ones. x = tmp_enter_result # Then the code of the "with" block. something( x ) except Exception: # Note: This part of the code must not set line numbers, which we # indicate with special source code references, which we call "internal". # Otherwise the line of the frame would get corrupted. tmp_indicator = True if not tmp_exit(*sys.exc_info()): raise finally: if not tmp_indicator # Call the exit if no exception occurred with all arguments # as "None". tmp_exit(None, None, None) .. note:: We don't refer really to ``sys.exc_info()`` at all, instead, we have fast references to the current exception type, value and trace, taken directly from the caught exception object on the C level. If we had the ability to optimize ``sys.exc_info()`` to do that, we could use the same transformation, but right now we don't have it. For Loops +++++++++ The ``for`` loops use normal assignments and handle the iterator that is implicit in the code explicitly. .. code-block:: python for x, y in iterable: if something( x ): break else: otherwise() This is roughly equivalent to the following code: .. code-block:: python _iter = iter(iterable) _no_break_indicator = False while 1: try: _tmp_value = next(_iter) except StopIteration: # Set the indicator that the else branch may be executed. _no_break_indicator = True # Optimization should be able to tell that the else branch is run # only once. break # Normal assignment re-formulation applies to this assignment of course. x, y = _tmp_value del _tmp_value if something(x): break if _no_break_indicator: otherwise() .. note:: The ``_iter`` temporary variable is of course also in a ``try/finally`` construct, to make sure it releases after its used. The ``x, y`` assignment is of course subject to unpacking re-formulation. The ``try``/``except`` is detected to allow to use a variant of ``next`` that does not raise an exception, but to be fast check about the ``NULL`` return from ``next`` built-in. So no actual exception handling is happening in this case. While Loops +++++++++++ Quoting the ``nuitka.tree.ReformulationWhileLoopStatements`` documentation: Reformulation of while loop statements. Loops in Nuitka have no condition attached anymore, so while loops are re-formulated like this: .. code-block:: python while condition: something() .. code-block:: python while 1: if not condition: break something() This is to totally remove the specialization of loops, with the condition moved to the loop body in an initial conditional statement, which contains a ``break`` statement. That achieves, that only ``break`` statements exit the loop, and allow for optimization to remove always true loop conditions, without concerning code generation about it, and to detect such a situation, consider e.g. endless loops. .. note:: Loop analysis (not yet done) can then work on a reduced problem (which ``break`` statements are executed under what conditions) and is then automatically very general. The fact that the loop body may not be entered at all, is still optimized, but also in the general sense. Explicit breaks at the loop start and loop conditions are the same. End quoting the ``nuitka.tree.ReformulationWhileLoopStatements`` documentation: Exception Handlers ++++++++++++++++++ Exception handlers in Python may assign the caught exception value to a variable in the handler definition. And the different handlers are represented as conditional checks on the result of comparison operations. .. code-block:: python try: block() except A as e: handlerA(e) except B as e: handlerB(e) else: handlerElse() .. code-block:: python try: block() except: # These are special nodes that access the exception, and don't really # use the "sys" module. tmp_exc_type = sys.exc_info()[0] tmp_exc_value = sys.exc_info()[1] # exception_matches is a comparison operation, also a special node. if exception_matches(tmp_exc_type, (A,)): e = tmp_exc_value handlerA(e) elif exception_matches(tmp_exc_type, (B,)): e = tmp_exc_value handlerB(e) else: handlerElse() For Python3, the assigned ``e`` variables get deleted at the end of the handler block. Should that value be already deleted, that ``del`` does not raise, therefore it's tolerant. This has to be done in any case, so for Python3 it is even more complex. .. code-block:: python try: block() except: # These are special nodes that access the exception, and don't really # use the "sys" module. tmp_exc_type = sys.exc_info()[0] tmp_exc_value = sys.exc_info()[1] # exception_matches is a comparison operation, also a special node. if exception_matches(tmp_exc_type, (A,)): try: e = tmp_exc_value handlerA(e) finally: del e elif exception_matches(tmp_exc_type, (B,)): try: e = tmp_exc_value handlerB(e) finally: del e else: handlerElse() Should there be no ``else:`` branch, a default re-raise statement is used instead. And of course, the values of the current exception type and value, both use special references, that access the C++ and don't go via ``sys.exc_info`` at all, nodes called ``CaughtExceptionTypeRef`` and ``CaughtExceptionValueRef``. This means, that the different handlers and their catching run time behavior are all explicit and reduced the branches. Statement ``try``/``except`` with ``else`` ++++++++++++++++++++++++++++++++++++++++++ Much like ``else`` branches of loops, an indicator variable is used to indicate the entry into any of the exception handlers. Therefore, the ``else`` becomes a real conditional statement in the node tree, checking the indicator variable and guarding the execution of the ``else`` branch. Class Creation (Python2) ++++++++++++++++++++++++ Classes in Python2 have a body that only serves to build the class dictionary and is a normal function otherwise. This is expressed with the following re-formulation: .. code-block:: python # in module "SomeModule" # ... class SomeClass(SomeBase, AnotherBase) """ This is the class documentation. """ some_member = 3 .. code-block:: python def _makeSomeClass: # The module name becomes a normal local variable too. __module__ = "SomeModule" # The doc string becomes a normal local variable. __doc__ = """ This is the class documentation. """ some_member = 3 return locals() # force locals to be a writable dictionary, will be optimized away, but # that property will stick. This is only to express, that locals(), where # used will be writable to. exec "" SomeClass = make_class("SomeClass", (SomeBase, AnotherBase), _makeSomeClass()) That is roughly the same, except that ``_makeSomeClass`` is *not* visible to its child functions when it comes to closure taking, which we cannot express in Python language at all. Therefore, class bodies are just special function bodies that create a dictionary for use in class creation. They don't really appear after the tree building stage anymore. The type inference will of course have to become able to understand ``make_class`` quite well, so it can recognize the created class again. Class Creation (Python3) ++++++++++++++++++++++++ In Python3, classes are a complicated way to write a function call, that can interact with its body. The body starts with a dictionary provided by the metaclass, so that is different, because it can ``__prepare__`` a non-empty locals for it, which is hidden away in "prepare_class_dict" below. What's noteworthy, is that this dictionary, could e.g. be a ``OrderDict``. I am not sure, what ``__prepare__`` is allowed to return. .. code-block:: python # in module "SomeModule" # ... class SomeClass(SomeBase, AnotherBase, metaclass = SomeMetaClass) """ This is the class documentation. """ some_member = 3 .. code-block:: python # Non-keyword arguments, need to be evaluated first. tmp_bases = ( SomeBase, AnotherBase ) # Keyword arguments go next, __metaclass__ is just one of them. In principle # we need to forward the others as well, but this is ignored for the sake of # brevity. tmp_metaclass = select_metaclass(tmp_bases, SomeMetaClass ) tmp_prepared = tmp_metaclass.__prepare__("SomeClass", tmp_bases) # The function that creates the class dictionary. Receives temporary variables # to work with. def _makeSomeClass: # This has effect, currently I don't know how to force that in Python3 # syntax, but we will use something that ensures it. locals() = tmp_prepared # The module name becomes a normal local variable too. __module__ = "SomeModule" # The doc string becomes a normal local variable. __doc__ = """ This is the class documentation. """ some_member = 3 # Create the class, share the potential closure variable "__class__" # with others. __class__ = tmp_metaclass("SomeClass", tmp_bases, locals()) return __class__ # Build and assign the class. SomeClass = _makeSomeClass() Generator Expressions +++++++++++++++++++++ There are re-formulated as functions. Generally they are turned into calls of function bodies with (potentially nested) for loops: .. code-block:: python gen = (x*2 for x in range(8) if cond()) .. code-block:: python def _gen_helper(__iterator): for x in __iterator: if cond(): yield x*2 gen = _gen_helper(range(8)) List Contractions +++++++++++++++++ The list contractions of Python2 are different from those of Python3, in that they don't actually do any closure variable taking, and that no function object ever exists. .. code-block:: python list_value = [x*2 for x in range(8) if cond()] .. code-block:: python def _listcontr_helper(__iterator): result = [] for x in __iterator: if cond(): result.append(x*2) return result list_value = _listcontr_helper(range(8)) The difference is that with Python3, the function "_listcontr_helper" is really there and named ````, whereas with Python2 the function is only an outline, so it can readily access the name space. Set Contractions ++++++++++++++++ The set contractions of Python2.7 are like list contractions in Python3, in that they produce an actual helper function: .. code-block:: python set_value = {x*2 for x in range(8) if cond()} .. code-block:: python def _setcontr_helper(__iterator): result = set() for x in __iterator: if cond(): result.add(x*2) return result set_value = _setcontr_helper( range(8) ) Dictionary Contractions +++++++++++++++++++++++ The dictionary contractions of are like list contractions in Python3, in that they produce an actual helper function: .. code-block:: python dict_value = {x: x*2 for x in range(8) if cond()} .. code-block:: python def _dictcontr_helper(__iterator): result = {} for x in __iterator: if cond(): result[x] = x*2 return result set_value = _dictcontr_helper( range(8) ) Boolean expressions ``and`` and ``or`` ++++++++++++++++++++++++++++++++++++++ The short circuit operators ``or`` and ``and`` tend to be only less general that the ``if``/``else`` expressions, but have dedicated nodes. We used to have a re-formulation towards those, but we now do these via dedicated nodes too. These new nodes, present the evaluation of the left value, checking for its truth value, and depending on it, to pick it, or use the right value. Simple Calls ++++++++++++ As seen below, even complex calls are simple calls. In simple calls of Python there is still some hidden semantic going on, that we expose. .. code-block:: python func(arg1, arg2, named1 = arg3, named2 = arg4) On the C-API level there is a tuple and dictionary built. This one is exposed: .. code-block:: python func(*(arg1, arg2), **{"named1" : arg3, "named2" : arg4}) A called function will access this tuple and the dictionary to parse the arguments, once that is also re-formulated (argument parsing), it can then lead to simple in-lining. This way calls only have 2 arguments with constant semantics, that fits perfectly with the C-API where it is the same, so it is actually easier for code generation. Although the above looks like a complex call, it actually is not. No checks are needed for the types of the star arguments and it's directly translated to ``PyObject_Call``. Complex Calls +++++++++++++ The call operator in Python allows to provide arguments in 4 forms. * Positional (or normal) arguments * Named (or keyword) arguments * Star list arguments * Star dictionary arguments The evaluation order is precisely that. An example would be: .. code-block:: python something(pos1, pos2, name1 = named1, name2 = named2, *star_list, **star_dict) The task here is that first all the arguments are evaluated, left to right, and then they are merged into only two, that is positional and named arguments only. for this, the star list argument and the star dictionary arguments, are merged with the positional and named arguments. What's peculiar, is that if both the star list and dictionary arguments are present, the merging is first done for star dictionary, and only after that for the star list argument. This makes a difference, because in case of an error, the star argument raises first. .. code-block:: python something(*1, **2) This raises "TypeError: something() argument after ** must be a mapping, not int" as opposed to a possibly more expected "TypeError: something() argument after * must be a sequence, not int." That doesn't matter much though, because the value is to be evaluated first anyway, and the check is only performed afterwards. If the star list argument calculation gives an error, this one is raised before checking the star dictionary argument. So, what we do, is we convert complex calls by the way of special functions, which handle the dirty work for us. The optimization is then tasked to do the difficult stuff. Our example becomes this: .. code-block:: python def _complex_call(called, pos, kw, star_list_arg, star_dict_arg): # Raises errors in case of duplicate arguments or tmp_star_dict not # being a mapping. tmp_merged_dict = merge_star_dict_arguments( called, tmp_named, mapping_check( called, tmp_star_dict ) ) # Raises an error if tmp_star_list is not a sequence. tmp_pos_merged = merge_pos_arguments( called, tmp_pos, tmp_star_list ) # On the C-API level, this is what it looks like. return called( *tmp_pos_merged, **tmp_merged_dict ) returned = _complex_call( called = something, pos = (pos1, pos2), named = { "name1" : named1, "name2" = named2 }, star_list_arg = star_list, star_list_arg = star_dict ) The call to ``_complex_call`` is be a direct function call with no parameter parsing overhead. And the call in its end, is a special call operation, which relates to the "PyObject_Call" C-API. Print statements ++++++++++++++++ The ``print`` statement exists only in Python2. It implicitly converts its arguments to strings before printing them. In order to make this accessible and compile time optimized, this is made visible in the node tree. .. code-block:: python print arg1, "1", 1 .. code-block:: python print str(arg1), "1", str(1) Only string objects are spared from the ``str`` built-in wrapper, because that would only cause noise in optimization stage. Additionally, each ``print`` may have a target, and multiple arguments, which we break down as well for dumber code generation. The target is evaluated first and should be a file, kept referenced throughout the whole print statement. .. code-block:: python print >>target_file, str(arg1), "1", str(1) This is being reformulated to: try: tmp_target = target_file print >>tmp_target, str(arg1), print >>tmp_target, "1", print >>tmp_target, str(1), print >>tmp_target finally: del tmp_target This allows code generation to not deal with arbitrary amount of arguments to ``print``. It also separates the newline indicator from the rest of things, which makes sense too, having it as a special node, as it's behavior with regards to soft-space is different of course. And finally, for ``print`` without a target, we still assume that a target was given, which would be ``sys.stdout`` in a rather hard-coded way (no variable look-ups involved). Call to ``dir`` without arguments --------------------------------- This expression is reformulated to ``locals().keys()`` for Python2, and ``list(locals.keys())``. Nodes that serve special purposes --------------------------------- Try statements ++++++++++++++ In Python, there is ``try``/``except`` and ``try``/``finally``. In Nuitka there is only a ``try``, which then has blocks to handle exceptions, ``continue``, or ``break``, or ``return``. There is no ``else`` to this node type. This is more low level and universal. Code for the different handlers can be different. User provided ``finally`` blocks become copied into the different handlers. Releases ++++++++ When a function exits, the local variables are to be released. The same applies to temporary variables used in re-formulations. These releases cause a reference to the object to the released, but no value change. They are typically the last use of the object in the function. The are similar to ``del``, but make no value change. For shared variables this effect is most visible. Side Effects ++++++++++++ When an exception is bound to occur, and this can be determined at compile time, Nuitka will not generate the code the leads to the exception, but directly just raise it. But not in all cases, this is the full thing. Consider this code: .. code-block:: python f(a(), 1 / 0) The second argument will create a ``ZeroDivisionError`` exception, but before that ``a()`` must be executed, but the call to ``f`` will never happen and no code is needed for that, but the name look-up must still succeed. This then leads to code that is internally like this: .. code-block:: python f(a(), raise ZeroDivisionError) which is then modeled as: .. code-block:: python side_effect(a(), f, raise ZeroDivisionError) where we can consider "side_effect" to be a function that returns the last expression. Of course, if this is not part of another expression, but close to statement level, side effects, can be converted to multiple statements simply. Another use case, is that the value of an expression can be predicted, but that the language still requires things to happen, consider this: .. code-block:: python a = len( ( f(), g() ) ) We can tell that ``a`` will be 2, but the call to ``f`` and ``g`` must still be performed, so it becomes: .. code-block:: python a = side_effects(f(), g(), 2) Modelling side effects explicitely has the advantage of recognizing them easily and allowing to drop the call to the tuple building and checking its length, only to release it. Caught Exception Type/Value References ++++++++++++++++++++++++++++++++++++++ When catching an exception, these are not directly put to ``sys.exc_info()``, but remain as mere C variables. From there, they can be accessed with these nodes, or if published then from the thread state. Hard Module Imports +++++++++++++++++++ These are module look-ups that don't depend on any local variable for the module to be looked up, but with hard-coded names. These may be the result of optimization gaining such level of certainty. Currently they are used to represent ``sys.stdout`` usage for ``print`` statements, but other usages will follow. Locals Dict Update Statement ++++++++++++++++++++++++++++ For the ``exec`` re-formulation, we apply an explicit sync back to locals as an explicit node. It helps us to tell the affected local variable traces that they might be affected. It represents the bit of ``exec`` in Python2, that treats ``None`` as the locals argument as an indication to copy back. Plan to replace "python-qt" for the GUI ======================================= Porting the tree inspector available with ``--dump-gui`` to "wxWindows" is very much welcome as the "python-qt4" bindings are severely under documented. Plan to add "ctypes" support ============================ Add interfacing to C code, so Nuitka can turn a ``ctypes`` binding into an efficient binding as if it were written manually with Python C-API or better. Goals/Allowances to the task ---------------------------- 1. Goal: Must not directly use any pre-existing C/C++ language file headers, only generate declarations in generated C code ourselves. We would rather write or use tools that turn an existing a C header to some ``ctypes`` declarations if it needs to be, but not mix and use declarations from existing header code. ..note:: The "cffi" interface maybe won't have the issue, but it's not something we need to write or test the code for. 2. Allowance: May use ``ctypes`` module at compile time to ask things about ``ctypes`` and its types. 3. Goal: Should make use of ``ctypes``, to e.g. not hard code in Nuitka what ``ctypes.c_int()`` gives on the current platform, unless there is a specific benefit. 4. Allowance: Not all ``ctypes`` usages must be supported immediately. 5. Goal: Try and be as general as possible. For the compiler, ``ctypes`` support should be hidden behind a generic interface of some sort. Supporting ``math`` module should be the same thing. Type Inference - The Discussion ------------------------------- Main initial goal is to forward value knowledge. When you have ``a = b``, that means that a and b now "alias". And if you know the value of ``b`` you can assume to know the value of ``a``. This is called "aliasing". When assigning ``a`` to something new, that won't change ``b`` at all. But when an attribute is set, a method called of it, that might impact the actual value, referenced by both. We need to understand mutable vs. immutable though, as some things are not affectable by aliasing in any way. .. code-block:: python a = 3 b = 3 b += 4 # a is not changed a = [ 3 ] b = a b += [4] # a is changed indeed If we cannot tell, we must assume that ``a`` might be changed. It's either ``b`` or what ``a`` was before. If the type is not mutable, we can assume the aliasing to be broken up, and if it is, we can assume both to be the same value still. When that value is a compile time constant, we will want to push it forward, and we do that with "(Constant) Value Propagation", which is implemented already. We avoid too large constants, and we properly trace value assignments, but not yet aliases. In order to fully benefit from type knowledge, the new type system must be able to be fully friends with existing built-in types, but for classes to also work with it, it should not be tied to them. The behavior of a type ``long``, ``str``, etc. ought to be implemented as far as possible with the built-in ``long``, ``str`` at compiled time as well. .. note:: This "use the real thing" concept extends beyond builtin types, e.g. ``ctypes.c_int()`` should also be used, but we must be aware of platform dependencies. The maximum size of ``ctypes.c_int`` values would be an example of that. Of course that may not be possible for everything. This approach has well proven itself with built-in functions already, where we use real built-ins where possible to make computations. We have the problem though that built-ins may have problems to execute everything with reasonable compile time cost. Another example, consider the following code: .. code-block:: python len("a" * 1000000000000) To predict this code, calculating it at compile time using constant operations, while feasible, puts an unacceptable burden on the compilation. Esp. we wouldn't want to produce such a huge constant and stream it, the C++ code would become too huge. So, we need to stop the ``*`` operator from being used at compile time and cope with reduced knowledge, already here: .. code-block:: python "a" * 10000000000000 Instead, we would probably say that for this expression: - The result is a ``str`` or a C level ``PyStringObject *``. - We know its length exactly, it's ``10000000000000``. - Can predict every of its elements when sub-scripted, sliced, etc., if need be, with a function we may create. Similar is true for this horrible (in Python2) thing: .. code-block:: python range(10000000000000) So it's a rather general problem, this time we know: - The result is a ``list`` or C level ``PyListObject *`` - We know its length exactly, ``10000000000000`` - Can predict every of its elements when index, sliced, etc., if need be, with a function. Again, we wouldn't want to create the list. Therefore Nuitka avoids executing these calculation, when they result in constants larger than a threshold of e.g. 256 elements. This concept has to be also applied to large integers and more CPU and memory traps. Now lets look at a more complete use case: .. code-block:: python for x in range( 10000000000000 ): doSomething() Looking at this example, one traditional way to look at it, would be to turn ``range`` into ``xrange``, and to note that ``x`` is unused. That would already perform better. But really better is to notice that ``range()`` generated values are not used at all, but only the length of the expression matters. And even if ``x`` were used, only the ability to predict the value from a function would be interesting, so we would use that computation function instead of having an iteration source. Being able to predict from a function could mean to have Python code to do it, as well as C code to do it. Then code for the loop can be generated without any CPython library usage at all. .. note:: Of course, it would only make sense where such calculations are "O(1)" complexity, i.e. do not require recursion like "n!" does. The other thing is that CPython appears to at - run time - take length hints from objects for some operations, and there it would help too, to track length of objects, and provide it, to outside code. Back to the original example: .. code-block:: python len("a" * 1000000000000) The theme here, is that when we can't compute all intermediate expressions, and we sure can't do it in the general case. But we can still, predict some of properties of an expression result, more or less. Here we have ``len`` to look at an argument that we know the size of. Great. We need to ask if there are any side effects, and if there are, we need to maintain them of course. This is already done by existing optimization if an operation generates an exception. .. note:: The optimization of ``len`` has been implemented and works for all kinds of container creation and ranges. Applying this to "ctypes" ------------------------- The *not so specific* problem to be solved to understand ``ctypes`` declarations is maybe as follows: .. code-block:: python import ctypes This leads to Nuitka in its tree to have an assignment from a ``__import__`` expression to the variable ``ctypes``. It can be predicted by default to be a module object, and even better, it can be known as ``ctypes`` from standard library with more or less certainty. See the section about "Importing". So that part is "easy", and it's what will happen. During optimization, when the module ``__import__`` expression is examined, it should say: - ``ctypes`` is a module - ``ctypes`` is from standard library (if it is, might not be true) - ``ctypes`` then has code behind it, called ``ModuleFriend`` that knows things about it attributes, that should be asked. The later is the generic interface, and the optimization should connect the two, of course via package and module full names. It will need a ``ModuleFriendRegistry``, from which it can be pulled. It would be nice if we can avoid ``ctypes`` to be loaded into Nuitka unless necessary, so these need to be more like a plug-in, loaded only if necessary, i.e. the user code actually uses ``ctypes``. Coming back to the original expression, it also contains an assignment expression, because it re-formulated to be more like this: .. code-block:: python ctypes = __import__("ctypes") The assigned to object, simply gets the type inferred propagated as part of an SSA form. Ideally, we could be sure that nothing in the program changes the variable, and therefore have only one version of that variable. For module variables, when the execution leaves the module to unknown code, or unclear code, it might change the variable. Therefore, likely we will often only assume that it could still be ``ctypes``, but also something else. Depending on how well we control module variable assignment, we can decide this more of less quickly. With "compiled modules" types, the expectation is that it's merely a quick C `==` comparison check. The module friend should offer code to allow a check if it applies, for uncertain cases. Then when we come to uses of it: .. code-block:: python ctypes.c_int() At this point, using SSA, we are more of less sure, that ``ctypes`` is at that point the module, and that we know what it's ``c_int`` attribute is, at compile time, and what it's call result is. We will use the module friend to help with that. It will attach knowledge about the result of that expression during the SSA collection process. This is more like a value forward propagation than anything else. In fact, constant propagation should only be the special case of it, and one design goal of Nuitka was always to cover these two cases with the same code. Excursion to Functions ---------------------- In order to decide what this means to functions and their call boundaries, if we propagate forward, how to handle this: .. code-block:: python def my_append(a, b): a.append(b) return a We annotate that ``a`` is first a "unknown but defined parameter object", then later on something that definitely has an ``append`` attribute, when returned, as otherwise an exception occurs. The type of ``a`` changes to that after ``a.append`` look-up succeeds. It might be many kinds of an object, but e.g. it could have a higher probability of being a ``PyListObject``. And we would know it cannot be a ``PyStringObject``, as that one has no ``append`` method, and would have raised an exception therefore. .. note:: If classes, i.e. other types in the program, have an ``append`` attribute, it should play a role too, there needs to be a way to plug-in to this decisions. .. note:: On the other hand, types without ``append`` attribute can be eliminated. Therefore, functions through SSA provide an automatic analysis on their return state, or return value types, or a quick way to predict return value properties, based on input value knowledge. So this could work: .. code-block:: python b = my_append([], 3) assert b == [3] # Could be decided now Goal: The structure we use makes it easy to tell what ``my_append`` may be. So, there should be a means to ask it about call results with given type/value information. We need to be able to tell, if evaluating ``my_append`` makes sense with given parameters or not, if it does impact the return value. We should e.g. be able to make ``my_append`` tell, one or more of these: - Returns the first parameter value as return value (unless it raises an exception). - The return value has the same type as ``a`` (unless it raises an exception). - The return value has an ``append`` attribute. - The return value might be a ``list`` object. - The return value may not be a ``str`` object. - The function will raise if first argument has no ``append`` attribute. The exactness of statements may vary. But some things may be more interesting. If e.g. the aliasing of a parameter value to the return value is known exactly, then information about it need to all be given up, but some can survive. It would be nice, if ``my_append`` had sufficient information, so we could specialize with ``list`` and ``int`` from the parameters, and then e.g. know at least some things that it does in that case. Such specialization would have to be decided if it makes sense. In the alternative, it could be done for each variant anyway, as there won't be that many of them. Doing this "forward" analysis appears to be best suited for functions and therefore long term. We will try it that way. Excursion to Loops ------------------ .. code-block:: python a = 1 while 1: # think loop: here b = a + 1 a = b if cond(): break print a The handling of loops (both ``for`` and ``while`` are re-formulated to this kind of loops with ``break`` statements) has its own problem. The loop start and may have an assumption from before it started, that ``a`` is constant, but that is only true for the first iteration. So, we can't pass knowledge from outside loop forward directly into the for loop body. So the collection for loops needs to be two pass for loops. First, to collect assignments, and merge these into the start state, before entering the loop body. The need to make two passes is special to loops. For a start, it is done like this. At loop entry, all pre-existing, but written traces, are turned into loop merges. Knowledge is not completely removed about everything assigned or changed in the loop, but then it's not trusted anymore. From that basis, the ``break`` exits are analysed, and merged, building up the post loop state, and ``continue`` exits of the loop replacing the unknown part of the loop entry state. The loop end is considered a ``continue`` for this purpose. Excursion to Conditions ----------------------- .. code-block:: python if cond: x = 1 else: x = 2 b = x < 3 The above code contains a condition, and these have the problem, that when exiting the conditional block, a merge must be done, of the ``x`` versions. It could be either one. The merge may trace the condition under which a choice is taken. That way, we could decide pairs of traces under the same condition. These merges of SSA variable "versions", represent alternative values. They pose difficulties, and might have to be reduced to commonality. In the above example, the ``<`` operator will have to check for each version, and then to decide that both indeed give the same result. The trace collection tracks variable changes in conditional branches, and then merges the existing state at conditional statement exits. .. note:: A branch is considered "exiting" if it is not abortive. Should it end in a ``raise``, ``break``, ``continue``, or ``return``, there is no need to merge that branch, as execution of that branch is terminated. Should both branches be abortive, that makes things really simple, as there is no need to even continue. Should only one branch exist, but be abortive, then no merge is needed, and the collection can assume after the conditional statement, that the branch was not taken, and continue. When exiting both the branches, these branches must both be merged, with their new information. In the above case: - The "yes" branch knows variable ``x`` is an ``int`` of constant value ``1`` - The "no" branch knows variable ``x`` is an ``int`` of constant value ``2`` That might be collapsed to: - The variable ``x`` is an integer of value in ``(1,2)`` Given this, we then should be able to precompute the value of this: .. code-block:: python b = x < 3 The comparison operator can therefore decide and tell: - The variable ``b`` is a boolean of constant value ``True``. Were it unable to decide, it would still be able to say: - The variable ``b`` is a boolean. For conditional statements optimization, it's also noteworthy, that the condition is known to pass or not pass the truth check, inside branches, and in the case of non-exiting single branches, after the statement it's not true. We may want to take advantage of it. Consider e.g. .. code-block:: python if type( a ) is list: a.append( x ) else: a += ( x, ) In this case, the knowledge that ``a`` is a list, could be used to generate better code and with the definite knowledge that ``a`` is of type list. With that knowledge the ``append`` attribute call will become the ``list`` built-in type operation. Excursion to ``return`` statements ---------------------------------- The ``return`` statement (like ``break``, ``continue``, ``raise``) is "aborting" to control flow. It is always the last statement of inspected block. When there statements to follow it, optimization will remove it as "dead code". If all branches of a conditional statement are "aborting", the statement is decided "aborting" too. If a loop doesn't abort with a break, it should be considered "aborting" too. Excursion to ``yield`` expressions ---------------------------------- The ``yield`` expression can be treated like a normal function call, and as such invalidates some known constraints just as much as they do. It executes outside code for an unknown amount of time, and then returns, with little about the outside world known anymore, if it's accessible from there. Mixed Types ----------- Consider the following inside a function or module: .. code-block:: python if cond is not None: a = [x for x in something() if cond(x)] else: a = () A programmer will often not make a difference between ``list`` and ``tuple``. In fact, using a ``tuple`` is a good way to express that something won't be changed later, as these are mutable. .. note:: Better programming style, would be to use this: .. code-block:: python if cond is not None: a = tuple(x for x in something() if cond(x)) else: a = () People don't do it, because they dislike the performance hit encountered by the generator expression being used to initialize the tuple. But it would be more consistent, and so Nuitka is using it, and of course one day Nuitka ought to be able to make no difference in performance for it. To Nuitka though this means, that if ``cond`` is not predictable, after the conditional statement we may either have a ``tuple`` or a ``list`` type object in ``a``. In order to represent that without resorting to "I know nothing about it", we need a kind of ``min``/``max`` operating mechanism that is capable of say what is common with multiple alternative values. .. note:: At this time, we don't really have that mechanism to find the commonality between values. Back to "ctypes" ---------------- .. code-block:: python v = ctypes.c_int() Coming back to this example, we needed to propagate ``ctypes``, then we can propagate "something" from ``ctypes.int`` and then known what this gives with a call and no arguments, so the walk of the nodes, and diverse operations should be addressed by a module friend. In case a module friend doesn't know what to do, it needs to say so by default. This should be enforced by a base class and give a warning or note. Now to the interface -------------------- The following is the intended interface: - Iteration with node methods ``computeStatement`` and ``computeExpression``. These traverse modules and functions (i.e. scopes) and visit everything in the order that Python executes it. The visiting object is ``TraceCollection`` and pass forward. Some node types, e.g. ``StatementConditional`` new create branch trace collections and handle the SSA merging at exit. - Replacing nodes during the visit. Both ``computeStatement`` and ``computeExpression`` are tasked to return potential replacements of themselves, together with "tags" (meaningless now), and a "message", used for verbose tracing. The replacement node of ``+`` operator, may e.g. be the pre-computed constant result, wrapped in side effects of the node, or the expression raised, again wrapped in side effects. - Assignments and references affect SSA. The SSA tree is initialized every time a scope is visited. Then during traversal, traces are built up. Every assignment and merge starts a new trace for that matter. References to a given variable version are traced that way. - Value escapes are traced too. When an operation hands over a value to outside code, it indicates so to the trace collection. This is for it to know, when e.g. a constant value, might be mutated meanwhile. - Nodes can be queried about their properties. There is a type shape and a value shape that each node can be asked about. The type shape offers methods that allow to check if certain operations are at all supported or not. These can always return ``True`` (yes), ``False`` (no), and ``None`` (cannot decide). In the case of the later, optimizations may not be able do much about it. Lets call these values "tri-state". There is also the value shape of a node. This can go deeper, and be more specific to a given node. The default implementation will be very pessimistic. Specific node types and shapes may then declare, that they e.g. have no side effects, will not raise for certain operations, have a known truth value, have a known iteration length, can predict their iteration values, etc. - Nodes are linked to certain states. During the collect, a variable reference, is linked to a certain trace state, and that can be used by parent operations. .. code-block:: python a = 1 b = a + a In this example, the references to ``a``, can look-up the ``1`` in the trace, and base value shape response to ``+`` on it. For compile time evaluation, it may also ask ``isCompileTimeConstant()`` and if both nodes will respond ``True``, then "getCompileTimeConstant()" will return ``1``, which will be be used in computation. Then ``extractSideEffects()`` for the ``a`` reference will return ``()`` and therefore, the result ``2`` will not be wrapped. An alternative approach would be ``hasTypeSlotAdd()`` on the both nodes, and they both do, to see if the selection mechanism used by CPython can be used to find which types ``+`` should be used. - Class for module import expression ``ExpressionImportModule``. This one just knows that something is imported, but not how or what it is assigned to. It will be able in a recursive compile, to provide the module as an assignment source, or the module variables or submodules as an attribute source when referenced from a variable trace or in an expression. - Base class for module friend ``ModuleFriendBase``. This is intended to provide something to overload, which e.g. can handle ``math`` in a better way. - Module ``ModuleFriendRegistry`` Provides a register function with ``name`` and instances of ``ValueFriendModuleBase`` to be registered. Recursed to modules should integrate with that too. The registry could well be done with a metaclass approach. - The module friends should each live in a module of their own. With a naming policy to be determined. These modules should add themselves via above mechanism to ``ModuleFriendRegistry`` and all shall be imported and register. Importing of e.g. ``ctypes`` should be delayed to when the friend is actually used. A meta class should aid this task. The delay will avoid unnecessary blot of the compiler at run time, if no such module is used. For "qt" and other complex stuff, this will be a must. - The walk should initially be single pass, and not maintain history. Instead optimization that needs to look at multiple things, e.g. "unused assignment", will look at the whole SSA collection afterwards. Discussing with examples ------------------------ The following examples: .. code-block:: python # Assignment, the source decides the type of the assigned expression a = b # Operator "attribute look-up", the looked up expression "ctypes" decides # via its trace. ctypes.c_int # Call operator, the called expressions decides with help of arguments, # which have been walked, before the call itself. called_expression_of_any_complexity() # import gives a module any case, and the "ModuleRegistry" may say more. import ctypes # From import need not give module, "x" decides what it is. from x import y # Operations are decided by arguments, and CPython operator rules between # argument states. a + b The optimization is mostly performed by walking of the tree and performing trace collection. When it encounters assignments and references to them, it considers current state of traces and uses it for ``computeExpression``. .. note:: Assignments to attributes, indexes, slices, etc. will also need to follow the flow of ``append``, so it cannot escape attention that a list may be modified. Usages of ``append`` that we cannot be sure about, must be traced to exist, and disallow the list to be considered known value again. Code Generation Impact ---------------------- Right now, code generation assumes that everything is a ``PyObject *``, i.e. a Python object, and does not take knowledge of ``int`` or other types into consideration at all, and it should remain like that for some time to come. Instead, ``ctypes`` value friend will be asked give ``Identifiers``, like other codes do too. And these need to be able to convert themselves to objects to work with the other things. But Code Generation should no longer require that operations must be performed on that level. Imagine e.g. the following calls: .. code-block:: python c_call( other_c_call() ) Value returned by "other_c_call()" of say ``c_int`` type, should be possible to be fed directly into another call. That should be easy by having a ``asIntC()`` in the identifier classes, which the ``ctypes`` Identifiers handle without conversions. Code Generation should one day also become able to tell that all uses of a variable have only ``c_int`` value, and use ``int`` instead of ``PyObjectLocalVariable`` more or less directly. We could consider ``PyIntLocalVariable`` of similar complexity as ``int`` after the C++ compiler performed its in-lining. Such decisions would be prepared by finalization, which then would track the history of values throughout a function or part of it. Initial Implementation ---------------------- The basic interface will be added to *all* expressions and a node may override it, potentially using trace collection state, as attached during ``computeExpression``. Goal 1 (Reached) ++++++++++++++++ Initially most things will only be able to give up on about anything. And it will be little more than a tool to do simple look-ups in a general form. It will then be the first goal to turn the following code into better performing one: .. code-block:: python a = 3 b = 7 c = a / b return c to: .. code-block:: python a = 3 b = 7 c = 3 / 7 return c and then: .. code-block:: python a = 3 b = 7 c = 0 return c and then: .. code-block:: python a = 3 b = 7 c = 0 return 0 This depends on SSA form to be able to tell us the values of ``a``, ``b``, and ``c`` to be written to by constants, which can be forward propagated at no cost. Goal 2 (Reached) ++++++++++++++++ The assignments to ``a``, ``b``, and ``c`` shall all become prey to "unused" assignment analysis in the next step. They are all only assigned to, and the assignment source has no effect, so they can be simply dropped. .. code-block:: python return 0 In the SSA form, these are then assignments without references. These assignments, can be removed if the assignment source has no side effect. Or at least they could be made "anonymous", i.e. use a temporary variable instead of the named one. That would have to take into account though, that the old version still needs a release. The most general form would first merely remove assignments that have no impact, and leave the value as a side effect, so we arrive at this first: .. code-block:: python 3 7 0 return 0 When applying the removal of expression only statements without effect, this gives us: .. code-block:: python return 0 which is the perfect result. Doing it in one step would only be an optimization. In order to be able to manipulate nodes related to a variable trace, we need to attach the nodes that did it. Consider this: .. code-block:: python if cond(): x = 1 elif other(): x = 3 # Not using "x". return 0 In the above case, the merge of the value friends, should say that ``x`` may be undefined, or one of ``1`` or ``3``, but since ``x`` is not used, apply the "dead value" trick to each branch. The removal of the "merge" of the 3 ``x`` versions, should exhibit that the other versions are also only assigned to, and can be removed. These merges of course appear as usages of the ``x`` versions. Goal 3 ++++++ Then third goal is to understand all of this: .. code-block:: python def f(): a = [] print a for i in range(1000): print a a.append(i) return len(a) .. note:: There are many operations in this, and all of them should be properly handled, or at least ignored in safe way. The first goal code gave us that the ``list`` has an annotation from the assignment of ``[]`` and that it will be copied to ``a`` until the for loop in encountered. Then it must be removed, because the ``for`` loop somehow says so. The ``a`` may change its value, due to the unknown attribute look-up of it already, not even the call. The for loop must be able to say "may change value" due to that, of course also due to the call of that attribute too. The code should therefore become equivalent to: .. code-block:: python def f(): a = [] print [] for i in range(1000): print a a.append(i) return len(a) But no other changes must occur, especially not to the ``return`` statement, it must not assume ``a`` to be constant "[]" but an unknown ``a`` instead. With that, we would handle this code correctly and have some form constant value propagation in place, handle loops at least correctly, and while it is not much, it is important demonstration of the concept. Goal 4 ++++++ The fourth goal is to understand the following: .. code-block:: python def f(cond): y = 3 if cond: x = 1 else: x = 2 return x < y In this we have a branch, and we will be required to keep track of both the branches separately, and then to merge with the original knowledge. After the conditional statement we will know that "x" is an "int" with possible values in "(1,2)", which can be used to predict that the return value is always "True". The forth goal will therefore be that the "ValueFriendConstantList" knows that append changes "a" value, but it remains a list, and that the size increases by one. It should provide an other value friend "ValueFriendList" for "a" due to that. In order to do that, such code must be considered: .. code-block:: python a = [] a.append( 1 ) a.append( 2 ) print len( a ) It will be good, if ``len`` still knows that "a" is a list, but not the constant list anymore. From here, work should be done to demonstrate the correctness of it with the basic tests applied to discover undetected issues. Fifth and optional goal: Extra bonus points for being able to track and predict "append" to update the constant list in a known way. Using "list.append" that should be done and lead to a constant result of "len" being used. The sixth and challenging goal will be to make the code generation be impacted by the value friends types. It should have a knowledge that "PyList_Append" does the job of append and use "PyList_Size" for "len". The "ValueFriends" should aid the code generation too. Last and right now optional goal will be to make "range" have a value friend, that can interact with iteration of the for loop, and "append" of the "list" value friend, so it knows it's possible to iterate 5000 times, and that "a" has then after the "loop" this size, so "len( a )" could be predicted. For during the loop, about a the range of its length should be known to be less than 5000. That would make the code of goal 2 completely analyzed at compile time. Limitations for now ------------------- - Aim only for limited examples. For ``ctypes`` that means to compile time evaluate: .. code-block:: python print ctypes.c_int( 17 ) + ctypes.c_long( 19 ) Later then call to "libc" or something else universally available, e.g. "strlen()" or "strcmp()" from full blown declarations of the callable. - We won't have the ability to test that optimization are actually performed, we will check the generated code by hand. With time, we will add XML based checks with "xpath" queries, expressed as hints, but that is some work that will be based on this work here. The "hints" fits into the "ValueFriends" concept nicely or so the hope is. - No inter-function optimization functions yet Of course, once in place, it will make the ``ctypes`` annotation even more usable. Using ``ctypes`` objects inside functions, while creating them on the module level, is therefore not immediately going to work. - No loops yet Loops break value propagation. For the ``ctypes`` use case, this won't be much of a difficulty. Due to the strangeness of the task, it should be tackled later on at a higher priority. - Not too much. Try and get simple things to work now. We shall see, what kinds of constraints really make the most sense. Understanding ``list`` subscript/slice values e.g. is not strictly useful for much code and should not block us. .. note:: This design is not likely to be the final one. .. raw:: pdf PageBreak Idea Bin ======== This an area where to drop random ideas on our minds, to later sort it out, and out it into action, which could be code changes, plan changes, issues created, etc. * Make "SELECT_METACLASS" meta class selection transparent. Looking at the "SELECT_METACLASS" it should become an anonymous helper function. In that way, the optimization process can remove choices at compile time, and e.g. in-line the effect of a meta class, if it is known. This of course makes most sense, if we have the optimizations in place that will allow this to actually happen. * Keeping track of iterations The trace collection trace should become the place, where variables or values track their use state. The iterator should keep track of the "next()" calls made to it, so it can tell which value to given in that case. That would solve the "iteration of constants" as a side effect and it would allow to tell that they can be removed. That would mean to go back in the tree and modify it long after. .. code-block:: python a = iter( ( 2, 3 ) ) b = next( a ) c = next( a ) del a It would be sweet if we could recognize that: .. code-block:: python a = iter( ( 2, 3 ) ) b = side_effect( next( a ), 2 ) c = side_effect( next( a ), 3 ) del a That trivially becomes: .. code-block:: python a = iter( ( 2, 3 ) ) next( a ) b = 2 next( a ) c = 3 del a When the "del a" is examined at the end of scope, or due to another assignment to the same variable, ending the trace, we would have to consider of the "next" uses, and retrofit the information that they had no effect. .. code-block:: python a = iter( ( 2, 3 ) ) b = 2 b = 3 del a * Aliasing Each time an assignment is made, an alias is created. A value may have different names. .. code-block:: python a = iter( range(9 )) b = a c = next(b) d = next(a) If we fail to detect the aliasing nature, we will calculate "d" wrongly. We may incref and decref values to trace it. Aliasing is automatically traced already in SSA form. The "b" is assigned to version of "a". So, that should allow to replace it with this: .. code-block:: python a = iter( range(9 )) c = next(a) d = next(a) Which then will be properly handled. * Tail recursion optimization. Functions that return the results of calls, can be optimized. The Stackless Python does it already. * Integrate with "upx" compression. Calling "upx" on the created binaries, would be easy. * In-lining constant "exec" and "eval". It should be possible to re-formulate at least cases without "locals" or "globals" given. .. code-block:: python def f(): a = 1 b = 2 exec("""a+=b;c=1""") return a, c Should become this here: .. code-block:: python def f(): a = 1 b = 2 a += b # c = 1 # MaybeLocalVariables for everything except known local ones. return a, c If this holds up, inlining ``exec`` should be relatively easy. * Original and overloaded built-ins This is about making things visible in the node tree. In Nuitka things that are not visible in the node tree tend to be wrong. We already pushed around information to the node tree a lot. Later versions, Nuitka will become able to determine it has to be the original built-in at compile time, then a condition that checks will be optimized away, together with the slow path. Or the other path, if it won't be. Then it will be optimized away, or if doubt exists, it will be correct. That is the goal. Right now, the change would mean to effectively disable all built-in call optimization, which is why we don't immediately do it. Making the compatible version, will also require a full listing of all built-ins, which is typing work merely, but not needed now. And a way to stop built-in optimization from optimizing built-in calls that it used in a wrap. Probably just some flag to indicate it when it visits it to skip it. That's for later. But should we have that both, I figure, we could not raise a ``RuntimeError`` error, but just do the correct thing, in all cases. An earlier step may raise ``RuntimeError`` error, when built-in module values are written to, that we don't support. .. raw:: pdf PageBreak * Recursion checks are expensive. If the "caller" or the "called" can declare that it cannot be called by itself, we could leave it out. TODO: Are they really that expensive? Unnecessary yes, but expensive may not be true. .. raw:: pdf PageBreak Prongs of Action ================ In this chapter, we keep track of prongs of action currently ongoing. This can get detailed and shows things we strive for. C types ------- The ultimate goal is of course to get C types to be used instead of the Python object type in as many places as possible. Currently stuck on how to reflect the types intermediate expressions towards code generation. Builtin optimization -------------------- Definitely want to get built-in names under full control, so that variable references to module variables do not have a twofold role. Currently they reference the module variable and also the potential built-in as a fallback. In terms of generated code size and complexity for modules with many variables and uses of them that is horrible. But ``some_var`` (normally) cannot be a built-in and therefore needs no code to check for that each time. This is also critical to getting to whole program optimization. Being certain what is what there on module level, will enable more definitely knowledge about data flows and module interfaces. Generator memory usage / Goto generators ---------------------------------------- Generator, coroutine, asyncgen memory usage. Currently, with fibers, every started generator alike object uses 1 MB of memory. That does not scale well. Already have we started code that will make generators re-entrant, with an initial goto dispatcher that continues code. The generator function will exit with a return when ``yield`` happens and resume at function entry again, going to the code portion in question. The storage of local variables and temporary variables in memory allocated for generator object is missing still. The benefit will be to be able to throw away all the platform dependent stuff for fibers, and hopefully this also could be faster for setup and maybe even the operation of generators. Class Creation Overhead Reduction --------------------------------- This is more of a meta goal. Some work for the metaclass has already been done, but that is Python2 only currently. Being able to to decide built-ins and to distinguish between global only variables, and built-ins more clearly will help this a lot. In the end, empty classes should be able to be statically converted to calls to ``type`` with static dictionaries. The inlining of class creation function is also needed for this, but on Python3 cannot happen yet. Memory Usage at Compile Time ---------------------------- We will need to store more and more information in the future. Getting the tree to be tight shaped is therefore an effort, where we will be spending time too. The mix-ins prevent slots usage, so lets try and get rid of those. The "children having" should become more simple and faster code. I am even thinking of even generating code in the meta class, so it's both optimal and doesn't need that mix-in any more. This is going to be ugly then. Plugins API and Options ----------------------- Plugins need options and should be documented API. So should the doxygen be generated automatically and published. Coverage Testing ---------------- And then there is coverage, it should be taken and merged from all Python versions and OSes, but I never managed to merge between Windows and Linux for unknown reasons. Python3 Performance ------------------- The Python3 lock for thread state is making it slower by a lot. I have only experimental code that just ignores the lock, but it likely only works on Linux, and I wonder why there is that lock in the first place. Ignoring the locks cannot be good. But what updates that thread state pointer ever without a thread change, and is this what abiflags are about in this context, are there some that allow us to ignore the locks. Test Runners ------------ Proper support for running tests against compiled packages. This is mostly done and needs documentation only. Distutils Integration --------------------- Proper target to build a wheel with Nuitka compiled stuff in there. This is mostly done and needs testing and documentation only. Updates for this Manual ======================= This document is written in REST. That is an ASCII format which is readable as ASCII, but used to generate PDF or HTML documents. You will find the current source under: http://nuitka.net/gitweb/?p=Nuitka.git;a=blob_plain;f=Developer_Manual.rst And the current PDF under: http://nuitka.net/doc/Developer_Manual.pdf Nuitka-0.5.28.2/tests/0000755000372000001440000000000013207540420014622 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/standalone/0000755000372000001440000000000013207540420016752 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/standalone/GtkUsing.py0000644000372000001440000000175713112214770021072 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import warnings warnings.filterwarnings("ignore", "") import sys import gtk import pygtk pygtk.require("2.0") Nuitka-0.5.28.2/tests/standalone/run_all.py0000755000372000001440000004077713207537242021012 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os import sys # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( my_print, setup, hasModule, compareWithCPython, decideFilenameVersionSkip, getRuntimeTraceOfLoadedFiles, createSearchMode, reportSkip ) from nuitka.utils.FileOperations import removeDirectory python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() search_mode.mayFailFor( # Do not expect PySide to work yet, because it has that bug still # where it won't call compiled functions as slots. "PySideUsing.py" ) for filename in sorted(os.listdir('.')): if not filename.endswith(".py"): continue if not decideFilenameVersionSkip(filename): continue path = os.path.relpath(filename) active = search_mode.consider(dirname = None, filename = filename) if not active: my_print("Skipping", filename) continue extra_flags = [ "expect_success", "standalone", "remove_output" ] if filename == "PySideUsing.py": # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): reportSkip("irrelevant Python version", ".", filename) continue if not hasModule("PySide.QtCore"): reportSkip("PySide not installed for this Python version, but test needs it", ".", filename ) continue # For the warnings. extra_flags.append("ignore_stderr") if "PyQt4" in filename: # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): reportSkip("irrelevant Python version", ".", filename) continue if not hasModule("PyQt4.QtGui"): reportSkip("PyQt4 not installed for this Python version, but test needs it", ".", filename) continue # For the plug-in information. extra_flags.append("ignore_infos") if "Idna" in filename: if not hasModule("idna.core"): reportSkip("idna not installed for this Python version, but test needs it", ".", filename) continue # For the warnings of Python2. if python_version.startswith("2"): extra_flags.append("ignore_stderr") if "PyQt5" in filename: # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): reportSkip("irrelevant Python version", ".", filename) continue if not hasModule("PyQt5.QtGui"): reportSkip("PyQt5 not installed for this Python version, but test needs it", ".", filename) continue # For the plug-in information. extra_flags.append("ignore_infos") # TODO: Temporary only if os.name == "nt" and "PyQt" in filename: continue if "PySide" in filename or "PyQt" in filename: extra_flags.append("plugin_enable:qt-plugins") if filename == "CtypesUsing.py": extra_flags.append("plugin_disable:pylint-warnings") if filename == "GtkUsing.py": # Don't test on platforms not supported by current Debian testing, and # which should be considered irrelevant by now. if python_version.startswith("2.6") or \ python_version.startswith("3.2"): reportSkip("irrelevant Python version", ".", filename) continue if not hasModule("pygtk"): reportSkip("pygtk not installed for this Python version, but test needs it", ".", filename) continue # For the warnings. extra_flags.append("ignore_stderr") if filename.startswith("Win"): if os.name != "nt": reportSkip("Windows only test", ".", filename) continue if filename == "Win32ComUsing.py": if not hasModule("win32com"): reportSkip("win32com not installed for this Python version, but test needs it", ".", filename) continue if filename == "LxmlUsing.py": if not hasModule("lxml.etree"): reportSkip("lxml.etree not installed for this Python version, but test needs it", ".", filename) continue if filename == "TkInterUsing.py": if python_version.startswith("2"): if not hasModule("Tkinter"): reportSkip("Tkinter not installed for this Python version, but test needs it", ".", filename) continue else: if not hasModule("tkinter"): reportSkip("tkinter not installed for this Python version, but test needs it", ".", filename) continue # For the warnings. extra_flags.append("ignore_stderr") if filename == "FlaskUsing.py": if not hasModule("flask"): reportSkip( "flask not installed for this Python version, but test needs it", ".", filename) continue # For the warnings. extra_flags.append("ignore_stderr") if filename == "NumpyUsing.py": # TODO: Disabled for now. reportSkip("numpy.test not fully working yet", ".", filename) continue if not hasModule("numpy"): reportSkip("numpy not installed for this Python version, but test needs it", ".", filename) continue extra_flags.append("plugin_enable:data-files") if filename == "PmwUsing.py": if not hasModule("Pwm"): reportSkip("Pwm not installed for this Python version, but test needs it", ".", filename) continue extra_flags.append("plugin_enable:pmw-freeze") my_print("Consider output of recursively compiled program:", filename) # First compare so we know the program behaves identical. compareWithCPython( dirname = None, filename = filename, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) # Second use "strace" on the result. loaded_filenames = getRuntimeTraceOfLoadedFiles( path = os.path.join( filename[:-3] + ".dist", filename[:-3] + ".exe" ) ) current_dir = os.path.normpath(os.getcwd()) current_dir = os.path.normcase(current_dir) illegal_access = False for loaded_filename in loaded_filenames: loaded_filename = os.path.normpath(loaded_filename) loaded_filename = os.path.normcase(loaded_filename) loaded_basename = os.path.basename(loaded_filename) if loaded_filename.startswith(current_dir): continue if loaded_filename.startswith(os.path.abspath(current_dir)): continue if loaded_filename.startswith("/etc/"): continue if loaded_filename.startswith("/proc/") or loaded_filename == "/proc": continue if loaded_filename.startswith("/dev/"): continue if loaded_filename.startswith("/tmp/"): continue if loaded_filename.startswith("/run/"): continue if loaded_filename.startswith("/usr/lib/locale/"): continue if loaded_filename.startswith("/usr/share/locale/"): continue if loaded_filename.startswith("/usr/share/X11/locale/"): continue # Themes may of course be loaded. if loaded_filename.startswith("/usr/share/themes"): continue if "gtk" in loaded_filename and "/engines/" in loaded_filename: continue # Taking these from system is harmless and desirable if loaded_basename.startswith(( "libz.so", "libutil.so", "libgcc_s.so", "libm.so.", )): continue # System C libraries are to be expected. if loaded_basename.startswith(( "libc.so.", "libpthread.so.", "libdl.so.", "libm.so.", )): continue # Loaded by C library potentially for DNS lookups. if loaded_basename.startswith(( "libnss_", "libnsl", )): continue # Loaded by dtruss on MacOS X. if loaded_filename.startswith("/usr/lib/dtrace/"): continue # Loaded by cowbuilder and pbuilder on Debian if loaded_basename == ".ilist": continue if "cowdancer" in loaded_filename: continue if "eatmydata" in loaded_filename: continue # Loading from home directories is OK too. if loaded_filename.startswith("/home/") or \ loaded_filename.startswith("/data/") or \ loaded_filename.startswith("/root/") or \ loaded_filename in ("/home", "/data", "/root"): continue # For Debian builders, /build is OK too. if loaded_filename.startswith("/build/") or \ loaded_filename == "/build": continue # TODO: Unclear, loading gconv from filesystem of installed system # may be OK or not. I think it should be. if loaded_basename == "gconv-modules.cache": continue if "/gconv/" in loaded_filename: continue if loaded_basename.startswith("libicu"): continue # Loading from caches is OK. if loaded_filename.startswith("/var/cache/"): continue # PySide accesses its directory. if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/PySide": continue # GTK accesses package directories only. if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/gtk-2.0/gtk": continue if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/glib": continue if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/gtk-2.0/gio": continue if loaded_filename == "/usr/lib/python" + \ python_version[:3] + \ "/dist-packages/gobject": continue # PyQt5 seems to do this, but won't use contents then. if loaded_filename in ( "/usr/lib/x86_64-linux-gnu/qt5/plugins", "/usr/lib/x86_64-linux-gnu/qt5", "/usr/lib/x86_64-linux-gnu", "/usr/lib" ): continue if loaded_filename == "/usr/bin/python3.2mu": continue # Accessing SE-Linux is OK. if loaded_filename in ("/sys/fs/selinux", "/selinux"): continue # Allow reading time zone info of local system. if loaded_filename.startswith("/usr/share/zoneinfo/"): continue # The access to .pth files has no effect. if loaded_filename.endswith(".pth"): continue # Looking at site-package dir alone is alone. if loaded_filename.endswith(("site-packages", "dist-packages")): continue # Windows baseline DLLs if loaded_basename.upper() in ( "SHELL32.DLL","USER32.DLL","KERNEL32.DLL", "NTDLL.DLL", "NETUTILS.DLL", "LOGONCLI.DLL", "GDI32.DLL", "RPCRT4.DLL", "ADVAPI32.DLL", "SSPICLI.DLL", "SECUR32.DLL", "KERNELBASE.DLL", "WINBRAND.DLL", "DSROLE.DLL", "DNSAPI.DLL", "SAMCLI.DLL", "WKSCLI.DLL", "SAMLIB.DLL", "WLDAP32.DLL", "NTDSAPI.DLL", "CRYPTBASE.DLL", "W32TOPL", "WS2_32.DLL", "SPPC.DLL", "MSSIGN32.DLL", "CERTCLI.DLL", "WEBSERVICES.DLL", "AUTHZ.DLL", "CERTENROLL.DLL", "VAULTCLI.DLL", "REGAPI.DLL", "BROWCLI.DLL", "WINNSI.DLL", "DHCPCSVC6.DLL", "PCWUM.DLL", "CLBCATQ.DLL", "IMAGEHLP.DLL", "MSASN1.DLL", "DBGHELP.DLL", "DEVOBJ.DLL", "DRVSTORE.DLL", "CABINET.DLL", "SCECLI.DLL", "SPINF.DLL", "SPFILEQ.DLL", "GPAPI.DLL", "NETJOIN.DLL", "W32TOPL.DLL", "NETBIOS.DLL", "DXGI.DLL", "DWRITE.DLL", "D3D11.DLL", "WLANAPI.DLL", "WLANUTIL.DLL", "ONEX.DLL", "EAPPPRXY.DLL", "MFPLAT.DLL", "AVRT.DLL", "ELSCORE.DLL", "INETCOMM.DLL", "MSOERT2.DLL", "IEUI.DLL", "MSCTF.DLL", "MSFEEDS.DLL", "UIAUTOMATIONCORE.DLL", "PSAPI.DLL", "EFSADU.DLL", "MFC42U.DLL", "ODBC32.DLL", "OLEDLG.DLL", "NETAPI32.DLL", "LINKINFO.DLL", "DUI70.DLL", "ADVPACK.DLL", "NTSHRUI.DLL", "WINSPOOL.DRV", "EFSUTIL.DLL", "WINSCARD.DLL", "SHDOCVW.DLL", "IEFRAME.DLL", "D2D1.DLL", "GDIPLUS.DLL", "OCCACHE.DLL", "IEADVPACK.DLL", "MLANG.DLL", "MSI.DLL", "MSHTML.DLL", "COMDLG32.DLL", "PRINTUI.DLL", "PUIAPI.DLL", "ACLUI.DLL", "WTSAPI32.DLL", "FMS.DLL", "DFSCLI.DLL", "HLINK.DLL", "MSRATING.DLL", "PRNTVPT.DLL", "IMGUTIL.DLL", "MSLS31.DLL", "VERSION.DLL", "NORMALIZ.DLL", "IERTUTIL.DLL", "WININET.DLL", "WINTRUST.DLL", "XMLLITE.DLL", "APPHELP.DLL", "PROPSYS.DLL", "RSTRTMGR.DLL", "NCRYPT.DLL", "BCRYPT.DLL", "MMDEVAPI.DLL", "MSILTCFG.DLL", "DEVMGR.DLL", "DEVRTL.DLL", "NEWDEV.DLL", "VPNIKEAPI.DLL", "WINHTTP.DLL", "WEBIO.DLL", "NSI.DLL", "DHCPCSVC.DLL", "CRYPTUI.DLL", "ESENT.DLL", "DAVHLPR.DLL", "CSCAPI.DLL", "ATL.DLL", "OLEAUT32.DLL", "SRVCLI.DLL", "RASDLG.DLL", "MPRAPI.DLL", "RTUTILS.DLL", "RASMAN.DLL", "MPRMSG.DLL", "SLC.DLL", "CRYPTSP.DLL", "RASAPI32.DLL", "TAPI32.DLL", "EAPPCFG.DLL", "NDFAPI.DLL", "WDI.DLL", "COMCTL32.DLL", "UXTHEME.DLL", "IMM32.DLL", "OLEACC.DLL", "WINMM.DLL", "WINDOWSCODECS.DLL", "DWMAPI.DLL", "DUSER.DLL", "PROFAPI.DLL", "URLMON.DLL", "SHLWAPI.DLL", "LPK.DLL", "USP10.DLL", "CFGMGR32.DLL", "MSIMG32.DLL", "POWRPROF.DLL", "SETUPAPI.DLL", "WINSTA.DLL", "CRYPT32.DLL", "IPHLPAPI.DLL", "MPR.DLL", "CREDUI.DLL", "NETPLWIZ.DLL", "OLE32.DLL", "ACTIVEDS.DLL", "ADSLDPC.DLL", "USERENV.DLL", "APPREPAPI.DLL", "BCP47LANGS.DLL", "BCRYPTPRIMITIVES.DLL", "CERTCA.DLL", "CHARTV.DLL", "COMBASE.DLL", "COML2.DLL", "DCOMP.DLL", "DPAPI.DLL", "DSPARSE.DLL", "FECLIENT.DLL", "FIREWALLAPI.DLL", "FLTLIB.DLL", "MRMCORER.DLL", "NTASN1.DLL", "SECHOST.DLL", "SETTINGSYNCPOLICY.DLL", "SHCORE.DLL", "TBS.DLL", "TWINAPI.APPCORE.DLL", "TWINAPI.DLL", "VIRTDISK.DLL", "WEBSOCKET.DLL", "WEVTAPI.DLL", "WINMMBASE.DLL", "WMICLNT.DLL", "UCRTBASE.DLL"): continue # Win API can be assumed. if loaded_basename.upper().startswith("API-MS-WIN"): continue # MSVC run time DLLs, seem to sometimes come from system. TODO: # clarify if that means we did it wrong. if loaded_basename.upper() in ("MSVCRT.DLL", "MSVCR90.DLL"): continue # These stopped being loaded by system on Windows 10. if loaded_basename.upper() in ("MSVCP_WIN.DLL", "WIN32U.DLL"): continue my_print("Should not access '%s'." % loaded_filename) illegal_access = True if illegal_access: sys.exit(1) removeDirectory(filename[:-3] + ".dist", ignore_errors = True) search_mode.finish() Nuitka-0.5.28.2/tests/standalone/PyQt5Using.py0000644000372000001440000000373213112214770021322 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function # This test is playing with configuration settings and checking that works. from PyQt5.QtCore import QSettings # @UnresolvedImport from PyQt5.QtCore import QCoreApplication # @UnresolvedImport import sys app = QCoreApplication([]) app.setOrganizationName("BOGUS_NAME") app.setOrganizationDomain("bogosity.com") app.setApplicationName("BOGUS") print("OK.") # This test is using signals and will only work if PySide properly accepts # compiled functions as callables. from PyQt5.QtCore import pyqtSlot, pyqtSignal, QObject, QMetaObject class Communicate(QObject): speak = pyqtSignal(int) def __init__(self,name = "",parent = None): QObject.__init__(self,parent) self.setObjectName(name) class Speaker(QObject): @pyqtSlot(int) def on_communicator_speak(self, stuff): print(stuff) speaker = Speaker() someone = Communicate(name = "communicator",parent = speaker) QMetaObject.connectSlotsByName(speaker) print("The answer is:",end = "") # emit 'speak' signal someone.speak.emit(42) print("Slot should have made output by now.") Nuitka-0.5.28.2/tests/standalone/Win32ComUsing.py0000644000372000001440000000171413112214770021677 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from win32com import storagecon from win32com.shell import shell, shellcon Nuitka-0.5.28.2/tests/standalone/FlaskUsing.py0000644000372000001440000000211213112214770021367 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from flask import Flask app = Flask(__name__) @app.route("/") def main(): return "Welcome!" if __name__ == "__main__": pass # TODO: Find something other meaningful to do. # app.run() Nuitka-0.5.28.2/tests/standalone/ShlibUsing.py0000644000372000001440000000177113112214770021402 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: import pyexpat print(pyexpat.__doc__) except ImportError: print("Skipped, no pyexpat module installed.") Nuitka-0.5.28.2/tests/standalone/PyQt4Using.py0000644000372000001440000000316113112214770021315 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This test is using signals and will only work if PySide properly accepts # compiled functions as callables. from __future__ import print_function from PyQt4.QtCore import pyqtSlot, pyqtSignal, QObject, QMetaObject class Communicate(QObject): speak = pyqtSignal(int) def __init__(self,name = "",parent = None): QObject.__init__(self,parent) self.setObjectName(name) class Speaker(QObject): @pyqtSlot(int) def on_communicator_speak(self, stuff): print(stuff) speaker = Speaker() someone = Communicate(name = "communicator",parent = speaker) QMetaObject.connectSlotsByName(speaker) print("The answer is:",end = "") # emit 'speak' signal someone.speak.emit(42) print("Slot should have made output by now.") Nuitka-0.5.28.2/tests/standalone/run_pbuilder_all.py0000755000372000001440000000604313112214771022656 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import subprocess import sys import os import tempfile nuitka_dir = os.path.normcase(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")) my_dir = os.path.dirname(os.path.abspath(__file__)) verbose = False code_names = [] for filename in os.listdir("/var/cache/pbuilder/"): if not filename.endswith(".tgz"): continue code_names.append(filename[:-4]) if not code_names: sys.exit("Error, found no pbuilder images.") code_names.sort() print("Working on:", ",".join(code_names)) # For build the test case with everything. if True: for code_name in code_names: subprocess.check_call( [ os.path.join(my_dir, "run_pbuilder.py"), code_name, "1" if verbose else "0" ] ) # Running on host machine, for basic overview. error = False for code_name in code_names: exit_code = subprocess.call( [ "./Asserts-%s.dist/Asserts.exe" % code_name ], stdout = open("/dev/null") if not verbose else sys.stdout, stderr = subprocess.STDOUT ) if exit_code: print(code_name.title(), "FAIL") error = True else: print(code_name.title(), "OK") if not error: with tempfile.NamedTemporaryFile("w", delete = False) as script_file: script_file.write("cd %s\n" % nuitka_dir) for code_name in code_names: script_file.write( """\ (./Asserts-%(code_name)s.dist/Asserts.exe >/dev/null && echo %(code_name)s OK) || \ echo %(code_name)s FAIL\n""" % {"code_name" : code_name} ) tmp_script = script_file.name try: for code_name in code_names: print("CHECK", code_name, ":") subprocess.check_call( [ "sudo", "pbuilder", "--execute", "--basetgz", "/var/cache/pbuilder/" + code_name + ".tgz", "--bindmounts", nuitka_dir, tmp_script ] ) finally: os.unlink(tmp_script) Nuitka-0.5.28.2/tests/standalone/PythonExecuting.py0000644000372000001440000000225613112214770022467 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Execute CPython from a standalone binary. The standalone environment should not harm CPython execution by not having set modified PYTHONPATH or PYTHONHOME. """ import subprocess, sys subprocess.call( [ sys.executable, "-c", "print('From forked CPython', 27)" ] ) Nuitka-0.5.28.2/tests/standalone/NumpyUsing.py0000644000372000001440000000163113112214770021444 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import numpy numpy.test() Nuitka-0.5.28.2/tests/standalone/CtypesUsing.py0000644000372000001440000000465513122472300021610 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, ctypes if os.name == "nt": # adapted from http://code.activestate.com/recipes/578513 from ctypes import wintypes # Lets allow this to match Windows API it reflects, # pylint: disable=invalid-name class PROCESS_MEMORY_COUNTERS_EX(ctypes.Structure): _fields_ = [ ("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD), ("PeakWorkingSetSize", ctypes.c_size_t), ("WorkingSetSize", ctypes.c_size_t), ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), ("QuotaPagedPoolUsage", ctypes.c_size_t), ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), ("QuotaNonPagedPoolUsage", ctypes.c_size_t), ("PagefileUsage", ctypes.c_size_t), ("PeakPagefileUsage", ctypes.c_size_t), ("PrivateUsage", ctypes.c_size_t), ] GetProcessMemoryInfo = ctypes.windll.psapi.GetProcessMemoryInfo GetProcessMemoryInfo.argtypes = [ wintypes.HANDLE, ctypes.POINTER(PROCESS_MEMORY_COUNTERS_EX), wintypes.DWORD, ] GetProcessMemoryInfo.restype = wintypes.BOOL counters = PROCESS_MEMORY_COUNTERS_EX() rv = GetProcessMemoryInfo( ctypes.windll.kernel32.GetCurrentProcess(), ctypes.byref(counters), ctypes.sizeof(counters) ) if not rv: raise ctypes.WinError() print("OK.") else: # TODO: How to get this to work. if False: libc = ctypes.CDLL("libc") printf = libc.printf printf("Hello, %s\n", "World!") print("OK.") Nuitka-0.5.28.2/tests/standalone/IdnaUsing.py0000644000372000001440000000176413112214770021216 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import idna.core import sys print(idna.core, "idna.idnadata" in sys.modules) Nuitka-0.5.28.2/tests/standalone/PyQt4Plugins.py0000644000372000001440000000171213112214770021651 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from PyQt4 import QtGui print(QtGui.QImageReader.supportedImageFormats()) Nuitka-0.5.28.2/tests/standalone/TkInterUsing.py0000644000372000001440000000177113112214770021721 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Python3 changed module name. try: import Tkinter as tkinter except ImportError: import tkinter assert tkinter Nuitka-0.5.28.2/tests/standalone/LxmlUsing.py0000644000372000001440000000177313112214770021257 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import lxml.etree tree = lxml.etree.fromstring("value") assert tree.tag == "root" assert tree.text == "value" Nuitka-0.5.28.2/tests/standalone/PySideUsing.py0000644000372000001440000000314213112214770021530 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This test is using signals and will only work if PySide properly accepts # compiled functions as callables. from __future__ import print_function from PySide.QtCore import Slot, Signal, QObject, QMetaObject class Communicate(QObject): speak = Signal(int) def __init__(self,name = "",parent = None): QObject.__init__(self,parent) self.setObjectName(name) class Speaker(QObject): @Slot(int) def on_communicator_speak(self, stuff): print(stuff) speaker = Speaker() someone = Communicate(name = "communicator",parent = speaker) QMetaObject.connectSlotsByName(speaker) print("The answer is:",end = "") # emit 'speak' signal someone.speak.emit(42) print("Slot should have made output by now.") Nuitka-0.5.28.2/tests/standalone/Issue116_2.py0000644000372000001440000000163513112214770021073 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("4B4159".decode("hex")) Nuitka-0.5.28.2/tests/standalone/run_pbuilder.py0000755000372000001440000000510613112214771022025 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import subprocess import sys import os import tempfile import shutil import getpass nuitka_dir = os.path.normcase(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")) code_name = sys.argv[1] verbose = int(sys.argv[2]) if not verbose: sys.stdout = open("/dev/null") sys.stderr = open("/dev/null") shutil.rmtree("Asserts-%s.build" % code_name, ignore_errors = True) shutil.rmtree("Asserts-%s.dist" % code_name, ignore_errors = True) with tempfile.NamedTemporaryFile("w", delete = False) as script_file: script_file.write("apt-get install -y lsb-release python python-dev\n") script_file.write("CODE_NAME=`lsb_release -c -s`\n") script_file.write('echo Hello pbuilder for "$CODE_NAME".\n') script_file.write("cd %s\n" % nuitka_dir) script_file.write("python bin/nuitka --python-flag=-S --standalone tests/basics/Asserts.py\n") # script_file.write("python3 bin/nuitka --standalone tests/basics/Asserts.py\n") tmp_script = script_file.name try: subprocess.check_call( [ "sudo", "pbuilder", "--execute", "--basetgz", "/var/cache/pbuilder/" + code_name + ".tgz", "--bindmounts", nuitka_dir, tmp_script ] ) shutil.move( "Asserts.build", "Asserts-%s.build" % code_name ) shutil.move( "Asserts.dist", "Asserts-%s.dist" % code_name ) subprocess.check_call( [ "sudo", "chown", "-R", getpass.getuser() + ":", "Asserts-%s.build" % code_name, "Asserts-%s.dist" % code_name ] ) finally: os.unlink(tmp_script) Nuitka-0.5.28.2/tests/reflected/0000755000372000001440000000000013207540420016557 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/reflected/compile_itself.py0000755000372000001440000002525213207537242022150 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import difflib import os import shutil import subprocess import sys import time # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import getTempDir, my_print, setup # isort:skip from nuitka.utils.FileOperations import removeDirectory, listDir # isort:skip python_version = setup() nuitka_main_path = os.path.join("..", "..", "bin", "nuitka") tmp_dir = getTempDir() # Cannot detect this more automatic, either need whitelist or # blacklist not needed stuff. PACKAGE_LIST = ( "nuitka", "nuitka/nodes", "nuitka/nodes/shapes", "nuitka/tree", "nuitka/importing", "nuitka/build", "nuitka/freezer", "nuitka/gui", "nuitka/codegen", "nuitka/codegen/templates", "nuitka/codegen/c_types", "nuitka/optimizations", "nuitka/finalizations", "nuitka/plugins", "nuitka/plugins/standard", "nuitka/plugins/user", "nuitka/containers", "nuitka/utils", ) def readSource(filename): if python_version < "3": return open(filename, "rb").read() else: return open(filename, "rb").read().decode("latin1") def diffRecursive(dir1, dir2): done = set() for path1, filename in listDir(dir1): path2 = os.path.join(dir2, filename) done.add(path1) # Skip these binary files of course. if filename.endswith(".o") or \ filename.endswith(".os") or \ filename.endswith(".obj"): continue # Skip scons build database if filename == ".sconsign.dblite": continue if not os.path.exists(path2): sys.exit("Only in %s: %s" % (dir1, filename)) if os.path.isdir(path1): diffRecursive(path1, path2) elif os.path.isfile(path1): fromdate = time.ctime(os.stat(path1).st_mtime) todate = time.ctime(os.stat(path2).st_mtime) diff = difflib.unified_diff( a = readSource(path1).splitlines(), b = readSource(path2).splitlines(), fromfile = path1, tofile = path2, fromfiledate = fromdate, tofiledate = todate, n = 3 ) result = list(diff) if result: for line in result: my_print(line) sys.exit(1) else: assert False, path1 for path1, filename in listDir(dir2): path2 = os.path.join(dir2, filename) if path1 in done: continue if not os.path.exists(path1): sys.exit("Only in %s: %s" % (dir2, filename)) def executePASS1(): my_print("PASS 1: Compiling from compiler running from .py files.") base_dir = os.path.join("..", "..") for package in PACKAGE_LIST: package = package.replace('/', os.path.sep) source_dir = os.path.join(base_dir, package) target_dir = package removeDirectory( path = target_dir, ignore_errors = False ) os.mkdir(target_dir) for path, filename in listDir(target_dir): if filename.endswith(".so"): os.unlink(path) for path, filename in listDir(source_dir): if not filename.endswith(".py"): continue if filename.startswith(".#"): continue if filename != "__init__.py": my_print("Compiling '%s'." % path) command = [ os.environ["PYTHON"], nuitka_main_path, "--module", "--recurse-none", "--plugin-enable=pylint-warnings", "--output-dir=%s" % target_dir, "--no-pyi-file", path ] command += os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() result = subprocess.call( command ) if result != 0: sys.exit(result) else: shutil.copyfile(path, os.path.join(target_dir, filename)) my_print("Compiling '%s'." % nuitka_main_path) shutil.copyfile(nuitka_main_path, "nuitka.py") command = [ os.environ["PYTHON"], nuitka_main_path, "--recurse-none", "--plugin-enable=pylint-warnings", "--output-dir=.", "--python-flag=-S", "nuitka.py" ] command += os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() result = subprocess.call( command ) if result != 0: sys.exit(result) scons_inline_copy_path = os.path.join( base_dir, "nuitka", "build", "inline_copy" ) if os.path.exists(scons_inline_copy_path): shutil.copytree( scons_inline_copy_path, os.path.join("nuitka", "build", "inline_copy") ) shutil.copy( os.path.join(base_dir, "nuitka", "build", "SingleExe.scons"), os.path.join("nuitka", "build", "SingleExe.scons") ) shutil.copytree( os.path.join(base_dir, "nuitka", "build", "static_src"), os.path.join("nuitka", "build", "static_src") ) shutil.copytree( os.path.join(base_dir, "nuitka", "build", "include"), os.path.join("nuitka", "build", "include") ) def compileAndCompareWith(nuitka): if "PYTHONHASHSEED" not in os.environ: os.environ["PYTHONHASHSEED"] = '0' base_dir = os.path.join("..", "..") for package in PACKAGE_LIST: package = package.replace('/', os.path.sep) source_dir = os.path.join(base_dir, package) for path, filename in listDir(source_dir): if not filename.endswith(".py"): continue if filename.startswith(".#"): continue path = os.path.join(source_dir, filename) if filename != "__init__.py": my_print("Compiling '%s'." % path) target = filename.replace(".py", ".build") target_dir = os.path.join(tmp_dir, target) removeDirectory( path = target_dir, ignore_errors = False ) command = [ nuitka, "--module", "--recurse-none", "--plugin-enable=pylint-warnings", "--output-dir=%s"% tmp_dir, "--no-pyi-file", path ] command += os.environ.get("NUITKA_EXTRA_OPTIONS", "").split() result = subprocess.call( command ) if result != 0: sys.exit(result) diffRecursive(os.path.join(package, target), target_dir) shutil.rmtree(target_dir) if os.name == "nt": target_filename = filename.replace(".py", ".pyd") else: target_filename = filename.replace(".py", ".so") os.unlink(os.path.join(tmp_dir, target_filename)) def executePASS2(): my_print( "PASS 2: Compiling from compiler running from .exe and many .so files." ) # Windows will load the compiled modules (pyd) only from PYTHONPATH, so we # have to add it. if os.name == "nt": os.environ["PYTHONPATH"] = ':'.join(PACKAGE_LIST) compileAndCompareWith(os.path.join('.', "nuitka.exe")) # Undo the damage from above. if os.name == "nt": del os.environ["PYTHONPATH"] my_print("OK.") def executePASS3(): my_print( "PASS 3: Compiling from compiler running from .py files to single .exe." ) exe_path = os.path.join(tmp_dir, "nuitka.exe") if os.path.exists(exe_path): os.unlink(exe_path) build_path = os.path.join(tmp_dir, "nuitka.build") if os.path.exists(build_path): shutil.rmtree(build_path) path = os.path.join("..", "..", "bin", "nuitka") my_print("Compiling '%s'." % path) command = [ os.environ["PYTHON"], nuitka_main_path, path, "--output-dir=%s" % tmp_dir, "--python-flag=-S", "--recurse-all" ] result = subprocess.call( command ) if result != 0: sys.exit(result) shutil.rmtree(build_path) my_print("OK.") def executePASS4(): my_print("PASS 4: Compiling the compiler running from single exe.") exe_path = os.path.join(tmp_dir, "nuitka.exe") compileAndCompareWith(exe_path) my_print("OK.") def executePASS5(): my_print( "PASS 5: Compiling the compiler 'nuitka' package to single '.so' file." ) path = os.path.join("..", "..", "nuitka") command = [ os.environ["PYTHON"], nuitka_main_path, "--plugin-enable=pylint-warnings", "--output-dir=%s" % tmp_dir, "--recurse-all", "--recurse-not-to=PyQt5", "--recurse-not-to=nuitka.build.inline_copy", "--recurse-not-to=nuitka.build.include", "--recurse-dir=%s" % path, "--module", path ] result = subprocess.call( command ) if result != 0: sys.exit(result) os.unlink(os.path.join(tmp_dir, "nuitka.so")) os.unlink(os.path.join(tmp_dir, "nuitka.pyi")) shutil.rmtree(os.path.join(tmp_dir, "nuitka.build")) cross_compilation = False executePASS1() if cross_compilation: my_print("PASS 2: Skipped for cross-compilation case.") else: executePASS2() executePASS3() if cross_compilation: my_print("PASS 4: Skipped for cross-compilation case.") else: executePASS4() shutil.rmtree("nuitka") executePASS5() os.unlink(os.path.join(tmp_dir, "nuitka.exe")) os.rmdir(tmp_dir) Nuitka-0.5.28.2/tests/packages/0000755000372000001440000000000013207540420016400 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/packages/run_all.py0000755000372000001440000000557713122472300020424 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os, sys # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( my_print, setup, createSearchMode, compareWithCPython, withExtendedExtraOptions, getTempDir ) python_version = setup() search_mode = createSearchMode() for filename in sorted(os.listdir('.')): if not os.path.isdir(filename) or filename.endswith(".build"): continue extra_flags = [ "expect_success", "remove_output", "module_mode", "two_step_execution" ] # The use of "__main__" in the test package gives a warning. if filename == "sub_package": extra_flags.append("ignore_warnings") active = search_mode.consider( dirname = None, filename = filename ) if active: my_print("Consider output of recursively compiled program:", filename) for filename_main in os.listdir(filename): if not os.path.isdir(os.path.join(filename,filename_main)): continue if filename_main not in ("..", '.'): break else: sys.exit( """\ Error, no package in dir '%s' found, incomplete test case.""" % filename ) extensions = [ "--recurse-to=%s" % os.path.basename(filename_main) ] if not "--output-dir" in os.environ.get("NUITKA_EXTRA_OPTIONS", ""): extensions.append("--output-dir=%s" % getTempDir()) with withExtendedExtraOptions(*extensions): compareWithCPython( dirname = filename, filename = filename_main, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.28.2/tests/packages/sub_package/0000755000372000001440000000000013207540420020644 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/packages/sub_package/kitty/0000755000372000001440000000000013207540420022010 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/packages/sub_package/kitty/speak/0000755000372000001440000000000013207540420023113 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/packages/sub_package/kitty/speak/hello.py0000644000372000001440000000203413112214770024570 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def speak(): print( "hello kitty" ) # Test Issue#115, in recursing modules, this was misbehaving. import types assert type(speak) == types.FunctionType Nuitka-0.5.28.2/tests/packages/sub_package/kitty/speak/miau.py0000644000372000001440000000163513112214770024426 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def speak(): print "miau" Nuitka-0.5.28.2/tests/packages/sub_package/kitty/speak/purr.py0000644000372000001440000000163713112214770024465 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def speak(): print "mrrruu" Nuitka-0.5.28.2/tests/packages/sub_package/kitty/speak/__init__.py0000644000372000001440000000200013112214770025215 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: print("__loader__ present:", __loader__ is not None) except NameError: print("No __loader__ found, OK for Python2") Nuitka-0.5.28.2/tests/packages/sub_package/kitty/smallkitty.py0000644000372000001440000000161613112214770024564 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # size = "small" Nuitka-0.5.28.2/tests/packages/sub_package/kitty/bigkitty.py0000644000372000001440000000161413112214770024213 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # size = "big" Nuitka-0.5.28.2/tests/packages/sub_package/kitty/__init__.py0000644000372000001440000000204713112214770024125 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("__name__ is ", __name__) print("__package__ is ", __package__) import sys print("From sys.modules", sys.modules["kitty"]) from kitty.speak.hello import speak Nuitka-0.5.28.2/tests/run-tests0000755000372000001440000000241113134660221016513 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Part of "Nuitka", an optimizing Python compiler that is compatible and # integrates with CPython, but also works on its own. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Main front-end to the tests of Nuitka. Has many options, read --help output. """ import os import sys # Unchanged, running from checkout, use the parent directory, the nuitka # package ought be there. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(__file__), "..", ) ) ) sys.path.insert( 1, "/usr/share/nuitka" ) from nuitka.tools.testing.run_nuitka_tests.__main__ import main # isort:skip main() Nuitka-0.5.28.2/tests/programs/0000755000372000001440000000000013207540420016454 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/dash_import/0000755000372000001440000000000013207540420020765 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/dash_import/DashImportMain.py0000644000372000001440000000157113112214770024223 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # __import__("dash-module") b = "dash-module" __import__(b) __import__("plus+module") c = "plus+module" __import__(c) Nuitka-0.5.28.2/tests/programs/dash_import/plus+module.py0000644000372000001440000000146213112214770023607 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Module with plus imported as", __name__) Nuitka-0.5.28.2/tests/programs/dash_import/dash-module.py0000644000372000001440000000146213112214770023545 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Module with dash imported as", __name__) Nuitka-0.5.28.2/tests/programs/run_all.py0000755000372000001440000001125213122472300020463 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os import sys # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( my_print, setup, createSearchMode, compareWithCPython, withPythonPathChange ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() extra_options = os.environ.get("NUITKA_EXTRA_OPTIONS","") for filename in sorted(os.listdir('.')): if not os.path.isdir(filename) or \ filename.endswith(".build") or \ filename.endswith(".dist"): continue filename = os.path.relpath(filename) # For these, we expect that they will fail. expected_errors = [ "module_exits", "main_raises", "main_raises2", "package_contains_main" ] # Allowed after Python3, packages need no more "__init__.py" if python_version < "3.3": expected_errors.append("package_missing_init") if filename not in expected_errors: extra_flags = ["expect_success"] else: extra_flags = ["expect_failure"] if filename in ("reimport_main_static", "package_missing_init", "dash_import", "package_contains_main", "case_imports3", "import_variants", "package_init_import"): extra_flags.append("ignore_warnings") extra_flags.append("remove_output") extra_flags.append("recurse_all") # Use the original __file__ value, at least one case warns about things # with filename included. extra_flags.append("original_file") # Cannot include the files with syntax errors, these would then become # ImportError, but that's not the test. In all other cases, use two # step execution, which will not add the program original source to # PYTHONPATH. if filename != "syntax_errors": extra_flags.append("two_step_execution") else: extra_flags.append("binary_python_path") if filename == "plugin_import": os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options + \ " --recurse-directory=%s/some_package" % ( os.path.abspath(filename) ) elif filename == "reimport_main_dynamic": if python_version < '3': os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options + \ " --recurse-directory=%s" % ( os.path.abspath(filename) ) else: os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options + \ " --recurse-pattern=%s/*.py" % ( os.path.abspath(filename) ) extra_flags.append("ignore_warnings") else: os.environ["NUITKA_EXTRA_OPTIONS"] = extra_options active = search_mode.consider( dirname = None, filename = filename ) if active: my_print("Consider output of recursively compiled program:", filename) for filename_main in os.listdir(filename): if filename_main.endswith("Main.py"): break if filename_main.endswith("Main"): break else: sys.exit( """\ Error, no file ends with 'Main.py' or 'Main' in %s, incomplete test case.""" % ( filename ) ) extra_python_path = [ os.path.abspath(os.path.join(filename,entry)) for entry in os.listdir(filename) if entry.startswith("path") ] with withPythonPathChange(extra_python_path): compareWithCPython( dirname = filename, filename = filename_main, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.28.2/tests/programs/package_program/0000755000372000001440000000000013207540420021576 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_program/PackageAsMain/0000755000372000001440000000000013207540420024222 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_program/PackageAsMain/__main__.py0000644000372000001440000000150713112214770026320 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Hello world!", __name__, sys.modules[ __name__ ]) Nuitka-0.5.28.2/tests/programs/plugin_import/0000755000372000001440000000000013207540420021344 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/plugin_import/some_package/0000755000372000001440000000000013207540420023762 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/plugin_import/some_package/some_module.py0000644000372000001440000000141713112214770026650 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "hi" ) Nuitka-0.5.28.2/tests/programs/plugin_import/some_package/__init__.py0000644000372000001440000000140313112214770026072 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Nuitka-0.5.28.2/tests/programs/plugin_import/PluginImportMain.py0000644000372000001440000000153613112214770025162 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # name = "some_module" module = getattr(__import__("some_package", fromlist = [name]), name) Nuitka-0.5.28.2/tests/programs/reimport_main_static/0000755000372000001440000000000013207540420022670 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/reimport_main_static/ImportItselfStaticMain.py0000644000372000001440000000160213112214770027640 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Here I am before import", __name__) import ImportItselfStaticMain print("Here I am after import", __name__) Nuitka-0.5.28.2/tests/programs/unicode_bom/0000755000372000001440000000000013207540420020737 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/unicode_bom/UnicodeBomMain.py0000644000372000001440000000150613112214770024145 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Importing unicode BOM file:") import unicode_bom print("OK.") Nuitka-0.5.28.2/tests/programs/unicode_bom/unicode_bom.py0000644000372000001440000000152113112214770023574 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # import unicodedata print( "This is from file with BOM unicode marker" ) Nuitka-0.5.28.2/tests/programs/reimport_main_dynamic/0000755000372000001440000000000013207540420023025 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/reimport_main_dynamic/ImportItselfDynamicMain.py0000644000372000001440000000162113112214770030133 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Here I am before import", __name__) c = "ImportItselfDynamicMain" __import__(c) print("Here I am after import", __name__) Nuitka-0.5.28.2/tests/programs/main_raises/0000755000372000001440000000000013207540420020746 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/main_raises/ErrorMain.py0000644000372000001440000000161713112214770023224 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Just plain exception from the module level, supposed to report the correct file and line import ErrorRaising ErrorRaising.raiseException() Nuitka-0.5.28.2/tests/programs/main_raises/ErrorRaising.py0000644000372000001440000000145213112214770023731 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def raiseException(): return 1 / 0 Nuitka-0.5.28.2/tests/programs/main_raises2/0000755000372000001440000000000013207540420021030 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/main_raises2/ErrorInFunctionMain.py0000644000372000001440000000207413112214770025301 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Just plain exception from the function level, supposed to report the correct file and line def generator_function(): import ErrorRaising x = ( lambda : ErrorRaising.raiseException() for z in range(3) ) next(x)() def normal_function(): y = generator_function() y() normal_function() Nuitka-0.5.28.2/tests/programs/main_raises2/ErrorRaising.py0000644000372000001440000000145213112214770024013 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def raiseException(): return 1 / 0 Nuitka-0.5.28.2/tests/programs/module_exits/0000755000372000001440000000000013207540420021155 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/module_exits/Main.py0000644000372000001440000000153213112214770022415 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( type(__builtins__) ) import ErrorExitingModule print( "Should not get here!" ) Nuitka-0.5.28.2/tests/programs/module_exits/ErrorExitingModule.py0000644000372000001440000000155313112214770025323 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print( type(__builtins__) ) sys.exit("Module doing sys.exit") print( "This won't happen!" ) Nuitka-0.5.28.2/tests/programs/package_init_import/0000755000372000001440000000000013207540420022464 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_init_import/some_package/0000755000372000001440000000000013207540420025102 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_init_import/some_package/PackageLocal.py0000644000372000001440000000150013207537242027766 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Imported PackageLocal", __name__, "in", __package__ ) Nuitka-0.5.28.2/tests/programs/package_init_import/some_package/__init__.py0000644000372000001440000000171413207537242027226 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is some_package", __name__, "in", __package__) try: import PackageLocal except ImportError: print( "This must be Python3, doing local import then." ) from . import PackageLocal Nuitka-0.5.28.2/tests/programs/package_init_import/Main.py0000644000372000001440000000147213112214770023727 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package print( some_package.PackageLocal ) Nuitka-0.5.28.2/tests/programs/case_imports1/0000755000372000001440000000000013207540420021225 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports1/CasedImportingMain.py0000644000372000001440000000145113112214770025316 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import Some_Module import some_package Nuitka-0.5.28.2/tests/programs/case_imports1/path2/0000755000372000001440000000000013207540420022243 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports1/path2/some_package/0000755000372000001440000000000013207540420024661 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports1/path2/some_package/__init__.py0000644000372000001440000000143713112214770027000 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is some_package") Nuitka-0.5.28.2/tests/programs/case_imports1/path2/some_module.py0000644000372000001440000000143613112214770025132 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is some_module") Nuitka-0.5.28.2/tests/programs/case_imports1/path1/0000755000372000001440000000000013207540420022242 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports1/path1/Some_Package/0000755000372000001440000000000013207540420024560 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports1/path1/Some_Package/__init__.py0000644000372000001440000000143713112214770026677 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package") Nuitka-0.5.28.2/tests/programs/case_imports1/path1/Some_Module.py0000644000372000001440000000143613112214770025031 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module") Nuitka-0.5.28.2/tests/programs/package_contains_main/0000755000372000001440000000000013207540420022751 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_contains_main/local.py0000644000372000001440000000144213112214770024417 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print ("Imported local package") Nuitka-0.5.28.2/tests/programs/package_contains_main/PackageContainsMain.py0000644000372000001440000000142613112214770027166 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from . import local Nuitka-0.5.28.2/tests/programs/package_contains_main/__init__.py0000644000372000001440000000140213112214770025060 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/pkgutil_usage/0000755000372000001440000000000013207540420021317 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/pkgutil_usage/PkgUtilUsageMain.py0000644000372000001440000000142013207537242025047 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import package Nuitka-0.5.28.2/tests/programs/pkgutil_usage/package/0000755000372000001440000000000013207540420022712 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/pkgutil_usage/package/DATA_FILE.txt0000644000372000001440000000001513207537242024767 0ustar hayenusers00000000000000DATA_CONTENT Nuitka-0.5.28.2/tests/programs/pkgutil_usage/package/__init__.py0000644000372000001440000000160613207537242025036 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import pkgutil __version__ = pkgutil.get_data(__package__ or __name__, 'DATA_FILE.txt').decode('ascii').strip() print(__version__) Nuitka-0.5.28.2/tests/programs/case_imports2/0000755000372000001440000000000013207540420021226 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports2/CasedImportingMain.py0000644000372000001440000000145113112214770025317 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import Some_Module import some_package Nuitka-0.5.28.2/tests/programs/case_imports2/path2/0000755000372000001440000000000013207540420022244 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports2/path2/Some_Package/0000755000372000001440000000000013207540420024562 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports2/path2/Some_Package/__init__.py0000644000372000001440000000143713112214770026701 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package") Nuitka-0.5.28.2/tests/programs/case_imports2/path2/Some_Module.py0000644000372000001440000000143613112214770025033 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module") Nuitka-0.5.28.2/tests/programs/case_imports2/path1/0000755000372000001440000000000013207540420022243 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports2/path1/some_package/0000755000372000001440000000000013207540420024661 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports2/path1/some_package/__init__.py0000644000372000001440000000140213112214770026770 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/case_imports2/path1/some_module.py0000644000372000001440000000140213112214770025123 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/package_overload/0000755000372000001440000000000013207540420021742 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_overload/foo/0000755000372000001440000000000013207540420022525 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_overload/foo/__init__.py0000644000372000001440000000143713112214770024644 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # bar = 42 not_overloaded = 45 Nuitka-0.5.28.2/tests/programs/package_overload/foo/bar2.py0000644000372000001440000000140213112214770023723 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/package_overload/foo/bar.py0000644000372000001440000000140213112214770023641 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/package_overload/Main.py0000644000372000001440000000156513112214770023210 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from foo import bar, not_overloaded print(bar, not_overloaded) from foo import bar2, not_overloaded print( bar2 ) Nuitka-0.5.28.2/tests/programs/dunderinit_imports/0000755000372000001440000000000013207540420022376 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/dunderinit_imports/package/0000755000372000001440000000000013207540420023771 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/dunderinit_imports/package/SubModule.py0000644000372000001440000000143613134660221026247 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from .__init__ import value Nuitka-0.5.28.2/tests/programs/dunderinit_imports/package/__init__.py0000644000372000001440000000153113134660221026103 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("Package as __name__", __name__) value = 1 Nuitka-0.5.28.2/tests/programs/dunderinit_imports/DunderInitImportsMain.py0000644000372000001440000000143213134660221027201 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import package.SubModule Nuitka-0.5.28.2/tests/programs/dash_main/0000755000372000001440000000000013207540420020377 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/dash_main/Dash-Main.py0000644000372000001440000000145113112214770022514 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is running from",__file__) Nuitka-0.5.28.2/tests/programs/package_module_collision/0000755000372000001440000000000013207540420023467 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_module_collision/something.py0000644000372000001440000000144113112214770026037 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Importing something.py") Nuitka-0.5.28.2/tests/programs/package_module_collision/Something/0000755000372000001440000000000013207540420025424 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_module_collision/Something/__init__.py0000644000372000001440000000145213112214770027540 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Importing Something/__init__.py") Nuitka-0.5.28.2/tests/programs/package_module_collision/PackageAndModuleNamedSameMain.py0000644000372000001440000000160513112214770031550 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import Something import something print("This would collide on Windows generated source names:") print(Something) print(something) Nuitka-0.5.28.2/tests/programs/relative_import/0000755000372000001440000000000013207540420021661 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/relative_import/dircache.py0000644000372000001440000000140213112214770023773 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/relative_import/RelativeImportMain.py0000644000372000001440000000151413112214770026010 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import absolute_import import dircache print( dircache ) Nuitka-0.5.28.2/tests/programs/package_code/0000755000372000001440000000000013207540420021041 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_code/some_package/0000755000372000001440000000000013207540420023457 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_code/some_package/SomeModule.py0000644000372000001440000000145413112214770026107 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Thanks for importing SomeModule" ) Nuitka-0.5.28.2/tests/programs/package_code/some_package/__init__.py0000644000372000001440000000145113112214770025572 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Thanks for importing", __name__) Nuitka-0.5.28.2/tests/programs/package_code/PackageInitCodeMain.py0000644000372000001440000000144013112214770025172 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package.SomeModule Nuitka-0.5.28.2/tests/programs/case_imports3/0000755000372000001440000000000013207540420021227 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports3/CasedImportingMain.py0000644000372000001440000000205713112214770025323 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: import some_module except ImportError: print("Cannot import wrongly cased module.") else: print("OK, imported wrongly cased module.") try: import some_package except ImportError: print("Cannot import wrongly cased package.") else: print("OK, imported wrongly cased package.") Nuitka-0.5.28.2/tests/programs/case_imports3/path2/0000755000372000001440000000000013207540420022245 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports3/path2/Some_Package/0000755000372000001440000000000013207540420024563 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports3/path2/Some_Package/__init__.py0000644000372000001440000000145213112214770026677 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package from path2") Nuitka-0.5.28.2/tests/programs/case_imports3/path2/Some_Module.py0000644000372000001440000000145113112214770025031 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module from path2") Nuitka-0.5.28.2/tests/programs/case_imports3/path1/0000755000372000001440000000000013207540420022244 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports3/path1/Some_Package/0000755000372000001440000000000013207540420024562 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/case_imports3/path1/Some_Package/__init__.py0000644000372000001440000000145213112214770026676 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Package from path1") Nuitka-0.5.28.2/tests/programs/case_imports3/path1/Some_Module.py0000644000372000001440000000145113112214770025030 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is Some_Module from path1") Nuitka-0.5.28.2/tests/programs/import_variants/0000755000372000001440000000000013207540420021675 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/import_variants/some_package/0000755000372000001440000000000013207540420024313 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/import_variants/some_package/__init__.py0000644000372000001440000000170113207537242026433 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** some_package: Coming from '%s'" % __file__.replace(".pyc", ".py") ) print( "*** some_package: Path is '%s'" % __path__ ) print( "*** some_package: Package is '%s'" % __package__ ) Nuitka-0.5.28.2/tests/programs/import_variants/some_package/Child1.py0000644000372000001440000000161713112214770025777 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** Child1: Begin" ) from . import Child3 as localname print("*** Child1: Imported Child3", localname) print( "*** Child1: End" ) Nuitka-0.5.28.2/tests/programs/import_variants/some_package/Child3.py0000644000372000001440000000147313112214770026001 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** Child3: Begin" ) print( "*** Child3: End" ) Nuitka-0.5.28.2/tests/programs/import_variants/some_package/Child2.py0000644000372000001440000000205013112214770025770 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("*** Child2: Begin", __name__) try: import Child1 except ImportError: print( "This must be Python3, doing local import then." ) from . import Child1 print("*** Child2: Child2 is in", __package__) print("*** Child2: Imported nearby child", Child1) print( "*** Child2: End" ) Nuitka-0.5.28.2/tests/programs/import_variants/ImportVariationsMain.py0000644000372000001440000000171213112214770026370 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "*** Main: Importing" ) import some_package.Child2 print( "*** Main: Imported" ) print("*** Main: Some package", some_package) print("*** Main: Imported package child", some_package.Child2) Nuitka-0.5.28.2/tests/programs/package_missing_init/0000755000372000001440000000000013207540420022623 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_missing_init/some_package/0000755000372000001440000000000013207540420025241 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/package_missing_init/some_package/some_module.py0000644000372000001440000000166513112214770030134 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "This must be Python3.3 or higher, which no longer needs __init__.py to accept a package." ) import sys print("The parent path is", sys.modules[ "some_package"].__path__) Nuitka-0.5.28.2/tests/programs/package_missing_init/PackageMissingInitMain.py0000644000372000001440000000144113112214770027514 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package.some_module Nuitka-0.5.28.2/tests/programs/with space/0000755000372000001440000000000013207540420020503 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/with space/Space Main.py0000755000372000001440000000147413112214770022767 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "Hello from main program with space in its paths" ) Nuitka-0.5.28.2/tests/programs/stdlib_overload/0000755000372000001440000000000013207540420021630 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/stdlib_overload/some_package/0000755000372000001440000000000013207540420024246 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/stdlib_overload/some_package/__init__.py0000644000372000001440000000140213112214770026355 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/stdlib_overload/some_package/star_importing.py0000644000372000001440000000166413112214770027671 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( "some_package.star_importing, doing the star import" ) print("Before", sorted(dir())) from .pyexpat import * lala = 1 print("After", sorted(dir())) print( "Finished" ) Nuitka-0.5.28.2/tests/programs/stdlib_overload/some_package/normal_importing.py0000644000372000001440000000153113112214770030201 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import pyexpat print( "Imported pyexpat, should use our one." ) print( dir(pyexpat) ) Nuitka-0.5.28.2/tests/programs/stdlib_overload/some_package/pyexpat.py0000644000372000001440000000145413112214770026317 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # defined_in_pyexpat_subpackage = "see me" Nuitka-0.5.28.2/tests/programs/stdlib_overload/StdlibOverloadMain.py0000644000372000001440000000222413112214770025725 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Main importing nearby package") import pyexpat try: print(pyexpat.defined_in_pyexpat ) except AttributeError: print("Must be Python3, where absolute imports are default.") print( "Main importing from package doing star import" ) from some_package import star_importing print( "Main importing from package doing normal import" ) from some_package import normal_importing print( "Done." ) Nuitka-0.5.28.2/tests/programs/stdlib_overload/pyexpat.py0000644000372000001440000000144113112214770023675 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # defined_in_pyexpat = "see me" Nuitka-0.5.28.2/tests/programs/absolute_import/0000755000372000001440000000000013207540420021664 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/absolute_import/foobar/0000755000372000001440000000000013207540420023134 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/absolute_import/foobar/local.py0000644000372000001440000000144713112214770024607 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("This is for relative import.") Nuitka-0.5.28.2/tests/programs/absolute_import/foobar/foobar.py0000644000372000001440000000166013112214770024762 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import absolute_import, print_function from foobar import util from . import local class Foobar(object): def __init__(self): print(util.foo()) Nuitka-0.5.28.2/tests/programs/absolute_import/foobar/util.py0000644000372000001440000000143613112214770024470 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def foo(): return "hi" Nuitka-0.5.28.2/tests/programs/absolute_import/foobar/__init__.py0000644000372000001440000000143613112214770025252 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from .foobar import Foobar Nuitka-0.5.28.2/tests/programs/absolute_import/AbsoluteImportMain.py0000644000372000001440000000150113112214770026012 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import foobar if __name__ == "__main__": foobar.Foobar() Nuitka-0.5.28.2/tests/programs/deep/0000755000372000001440000000000013207540420017371 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/deep/some_package/0000755000372000001440000000000013207540420022007 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/deep/some_package/deep_package/0000755000372000001440000000000013207540420024377 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/deep/some_package/deep_package/DeepDeepChild.py0000644000372000001440000000155613112214770027400 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print( "This is DeepDeepChild talking." ) from .. import DeepBrother Nuitka-0.5.28.2/tests/programs/deep/some_package/deep_package/__init__.py0000644000372000001440000000153713207537242026526 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("The deep package has __package__", __package__) Nuitka-0.5.28.2/tests/programs/deep/some_package/DeepBrother.py0000644000372000001440000000172213112214770024567 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("This is deep brother module talking.", __name__) def someBrotherFunction(): pass print("The __module__ of function here is", someBrotherFunction.__module__) Nuitka-0.5.28.2/tests/programs/deep/some_package/DeepChild.py0000644000372000001440000000161513112214770024206 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print( "Importing child." ) class A: pass print("Class defined here, has these vars", vars(A)) Nuitka-0.5.28.2/tests/programs/deep/some_package/__init__.py0000644000372000001440000000140213112214770024116 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/programs/deep/DeepProgramMain.py0000644000372000001440000000154113112214770022757 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import some_package.DeepChild import some_package.deep_package.DeepDeepChild print( "Done." ) Nuitka-0.5.28.2/tests/programs/syntax_errors/0000755000372000001440000000000013207540420021376 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/programs/syntax_errors/IndentationErroring.py0000644000372000001440000000142713112214770025741 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): x y Nuitka-0.5.28.2/tests/programs/syntax_errors/SyntaxErroring.py0000644000372000001440000000144013112214770024746 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class x(metaclass=y): pass Nuitka-0.5.28.2/tests/programs/syntax_errors/Main.py0000644000372000001440000000210213112214770022630 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import os try: from SyntaxErroring import x except Exception as e: print("Importing with syntax error gave:", type(e), e) try: from IndentationErroring import x except Exception as e: print("Importing with indentation error gave:", type(e), e) print("Finished.") Nuitka-0.5.28.2/tests/syntax/0000755000372000001440000000000013207540420016150 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/syntax/run_all.py0000755000372000001440000000354613122472300020166 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os import sys # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( my_print, setup, compareWithCPython, createSearchMode, decideFilenameVersionSkip ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() for filename in sorted(os.listdir('.')): if not filename.endswith(".py"): continue if not decideFilenameVersionSkip(filename): continue active = search_mode.consider( dirname = None, filename = filename ) if active: extra_flags = ["expect_failure", "remove_output", "syntax_errors"] compareWithCPython( dirname = None, filename = filename, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = False ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.28.2/tests/syntax/FutureUnknown.py0000644000372000001440000000144513112214771021362 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import not_existing Nuitka-0.5.28.2/tests/syntax/NonlocalNotFound32.py0000644000372000001440000000154213112214771022115 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def testNonlocal(): x = 0 y = 0 def f(): nonlocal z f() testNonlocal() Nuitka-0.5.28.2/tests/syntax/DuplicateArgument.py0000644000372000001440000000142713112214771022145 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(a, a): pass Nuitka-0.5.28.2/tests/syntax/ModuleReturn.py0000644000372000001440000000144113112214771021151 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # lalala = 3 return lelele = 7 Nuitka-0.5.28.2/tests/syntax/ExecWithNesting_2.py0000644000372000001440000000212713112214771022017 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # In Python2, it is not allowed to take closure variables, in a function that # has an "exec". For Python3, the problem doesn't exist, as there is no exec # statement anymore. def someFunctionWithUnqualifiedExecAndCallback(): exec "def f(): pass" def callback(): return nested someFunctionWithUnqualifiedExecAndCallback() Nuitka-0.5.28.2/tests/syntax/MisplacedFutureImport.py0000644000372000001440000000151213112214771023012 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): from __future__ import print_function print(locals()) Nuitka-0.5.28.2/tests/syntax/NonAsciiWithoutEncoding_2.py0000644000372000001440000000156313112214771023510 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # No encoding declared, but non-ASCII in that string. s = "'\0\"\n\r\t abcd\x85é\U00012fff\uD800\U0001D121xxx." Nuitka-0.5.28.2/tests/syntax/YieldInModule.py0000644000372000001440000000144613112214771021234 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # x = (i for i in (yield) if (yield)) Nuitka-0.5.28.2/tests/syntax/FutureBraces.py0000644000372000001440000000147413112214771021124 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import braces print( "Look ma, braces." ) Nuitka-0.5.28.2/tests/syntax/NonlocalForParameter32.py0000644000372000001440000000143313112214771022747 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(a): nonlocal a Nuitka-0.5.28.2/tests/syntax/BreakWithoutLoop.py0000644000372000001440000000154213112214771021770 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # marker1 def test_from_format(self): if sys.version_info >= (3, 3): break # marker2 Nuitka-0.5.28.2/tests/syntax/AsyncgenReturn36.py0000644000372000001440000000145113122472300021640 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # async def gen(): yield 1 return 2 Nuitka-0.5.28.2/tests/syntax/Importing32.py0000644000372000001440000000174713112214771020652 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def starImportFailure(): from doctest import * try: sys print( "but it does not" ) except NameError: print( "and it does" ) print("Star import needs to respect __all__", starImportFailure()) Nuitka-0.5.28.2/tests/syntax/LateFutureImport.py0000644000372000001440000000160413112214771022000 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Not allowed to do future imports that are not the first statements of the # module. a = 1 from __future__ import print_function Nuitka-0.5.28.2/tests/syntax/IndentationError.py0000644000372000001440000000143713112214771022017 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def someFunc(): a b Nuitka-0.5.28.2/tests/syntax/TryFinallyContinue.py0000644000372000001440000000155313112214771022332 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): for i in range(10): try: undefined finally: continue Nuitka-0.5.28.2/tests/syntax/UnpackNoTuple.py0000644000372000001440000000141013112214771021250 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # *a = 1 Nuitka-0.5.28.2/tests/syntax/ClosureDel_2.py0000644000372000001440000000171213112214771021007 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def deletingClosure(): a = 1 def closureTaker(): return a del a try: x = closureTaker() except Exception as e: print "Occurred %r" % e deletingClosure() Nuitka-0.5.28.2/tests/syntax/TryExceptAllNotLast.py0000644000372000001440000000153113112214771022411 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(): try: raise A except: print("caught") except A: print("hit") Nuitka-0.5.28.2/tests/syntax/YieldInAsync35.py0000644000372000001440000000144413112214771021232 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # async def g(): yield from None Nuitka-0.5.28.2/tests/syntax/ClassReturn.py0000644000372000001440000000142713112214771020775 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class X: return 3 Nuitka-0.5.28.2/tests/syntax/StarImportExtra.py0000644000372000001440000000157713112214771021646 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # try: from sys import not_there, * except Exception as e: print( "Star import with extra stuff not present gave", e ) Nuitka-0.5.28.2/tests/syntax/GlobalForParameter.py0000644000372000001440000000143113112214771022233 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def f(a): global a Nuitka-0.5.28.2/tests/syntax/UnpackTwoStars32.py0000644000372000001440000000145113112214771021622 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # foo, *bar, baz, *a, b = name.split('/') Nuitka-0.5.28.2/tests/syntax/SyntaxError.py0000644000372000001440000000154513112214771021031 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Test the syntax error case: def nested(): claxss ProxyBase(metaclass=ProxyType): pass Nuitka-0.5.28.2/tests/syntax/YieldFromInModule.py0000644000372000001440000000150213112214771022051 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # x = (i for i in (yield from range(7)) if (yield from range(2))) Nuitka-0.5.28.2/tests/syntax/GeneratorReturn_2.py0000644000372000001440000000155713112214771022103 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def g(): for a in range(3): yield a return 7 print("Yielder with return value", list(g())) Nuitka-0.5.28.2/tests/syntax/ContinueWithoutLoop.py0000644000372000001440000000152213112214771022526 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def test_from_format(self): if sys.version_info >= (3, 3): continue Nuitka-0.5.28.2/tests/basics/0000755000372000001440000000000013207540420016066 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/basics/run_all.py0000755000372000001440000001113313122472300020073 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os import sys # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) # Find nuitka package from system installation for package test too. try: import nuitka # @UnusedImport except ImportError: # Find nuitka in system install. sys.path[0] = "/usr/share/nuitka" import nuitka # @UnusedImport from nuitka.tools.testing.Common import ( my_print, setup, decideFilenameVersionSkip, compareWithCPython, hasDebugPython, withPythonPathChange, createSearchMode ) python_version = setup(needs_io_encoding = True) search_mode = createSearchMode() if python_version >= "3.4": # These tests don't work with 3.4 yet, and the list is considered the major # TODO for 3.4 support. search_mode.mayFailFor( # Prepared dictionaries of "enum.Enums" are not used early enough "Classes34.py", ) # Create large constants test on the fly, if it's not there, not going to # add it to release archives for no good reason. if not os.path.exists("BigConstants.py"): with open("BigConstants.py", 'w') as output: output.write( "# Automatically generated test, not part of releases or git.\n\n" ) output.write( "print('%s')\n" % ("1234" * 17000) ) # Now run all the tests in this directory. for filename in sorted(os.listdir('.')): if not filename.endswith(".py"): continue if not decideFilenameVersionSkip(filename): continue extra_flags = [ # No error exits normally, unless we break tests, and that we would # like to know. "expect_success", # Keep no temporary files. "remove_output", # Include imported files, mostly nothing though. "recurse_all", # Use the original __file__ value, at least one case warns about things # with filename included. "original_file" ] # This test should be run with the debug Python, and makes outputs to # standard error that might be ignored. if filename.startswith("Referencing"): extra_flags.append("python_debug") extra_flags.append("recurse_not:nuitka") # This tests warns about __import__() used. if filename == "OrderChecks.py": extra_flags.append("ignore_warnings") # This tests warns about an package relative import despite # being in no package. if filename == "Importing.py": extra_flags.append("ignore_warnings") # TODO: Nuitka does not give output for ignored exception in dtor, this is # not fully compatible and potentially an error. if filename == "YieldFrom33.py": extra_flags.append("ignore_stderr") # For Python2 there is a "builtins" package that gives warnings. TODO: We # ought to NOT import that package and detect statically that __builtins__ # import won't raise ImportError. if filename == "BuiltinOverload.py": extra_flags.append("ignore_warnings") active = search_mode.consider( dirname = None, filename = filename ) if active: if filename.startswith("Referencing") and not hasDebugPython(): my_print("Skipped (no debug Python)") continue needs_2to3 = python_version.startswith('3') and \ not filename.endswith("32.py") and \ not filename.endswith("33.py") with withPythonPathChange(".."): compareWithCPython( dirname = None, filename = filename, extra_flags = extra_flags, search_mode = search_mode, needs_2to3 = needs_2to3 ) else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.28.2/tests/basics/InplaceOperations.py0000644000372000001440000000241413112214770022061 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function x = 1 x += 2 print("Plain in-place:", x) z = [1, 2, 3] z[1] += 5 print("List in-place:", z[1]) h = { 'a' : 3 } h['a'] += 2 print("Dictionary in-place:", h['a']) class B: a = 1 B.a += 2 print("Class attribute in-place:", B.a) h = [1, 2, 3, 4] h[1:2] += (2,3) print("List 'sclice' in-place [x:y]", h) h[:1] += (9,9) print("List 'sclice' in-place [:y]", h) h[2:] += (6,6) print("List 'sclice' in-place [y:]", h) h[:] += (5,5,5) print("List 'sclice' in-place [:]", h) Nuitka-0.5.28.2/tests/basics/Importing.py0000644000372000001440000000460113112214770020412 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def localImporter1(): import os return os def localImporter1a(): import os as my_os_name return my_os_name def localImporter2(): from os import path return path def localImporter2a(): from os import path as renamed return renamed print("Direct module import", localImporter1()) print("Direct module import using rename", localImporter1a()) print("From module import", localImporter2()) print("From module import using rename", localImporter2a()) from os import * print("Star import gave us", path) import os.path as myname print("As import gave", myname) def localImportFailure(): try: from os import path, lala, listdir except Exception as e: print("gives", type(e), repr(e)) try: print(listdir) except UnboundLocalError: print("and listdir was not imported", end = ' ') print("but path was", path) print("From import that fails in the middle", end = ' ') localImportFailure() def nonPackageImportFailure(): try: # Not allowed without being a package, should raise ValueError from . import whatever except Exception as e: print(type(e), repr(e)) print("Package import fails in non-package:", end = ' ') nonPackageImportFailure() def importBuiltinTupleFailure(): try: value = "something", # Not allowed to not be constant string, optimization might be fooled # though. __import__(value) except Exception as e: print(type(e), repr(e)) print("The __import__ built-in optimization can handle tuples:", end = ' ') importBuiltinTupleFailure() Nuitka-0.5.28.2/tests/basics/TryContinueFinally.py0000644000372000001440000000406113112214770022244 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def tryWhileContinueFinallyTest(): print("Check if finally is executed in a continue using for loop:") x = 0 while x < 10: x += 1 try: if x % 2 == 1: continue finally: print(x, end = ' ') print('-', end = ' ') print() def tryForContinueFinallyTest(): print("Check if finally is executed in a continue using for loop:") for x in range(10): try: if x % 2 == 1: continue finally: print(x, end = ' ') print('-', end = ' ') print() def tryWhileBreakFinallyTest(): print("Check if finally is executed in a break using while loop:") x = 0 while x < 10: x += 1 try: if x == 5: break finally: print(x, end = ' ') print('-', end = ' ') print() def tryForBreakFinallyTest(): print("Check if finally is executed in a break using for loop:") for x in range(10): try: if x == 5: break finally: print(x, end = ' ') print('-', end = ' ') print() tryWhileContinueFinallyTest() tryWhileBreakFinallyTest() tryForContinueFinallyTest() tryForBreakFinallyTest() Nuitka-0.5.28.2/tests/basics/Constants27.py0000644000372000001440000000174513112214770020575 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("Clash in constants reveals order:") x = {True, 1} print(x) print("Complex constant using sets with clashes at run time:") y = True x = ({1, }, {1, True}, {1, 1.0}, {1, y}) print(x) Nuitka-0.5.28.2/tests/basics/Inspection.py0000644000372000001440000000776313112214770020571 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import inspect, types, sys, pprint def displayDict(d, remove_keys = ()): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" for remove_key in remove_keys: if remove_key in d: d = dict(d) del d[remove_key] return pprint.pformat(d) def compiledFunction(a, b): pass assert inspect.isfunction(compiledFunction) is True assert isinstance(compiledFunction, types.FunctionType) assert isinstance(compiledFunction, (int, types.FunctionType)) print("Compiled spec:", inspect.getargspec(compiledFunction)) print("Compiled args:", inspect.formatargspec(*inspect.getargspec(compiledFunction))) # Even this works. assert type(compiledFunction) == types.FunctionType class CompiledClass: def __init__(self): pass def compiledMethod(self): pass assert inspect.isfunction(CompiledClass) is False assert isinstance(CompiledClass, types.FunctionType) is False assert inspect.ismethod(compiledFunction) is False assert inspect.ismethod(CompiledClass) is False assert inspect.ismethod(CompiledClass.compiledMethod) == ( sys.version_info < (3,) ) assert inspect.ismethod(CompiledClass().compiledMethod) is True assert bool(type(CompiledClass.compiledMethod) == types.MethodType) == ( sys.version_info < (3,) ) print("Compiled method:", inspect.getargspec(CompiledClass().compiledMethod)) print("Compiled class:", inspect.formatargspec(*inspect.getargspec(CompiledClass().compiledMethod))) def compiledGenerator(): yield 1 assert inspect.isfunction(compiledGenerator) is True assert inspect.isgeneratorfunction(compiledGenerator) is True assert isinstance(compiledGenerator(), types.GeneratorType) is True assert type(compiledGenerator()) == types.GeneratorType assert isinstance(compiledGenerator, types.GeneratorType) is False assert inspect.ismethod(compiledGenerator()) is False assert inspect.isfunction(compiledGenerator()) is False assert inspect.isgenerator(compiledFunction) is False assert inspect.isgenerator(compiledGenerator) is False assert inspect.isgenerator(compiledGenerator()) is True def someFunction(a): assert inspect.isframe(sys._getframe()) # print("Running frame getframeinfo()", inspect.getframeinfo(sys._getframe())) # TODO: The locals of the frame are not updated. # print("Running frame arg values", inspect.getargvalues(sys._getframe())) someFunction(2) class C: print("Class locals", displayDict(sys._getframe().f_locals, remove_keys = ("__qualname__","__locals__"))) print("Class flags", sys._getframe().f_code.co_flags) def f(): print("Func locals", sys._getframe().f_locals) print("Func flags", sys._getframe().f_code.co_flags) f() def g(): yield("Generator object locals", sys._getframe().f_locals) yield("Generator object flags", sys._getframe().f_code.co_flags) for line in g(): print(*line) print("Generator function flags", g.__code__.co_flags) print("Module frame locals", displayDict(sys._getframe().f_locals)) print("Module flags", sys._getframe().f_code.co_flags) print("Module code name", sys._getframe().f_code.co_name) print("Module frame dir", dir(sys._getframe())) Nuitka-0.5.28.2/tests/basics/Operators.py0000644000372000001440000000371113112214770020421 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # a = 3 b = 7 c = [7, 8] d = 15 print '+', a + b print '-', a - b print '*', a * b print '/', a / b print "//", a // b print '%', b % a print "& (2)", a & b print "| (2)", a | b print "& (3)", a & b & d print "| (3)", a | b | d print "^ (2)", a ^ b print "^ (3)", a ^ b ^ d print "**", a ** b print "<<", a << b print ">>", b >> a print "in", b in c print "not in", b not in c print '<', a < b print '>', a > b print "==", a == b print "<=", a <= b print ">=", a >= b print "!=", a != b print "is", a is b print "is not", a is not b print '~', ~ b print '-', - b print '+', + b l = {('a', 'c') : "a,c", 'b' : 2, 'c' : 3, 'd' : 4 } l[ 'l', ] = '6' print "Extended slicing:" print "Should be a,c:", l[ 'a', 'c'] print "Short form of extended slicing:" d = {} # d[1] = 1 d[1,] = 2 d[1,2] = 3 d[1,2,3] = 4 L = list(d) L.sort() print L s = "Some information" ss = s[-1] print "Constant subscript of string", ss print "Repr" print `L`, `ss` print `0L` print repr(L), repr(ss) print repr(3L) print "Slicing on a list:" l = [1, 3, 5, 7, 11, 13, 17] print l[None:None] n = None print l[n:n] print l[3:n] print l[n:3] value = None try: x = value[1] except Exception as e: print "Indexing None gives", repr(e) Nuitka-0.5.28.2/tests/basics/ExtremeClosure.py0000644000372000001440000000256513112214770021417 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function a = 1 b = 1 def someFunction(): a = a class someClass(): b = b someClass() try: someFunction() except UnboundLocalError as e: print("Expected unbound local error occurred:", repr(e)) try: class anotherClass(): b = undefined_global # @UndefinedVariable except NameError as e: print("Expected name error occurred:", repr(e)) # TODO: This is not passing yet. if False: try: class yetanotherClass(): b = 1 del b print(b) except NameError as e: print("Expected name error occurred:", repr(e)) Nuitka-0.5.28.2/tests/basics/Functions32.py0000644000372000001440000000656313112214770020570 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def displayDict(d): result = '{' for key, value in sorted(d.items()): result += "%s: %s" % (key, value) result += '}' def kwonlysimple(*, a): return a print("Keyword only function case: ", kwonlysimple(a = 3)) def kwonlysimpledefaulted(*, a = 5): return a print("Keyword only function, using default value: ", kwonlysimpledefaulted()) def default1(): print("Called", default1) return 1 def default2(): print("Called", default2) return 2 def default3(): print("Called", default3) return 3 def default4(): print("Called", default4) return 4 def annotation1(): print("Called", annotation1) return "a1" def annotation2(): print("Called", annotation2) return "a2" def annotation3(): print("Called", annotation3) return "a3" def annotation4(): print("Called", annotation4) return "a4" def annotation5(): print("Called", annotation5) return "a5" def annotation6(): print("Called", annotation6) return "a6" def annotation7(): print("Called", annotation7) return "a7" def annotation8(): print("Called", annotation8) return "a8" def annotation9(): print("Called", annotation9) return "a9" print("Defining function with annotations, and defaults as functions for everything:") def kwonlyfunc(x: annotation1(), y: annotation2() = default1(), z: annotation3() = default2(), *, a: annotation4(), b: annotation5() = default3(), c: annotation6() = default4(), d: annotation7(), **kw: annotation8()) -> annotation9(): print(x, y, z, a, b, c, d) print("__kwdefaults__", displayDict(kwonlyfunc.__kwdefaults__)) print("Keyword only function called:") kwonlyfunc( 7, a = 8, d = 12 ) print("OK.") print("Annotations come out as", sorted( kwonlyfunc.__annotations__ )) kwonlyfunc.__annotations__ = {} print("After updating to None it is", kwonlyfunc.__annotations__) kwonlyfunc.__annotations__ = { "k" : 9 } print("After updating to None it is", kwonlyfunc.__annotations__) def kwonlystarfunc(*, a, b, **d): return a, b, d print("kwonlystarfunc", kwonlystarfunc(a = 8, b = 12, k = 9, j = 7)) def deeplyNestedNonLocalWrite(): x = 0 y = 0 def f(): def g(): nonlocal x x = 3 return x return g() return f(), x print("Deeply nested non local writing function", deeplyNestedNonLocalWrite()) def deletingClosureVariable(): try: x = 1 def g(): nonlocal x del x g() g() except Exception as e: return repr(e) print("Using deleted non-local variable:", deletingClosureVariable()) Nuitka-0.5.28.2/tests/basics/Referencing_2.py0000644000372000001440000000446013122472300021111 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython ) checkDebugPython() x = 17 # Python2 only syntax things are here. def simpleFunction1(): try: raise TypeError, (3,x,x,x) except TypeError: pass def simpleFunction2(): try: raise ValueError(1,2,3), ValueError(1,2,3) except Exception: pass def simpleFunction3(): try: raise ValueError, 2, None except Exception: pass def simpleFunction4(): try: raise ValueError, 2, 3 except Exception: pass def simpleFunction5(): def nested_args_function((a,b), c): return a, b, c nested_args_function((1, 2), 3) def simpleFunction6(): def nested_args_function((a,b), c): return a, b, c try: nested_args_function((1,), 3) except ValueError: pass def simpleFunction7(): def nested_args_function((a,b), c): return a, b, c try: nested_args_function((1, 2, 3), 3) except ValueError: pass # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/ExecEval.py0000644000372000001440000001522113112214770020136 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import tempfile, sys, os print "eval 3+3=", eval("3+3") print "eval 4+4=", eval(" 4+4") def functionEval1(): return eval(" 5+5") print "eval in a function with nothing provided", functionEval1() def functionEval2(): a = [2] g = {} r = eval("1+3", g) return r, g.keys(), a print "eval in a function with globals provided", functionEval2() def functionEval3(): result = [] for x in eval("(1,2)"): result.append(x) return result print "eval in a for loop as iterator giver", functionEval3() print "exec on a global level", exec( "d=2+2" ) print "2+2=",d # @UndefinedVariable def functionExec1(): a = 1 code = "a=2" exec( code ) return a def functionExec2(): a = 1 code = "a=2" exec code in globals(), locals() return a print "exec in function without and with locals() provided:", functionExec1(), functionExec2() tmp_filename = tempfile.gettempdir() + "/execfile.py" f = open(tmp_filename, 'w') f.write("e=7\nf=8\n") f.close() execfile(tmp_filename) print "execfile with defaults f,g=", e, f # @UndefinedVariable global_vars = { 'e' : '0', 'f' : 0 } local_vars = dict(global_vars) execfile(tmp_filename, global_vars) print "execfile with globals dict:", global_vars.keys() execfile(tmp_filename, global_vars, local_vars) print "execfile with globals and locals dict:", local_vars def functionExecfile(): e = 0 f = 0 global_vars = { 'e' : '0', 'f' : 0 } local_vars = dict(global_vars) print "execfile with globals and locals dict in a function:", x = execfile(tmp_filename, global_vars, local_vars) print x, print global_vars.keys(), local_vars, e, f functionExecfile() class classExecfile: e = 0 f = 0 print "execfile in a class:", # TODO: Won't work yet, Issue#5 # print execfile( tmp_filename ), execfile(tmp_filename) print "execfile changed local values:", e, f f = 7 def functionExecNonesTuple(): f = 0 exec("f=1", None, None) print "Exec with None as optimizable tuple args did update locals:", f def functionExecNonesSyntax(): f = 0 exec "f=2" in None, None print "Exec with None as optimizable normal args did update locals:", f functionExecNonesTuple() functionExecNonesSyntax() print "Global value is untouched", f def functionEvalNones2(): f = 11 code = 'f' g = None l = None f1 = eval (code, l, g) print "Eval with None arguments from variables did access locals:", f1 functionEvalNones2() def functionExecNonesTuple2(): f = 0 code = "f=1" g = None l = None exec(code, l, g) print "Exec with None as tuple args from variable did update locals:", f def functionExecNonesSyntax2(): f = 0 code = "f=2" g = None l = None exec code in l, g print "Exec with None as normal args did update locals:", f functionExecNonesTuple2() functionExecNonesSyntax2() print "Exec with a future division definition and one without:" exec """ from __future__ import division from __future__ import print_function print( "3/2 is with future division", 3/2 ) """ exec """ from __future__ import print_function print( "3/2 is without future division", 3/2 ) """ x = 1 y = 1 def functionGlobalsExecShadow(): global x print "Global x outside is", x y = 0 print "Local y is initially", y print "Locals initially", locals() exec """ from __future__ import print_function x = 2 print( "Exec local x is", x ) """ print "Function global x referenced as local x in exec is", x exec """ from __future__ import print_function print( "Re-exec local x", x ) """ print "Locals after exec assigning to local x", locals() exec """ from __future__ import print_function global x x = 3 print( "Exec global x is inside exec", x ) """ print "Global x referenced as global x in exec is", x exec """ from __future__ import print_function def change_y(): global y y = 4 print( "Exec function global y is", y ) y = 7 change_y() # TODO: The below will not work print( "Exec local y is", y ) """ # print "Local y is afterwards", y def print_global_y(): global y # TODO: The below will not work print "Global y outside", y print_global_y() print "Outside y", y functionGlobalsExecShadow() def functionWithClosureProvidedByExec(): code = "ValueError = TypeError" exec code in None, None def func(): print "Closure from exec not used", ValueError func() functionWithClosureProvidedByExec() x = 2 def functionWithExecAffectingClosure(): x = 4 code = "d=3;x=5" space = locals() exec code in space def closureMaker(): return x return d, closureMaker() # @UndefinedVariable print "Closure in a function with exec to not none", functionWithExecAffectingClosure() def generatorFunctionWithExec(): yield 1 code = "y = 2" exec code yield y print "Exec in a generator function", tuple(generatorFunctionWithExec()) def evalInContractions(): r1 = list( eval(str(s)) for s in range(3) ) r2 = [ eval(str(s)) for s in range(4) ] return r1, r2 print "Eval in a list contraction or generator expression", evalInContractions() def execDefinesFunctionToLocalsExplicity(): exec """\ def makeAddPair(a, b): def addPair(c, d): return (a + c, b + d) return addPair """ in locals() if sys.version_info < (3,): assert makeAddPair # @UndefinedVariable return "yes" print "Exec adds functions declares in explicit locals() given.", execDefinesFunctionToLocalsExplicity() os.unlink(tmp_filename) def execWithShortTuple(): try: exec("print hey",) except Exception as e: return "gives exception: " + repr(e) print "Exec with too short tuple argument:", execWithShortTuple() if str is not bytes: def evalMemoryView(value): return eval(memoryview(value)) print "Eval with memory view:", evalMemoryView(b"27") Nuitka-0.5.28.2/tests/basics/Classes.py0000644000372000001440000001123213122472300020031 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" # Avoid recursion that we don't offer for classes. if "__locals__" in d: d = dict(d) del d["__locals__"] import pprint return pprint.pformat(d) class SimpleClass: " The class documentation." # Leading space on purpose. # TODO: Doesn't work with Python3, because we don't yet make our own dict # visible. print locals() print "Class locals, while building", displayDict(locals()) class_var = 1 def __init__(self, init_parameter): self.x = init_parameter def normal_method(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 return self.arg1, self.arg2, self.x, self @staticmethod def static_method(): return "something" print "Simple class:", SimpleClass print "Lives in", SimpleClass.__module__ print "Documentation", SimpleClass.__doc__ print "Instantiate simple class:", SimpleClass(14) print "Call simple class normal method:", SimpleClass(11).normal_method(1, 2) print "Call simple class static method:", SimpleClass(11).static_method() class MetaClass(type): def __init__(cls, name, bases, dictionary): print "MetaClass is called." cls.addedin = 5 print MetaClass class ComplexClass: __metaclass__ = MetaClass print ComplexClass, dir(ComplexClass) print ComplexClass, hasattr(ComplexClass, "addedin") and ComplexClass.addedin def function(): x = 1 class DynamicClass: y = x x = 2 return DynamicClass print function(), function().y def strangeClassBehaviour(): class StrangeClass(object): count = 0 def __new__(cls): print "__new__" cls.count += 1 return object.__new__(cls) def __del__(self): print "__del__" cls = self.__class__ cls.count -= 1 assert cls.count >= 0 x = StrangeClass() return x.count print "Strange class with __new__ and __del__ overloads", strangeClassBehaviour() class ClosureLocalizer: function = function # Using what looks like a method as a decorator. def deco(f): # @NoSelf f.decorated = True return f @deco def x(self): pass print "Class with a name from module level renamed to local", ClosureLocalizer.function print "Class method decorated", ClosureLocalizer().x.decorated print "Class with decorator and meta class:" def classdecorator(cls): print "cls decorator", cls.addedin return cls @classdecorator class MyClass: __metaclass__ = MetaClass print "Class that updates its locals:", class DictUpdating: a = 1 locals().update({ 'b' : 2 }) for f in range(6): locals()[ "test_%s" % f ] = f print "Changed values", DictUpdating.b, DictUpdating.test_4 def functionThatOffersClosureToPassThroughClass(x): class Foo: global x x = 1 def __call__(self, y): return x + y return Foo() print functionThatOffersClosureToPassThroughClass(6)(2), print x class NameCollisionClosure: def x(self): return x print NameCollisionClosure, NameCollisionClosure().x() class ClassesWithNestedClass: class NestedClass(object): def getDict(self): return { 'a':2 } print "Classes:" print ClassesWithNestedClass, print ClassesWithNestedClass().NestedClass, print ClassesWithNestedClass().NestedClass().getDict() secondary = "global closure wins" class ClassWithModuleVariableCollisionMain(): secondary = None def __init__(self): self.secondary = self.Child() self.value = self.secondary.attr class Child(): def __init__(self): self.attr = secondary print ClassWithModuleVariableCollisionMain, ClassWithModuleVariableCollisionMain().value Nuitka-0.5.28.2/tests/basics/OrderChecks.py0000644000372000001440000003601413112214770020641 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def separator(): print('*' * 80) def dictOrderCheck(): def key1(): print("key1 called") return 1 def key2(): print("key2 called") return 2 def value1(): print("value1 called") return 11 def value2(): print("value2 called") return 22 print("Checking order of calls in dictionary creation from callables:") print( { key1() : value1(), key2() : value2() } ) try: (1/0)[1.0j/0] = (1.0/0) except ZeroDivisionError as e: print("Expected exception caught:", repr(e)) try: (1/0)[1.0/0] = 1 except ZeroDivisionError as e: print("Expected exception caught:", repr(e)) try: (1/0)[1] = (1.0/0) except ZeroDivisionError as e: print("Expected exception caught:", repr(e)) def listOrderCheck(): def value1(): print("value1 called") return 11 def value2(): print("value2 called") return 22 print([value1(), value2() ]) def sliceOrderCheck(): print("Slices:") d = list(range(10)) def lvalue(): print("lvalue", end = ' ') return d def rvalue(): print("rvalue", end = ' ') return range(2) def rvalue4(): print("rvalue", end = ' ') return range(4) def low(): print("low", end = ' ') return 0 def high(): print("high", end = ' ') return 4 def step(): print("step", end = ' ') return 2 print("Complex slice lookup:", end = ' ') print(lvalue()[ low() : high() : step() ]) print("Complex slice assignment:", end = ' ') lvalue()[ low() : high() : step() ] = rvalue() print(d) print("Complex slice del:", end = ' ') del lvalue()[ low() : high() : step() ] print(d) print("Complex inplace slice operation", end = ' ') # TODO: This gives an error in CPython, but not in Nuitka. # lvalue()[ low() : high() : step() ] += rvalue() print(d) d = list(range(10)) print("Simple slice lookup", end = ' ') print(lvalue()[ low() : high() ]) print("Simple slice assignment", end = ' ') lvalue()[ 3 + low() : 3 + high() ] = rvalue() print(d) print("Simple slice del", end = ' ') del lvalue()[ 3 + low() : 3 + high() ] print(d) print("Simple inplace slice operation", end = ' ') lvalue()[ low() : high() ] += rvalue4() print(d) def subscriptOrderCheck(): print("Subscripts:") d={} def lvalue(): print("lvalue", end = ' ') return d def rvalue(): print("rvalue", end = ' ') return 2 def subscript(): print("subscript", end = ' ') return 1 print("Assigning subscript:") lvalue()[ subscript() ] = rvalue() print(d) print("Lookup subscript:") print(lvalue()[ subscript() ]) print("Deleting subscript:") del lvalue()[ subscript() ] print(d) def attributeOrderCheck(): def lvalue(): print("lvalue", end = ' ') return lvalue def rvalue(): print("rvalue", end = ' ') return 2 print("Attribute assignment order:") lvalue().xxx = rvalue() print("Assigned was indeed:", lvalue.xxx) print("Checking attribute assignment to unassigned value from unassigned:") try: undefined_global_zzz.xxx = undefined_global_yyy # @UndefinedVariable except Exception as e: print("Expected exception caught:", repr(e)) else: assert False try: (1/0).x = (1.0/0) except ZeroDivisionError as e: print("Expected exception caught:", repr(e)) def compareOrderCheck(): def lvalue(): print("lvalue", end = ' ') return 1 def rvalue(): print("rvalue", end = ' ') return 2 print("Comparisons:") print("==", lvalue() == rvalue()) print("<=", lvalue() <= rvalue()) print(">=", lvalue() >= rvalue()) print("!=", lvalue() != rvalue()) print('>', lvalue() > rvalue()) print('<', lvalue() < rvalue()) print("Comparison used in bool context:") print("==", "yes" if lvalue() == rvalue() else "no") print("<=", "yes" if lvalue() <= rvalue() else "no") print(">=", "yes" if lvalue() >= rvalue() else "no") print("!=", "yes" if lvalue() != rvalue() else "no") print('>', "yes" if lvalue() > rvalue() else "no") print('<', "yes" if lvalue() < rvalue() else "no") def operatorOrderCheck(): def left(): print("left operand", end = ' ') return 1 def middle(): print("middle operand", end = ' ') return 3 def right(): print("right operand", end = ' ') return 2 print("Operations:") print('+', left() + middle() + right()) print('-', left() - middle() - right()) print('*', left() * middle() * right()) print('/', left() / middle() / right()) print('%', left() % middle() % right()) print("**", left() ** middle() ** right()) def generatorOrderCheck(): print("Generators:") def default1(): print("default1", end = ' ') return 1 def default2(): print("default2", end = ' ') return 2 def default3(): print("default3", end = ' ') return 3 def value(x): print("value", x, end = ' ') return x def generator(a = default1(), b = default2(), c = default3()): print("generator entry") yield value(a) yield value(b) yield value(c) print("generator exit") result = list(generator()) print("Result", result) def classOrderCheck(): print("Checking order of class constructions:") class B1: pass class B2: pass def base1(): print("base1", end = ' ') return B1 def base2(): print("base2", end = ' ') return B2 def deco1(cls): print("deco1", end = ' ') return cls def deco2(cls): print("deco2") return B2 @deco2 @deco1 class X(base1(), base2()): print("class body", end = ' ') print def inOrderCheck(): print("Checking order of in operator:") def container(): print("container", end = ' ') return [3] def searched(): print("searched", end = ' ') return 3 print("in:", searched() in container()) print("not in:", searched() not in container()) def unpackOrderCheck(): class TraceRelease: def __init__(self, value): self.value = value def __del__(self): print("Deleting iteration value", self.value) pass print("Unpacking values:") class Iterable: def __init__(self): self.consumed = 2 def __iter__(self): return Iterable() def __del__(self): print("Deleted iterable with", self.consumed) def next(self): print("Next with state", self.consumed) if self.consumed: self.consumed -=1 else: raise StopIteration return TraceRelease(self.consumed) __next__ = next iterable = Iterable() class RejectAttributeOwnership: def __setattr__(self, key, value): print("Setting", key, value.value) try: RejectAttributeOwnership().x, RejectAttributeOwnership().y = a, b = iterable except Exception as e: print("Caught", repr(e)) return a, b def superOrderCheck(): print("Built-in super:") try: super(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught super 2", repr(e)) def isinstanceOrderCheck(): print("Built-in isinstance:") try: isinstance(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught isinstance 2", repr(e)) def rangeOrderCheck(): print("Built-in range:") try: range(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught range 3", repr(e)) try: range(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught range 2", repr(e)) def importOrderCheck(): print("Built-in __import__:") def name(): print("name", end = ' ') def globals(): # @ReservedAssignment print("globals", end = ' ') def locals(): # @ReservedAssignment print("locals", end = ' ') def fromlist(): print("fromlist", end = ' ') def level(): print("level") try: __import__(name(), globals(), locals(), fromlist(), level()) except Exception as e: print("Expected exception caught __import__ 5", repr(e)) def hasattrOrderCheck(): print("Built-in hasattr:") try: hasattr(zzz, yyy) # @UndefinedVariable except Exception as e: print("Expected exception caught hasattr", repr(e)) def getattrOrderCheck(): print("Built-in getattr:") try: getattr(zzz, yyy) # @UndefinedVariable except Exception as e: print("Expected exception caught getattr 2", repr(e)) try: getattr(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught getattr 3", repr(e)) def default(): print("default used") print("Default side effects:", end = ' ') print(getattr(1, "real", default())) def typeOrderCheck(): print("Built-in type:") try: type(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught type 3", repr(e)) def iterOrderCheck(): print("Built-in iter:") try: iter(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught iter 2", repr(e)) def openOrderCheck(): print("Built-in open:") try: open(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught open 3", repr(e)) def unicodeOrderCheck(): print("Built-in unicode:") try: unicode(zzz, yyy, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught unicode", repr(e)) def longOrderCheck(): print("Built-in long:") try: long(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught long 2", repr(e)) def intOrderCheck(): print("Built-in int:") try: int(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught int", repr(e)) def nextOrderCheck(): print("Built-in next:") try: next(zzz, xxx) # @UndefinedVariable except Exception as e: print("Expected exception caught next 2", repr(e)) def callOrderCheck(): print("Checking nested call arguments:") class A: def __del__(self): print("Doing del inner object") def check(obj): print("Outer function") def p(obj): print("Inner function") check(p(A())) def boolOrderCheck(): print("Checking order of or/and arguments:") class A(int): def __init__(self, value): self.value = value def __del__(self): print("Doing del of %s" % self) def __bool__(self): print("Test of %s" % self) return self.value != 0 __nonzero__ = __bool__ def __str__(self): return "<%s %r>" % (self.__class__.__name__, self.value) class B(A): pass class C(A): pass print("Two arguments, A or B:") for a in range(2): for b in range(2): print("Case %d or %d" % (a,b)) r = A(a) or B(b) print(r) del r # TODO: The order of deletion does not exactly match, which we accept for # now. if True: print("Three arguments, A or B or C:") for a in range(2): for b in range(2): for c in range(2): print("Case %d or %d or %d" % (a,b,c)) r = A(a) or B(b) or C(c) print(r) del r print("Two arguments, A and B:") for a in range(2): for b in range(2): print("Case %d and %d" % (a,b)) r = A(a) and B(b) print(r) del r # See above if True: print("Three arguments, A and B and C:") for a in range(2): for b in range(2): for c in range(2): print("Case %d and %d and %d" % (a,b,c)) r = A(a) and B(b) and C(c) print(r) del r def comparisonChainOrderCheck(): print("Checking order of comparison chains:") class A(int): def __init__(self, value): self.value = value def __del__(self): print("Doing del of %s" % self) def __le__(self, other): print("Test of %s <= %s" % (self, other)) return self.value <= other.value def __str__(self): return "<%s %r>" % (self.__class__.__name__, self.value) class B(A): pass class C(A): pass # See above, not really doing it right. if False: print("Three arguments, A <= B <= C:") for a in range(3): for b in range(3): for c in range(3): print("Case %d <= %d <= %d" % (a,b,c)) r = A(a) <= B(b) <= C(c) print(r) del r dictOrderCheck() separator() listOrderCheck() separator() subscriptOrderCheck() separator() attributeOrderCheck() separator() operatorOrderCheck() separator() compareOrderCheck() separator() sliceOrderCheck() separator() generatorOrderCheck() separator() classOrderCheck() separator() inOrderCheck() separator() unpackOrderCheck() separator() superOrderCheck() separator() isinstanceOrderCheck() separator() rangeOrderCheck() separator() importOrderCheck() separator() hasattrOrderCheck() separator() getattrOrderCheck() separator() typeOrderCheck() separator() iterOrderCheck() separator() openOrderCheck() separator() unicodeOrderCheck() separator() nextOrderCheck() separator() longOrderCheck() separator() intOrderCheck() separator() callOrderCheck() separator() boolOrderCheck() separator() comparisonChainOrderCheck() Nuitka-0.5.28.2/tests/basics/run_xml.py0000755000372000001440000000566013134660221020137 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import os, sys, subprocess, tempfile, shutil # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import check_output # Go its own directory, to have it easy with path knowledge. nuitka1 = sys.argv[1] nuitka2 = sys.argv[2] search_mode = len(sys.argv) > 3 and sys.argv[3] == "search" start_at = sys.argv[4] if len(sys.argv) > 4 else None if start_at: active = False else: active = True my_dir = os.path.dirname(os.path.abspath(__file__)) for filename in sorted(os.listdir(my_dir)): if not filename.endswith(".py") or filename.startswith("run_"): continue path = os.path.relpath(os.path.join(my_dir, filename)) if not active and start_at in (filename, path): active = True if active: # TODO: Reactivate Python3 support here. if False: new_path = os.path.join(tempfile.gettempdir(), filename) shutil.copy(path, new_path) path = new_path # On Windows, we cannot rely on 2to3 to be in the path. if os.name == "nt": command = sys.executable + ' ' + os.path.join(os.path.dirname(sys.executable), "Tools/Scripts/2to3.py") else: command = "2to3" result = subprocess.call( command + " -w -n --no-diffs " + path, stderr = open(os.devnull, 'w'), shell = True ) command = "%s %s '%s' '%s' %s" % ( sys.executable, os.path.join(my_dir, "..", "..", "bin", "compare_with_xml"), nuitka1, nuitka2, path, ) result = subprocess.call( command, shell = True ) if result == 2: sys.stderr.write("Interrupted, with CTRL-C\n") sys.exit(2) if result != 0 and search_mode: print("Error exit!", result) sys.exit(result) else: print("Skipping", filename) Nuitka-0.5.28.2/tests/basics/Looping.py0000644000372000001440000000617213112214770020056 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def cond(): return False def loopingFunction(a = 1*2): c = [] f = [c, a] for a in range(6 or 8): for b in range(8): if a == b: c.append((a,b,True)) elif a < b: c.append((b,a,False)) else: c.append((a,b,False)) if a != b: z = 1 else: z = 0 if z == 0: continue if z == 1 and b == 6: break if a == b: z = 0 print(c) print(f) f = 1 while f < (10 or 8): m = 1 f += 1 print("m=", m) x = [u for u in range(8)] print(x) x = [(u,v) for (u,v) in zip(range(8),reversed(range(8))) ] print(x) x = [(u if u%2==0 else 0) for u in range(10)] print(x) x = [(u if u%2==0 else 0) for u in (a if cond() else range(9))] print(x) y = [ [ 3+ (l if l else -1) for l in [m,m+1] ] for m in [f for f in range(2)] ] print("f=", f) print("y=", y) if x: l = "YES" else: l = "NO" if x: l = "yes" else: if True: l = "no" print("Triple and chain") if m and l and f: print("OK") print("Triple or chain") if m or l or f: print("Okey") print("Nested if not chain") if not m: if not l: print("ok") print("Braced if not chain with 'or'") if not (m or l): print("oki") print("Braced if not chain with 'and'") if not (m and l): print("oki") d=1 print("Nested if chain with outer else") if a: if b or c: if d: print("inside nest") else: print("outer else") print(x) while False: pass else: print("Executed else branch for False condition while loop") while True: break else: print("Executed else branch for True condition while loop") for x in range(7): pass else: print("Executed else branch for no break for loop") for x in range(7): break else: print("Executed else branch despite break in for loop") x = iter(range(5)) while next(x): pass else: print("Executed else branch of while loop without break") loopingFunction() Nuitka-0.5.28.2/tests/basics/GlobalStatement.py0000644000372000001440000000720713112214770021534 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys x = 2 def someFunction1(): x = 3 return x def someFunction2(): global x x = 4 return x def someFunction3(): return x def someNestedGlobalUser1(): z = 1 # Nested function that uses a global z doesn't affect the local variable z at all. def setZ(): global z z = 3 setZ() return z def someNestedGlobalUser2(): z = 1 # Nested function that uses a global z doesn't affect the local variable z at # all. This doesn't change if it's done inside an exec block. exec(""" def setZ(): global z z = 3 setZ() """) return z def someNestedGlobalUser3a(): # Nested function that uses a exec variable scope z and a global z, changes z to be # the global one only. We verify that by looking at locals. This means that the global # statement inside the function of exec changes the effect of the z. exec(""" z = 1 def setZ(): global z z = 3 setZ() """) return z, locals().keys() == ["setZ"] def someNestedGlobalUser3b(): # Nested function that uses a exec variable scope z and a global z, changes # z to be the global one only. We verify that by looking at locals. exec(""" z = 1 """) if sys.version_info[0] < 3: return z, locals().keys() == ['z'] else: return locals().keys() == [] def someNestedGlobalUser4(): z = 1 # This one proves that the local variable z is entirely ignored, and that the global z # has the value 2 inside setZ(). exec(""" z = 2 def setZ(): global z z = 3*z setZ() """) return z def someNestedGlobalUser5(): z = 1 # Without a global statement, z affects the local variable z. exec(""" z = 3 """) return z def someNestedGlobalUser6(): # Without a global statement, a local variable z is created. exec(""" z = 7 """) return z print("Function that shadows a global variable with a local variable") print(someFunction1()) print("Function that accesses and changes a global variable declared with a global statement") print(someFunction2()) print("Function that uses a global variable") print(someFunction3()) print("Functions that uses a global variable in a nested function in various ways:") print(someNestedGlobalUser1, someNestedGlobalUser1()) del z print(someNestedGlobalUser2, someNestedGlobalUser2()) del z print(someNestedGlobalUser3a, someNestedGlobalUser3a()) del z print(someNestedGlobalUser3b, someNestedGlobalUser3b()) print(someNestedGlobalUser4, (someNestedGlobalUser4(), z)) del z print(someNestedGlobalUser5, someNestedGlobalUser5()) z = 9 print(someNestedGlobalUser6, (someNestedGlobalUser6(), z)) x = 7 def f(): x = 1 def g(): global x def i(): def h(): return x return h() return i() return g() print(f()) global global_already global_already = 1 Nuitka-0.5.28.2/tests/basics/ParameterErrors32.py0000644000372000001440000000361313112214770021726 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def kwfunc(a, *, k): pass print( "Call function with mixed arguments with too wrong keyword argument." ) try: kwfunc( k = 3, b = 5 ) except TypeError as e: print( repr(e) ) print( "Call function with mixed arguments with too little positional arguments." ) try: kwfunc( k = 3 ) except TypeError as e: print( repr(e) ) print( "Call function with mixed arguments with too little positional arguments." ) try: kwfunc( 3 ) except TypeError as e: print( repr(e) ) print( "Call function with mixed arguments with too many positional arguments." ) try: kwfunc( 1,2,k=3 ) except TypeError as e: print( repr(e) ) def kwfuncdefaulted(a, b = None, *, c = None): pass print( "Call function with mixed arguments and defaults but too many positional arguments." ) try: kwfuncdefaulted(1, 2, 3) except TypeError as e: print( repr(e) ) def kwfunc2(a, *, k, l, m): pass print( "Call function with mixed arguments with too little positional and keyword-only arguments." ) try: kwfunc2( 1, l = 2 ) except TypeError as e: print( repr(e) ) try: kwfunc2( 1 ) except TypeError as e: print( repr(e) ) Nuitka-0.5.28.2/tests/basics/DoubleDeletions.py0000644000372000001440000000210013112214770021513 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function a = 3 del a try: del a except NameError as e: print("Raised expected exception:", repr(e)) def someFunction(b, c): b = 1 del b try: del b except UnboundLocalError as e: print("Raised expected exception:", repr(e)) someFunction(3, 4) Nuitka-0.5.28.2/tests/basics/GeneratorExpressions.py0000644000372000001440000001255713112214770022644 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import inspect print("Generator expression that demonstrates the timing:") def iteratorCreationTiming(): def getIterable(x): print("Getting iterable", x) return Iterable(x) class Iterable: def __init__(self, x): self.x = x self.values = list(range(x)) self.count = 0 def __iter__(self): print("Giving iterator now", self.x) return self def __next__(self): print("Next of", self.x, "is", self.count) if len(self.values) > self.count: self.count += 1 return self.values[ self.count - 1 ] else: print("Raising StopIteration for", self.x) raise StopIteration # Python2/3 compatibility. next = __next__ def __del__(self): print("Deleting", self.x) gen = ( (y,z) for y in getIterable(3) for z in getIterable(2) ) print("Using generator", gen) next(gen) res = tuple(gen) print(res) print('*' * 20) try: next(gen) except StopIteration: print("Usage past end gave StopIteration exception as expected.") try: print("Generator state then is", inspect.getgeneratorstate(gen)) # @UndefinedVariable except AttributeError: pass print("Its frame is now", gen.gi_frame) print("Early aborting generator:") gen2 = ( (y,z) for y in getIterable(3) for z in getIterable(2) ) del gen2 iteratorCreationTiming() print("Generator expressions that demonstrate the use of conditions:") print(tuple( x for x in range(8) if x % 2 == 1 )) print(tuple( x for x in range(8) if x % 2 == 1 for z in range(8) if z == x )) print(tuple( x for (x,y) in zip(list(range(2)),list(range(4))))) print("Directory of generator expressions:") for_dir = ( x for x in [1] ) gen_dir = dir(for_dir) print(sorted( g for g in gen_dir )) def genexprSend(): x = ( x for x in range(9) ) print("Sending too early:") try: x.send(3) except TypeError as e: print("Gave expected TypeError with text:", e) z = next(x) y = x.send(3) print("Send return value", y) print("And then next gave", next(x)) print("Throwing an exception to it.") try: x.throw(2) assert False except TypeError as e: print("Gave expected TypeError:", e) print("Throwing an exception to it.") try: x.throw(ValueError(2)) except ValueError as e: print("Gave expected ValueError:", e) try: next(x) print("Next worked even after thrown error") except StopIteration as e: print("Gave expected stop iteration after throwing exception in it:", e) print("Throwing another exception from it.") try: x.throw(ValueError(5)) except ValueError as e: print("Gave expected ValueError with text:", e) print("Generator expressions have send too:") genexprSend() def genexprClose(): x = ( x for x in range(9) ) print("Immediate close:") x.close() print("Closed once") x.close() print("Closed again without any trouble") genexprClose() def genexprThrown(): def checked(z): if z == 3: raise ValueError return z x = ( checked(x) for x in range(9) ) try: for count, value in enumerate(x): print(count, value) except ValueError: print(count+1, ValueError) try: next(x) print("Allowed to do next() after raised exception from the generator expression") except StopIteration: print("Exception in generator, disallowed next() afterwards.") genexprThrown() def nestedExpressions(): a = [x for x in range(10)] b = (x for x in (y for y in a)) print("nested generator expression", list(b)) nestedExpressions() def lambdaGenerators(): a = 1 x = lambda : (yield a) print("Simple lambda generator", x, x(), list(x())) y = lambda : ((yield 1),(yield 2)) print("Complex lambda generator", y, y(), list(y())) lambdaGenerators() def functionGenerators(): # Like lambdaGenerators, to show how functions behave differently if at all. a = 1 def x(): yield a print("Simple function generator", x, x(), list(x())) def y(): yield((yield 1),(yield 2)) print("Complex function generator", y, y(), list(y())) functionGenerators() def strangeLambdaGeneratorExpression(): x = ((yield) for i in (1,2) if (yield)) print("Strange lambda generator expression") print(list(x)) strangeLambdaGeneratorExpression() Nuitka-0.5.28.2/tests/basics/Functions_2.py0000644000372000001440000000443513112214770020640 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def local_function(a,z = 9): b = `a*a+1` c = (a,b,a**32,a+a) d = long('0') # @UnusedVariable e = int("77") # @UnusedVariable d = long(b) e = long(1+1) return a, b, c, d, e, z print("Call function with many variables calculated and returned", local_function(1,z = 5)) print("Function with nested args:") def nested_args_function((a,b), c): return a, b, c print(nested_args_function((1, 2), 3)) try: nested_args_function((1, 2, 3), 3) except ValueError, e: print("Calling nested with too long tuple gave:", e) try: nested_args_function((1,), 3) except ValueError, e: print("Calling nested with too short tuple gave:", e) def deeply_nested_function(((a,), b, c, (d, (e,f)))): return a, b, c, d, e, f print("Deeply nested function", deeply_nested_function(((1,), 2, 3, (4, (5, 6))))) print("Function with nested args that have defaults:") def default_giver(): class R: def __iter__(self): print("Giving iter") return iter(range(2)) return R() def nested_args_function_with_defaults((a,b) = default_giver(), c = 5): return a, b, c print("Calling it.") print(nested_args_function_with_defaults()) def comp_args1((a, b)): return a,b def comp_args2((a, b) = (3, 4)): return a, b def comp_args3(a, (b, c)): return a, b, c def comp_args4(a = 2, (b, c) = (3, 4)): return a, b, c print("Complex args functions", comp_args1((2, 1)), comp_args2(), comp_args2((7,9)), comp_args3(7, (8,9)), comp_args4()) Nuitka-0.5.28.2/tests/basics/LateClosureAssignment.py0000644000372000001440000000521513112214770022717 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def closureTest1(): # Assign, but the value is not supposed to be used by the function, instead the later # update is effective. d = 1 def subby(): return d d = 22222*2222 return subby() def closureTest2(): # Using a closure variable that is not initialized at the time it is closured should # work as well. def subby(): return d d = 2222*2222 return subby() def closureTest3(): def subby(): return undefined_global # @UndefinedVariable try: return subby() except NameError: return 88 d = 1 def scopeTest4(): try: return d d = 1 except UnboundLocalError as e: return repr(e) print("Test closure where value is overwritten:", closureTest1()) print("Test closure where value is assigned only late:", closureTest2()) print("Test function where closured value is never assigned:", closureTest3()) print("Scope test where UnboundLocalError is expected:", scopeTest4()) def function(): pass class ClosureLocalizerClass: print("Function before assigned in a class:", function) function = 1 print("Function after it was assigned in class:", function) ClosureLocalizerClass() def ClosureLocalizerFunction(): try: function = function print("Function didn't give unbound local error") except UnboundLocalError as e: print("Function gave unbound local error when accessing function before assignment:", repr(e)) ClosureLocalizerFunction() class X: def __init__(self, x): self.x = x def changingClosure(): print("Changing a closure taken value after it was taken.") a = 1 def closureTaker(): return X(a) x = closureTaker() a=2 print("Closure value first time:", x.x) x = closureTaker() print("Closure value second time:", x.x) changingClosure() Nuitka-0.5.28.2/tests/basics/Recursion.py0000644000372000001440000000155513112214770020420 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # count = 0 def recurse(): global count count += 1 if count < 50: recurse() recurse() Nuitka-0.5.28.2/tests/basics/FunctionObjects.py0000644000372000001440000000310713112214770021541 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def func(arg1, arg2, arg3, **star): """ Some documentation. """ pass print("Starting out: func, __name__:", func, func.__name__) print("Changing its name:") func.__name__ = "renamed" print("With new name: func, __name__:", func, func.__name__) print("Documentation initially:", func.__doc__) print("Changing its doc:") func.__doc__ = "changed doc" + chr(0) + " with 0 character" print("Documentation updated:", repr(func.__doc__)) print("Setting its dict") func.my_value = "attached value" print("Reading its dict", func.my_value) print("__code__ dir") print(dir(func.__code__)) def func2(arg1, arg2 = "default_arg2", arg3 = "default_arg3"): x = arg2 + arg3 return x print("func __defaults__", func2.__defaults__) print("function varnames", func2.__code__.co_varnames) Nuitka-0.5.28.2/tests/basics/ThreadedGenerators.py0000644000372000001440000000220613112214770022213 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # From Issue#146, this has crashed in the past. import threading def some_generator(): yield 1 def run(): for i in range(10000): for j in some_generator(): pass def main(): workers = [threading.Thread(target = run) for i in range(5)] for t in workers: t.start() for t in workers: t.join() if __name__ == "__main__": main() Nuitka-0.5.28.2/tests/basics/ComparisonChains.py0000644000372000001440000000762313112214770021711 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def simple_comparisons(x, y): if 'a' <= x <= y <= 'z': print("One") if 'a' <= x <= 'z': print("Two") if 'a' <= x > 'z': print("Three") print("Simple comparisons:") simple_comparisons('c', 'd') def side_effect(): print("") return 7 def side_effect_comparisons(): print("Should have side effect:") print(1 < side_effect() < 9) print("Should not have side effect due to short circuit:") print(3 < 2 < side_effect() < 9) print("Check for expected side effects only:") side_effect_comparisons() def function_torture_is(): a = (1, 2, 3) for x in a: for y in a: for z in a: print(x, y, z, ':', x is y is z, x is not y is not z) function_torture_is() print("Check if lambda can have expression chains:", end = "") def function_lambda_with_chain(): a = (1, 2, 3) x = lambda x : x[0] < x[1] < x[2] print("lambda result is", x(a)) function_lambda_with_chain() print("Check if generators can have expression chains:", end = "") def generator_function_with_chain(): x = (1, 2, 3) yield x[0] < x[1] < x[2] print(list(generator_function_with_chain())) print("Check if list contractions can have expression chains:", end = "") def contraction_with_chain(): return [ x[0] < x[1] < x[2] for x in [(1, 2, 3) ] ] print(contraction_with_chain()) print("Check if generator expressions can have expression chains:", end = "") def genexpr_with_chain(): return ( x[0] < x[1] < x[2] for x in [(1, 2, 3) ] ) print(list(genexpr_with_chain())) print("Check if class bodies can have expression chains:", end = "") class class_with_chain: x = (1, 2, 3) print(x[0] < x[1] < x[2]) x = (1, 2, 3) print(x[0] < x[1] < x[2]) class CustomOps(int): def __lt__(self, other): print("enter <", self, other) return True def __gt__(self, other): print("enter >", self, other) return False print("Custom ops, to enforce chain eval order and short circuit:", end = "") print(CustomOps(7) < CustomOps(8) > CustomOps(6)) print("Custom ops, doing short circuit:", end = "") print(CustomOps(8) > CustomOps(7) < CustomOps(6)) def inOperatorChain(): print("In operator chains:") print(3 in [3,4] in [[3,4]]) print(3 in [3,4] not in [[3,4]]) if 3 in [3,4] in [[3,4]]: print("Yes") else: print("No") if 3 in [3,4] not in [[3,4]]: print("Yes") else: print("No") inOperatorChain() # Make sure the values are called and order is correct: class A(object): def __init__(self, name, value): self.name = name self.value = value def __repr__(self): return "" % (self.name, self.value) def __lt__(self, other): print("less than called for:", self, other, self.value, other.value, self.value < other.value) if self.value < other.value: print("good") return 7 else: print("bad") return 0 a = A('a',1) b = A('b',2) c = A('c',0) print(a < b < c) print('*' * 80) a = A('a',2) b = A('b',1) c = A('c',0) print(a < b < c) Nuitka-0.5.28.2/tests/basics/OrderChecks27.py0000644000372000001440000000345213112214770021012 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def setOrderCheck(): print("Checking order of set literals:") def one(): print("one") return "one" def two(): print("two") return "two" return {one(), two()} def raiseOrderCheck(): print("Checking order of raises:") def exception_type(): print("exception_type", end = "") return ValueError def exception_value(): print("exception_value", end = "") return 1 def exception_tb(): print("exception_value", end = "") return None print("3 args", end = "") try: raise exception_type(), exception_value(), exception_tb() except Exception as e: print("caught", repr(e)) print("2 args", end = "") try: raise exception_type(), exception_value() except Exception as e: print("caught", repr(e)) print("1 args", end = "") try: raise exception_type() except Exception as e: print("caught", repr(e)) setOrderCheck() raiseOrderCheck() Nuitka-0.5.28.2/tests/basics/Referencing27.py0000644000372000001440000000347413207537242021060 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython ) checkDebugPython() x = 17 # Python2.7 or higher syntax things are here. def simpleFunction1(): return {i:x for i in range(x)} def simpleFunction2(): try: return {y:i for i in range(x)} # @UndefinedVariable except NameError: pass def simpleFunction3(): return {i for i in range(x)} def simpleFunction4(): try: return {y for i in range(x)} # @UndefinedVariable @UnusedVariable except NameError: pass # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/ModuleAttributes.py0000644000372000001440000000327613112214770021745 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Some module documentation. With newline and stuff.""" from __future__ import print_function import os, sys print("doc:", __doc__) print("filename:", os.path.basename(__file__)) print("builtins:", __builtins__) print("debug", __debug__) print("debug in builtins", __builtins__.__debug__) print("__initializing__", end = ' ') try: print(__initializing__) # @UndefinedVariable except NameError: print("not found") def checkFromFunction(): frame = sys._getframe(1) def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" import pprint return pprint.pformat(d) print("Globals", displayDict(frame.f_globals)) print("Locals", displayDict(frame.f_locals)) print("Is identical", frame.f_locals is frame.f_globals) checkFromFunction() Nuitka-0.5.28.2/tests/basics/BuiltinSuper.py0000644000372000001440000000624613122472300021072 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function # Python2 will fallback to this variable, which Python3 will ignore. __class__ = "Using module level __class__ variable, would be wrong for Python3" # @ReservedAssignment class ClassWithUnderClassClosure: def g(self): def h(): print("Variable __class__ in ClassWithUnderClassClosure is", __class__) h() try: print("ClassWithUnderClassClosure: Super in ClassWithUnderClassClosure is", super()) except Exception as e: print("ClassWithUnderClassClosure: Occurred during super call", repr(e)) print("Class with a method that has a local function accessing __class__:") ClassWithUnderClassClosure().g() class ClassWithoutUnderClassClosure: def g(self): __class__ = "Providing __class__ ourselves, then it must be used" # @ReservedAssignment print(__class__) try: print("ClassWithoutUnderClassClosure: Super", super()) except Exception as e: print("ClassWithoutUnderClassClosure: Occurred during super call", repr(e)) ClassWithoutUnderClassClosure().g() # For Python2 only. __class__ = "Global __class__" # @ReservedAssignment def deco(C): print("Decorating", repr(C)) class D(C): pass return D @deco class X: __class__ = "some string" def f1(self): print("f1", locals()) try: print("f1", __class__) except Exception as e: print("Accessing __class__ in f1 gave", repr(e)) def f2(self): print("f2", locals()) def f4(self): print("f4", self) self = X() print("f4", self) try: print("f4", super()) print("f4", super().__self__) except TypeError: import sys assert sys.version_info < (3,) f5 = lambda x: __class__ def f6(self_by_another_name): # @NoSelf try: print("f6", super()) except TypeError: import sys assert sys.version_info < (3,) def f7(self): try: yield super() except TypeError: import sys assert sys.version_info < (3,) print("Early pre-class calls begin") print("Set in class __class__", __class__) # f1(1) f2(2) print("Early pre-class calls end") del __class__ x = X() x.f1() x.f2() x.f4() print("f5", x.f5()) x.f6() print("f7", list(x.f7())) Nuitka-0.5.28.2/tests/basics/Lamdas.py0000644000372000001440000000323213112214770017642 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def lamdaContainer(x): f = lambda c : c g = lambda c : c if x else c*c # h = lambda c: 'a' <= c <= 'z' y = f(x) z = g(4) print("Lambda with conditional expression gives", z) if 'a' <= x <= y <= 'z': print("Four") if 'a' <= x <= 'z': print("Yes") if 'a' <= x > 'z': print("Yes1") if 'a' <= ('1' if x else '2') > 'z': print("Yes2") if 'a' <= ('1' if x else '2') > 'z' > undefined_global: # @UndefinedVariable print("Yes3") z = lambda huhu = y : huhu print("Lambda defaulted gives", z()) lamdaContainer('b') def lambdaGenerator(): x = lambda : (yield 3) gen = x() print("Lambda generator gives", next(gen)) lambdaGenerator() def lambdaDirectCall(): args = range(7) x = (lambda *args:args)(*args) print("Lambda direct call gave", x) lambdaDirectCall() Nuitka-0.5.28.2/tests/basics/ClassMinimal.py0000644000372000001440000000154013112214770021015 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Very minimal class example, to be used for debugging. a = 1 class B: b = a print(B.b) Nuitka-0.5.28.2/tests/basics/TryExceptFrames.py0000644000372000001440000000421613122472300021525 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" import pprint return pprint.pformat(d) counter = 1 class X: def __init__(self): global counter self.counter = counter counter += 1 def __del__(self): print("X.__del__ occurred", self.counter) def raising(doit): _x = X() def nested(): if doit: 1 / 0 try: return nested() except ZeroDivisionError: print("Changing closure variable value.") # This is just to prove that closure variables get updates in frame # locals. doit = 5 raise # Call it without an exception raising(False) def catcher(): try: raising(True) except ZeroDivisionError: print("Catched.") print("Top traceback code is '%s'." % sys.exc_info()[2].tb_frame.f_code.co_name) print("Second traceback code is '%s'." % sys.exc_info()[2].tb_next.tb_frame.f_code.co_name) print("Third traceback code is '%s'." % sys.exc_info()[2].tb_next.tb_next.tb_frame.f_code.co_name) print("Previous frame locals (module) are", displayDict(sys.exc_info()[2].tb_next.tb_next.tb_frame.f_locals)) catcher() print("Good bye.") Nuitka-0.5.28.2/tests/basics/Referencing.py0000644000372000001440000003702613134660221020700 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython ) checkDebugPython() x = 17 # Just a function return a constant. Functions don't become any smaller. Let's # get that right. def simpleFunction1(): return 1 # Do a bit of math with a local variable, assigning to its value and then doing # an overwrite of that, trying that math again. This should cover local access # a bit. def simpleFunction2(): y = 3 * x # @UnusedVariable y = 3 return x*2*y # A local function is being returned. This covers creation of local functions # and their release. No closure variables involved yet. def simpleFunction3(): def contained(): return x return contained # Again, local function being return, but this time with local variable taken # as a closure. We use value from defaulted argument, so it cannot be replaced. def simpleFunction4(a = 1): y = a def contained(): return y return contained # Default argument and building a list as local variables. Also return them, # so they are not optimized away. def simpleFunction5(a = 2): c = 1 f = [a, a + c] return c, f def simpleFunction6(): for _b in range(6): pass for _c in (1, 2, 3, 4, 5, 6): pass def simpleFunction7(b = 1): for _b in range(6): pass def simpleFunction8(): c = [] c.append(x) def simpleFunction9(a = 1*2): if a == a: pass u = None def simpleFunction10(a = 1*2): x = [u for u in range(8)] def simpleFunction11(): f = 1 while f < 8: f += 1 v = None def simpleFunction12(): a = [(u,v) for (u,v) in zip(range(8),range(8))] return a def cond(): return 1 def simpleFunction13(a = 1*2): pass def simpleFunction14p(x): try: simpleFunction14p(1,1) except TypeError as _e: pass try: simpleFunction14p(1,1) except TypeError: pass def simpleFunction14(): simpleFunction14p(3) def simpleFunction15p(x): try: try: x += 1 finally: try: x *= 1 finally: _z = 1 except: pass def simpleFunction15(): simpleFunction15p([1]) def simpleFunction16(): class EmptyClass: pass return EmptyClass def simpleFunction17(): class EmptyObjectClass: pass return EmptyObjectClass() def simpleFunction18(): closured = 1 class NonEmptyClass: def __init__(self, a, b): self.a = a self.b = b inside = closured return NonEmptyClass(133, 135) def simpleFunction19(): lam = lambda l : l+1 return lam(9), lam def simpleFunction20(): try: a = [] a[1] except IndexError as _e: pass def simpleFunction21(): class EmptyBaseClass: def base(self): return 3 class EmptyObjectClass(EmptyBaseClass): pass result = EmptyObjectClass() c = result.base() return result, c def simpleFunction22(): return True is False and False is not None def simpleFunction23(): not 2 def simpleFunction24p(x): pass def simpleFunction24(): simpleFunction24p(x = 3) def simpleFunction25(): class X: f = 1 def inplace_adder(b): X.f += b return inplace_adder(6**8) def simpleFunction26(): class X: f = [5] def inplace_adder(b): X.f += b return inplace_adder([1, 2]) def simpleFunction27(): a = { 'g': 8 } def inplace_adder(b): a[ 'g' ] += b return inplace_adder(3) def simpleFunction28(): a = { 'g': [8], 'h': 2 } def inplace_adder(b): a[ 'g' ] += b return inplace_adder([3, 5]) def simpleFunction29(): return '3' in '7' def simpleFunction30(): def generatorFunction(): yield 1 yield 2 yield 3 def simpleFunction31(): def generatorFunction(): yield 1 yield 2 yield 3 a = [] for y in generatorFunction(): a.append(y) for z in generatorFunction(): a.append(z) def simpleFunction32(): def generatorFunction(): yield 1 gen = generatorFunction() next(gen) def simpleFunction33(): def generatorFunction(): a = 1 yield a a = [] for y in generatorFunction(): a.append(y) def simpleFunction34(): try: raise ValueError except: pass def simpleFunction35(): try: raise ValueError(1,2,3) except: pass def simpleFunction36(): try: raise (TypeError, (3,x,x,x)) except TypeError: pass def simpleFunction37(): l = [1, 2, 3] try: _a, _b = l except ValueError: pass def simpleFunction38(): class Base: pass class Parent(Base): pass def simpleFunction39(): class Parent(object): pass def simpleFunction40(): def myGenerator(): yield 1 myGenerator() def simpleFunction41(): a = b = 2 return a, b def simpleFunction42(): a = b = 2 * x return a, b def simpleFunction43(): class D: pass a = D() a.b = 1 def simpleFunction47(): def reraisy(): def raisingFunction(): raise ValueError(3) def reraiser(): raise try: raisingFunction() except: reraiser() try: reraisy() except: pass def simpleFunction48(): class BlockExceptions: def __enter__(self): pass def __exit__(self, exc, val, tb): return True with BlockExceptions(): raise ValueError() template = "lala %s lala" def simpleFunction49(): c = 3 d = 4 a = x, y = b, e = (c,d) return a, y, b, e b = range(10) def simpleFunction50(): def getF(): def f(): for i in b: yield i return f f = getF() for x in range(2): _r = list(f()) def simpleFunction51(): g = ( x for x in range(9) ) try: g.throw(ValueError, 9) except ValueError as _e: pass def simpleFunction52(): g = ( x for x in range(9) ) try: g.throw(ValueError(9)) except ValueError as _e: pass def simpleFunction53(): g = ( x for x in range(9) ) try: g.send(9) except TypeError as _e: pass def simpleFunction54(): g = ( x for x in range(9) ) next(g) try: g.send(9) except TypeError as _e: pass def simpleFunction55(): g = ( x for x in range(9) ) try: g.close() except ValueError as _e: pass def simpleFunction56(): def f(): f() try: f() except RuntimeError: pass def simpleFunction57(): x = 1 y = 2 def f(a = x, b = y): return a, b f() f(2) f(3,4) def simpleFunction58(): a = 3 b = 5 try: a = a * 2 return a finally: a / b def simpleFunction59(): a = 3 b = 5 try: a = a * 2 return a finally: return a / b class X: def __del__(self): # Super used to reference leak. x = super() raise ValueError(1) def simpleFunction63(): def superUser(): X() try: superUser() except Exception: pass def simpleFunction64(): x = 2 y = 3 # @UnusedVariable z = eval("x * y") return z def simpleFunction65(): import array a = array.array('b', b"") assert a == eval(repr(a), {"array": array.array}) d = { 'x' : 2, 'y' : 3 } z = eval(repr(d), d) return z def simpleFunction66(): import types return type(simpleFunction65) == types.FunctionType def simpleFunction67(): length = 100000 pattern = "1234567890\00\01\02\03\04\05\06" q, r = divmod(length, len(pattern)) teststring = pattern * q + pattern[:r] return teststring def simpleFunction68(): from random import randrange x = randrange(18) def simpleFunction69(): pools = [tuple() ] g = ((len(pool) == 0,) for pool in pools) next(g) def simpleFunction70(): def gen(): try: undefined_yyy # @UndefinedVariable except Exception: pass yield sys.exc_info() try: undefined_xxx # @UndefinedVariable except Exception: return list(gen()) def simpleFunction71(): try: undefined_global # @UndefinedVariable except Exception: try: try: raise finally: undefined_global # @UndefinedVariable except Exception: pass def simpleFunction72(): try: for _i in range(10): try: undefined_global # @UndefinedVariable finally: break except Exception: pass def simpleFunction73(): for _i in range(10): try: undefined_global # @UndefinedVariable finally: return 7 def simpleFunction74(): import os # @Reimport return os def simpleFunction75(): def raising_gen(): try: raise TypeError except TypeError: yield g = raising_gen() next(g) try: g.throw(RuntimeError()) except RuntimeError: pass def simpleFunction76(): class MyException(Exception): def __init__(self, obj): self.obj = obj class MyObj: pass def inner_raising_func(): raise MyException(MyObj()) try: inner_raising_func() except MyException: try: try: raise finally: raise except MyException: pass def simpleFunction77(): class weirdstr(str): def __getitem__(self, index): return weirdstr(str.__getitem__(self, index)) (weirdstr("1234")) # filter(lambda x: x>="33", weirdstr("1234")) def simpleFunction78(): a = "x = 2" exec(a) def simpleFunction79(): "some doc" simpleFunction79.__doc__ = simpleFunction79.__doc__.replace("doc", "dok") simpleFunction79.__doc__ += " and more" + simpleFunction79.__name__ def simpleFunction80(): "some doc" del simpleFunction80.__doc__ def simpleFunction81(): def f(): yield 1 j j = 1 x = list(f()) def simpleFunction82(): def f(): yield 1 j j = 1 x = f.__doc__ def simpleFunction83(): x = list(range(7)) x[2] = 5 j = 3 x += [h*2 for h in range(j)] def simpleFunction84(): x = tuple(range(7)) j = 3 x += tuple([h*2 for h in range(j)]) def simpleFunction85(): x = list(range(7)) x[2] = 3 x *= 2 def simpleFunction86(): x = "something" x += "" def simpleFunction87(): x = 7 x += 2000 def simpleFunction88(): class C: def __iadd__(self, other): return self x = C() x += C() def simpleFunction89(): x = [1,2] x += [3,4] def anyArgs(*args, **kw): pass def simpleFunction90(): some_tuple = ( simpleFunction89, simpleFunction89, simpleFunction89, ) anyArgs(*some_tuple) def simpleFunction91(): some_dict = { 'a' : simpleFunction90, } anyArgs(**some_dict) def simpleFunction92(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(*some_tuple, **some_dict) def simpleFunction93(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(some_tuple, *some_tuple, **some_dict) def simpleFunction94(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(*some_tuple, b = some_dict, **some_dict) def simpleFunction95(): some_tuple = ( simpleFunction89, ) some_dict = { 'a' : simpleFunction90, } anyArgs(some_tuple, *some_tuple, b = some_dict, **some_dict) def simpleFunction96(): some_tuple = ( simpleFunction89, ) anyArgs(some_tuple, *some_tuple) # Complex call with dictionary and key arguments only. def simpleFunction97(): some_dict = { 'a' : simpleFunction90, 'd' : simpleFunction91 } anyArgs(b = some_dict, c = 1, **some_dict) def simpleFunction98(): some_tuple = ( simpleFunction89, ) anyArgs(*some_tuple, b = some_tuple) def simpleFunction99(): some_dict = { 'a' : simpleFunction90, } anyArgs(some_dict, **some_dict) def simpleFunction100(): def h(f): def g(): return f return g def f(): pass h(f) def simpleFunction101(): def orMaking(a, b): x = "axa" x += a or b orMaking('x', "") #################################### class SomeClassWithAttributeAccess(object): READING = 1 def use(self): return self.READING def simpleFunction102(): SomeClassWithAttributeAccess().use() SomeClassWithAttributeAccess().use() #################################### def getInt(): return 3 def simpleFunction103(): try: raise getInt() except TypeError: pass #################################### class ClassWithGeneratorMethod: def generator_method(self): yield self def simpleFunction104(): return list(ClassWithGeneratorMethod().generator_method()) def simpleFunction105(): """ Delete a started generator, not properly closing it before releasing. """ def generator(): yield 1 yield 2 g = generator() next(g) del g def simpleFunction106(): # Call a PyCFunction with a single argument. return sys.getsizeof(type) def simpleFunction107(): # Call a PyCFunction with a single argument. return sum(i for i in range(x)) def simpleFunction108(): # Call a PyCFunction with a single argument. return sum((i for i in range(x)), 17) def simpleFunction109(): # Call a PyCFunction that looks like a method call. sys.exc_info() def simpleFunction110(): def my_open(*args, **kwargs): return(args, kwargs) __builtins__.open = my_open open("me", buffering = True) u = u'__name__' def simpleFunction111(): return getattr(simpleFunction111, u) #################################### # These need stderr to be wrapped. tests_stderr = (63,) # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/WithStatements.py0000644000372000001440000001140613112214770021426 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys x = 0 # This is used to trace the exact interaction with the context manager to # uncover and decide ordering and correctness of calls. class MyContextManager(object): def __getattribute__(self, attribute_name): print("Asking context manager attribute", attribute_name) return object.__getattribute__(self, attribute_name) def __enter__(self): global x x += 1 print("Entered context manager with counter value", x) return x def __exit__(self, exc_type, exc_value, traceback): print("Context manager exit sees", exc_type, exc_value, traceback) print("Published to context manager exit is", sys.exc_info()) return False print("Use context manager and raise no exception in the body:") with MyContextManager() as x: print("x has become", x) print("Use context manager and raise an exception in the body:") try: with MyContextManager() as x: print("x has become", x) raise Exception("Lalala") print(x) except Exception as e: print("Caught raised exception", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) # Python3 ranges are not lists l = list(range(3)) print("Use context manager and assign to subscription target:") with MyContextManager() as l[0]: print("Complex assignment target works", l[0]) try: with MyContextManager(): sys.exit(9) except BaseException as e: print("Caught base exception", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("Use context manager and fail to assign to attribute:") try: with MyContextManager() as l.wontwork: sys.exit(9) except BaseException as e: print("Caught base exception", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("Use context manager to do nothing inside:") with MyContextManager() as x: pass if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) # Use context manager and fail to assign. def returnFromContextBlock(): # Use context manager to do nothing. with MyContextManager() as x: return 7 if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("Use context manager to return value:") r = returnFromContextBlock() print("Return value", r) class NonContextManager1: def __enter__(self): return self class NonContextManager2: def __exit__(self): return self print("Use incomplete context managers:") try: with NonContextManager1() as x: print(x) except Exception as e: print("Caught for context manager without __exit__", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) try: with NonContextManager2() as x: print(x) except Exception as e: print("Caught for context manager without __enter__", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) class NotAtAllContextManager: pass try: with NotAtAllContextManager() as x: print(x) except Exception as e: print("Caught for context manager without any special methods", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) class MeanContextManager: def __enter__(self): raise ValueError("Nah, I won't play") def __exit__(self): print("Called exit, yes") print("Use mean context manager:") try: with MeanContextManager() as x: print(x) except Exception as e: print("Caught from mean manager", repr(e)) if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) class CatchingContextManager(object): def __enter__(self): pass def __exit__(self, exc_type, exc_value, traceback): return True print("Suppressing exception from context manager body:") with CatchingContextManager(): raise ZeroDivisionError if sys.version_info >= (3,): assert sys.exc_info() == (None, None, None) print("OK") Nuitka-0.5.28.2/tests/basics/TryExceptFinally.py0000644000372000001440000000436013112214770021712 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # "Some doc" from __future__ import print_function def one(): return 1 def tryScope1(x): try: try: x += one() finally: print("Finally is executed") try: _z = one() finally: print("Deep Nested finally is executed") except: print("Exception occurred") else: print("No exception occurred") tryScope1(1) print('*' * 20) tryScope1([1]) def tryScope2(x, someExceptionClass): try: x += 1 except someExceptionClass as e: print("Exception class from argument occurred:", someExceptionClass, repr(e)) else: print("No exception occurred") def tryScope3(x): if x: try: x += 1 except TypeError: print("TypeError occurred") else: print("Not taken") print('*' * 20) tryScope2(1, TypeError) tryScope2([1], TypeError) print('*' * 20) tryScope3(1) tryScope3([1]) tryScope3([]) print('*' * 20) def tryScope4(x): try: x += 1 except: print("exception occurred") else: print("no exception occurred") finally: print("finally obeyed") tryScope4(1) tryScope4([1]) def tryScope5(): import sys print("Exception info is initially", sys.exc_info()) try: try: undefined_global += 1 # @UndefinedVariable finally: print("Exception info in 'finally' clause is", sys.exc_info()) except: pass tryScope5() Nuitka-0.5.28.2/tests/basics/YieldFrom33.py0000644000372000001440000000626113112214770020506 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def g(): for a in range(3): yield a return 7 def h(): yield 4 yield 5 def f(): print("Yielded from returner", (yield from g())) print("Yielded from non-return value", (yield from h())) print( "Result", list( f() ) ) print( "Yielder with return value", list(g()) ) # This will raise when looking up any attribute. class Broken: def __iter__(self): return self def __next__(self): return 1 def __getattr__(self, attr): 1/0 def test_broken_getattr_handling(): def g(): yield from Broken() print( "Next with send: ", end = "" ) try: gi = g() next(gi) gi.send(1) except Exception as e: print( "Caught", repr(e) ) print( "Next with throw: ", end = "" ) try: gi = g() next(gi) gi.throw(AttributeError) except Exception as e: print( "Caught", repr(e) ) print( "Next with close: ", end = "" ) try: gi = g() next(gi) gi.close() print( "All good" ) except Exception as e: print( "Caught", repr(e) ) test_broken_getattr_handling() def test_throw_catched_subgenerator_handling(): def g1(): try: print("Starting g1") yield "g1 ham" yield from g2() yield "g1 eggs" finally: print("Finishing g1") def g2(): try: print("Starting g2") yield "g2 spam" yield "g2 more spam" except LunchError: print("Caught LunchError in g2") yield "g2 lunch saved" yield "g2 yet more spam" class LunchError(Exception): pass g = g1() for i in range(2): x = next(g) print("Yielded %s" % (x,)) e = LunchError("tomato ejected") print( "Throw returned", g.throw(e) ) print( "Sub thrown" ) for x in g: print("Yielded %s" % (x,)) test_throw_catched_subgenerator_handling() def give_cpython_generator(): # TODO: This relies on eval not being inlined, which will become untrue. return eval( "( x for x in range(3) )" ) def gen_compiled(): yield from give_cpython_generator() yield from range(7) print( list( gen_compiled() ) ) Nuitka-0.5.28.2/tests/basics/BuiltinOverload.py0000644000372000001440000000214513112214770021545 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function try: from __builtin__ import len as _len except ImportError: from builtins import len as _len # This kind of built-in overload will have to work. def len(x): # @ReservedAssignment print("Private built-in called with argument", repr(x)) return _len(x) print("Calling built-in len", len(range(9))) Nuitka-0.5.28.2/tests/basics/ListContractions.py0000644000372000001440000000511513112214770021745 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def displayDict(d): result = '{' for key, value in sorted(d.items()): result += "%s: %s" % (key, value) result += '}' print("List contraction on the module level:") x = [(u if u%2==0 else 0) for u in range(10)] print(x) print("List contraction on the function level:") def someFunction(): x = [(u if u%2==0 else 0) for u in range(10)] print(x) someFunction() print("List contractions with no, 1 one 2 conditions:") def otherFunction(): print([ x for x in range(8) ]) print([ x for x in range(8) if x % 2 == 1 ]) print([ x for x in range(8) if x % 2 == 1 if x > 4 ]) otherFunction() print("Complex list contractions with more than one for:") def complexContractions(): print([ (x,y) for x in range(3) for y in range(5) ]) seq = range(3) res = [(i, j, k) for i in iter(seq) for j in iter(seq) for k in iter(seq)] print(res) complexContractions() print("Contraction for 2 for statements and one final if referring to first for:") def trickyContraction(): class Range: def __init__(self, value): self.value = value def __iter__(self): print("Giving range iter to", self.value) return iter(range(self.value)) def Cond(y): print("Checking against", y) return y == 1 r = [ (x,z,y) for x in Range(3) for z in Range(2) for y in Range(4) if Cond(y) ] print("result is", r) trickyContraction() def lambdaWithcontraction(x): l = lambda x : [ z for z in range(x) ] r = l(x) print("Lambda contraction locals:", displayDict(locals())) lambdaWithcontraction(3) print("Contraction that gets a 'del' on the iterator variable:", end = ' ') def allowedDelOnIteratorVariable(z): x = 2 del x return [ x*z for x in range(z) ] print(allowedDelOnIteratorVariable(3)) Nuitka-0.5.28.2/tests/basics/TrickAssignments32.py0000644000372000001440000001340313112214770022077 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def someFunctionThatReturnsDeletedValueViaAttributeLookup(): class C: def __getattr__(self, attr_name): nonlocal a del a c = C() a = 1 c.something return a try: someFunctionThatReturnsDeletedValueViaAttributeLookup() except UnboundLocalError: print("OK, object attribute look-up correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaAttributeSetting(): class C: def __setattr__(self, attr_name, value): nonlocal a del a c = C() a = 1 c.something = 1 return a try: someFunctionThatReturnsDeletedValueViaAttributeSetting() except UnboundLocalError: print("OK, object attribute setting correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaAttributeDel(): class C: def __delattr__(self, attr_name): nonlocal a del a return True c = C() a = 1 del c.something return a try: someFunctionThatReturnsDeletedValueViaAttributeDel() except UnboundLocalError: print("OK, object attribute del correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaItemLookup(): class C: def __getitem__(self, attr_name): nonlocal a del a c = C() a = 1 c[2] return a try: someFunctionThatReturnsDeletedValueViaItemLookup() except UnboundLocalError: print("OK, object subscript look-up correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaItemSetting(): class C: def __setitem__(self, attr_name, value): nonlocal a del a c = C() a = 1 c[2] = 3 return a try: someFunctionThatReturnsDeletedValueViaItemSetting() except UnboundLocalError: print("OK, object subscript setting correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaItemDel(): class C: def __delitem__(self, attr_name): nonlocal a del a c = C() a = 1 del c[2] return a try: someFunctionThatReturnsDeletedValueViaItemDel() except UnboundLocalError: print("OK, object subscript del correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaCall(): class C: def __call__(self): nonlocal a del a c = C() a = 1 c() return a try: someFunctionThatReturnsDeletedValueViaCall() except UnboundLocalError: print("OK, object call correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaAdd(): class C: def __add__(self, other): nonlocal a del a c = C() a = 1 c += 1 return a try: someFunctionThatReturnsDeletedValueViaAdd() except UnboundLocalError: print("OK, object add correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaSub(): class C: def __add__(self, other): nonlocal a del a c = C() a = 1 c += 1 return a try: someFunctionThatReturnsDeletedValueViaSub() except UnboundLocalError: print("OK, object sub correctly deleted an item.") else: print("Ouch.!") # TODO: There is a whole lot more operations to cover. def someFunctionThatReturnsDeletedValueViaNot(): class C: def __bool__(self): nonlocal a del a return False c = C() a = 1 not c return a try: someFunctionThatReturnsDeletedValueViaNot() except UnboundLocalError: print("OK, object not correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaRepr(): class C: def __repr__(self): nonlocal a del a return "" c = C() a = 1 repr(c) return a try: someFunctionThatReturnsDeletedValueViaRepr() except UnboundLocalError: print("OK, object repr correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaStr(): class C: def __str__(self): nonlocal a del a return "" c = C() a = 1 str(c) return a try: someFunctionThatReturnsDeletedValueViaStr() except UnboundLocalError: print("OK, object str correctly deleted an item.") else: print("Ouch.!") def someFunctionThatReturnsDeletedValueViaCompare(): class C: def __lt__(self, other): nonlocal a del a return "" c = C() a = 1 c < None return a try: someFunctionThatReturnsDeletedValueViaCompare() except UnboundLocalError: print("OK, object compare correctly deleted an item.") else: print("Ouch.!") # TODO: The "del" operation may surrect a variable value by "__del__". # TODO: There must be way more than these. Nuitka-0.5.28.2/tests/basics/Unpacking35.py0000644000372000001440000000351213112214770020531 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def tupleUnpacking(): return (*a, b, *c) def listUnpacking(): return [*a, b, *c] def setUnpacking(): return {*a, b, *c} def dictUnpacking(): return {"a" : 1, **d} a = range(3) b = 5 c = range(8,10) d = {"a" : 2} print("Tuple unpacked", tupleUnpacking()) print("List unpacked", listUnpacking()) print("Set unpacked", setUnpacking()) print("Dict unpacked", dictUnpacking()) non_iterable = 2.0 def tupleUnpackingError(): try: return (*a,*non_iterable,*c) except Exception as e: return e def listUnpackingError(): try: return [*a,*non_iterable,*c] except Exception as e: return e def setUnpackingError(): try: return {*a,*non_iterable,*c} except Exception as e: return e def dictUnpackingError(): try: return {"a" : 1, **non_iterable} except Exception as e: return e print("Tuple unpacked error:", tupleUnpackingError()) print("List unpacked error:", listUnpackingError()) print("Set unpacked error:", setUnpackingError()) print("Dict unpacked error:", dictUnpackingError()) Nuitka-0.5.28.2/tests/basics/Slots.py0000644000372000001440000000311413112214770017544 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # class W1(object): def __init__(self): self.__hidden = 5 class W2(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 class _W1(object): def __init__(self): self.__hidden = 5 class _W2(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 class a_W1(object): def __init__(self): self.__hidden = 5 class a_W2(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 class W1_(object): def __init__(self): self.__hidden = 5 class W2_(object): __slots__=["__hidden"] def __init__(self): self.__hidden = 5 for w in (W1, W2, _W1, _W2, a_W1, a_W2, W1_, W2_): try: print(w) print(dir(w)) a = w() except AttributeError: print( "bug in %s" % w ) Nuitka-0.5.28.2/tests/basics/Crashers.py0000644000372000001440000000362413112214770020220 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import random def getrandom(): return random.random() def optimizerCrashIssue13(): try: print("Something with side effects that might raise.") except Exception as x: print("Caught it") raise print("Should not reach this") raise x # Just so it won't be optimized away entirely, the run time has no issue. optimizerCrashIssue13() def codegeneratorCrashIssue15(): f = float("nan") g = getrandom() # Prevent optimization of "nan"-constant return f+g # Just so it won't be optimized away entirely. codegeneratorCrashIssue15() def codegeneratorCrashIssue30(): f = getrandom() # Prevent optimization f # Will be optimized way in later versions of Nuitka. # TODO: May already be the case. def runtimeCrashIssue(): # Some C functions don't set an exception when called. from itertools import count import sys try: # This will set an error and return a value for at least Python3.2 count(1, sys.maxsize+5) except TypeError: # For Python2.6, the two arguments variant didn't exist yet. assert sys.version_info < (2,7) runtimeCrashIssue() Nuitka-0.5.28.2/tests/basics/Functions.py0000644000372000001440000002726413112214770020424 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function var_on_module_level = 1 def closureTest1(some_arg): x = 3 def enclosed(f = "default_value"): return x return enclosed print("Call closured function returning function:", closureTest1(some_arg = "ignored")()) def closureTest2(some_arg): def enclosed(f = "default_value"): return x x = 4 return enclosed print("Call closured function returning function:", closureTest2(some_arg = "ignored")()) def defaultValueTest1(no_default, some_default_constant = 1): return some_default_constant print("Call function with 2 parameters, one defaulted, and check that the default value is used:", defaultValueTest1("ignored")) def defaultValueTest1a(no_default, some_default_constant_1 = 1, some_default_constant_2 = 2): return some_default_constant_2 - some_default_constant_1 print("Call function with 3 parameters, 2 defaulted, check they are used correctly:", defaultValueTest1a("ignored")) def defaultValueTest2(no_default, some_default_variable = var_on_module_level*2): return some_default_variable print("Call function with 2 parameters, 1 defaulted with an expression, check its result", defaultValueTest2("ignored")) var_on_module_level = 2 print("Call function with 2 parameters, 1 defaulted with an expression, values have changed since, check its result", defaultValueTest2("ignored")) def contractionTest(): j = 2 return [ j + i for i in range(8) ] print("Call function that returns a list contraction:", contractionTest()) def defaultValueTest3a(no_default, funced_defaulted = defaultValueTest2(var_on_module_level)): return [ i + funced_defaulted for i in range(8) ] print("Call function that has a default value coming from a function call:", defaultValueTest3a("ignored")) def defaultValueTest3b(no_default, funced_defaulted = defaultValueTest2(var_on_module_level)): local_var = [ funced_defaulted + i for i in range(8) ] return local_var print("Call function that returns a list contraction result via a local variable:", defaultValueTest3b("ignored")) def defaultValueTest3c(no_default, funced_defaulted = defaultValueTest2(var_on_module_level)): local_var = [ [ j+funced_defaulted+1 for j in range(i) ] for i in range(8) ] return local_var print("Call function that returns a nested list contraction with input from default parameter", defaultValueTest3c("ignored")) def defaultValueTest4(no_default, funced_defaulted = lambda x: x**2): return funced_defaulted(4) print("Call function that returns value calculated by a lambda function as default parameter", defaultValueTest4("ignored")) def defaultValueTest4a(no_default, funced_defaulted = lambda x: x**2): c = 1 d = funced_defaulted(1) r = ( i+j+c+d for i, j in zip(range(8), range(9)) ) l = [] for x in r: l.append(x) return l print("Call function that has a lambda calculated default parameter and a generator expression", defaultValueTest4a("ignored")) def defaultValueTest4b(no_default, funced_defaulted = lambda x: x**3): d = funced_defaulted(1) # Nested generators l = [] for x in ( (d+j for j in range(4)) for i in range(8) ): for y in x: l.append(y) return l print("Call function that has a lambda calculated default parameter and a nested generator expression", defaultValueTest4b("ignored")) def defaultValueTest5(no_default, tuple_defaulted = (1,2,3)): return tuple_defaulted print("Call function with default value that is a tuple", defaultValueTest5("ignored")) def defaultValueTest6(no_default, list_defaulted = [1,2,3]): return list_defaulted print("Call function with default value that is a list", defaultValueTest6("ignored")) def lookup(unused, something): something.very.namelookup.chaining() something.very.namelookup.chaining() x = len("hey") def in_test(a): # if 7 in a: # print "hey" 8 in a # @NoEffect 9 not in a # @NoEffect def printing(): print("Hallo") print("du") print("da") def my_deco(function): def new_function(c, d): return function(d, c) return new_function @my_deco def decoriert(a,b): def subby(a): return 2 + a return 1+subby(b) print("Function with decoration", decoriert(3, 9)) #def var_test(a): # b = len(a) # c = len(a) def user(): global a return a a = "oh common" some_constant_tuple = (2,5,7) some_semiconstant_tuple = (2,5,a) f = a * 2 print(defaultValueTest1("ignored")) # The change of the default variable doesn't influence the default # parameter of defaultValueTest2, that means it's also calculated # at the time the function is defined. module_level = 7 print(defaultValueTest2("also ignored")) def starArgTest(a, b, c): return a, b, c print("Function called with star arg from tuple:") star_list_arg = (11, 44, 77) print(starArgTest(*star_list_arg)) print("Function called with star arg from list:") star_list_arg = [7, 8, 9] print(starArgTest(*star_list_arg)) star_dict_arg = { 'a' : 9, 'b' : 3, 'c': 8 } print("Function called with star arg from dict") print(starArgTest(**star_dict_arg)) lambda_func = lambda a, b : a < b lambda_args = (8, 9) print("Lambda function called with star args from tuple:") print(lambda_func(*lambda_args)) print("Lambda function called with star args from list:") lambda_args = [8, 7] print(lambda_func(*lambda_args)) print("Generator function without context:") def generator_without_context_function(): gen = ( x for x in range(9) ) return tuple(gen) print(generator_without_context_function()) print("Generator function with 2 iterateds:") def generator_with_2_fors(): return tuple( (x, y) for x in range(2) for y in range(3) ) print(generator_with_2_fors()) def someYielder(): yield 1 yield 2 def someYieldFunctionUser(): print("someYielder", someYielder()) result = [] for a in someYielder(): result.append(a) return result print("Function that uses some yielding function coroutine:") print(someYieldFunctionUser()) def someLoopYielder(): for i in (0, 1, 2): yield i def someLoopYieldFunctionUser(): result = [] for a in someLoopYielder(): result.append(a) return result print("Function that uses some yielding function coroutine that loops:") print(someLoopYieldFunctionUser()) def someGeneratorClosureUser(): def someGenerator(): def userOfGeneratorLocalVar(): return x+1 x = 2 yield userOfGeneratorLocalVar() yield 6 gen = someGenerator() return [next(gen), next(gen) ] print("Function generator that uses a local function accessing its local variables to yield:") print(someGeneratorClosureUser()) def someClosureUsingGeneratorUser(): offered = 7 def someGenerator(): yield offered return next(someGenerator()) print("Function generator that yield from its closure:") print(someClosureUsingGeneratorUser()) print("Function call with both star args and named args:") def someFunction(a, b, c, d): print(a, b, c, d) someFunction(a = 1, b = 2, **{ 'c' : 3, 'd' : 4 }) print("Order of evaluation of function and args:") def getFunction(): print("getFunction", end = "") def x(y, u, a, k): return y, u, k, a return x def getPlainArg1(): print("getPlainArg1", end = "") return 9 def getPlainArg2(): print("getPlainArg2", end = "") return 13 def getKeywordArg1(): print("getKeywordArg1", end = "") return 'a' def getKeywordArg2(): print("getKeywordArg2", end = "") return 'b' getFunction()(getPlainArg1(), getPlainArg2(), k = getKeywordArg1(), a = getKeywordArg2()) print() def getListStarArg(): print("getListStarArg", end = "") return [1] def getDictStarArg(): print("getDictStarArg", end = "") return { 'k' : 9 } print("Same with star args:") getFunction()(getPlainArg1(), a = getKeywordArg1(), *getListStarArg(), **getDictStarArg()) print() print("Dictionary creation order:") d = { getKeywordArg1() : getPlainArg1(), getKeywordArg2() : getPlainArg2() } print() print("Throwing an exception to a generator function:") def someGeneratorFunction(): try: yield 1 yield 2 except: yield 3 yield 4 gen1 = someGeneratorFunction() print("Fresh Generator Function throwing gives", end = "") try: print(gen1.throw(ValueError)), except ValueError: print("exception indeed") gen2 = someGeneratorFunction() print("Used Generator Function throwing gives", end = "") next(gen2) print(gen2.throw(ValueError), "indeed") gen3 = someGeneratorFunction() print("Fresh Generator Function close gives", end = "") print(gen3.close()) gen4 = someGeneratorFunction() print("Used Generator Function that mis-catches close gives", end = "") next(gen4) try: print(gen4.close(), end = "") except RuntimeError: print("runtime exception indeed") gen5 = someGeneratorFunction() print("Used Generator Function close gives", end = "") next(gen5) next(gen5) next(gen5) print(gen5.close(), end = "") def receivingGenerator(): while True: a = yield 4 yield a print("Generator function that receives", end = "") gen6 = receivingGenerator() print(next(gen6), end = "") print(gen6.send(5), end = "") print(gen6.send(6), end = "") print(gen6.send(7), end = "") print(gen6.send(8)) print("Generator function whose generator is copied", end = "") def generatorFunction(): yield 1 yield 2 gen7 = generatorFunction() next(gen7) gen8 = iter(gen7) print(next(gen8)) def doubleStarArgs(*a, **d): return a, d try: from UserDict import UserDict except ImportError: print("Using Python3, making own non-dict dict:") class UserDict(dict): pass print("Function that has keyword argument matching the list star arg name", end = "") print(doubleStarArgs(1, **UserDict(a = 2))) def generatorFunctionUnusedArg(a): yield 1 generatorFunctionUnusedArg(3) def closureHavingGenerator(arg): def gen(x = 1): yield arg return gen() print("Function generator that has a closure and default argument", end = "") print(list(closureHavingGenerator(3))) def functionWithDualStarArgsAndKeywordsOnly(a1, a2, a3, a4, b): return a1, a2, a3, a4, b l = [1, 2, 3] d = {'b': 8} print("Dual star args, but not positional call", functionWithDualStarArgsAndKeywordsOnly(a4 = 1, *l, **d)) def posDoubleStarArgsFunction(a, b, c, *l, **d): return a, b, c, l, d l = [2] d = { "other" : 7, 'c' : 3 } print("Dual star args consuming function", posDoubleStarArgsFunction(1, *l, **d)) import inspect, sys for value in sorted(dir()): main_value = getattr(sys.modules[ "__main__" ], value) if inspect.isfunction(main_value): print(main_value, main_value.__code__.co_varnames[:main_value.__code__.co_argcount]) # inspect.getargs( main_value.func_code ) # TODO: Make this work as well, currently disabled, because of nested arguments not # being compatible yet. # print main_value, main_value.func_code.co_varnames, inspect.getargspec( main_value ) pass Nuitka-0.5.28.2/tests/basics/Referencing33.py0000644000372000001440000000473513122472300021043 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython ) checkDebugPython() def simpleFunction1(): def g(): for a in range(20): yield a def h(): yield 4 yield 5 yield 6 def f(): yield from g() yield from h() x = list( f() ) def simpleFunction2(): def g(): for a in range(20): yield a def h(): yield 4 yield 5 yield 6 raise TypeError def f(): yield from g() yield from h() try: x = list( f() ) except TypeError: pass # Broken iterator class. class Broken: def __iter__(self): return self def __next__(self): return 1 def __getattr__(self, attr): 1/0 def simpleFunction3(): def g(): yield from Broken() try: gi = g() next(gi) except Exception: pass def simpleFunction4(): def g(): yield from Broken() try: gi = g() next(gi) gi.throw(AttributeError) except Exception: pass def simpleFunction5(): def g(): yield from (2,3) return list( g() ) # These need stderr to be wrapped. tests_stderr = (3, 4) # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/Varargs.py0000644000372000001440000000410413112214770020045 0ustar hayenusers00000000000000# -*- coding: utf-8 -*- # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def plain_list_dict_args_function(plain, *arg_list, **arg_dict): print("plain", plain, "arg_list", arg_list, "arg_dict", arg_dict) def plain_list_args_function(plain, *arg_list): print(plain, arg_list) def plain_dict_args_function(plain, **arg_dict): print(plain, arg_dict) print("Function with plain arg and varargs dict:") plain_dict_args_function(1, a = 2, b = 3, c = 4) plain_dict_args_function(1) print("Function with plain arg and varargs list:") plain_list_args_function(1, 2, 3, 4) plain_list_args_function(1) print("Function with plain arg, varargs list and varargs dict:") plain_list_dict_args_function(1, 2, z = 3) plain_list_dict_args_function(1, 2, 3) plain_list_dict_args_function(1, a = 2, b = 3, c = 4) def list_dict_args_function(*arg_list, **arg_dict): print(arg_list, arg_dict) def list_args_function(*arg_list): print(arg_list) def dict_args_function(**arg_dict): print(arg_dict) print("Function with plain arg and varargs dict:") dict_args_function(a = 2, b = 3, c = 4) dict_args_function() print("Function with plain arg and varargs list:") list_args_function(2, 3, 4) list_args_function() print("Function with plain arg, varargs list and varargs dict:") list_dict_args_function(2, z = 3) list_dict_args_function(2, 3) list_dict_args_function(a = 2, b = 3, c = 4) Nuitka-0.5.28.2/tests/basics/TryYieldFinally.py0000644000372000001440000000442513112214770021532 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def tryContinueFinallyTest(): for x in range(10): try: if x % 2 == 1: continue finally: yield x yield '-' def tryBreakFinallyTest(): for x in range(10): try: if x == 5: break finally: yield x yield '-' def tryFinallyAfterYield(): try: yield 3 finally: print("Executing finally") def tryReturnFinallyYield(): try: return finally: yield 1 def tryReturnExceptYield(): try: return except StopIteration: print("Caught StopIteration") yield 2 except: yield 1 else: print("No exception") def tryStopIterationExceptYield(): try: raise StopIteration except StopIteration: print("Caught StopIteration") yield 2 except: yield 1 else: print("No exception") print("Check if finally is executed in a continue using for loop:") print(tuple(tryContinueFinallyTest())) print("Check if finally is executed in a break using for loop:") print(tuple(tryBreakFinallyTest())) print("Check what try yield finally something does:") print(tuple(tryFinallyAfterYield())) print("Check if yield is executed in finally after return:") print(tuple(tryReturnFinallyYield())) print("Check if yield is executed in except after return:") print(tuple(tryReturnExceptYield())) print("Check if yield is executed in except after StopIteration:") print(tuple(tryReturnExceptYield())) Nuitka-0.5.28.2/tests/basics/Referencing36.py0000644000372000001440000000611513122472300021040 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os, types # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython, async_iterate, run_async ) checkDebugPython() def run_until_complete(coro): exc = False while True: try: if exc: exc = False fut = coro.throw(AwaitException) else: fut = coro.send(None) except StopIteration as ex: return ex.args[0] if fut == ('throw',): exc = True def simpleFunction1(): async def gen1(): try: yield except: pass async def run(): g = gen1() await g.asend(None) await g.asend(None) try: run_async(run()) except StopAsyncIteration: pass def simpleFunction2(): async def async_gen(): try: yield 1 yield 1.1 1 / 0 finally: yield 2 yield 3 yield 100 async_iterate(async_gen()) @types.coroutine def awaitable(*, throw=False): if throw: yield ('throw',) else: yield ('result',) async def gen2(): await awaitable() a = yield 123 # self.assertIs(a, None) await awaitable() yield 456 await awaitable() yield 789 def simpleFunction3(): def to_list(gen): async def iterate(): res = [] async for i in gen: res.append(i) return res return run_until_complete(iterate()) async def run2(): return to_list(gen2()) run_async(run2()) def simpleFunction4(): g = gen2() ai = g.__aiter__() an = ai.__anext__() an.__next__() try: ai.__anext__().__next__() except StopIteration as ex: pass ai.__anext__().__next__() # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/Branching.py0000644000372000001440000000770413112214770020344 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Some random branching to cover most common cases. """ from __future__ import print_function def branchingFunction(a, b, c): print("branchingFunction:", a, b, c) print("a or b", a or b) print("a and b", a and b) print("not a", not a) print("not b", not b) print("Simple branch with both branches") if a: l = "YES" else: l = "NO" print(a, "->", l) print("Simple not branch with both branches") if not a: l = "YES" else: l = "NO" print(not a, "->", l) print("Simple branch with a nested branch in else path:") if a: m = "yes" else: if True: m = "no" print(a, "->", m) print("Triple 'and' chain:") v = "NO" if a and b and c: v = "YES" print(a, b, c, "->", v) print("Triple or chain:") k = "NO" if a or b or c: k = "YES" print(a, b, c, "->", k) print("Nested 'if not' chain:") p = "NO" if not a: if not b: p = "YES" print("not a, not b", not a, not b, "->", p) print("or condition in braces:") q = "NO" if (a or b): q = "YES" print("(a or b) ->", q) print("Braced if not with two 'or'") if not (a or b or c): q = "YES" else: q = "NO" print("not (a or b or c)", q) print("Braced if not with one 'or'") q = "NO" if not (b or b): q = "YES" print("not (b or b)", q) print("Expression a or b", a or b) print("Expression not(a or b)", not(a or b)) print("Expression a and (b+5)", a and (b+5)) print("Expression (b if b else 2)", (b if b else 2)) print("Expression (a and (b if b else 2))", (a and (b if b else 2))) print("Braced if not chain with 'and' and conditional expression:") if not (a and (b if b else 2)): print("oki") print("Nested if chain with outer else:") d=1 if a: if b or c: if d: print("inside nest") else: print("outer else") print("Complex conditional expression:") v = (3 if a-1 else 0) or \ (b or (c*2 if c else 6) if b-1 else a and b and c) print(v) if True: print("Predictable branch taken") branchingFunction(1,0,3) x = 3 def optimizationVictim(): if x: pass else: pass if x: pass pass optimizationVictim() def dontOptimizeSideEffects(): print("Lets see, if conditional expression in known true values are correctly handled:") def returnTrue(): print("function 'returnTrue' was called as expected") return True def returnFalse(): print("function 'returnFalse' should not have beeen called") return False if (returnTrue() or returnFalse(),): print("Taken branch as expected.") else: print("Bad branch taken.") dontOptimizeSideEffects() def dontOptimizeTruthCheck(): class A: def __nonzero__(self): raise ValueError __bool__ = __nonzero__ a = A() if a: pass try: print("Check that branch conditions are not optimized way: ", end = "") dontOptimizeTruthCheck() print("FAIL.") except ValueError: print("OK.") Nuitka-0.5.28.2/tests/basics/Assignments32.py0000644000372000001440000001460213112214770021104 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Basic assignment forms from various iterables:") a, b = 1, 2 # simple sequence assignment print(a, b) a, b = ['green', 'blue'] # list assignment print(a, b) a, b = 'XY' # string assignment print(a, b) a, b = range(1,5,2) # any iterable will do print(a, b) print("Using braces on unpacking side:") (a,b), c = "XY", "Z" # a = 'X', b = 'Y', c = 'Z' print(a, b, c) print("Too many values:") try: (a,b), c = "XYZ" # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) print("Too few values:") try: (a,b), c = "XY" # ERROR -- need more than 1 value to unpack except Exception as e: print(repr(e)) print(a, b, c) print("More complex right hand side, consisting of multiple values:") (a,b), c, = [1,2],'this' # a = '1', b = '2', c = 'this' print(a, b, c) print("More complex right hand side, too many values:") try: (a,b), (c,) = [1,2],'this' # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) print("Extended sequence * unpacking:") a, *b = 1,2,3,4,5 # a = 1, b = [2,3,4,5] print(a, b) *a, b = 1,2,3,4,5 # a = [1,2,3,4], b = 5 print(a, b) a, *b, c = 1,2,3,4,5 # a = 1, b = [2,3,4], c = 5 print(a, b) a, *b = 'X' # a = 'X', b = [] print(a, b) *a, b = 'X' # a = [], b = 'X' print(a, b) a, *b, c = "XY" # a = 'X', b = [], c = 'Y' print(a, b) a, *b, c = "X...Y" # a = 'X', b = ['.','.','.'], c = 'Y' print(a, b, c) a, b, *c = 1,2,3 # a = 1, b = 2, c = [3] print(a, b, c) a, b, c, *d = 1,2,3 # a = 1, b = 2, c = 3, d = [] print(a, b, c, d) (a,b), c = [1,2],'this' # a = '1', b = '2', c = 'this' print(a, b, c) (a,b), *c = [1,2],'this' # a = '1', b = '2', c = ['this'] print(a, b, c) (a,b), c, *d = [1,2],'this' # a = '1', b = '2', c = 'this', d = [] print(a, b, c, d) (a,b), *c, d = [1,2],'this' # a = '1', b = '2', c = [], d = 'this' print(a, b, c, d) (a,b), (c, *d) = [1,2],'this' # a = '1', b = '2', c = 't', d = ['h', 'i', 's'] print(a, b, c, d) *a, = (1,2) # a = [1,2] print("Extended sequence * unpacking with non-iterable:") try: *a, = 1 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a) print("Extended sequence * unpacking with list:") *a, = [1] # a = [1] print(a) print("Extended sequence * unpacking with tuple:") *a, = (1,) # a = [1] print(a) print("Extended sequence * unpacking with fixed right side:") *a, b = [1] # a = [], b = 1 print(a, b) *a, b = (1,) # a = [], b = 1 print(a, b) print("Unpacking too many values:") try: (a,b),c = 1,2,3 # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) print("Unpacking with star argument changes error:") try: (a,b), *c = 1,2,3 # ERROR - 'int' object is not iterable except Exception as e: print(repr(e)) print(a, b, c) print("Unpacking with star argument after tuple unpack:") (a,b), *c = 'XY', 2, 3 # a = 'X', b = 'Y', c = [2,3] print(a, b, c) print("Extended sequence unpacking, nested:") try: (a,b),c = 1,2,3 # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) *(a,b), c = 1,2,3 # a = 1, b = 2, c = 3 print(a, b, c) *(a,b), = 1,2 # a = 1, b = 2 print(a, b) *(a,b), = 'XY' # a = 'X', b = 'Y' print(a, b) try: *(a, b), = 'this' # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b) *(a, *b), = 'this' # a = 't', b = ['h', 'i', 's'] print(a, b) *(a, *b), c = 'this' # a = 't', b = ['h', 'i'], c = 's' print(a, b, c) *(a,*b), = 1,2,3,3,4,5,6,7 # a = 1, b = [2, 3, 3, 4, 5, 6, 7] print(a, b) try: *(a,*b), (*c,) = 1,2,3,3,4,5,6,7 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print("unchanged", a, b, c) print("Unpacking with nested stars:") *(a,*b), c = 1,2,3,3,4,5,6,7 # a = 1, b = [2, 3, 3, 4, 5, 6], c = 7 print(a, b, c) print("Unpacking with even more nested stars:") *(a,*b), (*c,) = 1,2,3,4,5,'XY' # a = 1, b = [2, 3, 4, 5], c = ['X', 'Y'] print(a, b, c) *(a,*b), c, d = 1,2,3,3,4,5,6,7 # a = 1, b = [2, 3, 3, 4, 5], c = 6, d = 7 print("starting", a, b, c, d) try: *(a,*b), (c, d) = 1,2,3,3,4,5,6,7 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print("unchanged", a, b, c, d) try: *(a,*b), (*c, d) = 1,2,3,3,4,5,6,7 # ERROR -- 'int' object is not iterable except Exception as e: print(repr(e)) print(a, b, c, d) try: *(a,b), c = 'XY', 3 # ERROR -- need more than 1 value to unpack except Exception as e: print(repr(e)) print("unchanged", a, b, c) *(*a,b), c = 'XY', 3 # a = [], b = 'XY', c = 3 print(a, b, c) (a,b), c = 'XY', 3 # a = 'X', b = 'Y', c = 3 print(a, b, c) *(a,b), c = 'XY', 3, 4 # a = 'XY', b = 3, c = 4 print(a, b, c) *(*a,b), c = 'XY', 3, 4 # a = ['XY'], b = 3, c = 4 print(a, b, c) try: (a,b), c = 'XY', 3, 4 # ERROR -- too many values to unpack except Exception as e: print(repr(e)) print(a, b, c) Nuitka-0.5.28.2/tests/basics/MainPrograms.py0000644000372000001440000000266213112214770021046 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("Module name is", __name__) class SomeClass: pass print("Class inside main module names its module as", repr(SomeClass.__module__)) if __name__ == "__main__": print("Executed as __main__:") import sys, os # The sys.argv[0] might contain ".exe", ".py" or no suffix at all. # Remove it, so the "diff" output is more acceptable. args = sys.argv[:] args[0] = os.path.basename(args[0]).replace(".exe", ".py").replace(".py", "") print("Arguments were (stripped argv[0] suffix):", repr(args)) # Output the flags, so we can test if we are compatible with these too. print("The sys.flags are:", sys.flags) Nuitka-0.5.28.2/tests/basics/Empty.py0000644000372000001440000000140213112214770017534 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Nuitka-0.5.28.2/tests/basics/TryReturnFinally.py0000644000372000001440000000542313112214770021742 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # In this test we show that return in try/finally executes the finally part # just fine. from __future__ import print_function import sys def eight(): return 8 def nine(): return 9 def raisy1(): return 1 / 0 def raisy2(): return 1() def raisy3(arg): raise TypeError(arg) def returnInTried(for_call): try: print("returnInTried with exception info in tried block:", sys.exc_info()) return for_call() finally: print("returnInTried with exception info in final block:", sys.exc_info()) def returnInFinally(for_call): try: print("returnInFinally with exception info in tried block:", sys.exc_info()) finally: print("returnInFinally with exception info in final block:", sys.exc_info()) return for_call() print("Standard try finally with return in tried block:") print("result", returnInTried(eight)) print('*' * 80) print("Standard try finally with return in final block:") print("result", returnInFinally(nine)) print('*' * 80) print("Exception raising try finally with return in tried block:") try: print("result", returnInTried(raisy1)) except Exception as e: print("Gave exception", repr(e)) print('*' * 80) print("Exception raising try finally with return in final block:") try: print("result", returnInFinally(raisy2)) except Exception as e: print("Gave exception", repr(e)) print('*' * 80) try: raisy3("unreal 1") except Exception as outer_e: print("Exception raising try finally with return in tried block:") try: print("result", returnInTried(raisy1)) except Exception as e: print("Gave exception", repr(e)) print("Handler exception remains:", repr(outer_e)) print('*' * 80) try: raisy3("unreal 2") except Exception as outer_e: print("Exception raising try finally with return in final block:") try: print("result", returnInFinally(raisy2)) except Exception as e: print("Gave exception", repr(e)) print("Handler exception remains:", repr(outer_e)) print('*' * 80) Nuitka-0.5.28.2/tests/basics/Unicode.py0000644000372000001440000000210313112214770020023 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(u"gfcrk") print(repr(u"g\xfcrk")) print(r"""\x00""") print("\ttest\n") print(""" something with new lines""") print(u"favicon.ico (32\xd732)") # TODO: Python3 has a problem here, hard to find, disabled for now. if False: encoding = "utf-16-be" print("[\uDC80]".encode(encoding)) print("[\\udc80]") Nuitka-0.5.28.2/tests/basics/DefaultParameters.py0000644000372000001440000000416413112214770022056 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function module_level = 1 def defaultValueTest1(no_default, some_default_constant = 1): return some_default_constant def defaultValueTest2(no_default, some_default_computed = module_level*2): local_var = no_default return local_var, some_default_computed def defaultValueTest3(no_default, func_defaulted = defaultValueTest1(module_level)): return [ func_defaulted for _i in range(8) ] def defaultValueTest4(no_default, funced_defaulted = lambda x: x**2): c = 1 d = 1 return ( i+c+d for i in range(8) ) def defaultValueTest5(no_default, tuple_defaulted = (1,2,3)): return tuple_defaulted def defaultValueTest6(no_default, list_defaulted = [1,2,3]): list_defaulted.append(5) return list_defaulted print(defaultValueTest1("ignored")) # The change of the default variable doesn't influence the default # parameter of defaultValueTest2, that means it's also calculated # at the time the function is defined. module_level = 7 print(defaultValueTest2("also ignored")) print(defaultValueTest3("nono not again")) print(list(defaultValueTest4("unused"))) print(defaultValueTest5("unused")) print(defaultValueTest6("unused"), end = "") print(defaultValueTest6("unused")) print(defaultValueTest6.__defaults__) defaultValueTest6.func_defaults = ([1,2,3],) print(defaultValueTest6.__defaults__) print(defaultValueTest6(1)) Nuitka-0.5.28.2/tests/basics/Asserts.py0000644000372000001440000000322313112214770020065 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def testAssert1(): assert False return 1 def testAssert2(): assert True return 1 def testAssert3(): assert False, "argument" return 1 try: print("Function that will assert.") testAssert1() print("No exception.") except Exception as e: print("Raised", type(e), e) try: print("Function that will not assert.") testAssert2() print("No exception.") except Exception as e: print("Raised", type(e), e) try: print("Function that will assert with argument.") testAssert3() print("No exception.") except Exception as e: print("Raised", type(e), e) try: print("Assertion with tuple argument.", end = "") assert False, (3,) except AssertionError as e: print(str(e)) try: print("Assertion with plain argument.", end = "") assert False, 3 except AssertionError as e: print(str(e)) Nuitka-0.5.28.2/tests/basics/OverflowFunctions_2.py0000644000372000001440000000272513112214770022364 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def starImporterFunction(): from sys import * # @UnusedWildImport print "Version", version.split()[0].split('.')[:-1] starImporterFunction() def deepExec(): for_closure = 3 def deeper(): for_closure_as_well = 4 def execFunction(): code = "f=2" # Can fool it to nest exec code in None, None print "Locals now", locals() print "Closure one level up was taken", for_closure_as_well print "Closure two levels up was taken", for_closure print "Globals still work", starImporterFunction print "Added local from code", f # @UndefinedVariable execFunction() deeper() deepExec() Nuitka-0.5.28.2/tests/basics/ExceptionRaising33.py0000644000372000001440000000266013112214770022066 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys print("Testing exception changes between generator switches:") def yieldExceptionInteraction(): def yield_raise(): try: raise KeyError("caught") except KeyError: yield from sys.exc_info() yield from sys.exc_info() yield from sys.exc_info() g = yield_raise() print("Initial yield from catch in generator", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("Second yield from the catch reentered", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("After leaving the catch generator yielded", next(g)) yieldExceptionInteraction() Nuitka-0.5.28.2/tests/basics/TryExceptContinue.py0000644000372000001440000000425713112214770022105 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def tryWhileExceptContinueTest(): print("Check if continue is executed in a except handler using for loop:") global undefined x = 0 while x < 10: x += 1 try: if x % 2 == 1: undefined except: print(x, end = ' ') continue print('-', end = ' ') print() def tryForExceptContinueTest(): print("Check if continue is executed in a except handler using for loop:") for x in range(10): try: if x % 2 == 1: undefined except: print(x, end = ' ') continue print('-', end = ' ') print() def tryWhileExceptBreakTest(): print("Check if break is executed in a except handler using while loop:") x = 0 while x < 10: x += 1 try: if x == 5: undefined except: print(x, end = ' ') break print('-', end = ' ') print() def tryForExceptBreakTest(): print("Check if break is executed in a except handler using for loop:") for x in range(10): try: if x == 5: undefined except: print(x, end = ' ') break print('-', end = ' ') print() tryWhileExceptContinueTest() tryWhileExceptBreakTest() tryForExceptContinueTest() tryForExceptBreakTest() Nuitka-0.5.28.2/tests/basics/Printing_2.py0000644000372000001440000000263313112214770020460 0ustar hayenusers00000000000000# -*- coding: utf-8 -*- # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # All of these should be identical with correct software behavior. print "Output with newline." print "Output", "with", "newline." print "Output trailing spaces ", "with ", "newline." print "Output ", print "with ", print "newline." print "Output\twith tab" print "Output\t", print "with tab" # These ones gave errors with previous literal bugs: print "changed 2" print "foo%sbar%sfred%sbob?????" a = "partial print" # b doesn't exist try: print a, undefined_global # @UndefinedVariable except Exception, e: print "then occurred", repr(e) print "No newline at the end", x = 1 print """ New line is no soft space, is it """, x Nuitka-0.5.28.2/tests/basics/Constants.py0000644000372000001440000001041013134660221020411 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # """ Playing around with constants only. """ from __future__ import print_function try: long except NameError: long = int # @ReservedAssignment def displayDict(d): result = '{' for key, value in sorted(d.items()): result += "%s: %s" % (key, value) result += '}' print("A bunch of constants and their representation:") for value in (0, 3, -4, 17, "hey", (0,), 0.0, -0.0): print(value, ':', repr(value)) print("Comparing constants, optimizable:") print(1 == 0) print("Representation of long constants:") a = long(0) print(repr(long(0)), repr(a) == "0L") print("Identity of empty dictionary constants:") print({} is {}) a = ({}, []) a[0][1] = 2 a[1].append(3) print("Mutable list and dict inside an immutable tuple:") print(a) print("Empty list and dict are hopefully unchanged:") print(({}, [])) def argChanger(a): a[0][1] = 2 a[1].append(3) return a print("Mutable list and dict inside an immutable tuple as arguments:") print(argChanger(({}, []))) print("Empty list and dict are hopefully still unchanged:") print(({}, [])) print("Set constants:") print(set(["foo"])) def mutableConstantChanger(): a = ([1, 2], [3]) print("Start out with value:") print(a) a[1].append(5) print("Changed to value:") print(a) d = { 'l': [], 'm' : [] } print("Start out with value:") print(d) d['l'].append(7) print("Changed to value:") print(d) spec = dict(qual = [], storage = set(), type = [], function = set(), q = 1) spec["type"].insert(0, 2) spec["storage"].add(3) print("Dictionary created from dict built-in.") print(sorted(spec)) mutableConstantChanger() print("Redo constant changes, to catch corruptions:") mutableConstantChanger() def defaultKeepsIdentity(arg = "str_value"): print("Default constant values are still shared if immutable:") print(arg is "str_value") defaultKeepsIdentity() # Dictionary creation from call arguments. def dd(**d): return d def f(): def one(): print("one") def two(): print("two") a = dd(qual = one(), storage = two(), type = [], function = []) print("f mutable", displayDict(a)) a = dd(qual = 1, storage = 2, type = 3, function = 4) print("f immutable", displayDict(a)) x = { "p" : 7 } a = dd(qual=[], storage=[], type=[], function=[],**x) print("f ext mutable", displayDict(a)) x = { "p" : 8 } a = dd(qual=1, storage=2, type=3, function=4,**x) print("f ext immutable", displayDict(a)) f() # Dictionary creation one after another x={} x["function"] = [] x["type"] = [] x["storage"] = [] x["qual"] = [] print("Manual built dictionary:", x) x={} x["function"] = 1 x["type"] = 2 x["storage"] = 3 x["qual"] = 4 print("Manual built dictionary:", x) # Constants in the code must be created differently. d = { "qual" : [], "storage" : [], "type2" : [], "function" : [] } print("Mutable values dictionary constant:", displayDict(d)) d = { "qual" : 1, "storage" : 2, "type2" : 3, "function" : 4 } print("Immutable values dictionary constant:", displayDict(d)) # Constants that might be difficult min_signed_int = int(-(2**(8*8-1)-1)-1) print("Small int:", min_signed_int, type(min_signed_int)) min_signed_int = int(-(2**(8*4-1)-1)-1) print("Small int", min_signed_int, type(min_signed_int)) # Constants that might be difficult min_signed_long = long(-(2**(8*8-1)-1)-1) print("Small long", min_signed_long, type(min_signed_long)) min_signed_long = long(-(2**(8*4-1)-1)-1) print("Small long", min_signed_long, type(min_signed_long)) Nuitka-0.5.28.2/tests/basics/Future32.py0000644000372000001440000000160013112214770020055 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import barry_as_FLUFL print( eval("1 <> 2") ) print( eval('"a"<>"b"') ) print( eval("range(7) <> range(7)") ) Nuitka-0.5.28.2/tests/basics/Assignments.py0000644000372000001440000001300013112214770020726 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys def someFunction(): a = 2 print("Simple assignment to variable:", a) b = c = 3 print("Assignment to 2 variables", b, c) z = [1, 2, 3] z[2] = z[1] = 5 print("Assignment to list subscripts:", z) d, e = 1, 2 print("Assignment to variable tuple:", d, e) [f, g] = 7, 9 print("Assignment to variable list:", f, g) j = [h, i] = (7, 9) print("Complex Assignment from variable list:", j, type(j), h, i) a, (b,c) = 1, (2,3) print("Assignment to nested tuples:", a, b, c) v = [1, 2, 3, 4] v[2:3] = (8,9) print("Assignment to list slice", v) def varargsFunction(*args): f1, f2, f3, f4 = args print("Assignment from list", f1, f2, f3, f4) def otherFunction(): class Iterable: def __iter__(self): return iter(range(3)) a, b, c = Iterable() print("Assignments from iterable", a ,b ,c) print("Assignments from too small iterable",end = ' ') try: f, g = 1, except Exception as e: print("gave", type(e), repr(e)) try: print(f) except UnboundLocalError: print("Variable f is untouched") try: print(g) except UnboundLocalError: print("Variable g is untouched") print("Assignments from too large iterable", end = ' ') try: d, j = 1, 2, 3 except Exception as e: print("gave", type(e), repr(e)) try: print(d) except UnboundLocalError: print("Variable d is untouched") try: print(j) except UnboundLocalError: print("Variable j is untouched") class BasicIterClass: def __init__(self, n): self.n = n self.i = 0 def __next__(self): res = self.i if res >= self.n: raise StopIteration self.i = res + 1 return res if sys.version_info[0] < 3: def next(self): return self.__next__() class IteratingSequenceClass: def __init__(self, n): self.n = n def __iter__(self): return BasicIterClass(self.n) try: a, b, c = IteratingSequenceClass(2) except ValueError: print("Exception from iterating over too short class", sys.exc_info()) def anotherFunction(): d = {} print("Assignment to dictionary with comma subscript:", end = "") # d[ "f" ] = 3 d['a', 'b'] = 6 d['c', 'b'] = 9 print(sorted(d.items())) def swapVariables(): print("Strange swap form:") a = 1 b = 2 a, b, a = b, a, b print(a, b) def interuptedUnpack(): a = 1 b = 2 print("Assignment from a too short tuple to multiple targets:", end = ' ') try: s = a, c, d = s except ValueError as e: print("gives ValueError", repr(e)) try: print(c) except UnboundLocalError as e: print("and then nothing is assigned:", repr(e)) else: del d del a, b z = [] try: a, z.unknown, b = 1, 2, 3 except AttributeError: print("Interrupted unpack, leaves value assigned", a) def multiTargetInterrupt(): a = 1 b = 2 print("Multiple, overlapping targets", end = "") d = c, d = a, b # @UnusedVariable print(d, c, end = "") del c del d c, d = d = a, b print(d, c) print("Error during multiple assignments", end = "") del c del d e = 9 z = [] try: c, d = e, z.a = a, b except AttributeError: print("having attribute error", c, d, e) del c del d e = 9 print("Error during multiple assignments", end = "") try: c, d = z.a, e = a, b except AttributeError: print("having attribute error", c, d, e) def optimizeableTargets(): a = [1, 2] a[int(1)] = 3 print("Optimizable slice operation, results in", a) def complexDel(): a = b = c = d = 1 del a, b, (c, d) try: print(c) except UnboundLocalError as e: print("yes, del worked", repr(e)) def sliceDel(): # Python3 ranges are not lists. a = list(range(6)) del a[2:4] print("Del slice operation, results in", a) def globalErrors(): global unassigned_1, unassigned_2 try: unassigned_1 = unassigned_1 except NameError as e: print("Accessing unassigned global gives", repr(e)) try: del unassigned_2 except NameError as e: print("Del on unassigned global gives", repr(e)) someFunction() varargsFunction(1,2,3,4) otherFunction() anotherFunction() swapVariables() interuptedUnpack() multiTargetInterrupt() optimizeableTargets() complexDel() sliceDel() globalErrors() Nuitka-0.5.28.2/tests/basics/Classes34.py0000644000372000001440000000244213112214770020207 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from enum import Enum print("Enum class with duplicate enumeration values:") try: class Color(Enum): red = 1 green = 2 blue = 3 red = 4 print("not allowed to get here") except Exception as e: print("Occurred", e) print("Class variable that conflicts with closure variable:") def testClassNamespaceOverridesClosure(): # See #17853. x = 42 class X: locals()['x'] = 43 y = x print("should be 43:", X.y) testClassNamespaceOverridesClosure() Nuitka-0.5.28.2/tests/basics/HelloWorld_2.py0000644000372000001440000000240413122472300020731 0ustar hayenusers00000000000000# -*- coding: utf-8 -*- # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print "Hello World from module main code" def printHelloWorld(): print "Hello World from function main code" print printHelloWorld printHelloWorld() def printHelloWorld2(arg): print arg print printHelloWorld2 printHelloWorld2("Hello World from function positional argument") printHelloWorld2(arg = "Hello World from function keyword argument") def printHelloWorld3(arg = "Hello World from function default argument"): print arg print printHelloWorld3 printHelloWorld3() Nuitka-0.5.28.2/tests/basics/BuiltinsTest.py0000644000372000001440000002300213112214770021067 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def someFunctionWritingLocals(): x = 1 r = locals() # This is without effect on r. It doesn't mention y at all y = 2 # This adds z to the locals, but only that. r[ 'z' ] = 3 del x try: z except Exception as e: print("Accessing z writing to locals gives Exception", e) return r, y def someFunctionWritingLocalsContainingExec(): x = 1 r = locals() # This is without effect on r. It doesn't mention y at all y = 2 # This adds z to the locals, but only that. r[ 'z' ] = 3 try: z except Exception as e: print("Accessing z writing to locals in exec function gives Exception", e) return r, y # Note: This exec is dead code, and still changes the behaviour of # CPython, because it detects exec during parse already. exec("") print("Testing locals():") print(someFunctionWritingLocals()) print(someFunctionWritingLocalsContainingExec()) def displayDict(d): if "__loader__" in d: d = dict(d) d["__loader__"] = "<__loader__ removed>" if "__file__" in d: d = dict(d) d["__file__"] = "<__file__ removed>" import pprint return pprint.pformat(d) print("Vars on module level", displayDict(vars())) module_locals = locals() # Patch away "__file__" path in a hard to detect way. This will make sure, # repeated calls to locals really get the same dictionary. import os module_locals["__file__"] = os.path.basename(module_locals[ "__file__" ]) del module_locals print("Use of locals on the module level", displayDict(locals())) def someFunctionUsingGlobals(): g = globals() g["hallo"] = "du" global hallo print("hallo", hallo) print("Testing globals():") someFunctionUsingGlobals() print("Testing dir():") print("Module dir", end = ' ') def someFunctionUsingDir(): x = someFunctionUsingGlobals() print("Function dir", dir()) return x someFunctionUsingDir() print("Making a new type, with type() and 3 args:", end = ' ') new_class = type("Name", (object,), {}) print(new_class, new_class()) print("None has type", type(None)) print("Constant ranges", range(2), range(1, 6), range(3, 0, -1), range(3, 8, 2), range(5, -5, -3)) print("Border cases", range(0), range(-1), range(-1, 1)) print("Corner case large negative value", range(-2**100)) print("Corner case with large start/end values in small range", range(2**100,2**100+2)) try: print("Range with 0 step gives:", end = ' ') print(range(3, 8, 0)) except ValueError as e: print(repr(e)) try: print("Range with float:", end = ' ') print(range(1.0)) except TypeError as e: print("Gives exception:", repr(e)) try: print("Empty range call", end = ' ') print(range()) except TypeError as e: print("Gives exception:", e) print("List from iterable", list("abc"), list()) print("List from sequence", list(sequence = (0, 1, 2))) print("Tuple from iterable", tuple("cda"), tuple()) print("Tuple from sequence", tuple(sequence = (0, 1, 2))) print("Dictionary from iterable and keywords", displayDict(dict(("ab", (1, 2)), f = 1, g = 1))) print("More constant dictionaries", {"two": 2, "one": 1}, {}, dict()) g = {"two": 2, "one": 1} print("Variable dictionary", dict(g)) print("Found during optimization", dict(dict({"le": 2, "la": 1}), fu = 3), dict(named = dict({"le": 2, "la": 1}))) print("Floats from constants", float("3.0"), float(x = 9.0), float()) print("Found during optimization", float(float("3.2")), float(x = float(11.0))) print("Complex from constants", complex("3.0j"), complex(real = 9.0), complex(imag = 9.0), complex(1,2), complex()) print("Found during optimization", complex(float("3.2")), complex(real = float(11.0)), complex(imag = float(11.0))) print("Strs from constants", str("3.3"), str(object = 9.1), str()) print("Found during optimization", str(float("3.3")), str(object = float(12.0))) print("Bools from constants", bool("3.3"), bool(x = 9.1), bool(0), bool()) print("Found during optimization", bool(float("3.3")), bool(x = float(0.0))) print("Ints from constants", int('3'), int(x = '9'), int('f', 16), int(x = 'e', base = 16), int("0101", base = 2), int(0), int()) print("Found ints during optimization", int(int('3')), int(x = int(0.0))) try: print("Longs from constants", long('3'), long(x = '9'), long('f', 16), long(x = 'e', base = 16), long("0101", base = 2), long(0), long()) print("Found longs during optimization", long(long('3')), long(x = long(0.0))) except NameError: print("Python3 has no long type.") pass try: print("Int with only base", int(base = 2), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") try: print("Int with large base", int(2, 37), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") try: print("Long with only base", int(base = 2), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") print("Oct from constants", oct(467), oct(0)) print("Found during optimization", oct(int('3'))) print("Hex from constants", hex(467), hex(0)) print("Found during optimization", hex(int('3'))) print("Bin from constants", bin(467), bin(0)) print("Found during optimization", bin(int('3'))) try: int(1,2,3) except Exception as e: print("Too many args gave", repr(e)) try: int(y = 1) except Exception as e: print("Wrong arg", repr(e)) f = 3 print("Unoptimized call of int", int('0' * f, base = 16)) d = { 'x' : "12", "base" : 8 } print("Dict star argument call of int", int(**d)) base = 16 try: value = unicode("20") except NameError: print("Python3: Has unicode by default.") value = "20" print("Unoptimized calls of int with unicode args", int(value, base), int(value)) base = 37 try: print("Int with large base", int(2, base), end = ' ') except Exception as e: print("Caused", repr(e)) else: print("Worked") try: print(chr()) except Exception as e: print("Disallowed without args", repr(e)) try: print(ord()) except Exception as e: print("Disallowed without args", repr(e)) try: print(ord(s = 1)) except Exception as e: print("Disallowed keyword args", repr(e)) try: print(ord(1, 2)) except Exception as e: print("Too many plain args", repr(e)) try: print(ord(1, s = 2)) except Exception as e: print("Too many args, some keywords", repr(e)) try: print(sum()) except Exception as e: print("Disallowed without args", repr(e)) x = range(17) print("Sum of range(17) is", sum(x)) print("Sum of range(17) starting with 5 is", sum(x, 5)) try: print(str('1', offer = 2)) except Exception as e: print("Too many args, some keywords", repr(e)) # TODO: This is calls, not really builtins. a = 2 print("Can optimize the star list argness away", int(*(a,)), end = ' ') print("Can optimize the empty star list arg away", int(*tuple()), end = ' ') print("Can optimize the empty star dict arg away", int(**dict())) print("Dict building with keyword arguments", dict(), dict(a = f)) print( "Dictionary entirely from constant args", displayDict( dict(q = "Guido", w = "van", e = "Rossum", r = "invented", t = "Python", y = "") ) ) a = 5 print("Instance check recognises", isinstance(a, int)) try: print("Instance check with too many arguments", isinstance(a, long, int)) except Exception as e: print("Too many args", repr(e)) try: print("Instance check with too many arguments", isinstance(a)) except Exception as e: print("Too few args", repr(e)) def usingIterToCheckIterable(a): try: iter(a) except TypeError: print("not iterable") else: print("ok") usingIterToCheckIterable(1) print("Nested constant, dict inside a list, referencing a built-in compile time constant", end = ' ') print([dict(type = int)]) print("nan and -nan sign checks:") from math import copysign print(copysign(1.0, float("nan"))) print(copysign(1.0, float("-nan"))) print("Using != to detect nan floats:") a = float("nan") if a != a: print("is nan") else: print("isn't nan") class CustomStr(str): pass class CustomBytes(bytes): pass class CustomByteArray(bytearray): pass values = [ b'100', b'', bytearray(b'100'), CustomStr("100"), CustomBytes(b'100'), CustomByteArray(b'100') ] for x in values: try: print("int", repr(x), int(x), int(x,2)) except (TypeError, ValueError) as e: print("caught", repr(e)) try: print("long", repr(x), long(x), long(x,2)) except (TypeError, ValueError) as e: print("caught", repr(e)) except NameError: print("Python3 has no long") z = range(5) try: next(z) except TypeError as e: print("caught", repr(e)) try: open() except TypeError as e: print("Open without arguments gives", repr(e)) print("Type of id values:", type(id(id))) Nuitka-0.5.28.2/tests/basics/Decorators.py0000644000372000001440000000312313112214770020545 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function def decorator1(f): print("Executing decorator 1") def deco_f(): return f() + 2 return deco_f def decorator2(f): print("Executing decorator 2") def deco_f(): return f() * 2 return deco_f # Result of function now depends on correct order of applying the decorators @decorator1 @decorator2 def function1(): return 3 print(function1()) def deco_returner1(): print("Executing decorator returner D1") return decorator1 def deco_returner2(): print("Executing decorator returner D2") return decorator2 @deco_returner1() @deco_returner2() def function2(): return 3 print(function2()) # Same as function2, but without decorator syntax. def function3(): return 3 function3 = deco_returner1()(deco_returner2()(function3)) print(function3()) Nuitka-0.5.28.2/tests/basics/ExceptionRaising.py0000644000372000001440000003432613134660221021724 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function import sys print("Raising an exception type in a function:") def raiseExceptionClass(): raise ValueError try: raiseExceptionClass() except Exception as e: print("Caught exception type", e, repr(e), type(e)) print("Inside handler, sys.exc_info is this", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raising an exception instance in a function:") def raiseExceptionInstance(): raise ValueError("hallo") try: raiseExceptionInstance() except Exception as f: print("Caught exception instance", f, repr(f), type(f)) print("Inside handler, sys.exc_info is this", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raising an exception, then catch it to re-raise it:") def raiseExceptionAndReraise(arg): try: return arg / arg except: raise try: raiseExceptionAndReraise(0) except: print("Catched reraised", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Access an undefined global variable in a function:") def raiseNonGlobalError(): return undefined_value # @UndefinedVariable try: raiseNonGlobalError() except: print("NameError caught", sys.exc_info()) print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raise a new style class as an exception, should be rejected:") def raiseIllegalError(): class X(object): pass raise X() try: raiseIllegalError() except TypeError as E: print("New style class exception correctly rejected:", E) except: print(sys.exc_info()) assert False, "Error, new style class exception was not rejected" print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Raise an old-style class, version dependent outcome:") class ClassicClassException: pass def raiseCustomError(): raise ClassicClassException() try: try: raiseCustomError() except ClassicClassException: print("Caught classic class exception") except: print("Default catch", sys.exc_info()) assert False, "Error, old style class exception was not caught" except TypeError as e: print("Python3 hates to even try and catch classic classes", e) else: print("Classic exception catching was considered fine.") print("After catching, sys.exc_info is this", sys.exc_info()) print('*' * 20) print("Check lazy exception creation:") def checkExceptionConversion(): try: raise Exception("some string") except Exception as err: print("Catched raised object", err, type(err)) try: raise Exception, "some string" except Exception as err: print("Catched raised type, value pair", err, type(err)) checkExceptionConversion() print('*' * 20) print("Check exc_info scope:") def checkExcInfoScope(): try: raise ValueError except: assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None if sys.version_info[0] < 3: print("Exc_info remains visible after exception handler for Python2") assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None else: print("Exc_info is clear after exception handler for Python3") assert sys.exc_info()[0] is None assert sys.exc_info()[1] is None assert sys.exc_info()[2] is None def subFunction(): print("Entering with exception info", sys.exc_info()) assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None try: print("Trying") except: pass print("After trying something and didn't have an exception, info is", sys.exc_info()) print("Call a function inside the exception handler and check there too.") try: raise KeyError except: assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None subFunction() print("Call it twice and see.") try: raise "me" except: assert sys.exc_info()[0] is not None assert sys.exc_info()[1] is not None assert sys.exc_info()[2] is not None subFunction() subFunction() if sys.version_info[0] < 3: sys.exc_clear() checkExcInfoScope() print('*' * 20) # Check that the sys.exc_info is cleared again, after being set inside the # function checkExcInfoScope, it should now be clear again. assert sys.exc_info()[0] is None, sys.exc_info()[0] assert sys.exc_info()[1] is None assert sys.exc_info()[2] is None print("Check catching subclasses") def checkDerivedCatch(): class A(BaseException): pass class B(A): def __init__(self): pass a = A() b = B() try: raise A, b except B, v: print("Caught B", v) except A, v: print("Didn't catch as B, but as A, Python3 does that", v) else: print("Not caught A class, not allowed to happen.") try: raise B, a except TypeError, e: print("TypeError with pair form for class not taking args:", e) checkDerivedCatch() print('*' * 20) def checkNonCatch1(): print("Testing if the else branch is executed in the optimizable case:") try: 0 except TypeError: print("Should not catch") else: print("Executed else branch correctly") checkNonCatch1() print('*' * 20) def checkNonCatch2(): try: print("Testing if the else branch is executed in the non-optimizable case:") except TypeError: print("Should not catch") else: print("Executed else branch correctly") checkNonCatch2() print('*' * 20) print("Checking raise that with exception arguments that raise error themselves.") def checkRaisingRaise(): def geterror(): return 1/0 try: geterror() except Exception as e: print("Had exception", e) try: raise TypeError, geterror() except Exception as e: print("Had exception", e) try: raise TypeError, 7, geterror() except Exception as e: print("Had exception", e) checkRaisingRaise() print('*' * 20) print("Checking a re-raise that isn't one:") def checkMisRaise(): raise try: checkMisRaise() except Exception as e: print("Without existing exception, re-raise gives:", e) print('*' * 20) print("Raising an exception in an exception handler gives:") def nestedExceptions(a, b): try: a / b except ZeroDivisionError: a / b try: nestedExceptions(1, 0) except Exception as e: print("Nested exception gives", e) print('*' * 20) print("Checking unpacking from an exception as a sequence:") def unpackingCatcher(): try: raise ValueError(1,2) except ValueError as (a,b): print("Unpacking caught exception and unpacked", a, b) unpackingCatcher() print("Afterwards, exception info is", sys.exc_info()) print('*' * 20) print("Testing exception that escapes __del__ and therefore cannot be raised") def devide(a,b): return a / b def unraisableExceptionInDel(): class C: def __del__(self): c = devide(1,0) print(c) def f(): C() f() unraisableExceptionInDel() print('*' * 20) print("Testing exception changes between generator switches:") def yieldExceptionInteraction(): def yield_raise(): print("Yield finds at generator entry", sys.exc_info()[0]) try: raise KeyError("caught") except KeyError: yield sys.exc_info()[0] yield sys.exc_info()[0] yield sys.exc_info()[0] g = yield_raise() print("Initial yield from catch in generator", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("Second yield from the catch reentered", next(g)) print("Checking from outside of generator", sys.exc_info()[0]) print("After leaving the catch generator yielded", next(g)) yieldExceptionInteraction() print('*' * 20) print("Testing exception change between generator switches while handling an own exception") def yieldExceptionInteraction2(): def yield_raise(): print("Yield finds at generator entry", sys.exc_info()[0]) try: raise ValueError("caught") except ValueError: yield sys.exc_info()[0] yield sys.exc_info()[0] yield sys.exc_info()[0] try: undefined_global # @UndefinedVariable except Exception: print("Checking from outside of generator with", sys.exc_info()[0]) g = yield_raise() v = next(g) print("Initial yield from catch in generator:", v) print("Checking from outside the generator:", sys.exc_info()[0]) print("Second yield from the catch reentered:", next(g)) print("Checking from outside the generation again:", sys.exc_info()[0]) print("After leaving the catch generator yielded:", next(g)) print("After exiting the trying branch:", sys.exc_info()[0]) yieldExceptionInteraction2() print("After function exit, no exception", sys.exc_info()) print('*' * 20) print("Check what happens if a function attempts to clear the exception in a handler") def clearingException(): def clearit(): try: if sys.version_info[0] < 3: sys.exc_clear() except KeyError: pass try: raise KeyError except: print("Before clearing, it's", sys.exc_info()) clearit() print("After clearing, it's", sys.exc_info()) clearingException() print('*' * 20) print("Check that multiple exceptions can be caught in a handler through a variable:") def multiCatchViaTupleVariable(): some_exceptions = (KeyError, ValueError) try: raise KeyError except some_exceptions: print("Yes, indeed.") multiCatchViaTupleVariable() def raiseValueWithValue(): try: raise ValueError(1,2,3), (ValueError(1,2,3)) except Exception as e: print("Gives", e) print("Check exception given when value is raised with value", raiseValueWithValue()) # Make sure the "repr" of exceptions is fine a = IOError print("IOError is represented correctly:", repr(a)) def raising(): raise ValueError def not_raising(): pass def raiseWithFinallyNotCorruptingLineNumber(): try: try: raising() finally: not_raising() except ValueError: print("Traceback is in tried block line", sys.exc_info()[2].tb_lineno) raiseWithFinallyNotCorruptingLineNumber() def wideCatchMustPublishException(): print("At entry, no exception", sys.exc_info()) try: undefined_global # @UndefinedVariable except: print("Inside handler:", sys.exc_info()) pass print("Outside handler:", sys.exc_info()) print("Check that a unqualified catch properly preserves exception") wideCatchMustPublishException() print("Check if a nested exception handler does overwrite re-raised") def checkReraiseAfterNestedTryExcept(): def reraise(): try: raise TypeError("outer") except Exception: try: raise KeyError("nested") except KeyError: print("Current exception inside nested handler", sys.exc_info()) pass print("Current exception after nested handler exited", sys.exc_info()) # Which one does this pick raise try: reraise() except Exception as e: print("Catched", repr(e)) checkReraiseAfterNestedTryExcept() def checkReraiseByFunction(): def reraise(): raise try: try: raise TypeError("outer") except Exception: reraise() except Exception as e: import traceback print("Exception traceback of re-raise:") print('-' * 40) traceback.print_exc() print('-' * 40) print("OK.") # TODO: Enable this, once the actual traceback of a function # re-raise isn't wrong (contains itself) anymore. if False: checkReraiseByFunction() def checkNoRaiseExceptionDictBuilding(arg): a = { () : arg } b = { None : arg } c = { Ellipsis : arg } d = { 1.0j : arg } e = { 1.0 : arg } f = { long(0) : arg } g = { 0 : arg } h = { type : arg } return a, b, c, d, e, f, g, h checkNoRaiseExceptionDictBuilding(1) def checkRaiseExceptionDictBuildingRange(arg): try: i = { range(10) : arg } except Exception as e: print("Raised", repr(e)) else: print("No exception, OK for Python2") return i print("Check if range as dict key raises:") checkRaiseExceptionDictBuildingRange(2) def checkRaiseExceptionDictBuildingTuple(arg): try: i = { (2, []) : arg } except Exception as e: print("Raised", repr(e)) else: return i print("Check if mutable tuple as dict key raises:") checkRaiseExceptionDictBuildingTuple(3) def checkRaiseExceptionDictBuildingList(arg): try: i = { [2, ()] : arg } except Exception as e: print("Raised", repr(e)) else: return i print("Check if list as dict key raises:") checkRaiseExceptionDictBuildingList(4) Nuitka-0.5.28.2/tests/basics/ExceptionRaising32.py0000644000372000001440000000167713112214770022074 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def raisy(): raise ValueError() from None try: print("Raising exception in a function 'from None':") raisy() except (ValueError,TypeError) as e: print("Caught as", repr(e)) Nuitka-0.5.28.2/tests/basics/PrintFuture.py0000644000372000001440000000154013112214770020730 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print("hallo welt", end = ',') print("this is the end") Nuitka-0.5.28.2/tests/basics/Referencing32.py0000644000372000001440000000602613122472300021035 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython ) checkDebugPython() def simpleFunction1(): def abc(*, exc=IOError): pass for _ in range(100): abc() def simpleFunction2(): def abc(*, exc=IOError): raise ValueError from None try: abc() except (ValueError, TypeError): pass def simpleFunction3(): try: class A(Exception): pass class B(Exception): pass try: raise A('foo') except A as e1: raise B(str(e1)) from e1 except Exception: pass def simpleFunction4(): a = 1 def nonlocal_writer(): nonlocal a for a in range(10): pass nonlocal_writer() assert a == 9, a def simpleFunction5(): x = 2 def local_func(a: int, b: x*x): pass local_func(x, x) def simpleFunction6(): # Make sure exception state is cleaned up as soon as the except # block is left. class MyException(Exception): def __init__(self, obj): self.obj = obj class MyObj: pass def inner_raising_func(): local_ref = obj raise MyException(obj) # "except" block raising another exception obj = MyObj() try: try: inner_raising_func() except: raise KeyError except KeyError as e: pass range_low = 0 range_high = 256 range_step = 13 def simpleFunction7(): # Make sure xranges work nicely return range(range_low,range_high,range_step) def simpleFunction8(): # Make sure xranges work nicely return range(range_low,range_high) def simpleFunction9(): # Make sure xranges work nicely return range(range_high) # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/ParameterErrors.py0000644000372000001440000001304613112214770021562 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # def functionNoParameters(): pass print("Call a function with no parameters with a plain argument:") try: functionNoParameters(1) except TypeError as e: print(repr(e)) print("Call a function with no parameters with a keyword argument:") try: functionNoParameters(z = 1) except TypeError as e: print(repr(e)) def functionOneParameter(a): print(a) print("Call a function with one parameter with two plain arguments:") try: functionOneParameter(1, 1) except TypeError as e: print(repr(e)) print("Call a function with one parameter too many, and duplicate arguments:") try: functionOneParameter(6, a = 4, *(1, 2, 3)) except TypeError as e: print(repr(e)) print("Call a function with two parameters with three plain arguments:") def functionTwoParameters(a, b): print(a, b) try: functionTwoParameters(1, 2, 3) except TypeError as e: print(repr(e)) print("Call a function with two parameters with one plain argument:") try: functionTwoParameters(1) except TypeError as e: print(repr(e)) print("Call a function with two parameters with three plain arguments:") try: functionTwoParameters(1, 2, 3) except TypeError as e: print(repr(e)) print("Call a function with two parameters with one keyword argument:") try: functionTwoParameters(a = 1) except TypeError as e: print(repr(e)) print("Call a function with two parameters with three keyword arguments:") try: functionTwoParameters(a = 1, b = 2, c = 3) except TypeError as e: print(repr(e)) class MethodContainer: def methodNoParameters(self): pass def methodOneParameter(self, a): print(a) def methodTwoParameters(self, a, b): print(a, b) obj = MethodContainer() print("Call a method with no parameters with a plain argument:") try: obj.methodNoParameters(1) except TypeError as e: print(repr(e)) print("Call a method with no parameters with a keyword argument:") try: obj.methodNoParameters(z = 1) except TypeError as e: print(repr(e)) print("Call a method with one parameter with two plain arguments:") try: obj.methodOneParameter(1, 1) except TypeError as e: print(repr(e)) print("Call a method with two parameters with three plain arguments:") try: obj.methodTwoParameters(1, 2, 3) except TypeError as e: print(repr(e)) print("Call a method with two parameters with one plain argument:") try: obj.methodTwoParameters(1) except TypeError as e: print(repr(e)) print("Call a method with two parameters with one keyword argument:") try: obj.methodTwoParameters(a = 1) except TypeError as e: print(repr(e)) print("Call a method with two parameters with three keyword arguments:") try: obj.methodTwoParameters(a = 1, b = 2, c = 3) except TypeError as e: print(repr(e)) def functionPosBothStarArgs(a, b, c, *l, **d): print(a, b, c, l, d) l = [2] d = { "other" : 7 } print("Call a function with both star arguments and too little arguments:") try: functionPosBothStarArgs(1, *l, **d) except TypeError as e: print(repr(e)) print("Call a function with defaults with too little arguments:") def functionWithDefaults(a, b, c, d = 3): print(a, b, c, d) try: functionWithDefaults(1) except TypeError as e: print(repr(e)) print("Call a function with defaults with too many arguments:") try: functionWithDefaults(1) except TypeError as e: print(repr(e)) print("Complex call with both invalid star list and star arguments:") try: a = 1 b = 2.0 functionWithDefaults(1,c = 3,*a,**b) except TypeError as e: print(repr(e)) try: a = 1 b = 2.0 functionWithDefaults(1,*a,**b) except TypeError as e: print(repr(e)) try: a = 1 b = 2.0 functionWithDefaults(c = 1, *a,**b) except TypeError as e: print(repr(e)) try: a = 1 b = 2.0 functionWithDefaults(*a,**b) except TypeError as e: print(repr(e)) print("Complex call with both invalid star list argument:") try: a = 1 functionWithDefaults(*a) except TypeError as e: print(repr(e)) try: a = 1 MethodContainer(*a) except TypeError as e: print(repr(e)) try: a = 1 MethodContainer()(*a) except TypeError as e: print(repr(e)) try: a = 1 MethodContainer.methodTwoParameters(*a) except TypeError as e: print(repr(e)) try: a = 1 None(*a) except TypeError as e: print(repr(e)) try: a = 1 None(**a) except TypeError as e: print(repr(e)) print("Call object with name as both keyword and in star dict argument:") try: a = {'a' : 3} None(a = 2, **a) except TypeError as e: print(repr(e)) print("Call function with only defaulted value given as keyword argument:") def functionwithTwoArgsOneDefaulted(a, b = 5): pass try: functionwithTwoArgsOneDefaulted(b = 12) except TypeError as e: print(repr(e)) Nuitka-0.5.28.2/tests/basics/Referencing35.py0000644000372000001440000000606513122472300021043 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import sys, os, types # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( executeReferenceChecked, checkDebugPython, run_async ) checkDebugPython() def raisy(): raise TypeError def simpleFunction1(): async def foo(): return run_async(foo()) #################################### class AsyncIteratorWrapper: def __init__(self, obj): self._it = iter(obj) async def __aiter__(self): return self async def __anext__(self): try: value = next(self._it) except StopIteration: raise StopAsyncIteration return value def simpleFunction3(): async def f(): result = [] async for letter in AsyncIteratorWrapper("abcdefg"): result.append(letter) return result run_async(f()) #################################### def simpleFunction2(): async def foo(): return 7 run_async(foo()) def simpleFunction4(): async def foo(): raise StopIteration try: run_async(foo()) except RuntimeError: pass #################################### class ClassWithAsyncMethod: async def async_method(self): return self def simpleFunction5(): run_async(ClassWithAsyncMethod().async_method()) #################################### class BadAsyncIter: def __init__(self): self.weight = 1 async def __aiter__(self): return self def __anext__(self): return () def simpleFunction7(): async def foo(): async for i in BadAsyncIter(): print('never going to happen') try: run_async(foo()) except TypeError: pass def simpleFunction8(): async def bar(): return ("some", "thing") @types.coroutine def foo(): yield from bar() run_async(foo()) # These need stderr to be wrapped. tests_stderr = () # Disabled tests tests_skipped = {} result = executeReferenceChecked( prefix = "simpleFunction", names = globals(), tests_skipped = tests_skipped, tests_stderr = tests_stderr ) sys.exit(0 if result else 1) Nuitka-0.5.28.2/tests/basics/Classes32.py0000644000372000001440000000565413112214770020215 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python tests originally created or extracted from other peoples work. The # parts were too small to be protected. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print("Call order of Python3 metaclasses:") def a(): x = 1 class A: print("Class body a.A is evaluating closure x", x) print("Called", a) return A def b(): class B: pass print("Called", b) return B from collections import OrderedDict def displayable(dictionary): return sorted(dictionary.items()) def m(): class M(type): def __new__(cls, class_name, bases, attrs, **over): print("Metaclass M.__new__ cls", cls, "name", class_name, "bases", bases, "dict", displayable(attrs), "extra class defs", displayable(over)) return type.__new__(cls, class_name, bases, attrs) def __init__(self, name, bases, attrs, **over): print("Metaclass M.__init__", name, bases, displayable(attrs), displayable(over)) super().__init__(name, bases, attrs) def __prepare__(metacls, bases, **over): # @NoSelf print("Metaclass M.__prepare__", metacls, bases, displayable(over)) return OrderedDict() print("Called", m) return M def d(): print("Called", d) return 1 def e(): print("Called", e) return 2 class C1(a(), b(), other = d(), metaclass = m(), yet_other = e()): import sys # TODO: Enable this. # print("C1 locals type is", type(sys._getframe().f_locals)) print("OK, class created", C1) print("Attribute C1.__dict__ has type", type(C1.__dict__)) print("Function local classes can be made global and get proper __qualname__:") def someFunctionWithLocalClassesMadeGlobal(): # Affects __qualname__ only in Python3.4 or higher, not in Python3.2 global C class C: pass class D: pass try: print("Nested class qualname is", D.__qualname__) except AttributeError: # Python3.2 pass try: print("Local class made global qualname is", C.__qualname__) except AttributeError: pass someFunctionWithLocalClassesMadeGlobal() print("Function in a class with private name") class someClassWithPrivateArgumentNames: def f(self, *, __kw:1): pass print(someClassWithPrivateArgumentNames.f.__annotations__) print("OK.") Nuitka-0.5.28.2/tests/optimizations/0000755000372000001440000000000013207540420017533 5ustar hayenusers00000000000000Nuitka-0.5.28.2/tests/optimizations/run_all.py0000755000372000001440000001602613207537242021561 0ustar hayenusers00000000000000#!/usr/bin/env python # Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import os import sys # The test runner needs "lxml" itself. try: import lxml.etree except ImportError: print("Warning, no 'lxml' module installed, cannot do XML based tests.") sys.exit(0) # Find nuitka package relative to us. sys.path.insert( 0, os.path.normpath( os.path.join( os.path.dirname(os.path.abspath(__file__)), "..", ".." ) ) ) from nuitka.tools.testing.Common import ( # isort:skip check_output, convertUsing2to3, createSearchMode, hasModule, my_print, setup ) python_version = setup() # The Python version used by Nuitka needs "lxml" too. if not hasModule("lxml.etree"): print("Warning, no 'lxml' module installed, cannot run XML based tests.") sys.exit(0) search_mode = createSearchMode() def getKind(node): result = node.attrib[ "kind" ] result = result.replace("Statements", "") result = result.replace("Statement", "") result = result.replace("Expression", "") return result def getRole(node, role): for child in node: if child.tag == "role" and child.attrib["name"] == role: return child else: return None def getSourceRef(node): global filename return "%s:%s" % ( filename, node.attrib["line"] ) def isConstantExpression(expression): kind = getKind(expression) return kind.startswith("Constant") or \ kind in ("ImportModuleHard", "ImportName", "ModuleFileAttributeRef", "ModuleLoaderRef") def checkSequence(statements): for statement in statements: kind = getKind(statement) # Printing is fine. if kind == "PrintValue": print_arg, = getRole(statement, "value") if not isConstantExpression(print_arg): sys.exit( "%s: Error, print of non-constant '%s'." % ( getSourceRef(statement), getKind(print_arg) ) ) continue if kind == "PrintNewline": continue # Printing in Python3 is a function call whose return value is ignored. if kind == "Only": only_expression = getRole(statement, "expression")[0] if getKind(only_expression) == "CallNoKeywords": called_expression = getRole(only_expression, "called")[0] if getKind(called_expression) == "BuiltinRef": if called_expression.attrib["builtin_name"] == "print": continue if kind == "FrameModule": checkSequence(getRole(statement, "statements")) continue if kind == "AssignmentVariable": variable_name = statement.attrib["variable_name"] # Ignore "__spec__" assignment for Python3.4, it is not going # to be static. if variable_name == "__spec__": continue assign_source, = getRole(statement, "source") if getKind(assign_source) == "FunctionCreation": continue elif not isConstantExpression(assign_source): sys.exit("Error, assignment from non-constant '%s'." % getKind(assign_source)) continue print(lxml.etree.tostring(statement, pretty_print = True)) sys.exit("Error, non-print statement of unknown kind '%s'." % kind) for filename in sorted(os.listdir('.')): if not filename.endswith(".py") or filename.startswith("run_"): continue active = search_mode.consider( dirname = None, filename = filename ) extra_flags = ["expect_success"] if active: # Apply 2to3 conversion if necessary. if python_version.startswith('3'): filename, changed = convertUsing2to3(filename) else: changed = False my_print("Consider", filename, end = ' ') command = [ os.environ["PYTHON"], os.path.abspath(os.path.join("..", "..", "bin", "nuitka")), "--dump-xml", "--module", filename ] if search_mode.isCoverage(): # To avoid re-execution, which is not acceptable to coverage. if "PYTHONHASHSEED" not in os.environ: os.environ["PYTHONHASHSEED"] = '0' # Coverage modules hates Nuitka to re-execute, and so we must avoid # that. python_path = check_output( [ os.environ["PYTHON"], "-c" "import sys, os; print(os.pathsep.join(sys.path))" ] ) if sys.version_info >= (3,): python_path = python_path.decode("utf8") os.environ["PYTHONPATH"] = python_path.strip() command.insert(2, "--must-not-re-execute") command = command[0:1] + [ "-S", "-m", "coverage", "run", "--rcfile", os.devnull, "-a", ] + command[1:] result = check_output( command ) # Parse the result into XML and check it. root = lxml.etree.fromstring(result) module_body = root[0] module_statements_sequence = module_body[0] assert len(module_statements_sequence) == 1 module_statements = next(iter(module_statements_sequence)) try: checkSequence(module_statements) for function in root.xpath('role[@name="functions"]/node'): function_body, = function.xpath('role[@name="body"]') function_statements_sequence = function_body[0] assert len(function_statements_sequence) == 1 function_statements = next(iter(function_statements_sequence)) checkSequence(function_statements) if changed: os.unlink(filename) except SystemExit: my_print("FAIL.") raise my_print("OK.") else: my_print("Skipping", filename) search_mode.finish() Nuitka-0.5.28.2/tests/optimizations/Attributes.py0000644000372000001440000000204013112214770022230 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print((1).imag) print(int.__name__) print((1).__class__) print(getattr(1, "real")) print(getattr(1, "real", None)) print(hasattr(1, "real")) print((0.0).real) Nuitka-0.5.28.2/tests/optimizations/Conditions.py0000644000372000001440000000163113112214770022220 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(1 if [1,2] else 2) Nuitka-0.5.28.2/tests/optimizations/Calls.py0000644000372000001440000000173113112214770021146 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # TODO: This is only a placeholder, currently no calls will be optimized print(range(4)) Nuitka-0.5.28.2/tests/optimizations/Len.py0000644000372000001440000000233313112214770020625 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # from __future__ import print_function print(len(range(266))) print(len(range(266,9999))) print(len(range(266,9999,3))) print(len(range(266,9999,-3))) print(len(range(22266,9999,-3))) print(len(range(22266,9998,-3))) print(len(range(22266,9997,-3))) print(len(range(22266,9996,-3))) print(len(range(0,3,3))) print(len([3,3.3])) print(len((3,3.3))) Nuitka-0.5.28.2/tests/optimizations/DecodingOperations.py0000644000372000001440000000161513122472300023665 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(chr(7)) Nuitka-0.5.28.2/tests/optimizations/Subscripts.py0000644000372000001440000000175013112214770022252 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print( (1,2,3)[1] ) print( (1,2,3)[1:] ) print( (1,2,3)[:2] ) print( (1,2,3)[:] ) print( (1,2,3)[1:2] ) Nuitka-0.5.28.2/tests/optimizations/Iterations.py0000644000372000001440000000162513112214770022233 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # z = next(iter((3,4))) Nuitka-0.5.28.2/tests/optimizations/Operations.py0000644000372000001440000000270013112214770022230 0ustar hayenusers00000000000000# Copyright 2017, Kay Hayen, mailto:kay.hayen@gmail.com # # Python test originally created or extracted from other peoples work. The # parts from me are licensed as below. It is at least Free Software where # it's copied from other people. In these cases, that will normally be # indicated. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # print(not bool) print(not {}) print(not 7) print(bool or len) print(False or dict) print(type(Ellipsis)) print("a" in "abba") print("a" not in "abba") # TODO: Add support for functions # def testInplaceOperations(): # x = 2 # x += 1 # x *= 2 # x **= 2 # x -= 8 # x //= 5 # x %= 3 # x &= 2 # x |= 5 # x ^= 1 # x /= 2 # # print(x) print(len("a"*10000)) print(len(10000*"a")) print(len((1,) *20000)) print(len(20000*(1,))) print(len([1]*30000)) print(len(30000*[1])) print(len(unicode("a")*40000)) print(len(40000*unicode("a"))) Nuitka-0.5.28.2/LICENSE.txt0000644000372000037200000002613612711355343015270 0ustar hayenhayen00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Nuitka-0.5.28.2/README.rst0000644000372000001440000007435713207537706015204 0ustar hayenusers00000000000000Nuitka User Manual ~~~~~~~~~~~~~~~~~~ .. image:: doc/images/Nuitka-Logo-Symbol.png .. contents:: .. raw:: pdf PageBreak oneColumn SetPageCounter 1 Overview ======== This document is the recommended first read if you are interested in using Nuitka, understand its use cases, check what you can expect, license, requirements, credits, etc. Nuitka is **the** Python compiler. It is a seamless replacement or extension to the Python interpreter and compiles **every** construct that CPython 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, and 3.6 have. It then executes uncompiled code, and compiled code together in an extremely compatible manner. You can use all Python library modules or and all extension modules freely. It translates the Python into a C level program that then uses "libpython" to execute in the same way as CPython does. All optimization is aimed at avoiding overhead, where it's unnecessary. None is aimed at removing compatibility, although slight improvements will occassionally be done, where not every bug of standard Python is emulated, e.g. more complete error messages are given, but there is a full compatibility mode to disable even that. Usage ===== Requirements ------------ - C Compiler: You need a compiler with support for C11 or alternatively for C++03 [#]_ Currently this means, you need to use either of these compilers: * `The ``gcc`` compiler of at least version 5.1, or the ``g++`` compiler of at least version 4.4 as an alternative. * The ``clang`` compiler on MacOS X or FreeBSD, based on LLVM version 3.2 or higher. * The MinGW64 [#]_ C11 compiler on Windows, ideally the one based on gcc 5.1 or higher. Or the C++ compiler of at least version 4.4 as an alternative. * Visual Studio 2017 or higher on Windows [#]_, older versions may work, but are not officially supported. Configure to use English language pack for best results (Nuitka filters away garbage outputs, but only for that language). - Python: Version 2.6, 2.7 or 3.2, 3.3, 3.4, 3.5, 3.6 (yes, but read below) .. admonition:: Python3, but 3.2, 3.3, and 3.4 need other Python versions as a *compile time* dependency Nuitka itself is fully compatible with all mentions version, Scons is not. For these versions, you *need* a Python2 or Python3.5 installed as well, but only during the compile time only. That is for use with Scons (which orchestrates the C compilation), which does not support the same Python versions as Nuitka. .. admonition:: Moving to other machines The created binaries can be made executable independent of the Python installation, with ``--standalone`` option. .. admonition:: Binary filename suffix ".exe" even on Linux The created binaries have an ".exe" suffix, that you are free to remove that and yes, they are still Linux binaries. The suffix is just to be sure that the original script name and the binary name do not collide. .. admonition:: It has to be CPython or AnaConda You need the standard Python implementation, called "CPython", to execute Nuitka, because it is closely tied to using it. On Windows, the so called "WinPython" and "AnaConda" distributions work, but will cause issues for acceleration mode. Standalone mode and creating extension modules or packages will work. For acceleration mode, you need to copy the "PythonXX.DLL" alongside of it. - Operating System: Linux, FreeBSD, NetBSD, MacOS X, and Windows (32/64 bits). Others may work as well. The portability is expected to be generally good, but the e.g. Scons usage may have to be adapted. - Architectures: x86, x86_64 (amd64), and arm, likely more Other architectures are expected to also work, out of the box, as Nuitka is generally not using any hardware specifics. These are just the ones tested and known to be good. Feedback is welcome. Generally the architectures that Debian supports can be considered good and tested too. .. [#] Support for this C11 is given with gcc 5 or higher or clang. The MSVC compiler doesn't do it yet. But as a workaround, the C++03 language standard is very overlapping with C11 and is used instead where the C compiler is too old. Nuitka used to require a C++ compiler in the past, but it changed. .. [#] Download MinGW64 from here http://mingw-w64.org/ and choose 64 or 32 bits matching your Python. Use both MinGW64 and 64 bits Python if you have the choice of which Python to use. Install it to "C:\MinGW64" or "\MinGW64" (same disk root) to find it automatically. .. [#] Download for free from http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx (the Express editions work just fine). The latest version is recommend. There is no need to use older versions, they might in fact not work. Command Line ------------ No environment variable changes are needed, most noteworthy, you do not have to mess with ``PYTHONPATH`` at all for Nuitka. You just execute the ``nuitka`` and ``nuitka-run`` scripts directly without any changes to the environment. You may want to add the ``bin`` directory to your ``PATH`` for your convenience, but that step is optional. Nuitka has a ``--help`` option to output what it can do: .. code-block:: bash nuitka --help The ``nuitka-run`` command is the same as ``nuitka``, but with different default. It tries to compile *and* directly execute a Python script: .. code-block:: bash nuitka-run --help These option that is different is ``--run``, and passing on arguments after the first non-option to the created binary, so it is somewhat more similar to what plain ``python`` will do. License ------- Nuitka is licensed under the Apache License, Version 2.0; you may not use it except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Use Cases ========= Use Case 1 - Program compilation with all modules embedded ---------------------------------------------------------- If you want to compile a whole program recursively, and not only the single file that is the main program, do it like this: .. code-block:: bash nuitka --recurse-all program.py .. note:: There are more fine grained controls than ``--recurse-all`` available. Consider the output of ``nuitka --help``. In case you have a plugin directory, i.e. one which cannot be found by recursing after normal import statements via the ``PYTHONPATH`` (which would be recommended way), you can always require that a given directory shall also be included in the executable: .. code-block:: bash nuitka --recurse-all --recurse-directory=plugin_dir program.py .. note:: If you don't do any dynamic imports, simply setting your ``PYTHONPATH`` at compilation time will be sufficient for all your needs normally. Use ``--recurse-directory`` only if you make ``__import__()`` calls that Nuitka cannot predict, because they e.g. depend on command line parameters. Nuitka also warns about these, and point to the option. .. note:: The resulting binary still depends on CPython and used C extension modules being installed. If you want to be able to copy it to another machine, use ``--standalone`` and copy the created ``program.dist`` directory and execute the ``program.exe`` put inside. .. note:: The resulting filename will be ``program.exe`` on all platforms, that doesn't mean it doesn't run on non-Windows! But if you compile ``program`` we wouldn't want to overwrite it, or be unsure which one is the compiled form, and which one is not. Use Case 2 - Extension Module compilation ----------------------------------------- If you want to compile a single extension module, all you have to do is this: .. code-block:: bash nuitka --module some_module.py The resulting file "some_module.so" can then be used instead of "some_module.py". It's left as an exercise to the reader, what happens if both are present. .. note:: The option ``--recurse-all`` and other variants work as well. Use Case 3 - Package compilation -------------------------------- If you need to compile a whole package and embedded all modules, that is also feasible, use Nuitka like this: .. code-block:: bash nuitka --module some_package --recurse-directory=some_package .. note:: The recursion into the package directory needs to be provided manually, otherwise the package is empty. Data files located inside the package will not be embedded yet. Where to go next ================ Remember, this project is not completed yet. Although the CPython test suite works near perfect, there is still more work needed, esp. to make it do more optimization. Try it out. Subscribe to its mailing lists ------------------------------ Please visit the `mailing list page `__ in order to subscribe the relatively low volume mailing list. All Nuitka issues can be discussed there. Also this is the place to stay informed of what's coming. Report issues or bugs --------------------- Should you encounter any issues, bugs, or ideas, please visit the `Nuitka bug tracker `__ and report them. Best practices for reporting bugs: - Please aways include the following information in your report, for the underlying Python version. You can easily copy&paste this into your report. .. code-block:: sh nuitka --version - Try to make your example minimal. That is, try to remove code that does not contribute to the issue as much as possible. Ideally come up with a small reproducing program that illustrates the issue, using ``print`` with different results when that programs runs compiled or native. - If the problem occurs spuriously (i.e. not each time), try to set the environment variable ``PYTHONHASHSEED`` to ``0``, disabling hash randomization. If that makes the problem go away, try increasing in steps of 1 to a hash seed value that makes it happen every time. - Do not include the created code in your report. Given proper input, it's redundant, and it's not likely that I will look at it without the ability to change the Python or Nuitka source and re-run it. Follow me on Twitter -------------------- Nuitka announcements and interesting stuff is pointed to on the Twitter account, but obviously with no details. `@KayHayen `_. I will not answer Nuitka questions via Twitter though. Word of Warning --------------- Consider using this software with caution. Even though many tests are applied before releases, things are potentially breaking. Your feedback and patches to Nuitka are very welcome. Especially report it please, if you find that anything doesn't work, because the project is now at the stage that this should not happen and most definitely will mean you encountered an unknown bug. Join Nuitka =========== You are more than welcome to join Nuitka development and help to complete the project in all minor and major ways. The development of Nuitka occurs in git. We currently have these 3 branches: - `master `__: This branch contains the stable release to which only hotfixes for bugs will be done. It is supposed to work at all times and is supported. - `develop `__: This branch contains the ongoing development. It may at times contain little regressions, but also new features. On this branch the integration work is done, whereas new features might be developed on feature branches. - `factory `__: This branch contains unfinished and incomplete work. It is very frequently subject ``git rebase`` and the public staging ground, where my work for develop branch lives first. It is intended for testing only and recommended to base any of your own development on. When updating it, you very often will get merge conflicts. Simply resolve those by doing ``git reset --hard origin/factory`` and switch to the latest version. .. note:: I accept patch files, git formatted patch queues (use ``git format-patch origin`` command), or if you prefer git pull on the social code platforms. I will do the integration work. If you base your work on "master" or "develop" at any given time, I will do any re-basing required and keep your authorship intact. .. note:: The `Developer Manual `__ explains the coding rules, branching model used, with feature branches and hotfix releases, the Nuitka design and much more. Consider reading it to become a contributor. This document is intended for Nuitka users. Donations ========= Should you feel that you cannot help Nuitka directly, but still want to support, please consider `making a donation `__ and help this way. Unsupported functionality ========================= The ``co_code`` attribute of code objects ----------------------------------------- The code objects are empty for for native compiled functions. There is no bytecode with Nuitka's compiled function objects, so there is no way to provide it. Optimization ============ Constant Folding ---------------- The most important form of optimization is the constant folding. This is when an operation can be fully predicted at compile time. Currently Nuitka does these for some built-ins (but not all yet, somebody to look at this more closely will be very welcome!), and it does it e.g. for binary/unary operations and comparisons. Constants currently recognized: .. code-block:: python 5 + 6 # binary operations not 7 # unary operations 5 < 6 # comparisons range(3) # built-ins Literals are the one obvious source of constants, but also most likely other optimization steps like constant propagation or function inlining will be. So this one should not be underestimated and a very important step of successful optimizations. Every option to produce a constant may impact the generated code quality a lot. .. admonition:: Status The folding of constants is considered implemented, but it might be incomplete in that not all possible cases are caught. Please report it as a bug when you find an operation in Nuitka that has only constants as input and is not folded. Constant Propagation -------------------- At the core of optimizations there is an attempt to determine values of variables at run time and predictions of assignments. It determines if their inputs are constants or of similar values. An expression, e.g. a module variable access, an expensive operation, may be constant across the module of the function scope and then there needs to be none, or no repeated module variable look-up. Consider e.g. the module attribute ``__name__`` which likely is only ever read, so its value could be predicted to a constant string known at compile time. This can then be used as input to the constant folding. .. code-block:: python if __name__ == "__main__": # Your test code might be here use_something_not_use_by_program() .. admonition:: Status From modules attributes, only ``__name__`` are currently actually optimized. Also possible would be at least ``__doc__``. In the future, this may improve as SSA is expanded to module variables. Built-in Name Lookups --------------------- Also built-in exception name references are optimized if they are used as module level read only variables: .. code-block:: python try: something() except ValueError: # The ValueError is a slow global name lookup normally. pass .. admonition:: Status This works for all built-in names. When an assignment is done to such a name, or it's even local, then of course it is not done. Built-in Call Prediction ------------------------ For built-in calls like ``type``, ``len``, or ``range`` it is often possible to predict the result at compile time, esp. for constant inputs the resulting value often can be precomputed by Nuitka. It can simply determine the result or the raised exception and replace the built-in call with that value, allowing for more constant folding or code path reduction. .. code-block:: python type("string") # predictable result, builtin type str. len([1, 2]) # predictable result range(3, 9, 2) # predictable result range(3, 9, 0) # predictable exception, range raises due to 0. .. admonition:: Status The built-in call prediction is considered implemented. We can simply during compile time emulate the call and use its result or raised exception. But we may not cover all the built-ins there are yet. Sometimes the result of a built-in should not be predicted when the result is big. A ``range()`` call e.g. may give too big values to include the result in the binary. Then it is not done. .. code-block:: python range( 100000 ) # We do not want this one to be expanded .. admonition:: Status This is considered mostly implemented. Please file bugs for built-ins that are pre-computed, but should not be computed by Nuitka at compile time with specific values. Conditional Statement Prediction -------------------------------- For conditional statements, some branches may not ever be taken, because of the conditions being possible to predict. In these cases, the branch not taken and the condition check is removed. This can typically predict code like this: .. code-block:: python if __name__ == "__main__": # Your test code might be here use_something_not_use_by_program() or .. code-block:: python if False: # Your deactivated code might be here It will also benefit from constant propagations, or enable them because once some branches have been removed, other things may become more predictable, so this can trigger other optimization to become possible. Every branch removed makes optimization more likely. With some code branches removed, access patterns may be more friendly. Imagine e.g. that a function is only called in a removed branch. It may be possible to remove it entirely, and that may have other consequences too. .. admonition:: Status This is considered implemented, but for the maximum benefit, more constants need to be determined at compile time. Exception Propagation --------------------- For exceptions that are determined at compile time, there is an expression that will simply do raise the exception. These can be propagated upwards, collecting potentially "side effects", i.e. parts of expressions that were executed before it occurred, and still have to be executed. Consider the following code: .. code-block:: python print side_effect_having() + (1 / 0) print something_else() The ``(1 / 0)`` can be predicted to raise a ``ZeroDivisionError`` exception, which will be propagated through the ``+`` operation. That part is just Constant Propagation as normal. The call `side_effect_having()`` will have to be retained though, but the ``print`` statement does not and can be turned into an explicit raise. The statement sequence can then be aborted and as such the ``something_else`` call needs no code generation or consideration anymore. To that end, Nuitka works with a special node that raises an exception and is wrapped with a so called "side_effects" expression, but yet can be used in code as an expression having a value. .. admonition:: Status The propagation of exceptions is mostly implemented, but needs handling in every kind of operations, and not all of them might do it already. As work progresses or examples arise, the coverage will be extended. Feel free to generate bug reports with non-working examples. Exception Scope Reduction ------------------------- Consider the following code: .. code-block:: python try: b = 8 print range(3, b, 0) print "Will not be executed" except ValueError, e: print e The ``try`` block is bigger than it needs to be. The statement ``b = 8`` cannot cause a ``ValueError`` to be raised. As such it can be moved to outside the try without any risk. .. code-block:: python b = 8 try: print range(3, b, 0) print "Will not be executed" except ValueError as e: print e .. admonition:: Status This is considered done. For every kind of operation, we trace if it may raise an exception. We do however *not* track properly yes, what can do a ``ValueError`` and what cannot. Exception Block Inlining ------------------------ With the exception propagation it is then becomes possible to transform this code: .. code-block:: python try: b = 8 print range(3, b, 0) print "Will not be executed" except ValueError, e: print e .. code-block:: python try: raise ValueError, "range() step argument must not be zero" except ValueError, e: print e Which then can be reduced by avoiding the raise and catch of the exception, making it: .. code-block:: python e = ValueError( "range() step argument must not be zero" ) print e .. admonition:: Status This is not implemented yet. Empty Branch Removal -------------------- For loops and conditional statements that contain only code without effect, it should be possible to remove the whole construct: .. code-block:: python for i in range(1000): pass The loop could be removed, at maximum it should be considered an assignment of variable ``i`` to ``999`` and no more. .. admonition:: Status This is not implemented yet, as it requires us to track iterators, and their side effects, as well as loop values, and exit conditions. Too much yet, but we will get there. Another example: .. code-block:: python if side_effect_free: pass The condition check should be removed in this case, as its evaluation is not needed. It may be difficult to predict that ``side_effect_free`` has no side effects, but many times this might be possible. .. admonition:: Status This is considered implemented. The conditional statement nature is removed if both branches are empty, only the condition is evaluated, and checked for truth (in cases that could raise an exception). Unpacking Prediction -------------------- When the length of the right hand side of an assignment to a sequence can be predicted, the unpacking can be replaced with multiple assignments. .. code-block:: python a, b, c = 1, side_effect_free(), 3 .. code-block:: python a = 1 b = side_effect_free() c = 3 This is of course only really safe if the left hand side cannot raise an exception while building the assignment targets. We do this now, but only for constants, because we currently have no ability to predict if an expression can raise an exception or not. .. admonition:: Status Not implemented yet. Will need us to see through the unpacking of what is an iteration over a tuple, we created ourselves. We are not there yet, but we will get there. Built-in Type Inference ----------------------- When a construct like ``in xrange()`` or ``in range()`` is used, it is possible to know what the iteration does and represent that, so that iterator users can use that instead. I consider that: .. code-block:: python for i in xrange(1000): something(i) could translate ``xrange(1000)`` into an object of a special class that does the integer looping more efficiently. In case ``i`` is only assigned from there, this could be a nice case for a dedicated class. .. admonition:: Status Future work, not even started. Quicker Function Calls ---------------------- Functions are structured so that their parameter parsing and ``tp_call`` interface is separate from the actual function code. This way the call can be optimized away. One problem is that the evaluation order can differ. .. code-block:: python def f(a, b, c): return a, b, c f(c = get1(), b = get2(), a = get3()) This will have to evaluate first ``get1()``, then ``get2()`` and only then ``get3()`` and then make the function call with these values. Therefore it will be necessary to have a staging of the parameters before making the actual call, to avoid an re-ordering of the calls to ``get1()``, ``get2()``, and ``get3()``. .. admonition:: Status Not even started. A re-formulation that avoids the dictionary to call the function, and instead uses temporary variables appears to be relatively straight forward once we do that kind of parameter analysis. Lowering of iterated Container Types ------------------------------------ In some cases, accesses to ``list`` constants can become ``tuple`` constants instead. Consider that: .. code-block:: python for x in [a, b, c]: something(x) Can be optimized into this: .. code-block:: python for x in (a, b, c): something(x) This allows for simpler, faster code to be generated, and less checks needed, because e.g. the ``tuple`` is clearly immutable, whereas the ``list`` needs a check to assert that. This is also possible for sets. .. admonition:: Status Implemented, even works for non-constants. Needs other optimization to become generally useful, and will itself help other optimization to become possible. This allows us to e.g. only treat iteration over tuples, and not care about sets. In theory something similar is also possible for ``dict``. For the later it will be non-trivial though to maintain the order of execution without temporary values introduced. The same thing is done for pure constants of these types, they change to ``tuple`` values when iterated. Credits ======= Contributors to Nuitka ---------------------- Thanks go to these individuals for their much valued contributions to Nuitka. Contributors have the license to use Nuitka for their own code even if Closed Source. The order is sorted by time. - Li Xuan Ji: Contributed patches for general portability issue and enhancements to the environment variable settings. - Nicolas Dumazet: Found and fixed reference counting issues, ``import`` packages work, improved some of the English and generally made good code contributions all over the place, solved code generation TODOs, did tree building cleanups, core stuff. - Khalid Abu Bakr: Submitted patches for his work to support MinGW and Windows, debugged the issues, and helped me to get cross compile with MinGW from Linux to Windows. This was quite a difficult stuff. - Liu Zhenhai: Submitted patches for Windows support, making the inline Scons copy actually work on Windows as well. Also reported import related bugs, and generally helped me make the Windows port more usable through his testing and information. - Christopher Tott: Submitted patches for Windows, and general as well as structural cleanups. - Pete Hunt: Submitted patches for MacOS X support. - "ownssh": Submitted patches for built-ins module guarding, and made massive efforts to make high quality bug reports. Also the initial "standalone" mode implementation was created by him. - Juan Carlos Paco: Submitted cleanup patches, creator of the `Nuitka GUI `__, creator of the `Ninja IDE plugin `__ for Nuitka. - "dr. Equivalent": Submitted the Nuitka Logo. - Johan Holmberg: Submitted patch for Python3 support on MacOS X. - Umbra: Submitted patches to make the Windows port more usable, adding user provided application icons, as well as MSVC support for large constants and console applications. - David Cortesi: Submitted patches and test cases to make MacOS port more usable, specifically for the Python3 standalone support of Qt. Projects used by Nuitka ----------------------- * The `CPython project `__ Thanks for giving us CPython, which is the base of Nuitka. We are nothing without it. * The `GCC project `__ Thanks for not only the best compiler suite, but also thanks for supporting C++11 which helped to get Nuitka off the ground. Your compiler was the first usable for Nuitka and with little effort. * The `Scons project `__ Thanks for tackling the difficult points and providing a Python environment to make the build results. This is such a perfect fit to Nuitka and a dependency that will likely remain. * The `valgrind project `__ Luckily we can use Valgrind to determine if something is an actual improvement without the noise. And it's also helpful to determine what's actually happening when comparing. * The `NeuroDebian project `__ Thanks for hosting the build infrastructure that the Debian and sponsor Yaroslav Halchenko uses to provide packages for all Ubuntu versions. * The `openSUSE Buildservice `__ Thanks for hosting this excellent service that allows us to provide RPMs for a large variety of platforms and make them available immediately nearly at release time. * The `MinGW64 project `__ Thanks for porting the gcc to Windows. This allowed portability of Nuitka with relatively little effort. * The `Buildbot project `__ Thanks for creating an easy to deploy and use continuous integration framework that also runs on Windows and is written and configured in Python code. This allows to run the Nuitka tests long before release time. * The `Redbaron project `__ Thanks for creating a white space preserving and easy to use toolwork for refactoring Python. This has allows us to automatically format my Python code according to preferences and make global changes easily. * The `isort project `__ Thanks for making nice import ordering so easy. This makes it so easy to let your IDE do it and clean up afterwards. Updates for this Manual ======================= This document is written in REST. That is an ASCII format which is readable as ASCII, but used to generate PDF or HTML documents. You will find the current source under: http://nuitka.net/gitweb/?p=Nuitka.git;a=blob_plain;f=README.rst And the current PDF under: http://nuitka.net/doc/README.pdf