pax_global_header00006660000000000000000000000064145473515440014526gustar00rootroot0000000000000052 comment=1415547c33d527dd8653626a8746f26063b1e415 agate-sql-0.7.2/000077500000000000000000000000001454735154400134125ustar00rootroot00000000000000agate-sql-0.7.2/.github/000077500000000000000000000000001454735154400147525ustar00rootroot00000000000000agate-sql-0.7.2/.github/workflows/000077500000000000000000000000001454735154400170075ustar00rootroot00000000000000agate-sql-0.7.2/.github/workflows/ci.yml000066400000000000000000000013421454735154400201250ustar00rootroot00000000000000name: CI on: [push, pull_request] jobs: build: if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository runs-on: ${{ matrix.os }} strategy: matrix: os: [macos-latest, windows-latest, ubuntu-latest] python-version: [3.8, 3.9, '3.10', '3.11', '3.12', pypy-3.9] steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} cache: pip cache-dependency-path: setup.py - run: pip install .[test] coveralls - run: pytest --cov agatesql - env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: coveralls --service=github agate-sql-0.7.2/.github/workflows/lint.yml000066400000000000000000000010271454735154400205000ustar00rootroot00000000000000name: Lint on: [push, pull_request] jobs: build: if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: '3.10' cache: pip cache-dependency-path: setup.py - run: pip install --upgrade check-manifest flake8 isort setuptools - run: check-manifest - run: flake8 . - run: isort . --check-only agate-sql-0.7.2/.github/workflows/pypi.yml000066400000000000000000000013261454735154400205150ustar00rootroot00000000000000name: Publish to PyPI on: push jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: '3.10' - run: pip install --upgrade build - run: python -m build --sdist --wheel - name: Publish to TestPyPI uses: pypa/gh-action-pypi-publish@release/v1 with: password: ${{ secrets.TEST_PYPI_API_TOKEN }} repository-url: https://test.pypi.org/legacy/ skip-existing: true - name: Publish to PyPI if: startsWith(github.ref, 'refs/tags') uses: pypa/gh-action-pypi-publish@release/v1 with: password: ${{ secrets.PYPI_API_TOKEN }} agate-sql-0.7.2/.gitignore000066400000000000000000000001311454735154400153750ustar00rootroot00000000000000.DS_Store *.pyc *.swp *.swo *.egg-info docs/_build dist .coverage build .proof .test.png agate-sql-0.7.2/.readthedocs.yaml000066400000000000000000000002531454735154400166410ustar00rootroot00000000000000version: 2 build: os: ubuntu-20.04 tools: python: "3.9" python: install: - path: . - requirements: docs/requirements.txt sphinx: fail_on_warning: true agate-sql-0.7.2/AUTHORS.rst000066400000000000000000000010511454735154400152660ustar00rootroot00000000000000The following individuals have contributed code to agate-sql: * `Christopher Groskopf `_ * `Adrian Klaver `_ * `James McKinney `_ * `Chris Keller `_ * `git-clueless `_ * `z2s8 `_ * `Jake Zimmerman `_ * `Shige Takeda `_ * `Roger Webb `_ * `Steve Kowalik `_ agate-sql-0.7.2/CHANGELOG.rst000066400000000000000000000070441454735154400154400ustar00rootroot000000000000000.7.2 - Jan 9, 2024 -------------------- * fix: Remove internal use of transactions (added in 0.6.0), because csvkit's csvsql already starts a transaction. 0.7.1 - Jan 9, 2024 -------------------- * feat: Add experimental support for Ingres. * fix: Restore internal use of transactions instead of savepoints, because not all database engines support savepoints. 0.7.0 - Oct 18, 2023 -------------------- * feat: Use `Fast Executemany Mode `_ when using the PyODBC SQL Server dialect. * Add Python 3.12 support. * Drop support for Python 3.6 (2021-12-23), 3.7 (2023-06-27). 0.6.0 - Sep 26, 2023 -------------------- * Allow SQLAlchemy 2. Disallow SQLAlchemy < 1.4. 0.5.9 - Jan 28, 2023 -------------------- * Disallow SQLAlchemy 2. 0.5.8 - September 15, 2021 -------------------------- * Fix tests for Linux packages. 0.5.7 - July 13, 2021 --------------------- * Add wheels distribution. 0.5.6 - March 4, 2021 --------------------- * Fix test that fails in specific environments. 0.5.5 - July 7, 2020 -------------------- * Set type to ``DATETIME`` for datetime (MS SQL). * Drop support for Python 2.7 (EOL 2020-01-01), 3.4 (2019-03-18), 3.5 (2020-09-13). 0.5.4 - March 16, 2019 ---------------------- * Add ``min_col_len`` and ``col_len_multiplier`` options to :meth:`.Table.to_sql` to control the length of text columns. * agate-sql is now tested against Python 3.7. * Drop support for Python 3.3 (end-of-life was September 29, 2017). Dialect-specific: * Add support for CrateDB. * Set type to ``BIT`` for boolean (MS SQL). * Eliminate SQLite warning about Decimal numbers. 0.5.3 - January 28, 2018 ------------------------ * Add ``chunk_size`` option to :meth:`.Table.to_sql` to write rows in batches. * Add ``unique_constraint`` option to :meth:`.Table.to_sql` to include in a UNIQUE constraint. Dialect-specific: * Specify precision and scale for ``DECIMAL`` (MS SQL, MySQL, Oracle). * Set length of ``VARCHAR`` to ``1`` even if maximum length is ``0`` (MySQL). * Set type to ``TEXT`` if maximum length is greater than 21,844 (MySQL). 0.5.2 - April 28, 2017 ---------------------- * Add ``create_if_not_exists`` flag to :meth:`.Table.to_sql`. 0.5.1 - February 27, 2017 ------------------------- * Add ``prefixes`` option to :func:`.to_sql` to add expressions following the INSERT keyword, like OR IGNORE or OR REPLACE. * Use ``TIMESTAMP`` instead of ``DATETIME`` for DateTime columns. 0.5.0 - December 23, 2016 ------------------------- * ``VARCHAR`` columns are now generated with proper length constraints (unless explicilty disabled). * Tables can now be created from query results using :func:`.from_sql_query`. * Add support for running queries directly on tables with :func:`.sql_query`. * When creating tables, ``NOT NULL`` constraints will be created by default. * SQL create statements can now be generated without being executed with :func:`.to_sql_create_statement` 0.4.0 - December 19, 2016 ------------------------- * Modified ``example.py`` so it no longer depends on Postgres. * It is no longer necessary to run :code:`agatesql.patch()` after importing agatesql. * Upgrade required agate to ``1.5.0``. 0.3.0 - November 5, 2015 ------------------------ * Add ``overwrite`` flag to :meth:`.Table.to_sql`. * Removed Python 2.6 support. * Updated agate dependency to version 1.1.0. * Additional SQL types are now supported. (#4, #10) 0.2.0 - October 22, 2015 ------------------------ * Add explicit patch function. 0.1.0 - September 22, 2015 -------------------------- * Initial version. agate-sql-0.7.2/COPYING000066400000000000000000000021131454735154400144420ustar00rootroot00000000000000The MIT License Copyright (c) 2017 Christopher Groskopf and contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. agate-sql-0.7.2/MANIFEST.in000066400000000000000000000003451454735154400151520ustar00rootroot00000000000000include *.db include *.py include *.rst include COPYING recursive-include docs *.py recursive-include docs *.rst recursive-include docs *.txt recursive-include docs Makefile recursive-include tests *.py exclude .readthedocs.yaml agate-sql-0.7.2/README.rst000066400000000000000000000022531454735154400151030ustar00rootroot00000000000000.. image:: https://github.com/wireservice/agate-sql/workflows/CI/badge.svg :target: https://github.com/wireservice/agate-sql/actions :alt: Build status .. image:: https://coveralls.io/repos/wireservice/agate-sql/badge.svg?branch=master :target: https://coveralls.io/r/wireservice/agate-sql :alt: Coverage status .. image:: https://img.shields.io/pypi/dm/agate-sql.svg :target: https://pypi.python.org/pypi/agate-sql :alt: PyPI downloads .. image:: https://img.shields.io/pypi/v/agate-sql.svg :target: https://pypi.python.org/pypi/agate-sql :alt: Version .. image:: https://img.shields.io/pypi/l/agate-sql.svg :target: https://pypi.python.org/pypi/agate-sql :alt: License .. image:: https://img.shields.io/pypi/pyversions/agate-sql.svg :target: https://pypi.python.org/pypi/agate-sql :alt: Support Python versions agate-sql adds SQL read/write support to `agate `_. Important links: * agate https://agate.rtfd.org * Documentation: https://agate-sql.rtfd.org * Repository: https://github.com/wireservice/agate-sql * Issues: https://github.com/wireservice/agate-sql/issues agate-sql-0.7.2/agatesql/000077500000000000000000000000001454735154400152135ustar00rootroot00000000000000agate-sql-0.7.2/agatesql/__init__.py000066400000000000000000000000261454735154400173220ustar00rootroot00000000000000import agatesql.table agate-sql-0.7.2/agatesql/table.py000066400000000000000000000334711454735154400166640ustar00rootroot00000000000000""" This module contains the agatesql extensions to :class:`Table `. """ import datetime import decimal from urllib.parse import urlsplit import agate from sqlalchemy import Column, MetaData, Table, UniqueConstraint, create_engine, dialects from sqlalchemy.dialects.mssql import BIT from sqlalchemy.dialects.oracle import INTERVAL as ORACLE_INTERVAL from sqlalchemy.dialects.postgresql import INTERVAL as POSTGRES_INTERVAL from sqlalchemy.engine import Connection from sqlalchemy.schema import CreateTable from sqlalchemy.sql import select from sqlalchemy.types import BOOLEAN, DATE, DATETIME, DECIMAL, FLOAT, TEXT, TIMESTAMP, VARCHAR, Interval SQL_TYPE_MAP = { agate.Boolean: None, # See below agate.Number: None, # See below agate.Date: DATE, agate.DateTime: None, # See below agate.TimeDelta: None, # See below agate.Text: VARCHAR, } DATETIME_MAP = { 'mssql': DATETIME, } BOOLEAN_MAP = { 'mssql': BIT, } NUMBER_MAP = { 'crate': FLOAT, 'sqlite': FLOAT, } INTERVAL_MAP = { 'postgresql': POSTGRES_INTERVAL, 'oracle': ORACLE_INTERVAL, } def get_engine_and_connection(connection_or_string=None): """ Gets a connection to a specific SQL alchemy backend. If an existing connection is provided, it will be passed through. If no connection string is provided, then in in-memory SQLite database will be created. """ if connection_or_string is None: engine = create_engine('sqlite:///:memory:') connection = engine.connect() return None, connection if isinstance(connection_or_string, Connection): connection = connection_or_string return None, connection kwargs = {} if urlsplit(connection_or_string).scheme == 'mssql+pyodbc': kwargs = {'fast_executemany': True} engine = create_engine(connection_or_string, **kwargs) connection = engine.connect() return engine, connection def from_sql(cls, connection_or_string, table_name): """ Create a new :class:`agate.Table` from a given SQL table. Types will be inferred from the database schema. Monkey patched as class method :meth:`Table.from_sql`. :param connection_or_string: An existing sqlalchemy connection or connection string. :param table_name: The name of a table in the referenced database. """ engine, connection = get_engine_and_connection(connection_or_string) metadata = MetaData() sql_table = Table(table_name, metadata, autoload_with=connection) column_names = [] column_types = [] for sql_column in sql_table.columns: column_names.append(sql_column.name) if type(sql_column.type) in INTERVAL_MAP.values(): py_type = datetime.timedelta else: py_type = sql_column.type.python_type if py_type in [int, float, decimal.Decimal]: if py_type is float: sql_column.type.asdecimal = True column_types.append(agate.Number()) elif py_type is bool: column_types.append(agate.Boolean()) elif issubclass(py_type, str): column_types.append(agate.Text()) elif py_type is datetime.date: column_types.append(agate.Date()) elif py_type is datetime.datetime: column_types.append(agate.DateTime()) elif py_type is datetime.timedelta: column_types.append(agate.TimeDelta()) else: raise ValueError('Unsupported sqlalchemy column type: %s' % type(sql_column.type)) s = select(sql_table) rows = connection.execute(s) try: return agate.Table(rows, column_names, column_types) finally: if engine is not None: connection.close() engine.dispose() def from_sql_query(self, query): """ Create an agate table from the results of a SQL query. Note that column data types will be inferred from the returned data, not the column types declared in SQL (if any). This is more flexible than :func:`.from_sql` but could result in unexpected typing issues. :param query: A SQL query to execute. """ _, connection = get_engine_and_connection() # Must escape '%'. # @see https://github.com/wireservice/csvkit/issues/440 # @see https://bitbucket.org/zzzeek/sqlalchemy/commits/5bc1f17cb53248e7cea609693a3b2a9bb702545b rows = connection.execute(query.replace('%', '%%')) table = agate.Table(list(rows), column_names=rows._metadata.keys) return table def make_sql_column(column_name, column, sql_type_kwargs=None, sql_column_kwargs=None, sql_column_type=None): """ Creates a sqlalchemy column from agate column data. :param column_name: The name of the column. :param column: The agate column. :param sql_type_kwargs: Additional kwargs to passed through to the type constructor, such as ``length``. :param sql_column_kwargs: Additional kwargs to passed through to the Column constructor, such as ``nullable``. :param sql_column_type: The type of the column (optional). """ if not sql_column_type: for agate_type, sql_type in SQL_TYPE_MAP.items(): if isinstance(column.data_type, agate_type): sql_column_type = sql_type break if sql_column_type is None: raise ValueError('Unsupported column type: %s' % column.data_type) sql_type_kwargs = sql_type_kwargs or {} sql_column_kwargs = sql_column_kwargs or {} return Column(column_name, sql_column_type(**sql_type_kwargs), **sql_column_kwargs) def make_sql_table(table, table_name, dialect=None, db_schema=None, constraints=True, unique_constraint=[], connection=None, min_col_len=1, col_len_multiplier=1): """ Generates a SQL alchemy table from an agate table. """ metadata = MetaData() sql_table = Table(table_name, metadata, schema=db_schema) SQL_TYPE_MAP[agate.Boolean] = BOOLEAN_MAP.get(dialect, BOOLEAN) SQL_TYPE_MAP[agate.DateTime] = DATETIME_MAP.get(dialect, TIMESTAMP) SQL_TYPE_MAP[agate.Number] = NUMBER_MAP.get(dialect, DECIMAL) SQL_TYPE_MAP[agate.TimeDelta] = INTERVAL_MAP.get(dialect, Interval) for column_name, column in table.columns.items(): sql_column_type = None sql_type_kwargs = {} sql_column_kwargs = {} if constraints: if isinstance(column.data_type, agate.Text) and dialect in ('ingres', 'mysql'): length = table.aggregate(agate.MaxLength(column_name)) * decimal.Decimal(col_len_multiplier) if ( # https://dev.mysql.com/doc/refman/8.2/en/string-type-syntax.html dialect == 'mysql' and length > 21844 # 65,535 bytes divided by 3 # https://docs.actian.com/ingres/11.2/index.html#page/SQLRef/Character_Data_Types.htm or dialect == 'ingres' and length > 10666 # 32,000 bytes divided by 3 ): sql_column_type = TEXT # If length is zero, SQLAlchemy may raise "VARCHAR requires a length on dialect mysql". else: sql_type_kwargs['length'] = length if length >= min_col_len else min_col_len # PostgreSQL and SQLite don't have scale default 0. # @see https://www.postgresql.org/docs/9.2/static/datatype-numeric.html # @see https://www.sqlite.org/datatype3.html if isinstance(column.data_type, agate.Number) and dialect in ('ingres', 'mssql', 'mysql', 'oracle'): # Ingres has precision range 1-39 and default 5, scale default 0. # @see https://docs.actian.com/ingres/11.2/index.html#page/SQLRef/Storage_Formats_of_Data_Types.htm # MySQL has precision range 1-65 and default 10, scale default 0. # @see https://dev.mysql.com/doc/refman/8.2/en/fixed-point-types.html # Oracle has precision range 1-38 and default 38, scale default 0. # @see https://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm#CNCPT1832 # SQL Server has range 1-38 and default 18, scale default 0. # @see https://docs.microsoft.com/en-us/sql/t-sql/data-types/decimal-and-numeric-transact-sql sql_type_kwargs['precision'] = 38 sql_type_kwargs['scale'] = table.aggregate(agate.MaxPrecision(column_name)) # Avoid errors due to NO_ZERO_DATE. # @see https://dev.mysql.com/doc/refman/8.2/en/sql-mode.html#sqlmode_no_zero_date if not isinstance(column.data_type, agate.DateTime): sql_column_kwargs['nullable'] = table.aggregate(agate.HasNulls(column_name)) sql_table.append_column(make_sql_column(column_name, column, sql_type_kwargs, sql_column_kwargs, sql_column_type)) if unique_constraint: sql_table.append_constraint(UniqueConstraint(*unique_constraint)) return sql_table def to_sql(self, connection_or_string, table_name, overwrite=False, create=True, create_if_not_exists=False, insert=True, prefixes=[], db_schema=None, constraints=True, unique_constraint=[], chunk_size=None, min_col_len=1, col_len_multiplier=1): """ Write this table to the given SQL database. Monkey patched as instance method :meth:`Table.to_sql`. :param connection_or_string: An existing sqlalchemy connection or a connection string. :param table_name: The name of the SQL table to create. :param overwrite: Drop any existing table with the same name before creating. :param create: Create the table. :param create_if_not_exists: When creating the table, don't fail if the table already exists. :param insert: Insert table data. :param prefixes: Add prefixes to the insert query. :param db_schema: Create table in the specified database schema. :param constraints: Generate constraints such as ``nullable`` for table columns. :param unique_constraint: The names of the columns to include in a UNIQUE constraint. :param chunk_size: Write rows in batches of this size. If not set, rows will be written at once. :param col_min_len: The minimum length of text columns. :param col_len_multiplier: Multiply the maximum column length by this multiplier to accomodate larger values in later runs. """ engine, connection = get_engine_and_connection(connection_or_string) dialect = connection.engine.dialect.name sql_table = make_sql_table(self, table_name, dialect=dialect, db_schema=db_schema, constraints=constraints, unique_constraint=unique_constraint, connection=connection, min_col_len=min_col_len, col_len_multiplier=col_len_multiplier) if create: if overwrite: sql_table.drop(bind=connection, checkfirst=True) sql_table.create(bind=connection, checkfirst=create_if_not_exists) if insert: insert = sql_table.insert() for prefix in prefixes: insert = insert.prefix_with(prefix) if chunk_size is None: connection.execute(insert, [dict(zip(self.column_names, row)) for row in self.rows]) else: number_of_rows = len(self.rows) for index in range((number_of_rows - 1) // chunk_size + 1): end_index = (index + 1) * chunk_size if end_index > number_of_rows: end_index = number_of_rows connection.execute(insert, [dict(zip(self.column_names, row)) for row in self.rows[index * chunk_size:end_index]]) try: return sql_table finally: if engine is not None: connection.close() engine.dispose() def to_sql_create_statement(self, table_name, dialect=None, db_schema=None, constraints=True, unique_constraint=[]): """ Generates a CREATE TABLE statement for this SQL table, but does not execute it. :param table_name: The name of the SQL table to create. :param dialect: The dialect of SQL to use for the table statement. :param db_schema: Create table in the specified database schema. :param constraints: Generate constraints such as ``nullable`` for table columns. :param unique_constraint: The names of the columns to include in a UNIQUE constraint. """ sql_table = make_sql_table(self, table_name, dialect=dialect, db_schema=db_schema, constraints=constraints, unique_constraint=unique_constraint) if dialect: sql_dialect = dialects.registry.load(dialect)() else: sql_dialect = None return str(CreateTable(sql_table).compile(dialect=sql_dialect)).strip() + ';' def sql_query(self, query, table_name='agate'): """ Convert this agate table into an intermediate, in-memory sqlite table, run a query against it, and then return the results as a new agate table. Multiple queries may be separated with semicolons. :param query: One SQL query, or multiple queries to be run consecutively separated with semicolons. :param table_name: The name to use for the table in the queries, defaults to ``agate``. """ _, connection = get_engine_and_connection() # Execute the specified SQL queries queries = query.split(';') rows = None self.to_sql(connection, table_name) for q in queries: if q: rows = connection.exec_driver_sql(q) table = agate.Table(list(rows), column_names=rows._metadata.keys) return table agate.Table.from_sql = classmethod(from_sql) agate.Table.from_sql_query = classmethod(from_sql_query) agate.Table.to_sql = to_sql agate.Table.to_sql_create_statement = to_sql_create_statement agate.Table.sql_query = sql_query agate-sql-0.7.2/docs/000077500000000000000000000000001454735154400143425ustar00rootroot00000000000000agate-sql-0.7.2/docs/Makefile000066400000000000000000000107661454735154400160140ustar00rootroot00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " text to make text files" @echo " man to make manual pages" @echo " changes to make an overview of all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: -rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/agatesql.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/agatesql.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/agatesql" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/agatesql" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." make -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." agate-sql-0.7.2/docs/conf.py000066400000000000000000000024721454735154400156460ustar00rootroot00000000000000# Configuration file for the Sphinx documentation builder. # # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html import os import sys sys.path.insert(0, os.path.abspath('..')) # -- Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information project = 'agate-sql' copyright = '2017, Christopher Groskopf' version = '0.7.2' release = version # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx' ] templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output html_theme = 'furo' htmlhelp_basename = 'agatesqldoc' autodoc_default_options = { 'members': None, 'member-order': 'bysource', 'show-inheritance': True, } intersphinx_mapping = { 'python': ('https://docs.python.org/3', None), 'agate': ('https://agate.readthedocs.org/en/latest/', None) } agate-sql-0.7.2/docs/index.rst000066400000000000000000000036771454735154400162200ustar00rootroot00000000000000=================== agate-sql |release| =================== .. include:: ../README.rst Install ======= To install: .. code-block:: bash pip install agate-sql For details on development or supported platforms see the `agate documentation `_. .. warning:: You'll need to have the correct `sqlalchemy drivers `_ installed for whatever database you plan to access. For instance, in order to read/write tables in a Postgres database, you'll also need to ``pip install psycopg2``. Usage ===== agate-sql uses a monkey patching pattern to add SQL support to all :class:`agate.Table ` instances. .. code-block:: python import agate import agatesql Importing :mod:`.agatesql` attaches new methods to :class:`agate.Table `. For example, to import a table named :code:`doctors` from a local postgresql database named :code:`hospitals` you will use :meth:`.from_sql`: .. code-block:: python new_table = agate.Table.from_sql('postgresql:///hospitals', 'doctors') To save this table back to the database: .. code-block:: python new_table.to_sql('postgresql:///hospitals', 'doctors') The first argument to either function can be any valid `sqlalchemy connection string `_. The second argument must be a database name. (Arbitrary SQL queries are not supported.) That's all there is to it. === API === .. autofunction:: agatesql.table.from_sql .. autofunction:: agatesql.table.from_sql_query .. autofunction:: agatesql.table.to_sql .. autofunction:: agatesql.table.to_sql_create_statement .. autofunction:: agatesql.table.sql_query Authors ======= .. include:: ../AUTHORS.rst Changelog ========= .. include:: ../CHANGELOG.rst License ======= .. include:: ../COPYING Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` agate-sql-0.7.2/docs/requirements.txt000066400000000000000000000000351454735154400176240ustar00rootroot00000000000000furo sphinx>2 docutils>=0.18 agate-sql-0.7.2/example.db000066400000000000000000000040001454735154400153460ustar00rootroot00000000000000SQLite format 3@ - _!tabletesttestCREATE TABLE test ( number DECIMAL NOT NULL, text VARCHAR(1) NOT NULL ) b aagate-sql-0.7.2/example.py000077500000000000000000000004271454735154400154250ustar00rootroot00000000000000#!/usr/bin/env python import agate import agatesql table = agate.Table.from_sql('sqlite:///example.db', 'test') print(table.column_names) print(table.column_types) print(len(table.columns)) print(len(table.rows)) table.to_sql('sqlite:///example.db', 'test', overwrite=True) agate-sql-0.7.2/setup.cfg000066400000000000000000000002701454735154400152320ustar00rootroot00000000000000[flake8] max-line-length = 119 per-file-ignores = # imported but unused agatesql/__init__.py: F401 example.py: F401 [isort] line_length = 119 [bdist_wheel] universal = 1 agate-sql-0.7.2/setup.py000066400000000000000000000031071454735154400151250ustar00rootroot00000000000000from setuptools import find_packages, setup with open('README.rst') as f: long_description = f.read() setup( name='agate-sql', version='0.7.2', description='agate-sql adds SQL read/write support to agate.', long_description=long_description, long_description_content_type='text/x-rst', author='Christopher Groskopf', author_email='chrisgroskopf@gmail.com', url='https://agate-sql.readthedocs.org/', license='MIT', classifiers=[ 'Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: MIT License', 'Natural Language :: English', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Programming Language :: Python :: 3.12', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: Scientific/Engineering :: Information Analysis', 'Topic :: Software Development :: Libraries :: Python Modules', ], packages=find_packages(exclude=['tests', 'tests.*']), install_requires=[ 'agate>=1.5.0', 'sqlalchemy>=1.4', ], extras_require={ 'test': [ 'crate', 'geojson', 'pytest', 'pytest-cov', ], } ) agate-sql-0.7.2/tests/000077500000000000000000000000001454735154400145545ustar00rootroot00000000000000agate-sql-0.7.2/tests/__init__.py000066400000000000000000000000001454735154400166530ustar00rootroot00000000000000agate-sql-0.7.2/tests/test_agatesql.py000066400000000000000000000250631454735154400177740ustar00rootroot00000000000000from decimal import Decimal from textwrap import dedent import agate from sqlalchemy import create_engine from sqlalchemy.exc import IntegrityError import agatesql class TestSQL(agate.AgateTestCase): def setUp(self): self.rows = ( (1.123, 'a', True, '11/4/2015', '11/4/2015 12:22 PM'), (2, '👍', False, '11/5/2015', '11/4/2015 12:45 PM'), (2, 'c', False, '11/5/2015', '11/4/2015 12:45 PM'), (None, 'b', None, None, None), ) self.column_names = [ 'number', 'textcol', 'boolean', 'date', 'datetime', ] self.column_types = [ agate.Number(), agate.Text(), agate.Boolean(), agate.Date(), agate.DateTime(), ] self.table = agate.Table(self.rows, self.column_names, self.column_types) self.connection_string = 'sqlite:///:memory:' def test_back_and_forth(self): engine = create_engine(self.connection_string) connection = engine.connect() self.table.to_sql(connection, 'test') table = agate.Table.from_sql(connection, 'test') self.assertSequenceEqual(table.column_names, self.column_names) self.assertIsInstance(table.column_types[0], agate.Number) self.assertIsInstance(table.column_types[1], agate.Text) self.assertIsInstance(table.column_types[2], agate.Boolean) self.assertIsInstance(table.column_types[3], agate.Date) self.assertIsInstance(table.column_types[4], agate.DateTime) self.assertEqual(len(table.rows), len(self.table.rows)) self.assertSequenceEqual(table.rows[0], self.table.rows[0]) def test_create_if_not_exists(self): column_names = ['id', 'name'] column_types = [agate.Number(), agate.Text()] rows1 = ( (1, 'Jake'), (2, 'Howard'), ) rows2 = ( (3, 'Liz'), (4, 'Tim'), ) table1 = agate.Table(rows1, column_names, column_types) table2 = agate.Table(rows2, column_names, column_types) engine = create_engine(self.connection_string) connection = engine.connect() # Write two agate tables into the same SQL table table1.to_sql(connection, 'create_if_not_exists_test', create=True, create_if_not_exists=True, insert=True) table2.to_sql(connection, 'create_if_not_exists_test', create=True, create_if_not_exists=True, insert=True) table = agate.Table.from_sql(connection, 'create_if_not_exists_test') self.assertSequenceEqual(table.column_names, column_names) self.assertIsInstance(table.column_types[0], agate.Number) self.assertIsInstance(table.column_types[1], agate.Text) self.assertEqual(len(table.rows), len(table1.rows) + len(table1.rows)) self.assertSequenceEqual(table.rows[0], table1.rows[0]) def test_unique_constraint(self): engine = create_engine(self.connection_string) connection = engine.connect() with self.assertRaises(IntegrityError): self.table.to_sql(connection, 'unique_constraint_test', unique_constraint=['number']) def test_prefixes(self): engine = create_engine(self.connection_string) connection = engine.connect() self.table.to_sql(connection, 'prefixes_test', prefixes=['OR REPLACE'], unique_constraint=['number']) table = agate.Table.from_sql(connection, 'prefixes_test') self.assertSequenceEqual(table.column_names, self.column_names) self.assertIsInstance(table.column_types[0], agate.Number) self.assertIsInstance(table.column_types[1], agate.Text) self.assertIsInstance(table.column_types[2], agate.Boolean) self.assertIsInstance(table.column_types[3], agate.Date) self.assertIsInstance(table.column_types[4], agate.DateTime) self.assertEqual(len(table.rows), len(self.table.rows) - 1) self.assertSequenceEqual(table.rows[1], self.table.rows[2]) def test_to_sql_create_statement(self): statement = self.table.to_sql_create_statement('test_table') self.assertEqual(statement.replace('\t', ' '), dedent('''\ CREATE TABLE test_table ( number DECIMAL, textcol VARCHAR NOT NULL, boolean BOOLEAN, date DATE, datetime TIMESTAMP );''')) # noqa: W291 def test_to_sql_create_statement_no_constraints(self): statement = self.table.to_sql_create_statement('test_table', constraints=False) self.assertEqual(statement.replace('\t', ' '), dedent('''\ CREATE TABLE test_table ( number DECIMAL, textcol VARCHAR, boolean BOOLEAN, date DATE, datetime TIMESTAMP );''')) # noqa: W291 def test_to_sql_create_statement_unique_constraint(self): statement = self.table.to_sql_create_statement('test_table', unique_constraint=['number', 'textcol']) self.assertEqual(statement.replace('\t', ' '), dedent('''\ CREATE TABLE test_table ( number DECIMAL, textcol VARCHAR NOT NULL, boolean BOOLEAN, date DATE, datetime TIMESTAMP, UNIQUE (number, textcol) );''')) # noqa: W291 def test_to_sql_create_statement_with_schema(self): statement = self.table.to_sql_create_statement('test_table', db_schema='test_schema', dialect='mysql') # https://github.com/wireservice/agate-sql/issues/33#issuecomment-879267838 if 'CHECK' in statement: expected = '''\ CREATE TABLE test_schema.test_table ( number DECIMAL(38, 3), textcol VARCHAR(1) NOT NULL, boolean BOOL, date DATE, datetime TIMESTAMP NULL, CHECK (boolean IN (0, 1)) );''' # noqa: W291 else: expected = '''\ CREATE TABLE test_schema.test_table ( number DECIMAL(38, 3), textcol VARCHAR(1) NOT NULL, boolean BOOL, date DATE, datetime TIMESTAMP NULL );''' # noqa: W291 self.assertEqual(statement.replace('\t', ' '), dedent(expected)) def test_to_sql_create_statement_with_dialects(self): for dialect in ['crate', 'mssql', 'mysql', 'postgresql', 'sqlite']: self.table.to_sql_create_statement('test_table', dialect=dialect) def test_to_sql_create_statement_zero_width(self): rows = ((1, ''), (2, '')) column_names = ['id', 'name'] column_types = [agate.Number(), agate.Text()] table = agate.Table(rows, column_names, column_types) statement = table.to_sql_create_statement('test_table', db_schema='test_schema', dialect='mysql') self.assertEqual(statement.replace('\t', ' '), dedent('''\ CREATE TABLE test_schema.test_table ( id DECIMAL(38, 0) NOT NULL, name VARCHAR(1) );''')) # noqa: W291 def test_to_sql_create_statement_wide_width(self): rows = ((1, 'x' * 21845), (2, '')) column_names = ['id', 'name'] column_types = [agate.Number(), agate.Text()] table = agate.Table(rows, column_names, column_types) statement = table.to_sql_create_statement('test_table', db_schema='test_schema', dialect='mysql') self.assertEqual(statement.replace('\t', ' '), dedent('''\ CREATE TABLE test_schema.test_table ( id DECIMAL(38, 0) NOT NULL, name TEXT );''')) # noqa: W291 def test_make_sql_table_col_len_multiplier(self): rows = ((1, 'x' * 10), (2, '')) column_names = ['id', 'name'] column_types = [agate.Number(), agate.Text()] table = agate.Table(rows, column_names, column_types) sql_table = agatesql.table.make_sql_table(table, 'test_table', dialect='mysql', db_schema='test_schema', constraints=True, col_len_multiplier=1.5) self.assertEqual(sql_table.columns.get('name').type.length, 15) def test_make_sql_table_min_col_len(self): rows = ((1, 'x' * 10), (2, '')) column_names = ['id', 'name'] column_types = [agate.Number(), agate.Text()] table = agate.Table(rows, column_names, column_types) sql_table = agatesql.table.make_sql_table(table, 'test_table', dialect='mysql', db_schema='test_schema', constraints=True, min_col_len=20) self.assertEqual(sql_table.columns.get('name').type.length, 20) def test_sql_query_simple(self): results = self.table.sql_query('select * from agate') self.assertColumnNames(results, self.table.column_names) self.assertRows(results, self.table.rows) def test_sql_query_limit(self): results = self.table.sql_query('select * from agate limit 2') self.assertColumnNames(results, self.table.column_names) self.assertRows(results, self.table.rows[:2]) def test_sql_query_select(self): results = self.table.sql_query('select number, boolean from agate') self.assertColumnNames(results, ['number', 'boolean']) self.assertColumnTypes(results, [agate.Number, agate.Boolean]) self.assertRows(results, [ [Decimal('1.123'), True], [2, False], [2, False], [None, None], ]) def test_sql_query_aggregate(self): results = self.table.sql_query('select sum(number) as total from agate') self.assertColumnNames(results, ['total']) self.assertColumnTypes(results, [agate.Number]) self.assertRows(results, [[Decimal('5.123')]]) def test_chunk_size(self): column_names = ['number'] column_types = [agate.Number()] rows = [] expected = 0 for n in range(9999): rows.append((n,)) expected += n engine = create_engine(self.connection_string) connection = engine.connect() try: table = agate.Table(rows, column_names, column_types) table.to_sql(connection, 'test_chunk_size', overwrite=True, chunk_size=100) table = agate.Table.from_sql(connection, 'test_chunk_size') actual = sum(r[0] for r in table.rows) self.assertEqual(len(table.rows), len(rows)) self.assertEqual(expected, actual) finally: connection.close() engine.dispose()