SQLite 3.6 has new VFS functionality which defines the interface between the SQLite core and the underlying operating system. The majority of the functionality deals with files. APSW exposes this functionality letting you provide your own routines. You can also inherit from an existing vfs making it easy to augment or override specific routines. For example you could obfuscate your database by XORing the data implemented by augmenting the read and write methods. The method names are exactly the same as SQLite uses making it easier to read the SQLite documentation, trouble tickets, web searches or mailing lists. The SQLite convention results in names like xAccess, xCurrentTime and xWrite.
You specify which VFS to use as a parameter to the Connection constructor.
db=apsw.Connection("file", vfs="myvfs")
The easiest way to get started is to make a VFS derived class that inherits from the default vfs. Then override methods you want to change behaviour of. If you want to just change how file operations are done then you have to override VFS.xOpen() to return a file instance that has your overridden VFSFile methods. The example demonstrates obfuscating the database file contents.
To return an error from any routine you should raise an exception. The exception will be translated into the appropriate SQLite error code for SQLite. To return a specific SQLite error code use exceptionfor(). If the exception does not map to any specific error code then SQLITE_ERROR which corresponds to SQLError is returned to SQLite.
The SQLite code that deals with VFS errors behaves in varying ways. Some routines have no way to return an error (eg xDlOpen just returns zero/NULL on being unable to load a library, xSleep has no error return parameter), others are unified (eg almost any error in xWrite will be returned to the user as disk full error). Sometimes errors are ignored as they are harmless such as when a journal can’t be deleted after a commit (the journal is marked as obsolete before being deleted). Simple operations such as opening a database can result in many different VFS function calls such as hot journals being detected, locking, and read/writes for playback/rollback.
To avoid confusion with exceptions being raised in the VFS and exceptions from normal code to open Connections or execute SQL queries, VFS exceptions are not raised in the normal way. (If they were, only one could be raised and it would obscure whatever exceptions the Connection open or SQL query execute wanted to raise.) Instead the VFS.excepthook() or VFSFile.excepthook() method is called with a tuple of exception type, exception value and exception traceback. The default implementation of excepthook calls sys.excepthook() which under Python 2 shows the stack trace and under Python 3 merely prints the exception value. (If sys.excepthook fails then PyErr_Display() is called.)
In normal VFS usage there will be no exceptions raised, or specific expected ones which APSW clears after noting them and returning the appropriate value back to SQLite. The exception hooking behaviour helps you find issues in your code or unexpected behaviour of the external environment. Remember that augmented stack traces are available which significantly increase detail about the exceptions.
As an example, lets say you have a divide by zero error in your xWrite routine. The table below shows what happens with time going down and across.
| Python Query Code | SQLite and APSW C code | Python VFS code |
|---|---|---|
| cursor.execute("update table set foo=3") | ||
| SQLite starts executing query | ||
| Your VFS routines are called | ||
| Your xWrite divides by zero | ||
| VFSFile.excepthook() is called with ZeroDivision exception | ||
| SQLITE_ERROR (closest matching SQLite error code) is returned to SQLite by APSW | ||
| SQLite error handling and recovery operates which calls more VFS routines. | More VFS routines are called. Any exceptions in these routines will result in VFSFile.excepthook() being called with them. | |
| SQLite returns SQLITE_FULL to APSW | ||
| APSW returns apsw.FullError |
Provides operating system access. You can get an overview in the SQLite documentation. To create a VFS your Python class must inherit from VFS.
| Parameters: | name – The name to register this vfs under. If the name already exists then this vfs will replace the prior one of the same name. Use apsw.vfsnames() to get a list of registered vfs names. :param base: If you would like to inherit behaviour from an already registered vfs then give their name. To inherit from the default vfs, use a zero length string "" as the name. :param makedefault: If true then this vfs will be registered as the default, and will be used by any opens that don’t specify a vfs. :param maxpathname: The maximum length of database name in bytes when represented in UTF-8. If a pathname is passed in longer than this value then SQLite will not be able to open it. |
|---|---|
| Raises ValueError: | |
| If base is not None and the named vfs is not currently registered. | |
Called when there has been an exception in a VFS routine. The default implementation calls sys.excepthook and if that fails then PyErr_Display. The three arguments correspond to what sys.exc_info() would return.
| Parameters: |
|
|---|
Unregisters the VFS making it unavailable to future database opens. You do not need to call this as the VFS is automatically unregistered by when the VFS has no more references or open datatabases using it. It is however useful to call if you have made your VFS be the default and wish to immediately make it be unavailable. It is safe to call this routine multiple times.
Calls: sqlite3_vfs_unregister
SQLite wants to check access permissions. Return True or False accordingly.
| Parameters: |
|
|---|
Return the Julian Day Number as a floating point number where the integer portion is the day and the fractional part is the time. Do not adjust for timezone (ie use UTC).
Delete the named file.
Note
SQLite has 3 different behaviours depending on version for how to handle missing files.
| SQLite < 3.7.8 | Raise an IOError if the file does not exist. |
| SQLite >= 3.7.8 and SQLite < 3.7.15 | Do not raise an exception |
| SQLite >= 3.7.15 | Raise an IOError exception with extendedresult SQLITE_IOERR_DELETE_NOENT |
| Parameters: |
|
|---|
Close and unload the library corresponding to the handle you returned from xDlOpen(). You can use ctypes to do this:
def xDlClose(handle):
# Note leading underscore in _ctypes
_ctypes.dlclose(handle) # Linux/Mac/Unix
_ctypes.FreeLibrary(handle) # Windows
Return an error string describing the last error of xDlOpen() or xDlSym() (ie they returned zero/NULL). If you do not supply this routine then SQLite provides a generic message. To implement this method, catch exceptions in xDlOpen() or xDlSym(), turn them into strings, save them, and return them in this routine. If you have an error in this routine or return None then SQLite’s generic message will be used.
Load the shared library. You should return a number which will be treated as a void pointer at the C level. On error you should return 0 (NULL). The number is passed as is to xDlSym()/xDlClose() so it can represent anything that is convenient for you (eg an index into an array). You can use ctypes to load a library:
def xDlOpen(name):
return ctypes.cdll.LoadLibrary(name)._handle
Returns the address of the named symbol which will be called by SQLite. On error you should return 0 (NULL). You can use ctypes:
def xDlSym(ptr, name):
return _ctypes.dlsym (ptr, name) # Linux/Unix/Mac etc (note leading underscore)
return ctypes.win32.kernel32.GetProcAddress (ptr, name) # Windows
| Parameters: |
|
|---|---|
| Return type: | An int/long with the symbol address |
Return the absolute pathname for name. You can use os.path.abspath to do this.
This method is to return text describing the last error that happened in this thread. If not implemented SQLite’s more generic message is used. However the method is never called by SQLite.
Returns a pointer for the current method implementing the named system call. Return None if the call does not exist.
This method is repeatedly called to iterate over all of the system calls in the vfs. When called with None you should return the name of the first system call. In subsequent calls return the name after the one passed in. If name is the last system call then return None.
Note
Because of internal SQLite implementation semantics memory will be leaked on each call to this function. Consequently you should build up the list of call names once rather than repeatedly doing it.
This method should return a new file object based on name. You can return a VFSFile from a completely different VFS.
| Parameters: |
|
|---|
This method is called once when SQLite needs to seed the random number generator. It is called on the default VFS only. It is not called again, even across apsw.shutdown() calls. You can return less than the number of bytes requested including None. If you return more then the surplus is ignored.
| Return type: | (Python 2) string, buffer (Python 3) bytes, buffer |
|---|
Change a system call used by the VFS. This is useful for testing and some other scenarios such as sandboxing.
| Parameters: |
|
|---|
Raise an exception to return an error. If the system call does not exist then raise NotFoundError.
| Returns: | True if the system call was set. False if the system call is not known. |
|---|
Pause exection of the thread for at least the specified number of microseconds (millionths of a second). This routine is typically called from the busy handler.
| Returns: | How many microseconds you actually requested the operating system to sleep for. For example if your operating system sleep call only takes seconds then you would have to have rounded the microseconds number up to the nearest second and should return that rounded up value. |
|---|
Wraps access to a file. You only need to derive from this class if you want the file object returned from VFS.xOpen() to inherit from an existing VFS implementation.
Note
All file sizes and offsets are 64 bit quantities even on 32 bit operating systems.
| Parameters: |
|
|---|---|
| Raises ValueError: | |
If the named VFS is not registered. |
|
Note
If the VFS that you inherit from supports write ahead logging then your VFSFile will also support the xShm methods necessary to implement wal.
See also
Called when there has been an exception in a VFSFile routine. The default implementation calls sys.excepthook and if that fails then PyErr_Display. The three arguments correspond to what sys.exc_info() would return.
| Parameters: |
|
|---|
Returns True if any database connection (in this or another process) has a lock other than SQLITE_LOCK_NONE or SQLITE_LOCK_SHARED.
Close the database. Note that even if you return an error you should still close the file. It is safe to call this method mutliple times.
Return I/O capabilities (bitwise or of appropriate values). If you do not implement the function or have an error then 0 (the SQLite default) is returned.
Receives file control request typically issued by Connection.filecontrol(). See Connection.filecontrol() for an example of how to pass a Python object to this routine.
| Parameters: |
|
|---|---|
| Returns: | A boolean indicating if the op was understood |
As of SQLite 3.6.10, this method is called by SQLite if you have inherited from an underlying VFSFile. Consequently ensure you pass any unrecognised codes through to your super class. For example:
def xFileControl(self, op, ptr):
if op==1027:
process_quick(ptr)
elif op==1028:
obj=ctypes.py_object.from_address(ptr).value
else:
# this ensures superclass implementation is called
return super(MyFile, self).xFileControl(op, ptr)
# we understood the op
return True
Return the size of the file in bytes. Remember that file sizes are 64 bit quantities even on 32 bit operating systems.
Increase the lock to the level specified which is one of the SQLITE_LOCK family of constants. If you can’t increase the lock level because someone else has locked it, then raise BusyError.
Read the specified amount of data starting at offset. You should make every effort to read all the data requested, or return an error. If you have the file open for non-blocking I/O or if signals happen then it is possible for the underlying operating system to do a partial read. You will need to request the remaining data. Except for empty files SQLite considers short reads to be a fatal error.
| Parameters: |
|
|---|---|
| Return type: | (Python 2) string, buffer. (Python 3) bytes, buffer |
Return the native underlying sector size. SQLite uses the value returned in determining the default database page size. If you do not implement the function or have an error then 4096 (the SQLite default) is returned.
Ensure data is on the disk platters (ie could survive a power failure immediately after the call returns) with the sync flags detailing what needs to be synced. You can sync more than what is requested.
Set the file length to newsize (which may be more or less than the current length).
Decrease the lock to the level specified which is one of the SQLITE_LOCK family of constants.
Write the data starting at absolute offset. You must write all the data requested, or return an error. If you have the file open for non-blocking I/O or if signals happen then it is possible for the underlying operating system to do a partial write. You will need to write the remaining data.
| Parameters: |
|
|---|
SQLite uses a convoluted method of storing uri parameters after the filename binding the C filename representation and parameters together. This class encapsulates that binding. The example shows usage of this class.
Your VFS.xOpen() method will generally be passed one of these instead of a string as the filename if the URI flag was used or the main database flag is set.
You can safely pass it on to the VFSFile constructor which knows how to get the name back out.
Returns the filename.
Returns the boolean value for parameter name or default if not present.
Calls: sqlite3_uri_boolean
Returns the integer value for parameter name or default if not present.
Calls: sqlite3_uri_int64
Returns the value of parameter name or None.
Calls: sqlite3_uri_parameter
You can download this release as binaries for Windows. Just run the executable corresponding with the Python version you are using. The Windows binaries all include the FTS and RTree extensions. (FTS3_PARENTHESIS is on.)
Download in source form for other platforms or if you want to compile yourself on Windows. See the recommended way to build or all the options available.
Some Linux distributions also have packages which may trail the SQLite and APSW releases by a year, or more. It is also possible to build RPMs and DEB packages from the source, although this involves setting up package management tools and various dependencies on your build machine.
| Debian | Install python-apsw |
| Fedora | Install python-apsw |
| Ubuntu | Install python-apsw |
| Ubuntu PPA | PPA building has been broken for over two years because Canonical/Ubuntu add a broken flag to the PPA. |
| Gentoo | Install dev-python/apsw |
| Arch Linux | Install python-apsw |
Downloads are digitally signed so you can verify they have not been tampered with. Download and extract the zip file of signatures listed above. These instructions are for GNU Privacy Guard. (GPG is installed as standard on most Unix/Linux platforms and can be downloaded for Windows.)
Verify
To verify a file just use –verify specifying the corresponding .asc filename. This example verifies the source:
$ gpg --verify apsw-3.8.11.1-r1.zip.asc gpg: Signature made ... date ... using DSA key ID 0DFBD904 gpg: Good signature from "Roger Binns <rogerb@rogerbinns.com>"If you get a “good signature” then the file has not been tampered with and you are good to go.
Getting the signing key
You may not have the signing key available in which case the last line will be something like this:
gpg: Can't check signature: public key not foundYou can get a copy of the key using this command:
$ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 0DFBD904 gpg: requesting key 0DFBD904 from hkp server keyserver.ubuntu.com gpg: /home/username/.gnupg/trustdb.gpg: trustdb created gpg: key 0DFBD904: public key "Roger Binns <rogerb@rogerbinns.com>" imported gpg: Total number processed: 1 gpg: imported: 1Repeat the verify step.
The source is controlled by Git - start at https://github.com/rogerbinns/apsw
APSW is not available at the Python Package Index (pypi) and hence cannot be installed using easy_install, pip or similar tools. The reason for this is that the tools do not provide a way of providing options to the setup.py included with APSW and hence there is no way for APSW to know if you want SQLite downloaded, a consistent version of SQLite or the latest, to use a system SQLite instead, error if an a system version is not available etc. I could pick a sensible default but everyone else using pypi would be disadvantaged or worse get undesired behaviour (eg different versions of SQLite depending on when a machine did an install). Additionally the world of Python packaging is going through another series of changes (distutils2 aka packaging) so some solution may come out of that.
I’m happy to work with anyone who has a solution to this problem.
Copyright (C) 2004-2012 Roger Binns See src/traceback.c for code by Greg Ewing.
This software is provided ‘as-is’, without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software.
Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions:
Alternatively you may strike the license above and use it under any OSI approved open source license such as those listed at http://opensource.org/licenses/alphabetical
apsw.Error is the base for APSW exceptions.
For exceptions corresponding to SQLite error codes codes this attribute is the numeric error code.
APSW runs with extended result codes turned on. This attribute includes the detailed code.
As an example, if SQLite issued a read request and the system returned less data than expected then result would have the value SQLITE_IOERR while extendedresult would have the value SQLITE_IOERR_SHORT_READ.
The following exceptions happen when APSW detects various problems.
You have used an object concurrently in two threads. For example you may try to use the same cursor in two different threads at the same time, or tried to close the same connection in two threads at the same time.
You can also get this exception by using a cursor as an argument to itself (eg as the input data for Cursor.executemany()). Cursors can only be used for one thing at a time.
See apsw.fork_checker().
You have tried to start a new SQL execute call before executing all the previous ones. See the execution model for more details.
This exception is no longer generated. It was required in earlier releases due to constraints in threading usage with SQLite.
You have called Connection.close() and then continued to use the Connection or associated cursors.
You have called Cursor.close() and then tried to use the cursor.
There are several causes for this exception. When using tuples, an incorrect number of bindings where supplied:
cursor.execute("select ?,?,?", (1,2)) # too few bindings
cursor.execute("select ?,?,?", (1,2,3,4)) # too many bindings
You are using named bindings, but not all bindings are named. You should either use entirely the named style or entirely numeric (unnamed) style:
cursor.execute("select * from foo where x=:name and y=?")
Note
It is not considered an error to have missing keys in a dictionary. For example this is perfectly valid:
cursor.execute("insert into foo values($a,:b,$c)", {'a': 1})
b and c are not in the dict. For missing keys, None/NULL will be used. This is so you don’t have to add lots of spurious values to the supplied dict. If your schema requires every column have a value, then SQLite will generate an error due to some values being None/NULL so that case will be caught.
A statement is complete but you try to run it more anyway!
The execution tracer returned False so execution was aborted.
A call cannot be made to an inherited Virtual File System (VFS) method as the VFS does not implement the method.
The VFS file is closed so the operation cannot be performed.
The following lists which Exception classes correspond to which SQLite error codes.
SQLITE_ERROR. This error is documented as a bad SQL query or missing database, but is also returned for a lot of other situations. It is the default error code unless there is a more specific one.
SQLITE_MISMATCH. Data type mismatch. For example a rowid or integer primary key must be an integer.
SQLITE_NOTFOUND. Returned when various internal items were not found such as requests for non-existent system calls or file controls.
SQLITE_INTERNAL. (No longer used) Internal logic error in SQLite.
SQLITE_PROTOCOL. (No longer used) Database lock protocol error.
SQLITE_MISUSE. SQLite library used incorrectly.
SQLITE_RANGE. (Cannot be generated using APSW). 2nd parameter to sqlite3_bind out of range
SQLITE_PERM. Access permission denied by the operating system, or parts of the database are readonly such as a cursor.
SQLITE_READONLY. Attempt to write to a readonly database.
SQLITE_CANTOPEN. Unable to open the database file.
SQLITE_AUTH. Authorization denied.
SQLITE_ABORT. Callback routine requested an abort.
SQLITE_BUSY. The database file is locked. Use Connection.setbusytimeout() to change how long SQLite waits for the database to be unlocked or Connection.setbusyhandler() to use your own handler.
SQLITE_LOCKED. A table in the database is locked.
SQLITE_INTERRUPT. Operation terminated by sqlite3_interrupt - use Connection.interrupt().
SQLITE_SCHEMA. The database schema changed. A prepared statement becomes invalid if the database schema was changed. Behind the scenes SQLite reprepares the statement. Another or the same Connection may change the schema again before the statement runs. SQLite will attempt up to 5 times before giving up and returning this error.
SQLITE_CONSTRAINT. Abort due to constraint violation. This would happen if the schema required a column to be within a specific range. If you have multiple constraints, you can’t tell which one was the cause.
SQLITE_NOMEM. A memory allocation failed.
SQLITE_IOERR. Some kind of disk I/O error occurred. The extended error code will give more detail.
SQLITE_CORRUPT. The database disk image appears to be a SQLite database but the values inside are inconsistent.
SQLITE_FULL. The disk appears to be full.
SQLITE_TOOBIG. String or BLOB exceeds size limit. You can change the limits using Connection.limit().
SQLITE_NOLFS. SQLite has attempted to use a feature not supported by the operating system such as large file support.
SQLITE_EMPTY. Database is completely empty.
SQLITE_FORMAT. (No longer used) Auxiliary database format error.
SQLITE_NOTADB. File opened that is not a database file. SQLite has a header on database files to verify they are indeed SQLite databases.
When an exception occurs, Python does not include frames from non-Python code (ie the C code called from Python). This can make it more difficult to work out what was going on when an exception occurred for example when there are callbacks to collations, functions or virtual tables, triggers firing etc.
This is an example showing the difference between the tracebacks you would have got with earlier versions of apsw and the augmented traceback:
import apsw
def myfunc(x):
1/0
con=apsw.Connection(":memory:")
con.createscalarfunction("foo", myfunc)
con.createscalarfunction("fam", myfunc)
cursor=con.cursor()
cursor.execute("create table bar(x,y,z);insert into bar values(1,2,3)")
cursor.execute("select foo(1) from bar")
| Original Traceback | Augmented Traceback |
|---|---|
Traceback (most recent call last):
File "t.py", line 11, in <module>
cursor.execute("select foo(1) from bar")
File "t.py", line 4, in myfunc
1/0
ZeroDivisionError: integer division or modulo by zero
|
Traceback (most recent call last):
File "t.py", line 11, in <module>
cursor.execute("select foo(1) from bar")
File "apsw.c", line 3412, in resetcursor
File "apsw.c", line 1597, in user-defined-scalar-foo
File "t.py", line 4, in myfunc
1/0
ZeroDivisionError: integer division or modulo by zero
|
In the original traceback you can’t even see that code in apsw was involved. The augmented traceback shows that there were indeed two function calls within apsw and gives you line numbers should you need to examine the code. Also note how you are told that the call was in user-defined-scalar-foo (ie you can tell which function was called.)
But wait, there is more!!! In order to further aid troubleshooting, the augmented stack traces make additional information available. Each frame in the traceback has local variables defined with more information. You can print out the variables using ASPN recipe 52215
In the recipe, the initial code in print_exc_plus() is far more complicated than need be, and also won’t work correctly with all tracebacks (it depends on f_prev being set which isn’t always the case). Change the function to start like this:
tb = sys.exc_info()[2] stack = [] while tb: stack.append(tb.tb_frame) tb = tb.tb_next traceback.print_exc() print "Locals by frame, innermost last"
Here is a far more complex example from some virtual tables code I was writing. The BestIndex method in my code had returned an incorrect value. The augmented traceback includes local variables using recipe 52215. I can see what was passed in to my method, what I returned and which item was erroneous. The original traceback is almost completely useless.
Original traceback:
Traceback (most recent call last):
File "tests.py", line 1387, in testVtables
cursor.execute(allconstraints)
TypeError: Bad constraint (#2) - it should be one of None, an integer or a tuple of an integer and a boolean
Augmented traceback with local variables:
Traceback (most recent call last):
File "tests.py", line 1387, in testVtables
cursor.execute(allconstraints)
VTable = __main__.VTable
cur = <apsw.Cursor object at 0x988f30>
i = 10
self = testVtables (__main__.APSW)
allconstraints = select rowid,* from foo where rowid>-1000 ....
File "apsw.c", line 4050, in Cursor_execute.sqlite3_prepare
Connection = <apsw.Connection object at 0x978800>
statement = select rowid,* from foo where rowid>-1000 ....
File "apsw.c", line 2681, in VirtualTable.xBestIndex
self = <__main__.VTable instance at 0x98d8c0>
args = (((-1, 4), (0, 32), (1, 8), (2, 4), (3, 64)), ((2, False),))
result = ([4, (3,), [2, False], [1], [0]], 997, u'\xea', False)
File "apsw.c", line 2559, in VirtualTable.xBestIndex.result_constraint
indices = [4, (3,), [2, False], [1], [0]]
self = <__main__.VTable instance at 0x98d8c0>
result = ([4, (3,), [2, False], [1], [0]], 997, u'\xea', False)
constraint = (3,)
TypeError: Bad constraint (#2) - it should be one of None, an integer or a tuple of an integer and a boolean
A cursor encapsulates a SQL query and returning results. To make a new cursor you should call cursor() on your database:
db=apsw.Connection("databasefilename")
cursor=db.cursor()
A cursor executes SQL:
cursor.execute("create table example(title, isbn)")
You can also read data back. The row is returned as a tuple of the column values:
for row in cursor.execute("select * from example"):
print row
There are two ways of supplying data to a query. The really bad way is to compose a string:
sql="insert into example values('%s', %d)" % ("string", 8390823904)
cursor.execute(sql)
If there were any single quotes in string then you would have invalid syntax. Additionally this is how SQL injection attacks happen. Instead you should use bindings:
sql="insert into example values(?, ?)"
cursor.execute(sql, ("string", 8390823904))
# You can also use dictionaries
sql="insert into example values(:title, :isbn)"
cursor.execute(sql, {"title": "string", "isbn": 8390823904})
# You can use local variables as the dictionary
title="..."
isbn="...."
cursor.execute(sql, locals())
Cursors are cheap. Use as many as you need. It is safe to use them across threads, such as calling execute() in one thread, passing the cursor to another thread that then calls Cursor.next(). The only thing you can’t do is call methods at exactly the same time on the same cursor in two different threads - eg trying to call execute() in both at the same time, or execute() in one and Cursor.next() in another. (If you do attempt this, it will be detected and ThreadingViolationError will be raised.)
Behind the scenes a Cursor maps to a SQLite statement. APSW maintains a cache so that the mapping is very fast, and the SQLite objects are reused when possible.
A unique feature of APSW is that your query can be multiple semi-colon separated statements. For example:
cursor.execute("select ... ; insert into ... ; update ... ; select ...")
Note
SQLite fetches data as it is needed. If table example had 10 million rows it would only get the next row as requested (the for loop effectively calls next() to get each row). This code would not work as expected:
for row in cursor.execute("select * from example"):
cursor.execute("insert .....")
The nested execute() would start a new query abandoning any remaining results from the SELECT cursor. There are two ways to work around this. Use a different cursor:
for row in cursor1.execute("select * from example"):
cursor2.execute("insert ...")
You can also get all the rows immediately by filling in a list:
rows=list( cursor.execute("select * from example") )
for row in rows:
cursor.execute("insert ...")
This last approach is recommended since you don’t have to worry about the database changing while doing the select. You should also understand transactions and where to put the transaction boundaries.
Note
Cursors on the same Connection are not isolated from each other. Anything done on one cursor is immediately visible to all other Cursors on the same connection. This still applies if you start transactions. Connections are isolated from each other with cursors on other connections not seeing changes until they are committed.
See also
You obtain cursors by calling Connection.cursor().
It is very unlikely you will need to call this method. It exists because older versions of SQLite required all Connection/Cursor activity to be confined to the same thread. That is no longer the case. Cursors are automatically garbage collected and when there are none left will allow the connection to be garbage collected if it has no other references.
A cursor is open if there are remaining statements to execute (if your query included multiple statements), or if you called executemany() and not all of the sequenceofbindings have been used yet.
| Parameters: | force – If False then you will get exceptions if there is remaining work to do be in the Cursor such as more statements to execute, more data from the executemany binding sequence etc. If force is True then all remaining work and state information will be silently discarded. |
|---|
Based on the DB-API cursor property, this returns the same as getdescription() but with 5 Nones appended. See also APSW issue 131.
Executes the statements using the supplied bindings. Execution returns when the first row is available or all statements have completed.
| Parameters: |
|
|---|
If you use numbered bindings in the query then supply a sequence. Any sequence will work including lists and iterators. For example:
cursor.execute("insert into books values(?,?)", ("title", "number"))
Note
A common gotcha is wanting to insert a single string but not putting it in a tuple:
cursor.execute("insert into books values(?)", "a title")
The string is a sequence of 8 characters and so it will look like you are supplying 8 bindings when only one is needed. Use a one item tuple with a trailing comma like this:
cursor.execute("insert into books values(?)", ("a title",) )
If you used names in the statement then supply a dictionary as the binding. It is ok to be missing entries from the dictionary - None/null will be used. For example:
cursor.execute("insert into books values(:title, :isbn, :rating)",
{"title": "book title", "isbn": 908908908})
The return is the cursor object itself which is also an iterator. This allows you to write:
for row in cursor.execute("select * from books"):
print row
| Raises: |
|
|---|
See also
This method is for when you want to execute the same statements over a sequence of bindings. Conceptually it does this:
for binding in sequenceofbindings:
cursor.execute(statements, binding)
Example:
rows=( (1, 7),
(2, 23),
(4, 92),
(12, 12) )
cursor.executemany("insert into nums values(?,?)", rows)
The return is the cursor itself which acts as an iterator. Your statements can return data. See execute() for more information.
Returns all remaining result rows as a list. This method is defined in DBAPI. It is a longer way of doing list(cursor).
Returns the next row of data or None if there are no more rows.
Returns the Connection this cursor belongs to. An example usage is to get another cursor:
def func(cursor):
# I don't want to alter existing cursor, so make a new one
mycursor=cursor.getconnection().cursor()
mycursor.execute("....")
Returns a tuple describing each column in the result row. The return is identical for every row of the results. You can only call this method once you have started executing a statement and before you have finished:
# This will error
cursor.getdescription()
for row in cursor.execute("select ....."):
# this works
print cursor.getdescription()
print row
The information about each column is a tuple of (column_name, declared_column_type). The type is what was declared in the CREATE TABLE statement - the value returned in the row will be whatever type you put in for that row and column. (This is known as manifest typing which is also the way that Python works. The variable a could contain an integer, and then you could put a string in it. Other static languages such as C or other SQL databases only let you put one type in - eg a could only contain an integer or a string, but never both.)
Example:
cursor.execute("create table books(title string, isbn number, wibbly wobbly zebra)")
cursor.execute("insert into books values(?,?,?)", (97, "fjfjfj", 3.7))
cursor.execute("insert into books values(?,?,?)", ("fjfjfj", 3.7, 97))
for row in cursor.execute("select * from books"):
print cursor.getdescription()
print row
Output:
# row 0 - description
(('title', 'string'), ('isbn', 'number'), ('wibbly', 'wobbly zebra'))
# row 0 - values
(97, 'fjfjfj', 3.7)
# row 1 - description
(('title', 'string'), ('isbn', 'number'), ('wibbly', 'wobbly zebra'))
# row 1 - values
('fjfjfj', 3.7, 97)
Returns the currently installed (via setexectrace()) execution tracer.
See also
Returns the currently installed (via setrowtrace()) row tracer.
See also
callable is called with the cursor, statement and bindings for each execute() or executemany() on this cursor.
If callable is None then any existing execution tracer is removed.
callable is called with cursor and row being returned. You can change the data that is returned or cause the row to be skipped altogether.
If callable is None then any existing row tracer is removed.
See also
A backup object encapsulates copying one database to another. You call Connection.backup() on the destination database to get the backup object. Call step() to copy some pages repeatedly dealing with errors as appropriate. Finally finish() cleans up committing or rolling back and releasing locks.
Here is an example usage using the with statement to ensure finish() is called:
# copies source.main into db
with db.backup("main", source, "main") as b:
while not b.done:
b.step(100)
print b.remaining, b.pagecount, "\r",
If you are not using with then you’ll need to ensure finish() is called:
# copies source.main into db
b=db.backup("main", source, "main")
try:
while not b.done:
b.step(100)
print b.remaining, b.pagecount, "\r",
finally:
b.finish()
The database is copied page by page. This means that there is not a round trip via SQL. All pages are copied including free ones.
The destination database is locked during the copy. You will get a ThreadingViolationError if you attempt to use it.
You create a backup instance by calling Connection.backup().
You can use the backup object as a context manager as defined in PEP 0343. The __exit__() method ensures that backup is finished.
Implements context manager in conjunction with __enter__() ensuring that the copy is finished.
Does the same thing as finish(). This extra api is provided to give the same api as other APSW objects such as Connection.close(), blob.close() and Cursor.close(). It is safe to call this method multiple times.
| Parameters: | force – If true then any exceptions are ignored. |
|---|
Completes the copy process. If all pages have been copied then the transaction is committed on the destination database, otherwise it is rolled back. This method must be called for your backup to take effect. The backup object will always be finished even if there is an exception. It is safe to call this method multiple times.
Calls: sqlite3_backup_finish
Read only. How many pages were in the source database after the last step. If you haven’t called step() or the backup object has been finished then zero is returned.
Calls: sqlite3_backup_pagecount
Read only. How many pages were remaining to be copied after the last step. If you haven’t called step() or the backup object has been finished then zero is returned.
Calls: sqlite3_backup_remaining
Copies npages pages from the source to destination database. The source database is locked during the copy so using smaller values allows other access to the source database. The destination database is always locked until the backup object is finished.
| Parameters: | npages – How many pages to copy. If the parameter is omitted or negative then all remaining pages are copied. The default page size is 1024 bytes (1kb) which can be changed before database creation using a pragma. |
|---|
This method may throw a BusyError or LockedError if unable to lock the source database. You can catch those and try again.
| Returns: | True if this copied the last remaining outstanding pages, else false. This is the same value as done |
|---|
Calls: sqlite3_backup_step
pysqlite and APSW approached the problem of providing access to SQLite from Python from fundamentally different directions.
APSW only wraps version 3 of SQLite and provides access in whatever way is normal for SQLite. It makes no effort to hide how SQLite is different from other databases.
pysqlite tries to provide a DBAPI compliant wrapper for SQLite and in doing so needs to make it have the same behaviour as other databases. Consequently it does hide some of SQLite’s nuances.
Note
I suggest using APSW when you want to directly use SQLite and its functionality or are using your own code to deal with database independence rather than DBAPI. Use pysqlite and DBAPI if your needs are simple, and you don’t want to use SQLite features.
APSW has the following enhancements/differences over pysqlite 2 (wrapping SQLite 3):
APSW stays up to date with SQLite. As features are added and functionality changed in SQLite, APSW tracks them.
APSW gives all functionality of SQLite including virtual tables, Virtual File System (VFS), BLOB I/O, backups and file control.
You can use the same Connection across threads with APSW without needing any additional level of locking. pysqlite requires that the Connection and any cursors are used in the same thread. You can disable its checking, but unless you are very careful with your own mutexes you will have a crash or a deadlock.
APSW is a single file for the extension, apsw.pyd on Windows and apsw.so on Unix/Mac (Note PEP 3149). There are no other files needed and the build instructions show you how to include SQLite statically in this file. You can put this file anywhere your Python session can reach. pysqlite is one binary file and several .py files, all of which need to be available.
Nothing happens behind your back. By default pysqlite tries to manage transactions by parsing your SQL for you, but you can turn it off. This can result in very unexpected behaviour with pysqlite.
When using a Connection as a context manager APSW uses SQLite’s ability to have nested transactions. pysqlite only deals with one transaction at a time and cannot nest them. (Savepoints were introduced in SQLite 3.6.8 - another illustration of the benefits of keeping up to date with SQLite.)
APSW always handles Unicode correctly (this was one of the major reasons for writing it in the first place). pysqlite has since fixed many of its issues but you are still stuck with some.
You can use semi-colons at the end of commands and you can have multiple commands in the execute string in APSW. There are no restrictions on the type of commands used. For example this will work fine in APSW but is not allowed in pysqlite:
import apsw
con=apsw.Connection(":memory:")
cur=con.cursor()
for row in cur.execute("create table foo(x,y,z);insert into foo values (?,?,?);"
"insert into foo values(?,?,?);select * from foo;drop table foo;"
"create table bar(x,y);insert into bar values(?,?);"
"insert into bar values(?,?);select * from bar;",
(1,2,3,4,5,6,7,8,9,10)):
print row
And the output as you would expect:
(1, 2, 3)
(4, 5, 6)
(7, 8)
(9, 10)
Cursor.executemany() also works with statements that return data such as selects, and you can have multiple statements. pysqlite’s executescript() method doesn’t allow any form of data being returned (it silently ignores any returned data).
pysqlite swallows exceptions in your callbacks making it far harder to debug problems. That also prevents you from raising exceptions in your callbacks to be handled in your code that called SQLite. pysqlite does let you turn on printing of tracebacks, but that is a poor substitute. apsw does the right thing as demonstrated by this example.
Source:
def badfunc(t):
return 1/0
# pysqlite
from pysqlite2 import dbapi2 as sqlite
con = sqlite.connect(":memory:")
con.create_function("badfunc", 1, badfunc)
cur = con.cursor()
cur.execute("select badfunc(3)")
# apsw
import apsw
con = apsw.Connection(":memory:")
con.createscalarfunction("badfunc", badfunc, 1)
cur = con.cursor()
cur.execute("select badfunc(3)")
Exceptions:
# pysqlite
Traceback (most recent call last):
File "func.py", line 8, in ?
cur.execute("select badfunc(3)")
pysqlite2.dbapi2.OperationalError: user-defined function raised exception
# apsw
Traceback (most recent call last):
File "t.py", line 8, in ?
cur.execute("select badfunc(3)")
File "apsw.c", line 3660, in resetcursor
File "apsw.c", line 1871, in user-defined-scalar-badfunc
File "t.py", line 3, in badfunc
return 1/0
APSW has significantly enhanced debuggability. More details are available than just what is printed out when exceptions happen like above. See augmented stack traces
APSW has execution and row tracers. pysqlite has no equivalent to execution tracers and does have data adaptors which aren’t the same thing as a row tracer (for example you can’t skip rows or add a new column to each row returned). pysqlite does have a row factory but you can easily emulate that with the row tracer and Cursor.getdescription().
APSW has an apswtrace utility script that traces execution and results in your code without having to modify it in any way. It also outputs summary reports making it easy to see what your most time consuming queries are, which are most popular etc.
APSW has an exception corresponding to each SQLite error code and provides the extended error code. pysqlite combines several SQLite error codes into corresponding DBAPI exceptions. This is a good example of the difference in approach of the two wrappers.
The APSW test suite is larger and tests more functionality. Code coverage by the test suite is 99.6%. pysqlite is good at 81% for C code although there are several places that coverage can be improved. I haven’t measured code coverage for pysqlite’s Python code. The consequences of this are that APSW catches issues earlier and gives far better diagnostics. As an example try returning an unsupported type from a registered scalar function.
APSW is faster than pysqlite in my testing. Try the speedtest benchmark.
pysqlite has an adaptor system that lets you pretend SQLite stores and returns more types than it really supports. Note that the database won’t be useful in a non-pysqlite context (eg PHP code looking at the same database isn’t going to recognise your Point class). You can implement something similar in APSW by intercepting Cursor.execute() calls that suitably mangles the bindings going to SQLite and does something similar to the rows the iterator returns.
pysqlite lets you work with a database that contains invalid Unicode data by setting a text factory that deals with the text data.
APSW does not let you put non-Unicode data into the database in the first place and it will be considered invalid by other tools reading the data (eg Java, PHP). If you somehow do manage to get non-Unicode data as a SQLite string, you can cast it to a blob:
for row in cursor.execute("select CAST(column as BLOB) from table"):
# row[0] is buffer (py2) or bytes (py3) here
deal_with_binary_data(row[0])