pax_global_header 0000666 0000000 0000000 00000000064 11632421554 0014515 g ustar 00root root 0000000 0000000 52 comment=57c982a8e5e8950a695fbc83ba78dba842652df0
pyxnat-0.9.0~dev0/ 0000775 0000000 0000000 00000000000 11632421554 0014023 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/.gitignore 0000775 0000000 0000000 00000000117 11632421554 0016015 0 ustar 00root root 0000000 0000000 *.pyc
MANIFEST
pyxnat.egg-info/
build/
dist/
*.DS_Store*
.project
.pydevproject pyxnat-0.9.0~dev0/CHANGES.rst 0000664 0000000 0000000 00000004631 11632421554 0015631 0 ustar 00root root 0000000 0000000 Latest changes
===============
Release 0.9.0
----------------
* New features:
- Global scan listing function: interface.select.scans()
- Support for XNAT configuration file format
- Prearchive services
- Batch function for downloading all files related to a scan or an assessor
- Create element with an XML document
- New xpath function for EObjects
* Improvements:
- Catching authencation errors
- Toggle option for cache warnings
- Description for search templates is displayed
Release 0.8.0
----------------
* Compatible with XNAT 1.5
* New features:
- provenance annotation on assessors and recontructions
- search templates
- callback system to monitor data streams to and from the server
* Improvements:
- support for proxies in the Interface object
- a description can be added when a search is saved on the server
- python strings can be uploaded and saved just like files
* Bug fixes including:
- improved unicode support for uploaded files
- solved cache issue on Windows
- a major bug in the Collection.where method
Release 0.7.0
----------------
* Errors following the PEP-249
* Some operations follow the PEP-249 - e.g. `fetchall` replaces `get`
* New inspection functions:
- experiement_types
- assessor_types
- scan_types
- reconstruction_types
- project_values
- subject_values
- experiment_values
- assessor_values
- scan_values
- reconstruction_values
* Inspect method `fieldvalues` changed to `field_values`
* `Interface` Object now supports config files.
* Bug fix regarding the file names in the cache. It means that cached data
from older versions has to be re-downloaded.
* The disk check for available space is performed against a timer instead
of always.
* The default `get` function to download file now supports custom paths.
* Bug fix for HTTP sessions management.
* New `last_modified` method for project to get subjects last modified
date.
* Resource elements are now fully configurable at creation.
* Added support for XNAT pipelines.
* Added push and pull zip files at the resource level.
* Added simple schema parsing capabilities.
* Add a global management interface to gather different managers.
* Interface now follows redirections on the server url.
pyxnat-0.9.0~dev0/MANIFEST.in 0000664 0000000 0000000 00000000135 11632421554 0015560 0 ustar 00root root 0000000 0000000 include *.txt *.py
recursive-include pyxnat *.py *.txt *.csv
graft doc
prune pyxnat/examples
pyxnat-0.9.0~dev0/README.rst 0000664 0000000 0000000 00000007567 11632421554 0015531 0 ustar 00root root 0000000 0000000 The homepage of pyxnat with user documentation is located on:
http://packages.python.org/pyxnat/
Getting the latest code
=========================
To get the latest code using git, simply type::
git clone git://github.com/pyxnat/pyxnat.git
If you don't have git installed, you can download a zip or tarball
of the latest code: http://github.com/pyxnat/pyxnat/archives/master
Or the lastest stable version: http://pypi.python.org/pypi/pyxnat
Installing
=========================
As any Python packages, to install pyxnat, simply do::
python setup.py install
in the source code directory.
Workflow to contribute
=========================
To contribute to pyxnat, first create an account on `github
`_. Once this is done, fork the `pyxnat repository
`_ to have you own repository,
clone it using 'git clone' on the computers where you want to work. Make
your changes in your clone, push them to your github account, test them
on several computer, and when you are happy with them, send a pull
request to the main repository.
Running the test suite
=========================
To run the test suite, you need nosetests and the coverage modules.
Run the test suite using::
nosetests
from the root of the project.
Building the docs
=========================
To build the docs you need to have setuptools and sphinx (>=0.5) installed.
Run the command::
python setup.py build_sphinx
The docs are built in the build/sphinx/html directory.
Making a source tarball
=========================
To create a source tarball, eg for packaging or distributing, run the
following command::
python setup.py sdist
The tarball will be created in the `dist` directory. This command will
compile the docs, and the resulting tarball can be installed with
no extra dependencies than the Python standard library. You will need
setuptool and sphinx.
Making a release and uploading it to PyPI
==================================================
This command is only run by project manager, to make a release, and
upload in to PyPI::
python setup.py sdist bdist_egg register upload
Licensing
----------
pyxnat is **BSD-licenced** (3 clause):
This software is OSI Certified Open Source Software.
OSI Certified is a certification mark of the Open Source Initiative.
Copyright (c) 2010-2011, Yannick Schwartz
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of Yannick Schwartz. nor the names of other pyxnat
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
**This software is provided by the copyright holders and contributors
"as is" and any express or implied warranties, including, but not
limited to, the implied warranties of merchantability and fitness for
a particular purpose are disclaimed. In no event shall the copyright
owner or contributors be liable for any direct, indirect, incidental,
special, exemplary, or consequential damages (including, but not
limited to, procurement of substitute goods or services; loss of use,
data, or profits; or business interruption) however caused and on any
theory of liability, whether in contract, strict liability, or tort
(including negligence or otherwise) arising in any way out of the use
of this software, even if advised of the possibility of such
damage.**
pyxnat-0.9.0~dev0/TODO.rst 0000664 0000000 0000000 00000001152 11632421554 0015321 0 ustar 00root root 0000000 0000000
* Add some in-code comments on the inner workings of the mod ule
* Support current reserved keywords as resource identifiers
* Improve get file
* Add support for subfolders in files
- provenance
- nipype and cff integration
* Add/update tests for:
- provenance
- templates
* Seems to be a bug in select('/project/PROJ//files') syntax
* bug in provenance? --> bug with the allowDataDeletion flag
* BUG: have to go to the dropdown menu of custom variables before being
able to add custom variables
* BUG in manage.schemas.add?
* check ex2rst in pymvpa.tools to generate sphinx from example scripts
pyxnat-0.9.0~dev0/debian/ 0000775 0000000 0000000 00000000000 11632421554 0015245 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/debian/changelog 0000664 0000000 0000000 00000000245 11632421554 0017120 0 ustar 00root root 0000000 0000000 pyxnat (0.9.0~dev0-1) unstable; urgency=low
* Initial release (Closes: #609820).
-- Yannick Schwartz Thu, 08 Sep 2011 15:31:48 -0400
pyxnat-0.9.0~dev0/debian/compat 0000664 0000000 0000000 00000000002 11632421554 0016443 0 ustar 00root root 0000000 0000000 7
pyxnat-0.9.0~dev0/debian/control 0000664 0000000 0000000 00000002421 11632421554 0016647 0 ustar 00root root 0000000 0000000 Source: pyxnat
Section: python
Priority: extra
Maintainer: Yannick Schwartz
Uploaders: Yaroslav Halchenko , Michael Hanke
Build-Depends: debhelper (>= 7.2.18),
python-all,
python-lxml,
python-httplib2 (>= 0.7.0),
python-simplejson,
python-nose
Standards-Version: 3.9.2
Homepage: http://packages.python.org/pyxnat/
Vcs-Git: git://github.com/pyxnat/pyxnat.git
Vcs-Browser: https://github.com/pyxnat/pyxnat.git
Package: python-pyxnat
Architecture: all
Depends: ${python:Depends}, ${misc:Depends},
python-lxml,
python-simplejson,
python-httplib2 (>= 0.7.0),
Recommends: python-networkx,
python-matplotlib
Provides: ${python:Provides}
Description: Interface to access neuroimaging data on XNAT servers
pyxnat is a simple Python library that relies on the REST API provided
by the XNAT platform since its 1.4 version. XNAT is an extensible
database for neuroimaging data. The main objective is to ease
communications with an XNAT server to plug-in external tools or Python
scripts to process the data. It features:
.
- resources browsing capabilities
- read and write access to resources
- complex searches
- disk-caching of requested files and resources
pyxnat-0.9.0~dev0/debian/copyright 0000664 0000000 0000000 00000006050 11632421554 0017201 0 ustar 00root root 0000000 0000000 Format-Specification: http://svn.debian.org/wsvn/dep/web/deps/dep5.mdwn?op=file&rev=173
Upstream-Name: pyxnat
Upstream-Contact: Yannick Schwartz
Source: http://github.com/pyxnat/pyxnat
Files: *
Copyright: 2010-2011, Yannick Schwartz
License: BSD-3
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
* Neither the name of the Yannick Schwartz nor the names of any
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
.
This software is provided by the copyright holders and contributors
"as is" and any express or implied warranties, including, but not
limited to, the implied warranties of merchantability and fitness
for a particular purpose are disclaimed. In no event shall the
copyright owner or contributors be liable for any direct, indirect,
incidental, special, exemplary, or consequential damages (including,
but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however
caused and on any theory of liability, whether in contract, strict
liability, or tort (including negligence or otherwise) arising in
any way out of the use of this software, even if advised of the
possibility of such damage.
Files: doc/sphinxext/*
Copyright: 2008, Stefan van der Walt
2008, Pauli Virtanen
License: BSD-2
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
.
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
pyxnat-0.9.0~dev0/debian/gbp.conf 0000664 0000000 0000000 00000000363 11632421554 0016666 0 ustar 00root root 0000000 0000000 [DEFAULT]
# the default branch for upstream sources:
upstream-branch = master
# the default branch for the debian patch:
debian-branch = debian
# the default tag formats used:
upstream-tag = release/%(version)s
debian-tag = debian/%(version)s
pyxnat-0.9.0~dev0/debian/pyversions 0000664 0000000 0000000 00000000005 11632421554 0017404 0 ustar 00root root 0000000 0000000 2.6-
pyxnat-0.9.0~dev0/debian/rules 0000775 0000000 0000000 00000001463 11632421554 0016331 0 ustar 00root root 0000000 0000000 #!/usr/bin/make -f
# -*- makefile -*-
PACKAGE_NAME = python-pyxnat
PACKAGE_ROOT_DIR = debian/${PACKAGE_NAME}
INSTALL_PATH = $(CURDIR)/debian/${PACKAGE_NAME}
# default Python
PYTHON=$(shell pyversions -d)
%:
dh $@
override_dh_auto_test:
: # Do not test just after build, lets install and then test
override_dh_auto_install:
dh_auto_install
# All tests later on
# cd build to prevent use of local/not-built source tree
ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
cd build; \
for PYTHON in $(shell pyversions -r); do \
echo "I: Running pyxnat unittests using $$PYTHON"; \
PYTHONPATH=$$(/bin/ls -d $(INSTALL_PATH)/usr/lib/$$PYTHON/*-packages):$$PYTHONPATH \
$$PYTHON /usr/bin/nosetests --exclude=test_del_provenance pyxnat; \
done
endif
override_dh_auto_clean:
dh_auto_clean
-rm -rf doc/build build
pyxnat-0.9.0~dev0/debian/source/ 0000775 0000000 0000000 00000000000 11632421554 0016545 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/debian/source/format 0000664 0000000 0000000 00000000014 11632421554 0017753 0 ustar 00root root 0000000 0000000 3.0 (quilt)
pyxnat-0.9.0~dev0/debian/watch 0000664 0000000 0000000 00000000215 11632421554 0016274 0 ustar 00root root 0000000 0000000 version=3
opts="filenamemangle=s/.*\/(.*)/pyxnat-$1\.tar\.gz/" \
http://github.com/pyxnat/pyxnat/downloads .*tarball/release/([\d\.a-z]+)
pyxnat-0.9.0~dev0/doc/ 0000775 0000000 0000000 00000000000 11632421554 0014570 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/doc/CHANGES.rst 0000777 0000000 0000000 00000000000 11632421554 0020402 2../CHANGES.rst ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/doc/__init__.py 0000664 0000000 0000000 00000000136 11632421554 0016701 0 ustar 00root root 0000000 0000000 """
This is a phony __init__.py file, so that nose finds the doctests in this
directory.
"""
pyxnat-0.9.0~dev0/doc/advanced_tutorial.rst 0000664 0000000 0000000 00000022074 11632421554 0021017 0 ustar 00root root 0000000 0000000 ==============================
Advanced Tutorial
==============================
.. currentmodule:: pyxnat
This advanced tutorial is not much more complicated than the starters
one. It just goes over parts of the API that may be less used (not
that they are less useful!) and that are more likely to change in
future releases.
Introspection
-------------
In order to browse a database people have to be aware of:
- the REST hierarchy
- schema types and fields
- values of fields and resources within a project
The idea of this interface is to help users find their way around a
XNAT server by making it easier to gather the preceding information.
Searchable datatypes and fields
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> # simple datatypes listing
>>> central.inspect.datatypes()
[..., 'xnat:subjectData', 'xnat:projectData', 'xnat:mrSessionData', ...]
>>> # datatypes listing with filter
>>> central.inspect.datatypes('cnda:*')
['cnda:manualVolumetryData',
'cnda:clinicalAssessmentData',
'cnda:psychometricsData',
'cnda:dtiData',
'cnda:atlasScalingFactorData',
'cnda:segmentationFastData',
'cnda:modifiedScheltensData']
>>> # simple fields listing
>>> central.inspect.datatypes('xnat:subjectData')
['xnat:subjectData/SUBJECT_ID',
'xnat:subjectData/INSERT_DATE',
'xnat:subjectData/INSERT_USER',
'xnat:subjectData/GENDER_TEXT',
...]
>>> # field listing with filter
>>> central.inspect.datatypes('xnat:subjectData', '*ID*')
['xnat:subjectData/SUBJECT_ID', 'xnat:subjectData/ADD_IDS']
>>> # field listing on multiple types
>>> central.inspect.datatypes('cnda:*', 'EXPT_ID')
['cnda:manualVolumetryData/EXPT_ID',
'cnda:clinicalAssessmentData/EXPT_ID',
'cnda:psychometricsData/EXPT_ID',
'cnda:dtiData/EXPT_ID',
'cnda:atlasScalingFactorData/EXPT_ID',
'cnda:segmentationFastData/EXPT_ID',
'cnda:modifiedScheltensData/EXPT_ID']
To known what values fields can take in the database::
>>> central.inspect.field_values('xnat:mrSessionData/SESSION_ID')
REST hierarchy
~~~~~~~~~~~~~~
pyxnat does not support all the REST resources. The reasons for this
is that, some of these resources are still experimental, or do not
work exactly the same way which would make it difficult to provide a
consistent interface at the Python level. However support for these
exotic resources will increase in future releases. A good way to know
what is the supported REST hierarchy is to use the following method::
>>> central.inspect.architecture() # rest_hierarchy() still works
- PROJECTS
+ SUBJECTS
+ EXPERIMENTS
+ ASSESSORS
+ RESOURCES
+ FILES
+ IN_RESOURCES
+ FILES
+ OUT_RESOURCES
+ FILES
+ RECONSTRUCTIONS
+ IN_RESOURCES
+ FILES
+ OUT_RESOURCES
+ FILES
+ SCANS
+ RESOURCES
+ FILES
+ RESOURCES
+ FILES
+ RESOURCES
+ FILES
+ RESOURCES
+ FILES
Naming conventions
~~~~~~~~~~~~~~~~~~
Administrators usually use a consistent vocabulary across single
projects, that maps to XNAT datatypes. A new feature in introduced in
0.6 and improved in 0.7 is to be able to define a mapping so that
specific name patterns can be used to cast a resource when creating a
new one.
For example with the following mapping::
'/projects/my_project/subjects/*/experiments/SessionA_*':'xnat:mrSessionData'
Creating an experiment in ``my_project`` that matches `Session_*`, creates an `xnat:mrSessionData`::
>>> central.select('/projects/my_project/subjects/*/experiments/SessionA_new').create()
In the 0.7, it is no longer up to the user to manually save and load
the mapping file. Files are created automatically and the mappings are
discovered `on the fly` when queries are issued on the server. Files
are loaded at the ``Interface`` creation and the mappings are updated
regularly. This functionality can be configured with the following
method::
>>> # activate (default)
>>> central.inspect.set_autolearn('True')
>>> # setup update frequency
>>> central.inspect.set_autolearn(tick=10)
When a mapping is available, re-running the ``rest_hierarchy`` method will display additional information such as::
- PROJECTS
+ SUBJECTS
+ EXPERIMENTS
-----------
- xnat:mrSessionData
- xnat:petSessionData
+ASSESSORS
....
There are additional methods to visualize and display the mappings::
>>> central.inspect.experiment_types()
>>> central.inspect.assessor_types()
>>> central.inspect.scan_types()
>>> central.inspect.reconstruction_types()
Methods also allow to have a quick look on the values at those levels
on the database::
>>> central.inspect.experiment_values('xnat:mrSessionData')
>>> central.inspect.assessor_values('xnat:mrSessionData')
>>> central.inspect.scan_values('xnat:mrSessionData')
>>> central.inspect.reconstruction_values('xnat:mrSessionData')
For more details check the reference documentation.
.. note::
With ``networkx`` and ``matplotlib`` installed, a ``draw``
subinterface will be made available to display some data from the
inspect subinterface as a graph::
>>> central.draw.experiments()
>>> central.draw.assessors()
>>> central.draw.scans()
>>> central.draw.reconstructions()
>>> central.draw.architecture()
>>> central.draw.field_values()
Sharing
-------
It is possible to share ``Subjects``, ``Experiments`` and
``Assessors`` via the REST API. The methods to control sharing are::
>>> subject = interface.select('/project/project1/subject/subject1')
>>> subject.share('project2')
>>> subject.unshare('project2')
>>> # to know to in which projects a subject is available
>>> subject.shares()
Almost the same interface is available for collection objects::
>>> subjects = interface.select('/project/project1/subjects')
>>> subjects.share('project2')
>>> subjects.unshare('project2')
>>> # to retrieve the subjects sharing a list of projects
>>> subjects.sharing(['project1', 'project2'])
.. note::
Of course the permissions policies (user level and project
accessibility)still apply.
.. warning::
The ``shares`` and ``sharing`` methods are not implemented in an
efficient way at the moment. There is another more concerning
issue: subjects for example are accessible through their ID or
label. But labels stop working when trying to access a subject
through a project that is not its orginial one.
Search templates
---------------
PyXNAT is also able to define templates to use with XNAT search engine.
They work basically the same way as usual searches but instead of defining
values to filter the data, one need to define keywords to replace them later
with the actual values::
>>> contraints = [('xnat:subjectData/SUBJECT_ID','LIKE','subject_id'),
('xnat:subjectData/PROJECT', '=', 'project_id'),
'OR',
[('xnat:subjectData/AGE','>','age'),
'AND'
]
]
>>> columns = ['xnat:subjectData/PROJECT', 'xnat:subjectData/SUBJECT_ID']
>>> interface.manage.search.save_template('name',
'xnat:subjectData',
columns,
criteria,
sharing='public',
description='my first template'
)
>>> interface.manage.search.use_template('name',
{'subject_id':'%',
'project_id':'my_project',
'age':'42'
}
)
>>> interface.select(...).where(template=('name',
{'subject_id':'%',
'project_id':'my_project',
'age':'42'}
)
)
And now it is also possible to re-use saved searches in the where clause in the
same way as the templates. It means that you re-use the contraints but not the
data selection which still changes:
>>> interface.select(...).where(query='saved_name')
Provenance definition
--------------------
PyXNAT 0.8 introduces a way to store provenance i.e. to describe the steps
that were performed on an initial data to produce this one. Reconstructions
and assessors only can be annotated with provenace information:
>>> prov = {'program':'young',
'timestamp':'2011-03-01T12:01:01.897987',
'user':'angus',
'machine':'war',
'platform':'linux',
}
>>> element.provenance.attach(prov)
>>> element.provenance.get()
>>> element.dettach()
The provenance attach method adds new steps with each call, unless the overwrite
parameter is set to True. The following keywords for the provenance dictionnay are available:
- program
- program_version
- program_arguments
- timestamp
- cvs
- user
- machine
- platform
- platform_version
- compiler
- compiler_version
pyxnat-0.9.0~dev0/doc/conf.py 0000664 0000000 0000000 00000015000 11632421554 0016063 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
#
# pyxnat documentation build configuration file, created by
# sphinx-quickstart on Tue Nov 24 11:04:02 2009.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# The contents of this file are pickled, so don't put values in the namespace
# that aren't pickleable (module imports are okay, they're removed automatically).
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys, os
import pyxnat
sys.path.append(os.path.abspath('./sphinxext'))
# If your extensions (or modules documented by autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.append(os.path.abspath('.'))
# General configuration
# ---------------------
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [ 'sphinx.ext.autodoc',
'sphinx.ext.pngmath',
'numpydoc',
'phantom_import',
'autosummary',
'sphinx.ext.coverage'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'pyxnat'
copyright = u'2010, Yannick Schwartz'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = pyxnat.__version__[:3]
# The full version, including alpha/beta/rc tags.
release = pyxnat.__version__
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of documents that shouldn't be included in the build.
#unused_docs = []
# List of directories, relative to source directory, that shouldn't be searched
# for source files.
exclude_trees = ['_build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# Options for HTML output
# -----------------------
# The style sheet to use for HTML and HTML Help pages. A file of that name
# must exist either in Sphinx' static/ path, or in one of the custom paths
# given in html_static_path.
# html_style = 'default.css'
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
html_theme = 'nature'
# The name for this set of Sphinx documents. If None, it defaults to
# " v documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_use_modindex = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, the reST sources are included in the HTML build as _sources/.
#html_copy_source = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = ''
# Output file base name for HTML help builder.
htmlhelp_basename = 'pyxnatdoc'
# Options for LaTeX output
# ------------------------
# The paper size ('letter' or 'a4').
#latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
#latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, document class [howto/manual]).
latex_documents = [
('index', 'pyxnat.tex', ur'pyxnat Documentation',
ur'Yannick Schwartz', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_use_modindex = True
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'http://docs.python.org/dev': None}
pyxnat-0.9.0~dev0/doc/generated/ 0000775 0000000 0000000 00000000000 11632421554 0016526 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/doc/generated/pyxnat.Inspector.rst 0000664 0000000 0000000 00000000127 11632421554 0022550 0 ustar 00root root 0000000 0000000 pyxnat.Inspector
================
.. currentmodule:: pyxnat
.. autoclass:: Inspector
pyxnat-0.9.0~dev0/doc/generated/pyxnat.Interface.rst 0000664 0000000 0000000 00000000127 11632421554 0022502 0 ustar 00root root 0000000 0000000 pyxnat.Interface
================
.. currentmodule:: pyxnat
.. autoclass:: Interface
pyxnat-0.9.0~dev0/doc/generated/pyxnat.SearchManager.rst 0000664 0000000 0000000 00000000143 11632421554 0023300 0 ustar 00root root 0000000 0000000 pyxnat.SearchManager
====================
.. currentmodule:: pyxnat
.. autoclass:: SearchManager
pyxnat-0.9.0~dev0/doc/generated/pyxnat.Select.rst 0000664 0000000 0000000 00000000116 11632421554 0022017 0 ustar 00root root 0000000 0000000 pyxnat.Select
=============
.. currentmodule:: pyxnat
.. autoclass:: Select
pyxnat-0.9.0~dev0/doc/generated/pyxnat.Users.rst 0000664 0000000 0000000 00000000113 11632421554 0021676 0 ustar 00root root 0000000 0000000 pyxnat.Users
============
.. currentmodule:: pyxnat
.. autoclass:: Users
pyxnat-0.9.0~dev0/doc/index.rst 0000664 0000000 0000000 00000001322 11632421554 0016427 0 ustar 00root root 0000000 0000000 .. pyxnat documentation master file, created by sphinx-quickstart on Tue Nov 24 11:04:02 2009.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
pyxnat: Xnat in Python
============================================
.. topic:: **User guide contents:**
.. toctree::
:maxdepth: 1
installing.rst
under_the_hood.rst
starters_tutorial.rst
advanced_tutorial.rst
reference_documentation.rst
CHANGES.rst
Introduction
------------
.. automodule:: pyxnat
Module contents
----------------
.. currentmodule :: pyxnat
.. autosummary::
:toctree: generated/
Interface
Select
SearchManager
CacheManager
Users
pyxnat-0.9.0~dev0/doc/installing.rst 0000664 0000000 0000000 00000010073 11632421554 0017467 0 ustar 00root root 0000000 0000000 Installing pyxnat
===================
Platforms
---------
OS Independent.
It was tested under:
- Ubuntu 10.04 with python 2.6
- Ubuntu 8.04 with python 2.5
- Mandriva 2008 with python 2.5
- Windows XP with python 2.6
- Mac OS X 10.5 Leopard with python 2.5
Prerequisites
-------------
- *python* v2.5+
- *python-lxml* v1.3.6+ recommanded, earlier versions may work:
All Python dependencies but lxml are shipped with this package. Lxml may be
installed using standard Python distribution tools, here are the recommanded
ones. For more details on installing procedures please see the sections
following this one.
#. easy_install: lxml may be installed via easy_install under
Unix and Windows
#. Windows only: a .exe installer is provided
#. Mac OS X: a macport of lxml is available, see lxml site for further information.
Alternatively, you can compile the package from sources by following
the instructions. If the recommanded command on the site fails::
python setup.py build --static-deps
Try forcing an older version of libxml2::
python setup.py build --static-deps --libxml2-version=2.7.3
Do not forget to add lxml to your PYTHONPATH afterwards.
- *python-nose* v0.10+ to run the unit tests
- *networkx* and *matplotlib* are not mandatory:
These libraries are used to make some graphical representations.
The `easy_install` way
-----------------------
The easiest way to install pyxnat is (provided that you have `setuptools`
installed) to run::
easy_install pyxnat
You may need to run the above command as administrator
On a unix environment, it is better to install outside of the hierarchy
managed by the system:
easy_install --prefix /usr/local pyxnat
.. warning::
Packages installed via `easy_install` override the Python module look
up mechanism and thus can confused people not familiar with
setuptools. Although it may seem harder, we suggest that you use the
manual way, as described in the following paragraph.
The manual way
---------------
To install pyxnat first download the latest tarball (follow the link on
the bottom of http://pypi.python.org/pypi/pyxnat) and expand it.
Installing in a local environment
..................................
If you don't need to install for all users, we strongly suggest that you
create a local environment and install `pyxnat` in it. One of the pros of
this method is that you never have to become administrator, and thus all
the changes are local to your account and easy to clean up.
#. First, create the following directory (where `~` is your home
directory, or any directory that you want to use as a base for
your local Python environment, and `X` is your Python version
number, e.g. `2.6`)::
~/usr/lib/pythonX/site-packages
#. Second, make sure that you add this directory in your environment
variable `PYTHONPATH`. Under window you can do this by editing
your environment variables in the system parameters dialog. Under
Unix you can add the following line to your `.bashrc` or any file
source at login::
export PYTHONPATH=$HOME/usr/lib/python2.6/site-packages:$PYTHONPATH
#. In the directory created by expanding the `pyxnat` tarball, run the
following command::
python setup.py install --prefix ~/usr
You should not be required to become administrator, if you have
write access to the directory you are installing to.
Installing for all users
........................
If you have administrator rights and want to install for all users, all
you need to do is to go in directory created by expanding the `pyxnat`
tarball and run the following line::
python setup.py install
If you are under Unix, we suggest that you install in '/usr/local' in
order not to interfere with your system::
python setup.py install --prefix /usr/local
Testing
.......
Go in the directory 'pyxnat/tests' and run the `nosetests` command.
pyxnat-0.9.0~dev0/doc/reference_documentation.rst 0000664 0000000 0000000 00000003277 11632421554 0022222 0 ustar 00root root 0000000 0000000 ==============================
Reference documentation
==============================
.. currentmodule:: pyxnat.core
The `Interface` class
~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.Interface
:members:
:inherited-members:
The `Select` class
~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.select.Select
:members: __call__, project, projects
The `Inspect` class
~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.help.Inspector
:members:
The `SearchManager` class
~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.search.SearchManager
:members:
The `Search` class
~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.search.Search
:members:
The `EObject` class
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.resources.EObject
:members:
The `CObject` class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.resources.CObject
:members:
The `Project` class
~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.resources.Project
:members:
The `File` class
~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.resources.File
:members:
The `Attributes` class
~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.attributes.EAttrs
:members:
The `Provenance` class
~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.provenance.Provenance
:members:
The `Users` class
~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.users.Users
:members:
The `Inspector` class
~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.help.Inspector
:members:
The `CacheManager` class
~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.cache.CacheManager
:members:
The `SchemaManager` class
~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: pyxnat.core.manage.SchemaManager
:members:
pyxnat-0.9.0~dev0/doc/sphinxext/ 0000775 0000000 0000000 00000000000 11632421554 0016622 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/doc/sphinxext/LICENSE.txt 0000664 0000000 0000000 00000003027 11632421554 0020447 0 ustar 00root root 0000000 0000000 -------------------------------------------------------------------------------
The files
- numpydoc.py
- autosummary.py
- autosummary_generate.py
- docscrape.py
- docscrape_sphinx.py
- phantom_import.py
have the following license:
Copyright (C) 2008 Stefan van der Walt , Pauli Virtanen
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
pyxnat-0.9.0~dev0/doc/sphinxext/__init__.py 0000664 0000000 0000000 00000000000 11632421554 0020721 0 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/doc/sphinxext/autosummary.py 0000664 0000000 0000000 00000024741 11632421554 0021572 0 ustar 00root root 0000000 0000000 """
===========
autosummary
===========
Sphinx extension that adds an autosummary:: directive, which can be
used to generate function/method/attribute/etc. summary lists, similar
to those output eg. by Epydoc and other API doc generation tools.
An :autolink: role is also provided.
autosummary directive
---------------------
The autosummary directive has the form::
.. autosummary::
:nosignatures:
:toctree: generated/
module.function_1
module.function_2
...
and it generates an output table (containing signatures, optionally)
======================== =============================================
module.function_1(args) Summary line from the docstring of function_1
module.function_2(args) Summary line from the docstring
...
======================== =============================================
If the :toctree: option is specified, files matching the function names
are inserted to the toctree with the given prefix:
generated/module.function_1
generated/module.function_2
...
Note: The file names contain the module:: or currentmodule:: prefixes.
.. seealso:: autosummary_generate.py
autolink role
-------------
The autolink role functions as ``:obj:`` when the name referred can be
resolved to a Python object, and otherwise it becomes simple emphasis.
This can be used as the default role to make links 'smart'.
"""
import sys, os, posixpath, re
from docutils.parsers.rst import directives
from docutils.statemachine import ViewList
from docutils import nodes
import sphinx.addnodes, sphinx.roles
from sphinx.util import patfilter
from docscrape_sphinx import get_doc_object
def setup(app):
app.add_directive('autosummary', autosummary_directive, True, (0, 0, False),
toctree=directives.unchanged,
nosignatures=directives.flag)
app.add_role('autolink', autolink_role)
app.add_node(autosummary_toc,
html=(autosummary_toc_visit_html, autosummary_toc_depart_noop),
latex=(autosummary_toc_visit_latex, autosummary_toc_depart_noop))
app.connect('doctree-read', process_autosummary_toc)
#------------------------------------------------------------------------------
# autosummary_toc node
#------------------------------------------------------------------------------
class autosummary_toc(nodes.comment):
pass
def process_autosummary_toc(app, doctree):
"""
Insert items described in autosummary:: to the TOC tree, but do
not generate the toctree:: list.
"""
env = app.builder.env
crawled = {}
def crawl_toc(node, depth=1):
crawled[node] = True
for j, subnode in enumerate(node):
try:
if (isinstance(subnode, autosummary_toc)
and isinstance(subnode[0], sphinx.addnodes.toctree)):
env.note_toctree(env.docname, subnode[0])
continue
except IndexError:
continue
if not isinstance(subnode, nodes.section):
continue
if subnode not in crawled:
crawl_toc(subnode, depth+1)
crawl_toc(doctree)
def autosummary_toc_visit_html(self, node):
"""Hide autosummary toctree list in HTML output"""
raise nodes.SkipNode
def autosummary_toc_visit_latex(self, node):
"""Show autosummary toctree (= put the referenced pages here) in Latex"""
pass
def autosummary_toc_depart_noop(self, node):
pass
#------------------------------------------------------------------------------
# .. autosummary::
#------------------------------------------------------------------------------
def autosummary_directive(dirname, arguments, options, content, lineno,
content_offset, block_text, state, state_machine):
"""
Pretty table containing short signatures and summaries of functions etc.
autosummary also generates a (hidden) toctree:: node.
"""
names = []
names += [x.strip() for x in content if x.strip()]
table, warnings, real_names = get_autosummary(names, state,
'nosignatures' in options)
node = table
env = state.document.settings.env
suffix = env.config.source_suffix
all_docnames = env.found_docs.copy()
dirname = posixpath.dirname(env.docname)
if 'toctree' in options:
tree_prefix = options['toctree'].strip()
docnames = []
for name in names:
name = real_names.get(name, name)
docname = tree_prefix + name
if docname.endswith(suffix):
docname = docname[:-len(suffix)]
docname = posixpath.normpath(posixpath.join(dirname, docname))
if docname not in env.found_docs:
warnings.append(state.document.reporter.warning(
'toctree references unknown document %r' % docname,
line=lineno))
docnames.append(docname)
tocnode = sphinx.addnodes.toctree()
tocnode['includefiles'] = docnames
tocnode['maxdepth'] = -1
tocnode['glob'] = None
tocnode['entries'] = []
tocnode = autosummary_toc('', '', tocnode)
return warnings + [node] + [tocnode]
else:
return warnings + [node]
def get_autosummary(names, state, no_signatures=False):
"""
Generate a proper table node for autosummary:: directive.
Parameters
----------
names : list of str
Names of Python objects to be imported and added to the table.
document : document
Docutils document object
"""
document = state.document
real_names = {}
warnings = []
prefixes = ['']
prefixes.insert(0, document.settings.env.currmodule)
table = nodes.table('')
group = nodes.tgroup('', cols=2)
table.append(group)
group.append(nodes.colspec('', colwidth=30))
group.append(nodes.colspec('', colwidth=70))
body = nodes.tbody('')
group.append(body)
def append_row(*column_texts):
row = nodes.row('')
for text in column_texts:
node = nodes.paragraph('')
vl = ViewList()
vl.append(text, '')
state.nested_parse(vl, 0, node)
row.append(nodes.entry('', node))
body.append(row)
for name in names:
try:
obj, real_name = import_by_name(name, prefixes=prefixes)
except ImportError:
warnings.append(document.reporter.warning(
'failed to import %s' % name))
append_row(":obj:`%s`" % name, "")
continue
real_names[name] = real_name
doc = get_doc_object(obj)
if doc['Summary']:
title = " ".join(doc['Summary'])
else:
title = ""
col1 = ":obj:`%s <%s>`" % (name, real_name)
if doc['Signature']:
sig = re.sub('^[a-zA-Z_0-9.-]*', '', doc['Signature'])
if '=' in sig:
# abbreviate optional arguments
sig = re.sub(r', ([a-zA-Z0-9_]+)=', r'[, \1=', sig, count=1)
sig = re.sub(r'\(([a-zA-Z0-9_]+)=', r'([\1=', sig, count=1)
sig = re.sub(r'=[^,)]+,', ',', sig)
sig = re.sub(r'=[^,)]+\)$', '])', sig)
# shorten long strings
sig = re.sub(r'(\[.{16,16}[^,)]*?),.*?\]\)', r'\1, ...])', sig)
else:
sig = re.sub(r'(\(.{16,16}[^,)]*?),.*?\)', r'\1, ...)', sig)
col1 += " " + sig
col2 = title
append_row(col1, col2)
return table, warnings, real_names
def import_by_name(name, prefixes=[None]):
"""
Import a Python object that has the given name, under one of the prefixes.
Parameters
----------
name : str
Name of a Python object, eg. 'numpy.ndarray.view'
prefixes : list of (str or None), optional
Prefixes to prepend to the name (None implies no prefix).
The first prefixed name that results to successful import is used.
Returns
-------
obj
The imported object
name
Name of the imported object (useful if `prefixes` was used)
"""
for prefix in prefixes:
try:
if prefix:
prefixed_name = '.'.join([prefix, name])
else:
prefixed_name = name
return _import_by_name(prefixed_name), prefixed_name
except ImportError:
pass
raise ImportError
def _import_by_name(name):
"""Import a Python object given its full name"""
try:
# try first interpret `name` as MODNAME.OBJ
name_parts = name.split('.')
try:
modname = '.'.join(name_parts[:-1])
__import__(modname)
return getattr(sys.modules[modname], name_parts[-1])
except (ImportError, IndexError, AttributeError):
pass
# ... then as MODNAME, MODNAME.OBJ1, MODNAME.OBJ1.OBJ2, ...
last_j = 0
modname = None
for j in reversed(range(1, len(name_parts)+1)):
last_j = j
modname = '.'.join(name_parts[:j])
try:
__import__(modname)
except ImportError:
continue
if modname in sys.modules:
break
if last_j < len(name_parts):
obj = sys.modules[modname]
for obj_name in name_parts[last_j:]:
obj = getattr(obj, obj_name)
return obj
else:
return sys.modules[modname]
except (ValueError, ImportError, AttributeError, KeyError), e:
raise ImportError(e)
#------------------------------------------------------------------------------
# :autolink: (smart default role)
#------------------------------------------------------------------------------
def autolink_role(typ, rawtext, etext, lineno, inliner,
options={}, content=[]):
"""
Smart linking role.
Expands to ":obj:`text`" if `text` is an object that can be imported;
otherwise expands to "*text*".
"""
r = sphinx.roles.xfileref_role('obj', rawtext, etext, lineno, inliner,
options, content)
pnode = r[0][0]
prefixes = [None]
#prefixes.insert(0, inliner.document.settings.env.currmodule)
try:
obj, name = import_by_name(pnode['reftarget'], prefixes)
except ImportError:
content = pnode[0]
r[0][0] = nodes.emphasis(rawtext, content[0].astext(),
classes=content['classes'])
return r
pyxnat-0.9.0~dev0/doc/sphinxext/autosummary_generate.py 0000775 0000000 0000000 00000016471 11632421554 0023450 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python
r"""
autosummary_generate.py OPTIONS FILES
Generate automatic RST source files for items referred to in
autosummary:: directives.
Each generated RST file contains a single auto*:: directive which
extracts the docstring of the referred item.
Example Makefile rule::
generate:
./ext/autosummary_generate.py -o source/generated source/*.rst
"""
import glob, re, inspect, os, optparse, pydoc
from autosummary import import_by_name
try:
from phantom_import import import_phantom_module
except ImportError:
import_phantom_module = lambda x: x
def main():
p = optparse.OptionParser(__doc__.strip())
p.add_option("-p", "--phantom", action="store", type="string",
dest="phantom", default=None,
help="Phantom import modules from a file")
p.add_option("-o", "--output-dir", action="store", type="string",
dest="output_dir", default=None,
help=("Write all output files to the given directory (instead "
"of writing them as specified in the autosummary:: "
"directives)"))
options, args = p.parse_args()
if len(args) == 0:
p.error("wrong number of arguments")
if options.phantom and os.path.isfile(options.phantom):
import_phantom_module(options.phantom)
# read
names = {}
for name, loc in get_documented(args).items():
for (filename, sec_title, keyword, toctree) in loc:
if toctree is not None:
path = os.path.join(os.path.dirname(filename), toctree)
names[name] = os.path.abspath(path)
# write
for name, path in sorted(names.items()):
if options.output_dir is not None:
path = options.output_dir
if not os.path.isdir(path):
os.makedirs(path)
try:
obj, name = import_by_name(name)
except ImportError, e:
print "Failed to import '%s': %s" % (name, e)
continue
fn = os.path.join(path, '%s.rst' % name)
if os.path.exists(fn):
# skip
continue
f = open(fn, 'w')
try:
f.write('%s\n%s\n\n' % (name, '='*len(name)))
if inspect.isclass(obj):
if issubclass(obj, Exception):
f.write(format_modulemember(name, 'autoexception'))
else:
f.write(format_modulemember(name, 'autoclass'))
elif inspect.ismodule(obj):
f.write(format_modulemember(name, 'automodule'))
elif inspect.ismethod(obj) or inspect.ismethoddescriptor(obj):
f.write(format_classmember(name, 'automethod'))
elif callable(obj):
f.write(format_modulemember(name, 'autofunction'))
elif hasattr(obj, '__get__'):
f.write(format_classmember(name, 'autoattribute'))
else:
f.write(format_modulemember(name, 'autofunction'))
finally:
f.close()
def format_modulemember(name, directive):
parts = name.split('.')
mod, name = '.'.join(parts[:-1]), parts[-1]
return ".. currentmodule:: %s\n\n.. %s:: %s\n" % (mod, directive, name)
def format_classmember(name, directive):
parts = name.split('.')
mod, name = '.'.join(parts[:-2]), '.'.join(parts[-2:])
return ".. currentmodule:: %s\n\n.. %s:: %s\n" % (mod, directive, name)
def get_documented(filenames):
"""
Find out what items are documented in source/*.rst
See `get_documented_in_lines`.
"""
documented = {}
for filename in filenames:
f = open(filename, 'r')
lines = f.read().splitlines()
documented.update(get_documented_in_lines(lines, filename=filename))
f.close()
return documented
def get_documented_in_docstring(name, module=None, filename=None):
"""
Find out what items are documented in the given object's docstring.
See `get_documented_in_lines`.
"""
try:
obj, real_name = import_by_name(name)
lines = pydoc.getdoc(obj).splitlines()
return get_documented_in_lines(lines, module=name, filename=filename)
except AttributeError:
pass
except ImportError, e:
print "Failed to import '%s': %s" % (name, e)
return {}
def get_documented_in_lines(lines, module=None, filename=None):
"""
Find out what items are documented in the given lines
Returns
-------
documented : dict of list of (filename, title, keyword, toctree)
Dictionary whose keys are documented names of objects.
The value is a list of locations where the object was documented.
Each location is a tuple of filename, the current section title,
the name of the directive, and the value of the :toctree: argument
(if present) of the directive.
"""
title_underline_re = re.compile("^[-=*_^#]{3,}\s*$")
autodoc_re = re.compile(".. auto(function|method|attribute|class|exception|module)::\s*([A-Za-z0-9_.]+)\s*$")
autosummary_re = re.compile(r'^\.\.\s+autosummary::\s*')
module_re = re.compile(r'^\.\.\s+(current)?module::\s*([a-zA-Z0-9_.]+)\s*$')
autosummary_item_re = re.compile(r'^\s+([_a-zA-Z][a-zA-Z0-9_.]*)\s*')
toctree_arg_re = re.compile(r'^\s+:toctree:\s*(.*?)\s*$')
documented = {}
current_title = []
last_line = None
toctree = None
current_module = module
in_autosummary = False
for line in lines:
try:
if in_autosummary:
m = toctree_arg_re.match(line)
if m:
toctree = m.group(1)
continue
if line.strip().startswith(':'):
continue # skip options
m = autosummary_item_re.match(line)
if m:
name = m.group(1).strip()
if current_module and not name.startswith(current_module + '.'):
name = "%s.%s" % (current_module, name)
documented.setdefault(name, []).append(
(filename, current_title, 'autosummary', toctree))
continue
if line.strip() == '':
continue
in_autosummary = False
m = autosummary_re.match(line)
if m:
in_autosummary = True
continue
m = autodoc_re.search(line)
if m:
name = m.group(2).strip()
if m.group(1) == "module":
current_module = name
documented.update(get_documented_in_docstring(
name, filename=filename))
elif current_module and not name.startswith(current_module+'.'):
name = "%s.%s" % (current_module, name)
documented.setdefault(name, []).append(
(filename, current_title, "auto" + m.group(1), None))
continue
m = title_underline_re.match(line)
if m and last_line:
current_title = last_line.strip()
continue
m = module_re.match(line)
if m:
current_module = m.group(2)
continue
finally:
last_line = line
return documented
if __name__ == "__main__":
main()
pyxnat-0.9.0~dev0/doc/sphinxext/docscrape.py 0000664 0000000 0000000 00000034605 11632421554 0021147 0 ustar 00root root 0000000 0000000 """Extract reference documentation from the NumPy source tree.
"""
import inspect
import textwrap
import re
import pydoc
from StringIO import StringIO
from warnings import warn
4
class Reader(object):
"""A line-based string reader.
"""
def __init__(self, data):
"""
Parameters
----------
data : str
String with lines separated by '\n'.
"""
if isinstance(data,list):
self._str = data
else:
self._str = data.split('\n') # store string as list of lines
self.reset()
def __getitem__(self, n):
return self._str[n]
def reset(self):
self._l = 0 # current line nr
def read(self):
if not self.eof():
out = self[self._l]
self._l += 1
return out
else:
return ''
def seek_next_non_empty_line(self):
for l in self[self._l:]:
if l.strip():
break
else:
self._l += 1
def eof(self):
return self._l >= len(self._str)
def read_to_condition(self, condition_func):
start = self._l
for line in self[start:]:
if condition_func(line):
return self[start:self._l]
self._l += 1
if self.eof():
return self[start:self._l+1]
return []
def read_to_next_empty_line(self):
self.seek_next_non_empty_line()
def is_empty(line):
return not line.strip()
return self.read_to_condition(is_empty)
def read_to_next_unindented_line(self):
def is_unindented(line):
return (line.strip() and (len(line.lstrip()) == len(line)))
return self.read_to_condition(is_unindented)
def peek(self,n=0):
if self._l + n < len(self._str):
return self[self._l + n]
else:
return ''
def is_empty(self):
return not ''.join(self._str).strip()
class NumpyDocString(object):
def __init__(self,docstring):
docstring = textwrap.dedent(docstring).split('\n')
self._doc = Reader(docstring)
self._parsed_data = {
'Signature': '',
'Summary': [''],
'Extended Summary': [],
'Parameters': [],
'Returns': [],
'Raises': [],
'Warns': [],
'Other Parameters': [],
'Attributes': [],
'Methods': [],
'See Also': [],
'Notes': [],
'Warnings': [],
'References': '',
'Examples': '',
'index': {}
}
self._parse()
def __getitem__(self,key):
return self._parsed_data[key]
def __setitem__(self,key,val):
if not self._parsed_data.has_key(key):
warn("Unknown section %s" % key)
else:
self._parsed_data[key] = val
def _is_at_section(self):
self._doc.seek_next_non_empty_line()
if self._doc.eof():
return False
l1 = self._doc.peek().strip() # e.g. Parameters
if l1.startswith('.. index::'):
return True
l2 = self._doc.peek(1).strip() # ---------- or ==========
return l2.startswith('-'*len(l1)) or l2.startswith('='*len(l1))
def _strip(self,doc):
i = 0
j = 0
for i,line in enumerate(doc):
if line.strip(): break
for j,line in enumerate(doc[::-1]):
if line.strip(): break
return doc[i:len(doc)-j]
def _read_to_next_section(self):
section = self._doc.read_to_next_empty_line()
while not self._is_at_section() and not self._doc.eof():
if not self._doc.peek(-1).strip(): # previous line was empty
section += ['']
section += self._doc.read_to_next_empty_line()
return section
def _read_sections(self):
while not self._doc.eof():
data = self._read_to_next_section()
name = data[0].strip()
if name.startswith('..'): # index section
yield name, data[1:]
elif len(data) < 2:
yield StopIteration
else:
yield name, self._strip(data[2:])
def _parse_param_list(self,content):
r = Reader(content)
params = []
while not r.eof():
header = r.read().strip()
if ' : ' in header:
arg_name, arg_type = header.split(' : ')[:2]
else:
arg_name, arg_type = header, ''
desc = r.read_to_next_unindented_line()
desc = dedent_lines(desc)
params.append((arg_name,arg_type,desc))
return params
_name_rgx = re.compile(r"^\s*(:(?P\w+):`(?P[a-zA-Z0-9_.-]+)`|"
r" (?P[a-zA-Z0-9_.-]+))\s*", re.X)
def _parse_see_also(self, content):
"""
func_name : Descriptive text
continued text
another_func_name : Descriptive text
func_name1, func_name2, :meth:`func_name`, func_name3
"""
items = []
def parse_item_name(text):
"""Match ':role:`name`' or 'name'"""
m = self._name_rgx.match(text)
if m:
g = m.groups()
if g[1] is None:
return g[3], None
else:
return g[2], g[1]
raise ValueError("%s is not a item name" % text)
def push_item(name, rest):
if not name:
return
name, role = parse_item_name(name)
items.append((name, list(rest), role))
del rest[:]
current_func = None
rest = []
for line in content:
if not line.strip(): continue
m = self._name_rgx.match(line)
if m and line[m.end():].strip().startswith(':'):
push_item(current_func, rest)
current_func, line = line[:m.end()], line[m.end():]
rest = [line.split(':', 1)[1].strip()]
if not rest[0]:
rest = []
elif not line.startswith(' '):
push_item(current_func, rest)
current_func = None
if ',' in line:
for func in line.split(','):
push_item(func, [])
elif line.strip():
current_func = line
elif current_func is not None:
rest.append(line.strip())
push_item(current_func, rest)
return items
def _parse_index(self, section, content):
"""
.. index: default
:refguide: something, else, and more
"""
def strip_each_in(lst):
return [s.strip() for s in lst]
out = {}
section = section.split('::')
if len(section) > 1:
out['default'] = strip_each_in(section[1].split(','))[0]
for line in content:
line = line.split(':')
if len(line) > 2:
out[line[1]] = strip_each_in(line[2].split(','))
return out
def _parse_summary(self):
"""Grab signature (if given) and summary"""
if self._is_at_section():
return
summary = self._doc.read_to_next_empty_line()
summary_str = " ".join([s.strip() for s in summary]).strip()
if re.compile('^([\w., ]+=)?\s*[\w\.]+\(.*\)$').match(summary_str):
self['Signature'] = summary_str
if not self._is_at_section():
self['Summary'] = self._doc.read_to_next_empty_line()
else:
self['Summary'] = summary
if not self._is_at_section():
self['Extended Summary'] = self._read_to_next_section()
def _parse(self):
self._doc.reset()
self._parse_summary()
for (section,content) in self._read_sections():
if not section.startswith('..'):
section = ' '.join([s.capitalize() for s in section.split(' ')])
if section in ('Parameters', 'Attributes', 'Methods',
'Returns', 'Raises', 'Warns'):
self[section] = self._parse_param_list(content)
elif section.startswith('.. index::'):
self['index'] = self._parse_index(section, content)
elif section == 'See Also':
self['See Also'] = self._parse_see_also(content)
else:
self[section] = content
# string conversion routines
def _str_header(self, name, symbol='-'):
return [name, len(name)*symbol]
def _str_indent(self, doc, indent=4):
out = []
for line in doc:
out += [' '*indent + line]
return out
def _str_signature(self):
if self['Signature']:
return [self['Signature'].replace('*','\*')] + ['']
else:
return ['']
def _str_summary(self):
if self['Summary']:
return self['Summary'] + ['']
else:
return []
def _str_extended_summary(self):
if self['Extended Summary']:
return self['Extended Summary'] + ['']
else:
return []
def _str_param_list(self, name):
out = []
if self[name]:
out += self._str_header(name)
for param,param_type,desc in self[name]:
out += ['%s : %s' % (param, param_type)]
out += self._str_indent(desc)
out += ['']
return out
def _str_section(self, name):
out = []
if self[name]:
out += self._str_header(name)
out += self[name]
out += ['']
return out
def _str_see_also(self, func_role):
if not self['See Also']: return []
out = []
out += self._str_header("See Also")
last_had_desc = True
for func, desc, role in self['See Also']:
if role:
link = ':%s:`%s`' % (role, func)
elif func_role:
link = ':%s:`%s`' % (func_role, func)
else:
link = "`%s`_" % func
if desc or last_had_desc:
out += ['']
out += [link]
else:
out[-1] += ", %s" % link
if desc:
out += self._str_indent([' '.join(desc)])
last_had_desc = True
else:
last_had_desc = False
out += ['']
return out
def _str_index(self):
idx = self['index']
out = []
out += ['.. index:: %s' % idx.get('default','')]
for section, references in idx.iteritems():
if section == 'default':
continue
out += [' :%s: %s' % (section, ', '.join(references))]
return out
def __str__(self, func_role=''):
out = []
out += self._str_signature()
out += self._str_summary()
out += self._str_extended_summary()
for param_list in ('Parameters','Returns','Raises'):
out += self._str_param_list(param_list)
out += self._str_section('Warnings')
out += self._str_see_also(func_role)
for s in ('Notes','References','Examples'):
out += self._str_section(s)
out += self._str_index()
return '\n'.join(out)
def indent(str,indent=4):
indent_str = ' '*indent
if str is None:
return indent_str
lines = str.split('\n')
return '\n'.join(indent_str + l for l in lines)
def dedent_lines(lines):
"""Deindent a list of lines maximally"""
return textwrap.dedent("\n".join(lines)).split("\n")
def header(text, style='-'):
return text + '\n' + style*len(text) + '\n'
class FunctionDoc(NumpyDocString):
def __init__(self, func, role='func'):
self._f = func
self._role = role # e.g. "func" or "meth"
try:
NumpyDocString.__init__(self,inspect.getdoc(func) or '')
except ValueError, e:
print '*'*78
print "ERROR: '%s' while parsing `%s`" % (e, self._f)
print '*'*78
#print "Docstring follows:"
#print doclines
#print '='*78
if not self['Signature']:
func, func_name = self.get_func()
try:
# try to read signature
argspec = inspect.getargspec(func)
argspec = inspect.formatargspec(*argspec)
argspec = argspec.replace('*','\*')
signature = '%s%s' % (func_name, argspec)
except TypeError, e:
signature = '%s()' % func_name
self['Signature'] = signature
def get_func(self):
func_name = getattr(self._f, '__name__', self.__class__.__name__)
if inspect.isclass(self._f):
func = getattr(self._f, '__call__', self._f.__init__)
else:
func = self._f
return func, func_name
def __str__(self):
out = ''
func, func_name = self.get_func()
signature = self['Signature'].replace('*', '\*')
roles = {'func': 'function',
'meth': 'method'}
if self._role:
if not roles.has_key(self._role):
print "Warning: invalid role %s" % self._role
out += '.. %s:: %s\n \n\n' % (roles.get(self._role,''),
func_name)
out += super(FunctionDoc, self).__str__(func_role=self._role)
return out
class ClassDoc(NumpyDocString):
def __init__(self,cls,modulename='',func_doc=FunctionDoc):
if not inspect.isclass(cls):
raise ValueError("Initialise using a class. Got %r" % cls)
self._cls = cls
if modulename and not modulename.endswith('.'):
modulename += '.'
self._mod = modulename
self._name = cls.__name__
self._func_doc = func_doc
NumpyDocString.__init__(self, pydoc.getdoc(cls))
@property
def methods(self):
return [name for name,func in inspect.getmembers(self._cls)
if not name.startswith('_') and callable(func)]
def __str__(self):
out = ''
out += super(ClassDoc, self).__str__()
out += "\n\n"
#for m in self.methods:
# print "Parsing `%s`" % m
# out += str(self._func_doc(getattr(self._cls,m), 'meth')) + '\n\n'
# out += '.. index::\n single: %s; %s\n\n' % (self._name, m)
return out
pyxnat-0.9.0~dev0/doc/sphinxext/docscrape_sphinx.py 0000664 0000000 0000000 00000010074 11632421554 0022532 0 ustar 00root root 0000000 0000000 import re, inspect, textwrap, pydoc
from docscrape import NumpyDocString, FunctionDoc, ClassDoc
class SphinxDocString(NumpyDocString):
# string conversion routines
def _str_header(self, name, symbol='`'):
return ['.. rubric:: ' + name, '']
def _str_field_list(self, name):
return [':' + name + ':']
def _str_indent(self, doc, indent=4):
out = []
for line in doc:
out += [' '*indent + line]
return out
def _str_signature(self):
return ['']
if self['Signature']:
return ['``%s``' % self['Signature']] + ['']
else:
return ['']
def _str_summary(self):
return self['Summary'] + ['']
def _str_extended_summary(self):
return self['Extended Summary'] + ['']
def _str_param_list(self, name):
out = []
if self[name]:
out += self._str_field_list(name)
out += ['']
for param,param_type,desc in self[name]:
out += self._str_indent(['**%s** : %s' % (param.strip(),
param_type)])
out += ['']
out += self._str_indent(desc,8)
out += ['']
return out
def _str_section(self, name):
out = []
if self[name]:
out += self._str_header(name)
out += ['']
content = textwrap.dedent("\n".join(self[name])).split("\n")
out += content
out += ['']
return out
def _str_see_also(self, func_role):
out = []
if self['See Also']:
see_also = super(SphinxDocString, self)._str_see_also(func_role)
out = ['.. seealso::', '']
out += self._str_indent(see_also[2:])
return out
def _str_warnings(self):
out = []
if self['Warnings']:
out = ['.. warning::', '']
out += self._str_indent(self['Warnings'])
return out
def _str_index(self):
idx = self['index']
out = []
if len(idx) == 0:
return out
out += ['.. index:: %s' % idx.get('default','')]
for section, references in idx.iteritems():
if section == 'default':
continue
elif section == 'refguide':
out += [' single: %s' % (', '.join(references))]
else:
out += [' %s: %s' % (section, ','.join(references))]
return out
def _str_references(self):
out = []
if self['References']:
out += self._str_header('References')
if isinstance(self['References'], str):
self['References'] = [self['References']]
out.extend(self['References'])
out += ['']
return out
def __str__(self, indent=0, func_role="obj"):
out = []
out += self._str_signature()
out += self._str_index() + ['']
out += self._str_summary()
out += self._str_extended_summary()
for param_list in ('Parameters', 'Attributes', 'Methods',
'Returns','Raises'):
out += self._str_param_list(param_list)
out += self._str_warnings()
out += self._str_see_also(func_role)
out += self._str_section('Notes')
out += self._str_references()
out += self._str_section('Examples')
out = self._str_indent(out,indent)
return '\n'.join(out)
class SphinxFunctionDoc(SphinxDocString, FunctionDoc):
pass
class SphinxClassDoc(SphinxDocString, ClassDoc):
pass
def get_doc_object(obj, what=None):
if what is None:
if inspect.isclass(obj):
what = 'class'
elif inspect.ismodule(obj):
what = 'module'
elif callable(obj):
what = 'function'
else:
what = 'object'
if what == 'class':
return SphinxClassDoc(obj, '', func_doc=SphinxFunctionDoc)
elif what in ('function', 'method'):
return SphinxFunctionDoc(obj, '')
else:
return SphinxDocString(pydoc.getdoc(obj))
pyxnat-0.9.0~dev0/doc/sphinxext/numpydoc.py 0000664 0000000 0000000 00000007370 11632421554 0021041 0 ustar 00root root 0000000 0000000 """
========
numpydoc
========
Sphinx extension that handles docstrings in the Numpy standard format. [1]
It will:
- Convert Parameters etc. sections to field lists.
- Convert See Also section to a See also entry.
- Renumber references.
- Extract the signature from the docstring, if it can't be determined otherwise.
.. [1] http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines#docstring-standard
"""
import os, re, pydoc
from docscrape_sphinx import get_doc_object, SphinxDocString
import inspect
def mangle_docstrings(app, what, name, obj, options, lines,
reference_offset=[0]):
if what == 'module':
# Strip top title
title_re = re.compile(r'^\s*[#*=]{4,}\n[a-z0-9 -]+\n[#*=]{4,}\s*',
re.I|re.S)
lines[:] = title_re.sub('', "\n".join(lines)).split("\n")
else:
doc = get_doc_object(obj, what)
lines[:] = str(doc).split("\n")
if app.config.numpydoc_edit_link and hasattr(obj, '__name__') and \
obj.__name__:
v = dict(full_name=obj.__name__)
lines += [''] + (app.config.numpydoc_edit_link % v).split("\n")
# replace reference numbers so that there are no duplicates
references = []
for l in lines:
l = l.strip()
if l.startswith('.. ['):
try:
references.append(int(l[len('.. ['):l.index(']')]))
except ValueError:
print "WARNING: invalid reference in %s docstring" % name
# Start renaming from the biggest number, otherwise we may
# overwrite references.
references.sort()
if references:
for i, line in enumerate(lines):
for r in references:
new_r = reference_offset[0] + r
lines[i] = lines[i].replace('[%d]_' % r,
'[%d]_' % new_r)
lines[i] = lines[i].replace('.. [%d]' % r,
'.. [%d]' % new_r)
reference_offset[0] += len(references)
def mangle_signature(app, what, name, obj, options, sig, retann):
# Do not try to inspect classes that don't define `__init__`
if (inspect.isclass(obj) and
'initializes x; see ' in pydoc.getdoc(obj.__init__)):
return '', ''
if not (callable(obj) or hasattr(obj, '__argspec_is_invalid_')): return
if not hasattr(obj, '__doc__'): return
doc = SphinxDocString(pydoc.getdoc(obj))
if doc['Signature']:
sig = re.sub("^[^(]*", "", doc['Signature'])
return sig, ''
def initialize(app):
try:
app.connect('autodoc-process-signature', mangle_signature)
except:
monkeypatch_sphinx_ext_autodoc()
def setup(app, get_doc_object_=get_doc_object):
global get_doc_object
get_doc_object = get_doc_object_
app.connect('autodoc-process-docstring', mangle_docstrings)
app.connect('builder-inited', initialize)
app.add_config_value('numpydoc_edit_link', None, True)
#------------------------------------------------------------------------------
# Monkeypatch sphinx.ext.autodoc to accept argspecless autodocs (Sphinx < 0.5)
#------------------------------------------------------------------------------
def monkeypatch_sphinx_ext_autodoc():
global _original_format_signature
import sphinx.ext.autodoc
if sphinx.ext.autodoc.format_signature is our_format_signature:
return
print "[numpydoc] Monkeypatching sphinx.ext.autodoc ..."
_original_format_signature = sphinx.ext.autodoc.format_signature
sphinx.ext.autodoc.format_signature = our_format_signature
def our_format_signature(what, obj):
r = mangle_signature(None, what, None, obj, None, None, None)
if r is not None:
return r[0]
else:
return _original_format_signature(what, obj)
pyxnat-0.9.0~dev0/doc/sphinxext/phantom_import.py 0000664 0000000 0000000 00000013064 11632421554 0022240 0 ustar 00root root 0000000 0000000 """
==============
phantom_import
==============
Sphinx extension to make directives from ``sphinx.ext.autodoc`` and similar
extensions to use docstrings loaded from an XML file.
This extension loads an XML file in the Pydocweb format [1] and
creates a dummy module that contains the specified docstrings. This
can be used to get the current docstrings from a Pydocweb instance
without needing to rebuild the documented module.
.. [1] http://code.google.com/p/pydocweb
"""
import imp, sys, compiler, types, os, inspect, re
def setup(app):
app.connect('builder-inited', initialize)
app.add_config_value('phantom_import_file', None, True)
def initialize(app):
fn = app.config.phantom_import_file
if (fn and os.path.isfile(fn)):
print "[numpydoc] Phantom importing modules from", fn, "..."
import_phantom_module(fn)
#------------------------------------------------------------------------------
# Creating 'phantom' modules from an XML description
#------------------------------------------------------------------------------
def import_phantom_module(xml_file):
"""
Insert a fake Python module to sys.modules, based on a XML file.
The XML file is expected to conform to Pydocweb DTD. The fake
module will contain dummy objects, which guarantee the following:
- Docstrings are correct.
- Class inheritance relationships are correct (if present in XML).
- Function argspec is *NOT* correct (even if present in XML).
Instead, the function signature is prepended to the function docstring.
- Class attributes are *NOT* correct; instead, they are dummy objects.
Parameters
----------
xml_file : str
Name of an XML file to read
"""
import lxml.etree as etree
object_cache = {}
tree = etree.parse(xml_file)
root = tree.getroot()
# Sort items so that
# - Base classes come before classes inherited from them
# - Modules come before their contents
all_nodes = dict([(n.attrib['id'], n) for n in root])
def _get_bases(node, recurse=False):
bases = [x.attrib['ref'] for x in node.findall('base')]
if recurse:
j = 0
while True:
try:
b = bases[j]
except IndexError: break
if b in all_nodes:
bases.extend(_get_bases(all_nodes[b]))
j += 1
return bases
type_index = ['module', 'class', 'callable', 'object']
def base_cmp(a, b):
x = cmp(type_index.index(a.tag), type_index.index(b.tag))
if x != 0: return x
if a.tag == 'class' and b.tag == 'class':
a_bases = _get_bases(a, recurse=True)
b_bases = _get_bases(b, recurse=True)
x = cmp(len(a_bases), len(b_bases))
if x != 0: return x
if a.attrib['id'] in b_bases: return -1
if b.attrib['id'] in a_bases: return 1
return cmp(a.attrib['id'].count('.'), b.attrib['id'].count('.'))
nodes = root.getchildren()
nodes.sort(base_cmp)
# Create phantom items
for node in nodes:
name = node.attrib['id']
doc = (node.text or '').decode('string-escape') + "\n"
if doc == "\n": doc = ""
# create parent, if missing
parent = name
while True:
parent = '.'.join(parent.split('.')[:-1])
if not parent: break
if parent in object_cache: break
obj = imp.new_module(parent)
object_cache[parent] = obj
sys.modules[parent] = obj
# create object
if node.tag == 'module':
obj = imp.new_module(name)
obj.__doc__ = doc
sys.modules[name] = obj
elif node.tag == 'class':
bases = [object_cache[b] for b in _get_bases(node)
if b in object_cache]
bases.append(object)
init = lambda self: None
init.__doc__ = doc
obj = type(name, tuple(bases), {'__doc__': doc, '__init__': init})
obj.__name__ = name.split('.')[-1]
elif node.tag == 'callable':
funcname = node.attrib['id'].split('.')[-1]
argspec = node.attrib.get('argspec')
if argspec:
argspec = re.sub('^[^(]*', '', argspec)
doc = "%s%s\n\n%s" % (funcname, argspec, doc)
obj = lambda: 0
obj.__argspec_is_invalid_ = True
obj.func_name = funcname
obj.__name__ = name
obj.__doc__ = doc
if inspect.isclass(object_cache[parent]):
obj.__objclass__ = object_cache[parent]
else:
class Dummy(object): pass
obj = Dummy()
obj.__name__ = name
obj.__doc__ = doc
if inspect.isclass(object_cache[parent]):
obj.__get__ = lambda: None
object_cache[name] = obj
if parent:
if inspect.ismodule(object_cache[parent]):
obj.__module__ = parent
setattr(object_cache[parent], name.split('.')[-1], obj)
# Populate items
for node in root:
obj = object_cache.get(node.attrib['id'])
if obj is None: continue
for ref in node.findall('ref'):
if node.tag == 'class':
if ref.attrib['ref'].startswith(node.attrib['id'] + '.'):
setattr(obj, ref.attrib['name'],
object_cache.get(ref.attrib['ref']))
else:
setattr(obj, ref.attrib['name'],
object_cache.get(ref.attrib['ref']))
pyxnat-0.9.0~dev0/doc/starters_tutorial.rst 0000664 0000000 0000000 00000033772 11632421554 0021130 0 ustar 00root root 0000000 0000000 ==============================
Starters Tutorial
==============================
.. currentmodule:: pyxnat
This is a short tutorial going through the main features of this API.
Depending on the policy of the XNAT server you are using, and your
user level, you can have read and write access to a specific set of
resources. During this tutorial, we will use a `fake` standard user
account on the XNAT central repository, where you will have limited
access to a number of projects and full access to the projects you
own.
.. note::
XNAT Central is a public XNAT repository managed by the XNAT team
and updated regularly with the latest improvements of the
development branch.
_____
.. [#] http://central.xnat.org
Getting started
---------------
Connecting to a XNAT server requires valid credentials so you might want to
start by requesting those on the Web interface of your server.
>>> from pyxnat import Interface
>>> central = Interface(
server='http://central.xnat.org:8080',
user='my_login',
password='my_pass',
cachedir=os.path.join(os.path.expanduser('~'), '.store')
)
The cachedir argument specifies where the local disk-cache will be
stored. Every query and downloaded file will be stored here so choose
a location with sufficient free space. If the datastore argument if
not given, a temporary location is picked automatically but it may be
flushed at every reboot of the machine. Here the cachedir argument is
set to a `.store` directory in the user's home directory in a cross
platform manner. If the `cachedir` starts getting full, a warning will
be printed in the stdout.
It is also possible to define an :class:`Interface` object without
specifying all the connection parameters. Pyxnat switches to
interactive mode and prompts the user for the missing information. In
that case the object checks that the parameters are correct by
connecting to the server.
>>> central = Interface(server='http://central.xnat.org:8080')
>>> User:my_login
>>> Password:
You can also use a configuration file. The best way to create the file
is to use the ``save_config()`` method on an existing interface.
>>> central.save_config('central.cfg')
>>> central2 = Interface(config='central.cfg')
.. warning::
Depending on the server configuration, you may have to include the port
in the server url, as well as the name of the XNAT tomcat application.
So you might end up with something like:
http://server_ip:port/xnat
The main interface class is now divided into logical subinterfaces:
- data selection
- general management
- cache management
- server instrospection
Data selection
--------------
Now that we have an `Interface` object, we can start browsing the
server with the ``select`` subinterface which can be used, either with
expicit Python objects and methods, or through a ``path`` describing
the data.
Simple requests::
>>> interface.select.projects().get()
[..., u'CENTRAL_OASIS_CS', u'CENTRAL_OASIS_LONG', ...]
>>> interface.select('/projects').get()
[..., u'CENTRAL_OASIS_CS', u'CENTRAL_OASIS_LONG', ...]
Nested requests::
>>> interface.select.projects().subjects().get()
>>> interface.select('/projects/*/subjects').get()
>>> interface.select('/projects/subjects').get()
>>> interface.select('//subjects').get()
['IMAGEN_000000001274', 'IMAGEN_000000075717', ...,'IMAGEN_000099954902']
Filtered requests::
>>> interface.select.projects('*OASIS_CS*').get()
>>> interface.select('/projects/*OASIS_CS*').get()
[u'CENTRAL_OASIS_CS']
>>> interface.select.project('IMAGEN').subjects('*55*42*').get()
>>> interface.select('/projects/IMAGEN/subjects/*55*42*').get()
['IMAGEN_000055203542', 'IMAGEN_000055982442', 'IMAGEN_000097555742']
Resources paths
---------------
The resources paths that can be passed as an argument to ``select`` is
a powerful tool but can easily generate thousands of queries so one
has to be careful when using it.
Absolute paths
~~~~~~~~~~~~~~
A full path to a resource is a sequence of resource level and
resource_id pairs::
/project/IMAGEN/subject/IMAGEN_000055982442
A full path to a resource listing is a sequence of resource level and
resource_id pairs finishing by a plural resource level (i.e. with an
's')::
/project/IMAGEN/subject/IMAGEN_000055982442/experiments
The first nice thing here is that you actually don't have to worry about
resource level to be plural or singular within the path::
/project/IMAGEN/subject/IMAGEN_000055982442
EQUALS
/projects/IMAGEN/subjects/IMAGEN_000055982442
/project/IMAGEN/subject/IMAGEN_000055982442/experiments
EQUALS
/project/IMAGEN/subjects/IMAGEN_000055982442/experiment
Relative paths and shortcuts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When browsing resources, some levels are often left without any IDs
and filled with ``*`` filters instead, which leads to paths like::
/projects/*/subjects/*/experiments/*
That can instead be written::
/projects/subjects/experiments OR //experiments
To have all the experiments from a specific project::
/project/IMAGEN//experiments
EQUALS
/project/IMAGEN/subjects/*/experiments
The double slash syntax can be used anywhere in the path and any
number of time::
//subjects//assessors
EQUALS
/projects/*/subjects/*/experiments/*/assessors/*
Sometimes, a path will generate more than one path because it can be
interpreted in different way::
//subjects//assessors//files
Generates::
/projects/*/subjects/*/experiments/*/assessors/*/in_resources/*/files/*
/projects/*/subjects/*/experiments/*/assessors/*/out_resources/*/files/*
/projects/*/subjects/*/experiments/*/assessors/*/resources/*/files/*
.. warning::
If you try ``//files``, it will generate all the possible descendant paths:
| /projects/*/subjects/*/experiments/*/resources/*/files/*
| /projects/*/subjects/*/experiments/*/reconstructions/*/in_resources/*/files/*
| /projects/*/subjects/*/experiments/*/scans/*/resources/*/files/*
| /projects/*/subjects/*/experiments/*/assessors/*/out_resources/*/files/*
| /projects/*/subjects/*/resources/*/files/*
| /projects/*/resources/*/files/*
| /projects/*/subjects/*/experiments/*/reconstructions/*/out_resources/*/files/*
| /projects/*/subjects/*/experiments/*/assessors/*/in_resources/*/files/*
| /projects/*/subjects/*/experiments/*/assessors/*/resources/*/files/*
If the server has decent amount a data it will take ages to go through all the resources.
Resources operations
--------------------
Several operations are accessible for every resource level. The most
importants are responsible for creating new resources, deleting
existing ones and testing whether a given resource exists or not::
>>> my_project = central.select.project('my_project')
>>> my_project.exists()
False
>>> my_project.create()
>>> my_project.exists()
True
>>> subject = my_project.subject('first_subject')
>>> subject.create()
>>> subject.delete()
>>> subject.exists()
False
An optional keyword argument is available to specify the datatype from
the XNAT schema. The keyword must match the name of the REST level.
>>> subject.create()
>>> subject.experiment('pet_session'
).create(experiments='xnat:petSessionData')
It is also possible to create resources without having to create the
parent resources first. For example::
>>> central.select('/project/PROJECT/subject/SUBJECT').exists()
False
>>> central.select('/project/PROJECT/subject/SUBJECT/experiment/EXP').exists()
False
Specifiy the datatype on multiple levels::
>>> central.select('/project/PROJECT/subject/SUBJECT/experiment/EXP/scan/SCAN'
).create(experiments='xnat:mrSessionData', scans='xnat:mrScanData')
Use default datatypes::
>>> central.select('/project/PROJECT/subject/SUBJECT/experiment/EXP/scan/SCAN'
).create()
Additional fields can be configured at the resource creation. It can
be especially useful for datatypes that have some mandatory fields,
and thus would not be created if not specified (this is not a best
practice for XML Schema writers though). It also enables users to set
the resource ID through the REST API instead of just the label (the ID
in this case is generated automatically).
Custom ID example::
>>> experiment.create(experiments='xnat:mrSessionData',
ID='my_custom_ID'
)
With additional fields::
>>> experiment.create(**{'experiments':'xnat:mrSessionData',
'ID':'mr_custom_ID',
'xnat:mrSessionData/age':'42'}
)
.. warning:: When using xpath syntax to declare fields, it is
mandatory to pass the arguments using a dictionnary because of
the ``/`` and ``:`` characters. And do not forget to expand
the dict with the ``**``.
Since you can create different resource levels in a single create call
in pyxnat, it is also possible to configure those levels in a single
call. For example if the subject for that experiment was not created,
you could have specified::
>>> experiment.create(
**{'experiments':'xnat:mrSessionData',
'ID':'mr_custom_ID',
'xnat:mrSessionData/age':'42',
'xnat:subjectData/investigator/lastname':'doe',
'xnat:subjectData/investigator/firstname':'john',
'xnat:subjectData/ID':'subj_custom_ID'
})
File support
------------
It is possible to upload and then download files at every REST resource level::
>>> my_project.files()
[]
>>> my_project.file('image.nii').put('/tmp/image.nii')
>>> # you can add any of the following arguments to give additional
>>> # information on the file you are uploading
>>> my_project.file('image.nii').put( '/tmp/image.nii',
content='T1',
format='NIFTI'
tags='image test'
)
>>> my_project.resource('NIFTI').file('image.nii').size()
98098
>>> my_project.resource('NIFTI').file('image.nii').content()
'T1'
>>> my_project.resource('NIFTI').file('image.nii').format()
'NIFTI'
>>> my_project.resource('NIFTI').file('image.nii').tags()
'image test'
>>> my_project.resource('NIFTI').file('image.nii').get()
'~/.store/nosetests@central.xnat.org/c7a5b961fc504ffc9aa292f76d75fb0c_image.nii'
>>> my_project.file('image.nii').get_copy()
'~/.store/nosetests@central.xnat.org/workspace/projects/Volatile/resources/123150742/files/image.nii'
>>> my_project.file('image.nii').get_copy('/tmp/test.nii')
'/tmp/test.nii'
>>> # the resource level can be used to group files
>>> my_project.resource('ANALYZE').file('image.hdr').put('/tmp/image.hdr')
>>> my_project.resource('ANALYZE').file('image.img').put('/tmp/image.img')
>>> my_project.resources()
['NIFTI', 'ANALYZE']
>>> my_project.resource('ANALYZE').files()
['image.hdr', 'image.img']
.. tip::
New since 0.7, the default ``get()`` method on a file can be given
a custom path. It will still be handled and tracked by the cache in
the same way as other files.
Attributes support
------------------
Each resource level also has a set of metadata fields that can be
informed. This set of fields depends on the resource level and on its
type in the XNAT schema.
>>> # use hard-coded shortcuts from the REST API
>>> my_project.attrs.set('secondary_ID', 'myproject')
>>> my_project.attrs.get('secondary_ID')
'myproject'
>>> # use XPATH from standard or custom XNAT Schema
>>> my_project.attrs.set('xnat:projectData/keywords', 'test project')
>>> my_project.attrs.get('xnat:projectData/keywords')
'test project'
>>> # get or set multiple attributes in a single request to improve performance
>>> my_project.attrs.mset({'xnat:projectData/keywords':'test project', 'secondary_ID':'myproject'})
>>> my_project.attrs.mget(['xnat:projectData/keywords', 'secondary_ID'])
['test porject', 'myproject']
_____
.. [#] http://www.xnat.org/XNAT+REST+XML+Path+Shortcuts
The search engine
------------------
The XNAT search engine can be queried via the REST model. It can be
used to retrieve a specific subset of REST resources or a table
containing the relevant values. The following queries find all the
subjects that are within `my_project` or that have an age superior to
14::
>>> contraints = [('xnat:subjectData/SUBJECT_ID','LIKE','%'),
('xnat:subjectData/PROJECT', '=', 'my_project'),
'OR',
[('xnat:subjectData/AGE','>','14'),
'AND'
]
]
>>> # retrieve experiments
>>> interface.select('//experiments').where(contraints)
>>> # retrieve table with one subject per row and the columns SUBJECT_ID and AGE
>>> interface.select('xnat:subjectData', ['xnat:subjectData/SUBJECT_ID', 'xnat:subjectData/AGE']).where(contraints)
See the ``Search``, ``SeachManager`` and ``CObject`` classes reference
documentation for further details.
To get the searchable types and fields to put in the contraints, rows
and columns parameters, use the ``Interface.inspect.datatypes``
method::
>>> central.inspect.datatypes(optional_filter)
[..., 'xnat:subjectData', 'xnat:projectData', 'xnat:mrSessionData', ...]
>>> central.inspect.datatypes('xnat:subjectData', optional_filter)
['xnat:subjectData/SUBJECT_ID',
'xnat:subjectData/INSERT_DATE',
'xnat:subjectData/INSERT_USER',
'xnat:subjectData/GENDER_TEXT',
...]
.. tip::
How to get all the results in a query?
>>> interface.select('xnat:subjectData',
['xnat:subjectData/SUBJECT_ID',
'xnat:subjectData/AGE']).all()
.. tip::
How to get all the columns from a datatype?
>>> table = interface.select('xnat:subjectData').where(...)
.. tip::
Then to get everything:
>>> table = interface.select('xnat:subjectData').all()
pyxnat-0.9.0~dev0/doc/under_the_hood.rst 0000664 0000000 0000000 00000005046 11632421554 0020315 0 ustar 00root root 0000000 0000000 ====================
Under the Hood
====================
The REST model
--------------
Check out the wikipedia page if you are not familiar with REST.
_____
.. [#] http://en.wikipedia.org/wiki/REST
The REST model in XNAT
----------------------
Basically the resources are accessible pairing a level and its ID in a sequence
representing a URI starting by /REST.
Uniquely identifiable resources:
- /REST/projects/{ID}
- /REST/projects/{ID}/subjects/{ID}
Listing the resources is done by not giving an ID at the end of a URI:
- /REST/projects
- /REST/projects/{ID}/subjects
Requesting specific data in specific format is done with a query string:
- /REST/projects?format=csv
- /REST/projects/{ID}?format=xml
_____
.. [#] http://www.xnat.org/Web+Services
REST in Python
--------------
The REST resource levels supported by this Python library are the following::
- projects
- subjects
- experiments
- scans
- reconstructions
- assessors
Attached to each resource level are::
- resources
- files
Every level corresponds to a Python class sharing common methods from
ResourceObject and adding methods relevant to the specific resource.
+-----------------+----------------+
| REST resource | Python Class |
+=================+================+
| REST | Interface |
+-----------------+----------------+
| projects | Project |
+-----------------+----------------+
| subjects | Subject |
+-----------------+----------------+
| experiments | Experiment |
+-----------------+----------------+
| scans | Scan |
+-----------------+----------------+
| reconstructions | Reconstruction |
+-----------------+----------------+
| assessors | Assessor |
+-----------------+----------------+
| resources | Resource |
+-----------------+----------------+
| files | File |
+-----------------+----------------+
This classes may but should not be instantiated manually.
The HTTP requests are handled by a slightly modified version of httplib2, which is
shipped the package. The main fix was made to enable back the built-in HTTP cache
to avoid downloading again files that are already in the cache. This changes nothing
for other resources that don't have to necessary HTTP headers to enable a cache service.
pyxnat-0.9.0~dev0/pyxnat/ 0000775 0000000 0000000 00000000000 11632421554 0015346 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/pyxnat/__init__.py 0000664 0000000 0000000 00000004530 11632421554 0017461 0 ustar 00root root 0000000 0000000 """ pyxnat is a simple python library that relies on the REST API provided
by the XNAT platform since its 1.4 version. XNAT is an extensible database for neuroimaging data. The main objective is to ease communications with an XNAT
server to plug-in external tools or python scripts to process the data. It
features:
#. resources browsing capabilities
#. read and write access to resources
#. complex searches
#. disk-caching of requested files and resources
.. [#] XNAT home: http://www.xnat.org/
.. [#] pyxnat documentation: http://packages.python.org/pyxnat/
.. [#] pyxnat download: http://pypi.python.org/pypi/pyxnat#downloads
.. [#] pyxnat sources: http://github.com/pyxnat/pyxnat
____
**A short overview**
*Setup the connection*
>>> from pyxnat import Interface
>>> interface = Interface(
server='http://central.xnat.org:8080',
user='login',
password='pass',
cachedir=os.path.join(os.path.expanduser('~'), '.store')
)
*Browse the resources*
>>> interface.select.projects().get()
[u'CENTRAL_OASIS_CS', u'CENTRAL_OASIS_LONG', ...]
*Create new resources*
>>> interface.select.project('my_project').create()
>>> interface.select.project('my_project').resource('images'
).file('image.nii').put('/tmp/image.nii')
*Make complex searches*
>>> table = interface.select('xnat:subjectData',
['xnat:subjectData/PROJECT',
'xnat:subjectData/SUBJECT_ID'
]
).where([('xnat:subjectData/SUBJECT_ID','LIKE','%'),
('xnat:subjectData/PROJECT', '=', 'my_project'),
'AND'
])
"""
__version__ = '0.9.0dev'
from .core import Interface
from .core import SearchManager
from .core import CacheManager
from .core import Select
from .core import Inspector
from .core import Users
from .core import attributes
from .core import cache
from .core import help
from .core import interfaces
from .core import resources
from .core import schema
from .core import select
from .core import users
from .core import jsonutil
from .core import uriutil
from .core import xpass
from .core import xpath_store
pyxnat-0.9.0~dev0/pyxnat/core/ 0000775 0000000 0000000 00000000000 11632421554 0016276 5 ustar 00root root 0000000 0000000 pyxnat-0.9.0~dev0/pyxnat/core/__init__.py 0000664 0000000 0000000 00000000312 11632421554 0020403 0 ustar 00root root 0000000 0000000 import os
import sys
from .interfaces import Interface
from .search import SearchManager
from .cache import CacheManager
from .select import Select
from .help import Inspector
from .users import Users
pyxnat-0.9.0~dev0/pyxnat/core/array.py 0000664 0000000 0000000 00000015056 11632421554 0017775 0 ustar 00root root 0000000 0000000 from .jsonutil import JsonTable
class ArrayData(object):
def __init__(self, interface):
self._intf = interface
def _get_array(self, query_string, project_id=None,
subject_id=None, subject_label=None,
experiment_id=None, experiment_label=None,
experiment_type='xnat:imageSessionData',
columns=None, constraints=None
):
if constraints is None:
constraints = {}
uri = '/data/experiments?xsiType=%s' % experiment_type
if project_id is not None:
uri += '&project=%s' % project_id
if subject_id is not None:
uri += '&%s/subject_id=%s' % (experiment_type, subject_id)
if subject_label is not None:
uri += '&%s/subject_label=%s' % (experiment_type, subject_label)
if experiment_id is not None:
uri += '&ID=%s' % experiment_id
if experiment_label is not None:
uri += '&label=%s' % experiment_label
uri += query_string
if constraints != {}:
uri += ',' + ','.join(constraints.keys())
if columns is not None:
uri += ',' + ','.join(columns)
c = {}
[c.setdefault(key.lower(), value)
for key, value in constraints.items()
]
return JsonTable(self._intf._get_json(uri)).where(**c)
def experiments(self, project_id=None, subject_id=None, subject_label=None,
experiment_id=None, experiment_label=None,
experiment_type='xnat:mrSessionData',
columns=None,
constraints=None
):
""" Returns a list of all visible experiment IDs of the specified
type, filtered by optional constraints.
Parameters
----------
project_id: string
Name pattern to filter by project ID.
subject_id: string
Name pattern to filter by subject ID.
subject_label: string
Name pattern to filter by subject ID.
experiment_id: string
Name pattern to filter by experiment ID.
experiment_label: string
Name pattern to filter by experiment ID.
experiment_type: string
xsi path type; e.g. 'xnat:mrSessionData'
constraints: dict
Dictionary of xsi_type (key--) and parameter (--value)
pairs by which to filter.
"""
query_string = '&columns=ID,project,%s/subject_id' % experiment_type
return self._get_array(query_string, project_id,
subject_id, subject_label,
experiment_id, experiment_label,
experiment_type, constraints
)
def scans(self, project_id=None, subject_id=None, subject_label=None,
experiment_id=None, experiment_label=None,
experiment_type='xnat:mrSessionData',
scan_type='xnat:mrScanData',
columns=None,
constraints=None
):
""" Returns a list of all visible scan IDs of the specified type,
filtered by optional constraints.
Parameters
----------
project_id: string
Name pattern to filter by project ID.
subject_id: string
Name pattern to filter by subject ID.
subject_label: string
Name pattern to filter by subject ID.
experiment_id: string
Name pattern to filter by experiment ID.
experiment_label: string
Name pattern to filter by experiment ID.
experiment_type: string
xsi path type; e.g. 'xnat:mrSessionData'
scan_type: string
xsi path type; e.g. 'xnat:mrScanData', etc.
constraints: dict
Dictionary of xsi_type (key--) and parameter (--value)
pairs by which to filter.
"""
query_string = '&columns=ID,project,%s/subject_id,%s/ID' % (
experiment_type, scan_type)
return self._get_array(query_string, project_id,
subject_id, subject_label,
experiment_id, experiment_label,
experiment_type, constraints
)
def search_experiments(self,
project_id=None,
subject_id=None,
subject_label=None,
experiment_type='xnat:mrSessionData',
columns=None,
constraints=None
):
""" Returns a list of all visible experiment IDs of the
specified type, filtered by optional constraints. This
function is a shortcut using the search engine.
Parameters
----------
project_id: string
Name pattern to filter by project ID.
subject_id: string
Name pattern to filter by subject ID.
subject_label: string
Name pattern to filter by subject ID.
experiment_type: string
xsi path type must be a leaf session type.
defaults to 'xnat:mrSessionData'
columns: List[string]
list of xsi paths for names of columns to return.
constraints: list[(tupple)]
List of tupples for comparison in the form (key, comparison, value)
valid comparisons are: =, <, <=,>,>=, LIKE
"""
if columns is None:
columns = []
where_clause = []
if project_id is not None:
where_clause.append(('%s/project' % experiment_type, "=", project_id))
if subject_id is not None:
where_clause.append(('xnat:subjectData/ID', "=", subject_id))
if subject_label is not None:
where_clause.append(('xnat:subjectData/LABEL', "=", subject_label))
if constraints is not None:
where_clause.extend(constraints)
if where_clause != []:
where_clause.append('AND')
if where_clause != []:
table = self._intf.select(experiment_type, columns=columns).where(where_clause)
return table
else:
table = self._intf.select(experiment_type, columns=columns)
return table.all()
pyxnat-0.9.0~dev0/pyxnat/core/attributes.py 0000664 0000000 0000000 00000014243 11632421554 0021042 0 ustar 00root root 0000000 0000000 import difflib
import urllib
from .jsonutil import JsonTable
from .uriutil import uri_parent
from .schema import datatype_attributes
class EAttrs(object):
""" Accessor class to resource fields.
Help to retrieve the attributes paths relevant to this element::
>>> subject.attrs()
['xnat:subjectData/sharing',
'xnat:subjectData/sharing/share',
'xnat:subjectData/resources',
...
'xnat:subjectData/experiments/experiment'
]
All paths are not valid but they give an indication of what
is available. To retrieve the paths, the corresponding
schemas must be downloaded first through the schema
management interface in order to be parsed::
>>> interface.manage.schemas.add('xnat.xsd')
>>> interface.manage.schemas.add('myschema/myschema.xsd')
"""
def __init__(self, eobj):
"""
Parameters
----------
eobj:
:class:`EObject` Object
"""
self._eobj = eobj
self._intf = eobj._intf
self._datatype = None
self._id = None
def __call__(self):
""" List the attributes paths relevant to this element.
"""
paths = []
self._intf.manage.schemas._init()
for root in self._intf.manage.schemas._trees.values():
paths.extend(datatype_attributes(root, self._get_datatype()))
return paths
def _get_datatype(self):
if self._datatype is None:
self._datatype = self._eobj.datatype()
return self._datatype
def _get_id(self):
if self._id is None:
self._id = self._eobj.id()
return self._id
def set(self, path, value):
""" Set an attribute.
Parameters
----------
path: string
The xpath of the attribute relative to the element.
value: string
The attribute's value. Note that the python type is
always a string but the content of the value must
match what is defined in the schema.
e.g. an element defined as a float in the schema
must be given a string containing a number, a
valid date must follow the ISO 8601 which is the
standard representation for dates and times
established by the W3C.
"""
put_uri = self._eobj._uri+'?%s=%s' % (urllib.quote(path),
urllib.quote(value)
)
self._intf._exec(put_uri, 'PUT')
def mset(self, dict_attrs):
""" Set multiple attributes at once.
It is more efficient to use this method instead of
multiple times the `set()` method when setting more than
one attribute because only a single HTTP call is issued to
the server.
Parameters
----------
dict_attrs: dict
The dict of key values to set. It follows the same
principles as the single `set()` method.
"""
query_str = '?' + '&'.join(['%s=%s' % (urllib.quote(path),
urllib.quote(val)
)
for path, val in dict_attrs.items()
]
)
put_uri = self._eobj._uri + query_str
self._intf._exec(put_uri, 'PUT')
def get(self, path):
""" Get an attribute value.
.. note::
The value is always returned in a Python string. It must
be explicitly casted or transformed if needed.
Parameters
----------
path: string
The xpath of the attribute relative to the element.
Returns
-------
A string containing the value.
"""
query_str = '?columns=ID,%s' % path
get_uri = uri_parent(self._eobj._uri) + query_str
jdata = JsonTable(self._intf._get_json(get_uri)
).where(ID=self._get_id())
# unfortunately the return headers do not always have the
# expected name
header = difflib.get_close_matches(path.split('/')[-1],
jdata.headers()
)
if header == []:
header = difflib.get_close_matches(path, jdata.headers())[0]
else:
header = header[0]
replaceSlashS = lambda x : x.replace('\s',' ')
if type(jdata.get(header)) == list:
return map(replaceSlashS, jdata.get(header))
else:
return jdata.get(header).replace('\s', ' ')
def mget(self, paths):
""" Set multiple attributes at once.
It is more efficient to use this method instead of
multiple times the `get()` method when getting more than
one attribute because only a single HTTP call is issued to
the server.
Parameters
----------
paths: list
List of attributes' paths.
Returns
-------
list: ordered list of values (in the order of the
requested paths)
"""
query_str = '?columns=ID,%s' % ','.join(paths)
get_uri = uri_parent(self._eobj._uri) + query_str
jdata = JsonTable(self._intf._get_json(get_uri)
).where(ID=self._get_id())
results = []
# unfortunately the return headers do not always have the
# expected name
for path in paths:
header = difflib.get_close_matches(path.split('/')[-1],
jdata.headers())
if header == []:
header = difflib.get_close_matches(path, jdata.headers())[0]
else:
header = header[0]
results.append(jdata.get(header).replace('\s', ' '))
return results
pyxnat-0.9.0~dev0/pyxnat/core/cache.py 0000664 0000000 0000000 00000027005 11632421554 0017717 0 ustar 00root root 0000000 0000000 from __future__ import with_statement
import os
import platform
import ctypes
import glob
import hashlib
import re
import time
import shutil
from StringIO import StringIO
_platform = platform.system()
DEBUG = False
def md5name(key):
""" Generates a unique path to store server responses.
"""
ext = re.findall('.*?(?=(\?format=.+?(?=\?|&|$)|'
'\&format=.+?(?=\?|&|$)))', key)
if ext != []:
last = key.split('?')[0].split('/')[-1]
key = key.replace(ext[0], '')
ext = ext[0].split('format')[1]
key += ext
last += ext
else:
last = key.split('/')[-1]
return '%s_%s' % (hashlib.md5(key).hexdigest(), last.replace('=', '.'))
def bytes_to_human(size, unit):
""" Returns a more human readable version of a size in bytes.
"""
if unit == 'mega':
return float(size) / 1024 ** 2
elif unit == 'giga':
return float(size) / 1024 ** 3
else:
return size
def memstr_to_bytes(text):
""" Convert a memory text to its value in kilobytes.
"""
kilo = 1024**2
units = dict(K=1, M=kilo, G=kilo**2)
try:
size = int(units[text[-1]]*float(text[:-1]))
except (KeyError, ValueError):
raise ValueError(
"Invalid literal for size give: %s (type %s) should be "
"alike '10G', '500M', '50K'." % (text, type(text))
)
return size
class HTCache(object):
def __init__(self, cachedir, interface, safe=md5name):
"""
Parameters
----------
cachedir: string
The cache path.
interface:
:class:`Interface` Object
safe: callable
The function used to generate the responses cache paths.
"""
self._intf = interface
self.cache = cachedir
self.safe = safe
self._cachepath = None
if not os.path.exists(cachedir):
os.makedirs(cachedir)
def get(self, key):
retval = None
_cachepath = os.path.join(self.cache, self.safe(key))
if DEBUG:
print 'cache get:', key,
print '\n\t', _cachepath
try:
f = file('%s.headers' % _cachepath, "rb")
retval = f.read()
f.close()
f = file(_cachepath, "rb")
retval += f.read()
f.close()
except IOError:
try:
f = file('%s.alt' % _cachepath, "rb")
_altpath = f.read()
f.close()
f = file(_altpath, "rb")
retval += f.read()
f.close()
except:
return
return retval
def set(self, key, value):
""" Sets cache entry.
"""
_fakepath = '%s.alt' % os.path.join(self.cache, self.safe(key))
_headerpath = '%s.headers' % os.path.join(self.cache, self.safe(key))
_cachepath = os.path.join(self.cache, self.safe(key))
if self._cachepath is not None: # when using custom path
if _cachepath != self._cachepath and os.path.exists(_cachepath):
os.remove(_cachepath) # remove default file if exists
if os.path.exists(_fakepath):
f = file(_fakepath, "rb")
_altpath = f.read()
f.close()
if _altpath != self._cachepath and os.path.exists(_altpath):
# remove old custom file if different location
os.remove(_altpath)
_cachepath = self._cachepath
if DEBUG:
print 'cache set custom:', key
print '\n\t', _cachepath
f = open(_fakepath, 'w')
f.write(_cachepath)
f.close()
self._cachepath = None
else: # when using default cache path
if os.path.exists(_fakepath): # if alternate location exists
f = file(_fakepath, "rb")
_altpath = f.read()
f.close()
if _altpath != self._cachepath and os.path.exists(_altpath):
os.remove(_altpath) # remove actual file
os.remove(_fakepath) # remove pointer file
if DEBUG:
print 'cache set default:', key
print '\n\t', _cachepath
header = ''
value = StringIO(value)
# avoid checking disk status each time
if time.gmtime(time.time())[5] % 10 == 0:
disk_status = self._intf.cache.disk_ready(_cachepath)
if not disk_status[0] and self._intf.cache._warn:
print 'Warning: %s is %.2f%% full' % \
(os.path.dirname(_cachepath), disk_status[1])
while len(header) < 4 or header[-4:] != '\r\n\r\n':
header += value.read(1)
content = value.read()
f = file(_headerpath, "wb")
f.write(header)
f.close()
f = file(_cachepath, "wb")
f.write(content)
f.close()
def preset(self, path):
""" Sets and forces a path for the next entry to be set.
.. note::
Basically it is a way to trick the cache mechanism
into using something else than the default path to
store entries.
"""
if self._intf._mode != 'offline':
self._cachepath = path
def delete(self, key):
""" Deletes the entry.
"""
if DEBUG:
print 'cache del:', key
_cachepath = os.path.join(self.cache, self.safe(key))
_fakepath = '%s.alt' % _cachepath
_headerpath = '%s.headers' % _cachepath
if os.path.exists(_fakepath):
f = file(_fakepath, "rb")
_altpath = f.read()
f.close()
os.remove(_fakepath)
if os.path.exists(_altpath):
os.remove(_altpath)
if os.path.exists(_cachepath):
os.remove(_cachepath)
if os.path.exists(_headerpath):
os.remove(_headerpath)
def get_diskpath(self, key, force_default=False):
""" Gets the disk path where the entry is stored (default path
or not)
"""
_cachepath = os.path.join(self.cache, self.safe(key))
_fakepath = '%s.alt' % _cachepath
if os.path.exists(_fakepath) and not force_default:
f = file(_fakepath, "rb")
_altpath = f.read()
f.close()
return _altpath
else:
return _cachepath
class CacheManager(object):
""" Management interface for the cache.
It provides a few methods to::
- evaluate the size a the cache
- check if there is space left on the disk
- clear the cache
- define cache usage parameters
"""
def __init__(self, interface):
"""
Parameters
----------
interface:
:class:`Interface` Object
"""
self._intf = interface
self._cache = interface._http.cache
self._warn = True
def enable_warnings(self, toggle=True):
self._warn = toggle
def clear(self):
""" Clears all files tracked by pyxnat.
"""
for _fakepath in glob.iglob('%s/*.alt' % self._cache.cache):
f = file(_fakepath, "rb")
_altpath = f.read()
f.close()
if os.path.exists(_altpath):
os.remove(_altpath)
shutil.rmtree(self._cache.cache)
os.makedirs(self._cache.cache)
def size(self, unit='bytes'):
""" Returns the amount of space taken by the cache.
Parameters
----------
unit: str
unit in which to return the size
can be bytes (default), mega or giga
Returns
-------
size: float
"""
size = 0
for cachefile in glob.iglob('%s/*' % self._cache.cache):
size += os.path.getsize(cachefile)
if cachefile.endswith('.alt'):
f = file(cachefile, "rb")
_altpath = f.read()
f.close()
if os.path.exists(_altpath):
size += os.path.getsize(_altpath)
return bytes_to_human(size, unit)
def available_disk(self, path=None, unit='bytes'):
""" Available disk on partition. Default location is cache folder.
"""
if path is None:
path = self._cache.cache
if not os.path.isdir(path):
path = os.path.dirname(path.rstrip('/'))
if _platform == 'Windows':
available = ctypes.c_ulonglong(0)
ctypes.windll.kernel32.GetDiskFreeSpaceExW(
ctypes.c_wchar_p(path),
None,
None,
ctypes.pointer(available))
return bytes_to_human(available.value, unit)
else:
cache_st = os.statvfs(path)
return bytes_to_human(cache_st.f_bavail * cache_st.f_bsize, unit)
def used_disk(self, path=None, unit='bytes'):
""" Used disk on partition. Default location is cache folder.
"""
if path is None:
path = self._cache.cache
return self.total_disk(path, unit) - self.available_disk(path, unit)
def total_disk(self, path=None, unit='bytes'):
""" Total disk on partition. Default location is cache folder.
"""
if path is None:
path = self._cache.cache
if not os.path.isdir(path):
path = os.path.dirname(path.rstrip('/'))
if _platform == 'Windows':
total = ctypes.c_ulonglong(0)
ctypes.windll.kernel32.GetDiskFreeSpaceExW(
ctypes.c_wchar_p(path),
None,
ctypes.pointer(total),
None,
)
return bytes_to_human(total.value, unit)
else:
cache_st = os.statvfs(path)
return bytes_to_human(cache_st.f_blocks * cache_st.f_bsize, unit)
def disk_ready(self, path=None, ready_ratio=90.0):
""" Checks the status of the disk. Basically if there is enough space
left. The default location is cache dir.
"""
if path is None:
path = self._cache.cache
disk_ratio = ( float(self.used_disk(path)) / \
float(self.total_disk(path)) * 100
)
return disk_ratio < ready_ratio, disk_ratio
def set_usage(self, mode=None, expiration=1.0):
""" Customize cache usage.
Parameters
----------
mode: string
'online' or 'offline':
- online will always query the server to have up
to date data.
- offline will only try to query the server if the
data is not cached.
expiration: float
Relevant only to online mode. The cache has an
expiration mechanism. If two queries on the same
resource are issued under the specified value, the
cache will be used and the server will not be
requested.
"""
if mode == 'online':
self._intf._mode = 'online'
self._intf._memtimeout = expiration
elif mode == 'offline':
self._intf._mode = 'offline'
self._intf._memtimeout = expiration
else:
self._intf._memtimeout = expiration
pyxnat-0.9.0~dev0/pyxnat/core/downloadutils.py 0000664 0000000 0000000 00000014773 11632421554 0021554 0 ustar 00root root 0000000 0000000 import os
import zipfile
from .schema import class_name
from . import uriutil
from . import schema
def unzip(fzip,
dest_dir,
check={'run':lambda z,d: True,'desc':""}):
"""
Extracts the given zip file to the given directory, but only if all members of the
archive pass the given check.
Parameters
----------
src: fzip
zipfile
dest_dir: string
directory into which to extract the archive
check: dict
An dictionary that has the keys:
'run' : A function that takes a filename and parent directory and returns Bool. By default
this function always returns True.
'dest' : A string description of this test. By default this is empty.
Returns a tuple of type (bool,[string]) where if the extraction ran successfully the first is true and the
second is a list of files that were extracted, and if not the first is false and the second is the name
of the failing member.
"""
for member in fzip.namelist():
if not check['run'](member,dest_dir):
return (False,member)
fzip.extractall(path=dest_dir)
return (True, map (lambda f: os.path.join(dest_dir,f),fzip.namelist()))
def download (dest_dir, instance=None,type="ALL",name=None, extract=False, safe=False):
"""
Download all the files at this level that match the given constraint as a zip archive. Should not be called directly
but from a instance of class that supports bulk downloading eg. "Scans"
Parameters
----------
instance : 'object
The instance that contains local values needed by this function eg. instance._cbase stores the URI.
dest_dir : string
directory into which to place the downloaded archive
type: string
a comma separated list of file types, eg. "T1,T2". Default is "ALL"
name: string
the name of the zip archive. Defaults to None. See below for the default naming scheme
extract: bool
If True, the files are left extracted in the parent directory. Default is False
safe: bool
If true, run safety checks on the files before extracting, eg. check that the file
doesn't exist in the parent directory before overwriting it. Default is False
Default Zip Name
----------------
Given the project "p", subject "s" and experiment "e", and that the "Scans" (as opposed to "Assessors"
or "Reconstructions") are being downloaded, and the scan types are constrained to "T1,T2",
the name of the zip file defaults to:
"p_s_e_scans_T1_T2.zip"
Exceptions
----------
A generic Exception will be raised if any of the following happen:
- This function is called directly and not from an instance of a class that supports bulk downloading eg."Scans"
- The destination directory is unspecified
A LookupError is raised if there are no resources to download
A ValueError is raised if any of the following happen:
- The project, subject and experiment names could not be extracted from the URI
- The type constraint "ALL" is used with other constraints. eg. "ALL,T1,T2"
- The URI associated with this class contains wildcards eg. /projects/proj/subjects/*/experiments/scans
An EnvironmentError is raised if any of the following happen:
- If "safe" is true, and (a) a zip file with the same name exists in given destination directory or
(b) extracting the archive overrides an existing file. In the second case the downloaded archive
is left in the parent directory.
Return
------
A path to the zip archive if "extract" is False, and a list of extracted files if True.
"""
if instance is None:
raise Exception('This function should be called directly but from an instance of a class that supports bulk downloading, eg. "Scans"')
if dest_dir is None:
raise Exception('Destination directory is unspecified')
# the URI must be fully qualified with a project,subject and experiment
if '%2A' in instance._cbase:
raise ValueError('URI contains wildcards :' + instance._cbase)
# Check that there are resources at this level
available = instance.get()
if len(available) == 0:
raise LookupError(
'There are no %s to download' % class_name(instance).lower())
pse = uriutil.extract_uri(instance._cbase)
if pse is None:
raise ValueError("Could not extract project, subject and experiment from the uri: " + instance._cbase)
# Extract the desired scan types. Clean up whitespace and remove dupes
types = {}
for t in type.split(','):
cleaned = t.strip()
if cleaned != "":
types[cleaned] = cleaned
# Make sure the user hasn't asked us to download ALL the scans and then asked
# for them to be constrained to a type.
if len(types) > 1 and 'ALL' in types:
raise ValueError('The \"ALL\" scan type constraint cannot be used with any other constraint')
(p,s,e) = pse
# Make the name of the zip file
default_zip_name = lambda: '%s_%s_%s_%s_%s' % (
p, s, e, class_name(instance).lower(), '_'.join(types.values()))
zip_name = name if name != None else default_zip_name()
zip_location = os.path.join(dest_dir, zip_name + '.zip')
if safe:
if os.path.exists(zip_location):
raise EnvironmentError("Unable to download to " + zip_location + " because this file already exists.")
# Download from the server
instance._intf._http.cache.preset(zip_location)
instance._intf._exec(uriutil.join_uri(
instance._cbase,','.join(types.values())) + '/files?format=zip')
# Extract the archive
fzip = zipfile.ZipFile(zip_location,'r')
if extract:
check = {'run': lambda f, d: not os.path.exists(os.path.join(dest_dir,f)),
'desc': 'File does not exist in the parent directory'}
safeUnzip = lambda: unzip(fzip, dest_dir, check) if safe else lambda:unzip(fzip,dest_dir)
(unzipped, paths) = safeUnzip()()
if not unzipped:
fzip.close()
raise EnvironmentError("Unable to extract " + zip_location + " because file " + paths + " failed the following test: " + check['desc'])
else:
return paths
else:
fzip.close()
return zip_location
pyxnat-0.9.0~dev0/pyxnat/core/errors.py 0000664 0000000 0000000 00000005535 11632421554 0020174 0 ustar 00root root 0000000 0000000 import re
import httplib2
# parsing functions
def is_xnat_error(message):
return message.startswith('')
def parse_error_message(message):
if message.startswith(''):
error = message.split('
')[0]
required_fields = []
for line in error.split('\n'):
try:
datatype_name = re.findall("\'.*?\'",line)[0].strip('\'')
element_name = re.findall("\'.*?\'",line
)[1].rsplit(':', 1)[1].strip('}\'')
required_fields.append((datatype_name, element_name))
except:
continue
return required_fields
def catch_error(msg_or_exception):
# handle errors returned by the xnat server
if isinstance(msg_or_exception, (str, unicode)):
# parse the message
msg = msg_or_exception
if msg.startswith(''):
error = msg.split('
')[0]
else:
error = msg
# choose the exception
if error == 'The request requires user authentication':
raise OperationalError('Authentication failed')
elif 'Not Found' in error:
raise OperationalError('Connection failed')
else:
raise DatabaseError(error)
# handle other errors, raised for instance by the http layer
else:
if isinstance(msg_or_exception, httplib2.ServerNotFoundError):
raise OperationalError('Connection failed')
else:
raise DatabaseError(str(msg_or_exception))
# Exceptions as defined in PEP-249, the module treats errors using thoses
# classes following as closely as possible the original definitions.
class Warning(StandardError):
pass
class Error(StandardError):
pass
class InterfaceError(Error):
pass
class DatabaseError(Error):
pass
class DataError(DatabaseError):
pass
class OperationalError(DatabaseError):
pass
class IntegrityError(DatabaseError):
pass
class InternalError(DatabaseError):
pass
class ProgrammingError(DatabaseError):
pass
class NotSupportedError(DatabaseError):
pass
pyxnat-0.9.0~dev0/pyxnat/core/help.py 0000664 0000000 0000000 00000053613 11632421554 0017610 0 ustar 00root root 0000000 0000000 import glob
try:
import networkx as nx
import matplotlib.pyplot as plt
_DRAW_GRAPHS = True
except:
_DRAW_GRAPHS = False
import json
from . import schema
from .jsonutil import get_column
from .search import Search
from .uriutil import check_entry
class Inspector(object):
""" Database introspection interface.
"""
def __init__(self, interface):
"""
Parameters
----------
interface:
:class:`Interface` Object
"""
self._intf = interface
self._get_json = interface._get_json
self._auto = True
self._tick = 30
self.schemas = SchemasInspector(interface)
def __call__(self):
for name in ['experiment', 'assessor', 'scan', 'reconstruction']:
self._resource_struct(name)
def set_autolearn(self, auto=None, tick=None):
""" Once in a while queries will persist additional
information on the server. This information is available
through the following methods of this class:
- experiment_types
- assessor_types
- scan_types
- reconstruction_types
It is also transparently used in insert operations.
Parameters
----------
auto: boolean
True to enable auto learn. False to disable.
tick: int
Every 'tick' seconds, if a query is issued, additional
information will be persisted.
See Also
--------
EObject.insert()
"""
if auto is not None:
self._auto = auto
if tick is not None:
self._tick = tick
def datatypes(self, pattern='*', fields_pattern=None):
""" Discovers the datatypes and datafields of the database.
Parameters
----------
pattern: string
Pattern for the datatype. May include wildcards.
fields_pattern: string
- Pattern for the datafields -- may include wildcards.
- If specified, datafields will be returned instead of
datatypes.
Returns
-------
list : datatypes or datafields depending on the argument usage.
"""
self._intf._get_entry_point()
search_els = self._get_json('%s/search/elements?format=json' %
self._intf._entry)
if not fields_pattern and ('*' in pattern or '?' in pattern):
return get_column(search_els , 'ELEMENT_NAME', pattern)
else:
fields = []
for datatype in get_column(search_els , 'ELEMENT_NAME', pattern):
fields.extend(self._datafields(datatype,
fields_pattern or '*', True)
)
return fields
def _datafields(self, datatype, pattern='*', prepend_type=True):
self._intf._get_entry_point()
search_fds = self._get_json('%s/search/elements/%s?format=json'
% (self._intf._entry, datatype)
)
fields = get_column(search_fds, 'FIELD_ID', pattern)
return ['%s/%s' % (datatype, field)
if prepend_type else field
for field in fields
if '=' not in field and 'SHARINGSHAREPROJECT' not in field
]
def experiment_types(self):
""" Returns the datatypes used at the experiment level in this
database.
See Also
--------
Inspector.set_autolearn()
"""
return self._resource_types('experiment')
def assessor_types(self):
""" Returns the datatypes used at the assessor level in this
database.
See Also
--------
Inspector.set_autolearn()
"""
return self._resource_types('assessor')
def reconstruction_types(self):
""" Returns the datatypes used at the reconstruction level in this
database.
See Also
--------
Inspector.set_autolearn()
"""
return self._resource_types('reconstruction')
def scan_types(self):
""" Returns the datatypes used at the scan level in this
database.
See Also
--------
Inspector.set_autolearn()
"""
return self._resource_types('scan')
def field_values(self, field_name):
""" Look for the values a specific datafield takes in the database.
"""
self._intf._get_entry_point()
search_tbl = Search(field_name.split('/')[0],
[field_name], self._intf
)
criteria = [('%s/ID' % field_name.split('/')[0], 'LIKE', '%'), 'AND']
return list(set([val
for entry in search_tbl.where(criteria)
for val in entry.values()
])
)
def project_values(self):
""" Look for the values a the project level in the database.
.. note::
Is equivalent to interface.select.projects().get()
"""
self._intf._get_entry_point()
return get_column(self._get_json(
'%s/projects' % self._intf._entry), 'ID')
def subject_values(self, project=None):
""" Look for the values a the subject level in the database.
.. note::
Is equivalent to interface.select('//subjects').get()
"""
self._intf._get_entry_point()
uri = '%s/subjects?columns=ID' % self._intf._entry
if project is not None:
uri += '&project=%s' % project
return get_column(self._get_json(uri), 'ID')
def experiment_values(self, datatype, project=None):
""" Look for the values a the experiment level for a given datatype
in the database.
.. note::
The datatype should be one of Inspector.experiment_types()
Parameters
----------
datatype: string
An experiment type.
project: string
Optional. Restrict operation to a project.
project: string
Optional. Restrict operation to a project.
"""
self._intf._get_entry_point()
uri = '%s/experiments?columns=ID' % self._intf._entry
if datatype is not None:
uri += '&xsiType=%s' % datatype
if project is not None:
uri += '&project=%s' % project
return get_column(self._get_json(uri), 'ID')
def assessor_values(self, experiment_type, project=None):
""" Look for the values at the assessor level for a given experiment
type in the database.
.. note::
The experiment type should be one of
Inspector.experiment_types()
.. warning::
Depending on the number of elements the operation may
take a while.
Parameters
----------
datatype: string
An experiment type.
project: string
Optional. Restrict operation to a project.
"""
return self._sub_experiment_values('assessor',
project, experiment_type)
def scan_values(self, experiment_type, project=None):
""" Look for the values at the scan level for a given experiment
type in the database.
.. note::
The experiment type should be one of
Inspector.experiment_types()
.. warning::
Depending on the number of elements the operation may
take a while.
Parameters
----------
datatype: string
An experiment type.
project: string
Optional. Restrict operation to a project.
"""
return self._sub_experiment_values('scan', project, experiment_type)
def reconstruction_values(self, experiment_type, project=None):
""" Look for the values at the reconstruction level for a given
experiment type in the database.
.. note::
The experiment type should be one of
Inspector.experiment_types()
.. warning::
Depending on the number of elements the operation may
take a while.
Parameters
----------
datatype: string
An experiment type.
project: string
Optional. Restrict operation to a project.
"""
return self._sub_experiment_values('reconstruction',
project, experiment_type)
def structure(self):
""" Displays the keywords structure used in XNAT REST API.
"""
def traverse(coll, lvl):
for key in schema.resources_tree[coll]:
print '%s+%s' % (' ' * lvl, key.upper())
datatypes = set([
self._intf._struct[uri]
for uri in self._intf._struct.keys()
if uri.split('/')[-2] == key
])
if datatypes != set():
print '%s %s' % (' ' * lvl, '-' * len(key))
for datatype in datatypes:
print '%s- %s' % (' ' * lvl, datatype)
if schema.resources_tree.has_key(key):
traverse(key, lvl + 4)
print '- %s' % 'PROJECTS'
traverse('projects', 4)
def _sub_experiment_values(self, sub_exp, project, experiment_type):
self._intf._get_entry_point()
values = []
column = '%s/%ss/%s/id' % \
(experiment_type.lower(), sub_exp, sub_exp)
sub_exps = '%s/experiments?columns=ID,%s' % (self._intf._entry,
column
)
if project is not None:
sub_exps += '&project=%s' % project
values = get_column(self._get_json(sub_exps), column)
return list(set(values))
def _resource_struct(self, name):
kbase = {}
for kfile in glob.iglob('%s/*.struct' % self._intf._cachedir):
kdata = json.load(open(kfile, 'rb'))
if kdata == {}:
continue
if name in kdata.keys()[0]:
kbase.update(kdata)
self._intf._struct.update(kbase)
return kbase
def _resource_types(self, name):
return list(set(self._resource_struct(name).values()))
class GraphData(object):
def __init__(self, interface):
self._intf = interface
self._struct = interface._struct
# def link(self, subjects, fields):
# criteria = [('xnat:subjectData/SUBJECT_ID', '=', _id)
# for _id in subjects
# ]
# criteria += ['OR']
# # variables = ['xnat:subjectData/SUBJECT_ID'] + fields
# subject_id = 'xnat:subjectData/SUBJECT_ID'
# for field in fields:
# field_tbl = self._intf.select('xnat:subjectData',
# [subject_id, field]
# ).where(criteria)
# head = field_tbl.headers()
# head.remove('subject_id')
# head = head[0]
# possible = set(field_tbl.get(head))
# groups = {}
# for val in possible:
# groups[val] = field_tbl.where(**{head:val}
# ).select('subject_id')
# return groups
def datatypes(self, pattern='*'):
graph = nx.DiGraph()
graph.add_node('datatypes')
graph.labels = {'datatypes':'datatypes'}
graph.weights = {'datatypes':100.0}
datatypes = self._intf.inspect.datatypes(pattern)
namespaces = set([dat.split(':')[0] for dat in datatypes])
for ns in namespaces:
graph.add_edge('datatypes', ns)
graph.weights[ns] = 70.0
for dat in datatypes:
if dat.startswith(ns):
graph.add_edge(ns, dat)
graph.weights[dat] = 40.0
return graph
def rest_resource(self, name):
resource_types = self._intf.inspect._resource_types(name)
graph = nx.DiGraph()
graph.add_node(name)
graph.labels = {name:name}
graph.weights = {name:100.0}
namespaces = set([exp.split(':')[0] for exp in resource_types])
for ns in namespaces:
graph.add_edge(name, ns)
graph.weights[ns] = 70.0
for exp in resource_types:
if exp.startswith(ns):
graph.add_edge(ns, exp)
graph.weights[exp] = 40.0
return graph
def field_values(self, field_name):
search_tbl = Search(field_name.split('/')[0],
[field_name], self._intf
)
criteria = [('%s/ID' % field_name.split('/')[0], 'LIKE', '%'), 'AND']
dist = {}
for entry in search_tbl.where(criteria):
for val in entry.values():
dist.setdefault(val, 1.0)
dist[val] += 1
graph = nx.Graph()
graph.add_node(field_name)
graph.weights = dist
graph.weights[field_name] = 100.0
for val in dist.keys():
graph.add_edge(field_name, val)
return graph
def architecture(self, with_datatypes=True):
graph = nx.DiGraph()
graph.add_node('projects')
graph.labels = {'projects':'projects'}
graph.weights = {'projects':100.0}
def traverse(lkw, as_lkw):
for key in schema.resources_tree[lkw]:
as_key = '%s_%s' % (as_lkw, key)
weight = (1 - len(as_key*2)/100.0) * 100
graph.add_edge(as_lkw, as_key)
graph.labels[as_key] = key
graph.weights[as_key] = weight
if with_datatypes:
for uri in self._struct.keys():
if uri.split('/')[-2] == key:
datatype = self._struct[uri]
graph.add_edge(as_key, datatype)
graph.weights[datatype] = 10
graph.labels[datatype] = datatype
traverse(key, as_key)
traverse('projects', 'projects')
return graph
class PaintGraph(object):
def __init__(self, interface):
self._intf = interface
self.get_graph = interface._get_graph
def architecture(self, with_datatypes=True, save=None):
graph = self.get_graph.architecture(with_datatypes)
plt.figure(figsize=(8,8))
pos = nx.graphviz_layout(graph, prog='twopi', args='')
# node_size = [(float(graph.degree(v)) * 5)**3 for v in graph]
# node_size = [graph.weights[v] ** 2 for v in graph]
# node_color = [float(graph.degree(v)) for v in graph]
# node_color = [graph.weights[v] ** 2 for v in graph]
cost = lambda v: float(graph.degree(v)) ** 3 + \
graph.weights[v] ** 2
costs = norm_costs([cost(v) for v in graph], 10000)
nx.draw(graph, pos, labels=graph.labels,
node_size=costs, node_color=costs,
font_size=13, font_color='orange', font_weight='bold'
)
plt.axis('off')
if save is not None:
plt.savefig(save)
plt.show()
def experiments(self, save=None):
graph = self.get_graph.rest_resource('experiments')
self._draw_rest_resource(graph, save=None)
def assessors(self, save=None):
graph = self.get_graph.rest_resource('assessors')
self._draw_rest_resource(graph, save=None)
def reconstructions(self, save=None):
graph = self.get_graph.rest_resource('reconstructions')
self._draw_rest_resource(graph, save=None)
def scans(self):
graph = self.get_graph.rest_resource('scans')
self._draw_rest_resource(graph)
def _draw_rest_resource(self, graph, save=None):
plt.figure(figsize=(8,8))
pos = nx.graphviz_layout(graph, prog='twopi', args='')
cost = lambda v: float(graph.degree(v)) ** 3 + \
graph.weights[v] ** 2
node_size = [cost(v) for v in graph]
# node_size = [graph.weights[v] ** 2 for v in graph]
# node_color = [float(graph.degree(v)) for v in graph]
node_color = [cost(v) for v in graph]
nx.draw(graph, pos,
node_size=node_size, node_color=node_color,
font_size=13, font_color='green', font_weight='bold'
)
plt.axis('off')
if save is not None:
plt.savefig(save)
plt.show()
def datatypes(self, pattern='*', save=None):
graph = self.get_graph.datatypes(pattern)
plt.figure(figsize=(8,8))
pos = nx.graphviz_layout(graph, prog='twopi', args='')
cost = lambda v: float(graph.degree(v)) ** 3 + \
graph.weights[v] ** 2
node_size = [cost(v) for v in graph]
# node_size = [graph.weights[v] ** 2 for v in graph]
# node_color = [float(graph.degree(v)) for v in graph]
node_color = [cost(v) for v in graph]
nx.draw(graph, pos,
node_size=node_size, node_color=node_color,
font_size=13, font_color='green', font_weight='bold'
)
plt.axis('off')
if save is not None:
plt.savefig(save)
plt.show()
def field_values(self, field_name, save=None):
graph = self.get_graph.field_values(field_name)
plt.figure(figsize=(8,8))
pos = nx.graphviz_layout(graph, prog='twopi', args='')
cost = lambda v: graph.weights[v]
graph.weights[field_name] = max([cost(v) for v in graph]) / 2.0
costs = norm_costs([cost(v) for v in graph], 10000)
nx.draw(graph, pos,
node_size=costs, node_color=costs,
font_size=13, font_color='black', font_weight='bold'
)
plt.axis('off')
if save is not None:
plt.savefig(save)
plt.show()
def norm_costs(costs, norm=1000):
max_cost = max(costs)
return [ (cost / max_cost) * norm for cost in costs]
# class GraphDrawer(object):
# def __init__(self, interface):
# self._intf = interface
# def datatypes(self, project):
# self._intf.connection.set_strategy('offline')
# experiments_types = self._intf.inspect.datatypes.experiments(project)
# labels = {project:project, 'Experiments':'Experiments'}
# g = nx.Graph()
# g.add_edge(project, 'Experiments', {'weight':1})
# for exp_type in experiments_types:
# g.add_edge('Experiments', exp_type, {'weight':8})
# labels[exp_type] = exp_type
# pos = nx.graphviz_layout(g, prog='twopi', args='')
# nx.draw_networkx_nodes(g, pos,
# nodelist=[project],
# node_color='green', alpha=0.7,
# node_size=2500, node_shape='s')
# nx.draw_networkx_nodes(g, pos,
# nodelist=['Experiments'],
# node_color='blue', alpha=0.7,
# node_size=2000, node_shape='p')
# nx.draw_networkx_nodes(g, pos,
# nodelist=experiments_types,
# node_color='red', alpha=0.7,
# node_size=1500, node_shape='o')
# nx.draw_networkx_edges(g, pos, width=2, alpha=0.5,
# edge_color='black')
# nx.draw_networkx_labels(g, pos, labels,
# alpha=0.9, font_size=8,
# font_color='black', font_weight='bold')
# plt.axis('off')
# plt.show()
# self._intf.connection.revert_strategy()
class SchemasInspector(object):
def __init__(self, interface):
self._intf = interface
def __call__(self):
self._intf.manage.schemas._init()
for xsd in self._intf.manage.schemas():
print '-'*40
print xsd.upper()
print '-'*40
print
for datatype in schema.datatypes(
self._intf.manage.schemas._trees[xsd]):
print '[%s]'%datatype
print
for path in schema.datatype_attributes(
self._intf.manage.schemas._trees[xsd], datatype):
print path
print
def look_for(self, element_name, datatype_name=None):
paths = []
self._intf.manage.schemas._init()
if ':' in element_name:
for root in self._intf.manage.schemas._trees.values():
paths.extend(schema.datatype_attributes(root, element_name))
return paths
for xsd in self._intf.manage.schemas():
# nsmap = self._intf.manage.schemas._trees[xsd].nsmap
if datatype_name is not None:
datatypes = [datatype_name]
else:
datatypes = schema.datatypes(
self._intf.manage.schemas._trees[xsd]
)
for datatype in datatypes:
for path in schema.datatype_attributes(
self._intf.manage.schemas._trees[xsd], datatype):
if element_name in path:
paths.append(path)
return paths
pyxnat-0.9.0~dev0/pyxnat/core/httputil.py 0000664 0000000 0000000 00000001100 11632421554 0020515 0 ustar 00root root 0000000 0000000
_boundary = '----------ThIs_Is_tHe_bouNdaRY_$'
_crlf = '\r\n'
def file_message(content, content_type, path, name):
body = []
body.append('--' + _boundary)
body.append('Content-Disposition: form-data; '
'name="%s"; filename="%s"' % (path, name)
)
body.append('Content-Type: %s' % content_type)
body.append('')
body.append(content)
body.append('--' + _boundary + '--')
body.append('')
body = _crlf.join(body)
content_type = 'multipart/form-data; boundary=%s' % _boundary
return body, content_type
pyxnat-0.9.0~dev0/pyxnat/core/interfaces.py 0000664 0000000 0000000 00000033543 11632421554 0021003 0 ustar 00root root 0000000 0000000 import os
import time
import tempfile
import email
import getpass
import httplib2
import json
from .select import Select
from .cache import CacheManager, HTCache
from .help import Inspector, GraphData, PaintGraph, _DRAW_GRAPHS
from .manage import GlobalManager
from .uriutil import join_uri
from .jsonutil import csv_to_json
from .errors import is_xnat_error
from .errors import catch_error
from .array import ArrayData
from .xpath_store import XpathStore
from . import xpass
DEBUG = False
# main entry point
class Interface(object):
""" Main entry point to access a XNAT server.
>>> central = Interface(server='http://central.xnat.org:8080',
user='login',
password='pwd',
cachedir='/tmp'
)
Or with config file:
>>> central = Interface(config='/home/me/.xnat.cfg')
Or for interactive use:
>>> central = Interface('http://central.xnat.org')
.. note::
The interactive mode is activated whenever an argument within
server, user or password is missing. In interactive mode pyxnat
tries to check the validity of the connection parameters.
Attributes
----------
_mode: online | offline
Online or offline mode
_memtimeout: float
Lifespan of in-memory cache
"""
def __init__(self, server=None, user=None, password=None,
cachedir=tempfile.gettempdir(), config=None):
"""
Parameters
----------
server: string | None
The server full URL (including port and XNAT instance name
if necessary) e.g. http://central.xnat.org,
http://localhost:8080/xnat_db
Or a path to an existing config file. In that case the other
parameters (user etc..) are ignored if given.
If None the user will be prompted for it.
user: string | None
A valid login registered through the XNAT web interface.
If None the user will be prompted for it.
password: string | None
The user's password.
If None the user will be prompted for it.
cachedir: string
Path of the cache directory (for all queries and
downloaded files)
If no path is provided, a platform dependent temp dir is
used.
v config: string
Reads a config file in json to get the connection parameters.
If a config file is specified, it will be used regardless of
other parameters that might have been given.
"""
self._interactive = False
if not all([server, user, password]) and not config:
self._interactive = True
if all(arg is None
for arg in [server, user, password, config]):
connection_args = xpass.read_xnat_pass(xpass.path())
if connection_args is None:
raise Exception('XNAT configuration file not found '
'or formated incorrectly.')
self._server = connection_args['host']
self._user = connection_args['u']
self._pwd = connection_args['p']
self._cachedir = os.path.join(
cachedir, '%s@%s' % (
self._user,
self._server.split('//')[1].replace(
'/', '.').replace(':', '_')
)
)
elif config is not None:
self.load_config(config)
else:
if server is None:
self._server = raw_input('Server: ')
else:
self._server = server
if user is None:
user = raw_input('User: ')
if password is None:
password = getpass.getpass()
self._user = user
self._pwd = password
self._cachedir = os.path.join(
cachedir, '%s@%s' % (
self._user,
self._server.split('//')[1].replace(
'/', '.').replace(':', '_')
)
)
self._callback = None
self._memcache = {}
self._memtimeout = 1.0
self._mode = 'online'
self._struct = {}
self._entry = None
self._last_memtimeout = 1.0
self._last_mode = 'online'
self._jsession = 'authentication_by_credentials'
self._connect_extras = {}
self._connect()
self.inspect = Inspector(self)
self.select = Select(self)
self.array = ArrayData(self)
self.cache = CacheManager(self)
self.manage = GlobalManager(self)
self.xpath = XpathStore(self)
if _DRAW_GRAPHS:
self._get_graph = GraphData(self)
self.draw = PaintGraph(self)
if self._interactive:
self._get_entry_point()
self.inspect()
def _get_entry_point(self):
if self._entry is None:
# /REST for XNAT 1.4, /data if >=1.5
self._entry = '/REST'
try:
self._jsession = self._exec('/data/JSESSION')
self._entry = '/data'
if is_xnat_error(self._jsession):
catch_error(self._jsession)
except Exception, e:
if not '/data/JSESSION' in e.message:
raise e
return self._entry
def _connect(self, **kwargs):
""" Sets up the connection with the XNAT server.
Parameters
----------
kwargs: dict
Can be used to pass additional arguments to
the Http constructor. See the httplib2 documentation
for details. http://code.google.com/p/httplib2/
"""
if kwargs != {}:
self._connect_extras = kwargs
else:
kwargs = self._connect_extras
kwargs['disable_ssl_certificate_validation'] = True
if DEBUG:
httplib2.debuglevel = 2
self._http = httplib2.Http(HTCache(self._cachedir, self), **kwargs)
self._http.add_credentials(self._user, self._pwd)
def _exec(self, uri, method='GET', body=None, headers=None):
""" A wrapper around a simple httplib2.request call that:
- avoids repeating the server url in the request
- deals with custom caching mechanisms
- manages a user session with cookies
- catches and broadcast specific XNAT errors
Parameters
----------
uri: string
URI of the resource to be accessed. e.g. /REST/projects
method: GET | PUT | POST | DELETE
HTTP method.
body: string
HTTP message body
headers: dict
Additional headers for the HTTP request.
"""
if headers is None:
headers = {}
self._get_entry_point()
uri = join_uri(self._server, uri)
if DEBUG:
print uri
# using session authentication
headers['cookie'] = self._jsession
headers['connection'] = 'keep-alive'
# reset the memcache when client changes something on the server
if method in ['PUT', 'DELETE']:
self._memcache = {}
if self._mode == 'online' and method == 'GET':
if time.time() - self._memcache.get(uri, 0) < self._memtimeout:
if DEBUG:
print 'send: GET CACHE %s' % uri
info, content = self._http.cache.get(uri
).split('\r\n\r\n', 1)
self._memcache[uri] = time.time()
response = None
else:
response, content = self._http.request(uri, method,
body, headers)
self._memcache[uri] = time.time()
elif self._mode == 'offline' and method == 'GET':
cached_value = self._http.cache.get(uri)
if cached_value is not None:
if DEBUG:
print 'send: GET CACHE %s' % uri
info, content = cached_value.split('\r\n\r\n', 1)
response = None
else:
try:
self._http.timeout = 10
response, content = self._http.request(uri, method,
body, headers)
self._http.timeout = None
self._memcache[uri] = time.time()
except Exception, e:
catch_error(e)
else:
response, content = self._http.request(uri, method,
body, headers)
if DEBUG:
if response is None:
response = httplib2.Response(email.message_from_string(info))
print 'reply: %s %s from cache' % (response.status,
response.reason
)
for key in response.keys():
print 'header: %s: %s' % (key.title(), response.get(key))
if response is not None and response.has_key('set-cookie'):
self._jsession = response.get('set-cookie')[:44]
if response is not None and response.get('status') == '404':
r,_ = self._http.request(self._server)
if self._server.rstrip('/') != r.get('content-location',
self._server).rstrip('/'):
old_server = self._server
self._server = r.get('content-location').rstrip('/')
return self._exec(uri.replace(old_server, ''), method, body)
else:
raise httplib2.HttpLib2Error('%s %s %s' % (uri,
response.status,
response.reason
)
)
if is_xnat_error(content):
catch_error(content)
return content
def _get_json(self, uri):
""" Specific Interface._exec method to retrieve data.
It forces the data format to csv and then puts it back to a
json-like format.
Parameters
----------
uri: string
URI of the resource to be accessed. e.g. /REST/projects
Returns
-------
List of dicts containing the results
"""
if 'format=json' in uri:
uri = uri.replace('format=json', 'format=csv')
else:
if '?' in uri:
uri += '&format=csv'
else:
uri += '?format=csv'
content = self._exec(uri, 'GET')
if is_xnat_error(content):
catch_error(content)
return csv_to_json(content)
def _get_head(self, uri):
if DEBUG:
print 'GET HEAD'
_nocache = httplib2.Http()
_nocache.add_credentials(self._user, self._pwd)
rheaders = {'cookie':self._jsession}
try:
head = _nocache.request(
'%s%s' % (self._server, uri), 'HEAD', headers=rheaders)[0]
except:
time.sleep(1)
head = _nocache.request(
'%s%s' % (self._server, uri), 'HEAD', headers=rheaders)[0]
info = email.Message.Message()
for key, value in head.iteritems():
if key == 'content-disposition':
info['content-location'] = '%s%s' % (self._server, uri)
if key not in ['set-cookie']:
info[key] = value
return info
def save_config(self, location):
""" Saves current configuration - including password - in a file.
.. warning::
Since the password is saved as well, make sure the file
is saved at a safe location with appropriate permissions.
Parameters
----------
location: string
Destination config file.
"""
if not os.path.exists(os.path.dirname(location)):
os.makedirs(os.path.dirname(location))
fp = open(location, 'w')
config = {'server':self._server,
'user':self._user,
'password':self._pwd,
'cachedir':os.path.split(self._cachedir)[0],
}
json.dump(config, fp)
fp.close()
def load_config(self, location):
""" Loads a configuration file and replaces current connection
parameters.
Parameters
----------
location: string
Configuration file path.
"""
if os.path.exists(location):
fp = open(location, 'rb')
config = json.load(fp)
fp.close()
self._server = str(config['server'])
self._user = str(config['user'])
self._pwd = str(config['password'])
self._cachedir = str(config['cachedir'])
self._cachedir = os.path.join(
self._cachedir, '%s@%s' % (
self._user,
self._server.split('//')[1].replace(
'/', '.').replace(':', '_')
)
)
else:
raise Exception('Configuration file does not exists.')
def version(self):
return self._exec('/data/version')
def set_logging(self, level=0):
pass
pyxnat-0.9.0~dev0/pyxnat/core/jsonutil.py 0000664 0000000 0000000 00000022434 11632421554 0020524 0 ustar 00root root 0000000 0000000 import csv
import copy
from fnmatch import fnmatch
from StringIO import StringIO
import json
# jdata is a list of dicts
def join_tables(join_column, jdata, *jtables):
indexes = []
for jtable in [jdata]+list(jtables):
if isinstance(jtable, dict):
jtable = [jtable]
index = {}
[index.setdefault(entry[join_column], entry) for entry in jtable]
indexes.append(index)
merged_jdata = []
for join_id in indexes[0].keys():
for index in indexes[1:]:
indexes[0][join_id].update(index[join_id])
merged_jdata.append(indexes[0][join_id])
return merged_jdata
def get_column(jdata, col, val_pattern='*'):
if isinstance(jdata, dict):
jdata = [jdata]
if val_pattern == '*':
return [entry[col] for entry in jdata if entry.has_key(col)]
else:
return [entry[col] for entry in jdata
if fnmatch(entry.get(col), val_pattern)
]
def get_where(jdata, *args, **kwargs):
if isinstance(jdata, dict):
jdata = [jdata]
match = []
for entry in jdata:
match_args = all([arg in entry.keys() or arg in entry.values()
for arg in args
])
match_kwargs = all([entry[key] == kwargs[key]
for key in kwargs.keys()
])
if match_args and match_kwargs:
match.append(entry)
return match
def get_where_not(jdata, *args, **kwargs):
if isinstance(jdata, dict):
jdata = [jdata]
match = []
for entry in jdata:
match_args = all([arg in entry.keys() or arg in entry.values()
for arg in args
])
match_kwargs = all([entry[key] == kwargs[key]
for key in kwargs.keys()
])
if not match_args and not match_kwargs:
match.append(entry)
return match
def get_headers(jdata):
if isinstance(jdata, dict):
jdata = [jdata]
return [] if jdata == [] else jdata[0].keys()
def get_selection(jdata, columns):
if isinstance(jdata, dict):
jdata = [jdata]
sub_table = copy.deepcopy(jdata)
rmcols = set(get_headers(jdata)).difference(columns)
for entry in sub_table:
for col in rmcols:
if entry.has_key(col):
del entry[col]
return sub_table
def csv_to_json(csv_str):
csv_reader = csv.reader(StringIO(csv_str), delimiter=',', quotechar='"')
headers = csv_reader.next()
return [dict(zip(headers, entry)) for entry in csv_reader]
class JsonTable(object):
""" Wrapper around a list of dictionnaries to provide utility functions.
"""
def __init__(self, jdata, order_by=[]):
self.data = jdata
self.order_by = order_by
def __repr__(self):
# if len(self.data) == 0:
# return '[]'
# elif len(self.data) == 1:
# return str(self.data[0])
if len(self.headers()) <= 5:
_headers = ','.join(self.headers())
else:
_headers = '%s ... %s' % (','.join(self.headers()[:2]),
','.join(self.headers()[-2:])
)
return ' %s' % (
len(self), len(self.headers()), _headers
)
# return ('[%s\n .\n .\n . \n%s]\n\n'
# '------------\n'
# '%s rows\n'
# '%s columns\n'
# '%s characters') % (str(self.data[0]),
# str(self.data[-1]),
# len(self),
# len(self.headers()),
# len(self.dumps_csv())
# )
def __str__(self):
return self.dumps_csv()
def __len__(self):
return len(self.data)
def __iter__(self):
return iter(self.data)
def __getitem__(self, name):
if isinstance(name, (str, unicode)):
return self.get(name)
elif isinstance(name, int):
return self.__class__([self.data[name]], self.order_by)
elif isinstance(name, list):
return self.select(name)
def __getslice__(self, i, j):
return self.__class__(self.data[i:j], self.order_by)
def join(self, join_column, *jtables):
""" Join jsontables with a common column.
Parameters
----------
join_column: string
The name or header of the join column.
jtables: *args
Other jtables.
"""
return self.__class__(
join_tables(join_column, self.data,
*[jtable.data for jtable in jtables]),
self.order_by
)
def has_header(self, name):
return name in self.headers()
def headers(self):
""" Returns the headers of the object.
"""
return get_headers(self.data)
def get(self, col, val_pattern='*', always_list=False):
""" Gets a single column value.
Parameters
----------
col: string
The column name
val_pattern: string
Enable a filter on the values to be returned.
always_list: boolean
If only a single value is to be returned - i.e. there
is only on element in the list of dicts or there is only
one match against the value filter - is can be returned
within a list (with True) or not (default).
"""
res = get_column(self.data, col, val_pattern)
if always_list:
return res
if len(self.data) == 1:
return res[0]
return res
def where(self, *args, **kwargs):
""" Filters the object.
Paramaters
----------
args:
Value must be matched in the key or the value of an entry.
kwargs:
Value for a specific key must be matched in an entry.
Returns
-------
A :class:`JsonTable` containing the matches.
"""
return self.__class__(get_where(self.data, *args, **kwargs),
self.order_by
)
def where_not(self, *args, **kwargs):
""" Filters the object. Conditions must not be matched.
Paramaters
----------
args:
Value must not be matched in the key or the value of an
entry.
kwargs:
Value for a specific key must not be matched in an entry.
Returns
-------
A :class:`JsonTable` containing the not matches.
"""
return self.__class__(get_where_not(self.data, *args, **kwargs),
self.order_by
)
def select(self, columns):
""" Select only some columns of interest.
Returns
-------
A :class:`JsonTable` with the selected columns.
"""
return self.__class__(get_selection(self.data, columns),
self.order_by
)
def dump_csv(self, dest, delimiter=','):
""" Dumps the object content in a csv file format.
Parameters
----------
dest: string
Destination file path.
delimiter: char
Character to separate values in the csv file.
"""
fd = open(dest, 'w')
fd.write(self.dumps_csv(delimiter))
fd.close()
def dumps_csv(self, delimiter=','):
str_buffer = StringIO()
csv_writer = csv.writer(str_buffer, delimiter=delimiter,
quotechar='"', quoting=csv.QUOTE_MINIMAL)
for entry in self.as_list():
csv_writer.writerow(entry)
return str_buffer.getvalue()
def dump_json(self, dest):
fd = open(dest, 'w')
fd.write(self.dumps_json())
fd.close()
def dumps_json(self):
return json.dumps(self.data)
def as_list(self):
table = [[]]
for header in self.order_by:
if header in self.headers():
table[0].append(header)
for header in self.headers():
if header not in self.order_by:
table[0].append(header)
for entry in self.data:
row = []
for header in self.order_by:
if entry.has_key(header):
row.append(entry.get(header))
for header in self.headers():
if header not in self.order_by:
row.append(entry.get(header))
table.append(row)
return table
def items(self):
table = []
for entry in self.data:
row = ()
for header in self.order_by:
if entry.has_key(header):
row += (entry.get(header), )
for header in self.headers():
if header not in self.order_by:
row += (entry.get(header), )
table.append(row)
return table
pyxnat-0.9.0~dev0/pyxnat/core/manage.py 0000664 0000000 0000000 00000026236 11632421554 0020111 0 ustar 00root root 0000000 0000000 import re
import glob
import urllib
from lxml import etree
from .search import SearchManager
from .users import Users
from .resources import Project
from .tags import Tags
from .uriutil import join_uri, check_entry
from .jsonutil import JsonTable
from . import httputil
class GlobalManager(object):
""" Mainly a container class to provide a clean interface for all
management classes.
"""
def __init__(self, interface):
self._intf = interface
self.search = SearchManager(self._intf)
self.users = Users(self._intf)
self.tags = Tags(self._intf)
self.schemas = SchemaManager(self._intf)
self.prearchive = PreArchive(self._intf)
def register_callback(self, func):
""" Defines a callback to execute when collections of resources are
accessed.
Parameters
----------
func: callable
A callable that takes the current collection object as first
argument and the current element object as second argument.
Examples
--------
>>> def notify(cobj, eobj):
>>> print eobj._uri
>>> interface.manage.register_callback(notify)
"""
self._intf._callback = func
def unregister_callback(self):
""" Unregisters the callback.
"""
self._intf._callback = None
def project(self, project_id):
""" Returns a project manager.
"""
return ProjectManager(project_id, self._intf)
class ProjectManager(object):
""" Management interface for projects.
This functionalities are also available through `Project` objects.
"""
def __init__(self, project_id, interface):
self._intf._get_entry_point()
self._intf = interface
project = Project('%s/projects/%s' % (self._intf._entry,
project_id
),
self._intf
)
self.prearchive_code = project.prearchive_code
self.set_prearchive_code = project.set_prearchive_code
self.quarantine_code = project.quarantine_code
self.set_quarantine_code = project.set_quarantine_code
self.current_arc = project.current_arc
self.set_subfolder_in_current_arc = \
project.set_subfolder_in_current_arc
self.accessibility = project.accessibility
self.users = project.users
self.owners = project.owners
self.members = project.members
self.collaborators = project.collaborators
self.user_role = project.user_role
self.add_user = project.add_user
self.remove_user = project.remove_user
class SchemaManager(object):
""" Management interface for XNAT schemas.
The aim is to provide a minimal set of functionalities to parse
and look at XNAT schemas.
"""
def __init__(self, interface):
self._intf = interface
self._trees = {}
def _init(self):
if self._trees == {}:
cache_template = '%s/*.xsd.headers' % self._intf._cachedir
for entry in glob.iglob(cache_template):
hfp = open(entry, 'rb')
content = hfp.read()
hfp.close()
url = re.findall('(?<=content-location:\s%s)'
'.*?(?=\r{0,1}\n)' % self._intf._server,
content)[0]
self._trees[url.split('/')[-1]] = \
etree.fromstring(self._intf._exec(url))
def __call__(self):
self._init()
return self._trees.keys()
def add(self, url):
""" Loads an additional schema.
Parameters
----------
url: str
url of the schema relative to the server.
e.g. for
http://central.xnat.org/schemas/xnat/xnat.xsd give
``schemas/xnat/xnat.xsd`` or even only
``xnat.xsd``
"""
self._init()
if not re.match('/?schemas/.*/.*\.xsd', url):
if not 'schemas' in url and re.match('/?\w+/\w+[.]xsd', url):
url = join_uri('/schemas', url)
elif not re.match('^[^/].xsd', url):
url = '/schemas/%s/%s'%(url.split('.xsd')[0], url)
else:
raise NotImplementedError
self._trees[url.split('/')[-1]] = \
etree.fromstring(self._intf._exec(url))
def remove(self, name):
""" Removes a schema.
"""
if self._trees.has_key(name):
del self._trees[name]
"""
Fields Of the Prearchive
------------------------
Each session in the prearchive
has the following fields:
"project" - The name of the project. "Unassigned" if the session is unassigned.
"timestamp" - The time (down to millisecond) that this session was received
by XNAT. "20110603_124835868" for example.
"lastmod" - The time this session as last modified. Moving, resetting etc.
updates this time
"uploaded" - The time this session was uploaded to XNAT. Usually the same as
"timestamp"
"scan_date" - The date this session was scanned.
"scan_time" - The time this session was scanned.
"subject" - The name of the subject
"folderName" - The id of this session. Corresponds to XNAT's session id,
"name" - The name of this session. Corresponds to XNAT's session label.
"tag" - This session's unique DICOM identifier. Usually the SOP instance UID.
"status" - The current status of this session
"url" - The unique uri of this session.
"autoarchive" - Whether this session should be auto-archived.
For more in-depth information about the prearchive see:
http://docs.xnat.org/Using+the+Prearchive#XNAT%201.5:%20Managing%20Data%20with%20the%20Prearchive-Operations-Session%20Status
Uniquely identifying a session
------------------------------
Every session in the prearchive uniquely is identified by "project",
"timestamp" and "folderName".
(13/7/2011) - Each session could have been uniquely identified by the "url"
field or the "tag" field. These are arguably more elegant
identifiers but for now the "project", "timestamp", "folderName"
triple is used.
"""
class PreArchive(object):
def __init__(self, interface):
self._intf = interface
"""
Retrieve the status of a session
Parameters
----------
triple - A list containing the project, timestamp and session id, in that order.
"""
def status(self, triple):
return JsonTable(
self._intf._get_json('/data/prearchive/projects')
).where(
project=triple[0], timestamp=triple[1], folderName=triple[2]
).get('status')
"""
Retrieve the contents of the prearchive.
"""
def get(self):
return JsonTable(self._intf._get_json('/data/prearchive/projects'),
['project', 'timestamp', 'folderName']
).select(['project', 'timestamp', 'folderName']
).as_list()[1:]
"""
Retrieve the scans of a give session triple
Parameters
----------
triple - A list containing the project, timestamp and session id, in that order.
"""
def get_scans(self, triple):
return JsonTable(self._intf._get_json(
'/data/prearchive/projects/%s/scans' \
% '/'.join(triple)
)).get('ID')
"""
Retrieve the resource of a session triple
Parameters
----------
triple - A list containing the project, timestamp and session id, in that order.
scan_id - id of the scan
"""
def get_resources(self, triple, scan_id):
return JsonTable(self._intf._get_json(
'/data/prearchive/projects/%s'
'/scans/%s/resources' % ('/'.join(triple), scan_id)
)).get('label')
"""
Retrieve a list of files in a given session triple
Parameters
----------
triple - A list containing the project, timestamp and session id, in that order.
scan_id - id of the scan
resource_id - id of the resource
"""
def get_files(self, triple, scan_id, resource_id):
return JsonTable(self._intf._get_json(
'/data/prearchive/projects/%s'
'/scans/%s/resources/%s/files' % ('/'.join(triple),
scan_id,
resource_id)
)).get('Name')
"""
Move multiple sessions to a project in the prearchive asynchronously.
If only one session is it is done now.
This does *not* archive a session.
Parameters
----------
uris - a list of session uris
new_project - The name of the project to which to move the sessions.
"""
def move (self, uris, new_project):
add_src = lambda u: urllib.urlencode({'src':u})
async = len(uris) > 1 and 'true' or 'false'
print async
post_body = '&'.join ((map(add_src,uris))
+ [urllib.urlencode({'newProject':new_project})]
+ [urllib.urlencode({'async':async})])
request_uri = '/data/services/prearchive/move?format=csv'
return self._intf._exec(request_uri ,'POST', post_body,
{'content-type':'application/x-www-form-urlencoded'})
"""
Reinspect the file on the filesystem on the XNAT server and recreate the
parameters of the file. Essentially a refresh of the file.
Be warned that if this session has been scheduled for an operation, that
operation is cancelled.
Parameters
----------
uris - a list of session uris
new_project - The name of the project to which to move the sessions.
"""
def reset(self, triple):
post_body = "action=build"
request_uri = '/data/prearchive/projects/%s?format=single' \
% '/'.join(triple)
return self._intf._exec(request_uri ,'POST', post_body,
{'content-type':'application/x-www-form-urlencoded'})
"""
Delete a session from the prearchive
Parameters
----------
uri - The uri of the session to delete
"""
def delete (self, uri):
post_body = "src=" + uri + "&" + "async=false"
request_uri = "/data/services/prearchive/delete?format=csv"
return self._intf._exec(request_uri ,'POST', post_body,
{'content-type':'application/x-www-form-urlencoded'})
"""
Get the uri of the given session.
Parameters
----------
triple - A list containing the project, timestamp and session id, in that order.
"""
def get_uri(self, triple):
return JsonTable(
self._intf._get_json('/data/prearchive/projects')
).where(
project=triple[0], timestamp=triple[1], folderName=triple[2]
).get('url')
pyxnat-0.9.0~dev0/pyxnat/core/pathutil.py 0000664 0000000 0000000 00000001135 11632421554 0020502 0 ustar 00root root 0000000 0000000 import os
def find_files(src):
names = os.listdir(src)
errors = []
files = []
for name in names:
srcname = os.path.join(src, name)
try:
if os.path.islink(srcname):
linkto = os.readlink(srcname)
files.extend(find_files(linkto))
# os.symlink(linkto, dstname)
elif os.path.isdir(srcname):
files.extend(find_files(srcname))
else:
files.append(srcname)
except (IOError, os.error), why:
errors.append((srcname, str(why)))
return files
pyxnat-0.9.0~dev0/pyxnat/core/pipelines.py 0000664 0000000 0000000 00000022057 11632421554 0020646 0 ustar 00root root 0000000 0000000 import sys
import os
import json
import datetime
import urllib2
import smtplib
from copy import deepcopy
import email.mime.text
import xml.dom.minidom
import suds.client
import suds.xsd.doctor
from . import httputil
class PipelineNotFoundError(Exception):
"""workflow not found"""
class Pipelines(object):
def __init__(self, project, interface):
self._intf = interface
self._project = project
def get(self):
response = self._intf._exec('%s/projects/%s/pipelines' % (
self._intf._get_entry_point(),
self._project,
))
return json.loads(response)['ResultSet']['Result']
def add(self, location):
f = open(location, 'rb')
pip_doc = f.read()
f.close()
body, content_type = httputil.file_message(
pip_doc, 'text/xml', location, os.path.split(location)[1])
pipeline_uri = '%s/projects/%s/pipelines/%s' % (
self._intf._get_entry_point(),
self._project,
os.path.split(location)[1]
)
self._intf._exec(pipeline_uri,
method='PUT',
body=body,
headers={'content-type':content_type}
)
def delete(self, pipeline_id):
pass
class Pipeline(object):
def __init__(self, pipeline_id, interface):
self._intf = interface
self._id = pipeline_id
def run(self):
pass
def stop(self):
pass
def update(self):
pass
def complete(self):
pass
def fail(self):
pass
# class Pipeline(object):
# """class for mirroring workflow information (XML) in XNAT"""
# def __init__(self, base_url, username, password, workflow_id):
# self._base_url = base_url
# self._username = username
# self._password = password
# self._cookiejar = None
# res = self._call('CreateServiceSession.jws',
# 'execute',
# (),
# authenticated=True)
# self._session = str(res)
# args = (('ns1:string', self._session),
# ('ns1:string', 'wrk:workflowData.ID'),
# ('ns1:string', '='),
# ('ns1:string', workflow_id),
# ('ns1:string', 'wrk:workflowData'))
# workflow_ids = self._call('GetIdentifiers.jws', 'search', args)
# self._doc = None
# for w_id in workflow_ids:
# url = '%s/app/template/XMLSearch.vm/id/%s/data_type/wrk:workflowData' % (self._base_url, str(w_id))
# r = urllib2.Request(url)
# self._cookiejar.add_cookie_header(r)
# data = urllib2.urlopen(r).read()
# doc = xml.dom.minidom.parseString(data)
# workflow_node = doc.getElementsByTagName('wrk:Workflow')[0]
# status = workflow_node.getAttribute('status').lower()
# if status in ('queued', 'awaiting action', 'hold'):
# self._doc = doc
# break
# if self._doc is None:
# raise PipelineNotFoundError
# return
# def _call(self,
# jws,
# operation,
# inputs,
# authenticated=False,
# fix_import=False):
# """perform a SOAP call"""
# url = '%s/axis/%s' % (self._base_url, jws)
# if authenticated:
# t = suds.transport.http.HttpAuthenticated(username=self._username,
# password=self._password)
# else:
# t = suds.transport.http.HttpTransport()
# if self._cookiejar is not None:
# t.cookiejar = self._cookiejar
# if fix_import:
# xsd_url = 'http://schemas.xmlsoap.org/soap/encoding/'
# imp = suds.xsd.doctor.Import(xsd_url)
# doctor = suds.xsd.doctor.ImportDoctor(imp)
# client = suds.client.Client('%s?wsdl' % url,
# transport=t,
# doctor=doctor)
# else:
# client = suds.client.Client('%s?wsdl' % url, transport=t)
# typed_inputs = []
# for (dtype, val) in inputs:
# ti = client.factory.create(dtype)
# ti.value = val
# typed_inputs.append(ti)
# # the WSDL returns the local IP address in the URLs; these need
# # to be corrected if XNAT is behind a proxy
# client.set_options(location=url)
# f = getattr(client.service, operation)
# result = f(*typed_inputs)
# if self._cookiejar is None:
# self._cookiejar = t.cookiejar
# return result
# def _close(self):
# """close the XNAT session (log out)"""
# self._call('CloseServiceSession.jws', 'execute', ())
# return
# def _update_xnat(self):
# """update XNAT with the current state of this (WorkflowInfo) object"""
# inputs = (('ns0:string', self._session),
# ('ns0:string', self._doc.toxml()),
# ('ns0:boolean', False),
# ('ns0:boolean', True))
# self._call('StoreXML.jws',
# 'store',
# inputs,
# authenticated=True,
# fix_import=True)
# return
# def _append_node(self, root, name, value):
# """add a simple text node with tag "name" and data "value" under
# the node "root"
# """
# node = self._doc.createElement(name)
# node.appendChild(self._doc.createTextNode(value))
# root.appendChild(node)
# return
# def set_environment(self, arguments, parameters):
# """set the execution environment
# should be run only once before update() is called
# """
# # order is important
# workflow_node = self._doc.getElementsByTagName('wrk:Workflow')[0]
# ee_node = self._doc.createElement('wrk:executionEnvironment')
# ee_node.setAttribute('xsi:type', 'wrk:xnatExecutionEnvironment')
# workflow_node.appendChild(ee_node)
# self._append_node(ee_node, 'wrk:pipeline', arguments['pipeline'])
# self._append_node(ee_node, 'wrk:xnatuser', arguments['u'])
# self._append_node(ee_node, 'wrk:host', arguments['host'])
# params_node = self._doc.createElement('wrk:parameters')
# ee_node.appendChild(params_node)
# for key in parameters:
# param_node = self._doc.createElement('wrk:parameter')
# param_node.setAttribute('name', key)
# for val in parameters[key]:
# param_node.appendChild(self._doc.createTextNode(val))
# params_node.appendChild(param_node)
# for email in arguments['notify_emails']:
# self._append_node(ee_node, 'wrk:notify', email)
# self._append_node(ee_node, 'wrk:dataType', arguments['dataType'])
# self._append_node(ee_node, 'wrk:id', arguments['id'])
# if arguments['notify_flag']:
# self._append_node(ee_node, 'wrk:supressNotification', '0')
# else:
# self._append_node(ee_node, 'wrk:supressNotification', '1')
# return
# def update(self, step_id, step_description, percent_complete):
# """update the workflow in XNAT"""
# workflow_node = self._doc.getElementsByTagName('wrk:Workflow')[0]
# workflow_node.setAttribute('status', 'Running')
# t = datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S')
# workflow_node.setAttribute('current_step_launch_time', t)
# workflow_node.setAttribute('current_step_id', step_id)
# workflow_node.setAttribute('step_description', step_description)
# workflow_node.setAttribute('percentageComplete', percent_complete)
# self._update_xnat()
# return
# def complete(self):
# """mark the workflow comleted in XNAT and close the session"""
# workflow_node = self._doc.getElementsByTagName('wrk:Workflow')[0]
# workflow_node.setAttribute('status', 'Complete')
# t = datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S')
# workflow_node.setAttribute('current_step_launch_time', t)
# workflow_node.setAttribute('percentageComplete', '100.0')
# try:
# workflow_node.removeAttribute('current_step_id')
# except xml.dom.NotFoundErr:
# pass
# try:
# workflow_node.removeAttribute('step_description')
# except xml.dom.NotFoundErr:
# pass
# self._update_xnat()
# self._close()
# return
# def fail(self, description=None):
# """mark the workflow failed in XNAT and close the session"""
# workflow_node = self._doc.getElementsByTagName('wrk:Workflow')[0]
# workflow_node.setAttribute('status', 'Failed')
# if description is not None:
# workflow_node.setAttribute('step_description', description)
# self._update_xnat()
# self._close()
# return
pyxnat-0.9.0~dev0/pyxnat/core/provenance.py 0000664 0000000 0000000 00000023075 11632421554 0021017 0 ustar 00root root 0000000 0000000 import time
import platform
import socket
from lxml import etree
from lxml.etree import Element, QName
from .uriutil import uri_parent
from .jsonutil import JsonTable
from . import httputil
_nsmap = {'xnat':'http://nrg.wustl.edu/xnat',
'prov':'http://www.nbirn.net/prov',
'xsi':'http://www.w3.org/2001/XMLSchema-instance'
}
_required = ['program', 'timestamp', 'user', 'machine', 'platform']
_optional = ['program_version', 'program_arguments',
'cvs',
'platform_version',
'compiler', 'compiler_version',
'library', 'library_version'
]
_all = ['program', 'program_version', 'program_arguments',
'timestamp',
'cvs',
'user',
'machine',
'platform', 'platform_version',
'compiler', 'compiler_version',
# 'library', 'library_version'
]
_platform_name, _hostname, \
_platform_version, _platform_version2,\
_machine, _machine2 = platform.uname()
_machine = socket.gethostname()
def provenance_document(eobj, process_steps, overwrite):
root_node = etree.fromstring(eobj.get())
existing_prov = None
for child in root_node.getchildren():
if str(child.tag).endswith('provenance'):
existing_prov = child
break
if existing_prov is not None and not overwrite:
prov_node = existing_prov
else:
if existing_prov is not None and overwrite:
root_node.remove(existing_prov)
prov_node = Element(QName(_nsmap['xnat'], 'provenance'),
nsmap=_nsmap
)
root_node.insert(0, prov_node)
prov_node.extend(provenance_parameters(process_steps))
return etree.tostring(root_node.getroottree())
def provenance_parameters(process_steps):
prov = []
for step in process_steps:
if not set(_required).issubset(step.keys()):
missing = list(set(_required).difference(step.keys()))
raise Exception(('Following attributes are '
'required to define provenance: %s' % missing
)
)
prov.append(process_step_xml(**step))
return prov
def process_step_xml(**kwargs):
step_node = Element(QName(_nsmap['prov'], 'processStep'), nsmap=_nsmap)
program_node = Element(QName(_nsmap['prov'], 'program'), nsmap=_nsmap)
program_node.text = kwargs['program']
if kwargs.has_key('program_version'):
program_node.set('version', kwargs['program_version'])
if kwargs.has_key('program_arguments'):
program_node.set('arguments', kwargs['program_arguments'])
step_node.append(program_node)
timestamp_node = Element(QName(_nsmap['prov'], 'timestamp'),
nsmap=_nsmap
)
timestamp_node.text = kwargs['timestamp']
step_node.append(timestamp_node)
if kwargs.has_key('cvs'):
cvs_node = Element(QName(_nsmap['prov'], 'cvs'), nsmap=_nsmap)
cvs_node.text = kwargs['cvs']
step_node.append(cvs_node)
user_node = Element(QName(_nsmap['prov'], 'user'), nsmap=_nsmap)
user_node.text = kwargs['user']
step_node.append(user_node)
machine_node = Element(QName(_nsmap['prov'], 'machine'), nsmap=_nsmap)
machine_node.text = kwargs['machine']
step_node.append(machine_node)
platform_node = Element(QName(_nsmap['prov'], 'platform'), nsmap=_nsmap)
platform_node.text = kwargs['platform']
if kwargs.has_key('platform_version'):
platform_node.set('version', kwargs['platform_version'])
step_node.append(platform_node)
if kwargs.has_key('compiler'):
compiler_node = Element(QName(_nsmap['prov'], 'compiler'),
nsmap=_nsmap
)
compiler_node.text = kwargs['compiler']
if kwargs.has_key('compiler_version'):
compiler_node.set('version', kwargs['compiler_version'])
step_node.append(compiler_node)
if kwargs.has_key('library'):
library_node = Element(QName(_nsmap['prov'], 'library'),
nsmap=_nsmap
)
library_node.text = kwargs['library']
if kwargs.has_key('library_version'):
library_node.set('version', kwargs['library_version'])
step_node.append(library_node)
return step_node
class Provenance(object):
""" Class to annotate processed data with provenance information.
The following parameters are available:
- program
- program_version
- program_arguments
- timestamp
- cvs
- user
- machine
- platform
- platform_version
- compiler
- compiler_version
Examples
--------
>>> prov = {'program':'young',
'timestamp':'2011-03-01T12:01:01.897987',
'user':'angus',
'machine':'war',
'platform':'linux',
}
>>> element.provenance.set(prov)
>>> element.provenance.get()
>>> element.delete()
"""
def __init__(self, eobject):
self._intf = eobject._intf
self._eobject = eobject
def set(self, process_steps, overwrite=False):
""" Set provenance information for the data within this element.
.. note::
If some required parameters are not provided, theses
parameters will be extracted from the current machine
and set automatically. Those parameters are:
- machine
- platform
- timestamp
- user
.. warning::
overwrite option doesn't work because of a bug with the
allowDataDeletion flag in XNAT
Parameters
----------
process_steps: list or dict
dict or list of dicts to define the processing steps
of the data. The minimum set of information to give
is: program, timestamp, user, machine and platform. More
keywords in the class documentation.
overwrite: boolean
If False the process_steps are added to the existing ones.
Else the processing steps overwrite any existing provenance.
"""
if isinstance(process_steps, dict):
process_steps = [process_steps]
for process_step in process_steps:
_timestamp = time.strftime('%Y-%m-%dT%H:%M:%S',
time.localtime()
)
if not process_step.has_key('machine'):
process_step['machine'] = _machine
if not process_step.has_key('platform'):
process_step['platform'] = _platform_name
process_step['platform_version'] = _platform_version
if not process_step.has_key('timestamp'):
process_step['timestamp'] = _timestamp
if not process_step.has_key('user'):
process_step['user'] = self._intf._user
doc = provenance_document(self._eobject, process_steps, overwrite)
body, content_type = httputil.file_message(
doc, 'text/xml', 'prov.xml', 'prov.xml')
prov_uri = self._eobject._uri
if overwrite:
prov_uri += '?allowDataDeletion=true'
self._intf._exec(prov_uri,
method='PUT',
body=body,
headers={'content-type':content_type}
)
def get(self):
""" Gets all the provenance information for that object.
Returns
-------
A list of dicts.
"""
datatype = self._eobject.datatype()
columns = ['%s/ID' % datatype] + [
'%s/provenance/processStep/%s' % (datatype, field)
for field in _all
]
prov_uri = uri_parent(self._eobject._uri)
prov_uri += '?columns='
prov_uri += ','.join(columns)
steps = []
table = JsonTable(self._intf._get_json(prov_uri))
id_header = 'ID' if table.has_header('ID') \
else '%s/id' % datatype.lower()
for step in table.where(**{id_header:self._eobject.id()}):
step_dict = {}
for key in step.keys():
if 'processstep' in key:
step_dict[key.split('processstep/')[1]] = step[key]
steps.append(step_dict)
return steps
def delete(self):
""" Removes the provenance attached to this object.
.. warning::
doesn't work because of a bug with the allowDataDeletion
flag in XNAT
"""
provenance_node = self._eobject.xpath('//xnat:provenance')
if provenance_node != []:
provenance_node = provenance_node[0]
parent_node = provenance_node.getparent()
parent_node.remove(provenance_node)
doc = etree.tostring(parent_node.getroottree())
body, content_type = httputil.file_message(
doc, 'text/xml', 'prov.xml', 'prov.xml')
self._intf._exec(
'%s?allowDataDeletion=true' % self._eobject._uri,
method='PUT',
body=body,
headers={'content-type':content_type}
)
pyxnat-0.9.0~dev0/pyxnat/core/resources.py 0000664 0000000 0000000 00000207423 11632421554 0020672 0 ustar 00root root 0000000 0000000 from __future__ import with_statement
import lxml
import os
import re
import shutil
import tempfile
import mimetypes
import zipfile
import time
import urllib
import codecs
from fnmatch import fnmatch
import json
from lxml import etree
from .uriutil import join_uri, translate_uri, uri_segment
from .uriutil import uri_last, uri_nextlast
from .uriutil import uri_parent, uri_grandparent
from .uriutil import uri_shape
from .jsonutil import JsonTable, get_selection
from .pathutil import find_files
from .attributes import EAttrs
from .search import build_search_document, rpn_contraints, query_from_xml
from .errors import is_xnat_error, parse_put_error_message
from .errors import DataError, ProgrammingError, catch_error
from .cache import md5name
from .provenance import Provenance
# from .pipelines import Pipelines
from . import schema
from . import httputil
from . import downloadutils
DEBUG = False
# metaclasses
def get_element_from_element(rsc_name):
def getter(self, ID):
Element = globals()[rsc_name.title()]
return Element(join_uri(self._uri, rsc_name + 's', ID), self._intf)
return getter
def get_element_from_collection(rsc_name):
def getter(self, ID):
Element = globals()[rsc_name.title()]
Collection = globals()[rsc_name.title() + 's']
return Collection([Element(join_uri(eobj._uri, rsc_name + 's', ID),
self._intf
)
for eobj in self
],
self._intf
)
return getter
def get_collection_from_element(rsc_name):
def getter(self, id_filter='*'):
Collection = globals()[rsc_name.title()]
return Collection(join_uri(self._uri, rsc_name),
self._intf, id_filter
)
return getter
def get_collection_from_collection(rsc_name):
def getter(self, id_filter='*'):
Collection = globals()[rsc_name.title()]
return Collection(self, self._intf, id_filter,
rsc_name, self._id_header, self._columns)
return getter
class ElementType(type):
def __new__(cls, name, bases, dct):
rsc_name = name.lower()+'s' \
if name.lower() in schema.resources_singular \
else name.lower()
for child_rsc in schema.resources_tree[rsc_name]:
dct[child_rsc] = get_collection_from_element(child_rsc)
dct[child_rsc.rstrip('s')] = \
get_element_from_element(child_rsc.rstrip('s'))
return type.__new__(cls, name, bases, dct)
def __init__(cls, name, bases, dct):
super(ElementType, cls).__init__(name, bases, dct)
class CollectionType(type):
def __new__(cls, name, bases, dct):
rsc_name = name.lower()+'s' \
if name.lower() in schema.resources_singular \
else name.lower()
for child_rsc in schema.resources_tree[rsc_name]:
dct[child_rsc] = get_collection_from_collection(child_rsc)
dct[child_rsc.rstrip('s')] = \
get_element_from_collection(child_rsc.rstrip('s'))
return type.__new__(cls, name, bases, dct)
def __init__(cls, name, bases, dct):
super(CollectionType, cls).__init__(name, bases, dct)
# generic classes
class EObject(object):
""" Generic Object for an element URI.
"""
def __init__(self, uri, interface):
"""
Parameters
----------
uri: string
URI for an element resource.
e.g. /REST/projects/my_project
interface: :class:`Interface`
Main interface reference.
"""
self._uri = urllib.quote(translate_uri(uri))
self._urn = urllib.unquote(uri_last(self._uri))
self._urt = uri_nextlast(self._uri)
self._intf = interface
self.attrs = EAttrs(self)
def __repr__(self):
return '<%s Object> %s' % (self.__class__.__name__,
urllib.unquote(uri_last(self._uri))
)
def _getcell(self, col):
""" Gets a single property of the element resource.
"""
return self._getcells([col])
def _getcells(self, cols):
""" Gets multiple properties of the element resource.
"""
p_uri = uri_parent(self._uri)
id_head = schema.json[self._urt][0]
lbl_head = schema.json[self._urt][1]
filters = {}
columns = set([col for col in cols
if col not in schema.json[self._urt] \
or col != 'URI'] + schema.json[self._urt]
)
get_id = p_uri + '?format=json&columns=%s' % ','.join(columns)
for pattern in self._intf._struct.keys():
if fnmatch(uri_segment(
self._uri.split(self._intf._entry, 1)[1], -2), pattern):
reg_pat = self._intf._struct[pattern]
filters.setdefault('xsiType', set()).add(reg_pat)
if filters != {}:
get_id += '&' + \
'&'.join('%s=%s' % (item[0], item[1])
if isinstance(item[1], basestring)
else '%s=%s' % (item[0],
','.join([val for val in item[1]])
)
for item in filters.items()
)
for res in self._intf._get_json(get_id):
if self._urn in [res.get(id_head), res.get(lbl_head)]:
if len(cols) == 1:
return res.get(cols[0])
else:
return get_selection(res, cols)[0]
def exists(self, consistent=False):
""" Test whether an element resource exists.
"""
try:
return self.id() != None
except Exception, e:
if DEBUG:
print e
return False
def id(self):
""" Returns the element resource id.
"""
return self._getcell(schema.json[self._urt][0])
def label(self):
""" Returns the element resource label.
"""
return self._getcell(schema.json[self._urt][1])
def datatype(self):
""" Returns the type defined in the XNAT schema for this element
resource.
+----------------+-----------------------+
| EObject | possible xsi types |
+================+=======================+
| Project | xnat:projectData |
+----------------+-----------------------+
| Subject | xnat:subjectData |
+----------------+-----------------------+
| Experiment | xnat:mrSessionData |
| | xnat:petSessionData |
+----------------+-----------------------+
"""
return self._getcell('xsiType')
def create(self, **params):
""" Creates the element if it does not exists.
Any non-existing ancestor will be created as well.
.. warning::
An element resource both have an ID and a label that
can be used to access it. At the moment, XNAT REST API
defines the label when creating an element, but not
the ID, which is generated. It means that the `name`
given to a resource may not appear when listing the
resources because the IDs will appear, not the labels.
.. note::
To set up additional variables for the element at its
creation it is possible to use shortcuts defined in the
XNAT REST documentation or xpath in the schema:
- element.create(ID='theid')
- subject.create(**{'xnat:subjectData/ID':'theid'})
Parameters
----------
params: keywords
- Specify the datatype of the element resource and of
any ancestor that may need to be created. The
keywords correspond to the levels in the REST
hierarchy, see Interface.inspect.architecture()
- If an element is created with no specified type:
- if its name matches a naming convention, this type
will be used
- else a default type is defined in the schema module
- To give the ID the same value as the label use
use_label=True e.g element.create(use_label=True)
Examples
--------
>>> interface.select('/project/PROJECT/subject'
'/SUBJECT/experiment/EXP/scan/SCAN'
).create(experiments='xnat:mrSessionData',
scans='xnat:mrScanData'
)
See Also
--------
EObject.id
EObject.label
EObject.datatype
"""
if params.has_key('xml') and os.path.exists(params.get('xml')):
f = codecs.open(params.get('xml'))
doc = f.read()
f.close()
try:
doc_tree = etree.fromstring(doc)
doc_tree.xpath('//*')[0].set('label', uri_last(self._uri))
doc = etree.tostring(doc_tree)
except:
pass
body, content_type = httputil.file_message(
doc, 'text/xml', 'data.xml', 'data.xml')
_uri = self._uri
_uri += '?allowDataDeletion=true'
self._intf._exec(_uri,
method='PUT',
body=body,
headers={'content-type':content_type}
)
return self
datatype = params.get(uri_nextlast(self._uri))
struct = self._intf._struct
if datatype is None:
for uri_pattern in struct.keys():
if fnmatch(
self._uri.split(self._intf._entry, 1)[1], uri_pattern):
datatype = struct.get(uri_pattern)
break
else:
datatype = schema.default_datatypes.get(
uri_nextlast(self._uri))
if datatype is None:
create_uri = self._uri
else:
local_params = \
[param for param in params
if param not in schema.resources_types + ['use_label'] \
and (param.startswith(datatype) or '/' not in param)
]
create_uri = '%s?xsiType=%s' % (self._uri, datatype)
if 'ID' not in local_params \
and '%s/ID' % datatype not in local_params \
and params.get('use_label'):
create_uri += '&%s/ID=%s' % (datatype, uri_last(self._uri))
if local_params != []:
create_uri += '&' + '&'.join('%s=%s' % (key,
params.get(key)
)
for key in local_params
)
# avoid to reuse relative parameters
for key in local_params:
del params[key]
parent_element = self._intf.select(uri_grandparent(self._uri))
if not uri_nextlast(self._uri) == 'projects' \
and not parent_element.exists():
parent_datatype = params.get(uri_nextlast(parent_element._uri))
if DEBUG:
print 'CREATE', parent_element, parent_datatype
parent_element.create(**params)
if DEBUG:
print 'PUT', create_uri
output = self._intf._exec(create_uri, 'PUT')
if is_xnat_error(output):
paths = []
for datatype_name, element_name \
in parse_put_error_message(output):
path = self._intf.inspect.schemas.look_for(
element_name, datatype_name)
paths.extend(path)
if DEBUG:
print path, 'is required'
return paths
return self
insert = create
def delete(self, delete_files=True):
""" Deletes an element resource.
Parameters
----------
delete_files: boolean
Tells if files attached to the element resources are
removed as well from the server filesystem.
"""
delete_uri = self._uri if not delete_files \
else self._uri + '?removeFiles=true'
out = self._intf._exec(delete_uri, 'DELETE')
if is_xnat_error(out):
catch_error(out)
def get(self):
""" Retrieves the XML document corresponding to this element.
"""
return self._intf._exec(self._uri+'?format=xml', 'GET')
def xpath(self, xpath):
root = etree.fromstring(self.get())
return root.xpath(xpath, namespaces=root.nsmap)
def namespaces(self):
pass
def parent(self):
uri = uri_grandparent(self._uri)
Klass = globals()[uri_nextlast(uri).title().rsplit('s', 1)[0]]
return Klass(uri, self._intf)
def children(self, show_names=True):
""" Returns the children levels of this element.
Parameters
----------
show_name: boolean
If True returns a list of strings. If False returns a
collection object referencing all child objects of
this elements.
Examples
--------
>>> subject_object.children()
['experiments', 'resources']
>>> subject_object.children(False)
170976556
"""
children = schema.resources_tree.get(uri_nextlast(self._uri))
if show_names:
return children
return CObject([getattr(self, child)() for child in children],
self._intf
)
def tag(self, name):
""" Tag the element.
"""
tag = self._intf.manage.tags.get(name)
if not tag.exists():
tag.create()
tag.reference(self._uri)
return tag
def untag(self, name):
""" Remove a tag for the element.
"""
tag = self._intf.manage.tags.get(name)
tag.dereference(self._uri)
if tag.references().get() == []:
tag.delete()
class CObject(object):
""" Generic Object for a collection resource.
A collection resource is a list of element resources. There is
however several ways to obtain such a list:
- a collection URI e.g. /REST/projects
- a list of element URIs
- a list of collections
e.g. /REST/projects/ONE/subjects **AND**
/REST/projects/TWO/subjects
- a list of element objects
- a list a collection objects
Collections objects built in different ways share the same behavior:
- they behave as iterators, which enables a lazy access to
the data
- they always yield EObjects
- they can be nested with any other collection
Examples
--------
No access to the data:
>>> interface.select.projects()
173667084
Lazy access to the data:
>>> for project in interface.select.projects():
>>> print project
Nesting:
>>> for subject in interface.select.projects().subjects():
>>> print subject
"""
def __init__(self, cbase, interface, pattern='*', nested=None,
id_header='ID', columns=[], filters={}):
"""
Parameters
----------
cbase: string | list | CObject
Object from which the collection is built.
interface: :class:`Interface`
Main interface reference.
pattern: string
Only resource element whose ID match the pattern are
returned.
nested: None | string
Parameter used to nest collections.
id_header: ID | label
Defines whether the element label or ID is returned as the
identifier.
columns: list
Defines additional columns to be returned.
filters: dict
Defines additional filters for the query, typically options
for the query string.
"""
self._intf = interface
self._cbase = cbase
self._id_header = id_header
self._pattern = pattern
self._columns = columns
self._filters = filters
self._nested = nested
if isinstance(cbase, basestring):
self._ctype = 'cobjectcuri'
elif isinstance(cbase, CObject):
self._ctype = 'cobjectcobject'
elif isinstance(cbase, list) and cbase != []:
if isinstance(cbase[0], basestring):
self._ctype = 'cobjecteuris'
if isinstance(cbase[0], EObject):
self._ctype = 'cobjecteobjects'
if isinstance(cbase[0], CObject):
self._ctype = 'cobjectcobjects'
elif isinstance(cbase, list) and cbase == []:
self._ctype = 'cobjectempty'
else:
raise Exception('Invalid collection accessor type: %s'%cbase)
def __repr__(self):
return ' %s' % id(self)
def _call(self, columns):
try:
uri = translate_uri(self._cbase)
uri = urllib.quote(uri)
request_shape = uri_shape(
'%s/0' % uri.split(self._intf._entry, 1)[1])
reqcache = os.path.join(self._intf._cachedir,
'%s.struct' % md5name(request_shape)
).replace('_*', '')
gather = uri.split('/')[-1] in ['experiments', 'assessors',
'scans', 'reconstructions']
tick = time.gmtime(time.time())[5] % \
self._intf.inspect._tick == 0 and\
self._intf.inspect._auto
if (not os.path.exists(reqcache) and gather) \
or (gather and tick):
columns += ['xsiType']
# struct = {}
# if self._intf._struct.has_key(reqcache):
# struct = self._intf._struct[reqcache]
# else:
# struct = json.load(open(reqcache, 'rb'))
# self._intf._struct[reqcache] = struct
query_string = '?format=json&columns=%s' % ','.join(columns)
# struct = {}
# for pattern in struct.keys():
# request_pat = uri_segment(
# join_uri(uri, self._pattern).split('/REST')[1], -2
# )
# # print pattern, request_pat, fnmatch(pattern, request_pat)
# if (fnmatch(pattern, request_pat)
# and struct[pattern] is not None):
# self._filters.setdefault('xsiType', set()
# ).add(struct[pattern])
if self._filters != {}:
query_string += '&' + '&'.join(
'%s=%s' % (item[0], item[1])
if isinstance(item[1], (str, unicode))
else '%s=%s' % (
item[0], ','.join([val for val in item[1]]) )
for item in self._filters.items()
)
jtable = self._intf._get_json(uri + query_string)
if (not os.path.exists(reqcache) and gather) \
or (gather and tick):
_type = uri.split('/')[-1]
self._learn_from_table(_type, jtable, reqcache)
return jtable
except Exception, e:
if DEBUG:
raise e
return []
def _learn_from_table(self, _type, jtable, reqcache):
request_knowledge = {}
for element in jtable:
xsitype = element.get('xsiType')
uri = element.get('URI').split(self._intf._entry, 1)[1]
uri = uri.replace(uri.split('/')[-2], _type)
shape = uri_shape(uri)
request_knowledge[shape] = xsitype
if os.path.exists(reqcache):
previous = json.load(open(reqcache, 'rb'))
previous.update(request_knowledge)
request_knowledge = previous
self._intf._struct.update(request_knowledge)
json.dump(request_knowledge, open(reqcache, 'w'))
def __iter__(self):
if self._ctype == 'cobjectcuri':
if self._id_header == 'ID':
id_header = schema.json[uri_last(self._cbase)][0]
elif self._id_header == 'label':
id_header = schema.json[uri_last(self._cbase)][1]
else:
id_header = self._id_header
for res in self._call([id_header] + self._columns):
try:
eid = urllib.unquote(res[id_header])
if fnmatch(eid, self._pattern):
klass_name = uri_last(self._cbase
).rstrip('s').title()
Klass = globals().get(klass_name, self._intf.__class__)
eobj = Klass(join_uri(self._cbase, eid), self._intf)
if self._nested is None:
self._run_callback(self, eobj)
yield eobj
else:
Klass = globals().get(self._nested.title(),
self._intf.__class__)
for subeobj in Klass(
cbase=join_uri(eobj._uri, self._nested),
interface=self._intf,
pattern=self._pattern,
id_header=self._id_header,
columns=self._columns):
try:
self._run_callback(self, subeobj)
yield subeobj
except RuntimeError:
pass
except KeyboardInterrupt:
self._intf._connect()
raise StopIteration
elif self._ctype == 'cobjecteuris':
for uri in self._cbase:
try:
Klass = globals().get(uri_nextlast(uri).rstrip('s').title(),
self._intf.__class__)
eobj = Klass(uri, self._intf)
if self._nested is None:
self._run_callback(self, eobj)
yield eobj
else:
Klass = globals().get(self._nested.title(),
self._intf.__class__)
for subeobj in Klass(
cbase=join_uri(eobj._uri, self._nested),
interface=self._intf,
pattern=self._pattern,
id_header=self._id_header,
columns=self._columns):
try:
self._run_callback(self, subeobj)
yield subeobj
except RuntimeError:
pass
except KeyboardInterrupt:
self._intf._connect()
raise StopIteration
elif self._ctype == 'cobjecteobjects':
for eobj in self._cbase:
try:
if self._nested is None:
self._run_callback(self, eobj)
yield eobj
else:
Klass = globals().get(self._nested.rstrip('s').title(),
self._intf.__class__)
for subeobj in Klass(
cbase=join_uri(eobj._uri, self._nested),
interface=self._intf,
pattern=self._pattern,
id_header=self._id_header,
columns=self._columns):
try:
self._run_callback(self, subeobj)
yield subeobj
except RuntimeError:
pass
except KeyboardInterrupt:
self._intf._connect()
raise StopIteration
elif self._ctype == 'cobjectcobject':
for eobj in self._cbase:
try:
if self._nested is None:
self._run_callback(self, eobj)
yield eobj
else:
Klass = globals().get(self._nested.title(),
self._intf.__class__)
for subeobj in Klass(
cbase=join_uri(eobj._uri, self._nested),
interface=self._intf,
pattern=self._pattern,
id_header=self._id_header,
columns=self._columns):
try:
self._run_callback(self, eobj)
yield subeobj
except RuntimeError:
pass
except KeyboardInterrupt:
self._intf._connect()
raise StopIteration
elif self._ctype == 'cobjectcobjects':
for cobj in self._cbase:
try:
for eobj in cobj:
if self._nested is None:
self._run_callback(self, eobj)
yield eobj
else:
Klass = globals().get(cobj._nested.title(),
self._intf.__class__)
for subeobj in Klass(
cbase=join_uri(eobj._uri, cobj._nested),
interface=cobj._intf,
pattern=cobj._pattern,
id_header=cobj._id_header,
columns=cobj._columns):
try:
self._run_callback(self, eobj)
yield subeobj
except RuntimeError:
pass
except KeyboardInterrupt:
self._intf._connect()
raise StopIteration
elif self._ctype == 'cobjectempty':
for empty in []:
yield empty
def _run_callback(self, cobj, eobj):
if self._intf._callback is not None:
self._intf._callback(cobj, eobj)
def first(self):
""" Returns the first element of the collection.
"""
for eobj in self:
return eobj
fetchone = first
def get(self, *args):
""" Returns every element.
.. warning::
If a collection needs to issue thousands of queries it may
be better to access the resources within a `for-loop`.
Parameters
----------
args: strings
- Specify the information to return for the elements
within ID, label and Object.
- Any combination of ID, label and obj is valid, if
more than one is given, a list of tuple is returned
instead of a list.
"""
if args == ():
return [urllib.unquote(uri_last(eobj._uri)) for eobj in self]
else:
ret = ()
for arg in args:
if arg == 'id':
self._id_header = 'ID'
ret += ([urllib.unquote(uri_last(eobj._uri))
for eobj in self
], )
elif arg == 'label':
self._id_header = 'label'
ret += ([urllib.unquote(uri_last(eobj._uri))
for eobj in self
], )
else:
ret += ([eobj for eobj in self], )
if len(args) != 1:
return ret
else:
return ret[0]
fetchall = get
def tag(self, name):
""" Tag the collection.
"""
tag = self._intf.manage.tags.get(name)
if not tag.exists():
tag.create()
tag.reference_many([eobj._uri for eobj in self])
return tag
def untag(self, name):
""" Remove the tag from the collection.
"""
tag = self._intf.manage.tags.get(name)
tag.dereference_many([eobj._uri for eobj in self])
if tag.references().get() == []:
tag.delete()
def where(self, constraints=None, template=None, query=None):
""" Only the element objects whose subject that are matching the
constraints will be returned. It means that it is not possible
to use this method on an element that is not linked to a
subject, such as a project.
Examples
--------
The ``where`` clause should be on the first select:
>>> for experiment in interface.select('//experiments'
).where([('atest/FIELD', '=', 'value'), 'AND']):
>>> print experiment
Do **NOT** do this:
>>> for experiment in interface.select('//experiments'):
for assessor in experiment.assessors(
).where([('atest/FIELD', '=', 'value'), 'AND']):
>>> print assessor
Or this:
>>> for project in interface.select('//projects'
).where([('atest/FIELD', '=', 'value'), 'AND']):
>>> print project
See Also
--------
search.Search()
"""
if isinstance(constraints, (str, unicode)):
constraints = rpn_contraints(constraints)
elif isinstance(template, (tuple)):
tmp_bundle = self._intf.manage.search.get_template(
template[0], True)
tmp_bundle = tmp_bundle % template[1]
constraints = query_from_xml(tmp_bundle)['constraints']
elif isinstance(query, (str, unicode)):
tmp_bundle = self._intf.manage.search.get(query, 'xml')
constraints = query_from_xml(tmp_bundle)['constraints']
elif isinstance(constraints, list):
pass
else:
raise ProgrammingError('One in [contraints, template and '
'query] parameters must be correctly '
'set.'
)
# _columns = [
# 'xnat:subjectData/PROJECT',
# 'xnat:subjectData/SUBJECT_ID',
# ] + ['%s/ID' %qtype for qtype in _queried_types]
# bundle = build_search_document(
# 'xnat:imageSessionData', _columns, constraints)
# content = self._intf._exec(
# "%s/search?format=json" % self._intf._entry,
# 'POST', bundle)
# if content.startswith(''):
# raise Exception(content.split('
')[1].split('
')[0])
# results = JsonTable(json.loads(content)['ResultSet']['Result'])
# return results
results = query_with(
interface=self._intf,
join_field='xnat:subjectData/SUBJECT_ID',
common_field='SUBJECT_ID',
return_values=['xnat:subjectData/PROJECT',
'xnat:subjectData/SUBJECT_ID'],
_filter=constraints
)
searchpop = ['%s/projects/' % self._intf._entry + \
'%(project)s/subjects/%(subject_id)s' % res
for res in results
]
cobj = self
while cobj:
first = cobj.first()
if not first:
break
if uri_nextlast(first._uri) == 'subjects':
break
else:
cobj = getattr(cobj, '_cbase')
backup_header = cobj._id_header
if cobj._pattern != '*':
cobj._id_header = 'ID'
poi = set(searchpop
).intersection([eobj._uri for eobj in cobj])
else:
poi = searchpop
cobj._cbase = list(poi)
cobj._ctype = 'cobjecteuris'
cobj._nested = None
cobj._id_header = backup_header
return self
# specialized classes
class Project(EObject):
__metaclass__ = ElementType
def __init__(self, uri, interface):
"""
Parameters
----------
uri: string
The file resource URI
interface: Interface Object
"""
EObject.__init__(self, uri, interface)
# self.pipelines = Pipelines(self.id(), self._intf)
def prearchive_code(self):
""" Gets project prearchive code.
"""
return int(self._intf._exec(join_uri(self._uri, 'prearchive_code')))
def set_prearchive_code(self, code):
""" Sets project prearchive code.
Parameters
----------
code: 0 to 4
"""
self._intf._exec(join_uri(self._uri, 'prearchive_code', code),
'PUT')
def quarantine_code(self):
""" Gets project quarantine code.
"""
return int(self._intf._exec(join_uri(self._uri, 'quarantine_code')))
def set_quarantine_code(self, code):
""" Sets project quarantine code.
Parameters
----------
code: 0 to 1
"""
self._intf._exec(join_uri(self._uri, 'quarantine_code', code),
'PUT')
def current_arc(self):
""" Gets project current archive folder on the server.
"""
return self._intf._exec(join_uri(self._uri, 'current_arc'))
def set_subfolder_in_current_arc(self, subfolder):
""" Changes project current archive subfolder on the server.
"""
current_arc = self._intf._exec(join_uri(self._uri, 'current_arc'))
self._intf._exec(join_uri(self._uri, 'current_arc',
current_arc, subfolder),
'PUT')
def accessibility(self):
""" Gets project accessibility.
"""
return self._intf._exec(join_uri(self._uri, 'accessibility'), 'GET')
def set_accessibility(self, accessibility='protected'):
""" Sets project accessibility.
.. note::
Write access is given or not by the user level for a
specific project.
Parameters
----------
accessibility: public | protected | private
Sets the project accessibility:
- public: the project is visible and provides read
access for anyone.
- protected: the project is visible by anyone but the
data is accessible for allowed users only.
- private: the project is visible by allowed users only.
"""
return self._intf._exec(join_uri(self._uri, 'accessibility',
accessibility), 'PUT')
def users(self):
""" Gets all registered users for this project.
"""
return JsonTable(self._intf._get_json(join_uri(self._uri, 'users'))
).get('login', always_list=True)
def owners(self):
""" Gets owners of this project.
"""
return JsonTable(self._intf._get_json(join_uri(self._uri, 'users'))
).where(displayname='Owners'
).get('login', always_list=True)
def members(self):
""" Gets members of this project.
"""
return JsonTable(self._intf._get_json(join_uri(self._uri, 'users'))
).where(displayname='Members'
).get('login', always_list=True)
def collaborators(self):
""" Gets collaborator of this project.
"""
return JsonTable(self._intf._get_json(join_uri(self._uri, 'users'))
).where(displayname='Collaborators'
).get('login', always_list=True)
def user_role(self, login):
""" Gets the user level of the user for this project.
Parameters
----------
login: string
A user of the project.
Returns
-------
string : owner | member | collaborator
"""
return JsonTable(self._intf._get_json(join_uri(self._uri, 'users'))
).where(login=login
)['displayname'].lower().rstrip('s')
def add_user(self, login, role='member'):
""" Adds a user to the project. The user must already exist on
the server.
Parameters
----------
login: string
Valid username for the XNAT database.
role: owner | member | collaborator
The user level for this project:
- owner: read and write access, as well as
administrative privileges such as adding and removing
users.
- member: read access and can create new resources but
not remove them.
- collaborator: read access only.
"""
self._intf._exec(join_uri(self._uri, 'users',
role.lstrip('s').title() + 's',
login
),
'PUT')
def remove_user(self, login):
""" Removes a user from the project.
Parameters
----------
login: string
Valid username for the XNAT database.
"""
self._intf._exec(join_uri(self._uri, 'users',
self.user_role(login).title()+'s',
login
),
'DELETE')
def datatype(self):
return 'xnat:projectData'
def experiments(self, id_filter='*'):
return Experiments('%s/experiments' % self._intf._entry,
self._intf,
id_filter,
filters={'project':self.id()}
)
def experiment(self, ID):
return Experiment('%s/experiments/%s' % (self._intf._entry, ID),
self._intf
)
def last_modified(self):
""" Gets the last modified dates for all the project subjects.
If any element related to a subject changes, experiment,
variable, scan, image etc... the date will be changed.
"""
uri = '%s/subjects?columns=last_modified' % self._uri
return dict(JsonTable(self._intf._get_json(uri),
order_by=['ID', 'last_modified']
).select(['ID', 'last_modified']
).items()
)
def add_custom_variables(self, custom_variables, allow_data_deletion=False):
"""Adds a custom variable to a specified group
Parameters
----------
custom_variables: a dictionary
allow_data_deletion : a boolean
Examples
--------
>>> variables = {'Subjects' : {'newgroup' : {'foo' : 'string', 'bar' : 'int'}}}
>>> project.add_custom_variables(variables)
"""
tree = lxml.etree.fromstring(self.get())
update = False
for protocol, value in custom_variables.items():
try:
protocol_element = tree.xpath(
"//xnat:studyProtocol[@name='%s']" % protocol,
namespaces=tree.nsmap).pop()
except IndexError:
raise ValueError(
'Protocol %s not in current schema' % protocol
)
try:
definitions_element = protocol_element.xpath(
'xnat:definitions', namespaces = tree.nsmap).pop()
except IndexError:
update = True
definitions_element = lxml.etree.Element(
lxml.etree.QName(tree.nsmap['xnat'],'definitions'),
nsmap=tree.nsmap
)
protocol_element.append(definitions_element)
for group, fields in value.items():
try:
group_element = definitions_element.xpath(
"xnat:definition[@ID='%s']" % group,
namespaces = tree.nsmap).pop()
fields_element = group_element.xpath(
"xnat:fields",
namespaces = tree.nsmap).pop()
except IndexError:
update = True
group_element = lxml.etree.Element(
lxml.etree.QName(tree.nsmap['xnat'],'definition'),
nsmap = tree.nsmap
)
group_element.set('ID', group)
group_element.set(
'data-type', protocol_element.get('data-type'))
group_element.set('description','')
group_element.set('project-specific','1')
definitions_element.append(group_element)
fields_element = lxml.etree.Element(
lxml.etree.QName(tree.nsmap['xnat'],'fields'),
nsmap = tree.nsmap
)
group_element.append(fields_element)
for field, datatype in fields.items():
try:
field_element = fields_element.xpath(
"xnat:field[@name='%s']" % field,
namespaces = tree.nsmap).pop()
except IndexError:
field_element = lxml.etree.Element(
lxml.etree.QName(tree.nsmap['xnat'],'field'),
nsmap = tree.nsmap)
field_element.set('name', field)
field_element.set('datatype', datatype)
field_element.set('type', 'custom')
field_element.set('required', '0')
field_element.set(
'xmlPath',
"xnat:%s/fields/field[name=%s]/field" % (
protocol_element.get(
'data-type').split(':')[-1], field)
)
fields_element.append(field_element)
update = True
if update:
body, content_type = httputil.file_message(
lxml.etree.tostring(tree),
'text/xml',
'cust.xml',
'cust.xml'
)
uri = self._uri
if allow_data_deletion:
uri = self._uri + '?allowDataDeletion=true'
self._intf._exec(uri, method='PUT', body=body,
headers= {'content-type':content_type})
def get_custom_variables(self):
"""Retrieves custom variables as a dictionary
It has the format {studyProtocol: { setname : {field: type, ...}}}
"""
tree = lxml.etree.fromstring(self.get())
nsmap = tree.nsmap
custom_variables = {}
for studyprotocols in tree.xpath('//xnat:studyProtocol',
namespaces=nsmap):
protocol_name = studyprotocols.get('name')
custom_variables[protocol_name] = {}
for definition in studyprotocols.xpath(('xnat:definitions'
'/xnat:definition'),
namespaces=nsmap):
definition_id = definition.get('ID')
custom_variables[protocol_name][definition_id] = {}
for field in definition.xpath('xnat:fields/xnat:field',
namespaces=nsmap):
field_name = field.get('name')
if field.get('type') == 'custom':
custom_variables[protocol_name][definition_id][
field_name] = field.get('datatype')
return custom_variables
class Subject(EObject):
__metaclass__ = ElementType
def datatype(self):
return 'xnat:subjectData'
def shares(self, id_filter='*'):
""" Returns the projects sharing this subject.
Returns
-------
Collection object.
"""
return Projects(join_uri(self._uri, 'projects'),
self._intf, id_filter)
def share(self, project):
""" Share this subject with another project.
Parameters
----------
project: string
The other project name.
"""
self._intf._exec(join_uri(self._uri, 'projects', project), 'PUT')
def unshare(self, project):
""" Remove subject from another project in which it was shared.
Parameters
----------
project: string
The other project name.
"""
self._intf._exec(join_uri(self._uri, 'projects', project), 'DELETE')
class Experiment(EObject):
__metaclass__ = ElementType
def shares(self, id_filter='*'):
""" Returns the projects sharing this experiment.
Returns
-------
Collection object.
"""
return Projects(join_uri(self._uri, 'projects'),
self._intf, id_filter)
def share(self, project):
""" Share this experiment with another project.
Parameters
----------
project: string
The other project name.
"""
self._intf._exec(join_uri(self._uri, 'projects', project), 'PUT')
def unshare(self, project):
""" Remove experiment from another project in which it was shared.
Parameters
----------
project: string
The other project name.
"""
self._intf._exec(join_uri(self._uri, 'projects', project), 'DELETE')
def trigger_pipelines(self):
""" Triggers the AutoRun pipeline.
"""
self._intf._exec(self._uri+'?triggerPipelines=true', 'PUT')
def fix_scan_types(self):
""" Populate empty scan TYPE attributes based on how similar
scans were populated.
"""
self._intf._exec(self._uri+'?fixScanTypes=true', 'PUT')
def pull_data_from_headers(self):
""" Pull DICOM header values into the session.
"""
self._intf._exec(self._uri+'?pullDataFromHeaders=true', 'PUT')
def trigger(self, pipelines=True, fix_types=True, scan_headers=True):
""" Run several triggers in a single call.
Parameters
----------
pipelines: boolean
Same as trigger_pipelines.
fix_types: boolean
Same as fix_scan_types.
scan_headers: boolean
Same as pull_data_from headers.
"""
if not all([not pipelines, not fix_types, not scan_headers]):
options = []
if pipelines:
options.append('triggerPipelines=true')
if fix_types:
options.append('fixScanTypes=true')
if scan_headers:
options.append('pullDataFromHeaders=true')
options = '?' + '&'.join(options)
self._intf._exec(self._uri + options, 'PUT')
class Assessor(EObject):
__metaclass__ = ElementType
def __init__(self, uri, interface):
EObject.__init__(self, uri, interface)
self.provenance = Provenance(self)
def shares(self, id_filter='*'):
""" Returns the projects sharing this assessor.
Returns
-------
Collection object.
"""
return Projects(join_uri(self._uri, 'projects'),
self._intf, id_filter)
def share(self, project):
""" Share this assessor with another project.
Parameters
----------
project: string
The other project name.
"""
self._intf._exec(join_uri(self._uri, 'projects', project), 'PUT')
def unshare(self, project):
""" Remove assessor from another project in which it was shared.
Parameters
----------
project: string
The other project name.
"""
self._intf._exec(join_uri(self._uri, 'projects', project), 'DELETE')
def set_param(self, key, value):
self.attrs.set('%s/parameters/addParam[name=%s]/addField' \
% (self.datatype(), key),
value
)
def get_param(self, key):
return self.xpath(
"//xnat:addParam[@name='%s']/child::text()" % key)[-1]
def get_params(self):
return self.xpath("//xnat:addParam/child::text()")[1::2]
def params(self):
return self.xpath('//xnat:addParam/attribute::*')
class Reconstruction(EObject):
__metaclass__ = ElementType
def __init__(self, uri, interface):
EObject.__init__(self, uri, interface)
self.provenance = Provenance(self)
def datatype(self):
return (super(Reconstruction, self).datatype()
or 'xnat:reconstructedImageData'
)
class Scan(EObject):
__metaclass__ = ElementType
def set_param(self, key, value):
self.attrs.set('%s/parameters/addParam[name=%s]/addField' \
% (self.datatype(), key),
value
)
def get_param(self, key):
return self.xpath(
"//xnat:addParam[@name='%s']/child::text()" % key)[-1]
def get_params(self):
return self.xpath("//xnat:addParam/child::text()")[1::2]
def params(self):
return self.xpath('//xnat:addParam/attribute::*')
class Resource(EObject):
__metaclass__ = ElementType
def get(self, dest_dir, extract=False):
""" Downloads all the files within a resource.
..warning::
Currently XNAT adds parent folders in the zip file that
is downloaded to avoid name clashes if several resources
are downloaded in the same folder. In order to be able to
download the data uploaded previously with the same
structure, pyxnat extracts the zip file, remove the exra
paths and if necessary re-zips it. Careful, it may take
time, and there is the problem of name clashes.
Parameters
----------
dest_dir: string
Destination directory for the resource data.
extract: boolean
If True, the downloaded zip file is extracted.
If False, not extracted.
Returns
-------
If extract is False, the zip file path.
If extract is True, the list of file paths previously in
the zip.
"""
zip_location = os.path.join(dest_dir, uri_last(self._uri) + '.zip')
if dest_dir is not None:
self._intf._http.cache.preset(zip_location)
self._intf._exec(join_uri(self._uri, 'files')+'?format=zip')
fzip = zipfile.ZipFile(zip_location, 'r')
fzip.extractall(path=dest_dir)
fzip.close()
members = []
for member in fzip.namelist():
old_path = os.path.join(dest_dir, member)
print member
print member.split('files',1)
new_path = os.path.join(
dest_dir,
uri_last(self._uri)
#, member.split('files', 1)[1].split(os.sep, 2)[2]
)
if not os.path.exists(os.path.dirname(new_path)):
os.makedirs(os.path.dirname(new_path))
shutil.move(old_path, new_path)
members.append(new_path)
# TODO: cache.delete(...)
for extracted in fzip.namelist():
pth = os.path.join(dest_dir, extracted.split(os.sep, 1)[0])
if os.path.isdir(pth):
shutil.rmtree(pth)
os.remove(zip_location)
if not extract:
fzip = zipfile.ZipFile(zip_location, 'w')
arcprefix = os.path.commonprefix(members)
arcroot = '/%s' % os.path.split(arcprefix.rstrip('/'))[1]
for member in members:
fzip.write(member, os.path.join(arcroot,
member.split(arcprefix)[1])
)
fzip.close()
unzippedTree = os.path.join(dest_dir, uri_last(self._uri))
if os.path.exists(unzippedTree):
if os.path.isdir(unzippedTree):
shutil.rmtree(os.path.join(dest_dir, uri_last(self._uri)))
else :
os.remove(unzippedTree)
return zip_location if os.path.exists(zip_location) else members
def put(self, sources, **datatypes):
""" Insert a list of files in a single resource element.
This method takes all the files an creates a zip with them
which will be the element to be uploaded and then extracted on
the server.
"""
zip_location = tempfile.mkstemp(suffix='.zip')[1]
arcprefix = os.path.commonprefix(sources)
arcroot = '/%s' % os.path.split(arcprefix.rstrip('/'))[1]
fzip = zipfile.ZipFile(zip_location, 'w')
for src in sources:
fzip.write(src, os.path.join(arcroot, src.split(arcprefix)[1]))
fzip.close()
self.put_zip(zip_location, **datatypes)
os.remove(zip_location)
def put_zip(self, zip_location, **datatypes):
""" Uploads a zip or tgz file an then extracts it on the server.
After the compressed file is extracted the individual
files are accessible separately, or as a whole using get_zip.
"""
if not self.exists():
self.create(**datatypes)
self.file(os.path.split(zip_location)[1] + '?extract=true'
).put(zip_location)
def put_dir(self, src_dir, **datatypes):
""" Finds recursively all the files in a folder and uploads
them using `insert`.
"""
self.put(find_files(src_dir), **datatypes)
batch_insert = put
zip_insert = put_zip
dir_insert = put_dir
def datatype(self):
return (super(Reconstruction, self).datatype()
or 'xnat:abstractResource'
)
class In_Resource(Resource):
__metaclass__ = ElementType
class Out_Resource(Resource):
__metaclass__ = ElementType
class File(EObject):
""" EObject for files stored in XNAT.
"""
__metaclass__ = ElementType
def __init__(self, uri, interface):
"""
Parameters
----------
uri: string
The file resource URI
interface: Interface Object
"""
EObject.__init__(self, uri, interface)
self._absuri = None
def attributes(self):
""" Files attributes include:
- URI
- Name
- Size in bytes
- file_tags
- file_format
- file_content
Returns
-------
dict : a dictionnary with the file attributes
"""
return self._getcells(['URI', 'Name', 'Size',
'file_tags', 'file_format', 'file_content'])
def get(self, dest=None, force_default=False):
""" Downloads the file to the cache directory.
.. note::
The default cache path is computed like this:
``path_to_cache/md5(uri + query_string)_filename``
Parameters
----------
dest: string | None
- If None a default path in the cache folder is
automatically computed.
- Else the file is downloaded at the requested location.
force_default: boolean
- Has no effect if the file is downloaded for the first time
- If the file was previously download with a custom path,
calling get() will remember the custom location unless:
- another custom location is set in dest
- force_default is set to True and the file will be
moved to the cache
Returns
-------
string : the file location.
"""
if not self._absuri:
self._absuri = self._getcell('URI')
if self._absuri is None:
raise DataError('Cannot get file: does not exists')
if dest is not None:
self._intf._http.cache.preset(dest)
elif not force_default:
_location = \
self._intf._http.cache.get_diskpath(
'%s%s' % (self._intf._server, self._absuri)
)
self._intf._http.cache.preset(_location)
self._intf._exec(self._absuri, 'GET')
return self._intf._http.cache.get_diskpath(
'%s%s' % (self._intf._server, self._absuri)
)
def get_copy(self, dest=None):
""" Downloads the file to the cache directory but creates a copy at
the specified location.
Parameters
----------
dest: string | None
- file path for the copy
- if None a copy is created at a default location based
on the file URI on the server
Returns
-------
string : the copy location.
"""
if not dest:
dest = os.path.join(self._intf._http.cache.cache, 'workspace',
*self._absuri.strip('/').split('/')[1:])
if not os.path.exists(os.path.dirname(dest)):
os.makedirs(os.path.dirname(dest))
src = self.get()
if src != dest:
shutil.copy2(src, dest)
return dest
def put(self, src, format='U', content='U', tags='U', **datatypes):
""" Uploads a file to XNAT.
Parameters
----------
src: string
Location of the local file to upload or the actual content
to upload.
format: string
Optional parameter to specify the file format.
Defaults to 'U'.
content: string
Optional parameter to specify the file content.
Defaults to 'U'.
tags: string
Optional parameter to specify tags for the file.
Defaults to 'U'.
"""
format = urllib.quote(format)
content = urllib.quote(content)
tags = urllib.quote(tags)
try:
if os.path.exists(src):
path = src
name = os.path.basename(path).split('?')[0]
src = codecs.open(src).read()
else:
path = self._uri.split('/')[-1]
name = path
except:
path = self._uri.split('/')[-1]
name = path
content_type = mimetypes.guess_type(path)[0] or \
'application/octet-stream'
body, content_type = httputil.file_message(src, content_type,
path, name
)
guri = uri_grandparent(self._uri)
if not self._intf.select(guri).exists():
self._intf.select(guri).insert(**datatypes)
resource_id = self._intf.select(guri).id()
self._absuri = urllib.unquote(
re.sub('resources/.*?/',
'resources/%s/' % resource_id, self._uri)
)
query_args = {
'format': format,
'content': content,
'tags': tags,
}
if '?' in self._absuri:
k, v = self._absuri.split('?')[1].split('=')
query_args[k] = v
self._absuri = self._absuri.split('?')[0]
put_uri = '%s?%s' % (
self._absuri,
'&'.join('%s=%s' % (k,v) for k, v in query_args.items())
)
# print 'INSERT FILE', os.path.exists(src)
self._intf._exec(
put_uri, 'PUT', body,
headers={'content-type':content_type}
)
# track the uploaded file as one of the cache
# print 'GET DISKPATH', os.path.exists(src)
# _cachepath = self._intf._http.cache.get_diskpath(
# '%s%s' % (self._intf._server, self._absuri),
# force_default=True
# )
# _fakepath = '%s.alt' % _cachepath
# _headerpath = '%s.headers' % _cachepath
# print 'WRITE REFFILE', os.path.exists(src)
# reffile = open(_fakepath, 'wb')
# reffile.write(src)
# reffile.close()
# info_head = self._intf._get_head(self._absuri)
# print 'WRITE HEADER FILE', os.path.exists(src)
# headerfile = open(_headerpath, 'wb')
# headerfile.write(info_head.as_string())
# headerfile.close()
insert = put
create = put
def delete(self):
""" Deletes the file on the server.
"""
if not self._absuri:
self._absuri = self._getcell('URI')
if self._absuri is None:
raise DataError('Cannot delete file: does not exists')
return self._intf._exec(self._absuri, 'DELETE')
def size(self):
""" Gets the file size.
"""
return self._getcell('Size')
def labels(self):
""" Gets the file labels.
"""
return self._getcell('file_tags')
def format(self):
""" Gets the file format.
"""
return self._getcell('file_format')
def content(self):
""" Gets the file content description.
"""
return self._getcell('file_content')
class In_File(File):
__metaclass__ = ElementType
class Out_File(File):
__metaclass__ = ElementType
class Projects(CObject):
__metaclass__ = CollectionType
class Subjects(CObject):
__metaclass__ = CollectionType
def sharing(self, projects=[]):
return Subjects([eobj for eobj in self
if set(projects).issubset(eobj.shares().get())
],
self._intf
)
def share(self, project):
for eobj in self:
eobj.share(project)
def unshare(self, project):
for eobj in self:
eobj.unshare(project)
class Experiments(CObject):
__metaclass__ = CollectionType
def sharing(self, projects=[]):
return Experiments([eobj for eobj in self
if set(projects).issubset(eobj.shares().get())
],
self._intf
)
def share(self, project):
for eobj in self:
eobj.share(project)
def unshare(self, project):
for eobj in self:
eobj.unshare(project)
class Assessors(CObject):
__metaclass__ = CollectionType
def sharing(self, projects=[]):
return Assessors([eobj for eobj in self
if set(projects).issubset(eobj.shares().get())
],
self._intf
)
def share(self, project):
for eobj in self:
eobj.share(project)
def unshare(self, project):
for eobj in self:
eobj.unshare(project)
def download (self, dest_dir, type="ALL",
name=None, extract=False, safe=False):
"""
A wrapper around :func:`downloadutils.download`
"""
return downloadutils.download(dest_dir, self, type, name,
extract, safe)
class Reconstructions(CObject):
__metaclass__ = CollectionType
def download (self, dest_dir, type="ALL",
name=None, extract=False, safe=False):
"""
A wrapper around :func:`downloadutils.download`
"""
return downloadutils.download(dest_dir, self, type, name,
extract, safe)
class Scans(CObject):
__metaclass__ = CollectionType
def download (self, dest_dir, type="ALL",
name=None, extract=False, safe=False):
"""
A wrapper around :func:`downloadutils.download`
"""
return downloadutils.download(dest_dir, self, type, name,
extract, safe)
class Resources(CObject):
__metaclass__ = CollectionType
class In_Resources(Resources):
__metaclass__ = CollectionType
class Out_Resources(Resources):
__metaclass__ = CollectionType
class Files(CObject):
__metaclass__ = CollectionType
class In_Files(Files):
__metaclass__ = CollectionType
class Out_Files(Files):
__metaclass__ = CollectionType
## Utility functions for downloading and extracting zip archives
def _datatypes_from_query(query):
datatypes = []
for constraint in query:
if isinstance(constraint, list):
datatypes.extend(_datatypes_from_query(constraint))
elif isinstance(constraint, tuple):
datatypes.append(constraint[0].split('/')[0])
return datatypes
def query_with(interface, join_field,
common_field, return_values, _filter):
_stm = (join_field.split('/')[0], return_values)
_cls = rewrite_query(interface, join_field,
common_field, _filter)
return interface.select(*_stm).where(_cls)
def rewrite_query(interface, join_field,
common_field, _filter):
_new_filter = []
for _f in _filter:
if isinstance(_f, list):
_new_filter.append(rewrite_query(
interface, join_field, common_field, _f))
elif isinstance(_f, tuple):
_datatype = _f[0].split('/')[0]
_res = interface.select(
_datatype, ['%s/%s' % (_datatype, common_field)]
).where([_f, 'AND'])
_new_f = [(join_field, '=', '%s' % sid)
for sid in _res['subject_id']
]
_new_f.append('OR')
_new_filter.append(_new_f)
elif isinstance(_f, (str, unicode)):
_new_filter.append(_f)
else:
raise Exception('Invalid filter')
return _new_filter
pyxnat-0.9.0~dev0/pyxnat/core/schema.py 0000664 0000000 0000000 00000012217 11632421554 0020113 0 ustar 00root root 0000000 0000000
# REST collection resources tree
resources_tree = {
'projects' :['subjects', 'resources'],
'subjects' :['experiments', 'resources'],
'experiments' :['assessors', 'reconstructions', 'scans', 'resources'],
'assessors' :['resources', 'in_resources','out_resources'],
'reconstructions' :['in_resources','out_resources'],
'scans' :['resources'],
'resources' :['files'],
'files' :[],
'in_resources' :['files'],
'in_files' :[],
'out_resources' :['files'],
'out_files' :[],
}
prearc_tree = {'projects' :['scans'],
'scans' :['resources'],
'resources' :['files']
}
# REST resources that are not natively supported
extra_resources_tree = {'projects':['assessors', 'scans', 'reconstructions'],
'subjects':['assessors', 'scans', 'reconstructions'],
}
# REST translation table
rest_translation = {'in_resources':'in/resources',
'in_files':'in/files',
'out_resources':'out/resources',
'out_files':'out/files',
'in_resource':'in/resource',
'in_file':'in/file',
'out_resource':'out/resource',
'out_file':'out/file',
}
# REST json format
json = {'projects':['ID', 'ID'],
'subjects':['ID', 'label'],
'experiments':['ID', 'label'],
'assessors':['ID', 'label'],
'reconstructions':['ID', 'label'],
'scans':['ID', 'ID'],
'resources':['xnat_abstractresource_id', 'label'],
'out_resources':['xnat_abstractresource_id', 'label'],
'in_resources':['xnat_abstractresource_id', 'label'],
'files':['Name', 'Name'],
}
resources_singular = [key.rsplit('s', 1)[0] for key in resources_tree.keys()]
resources_plural = resources_tree.keys()
resources_types = resources_singular + resources_plural
default_datatypes = {'projects':'xnat:projectData',
'subjects':'xnat:subjectData',
'experiments':'xnat:mrSessionData',
'assessors':'xnat:mrAssessorData',
'reconstructions':'xnat:reconstructedImageData',
'scans':'xnat:mrScanData',
'resources':None,
'in_resources':None,
'out_resources':None,
}
def datatype_attributes(root, datatype):
def _iterchildren(node, pathsofar):
elements = []
for child in node.iterchildren():
if isinstance(child.tag, basestring) \
and child.tag.split('}')[1] == 'element':
elements.append('%s/%s'%(pathsofar, child.get('name')))
elements.extend(_iterchildren(child, '%s/%s'%
(pathsofar, child.get('name')))
)
elif isinstance(child.tag, basestring) \
and child.tag.split('}')[1] == 'attribute':
elements.append('%s/%s'%(pathsofar, child.get('name')))
elif isinstance(child.tag, basestring) \
and child.tag.split('}')[1] == 'extension':
ct_xpath = "/xs:schema/xs:complexType[@name='%s']"% \
child.get('base').split(':')[1]
for complex_type in node.getroottree(
).xpath(ct_xpath, namespaces=child.nsmap):
same = False
for ancestor in child.iterancestors():
if ancestor.get('name') == \
child.get('base').split(':')[1]:
same = True
break
if not same:
elements.extend(_iterchildren(complex_type,
pathsofar)
)
elements.extend(_iterchildren(child, pathsofar))
else:
elements.extend(_iterchildren(child, pathsofar))
return elements
ct_xpath = "/xs:schema/xs:complexType[@name='%s']" % \
datatype.split(':')[1]
attributes = []
for complex_type in root.xpath(ct_xpath, namespaces=root.nsmap):
for child in complex_type.iterchildren():
attributes.extend(_iterchildren(child, datatype))
return attributes
def datatypes(root):
nsmap = get_nsmap(root)
return [element.get('type')
for element in root.xpath('/xs:schema/xs:element',
namespaces=nsmap)
]
def get_nsmap(node):
nsmap = node.nsmap
none_ns = node.nsmap.get(None)
if none_ns != None:
nsmap[none_ns.rsplit('/', 1)[1]] = none_ns
del nsmap[None]
return nsmap
def class_name(self):
"""
Return the name of this class without qualification.
eg. If the class name is "x.y.class" return only "class"
"""
return self.__class__.__name__.split('.')[-1]
pyxnat-0.9.0~dev0/pyxnat/core/search.py 0000664 0000000 0000000 00000061735 11632421554 0020131 0 ustar 00root root 0000000 0000000 import os
import re
import glob
import csv
import difflib
from StringIO import StringIO
from lxml import etree
import json
from .jsonutil import JsonTable, get_column, get_where, get_selection
from .errors import is_xnat_error, catch_error
from .errors import ProgrammingError, NotSupportedError
from .errors import DataError, DatabaseError
from .uriutil import check_entry
search_nsmap = {'xdat':'http://nrg.wustl.edu/security',
'xsi':'http://www.w3.org/2001/XMLSchema-instance'}
special_ops = {'*':'%', }
def build_search_document(root_element_name, columns, criteria_set,
brief_description='', long_description='',
allowed_users=[]):
root_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'bundle'),
nsmap=search_nsmap
)
root_node.set('ID', "@%s" % root_element_name)
root_node.set('brief-description', brief_description)
root_node.set('description', long_description)
root_node.set('allow-diff-columns', "0")
root_node.set('secure', "false")
root_element_name_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'root_element_name'),
nsmap=search_nsmap
)
root_element_name_node.text = root_element_name
root_node.append(root_element_name_node)
for i, column in enumerate(columns):
element_name, field_ID = column.split('/')
search_field_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'search_field'),
nsmap=search_nsmap
)
element_name_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'element_name'),
nsmap=search_nsmap
)
element_name_node.text = element_name
field_ID_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'field_ID'),
nsmap=search_nsmap
)
field_ID_node.text = field_ID
sequence_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'sequence'),
nsmap=search_nsmap
)
sequence_node.text = str(i)
type_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'type'),
nsmap=search_nsmap
)
type_node.text = 'string'
header_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'header'),
nsmap=search_nsmap
)
header_node.text = column
search_field_node.extend([element_name_node,
field_ID_node,
sequence_node,
type_node, header_node
])
root_node.append(search_field_node)
search_where_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'search_where'),
nsmap=search_nsmap
)
root_node.append(build_criteria_set(search_where_node, criteria_set))
if allowed_users != []:
allowed_users_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'allowed_user'),
nsmap=search_nsmap
)
for allowed_user in allowed_users:
login_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'login'),
nsmap=search_nsmap
)
login_node.text = allowed_user
allowed_users_node.append(login_node)
root_node.append(allowed_users_node)
return etree.tostring(root_node.getroottree())
def build_criteria_set(container_node, criteria_set):
for criteria in criteria_set:
if isinstance(criteria, basestring):
container_node.set('method', criteria)
if isinstance(criteria, (list)):
sub_container_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'child_set'),
nsmap=search_nsmap
)
container_node.append(
build_criteria_set(sub_container_node, criteria))
if isinstance(criteria, (tuple)):
if len(criteria) != 3:
raise ProgrammingError('%s should be a 3-element tuple' %
str(criteria)
)
constraint_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'criteria'),
nsmap=search_nsmap
)
constraint_node.set('override_value_formatting', '0')
schema_field_node = \
etree.Element(etree.QName(search_nsmap['xdat'],
'schema_field'
),
nsmap=search_nsmap
)
schema_field_node.text = criteria[0]
comparison_type_node = \
etree.Element(etree.QName(search_nsmap['xdat'],
'comparison_type'
),
nsmap=search_nsmap
)
comparison_type_node.text = special_ops.get(criteria[1],
criteria[1]
)
value_node = \
etree.Element(etree.QName(search_nsmap['xdat'], 'value'),
nsmap=search_nsmap
)
value_node.text = criteria[2].replace('*', special_ops['*'])
constraint_node.extend([
schema_field_node, comparison_type_node, value_node])
container_node.append(constraint_node)
return container_node
def query_from_xml(document):
query = {}
root = etree.fromstring(document)
_nsmap = root.nsmap
query['description'] = root.get('description', default="")
query['row'] = root.xpath('xdat:root_element_name',
namespaces=root.nsmap)[0].text
query['columns'] = []
for node in root.xpath('xdat:search_field',
namespaces=_nsmap):
en = node.xpath('xdat:element_name', namespaces=root.nsmap)[0].text
fid = node.xpath('xdat:field_ID', namespaces=root.nsmap)[0].text
query['columns'].append('%s/%s' % (en, fid))
query['users'] = [
node.text
for node in root.xpath('xdat:allowed_user/xdat:login',
namespaces=root.nsmap
)
]
try:
search_where = root.xpath('xdat:search_where',
namespaces=root.nsmap)[0]
query['constraints'] = query_from_criteria_set(search_where)
except:
query['constraints'] = [('%s/ID' % query['row'], 'LIKE', '%'), 'AND']
return query
def query_from_criteria_set(criteria_set):
query = []
query.append(criteria_set.get('method'))
_nsmap = criteria_set.nsmap
for criteria in criteria_set.xpath('xdat:criteria',
namespaces=_nsmap):
_f = criteria.xpath('xdat:schema_field', namespaces=_nsmap)[0]
_o = criteria.xpath('xdat:comparison_type', namespaces=_nsmap)[0]
_v = criteria.xpath('xdat:value', namespaces=_nsmap)[0]
constraint = (_f.text, _o.text, _v.text)
query.insert(0, constraint)
for child_set in criteria_set.xpath('xdat:child_set',
namespaces=_nsmap):
query.insert(0, query_from_criteria_set(child_set))
return query
def rpn_contraints(rpn_exp):
left = []
right = []
triple = []
for i, t in enumerate(rpn_exp.split()):
if t in ['AND', 'OR']:
if 'AND' in right or 'OR' in right and left == []:
try:
operator = right.pop(right.index('AND'))
except:
operator = right.pop(right.index('OR'))
left = [right[0]]
left.append(right[1:] + [t])
left.append(operator)
right = []
elif right != []:
right.append(t)
if left != []:
left.append(right)
else:
left = right[:]
right = []
elif right == [] and left != []:
left = [left]
left.append(t)
right = left[:]
left = []
else:
raise ProgrammingError('in expression %s' % rpn_exp)
else:
triple.append(t)
if len(triple) == 3:
right.append(tuple(triple))
triple = []
return left if left != [] else right
# ---------------------------------------------------------------
class SearchManager(object):
""" Search interface.
Handles operations to save and get back searches on the server.
Examples
--------
>>> row = 'xnat:subjectData'
>>> columns = ['xnat:subjectData/PROJECT',
'xnat:subjectData/SUBJECT_ID'
]
>>> criteria = [('xnat:subjectData/SUBJECT_ID', 'LIKE', '*'),
'AND'
]
>>> interface.manage.search.save('mysearch', row, columns,
criteria, sharing='public',
description='my first search'
)
"""
def __init__(self, interface):
self._intf = interface
def _save_search(self, row, columns, constraints, name, desc, sharing):
self._intf._get_entry_point()
name = name.replace(' ', '_')
if sharing == 'private':
users = [self._intf._user]
elif sharing == 'public':
users = []
elif isinstance(sharing, list):
users = sharing
else:
raise NotSupportedError('Share mode %s not valid' % sharing)
self._intf._exec(
'%s/search/saved/%s?inbody=true' % (self._intf._entry, name),
method='PUT',
body=build_search_document(row, columns,
constraints,
name, desc.replace('%', '%%'),
users
)
)
def save(self, name, row, columns, constraints,
sharing='private', description=''):
""" Saves a query on the XNAT server.
Parameters
----------
name: string
Name of the query displayed on the Web Interface and
used to get back the results.
row: string
Datatype from `Interface.inspect.datatypes()`.
Usually ``xnat:subjectData``
columns: list
List of data fields from
`Interface.inspect.datatypes('*', '*')`
constraints: list
See also: `Search.where()`
sharing: string | list
Define by whom the query is visible.
If sharing is a string it may be either
``private`` or ``public``.
Otherwise a list of valid logins for the XNAT server
from `Interface.users()`.
See Also
--------
Search.where
"""
self._save_search(row, columns, constraints,
name, description, sharing)
def saved(self, with_description=False):
""" Returns the names of accessible saved search on the server.
"""
self._intf._get_entry_point()
jdata = self._intf._get_json(
'%s/search/saved?format=json' % self._intf._entry)
if with_description:
return [(ld['brief_description'],
ld['description'].replace('%%', '%'))
for ld in get_selection(jdata, ['brief_description',
'description'
]
)
if not ld['brief_description'].startswith('template_')]
else:
return [name
for name in get_column(jdata, 'brief_description')
if not name.startswith('template_')]
def get(self, name, out_format='results'):
""" Returns the results of the query saved on the XNAT server or
the query itself to know what it does.
Parameters
----------
name: string
Name of the saved search. An exception is raised if the name
does not exist.
out_format: string
Can take the following values:
- results to download the results of the search
- xml to download the XML document defining the search
- query to get the pyxnat representation of the search
"""
self._intf._get_entry_point()
jdata = self._intf._get_json(
'%s/search/saved?format=json' % self._intf._entry)
try:
search_id = get_where(jdata, brief_description=name)[0]['id']
except IndexError:
raise DatabaseError('%s not found' % name)
if out_format in ['xml', 'query']:
bundle = self._intf._exec(
'%s/search/saved/%s' % (self._intf._entry,
search_id
), 'GET')
if out_format == 'xml':
return bundle
else:
return query_from_xml(bundle)
content = self._intf._exec(
'%s/search/saved/%s/results?format=csv' % (self._intf._entry,
search_id), 'GET')
results = csv.reader(StringIO(content), delimiter=',', quotechar='"')
headers = results.next()
return JsonTable([dict(zip(headers, res))
for res in results
],
headers
)
def delete(self, name):
""" Removes the search from the server.
"""
self._intf._get_entry_point()
jdata = self._intf._get_json(
'%s/search/saved?format=json' % self._intf._entry)
try:
search_id = get_where(jdata, brief_description=name)[0]['id']
except IndexError:
raise DatabaseError('%s not found' % name)
self._intf._exec('%s/search/saved/%s' % (self._intf._entry,
search_id
), 'DELETE')
def save_template(self, name, row=None, columns=[],
constraints=[], sharing='private', description=''):
"""
Define and save a search template. Same as the save method, but
the values in the constraints are used as keywords for value
replacement when using the template.
Parameters
----------
name: string
Name under which the template is save in XNAT. A 'template_' is
prepended to the name so that it appear clearly as a template
on the web interface.
row: string
Datatype from `Interface.inspect.datatypes()`.
Usually ``xnat:subjectData``
columns: list
List of data fields from
`Interface.inspect.datatypes('*', '*')`
constraints: list
See also: `Search.where()`, values are keywords for the template
sharing: string | list
Define by whom the query is visible.
If sharing is a string it may be either
``private`` or ``public``.
Otherwise a list of valid logins for the XNAT server
from `Interface.users()`.
"""
def _make_template(query):
query_template = []
for constraint in query:
if isinstance(constraint, tuple):
query_template.append((constraint[0],
constraint[1],
'%%(%s)s' % constraint[2])
)
elif isinstance(constraint, (unicode, str)):
query_template.append(constraint)
elif isinstance(constraint, list):
query_template.append(_make_template(constraint))
else:
raise ProgrammingError('Unrecognized token '
'in query: %s' % constraint
)
return query_template
self._save_search(
row, columns, _make_template(constraints),
'template_%s' % name, description, sharing
)
def saved_templates(self, with_description=False):
""" Returns the names of accessible saved search templates on the server.
"""
self._intf._get_entry_point()
jdata = self._intf._get_json(
'%s/search/saved?format=json' % self._intf._entry)
if with_description:
return [
(ld['brief_description'].split('template_')[1],
ld['description'].replace('%%', '%')
)
for ld in get_selection(jdata, ['brief_description',
'description'
]
)
if ld['brief_description'].startswith('template_')
]
else:
return [name.split('template_')[1]
for name in get_column(jdata, 'brief_description')
if name.startswith('template_')]
def use_template(self, name, values):
"""
Parameters
----------
name: string
Name of the template.
values: dict
Values to put in the template, get the valid keys using
the get_template method.
Examples
--------
>>> interface.manage.search.use_template(name,
{'subject_id':'ID',
'age':'32'
})
"""
self._intf._get_entry_point()
bundle = self.get_template(name, True) % values
# have to remove search_id information before re-posting it
_query = query_from_xml(bundle)
bundle = build_search_document(_query['row'],
_query['columns'],
_query['constraints']
)
content = self._intf._exec(
"%s/search?format=csv" % self._intf._entry, 'POST', bundle)
results = csv.reader(StringIO(content), delimiter=',', quotechar='"')
headers = results.next()
return JsonTable([dict(zip(headers, res))
for res in results
],
headers
)
def get_template(self, name, as_xml=False):
""" Get a saved template, either as an xml document, or as a pyxnat
representation, with the keys to be used in the template
between the parentheses in %()s.
Parameters
----------
name: str
Name under which the template is saved
as_xml: boolean
If True returns an XML document, else return a list of
constraints. Defaults to False.
"""
self._intf._get_entry_point()
jdata = self._intf._get_json(
'%s/search/saved?format=json' % self._intf._entry)
try:
search_id = get_where(jdata,
brief_description='template_%s' % name
)[0]['id']
except IndexError:
raise DatabaseError('%s not found' % name)
bundle = self._intf._exec(
'%s/search/saved/%s' % (self._intf._entry,
search_id
), 'GET')
if as_xml:
return bundle
else:
_query = query_from_xml(bundle)
return _query['row'], _query['columns'], _query['constraints'], _query['description']
def delete_template(self, name):
""" Deletes a search template.
"""
self.delete('template_%s' % name)
def eval_rpn_exp(self, rpnexp):
return rpn_contraints(rpnexp)
class Search(object):
""" Define constraints to make a complex search on the database.
This :class:`Search` is available at different places throughout
the API:
>>> interface.select(DATA_SELECTION).where(QUERY)
>>> interface.manage.search.save('name', TABLE_DEFINITION, QUERY)
Examples
--------
>>> query = [('xnat:subjectData/SUBJECT_ID', 'LIKE', '%'),
('xnat:projectData/ID', '=', 'my_project'),
[('xnat:subjectData/AGE', '>', '14'),
'AND'
],
'OR'
]
"""
def __init__(self, row, columns, interface):
""" Configure the result table.
Parameters
----------
row: string
The returned table will have one line for every matching
occurence of this type.
e.g. xnat:subjectData
--> table with one line per matching subject
columns: list
The returned table will have all the given columns.
"""
self._row = row
self._columns = columns
self._intf = interface
def where(self, constraints=None, template=None, query=None):
""" Triggers the search.
Parameters
----------
contraints: list
A query is an unordered list that contains
- 1 or more constraints
- 0 or more sub-queries (lists as this one)
- 1 comparison method between the constraints
('AND' or 'OR')
A constraint is an ordered tuple that contains
- 1 valid searchable_type/searchable_field
- 1 operator among '=', '<', '>', '<=', '>=', 'LIKE'
Returns
-------
results: JsonTable object
An table-like object containing the results. It is
basically a list of dictionaries that has additional
helper methods.
"""
self._intf._get_entry_point()
if isinstance(constraints, (str, unicode)):
constraints = rpn_contraints(constraints)
elif isinstance(template, (tuple)):
tmp_bundle = self._intf.manage.search.get_template(
template[0], True)
tmp_bundle = tmp_bundle % template[1]
constraints = query_from_xml(tmp_bundle)['constraints']
elif isinstance(query, (str, unicode)):
tmp_bundle = self._intf.manage.search.get(query, 'xml')
constraints = query_from_xml(tmp_bundle)['constraints']
elif isinstance(constraints, list):
pass
else:
raise ProgrammingError('One of contraints, template and query'
'parameters must be correctly set.')
bundle = build_search_document(self._row, self._columns, constraints)
content = self._intf._exec(
"%s/search?format=csv" % self._intf._entry, 'POST', bundle)
if is_xnat_error(content):
catch_error(content)
results = csv.reader(StringIO(content), delimiter=',', quotechar='"')
headers = results.next()
headers_of_interest = []
for column in self._columns:
headers_of_interest.append(
difflib.get_close_matches(
column.split(self._row + '/')[0].lower() \
or column.split(self._row + '/')[1].lower(),
headers)[0]
)
if len(self._columns) != len(headers_of_interest):
raise DataError('unvalid response headers')
return JsonTable([dict(zip(headers, res)) for res in results],
headers_of_interest).select(headers_of_interest)
def all(self):
return self.where([(self._row + '/ID', 'LIKE', '%'), 'AND'])
pyxnat-0.9.0~dev0/pyxnat/core/select.py 0000664 0000000 0000000 00000034765 11632421554 0020146 0 ustar 00root root 0000000 0000000 import re
from . import schema
from .search import Search
from .resources import CObject, Project, Projects # imports used implicitly
from .uriutil import inv_translate_uri, check_entry
# from .uriutil import uri_last
from .errors import ProgrammingError
DEBUG = False
def is_type_level(element):
return element.strip('/') in schema.resources_types and \
not is_expand_level(element)
def is_singular_type_level(element):
return element.strip('/') in schema.resources_singular and \
not is_expand_level(element)
def is_expand_level(element):
return element.startswith('//') and \
element.strip('/') in schema.resources_types
def is_id_level(element):
return element is not None and \
element.strip('/') not in schema.resources_types
def is_wildid_level(element):
return element is not None and \
element.strip('/') not in schema.resources_types and \
('?' in element or '*' in element)
def expand_level(element, fullpath):
def find_paths(element, path=[]):
resources_dict = schema.resources_tree
element = element.strip('/')
paths = []
if path == []:
path = [element]
init_path = path[:]
for key in resources_dict.keys():
path = init_path[:]
if element in resources_dict[key]:
path.append(key)
look_again = find_paths(key, path)
if look_again != []:
paths.extend(look_again)
else:
path.reverse()
paths.append('/' + '/'.join(path))
return paths
absolute_paths = find_paths(element)
els = re.findall('/{1,2}.*?(?=/{1,2}|$)', fullpath)
index = els.index(element)
if index == 0:
return absolute_paths
else:
for i in range(1, 4):
if is_type_level(els[index - i]) or is_expand_level(els[index - i]):
parent_level = els[index - i]
break
if parent_level.strip('/') in schema.resources_singular:
parent_level += 's'
return [abspath.split(parent_level)[1]
for abspath in absolute_paths
if parent_level in abspath]
def mtransform(paths):
tpaths = []
for path in paths:
els = re.findall('/{1,2}.*?(?=/{1,2}|$)', path)
tels = []
ignore_path = False
for i, curr_el in enumerate(els):
if i + 1 < len(els):
next_el = els[i + 1]
else:
next_el = None
if is_type_level(curr_el):
if not is_id_level(next_el):
if not is_singular_type_level(curr_el):
tels.append(curr_el)
tels.append('/*')
else:
tels.append(curr_el + 's')
tels.append('/*')
else:
if not is_singular_type_level(curr_el):
if not is_wildid_level(next_el):
tels.append(curr_el.rstrip('s'))
else:
tels.append(curr_el)
else:
if not is_wildid_level(next_el):
tels.append(curr_el)
else:
tels.append(curr_el + 's')
elif is_expand_level(curr_el):
exp_paths = [''.join(els[:i] + [rel_path] + els[i + 1:])
for rel_path in expand_level(curr_el, path)
]
tpaths.extend(mtransform(exp_paths))
ignore_path = True
break
elif is_id_level(curr_el):
tels.append(curr_el)
else:
raise ProgrammingError('in %s' % path)
if not ignore_path:
tpaths.append(''.join(tels))
return tpaths
def group_paths(paths):
groups = {}
for path in paths:
resources = [el
for el in re.findall('/{1,2}.*?(?=/{1,2}|$)', path)
if el.strip('/') in schema.resources_types \
and el.strip('/') not in ['files', 'file']
]
if len(resources) == 1:
groups.setdefault(resources[0], set()).add(path)
continue
for alt_path in paths:
if alt_path.endswith(path):
alt_rsc = \
[el for el in re.findall('/{1,2}.*?(?=/{1,2}|$)',
alt_path
)
if el.strip('/') in schema.resources_types \
and el.strip('/') not in ['files', 'file']
]
if alt_rsc[-1].strip('/') in \
['files', 'file', 'resources', 'resource'] + \
schema.rest_translation.keys():
groups.setdefault(alt_rsc[-2] + alt_rsc[-1], set()
).add(alt_path)
else:
groups.setdefault(alt_rsc[-1], set()).add(alt_path)
return groups
def compute(path):
if not re.match('/project(s)?|//.+', path):
path = '/' + path
path = inv_translate_uri(path)
try:
groups = group_paths(mtransform([path]))
except:
raise ProgrammingError('in %s' % path)
best = []
for name in groups:
lightest = (0, None)
for path in groups[name]:
score = len(path.split('/'))
if lightest == (0, None) or lightest[0] > score:
lightest = (score, path)
best.append(lightest[1])
return best
class Select(object):
""" Data selection interface. Callable object that indicates the
data to be returned to the user.
Examples
--------
Select with a path:
>>> interface.select('/projects/myproj/subjects').get()
Select with a datatype:
>>> columns = ['xnat:subjectData/PROJECT',
'xnat:subjectData/SUBJECT_ID'
]
>>> criteria = [('xnat:subjectData/SUBJECT_ID', 'LIKE', '*'),
'AND'
]
>>> interface.select('xnat:subjectData', columns
).where(criteria)
"""
def __init__(self, interface):
"""
Parameters
----------
interface: :class:`Interface`
Main interface reference.
"""
self._intf = interface
def project(self, ID):
""" Access a particular project.
Parameters
----------
ID: string
ID of the project.
"""
self._intf._get_entry_point()
return globals()['Project'](
'%s/projects/%s' % (self._intf._entry, ID), self._intf)
def projects(self, id_filter='*'):
""" Returns the list of all visible projects for the server.
Parameters
----------
id_filter: string
Name pattern to filter the returned projects.
"""
self._intf._get_entry_point()
return globals()['Projects'](
'%s/projects' % self._intf._entry, self._intf, id_filter)
def experiments(self, project_id=None, subject_id=None, subject_label=None
, experiment_type='xnat:mrSessionData'
, columns=None
, constraints=None
):
""" Returns a list of all visible experiment IDs of the specified type,
filtered by optional constraints.
Parameters
----------
project_id: string
Name pattern to filter by project ID.
subject_id: string
Name pattern to filter by subject ID.
subject_label: string
Name pattern to filter by subject ID.
experiment_type: string
xsi path type must be a leaf session type. defaults to 'xnat:mrSessionData'
columns: List[string]
list of xsi paths for names of columns to return.
constraints: list[(tupple)]
List of tupples for comparison in the form (key, comparison, value)
valid comparisons are: =, <, <=,>,>=, LIKE
"""
if columns is None:
columns = []
where_clause = []
if project_id is not None:
where_clause.append(('%s/project' % experiment_type, "=", project_id))
if subject_id is not None:
where_clause.append(('xnat:subjectData/ID', "=", subject_id))
if subject_label is not None:
where_clause.append(('xnat:subjectData/LABEL', "=", subject_label))
if constraints is not None:
where_clause.extend(constraints)
if where_clause != []:
where_clause.append('AND')
if where_clause != []:
print where_clause
table = self.__call__(experiment_type, columns=columns).where(where_clause)
return table
else:
table = self.__call__(experiment_type, columns=columns)
return table.all()
pass
def scans(self, project_id=None, subject_id=None, subject_label=None,
experiment_id=None, experiment_label=None,
experiment_type='xnat:imageSessionData',
scan_type='xnat:imageScanData',
columns=None,
constraints=None
):
""" Returns a list of all visible scan IDs of the specified type,
filtered by optional constraints.
Parameters
----------
project_id: string
Name pattern to filter by project ID.
subject_id: string
Name pattern to filter by subject ID.
subject_label: string
Name pattern to filter by subject ID.
experiment_id: string
Name pattern to filter by experiment ID.
experiment_label: string
Name pattern to filter by experiment ID.
experiment_type: string
xsi path type; e.g. 'xnat:mrSessionData'
scan_type: string
xsi path type; e.g. 'xnat:mrScanData', etc.
constraints: dict
Dictionary of xsi_type (key--) and parameter (--value)
pairs by which to filter.
"""
if constraints is None:
constraints = {}
uri = '/data/experiments?xsiType=%s' % experiment_type
if project_id is not None:
uri += '&project=%s' % project_id
if subject_id is not None:
uri += '&subject_id=%s' % subject_id
if subject_label is not None:
uri += '&subject_label=%s' % subject_label
if experiment_id is not None:
uri += '&ID=%s' % experiment_id
if experiment_label is not None:
uri += '&label=%s' % experiment_label
uri += '&columns=ID,project,subject_id,%s/ID' % scan_type
if constraints != {}:
uri += ',' + ','.join(constraints.keys())
if columns is not None:
uri += ',' + ','.join(columns)
c = {}
[c.setdefault(key.lower(), value)
for key, value in constraints.items()
]
return JsonTable(self._intf._get_json(uri)).where(**c)
def tag(self, name):
self._intf._get_entry_point()
return self._intf.manage.tags.get(name).references()
def tags(self):
self._intf._get_entry_point()
return self._intf.manage.tags()
def __repr__(self):
return ''
def __call__(self, datatype_or_path, columns=[]):
""" Select clause to specify what type of data is to be returned.
Parameters
----------
datatype_or_path: string
Can either be a resource path or a datatype:
- when a path, REST resources are returned, the
`columns` argument is useless.
- when a datatype, a search Object is returned,
the `columns` argument has to be specified.
columns: list
List of fieldtypes e.g. xnat:subjectData/SUBJECT_ID
Datatype and columns are used to specify the search table
that has to be returned. Use the method `where` on the
`Search` object to trigger a search on the database.
"""
self._intf._get_entry_point()
if datatype_or_path.startswith('/tag'):
if len(datatype_or_path.split('/')) == 3:
return self.tag(datatype_or_path.split('/')[-1])
else:
return self.tags()
if datatype_or_path in ['/', '//', self._intf._entry]:
return self
if datatype_or_path.startswith(self._intf._entry):
datatype_or_path = datatype_or_path.split(
self._intf._entry, 1)[1]
if datatype_or_path.startswith('/'):
return_list = []
try:
for path in compute(datatype_or_path):
if DEBUG:
print 'path: %s' % path
pairs = zip(path.split('/')[1::2], path.split('/')[2::2])
# # in case a level id has a / - allowed for files only
# if len(path.split('/')[1:]) % 2 == 1 \
# and uri_last(path) not in schema.resources_types:
# pairs[-1] = (pairs[-1][0], uri_last(path))
obj = self
for resource, identifier in pairs:
if isinstance(obj, list):
obj = [getattr(sobj, resource)(identifier)
for sobj in obj]
else:
obj = getattr(obj, resource)(identifier)
return_list.append(obj)
if len(return_list) == 1:
return return_list[0]
else:
return CObject(return_list, self._intf)
except Exception, e:
if DEBUG:
print e
raise ProgrammingError('in %s' % datatype_or_path)
else:
if columns == []:
columns = self._intf.inspect.datatypes(datatype_or_path)
return Search(datatype_or_path, columns, self._intf)
pyxnat-0.9.0~dev0/pyxnat/core/tags.py 0000664 0000000 0000000 00000006706 11632421554 0017617 0 ustar 00root root 0000000 0000000 import os
import tempfile
from .jsonutil import JsonTable, csv_to_json
from .resources import CObject
class Tags(object):
""" Database tags sub-interface.
It is meant to be an attribute of the main Interface class.
"""
def __init__(self, interface):
self._intf = interface
self._meta_project = None
def __call__(self):
self._init()
return self._meta_project.subject(self._intf._user
).resource('tags').files().get()
def _init(self):
if self._meta_project is None:
self._meta_project = \
self._intf.select.project('metabase_%s' % self._intf._user)
if self._meta_project.accessibility() != 'private':
self._meta_project.set_accessibility('private')
if not self._meta_project.exists():
self._meta_project.create()
self._meta_project.set_accessibility('private')
def new(self, name):
return self.get(name).create()
def delete(self, name):
self.get(name).delete()
def get(self, name):
return Tag(name, self._intf)
def share(self, other_user):
if self._intf.select.project('metabase_%s' % other_user).exists():
self._meta_project.subject(self._intf._user
).share('metabase_%s' % other_user)
class Tag(object):
def __init__(self, name, interface):
self._name = name
self._intf = interface
self._intf.manage.tags._init()
self._file = self._intf.manage.tags._meta_project.subject(
self._intf._user).resource('tags').file(name)
def __repr__(self):
return ' %s'%self._name
def _read(self):
fd = open(self._file.get(), 'rb')
jtag = JsonTable(csv_to_json(fd.read()))
fd.close()
return jtag
def create(self):
if not self.exists():
jtag = JsonTable([])
tmp = tempfile.mkstemp()[1]
jtag.dump_csv(tmp)
self._file.put(tmp)
os.remove(tmp)
return self
def delete(self):
if self.exists():
self._file.delete()
def exists(self):
return self._file.exists()
def dereference(self, uri):
jtag = self._read()
tmp = tempfile.mkstemp()[1]
jtag.where_not(URI=uri).dump_csv(tmp)
self._file.put(tmp)
os.remove(tmp)
def dereference_many(self, uris=[]):
jtag = self._read()
for uri in uris:
jtag = jtag.where_not(URI=uri)
tmp = tempfile.mkstemp()[1]
jtag.dump_csv(tmp)
self._file.put(tmp)
os.remove(tmp)
def reference(self, uri):
uri = self._intf.select(uri)._uri
jtag = self._read()
if uri not in jtag.get('URI', always_list=True):
jtag.data.append({'URI':uri})
tmp = tempfile.mkstemp()[1]
jtag.dump_csv(tmp)
self._file.put(tmp)
os.remove(tmp)
def reference_many(self, uris=[]):
jtag = self._read()
for uri in uris:
jtag.data.append({'URI':uri})
tmp = tempfile.mkstemp()[1]
jtag.dump_csv(tmp)
self._file.put(tmp)
os.remove(tmp)
def references(self, show_uris=False):
jtag = self._read()
uris = jtag.get('URI', always_list=True)
if not show_uris:
return CObject(uris, self._intf)
return uris
pyxnat-0.9.0~dev0/pyxnat/core/uriutil.py 0000664 0000000 0000000 00000006637 11632421554 0020361 0 ustar 00root root 0000000 0000000 import os
import re
from .schema import rest_translation
# from .schema import resources_types
def translate_uri(uri):
segs = uri.split('/')
for key in rest_translation.keys():
if key in segs[-2:]:
uri = uri.replace(key, rest_translation[key])
return uri
def inv_translate_uri(uri):
inv_table = dict(zip(rest_translation.values(), rest_translation.keys()))
for key in inv_table.keys():
uri = uri.replace('/%s' % key, '/%s' % inv_table[key])
return uri
def join_uri(uri, *segments):
return '/'.join(uri.split('/') + \
[seg.lstrip('/') for seg in segments]).rstrip('/')
def uri_last(uri):
# return uri.split(uri_parent(uri))[1].strip('/')
return uri.split('/')[-1]
def uri_nextlast(uri):
# return uri_last(uri.split(uri_last(uri))[0].strip('/'))
return uri.split('/')[-2]
def uri_parent(uri):
# parent = uri
# if not os.path.split(uri)[1] in resources_types:
# while os.path.split(parent)[1] not in resources_types:
# parent = os.path.split(parent)[0]
# return parent
return uri_split(uri)[0]
def uri_grandparent(uri):
return uri_parent(uri_parent(uri))
def uri_split(uri):
return uri.rsplit('/', 1)
def uri_segment(uri, start=None, end=None):
if start is None and end is None:
return uri
elif start is None:
return '/'+'/'.join(uri.split('/')[:end])
elif end is None:
return '/'+'/'.join(uri.split('/')[start:])
else:
return '/'+'/'.join(uri.split('/')[start:end])
def uri_shape(uri):
kwid_map = dict(zip(uri.split('/')[1::2], uri.split('/')[2::2]))
shapes = {}
for kw in kwid_map:
seps = kwid_map[kw]
for char in re.findall('[a-zA-Z0-9]', seps):
seps = seps.replace(char, '')
chunks = []
for chunk in re.split('|'.join(seps), kwid_map[kw]):
try:
float(chunk)
chunk = '*'
except:
pass
chunks.append(chunk)
shapes[kw] = '?'.join(chunks)
return make_uri(shapes)
def make_uri(_dict):
uri = ''
kws = ['projects', 'subjects', 'experiments', 'assessors',
'reconstructions', 'scans', 'resources', 'in_resources',
'out_resources', 'files', 'in_files', 'out_files']
for kw in kws:
if _dict.has_key(kw):
uri += '/%s/%s' % (kw, _dict.get(kw))
return uri
def check_entry(func):
def inner(*args, **kwargs):
args[0]._intf._get_entry_point()
return func(*args, **kwargs)
return inner
def extract_uri(uri) :
"""
Destructure the given REST uri into project,subject and experiment.
Returns None if any one of project,subject or experiment is unspecified in the URI and a
(project,subject,experiment) triple otherwise.
"""
split = uri.split(os.sep)
# a well qualified uri has a project subject, and experiment name
# so when split the following items should be present:
# ['', 'data', 'projects', 'project-name', 'subjects', 'subject-name', 'experiments', 'experiment-name', 'scans']
# Based on the above comment if there aren't 9 items in the split list the uri isn't well qualified
if (len(split) != 9): return None
project = split[3]
subject = split[5]
experiment = split[7]
return (project,subject,experiment)
pyxnat-0.9.0~dev0/pyxnat/core/users.py 0000664 0000000 0000000 00000004620 11632421554 0020013 0 ustar 00root root 0000000 0000000 from .uriutil import check_entry
from .jsonutil import JsonTable
class Users(object):
""" Database user management interface. It is used to retrieve information
on users registered on a server.
.. note::
At the moment user creation and deletion is not supported through
the REST API but it will be at some point.
Examples
--------
>>> interface.manage.users()
['list_of_users']
>>> interface.manage.users.firstname('nosetests')
'nose'
See Also
--------
Project.users()
Project.add_user()
Project.remove_user()
"""
def __init__(self, interface):
"""
Parameters
----------
interface: :class:`Interface`
Main interface reference.
"""
self._intf = interface
def __call__(self):
""" Returns the list of all registered users on the server.
"""
self._intf._get_entry_point()
return JsonTable(self._intf._get_json('%s/users' % self._intf._entry)
).get('login', always_list=True)
def firstname(self, login):
""" Returns the firstname of the user.
"""
self._intf._get_entry_point()
return JsonTable(self._intf._get_json('%s/users' % self._intf._entry)
).where(login=login)['firstname']
def lastname(self, login):
""" Returns the lastname of the user.
"""
self._intf._get_entry_point()
return JsonTable(self._intf._get_json('%s/users' % self._intf._entry)
).where(login=login)['lastname']
def id(self, login):
""" Returns the id of the user.
"""
self._intf._get_entry_point()
return JsonTable(self._intf._get_json('%s/users' % self._intf._entry)
).where(login=login)['xdat_user_id']
def email(self, login):
""" Returns the email of the user.
"""
self._intf._get_entry_point()
return JsonTable(self._intf._get_json('%s/users' % self._intf._entry)
).where(login=login)['email']
def resources(self):
""" Returns the resources of the user.
"""
self._intf._get_entry_point()
print self._intf._get_json(
'%s/user/cache/resources' % self._intf._entry)
pyxnat-0.9.0~dev0/pyxnat/core/xpass.py 0000664 0000000 0000000 00000003560 11632421554 0020012 0 ustar 00root root 0000000 0000000 import os
from functools import partial
# default path to xnat pass file
def path():
return os.getenv('USERPROFILE') or os.getenv('HOME') + "/.xnatPass"
# str -> {'host': ..., 'u': ..., 'p': ..., 'port': ...} | None
def read_xnat_pass(f):
if os.path.exists(f) and os.path.isfile(f):
infile = open(f)
return parse_xnat_pass(infile.readlines())
else:
# raise IOError('XNAT Pass file :' + f + " does not exist")
return None
# [str] -> {'host': ..., 'u': ..., 'p': ..., 'port':...} | None
def parse_xnat_pass(lines):
empty = {'host': None, 'u': None, 'p': None}
line = find_plus_line(lines)
u = ('u',partial(find_token,'@'),True)
host = ('host',partial(find_token,'='),True)
p = ('p',partial(lambda x : (x,x)),True)
def update_state(x,k,state):
state[k] = x
return state
if line == None:
return None
else:
return chain([u,host,p],line,empty,update_state)
# [(str, str -> str, bool)] ->
# str ->
# dict
# dict -> dict
# dict | None
def chain(ops, initEnv, initState, update_statef):
env = initEnv
state = initState
for op in ops:
(k,_op,mandatory) = op
tmp = _op(env)
if tmp == None:
if mandatory:
return None
else:
(v,rest) = tmp
env = rest
state = update_statef(v,k,state)
return state
# [str] -> str | None
def find_plus_line(lines):
plusLines = filter (lambda x: x.startswith('+'), lines)
if len(plusLines) == 0:
return None
else:
return plusLines[0][1:]
# char -> str -> (str,str) | None
def find_token(tok,line):
splitString = map(lambda x: x.strip(), line.split(tok))
if len(splitString) == 0 or len(splitString) == 1 or splitString[0] == '':
return None
else:
return (splitString[0],splitString[1])
pyxnat-0.9.0~dev0/pyxnat/core/xpath_store.py 0000664 0000000 0000000 00000012657 11632421554 0021223 0 ustar 00root root 0000000 0000000 import re
from datetime import datetime
from os import path
from glob import iglob
from lxml import etree
from .jsonutil import JsonTable
def get_subject_id(location):
f = open(location, 'rb')
content = f.read()
f.close()
subject = re.findall('